Fix tiers
Every fix falls in a cell: the recipe that made it (or Claude), the provider, and the kind of change. Each cell has a tier that decides what SHAPI does with a verified fix:
- REVIEW: the fix PR opens as a draft, for a person to check. Every cell starts here.
- AUTO: the fix PR opens ready for review. A cell earns it only on the benchmark: 5 instances fixed with no edit outside the predicted lines, and no instance in the run that the fix failed.
- ALERT: no fix PR. The change is listed on the Dashboard issue as a manual fix.
The tiers come from data, never from hand:
- Authentication and scope changes stay at REVIEW, whatever the benchmark says.
- Merge outcomes lower a tier. When three or more of your fix PRs in a cell were closed without merging, more than were merged, that cell drops one tier for your account: AUTO to REVIEW, REVIEW to ALERT.
- SHAPI can demote a cell, with a reason shown below. A demotion can only lower a tier.
Every fix PR shows its tier next to its size and confidence.
The registry
From the benchmark run on APIBreak-Replay v0, SHAPI 0.3.1, 2026-10-30: 0 cells at AUTO, 33 at REVIEW, 0 at ALERT.
| Recipe | Provider | Change | Tier | Benchmark | Why |
|---|---|---|---|---|---|
apply-fix | hubspot | field-rename | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
apply-fix | hubspot | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
apply-fix | meta-graph | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
apply-fix | quickbooks | version-sunset | REVIEW | 0 fixed, 2 not Instances❌ quickbooks-minorversion-below-75__hfscdogg__outgrow❌ quickbooks-minorversion-below-75__RCOLKITT__AccountingQB-MCP | 2 benchmark instances not fixed cleanly |
apply-fix | shopify | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
apply-fix | stripe | enum-added | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | anthropic | status-code | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | anthropic | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | google-gemini | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs | |
llm | hubspot | field-rename | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | hubspot | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | meta-graph | field-rename | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | meta-graph | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | openai | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | quickbooks | auth | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | quickbooks | field-rename | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | salesforce | auth | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | shopify | auth | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | shopify | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | slack | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | twilio | enum-added | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | twilio | field-rename | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | twilio | required-field | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | twilio | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | x | enum-added | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | x | scope-change | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | xero | enum-added | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | xero | required-field | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | xero | scope-change | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | xero | status-code | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
llm | xero | version-sunset | REVIEW | 0 fixed, 0 not | 0 of the 5 benchmark passes AUTO needs |
status-code-accept-set | github | status-code | REVIEW | 0 fixed, 2 not Instances❌ github-workflow-dispatch-204-to-200__vincentng295__gemini_fbchat❌ github-workflow-dispatch-204-to-200__pyzz58__gpt-recommendations | 2 benchmark instances not fixed cleanly |
The benchmark and its instances are in packages/bench; the registry is packages/core/tiers.json, rebuilt from the latest run.