With the rules, models still score 76 to 87 of 87; 3 amended rules meant to fix the last 32 misses raised them to 40.
Can we fix wheel 002's 32 misses by amending 3 of its rules, without causing new ones on the same 10 models?
In wheel 002, 10 models chose button variants with the same 12 rules and scored 80 to 87 of 87. Their 32 misses, the answers that differed from the designer's, all came from 3 gaps in the rules. While those gaps remain, the rules cannot enter the design system as tested on 10 models.
In scope: wheel 001's 29 scenarios and correct answers, wheel 002's 10 models at the same settings, and the 12 rules, 3 of them amended.
Out of scope: wheel 002's other 2 levels, names only and human written documentation, which did not change; any other change to the rules; new scenarios.
The rules still work: with the amended rules, every model scored 76 to 87 of 87. But the amendments raised the misses from 32 to 40 of 870, where we expected fewer than 10.
One amended rule fixed all 22 misses it was aimed at, and the other 2 each improved one button and made others worse. The models followed the amended rules as written, and those rules said something the designer did not mean.
Based on: one component, 10 models, and the 3 amended rules tested together. It does not show how each amended rule works on its own.
'Revert to original' in an image editor toolbar, as seen by every model. The answers are Claude Opus 5.5's.
The taskContext: An image editor. The user has applied several crops and filters. In the toolbar is a button that removes all of them and shows the original photo again. The user can apply edits again afterwards. Button label: "Revert to original" Which variant should this button use?
ghost Under H10, an action that undoes the user's own edits, is rarely needed and cannot cause permanent loss is ghost, and H3 rules out destructive because the edits can be applied again.
Opus applied H10, the rule for undoing the user's own edits, and gave the designer's answer in 3 of 3 repeats.
destructive Reversibility comes first in the precedence order: reverting discards all crops and filters, and the only way back is to redo them by hand, which H3 says does not count as reversible, so H2 makes it destructive (H10's ghost does not apply because the edits are lost permanently).
Opus applied the amended undo rule and missed in 3 of 3 repeats. That rule says redoing work by hand does not count as undoing it. 5 other models did the same in at least 2 of their 3.
For each gap that wheel 002 found, we amended one rule, and we tested all 3 together.
| Amended rule | What changed |
|---|---|
| 1. Where 'optional' ends | We rewrote the last step in the order the rules are checked. A reversible action the user came to the view for, or may reasonably take there, is secondary, such as archive, export, or log out. Ghost is only for actions most users never take on that view. |
| 2. Which rule wins | We added to step 3 of the order: if 2 rules about the button's role fit, the more specific one wins. An equal alternative to a recommended option (H8) beats a bypass (H12), and a bypass beats a plain dismiss (H5). We also moved H8 into step 3 from step 4. |
| 3. What 'undone' means | We added to rule H3, the undo rule: an action is reversible only if the system can restore the previous state. Redoing the work by hand does not count. |
Amended rule 1 fixed all 22 misses on the 3 buttons it was aimed at. Each button got 30 answers: 10 models, 3 repeats each.
In this table and the next 2, green marks a result better than with the original rules, and red a worse one.
| Amended rule | Button | Misses with the original rules | Misses with the amended rules |
|---|---|---|---|
| 1 | 'Archive project' on a project settings page | 13 of 30 | 0 of 30 |
| 1 | 'Log out' in an account menu | 6 of 30 | 0 of 30 |
| 1 | 'Export CSV' above a data table | 3 of 30 | 0 of 30 |
| 1 | 'New invoice' in the header of an invoices list page | 0 of 30 | 2 of 30 |
| 2 | 'Not now' on a prompt to turn on notifications | 3 of 30 | 0 of 30 |
| 2 | 'Stay on Free' on a pricing screen | 4 of 30 | 12 of 30 |
| 3 | 'Discard changes' in a dialog about unsaved changes | 3 of 30 | 1 of 30 |
| 3 | 'Reset to defaults' on a profile settings page | 0 of 30 | 8 of 30 |
| 3 | 'Revert to original' in an image editor toolbar | 0 of 30 | 16 of 30 |
| 3 | 'Save changes' at the bottom of a profile settings page | 0 of 30 | 1 of 30 |
| All 29 scenarios | 32 of 870 | 40 of 870 |
The other 19 scenarios had no misses in either run. All 3 amended rules ran together, so we assign each miss to one of them by the rule the model cited.
Where misses rose, the models' reasons quote the amended rules. On 'Reset to defaults' on a profile settings page and 'Revert to original' in an image editor, 24 of 60 answers missed. All 24 say that redoing work by hand does not count. All 36 correct answers on those 2 buttons cite H10, the rule for undoing the user's own edits, instead.
The table counts the models whose most common answer, over 3 repeats, was destructive.
| Button | Designer's answer | Original rules | Amended rules |
|---|---|---|---|
| 'Discard changes' in a dialog about unsaved changes | destructive | 9 of 10 models | 10 of 10 models |
| 'Reset to defaults' on a profile settings page | ghost | 0 of 10 models | 2 of 10 models |
| 'Revert to original' in an image editor toolbar | ghost | 0 of 10 models | 6 of 10 models |
No single meaning of 'undone' fits all 3 of the designer's answers. All 3 buttons lose work that comes back only if the user redoes it by hand. The designer calls the first destructive and the other 2 ghost. The designer sees a difference, but it is not about undo.
Amended rule 2 had the same problem on 'Stay on Free' on a pricing screen. It works only if a model first decides that H8, the rule for an equal alternative, applies. H8's wording fits choices such as rejecting cookies: "a legally or ethically equal alternative". Fable, Opus, Sonnet, and GPT-5.6 cited H8 and chose secondary. GPT-6 Astra, both Grok models, and Gemini 3.1 Pro never cited it, and chose ghost in 11 of their 12 answers.
The models also agree with each other more than before. Taking each model's most common answer, all 10 agreed on 25 of 29 scenarios, up from 23.
A reason is one line the model writes with its answer. It shows the rule the model named, not everything it weighed.
The top-tier models are each company's flagship that our account can reach, plus Claude Opus 5.5. All scores are out of 87.
| Model | Tier | Original rules | Amended rules |
|---|---|---|---|
| Claude Fable 5.1 | flagship | 87 of 87 | 85 of 87 |
| Claude Opus 5.5 | large | 87 of 87 | 81 of 87 |
| GPT-6 Astra | flagship | 84 of 87 | 76 of 87 |
| Grok 4.7 | flagship | 83 of 87 | 79 of 87 |
| Gemini 3.1 Pro | top Pro model our account can reach | 82 of 87 | 82 of 87 |
| Claude Sonnet 5.5 | mid | 87 of 87 | 87 of 87 |
| Claude Haiku 4.5 | small | 84 of 87 | 86 of 87 |
| GPT-5.6 sol | previous flagship | 80 of 87 | 83 of 87 |
| Grok 4.3 | older | 82 of 87 | 85 of 87 |
| Gemini 3.8 Flash | small | 82 of 87 | 86 of 87 |
The top-tier models made 20 of the 24 misses on the 2 buttons that reset the user's edits. The other 5 had made 16 of the 22 misses that amended rule 1 fixed, so they gained most from it. A rule that says the wrong thing costs most on the models that follow rules most closely.
This does not show that top-tier models are worse at rules. They followed the amended rules more closely than the others.
A reworded pair is a second scenario that tests the same rule in different words. These went wrong, in our setup as well as in the answers:
Nothing changes in the system, and the 12 rules stay as they are. Amended rule 1 goes into the next run unchanged. Before that run, the designer needs to make 2 decisions. The first is what makes 'Discard changes' in a dialog destructive, but 'Reset to defaults' on a settings page ghost. The second is whether H8 covers 'Stay on Free' on a pricing screen.
Read the 12 rules↗This run has these limits:
| Setting | Value |
|---|---|
| Models | wheel 002's 10, from Anthropic, OpenAI, Google, and xAI, at the same settings, each through its own application programming interface (API) |
| Information levels | rules only: wheel 001's 12 rules, with 3 of them amended |
| Scenarios | wheel 001's 29, unchanged, with the same correct answers |
| Repeats | 3 per scenario per model, each a fresh conversation: 870 answers |
| Scoring | wheel 001's script, against the same correct answers |
| Dates | 8 October 2026 |
| Cost | about €2.70 in API credit, pay as you go, for all 10 runs, with no reruns: from €0.10 for Sonnet 5.5 to €0.54 for Gemini 3.1 Pro. We worked it out from the tokens each model used and each company's published prices |
We wrote the 3 amended rules, the expected result, and the success criteria before the first run, and changed nothing after.
Reproduce it