New Claude models? Test them before you switch

Anthropic released Opus 5.5, Sonnet 5.5 and Haiku 5.5 this week for Claude. I usually just switch everything over on day one. This time, as I am trying to understand how agentic workflows can affect token use, I am happily going to follow my Scottish stereotype and I want to be 'tight' with my token use.

My creative bloke backend, that I use for a bunch of uses, including managing this site, has a spellchecking engine that I custom-made (I am especially proud of the 'DIMS' button, which checks to see if my usual garbled brain vomit is legible to another human). This is run through AI models, and I go for speed for the model choice.

Running multiple machines is great, but upkeep is a pain.

My house proofreader, the small service on my Mac Mini that checks my writing, asked for a model called “haiku”. That is a nickname, not a model. On my MacBook Pro it already meant Haiku 5.5. On the Mac Mini, which runs a slightly older Claude Code, it still meant Haiku 4.5. So the day the Mini updated itself, my proofreader would have changed model without anyone deciding it.

The fix is boring and important: in the script, name the model in full (claude-haiku-5-5), and have a test that refuses the nickname.

So I got Claude to run some tests.

Three tests, on real work

1. My internal proofreader: Haiku 4.5 against Haiku 5.5. The same planted mistakes and the same clean paragraphs, two runs each. Both caught 8 of 8 slips and invented nothing. Haiku 5.5 did it in 6.6 seconds against 17.1 = excellent and upgrade please to Haiku 5.5.

2. The search helpers: Haiku 5.5 against Sonnet 5.5. These are the small helpers Claude Code sends off to dig through my projects. I re-ran six real jobs from the last fortnight, word for word. Both got the main answer right on all six. Haiku was slower and read more, but each token costs a twentieth of Sonnet's, so a job came in at about a fourteenth of the cost. Haiku also found a real bug in one of my tools that Sonnet walked straight past. Thanks to the test, I am going to run with Haiku in the background, and hopefully I won't notice and also did I tell you I was Scottish.

3. The design critic: Sonnet 5.5 against Opus 5.5. I have a helper that gives an art director's crit of anything I am looking at (based on a lot of training of what I like when I am Art Directing, and this critical helper cuts a lot of wheat from the chaff and I can choose the number of options I want).

I gave both models four pages, one of them a copy of my proofreader page with six faults planted on purpose. Both found all six. Both gave the same pass or fail on every page, and named the same worst problem. Each caught a few things the other missed, which is a good argument for a second pair of eyes. So I need to make sure there is a degree of model flexibility on this process, which I will see if I can partially automate.

What actually saves tokens

What the test threw up though was that there is an even better way to economise on tokens. Every new message makes Claude re-read the whole conversation, and with the new million-token memory a chat could grow to nearly a million tokens before it tidied itself up. I now cap that at 300,000, and start a fresh chat with a short handoff note when the job changes.

So, from today:

• Chats tidy themselves up at 300,000 tokens.
• The search helpers run on Haiku 5.5.
• The design critic runs on Sonnet 5.5, with Opus called in for a final look when it matters.
• The main chat, where the planning and judgement happen, stays on Opus 5.5.

My biggest takeaway is never assume a new model is 'better' for you and your use case, (Gemini 3.1 Pro and Opus 5, I am looking at both of you when I say that). When a new model arrives test it against how you use A.I. as no one else uses it the way you do!

glossary:

Token = the unit an AI model reads and writes in, roughly three quarters of a word. It is what your plan or your bill is measured in.