Gemini 4 Hydrogen and Claude Sonnet 5.5: 57% and 70.6% are not the same paper at all

The 57% score Google gave Gemini 4 Hydrogen is a Terminal-Bench 4.1 score. Anthropic gave Claude Sonnet 5.5 a score of 70.6%, which is Terminal-Bench 4.0.

Two papers.

On the day of the release, many posts directly put the two numbers side by side and concluded that "Sonnet is ahead by 13 points." This conclusion is probably correct, but the number 13 itself is not valid. 4.1 is a new version of the question bank. The questions and scores have been changed. The scores of the same model in the two versions cannot be interchanged. To compare, you have to find the same paper.

The only ones I could find were third parties. Artificial Analysis ran it on 4.0 itself: Hydrogen was around 54% in both high and xhigh gears; Sonnet 5.5 was 63.6%. There's also a third-party comparison that rates Hydrogen at 53.5% on 4.0. The gap is nine to ten points, which is smaller than "13 points" and much calmer than the impression given by Google's "jump from 18.2% to 57%".

Incidentally, there’s also a 7-point difference between Anthropic’s self-reported 70.6% and third-party’s 63.6%. Both companies have the spirit of "testing themselves", but the direction is different: Google changed a paper that is updated to itself, and Anthropic uses its own test environment.

Where did the two models come from?

Claude Sonnet 5.5 was released on September 28, so it’s been a month and a half now. Its selling point was clear when it was released: the programming benchmark outperformed its own more expensive Opus 5.5 (official figures 70.6% vs. 66.4%), but the price was only half the price. The input was $2 and the output was $10 per million tokens. Like the previous generation Sonnet 5, the speed was more than 30% faster.

Gemini 4 Hydrogen was only released on November 12th, nominally succeeding Gemini 3.8 Flash. But its pricing is $1.50 and $7.50, which is exactly twice as expensive as the 3.8 Flash. Calling it a "Flash successor" is actually a bit misleading. It has already stood at the doorstep of Sonnet's price range.

Google’s abacus isn’t hard to understand either. The flagship Argon is on sale for $2/$10 right now, and will be on sale for $4/$20 starting December 1st. Hydrogen is stuck in one position: 25% cheaper than Argon’s discount price, 62.5% cheaper than Argon’s regular price, and it scored 51 points on the default Artificial Analysis comprehensive index, 1 point more than Argon’s 50 points. After December, Google hopes to move a large number of daily coding tasks that were originally running on Argon to Hydrogen.

The list price is 25% cheaper, the bill is not guaranteed

Just looking at list price, the Hydrogen is exactly three-quarters the price of the Sonnet.

However, the bulk of the bill for programming an agent is often not the output, but the large piece of context that is sent in repeatedly: system prompt words, tool definitions, code library fragments, and each round of dialogue must be sent again. The cached read price of Sonnet 5.5 is $0.20, which is equivalent to a discount of 10% off the original price.

Let’s do the math with a common shape: each call to a cached context with 100,000 tokens outputs 2,000 tokens. Sonnet 5.5 is $0.02 cached reads plus $0.02 output, for a total of about $0.04. If Hydrogen charges the 100,000 tokens at the original price of $1.5, the input alone is $0.15.

Of course, Gemini has context caching. Argon’s cached read price is $0.10, which is a 95% discount. If Hydrogen has the same discount, then 100,000 tokens will be about $0.0075, and the entire account will in turn be cheaper than Sonnet. The problem is that as of press time, I have not found Hydrogen’s cache unit price on Google’s public page, and several third-party reviews have not written it either.

So don't just look at the numbers 1.5 and 2 in this matter. Before going online, run on your own traffic for a day and speak with real bills.

Hydrogen’s Weakness: He speaks quickly and makes up quickly.

In a release, Google said Hydrogen had "significantly fewer false claims" compared to 3.8 Flash. It doesn’t look so good if you change the reference: According to an analysis and statistics, 8.5% of Hydrogen’s test questions contain fabricated content, and Argon’s is 4.3%, which is almost double.

Doubling is not a decimal.

For writing code, the manifestations of this problem are very specific: calling a library function that does not exist, giving a misremembered configuration item name, and quoting the old parameters of an API seriously. If the agent runs the test on its own, it can cover part of the problem, but if it is used to directly change the production configuration or generate documents for people to see, it will require an additional manual check.

Another problem is that it is expensive. For the team that switched from 3.8 Flash, the bill for the same call amount directly doubled; compared with OpenAI's GPT-6.1 Terra, it is also 50% more expensive. The original scenario where Flash was used as "cheap and high-volume" now has to be chosen again.

Sonnet’s weakness: 128,000 output ceiling, and top-end bill

The context of Sonnet 5.5 is 1 million, but the upper limit of a single output is only 128,000 tokens. The batch interface plus beta version request header can be put into 300,000.

Most coding tasks won't touch it.

What I encountered was another type of work: generating an entire set of test data at once, translating a long document in its entirety, and rewriting hundreds of files in batches. Hydrogen's single output can reach 1 million. Sonnet has to split the same task into several requests and write the splicing logic by itself.

The second shortcoming is hidden in the "effort" gear. Some calculations have found that when the effort of Sonnet 5.5 is turned to the maximum, the cost of completing a question actually exceeds that of Opus 5.5. The official promotion is "beat Opus at half the price", provided you don't turn the gears all the way. Many people habitually turn the reasoning setting to the maximum, and the logic of saving money is gone.

In terms of speed, Artificial Analysis measured Sonnet 5.5 at about 95 tokens per second in high mode. The independent speed test data of Hydrogen has not yet been released. Google only said that it is 2.5 times faster than 3.8 Flash. This is a relative number and cannot be converted into an absolute value.

Google’s placement for Hydrogen

Orchestrator, which launched on the same day as Hydrogen, says more about what Google thinks than the model itself.

In Antigravity, the main agent is played by Carbon (Pro users) or Argon (Ultra users), who is responsible for figuring out how to split the tasks, and then assign the split tasks to multiple Hydrogen sub-agents to do it. In other words, Google itself has no intention of letting Hydrogen handle complex tasks independently. Its design role is to be a "working hand."

The position of Sonnet 5.5 in Claude Code is exactly the opposite. Many people use it as the main workhorse alone, leaving all planning and execution to it.

One is the hand and the other is the brain.

This also explains why the gap in running scores is acceptable: Hydrogen does not need to win on Terminal-Bench, it only needs to complete each small piece quickly and cheaply on the premise that "others have already broken down the task."

There is a small pit to be reminded of. Some people in the community have reported that "discounted Pro" accounts such as trial periods and student discounts can see Argon in Antigravity but cannot use it. Which model Orchestrator's main agent uses depends on your subscription level. If you plan to save money by relying on it, first confirm which level your account can be adjusted to.

How would I allocate

When a single agent performs coding tasks from beginning to end, such as fixing a bug that spans several files or making up for testing on an old project, Sonnet 5.5 is more stable. The third-party 4.0 score is nine to ten points higher, and the hallucination problem has not been singled out for criticism. Just use high for effort, don't use max.

For the execution layer in the multi-agent pipeline, such as batch renaming, generating code according to templates, and running a large number of format conversions, Hydrogen's speed and price are more suitable, provided that there is a stronger model for planning above and testing below.

For long output tasks that require spitting out hundreds of thousands of tokens at a time, choose Hydrogen. Sonnet has a hard upper limit on this matter.

In a scenario where the prompt word is very long and needs to be sent in repeatedly every round, Sonnet's $0.2 cache read is the side that can be calculated clearly before Hydrogen announces the cache price.

Both can be tried on the web: Hydrogen can be adjusted directly in Google AI Studio, and Sonnet 5.5 can be selected in the Claude application and API console.

Related comparisons: "Claude Fable 5.2 and Gemini 4 Carbon: Carbon is four times cheaper, but each question in the low range is more expensive" | "Gemini 4 Carbon, Claude Opus 5.5, GPT-6 Astra: Carbon is one point ahead, most people can only look at the list this month" | "Muse Ember, Claude Haiku 5.5, GPT-6 Luna: The price is the same, but the difference in the account for each question is almost three times”