The wrong opponent

TypeSafe released Jev a week ago and the two numbers on the homepage are 193.6x faster and 444.6x cheaper. I believe both of them, roughly, in the way you believe a number that has been rounded in a friendly direction. What I could not work out from the homepage was faster and cheaper than what. Neither figure names an opponent. So I went and read the benchmark they come from, did the arithmetic on the published cells, and then ran the comparison that is missing from it on my own laptop.

The short version: the two headlines are measured against two different models, one of which is more accurate than Jev. The comparison set contains sixteen configurations and every single one is a frontier LLM. And when somebody finally did run a small model against Jev on a neutral harness, the small model landed inside Jev's accuracy band, led it on the composite score, and the cost gap collapsed from four hundred and forty five to about six.

What the headline is measured against

TypeSafe publishes its benchmark, which is more than most vendors do. Four workflows, 711 cases, seventeen configurations, and a page of self-declared limitations. Go one level down into the methodology and you find this:

For this eval, the reference labels are generated via an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking, answering every question in the harness.

There is no ground truth. "Accuracy" means agreement with what two frontier models said the answer was. TypeSafe is explicit about why, and the reasoning is defensible: "Instead of debating the correctness of the harness and labels, we assume that the code is correct, and measure against the current smartest large models." If your actual question is "can I swap out my Opus call for this," then agreement with Opus is the metric you want. It is a migration-risk number, and it is honest about being one.1

The problem is who else is on the chart. I counted the opponents: sixteen configurations, all of them frontier LLMs, run either through TypeSafe's own wrapper or as one shot prompts. No encoder. No BERT-family model. No fitted classifier. No small instruction model. Nothing that a team with labelled data would actually reach for.

Which is odd, because TypeSafe's own cookbook tells you to add TF-IDF and embedding baselines before trusting a result. The advice is right. It just was not taken on the launch benchmark.

Two headlines, two opponents

Jev's line on the aggregate chart is 67.8% agreement at $0.0004 and 0.4 seconds per case. Those are rounded, which turns out to matter, because you can use the rounding to work out which opponent each headline is talking about.

Take the cost claim. 444.6x means Jev's true mean cost is the opponent's cost divided by 444.6. Rounding bounds Jev to somewhere in $0.000350 to $0.000450. Of the sixteen opponents, exactly one lands inside that window: Opus 5 on the workflow route, at $0.17605, which implies $0.000396. The next nearest candidate implies $0.000451 and falls outside. So the cheaper-by-444.6x number is against Opus 5.

Now the speed claim. 193.6x against a 0.4 second mean bounds Jev to 0.375 to 0.475 seconds, and only two opponents have latencies consistent with that: Sonnet 5 on the workflow route at 78.1 seconds, or DeepSeek v4 pro at 86.5. It cannot be Opus 5, which works out to 88.9x on time, not 193.6x.

Nothing here is fabricated and I want to be careful about that. Every number I just used is TypeSafe's own, and they say out loud that "we expect that these are on the higher end of real world gains." The charge is not that the figures are false. It is that a multiplier without a named opponent is not a measurement, and that these two were picked from opposite ends of a table.

Watch how far the number moves if you slide the opponent along that same table, using the same published cells:

Two orders of magnitude, same model, same benchmark, same day. The green row is the opponent closest to Jev on accuracy, which is the comparison a buyer actually wants. The amber row is the one that produces the biggest number. Pick your opponent and you pick your headline.

One more thing about that benchmark: 691 of the 711 cases are not published, the harness is not released, and there are no token counts, no hardware, no batch sizes, no seeds and no error bars. Every cost cell carries "basis": "estimated_uncached" against a pricing revision hash with no price table behind it. So the cost axis is a model nobody outside the company can recompute. TypeSafe also notes that their timings are "generally run from our laptops on the West Coast."

The opponent nobody ran

Someone did run the missing comparison. JevBench is an MIT-licensed harness by Benchmark Heaven, unaffiliated with TypeSafe, and version 1.3.0 scores 534 frozen typed decisions across four axes at equal weight: intelligence, calibration, speed and cost.2

Jev takes first place overall at 74.4. That is a real result and it is the headline of the leaderboard. But look at the intelligence column on its own and the top row is not Jev. It is GPT-5.6 Luna at low reasoning effort, scoring 95.3 against Jev's 85.7. The benchmark's author puts it more plainly than I would:

An ordinary small instruction model is not out of its depth here. GPT-5.6 Luna (low reasoning effort) reaches 97.1%. What the purpose-built models buy you shows up in the other four columns, not in accuracy.

Luna finishes fourteenth overall for one reason: it costs $0.2419 per thousand decisions against Jev's $0.0399. That is a 6.1x gap. Not 445x. Six.

To be fair to Jev on the raw accuracy figure, the author's own instruction is to "treat the first few rows as a group, not a podium," because Luna's 97.1% and Jev's 96.3% have overlapping confidence intervals. Read it that way and the claim is not that Luna wins, it is that a cheap general model is already inside the same band. On the composite intelligence score across 534 decisions the gap is wider and the ordering is not in doubt.

That is the whole argument of this piece in one comparison. Jev is genuinely fast and genuinely cheap and the parallelism is genuinely the product, and none of that requires a 445x number. Against an opponent that a sensible engineer would actually consider, the advantage is single digit on cost, and on the composite intelligence score Jev is the one behind.

I ran the frozen-task baseline myself

There is a third regime nobody's leaderboard covers: you have one task, it is not changing, and you own labels for it. That is most classification work in production. For that case the opponent is a small supervised encoder, and I wanted a number I had measured rather than read.

So I pulled com-kotobalabs/open-jev-deberta-v3-large, an open reproduction of Jev's shape (one state, many typed questions, one forward pass) built by an unaffiliated team on public gold labels. Its card is scrupulous about not being Jev and about its numbers not being comparable to TypeSafe's. It ran on an M1 Pro, torch 2.14.

First I reproduced the card's own worked example. The chosen option came back identical. The three probabilities drifted a little: confidence 0.701 against their 0.733, score 0.819 against 0.846, noul 0.941 against 0.892. Close enough to call the artifact loadable and working.

Then the real test, banking77 intent classification, 77 options, all 3,076 test examples, against gold labels rather than consensus labels. I ran it twice, because partway through I discovered that how you spell the option strings is not a cosmetic choice. The next section explains why. The results first:

Condition A is the faithful reproduction, and their headline holds. I get 0.8992 (2766/3076), 95% interval [0.888, 0.909], against the card's claimed 0.916. That reads like a 1.7-point shortfall until you look at how their test set is built. They shuffle each source and take a slice, and with 1,500 test states spread across three datasets the banking77 intent figure rests on roughly 500 examples rather than the 1,500 the card implies. At that size a 0.916 carries an interval of about [0.888, 0.937], which contains my number without strain (two-proportion z = 1.17, p = 0.24). So this is not a discrepancy. It is the same result on six times the data, and I would rather say so than manufacture a gap out of a sample-size difference.

Condition B is where it gets interesting, and not in the way I expected. Same 77 labels, same meanings, underscores left in, context raised until they fit. Accuracy holds up: 0.8901, statistically indistinguishable from condition A (z = 1.16, p = 0.24). I had assumed that was the whole story, that formatting costs you feasibility and nothing else. Then I looked at the calibration column.

Expected calibration error goes from 0.011 to 0.089, 7.8 times worse. Mean confidence falls to 0.801 while accuracy stays near 0.89, so the model is running about 9 points under-confident instead of 1. Latency also goes from 354 ms to 657 ms, which is just the longer sequence. The accuracy is fine and the number you would threshold on is wrecked.

I want to state that result narrowly, because condition B changes two things at once: the spelling and the sequence length.3 The mechanism is almost certainly the temperature. This model ships a single post-hoc temperature of 1.05, fitted on a validation split in the format its authors used. Feed it a distribution that split did not contain and the scaling is no longer the right scaling. So the honest claim is not "underscores are bad". It is that a confidence number carries an unstated dependency on the input format it was calibrated against, and changing that format silently invalidates it.

Either way it matters more than a point of accuracy, because calibrated confidence is the entire reason to buy a typed-decision model instead of asking an LLM to emit JSON. A silent formatting choice in glue code left the headline metric intact and quietly degraded the one you would actually build a review queue on. Nobody would catch that from an accuracy dashboard.

The trap that explains 51.7%

The two rows in that table are the same 77 labels with the same meanings. The only difference is whether the underscores in the dataset's own label_text field are replaced with spaces. That is a formatting decision in glue code, the kind nobody writes down.

It costs 35% of the context budget. With spaces, the 77-option question encodes to 348 to 423 tokens across the test set. With the raw underscored field, it encodes to 531 to 606. The shipped configuration allows 512.

options as "activate my card"   median 356   over 512:     0 / 3076
options as activate_my_card     median 539   over 512:  3076 / 3076

Pass the dataset field straight through, which is the obvious thing to do, and every single input raises ValueError before the model runs. Replace one character and it all fits with room to spare. Nothing in the model card mentions this, and the naive path is the broken one.

This is not a footnote about my afternoon. It is the explanation for the most-quoted number in the entire Jev debate. On JevBench v1 that same open encoder scored 51.7% against Jev's 96.3%, and in v1.3.0 it ranks 42nd of 44. Everyone cites it as proof that small encoders cannot do typed decisions. Here is the benchmark author, who deserves credit for saying so in the results themselves:

open-jev-deberta-v3-large (local CPU) did not see the whole question 80 times out of 242. Its encoder window is 256 tokens and several routing requests are longer than that. Its 51.7% is a context limit as much as a judgement one, worth knowing before anyone reads it as "small encoders cannot do this."

Both numbers in that paragraph are right, and they are different budgets: the shipped config allows 512 tokens in total and caps the state at 256, so a long document gets cut at 256 while a long question set runs out of room at 512. My banking77 question hit the second limit. JevBench's routing requests hit the first.

A third of that benchmark never reached the model. And when I diffed the snapshot on the model hub against the project's own repository, the repository had already fixed it: the newer encoder computes the question budget first and clips the state to fit, instead of failing after the fact. The published artifact still carries the 18 September version. The fix landed upstream at 09:08 UTC on the day I wrote this, so it is real, it is hours old, and it is not yet in the file you download from the hub. I am not scoring points off that gap. I am pointing out that JevBench ran on 19 September, when the fix did not exist at all, and that its 51.7% is therefore a measurement of the truncating encoder.

So the case against small encoders here rests substantially on a truncation bug in a wrapper, not on a finding about encoders.

What I would actually do

Three regimes, three different right answers, and the leaderboards only cover one of them.

Your task is fixed and you have labels. Train the encoder. On banking77 a 22M-parameter supervised encoder gets 93.2% at 8 ms on CPU against Jev's 80.1% zero shot, which is somebody else's measurement, not mine. There is no contest and there should not be, because one of them was shown the answers. Self-hosting has a floor though: the open model only beats Jev's list price above roughly 14% GPU duty cycle, which is about 188 million decisions a month. Below that, paying for the API is the cheaper engineering decision, and a network round trip of 69 to 105 ms erases the compute advantage anyway.

Your questions change at runtime. This is the case Jev is actually built for, and the case the encoder loses badly. An honest critic on Hacker News put the steelman better than the marketing did: you are competitive with Jev "only if you fine tune on the train dataset and calibrate per question," whereas TypeSafe has "an API model that works on any problem without fine tuning, and that is the key." The open reproduction concedes the same thing in its own numbers, 0.854 in domain falling to 0.690 once the instructions and option sets change, and it leads its limitations with "It reads the question only partly."

You need many judgements on one document. Buy Jev, and this is where it stops being a benchmark argument. An independent pre-registered evaluation found that going from one question to twelve costs minus nine milliseconds, and that 800 typed judgements on a single state came back in 985 ms for $0.000747. The same evaluation registered 50 predictions before collecting data, and reports that 18 of its 21 falsified predictions missed in the same direction: they kept predicting failure and kept being wrong.

That same evaluation is also where the sharpest criticism comes from, and it is about the number people will most want to use. Jev's confidence ranks well and calibrates badly: ECE 0.051, AUROC 0.837. Use it to sort, not to threshold. It called a cake recipe a technical issue at 0.94 confidence and flagged zero of thirty out-of-scope messages. TypeSafe's own jaggedness page says much the same in blunter language, including "jev-1.13 does not count reliably" and "Jev suffers from context rot." A vendor publishing a page of its model's failure modes is worth more than a 445x on a homepage.