Turn GLM-5.3-Flash into a Jev-like System One model

TL;DR: An ordinary LLM can make typed decisions in a single forward pass, with no retraining. In this post, we show how this works with GLM-5.3-Flash, hosted on Privatemode. On 28 public datasets, it matches the accuracy of TypeSafe's Jev and is faster when used from Europe (since Privatemode hosts in Europe), at 152 ms against 251 ms per decision from Germany; from the US, Jev is faster. Cost-wise it plays in a similar league as Jev, with about EUR 62 per million decisions. Jev in this benchmark costs EUR 16 per million decisions. Unlike Jev, it also reads images, works with other models, and is protected by confidential computing.

Why typed decisions

Much of what software asks a language model is a decision. "Which team should handle this ticket?", or "Does this contract clause belong in the liability section?". Source code needs one of a fixed set of answers. Structured output already guarantees that: the reply is valid JSON, and the value is one of the allowed ones. What it doesn't give you is how sure the model is, since the reply names one option and says nothing about the others. It is also slower than it needs to be: the model writes a whole JSON object, and a reasoning model may think for hundreds of tokens before that. For a decision in the middle of a request, speed and a probability for each option are what matter.

Models such as TypeSafe's Jev and Convai's Laya are built for exactly this. You pass in a piece of state and a set of named options, and you get back the chosen option together with a probability for each one. The name borrows from Daniel Kahneman's System 1, the fast and intuitive mode of thinking that answers without deliberating.

We like this approach. A fixed set of answers with a probability for each makes a language model feel much more deterministic than free text does, and it is what software needs. Customers have asked us whether they could have something like Jev on Privatemode, which is what this post is about.

An ordinary LLM can do the same, with no retraining. It comes down to how the model is queried, and we provide an open-source repository that does that part for you. We use GLM-5.3-Flash here; other models that return log probabilities for chosen tokens work the same way. Queried like this, the model answers each question in a single forward pass, which makes it an alternative to Jev. We run it on Privatemode, on hardware in Europe. In this post, we explain the technique, let you try it, and compare it with Jev on 29 public datasets for accuracy, latency, and cost. For accuracy, we also benchmark Laya.

How it works

For every position in a prompt, an LLM computes a probability distribution over its entire vocabulary. Text generation samples one token from that distribution, appends it, and repeats. The technique arranges the prompt so that the whole answer sits at one position, and then reads the distribution at that position instead of sampling from it. It takes three steps:

1. Number the options. The state, the question, and the options go into the prompt as JSON, with an index on every option. The instruction asks the model to answer with choice_index: followed by an index.
2. Prefill the answer. The assistant's turn already starts with choice_index:, and the model continues that turn instead of starting a new one. The next token it produces is therefore the index.
3. Restrict the vocabulary. vLLM's allowed_token_ids sets the probability of every other token to zero, so the model can only produce one of the option indexes. There is no malformed output to handle.

The response contains that single token and, via logprob_token_ids, the log probability of every option index. Normalized over the options, these give the probability of each answer.

Two further details matter in practice.

1. top_logprobs is not enough. It reports the distribution before the restriction is applied, so formatting tokens such as a leading space take up the top slots, and some options drop off the list and appear to have a probability of zero.
2. The token ids of the indexes depend on the model's tokenizer. Digits aren't always single tokens; GLM-5.3-Flash, for example, has a single token for 12. Rather than shipping a model-specific tokenizer, the library gets the token ids from the server, which keeps it simple to use with any model: sending a prompt to /completions with echo returns its exact tokenization by the model that is actually serving.

You can find our implementation in the below repository.

Try it

The playground below runs GLM-5.3-Flash modified with the above setup on Privatemode, directly from your browser. Pick one of the examples, among them a scanned invoice and a question that depends on your local time, or write your own questions and add images. Each answer comes back as a distribution over its options, typically within a few hundred milliseconds.

The distribution is often very useful, e.g. to decide whether to include a human-in-the-loop. The model solves most classic trick questions, but not all of them.

Benchmark results

Please find our source code for the benchmarks below; if you have suggestions to add, let us know via issues/PR.

We compared three systems on 29 public, labeled datasets: GLM-5.3-Flash hosted on Privatemode and queried with the technique above, TypeSafe's Jev, and Convai's Laya.

The datasets have between 2 and 151 options and cover English and German, intent routing, sentiment, topic classification, moderation, entailment, question answering, legal text, and scanned documents. All three systems receive the same state, the same option names in the same order, and the same instruction.

We ran each dataset twice. Even at temperature 0, both hosted services changed up to 3.5% of their answers between identical runs, so we treat smaller differences as noise.

We are one of the parties in this comparison, so everything behind it is public: the methodology, the dataset specifications, the test harness, and the aggregation. We ran Jev and Laya with their default settings and did not tune our prompt on these datasets. Our intent is in no way to make Jev or Laya look bad (actually not at all, both are very fascinating models and very much needed!), but to provide the reader with another option to set up generalistic classifiers. Since we are a company providing confidential computing services, our motivation lies mainly behind use cases with sensitive data.

Accuracy

On the 28 datasets that both hosted systems can answer, GLM-5.3-Flash and Jev are equally accurate. Each is more accurate on 10 datasets; on the remaining 8, the two are within one percentage point of each other. The median gap is 0.7 percentage points in Jev's favor, which is not statistically significant (p = 0.64). Laya, a model with 421 million parameters that we ran locally, scores lower than both on most datasets. Its median gap is 13 to 15 percentage points, which is statistically significant (p < 0.001). Laya's strengths lie elsewhere: it is small enough to run on a laptop and to fine-tune on your own labels. We used it as it comes, without fine-tuning.

The number of options has a larger effect on accuracy than the choice between Jev and GLM-5.3-Flash.

Across datasets, the number of options changes along with everything else about the task. Three datasets, however, label the same questions twice, once coarsely and once finely, so the task stays the same and only the number of options changes. On TREC, going from 6 to 42 options, Jev drops from 92.1% to 85.6%, GLM-5.3-Flash from 91.2% to 79.6%, and Laya from 88.4% to 51.2%. On MASSIVE, going from 18 scenarios to 59 intents raises the scores of Jev and GLM-5.3-Flash in both languages, while Laya's score drops. The number of options alone doesn't determine how hard a task is.

Latency and cost

We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model. The times are what a user sees: the model call, network included. From Germany, across all datasets, a typical decision takes 152 ms with GLM-5.3-Flash on Privatemode and 251 ms with Jev. Neither account was rate-limited during the runs.

Where you measure from matters. We ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode. Privatemode runs in Europe and Jev apparently in the US, so each is faster close to home.

On cost, Jev is cheaper. One million decisions cost about EUR 62 with GLM-5.3-Flash and about EUR 16 with Jev, at each service's list prices.

Most of the difference comes from the price per input token, and some from how each system packages a question. Jev adds roughly 270 tokens of fixed overhead and about 10 tokens per option. The GLM-5.3-Flash prompt adds about 55 tokens of fixed overhead and about 20 per option. Below about 21 options, it sends fewer tokens than Jev; above that, it sends more.

Laya runs locally, so there is no comparable latency or price per decision for it.

Multimodal decisions

The state doesn't have to be text. GLM-5.3-Flash is a vision-capable model, so a question can come with images, such as a scanned invoice, a photo of a damaged parcel, or a screenshot. The image goes into the same prompt, and the answer is still a single token with a probability for every option. The playground's Scanned document example shows this; you can also paste or drop in your own image.

According to its documentation, Jev works on text, and Laya is a text encoder, so neither takes images as input. On RVL-CDIP, a set of 1,600 scanned business documents in 16 classes, GLM-5.3-Flash reaches an accuracy of 70.2% and is the only one of the three that can answer. A document costs more than a sentence: the image adds about 1,350 input tokens, so a million document decisions cost about EUR 270.

Capabilities

With many options, each system hits a limit. Laya's option names share a budget of 192 tokens, which is enough for the 77 intents of banking77 but not for the 151 intents of CLINC150. Privatemode's deployment of GLM-5.3-Flash reports at most 128 entries in logprob_token_ids, while the mask in allowed_token_ids takes every option. So the library sends a question with 151 options twice, identically, and reads the probabilities of 128 options from the first response and of the other 23 from the second. At temperature 0 both run the same forward pass, so the merged result is the distribution a single request would return. On CLINC150, GLM-5.3-Flash reaches 87.5% this way, against 78.4% for Jev. The second request and the long option list cost time: a decision takes 719 ms, against 249 ms for Jev. The limit is then the model's 191 single-token indexes.

Some tasks only one or two of the systems can handle at all.

Further findings

Some of the remaining errors are in the labels. When most systems agree on an answer and the dataset's label disagrees, the label is often one of two defensible answers. banking77 has many such pairs, for example get_physical_card and order_physical_card, or declined_transfer and failed_transfer. About 17% of banking77's examples fall into this category, so the highest score any system could reach there is about 85% rather than 100%.

Reasoning helps, at a price. As a control, we let the same GLM-5.3-Flash reason before it answers, on all 29 datasets. It is more accurate in every band of option counts, from 89.9% against 85.5% with two options to 82.0% against 79.2% between 21 and 80. It also writes hundreds of tokens per decision instead of one and costs about EUR 350 per million decisions, against EUR 62. At the other end, plain embedding similarity, which picks the option closest to the text with no decision model at all, reaches between 45.9% and 72.8% depending on the band.

Renaming the options affects the systems differently. We re-ran every dataset with each option replaced by a synonym and nothing else changed. On boolq, where true and false became correct and wrong, GLM-5.3-Flash lost 20 points, while the other two lost less than three. Because this renaming also changes the meaning of the question, it doesn't isolate memorization. The results for every dataset are in the benchmark repository.

Build it yourself

The library is written in Python and works with any vLLM-backed endpoint; it relies on vLLM's extensions to the OpenAI API, such as allowed_token_ids and logprob_token_ids. Its README explains how to set it up with Privatemode, and an AGENTS.md file tells coding agents what an implementation has to get right. The benchmark repository contains the methodology, the dataset specifications, the test harness, the aggregation, and every raw run as a download, so every number in this post can be reproduced without running anything again. If you know a setting that serves any of the three systems better, find a mistake, or want to add a dataset or another system, we welcome pull requests.

With Privatemode, these decisions are protected by confidential computing: your data stays encrypted in memory even during processing, and the client verifies the deployment's attestation report before sending anything. You can read more on our security page.