April 30, 2026 · TrueLLMs

How Logprobs Quietly Reveal Model Identity

A 30-line tokenizer-boundary test will tell you whether your proxy is really running GPT-5 or quietly redirecting to a cheaper model.

Of TrueLLMs' twelve detection dimensions, exactly one carries 20% of the weight on its own: logprobs. Every other dimension is built to cover the case where logprobs are unavailable. This post is about why.

What logprobs actually return

Set logprobs: true, top_logprobs: 5 on a chat-completions call. For every position in the response, the API returns the chosen token plus the five most-likely alternatives, each with their log probability. The chosen token is boring. The five alternatives are diagnostic.

Different model families have different vocabularies. GPT-4o and GPT-5 use o200k_base — about 200,000 BPE merges. GPT-3.5 and GPT-4 use cl100k_base — about 100,000 merges. Claude has its own vocabulary, not exposed by name. Gemini has another. Each of these vocabularies splits unusual characters at slightly different boundaries.

A 30-line test that distinguishes them

Send a stable prompt that triggers a known unicode-heavy completion (a Chinese place name, a JSON blob with emoji keys). Collect top_logprobs at every position. Look for the alternative-token strings.

See " the" with a leading space and " The" as alternatives? cl100k_base or o200k_base — OpenAI family.
See multi-byte fragments split into 3-byte chunks? Older cl100k_base.
See native Chinese characters as single tokens (e.g. "北京")? Probably o200k_base or a Chinese-tuned vocab.
See alternatives without leading-space tokens at all? Probably Claude.

Why proxies sometimes strip logprobs

There are three real reasons:

The upstream model genuinely does not expose them.
Logprobs increase response size and cost. A cost-conscious gateway strips them by default.
Returning logprobs would make a substitution detectable. Stripping them is the cheapest way to defeat half the published fingerprint techniques.

From the auditor's seat, you cannot tell the three apart. So TrueLLMs treats logprobs unavailability as a yellow flag, not a verdict — but caps the final confidence at 70 if logprobs are stripped. Other dimensions can still push the verdict to likely-substituted, but they cannot push it to confirmed.

Try it yourself

TrueLLMs runs this exact test. Open the auditor, paste your Base URL and key, and click Quick. The Logprobs Fingerprint card will tell you the inferred tokenizer family and whether it matches the claimed model.

See /dimensions/logprobs-fingerprint for the full algorithm and threshold table.