In 2026 you don’t really write prompts anymore, right? You drop a document into the chat, state what you want in one sentence, and the model figures out the details. The first demo works, the stakeholders nod, and you move on to the next ticket.

Well, not so fast.

That is more or less what our work degraded to when we wired invoice extraction into our document pipeline: describe the fields, ask for JSON, let the model do the rest. This post is the story of how that theory died, what the autopsy reveals, and why the fix is, of all things, a carefully written prompt.

The Setup

We run a small NestJS service that accepts image and PDF uploads, rasterizes PDFs with pdftoppm at 150 DPI, and OCRs each page with Nvidia’s Nemotron Parse 2.0 on a local vLLM instance. Picture blocks get cropped and described by a skainet-hosted vision model, an Nvidia Omni Model. In the last stage, the service flattens all pages into a single text context and asks that same model to return the invoice details (sender, recipient, IBAN, totals, line items) as one JSON object. Text only at that point, no images.

The prompt doing the asking is six keys and one guard sentence:

You are a data extraction specialist. From the following document
context, extract the invoice details. Your response MUST be a single,
valid JSON object with exactly these keys:
- "sender": the entity sending the invoice.
- "recipient": the target of the invoice (the recipient).
- "iban": the bank routing number (IBAN).
- "total_amount": the invoice total.
- "net_amount": the invoice net amount.
- "components": the components that comprise the total, as a table
rendered as an array of {"label": ..., "amount": ...} objects.
Use null for any value that is not present in the document. Do not
invent values.

- DOCUMENT CONTEXT
{context}

Provide ONLY the JSON object and nothing else.
src/ocr/prompts.ts (initial version)

{context} is where the flattened document text goes.

We test it with a synthetic invoice: clean HTML rendering, one page, tidy fields. The model returns a perfect JSON object on the first try.

So we are done?

We are not done.

The Impasse

A real invoice, two pages, German, from a hardware store, returns {}. Not a malformed answer, not an error. An empty JSON object, twice in a row, because the code politely retries at double the token budget. Politeness does not help.

What follows is a long evening of probing the gateway directly, and the findings are worth listing, in the order they embarrass us.

First: response_format: {type: "json_object"} is not a polite hint. It is a grammar. The server masks the model’s token choices at every step so the output stays syntactically valid JSON, which means that as long as the model wants to think in prose, the grammar makes thinking illegal. Our extraction model, trying to reason under a JSON grammar, emits the literal string null and burns its entire token budget. finish_reason: "length". The grammar works perfectly. The model just has nothing legal left to say.

Fine. We turn JSON mode off and let it think. It thinks. And thinks. At an 8,192-token budget it produces 22,694 characters of deliberation and never even closes its think tag, let alone answers. Our favorite transcript contains 22 instances of “Wait” and 30 instances of “Actually,” as the model re-decides the same four questions in a loop: should the sender include the address, should the label be the article number or the full description, which of the three IBANs on the invoice is the IBAN, and what, philosophically, “comprises the total”.

A borrowed Qwen3.8-27B needs 16,000 tokens to produce 50,141 characters of the same stall. Different engine, same disease. So it’s not the Nvidia model, it’s something else. Inside that rumination even sits a complete, correct JSON object. The model understands the invoice the whole time. What it cannot resolve is us.

The Learnings

Constrained decoding and chain-of-thought exclude each other in the same token stream, under the initially adapted prompt, at least.

The model is never confused about the document. It reads a two-page German invoice, multiple IBANs, a cross-page carry-over subtotal, without a single comprehension error. Every ambiguity it deliberates on is one our prompt has created.

At temperature 0, a verification loop is deterministic. Once the “let me double-check” pattern fires, it regenerates itself forever. The original notebook this pipeline is ported from runs its reasoning stage at temperature 0.6 per the model card, and now we know why.

And the best one: fast modes hallucinate to fill schema slots. Asked for a per-item net amount that the invoice never prints, both no-think models happily compute one, dividing by 1.19, or sometimes 1.2, with the confidence of a drunk calculator. The thinking model is the only one that refuses. The bug is never the model; it is our schema demanding data that isn’t there.

We adapted the prompt to resolve the issues the models were thinking about the most: which address information to include. This made the models converge. Table 1 shows the three configurations that produce an answer at all. Qwen3.8-27B in thinking mode is absent for a reason: it is still busy producing 50,141 characters of not answering.

Nvidia Omni Model, no-think + JSON mode Qwen3.8-27B, no-think + JSON mode Nvidia Omni Model, thinking, JSON mode off
Latency / output tokens 4.9 s / 582 17.0 s / 915 105.1 s / 13,241
sender ✅ perfect ✅ perfect ✅ perfect
recipient phones ❌ missed both ⚠️ found 1 of 2 ✅ found both
iban ✅ all 3, as printed ⚠️ all 3, spaces stripped ✅ all 3, as printed
total_amount / net_amount ✅ correct, as floats ✅ correct, as floats ✅ correct, German strings
Per-component net_amount ❌ invented (÷1.19-ish, sometimes ÷1.2) ❌ invented (all 5 absent from the document) ✅ no invention (reuses the printed line totals)
label short article codes full multi-line descriptions full multi-line descriptions
number Pos 1-5 Pos 1-5 quantity with unit
No-amount carry-over row included, nulls omitted omitted
Table 1: extraction quality for fixed but unoptimized prompt

Read that table top to bottom and the pattern is the same in every row: the models understand the document just fine, and then answer a different question: ours. Three models, two interpretations of “label”, two of “number”, three phone-number policies, and two fast modes computing per-item net amounts the invoice never printed, each arriving at different wrong numbers.

The Fine-Tuning

The next round of prompt tuning follows one principle: prescribe, don’t describe. Every “usually”, every “often”, every “e.g.” is a fork where a model invents its own interpretation, and different models pick different forks. So the schema is pinned down to the type level: multi-line strings for sender and recipient, a list of IBANs with an explicit spacing rule, JSON numbers for amounts, an ISO currency code, and the line items as a markdown table that mirrors the invoice’s own columns instead of forcing a semantic net/total split the document doesn’t contain.

Plus one sentence that does more than everything else combined: reformatting a printed number is allowed, deriving an unprinted number is not.

Every rule removes one decision the models would otherwise make on their own. Deterministic behavior is what remains, or at least as deterministic as anything with 120 billion parameters gets. We don’t make the models smarter; we make the questions smaller.

The final prompt isn’t even large, just carefully worded:

You are a data extraction specialist. From the following document
context, extract the invoice details. Your response MUST be a single,
valid JSON object with exactly these keys:
- "sender": Details about the invoice sender. Usually the letterhead
information: name, address, telephone number(s), email. Format as
multi-line string.
- "recipient": Details about the invoice recipient. Usually the
letterhead information: name, address, telephone number(s), email.
Format as multi-line string
- "iban": List of bank routing number (IBAN) where to send the invoice
amount to. List all IBAN you can find. Reformat to the spaced
formatting: "DE89 3704 0044 0532 0130 00" even if printed without
spaces. List of strings.
- "total_amount": the invoice total after taxes.
- "net_amount": the invoice net amount before taxes.
- "components": the components that comprise the total, as a string
containing a markdown table structured like the original list. Often
contains label, number, net amount, total amount.
- "currency": the three-letter ISO currency code (e.g. 'EUR') of the
currency the invoice uses.

Use null for any value that is not present in the document. Do not
invent values. Reformatting a printed number into a float is allowed;
deriving a number that is not printed is not. All amounts should be
formatted as JSON numbers, e.g. 2419.46.

- DOCUMENT CONTEXT
{context}

Provide ONLY the JSON object and nothing else.

prompts/invoice-details.txt

After the fine-tuning, same invoice, same document context, the full matrix:

Nvidia Omni Model, thinking, JSON mode off Nvidia Omni Model, no-think + JSON mode Qwen3.8-27B, thinking (xhigh), JSON mode off Qwen3.8-27B, no-think + JSON mode
Latency / output tokens 121.4 s / 15,291 8.6 s / 1,036 ~3 min / 13,815 16.1 s / 993
All 7 keys, incl. currency ✅ ✅ ✅ ✅
sender/recipient as multi-line strings ✅ ✅ ✅ ✅
IBANs spaced, incl. the unspaced-printed one ✅ ✅ ✅ ✅
total_amount / net_amount as JSON numbers ✅ 2419.46 / 2033.16 ✅ 2419.46 / 2033.16 ❌ null / 2033.16 ✅ 2419.46 / 2033.16
currency “EUR” “EUR” “EUR” “EUR”
components: markdown table, original columns ✅ German formatting preserved ✅ German preserved, most complete table ⚠️ floats inside the table, German formatting lost ✅ German preserved, incl. carry-over row
Invented values none none none none
Customer phones ❌ missed ❌ missed ✅ found (only cell) ❌ missed
Table 2: extraction quality for fine-tuned prompt, same real invoice

All four cells return valid JSON with all seven keys, spaced IBANs, and "currency": "EUR". Nobody invents values anymore; the prompt fine-tuning closes that class of bug for both models in both modes. The differences that remain are honest misses, not different answers to the same question: Qwen’s thinking cell is the only one that finds the customer phone numbers, and the only one that loses the total.

The service now runs the second column of Table 2. End-to-end extraction goes from 221 seconds to 8.2 seconds on the same invoice, a factor of 27, and the output is typesafe, which is the part we care about: amounts are JSON numbers instead of sometimes-strings-with-German-decimal-commas, the IBAN field is a list, the sender is a string. The TypeScript side finally knows what it will be parsing before it parses it.

Conclusion

We went from an empty output after several minutes to a perfectly interpreted, guaranteed JSON formed output taking only 8.2 seconds just by fixing our prompt. Main learning: reasoning on temperature 0 with vague prompts is a recipe for disaster.

So, does the model “just know what you want” in 2026? It knows what your document says. That part is genuinely solved: our OCR pipeline hands it clean text and it reads it flawlessly. What it cannot know is what you meant. That part is still called a specification, it still has no autocomplete, and no amount of parameters will write it for you.