Generating questions is easy. Verifying them is the product.

Why is generation the wrong thing to optimise?

Because it is already solved, and it was never the hard part.

Ask any current model for a physics multiple-choice question on ray optics and you will get something that reads exactly like the real thing: correct register, plausible numbers, four options, a worked solution underneath. It will look right to a non-specialist, and it will look right to the person building the product.

The failure is not fluency. It is that the question quietly has two defensible answers, or none, or a distractor that is accidentally also correct under a reasonable reading. You cannot see this by looking at the question. You can only see it by solving it.

This is why demos of AI question generation are so much more convincing than the products. The demo shows you output. The product has to be right ten thousand times without anyone checking.

What does a verification gate look like?

A second, stronger model independently solves each question — it does not review it.

The distinction matters more than it sounds. If you hand a model a question and its proposed answer and ask 'is this correct?', you have created an agreement machine. Models are strongly biased toward confirming a supplied answer, and you will get a high pass rate that means nothing.

If instead you hand the model only the question and ask it to solve it, you get an independent signal. You compare the two answers. If they disagree, the question does not ship.

The gate is binary and it is not a quality score. Exactly one option must be provably correct. A question that is 'probably fine' is a question that will be screenshotted by a student and posted with the caption that your product is broken.

• Generate with a fast, cheap model — this stage is commoditised and there is no reason to pay for it.
• Verify with the strongest model you can afford. Verification output is tiny, so the stronger model costs almost nothing here.
• Fix or regenerate on disagreement, then re-verify. Do not hand-wave a near miss through.
• Track accept, fix and regenerate rates as the real quality metric of the pipeline.

Why is the cost asymmetry so favourable?

Because checking produces far fewer tokens than creating.

A generated question with a full worked solution and four distractors is a substantial piece of output. The verification pass produces an answer and a short justification. That asymmetry means you can afford to check with a materially better model than the one you generated with, and the total bill barely moves.

In practice this inverts the usual instinct. Teams reach for the strongest model at generation time, where it is expensive and least necessary, and skip verification entirely because it feels like overhead. The economics point the other way.

How do you stop an AI pipeline from bankrupting you?

Put a hard rupee or dollar meter in the loop, and a kill switch outside it.

Any system that calls a paid model in a loop can, given one bad retry condition, spend without limit. Estimating cost afterwards from token counts is not a control — it is a post-mortem.

A spend meter

A hard cap measured in currency, not tokens. The run halts the instant it is crossed, mid-batch, without negotiation.

A kill switch

A file or flag that causes every billable call to refuse. It should be usable by someone who cannot deploy code.

Measured cost, not estimated

Report the real spend of every run. An estimate that drifts from reality is worse than no estimate.

Does any of this generalise beyond exam questions?

Yes — the pattern applies wherever output has a checkable ground truth.

Exam questions are unusually well suited to this because correctness is decidable: the question either has exactly one right answer or it does not. But the same structure holds for generated code that must compile, extracted data that must reconcile against a source, calculations that must balance, and translations that must round-trip.

The general form is: generate cheaply, verify independently with something stronger, gate on a binary condition rather than a score, and cap the spend. Where the ground truth is genuinely unavailable — open-ended writing, subjective judgement — this pattern does not rescue you, and you should be suspicious of anyone claiming it does.

What does it look like when it works?

The output becomes boring, which is the point.

In blind testing, verified generated questions were rated on par with or better than questions written by human subject setters. Not because generation improved, but because the bad ones never reached the page.

This is the part that is difficult to demonstrate and easy to undervalue. The visible artefact of a verification gate is an absence: the questions that would have embarrassed you, which nobody ever sees, and which therefore never appear in a case study.