LLM vs. planner

Two official Swiss financial-planner certification cases, both with an official solution key – which means there is a ground truth, not an opinion. A practising planner solved them without AI tools. A purpose-built assistant solved the same cases in a single call, with no follow-up questions allowed. Neither saw the other's work.

Scoring used an LLM as judge, given the official key as reference, across five dimensions: factual accuracy, calculation correctness, completeness, recommendation quality and structural clarity. Every output was scored twice, independently. All twenty dimension pairs matched across both runs.

It is the subject. And the interesting finding is not that the assistant scored higher – with two cases and one planner, that is a probe, not evidence about machines and people.

The errors are what the study is about. On one case the model applied 2026 pension parameters to a 2024 situation: it took the current maximum disability pension from its knowledge base, used it in a case set two years earlier, and propagated that figure into the children's benefits. Everything about that answer looks competent. The structure is clean, the reasoning is explicit, the number is wrong.

That is the whole point, and it is why the assistant is deliberately unclever – no retrieval, no agent loop, one call. What carries it is the knowledge base: 62 structured documents from the official Swiss sources, one tax file per canton, all valid as of 1 January 2026, and a system prompt written against the failure modes documented in the literature. Get the ground truth right and the model is useful. Get it slightly wrong and it is confidently, invisibly wrong – which in financial planning is worse than useless.