9
The demo is the easiest meeting of the buying cycle. The vendor types a question into a clean window; the answer appears before the sentence finishes; the partners in the room nod. Then the tool lands on your desk, someone pastes in a ninety-page contract, and the tempo changes.
It was never a lie, exactly. Performance figures quoted for these systems are usually measured under the best possible conditions: an empty conversation, one question, nothing loaded. Speed falls as the conversation fills. The acceleration techniques that inflate a number on a fresh session pay their largest dividends when the window is empty and their smallest once a real matter sits inside it. A throughput claim quoted without its context depth, its before-and-after baseline, and the settings it was run under is not a measurement. It is a marketing asset.
Numbers like these travel because they are repeated, not because they are verified. Confidence outruns rigor in public: a figure posted early and stated plainly will out-circulate a careful correction every time, and a small cluster of accounts can amplify an unverified claim across an entire industry in an afternoon. Procurement committees absorb the figure and never ask what was in the window when it was taken.
Three questions separate a benchmark from a brochure:
- At what context depth was this measured? One fresh question tells you nothing about how the system behaves in hour four of a document review, when the case file, the correspondence and the prior drafts are all still loaded.
- What is the baseline on the same hardware and the same build? An improvement quoted without a before number is an assertion, not evidence — and the honest before-and-after is the only thing that tells you what you are actually buying.
- Which acceleration settings were enabled? Techniques that flatter a benchmark on an empty session often contribute far less once real context is present. Ask what happens with them off.
If a vendor cannot answer all three, treat the number as marketing until you have run it yourself.
A purchase qualified on a slideware number has a second life inside the firm. The tool arrives, it sags under real load, and the people who needed it quietly return to whatever they used before — frequently a public assistant on a personal account, with client material attached. A figure that never survived contact with your workload does not merely waste a budget line. It pushes the work straight back toward the shadow tools the engagement was meant to eliminate.
For a firm that cannot risk its privilege, the deeper point is this: the only test that means anything is run against your own matters, at your own volume, over hours — and that test cannot honestly be conducted on someone else’s servers without handing them the exact material you are trying to protect. Evaluation and deployment are the same decision. Test in the environment you intend to work in: your infrastructure, your documents, full context, sustained load, nobody watching.
The number that matters is not the one on the slide. It is how the system performs on your work, in your control, months later — and whether the people responsible for it can reproduce that result without a demonstration standing behind them. Firms that can answer that question do not buy AI on faith. They qualify it, they measure it, and they keep the measurement where the work is.
Five minutes. Zero risk. See exactly what your firm is exposing right now. The Confidential AI Exposure Checklist walks through where your data actually goes before a single tool is procured — take the checklist.
Nine lives. Zero leaks.