Every AI launch arrives with a chart showing the new model beating its rivals. Some of those charts are useful. Many are not about your work at all. This checklist is for anyone choosing a model for a product or a team: how to cut through launch claims, test what actually matters, and stay on the right side of UK data protection rules.
Step 1: Read launch claims like a sceptic
Before you run a single test, sort the claims you have seen into three piles.
- Company-reported results. These are real numbers, but the company chose which tests to show. Treat them as a starting point.
- Independent results. Groups such as Artificial Analysis and Epoch AI run their own tests across many models. They are more comparable, but still not your tasks.
- Anecdotes. Early testers' impressions can flag problems benchmarks miss, but one person's experience is not data.
November 2026 gave a clear example of why this matters. OpenAI's GPT-6.2 Astra rose 5 points on the Artificial Analysis Intelligence Index, yet was flat on the same group's coding agent index, and Epoch AI found it trailed a rival on hand-graded realistic tasks. One model, three different stories, all true. The only way to know which story applies to you is to test it yourself.
Step 2: Build a test set from your own work
The single most valuable thing you can do is collect 30 to 100 real examples of the task you want the AI to do, along with what a good answer looks like. Include:
- typical, everyday cases;
- awkward edge cases that trip up new staff;
- a few questions the model should refuse or say it cannot answer;
- for agents, tasks where something is deliberately missing, so you can see if the model reports it.
Strip out or replace personal data before you use real examples (see Step 6). Keep the test set fixed so you can rerun it on every new model and compare fairly.
Step 3: Measure accuracy and honesty
Score each answer as correct, partly correct or wrong. Then look separately at hallucination: answers that are confident but made up. A model that says "I don't know" when it should is often more valuable than one that is slightly more accurate but guesses.
Independent tests show why this deserves its own line. Artificial Analysis reported hallucination rates of 59% and 55% for OpenAI's two November 2026 models at maximum effort, on a test designed to see whether models guess when they don't know. Your rate on your own data may be very different, which is the point of measuring it.
For agents, add a check for silent failure. OpenAI disclosed that its new models were more likely than earlier ones to leave out obstacles that blocked their work. Plant a missing file or broken link in some test tasks and record whether the model tells you.
Step 4: Measure speed under real conditions
Latency is how long you wait for an answer. Measure two things: time until the first words appear, and time until the full answer is complete. Reasoning models that think harder can be much slower, so test at the effort level you would actually use. Run tests at different times of day, because response times can vary with demand.
Step 5: Work out the total cost per task
Price per million tokens is only part of the bill. What you really pay is the cost of a finished, correct task. Calculate:
- Tokens per task: input plus output, including any hidden reasoning tokens you are charged for.
- Retries and failures: a cheaper model that needs three attempts may cost more than a pricier one that succeeds first time.
- Caching: many providers charge much less for repeated input; OpenAI, for example, prices cached input for GPT-6.2 Astra at $0.40 per million tokens against $8 for fresh input.
- Human checking time: often the biggest cost of all.
- Currency: API prices are often quoted in US dollars, so budget for exchange-rate movement.
Step 6: Check data protection and residency
If any personal data will pass through the model, UK GDPR applies. The Information Commissioner's Office (ICO) publishes detailed guidance on AI and data protection, covering accountability and governance, transparency, lawfulness, accuracy, fairness, security and data minimisation, and individual rights. It includes an AI and data protection risk toolkit. Note that the ICO says the guidance is under review following the Data (Use and Access) Act, so check for updates.
Practical questions to put to any provider:
- Where is our data processed and stored, and can we choose a UK or European region?
- Is our data used to train your models, and can we opt out?
- How long are prompts and outputs kept, and who can see them?
- Do we need a Data Protection Impact Assessment (DPIA)? The ICO's guidance covers what to consider.
For very sensitive data, consider models that can run on your own infrastructure. Some research tools, such as the open-source OpenScientist system published in Nature Medicine, are designed to run fully offline with open-source models for exactly this reason.
Step 7: Test security and safety
If the model will read emails, documents or web pages, test it with content containing hidden instructions (prompt injection). If it will take actions, follow the National Cyber Security Centre's advice on agentic AI: sandbox it, limit its permissions, log its actions and make sure you can shut it down. Our guide to AI agents in practice covers this in more depth.
Step 8: Plan for change
Models are updated every few weeks. Prices change, behaviour shifts, older versions are retired. Before you commit:
- keep your test set and rerun it on every new version;
- avoid locking your product so tightly to one provider that switching is impossible;
- agree who signs off a model change, and on what evidence.
The one-page checklist
- Sort claims into company, independent and anecdotal.
- Build a fixed test set of 30 to 100 real tasks.
- Score accuracy, hallucination and silent failures separately.
- Measure time to first word and time to finish.
- Calculate cost per correct task, including human review.
- Check UK GDPR, data location, retention and training use.
- Test for prompt injection and limit what agents can do.
- Rerun tests whenever the model changes.
The UK angle
The UK has no AI-specific safety law yet, though a Commons committee has called for one. In the meantime, existing rules, especially UK GDPR enforced by the ICO, already apply to how you use AI with personal data. The AI Security Institute tests frontier models for national security risks, but that is no substitute for your own evaluation of whether a model is right for your customers, your staff and your data.
FAQs
Can I trust AI benchmark scores?
Use them as a starting point only. Company-reported scores are chosen by the company, independent scores are more comparable, and neither reflects your exact tasks. Build your own test set and rerun it on each model you consider.
Does UK GDPR apply to using AI tools?
Yes, whenever personal data is processed. The ICO publishes guidance on AI and data protection, including a risk toolkit, though it says the guidance is under review following the Data (Use and Access) Act.
What is the best way to compare AI model costs?
Compare the cost of a finished, correct task rather than the price per token. Include retries, cached input discounts, hidden reasoning tokens, currency changes and the time people spend checking results.
Sources
- Artificial Analysis: GPT-6.2 Astra and Sol results
- OpenAI: Introducing GPT-6.2 Astra and Sol
- The Next Web: OpenAI launches GPT-6.2 Astra and Sol, and early testers push back
- ICO: Guidance on AI and data protection
- NCSC: Managing the cyber risk of agentic AI
- UK AI Security Institute
Follow AI Decoded on Substack
Weekly plain-English AI news and verified UK events.
Subscribe