AI Quality Engineering
AI is a capability here, not a slogan
Most QA vendors have recently added the word AI to their homepage. Here is the specific, unglamorous version: where we let models into our workflow, where we refuse to, and how we test products that have AI inside them.
The rule we do not bend
AI proposes, generates and analyses. Controlled automation and human review decide whether a result is trusted. We will never build a system where a model can silently turn a failing test into a pass.
This matters more than it sounds. The failure mode of AI in testing is not a wrong answer — it is a confident, plausible green tick over a real defect. Every place we use a model, the decision that gates your release stays deterministic, reviewable and owned by a person.
The position comes from building it rather than reading about it: AI-assisted automation solutions on LLM technologies in a production engineering context, plus internal tooling — including a complete Python framework — generated with AI assistance and kept under human review. Doing that work is precisely what makes the limits obvious. More on the background.
Part one
AI inside our delivery
This is how we build more coverage in less time than a traditional QA engagement. You never see it directly — you see it as velocity. Every item below has a human review gate.
Requirements → test candidates
A model drafts scenarios from a user story or spec. An engineer reviews, cuts and rewrites before anything is committed. It removes the blank page, not the judgement.
OpenAPI → API test skeletons
Endpoint definitions generate the structural tests — status codes, schema shape, required fields — so engineer time goes to business logic and negative cases.
Locator discovery & repair
When a UI change breaks selectors, we get candidate replacements ranked by stability instead of hand-searching the DOM.
Failure triage at scale
A few hundred failures across a nightly run get clustered by probable cause, so the morning starts with five groups rather than five hundred lines.
Synthetic test data
Realistic, non-sensitive datasets covering edge cases — unusual names, currencies, time zones, boundary values — without touching customer records.
Change-aware prioritisation
Given a diff, run the tests most likely to be affected first, so the useful signal arrives in minutes instead of at the end of the suite.
Flakiness classification
Historical results are analysed to separate genuine intermittent bugs from timing issues and shared-state collisions.
Draft defect reports
Reproduction steps, environment details and trace context assembled automatically. A human confirms severity and sends it.
Part two
Testing products that have AI in them
A specialised service, and a genuinely different discipline. Conventional test automation assumes the same input produces the same output. LLM features break that assumption, so the entire approach has to change — from assertions to evaluation.
Output correctness & regression
Golden datasets with graded expectations, so you can tell whether a prompt change, model upgrade or retrieval tweak made things better or quietly worse.
RAG evaluation
Retrieval quality measured separately from generation quality. Groundedness, citation accuracy, and behaviour when the correct answer genuinely is not in the corpus.
Hallucination & refusal testing
Does it invent facts under pressure? Does it refuse things it should answer? Both directions are failures, and both need measuring.
Prompt-injection & jailbreak resistance
Adversarial inputs through every channel the model can read — user input, retrieved documents, tool output, file uploads.
Tool use & agent behaviour
Does the agent call the right tool with the right arguments, handle tool failure, recognise when to stop, and avoid loops that burn money?
Latency & cost measurement
Tokens and response time per user journey, tracked across releases. AI features fail commercially long before they fail functionally.
Guardrail verification
PII handling, content filtering and policy boundaries tested as explicit requirements rather than assumed from the system prompt.
Non-determinism strategy
Fixed seeds and temperature where possible; statistical thresholds over repeated runs where not. A flaky AI test still has to mean something.
Why it needs a different approach
Conventional testing versus AI evaluation
| Dimension | Conventional feature | AI feature |
|---|---|---|
| Expected result | One known correct output | A range of acceptable outputs, graded by quality |
| Assertion style | Exact equality, schema match | Properties: grounded, in-format, in-policy, in-budget |
| Repeatability | Deterministic | Statistical — repeated runs over a dataset |
| Regression signal | Pass turns to fail | An aggregate score drops below an agreed threshold |
| What breaks it | A code change | Code, prompt, model version, retrieval corpus, or the provider |
| Security surface | Input validation, authorisation | Plus injection through any text the model reads |
| Cost of a bug | Broken function | Broken function, wrong advice, leaked data, or a large bill |
The practical consequence: an AI feature needs an evaluation suite that runs on every prompt and model change, not just on code changes. Most teams discover this after their first silent quality regression from a provider model upgrade.
FAQ
Fair questions about AI in testing
Is this just "we use ChatGPT to write tests"?
No. AI sits inside our delivery workflow at specific points — drafting, data generation, triage, prioritisation — each with a human review step. The tests that gate your releases are reviewed, version-controlled code in your repository, exactly as they would be without AI.
Can AI decide whether a test passed?
Not for anything that gates a release. Deterministic assertions decide deterministic outcomes. AI-as-judge is used only where the output is genuinely open-ended — an LLM response, a summary — and even then it is calibrated against human-graded examples and reported with its own error rate.
We do not have AI features. Is any of this relevant?
The second half is not. The first half still is — it is how we deliver faster on ordinary web and API testing. You do not need AI in your product to benefit from AI in our workflow.
Does our code or data get sent to a model provider?
Only with your explicit written agreement, scoped to what you approve. Many clients allow it for spec text and test code while excluding production data entirely. If you want a zero-external-inference engagement, we will do the work without it and say what that costs in speed.
How do you test something that gives a different answer every time?
You stop asserting exact strings and start measuring properties: is it grounded in the retrieved context, is it in the right format, does it stay inside policy, is it within the latency and cost budget. Then you run it enough times to make the measurement mean something.
Shipping AI features with no way to test them?
Start with the health check for the general picture, or write to us directly about the AI-specific problem. Either way the first conversation costs nothing and you get a concrete opinion.