PNSQC – What is AI Assurance? The next big thing for QA?

This year, I have the huge pleasure of delivering the opening keynote at PNSQC on Monday, October 12th at high noon. I guess the goal of any keynote is to inspire attendees and set the tone for the rest of the conference.

Unless you have been hiding out on Mount Rainier for the last five years, it has been nearly impossible to attend a conference without hearing another talk on how AI is changing the landscape, followed by a debate on its positive or negative impact.

Take it a step further by adding “assurance” or “quality” to the title, and the discussion becomes even more elevated: how do you measure quality, and who is responsible for it? These have been valid debates since the first “bug” was found back in 1947.

In this blog, let us avoid the minefields of controversy and instead explore a relatively new concept: “AI Assurance”. I first came across it while working with the UK Government back in 2020, when it published an independent report from the CDEI on AI Assurance, which I was lucky enough to contribute to as a fellow and committee member of the British Computer Society Specialist Interest Group in Software Testing (SIGiST), originally set up by Dorothy Graham.

Earlier this year, Jason Arbon published a Manifesto for Confidence Engineering, which stated: “Perfect software isn't achievable, but confidence is. Ship with real evidence, not optimism and vanity metrics.” As a co-founder of OpenTest.AI, it is very important to help contribute to the direction of the industry and, more importantly, to openness: built by the community, for the community.

Just over 10 years ago, I tried to capture this by publishing a similar manifesto with goals for how to test AI systems. I also gave a TEDx talk on Cognitive Learning in 2017, built around a simple “Think > Test > Learn” model. This was foundational to the AI-infused systems shown in the background of the image below, which I was involved in testing at the time.

The complexity of testing smart cities, robotics, and predictive crime platforms at the time highlighted a major challenge: testing a single AI-infused application was hard enough, but what happens when you combine Physical AI with Enterprise AI systems of systems? This is when the term “AI Assurance” was born back in 2020, and it later became my chapter in the book Artificial Intelligence for Software Testing (download for free), where I covered the use case of working with MIT during COVID-19 to test the rollout of contact tracing.

The complexity of testing Enterprise AI at scale, especially systems of systems that combine with Physical AI, is a real problem. I’ll be sharing recent real-world use cases in the keynote, including space systems, 6G holographic calls, and humanoid robots.

Across all these real-world projects, I use “AI Assurance” to define what needs to be tested, as set out in the AI Assurance Manifesto, and then leverage “Confidence Engineering” to answer how we know, as described in the Confidence Engineering Manifesto.

AI Assurance answers what to test. AI Confidence Engineering answers how we know our tests still mean anything. Together, they turn a historical all-green dashboard into justified confidence.

However, as Jason Arbon states in his latest book, Testing AI: Engineering Confidence in Non-Deterministic Systems: “AI makes software faster to create and harder to trust.” He goes on to cover quality, safety, evals, statistics, and judgement for the age of AI systems.

The Confidence Asymmetry principle explains why the cost of confidence, not the cost of code, will govern the economics of testing AI-infused software. As the cost of generating code falls, the complexity gap increases, while the undecidability floor of the halting problem and Rice’s theorem sits beneath the verification cost of execution, reinforcing why you must run it in practice.

As the theoretical and empirical foundations of Confidence Engineering continue to evolve, the bigger challenge of the what and the why remains unanswered. This is where “AI Assurance” steps in to address “AI testing — how to provide AI assurance through confidence engineering”, which, ironically, is the placeholder title for our upcoming book.

After catching up with Ken Johnston about this the other day, I learned that he has been working on the AIGovOps Foundation, which is trying to answer this exact question. Its tagline is “Ship Safe AI, Never Unsafe AI”, and it was co-founded by Bob Rapp to help advance responsible AI governance for organisations navigating the evolving AI landscape.

A recent study from the WorldDigitalReport 2026 found that 88% of organisations now use AI in at least one business function, up from 78% last year, but only 5% extract substantial value at scale. If you are interested in understanding how Open Assurance can help you unlock real business value from “AI Assurance”, through “Intent-Driven Testing” to “Confidence Engineering”, this is exactly the journey we need to explore.

As for “AI Agentic Testing”, don’t take my word for it. This week, Gartner released this year’s Magic Quadrant for AI Agentic Software Quality Assurance Platforms, where my work was featured in the Leaders quadrant again, building on the momentum of last year’s AI-Augmented Testing Magic Quadrant. This aligns perfectly with the topic of my keynote: “The AI Assurance Imperative: Quality Engineering for the Agentic Era”.