Testing Endgame or Encore? - Netflix, Dario and AI Assurance

Imagine an AI Agentic Testing platforms that outperform humans domain expert testers with clear AI transparency and observability of testing non-deterministic systems, with the expertise of the top 1%.

The Netflix AI documentary raises questions about deception, control, commercial pressure, employment, concentrated power and the distribution of benefits and harms. Its optimistic arguments about scientific progress and human capability matter too. Those competing possibilities lead to a practical responsibility: establish the conditions under which a particular AI system deserves to be used, and who can challenge that judgement.

Netflix AI Documentary - THE AI DOC: Or How I Became an Apocaloptimist

In my recent PNSQC and AutomationSTAR blogs, and my TestGuild conversation about becoming part of the irreplaceable 40%, I explored the value of domain knowledge, context and human judgement. Here I want to connect those ideas to an enterprise operating model: independent AI Assurance, supported by Confidence Engineering and global shared testing knowledge as part of Open Testing.

TestGuild - Automation Podcast - How to Stay Irreplaceable with Joe Colantonio (https://testguild.com/a604)

Dario Amodei recently argued that increasingly capable models could become better at deceiving evaluations, making broader tests and interpretability cross-checks especially valuable. He also proposed embedded external evaluators with substantial access and the ability to report uncomfortable findings. Both points belong in the same conversation. Better evaluation methods need institutions capable of using their results.

Dario Amodei - We Must Pace the Frontier - (https://darioamodei.com/post/we-must-pace-the-frontier).

There is empirical reason to take the evaluation problem seriously. Anthropic and Redwood Research demonstrated alignment-faking behaviour under experimental conditions that placed a model’s existing harmlessness preferences in conflict with a stated new training objective. This was a constructed setting, not a measurement of how often deployed AI secretly pursues harmful goals. ).

See content credentials

Anthropic - Alignment faking in LLMs (https://www.anthropic.com/research/alignment-faking).

Interpretation also needs challenge. A June 2026 preprint found that trying to satisfy perceived researcher expectations could explain some behaviour attributed to scheming in the evaluations it studied. Its results do not eliminate deception risk; they show why an assurance claim must distinguish observed actions from an inference about motivation. [Sycophancy Towards Researchers Drives Performative Misalignment](https://arxiv.org/abs/2606.08629).

An incorrect answer is not automatically deception. A fluent explanation is not evidence of faithful reasoning. Higher capability does not establish a universal law of increasing dishonesty.

The practical concern is that an evaluation may become less diagnostic even while its scores improve.

I propose the AI Assurance Asymmetry Principle as a research proposition:

As AI Agentic capabilities, autonomy and deployment scope increase, the ability to demonstrate acceptable behaviour can improve faster than our ability to establish that behaviour remains acceptable in use.

This is an independent companion to the AI Confidence Asymmetry Principle, which I developed around generation, verification and the cost of justified confidence. This new proposition focuses on the gap between apparent acceptability and the evidence supporting real deployment decisions.

The Confidence Asymmetry Principle - Jonathon Wright · 4 May 2026 · Working paper

It is conditional and testable. Better measurement, constrained architectures and stronger controls may narrow that gap. We should investigate when they do, using independently discovered failures and operational outcomes alongside benchmark scores.

Independent AI Assurance validation needs a real mandate

AI Assurance needs practitioners who can challenge the system, its intended use and the organisation’s reasons for trusting it.

Independence has several dimensions. Organisational independence protects findings from delivery pressure. Methodological independence introduces different scenarios, assumptions and ways of observing results. Evidential independence asks whether apparently separate checks share the same blind spots.

This time last year for the Gartner MQ & Forrester Wave, I introduced the concept of 'LLM as a Judge' into our AI Agentic Platform which can help with providing conference levels, but a different provider does not establish independent errors. Nor does an external assessor provide meaningful assurance if it receives only limited meta test data, screenshots and a context understanding.

AI Assurance needs access to all relevant multi-dimensional data (i.e., delta lakehouses / knowledge graphs), real world environments, failures and operational evidence; resources to reproduce findings; and a protected route to escalate unresolved concerns. Its remit should permit a conclusion of insufficient evidence.

OpenTest.AI - Decades of Domain Testing Context Understand

Development teams still test their own work. Independent validation challenges their conclusions. Internal AI Audit can assess whether governance works while remaining independent of operating those controls. These responsibilities can cooperate without becoming interchangeable. This separation is central to the OpenAssurance.AI governance methodology.

Open Assurance - AI-Assured - AI governance for AI-native organizations - openassurance.ai

The Confidence Engineer assesses evidence sufficiency. A named human decision owner authorises deployment, accepts residual risk or imposes restrictions. Those are related responsibilities with different decision rights.

Intent-Based Testing (IBT) connects mission statement to evidence

“Test the 6G Holographic calling” could be a mission for AI Agentic Swarm to work on. But it leaves the most important questions unanswered: what are the goals / outcomes, what success means, do I understand enough context understanding for the domain, industry and business value.

Intent-Based Testing makes those questions explicit. It builds on the [Intent Layer] in Confidence Engineering: clarify requirements, expose ambiguity, quantify outcomes where appropriate and connect examples to acceptance conditions.

Intent includes stakeholder needs and legitimate policy constraints; it is broader than obeying the latest prompt. Conflicting intentions must be resolved by authorised people. A request to “rebuild the 6G test lab” cannot silently override approval boundaries.

The resulting confidence claim is specific: supported for this purpose, with these controls, under these conditions. It is neither the model’s self-reported confidence nor an unexplained percentage on a dashboard.

AI Wrangler - The Human AI Overseer's Role - 30+ Agentic Swarm running 16hrs a day!

Scaling AI testing across the enterprise

Since April 1st this year my transcendence happened, the shift from Automation Cyborg fueled with my 30+ years experience in Automation. Now I'm just Jonathon.AI the human AI overseer, managing 30+ agents working on the most complex real world testing challenges ranging from testing Quantum, 6G, Cyber Robotics, Physical AI and even Space. Running 16hr days 7 days a week, avoiding tokenmaxing, providing context engineering and shared understanding (short > long term memory).

Transcendence from the Automation Cyborg to AI Agentic Agent (mcp.jonathon.ai)

However, unless you can scale this testing globally, then it is limited to few minds that have spent the last 15 years trying to solve the AI for Software Testing problem space. Which is one of the reasons I cofounded OpenTest.AI and OpenAssurance.AI to provide the blueprints and runbooks to make this a scalable method: starting with intent, prime the system, define risks and outcomes, plan and design the right tests, execute in the real world (not browsers), interpret findings, gather evidence, build confidence and reassess.

Upcoming book "Test AI: How to Provide AI Assurance through Confidence Engineering"

It combines two perspectives: the intelligence being delivered and the integrated experience through which people use it. Evidence is collected at model, system and user levels. A strong model evaluation cannot substitute for testing permissions, hand-offs, accessibility or the consequences of a failed business process.

The current [OpenAssurance foundations] organise this work through six connected modules:

| Intent | Define outcomes, affected people, system boundaries and accountable owners.

| Risks | Identify harms, exposure, obligations and priorities.

| Controls | Establish permissions, oversight, safeguards and recovery.

| Evidence | Test claims and controls; judge the sufficiency and limitations of results.

| Decisions | Record human authorisation, residual risk and operating conditions.

| Confidence | Monitor whether the justification remains valid and reopen decisions when it does not.

What to give it AI Confidence Engineering a try? run the AI Assessment against your company and see if it adds any value! (unless you have have AI Agent blocker lol)

World Digital Report (est. 2017) - A real-world example of testing 500+ apps, finding 35,000 bugs, and creating more than half a million dollars of value in a single day