AutomationSTAR - Testing the World in 24 Hours with AI Agentic Testing

For the last eight years, three times a year, I have run tests against 500+ global retailers, covering security, performance, usability, accessibility, privacy, reliability, and code quality. Back in 2018, as you can imagine, this required a huge amount of effort from thought leaders across the testing community, and I needed to draw on their specialist expertise. Thankfully, I was able to bring in my brother from another mother, Todd DeCapua, to support the Performance Engineering aspects, alongside experts such as Stuart Moncrieff, who helped with specialist areas of security testing.

Since the Agile Manifesto asked us to value working software, many people have moved away from mastering one specialism, such as automation engineering, and towards becoming a jack of all trades — and, too often, master of none. Now, with swarms of AI agents emerging, testers are understandably questioning the value they add through traditional methods, approaches, and tooling.

Thankfully, I have held the same belief throughout the last 30 years of my testing career (https://jonathon.ai/about): whether we are talking about automation or agentic AI, it will never replace human intelligence; it will simply give us superpowers.

However, since my AI transcendence on April 1st this year, my role has changed dramatically. I have moved from being an AI Wrangler, as I have been for the last few years, to becoming an AI Overseer. That shift has also changed the skills I need now, as well as the future skills I will need to survive in this new AI reality.

Sure, my role as AI Overseer sounds easy: talking to my coordinating agentic visual assistant — let’s call it JARVIS for simplicity, although it is actually just Jonathon.AI — and saying, “Hey, I have a new mission for you!”

Of course, I would love to have my “fantasy testers” team, made up of the heavy-hitting “world’s best testers”. Instead, I have to settle for 30+ AI agents with the limited skills and knowledge of the humanoid versions they are loosely based on. These agents are usually generated at runtime from the 1,000+ testers listed as speakers at this year’s conferences: https://www.testing-world.com/speakers

Unless you have been hiding under a rock for the last five years, you will have heard the claim that around 88% of people use “AI” every day as part of their company’s productivity efforts, while only about 5% are creating real business value with AI rather than simply generating AI slop.

So, what happens if you are already fed up with endless “AI slop” generated by generative AI platforms, where people simply paste anything into tools like Undetectable? After publishing the Artificial Intelligence for Software Testing book five years ago, the team has been working on a follow-up: AI Testing: How to Provide AI Assurance Through Confidence Engineering.

That work introduces both the AI Assurance and Confidence Engineering manifestos: “the WHAT” from the top down, and “the HOW WE KNOW” from the bottom up. These complementary disciplines help define the future skills and roles needed in this new AI-infused reality.

I am not going to dive into the semantics of the differences between using agentic AI and simply using generative AI or LLMs, as that is covered in this post. But, like your favorite Copilot coworker platform of choice, it usually starts with a goal or a plan. The AI then helps extract the intent — the mission — derive the goals and associated plans and define and prime a proposed agentic fleet suitable for the AI workload replacement. Usually, this is the cognitive thinking part of the human intelligence transcendence piece.

More recently, this has involved tracking down sources of knowledge and information — a new model for testing — to correctly prime my fleet with content understanding for industries such as 6G, Physical AI, and Cyber Robotics; domains such as testing, astrophysics, and quantum; and implementation specifics, including IP-sensitive information.

Then comes cognitive reasoning, critical thinking, and the understanding of intent and context. For me personally, in my AI Overseer role, this seems to be where I add most of my value today. Most of my time is spent providing additional business context, interpretation, and validation, along with legacy interactions with other humanoids to clarify systemic and epistemic entropy — usually leading back to the Information-Theoretic Imperative.

Finally, this all leads back to the Confidence Asymmetry Principle: the cost of Confidence Engineering the proof or evidence that the AI is doing the right thing, rather than just generating more slop.

Sure, that could mean having a five-minute conversation with JARVIS, aka Jonathon.AI, and asking it to test 500+ applications, run thousands of intent-driven tests, find thousands of defects, and generate more than half a million dollars of value in a single day.

Sure, testing a website is easy. I have no idea why so much testing effort is still spent on something we automated last century, alongside desktop, RPA, and CUA. But what happens when you are asked to test 6G holographic calls? Spacecraft? Physical AI and cyber-humanoid robotics? Yes, I have tested all of the above in the last few months — with no UIs.

If you think you are “not ready” for any of the above, then you need to register today for AutomationSTAR and learn the essential future skills you need to be ready to test the world.

Better still if you want to try all of the above with a hands-on tutorial then I will be supporting my padawan learner Gomathi Ramalingam at Tutorial F on Wednesday 4th November all morning at the AI Testing Dojo: Build, Break & Master Agents | AutomationSTAR