When Determinism Dies: Testing and Securing Agentic AI in Production

Show notes

The pilot episode where two enterprise veterans explain why your old QA pipeline breaks the moment an LLM enters the building. In this first episode of Ctrl-Alt-Deploy, host Alex Belotsky (CEO of Hesabon/TestSavant.AI) and co-host Kuba Fiatkevich unpack what changes when non-deterministic AI lands in high-stakes software. They argue that the classic assumption of "input A always yields output B" no longer holds, so pipelines built for binary pass/fail states need a new evaluation stage that judges the quality of probabilistic outputs. The conversation moves from LLM chatbots to the 2026 rush toward autonomous agents that wield tools and execute their own code — where a bad chatbot causes reputational damage but a bad agent can delete a database. They dig into treating agents as untrusted "employees," tightening permission scoping at the architectural level, stripping tool permissions the moment a task ends, and red-teaming for prompt injection, exfiltration, and denial-of-wallet attacks. Alex and Kuba also introduce economic and functional health metrics — cost per successful transaction, human intervention rate, benign rejection rate — and describe an adaptive loop where test resu