Red teaming an agentic AI system is different from traditional systems. Agentic AI and traditional AI systems are not deterministic and scripts will need to be run multiple times. Each time the scripts are run, the result will be different. You must account for this variability when testing each scenario. You should also keep in mind that due to the logic of the agentic workflow, the LLM itself, prompt variability, and agent behavior will lead to greater variability. You will also find that running the same task in the same scenario will react differently and you will need to run more tests and test cases to cover potential blind spots. Have your development teams create a map of all the rules and circulate possibilities throughout the process.

As with any tool, you won’t and shouldn’t always automate everything. Use a tool like PyRIT with manual testing. Manual testing will allow testers to test specific problem areas as well as perform deeper analysis of any areas discovered by automation testing.

Make sure you also provide monitoring and logging of your automation tests. This will test the issue tracing process, but also help the team further their manual testing. Test the process of using recorded data to ensure transparency and auditability at this point, rather than when an issue arises in production.

Latelywork with other cybersecurity experts to compare and contrast measures and practices. Continue to develop your governance framework and always add and refine your procedures.

The future of agentic AI: promising… and full of possibilities

The wide range of benefits, capabilities and efficiencies that can be offered to the business makes it the ideal time to explore this technology. However, the associated security risks and threats cannot be ignored. We must ensure that the company culture is broadened so that safety is everyone’s business. It is the teams’ responsibility to record all interactions, monitor the system, and ensure human controls are in place. Tools must be integrated into end-to-end processes to proactively detect issues before they erode user and business trust. Transparency, human oversight and AI safety must always be a priority.

Security teams must define security controls and governance, metrics and rules. Development teams should educate themselves not only about these rules and requirements, but also about the risks they will encounter and the mitigation measures they need to put in place.

Stephen Kaufman is chief architect in the Microsoft Customer Success Unit Office of the CTO focused on AI and cloud computing. He brings more than 30 years of experience working with some of the largest enterprise clients, helping them understand and use AI, from initial concepts to specific application architectures, design, development and implementation. delivery.

This article was made possible thanks to our partnership with IASA Chief Architects Forum. The goal of the CAF is to test, challenge and support the art and science of enterprise technology architecture and its evolution over time, as well as to increase the influence and leadership of chief architects, both inside and outside the profession. The CAF is a leadership community of IASAthe leading nonprofit professional association for enterprise technology architects.