Security of interacting AI agent systems

Multi-agent systems is a paradigm that encompasses all sorts of systems, ranging from classical distributed systems (composed of
deterministic components described by pure code) to agentic AI systems, including what we call here “interacting AI agent systems”,
combining pure code and LLM-based components. While security analysis of classical distributed systems is a well-developed field with a
large spectrum of tools such as taint analysis, the analysis of security properties of interacting AI agent systems is still in their infancy
because of key main reasons. First, the presence of LLM elements renders the traceability of data challenging because the original data is
modified through the attention layers of transformers in a highly complex manner. Second, new security issues have been observed from the
composition of LLM elements that would have their security vetted individually, but that may give rise to undesirable emergent overall
behaviour.
The thesis is to take place within the LogicA team which offers a relevant research environment with combined theoretical expertise in
formal methods for multi-agent systems, experience of development and deployment of LLM-based systems, as well as core AI and GenAI
model training and specialisation. Moreover, the thesis is grounded on the ongoing research collaboration between Université de Rennes
and Heriot-Watt University providing expertise in synergy with the LogicA team: machine learning and GenAI with Michael Lones,
verification of security protocols with Sasa Radomirovic, applied formal methods for the security of software systems with Manuel Maarek.
The topic of this thesis is concerned with the security analysis of interacting AI agent systems. Typically, prompt injection is known to be a
high security risk already for a single LLM-based component inference; many solutions have been proposed to mitigate this risk, for instance
LLMs can be fine-tuned to maintain the separation between user data and instructions in a prompt. However, there is a strong likelihood of
risks being compounded once an LLM interacts with other components of the overall system (for instance, fine-tuning of individual models
is unlikely to be a sufficient guard), and therefore protections need to be considered at a higher level. The thesis’ main research questions
are as follows.
• How effective are existing defences against prompt-based attacks in the context of a system with different forms of LLM
composition (effectiveness may be sensitive to the type of architecture)?
• What individual instrumentations of LLMs corroborate the characterisation of secure and insecure LLM inference?
• How can instrumentations of LLMs support compositional reasoning about the security of interacting AI agent systems?
• Which instrumentations of LLMs can serve the design of AI agent interaction protocols to guarantee secure behaviour of an
interacting AI agent system?

There will be four main steps in the thesis:

1. Design and implement interacting AI agent systems that evidence the bypassing of solutions against prompt-based attack onindividual LLMs.
2. Propose various individual instrumentations of LLMs (for example using metrics derived from the LLM models) tocharacterise secure and insecure LLM inferences.
3. Run experiments to select individual instrumentations that evidence interaction phenomena between components that revealsecurity flaws and make the evidence amenable to formal reasoning.
4. Starting from previously selected instrumentations, design remediation (such as interaction protocols) against prompt-based
5. attacks in interacting AI agent systems.

To achieve the four steps, we propose a methodology that takes advantage of the multi-competences of the research environment offered by
the LogicA team and the collaboration (cotutelle) with Heriot-Watt University, together keeping up with the fast moving applied and
theoretical field of GenAI: (1) targeted implementations of interacting AI agent systems can be hosted by an environment that already exists,
where team members have shown their skills on deploying agentic LLMs; (2) instrumentation of LLMs can be supported by team members’
knowledge in the training of LLMs and in existing tools on metrics derived from LLM models; (3) expected compositional reasoning is
supported by the team expertise in formal verification as well as security protocols