Emerging Risk

What Is an AI Agent?

An AI agent is a system that uses an AI model to take autonomous sequences of actions to complete a goal, browsing the web, calling APIs, writing and executing code, sending communications, or interacting with other software. Unlike a chatbot that produces text, an agent acts in the world. This creates governance challenges that are qualitatively different from ordinary AI tool deployment.

Agents vs. tools: the key distinction

A conventional AI tool, a chatbot, a document summariser, an image generator, takes an input and produces an output. A human then decides what to do with that output. An agent inverts this relationship: it is given a goal, and autonomously decides what sequence of actions to take to achieve it.

This distinction matters enormously for governance. When a human reviews a chatbot output before acting, the human is the control. When an agent acts before a human reviews, the human oversight mechanism must be designed into the system, it does not happen automatically.

Governance risks specific to AI agents

Action irreversibility

An agent that sends emails, submits forms, executes code, or transfers data may take irreversible actions before a human has a chance to review. Unlike a chatbot that outputs text, agent actions can have immediate real-world consequences.

Cascading errors

Agents take multiple sequential steps. An error at step 2 propagates through steps 3, 4, and 5. By the time the error is apparent, significant damage may have occurred. Control mechanisms must intercept errors early in the chain.

Prompt injection

An agent processing external content, emails, web pages, documents, can be manipulated by malicious instructions embedded in that content. A hostile actor can redirect the agent's behaviour by including instructions in data the agent reads.

Accountability gaps

When an agent takes a sequence of actions across multiple systems, tracing which decision caused a harmful outcome requires complete audit logging at every step. Most enterprise AI deployments do not yet have this capability.

Privilege escalation

Agents are often granted broad system permissions to complete tasks flexibly. This creates attack surface: if the agent is manipulated, it can use its legitimate permissions to cause harm at scale.

Regulatory classification

Agentic AI operating in high-stakes domains, healthcare, finance, employment, legal proceedings, may fall within the EU AI Act's high-risk AI categories, triggering conformity assessment and human oversight requirements.

Minimum governance requirements for agentic AI

Organisations deploying AI agents should establish, at minimum, the following controls before any agent takes autonomous action in a production environment:

Complete audit logging of every action the agent takes, at every step
Explicit scope limitation, agents should have access only to the systems and permissions required for their specific task
Human approval checkpoints before irreversible actions (sending communications, executing payments, modifying production systems)
A defined kill-switch or pause mechanism accessible to a named human responsible
Testing in a sandboxed environment before production deployment, including adversarial testing for prompt injection

Agentic AI governance guide What is an LLM?