How We Test Agent Boundaries (And What Our Score Does Not Mean)

The security question for AI has moved. A year ago the sharp edge was the single-prompt jailbreak: can someone talk the model into ignoring its instructions. That still matters, but it is no longer where the real damage happens. Damage happens when an autonomous agent with tools, memory, credentials, and the ability to talk to other agents does something it was never authorized to do — sends the wrong data to the wrong place, escalates its own permissions, or acts on an instruction hidden inside a document it was asked to summarize.

This post explains how ZeroLeaks tests that surface, how we do it without endangering the systems we scan, and — just as importantly — what our score means and what it does not. We would rather tell you the boundary of our confidence than sell you a number you cannot defend to your own security team.

From prompts to boundaries

We start every agent scan by profiling the target: its role, its declared tools and schemas, the data it can reach, and the trust boundaries implied by all of that. From this we derive a set of enforceable invariants — concrete statements of what the agent must never do. Examples: this identity may not invoke that tool, this data class may not leave the tenant, this action requires human approval, this credential may not be used on behalf of an unauthenticated caller.

Attacks are then generated against those specific invariants, not against a generic checklist. A frontier attacker model writes probes tailored to the agent in front of it, and every failed attempt is mutated and retried. The point is to test the agent you actually built, not a strawman.

The four things a generic scanner misses

Coverage is where most agent testing quietly falls short. Four areas matter and are easy to fake:

Multi-agent attacks. Agent graphs have their own trust surface. We forge inter-agent messages that impersonate a supervisor, spoof an agent-to-agent handoff, poison shared memory, mutate an MCP tool description mid-session, chain delegation to amplify privilege, and set up confused-deputy situations. A tool that only tests one model in isolation cannot see any of this.

Real indirect and multimodal payloads. Claiming a prompt is hidden inside a PDF is not the same as testing a real PDF parser. We generate genuine artifacts — valid PDFs, images with embedded text, HTML with concealed content, spreadsheets with formula injection, email, poisoned retrieval chunks, and MCP manifests — each carrying a hidden instruction, and deliver them through the agent's real ingestion path. When a channel cannot be exercised, we say so rather than pretend it passed.

Long-horizon campaigns. Some failures only appear after trust is built over many turns: memory poisoning, delayed execution, approval grooming, authority drift. We run budgeted attacker-in-the-loop campaigns that groom the agent across a long conversation, and the campaign state is durable so a paused scan resumes instead of restarting.

Cross-scan learning that cannot leak. Attacks that work get promoted into a reusable library, but strictly within a tenant boundary, only after independent verification and reproduction across distinct targets. One customer's prompts, tool names, or workflows never influence another customer's scan, and a malicious target cannot poison the shared corpus.

Contained by design

A red-team tool that can cause real harm is not one you should point at a production-shaped agent. The industry was reminded of this recently when models in cybersecurity evaluations were observed circumventing isolation and reaching unauthorized channels.

So containment is enforced, not documented. Every attack payload is rewritten on the exact bytes we send, so any destination an exploit tries to reach resolves to a sink we own and nowhere else. When ZeroLeaks executes tool calls itself, a policy proxy runs before execution: external network egress is denied by default, and destructive, remote-code, and reverse-shell commands are simulated rather than run. Detection is preserved — we still record that the agent tried — but the effect never lands. Authorized tests of an agent's own boundary, such as a request to its cloud metadata endpoint, are kept intact, because the risk there stays inside the target's own trust domain.

Fixes you can prove, not just findings

A finding without a fix is homework. Every result carries the least-privilege policy or configuration change that closes it — not only prompt hardening — and we automatically rerun the exploit to confirm the boundary now holds. Reports rank findings by severity, preserve reproducible evidence and the full causal trace, and export cleanly for engineering, security, and compliance.

What our score is, and what it is not

Here is the part most vendors leave out.

Our score is an observed adversarial risk score. It reflects what our current attacker and judge ensemble found, under the query budget of this scan, judged by multiple models. It is genuinely useful: it finds real, reproducible boundary failures, and it is honest about the evidence behind each one — an externally observable effect such as a real tool call or an echoed canary is weighted far above the model merely sounding compliant.

It is not, today, a statistical certification. We do not yet publish population-level false-positive and false-negative rates, per-category precision and recall, or a calibration of the number itself against human-adjudicated ground truth. Producing that — a sealed, expert-labeled evaluation set and the calibration study on top of it — is work in progress, and we will publish the numbers when they exist rather than imply them before. Until then, treat the score as strong signal for prioritization and design-partner assurance, not as a release gate that certifies an agent is safe.

We think being clear about that boundary is the whole point. The teams protecting critical systems do not need another confident dashboard. They need testing that is broad, safe to run, reproducible, and honest about its own uncertainty. That is what we are building, continuously, and in the open.