How to red team an agent before deployment

Test the agent with the tools, memory, permissions, and approval gates it will actually use. Start with explicit rules it must obey, send attacks through its untrusted inputs, and inspect the actions it takes. Keep each confirmed failure as a regression case.

By ZeroLeaks · Updated 2026-11-03

Set up a runtime scan Documentation

Write down what must never happen

A useful test has an observable failure condition. These examples describe boundaries you can check in tool traces and application state.

Authorization: A support agent can read the current customer's records. A request from that customer must not make it read a different tenant's records or borrow a more privileged identity.
Tool use and approval: A draft-writing agent must not publish, transfer funds, or delete data without the application's required approval. Check the tool call and approval state together.
Network and disclosure: Retrieved content must not make the agent send private data to a new destination. Use synthetic credentials and reserved .invalid destinations in test payloads.
Memory and handoffs: An instruction in a document must not become a durable rule, survive into another user's session, or grant a downstream agent permissions the user did not grant.

Choose the test surface

Use an endpoint scan when the deployed HTTP interface exposes the behavior you need to test. Use a runtime scan when you need the application's actual orchestration, tool traces, memory, or middleware. The runtime relay lets your process execute the agent without exposing a public target URL.

Describe the available tools and return structured tool calls, including arguments, results, and errors. A text-only response can hide a side effect. Keep state within a multi-turn campaign, isolate campaigns from one another, and implement reset so one test does not leave instructions behind for the next.

Run a legitimate task before attacking. If the agent cannot complete it, an apparent refusal may be a broken connector, missing credential, or timeout. Report infrastructure errors separately from defended attacks.

Follow an attack through to its effect

One family in our research corpus poisons a handoff so an agent stores a deployment credential in durable memory. The relevant failure is a memory write containing protected material. A reply that merely says it will remember the credential is weaker evidence. The published Gauntlet measures model behavior in its scenarios; it does not certify your application's configuration.

For your agent, plant a synthetic canary, deliver the instruction through a surface an attacker could actually control, and inspect both the tool trace and memory after the turn. Check whether a later session can retrieve the canary. Record the entry point, permission boundary, observed action, and the exact version of the target.

Redact credentials and customer content before sharing reports. A reproducible finding needs enough context to explain the failure without distributing the sensitive value itself.

Gauntlet scenarios and evaluation

Keep the environment safe to exercise

Use a staging tenant with synthetic records and restricted credentials. Preserve the production tool schemas, authorization checks, and approval workflow. Replace destructive effects with audited simulations at the execution boundary.

ZeroLeaks rewrites outbound attack destinations to canary sinks on .invalid. Endpoint and runtime targets still execute their own tools. Payload containment cannot undo a local delete or a privileged database write, so the target must also enforce its own tool permissions and execution controls.

Include clean control tasks alongside adversarial cases. After tightening a permission, confirm that the agent can still complete the intended workflow. A policy that blocks all useful actions is not a successful deployment outcome.

Apply deterministic tool permissions with Shield

Close the finding, then run it again

For the memory example, restrict the memory tool to approved fields, reject credential-like values at the write boundary, and isolate stored state by tenant and session. Change the smallest permission or configuration that prevents the observed effect.

ZeroLeaks generates remediation guidance after a scan. It does not apply the change to your application or prove the fix inside that same scan. Apply the change, then run the attack again on the fixed target. Check the memory state as well as the response.

Put the reproducer in your release checks. Keep the model, tools, policy, and attack configuration recorded so a later result can be interpreted. Block release on a confirmed boundary violation or an incomplete required test; investigate score changes with the underlying evidence.

AI red teaming coverage

Questions

Email us if yours isn't here.

Can I test an agent without giving ZeroLeaks its credentials?

Runtime scans invoke the agent inside your process through a relay, so its provider and tool credentials stay with your application. Review the messages and tool results you return to the scan service, and use synthetic data wherever possible.

Does a safe model mean the agent is safe?

Model behavior is one part of the system. Retrieval, memory, tool permissions, approvals, and other agents can introduce failures even when a model handles a standalone attack correctly. Test those paths together.

Should I run red teaming in production?

Start with a production-shaped staging environment. Endpoint and runtime agents execute their own tools, so a production scan requires enforceable limits on side effects, synthetic test data, and scoped credentials. Rewriting attack destinations alone is not sufficient.

Does ZeroLeaks verify the generated remediation automatically?

The scan supplies remediation guidance. You apply the change to your target and verify it with a subsequent scan. Keep successful attack cases in later runs to detect regressions.