Which prompt injection detector fits your agent?
Choose with a test set from the documents and tool results your agent actually reads. Compare attacks caught, legitimate inputs blocked, complete-input coverage, and latency at the same operating point. Then verify that the application handles both detections and service failures correctly.
By ZeroLeaks · Updated
Decide what a mistake costs
A detector that works well on short chat messages may behave differently on a retrieved page, a support ticket, or structured tool output.
- Attacks missed
- Measure recall on instructions embedded in the formats your application consumes. Include indirect attacks, obfuscation, and attempts to trigger tool use, alongside direct user attacks.
- Legitimate work blocked
- Measure false positives on ordinary traffic and difficult benign examples: security documentation, quoted attacks, code, and multilingual content. Pick an acceptable false-positive budget before selecting a threshold.
- Input coverage
- Check length limits, truncation, chunking, and supported content types. An unflagged result does not establish safety for text that the detector did not score.
- Operational fit
- Measure end-to-end latency on representative input lengths, including queues and transport. Check data retention, deployment licensing, service errors, and the integration's behavior before enabling blocking.
What our published comparison measures
ZeroLeaks' October 29, 2026 Shield 1.1 model card addendum compares detectors on 35 headline public datasets across five categories, using each detector's default threshold. The table below shows selected detectors from that evaluation. Mean balanced accuracy averages the category scores; it is not the percentage of attacks stopped.
This is our own evaluation, published with its methodology in the model card. The benchmark also guided model selection, so these are development results, not a fresh held-out test. Use them as a starting point for a shortlist. They do not measure your application's traffic, establish current hosted latency, or prove resistance to an attacker adapting to the current model.
| Detector | Mean balanced accuracy |
|---|---|
| Shield Large | 91.5% |
| GPT-6.1 Sol as a judge | 89.0% |
| Jev by TypeSafe | 84.6% |
| OpenAI Guardrails | 80.1% |
| Amazon Bedrock Guardrails | 72.8% |
| Azure AI Prompt Shields | 63.2% |
Read the weak spots before choosing
In that evaluation, GPT-6.1 Sol used as a judge scored higher than Shield Large on agent attacks and prompt injection. Jev left more benign text unflagged. Shield Large's overall score combines its results across all five categories, so it should not be read as a win on every workload.
The Shield 1.1 addendum states that adaptive attacks have not been rerun against that release. The original Shield 1 model card's adaptive experiments evaluated an earlier release, and the archived SDK benchmark page describes another historical configuration. Neither is evidence of the current hosted model's adaptive robustness.
For RAG and tool results, test instructions inside full documents and structured responses. Keep a private holdout that you do not use to tune thresholds. Evaluate new attack attempts against the chosen configuration after tuning, and record the model version and test date.
Build a small evaluation from your workflow
Collect representative benign inputs with permission and remove sensitive data. Add malicious counterparts that preserve each document's format and legitimate task. Split by source document or conversation so near-duplicates do not land in both tuning and test sets.
Score each candidate on the same inputs and record false positives, false negatives, errors, incomplete coverage, and latency. An unavailable detector is a separate outcome, not a clean verdict. Keep local model timings separate from hosted round-trip measurements.
Start in observation mode on an approved test environment. Review what would be blocked, then exercise the blocking path with a known attack, a clean input, a timeout, and an over-length input. Confirm that the agent's legitimate task still succeeds.
Choose the integration as well as the detector
Shield's hosted API is useful when you want a managed detector in front of retrieved text or tool results. Its default Shield tier is free with an account, API key, and research acknowledgement. Paid tiers have different data-handling terms; review those before sending sensitive content.
A local classifier may suit an environment that cannot send text to an external service. Check the model weights' license separately from the SDK's license, and measure the complete local pipeline. A general language model used as a judge is another candidate when its measured latency and cost fit the workflow.
Keep authorization, egress restrictions, and approval checks in the application regardless of which detector you select. A text classifier cannot decide every tenant boundary or tool permission. Use agent red teaming to test whether a missed injection can produce a forbidden action.
Questions
Email us if yours isn't here.
Which prompt injection detector is best?
The right choice depends on your input formats, acceptable false positives, latency, and data-handling requirements. Use published benchmarks to choose candidates, then compare them on a private holdout from your own workflow.
Is balanced accuracy the same as attack recall?
No. Balanced accuracy gives equal weight to correctly identifying attacks and correctly leaving benign inputs alone. Attack recall measures how many attacks were detected. Read both alongside the false-positive rate and the test-set composition.
Does a clean verdict mean a document is safe?
No. A detector can miss attacks, and a verdict may cover only part of a long input. Check coverage and service errors, then retain permission controls around the actions the agent can take.
Can I compare local inference time with hosted API latency?
Only if you clearly separate what each timing includes. Local inference excludes network transport and service queues. For deployment decisions, measure the end-to-end path your application will use.
Related testing
- How to Red Team AI Agents with Tools and MemoryA deployment checklist for testing AI agents with tools, memory, and credentials: define boundaries, exercise real tool paths, inspect evidence, and rerun after fixes.
- MCP Security: Protect Tools from Prompt InjectionProtect MCP agents from poisoned tool descriptions and results. Inspect definitions, pin changes, enforce tool permissions, and test the complete agent workflow.
- Prompt Injection Testing for AI SystemsTest AI applications for prompt injection, instruction hijacking, tool abuse, and multi-turn attacks before they reach production.
- AI Red Teaming for LLM Apps and AgentsAutomated AI red teaming for LLM applications, agents, system prompts, retrieval flows, and tool-calling workflows.
- AI Agent Security TestingTest AI agents for prompt injection, unsafe tool calls, over-permissioned actions, secret exposure, and workflow boundary failures.
- LLM Security Scanner for AI AppsRun automated LLM security scans for prompt injection, prompt leakage, tool abuse, model behavior regressions, and AI security reports.