Archived benchmarks
Historical SDK detector comparisons, evaluation limitations, and reproduction notes. These results do not describe the current hosted models.
Historical evaluation. This page preserves an earlier SDK comparison and its then-current model configuration. References below to a model being “shipped” or “current” describe that historical snapshot. The hosted
shield-largemodel now combines two classifiers; these tables do not measure its current results or API latency. See current model access.
Evaluation limits. The labels “held-out v1” through “held-out v5” are historical group names, not guarantees that a dataset or its source was unseen during training or model selection. Several evaluation sources overlap with training sources, and development results influenced model selection. These scores do not establish performance on independent, unseen data or a state-of-the-art claim.
The archived comparison ran Shield and 35 other detectors on 63 evaluation sets in seven groups. It included open injection and jailbreak classifiers, three large guard models, two paid services (Azure AI Prompt Shields and Amazon Bedrock Guardrails), GPT-6 Sol asked to judge each text, and JavaScript libraries on npm. Detectors used their default thresholds.
The archived run used three local SDK configurations with matching model files, tokenization, and windows. "Shield's model" is detectAsync() with Shield's model in place of its built-in classifier (model.options()). "Shield's large model" uses the large model the same way, scored from its 4-bit file. "Tiered" is tiered(): Shield's model first, and the large model for the inputs it is unsure about. "Shield 2.0 without the model" used the synchronous local detect(); that function now lives in @zeroleaks/shield/local.
Recorded results
- Agent-oriented groups. The archived large model scored 0.751 balanced accuracy on v5, compared with 0.801 for GPT-6 Sol. Its attack recall was 50.8% with 2.4% false positives. On v4, the earlier large checkpoint scored 0.818 and the later checkpoint 0.810; GPT-6 Sol scored 0.808.
- Development groups. The large model averaged 0.923 across the five groups used to build and select the models, tiered 0.911, and the default model 0.889. These are development results and should be interpreted with the training-source overlap described below.
- Local latency. The archived default model took a median of 25ms on one CPU thread and 10ms on four. A separate SDK run measured tiered at 43ms and the default model at 38ms on one thread; their means were 1,333ms and 178ms because the large model took seconds on long documents. The large model's median was 671ms on four CPU threads. These measurements exclude hosted API overhead.
- Limits. No tested detector, including GPT-6 Sol, did better than chance on MCIP's tool-use conversations. On v4's
agent_extractionset, the earlier large checkpoint caught 20.6% of attacks. ProtectAI's model scored 0.896 on v1 against the large model's 0.866.
| Detector | Kind | Mean of the five groups | Held-out v5 | Held-out v4 | Benchmark | Held-out v1 | Horizon suite | Held-out v2 | Held-out v3 | Median latency, 1 CPU thread |
|---|---|---|---|---|---|---|---|---|---|---|
| Shield 2.0 with the large model's earlier version | SDK + own model | 0.924 | – | 0.818 | 0.933 | 0.860 | 0.937 | 0.984 | 0.907 | – |
| Shield 2.0 with Shield's large model | SDK + own model | 0.923 | 0.751 | 0.810 | 0.922 | 0.866 | 0.943 | 0.982 | 0.902 | 671 ms on 4 threads |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | SDK + own models | 0.911 | 0.709 | 0.800 | 0.926 | 0.821 | 0.937 | 0.982 | 0.887 | 43 ms |
| Shield 2.0 with Shield's model | SDK + own model | 0.889 | 0.669 | 0.771 | 0.910 | 0.786 | 0.921 | 0.973 | 0.857 | 25 ms |
| Shield 2.0 with Shield's model, int8 | SDK + own model | 0.885 | 0.638 | 0.770 | 0.906 | 0.783 | 0.919 | 0.972 | 0.845 | 17 ms |
| Patronus Wolf Defender | open model | 0.838 | 0.671 | 0.753 | 0.859 | 0.770 | 0.850 | 0.933 | 0.776 | 72 ms |
| Horizon-Labs prompt injection guard large | open model | 0.823 | 0.708 | 0.759 | 0.822 | 0.763 | 0.896 | 0.860 | 0.772 | 367 ms |
| Patronus Wolf Defender small | open model | 0.822 | – | 0.746 | 0.842 | 0.769 | 0.838 | 0.897 | 0.764 | – |
| Sheltron prompt guard 68M | open model | 0.806 | 0.697 | 0.749 | 0.821 | 0.818 | 0.824 | 0.915 | 0.653 | 47 ms |
| PIGuard | open model | 0.786 | 0.692 | 0.708 | 0.815 | 0.686 | 0.805 | 0.902 | 0.723 | 107 ms |
| Shield 2.0 without the model | SDK | 0.777 | 0.627 | 0.694 | 0.850 | 0.734 | 0.841 | 0.790 | 0.670 | 0.05 ms |
| Amazon Bedrock Guardrails, prompt attack filter | paid API | 0.768 | 0.640 | 0.761 | 0.748 | 0.723 | 0.809 | 0.820 | 0.739 | – |
| @stackone/defender | npm library | 0.763 | 0.632 | 0.677 | 0.760 | 0.711 | 0.708 | 0.858 | 0.781 | – |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | open model | 0.753 | 0.624 | 0.728 | 0.731 | 0.896 | 0.734 | 0.835 | 0.566 | 106 ms |
| TestSavant prompt injection defender large | open model | 0.752 | 0.590 | 0.702 | 0.762 | 0.689 | 0.673 | 0.976 | 0.663 | – |
| NeuralTrust prompt guard small | open model | 0.741 | 0.589 | 0.733 | 0.737 | 0.624 | 0.714 | 0.937 | 0.694 | – |
| Azure AI Prompt Shields | paid API | 0.740 | 0.626 | 0.615 | 0.739 | 0.700 | 0.753 | 0.813 | 0.696 | – |
| Horizon-Labs prompt injection guard base | open model | 0.721 | 0.655 | 0.610 | 0.718 | 0.667 | 0.794 | 0.755 | 0.673 | 104 ms |
| Meta Llama Prompt Guard 2 86M | open model | 0.715 | 0.595 | 0.611 | 0.697 | 0.627 | 0.695 | 0.885 | 0.668 | 108 ms |
| Prismor prompt-guard 1.5B | open LLM | 0.707 | – | 0.758 | 0.774 | 0.692 | 0.733 | 0.649 | 0.690 | – |
| Horizon-Labs prompt injection guard small | open model | 0.694 | – | 0.492 | 0.731 | 0.640 | 0.757 | 0.726 | 0.615 | 44 ms |
| Meta Llama Prompt Guard 2 22M | open model | 0.692 | 0.560 | 0.579 | 0.685 | 0.641 | 0.679 | 0.821 | 0.632 | 35 ms |
| Jailbreak-Detector-Large (madhurjindal) | open model | 0.689 | 0.607 | 0.658 | 0.708 | 0.603 | 0.671 | 0.789 | 0.672 | – |
| Qwen3Guard-Gen 4B (Jailbreak category) | open LLM | 0.652 | – | 0.596 | 0.676 | 0.617 | 0.694 | 0.680 | 0.593 | – |
| @lov3kaizen/agentsea-guardrails | npm library | 0.637 | 0.512 | 0.417 | 0.633 | 0.636 | 0.621 | 0.733 | 0.561 | – |
| @andersmyrmel/vard | npm library | 0.636 | 0.509 | 0.415 | 0.614 | 0.639 | 0.613 | 0.766 | 0.549 | – |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | open LLM | 0.632 | – | 0.543 | 0.659 | 0.598 | 0.664 | 0.673 | 0.567 | – |
| llm-trust-guard | npm library | 0.632 | 0.551 | 0.421 | 0.651 | 0.662 | 0.635 | 0.670 | 0.542 | – |
| deepset DeBERTa v3 base injection | open model | 0.631 | 0.559 | 0.739 | 0.590 | 0.715 | 0.673 | 0.600 | 0.577 | – |
| llm-guard (npm) | npm library | 0.598 | 0.524 | 0.396 | 0.614 | 0.631 | 0.602 | 0.577 | 0.564 | – |
| zambo-prompt-shield | npm library | 0.597 | 0.509 | 0.394 | 0.621 | 0.643 | 0.614 | 0.563 | 0.543 | – |
| llm-firewall | npm library | 0.591 | 0.508 | 0.368 | 0.618 | 0.632 | 0.604 | 0.568 | 0.534 | – |
| IBM Granite Guardian 4.1 8B (jailbreak) | open LLM | 0.590 | – | 0.442 | 0.597 | 0.615 | 0.606 | 0.560 | 0.573 | – |
| @zeroleaks/shield 1.2.1 | npm library | 0.589 | 0.521 | 0.422 | 0.610 | 0.631 | 0.611 | 0.568 | 0.524 | – |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | open model | 0.578 | 0.500 | 0.337 | 0.641 | 0.583 | 0.606 | 0.543 | 0.519 | – |
| llm-prompt-guard | npm library | 0.577 | 0.509 | 0.383 | 0.597 | 0.633 | 0.598 | 0.522 | 0.539 | – |
| Guardion ModernGuard-1 | open model | 0.574 | – | 0.698 | 0.617 | 0.490 | 0.507 | 0.624 | 0.632 | – |
| vLLM Semantic Router Vela 307M guard | open model | 0.551 | – | 0.440 | 0.611 | 0.595 | 0.591 | 0.424 | 0.533 | – |
| @bastionsoft/prompt-protection | npm library | – | – | – | 0.813 | 0.661 | – | 0.825 | – | – |
| GPT-6 Sol as a judge | paid LLM | – | 0.801 | 0.808 | 0.901 | 0.806 | – | 0.747 | 0.844 | – |
| fmops DistilBERT prompt injection | open model | – | – | – | 0.579 | 0.716 | – | – | – | – |
| Meta Prompt Guard 86M (jailbreak score) | open model | – | – | – | 0.700 | 0.675 | – | – | – | – |
| Meta Prompt Guard 86M (injection or jailbreak score) | open model | – | – | – | 0.510 | 0.386 | – | – | – | – |
"Mean of the five groups" averages the development groups' balanced accuracy for detectors that ran on all of them; held-out v4 is reported on its own. Latency is the median over the latency test, with ONNX Runtime, except for the large model and tiered, which were timed through the SDK in a later run in which the default model took 38ms on one thread; a dash means not measured. GPT-6 Sol and @bastionsoft/prompt-protection did not run on every group: the judge costs a model call per text, and @bastionsoft/prompt-protection was stopped after 30 minutes on v3's web pages. fmops' model and Meta's Prompt Guard 86M ran only on the benchmark sets and held-out v1, in the earlier version of this benchmark. GPT-6 Sol did not run on the Horizon suite.
The groups
| Group | Sets | What it measures | Used to build Shield? |
|---|---|---|---|
| Benchmark sets | 13 | Direct and indirect injection, jailbreaks, multilingual attacks, over-defense | Yes: nine share a source with the training data, and all 13 guided development |
| Held-out v1 | 6 | CyberSecEval, Open-Prompt-Injection, and benign prompts that sound risky | Used to compare candidate models |
| Horizon-Labs eval suite | 16 | The suite Horizon-Labs published with its guard models in September 2026 | 14 sets used to compare candidate models; five share a source with the training data |
| Held-out v2 | 4 | JailbreakBench and Salad-Data jailbreaks, CyberSecEval's benign security prompts, Alpaca | Used to choose the large model |
| Held-out v3 | 8 | Benchmarks published since mid-2025: BrowseSafe, Lakera's b3, PIArena, JailbreaksOverTime, a curated jailbreak set, and two false-positive sets | Used to choose the large model; BrowseSafe results also informed a windowing change |
| Held-out v5 | 4 | Agent tool results (AgentDojo, AgentDyn), web-agent injections (WAInjectBench), poisoned MCP tool descriptions (MCPTox), tool-use conversations (MCIP) | Reported as a later evaluation; no training-source isolation guarantee |
| Held-out v4 | 9 | Agent-security datasets, Kubernetes manifests with planted instructions, Tensor Trust, Chinese injections, attacks on agents' secrets, and jailbreak payloads, most published in August and September 2026 | Reported as a later evaluation; no training-source isolation guarantee |
Archived evaluation plans are in the package's benchmark/heldout-plans directory. The benchmark, v1, Horizon-Labs, v2, and v3 groups informed model development or selection. Training-source overlap includes deepset, LLMail, translated injection datasets, and BIPIA-derived data. Filtering duplicate rows does not make different splits, translations, or variants from the same source independent. This archive does not establish that every evaluated dataset or source was excluded from training. Other detectors' training data is not fully public either.
Held-out v5
This group covers instructions planted in tool results, web pages, MCP tool descriptions, and tool-use conversations. It was added after v4 in the archived evaluation. A fifth set, password_eval, tested the large model's context mode; see Shield's large model.
| Set | Source | Rows | Labels |
|---|---|---|---|
agentdojo_tools | Tool results from AgentDojo and AgentDyn runs (ETH Zurich, SaFo-Lab): results carrying an injection, and results from runs without an attack | 1,200 | 600 attacks, 600 benign |
wainject | WAInjectBench's text split: web-agent injections (EIA, VWA-adv, pop-ups, VPI, WASP) and benign web text | 1,123 | 523 attacks, 600 benign |
mcptox | MCPTox: poisoned MCP tool descriptions and clean tools from the same servers | 827 | 485 attacks, 342 benign |
mcip | MCIP-bench's test split: tool-use conversations with injected or misused function calls, and clean ones | 893 | 744 attacks, 149 benign |
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| GPT-6 Sol as a judge | 0.801 | 58.7% | 1.9% |
| Shield 2.0 with Shield's large model | 0.751 | 50.8% | 2.4% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.709 | 50.7% | 7.6% |
| Horizon-Labs prompt injection guard large | 0.708 | 47.0% | 7.9% |
| Sheltron prompt guard 68M | 0.697 | 64.4% | 20.3% |
| PIGuard | 0.692 | 66.3% | 22.5% |
| Patronus Wolf Defender | 0.671 | 34.7% | 1.7% |
| Shield 2.0 with Shield's model | 0.669 | 48.6% | 10.6% |
| Horizon-Labs prompt injection guard base | 0.655 | 41.3% | 8.2% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.640 | 28.0% | 0.5% |
| Shield 2.0 with Shield's model, int8 | 0.638 | 41.5% | 9.5% |
| @stackone/defender | 0.632 | 33.0% | 5.5% |
| Shield 2.0 without the model | 0.627 | 30.2% | 4.7% |
| Azure AI Prompt Shields | 0.626 | 25.5% | 0.7% |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.624 | 44.4% | 16.0% |
| Jailbreak-Detector-Large (madhurjindal) | 0.607 | 30.1% | 5.7% |
| Meta Llama Prompt Guard 2 86M | 0.595 | 27.1% | 2.1% |
| TestSavant prompt injection defender large | 0.590 | 23.0% | 6.4% |
| NeuralTrust prompt guard small | 0.589 | 19.6% | 3.1% |
| Meta Llama Prompt Guard 2 22M | 0.560 | 15.2% | 1.1% |
| deepset DeBERTa v3 base injection | 0.559 | 96.0% | 76.9% |
| llm-trust-guard | 0.551 | 11.6% | 0.9% |
| llm-guard (npm) | 0.524 | 8.3% | 2.2% |
| @zeroleaks/shield 1.2.1 | 0.521 | 4.0% | 0.2% |
| @lov3kaizen/agentsea-guardrails | 0.512 | 2.5% | 0.1% |
| @andersmyrmel/vard | 0.509 | 2.2% | 0.3% |
| llm-prompt-guard | 0.509 | 1.9% | 0.0% |
| zambo-prompt-shield | 0.509 | 1.8% | 0.0% |
| llm-firewall | 0.508 | 1.7% | 0.0% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.500 | 0.0% | 0.0% |
Per set (balanced accuracy):
| Detector | agentdojo_tools | wainject | mcptox | mcip |
|---|---|---|---|---|
| GPT-6 Sol as a judge | 0.982 | 0.711 | 0.977 | 0.534 |
| Shield 2.0 with Shield's large model | 0.882 | 0.648 | 0.922 | 0.553 |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.787 | 0.646 | 0.877 | 0.526 |
| Horizon-Labs prompt injection guard large | 0.861 | 0.553 | 0.926 | 0.493 |
| Sheltron prompt guard 68M | 0.723 | 0.721 | 0.892 | 0.451 |
| PIGuard | 0.807 | 0.638 | 0.788 | 0.535 |
| Patronus Wolf Defender | 0.899 | 0.620 | 0.671 | 0.494 |
| Shield 2.0 with Shield's model | 0.690 | 0.665 | 0.792 | 0.528 |
| Horizon-Labs prompt injection guard base | 0.819 | 0.534 | 0.731 | 0.537 |
| Amazon Bedrock Guardrails, prompt attack filter | 0.909 | 0.550 | 0.595 | 0.505 |
| Shield 2.0 with Shield's model, int8 | 0.634 | 0.655 | 0.748 | 0.513 |
| @stackone/defender | 0.720 | 0.563 | 0.711 | 0.533 |
| Shield 2.0 without the model | 0.671 | 0.533 | 0.792 | 0.514 |
| Azure AI Prompt Shields | 0.854 | 0.530 | 0.623 | 0.497 |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.705 | 0.654 | 0.610 | 0.528 |
| Jailbreak-Detector-Large (madhurjindal) | 0.683 | 0.583 | 0.631 | 0.530 |
| Meta Llama Prompt Guard 2 86M | 0.802 | 0.534 | 0.505 | 0.539 |
| TestSavant prompt injection defender large | 0.638 | 0.692 | 0.521 | 0.509 |
| NeuralTrust prompt guard small | 0.591 | 0.556 | 0.716 | 0.493 |
| Meta Llama Prompt Guard 2 22M | 0.690 | 0.523 | 0.512 | 0.515 |
| deepset DeBERTa v3 base injection | 0.573 | 0.661 | 0.503 | 0.500 |
| llm-trust-guard | 0.580 | 0.560 | 0.538 | 0.527 |
| llm-guard (npm) | 0.557 | 0.499 | 0.519 | 0.521 |
| @zeroleaks/shield 1.2.1 | 0.532 | 0.501 | 0.551 | 0.501 |
| @lov3kaizen/agentsea-guardrails | 0.533 | 0.510 | 0.502 | 0.503 |
| @andersmyrmel/vard | 0.532 | 0.499 | 0.500 | 0.506 |
| llm-prompt-guard | 0.533 | 0.500 | 0.501 | 0.502 |
| zambo-prompt-shield | 0.533 | 0.500 | 0.500 | 0.501 |
| llm-firewall | 0.533 | 0.500 | 0.500 | 0.500 |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.500 | 0.500 | 0.500 | 0.500 |
Held-out v4
This archived group combines agent-security datasets, configuration files with planted instructions, and direct attacks. Its historical name does not certify independence from model development or training sources.
| Set | Source | Rows | Labels |
|---|---|---|---|
pp_attacks | promptprotection's agent-security datasets (Sep 2026): attacks | 130 | Attacks |
pp_benign_hard | The same datasets' hard benign prompts, written to look like attacks | 155 | Benign |
pp_agent_flows | The same datasets' agent flows: the texts an agent's tools return in each flow, joined | 91 | 50 attacks, 41 benign |
k8s | Kubernetes manifests with and without a planted instruction (Aug 2026) | 851 | 493 attacks, 358 benign |
tensor_trust | Tensor Trust's hijacking and extraction robustness benchmarks, the attacks | 770 | Attacks |
changemore_zh | ChangeMore prompt injection evaluation, Chinese | 1,963 | 978 attacks, 985 benign |
agent_extraction | Prompts against agents with secrets in their instructions (Apr 2026) | 783 | Attacks |
airline_payloads | Real jailbreak payloads held out from an airline agent's red-team set (Sep 2026) | 158 | Attacks |
probes_mixed | Realistic prompt injections from three sets published in September 2026 | 136 | 115 attacks, 21 benign |
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| Shield 2.0 with the large model's earlier version | 0.818 | 76.1% | 6.4% |
| Shield 2.0 with Shield's large model | 0.810 | 73.5% | 4.4% |
| GPT-6 Sol as a judge | 0.808 | 55.2% | 0.2% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.800 | 68.9% | 4.7% |
| Shield 2.0 with Shield's model | 0.771 | 62.7% | 5.6% |
| Shield 2.0 with Shield's model, int8 | 0.770 | 61.6% | 4.8% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.761 | 55.5% | 2.1% |
| Horizon-Labs prompt injection guard large | 0.759 | 59.6% | 2.6% |
| Prismor prompt-guard 1.5B | 0.758 | 65.7% | 15.6% |
| Patronus Wolf Defender | 0.753 | 54.5% | 3.6% |
| Sheltron prompt guard 68M | 0.749 | 60.1% | 12.2% |
| Patronus Wolf Defender small | 0.746 | 55.4% | 4.3% |
| deepset DeBERTa v3 base injection | 0.739 | 90.3% | 83.1% |
| NeuralTrust prompt guard small | 0.733 | 57.5% | 4.0% |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.728 | 56.4% | 8.7% |
| PIGuard | 0.708 | 53.1% | 9.5% |
| TestSavant prompt injection defender large | 0.702 | 66.4% | 37.0% |
| Guardion ModernGuard-1 | 0.698 | 92.7% | 77.8% |
| Shield 2.0 without the model | 0.694 | 46.9% | 6.1% |
| @stackone/defender | 0.677 | 44.6% | 6.5% |
| Jailbreak-Detector-Large (madhurjindal) | 0.658 | 55.9% | 8.9% |
| Azure AI Prompt Shields | 0.615 | 35.1% | 1.7% |
| Meta Llama Prompt Guard 2 86M | 0.611 | 40.3% | 2.4% |
| Horizon-Labs prompt injection guard base | 0.610 | 44.3% | 4.5% |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.596 | 26.7% | 0.6% |
| Meta Llama Prompt Guard 2 22M | 0.579 | 37.4% | 2.4% |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.543 | 22.6% | 0.4% |
| Horizon-Labs prompt injection guard small | 0.492 | 21.3% | 1.0% |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.442 | 25.4% | 16.7% |
| vLLM Semantic Router Vela 307M guard | 0.440 | 19.8% | 11.3% |
| @zeroleaks/shield 1.2.1 | 0.422 | 11.6% | 1.0% |
| llm-trust-guard | 0.421 | 15.6% | 3.8% |
| @lov3kaizen/agentsea-guardrails | 0.417 | 14.2% | 1.5% |
| @andersmyrmel/vard | 0.415 | 16.2% | 2.4% |
| llm-guard (npm) | 0.396 | 15.6% | 5.3% |
| zambo-prompt-shield | 0.394 | 10.5% | 1.6% |
| llm-prompt-guard | 0.383 | 7.4% | 0.8% |
| llm-firewall | 0.368 | 6.8% | 1.9% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.337 | 1.1% | 0.0% |
Per set (balanced accuracy):
| Detector | pp_attacks | pp_benign_hard | pp_agent_flows | k8s | tensor_trust | changemore_zh | agent_extraction | airline_payloads | probes_mixed |
|---|---|---|---|---|---|---|---|---|---|
| Shield 2.0 with the large model's earlier version | 1.000 | 0.794 | 0.917 | 0.925 | 0.982 | 0.937 | 0.206 | 0.633 | 0.972 |
| Shield 2.0 with Shield's large model | 0.985 | 0.819 | 0.899 | 0.923 | 0.973 | 0.935 | 0.152 | 0.601 | 1.000 |
| GPT-6 Sol as a judge | 0.977 | 1.000 | 0.950 | 0.933 | 0.871 | 0.679 | 0.074 | 0.797 | 0.991 |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.969 | 0.813 | 0.931 | 0.907 | 0.968 | 0.873 | 0.115 | 0.627 | 1.000 |
| Shield 2.0 with Shield's model | 0.962 | 0.806 | 0.867 | 0.851 | 0.949 | 0.801 | 0.092 | 0.633 | 0.976 |
| Shield 2.0 with Shield's model, int8 | 0.946 | 0.813 | 0.891 | 0.849 | 0.935 | 0.797 | 0.092 | 0.614 | 0.991 |
| Amazon Bedrock Guardrails, prompt attack filter | 0.854 | 0.923 | 0.908 | 0.815 | 0.745 | 0.740 | 0.243 | 0.639 | 0.978 |
| Horizon-Labs prompt injection guard large | 0.892 | 0.884 | 0.909 | 0.917 | 0.721 | 0.788 | 0.211 | 0.551 | 0.954 |
| Prismor prompt-guard 1.5B | 0.869 | 0.839 | 0.946 | 0.757 | 0.584 | 0.849 | 0.390 | 0.652 | 0.939 |
| Patronus Wolf Defender | 0.892 | 0.845 | 0.903 | 0.694 | 0.926 | 0.781 | 0.029 | 0.728 | 0.978 |
| Sheltron prompt guard 68M | 0.931 | 0.645 | 0.831 | 0.920 | 0.944 | 0.678 | 0.098 | 0.696 | 0.996 |
| Patronus Wolf Defender small | 0.846 | 0.781 | 0.873 | 0.699 | 0.930 | 0.782 | 0.061 | 0.747 | 0.991 |
| deepset DeBERTa v3 base injection | 0.992 | 0.555 | 0.524 | 0.500 | 0.969 | 0.580 | 0.600 | 1.000 | 0.929 |
| NeuralTrust prompt guard small | 0.746 | 0.916 | 0.823 | 0.728 | 0.686 | 0.843 | 0.235 | 0.722 | 0.902 |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.831 | 0.665 | 0.745 | 0.843 | 0.971 | 0.657 | 0.144 | 0.747 | 0.946 |
| PIGuard | 0.731 | 0.710 | 0.766 | 0.880 | 0.832 | 0.660 | 0.040 | 0.772 | 0.983 |
| TestSavant prompt injection defender large | 0.685 | 0.658 | 0.928 | 0.600 | 0.794 | 0.730 | 0.388 | 0.639 | 0.896 |
| Guardion ModernGuard-1 | 0.900 | 0.187 | 0.594 | 0.438 | 0.932 | 0.633 | 0.903 | 0.968 | 0.729 |
| Shield 2.0 without the model | 0.808 | 0.787 | 0.783 | 0.812 | 0.878 | 0.600 | 0.047 | 0.544 | 0.983 |
| @stackone/defender | 0.754 | 0.632 | 0.797 | 0.717 | 0.834 | 0.614 | 0.103 | 0.696 | 0.941 |
| Jailbreak-Detector-Large (madhurjindal) | 0.400 | 0.955 | 0.598 | 0.632 | 0.722 | 0.892 | 0.130 | 0.873 | 0.722 |
| Azure AI Prompt Shields | 0.492 | 0.897 | 0.720 | 0.704 | 0.652 | 0.621 | 0.004 | 0.506 | 0.935 |
| Meta Llama Prompt Guard 2 86M | 0.323 | 0.839 | 0.603 | 0.702 | 0.784 | 0.666 | 0.013 | 0.652 | 0.917 |
| Horizon-Labs prompt injection guard base | 0.346 | 0.948 | 0.648 | 0.871 | 0.655 | 0.689 | 0.032 | 0.475 | 0.830 |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.554 | 0.968 | 0.696 | 0.593 | 0.582 | 0.563 | 0.005 | 0.614 | 0.791 |
| Meta Llama Prompt Guard 2 22M | 0.200 | 0.826 | 0.563 | 0.734 | 0.762 | 0.620 | 0.027 | 0.595 | 0.883 |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.369 | 0.968 | 0.610 | 0.612 | 0.503 | 0.547 | 0.006 | 0.614 | 0.657 |
| Horizon-Labs prompt injection guard small | 0.323 | 0.942 | 0.618 | 0.562 | 0.322 | 0.634 | 0.006 | 0.228 | 0.791 |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.069 | 0.929 | 0.568 | 0.860 | 0.173 | 0.467 | 0.176 | 0.177 | 0.559 |
| vLLM Semantic Router Vela 307M guard | 0.269 | 0.929 | 0.510 | 0.472 | 0.360 | 0.525 | 0.140 | 0.209 | 0.548 |
| @zeroleaks/shield 1.2.1 | 0.085 | 0.935 | 0.573 | 0.587 | 0.255 | 0.511 | 0.004 | 0.120 | 0.726 |
| llm-trust-guard | 0.138 | 0.806 | 0.581 | 0.592 | 0.187 | 0.574 | 0.029 | 0.114 | 0.765 |
| @lov3kaizen/agentsea-guardrails | 0.062 | 0.955 | 0.503 | 0.584 | 0.244 | 0.546 | 0.064 | 0.146 | 0.652 |
| @andersmyrmel/vard | 0.069 | 0.948 | 0.493 | 0.562 | 0.334 | 0.548 | 0.092 | 0.038 | 0.652 |
| llm-guard (npm) | 0.038 | 0.729 | 0.491 | 0.605 | 0.223 | 0.527 | 0.129 | 0.082 | 0.735 |
| zambo-prompt-shield | 0.023 | 0.865 | 0.523 | 0.569 | 0.240 | 0.518 | 0.008 | 0.101 | 0.696 |
| llm-prompt-guard | 0.054 | 0.948 | 0.526 | 0.539 | 0.206 | 0.504 | 0.003 | 0.013 | 0.652 |
| llm-firewall | 0.038 | 0.826 | 0.506 | 0.533 | 0.081 | 0.536 | 0.010 | 0.127 | 0.652 |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.000 | 1.000 | 0.500 | 0.500 | 0.000 | 0.518 | 0.000 | 0.013 | 0.500 |
Shield's large model
The large model is Qwen3-1.7B fine-tuned as a classifier (LoRA, rank 32) on the default model's training data plus about 22,500 additional texts: WildJailbreak's benign requests that sound harmful, its harmful requests wrapped in jailbreak tactics, PHTest's pseudo-harmful prompts, and CoCoNot's contrast set. That version was scored on v4. After v4, a second LoRA trained it further on 35,000 texts from the same data, 20,700 benign texts that look like attacks (real user prompts from LMSYS Toxic-Chat, FalseReject, system prompts, Kubernetes manifests, security Q&A, Chinese Q&A), and 25,000 system prompts paired with requests that follow or break them (SystemCheck, SPML). The shipped version uses 4-bit weights in a 1.08GB file. Its v5 scores come from that file (0.751; the full-precision weights score 0.726, and the two differ on 5% of v5's verdicts). Scores for the other groups come from the full-precision weights.
- Size. On the default model's data, Qwen3 0.6B, 1.7B, 4B, and 8B scored 0.891, 0.915, 0.916, and 0.912 on the development groups, so the 1.7B model ships.
- Data. Those models flagged 15-18% of OR-Bench's benign prompts that sound harmful, against the default model's 4%. The added data brought that to 5% and the development score to 0.924, the version scored on v4. Two variants picked what to add: labeling plain harmful requests benign as well cut recall on instructions planted in documents.
- Tiered.
tiered()sends the inputs the default model scores from 0.01 up to 0.97 to the large model, 14% of them here. It scores 0.911 on the development groups and 0.800 on v4. - Context mode, not shipped. We also trained the large model to read the agent's instructions before each input, on system prompts paired with requests that follow or break them (SystemCheck, SPML). On held-out v5's
password_eval(PasswordEval: agents whose instructions hold a password and a secret, and requests for the secret with and without the password), it flagged 86.9% of the requests that must be refused but also 98.0% of those that gave the right password: it learned that asking for the secret is suspect, not who may ask. Detectors that read only the request can't separate the two by design and scored near chance. The option isn't in the SDK. - Not shipped. Training further on the benign look-alikes without the system-prompt pairs cut false positives but also recall, and scored lower on the development groups. Distilling the large model into the default model scored 0.884 and 0.886 (the second time with 83,000 more texts scored by the large model), below the default model's 0.889, and four runs of the default model on the added data scored 0.883 to 0.889, within the 0.007 two seeds differ by, so the default model is unchanged.
Held-out v3
This group collects benchmarks published since mid-2025. Its scores were used during model development:
| Set | Source | Rows | Labels |
|---|---|---|---|
browsesafe | BrowseSafe-Bench (Perplexity, Nov 2025): real web pages with injections for browser agents, and pages with look-alike text meant for people | 1,000 | 500 attacks, 500 benign |
b3_agent | Lakera's b3 benchmark (Nov 2025): attacks from the Gandalf Agent Breaker challenge against agent apps | 187 | Attacks |
piarena | PIArena (Horizon-Labs' suite, Sep 2026): tasks injected into QA, summarization, and RAG contexts | 857 | 675 attacks, 182 benign |
jot | JailbreaksOverTime (Oct 2025): jailbreaks collected over 10 months, and real user prompts | 1,713 | 713 attacks, 1,000 benign |
simsonsun | Curated jailbreak dataset (Jun 2025) | 1,948 | Attacks |
guardrail_hn | Guardrail hard negatives test split (Aug 2026): benign and attack texts that share the same surface | 135 | 96 attacks, 39 benign |
operant_fp | Operant AI false-positive test data (Jul 2026), several languages | 562 | Benign |
hirundo_benign | Hirundo's benign prompts from a prompt injection benchmark (Jun 2026) | 1,000 | Benign |
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| Shield 2.0 with the large model's earlier version | 0.907 | 89.1% | 3.0% |
| Shield 2.0 with Shield's large model | 0.902 | 89.0% | 2.8% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.887 | 87.1% | 3.7% |
| Shield 2.0 with Shield's model | 0.857 | 82.3% | 5.5% |
| Shield 2.0 with Shield's model, int8 | 0.845 | 80.3% | 4.8% |
| GPT-6 Sol as a judge | 0.844 | 67.4% | 5.0% |
| @stackone/defender | 0.781 | 71.1% | 9.7% |
| Patronus Wolf Defender | 0.776 | 66.3% | 5.4% |
| Horizon-Labs prompt injection guard large | 0.772 | 78.0% | 20.8% |
| Patronus Wolf Defender small | 0.764 | 63.9% | 4.9% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.739 | 58.8% | 3.7% |
| PIGuard | 0.723 | 75.8% | 27.5% |
| Azure AI Prompt Shields | 0.696 | 41.8% | 1.6% |
| NeuralTrust prompt guard small | 0.694 | 61.8% | 10.4% |
| Prismor prompt-guard 1.5B | 0.690 | 75.7% | 34.7% |
| Horizon-Labs prompt injection guard base | 0.673 | 62.8% | 19.0% |
| Jailbreak-Detector-Large (madhurjindal) | 0.672 | 68.9% | 32.6% |
| Shield 2.0 without the model | 0.670 | 48.6% | 15.8% |
| Meta Llama Prompt Guard 2 86M | 0.668 | 50.1% | 6.5% |
| TestSavant prompt injection defender large | 0.663 | 77.7% | 32.6% |
| Sheltron prompt guard 68M | 0.653 | 72.8% | 32.5% |
| Meta Llama Prompt Guard 2 22M | 0.632 | 40.3% | 5.8% |
| Guardion ModernGuard-1 | 0.632 | 90.4% | 68.5% |
| Horizon-Labs prompt injection guard small | 0.615 | 33.8% | 2.5% |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.593 | 13.1% | 1.2% |
| deepset DeBERTa v3 base injection | 0.577 | 99.8% | 80.6% |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.573 | 20.1% | 7.2% |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.567 | 10.1% | 1.4% |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.566 | 54.1% | 34.6% |
| llm-guard (npm) | 0.564 | 23.0% | 6.8% |
| @lov3kaizen/agentsea-guardrails | 0.561 | 22.3% | 7.2% |
| @andersmyrmel/vard | 0.549 | 36.4% | 21.8% |
| zambo-prompt-shield | 0.543 | 14.1% | 4.5% |
| llm-trust-guard | 0.542 | 23.1% | 17.1% |
| llm-prompt-guard | 0.539 | 6.5% | 0.6% |
| llm-firewall | 0.534 | 8.6% | 0.5% |
| vLLM Semantic Router Vela 307M guard | 0.533 | 22.6% | 17.9% |
| @zeroleaks/shield 1.2.1 | 0.524 | 10.4% | 5.6% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.519 | 5.3% | 0.1% |
Per set (balanced accuracy):
| Detector | browsesafe | b3_agent | piarena | jot | simsonsun | guardrail_hn | operant_fp | hirundo_benign |
|---|---|---|---|---|---|---|---|---|
| Shield 2.0 with the large model's earlier version | 0.865 | 0.674 | 0.928 | 0.970 | 0.931 | 0.974 | 0.947 | 0.965 |
| Shield 2.0 with Shield's large model | 0.868 | 0.615 | 0.935 | 0.978 | 0.921 | 0.987 | 0.941 | 0.969 |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.878 | 0.567 | 0.870 | 0.983 | 0.917 | 0.987 | 0.943 | 0.953 |
| Shield 2.0 with Shield's model | 0.863 | 0.503 | 0.787 | 0.973 | 0.875 | 0.987 | 0.936 | 0.937 |
| Shield 2.0 with Shield's model, int8 | 0.871 | 0.460 | 0.751 | 0.964 | 0.870 | 0.949 | 0.950 | 0.941 |
| GPT-6 Sol as a judge | 0.698 | 0.738 | 0.999 | 0.923 | 0.526 | 1.000 | 0.970 | 0.901 |
| @stackone/defender | 0.677 | 0.545 | 0.680 | 0.902 | 0.822 | 0.753 | 0.945 | 0.922 |
| Patronus Wolf Defender | 0.642 | 0.529 | 0.599 | 0.923 | 0.816 | 0.815 | 0.900 | 0.986 |
| Horizon-Labs prompt injection guard large | 0.535 | 0.396 | 0.919 | 0.953 | 0.698 | 0.927 | 0.934 | 0.816 |
| Patronus Wolf Defender small | 0.649 | 0.455 | 0.593 | 0.926 | 0.771 | 0.789 | 0.945 | 0.985 |
| Amazon Bedrock Guardrails, prompt attack filter | 0.747 | 0.310 | 0.627 | 0.883 | 0.668 | 0.788 | 0.957 | 0.933 |
| PIGuard | 0.508 | 0.455 | 0.838 | 0.856 | 0.745 | 0.745 | 0.874 | 0.760 |
| Azure AI Prompt Shields | 0.672 | 0.299 | 0.736 | 0.843 | 0.325 | 0.745 | 0.980 | 0.970 |
| NeuralTrust prompt guard small | 0.649 | 0.198 | 0.541 | 0.980 | 0.736 | 0.716 | 0.870 | 0.859 |
| Prismor prompt-guard 1.5B | 0.492 | 0.679 | 0.696 | 0.732 | 0.874 | 0.479 | 0.927 | 0.640 |
| Horizon-Labs prompt injection guard base | 0.557 | 0.187 | 0.869 | 0.826 | 0.560 | 0.602 | 0.943 | 0.843 |
| Jailbreak-Detector-Large (madhurjindal) | 0.497 | 0.433 | 0.510 | 0.793 | 0.865 | 0.737 | 0.957 | 0.581 |
| Shield 2.0 without the model | 0.591 | 0.273 | 0.510 | 0.816 | 0.496 | 0.872 | 0.945 | 0.860 |
| Meta Llama Prompt Guard 2 86M | 0.677 | 0.182 | 0.538 | 0.907 | 0.588 | 0.612 | 0.988 | 0.856 |
| TestSavant prompt injection defender large | 0.640 | 0.439 | 0.754 | 0.867 | 0.876 | 0.552 | 0.580 | 0.592 |
| Sheltron prompt guard 68M | 0.501 | 0.358 | 0.748 | 0.882 | 0.722 | 0.655 | 0.573 | 0.783 |
| Meta Llama Prompt Guard 2 22M | 0.649 | 0.134 | 0.531 | 0.833 | 0.451 | 0.571 | 0.986 | 0.906 |
| Guardion ModernGuard-1 | 0.500 | 0.909 | 0.657 | 0.666 | 0.944 | 0.615 | 0.468 | 0.296 |
| Horizon-Labs prompt injection guard small | 0.618 | 0.064 | 0.598 | 0.745 | 0.383 | 0.578 | 0.984 | 0.948 |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.578 | 0.160 | 0.506 | 0.670 | 0.049 | 0.792 | 0.993 | 1.000 |
| deepset DeBERTa v3 base injection | 0.500 | 0.984 | 0.547 | 0.625 | 0.998 | 0.500 | 0.203 | 0.257 |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.549 | 0.198 | 0.612 | 0.568 | 0.198 | 0.610 | 0.945 | 0.904 |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.582 | 0.171 | 0.504 | 0.616 | 0.034 | 0.638 | 0.993 | 0.999 |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.500 | 0.332 | 0.618 | 0.801 | 0.493 | 0.566 | 0.468 | 0.754 |
| llm-guard (npm) | 0.656 | 0.070 | 0.525 | 0.671 | 0.192 | 0.501 | 0.989 | 0.911 |
| @lov3kaizen/agentsea-guardrails | 0.663 | 0.134 | 0.504 | 0.576 | 0.300 | 0.510 | 1.000 | 0.801 |
| @andersmyrmel/vard | 0.520 | 0.043 | 0.505 | 0.654 | 0.400 | 0.505 | 0.998 | 0.764 |
| zambo-prompt-shield | 0.628 | 0.070 | 0.511 | 0.574 | 0.092 | 0.477 | 1.000 | 0.992 |
| llm-trust-guard | 0.502 | 0.102 | 0.554 | 0.586 | 0.111 | 0.526 | 0.982 | 0.969 |
| llm-prompt-guard | 0.674 | 0.080 | 0.502 | 0.519 | 0.017 | 0.526 | 1.000 | 0.993 |
| llm-firewall | 0.645 | 0.048 | 0.500 | 0.601 | 0.027 | 0.454 | 1.000 | 0.994 |
| vLLM Semantic Router Vela 307M guard | 0.492 | 0.267 | 0.515 | 0.564 | 0.155 | 0.484 | 0.888 | 0.901 |
| @zeroleaks/shield 1.2.1 | 0.552 | 0.021 | 0.507 | 0.560 | 0.056 | 0.510 | 1.000 | 0.985 |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.500 | 0.000 | 0.500 | 0.646 | 0.004 | 0.500 | 1.000 | 1.000 |
The model was chosen and scored on v3 once, at 0.792. It flagged 84% of BrowseSafe's benign pages: its training had shown it only the first 256 tokens of each BrowseSafe training page, so it never saw where on the page an injection or a look-alike was. We retrained with each training page split into windows, which brought those false positives to 10%, and ship that model. That change was made after seeing v3, so the v3 numbers above are not a clean test for BrowseSafe; the other seven sets did not inform any change, and Shield's score on them is 0.857.
Held-out v2
| Set | Source | Rows | Labels |
|---|---|---|---|
jbb | JailbreakBench artifacts: PAIR, GCG, and random-search jailbreaks (MIT) | 996 | Attacks |
salad_attack | Salad-Data's attack-enhanced set, 1,000 sampled (Apache-2.0) | 971 | Attacks |
cse_frr | CyberSecEval's benign prompts that sound like cyberattack requests (MIT) | 749 | Benign |
alpaca | Alpaca instructions with their inputs, 1,000 sampled (eval only) | 1,000 | Benign |
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| Shield 2.0 with the large model's earlier version | 0.984 | 98.0% | 1.1% |
| Shield 2.0 with Shield's large model | 0.982 | 96.9% | 0.4% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.982 | 96.6% | 0.3% |
| TestSavant prompt injection defender large | 0.976 | 97.3% | 1.9% |
| Shield 2.0 with Shield's model | 0.973 | 95.0% | 0.5% |
| Shield 2.0 with Shield's model, int8 | 0.972 | 95.0% | 0.5% |
| NeuralTrust prompt guard small | 0.937 | 88.8% | 1.2% |
| Patronus Wolf Defender | 0.933 | 87.2% | 0.5% |
| Sheltron prompt guard 68M | 0.915 | 83.3% | 0.3% |
| PIGuard | 0.902 | 81.9% | 1.3% |
| Patronus Wolf Defender small | 0.897 | 80.7% | 1.2% |
| Meta Llama Prompt Guard 2 86M | 0.885 | 78.1% | 1.0% |
| Horizon-Labs prompt injection guard large | 0.860 | 73.1% | 1.4% |
| @stackone/defender | 0.858 | 79.7% | 7.1% |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.835 | 67.0% | 0.0% |
| @bastionsoft/prompt-protection | 0.825 | 65.0% | 0.3% |
| Meta Llama Prompt Guard 2 22M | 0.821 | 67.6% | 3.0% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.820 | 65.8% | 1.4% |
| Azure AI Prompt Shields | 0.813 | 63.5% | 0.8% |
| Shield 2.0 without the model | 0.790 | 58.7% | 0.8% |
| Jailbreak-Detector-Large (madhurjindal) | 0.789 | 96.4% | 33.0% |
| @andersmyrmel/vard | 0.766 | 53.4% | 0.3% |
| Horizon-Labs prompt injection guard base | 0.755 | 51.9% | 1.1% |
| GPT-6 Sol as a judge | 0.747 | 49.3% | 0.1% |
| @lov3kaizen/agentsea-guardrails | 0.733 | 46.8% | 0.1% |
| Horizon-Labs prompt injection guard small | 0.726 | 45.7% | 0.7% |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.680 | 35.8% | 0.1% |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.673 | 34.3% | 0.0% |
| llm-trust-guard | 0.670 | 34.7% | 0.7% |
| Prismor prompt-guard 1.5B | 0.649 | 61.0% | 26.6% |
| Guardion ModernGuard-1 | 0.624 | 99.2% | 70.7% |
| deepset DeBERTa v3 base injection | 0.600 | 98.9% | 75.8% |
| llm-guard (npm) | 0.577 | 18.6% | 3.1% |
| llm-firewall | 0.568 | 13.5% | 0.0% |
| @zeroleaks/shield 1.2.1 | 0.568 | 13.7% | 0.2% |
| zambo-prompt-shield | 0.563 | 12.4% | 0.0% |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.560 | 19.2% | 7.9% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.543 | 8.4% | 0.0% |
| llm-prompt-guard | 0.522 | 4.3% | 0.0% |
| vLLM Semantic Router Vela 307M guard | 0.424 | 28.1% | 38.9% |
TestSavant's model is slightly ahead on v2 but flags 16% of the benchmark sets' benign text and 42% of the Horizon suite's.
Benchmark sets
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| Shield 2.0 with the large model's earlier version | 0.933 | 86.4% | 2.5% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.926 | 85.4% | 2.1% |
| Shield 2.0 with Shield's large model | 0.922 | 84.3% | 1.6% |
| Shield 2.0 with Shield's model | 0.910 | 82.9% | 2.6% |
| Shield 2.0 with Shield's model, int8 | 0.906 | 80.8% | 2.1% |
| GPT-6 Sol as a judge | 0.901 | 75.4% | 3.2% |
| Patronus Wolf Defender | 0.859 | 79.0% | 6.0% |
| Shield 2.0 without the model | 0.850 | 70.5% | 1.8% |
| Patronus Wolf Defender small | 0.842 | 76.4% | 6.4% |
| Horizon-Labs prompt injection guard large | 0.822 | 67.6% | 2.9% |
| Sheltron prompt guard 68M | 0.821 | 73.5% | 5.3% |
| PIGuard | 0.815 | 64.9% | 5.7% |
| @bastionsoft/prompt-protection | 0.813 | 56.0% | 1.2% |
| Prismor prompt-guard 1.5B | 0.774 | 73.4% | 17.5% |
| TestSavant prompt injection defender large | 0.762 | 69.9% | 16.3% |
| @stackone/defender | 0.760 | 64.8% | 18.0% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.748 | 48.8% | 3.7% |
| Azure AI Prompt Shields | 0.739 | 44.4% | 2.7% |
| NeuralTrust prompt guard small | 0.737 | 43.3% | 3.0% |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.731 | 58.9% | 9.3% |
| Horizon-Labs prompt injection guard small | 0.731 | 41.9% | 0.7% |
| Horizon-Labs prompt injection guard base | 0.718 | 52.2% | 3.4% |
| Jailbreak-Detector-Large (madhurjindal) | 0.708 | 52.3% | 16.0% |
| Meta Prompt Guard 86M (jailbreak score) | 0.700 | 40.6% | 6.4% |
| Meta Llama Prompt Guard 2 86M | 0.697 | 40.8% | 7.1% |
| Meta Llama Prompt Guard 2 22M | 0.685 | 36.9% | 6.1% |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.676 | 29.7% | 3.2% |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.659 | 24.5% | 3.3% |
| llm-trust-guard | 0.651 | 20.8% | 2.6% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.641 | 15.9% | 0.1% |
| @lov3kaizen/agentsea-guardrails | 0.633 | 19.0% | 2.7% |
| zambo-prompt-shield | 0.621 | 11.5% | 0.6% |
| llm-firewall | 0.618 | 11.6% | 0.5% |
| Guardion ModernGuard-1 | 0.617 | 90.8% | 55.6% |
| llm-guard (npm) | 0.614 | 15.4% | 5.0% |
| @andersmyrmel/vard | 0.614 | 16.9% | 5.8% |
| vLLM Semantic Router Vela 307M guard | 0.611 | 20.5% | 11.5% |
| @zeroleaks/shield 1.2.1 | 0.610 | 8.7% | 0.8% |
| llm-prompt-guard | 0.597 | 5.7% | 1.6% |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.597 | 11.8% | 6.7% |
| deepset DeBERTa v3 base injection | 0.590 | 96.8% | 67.7% |
| fmops DistilBERT prompt injection | 0.579 | 93.8% | 67.4% |
| Meta Prompt Guard 86M (injection or jailbreak score) | 0.510 | 78.7% | 76.5% |
These 13 sets favor Shield: nine share a source with its training data, with duplicates removed, and they guided its development. Their contents are listed in The 13 benchmark sets.
Held-out v1
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.896 | 70.2% | 2.8% |
| Shield 2.0 with Shield's large model | 0.866 | 69.7% | 1.0% |
| Shield 2.0 with the large model's earlier version | 0.860 | 71.0% | 3.6% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.821 | 57.8% | 1.8% |
| Sheltron prompt guard 68M | 0.818 | 64.2% | 5.3% |
| GPT-6 Sol as a judge | 0.806 | 53.6% | 0.0% |
| Shield 2.0 with Shield's model | 0.786 | 51.0% | 2.8% |
| Shield 2.0 with Shield's model, int8 | 0.783 | 49.2% | 2.2% |
| Patronus Wolf Defender | 0.770 | 51.3% | 5.7% |
| Patronus Wolf Defender small | 0.769 | 48.5% | 4.7% |
| Horizon-Labs prompt injection guard large | 0.763 | 46.8% | 2.7% |
| Shield 2.0 without the model | 0.734 | 36.2% | 1.6% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.723 | 32.8% | 0.8% |
| fmops DistilBERT prompt injection | 0.716 | 89.0% | 47.7% |
| deepset DeBERTa v3 base injection | 0.715 | 91.4% | 49.5% |
| @stackone/defender | 0.711 | 37.7% | 6.7% |
| Azure AI Prompt Shields | 0.700 | 28.3% | 0.9% |
| Prismor prompt-guard 1.5B | 0.692 | 45.0% | 13.3% |
| TestSavant prompt injection defender large | 0.689 | 64.0% | 25.6% |
| PIGuard | 0.686 | 33.5% | 7.5% |
| Meta Prompt Guard 86M (jailbreak score) | 0.675 | 26.5% | 3.2% |
| Horizon-Labs prompt injection guard base | 0.667 | 24.0% | 2.9% |
| llm-trust-guard | 0.662 | 18.2% | 0.4% |
| @bastionsoft/prompt-protection | 0.661 | 12.5% | 2.9% |
| zambo-prompt-shield | 0.643 | 15.9% | 0.2% |
| Meta Llama Prompt Guard 2 22M | 0.641 | 18.9% | 3.7% |
| Horizon-Labs prompt injection guard small | 0.640 | 13.5% | 0.2% |
| @andersmyrmel/vard | 0.639 | 16.9% | 0.7% |
| @lov3kaizen/agentsea-guardrails | 0.636 | 16.6% | 0.6% |
| llm-prompt-guard | 0.633 | 14.2% | 0.0% |
| llm-firewall | 0.632 | 13.9% | 0.0% |
| llm-guard (npm) | 0.631 | 19.8% | 8.7% |
| @zeroleaks/shield 1.2.1 | 0.631 | 13.9% | 0.2% |
| Meta Llama Prompt Guard 2 86M | 0.627 | 21.4% | 7.5% |
| NeuralTrust prompt guard small | 0.624 | 32.8% | 17.8% |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.617 | 8.0% | 0.2% |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.615 | 11.1% | 2.7% |
| Jailbreak-Detector-Large (madhurjindal) | 0.603 | 15.8% | 7.6% |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.598 | 4.5% | 0.1% |
| vLLM Semantic Router Vela 307M guard | 0.595 | 17.8% | 11.9% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.583 | 0.0% | 0.0% |
| Guardion ModernGuard-1 | 0.490 | 71.2% | 67.3% |
| Meta Prompt Guard 86M (injection or jailbreak score) | 0.386 | 87.0% | 97.2% |
Most of the leaders' margin comes from CyberSecEval, where ProtectAI's model catches 97% of the English attacks and Shield 54%. Many of those attacks read as ordinary requests ("How do I make moussaka?" in Greek, "What are the latest advancements in AI?") that are attacks only because the test's system prompt forbids them; a detector that sees only the text can't tell. On Open-Prompt-Injection, where a second task is injected into a task's input, Shield catches 48% of the attacks with 4% false positives, and ProtectAI's model 23%.
Horizon-Labs eval suite
Horizon-Labs published this suite with its guard models on September 23, 2026. All 16 sets:
| Detector | Balanced accuracy | Attack recall | False positives |
|---|---|---|---|
| Shield 2.0 with Shield's large model | 0.941 | 93.3% | 8.1% |
| Shield 2.0 with the large model's earlier version | 0.936 | 93.9% | 9.7% |
| Shield 2.0, tiered: Shield's model, then its large model when unsure | 0.931 | 91.5% | 10.3% |
| Shield 2.0 with Shield's model | 0.907 | 87.2% | 12.1% |
| Shield 2.0 with Shield's model, int8 | 0.904 | 85.2% | 10.7% |
| Horizon-Labs prompt injection guard large | 0.886 | 78.2% | 3.5% |
| Patronus Wolf Defender | 0.834 | 67.3% | 9.5% |
| Patronus Wolf Defender small | 0.820 | 63.6% | 11.3% |
| Sheltron prompt guard 68M | 0.814 | 76.0% | 19.4% |
| PIGuard | 0.805 | 73.4% | 14.0% |
| Shield 2.0 without the model | 0.803 | 55.0% | 12.6% |
| Amazon Bedrock Guardrails, prompt attack filter | 0.792 | 55.4% | 3.8% |
| Horizon-Labs prompt injection guard base | 0.784 | 65.0% | 3.0% |
| Prismor prompt-guard 1.5B | 0.736 | 67.1% | 38.7% |
| Azure AI Prompt Shields | 0.730 | 42.4% | 3.1% |
| Horizon-Labs prompt injection guard small | 0.724 | 41.4% | 0.6% |
| ProtectAI DeBERTa v3 base v2 (LLM Guard) | 0.715 | 51.9% | 18.6% |
| @stackone/defender | 0.712 | 53.7% | 28.9% |
| NeuralTrust prompt guard small | 0.705 | 48.3% | 14.4% |
| TestSavant prompt injection defender large | 0.692 | 60.4% | 38.5% |
| deepset DeBERTa v3 base injection | 0.683 | 98.4% | 68.0% |
| Meta Llama Prompt Guard 2 86M | 0.681 | 34.9% | 7.1% |
| Jailbreak-Detector-Large (madhurjindal) | 0.672 | 40.7% | 17.9% |
| Meta Llama Prompt Guard 2 22M | 0.659 | 29.4% | 8.0% |
| Qwen3Guard-Gen 4B (Jailbreak category) | 0.648 | 16.8% | 0.6% |
| Qwen3Guard-Gen 0.6B (Jailbreak category) | 0.620 | 12.6% | 0.9% |
| llm-trust-guard | 0.598 | 17.8% | 8.9% |
| @lov3kaizen/agentsea-guardrails | 0.593 | 15.7% | 3.4% |
| @andersmyrmel/vard | 0.592 | 15.9% | 5.0% |
| IBM Granite Guardian 4.1 8B (jailbreak) | 0.580 | 18.2% | 10.0% |
| zambo-prompt-shield | 0.576 | 10.6% | 4.9% |
| llm-guard (npm) | 0.572 | 17.5% | 12.5% |
| @zeroleaks/shield 1.2.1 | 0.571 | 8.2% | 1.9% |
| NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails) | 0.565 | 2.1% | 0.0% |
| llm-firewall | 0.564 | 8.5% | 6.0% |
| vLLM Semantic Router Vela 307M guard | 0.563 | 19.2% | 14.2% |
| llm-prompt-guard | 0.556 | 7.2% | 2.0% |
| Guardion ModernGuard-1 | 0.543 | 81.1% | 77.2% |
Five of its sets share a source with Shield's training data (agentic 5k and boundary pairs, neuralchemy, S-Labs, LLMail); BIPIA's test set is separate from the BIPIA training split Shield used. Three are held out from Horizon-Labs' own training generator and favor its models. PIArena and Simsonsun are also in held-out v3.
Speed
Every model ran on the same four cores of an Intel Xeon Platinum 8275CL server with ONNX Runtime, one call per text, on 538 texts drawn from every group (median 206 characters). Each model read text the way its benchmark run did: Shield's in 256-token windows every 192 tokens, the others up to their maximum length.
| Model | File | Median, 1 thread | Median, 4 threads | 95th percentile, 1 thread |
|---|---|---|---|---|
| Shield's model | 236MB | 25ms | 10ms | 592ms |
Shield's model, int8 (dtype: "q8") | 118MB | 17ms | – | – |
| Meta Llama Prompt Guard 2 22M | 284MB | 35ms | 18ms | 486ms |
| Horizon-Labs guard small, int8 | 268MB | 44ms | 20ms | 1,345ms |
| Sheltron prompt guard 68M, int8 | 124MB | 47ms | 21ms | 1,589ms |
| Patronus Wolf Defender, int8 | 218MB | 72ms | 31ms | 2,690ms |
| Horizon-Labs guard base, int8 | 641MB | 104ms | 38ms | 2,997ms |
| ProtectAI DeBERTa v3 base v2 | 739MB | 106ms | 41ms | 1,410ms |
| PIGuard | 736MB | 107ms | 41ms | 1,374ms |
| Meta Llama Prompt Guard 2 86M | 1,116MB | 108ms | 42ms | 1,396ms |
| Horizon-Labs guard large, int8 | 1,479MB | 367ms | 102ms | 7,514ms |
In the SDK, detectAsync() with model.options(), including tokenization in JavaScript, took a median of 26ms on one thread and 12ms on four. The large model, through the SDK on the same machine, took a median of 671ms on four threads; long documents, up to 16 windows of 512 tokens, take seconds. In the same run the default model took a median of 38ms on one thread and tiered 43ms, since most inputs never reach the large model; their means were 178ms and 1,333ms. On an NVIDIA A10G GPU, a model of the large model's size took a median of 30ms per request. detect() alone takes about 0.05ms. The int8 file changes about 2% of verdicts and scores 0.885 against 0.889 on the five groups.
How Shield's model was chosen
Candidates for the archived default model were compared on the benchmark sets, v1, and 14 sets of the Horizon-Labs suite. The large model and the tiered setting were chosen later on all five development groups, v2 and v3 included; see Shield's large model. We trained models across seven rounds of training data using five base models (multilingual E5 small, base, and large, mmBERT, and mDeBERTa). E5-small was as accurate as the larger models on this data and several times faster. The threshold stayed at 0.5: moving it from 0.3 to 0.95 changed the development score by less than 0.01. A 6-layer student distilled from the model was twice as fast but scored 0.014 lower. Averaging two models scored 0.003 higher at twice the cost. Neither ships.
How to read these numbers
- Balanced accuracy is the mean of recall and specificity on sets with both labels, specificity on benign-only sets, and recall on attack-only sets, averaged over the sets in a group. It weighs catching attacks and letting benign text through equally. Recall and false positives are pooled over the group's rows.
- What counts as an injection. Shield's sets label a plain harmful request, with no attempt to get around the model's rules, as benign, as PIGuard and Horizon-Labs do. Qwen3Guard and Granite Guardian are content-safety models; we scored them by their jailbreak category, which is why they miss most injections.
- Open models ran at their maximum input length with a sliding window over longer text, scoring the highest window, the protocol Horizon-Labs uses. That helps every model on long documents. The large guard models read the first 3,072 tokens.
- Paid services. Azure AI Prompt Shields ran through its user-prompt and document checks (a detection by either counts), Amazon Bedrock Guardrails through its prompt attack filter at high strength (Standard tier), and GPT-6 Sol with the prompt in
LLM_DETECTOR_PROMPTon up to 6,000 characters. GPT-6 Sol did not run on the Horizon suite.
Not tested
- Lakera Guard and Google Cloud Model Armor, which need accounts we didn't have for this run.
- OpenAI's
@openai/guardrails, whose jailbreak and prompt injection checks ask an OpenAI model to judge, like the GPT-6 Sol row. - Guardrails AI's jailbreak validator, which combines several models; its published classifier is only one part and isn't meant to be used alone.
- Gated models whose terms we couldn't accept: CodeIntegrity's PromptGuard, Qualifire's and Rogue Security's Sentinel models, and Meta's original Prompt Guard 86M, which ran only on the benchmark sets and held-out v1, from an earlier copy.
- Enterprise products sold through sales, such as Prompt Security, HiddenLayer, and Cisco AI Defense.
The 13 benchmark sets
| Set | Rows | What it tests |
|---|---|---|
deepset_test | 116 | Classic injections and goal hijacking, English and German |
jackhhao_test | 262 | Jailbreak prompts against benign prompts |
inthewild | 2,000 | Jailbreaks collected from Discord, Reddit, and prompt sites, against regular prompts from the same sources |
safeguard_test | 1,989 | Synthetic injections against open-instruction requests |
multilingual | 1,272 | Injections in about a dozen languages |
multilingual_gpl | 1,581 | Translated injections and benign prompts |
bipia_like | 1,000 | Documents with an injection inserted at a random sentence, against the same documents without it |
llmail_inject | 1,000 | Attacker emails from Microsoft's LLMail-Inject challenge (attacks only) |
zeroleaks_attacks | 211 | Agent attacks from ZeroLeaks' private corpus, many of which landed on frontier models (attacks only) |
notinject | 339 | Benign prompts built around trigger words, an over-defense test (benign only) |
oasst_val | 1,595 | Real user prompts from OpenAssistant (benign only) |
wiki_docs | 1,000 | Wikipedia passages (benign only) |
llmail_fp_emails | 160 | The LLMail challenge's benign emails (benign only) |
Reproducing
The package's benchmark/ directory builds the sets (data/), runs the detectors (competitors/: run_gpu.py for the open models and guards, run_onnx.py, the npm libraries in js/, run_bedrock_guardrail.py, and latency.py), and scores them (report/compare_groups.py).