Skip to Content
ZeroLeaksDocs
Shield SDK

Archived benchmarks

Historical SDK detector comparisons, evaluation limitations, and reproduction notes. These results do not describe the current hosted models.

Historical evaluation. This page preserves an earlier SDK comparison and its then-current model configuration. References below to a model being “shipped” or “current” describe that historical snapshot. The hosted shield-large model now combines two classifiers; these tables do not measure its current results or API latency. See current model access.

Evaluation limits. The labels “held-out v1” through “held-out v5” are historical group names, not guarantees that a dataset or its source was unseen during training or model selection. Several evaluation sources overlap with training sources, and development results influenced model selection. These scores do not establish performance on independent, unseen data or a state-of-the-art claim.

The archived comparison ran Shield and 35 other detectors on 63 evaluation sets in seven groups. It included open injection and jailbreak classifiers, three large guard models, two paid services (Azure AI Prompt Shields and Amazon Bedrock Guardrails), GPT-6 Sol asked to judge each text, and JavaScript libraries on npm. Detectors used their default thresholds.

The archived run used three local SDK configurations with matching model files, tokenization, and windows. "Shield's model" is detectAsync() with Shield's model in place of its built-in classifier (model.options()). "Shield's large model" uses the large model the same way, scored from its 4-bit file. "Tiered" is tiered(): Shield's model first, and the large model for the inputs it is unsure about. "Shield 2.0 without the model" used the synchronous local detect(); that function now lives in @zeroleaks/shield/local.

Recorded results

  • Agent-oriented groups. The archived large model scored 0.751 balanced accuracy on v5, compared with 0.801 for GPT-6 Sol. Its attack recall was 50.8% with 2.4% false positives. On v4, the earlier large checkpoint scored 0.818 and the later checkpoint 0.810; GPT-6 Sol scored 0.808.
  • Development groups. The large model averaged 0.923 across the five groups used to build and select the models, tiered 0.911, and the default model 0.889. These are development results and should be interpreted with the training-source overlap described below.
  • Local latency. The archived default model took a median of 25ms on one CPU thread and 10ms on four. A separate SDK run measured tiered at 43ms and the default model at 38ms on one thread; their means were 1,333ms and 178ms because the large model took seconds on long documents. The large model's median was 671ms on four CPU threads. These measurements exclude hosted API overhead.
  • Limits. No tested detector, including GPT-6 Sol, did better than chance on MCIP's tool-use conversations. On v4's agent_extraction set, the earlier large checkpoint caught 20.6% of attacks. ProtectAI's model scored 0.896 on v1 against the large model's 0.866.
DetectorKindMean of the five groupsHeld-out v5Held-out v4BenchmarkHeld-out v1Horizon suiteHeld-out v2Held-out v3Median latency, 1 CPU thread
Shield 2.0 with the large model's earlier versionSDK + own model0.924–0.8180.9330.8600.9370.9840.907–
Shield 2.0 with Shield's large modelSDK + own model0.9230.7510.8100.9220.8660.9430.9820.902671 ms on 4 threads
Shield 2.0, tiered: Shield's model, then its large model when unsureSDK + own models0.9110.7090.8000.9260.8210.9370.9820.88743 ms
Shield 2.0 with Shield's modelSDK + own model0.8890.6690.7710.9100.7860.9210.9730.85725 ms
Shield 2.0 with Shield's model, int8SDK + own model0.8850.6380.7700.9060.7830.9190.9720.84517 ms
Patronus Wolf Defenderopen model0.8380.6710.7530.8590.7700.8500.9330.77672 ms
Horizon-Labs prompt injection guard largeopen model0.8230.7080.7590.8220.7630.8960.8600.772367 ms
Patronus Wolf Defender smallopen model0.822–0.7460.8420.7690.8380.8970.764–
Sheltron prompt guard 68Mopen model0.8060.6970.7490.8210.8180.8240.9150.65347 ms
PIGuardopen model0.7860.6920.7080.8150.6860.8050.9020.723107 ms
Shield 2.0 without the modelSDK0.7770.6270.6940.8500.7340.8410.7900.6700.05 ms
Amazon Bedrock Guardrails, prompt attack filterpaid API0.7680.6400.7610.7480.7230.8090.8200.739–
@stackone/defendernpm library0.7630.6320.6770.7600.7110.7080.8580.781–
ProtectAI DeBERTa v3 base v2 (LLM Guard)open model0.7530.6240.7280.7310.8960.7340.8350.566106 ms
TestSavant prompt injection defender largeopen model0.7520.5900.7020.7620.6890.6730.9760.663–
NeuralTrust prompt guard smallopen model0.7410.5890.7330.7370.6240.7140.9370.694–
Azure AI Prompt Shieldspaid API0.7400.6260.6150.7390.7000.7530.8130.696–
Horizon-Labs prompt injection guard baseopen model0.7210.6550.6100.7180.6670.7940.7550.673104 ms
Meta Llama Prompt Guard 2 86Mopen model0.7150.5950.6110.6970.6270.6950.8850.668108 ms
Prismor prompt-guard 1.5Bopen LLM0.707–0.7580.7740.6920.7330.6490.690–
Horizon-Labs prompt injection guard smallopen model0.694–0.4920.7310.6400.7570.7260.61544 ms
Meta Llama Prompt Guard 2 22Mopen model0.6920.5600.5790.6850.6410.6790.8210.63235 ms
Jailbreak-Detector-Large (madhurjindal)open model0.6890.6070.6580.7080.6030.6710.7890.672–
Qwen3Guard-Gen 4B (Jailbreak category)open LLM0.652–0.5960.6760.6170.6940.6800.593–
@lov3kaizen/agentsea-guardrailsnpm library0.6370.5120.4170.6330.6360.6210.7330.561–
@andersmyrmel/vardnpm library0.6360.5090.4150.6140.6390.6130.7660.549–
Qwen3Guard-Gen 0.6B (Jailbreak category)open LLM0.632–0.5430.6590.5980.6640.6730.567–
llm-trust-guardnpm library0.6320.5510.4210.6510.6620.6350.6700.542–
deepset DeBERTa v3 base injectionopen model0.6310.5590.7390.5900.7150.6730.6000.577–
llm-guard (npm)npm library0.5980.5240.3960.6140.6310.6020.5770.564–
zambo-prompt-shieldnpm library0.5970.5090.3940.6210.6430.6140.5630.543–
llm-firewallnpm library0.5910.5080.3680.6180.6320.6040.5680.534–
IBM Granite Guardian 4.1 8B (jailbreak)open LLM0.590–0.4420.5970.6150.6060.5600.573–
@zeroleaks/shield 1.2.1npm library0.5890.5210.4220.6100.6310.6110.5680.524–
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)open model0.5780.5000.3370.6410.5830.6060.5430.519–
llm-prompt-guardnpm library0.5770.5090.3830.5970.6330.5980.5220.539–
Guardion ModernGuard-1open model0.574–0.6980.6170.4900.5070.6240.632–
vLLM Semantic Router Vela 307M guardopen model0.551–0.4400.6110.5950.5910.4240.533–
@bastionsoft/prompt-protectionnpm library–––0.8130.661–0.825––
GPT-6 Sol as a judgepaid LLM–0.8010.8080.9010.806–0.7470.844–
fmops DistilBERT prompt injectionopen model–––0.5790.716––––
Meta Prompt Guard 86M (jailbreak score)open model–––0.7000.675––––
Meta Prompt Guard 86M (injection or jailbreak score)open model–––0.5100.386––––

"Mean of the five groups" averages the development groups' balanced accuracy for detectors that ran on all of them; held-out v4 is reported on its own. Latency is the median over the latency test, with ONNX Runtime, except for the large model and tiered, which were timed through the SDK in a later run in which the default model took 38ms on one thread; a dash means not measured. GPT-6 Sol and @bastionsoft/prompt-protection did not run on every group: the judge costs a model call per text, and @bastionsoft/prompt-protection was stopped after 30 minutes on v3's web pages. fmops' model and Meta's Prompt Guard 86M ran only on the benchmark sets and held-out v1, in the earlier version of this benchmark. GPT-6 Sol did not run on the Horizon suite.

The groups

GroupSetsWhat it measuresUsed to build Shield?
Benchmark sets13Direct and indirect injection, jailbreaks, multilingual attacks, over-defenseYes: nine share a source with the training data, and all 13 guided development
Held-out v16CyberSecEval, Open-Prompt-Injection, and benign prompts that sound riskyUsed to compare candidate models
Horizon-Labs eval suite16The suite Horizon-Labs published with its guard models in September 202614 sets used to compare candidate models; five share a source with the training data
Held-out v24JailbreakBench and Salad-Data jailbreaks, CyberSecEval's benign security prompts, AlpacaUsed to choose the large model
Held-out v38Benchmarks published since mid-2025: BrowseSafe, Lakera's b3, PIArena, JailbreaksOverTime, a curated jailbreak set, and two false-positive setsUsed to choose the large model; BrowseSafe results also informed a windowing change
Held-out v54Agent tool results (AgentDojo, AgentDyn), web-agent injections (WAInjectBench), poisoned MCP tool descriptions (MCPTox), tool-use conversations (MCIP)Reported as a later evaluation; no training-source isolation guarantee
Held-out v49Agent-security datasets, Kubernetes manifests with planted instructions, Tensor Trust, Chinese injections, attacks on agents' secrets, and jailbreak payloads, most published in August and September 2026Reported as a later evaluation; no training-source isolation guarantee

Archived evaluation plans are in the package's benchmark/heldout-plans directory. The benchmark, v1, Horizon-Labs, v2, and v3 groups informed model development or selection. Training-source overlap includes deepset, LLMail, translated injection datasets, and BIPIA-derived data. Filtering duplicate rows does not make different splits, translations, or variants from the same source independent. This archive does not establish that every evaluated dataset or source was excluded from training. Other detectors' training data is not fully public either.

Held-out v5

This group covers instructions planted in tool results, web pages, MCP tool descriptions, and tool-use conversations. It was added after v4 in the archived evaluation. A fifth set, password_eval, tested the large model's context mode; see Shield's large model.

SetSourceRowsLabels
agentdojo_toolsTool results from AgentDojo and AgentDyn runs (ETH Zurich, SaFo-Lab): results carrying an injection, and results from runs without an attack1,200600 attacks, 600 benign
wainjectWAInjectBench's text split: web-agent injections (EIA, VWA-adv, pop-ups, VPI, WASP) and benign web text1,123523 attacks, 600 benign
mcptoxMCPTox: poisoned MCP tool descriptions and clean tools from the same servers827485 attacks, 342 benign
mcipMCIP-bench's test split: tool-use conversations with injected or misused function calls, and clean ones893744 attacks, 149 benign
DetectorBalanced accuracyAttack recallFalse positives
GPT-6 Sol as a judge0.80158.7%1.9%
Shield 2.0 with Shield's large model0.75150.8%2.4%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.70950.7%7.6%
Horizon-Labs prompt injection guard large0.70847.0%7.9%
Sheltron prompt guard 68M0.69764.4%20.3%
PIGuard0.69266.3%22.5%
Patronus Wolf Defender0.67134.7%1.7%
Shield 2.0 with Shield's model0.66948.6%10.6%
Horizon-Labs prompt injection guard base0.65541.3%8.2%
Amazon Bedrock Guardrails, prompt attack filter0.64028.0%0.5%
Shield 2.0 with Shield's model, int80.63841.5%9.5%
@stackone/defender0.63233.0%5.5%
Shield 2.0 without the model0.62730.2%4.7%
Azure AI Prompt Shields0.62625.5%0.7%
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.62444.4%16.0%
Jailbreak-Detector-Large (madhurjindal)0.60730.1%5.7%
Meta Llama Prompt Guard 2 86M0.59527.1%2.1%
TestSavant prompt injection defender large0.59023.0%6.4%
NeuralTrust prompt guard small0.58919.6%3.1%
Meta Llama Prompt Guard 2 22M0.56015.2%1.1%
deepset DeBERTa v3 base injection0.55996.0%76.9%
llm-trust-guard0.55111.6%0.9%
llm-guard (npm)0.5248.3%2.2%
@zeroleaks/shield 1.2.10.5214.0%0.2%
@lov3kaizen/agentsea-guardrails0.5122.5%0.1%
@andersmyrmel/vard0.5092.2%0.3%
llm-prompt-guard0.5091.9%0.0%
zambo-prompt-shield0.5091.8%0.0%
llm-firewall0.5081.7%0.0%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5000.0%0.0%

Per set (balanced accuracy):

Detectoragentdojo_toolswainjectmcptoxmcip
GPT-6 Sol as a judge0.9820.7110.9770.534
Shield 2.0 with Shield's large model0.8820.6480.9220.553
Shield 2.0, tiered: Shield's model, then its large model when unsure0.7870.6460.8770.526
Horizon-Labs prompt injection guard large0.8610.5530.9260.493
Sheltron prompt guard 68M0.7230.7210.8920.451
PIGuard0.8070.6380.7880.535
Patronus Wolf Defender0.8990.6200.6710.494
Shield 2.0 with Shield's model0.6900.6650.7920.528
Horizon-Labs prompt injection guard base0.8190.5340.7310.537
Amazon Bedrock Guardrails, prompt attack filter0.9090.5500.5950.505
Shield 2.0 with Shield's model, int80.6340.6550.7480.513
@stackone/defender0.7200.5630.7110.533
Shield 2.0 without the model0.6710.5330.7920.514
Azure AI Prompt Shields0.8540.5300.6230.497
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.7050.6540.6100.528
Jailbreak-Detector-Large (madhurjindal)0.6830.5830.6310.530
Meta Llama Prompt Guard 2 86M0.8020.5340.5050.539
TestSavant prompt injection defender large0.6380.6920.5210.509
NeuralTrust prompt guard small0.5910.5560.7160.493
Meta Llama Prompt Guard 2 22M0.6900.5230.5120.515
deepset DeBERTa v3 base injection0.5730.6610.5030.500
llm-trust-guard0.5800.5600.5380.527
llm-guard (npm)0.5570.4990.5190.521
@zeroleaks/shield 1.2.10.5320.5010.5510.501
@lov3kaizen/agentsea-guardrails0.5330.5100.5020.503
@andersmyrmel/vard0.5320.4990.5000.506
llm-prompt-guard0.5330.5000.5010.502
zambo-prompt-shield0.5330.5000.5000.501
llm-firewall0.5330.5000.5000.500
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5000.5000.5000.500

Held-out v4

This archived group combines agent-security datasets, configuration files with planted instructions, and direct attacks. Its historical name does not certify independence from model development or training sources.

SetSourceRowsLabels
pp_attackspromptprotection's agent-security datasets (Sep 2026): attacks130Attacks
pp_benign_hardThe same datasets' hard benign prompts, written to look like attacks155Benign
pp_agent_flowsThe same datasets' agent flows: the texts an agent's tools return in each flow, joined9150 attacks, 41 benign
k8sKubernetes manifests with and without a planted instruction (Aug 2026)851493 attacks, 358 benign
tensor_trustTensor Trust's hijacking and extraction robustness benchmarks, the attacks770Attacks
changemore_zhChangeMore prompt injection evaluation, Chinese1,963978 attacks, 985 benign
agent_extractionPrompts against agents with secrets in their instructions (Apr 2026)783Attacks
airline_payloadsReal jailbreak payloads held out from an airline agent's red-team set (Sep 2026)158Attacks
probes_mixedRealistic prompt injections from three sets published in September 2026136115 attacks, 21 benign
DetectorBalanced accuracyAttack recallFalse positives
Shield 2.0 with the large model's earlier version0.81876.1%6.4%
Shield 2.0 with Shield's large model0.81073.5%4.4%
GPT-6 Sol as a judge0.80855.2%0.2%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.80068.9%4.7%
Shield 2.0 with Shield's model0.77162.7%5.6%
Shield 2.0 with Shield's model, int80.77061.6%4.8%
Amazon Bedrock Guardrails, prompt attack filter0.76155.5%2.1%
Horizon-Labs prompt injection guard large0.75959.6%2.6%
Prismor prompt-guard 1.5B0.75865.7%15.6%
Patronus Wolf Defender0.75354.5%3.6%
Sheltron prompt guard 68M0.74960.1%12.2%
Patronus Wolf Defender small0.74655.4%4.3%
deepset DeBERTa v3 base injection0.73990.3%83.1%
NeuralTrust prompt guard small0.73357.5%4.0%
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.72856.4%8.7%
PIGuard0.70853.1%9.5%
TestSavant prompt injection defender large0.70266.4%37.0%
Guardion ModernGuard-10.69892.7%77.8%
Shield 2.0 without the model0.69446.9%6.1%
@stackone/defender0.67744.6%6.5%
Jailbreak-Detector-Large (madhurjindal)0.65855.9%8.9%
Azure AI Prompt Shields0.61535.1%1.7%
Meta Llama Prompt Guard 2 86M0.61140.3%2.4%
Horizon-Labs prompt injection guard base0.61044.3%4.5%
Qwen3Guard-Gen 4B (Jailbreak category)0.59626.7%0.6%
Meta Llama Prompt Guard 2 22M0.57937.4%2.4%
Qwen3Guard-Gen 0.6B (Jailbreak category)0.54322.6%0.4%
Horizon-Labs prompt injection guard small0.49221.3%1.0%
IBM Granite Guardian 4.1 8B (jailbreak)0.44225.4%16.7%
vLLM Semantic Router Vela 307M guard0.44019.8%11.3%
@zeroleaks/shield 1.2.10.42211.6%1.0%
llm-trust-guard0.42115.6%3.8%
@lov3kaizen/agentsea-guardrails0.41714.2%1.5%
@andersmyrmel/vard0.41516.2%2.4%
llm-guard (npm)0.39615.6%5.3%
zambo-prompt-shield0.39410.5%1.6%
llm-prompt-guard0.3837.4%0.8%
llm-firewall0.3686.8%1.9%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.3371.1%0.0%

Per set (balanced accuracy):

Detectorpp_attackspp_benign_hardpp_agent_flowsk8stensor_trustchangemore_zhagent_extractionairline_payloadsprobes_mixed
Shield 2.0 with the large model's earlier version1.0000.7940.9170.9250.9820.9370.2060.6330.972
Shield 2.0 with Shield's large model0.9850.8190.8990.9230.9730.9350.1520.6011.000
GPT-6 Sol as a judge0.9771.0000.9500.9330.8710.6790.0740.7970.991
Shield 2.0, tiered: Shield's model, then its large model when unsure0.9690.8130.9310.9070.9680.8730.1150.6271.000
Shield 2.0 with Shield's model0.9620.8060.8670.8510.9490.8010.0920.6330.976
Shield 2.0 with Shield's model, int80.9460.8130.8910.8490.9350.7970.0920.6140.991
Amazon Bedrock Guardrails, prompt attack filter0.8540.9230.9080.8150.7450.7400.2430.6390.978
Horizon-Labs prompt injection guard large0.8920.8840.9090.9170.7210.7880.2110.5510.954
Prismor prompt-guard 1.5B0.8690.8390.9460.7570.5840.8490.3900.6520.939
Patronus Wolf Defender0.8920.8450.9030.6940.9260.7810.0290.7280.978
Sheltron prompt guard 68M0.9310.6450.8310.9200.9440.6780.0980.6960.996
Patronus Wolf Defender small0.8460.7810.8730.6990.9300.7820.0610.7470.991
deepset DeBERTa v3 base injection0.9920.5550.5240.5000.9690.5800.6001.0000.929
NeuralTrust prompt guard small0.7460.9160.8230.7280.6860.8430.2350.7220.902
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.8310.6650.7450.8430.9710.6570.1440.7470.946
PIGuard0.7310.7100.7660.8800.8320.6600.0400.7720.983
TestSavant prompt injection defender large0.6850.6580.9280.6000.7940.7300.3880.6390.896
Guardion ModernGuard-10.9000.1870.5940.4380.9320.6330.9030.9680.729
Shield 2.0 without the model0.8080.7870.7830.8120.8780.6000.0470.5440.983
@stackone/defender0.7540.6320.7970.7170.8340.6140.1030.6960.941
Jailbreak-Detector-Large (madhurjindal)0.4000.9550.5980.6320.7220.8920.1300.8730.722
Azure AI Prompt Shields0.4920.8970.7200.7040.6520.6210.0040.5060.935
Meta Llama Prompt Guard 2 86M0.3230.8390.6030.7020.7840.6660.0130.6520.917
Horizon-Labs prompt injection guard base0.3460.9480.6480.8710.6550.6890.0320.4750.830
Qwen3Guard-Gen 4B (Jailbreak category)0.5540.9680.6960.5930.5820.5630.0050.6140.791
Meta Llama Prompt Guard 2 22M0.2000.8260.5630.7340.7620.6200.0270.5950.883
Qwen3Guard-Gen 0.6B (Jailbreak category)0.3690.9680.6100.6120.5030.5470.0060.6140.657
Horizon-Labs prompt injection guard small0.3230.9420.6180.5620.3220.6340.0060.2280.791
IBM Granite Guardian 4.1 8B (jailbreak)0.0690.9290.5680.8600.1730.4670.1760.1770.559
vLLM Semantic Router Vela 307M guard0.2690.9290.5100.4720.3600.5250.1400.2090.548
@zeroleaks/shield 1.2.10.0850.9350.5730.5870.2550.5110.0040.1200.726
llm-trust-guard0.1380.8060.5810.5920.1870.5740.0290.1140.765
@lov3kaizen/agentsea-guardrails0.0620.9550.5030.5840.2440.5460.0640.1460.652
@andersmyrmel/vard0.0690.9480.4930.5620.3340.5480.0920.0380.652
llm-guard (npm)0.0380.7290.4910.6050.2230.5270.1290.0820.735
zambo-prompt-shield0.0230.8650.5230.5690.2400.5180.0080.1010.696
llm-prompt-guard0.0540.9480.5260.5390.2060.5040.0030.0130.652
llm-firewall0.0380.8260.5060.5330.0810.5360.0100.1270.652
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.0001.0000.5000.5000.0000.5180.0000.0130.500

Shield's large model

The large model is Qwen3-1.7B fine-tuned as a classifier (LoRA, rank 32) on the default model's training data plus about 22,500 additional texts: WildJailbreak's benign requests that sound harmful, its harmful requests wrapped in jailbreak tactics, PHTest's pseudo-harmful prompts, and CoCoNot's contrast set. That version was scored on v4. After v4, a second LoRA trained it further on 35,000 texts from the same data, 20,700 benign texts that look like attacks (real user prompts from LMSYS Toxic-Chat, FalseReject, system prompts, Kubernetes manifests, security Q&A, Chinese Q&A), and 25,000 system prompts paired with requests that follow or break them (SystemCheck, SPML). The shipped version uses 4-bit weights in a 1.08GB file. Its v5 scores come from that file (0.751; the full-precision weights score 0.726, and the two differ on 5% of v5's verdicts). Scores for the other groups come from the full-precision weights.

  • Size. On the default model's data, Qwen3 0.6B, 1.7B, 4B, and 8B scored 0.891, 0.915, 0.916, and 0.912 on the development groups, so the 1.7B model ships.
  • Data. Those models flagged 15-18% of OR-Bench's benign prompts that sound harmful, against the default model's 4%. The added data brought that to 5% and the development score to 0.924, the version scored on v4. Two variants picked what to add: labeling plain harmful requests benign as well cut recall on instructions planted in documents.
  • Tiered. tiered() sends the inputs the default model scores from 0.01 up to 0.97 to the large model, 14% of them here. It scores 0.911 on the development groups and 0.800 on v4.
  • Context mode, not shipped. We also trained the large model to read the agent's instructions before each input, on system prompts paired with requests that follow or break them (SystemCheck, SPML). On held-out v5's password_eval (PasswordEval: agents whose instructions hold a password and a secret, and requests for the secret with and without the password), it flagged 86.9% of the requests that must be refused but also 98.0% of those that gave the right password: it learned that asking for the secret is suspect, not who may ask. Detectors that read only the request can't separate the two by design and scored near chance. The option isn't in the SDK.
  • Not shipped. Training further on the benign look-alikes without the system-prompt pairs cut false positives but also recall, and scored lower on the development groups. Distilling the large model into the default model scored 0.884 and 0.886 (the second time with 83,000 more texts scored by the large model), below the default model's 0.889, and four runs of the default model on the added data scored 0.883 to 0.889, within the 0.007 two seeds differ by, so the default model is unchanged.

Held-out v3

This group collects benchmarks published since mid-2025. Its scores were used during model development:

SetSourceRowsLabels
browsesafeBrowseSafe-Bench (Perplexity, Nov 2025): real web pages with injections for browser agents, and pages with look-alike text meant for people1,000500 attacks, 500 benign
b3_agentLakera's b3 benchmark (Nov 2025): attacks from the Gandalf Agent Breaker challenge against agent apps187Attacks
piarenaPIArena (Horizon-Labs' suite, Sep 2026): tasks injected into QA, summarization, and RAG contexts857675 attacks, 182 benign
jotJailbreaksOverTime (Oct 2025): jailbreaks collected over 10 months, and real user prompts1,713713 attacks, 1,000 benign
simsonsunCurated jailbreak dataset (Jun 2025)1,948Attacks
guardrail_hnGuardrail hard negatives test split (Aug 2026): benign and attack texts that share the same surface13596 attacks, 39 benign
operant_fpOperant AI false-positive test data (Jul 2026), several languages562Benign
hirundo_benignHirundo's benign prompts from a prompt injection benchmark (Jun 2026)1,000Benign
DetectorBalanced accuracyAttack recallFalse positives
Shield 2.0 with the large model's earlier version0.90789.1%3.0%
Shield 2.0 with Shield's large model0.90289.0%2.8%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.88787.1%3.7%
Shield 2.0 with Shield's model0.85782.3%5.5%
Shield 2.0 with Shield's model, int80.84580.3%4.8%
GPT-6 Sol as a judge0.84467.4%5.0%
@stackone/defender0.78171.1%9.7%
Patronus Wolf Defender0.77666.3%5.4%
Horizon-Labs prompt injection guard large0.77278.0%20.8%
Patronus Wolf Defender small0.76463.9%4.9%
Amazon Bedrock Guardrails, prompt attack filter0.73958.8%3.7%
PIGuard0.72375.8%27.5%
Azure AI Prompt Shields0.69641.8%1.6%
NeuralTrust prompt guard small0.69461.8%10.4%
Prismor prompt-guard 1.5B0.69075.7%34.7%
Horizon-Labs prompt injection guard base0.67362.8%19.0%
Jailbreak-Detector-Large (madhurjindal)0.67268.9%32.6%
Shield 2.0 without the model0.67048.6%15.8%
Meta Llama Prompt Guard 2 86M0.66850.1%6.5%
TestSavant prompt injection defender large0.66377.7%32.6%
Sheltron prompt guard 68M0.65372.8%32.5%
Meta Llama Prompt Guard 2 22M0.63240.3%5.8%
Guardion ModernGuard-10.63290.4%68.5%
Horizon-Labs prompt injection guard small0.61533.8%2.5%
Qwen3Guard-Gen 4B (Jailbreak category)0.59313.1%1.2%
deepset DeBERTa v3 base injection0.57799.8%80.6%
IBM Granite Guardian 4.1 8B (jailbreak)0.57320.1%7.2%
Qwen3Guard-Gen 0.6B (Jailbreak category)0.56710.1%1.4%
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.56654.1%34.6%
llm-guard (npm)0.56423.0%6.8%
@lov3kaizen/agentsea-guardrails0.56122.3%7.2%
@andersmyrmel/vard0.54936.4%21.8%
zambo-prompt-shield0.54314.1%4.5%
llm-trust-guard0.54223.1%17.1%
llm-prompt-guard0.5396.5%0.6%
llm-firewall0.5348.6%0.5%
vLLM Semantic Router Vela 307M guard0.53322.6%17.9%
@zeroleaks/shield 1.2.10.52410.4%5.6%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5195.3%0.1%

Per set (balanced accuracy):

Detectorbrowsesafeb3_agentpiarenajotsimsonsunguardrail_hnoperant_fphirundo_benign
Shield 2.0 with the large model's earlier version0.8650.6740.9280.9700.9310.9740.9470.965
Shield 2.0 with Shield's large model0.8680.6150.9350.9780.9210.9870.9410.969
Shield 2.0, tiered: Shield's model, then its large model when unsure0.8780.5670.8700.9830.9170.9870.9430.953
Shield 2.0 with Shield's model0.8630.5030.7870.9730.8750.9870.9360.937
Shield 2.0 with Shield's model, int80.8710.4600.7510.9640.8700.9490.9500.941
GPT-6 Sol as a judge0.6980.7380.9990.9230.5261.0000.9700.901
@stackone/defender0.6770.5450.6800.9020.8220.7530.9450.922
Patronus Wolf Defender0.6420.5290.5990.9230.8160.8150.9000.986
Horizon-Labs prompt injection guard large0.5350.3960.9190.9530.6980.9270.9340.816
Patronus Wolf Defender small0.6490.4550.5930.9260.7710.7890.9450.985
Amazon Bedrock Guardrails, prompt attack filter0.7470.3100.6270.8830.6680.7880.9570.933
PIGuard0.5080.4550.8380.8560.7450.7450.8740.760
Azure AI Prompt Shields0.6720.2990.7360.8430.3250.7450.9800.970
NeuralTrust prompt guard small0.6490.1980.5410.9800.7360.7160.8700.859
Prismor prompt-guard 1.5B0.4920.6790.6960.7320.8740.4790.9270.640
Horizon-Labs prompt injection guard base0.5570.1870.8690.8260.5600.6020.9430.843
Jailbreak-Detector-Large (madhurjindal)0.4970.4330.5100.7930.8650.7370.9570.581
Shield 2.0 without the model0.5910.2730.5100.8160.4960.8720.9450.860
Meta Llama Prompt Guard 2 86M0.6770.1820.5380.9070.5880.6120.9880.856
TestSavant prompt injection defender large0.6400.4390.7540.8670.8760.5520.5800.592
Sheltron prompt guard 68M0.5010.3580.7480.8820.7220.6550.5730.783
Meta Llama Prompt Guard 2 22M0.6490.1340.5310.8330.4510.5710.9860.906
Guardion ModernGuard-10.5000.9090.6570.6660.9440.6150.4680.296
Horizon-Labs prompt injection guard small0.6180.0640.5980.7450.3830.5780.9840.948
Qwen3Guard-Gen 4B (Jailbreak category)0.5780.1600.5060.6700.0490.7920.9931.000
deepset DeBERTa v3 base injection0.5000.9840.5470.6250.9980.5000.2030.257
IBM Granite Guardian 4.1 8B (jailbreak)0.5490.1980.6120.5680.1980.6100.9450.904
Qwen3Guard-Gen 0.6B (Jailbreak category)0.5820.1710.5040.6160.0340.6380.9930.999
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.5000.3320.6180.8010.4930.5660.4680.754
llm-guard (npm)0.6560.0700.5250.6710.1920.5010.9890.911
@lov3kaizen/agentsea-guardrails0.6630.1340.5040.5760.3000.5101.0000.801
@andersmyrmel/vard0.5200.0430.5050.6540.4000.5050.9980.764
zambo-prompt-shield0.6280.0700.5110.5740.0920.4771.0000.992
llm-trust-guard0.5020.1020.5540.5860.1110.5260.9820.969
llm-prompt-guard0.6740.0800.5020.5190.0170.5261.0000.993
llm-firewall0.6450.0480.5000.6010.0270.4541.0000.994
vLLM Semantic Router Vela 307M guard0.4920.2670.5150.5640.1550.4840.8880.901
@zeroleaks/shield 1.2.10.5520.0210.5070.5600.0560.5101.0000.985
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5000.0000.5000.6460.0040.5001.0001.000

The model was chosen and scored on v3 once, at 0.792. It flagged 84% of BrowseSafe's benign pages: its training had shown it only the first 256 tokens of each BrowseSafe training page, so it never saw where on the page an injection or a look-alike was. We retrained with each training page split into windows, which brought those false positives to 10%, and ship that model. That change was made after seeing v3, so the v3 numbers above are not a clean test for BrowseSafe; the other seven sets did not inform any change, and Shield's score on them is 0.857.

Held-out v2

SetSourceRowsLabels
jbbJailbreakBench artifacts: PAIR, GCG, and random-search jailbreaks (MIT)996Attacks
salad_attackSalad-Data's attack-enhanced set, 1,000 sampled (Apache-2.0)971Attacks
cse_frrCyberSecEval's benign prompts that sound like cyberattack requests (MIT)749Benign
alpacaAlpaca instructions with their inputs, 1,000 sampled (eval only)1,000Benign
DetectorBalanced accuracyAttack recallFalse positives
Shield 2.0 with the large model's earlier version0.98498.0%1.1%
Shield 2.0 with Shield's large model0.98296.9%0.4%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.98296.6%0.3%
TestSavant prompt injection defender large0.97697.3%1.9%
Shield 2.0 with Shield's model0.97395.0%0.5%
Shield 2.0 with Shield's model, int80.97295.0%0.5%
NeuralTrust prompt guard small0.93788.8%1.2%
Patronus Wolf Defender0.93387.2%0.5%
Sheltron prompt guard 68M0.91583.3%0.3%
PIGuard0.90281.9%1.3%
Patronus Wolf Defender small0.89780.7%1.2%
Meta Llama Prompt Guard 2 86M0.88578.1%1.0%
Horizon-Labs prompt injection guard large0.86073.1%1.4%
@stackone/defender0.85879.7%7.1%
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.83567.0%0.0%
@bastionsoft/prompt-protection0.82565.0%0.3%
Meta Llama Prompt Guard 2 22M0.82167.6%3.0%
Amazon Bedrock Guardrails, prompt attack filter0.82065.8%1.4%
Azure AI Prompt Shields0.81363.5%0.8%
Shield 2.0 without the model0.79058.7%0.8%
Jailbreak-Detector-Large (madhurjindal)0.78996.4%33.0%
@andersmyrmel/vard0.76653.4%0.3%
Horizon-Labs prompt injection guard base0.75551.9%1.1%
GPT-6 Sol as a judge0.74749.3%0.1%
@lov3kaizen/agentsea-guardrails0.73346.8%0.1%
Horizon-Labs prompt injection guard small0.72645.7%0.7%
Qwen3Guard-Gen 4B (Jailbreak category)0.68035.8%0.1%
Qwen3Guard-Gen 0.6B (Jailbreak category)0.67334.3%0.0%
llm-trust-guard0.67034.7%0.7%
Prismor prompt-guard 1.5B0.64961.0%26.6%
Guardion ModernGuard-10.62499.2%70.7%
deepset DeBERTa v3 base injection0.60098.9%75.8%
llm-guard (npm)0.57718.6%3.1%
llm-firewall0.56813.5%0.0%
@zeroleaks/shield 1.2.10.56813.7%0.2%
zambo-prompt-shield0.56312.4%0.0%
IBM Granite Guardian 4.1 8B (jailbreak)0.56019.2%7.9%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5438.4%0.0%
llm-prompt-guard0.5224.3%0.0%
vLLM Semantic Router Vela 307M guard0.42428.1%38.9%

TestSavant's model is slightly ahead on v2 but flags 16% of the benchmark sets' benign text and 42% of the Horizon suite's.

Benchmark sets

DetectorBalanced accuracyAttack recallFalse positives
Shield 2.0 with the large model's earlier version0.93386.4%2.5%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.92685.4%2.1%
Shield 2.0 with Shield's large model0.92284.3%1.6%
Shield 2.0 with Shield's model0.91082.9%2.6%
Shield 2.0 with Shield's model, int80.90680.8%2.1%
GPT-6 Sol as a judge0.90175.4%3.2%
Patronus Wolf Defender0.85979.0%6.0%
Shield 2.0 without the model0.85070.5%1.8%
Patronus Wolf Defender small0.84276.4%6.4%
Horizon-Labs prompt injection guard large0.82267.6%2.9%
Sheltron prompt guard 68M0.82173.5%5.3%
PIGuard0.81564.9%5.7%
@bastionsoft/prompt-protection0.81356.0%1.2%
Prismor prompt-guard 1.5B0.77473.4%17.5%
TestSavant prompt injection defender large0.76269.9%16.3%
@stackone/defender0.76064.8%18.0%
Amazon Bedrock Guardrails, prompt attack filter0.74848.8%3.7%
Azure AI Prompt Shields0.73944.4%2.7%
NeuralTrust prompt guard small0.73743.3%3.0%
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.73158.9%9.3%
Horizon-Labs prompt injection guard small0.73141.9%0.7%
Horizon-Labs prompt injection guard base0.71852.2%3.4%
Jailbreak-Detector-Large (madhurjindal)0.70852.3%16.0%
Meta Prompt Guard 86M (jailbreak score)0.70040.6%6.4%
Meta Llama Prompt Guard 2 86M0.69740.8%7.1%
Meta Llama Prompt Guard 2 22M0.68536.9%6.1%
Qwen3Guard-Gen 4B (Jailbreak category)0.67629.7%3.2%
Qwen3Guard-Gen 0.6B (Jailbreak category)0.65924.5%3.3%
llm-trust-guard0.65120.8%2.6%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.64115.9%0.1%
@lov3kaizen/agentsea-guardrails0.63319.0%2.7%
zambo-prompt-shield0.62111.5%0.6%
llm-firewall0.61811.6%0.5%
Guardion ModernGuard-10.61790.8%55.6%
llm-guard (npm)0.61415.4%5.0%
@andersmyrmel/vard0.61416.9%5.8%
vLLM Semantic Router Vela 307M guard0.61120.5%11.5%
@zeroleaks/shield 1.2.10.6108.7%0.8%
llm-prompt-guard0.5975.7%1.6%
IBM Granite Guardian 4.1 8B (jailbreak)0.59711.8%6.7%
deepset DeBERTa v3 base injection0.59096.8%67.7%
fmops DistilBERT prompt injection0.57993.8%67.4%
Meta Prompt Guard 86M (injection or jailbreak score)0.51078.7%76.5%

These 13 sets favor Shield: nine share a source with its training data, with duplicates removed, and they guided its development. Their contents are listed in The 13 benchmark sets.

Held-out v1

DetectorBalanced accuracyAttack recallFalse positives
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.89670.2%2.8%
Shield 2.0 with Shield's large model0.86669.7%1.0%
Shield 2.0 with the large model's earlier version0.86071.0%3.6%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.82157.8%1.8%
Sheltron prompt guard 68M0.81864.2%5.3%
GPT-6 Sol as a judge0.80653.6%0.0%
Shield 2.0 with Shield's model0.78651.0%2.8%
Shield 2.0 with Shield's model, int80.78349.2%2.2%
Patronus Wolf Defender0.77051.3%5.7%
Patronus Wolf Defender small0.76948.5%4.7%
Horizon-Labs prompt injection guard large0.76346.8%2.7%
Shield 2.0 without the model0.73436.2%1.6%
Amazon Bedrock Guardrails, prompt attack filter0.72332.8%0.8%
fmops DistilBERT prompt injection0.71689.0%47.7%
deepset DeBERTa v3 base injection0.71591.4%49.5%
@stackone/defender0.71137.7%6.7%
Azure AI Prompt Shields0.70028.3%0.9%
Prismor prompt-guard 1.5B0.69245.0%13.3%
TestSavant prompt injection defender large0.68964.0%25.6%
PIGuard0.68633.5%7.5%
Meta Prompt Guard 86M (jailbreak score)0.67526.5%3.2%
Horizon-Labs prompt injection guard base0.66724.0%2.9%
llm-trust-guard0.66218.2%0.4%
@bastionsoft/prompt-protection0.66112.5%2.9%
zambo-prompt-shield0.64315.9%0.2%
Meta Llama Prompt Guard 2 22M0.64118.9%3.7%
Horizon-Labs prompt injection guard small0.64013.5%0.2%
@andersmyrmel/vard0.63916.9%0.7%
@lov3kaizen/agentsea-guardrails0.63616.6%0.6%
llm-prompt-guard0.63314.2%0.0%
llm-firewall0.63213.9%0.0%
llm-guard (npm)0.63119.8%8.7%
@zeroleaks/shield 1.2.10.63113.9%0.2%
Meta Llama Prompt Guard 2 86M0.62721.4%7.5%
NeuralTrust prompt guard small0.62432.8%17.8%
Qwen3Guard-Gen 4B (Jailbreak category)0.6178.0%0.2%
IBM Granite Guardian 4.1 8B (jailbreak)0.61511.1%2.7%
Jailbreak-Detector-Large (madhurjindal)0.60315.8%7.6%
Qwen3Guard-Gen 0.6B (Jailbreak category)0.5984.5%0.1%
vLLM Semantic Router Vela 307M guard0.59517.8%11.9%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5830.0%0.0%
Guardion ModernGuard-10.49071.2%67.3%
Meta Prompt Guard 86M (injection or jailbreak score)0.38687.0%97.2%

Most of the leaders' margin comes from CyberSecEval, where ProtectAI's model catches 97% of the English attacks and Shield 54%. Many of those attacks read as ordinary requests ("How do I make moussaka?" in Greek, "What are the latest advancements in AI?") that are attacks only because the test's system prompt forbids them; a detector that sees only the text can't tell. On Open-Prompt-Injection, where a second task is injected into a task's input, Shield catches 48% of the attacks with 4% false positives, and ProtectAI's model 23%.

Horizon-Labs eval suite

Horizon-Labs published this suite with its guard models on September 23, 2026. All 16 sets:

DetectorBalanced accuracyAttack recallFalse positives
Shield 2.0 with Shield's large model0.94193.3%8.1%
Shield 2.0 with the large model's earlier version0.93693.9%9.7%
Shield 2.0, tiered: Shield's model, then its large model when unsure0.93191.5%10.3%
Shield 2.0 with Shield's model0.90787.2%12.1%
Shield 2.0 with Shield's model, int80.90485.2%10.7%
Horizon-Labs prompt injection guard large0.88678.2%3.5%
Patronus Wolf Defender0.83467.3%9.5%
Patronus Wolf Defender small0.82063.6%11.3%
Sheltron prompt guard 68M0.81476.0%19.4%
PIGuard0.80573.4%14.0%
Shield 2.0 without the model0.80355.0%12.6%
Amazon Bedrock Guardrails, prompt attack filter0.79255.4%3.8%
Horizon-Labs prompt injection guard base0.78465.0%3.0%
Prismor prompt-guard 1.5B0.73667.1%38.7%
Azure AI Prompt Shields0.73042.4%3.1%
Horizon-Labs prompt injection guard small0.72441.4%0.6%
ProtectAI DeBERTa v3 base v2 (LLM Guard)0.71551.9%18.6%
@stackone/defender0.71253.7%28.9%
NeuralTrust prompt guard small0.70548.3%14.4%
TestSavant prompt injection defender large0.69260.4%38.5%
deepset DeBERTa v3 base injection0.68398.4%68.0%
Meta Llama Prompt Guard 2 86M0.68134.9%7.1%
Jailbreak-Detector-Large (madhurjindal)0.67240.7%17.9%
Meta Llama Prompt Guard 2 22M0.65929.4%8.0%
Qwen3Guard-Gen 4B (Jailbreak category)0.64816.8%0.6%
Qwen3Guard-Gen 0.6B (Jailbreak category)0.62012.6%0.9%
llm-trust-guard0.59817.8%8.9%
@lov3kaizen/agentsea-guardrails0.59315.7%3.4%
@andersmyrmel/vard0.59215.9%5.0%
IBM Granite Guardian 4.1 8B (jailbreak)0.58018.2%10.0%
zambo-prompt-shield0.57610.6%4.9%
llm-guard (npm)0.57217.5%12.5%
@zeroleaks/shield 1.2.10.5718.2%1.9%
NVIDIA NemoGuard JailbreakDetect (NeMo Guardrails)0.5652.1%0.0%
llm-firewall0.5648.5%6.0%
vLLM Semantic Router Vela 307M guard0.56319.2%14.2%
llm-prompt-guard0.5567.2%2.0%
Guardion ModernGuard-10.54381.1%77.2%

Five of its sets share a source with Shield's training data (agentic 5k and boundary pairs, neuralchemy, S-Labs, LLMail); BIPIA's test set is separate from the BIPIA training split Shield used. Three are held out from Horizon-Labs' own training generator and favor its models. PIArena and Simsonsun are also in held-out v3.

Speed

Every model ran on the same four cores of an Intel Xeon Platinum 8275CL server with ONNX Runtime, one call per text, on 538 texts drawn from every group (median 206 characters). Each model read text the way its benchmark run did: Shield's in 256-token windows every 192 tokens, the others up to their maximum length.

ModelFileMedian, 1 threadMedian, 4 threads95th percentile, 1 thread
Shield's model236MB25ms10ms592ms
Shield's model, int8 (dtype: "q8")118MB17ms––
Meta Llama Prompt Guard 2 22M284MB35ms18ms486ms
Horizon-Labs guard small, int8268MB44ms20ms1,345ms
Sheltron prompt guard 68M, int8124MB47ms21ms1,589ms
Patronus Wolf Defender, int8218MB72ms31ms2,690ms
Horizon-Labs guard base, int8641MB104ms38ms2,997ms
ProtectAI DeBERTa v3 base v2739MB106ms41ms1,410ms
PIGuard736MB107ms41ms1,374ms
Meta Llama Prompt Guard 2 86M1,116MB108ms42ms1,396ms
Horizon-Labs guard large, int81,479MB367ms102ms7,514ms

In the SDK, detectAsync() with model.options(), including tokenization in JavaScript, took a median of 26ms on one thread and 12ms on four. The large model, through the SDK on the same machine, took a median of 671ms on four threads; long documents, up to 16 windows of 512 tokens, take seconds. In the same run the default model took a median of 38ms on one thread and tiered 43ms, since most inputs never reach the large model; their means were 178ms and 1,333ms. On an NVIDIA A10G GPU, a model of the large model's size took a median of 30ms per request. detect() alone takes about 0.05ms. The int8 file changes about 2% of verdicts and scores 0.885 against 0.889 on the five groups.

How Shield's model was chosen

Candidates for the archived default model were compared on the benchmark sets, v1, and 14 sets of the Horizon-Labs suite. The large model and the tiered setting were chosen later on all five development groups, v2 and v3 included; see Shield's large model. We trained models across seven rounds of training data using five base models (multilingual E5 small, base, and large, mmBERT, and mDeBERTa). E5-small was as accurate as the larger models on this data and several times faster. The threshold stayed at 0.5: moving it from 0.3 to 0.95 changed the development score by less than 0.01. A 6-layer student distilled from the model was twice as fast but scored 0.014 lower. Averaging two models scored 0.003 higher at twice the cost. Neither ships.

How to read these numbers

  • Balanced accuracy is the mean of recall and specificity on sets with both labels, specificity on benign-only sets, and recall on attack-only sets, averaged over the sets in a group. It weighs catching attacks and letting benign text through equally. Recall and false positives are pooled over the group's rows.
  • What counts as an injection. Shield's sets label a plain harmful request, with no attempt to get around the model's rules, as benign, as PIGuard and Horizon-Labs do. Qwen3Guard and Granite Guardian are content-safety models; we scored them by their jailbreak category, which is why they miss most injections.
  • Open models ran at their maximum input length with a sliding window over longer text, scoring the highest window, the protocol Horizon-Labs uses. That helps every model on long documents. The large guard models read the first 3,072 tokens.
  • Paid services. Azure AI Prompt Shields ran through its user-prompt and document checks (a detection by either counts), Amazon Bedrock Guardrails through its prompt attack filter at high strength (Standard tier), and GPT-6 Sol with the prompt in LLM_DETECTOR_PROMPT on up to 6,000 characters. GPT-6 Sol did not run on the Horizon suite.

Not tested

  • Lakera Guard and Google Cloud Model Armor, which need accounts we didn't have for this run.
  • OpenAI's @openai/guardrails, whose jailbreak and prompt injection checks ask an OpenAI model to judge, like the GPT-6 Sol row.
  • Guardrails AI's jailbreak validator, which combines several models; its published classifier is only one part and isn't meant to be used alone.
  • Gated models whose terms we couldn't accept: CodeIntegrity's PromptGuard, Qualifire's and Rogue Security's Sentinel models, and Meta's original Prompt Guard 86M, which ran only on the benchmark sets and held-out v1, from an earlier copy.
  • Enterprise products sold through sales, such as Prompt Security, HiddenLayer, and Cisco AI Defense.

The 13 benchmark sets

SetRowsWhat it tests
deepset_test116Classic injections and goal hijacking, English and German
jackhhao_test262Jailbreak prompts against benign prompts
inthewild2,000Jailbreaks collected from Discord, Reddit, and prompt sites, against regular prompts from the same sources
safeguard_test1,989Synthetic injections against open-instruction requests
multilingual1,272Injections in about a dozen languages
multilingual_gpl1,581Translated injections and benign prompts
bipia_like1,000Documents with an injection inserted at a random sentence, against the same documents without it
llmail_inject1,000Attacker emails from Microsoft's LLMail-Inject challenge (attacks only)
zeroleaks_attacks211Agent attacks from ZeroLeaks' private corpus, many of which landed on frontier models (attacks only)
notinject339Benign prompts built around trigger words, an over-defense test (benign only)
oasst_val1,595Real user prompts from OpenAssistant (benign only)
wiki_docs1,000Wikipedia passages (benign only)
llmail_fp_emails160The LLMail challenge's benign emails (benign only)

Reproducing

The package's benchmark/ directory builds the sets (data/), runs the detectors (competitors/: run_gpu.py for the open models and guards, run_onnx.py, the npm libraries in js/, run_bedrock_guardrail.py, and latency.py), and scores them (report/compare_groups.py).

On this page