An LLM security scanner you can run from CI
Each scan attacks your agents and their tools, then returns a score, findings ranked by severity, full transcripts, and a fix for each one. Run it again and the score shows whether your last change helped.
Pick the scan that matches your stage
Different points in a project need different fidelity. A deployed agent is not meaningfully tested by scanning its prompt in isolation, and an agent that only exists on a developer machine still needs to be tested. ZeroLeaks exposes several scan types through one API.
- Standalone prompts
- Test a system prompt before an application exists around it. The source-available zeroleaks CLI runs the scan locally against your own model provider, ideal while instructions are still being drafted.
- Runtime scans
- Highest fidelity. Probes execute through your real application code with your models, tools, memory, and middleware in the loop, and credentials stay in your infrastructure.
- Endpoint scans
- Point the scanner at a deployed HTTP agent. Exercises the production contract including authentication, response parsing, and declared tools.
- Skill scans
- Static and behavioral analysis of agent skills and manifests from a repository, URL, or archive, for checking capability before it is installed.
- Framework adapters
- First-class adapters for the AI SDK and OpenAI, covering tool schemas, executions, and multi-turn state without writing a custom harness.
- CI integration
- The SDK and REST API are built for pipelines, so a scan can gate a release on its score and a regression fails the build rather than reaching production.
What a scan produces
The output is a report rather than a log. Each run yields an overall security score, a list of findings with severity, the full attack tree showing which paths were explored and which were pruned, conversation transcripts, and for runtime scans the actual tool calls with their arguments and results.
Severity reflects what was reachable rather than a generic category rating. A tool-abuse finding on a read-only lookup and one on a credential-bearing write are not the same risk even when the technique is identical, and the report distinguishes them.
Reports are available through the dashboard, the API, and as PDFs, and can be shared through a link for stakeholders who need the result without access to the platform. That tends to matter for security review and procurement, where the audience for the evidence is not the team that ran the scan.
Scanning as a regression test
The highest-value way to use a scanner is not the first run. It is the hundredth. The first run tells you where you stand. Every subsequent run tells you whether the change you just made broke something that previously held.
This matters more for AI systems than for conventional code because the failure is non-obvious. Tightening a system prompt to fix a support complaint can weaken an instruction that was holding an injection boundary. Adding a tool to an agent expands the reachable surface for every existing probe. Neither change looks like a security change in review.
Running scans in CI puts that check where the change is made. Scores are comparable across runs, so a drop is attributable to a specific pull request, and a fix can be confirmed to hold rather than assumed to.
Scanning and runtime defense are different jobs
Scanning is a development-time activity. It finds vulnerabilities so they can be fixed, and its value is measured in issues resolved before release. Runtime defense is a production activity, blocking or sanitizing traffic in the moment, and its value is measured in attacks stopped.
Teams sometimes treat these as competing purchases. They address different failure modes. A scanner will not stop an attack in progress, and a runtime filter will not tell you that your retrieval layer treats untrusted documents as instructions.
The Shield SDK covers the runtime side with local prompt hardening, injection detection, and output sanitization, including n-gram matching that catches system prompt fragments in output. It is heuristic and it says so plainly: it catches known patterns quickly and does not claim to catch novel semantic attacks or complex multi-turn grooming. Those are what scanning is for.
Questions
Email us if yours isn't here.
What does an LLM security scanner test?
It tests how an AI system responds to adversarial prompts, whether hidden instructions leak, and whether tools or agents can be abused. Unlike a code scanner it evaluates runtime behavior, because that is where the vulnerability lives.
Can ZeroLeaks be used before deployment?
Yes. Runtime scans work in CI so changes are tested before they ship, and the source-available zeroleaks CLI scans a standalone system prompt before any application exists.
How does this differ from an open-source scanner?
Open-source tools like garak, PyRIT, and Promptfoo are capable and worth knowing. The practical differences are the adaptive attack tree rather than a fixed probe battery, the relay protocol for testing production agents without exposing credentials, and reporting built for sharing with people outside the engineering team.
Do you have an open-source option?
Yes. The Shield SDK for runtime hardening and the @zeroleaks/sdk client are MIT-licensed. The zeroleaks npm package is a source-available scanner and CLI (FSL-1.1, which converts to Apache 2.0) that runs extraction and injection scans against a system prompt using your own model-provider credentials. All three are separate from the hosted platform.
How is the security score calculated?
From the findings a scan produces, weighted by severity, with leak depth factored in for leakage findings. The absolute number matters less than its movement across runs, which is what makes it usable as a regression signal.
Can I run scans from my own pipeline?
Yes. The SDK is server-side and calls the versioned REST API with a bearer key, so scans can be triggered from CI, a cron job, or an internal service, and results can be fetched programmatically.
Does the scanner need access to our infrastructure?
Not for endpoint scans, which need only a reachable HTTP URL. For runtime scans the relay protocol inverts the connection, so your SDK process claims probes and executes them locally. Credentials stay with you and the target needs no public URL.
Related testing
- Prompt Injection Testing for AI SystemsTest AI applications for prompt injection, instruction hijacking, tool abuse, and multi-turn attacks before they reach production.
- AI Red Teaming for LLM Apps and AgentsAutomated AI red teaming for LLM applications, agents, system prompts, retrieval flows, and tool-calling workflows.
- AI Agent Security TestingTest AI agents for prompt injection, unsafe tool calls, over-permissioned actions, secret exposure, and workflow boundary failures.
