Every agent security tool grades its own homework

A proposal for an open benchmark, and our own scores on it

There are at least a dozen commercial products for securing AI agents. Every one publishes performance numbers. Not one publishes a harness you can run.

I have spent a while reading their marketing pages, and the pattern is consistent: a detection claim, a percentage, occasionally a confidence-inspiring benchmark name, and no way to reproduce any of it. "Detects 200+ attack techniques." "Blocks 95% of prompt injections." "Covers the OWASP LLM Top 10." Each of those is a claim about a measurement, made without the measurement.

I want to be careful here, because I sell a competing product and there is an obvious self-serving version of this argument. So: the vendors are not lying. Their numbers are probably real, generated by real internal evaluations, on corpora they built. The problem is structural rather than dishonest. When each vendor builds its own test set, chooses its own scoring, and publishes only the aggregate, the numbers are not comparable to each other and not checkable by you. They are not evidence. They are assertions with decimal points.

I am proposing a shared benchmark, and submitting my own tool to it first, including the results that make it look bad.

Why this category has no benchmark and others do

Application security has managed this before. There is OWASP Benchmark for SAST tools, Juliet for static analyzers, DaCapo for JVM performance. They are imperfect, gamed at the edges, and vastly better than nothing, because they create a shared axis. Once an axis exists, "we score higher" becomes a checkable claim rather than a slogan.

Agent security has none, for three reasons that are worth stating because they constrain any proposal.

The subject is young. The OWASP Top 10 for Agentic Applications — the first flagship list built for autonomous agents rather than the models under them — was only announced on 9 December 2025. A taxonomy is a prerequisite for a benchmark, and we have had one for eight months.

Nobody agrees what the unit of analysis is. Static scanners look at code. Runtime firewalls look at prompts and tool calls. Red-team tools generate attacks and measure success rate. These are not three implementations of one task; they are three tasks. A benchmark has to pick, or explicitly partition.

And the incentive runs the wrong way. A vendor with a good internal number has nothing to gain from a shared axis that might disagree with it. This is why benchmarks in commercial categories almost always come from academia or from the smallest participant — the one with nothing to protect.

That last one describes me exactly. Lucin has, at time of writing, effectively no users. I am proposing a standard partly because I would benefit from a world where claims are checkable, and I would rather say that than pretend to neutrality.

The academic work that already exists

Two papers matter here, and the second one is the reason I think this is worth doing now.

AgentFuzz — published as "Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-based Agents," USENIX Security '25, from Fudan University and UC Davis. Directed greybox fuzzing combined with static taint analysis: an LLM generates functionality-specific seed prompts, a multifaceted feedback function scores seeds on both semantic coherence and control-flow distance to a sink, then mutators refine them. They found 34 high-risk zero-day vulnerabilities across 20 widely-used open-source agent applications, were assigned 23 CVEs, and report 33× higher precision than the prior state of the art. Fourteen of the affected agents had code-injection vulnerabilities; seven had over 10,000 GitHub stars.

That is what a real evaluation looks like in this space, and it is dynamic rather than static — which matters, because it establishes that taint-style vulnerabilities in real agents are abundant and findable. It also honestly describes the limitation of pure static approaches to the same problem: high false-positive and false-negative rates on source-to-sink call chains. I publish a 24% false-negative rate, so I am not going to argue with them.

AgentFlow — "Building Agent Dependency Graphs for Static Analysis of Agent Programs," arXiv 2607.01640, submitted 2 July 2026, from Huazhong University of Science and Technology and Macquarie University. It defines an Agent Dependency Graph: typed nodes for agent units, prompt contexts, model units, capabilities, memory states and control policies, decomposed into component-dependency, control-flow and data-flow subgraphs. It hand-models 143 framework constructs across five frameworks.

It is the closest published work to what Lucin does. And there is no public code or dataset artifact — the paper describes an 11.4 KLoC Python and 4.5 KLoC TypeScript implementation with no availability statement, and the AgentZoo benchmark of 5,399 programs is not published either.

Which is the situation in miniature. The best formalisation of this analysis exists as a paper you cannot run, and the deployed tools exist as products you cannot evaluate. There is no artifact anywhere.

What the benchmark needs

Six requirements. Each one exists because its absence is how a benchmark becomes marketing.

1. A public, pinned corpus. Real vulnerable agents from real repositories, at pinned commit SHAs, plus clearly labelled constructed cases. The distinction between real and constructed must be per-case and machine-readable, because constructed cases are easier and a corpus that hides the ratio is reporting a different number than it appears to.

2. Per-class recall, never aggregate. My own recall is 76%, and that number is close to useless on its own. Broken out it is six classes at 100%, one at 80%, and two — path traversal and SSRF — at 0% and 17%. The aggregate hides the only information a reader needs, which is which classes are covered. Any benchmark that reports a single recall figure has failed at the first requirement.

3. Precision on a separate benign corpus, with the adjudication rubric published. Precision requires deciding whether a finding on ostensibly clean code is a true positive, and that decision is where the number is made or fabricated. So the rubric goes in the repository. Mine is at benchmarks/ADJUDICATION_RUBRIC.md; it is not perfect, and it is at least arguable-with.

4. A published false-negative list. Every miss, by case name. This is the requirement that would take the most from vendors and give the most to users, and it is the one I expect to be refused. A recall number without the miss list is a claim about a set the reader cannot inspect.

5. One command. If reproducing the result takes a conversation with a sales engineer, it is not a benchmark.

6. Declared scope. Static pre-deploy, runtime enforcement, and adversarial testing are three different tasks. A tool declares which it is competing in, and cross-category comparisons are rejected rather than fudged. This is what stops the benchmark from turning into a category war.

What each vendor currently publishes

Assembled from public marketing pages. Every performance figure is a vendor claim, unverified — which is the point of the table.

I am aware of how that last row reads. It is five checkmarks against four empty rows, in a table I built, in a post I wrote. Treat it as an invitation rather than a result: the columns are the argument, not the row. If any of those vendors fills a column, the table improves and I lose the only structural advantage I have. That would be a good outcome and I mean it — a category where everyone publishes their misses is a category where I have to compete on the misses being fewer.

Our scores, including the bad ones

Submitting first, on the six requirements.

Scope declared: static pre-deploy analysis of agent tool code. Not runtime, not red-teaming.

Recall: 38/50 = 76% across 10 classes, on a held-out corpus of 22 real third-party cases and 28 labelled constructed cases. Real cases alone: 19/22 = 86%.

python benchmarks/recall_corpus.py

Path traversal is 0% deliberately. The detector is written, sound and unit-tested, and unregistered — the benign corpus contains byte-identical legitimate file tools, so registering it would have produced noise. I chose the honest 0% over 12 points of headline recall. That decision is arguable and the detector is in the repository if you want to argue with it.

SSRF is 17% because the detector only fires when tainted input forms the URL host. Coverage of that class is close to a placeholder.

False positives: 11 adjudicated across 54 real repositories, 9,520 files, counted per distinct (file, detector-id) pair against a published known-capability list. 45 of the 54 scan completely clean. python benchmarks/build_benign_corpus.py

That figure was wrong in public for months. It read "0 across 52 repositories, 2,732 files" long after the corpus had grown, because the corpus was re-run and the sentence was not. It was corrected on 4 August 2026, and benchmarks/regression_snapshot.py now fails the build when a published number drifts from what the harness prints. A benchmark proposal written by someone who has never had to retract a number is not worth much.

And the number that corpus cannot give you: precision on a broad, unscoped population is not currently measured. An earlier figure was withdrawn because it had been computed over the same adjudication labels used to build the precision filters, which is training on the test set. Withdrawn is the honest state, and stating it is the point — a benign corpus tells you how noisy a tool is on code chosen to be clean, and nothing at all about how it behaves on code nobody curated.

False negatives: all twelve published by name, printed by the benchmark, written up separately.

Known structural limit: taint analysis is intraprocedural plus a limited same-file cross-function pass. Not a whole-program call graph. Recall on heavily dynamic Python — getattr, reflection, generated tools — will be worse than 76%, and I have not measured how much worse.

What I am asking for

A repository. Corpus, rubric, harness, and a submissions directory where any tool can add a result — including closed-source ones, via a published harness they run themselves and sign.

The last part matters. A benchmark that only open-source tools can enter is a benchmark that measures openness, and the commercial products are where most of the deployed surface is. Let closed tools self-report through a fixed harness with a signed result. It is weaker evidence than a reproducible run, and it is far stronger than a marketing page.

Three things I would want from anyone who thinks this is worth doing:

Cases. A vulnerable agent I do not have, with provenance. This is the most valuable contribution and the one I most need — my corpus is 50 cases and it should be 500.

Disagreement with the rubric. If my adjudication is too generous to me, say where. That is the criticism I cannot generate myself.

A second implementation. Anything that reads agent code and reports findings. Two tools on one corpus is the minimum for the axis to mean anything, and right now the axis has one point on it, which is not a benchmark. It is a self-portrait.

I have sent the methodology to the OWASP GenAI Security Project, whose agentic taxonomy this maps to. If it belongs anywhere long-term, it is with a neutral body rather than a vendor — including this one.

The argument underneath

The reason to care is not academic tidiness.

If you are deploying agents with tool access, you are being asked to choose a security product from a set of unfalsifiable claims. You cannot compare two vendors, because their numbers measure different things on corpora you cannot see. You cannot check either one, because neither ships a harness. So the decision gets made on brand, on procurement relationships, and on which demo found the scariest thing — and none of those correlate with whether the tool will catch the finding that matters in your repository.

That is bad for you. It is also bad for the tools, because it means the winning strategy is impressive demos rather than measured precision, and the incentive gradient points away from the work that would actually help.

A shared benchmark changes what it is rational to compete on. That is worth more to this category than any individual product in it, including mine.

The corpus, the rubric and every number above are in the repository. Take them, disagree with them, or improve them.

Everything in this post, reproducible

Recall, per class, with the miss list:  python benchmarks/recall_corpus.py
Precision on the benign corpus:         python benchmarks/build_benign_corpus.py
The adjudication rubric:                benchmarks/ADJUDICATION_RUBRIC.md
Corpus provenance and pinned SHAs:      benchmarks/recall_corpus/manifest.json, corpus_shas.json

References. AgentFuzz — Make Agent Defeat Agent, USENIX Security '25 · AgentFuzz code · AgentFlow, arXiv 2607.01640 · OWASP Top 10 for Agentic Applications