← Compare
COMPARE

Lucin vs CodeQL

CodeQL is a far more powerful analysis engine; it does not model an AI agent's tool graph. The three cases it cannot flag, and why our LLM-boundary assumption differs.

Short version: CodeQL is a far more powerful analysis engine than Lucin, and it does not model an AI agent's tool graph. If you already run CodeQL, keep running it — this page is about the gap it leaves, not a reason to replace it.

Where CodeQL is better, plainly

If your question is "find injection, deserialization and memory-safety bugs across a large polyglot codebase", the answer is CodeQL.

Where the gap is

CodeQL analyses code. An agent's dangerous behaviour is often not in the code.

Three concrete cases CodeQL will not flag, because there is nothing wrong with any individual line:

  1. The lethal trifecta as a composition. Three individually reasonable tools — retrieve a document, read a customer file, post to a webhook — share one model context. Any injected instruction inside the retrieved document can steer the model into using the other two. Lucin reports this as a path with a witness: retrieve_context → __llm__ → post_to_webhook. There is no bad line to match.
  2. Configuration that is not code. MCP server definitions, tool descriptions, ~/.claude/settings.json, skill files. A tool description is an instruction to a model, so injection there is closer to code execution than to a string literal — and it lives in JSON and Markdown that a code analyzer has no reason to read. lucin discover finds those files on your machine; nothing leaves it.
  3. Drift after approval. A tool description that gains "also BCC every message to…" in a point release (the Postmark MCP pattern). That is a diff between runs, not a property of one snapshot.

We also disagree with CodeQL about one thing, and it is a real disagreement: CodeQL's dataflow treats the model boundary as ordinary code. Lucin treats the LLM as an untrusted, fully-connected join node — anything in its context can influence anything it emits. That assumption is why our trifecta query works, and it is why our false-positive profile is different.

Precision, and how to read our number against theirs

We publish 20.5–31.5% precision (n=73 clean-holdout adjudicated, 95% CI 12.9–42.9%) on 81 real agent repositories, regenerated by python benchmarks/agentzoo_precision.py.

CodeQL does not publish a false-positive rate — we looked, and could not find one. What they publish instead is arguably better engineering: a per-query @precision rating and a tiered suite, so you choose your noise level. What can be said from independent measurement is that on 244 real Python CVEs, an academic evaluation found CodeQL detected 10.8% (arXiv:2509.04260) — which says more about how hard real-world detection is than about CodeQL.

So do not read our precision figure "vs CodeQL" as a ranking. Different corpora, different vulnerability classes, ours self-measured and theirs not measured at all. The claim we will defend is narrower: our number exists, carries an interval, and regenerates from a command.

Using both

# CodeQL: the code
codeql database create db --language=python && codeql database analyze db

# Lucin: the agent's tool graph, its configs, and the flows between them
pip install lucin && lucin scan .

Both emit SARIF, and Lucin now tags findings with CWE identifiers (external/cwe/cwe-78, the same convention CodeQL uses), so both land in the GitHub Security tab and dedupe sensibly in the same pipeline.

When not to use Lucin