Best LLM Scanners
Home Dispatches Topics
☰
Scanner Picker Tools Glossary Resources Masthead
Defensive / Tools network
AI Defense AI Moderation Tools AI Sec Bench AI Sec Reviews Best AI Security Tools
Clean isometric vector illustration explaining site purpose and meta information structure
site

What this site is for

Best LLM Scanners is a practitioner's comparison of LLM security scanners — Garak, PyRIT, promptmap, vendor scanners — coverage gaps, false-positive profiles, integration cost, and when 'best' depends on what you're defending.

By Best LLM Scanners Editorial · · 8 min read

Best LLM Scanners exists for the engineer who got told to “scan the LLM before launch” and discovered there are a dozen scanners, no shared benchmark, and every comparison online is a feature table written by one of the vendors.

What we publish:

Side-by-side scanner comparisons on the same target. Garak, PyRIT, promptmap, Vigil, Promptfoo, LLM Guard, and the commercial LLM-specific scanners — run against the same model, the same attack corpora, the same harness. Which probe categories each one actually covers, and which gaps are real versus marketing.

Coverage maps, not feature lists. A scanner that ships 200 probes but none for tool-call injection is worse for an agent product than one with 30 well-targeted probes. We map each scanner’s real coverage to attack classes — direct injection, indirect/RAG injection, jailbreaks, data extraction, encoding bypasses — so you can match the tool to your threat model.

False-positive profiles. A scanner that flags every refusal as a vulnerability wastes triage hours. We report each scanner’s false-positive behavior on benign-but-adversarial-looking inputs, because that number decides whether a tool survives contact with a real backlog.

Integration cost, honestly. CI/CD fit, runtime-vs-pre-deploy use, how long a full run takes, how the scanner behaves against a rate-limited API, and what it takes to wire it to a non-OpenAI endpoint. The “best” scanner you can’t fit into a pipeline isn’t the best scanner for you.

What we don’t publish:

  • Vendor feature tables reprinted as “comparisons”
  • “Top 10 LLM scanners” listicles with no runs behind them
  • A single winner — “best” is conditional on what you’re defending, and we say so
  • Any ranking we can’t reproduce from a published methodology

Pseudonymous bylines, consistent across the site so the scoring rubric stays consistent. Tips, corrections, and “this scanner missed an obvious bypass on prod” reports go to the editor.

Real comparisons start shortly.

See also

#meta
Subscribe

Best LLM Scanners — in your inbox

Comparing LLM security scanners and detection tools. — delivered when there's something worth your inbox.

No spam. Unsubscribe anytime.

Related

Isometric vector illustration of LLM vulnerability scanners mapped against OWASP LLM Top 10 coverage
tools

Best LLM Vulnerability Scanners 2026: Garak, PyRIT, Promptfoo, and Mindgard Compared

A practitioner's guide to the best LLM vulnerability scanners in 2026 — Garak v0.15.0, PyRIT, Promptfoo (now OpenAI), and Mindgard. OWASP LLM Top 10 coverage, CI/CD fit, and buyer profiles.
Isometric diagram of a layered LLM red teaming pipeline with attack vectors and tool stages
tools

Open Source LLM Red Teaming Tools: PyRIT, Garak, HarmBench, and What to Use When

A practitioner's guide to the main open source LLM red teaming tools — PyRIT, Garak, HarmBench, TextAttack — what each does, what it misses, and how to build them into a real testing pipeline.
Isometric vector illustration showing automated tools garak, pyrit, promptfoo in CI pipeline for LLM red-teaming
tools

Automated LLM Red-Teaming in CI: garak vs PyRIT vs Promptfoo

Three open-source tools can gate your pipeline on LLM security findings — garak, PyRIT, and Promptfoo. A practitioner comparison of how each fits CI/CD, what it scans, and which to run where.

Comments