Your LLM app has the same five bugs as everyone else's
I’ve been reviewing LLM application code — mine and other people’s — and the same five defects show up in almost every codebase, regardless of language or framework:
- An API key hardcoded in the client constructor, “just for now.”
- Model output passed to
eval,exec, or a shell, because the demo worked and the demo shipped. - Model output rendered as HTML, unsanitized, straight into the DOM.
- User-controlled text interpolated into the system prompt.
- An agent tool whose implementation is “run whatever string the model hands me.”
None of these are exotic. All of them are the old vulnerabilities — command injection, XSS, hardcoded credentials — wearing a new SDK. What’s changed is the threat model: model output is attacker-influenced input. Anyone who can get text into your model’s context — a prompt, a retrieved document, a poisoned webpage — has a say in what comes out. If what comes out reaches a shell, your attacker has a shell.
A config, not a tool
So I built llm-security-rules: a tested semgrep ruleset for exactly these defects, mapped to the OWASP Top 10 for LLM Applications (2025), for Python, TypeScript/JavaScript, and Go.
There are good adjacent projects. agent-audit is a real scanner with real benchmarks — but it’s a separate tool with a custom engine, and it’s Python-only. Semgrep’s own LLM material is guidance for AI coding agents, not a tested ruleset. And nobody covers the TypeScript side, where a huge share of LLM apps actually get written (Vercel AI SDK, Node SDKs, React chat UIs).
This is deliberately not a tool. If semgrep already runs in your CI, this is one line:
semgrep scan --config https://github.com/gmhoward9289-ops/llm-security-rules/archive/refs/heads/main.tar.gz .
There’s a GitHub Action, a pre-commit hook, and — because a lot of LLM app code is now written by AI agents — a Claude Code skill that makes the agent scan its own output before it commits.
What “tested” means here
Every rule ships with fixtures that pin both directions: code that must fire,
and near-misses that must not. The near-misses are where rulesets earn trust.
A masked key like sk-proj-****T3BlbkFJ**** must not fire. A key leaked in a
comment must fire. A too-short sk-ant- stub must not. CI runs
semgrep --validate plus the full fixture suite on every PR — if a rule is in
the repo, there’s a machine-checked proof of what it catches and what it
deliberately ignores.
Taint mode does the heavy lifting where it matters: model output is tracked through assignments and destructuring, so renaming a variable doesn’t dodge the rule.
const { text } = await generateText({ model: "openai/gpt-4o", prompt });
exec(text); // caught — taint flows through destructuring
@tool
def run_python(code: str) -> str:
return str(eval(code)) # caught — tool args are model-chosen
And the React rule is sanitizer-aware — route the output through DOMPurify
first and it stays quiet, because the danger is gone, not because you renamed
something:
<div dangerouslySetInnerHTML={{ __html: modelOutput }} /> // caught
<div dangerouslySetInnerHTML={{ __html: DOMPurify.sanitize(modelOutput) }} /> // clean
Three lessons the rules taught me
Precision claims die on contact with real code. The first field test
scanned a real non-LLM TypeScript codebase. One finding: a “Gemini API key” —
which was actually a Firebase web client key. Same AIza prefix, and Firebase
web keys are designed to ship publicly. The rule now says what it can
actually know (“Google API key — check what it’s enabled for”), fires as
WARNING, and that exact near-miss is a fixture. If your secrets rule can’t tell
you its own known false positive, it’s not done.
A secrets ruleset trips over itself. The fixture file contains key-shaped
strings — necessarily, that’s what the rules detect — which means GitHub’s push
protection blocks the repo’s own first push, and any credential scanner run
against it reports findings on a security repo. The fix is to name that one
file, by exact path, in both .github/secret_scanning.yml and a .gitleaks.toml
allowlist — before the first push, because afterward the blobs are already in
history. File that one under “problems you only meet once.”
The repo’s first vulnerability report was against itself. While writing the
SECURITY.md, I found a script-injection hole in my own GitHub Action:
${{ inputs.paths }} interpolated straight into a run: block. GitHub expands
${{ }} textually before bash ever sees the script, so a consumer who wires
paths to a PR title in a pull_request_target workflow is handing command
execution to whoever opens a PR. The fix is the boring, correct one — route
inputs through env: so they arrive as data — and there’s now a CI job that
fails the build if any expression is ever interpolated into a shell block
again. The rules scan your code, but the packaging is code too, and the threat
model doesn’t care which directory it lives in.
What’s next
Full Top-10 coverage where it can be made precise (LLM08 and LLM10 stay out until they can be — fuzzy heuristics get rulesets uninstalled), AutoGen and Spring AI shapes, and a benchmarked precision corpus. The 2–3 strongest rules are headed for the semgrep registry.
MIT licensed. If you ship LLM code, point semgrep at it and tell me what it gets wrong: real false-positive reports are the most valuable contribution a ruleset can receive.
From my swamp to yours.