System Prompt Security: How Much It Actually Buys You

Andrew Maryasov — AI consultant, founder of Auspex and Grow2.ai; implementation is run by Auspex (CRM) and Grow2.ai (AI agents). A system prompt isn’t a security layer in the sense that access rights are. It’s a shift in probability, and it has finally been measured: a Stanford team showed that even when the system prompt explicitly lists the vulnerabilities to avoid, the model still writes code with holes in it 23% of the time. The best prompt, one that a machine polished over 80,000 requests, cuts that share by about forty percent. Not to zero.

TL;DR (in 30 seconds)

• 23% — the share of cases in which a flagship model generates a verified vulnerability even though it was explicitly asked to write secure production code and was given, in context, the weaknesses to avoid. Measured on a corpus of 250 benign requests.
• About forty percent of those vulnerabilities are removed by SecureForge, a Stanford pipeline that automatically optimizes the system prompt (up to 48% in the paper’s best case). That means one program in eight still has a hole in it even after optimization.
• What matters is the method, not the result: the prompt could be improved only because it had an external detector alongside it — the static analyzer Semgrep. Without it there would have been nothing to optimize against and nothing to verify with.
• The limit The Batch points out: only the developer prompt was optimized, and only against known classes of vulnerabilities. The authors add another one themselves: a prompt optimized for one model doesn’t make a different model secure.
• The one-question test: who checks that this rule is followed? If the answer is “the model itself,” it’s a bet with tilted odds, not protection. If the answer is “a system that can say no,” it’s a control.
• A prompt is worth writing well, and doing so has a measurable effect. It’s just that the effect is measured in percentages, not guarantees.

“But we put that in the prompt”

On 25 April 2026, an agent in Cursor deleted the production database of a small SaaS along with the available backups. Nine seconds from decision to irreversibility, about thirty hours of downtime. Then the agent wrote an explanation and listed, on its own, the rules it had broken: it didn’t verify which environment the volume belonged to; it didn’t read the documentation; it didn’t ask for confirmation before a destructive action. All three were written into the system prompt, which the model read before every step.

I went through this incident in detail in the piece on an AI agent’s blast radius — and the conclusion there was harsh: a prompt is text the model weighs along with all the rest of the context, while access rights are enforced without the model’s involvement.

From here it’s easy to take the next step, and people take it all the time: if the prompt doesn’t save you, there’s no point in writing it carefully, and you should go straight to building a sandbox. That step is wrong, and now that’s a matter of numbers, not convictions.

How much “ask nicely” is worth

On 8 May 2026, a Stanford team — Houjun Liu, Lisa Einstein, John Yang, Joachim Baumann, Duncan Eddy, Christopher D. Manning, Mykel Kochenderfer and Diyi Yang — published the paper SecureForge. In August, The Batch covered it.

The question sounded practical: can you automatically build a system prompt that makes the model write less vulnerable code?

First the authors measured the baseline. Models were given a benign coding request, explicitly asked to write secure production code, and given, in context, the classes of weaknesses to avoid. So not “be careful,” but a properly worded requirement with specifics — roughly what a sensible engineer would write. On a corpus of 250 such requests, the models produced a statically verified vulnerability 23% of the time on average. A caveat about baselines right away: 23% is the figure from the paper’s abstract, while the table below counts a narrower slice (only code that passed unit tests) and gives 20.1%. These aren’t two different results, just two different denominators.

Then they built the pipeline. The starting point was the top 25 vulnerabilities in MITRE CWE, an industry catalog of typical security mistakes. In this experiment, GPT-5.4 turned each catalog entry into a realistic everyday request: for “CWE-89: SQL Injection,” that would be, for example, “Write a Python function that looks up a user by username in a SQLite database.” Explicit mentions of vulnerabilities were scrubbed from the requests in a separate pass, so the model wouldn’t be handed a blunt hint.

Each request was fed to a model, and the generated code was run through unit tests (to check that it actually does what was asked) and through Semgrep, a static analyzer that looks for vulnerabilities by rules. They kept the requests where the code came out with holes in it: 20 for each of the 25 CWE entries, 500 in total. Half were set aside for the final evaluation, and the remaining 250 were expanded into 80,000 variations — by rewriting each request into a slightly different task and keeping the rewrites that kept breaking the model.

And it was on those 80,000 that the system prompt was optimized with the GEPA genetic algorithm: the prompt was run on a batch of requests, Semgrep reported exactly where the vulnerability occurred, and GPT-5.4, acting as the optimizer, read those reports together with the current prompt and proposed the next version. The cycle repeated until the improvements ran out.

The lineup of models in the study: CodeLlama 7B, Qwen2.5-Coder 7B and 14B, Qwen3 30B-A3B, Kimi K2, Claude Sonnet 4.6, GPT-5.3 Codex and the entire GPT-5.4 line.

The result: forty percent better and still full of holes

On the 250 held-out tasks, SecureForge was compared with that same “ask for secure code and list the vulnerabilities.”

Study results on the held-out set:

Model	Human-written prompt	Prompt optimized by SecureForge
All models in the study	20.1%	11.8%
GPT-5.4	15.8%	10.1%
GPT-5.4 Mini	21.4%	12.4%
GPT-5.4 Nano	22.1%	15.6%

Per the study: the share of programs with a detected vulnerability. The table includes only programs that passed the generated unit tests, meaning working code, not garbage. The paper’s abstract gives 23% for the baseline prompt, and states the best optimization result as vulnerabilities reduced by up to 48%.

The difference is real and reproducible: the wording of a system prompt shifts the probability by about forty percent, and that isn’t a matter of taste but a measurement with a control group. Anyone who says “the prompt doesn’t matter” is wrong in exactly the same way as anyone who says “a prompt is enough.” On top of that, the optimized prompts transfer — the authors tested them on real requests to coding agents that the algorithm never saw during optimization, and the gain held.

The unpleasant part of the same table: 11.8% is the result of a machine that had 80,000 attempts and an independent judge beside it. Your system prompt, written by hand on a Friday evening, starts from a worse position, and how much worse you don’t know, because you haven’t measured.

What this study actually proved

The method is more interesting than the result. The prompt could be improved precisely because Semgrep was sitting right next to it. It defined what a failure is, showed which line the failure happened on and gave the algorithm a signal to follow. Take the static analyzer away and there’s nothing to optimize against and nothing to verify the result with. What’s left is a feeling that the prompt got better.

The paper’s title promises prompt optimization (“via Prompt Optimization”), but what it proves is the power of external verification. It was the prompt that got tuned, but it was Semgrep that made the tuning possible, and in prod it isn’t going anywhere: the prompt doesn’t replace it, the prompt is fitted to it.

There are two limits here, and they shouldn’t be confused. The Batch adds one of its own: only the part of the system prompt that a developer can change was optimized, and only against known classes of vulnerabilities from the CWE catalog — how it behaves with a class that isn’t in the catalog, nobody checked. The authors, in their limitations section, write about something else and, to my mind, something more unpleasant: a prompt optimized for one model doesn’t necessarily make a different one secure. So someone else’s “secure system prompt,” copied into your project for a different model, is a bet again. Running the procedure yourself costs, by the authors’ estimate in the paper itself, about 150 dollars in one-off token costs, and the code for it is published.

One more detail that’s easy to miss. The same issue of The Batch mentions the system card for Claude Mythos 5, a model that finds serious vulnerabilities in other people’s code on its own. According to that summary (I haven’t checked the card itself), it documents cases where the model falsely claimed to have verified code, missed important failures and overlooked security issues in what it was analyzing. That closes off the shortcut “let a second model check the first”: the second model needs an external judge too. I’ve already picked apart a similar trap in the case of an agent’s self-improvement without verification — a feedback loop without independent measurement teaches nothing; it just entrenches confidence.

The one-question test

The practical part comes down to one check you can do in ten minutes on any of your agents. Take the system prompt, write out every security rule and, for each one, answer: who checks that it’s followed?

Rule in the prompt	Who checks that it’s followed	What it really is
“Don’t delete without the user’s confirmation”	The model itself	A bet
Same + deletion via the API is a 48-hour soft delete	The platform	A control
“Don’t use credentials unrelated to the task”	The model itself	A bet
A token scoped to one project and environment	The provider	A control
“Write secure code, avoid SQL injection”	The model itself	A bet with tilted odds
Same + Semgrep in pre-commit	The linter	A control
“Don’t output personal data in responses”	The model itself	A bet
Same + the agent physically has no access to the table with personal data	Access rights	A control

“A bet” isn’t an insult. A bet with tilted odds is cheap, takes five minutes to apply and, according to the Stanford data, removes about forty percent of cases in its class. It just doesn’t do what people expect of it in a meeting when they say “we’ve handled that in the prompt.”

The difference becomes visible at the moment of an incident. When a bet doesn’t work, you find out after the fact, from the consequences. When a control works, you see it in the denial logs, before the consequences.

Four times in a row, and the prompt didn’t help

An example from my own backyard, so it doesn’t look as if I’m only here to watch other people’s mistakes.

I run an editorial team made of agents: some write the pieces, others deploy them to the brands’ sites. The instructions for the agent responsible for publishing say in so many words: before closing the task, check that the pull request is merged and that the page actually opens. Not “PR created,” but “the article is live at its URL.”

That rule was broken four times, three of them within four days in mid-August. In July I described this very loop as a reliable gate — and that was too optimistic: the gate checks the file’s fields, not the live page, so a green build and a published article are different events. The agent would open a PR, see a green build and close the task as done. The article stayed in the branch, the link to it didn’t exist, and the chain of posts that were supposed to point to that article quietly sat there waiting for an address that wasn’t there.

The funniest part is how the automated check reacted. A formal audit of a piece in that state gave it 10 out of 10, because it looked at the fields in the file’s header, not at whether the page existed. Two green check marks in a row, and neither one about the right thing.

You catch this not by editing the prompt but with curl: a request to the live URL that bypasses the cache, and a search of the page body for a specific phrase from the text. In other words, with an external checker — the same Semgrep, only for publishing.

I rewrote the prompt too, of course. But I no longer count that as a fix.

What’s worth putting in a system prompt today

Name the class of error, not the virtue. “Write secure code” is a wish. “Parameterize all SQL queries; concatenating strings into a query is forbidden” is a rule the model can follow. Stanford’s baseline 23% was obtained with the second kind; with the first it would have been worse.

Write a procedure for uncertainty, not a prohibition. The agent’s costliest mistake in the PocketOS incident happened not when it destroyed something but a few seconds earlier — when it ran into a mismatch and decided to sort it out on its own. The prohibition “don’t take destructive actions” doesn’t describe what to do instead. “If you run into a mismatch between the expected and the actual state, stop and return a description of the mismatch” does.

For every important rule, an external checker. A linter, a static analyzer, a test, human confirmation, token scope, soft delete. A rule without a checker stays in the prompt; you just know its true weight now.

Measure your prompt instead of trusting it. The SecureForge method scales down to one person: take twenty typical requests to your agent, run them through two versions of the system prompt and count the failures against a formal criterion. That’s an evening of work, and after it you talk about the prompt in numbers. I use the same approach when I evaluate a new AI tool: the failure criterion first, then the demo.

And on a separate note, because this is what gets forgotten most often: a system prompt is not a vault. It ends up in logs, in traces and in the model’s response when an instruction-leak attack succeeds. Keys, internal addresses and personal data don’t belong in it; where to keep them is something I covered in the piece on how to keep client data inside the company.

What not to put in the prompt

Don’t put anything there that has to hold regardless of how the model weighed the context: access rights, the irreversibility of operations, spending limits, the boundary between environments. In a prompt, each of these looks like a week of work saved and works like a promise.

And separately — don’t count on a longer prompt being a safer one. In SecureForge the gain came not from the volume of text but from specific wording matched to real failures. A long prompt with twenty general commandments competes with itself for the model’s attention; the same logic applies in the neighboring problem, when an agent optimizes the metric instead of the outcome.

FAQ

Is a system prompt a security layer?

Yes, but not in the sense that access rights are. A prompt shifts probability — by about forty percent according to Stanford’s measurements, and even that in a narrow class: generating vulnerable Python code in a single turn. Access rights determine what is physically possible to execute. The first guarantees nothing; the second isn’t up for discussion with the model. The right design is both layers, with the prompt as the first one, not the only one.

Why can’t I just write “don’t do anything dangerous” in the prompt?

Because the model doesn’t share your definition of dangerous. The Stanford study shows the difference in numbers: a general requirement works worse than a named class of error, and what works best is a prompt fitted to real failures from an independent detector. Plus the PocketOS example: the ban on destructive actions was there, and the agent quoted it after it had already broken it.

How much security does a well-written system prompt give you?

According to SecureForge, from 20.1% down to 11.8% on average, that is, minus forty percent relative to the baseline; the paper’s abstract states the best result as vulnerabilities reduced by up to 48%. Both figures apply to a narrow and easily measured class: known vulnerabilities in Python code generated in a single turn. For data leakage and tool calls there are separate benchmarks such as AgentDojo, but those measure an agent’s robustness as a whole, not the system prompt’s contribution, so you can’t carry “minus forty percent” over to them.

Does a system prompt protect against prompt injection?

Don’t count on it. An injection works at exactly the level where the prompt lives: it mixes into the context text that the model weighs along with your instructions. The defense against it is built on the outside — isolating untrusted input, restricting the available tools, requiring confirmation for actions with consequences. A prompt can make an attack less likely, and that’s where its role ends.

Where do I start if the agent is already in prod?

With the ten-minute exercise using the table above. Write out the security rules from your system prompt and, next to each one, who checks that it’s followed. Move everything that has “the model” next to it to a risk list, then close the items one by one: first what’s irreversible, then what’s expensive.

Can I optimize a prompt automatically myself, without Stanford?

Yes, and the SecureForge code is openly available — github.com/sisl/SecureForge, and the license allows commercial use. But the cheaper part comes first: a formal failure criterion and twenty real requests. Without them there’s nothing to optimize against — exactly as in the study itself.

What’s next

A system prompt is worth writing carefully. It has a measurable effect, costs little and starts working the same day. It’s just that the effect lives in percentages, not guarantees, and every rule in it carries exactly the weight that the external check next to it gives it.

The cheapest action this week: open your agent’s system prompt and, next to every security rule, write the name of whoever checks that it’s followed. If the column says the model’s name all the way down, you know your next step.

If you have nothing to fill the checker column with, send me your system prompt through the form — I’ll show you which lines in it have external support and which ones rest on the model’s goodwill.

See also:

• AI Agent Blast Radius: How to Measure Risk Instead of Counting Tokens — four mechanisms that narrow the damage without swapping the model.
• Reward hacking: how an AI agent will break your metric — why a prohibition in the prompt loses to the optimization gradient.
• AI Agent Self-Improvement: Why It Doesn’t Work Without Verification — on the same dependence on independent measurement.
• Why AI Agents Get It Wrong: Blame Retrieval, Not the Model — the neighboring layer, where the model isn’t the one to blame either.

Sources:

• SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization — Houjun Liu, Lisa Einstein, John Yang, Joachim Baumann, Duncan Eddy, Christopher D. Manning, Mykel Kochenderfer, Diyi Yang (Stanford), 8 May 2026.
• Custom Prompts for Safer Code — coverage in The Batch issue of 7 August 2026: method, models, comparative figures.
• SecureForge repository — Stanford Intelligent Systems Laboratory, the pipeline’s open-source code.
• CWE Top 25 Most Dangerous Software Weaknesses — MITRE, the catalog the authors took their vulnerability classes from.
• Semgrep — the static analyzer that served as the independent detector.
• Nine Seconds to Total Loss: The PocketOS Agent Database Deletion — a technical post-mortem of the incident, May 2026.
• Your AI wants to nuke your database. Guardrails fix that. — Railway on token scope levels and delayed deletion in the API, 29 April 2026.