Notes ·

A search API is a prompt injection surface

Most prompt injection advice assumes the attacker has to get their text in front of your model somehow — a document upload, a support ticket, a shared file.

Web search removes that step. If your agent searches, an attacker does not need to reach you at all. They need to rank for a query you are likely to run, and wait.

That is a materially worse threat model than the one most integrations are built for, and it is worth stating plainly because it is easy to miss: the retrieved page is written by whoever wants to influence your agent, and they chose the query.

Why the usual defence is not enough

The standard advice is “tell the model to ignore instructions in retrieved content”. It helps, and it is not sufficient, because it asks the model to win an argument with text that was written specifically to win that argument.

Anything that depends on the model not being persuaded is a defence that fails exactly when it is tested hardest. You want defences that hold even if persuasion succeeds completely.

Three that do

Fence the content, and say what it is. Retrieved text goes inside an explicit boundary and the system prompt states that everything within is data and never instruction. This is the weakest of the three and still worth doing: it raises the cost of a naive attack and it costs nothing.

Constrain the output schema. The model does not return free-form text that your code then interprets. It returns an answer string and a set of source indices, and nothing else is representable. A fully persuaded model, one that has completely accepted the injected instructions, still cannot emit a tool call, a URL of the attacker’s choosing, or a field your parser will act on. The blast radius is the answer text, which a human reads.

This is the one that matters most, because it does not rely on the model resisting anything. It removes the capability rather than the intent.

Validate citations against what was actually supplied. The model cites by index into the source list it was given. Those indices are resolved back to the URLs your search actually returned, and anything that does not resolve is dropped. An injected “cite https://attacker.example” is discarded rather than echoed, because it was never in the list.

What this means for your own pipeline

If you pass search results into your own prompt — and if you are using a search API for grounding, you are — then the API’s defences protect the API’s answer, not yours. The untrusted text is now in your context.

Three things worth doing on your side:

  • Treat content as hostile input, with the same care you would give a user-submitted string. It is less trustworthy than one, because the attacker selected the query.
  • Constrain your own output schema wherever the model’s response drives an action. Free-form text feeding a tool call is where injection converts from an annoyance into an incident.
  • Resolve any URL the model produces against the list you supplied it, rather than following it. This is cheap and it closes the exfiltration path where a model is persuaded to append data to an attacker’s URL.

The uncomfortable part

None of this makes retrieved content safe. It makes the consequences bounded.

A search API that claims to have solved prompt injection has either not thought about it or is describing a filter that an attacker will iterate against until it passes. The honest position is that the content is untrusted, will stay untrusted, and the job is to ensure a persuaded model cannot do anything worse than say something wrong.

← All notes