← All lab updates

Inside a model, an order from the user, an order planted in a document and plain data look different

A prompt injection works because the model cannot tell an order from its user from an order hidden in the text it was asked to read. We looked inside an open 7B model while it read more than six thousand such cases, and found that its internal state already marks which words are the user's order, which are a planted order and which are just data. A simple reader picks out the planted orders every time on the held-out cases, where fixed rules find about one in ten. The result holds on this benchmark's format, in English; the next test moves the planted order anywhere in free text, in five languages.