← All lab updates

We can see a model start to cave before it does

A model has an internal signal that lights up when it is about to cave to the user. Reading it token by token as the model writes, the first quarter of the reply already predicts whether it will cave by the end, a touch better than reading the whole reply, and the two cases pull apart from the first word. Like a knock sensor in an engine: we see the drift forming before it reaches the output.