← All lab updates

Does a model understand you, and does its reply show it?

A new test asks each model two things about the same message: what does the person mean, and does your normal reply act on it? Six models, five languages, three rounds. Understanding grows with size and then saturates; using it does not. On messages written natively in each language, large models ignore what they understood on 18 to 45 messages in 100, most often by answering a rhetorical question as a real one.