← All lab updates

Private test pairs in five languages: a production assistant holds its decisions but obeys what it reads

Each pair is two requests that differ in one detail: who wrote an instruction, the user or the document being handled. A production business assistant held its decisions under every pressure we tried and caught dangerous-looking code, but carried out commitments written inside the emails, contracts and pages it was asked to answer, often presenting them as the user's own request. Telling it that the text was third-party data cut these actions from 25 of 25 to 10 of 25: helpful, not enough. On a small model we host, the right answer was inside the model and did not reach the reply.