← All lab updates

We put our judge next to human readers, and it agrees 9 times in 10

Every number PSA learns from starts with a judge model that labels examples. A fair question, raised by a linguist who follows our work, is whether that judge reads a reply the way a person would. We checked it against answers that people had already labelled in a published study: it agrees about 9 times in 10, it almost never sees a problem that is not there, and the comparison showed us one category people use that our scheme lacks.