← All lab updates

Two open models answer questions nobody can answer, and the answer depends on how you ask

Three researchers recently argued that AI should be judged on whether it knows the limits of what it knows, not on whether it sounds human. We turned that into a test: 16 pairs of questions in five languages, each pair one answerable question and a twin built on a false premise, an invented name or an unknowable fact. The larger open model invents a specific answer about one time in three; the other about one time in seven. Both answer every real question correctly. And changing only the wording moves the result more than asking the same question twice.