Catch the AI Lying: testing an AI on public-safety and advocacy facts

Exact match and the judge gave the same verdict on every item. That is a limit of this question set, not a finding about judges: every answer was either word for word the gold answer or wrong, so no item tested whether a judge can recognise a correct answer worded differently, or reject a fluent wrong one. On this set the judge added nothing that a word-for-word check did not. The judge gave the same verdict on both runs for all 10 items (kappa 1.0, almost perfect). That describes these items, on one day, at these two temperatures. It is not a guarantee: hosted models change with load and version. The judge's 0-10 score took only the values 0 and 10, so in practice it gave a pass or a fail, not partial credit.

Size. 10 items: one flipped verdict moves any percentage by 10 points, so read every number with its denominator. One model, one judge, one day. A single small free model answered and a single free model graded. Free hosted models change without notice, so a re-run can differ. The gold answers were written from public sources and have not been independently reviewed by an advocacy, legal or law-enforcement expert. Hotline numbers, statutes and travel rules change; verify any fact before relying on it. No hard cases yet. To make a set like this fit to inform any real decision it would need items built to break exact match (the right answer in different words or units), items built to fool a judge (a fluent wrong answer, or an answer that argues the gold answer is wrong), a way to tell an honest "I don't know" from a confident wrong answer, crisis and escalation questions that people review rather than a judge, and a sample graded by two or three advocates so the judge can be compared with people and not only with itself. No survivor, client or case data was used: every question is a general public fact.

Ten gold question/answer pairs on public-safety and advocacy facts (crisis hotline numbers, a Nevada statute, a federal law, a travel rule for personal-safety products, sexual-assault care terminology, and questions about drink spiking and consent) were written before any model output was looked at and saved as data/gold_qa.csv. One model (liquid/lfm-2.5-2.6b:free) answered all ten at temperature 0.0 through OpenRouter, with a fixed system prompt that is identical for every item, because the prompt is part of the instrument. Three graders scored the same answers. Exact match normalizes case, Unicode look-alikes, Markdown emphasis, answer labels, surrounding quotes and trailing punctuation, then requires the whole string to equal the gold answer or one of its accepted variants (for example a hotline number written with different punctuation). Contains-gold is a deliberately lenient diagnostic that produces false passes. The AI judge (nvidia/nemotron-3-ultra-550b-a55b:free) reads the question, the gold answer and the model's answer, and returns VERDICT / SCORE / REASON in three lines. It is told that the gold answer is authoritative, that the same quantity in a different unit or the same number in a different format is a pass, and that anything that could put a person at risk if acted on, such as a wrong phone number, is a fail. The parser clamps the score to 0-10 and returns UNPARSEABLE rather than guessing when the format was not respected. The judge was run twice, at temperature 0.7 and again at 1.1, and the two runs are compared with raw agreement and Cohen's kappa (chance-corrected, read on the Landis and Koch scale). Kappa bands are a convention, not a law, and at n=10 a single flipped item moves accuracy by 10 points, so treat every number here as a pilot measurement with its denominator attached. Every model called is on OpenRouter's free tier at $0 per token, verified against the live catalog at run time. Answers and verdicts are cached to disk, so re-running a cell never spends quota twice, and the harness counts requests against a daily budget and refuses to exceed it. This is a ten-item pilot, not a validated safety tool. It does not use real survivor, client or case data, and nothing here is legal, medical or safety advice.