Module 4 · ~30 min · draft
Deciding what to trust
- V1 Check a model's factual claims against a source and mark which held up.
- V3 Decide when to trust, verify, or reject a model output based on stakes and reversibility.
How to check a claim
A model's output is a draft by a fluent stranger. You wouldn't forward a stranger's numbers to your boss without looking; the same rule applies. Checking is quick if you do it in order:
- Find the claims. Underline every sentence that could be false: a number, a date, a name, a "studies show," a "the law requires." Opinions and summaries of your own material aren't claims; facts about the world are.
- Pick the ones that matter. You won't check everything. Check the ones that would embarrass you or change a decision if wrong.
- Go to a source, not to the model. Asking the model "are you sure?" is not checking; it will agree with itself. Use a primary source (the actual document, the official site, the original paper) or a reference you already trust. Note where you looked.
- Mark each one held, failed, or couldn't verify. Couldn't verify counts as failed for anything that matters.
Deciding what to do with the output
Two questions settle it. What's at stake if it's wrong? An internal brainstorm versus a customer commitment. How reversible is a mistake? A draft you'll revise versus an email already sent. Low stakes and reversible: use it. High stakes or hard to undo: verify first, or don't use it. That's the whole rule, and it is the same rule you already use for advice from people.
The task
Ask a model to write a short factual explainer (about 200 words) on a topic in your own field that a new colleague might ask about. Choose something with real facts in it: a process, a regulation, a tool, a history. Then:
- Underline the factual claims. Pick three that would matter if wrong.
- Check each against a real source you name. Mark it held, failed, or couldn't verify.
- Classify the whole explainer as safe to use, use after fixes, or reject, and say why in terms of stakes and reversibility for the situation you'd actually use it in.
What to submit
The explainer, your claim-check table (claim, where you checked, held or failed), and your trust decision with the reasoning.
How it's graded
- Sources are real and cited.
- At least one claim was actually checked, not "seems right."
- The trust decision references stakes or reversibility.