How a question is made
No scored question in ResidentReady is released from model memory. Every one is written against sources that were retrieved for that topic, then attacked by a separate pass that answers it blind before it is shown the key. Items that fail any gate are discarded and the resident never sees them. If nothing survives, the assignment fails and says so.
The pipeline
- 1Topic normalisationThe entered text is mapped to canonical clinical wording and to a knowledge domain in that specialty. Input that looks patient-identifiable is refused here.
- 2Source retrievalPubMed is searched through NCBI E-utilities, starting with guidelines, reviews and consensus statements from the last fifteen years and relaxing the filters only until something is found. Every result must clear the specialty source allowlist.
- 3Injection screeningRetrieved text is scanned for model-directed instructions. A passage that argues with the pipeline is dropped rather than cleaned, and everything that survives is fenced as data.
- 4Candidate generationMore items are written than are needed, so the gates have something to discard. The writer is told to return fewer items when the evidence is thin.
- 5Deterministic gatesTwelve structural checks that no model can talk its way past. Listed below.
- 6Independent verificationA separate pass answers the item from the evidence before seeing the key, lists every other defensible option, and decides fourteen named gates.
- 7Release or discardDisagreement with the key, more than one defensible option, any failed gate, or injected text in the evidence all mean discard.
Deterministic gates (12)
four_distinct_labels
Exactly four options, labelled A–D, each label used once.
choices_distinct
No two options say the same thing.
correct_label_exists
The key names an option that exists.
explanation_matches_answer
The explanation addresses the three non-key options and not the key.
no_answer_leakage_in_stem
The key does not appear verbatim in the stem.
no_absolute_language_tell
No distractor is given away by "always" or "never".
no_length_tell
The key is not conspicuously longer than every distractor.
citations_resolve
Every cited source id is one that was actually retrieved.
no_invented_urls
No URL appears that was not in the retrieved set.
no_unsupported_percentage
Every percentage in the explanation appears in the evidence.
no_patient_identifiers
No MRN, date of birth, phone number, email or named patient.
not_verbatim_source_copy
The stem shares no long verbatim run with a source.
Verifier gates (15)
stem_medically_defensible
The clinical scenario holds up.
answer_source_supported
The key is supported by the retrieved evidence, not by recall.
distractors_inferior
Each distractor is false or clearly worse.
exactly_one_best_answer
Only one option can be defended as best.
no_unsupported_numeric_claim
No figure is asserted that the evidence does not carry.
no_hallucinated_citation
No source is invented or misattributed.
no_answer_leakage
Nothing in the wording gives the key away.
not_copied_proprietary_item
The item is not a reproduction of a real examination question.
pgy_level_reasonable
The difficulty matches the stated training level.
tests_meaningful_concept
specialty_correct
The item belongs to the specialty it is filed under.
case_relevance_real
The item is genuinely about the assigned topic.
explanation_matches_answer
The explanation and the key agree.
no_patient_identifiers
Nothing identifiable about any person.
claims_current_enough
The claims are current enough to teach from.
Source policy
Sources are governed by an allowlist rather than a blocklist, because the failure mode we care about — an answer-dump site that looks authoritative — is unbounded, and only an allowlist is closed. Every specialty inherits this universal set and adds its own society and journal domains:
- pubmed.ncbi.nlm.nih.gov
- www.ncbi.nlm.nih.gov
- pmc.ncbi.nlm.nih.gov
- www.nih.gov
- www.cdc.gov
- www.fda.gov
- medlineplus.gov
- www.who.int
- www.ahrq.gov
- www.uspreventiveservicestaskforce.org
Answer-dump and flashcard sites are refused outright even if some other rule would admit them. Every released item keeps the URL, publisher, identifier and retrieval date of the sources that justified it, and those are shown on the explanation.
What this does not prove
A verified item is an item that survived these checks. That is a meaningful bar and it is not the same as clinical validation, editorial board review, or psychometric calibration. ResidentReady records how every released item behaves — how many residents answered it, how many got it right, how long they took, and whether anyone flagged it — so calibration can rest on measurement when there is enough of it. Until then, nothing here is presented as a validated instrument.