FAQ for medical educators

How you read a report matters as much as what it says: treat it as evidence, not a verdict.

Every encounter produces one feedback report, and every score in it traces back to a verbatim line from the transcript, not an unexplained rating. The report opens with an overall impression (strengths, priority gaps, top action items) and then gives targeted recommendations. For GME cases, that means a milestone placement on the Dreyfus scale, read against the specialty's own verbatim level descriptors. For UME cases there is no numeric level, since the Foundational Competencies were not published with one. Instead the report records whether each competency was demonstrated, partially demonstrated, or not demonstrated, and scores performance through whichever communication rubric was applied.

Treat the milestone-aligned score as evidence for your Clinical Competency Committee discussion, not a stand-alone verdict. The ACGME milestones are formative and were not designed for high-stakes decisions, so ClinicalSim does not use them that way either. The full standards-alignment logic, including how level descriptors and rubrics are applied, is documented on the methodology page.

Each case is a voice conversation between the learner and an AI role built for that case, recorded as a timestamped transcript. For every scored competency and framework step, the platform pulls one or two verbatim excerpts that show the behavior, or notes that it did not happen. That is what makes each score traceable to a specific moment instead of a black box.

Scoring runs on two tracks that stay separate. The competency framework (ACGME Milestones 2.0 for GME, the Foundational Competencies for UME) reflects where the learner sits developmentally. Communication frameworks like SPIKES or Calgary-Cambridge, plus any rubric your program already uses, score the specific technique independently. Both show up together with their evidence, so you see the full picture. The methodology page walks through this end to end, from case creation to how each score is generated.

The method is the same across levels. What changes is the competency framework a case is anchored to and how it is scored.

For residents and fellows (GME), cases align to the specialty-specific ACGME Milestones 2.0, with the milestone text quoted verbatim from the specialty's own document. The subcompetencies the scenario exercises, most often interpersonal and communication skills and professionalism, are placed on the Dreyfus scale (Level 1 Novice through Level 5 Expert), read against those verbatim descriptors. If a case cannot reach the complexity needed to evaluate through Level 4, we do not assign a milestone score rather than force one.

For medical students (UME), the Foundational Competencies (AAMC, AACOM, and ACGME) are not published with the milestones' five-level scale, so there is no numeric level. The report records each competency as demonstrated, partially demonstrated, or not demonstrated, and scores technique through the applied communication or skill rubric. We map UME cases to the Foundational Competencies and to the AAMC Core EPAs independently, with no fixed crosswalk, since an EPA set aligned to the 2024 Foundational Competencies has not been published yet. Entrustment stays a program decision that this evidence informs.

The full logic for both levels is on the methodology page.

The short answer is transparency plus human judgment, not a black box you are asked to trust.

Every score traces to one or two verbatim excerpts from the transcript, or a note that the behavior did not occur, so any reviewer can check the basis of a rating rather than accept it on faith. The same scoring logic applies to every case regardless of specialty, learner level, or which frameworks are applied, which keeps results consistent across learners. And each case is built from primary sources, reviewed by practicing physicians before release, and run repeatedly beforehand to confirm the scoring and feedback perform as intended.

Just as important is what the output is for: it is formative evidence that informs a faculty or committee decision, and it never replaces human judgment. When a rating looks off, the evidence is right there to check and, if needed, override. The methodology page documents the review process and scoring logic in full.

A ClinicalSim encounter is, in effect, a direct observation of a clinical encounter scored on a structured tool, the same modality medical schools and residencies already rely on, from the mini-CEX to EPA-based workplace assessment. So the direct-observation literature applies to us directly, and it identifies two well-documented constraints on that modality generally, not on any individual faculty member's judgment.

The first is that a single observation is not reliable, and reaching a defensible level takes many. Direct observation is nearly universal in undergraduate medical education (almost every student is observed at least once during core clerkships), but its frequency and quality are not well established (Kogan et al., 2009), and in practice learners are judged on far too few observations to be reliable, mostly because standardized-patient scheduling and faculty time cap how many a rotation can offer. The second is that a given score reflects the rater as well as the learner. In a seven-institution study of more than 53,000 assessments, less than 10% of score variance was attributable to the learner (Ryan et al., 2024). Different faculty watching the same encounter reach different conclusions, the well-described assessor-cognition problem (Gingerich et al., 2014), a property of holistic human judgment applied across many raters, not a reflection on any one evaluator.

And the inconsistency does not stop at the individual score. The committees that weigh these data vary too. This is most visible in UME, where the LCME requires only a fair, timely process for decisions affecting a student's standing (Element 9.9) and prescribes no committee structure or meeting cadence, leaving those to each school, so how a progress or promotions committee is composed, and how often it meets, differ from one institution to the next. That variability is inherent to holistic human judgment. It is one more reason consistent, evidence-linked input matters.

ClinicalSim's structure speaks to the first two. Because there is no standardized-patient schedule or faculty roster capping how often a case can run, a learner can accumulate the observations reliability actually requires, distributed over time, rather than the handful a rotation allows. And because one consistent rubric and scoring process applies to every encounter, a learner's score does not shift depending on which faculty member happened to be observing, a consequence of being a single repeatable scorer, not a claim that the AI judges better than a trained clinician would. Every score stays tied to verbatim evidence, and the output remains formative: it informs the Clinical Competency Committee or student progress committee, which continues to own the decision (how each score is made auditable and kept in human hands is covered in the question above).

To be clear about the boundary: we are not claiming ClinicalSim's scoring is more accurate or more valid than a faculty member's. We do not have the validation data to say that, and we will not claim it until we do. What we can say plainly is that the platform adds observation capacity and holds scoring consistent across encounters, which speaks to the frequency and rater-variance findings above, not to whether the AI's judgment is better than a clinician's. Because a ClinicalSim encounter mirrors a direct-observation tool like the mini-CEX rather than an EPA entrustment scale, the relevant target is modest: on the order of four to five spaced encounters per skill (the range programs commonly use to document competence), rising toward eight for higher statistical reliability, set by the instrument rather than the training level and tuned by in-product performance. Because these figures come from real-patient, human-rated observation, we treat them as a guide for a simulated, single-scorer setting rather than an exact transfer.

Neither ClinicalSim nor the mastery-learning literature sets a fixed number, and the communication-skills evidence argues against picking one. McGaghie and colleagues describe simulation-based mastery learning as outcomes-based, not time- or repetition-based: learners practice and get reassessed until they clear a minimum passing standard, and how many attempts that takes differs learner to learner. No study establishes a novice-to-competent rep count for a purely interpersonal skill, so any run-it-five-times rule would be invented rather than evidenced.

Two findings shape how we think about it instead. First, competence in communication is scenario-specific: performance on one case predicts performance on another only weakly, so breadth across varied scenarios matters more than depth on one. Second, learners cannot reliably judge their own competence, which is why the standard has to be observed and objective rather than a learner's own sense of readiness. Repetition still matters (Issenberg's BEME review found it was the second most cited feature of effective simulation-based learning, present in 39% of studies, right behind feedback), but it is repetition across cases, measured against a rubric, that does the work.

So the expectation is not a rep count. It is this: keep practicing across varied cases until performance holds up against the standard, then come back before it slips. Most programs could not run cases to that standard before now, because standardized-patient scheduling and faculty time capped how many reps a learner could get. That is the ceiling ClinicalSim removes.

For the pilot, we recommend being prescriptive: each learner takes every case assigned to them and keeps retaking it until they meet the minimum standard for their PGY year. The logic is the one your program already applies to procedures. You would not clear a resident on a central line or an intubation until they had shown they could do it to standard, and communication deserves the same treatment. Milestone level descriptors set the bar, the report shows where the learner sits against it, and progression follows the standard, not the number of attempts.

For acquisition, the defensible cadence is spaced, coached practice distributed across days over several weeks, not a single crammed workshop. A workable default is two to three coached, feedback-rich scenario sessions per week for three to six weeks, each covering varied cases rather than the same one repeated. Three things support that shape: spacing helps during learning itself and not just for later recall, the benefit carries into applied training and not just the laboratory, and deliberate practice with feedback (individualized, coach-designed, at the edge of ability) is the active ingredient rather than mere exposure.

The number of weeks is a starting point, not a law. Gate progression to the proficiency standard, not the calendar: a learner is done acquiring when they clear the rubric across varied cases. If in-product data show most learners plateau before three weeks, shorten the block. If most still fail the standard at six weeks, the standard or the scenario difficulty needs recalibration.

Communication skills demonstrably decay within about six months without reinforcement, and coaching is what prevents it. Rather than fix a maintenance interval by rule, gate it to performance and let spaced booster practice carry the load between checks. Cognitive-psychology spacing research gives a useful heuristic for the interval: the optimal gap between practice sessions scales with how long the skill needs to last, so a six-to-twelve-month retention horizon points to booster practice on the order of every one to two months, lengthening as proficiency proves durable.

Which sub-skill decays depends on what the learner's real-world environment reinforces, so maintenance practice should preferentially re-target whatever the clinic fails to keep sharp, often the warmer, emotionally demanding skills for busy clinicians. The schedule is only a fallback: whenever an in-product assessment shows performance below standard, re-trigger a short acquisition-style block. If scores hold at or above standard at three months with no touch, lengthen the interval. If they drop by a standard deviation or more before the next scheduled touch, shorten it.

We would rather be straight about the limits than oversell the numbers. The biggest one is a domain-transfer gap: almost all of the quantitative cadence and decay figures come from procedural skills, fact learning, or motivational interviewing specifically, not from breaking bad news, goals-of-care conversations, or general interpersonal skill. That is exactly why we do not put a fixed maintenance interval on communication skills. Importing a motivational-interviewing schedule to those harder conversations would be extrapolation, not a measured result. We present cadences as proficiency-anchored defaults, tuned by in-product performance data, rather than as settled facts.

Two smaller limits matter too. There is no published rep count for reaching competence on a purely interpersonal skill, which is why the acquisition cadence is a starting hypothesis rather than a proven dose. And the results are genuinely mixed in places: which sub-skill decays runs in opposite directions across studies, and not every retention study shows decay by six months. The honest read is that the direction of the evidence is clear (space it, coach it, reassess it) while the exact numbers deserve the humility of a default you keep testing.

Because there is no standardized patient to schedule and no faculty member needed to staff each encounter, cases go wherever the curriculum needs them, in a few common shapes.

As a rotation or clerkship set, assign the cases that match the high-stakes conversations the rotation is meant to exercise, and have each learner retake them until they meet the standard for their level. As a remediation plan, use a short, spaced block of coached practice across varied cases (the acquisition cadence above), then reassess against the rubric. As maintenance, space lighter touches to keep a proficient skill from slipping. In every shape, the principle is the same: gate progression to the proficiency standard, not a rep count or the calendar.

If you are standing up a pilot, we will help you map cases to the rotation and set the standard for each learner level. The methodology page covers the scoring logic behind it, and you can talk to us about program design.

Yes, and the format is built for it. Each encounter produces a single transcript-grounded report: verbatim evidence for each rating, an overall impression, and targeted recommendations. Run across a remediation block, those reports give a defensible, time-stamped record of what a learner practiced and where their performance landed against the standard over time.

One important caveat on how you frame that record. GME milestone-aligned output is formative. The ACGME is explicit that individual milestones were not designed for high-stakes external decisions, so it should inform your Clinical Competency Committee's judgment rather than stand alone as a verdict. The same holds for UME: entrustment remains a program decision the evidence informs. Used that way, the reports strengthen a remediation file. They do not replace the committee that owns the decision.

Less than standardized-patient practice, and spent differently. There is no SP to recruit, train, or schedule and no faculty member needed to observe and score each encounter. The platform captures the transcript and generates the report. That removes the scheduling-and-staffing ceiling that normally caps how much practice a learner can get.

What faculty time remains shifts to the part that actually moves the needle: reviewing reports and coaching where the evidence points. That is deliberate practice with feedback, the active ingredient in the acquisition and maintenance research above, rather than time spent running and hand-scoring encounters.

Case content is entirely synthetic. Scenarios are authored from the clinical literature, not from real patient records, so there is no protected health information in the cases themselves.

For how learner data (transcripts and reports) is stored and handled, see our privacy policy, or talk to us and we will walk your privacy or IRB reviewers through the specifics for your institution.

A typical case takes between three and ten minutes. It is a single voice encounter captured as one transcript, not a multi-hour simulation block, and length varies with the complexity of the conversation the case is built to exercise. There is no scheduling step and no standardized patient to coordinate, which is exactly what lets a learner run a case again the same day if they need another rep.