Methodology: case creation, standards alignment, and feedback generation

2. How the methodology works

The method below applies to every case, regardless of learner level. Within Evidence and scoring, the level subsections describe what varies by learner level.

2.1 Building a case

Every case begins with a defined purpose: the communication and clinical skills it should exercise and the competencies it should assess. Content is written to that purpose, with explicit learning objectives and a clinical evidence base drawn from foundational and other applicable literature.

Physicians then review each case for accuracy, content, alignment, and fit to its objectives; reviewers are practicing physicians with strong academic backgrounds and decades of collective experience, including program directors, simulation facilitators, and UME and GME educators. Faculty development cases are also reviewed by someone with faculty development or clinical teaching expertise.

Before release, each case is run repeatedly to confirm three things: that the AI character convincingly plays the role the case requires; that scoring and feedback perform as intended; and that what the case asks can be assessed within the limits of voice-based simulation. Refinements are made in coordination with ClinicalSim’s clinical and technical leadership.

2.2 Competency alignment and communication frameworks

Three terms recur here. The competency framework is the anchor for the competency assessment; where it defines level descriptors, as the ACGME milestones do, those are quoted verbatim from the primary source. The communication frameworks are then applied to characterize how the learner communicated. The two are distinct: the competency score reflects the learner’s developmental level, while the communication frameworks capture the specific skills underlying communication technique.

Competency framework

The governing-body standard a case is assessed against: the ACGME Milestones 2.0 in graduate medical education, the Foundational Competencies in undergraduate medical education, and the ACGME Clinician Educator Milestones in faculty development.

Communication framework

A validated, published model of communication behavior, such as SPIKES or Calgary-Cambridge, applied to characterize how the learner communicated.

Rubric

The scored instrument that turns a framework into rated items, including a program's own internal or externally validated tools.

Each communication framework comes from a cited, published source and is a floor, not a ceiling: one or more may be applied to a case, each scored independently, and programs may add their own internal or externally validated rubrics. Because these frameworks and rubrics operate at different scopes, from whole-encounter structures to task-specific routines to discrete micro-skills, ClinicalSim selects those best suited to each case’s communication task.

2.3 Evidence and scoring

Each encounter is a voice conversation between the learner and an AI role designed for the case, captured as a timestamped transcript. For every scored competency and framework step, the platform draws one or two verbatim excerpts that demonstrate the behavior, or documents its absence. Because each score is traceable to the moment that supports it, the output withstands review rather than serving as an unexplained rating.

Scoring follows the competency framework on which a case is built, and the unit of assessment is the individual competency the case exercises. Each applied communication framework or program rubric is scored independently of the competency.

Where an instrument publishes its own rating scale, we use it. Most communication frameworks do not, and there we apply a ClinicalSim scale to the framework’s own steps and say so in the case, so a score is never read as though the framework’s validation stood behind it.

Because these frameworks are developmental, a given result carries different meaning at different stages of training and is always interpreted accordingly. All scores are presented together, with their verbatim evidence, so the learner or reviewer sees a complete picture. What varies is the competency framework a case is anchored to and how the competency itself is scored, described by learner level below.

2.4 Feedback

Each encounter produces a single feedback report.

Verbatim evidence is incorporated into the grading rubrics, justifying the level a learner reached or the specific step assessed. The report then offers an overall impression (strengths, priority gaps, and top action items) and targeted recommendations. Depending on the case, it indicates where a learner sits developmentally and provides reviewers with transcript-grounded evidence for decisions about progression, remediation, readiness for practice, readiness to perform a particular task, or familiarity with a given subject area.

2.5 What a score claims, and what it does not

The frameworks ClinicalSim applies were built for trained human raters observing real encounters, and that is how their published reliability was established. Scoring them with AI in a simulated encounter is an extension beyond that context, so a framework’s reliability does not carry over to a ClinicalSim score. Each score is a formative signal backed by verbatim transcript evidence.

We are testing that rather than asserting it. In our current pilot, program directors review ClinicalSim output alongside their own assessment of the same encounters, which is how we find out where the platform holds up against the standard and where it complements faculty judgment rather than substituting for it.