Questions to ask an AI patient simulation vendor before you buy
Twelve questions before an AI patient simulation purchase: the framework behind each score, transcript evidence, limits on use, and who wrote the cases.
ClinicalSim Team
ClinicalSim
A hospital evaluating an AI patient simulation vendor should ask what each score is built on, whether a reviewer can read the words behind it, what the score must not be used for, what has been studied, how participant recordings are handled, and who wrote the cases. A demo answers few of these questions, because it shows one conversation chosen by the seller, and a purchase puts your own clinicians' conversations into the vendor's scoring system.
This guide is for the program director, simulation director, or nurse educator who runs the evaluation, and it applies to any vendor. We sell one of these products, so each question ends with ClinicalSim's answer and a link to the page where you can check it. Hold us to the same list.
What should a hospital ask an AI patient simulation vendor?
The twelve questions fall into four groups, and each group has a natural owner on an evaluation committee:
- Faculty who will read reports ask about scoring: the framework behind each score, the transcript evidence, and any comparison with human raters.
- The people who will practice ask about the practice itself: speaking or typing, a complete encounter to read, and who wrote the cases.
- Whoever sets evaluation policy asks how scores are used: the limits on use, who sees a participant's results, the evidence, and endorsement claims.
- IT, privacy, and procurement ask what data the product holds and what it needs to run.
Five more questions cover exports and what happens if the vendor closes. We published those in an earlier post on vendor shutdowns, and the last section here summarizes them.
Questions about scoring
Which published framework is each score built on?
Faculty can't weigh a score that has no named source. Ask the vendor to name the framework or rubric behind every score, to cite where it was published, and to say whether your program can supply its own standard. Ask too whether separate instruments are reported separately or blended into one number.
ClinicalSim's answer: every score traces to a competency standard the program approves or to a published communication framework, and the two are scored separately. Our frameworks page lists the frameworks attached to published simulations and names the owner of each, including SPIKES, NURSE, AHRQ CANDOR, and the Calgary-Cambridge Guide. Our methodology explains how a case is mapped to a standard.
Can a reviewer read the words behind each score?
A faculty member who disagrees with a rating needs to see what the participant said. Ask whether each scored item quotes the transcript, whether the report says when a behavior was absent or couldn't be assessed, and whether the rationale is written out in the report.
ClinicalSim's answer: each scored item carries one or two verbatim excerpts from the transcript, or a note that the behavior was absent. In our published vaccine hesitancy counseling example, the SEGUE rubric gives "Understand the patient's perspective" 5 out of 5 and quotes the participant: "I hear the worry, um, in your voice, and I also hear how important it is to protect Ava." The same report gives "End the encounter" 2 out of 5, quotes the closing line, and explains that the participant didn't ask whether the parent had any remaining questions.
Has the automated score been compared with expert human raters?
The communication frameworks ClinicalSim applies were built for trained human raters observing real encounters, and a vendor that scores against a published framework should say whether the same is true of its own. An AI score in a simulated encounter is a different use, so ask whether the vendor has compared its scores with blinded expert raters, on which rubric, and where the results are published. A score that repeats from one model run to the next doesn't settle this.
ClinicalSim's answer: Consistency between model runs is not the same thing as agreement with expert human raters. Faculty-rater validation on a customer's own rubric is the gate before any high-stakes use, and that work has not been done. We don't claim our scoring is more accurate or more valid than a faculty member's read. In our current pilot, program directors review ClinicalSim output alongside their own assessment of the same encounters, as section 2.5 of the methodology describes.
Questions about the practice itself
Do participants speak, or type?
Spoken and typed practice exercise different skills. Speaking aloud exercises tone, pacing, silence, and responding to emotion in the moment, while typing gives a learner time to compose and suits step by step reasoning. Ask which format the product uses, and ask what that format can't assess.
ClinicalSim's answer: ClinicalSim is audio only. Participants speak with an AI patient, and the conversation is captured as a timestamped transcript. A product that is audio only can't assess eye contact, body language, or physical presence, so those skills stay with live encounters and human review. Our comparison of voice-based and text-based simulation covers where each format fits.
Can we read a complete encounter before the demo?
The person running a demo knows the product and has rehearsed the conversation. Ask instead for a whole encounter, start to finish, with the recording, the full transcript, and the feedback report, and read it before the call. You're checking whether the AI patient responds like a person, and whether the feedback names what the participant did well along with what to practice next.
ClinicalSim's answer: encounters are published on our examples page with audio, the full transcript, and feedback, with no sign-in and no form. Each report shows strengths, areas for focused practice, and the participant's words behind the feedback.
Who writes and reviews the cases?
Ask who wrote each case, what clinical source it was written from, who reviewed it, and how it was tested before release. Ask whether any case draws on real patient records. Ask too whether a published case or rubric can change after your participants have been scored on it, and how you'd know.
ClinicalSim's answer: every patient is synthetic and written from clinical literature, not from a patient record. Practicing physicians, including program directors, simulation facilitators, and medical educators, review each case for accuracy and fit to its objectives, and each case is run repeatedly before release. Once a case is published, its text, rubric, and scoring prompts are versioned and locked. The methodology and our trust and data handling page document this.
Questions about how scores will be used
What must the score not be used for?
Every assessment tool has uses it wasn't built for, and a vendor should be able to name them. Ask for a written statement of what the score is not intended for, and ask whether that statement appears in the agreement as well as on the website.
ClinicalSim's answer: ClinicalSim output is formative. It is evidence that informs program judgment, and it is not intended for employment, credentialing, privileging, licensure, or other high-stakes decisions.
Formative scores must not be used for employment decisions, discipline, compensation, credentialing, privileging, or licensure. Institutions should set access, retention, and reporting rules before staff participate. Our evaluation page lists the other claims we don't make.
Who sees a participant's recording and scores?
Participants should know who can listen before they practice. Ask which roles can open a participant's recording, transcript, and scores, and whether one organization's data is ever visible to another. Ask what the participant gets back, because the people practicing should receive feedback they can use and another chance to practice, while leaders review patterns across a group.
ClinicalSim's answer: there are four roles. Members practice and see only their own work, Project Managers see the results of the cohorts they run, and Admins see every program in their own organization. Nothing crosses organizations, as our roles and permissions article explains. Each participant's report shows strengths and what to practice next.
What evidence exists for the product, and what doesn't?
Ask for studies of the product itself, as distinct from studies of simulation in general or of the frameworks the product borrows. Ask what was measured, in whom, by whom, and whether it has been peer reviewed. Then ask what the vendor does not claim, since a clear list of what hasn't been studied makes the other answers easier to trust.
ClinicalSim's answer: studies of ClinicalSim are underway at partner institutions, and we won't publish an outcome from any of them until that study's owners have confirmed the record in writing. What is public today is the conference presentation list on our research page. ClinicalSim does not predict patient experience scores, readmissions, safety events, claims, or other clinical or business outcomes. Any comparison with institution-held outcome data requires a separate study plan.
Does the vendor claim an accreditor endorses it?
Ask whether any accreditor or framework owner named in the pitch has endorsed the product, and ask to see that in writing. The ACGME's non-endorsement policy for data collection software, last modified on February 10, 2015, shows why: "To the extent any vendor states that it is endorsed by the ACGME or that its product will allow compliance with or meets ACGME standards, such statement is false." That policy covers one accreditor and one category of software, and it leaves each institution or program to decide whether a product meets its needs.
ClinicalSim's answer: ClinicalSim is not endorsed by, affiliated with, or approved by the ACGME or AHRQ. Scores map to published frameworks; they do not establish compliance with any standard, which a surveyor determines against the institution's own policy.
Questions for IT, privacy, and procurement
What data does the product hold, and what security documentation can we review?
A simulation product can hold no patient data and still hold sensitive records, including voice recordings of your clinicians, transcripts, scores, and account data. Ask what is collected, whether recording requires consent, whether a participant can have a recording erased, and how long records are kept. Then ask for the security documentation your institution's review requires, and ask the vendor to say plainly which documents exist and which don't.
ClinicalSim's answer: no patient record enters through a case, because every patient is synthetic. The platform does handle learner recordings, transcripts, account data, and institutional data. Voice collection is consent-gated, and learners can request erasure.
Our trust and data handling page describes this, and learner data inside the product is governed by the institutional agreement. If your security review needs something that page doesn't cover, ask us and we'll tell you what exists and what doesn't.
What does it need to run, and who can't it serve?
Ask what participants have to install, which devices and languages are supported, and what your network has to allow. Ask about accessibility directly: how a participant who is deaf or hard of hearing, or who uses assistive technology, takes part, and what accommodations the vendor documents.
ClinicalSim's answer: ClinicalSim runs in a browser on a phone, tablet, or desktop with nothing to install, and a typical encounter takes 3 to 10 minutes. It supports English only today. IT may need to allow the voice service through a hospital firewall, VPN, or web filter, and the briefing page runs a connection test that names what to ask IT for. The product is audio only, so raise accessibility needs with us before a rollout.
What to ask about records and continuity
A product that holds your assessment records is a dependency, and the time to ask how you'd leave is before you sign. Our post on what a program still owes when a simulation vendor shuts down sets out five questions for that conversation. They ask what you receive on the last day and in what format, what happens to learner data if the company is acquired, whether a wind down runs on your date or the vendor's, whether you can reconstruct a participant's assessment record without logging in to the vendor, and whether any of it is in the contract.
That post uses Kognito, which stopped selling in August 2023, to show why a published trial isn't a continuity guarantee. We're a young company, so a program should put those five questions to ClinicalSim too. Our answers on export, retention, and handover belong in the agreement with each institution.
How to use the list
Send the questions before the demo and ask for answers in writing. Give the scoring questions to the faculty who will read reports, the data questions to privacy and IT, and the use questions to whoever owns evaluation policy for residents or staff.
Then have a few clinicians run an encounter and read their own reports, because they're the people who will use the product. Compare vendors on what they put in writing, including what each one says it can't do.
References
- ACGME Non-Endorsement Policy on Data Collection Software Policy, last modified February 10, 2015. Accreditation Council for Graduate Medical Education. Accessed November 15, 2026.