Skip to main content

Setting up the Assessment using AI-based Quantitative Assessment - SHORT

Leverage the power of AI to create accurate scoring rubrics for your simulations.

Written by Szabolcs Arnoczki

When you copy a bundled patient, your copied custom patient will come with the basic structure of the Objectives and Credits, but you might need to define what the exact requirements are.

The layout of the Objectives tab is the following:

To customize a credit, click the pencil icon next to a credit to open the editing window.

Creating AI-Based Assessment Credits

The AI-based Assessment is only available for simulations utilizing the Generative AI upgrade, but it allows you to create accurate quantitative assessments focued on communication.

The AI reads the entire conversation and looks for the information you define in each credit.

The key to accurate scoring is writing clear credit descriptions.


Understanding the Credit Editor

Every credit has two fields.

Credit Name

The Credit Name is a label.

It helps organize your checklist and is shown to learners by default.

Examples:

  • Onset

  • Medical History

  • Tobacco Use

  • Family History

The name itself does not affect scoring.


Credit Description

This tells the AI what information should receive credit.

This is the most important field.

For example:

Broad (more lenient)

Pain onset

Any discussion about when the pain started can receive credit.

Specific (stricter)

Pain started two days ago

Only that information receives credit.

Both are valid.

Choose the level of detail that matches what you want to assess.


In-Session Credit Guidance

This is shown to learners during the simulation.

Use it for hints or reminders.

Example:

Discuss when the pain started.

This field does not affect AI scoring.


How the AI Scores Credits

The AI reviews the entire session transcript.

It doesn't matter:

  • which question was asked

  • when the information was discussed

  • whether the patient volunteered the information

If the required information appears anywhere in the conversation, the credit can be awarded.


Writing Good Credit Descriptions

Broad is okay. Vague is not.

A broad credit simply gives learners more ways to earn the point.

A vague credit is unclear.

✅ Broad:

Pain onset

Any discussion about when the pain began.

✅ Specific:

Pain started two days ago

Only that specific answer earns the point.

❌ Vague:

Pain

Does this mean:

  • onset?

  • severity?

  • location?

  • quality?

  • duration?

The AI—and other people—shouldn't have to guess.


Another example:

Protection

Could mean:

  • pregnancy prevention

  • STI prevention

  • physical safety

  • financial security

Better:

✅ Uses condoms for STI prevention

✅ Uses contraception to prevent pregnancy

✅ Has financial support from family


Don't expect the AI to make assumptions

The AI recognizes information that appears in the transcript.

It does not automatically infer related information.

Example

Patient

"I have chronic bowel problems."

Credit:

Credit

Awarded?

GI disease

✅ Yes

Diverticulitis

❌ No

GI disease, diverticulitis EITHER

✅ Yes

The patient never said they have diverticulitis.


Combining Information

By default, commas mean AND.

Credit:

hospitalization, 4 surgeries

To earn the point, the transcript must include:

  • hospitalization

  • AND four surgeries


OR (EITHER)

If any one item should earn the point, add EITHER.

Credit:

patient's hospitalization, patient's surgeries EITHER

The learner receives credit if they discuss:

  • hospitalization

OR

  • surgeries


Every word matters

Credit:

lives with husband and children in a suburban condo

The transcript must include:

  • husband

  • children

  • suburban

  • condo

If you want any of those pieces of information to be worth the point, write:

lives with husband, lives with children, lives in a suburban condo EITHER


Common Mistakes

Writing questions instead of information

❌ Any family history?

✅ Mother had lung cancer.


Using vague descriptions

❌ Protection

✅ Uses condoms for STI prevention.


Combining unrelated ideas

Instead of: worried her COPD is getting worse, worried about breathing, worried she'll need to move into a home EITHER

Split it into separate credits.

Current concerns: worried about breathing, worried COPD is getting worse

Future concerns: worried she'll need to move into a home

Smaller credits are easier to understand and score more consistently.


Making credits harder than they need to be

This works:

family history of inflammatory bowel disease and colon cancer, mother had ulcerative colitis and colon cancer, grandfather had colon cancer, sibling has ulcerative colitis EITHER

This is much harder to interpret:

family history of both inflammatory bowel disease and colon cancer, mother had ulcerative colitis diagnosed at age 42 and colon cancer diagnosed at age 61, maternal grandfather had colon cancer diagnosed at age 62, one male sibling who has ulcerative colitis diagnosed at age 19 EITHER

Only include details that matter for scoring.


Examples

Credit Description

Type

Why

Pain onset

✅ Broad

Awards credit for any discussion about when the pain started.

Pain started two days ago

✅ Specific

Only awards credit if that exact information is discussed.

Pain

❌ Vague

Doesn't define whether you're scoring onset, severity, location, duration, or quality.

Family history

✅ Broad

Awards credit for any relevant family medical history.

Mother has hypertension

✅ Specific

Only awards credit for that specific family history.

History

❌ Vague

Doesn't specify medical, surgical, family, social, or medication history.

Medication use

✅ Broad

Awards credit for any discussion about medications.

Takes Metformin 500 mg twice daily

✅ Specific

Only awards credit for that medication information.

Protection

❌ Vague

Could mean pregnancy prevention, STI prevention, physical safety, financial security, or something else.


Before You Save

Ask yourself:

  • Am I describing information, not a question?

  • Would another person know exactly when to award this point?

  • Am I intentionally broad, or intentionally specific?

  • Could the AI recognize this information from the transcript without making assumptions?

  • If I used EITHER, does any one item deserve the point?

Did this answer your question?