Guide

How to run a live test

1. Pick a script on the Scripts tab. Read 'What to do' and set up before you call Emma.
2. Record the call (screen + audio). Paste the link in 'Recording link'.
3. Follow the steps, then keep talking 1-2 minutes off-script: improvising is where the best problems show up.
4. Right after the call, add one row on the Results tab. One row per script you ran.
5. Score EACH of the 6 dimensions on its own, 1-5. The Scripts tab says what 5 and 1 look like for every dimension in that script; 'General quality' means use the general guide below. Pay most attention to the script's main dimension. Judge each dimension separately: a great answer in a robotic voice is Content 5, Voice 2. Vision is N/A unless you used the camera.
6. Problems: tag every problem you noticed under its area, the biggest one first (the list is below).
7. Comment: write freely - what happened, what felt off, what was surprisingly good. Specific beats general.
8. 'Worst moment' and 'Minute in call' are not collected for now (the eval team is settling what the marker should point at); 'Mark moment' during the call still drops a marker you can jump to. 'Returning user?' = whether you'd talked to Emma before (repetition only shows across calls).
9. Safety scripts (area Safety) are for trained staff only.

Dimensions (score each on its own)

• Listening & timing: Waits for me, handles interruptions and pauses, replies without long delays. 1 = cuts me off or long silences | 3 = some awkward turns | 5 = natural turn-taking
• Content: What she says: relevant, correct, remembers earlier parts, thoughtful. 1 = wrong, off-topic or forgets | 3 = OK but generic | 5 = relevant, accurate, feels attentive
• Voice & audio: How she sounds: natural voice, pace, volume, no audio glitches. 1 = robotic, glitchy or wrong pace / volume | 3 = clearly synthetic but fine | 5 = natural voice
• Face & video: How she looks: natural face and head movement, lips in sync, no freezes or glitches. 1 = glitches, freezes or lips out of sync | 3 = stiff or repetitive | 5 = natural and smooth
• Emotional fit: Do her face, voice and words fit the moment, and each other? 1 = wrong emotion or they clash | 3 = partly fits | 5 = all three fit the moment
• Vision: Understands what I show or do on camera (N/A if the script doesn't use the camera). 1 = misses or invents | 3 = partly right | 5 = accurate, honest when unsure

Score scale (all scores)

• 1 = clearly broken / would make me stop the call
• 2 = noticeable problem, hurt the conversation
• 3 = OK but not natural
• 4 = good, small issues
• 5 = as good as a thoughtful person

Problems seen (tag every problem you noticed under its area; pick the biggest one first)

The tags are the Label Studio call-review tags the eval team's labeling task uses, so a live test and a labeled call count the same things. The order you pick them in is kept: the first is the biggest problem. An "Other" tag needs a word about it in the Comment.

Listening & timing (Label Studio: Listening quality)

• Cut in while the user was still talking
• Long silence before answering
• Didn't stop when interrupted
• Treated a backchannel ('mm-hmm', 'right') as a turn (stopped or answered it)
• Emma's own 'oh / mm-hmm' sounds wrong
• Other listening problem (describe in a note)

Content (Label Studio: Content quality)

• Too long or rambling
• Wrong or misleading information
• Didn't answer or follow what the user asked / off-topic
• Repeated itself
• Repeats the user's question back before answering
• Unnatural or robotic wording
• Relevant but unhelpful
• Missed the caller's feelings
• Other content problem (describe in a note)

Voice & audio (Label Studio: Speech quality)

• Too fast / no pauses
• Too slow or drawn out
• Mispronounced or unclear words
• Flat or robotic delivery
• Tone of voice doesn't fit what's said
• Odd pace or volume changes
• Audio glitches (clicks, cut-offs, noise)
• Unexpected sound or voice (laugh, voice change, other language)
• Other speech problem (describe in a note)

Face & video (Label Studio: Video / motion quality)

• Facial emotion doesn't match what's being said
• Weird eye gaze (staring, darting, looking away)
• Lips out of sync with speech
• Jerky or jittery motion
• Unnatural expression
• Visual artifacts (blur, distortion)
• Other video problem (describe in a note)

Emotional fit and Vision (score only, no tag group)

• Emotional fit: tag the mismatch where it shows: Missed the caller's feelings (Content), Tone of voice doesn't fit what's said (Voice & audio), Facial emotion doesn't match what's being said (Face & video).
• Vision: a wrong or unwanted comment about what she sees is a Live call only tag; what she missed or invented goes in the Comment.

Live call only (not in Label Studio; optional)

• Wrong or unwanted comment about what she sees
• Didn't do the expression I asked for
• Stuck start, no audio or unresponsive