Methodology

The task

Players hear one 5–15 second clip and vote Human or AI, then get instant feedback. Clips come from a fixed pre-generated deck; nothing is synthesized at request time. The deck covers 24 scripts × 2 languages (English, Polish) across 8 emotion categories: neutral narration, angry rant, grief, sarcasm, joke delivery with timing, whisper, excited sports commentary, and hesitant filler-heavy speech. Scripts were written with natural disfluencies, breaths, and mid-sentence restarts baked into the text — both humans and the AI read the same words.

Scoring definitions (exact)

• fool_rate_ai — the headline stat: share of counted votes on AI clips where the guess was “human”. fooled / all counted votes on AI clips.
• false_alarm_rate — the counter-stat: share of counted votes on human clips where the guess was “AI”. Humans mistaken for machines are just as much part of the result.

Both are reported with their n. Rates are rounded to one decimal by standard rounding — never truncated or nudged in either direction.

A vote is “counted” when all of these hold:

• it is the first vote by that session on that clip (enforced by a unique database constraint; replays are rejected at write time);
• ms_to_vote ≥ 700ms — faster votes are stored but excluded from every stat on this site, on the assumption they are reflexive taps rather than listening. The threshold is fixed, chosen before launch, and applied identically to right and wrong answers;
• the clip is an active deck clip (the private holdout deck — see below — has no votes by construction).

As a robustness check, /stats also reports the fool rate counting at most one vote per (IP, clip), using salted IP hashes. Raw IP addresses are never stored.

Human recordings

• All recordings are fresh, made for this project by the author and consenting friends/family. No LibriVox or public-domain audio — audiobook register is exactly what TTS mimics best and older public audio may be in training data.
• Mic diversity is deliberate (podcast mics and phone mics), so “studio quality” can't serve as an AI tell in either direction.
• Speakers were directed to read naturally, keep stumbles, and not act. Consent notes are stored for every speaker; Clone Roulette participants additionally consented to an instant voice clone, revocable at any time.

AI generation

• Model: eleven_v3 (ElevenLabs), generated July 2026 via the official TypeScript SDK. Scripts used v3 audio tags (e.g. [whispers], [sighs], [excited]) per the current prompting guide.
• Condition A (designed): six Voice Library / Voice Design voices roughly matched to the human speakers' demographics, each rendering the same script assignments as a human speaker row.
• Condition B (clone): instant voice clones (IVC) of consenting speakers, built from ~2 minutes of their own recording, rendering the same scripts those speakers recorded — the apples-to-apples deck behind Clone Roulette.
• Takes policy: 2 takes were generated per (voice, script); the better take was picked by ear before any votes existed. Take picks are frozen in the repo. Note v3's own docs recommend prompts >250 characters; our scripts are shorter (15–40 words), which may reduce v3 consistency — takes mitigate but don't remove this. This favors neither side systematically but is disclosed.
• No competitor models, by design: this measures human vs ElevenLabs only, and either outcome is an interesting result.

Normalization (identical for both kinds)

Every clip — human and AI — passes the exact same chain, so loudness or format artifacts can't leak the label:

ffmpeg -i in -af "silenceremove(head/tail → uniform 250ms),
               loudnorm=I=-16:TP=-1.5:LRA=11"
       -ar 44100 -ac 1 -c:a libmp3lame -b:a 128k
       -map_metadata -1  out.mp3

• No denoising of human clips. No room tone added to AI clips. What the mic heard is what you hear, loudness-normalized.
• Filenames are opaque UUIDs; all metadata/ID3 tags stripped.
• The animated waveform in the game is drawn from peak buckets precomputed from the normalized file — it contains no information the audio itself doesn't.

Integrity

• All vote counting is server-side. Before a vote is cast, the client receives only: clip id (random UUID), audio URL (random UUID filename), duration, waveform peaks. Clip provenance appears in no payload, header, or URL until after the vote — enforced by an automated test in the repo, not by promise.
• Rate limits: 3 votes/second per session, 500 votes/day per IP.
• Decks are sampled per session (stratified 5 human / 5 AI in Quick Run) and exclude clips the session has already voted on; clip ids never appear in shareable URLs.
• A private 10-clip holdout deck (same pipeline, never uploaded) is reserved to detect answer-sharing distortion later by rotation.

Known biases and limitations

• Players are self-selected and primed: they know AI clips exist, which lowers the fool rate versus an unsuspecting listener. This benchmark measures detection when trying — the hardest condition for the AI.
• The audience skews English-speaking and tech-adjacent (launch channels: HN, X). Per-language tables are provided; Polish results come mostly from Polish-speaking players.
• Small per-cell n early on — per-emotion numbers stabilize as votes accumulate; every table shows its n.
• The human deck is a handful of speakers, not a population sample. Speaker-level effects exist; per-clip data is in the CSV.
• Fool rate can drift over time (players learn, clips circulate). The stat is always reported with its running n; the holdout deck exists to quantify contamination.

Data access

Per-clip aggregates (kind, condition, language, emotion, votes, fooled count, average decision time): /api/stats/clips.csv. Aggregate endpoint: /api/stats.

One deliberate limitation: the CSV identifies clips by a stable pseudonymous clip_ref (salted hash), not by the live clip UUID. Publishing the UUID→kind mapping would hand bots a downloadable answer key for gaming the benchmark. Everything else about each clip is disclosed.