Turing test protocols

Description

Blinded judges exchange typed text with hidden humans and machines and decide which is which; the proportion of judges fooled measures human-likeness under a stated protocol.

Turing replaced 'can machines think?' with whether a computer could do well in the imitation game, in which a judge exchanging only typed text tries to tell machine from human. He left judges, duration and topics open but predicted that average interrogators would be right no more than 70 per cent of the time after five minutes. Protocols differ: the 1950 three-party parallel game versus the 1952 serial jury version, in which judges question contestants one at a time without knowing the ratio and over-call machines; the 2014 Royal Society event with five-minute paired conversations; and the Kapor-Kurzweil wager with three judges, two-hour interviews and a rank-order criterion. A small offline variant mixes machine and human texts for judges to identify. Short sessions, lay judges, evasive personas and anthropomorphism inflate pass rates, and the test measures indistinguishability, not usefulness or understanding.

Attributes

Loss type
Latent
Stage
Evaluation
Effort
Days

Steps

  1. Define what counts as a human and a machine contestant and state the purpose (for example disclosure decisions).
  2. Have a committee select judges and human foils, varying expertise, age and language background as required.
  3. Run blinded typed conversations for a fixed duration, choosing parallel (one human, one machine) or serial format and stating it.
  4. Collect a human or machine verdict per candidate and, optionally, a rank of all candidates from least to most human.
  5. Apply a pre-stated pass criterion, for example more than 30 per cent of judges fooled after five minutes, or the Kapor-Kurzweil rule.
  6. Report judge selection, session length, sample selection (best versus typical outputs) and identification accuracy against a human-only baseline.

Sources

  • AI (Mitchell, 2019)
  • Gödel, Escher, Bach (Hofstadter, 1999)
  • Mind Design II (Haugeland, 1997)
  • Handbook of the Philosophy of Science (Thagard, 2007)

Also known as

  • Turing test
  • Imitation game
  • Human determination test
  • Rank order test
  • A Little Turing Test
  • Blind human-versus-machine text discrimination
  • Loebner contest format

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning