Oregon State University
2025-07-16
Billy Wisse, lead Jeopardy clue writer:
a good question has to get one of three reactions: “I knew that,” or “darn I should have known that,” or “I didn’t know that, but now I’m glad I do.”1
N
gives ±2% margin of error for 550k questionso3
gives the best resultModel | Precision | Recall | F1 | N |
---|---|---|---|---|
o4-mini | 0.925 | 0.877 | 0.9 | 1250 |
gpt-4.1 | 0.85 | 0.944 | 0.895 | 1250 |
o3 | 0.842 | 0.952 | 0.894 | 1250 |
gpt-4.1-mini | 0.769 | 0.937 | 0.844 | 1250 |
gpt-4.1-nano | 0.593 | 0.821 | 0.689 | 1250 |
Model | Precision | Recall | F1 | N |
---|---|---|---|---|
o3 | 0.848 | 0.932 | 0.888 | 1250 |
gpt-4.1 | 0.803 | 0.963 | 0.876 | 1250 |
o4-mini | 0.919 | 0.827 | 0.871 | 1250 |
gpt-4.1-mini | 0.707 | 0.948 | 0.81 | 1250 |
gpt-4.1-nano | 0.737 | 0.791 | 0.763 | 1250 |
Only for questions that reference literature, use an LLM to output Pydantic-validated JSON.
Percentage of Jeopardy questions about literature over time.
writer | count |
---|---|
William Shakespeare (1564-1616) | 5285 |
Charles Dickens (1812-1870) | 1188 |
Mark Twain (1835-1910) | 879 |
Edgar Allan Poe (1809-1849) | 711 |
Homer | 665 |
Lewis Carroll (1832-1898) | 633 |
Ernest Hemingway (1899-1961) | 612 |
John Steinbeck (1902-1968) | 499 |
Stephen King (1947- ) | 498 |
Herman Melville (1819-1891) | 478 |
Arthur Conan Doyle (1859-1930) | 457 |
Tennessee Williams (1911-1983) | 452 |
George Bernard Shaw (1856-1950) | 437 |
L. Frank Baum (1856-1919) | 424 |
Henry Wadsworth Longfellow (1807-1882) | 424 |
Rudyard Kipling (1865-1936) | 414 |
J.R.R. Tolkien (1892-1973) | 414 |
Robert Louis Stevenson (1850-1894) | 401 |
Nathaniel Hawthorne (1804-1864) | 400 |
Victor Hugo (1802-1885) | 396 |
Jane Austen (1775-1817) | 395 |
F. Scott Fitzgerald (1896-1940) | 385 |
J.K. Rowling (1965- ) | 380 |
Dr. Seuss (1904-1991) | 377 |
Oscar Wilde (1854-1900) | 374 |
Margaret Mitchell (1900-1949) | 371 |
George Orwell (1903-1950) | 347 |
Agatha Christie (1890-1976) | 345 |
Brothers Grimm (Jacob Grimm: 1785-1863, Wilhelm Grimm: 1786-1859) | 329 |
James Joyce (1882-1941) | 323 |
Alexandre Dumas (1802-1870) | 322 |
Miguel de Cervantes (1547-1616) | 321 |
Leo Tolstoy (1828-1910) | 315 |
T.S. Eliot (1888-1965) | 305 |
William Faulkner (1897-1962) | 303 |
Alfred, Lord Tennyson (1809-1892) | 302 |
Washington Irving (1783-1859) | 294 |
Hans Christian Andersen (1805-1875) | 286 |
Robert Frost (1874-1963) | 283 |
Jules Verne (1828-1905) | 281 |
Louisa May Alcott (1832-1888) | 271 |
Arthur Miller (1915-2005) | 267 |
Walt Whitman (1819-1892) | 266 |
Geoffrey Chaucer (c. 1343-1400) | 259 |
James Fenimore Cooper (1789-1851) | 257 |
John Milton (1608-1674) | 256 |
Dante Alighieri (1265-1321) | 256 |
Jonathan Swift (1667-1745) | 246 |
Eugene O’Neill (1888-1953) | 243 |
J.M. Barrie (1860-1937) | 243 |
!=
clue difficultyRound | Row | DD | Med. Value | p(correct) | Questions |
---|---|---|---|---|---|
DJ | 5 | True | $2,000 | 58.5% | 4,311 |
J | 5 | True | $1,000 | 62.5% | 2,919 |
FJ | FJ | False | N/A | 63.0% | 9,223 |
DJ | 4 | True | $2,000 | 63.7% | 6,874 |
DJ | 3 | True | $2,000 | 66.2% | 5,071 |
J | 4 | True | $1,000 | 66.3% | 3,241 |
J | 1 | True | $550 | 66.7% | 6 |
J | 3 | True | $1,000 | 70.1% | 2,247 |
DJ | 2 | True | $2,000 | 71.8% | 1,842 |
J | 2 | True | $1,000 | 73.7% | 700 |
DJ | 1 | True | $1,400 | 79.4% | 34 |
DJ | 5 | False | $2,000 | 82.9% | 37,620 |
J | 5 | False | $1,000 | 86.6% | 42,792 |
DJ | 4 | False | $1,600 | 86.9% | 40,019 |
DJ | 3 | False | $1,200 | 89.6% | 44,382 |
J | 4 | False | $800 | 89.9% | 46,188 |
J | 3 | False | $600 | 91.6% | 49,069 |
DJ | 2 | False | $800 | 92.0% | 49,273 |
J | 2 | False | $400 | 93.2% | 51,865 |
DJ | 1 | False | $400 | 94.7% | 52,443 |
J | 1 | False | $200 | 95.5% | 53,580 |
Rank | Author | Avg Difficulty | Z-Score | Clue Count |
---|---|---|---|---|
1 | Robert Graves | 41.82 | 2.281 | 58 |
2 | Nikolai Gogol | 41.66 | 2.252 | 43 |
3 | Thomas Pynchon | 40.91 | 2.114 | 48 |
4 | Aristophanes | 40.17 | 1.978 | 79 |
5 | Petrarch | 39.95 | 1.938 | 46 |
6 | August Strindberg | 39.87 | 1.923 | 53 |
7 | H.P. Lovecraft | 39.64 | 1.881 | 41 |
8 | Robert Penn Warren | 39.6 | 1.873 | 68 |
9 | Lillian Hellman | 39.46 | 1.848 | 82 |
10 | Edgar Lee Masters | 39.23 | 1.805 | 54 |
11 | Daphne du Maurier | 38.96 | 1.756 | 99 |
12 | Bret Harte | 38.92 | 1.748 | 44 |
13 | Umberto Eco | 38.73 | 1.713 | 53 |
14 | Wilkie Collins | 38.55 | 1.68 | 53 |
15 | Alexander Hamilton | 38.35 | 1.644 | 44 |
16 | Stephen Vincent Benét | 37.98 | 1.576 | 57 |
17 | Saul Bellow | 37.8 | 1.543 | 76 |
18 | George Sand | 37.57 | 1.5 | 62 |
19 | Albert Camus | 37.51 | 1.489 | 112 |
20 | James M. Cain | 37.47 | 1.482 | 44 |
21 | John Donne | 37.46 | 1.48 | 115 |
22 | Alexander Pushkin | 37.41 | 1.471 | 82 |
23 | Aeschylus | 37.4 | 1.469 | 94 |
24 | William Gibson | 37.28 | 1.447 | 70 |
25 | William Butler Yeats | 37.21 | 1.434 | 92 |
This only counts authors who have appeared in 40 or more clues, i.e., more than once per season on average.
Rank | Author | Avg Difficulty | Z-Score | Clue Count |
---|---|---|---|---|
357 | Carolyn Keene | 20.81 | -1.579 | 46 |
358 | L. Frank Baum | 20.75 | -1.59 | 422 |
359 | Ian Fleming | 20.73 | -1.594 | 185 |
360 | Thomas Jefferson | 20.66 | -1.607 | 65 |
361 | Miguel de Cervantes | 20.42 | -1.651 | 321 |
362 | Thomas Harris | 20.35 | -1.664 | 107 |
363 | Brothers Grimm | 19.95 | -1.737 | 328 |
364 | Daniel Defoe | 19.72 | -1.779 | 232 |
365 | Stephenie Meyer | 19.59 | -1.803 | 94 |
366 | Harriet Beecher Stowe | 19.54 | -1.812 | 219 |
367 | Helen Keller | 19.54 | -1.812 | 63 |
368 | J.R.R. Tolkien | 19.45 | -1.829 | 414 |
369 | Tim Rice | 19.36 | -1.845 | 41 |
370 | Dr. Seuss | 19.36 | -1.845 | 376 |
371 | George R.R. Martin | 19.32 | -1.853 | 97 |
372 | Louisa May Alcott | 19.31 | -1.855 | 271 |
373 | Anne Frank | 19.3 | -1.856 | 92 |
374 | P.L. Travers | 19.09 | -1.895 | 52 |
375 | Peter Benchley | 18.86 | -1.937 | 51 |
376 | Margaret Mitchell | 18.6 | -1.985 | 370 |
377 | Erle Stanley Gardner | 18.38 | -2.026 | 47 |
378 | J.K. Rowling | 17.91 | -2.112 | 378 |
379 | Suzanne Collins | 17.79 | -2.134 | 82 |
380 | Charles M. Schulz | 17.36 | -2.213 | 69 |
381 | Aesop | 15.71 | -2.516 | 169 |
In 2011, IBM’s Watson defeated Ken Jennings and Brad Rutter.1
THE OLD TESTAMENT
In Genesis 21, Abraham banishes Hagar & this son of theirs to the desert; call him…
Ishmael
Ways in:
fredner.org/slides