Oregon State University
2025-07-16
Billy Wisse, lead Jeopardy clue writer:
a good question has to get one of three reactions: “I knew that,” or “darn I should have known that,” or “I didn’t know that, but now I’m glad I do.”1
N gives ±2% margin of error for 550k questionso3 gives the best result| Model | Precision | Recall | F1 | N |
|---|---|---|---|---|
| o4-mini | 0.925 | 0.877 | 0.9 | 1250 |
| gpt-4.1 | 0.85 | 0.944 | 0.895 | 1250 |
| o3 | 0.842 | 0.952 | 0.894 | 1250 |
| gpt-4.1-mini | 0.769 | 0.937 | 0.844 | 1250 |
| gpt-4.1-nano | 0.593 | 0.821 | 0.689 | 1250 |
| Model | Precision | Recall | F1 | N |
|---|---|---|---|---|
| o3 | 0.848 | 0.932 | 0.888 | 1250 |
| gpt-4.1 | 0.803 | 0.963 | 0.876 | 1250 |
| o4-mini | 0.919 | 0.827 | 0.871 | 1250 |
| gpt-4.1-mini | 0.707 | 0.948 | 0.81 | 1250 |
| gpt-4.1-nano | 0.737 | 0.791 | 0.763 | 1250 |
Only for questions that reference literature, use an LLM to output Pydantic-validated JSON.
Percentage of Jeopardy questions about literature over time.
| writer | count |
|---|---|
| William Shakespeare (1564-1616) | 5285 |
| Charles Dickens (1812-1870) | 1188 |
| Mark Twain (1835-1910) | 879 |
| Edgar Allan Poe (1809-1849) | 711 |
| Homer | 665 |
| Lewis Carroll (1832-1898) | 633 |
| Ernest Hemingway (1899-1961) | 612 |
| John Steinbeck (1902-1968) | 499 |
| Stephen King (1947- ) | 498 |
| Herman Melville (1819-1891) | 478 |
| Arthur Conan Doyle (1859-1930) | 457 |
| Tennessee Williams (1911-1983) | 452 |
| George Bernard Shaw (1856-1950) | 437 |
| L. Frank Baum (1856-1919) | 424 |
| Henry Wadsworth Longfellow (1807-1882) | 424 |
| Rudyard Kipling (1865-1936) | 414 |
| J.R.R. Tolkien (1892-1973) | 414 |
| Robert Louis Stevenson (1850-1894) | 401 |
| Nathaniel Hawthorne (1804-1864) | 400 |
| Victor Hugo (1802-1885) | 396 |
| Jane Austen (1775-1817) | 395 |
| F. Scott Fitzgerald (1896-1940) | 385 |
| J.K. Rowling (1965- ) | 380 |
| Dr. Seuss (1904-1991) | 377 |
| Oscar Wilde (1854-1900) | 374 |
| Margaret Mitchell (1900-1949) | 371 |
| George Orwell (1903-1950) | 347 |
| Agatha Christie (1890-1976) | 345 |
| Brothers Grimm (Jacob Grimm: 1785-1863, Wilhelm Grimm: 1786-1859) | 329 |
| James Joyce (1882-1941) | 323 |
| Alexandre Dumas (1802-1870) | 322 |
| Miguel de Cervantes (1547-1616) | 321 |
| Leo Tolstoy (1828-1910) | 315 |
| T.S. Eliot (1888-1965) | 305 |
| William Faulkner (1897-1962) | 303 |
| Alfred, Lord Tennyson (1809-1892) | 302 |
| Washington Irving (1783-1859) | 294 |
| Hans Christian Andersen (1805-1875) | 286 |
| Robert Frost (1874-1963) | 283 |
| Jules Verne (1828-1905) | 281 |
| Louisa May Alcott (1832-1888) | 271 |
| Arthur Miller (1915-2005) | 267 |
| Walt Whitman (1819-1892) | 266 |
| Geoffrey Chaucer (c. 1343-1400) | 259 |
| James Fenimore Cooper (1789-1851) | 257 |
| John Milton (1608-1674) | 256 |
| Dante Alighieri (1265-1321) | 256 |
| Jonathan Swift (1667-1745) | 246 |
| Eugene O’Neill (1888-1953) | 243 |
| J.M. Barrie (1860-1937) | 243 |
!= clue difficulty| Round | Row | DD | Med. Value | p(correct) | Questions |
|---|---|---|---|---|---|
| DJ | 5 | True | $2,000 | 58.5% | 4,311 |
| J | 5 | True | $1,000 | 62.5% | 2,919 |
| FJ | FJ | False | N/A | 63.0% | 9,223 |
| DJ | 4 | True | $2,000 | 63.7% | 6,874 |
| DJ | 3 | True | $2,000 | 66.2% | 5,071 |
| J | 4 | True | $1,000 | 66.3% | 3,241 |
| J | 1 | True | $550 | 66.7% | 6 |
| J | 3 | True | $1,000 | 70.1% | 2,247 |
| DJ | 2 | True | $2,000 | 71.8% | 1,842 |
| J | 2 | True | $1,000 | 73.7% | 700 |
| DJ | 1 | True | $1,400 | 79.4% | 34 |
| DJ | 5 | False | $2,000 | 82.9% | 37,620 |
| J | 5 | False | $1,000 | 86.6% | 42,792 |
| DJ | 4 | False | $1,600 | 86.9% | 40,019 |
| DJ | 3 | False | $1,200 | 89.6% | 44,382 |
| J | 4 | False | $800 | 89.9% | 46,188 |
| J | 3 | False | $600 | 91.6% | 49,069 |
| DJ | 2 | False | $800 | 92.0% | 49,273 |
| J | 2 | False | $400 | 93.2% | 51,865 |
| DJ | 1 | False | $400 | 94.7% | 52,443 |
| J | 1 | False | $200 | 95.5% | 53,580 |
| Rank | Author | Avg Difficulty | Z-Score | Clue Count |
|---|---|---|---|---|
| 1 | Robert Graves | 41.82 | 2.281 | 58 |
| 2 | Nikolai Gogol | 41.66 | 2.252 | 43 |
| 3 | Thomas Pynchon | 40.91 | 2.114 | 48 |
| 4 | Aristophanes | 40.17 | 1.978 | 79 |
| 5 | Petrarch | 39.95 | 1.938 | 46 |
| 6 | August Strindberg | 39.87 | 1.923 | 53 |
| 7 | H.P. Lovecraft | 39.64 | 1.881 | 41 |
| 8 | Robert Penn Warren | 39.6 | 1.873 | 68 |
| 9 | Lillian Hellman | 39.46 | 1.848 | 82 |
| 10 | Edgar Lee Masters | 39.23 | 1.805 | 54 |
| 11 | Daphne du Maurier | 38.96 | 1.756 | 99 |
| 12 | Bret Harte | 38.92 | 1.748 | 44 |
| 13 | Umberto Eco | 38.73 | 1.713 | 53 |
| 14 | Wilkie Collins | 38.55 | 1.68 | 53 |
| 15 | Alexander Hamilton | 38.35 | 1.644 | 44 |
| 16 | Stephen Vincent Benét | 37.98 | 1.576 | 57 |
| 17 | Saul Bellow | 37.8 | 1.543 | 76 |
| 18 | George Sand | 37.57 | 1.5 | 62 |
| 19 | Albert Camus | 37.51 | 1.489 | 112 |
| 20 | James M. Cain | 37.47 | 1.482 | 44 |
| 21 | John Donne | 37.46 | 1.48 | 115 |
| 22 | Alexander Pushkin | 37.41 | 1.471 | 82 |
| 23 | Aeschylus | 37.4 | 1.469 | 94 |
| 24 | William Gibson | 37.28 | 1.447 | 70 |
| 25 | William Butler Yeats | 37.21 | 1.434 | 92 |
This only counts authors who have appeared in 40 or more clues, i.e., more than once per season on average.
| Rank | Author | Avg Difficulty | Z-Score | Clue Count |
|---|---|---|---|---|
| 357 | Carolyn Keene | 20.81 | -1.579 | 46 |
| 358 | L. Frank Baum | 20.75 | -1.59 | 422 |
| 359 | Ian Fleming | 20.73 | -1.594 | 185 |
| 360 | Thomas Jefferson | 20.66 | -1.607 | 65 |
| 361 | Miguel de Cervantes | 20.42 | -1.651 | 321 |
| 362 | Thomas Harris | 20.35 | -1.664 | 107 |
| 363 | Brothers Grimm | 19.95 | -1.737 | 328 |
| 364 | Daniel Defoe | 19.72 | -1.779 | 232 |
| 365 | Stephenie Meyer | 19.59 | -1.803 | 94 |
| 366 | Harriet Beecher Stowe | 19.54 | -1.812 | 219 |
| 367 | Helen Keller | 19.54 | -1.812 | 63 |
| 368 | J.R.R. Tolkien | 19.45 | -1.829 | 414 |
| 369 | Tim Rice | 19.36 | -1.845 | 41 |
| 370 | Dr. Seuss | 19.36 | -1.845 | 376 |
| 371 | George R.R. Martin | 19.32 | -1.853 | 97 |
| 372 | Louisa May Alcott | 19.31 | -1.855 | 271 |
| 373 | Anne Frank | 19.3 | -1.856 | 92 |
| 374 | P.L. Travers | 19.09 | -1.895 | 52 |
| 375 | Peter Benchley | 18.86 | -1.937 | 51 |
| 376 | Margaret Mitchell | 18.6 | -1.985 | 370 |
| 377 | Erle Stanley Gardner | 18.38 | -2.026 | 47 |
| 378 | J.K. Rowling | 17.91 | -2.112 | 378 |
| 379 | Suzanne Collins | 17.79 | -2.134 | 82 |
| 380 | Charles M. Schulz | 17.36 | -2.213 | 69 |
| 381 | Aesop | 15.71 | -2.516 | 169 |
In 2011, IBM’s Watson defeated Ken Jennings and Brad Rutter.1
THE OLD TESTAMENT
In Genesis 21, Abraham banishes Hagar & this son of theirs to the desert; call him…
Ishmael
Ways in:
fredner.org/slides