The New York Times published an interactive quiz pitting AI-generated prose against passages from literary authors, including Cormac McCarthy and Ursula K. Le Guin. Fifty-four percent of the more than 86,000 people who took the quiz said they preferred the AI-written passages.
The methodology is debatable. But the result confirms what people who use these models every day know: the writing gap has closed. Every professional who writes will eventually have to decide where on the spectrum they draw the line between their own voice and AI's.
The New York Times published an interactive quiz presenting readers with five pairs of short passages. In each pair, one passage was written by a published human author and the other was generated by AI. Readers were asked to identify which was which and, separately, which they preferred.
More than 86,000 people took the quiz. Of those readers, 54% said they preferred the AI-written passages over the human originals. And readers struggled to consistently tell the difference.
The human passages came from Cormac McCarthy, Ursula K. Le Guin, Carl Sagan, Hilary Mantel, and Elizabeth Bishop. The AI passages were generated by Anthropic's Claude Opus 4.5.
The methodology drew immediate criticism from writers and literary critics. The quiz used short, decontextualized fragments, just a few sentences stripped from longer works. Writer Ben Mauk argued that evaluating individual paragraphs is not how literature works. Writing as a craft involves voice, sustained argument, and ideas developed across thousands of words. Reducing the test to which sentence was more polished measures exactly the thing AI is best at and ignores everything else.
Nevertheless, supporters of the quiz countered that the results are significant. If readers genuinely prefer AI prose at the sentence level, the gap in the quality of surface-level writing has effectively closed.
The results have come in the middle of an already intense conversation. In recent weeks, Cleveland.com announced it would use AI for writing and the Associated Press drew scrutiny for how it described AI writing within its newsroom. The question of what AI means for professional writing is no longer theoretical.
SmarterX and Marketing AI Institute founder and CEO Paul Roetzer broke down what the quiz actually tells us, and what it misses, on Episode 203 of The Artificial Intelligence Show.
86,000+ - Number of people who took the NYT's AI vs. human writing quiz
54% - Quiz-takers who said they preferred the AI-written passages
5 - Genres tested (literary fiction, fantasy, science writing, historical fiction, and poetry)
5 - Levels in Roetzer's Human-to-Machine Scale for writers, from all-human to all-machine
Anyone using generative AI daily already knows the answer. Roetzer's reaction was frustration that the quiz asks the wrong question entirely.
"I think we can assume that for most people, it will become increasingly difficult to tell what a human wrote versus what an AI wrote, especially when the AI is trained to write in a specific writer's style and tone."
— Paul Roetzer, founder and CEO of SmarterX
"I don't have any problem saying it's really, really good," he says of AI's writing capability. "Is it creative like a human? No. It doesn't have the life experiences we have. It doesn't feel emotions, but it can simulate them."
Roetzer thinks the quiz is asking a question that was already answered. What matters more:
"The bigger question becomes: When should we use AI to write?"
— Paul Roetzer, founder and CEO of SmarterX
Writing is thinking. Roetzer approaches this as a writer, not a tech expert. He graduated from the E.W. Scripps School of Journalism at Ohio University, has authored three books, and hosts a weekly AI podcast. For him, the question is deeply personal.
"For me, writing is thinking," he says. "It's how I process information, comprehend concepts, build competency, and pursue mastery of topics. I write for myself to learn, understand, and grow, and I write for audiences to educate, entertain, and inspire."
"There are absolutely use cases like my LinkedIn posts, my podcast commentary, and my personal ExecAI newsletter when I want the writing to be 100% authentic and personal," Roetzer says. "I have no use for AI in these instances. No matter how much time it would save, the process is the purpose."
Five levels of AI writing. At last year's AI for Writers Summit, Roetzer introduced a framework called the Human-to-Machine Scale, mapping writing tasks across five levels. At level zero, the writer is the sole creator: essays, investigative journalism, keynote speeches. At levels one and two, AI assists with research and drafting while the author leads. At levels three and four, AI leads and the human approves: product descriptions, SEO content, routine communications.
"The decision of when and how to use AI in your writing isn't binary. It exists on a spectrum and it is a subjective and personal choice."
— Paul Roetzer, founder and CEO of SmarterX
The tension is real, even for the people who understand it best. Roetzer is candid about wrestling with this himself. He has book ideas he considers genuinely important. Writing a business book takes 300 to 500 hours. As a CEO, he does not have that time.
"I got this idea. I think it's really, really important. I could get this to market in probably 30 days if I work with an AI to do it," he says. "But if I stop and have to write it, it's not going to happen this year."
"It comes down to this authenticity and value creation," he says. "What is the most important thing to do here? For me to write it all? And for me to get the ideas out and be the shepherd of those ideas?"
The quiz is a useful conversation starter, but it answers a question most AI-forward professionals moved past months ago. AI writes well. The models are getting better fast. The gap is closed or at least significantly closing.
The more important question for you is: Where on the spectrum does your work need to be authentically yours? That question matters more than any quiz. And writing as thinking is a skill-building mechanism that organizations cannot afford to lose. If organizations let AI handle all the prose, they risk losing the training ground for the critical thinking that produced it.
"As AI can take on more of the go-to-market side, such as marketing, sales, success, content, how do you train those people to think if they're not having to write anymore?"
— Paul Roetzer, founder and CEO of SmarterX
At the same time, Roetzer is actively exploring how the disruption creates opportunity for people with writing skills. "I'm also actively trying to figure out how do we scale around former journalists," he says. "Because there's a lot of extremely talented people who are either unemployed right now, or face challenging career path futures within the journalism industry."
The skills that make a great writer, critical thinking, storytelling, the ability to process complex information and communicate it clearly, are precisely the skills most valuable when working alongside AI.
The AI for Writers Summit is returning later this spring, and the question of when and how to use AI in writing will be a central topic. The Cleveland.com and Associated Press controversies are not going away. Expect more newsrooms and content organizations to publicly wrestle with where they draw the line.
Meanwhile, the models keep improving. The quiz used Claude Opus 4.5. The next generation will be better. The window during which human writers can point to a clear quality gap at the sentence level is closing, which makes the "when should we use it" question more urgent, not less.
Who's a Better Writer: A.I. or Humans? Take Our Quiz → nytimes.com
B.D. McClay: Some Cranky Thoughts About That NYT AI Quiz → bdmcclay.com
Ben Mauk: A Simple Question About AI and Art → benmauk.substack.com
Nathan Yau: AI or Human, Writing Passages → flowingdata.com
Heard on The Artificial Intelligence Show, Episode 203
Paul Roetzer and Mike Kaput discuss the NYT AI writing quiz, the framework for deciding when to use AI to write, and what the closing quality gap means for professional writers. Listen →