AI Outperforms Law Professors in Legal Reasoning Study
A recent study spearheaded by Stanford University has revealed a surprising outcome: law professors demonstrated a preference for answers generated by artificial intelligence over those written by their human colleagues when evaluating legal reasoning tasks.
The research involved 16 professors from 14 prominent U.S. law schools, including Stanford, Yale, New York University, the University of Chicago, Georgetown, UCLA, and the University of Virginia. These academics developed 40 contract law questions encompassing legal doctrine, case law, hypothetical scenarios, and policy considerations, providing an ideal testing ground for the capabilities of modern AI.
“Large language models (LLMs) are increasingly promoted as educational tutors, yet most evaluations focus on domains with a single ground truth,” the researchers noted. “Many disciplines, however, hinge on judgment: reasoning, weighing ambiguity, and reaching defensible conclusions. Law provides a sharp test.”
In a series of 2,918 blinded comparisons, professors were asked to select the answer they would prefer to give to a student. Google’s Gemini 2.5 Pro emerged victorious in 75.92% of its matchups against human instructors, while the tech giant’s NotebookLM won 74.75% of the time. This indicates that AI-generated responses were favored over human ones in approximately three-quarters of the evaluations.
To ascertain if these results reflected a broader professional consensus, the researchers analyzed the extent to which professors agreed when assessing the same answer pairs. “Observed agreement exceeded the level expected if judgments were entirely idiosyncratic, indicating that the LLMs’ success reflects alignment with common disciplinary criteria,” they stated.
The study further found that AI models surpassed human instructors across various categories, including recall questions related to case law, statutes, or doctrine, as well as hypotheticals and policy discussions.
“To probe whether any LLM advantage might be driven by surface-level writing style rather than substantive content, we additionally engineered a set of lexico-syntactic features—answer length, structural organization, reasoning nuance, legal anchors, confidence tone, clarity, and pedagogical support—and tested how much of the preference pattern they could explain,” the study elaborated.
AI-generated answers were also flagged as harmful less frequently than those authored by professors. Gemini exhibited a 3.41% harmfulness rate and NotebookLM 3.64%, a stark contrast to the 12.06% rate for human instructors. In a separate analysis involving additional models, Anthropic’s Claude Opus ranked first with a 4.7% harmfulness rate, followed by OpenAI’s ChatGPT at 5.4% and Gemini 2.5 Pro. Notably, every AI model evaluated outperformed human instructors on average in this regard.
However, the researchers cautioned that the study did not measure whether the AI-generated answers aligned with individual professors’ specific teaching preferences. This leaves open the possibility that the AI responses were perceived as generally acceptable rather than tailored to a particular instructor’s pedagogical approach.
“While LLM responses are generally preferred over those of human instructors, our evaluation setting does not allow us to directly measure the extent to which instructor preferences are satisfied,” the study acknowledged. “It is at least theoretically possible that LLMs, although generally delivering stronger responses, still generate answers that are merely viewed as ‘good enough.’”
This study emerges as courts, law firms, and law schools increasingly grapple with the integration of artificial intelligence into the legal profession. In March, the Los Angeles Superior Court initiated testing of AI tools to assist judges in managing burgeoning caseloads, while law schools are actively incorporating AI training programs.
“The potential benefits of these new technologies as a force multiplier in the practice of law just can’t be ignored,” John P. Anderson, Dean of Mississippi College School of Law, previously told Decrypt. “Whether our students plan to be litigators or transactional attorneys, their future employers will expect familiarity with these AI tools. We want the firms hiring our students to be confident that every MC Law grad is competent in AI technologies.”
Concurrently, law firms continue to encounter cases complicated by AI-generated errors, including hallucinations. In April, the law firm Sullivan & Cromwell admitted to a U.S. bankruptcy court that a recent filing in a high-profile case contained fabricated citations produced by AI.
Leave a Reply