Recent months have cast a spotlight on the growing, yet fraught, relationship between artificial intelligence and the US legal system. As MIT Technology Review reports, a series of troubling incidents have revealed how AI-generated errors can slip into legal documents and even court orders, sometimes with significant consequences. Lawyers from prestigious firms have submitted filings citing fictitious cases, and in one striking example, a Stanford professor offered sworn testimony riddled with AI hallucinations during a case concerning deepfakes—despite his own expertise in AI and misinformation. While judges have responded with reprimands and fines, these episodes have exposed the legal community’s uneasy struggle to adapt to AI’s limitations.
Despite these setbacks, some judges have embraced generative AI, hoping it might ease the severe backlog plaguing many US courts. Early adopters see promise in using AI to expedite legal research, summarise lengthy cases, draft routine orders, and streamline administrative tasks. Yet, as several mistakes linked to AI have already been cited by judges themselves, doubts remain about how safely and effectively these tools can be integrated into the judicial process. For instance, a federal judge in New Jersey recently had to withdraw a flawed order that likely contained AI-generated errors, and a judge in Mississippi declined to explain why his order included what appeared to be hallucinations from AI. Such incidents reveal two challenges: defining which routine judicial tasks are appropriate for AI assistance, and the lack of accountability judges face when AI-related errors slip through, unlike lawyers who face professional sanctions.
Federal judge Xavier Rodriguez of the Western District of Texas offers a measured perspective grounded in firsthand experience. He recounts a case where both self-represented parties used AI tools and submitted filings referencing non-existent cases. While he had the authority to sanction them, Rodriguez chose leniency, pointing out that mistakes are hardly new to the legal field and that “lawyers have been hallucinating well before AI.” He uses AI tools to summarise cases and prepare hearing questions, tasks he believes do not require human judgment and allow him ample opportunity to catch errors before they reach the courtroom. However, he draws a firm line at decisions involving discretion or judgment—such as bail determinations—deeming those beyond the safe scope of AI assistance.
Erin Solovey, a human-AI interaction researcher at Worcester Polytechnic Institute, echoes the complexity of defining appropriate AI uses in the judiciary. Her research in the UK shows that what judges consider safe AI tasks varies widely. Moreover, AI’s limitations remain acute: summarising documents can yield inconsistent results depending on the training data, and AI’s logical sequencing of case events can produce plausible but factually inaccurate timelines. In response to these concerns, Rodriguez and colleagues helped develop guidelines published by the Sedona Conference, recommending cautious AI use restricted to tasks like legal research or preliminary transcript creation, with an emphasis on verifying AI outputs due to the persistent risk of hallucinations.
Judge Allison Goddard of California shares a pragmatic approach to AI. After witnessing AI’s impact even in a high school classroom, she now experiments with various models including ChatGPT and Claude. Goddard uses AI as a “thought partner” to digest complex orders and suggest relevant questions for attorneys, while cautioning against relying on it for nuanced legal analysis or criminal matters where bias and precision are critical. She encourages her clerks to use AI where appropriate but still relies on specialised legal AI tools for substantive law-related tasks. Yet, she recognises the broader dilemma facing many users of AI: the allure of fluent, humanlike responses masks persistent risks, including errors and data privacy concerns. For judges, the stakes are particularly high, as mistakes can have profound and lasting effects on people’s lives. Goddard stresses the professional embarrassment and consequences that come with citing AI-generated inaccuracies.
Not all judges share the same caution. The idea that AI’s supposed objectivity might outperform human fallibility tempts some on the bench to adopt the technology rapidly, fearing that lagging behind could pose its own risks. But as Judge Scott Schlegel of the Fifth Circuit Court of Appeal warns, early adoption also risks creating a “crisis waiting to happen.” Unlike lawyers, judges have little room to retract mistakes once made; their rulings set binding precedent. Errors in crucial decisions—such as those involving child custody or bail—can cause irreparable harm if based on AI-generated fabrications or inaccuracies. Recent cases, including a Georgia appellate judge’s reliance on fabricated cases and a Mississippi federal judge’s unexplained errors in a civil rights ruling, underscore these dangers.
Schlegel emphasises that while AI can be a useful tool for rote tasks—summarising testimony, providing quick feedback—judges must treat such assistance like the work of a junior associate, checking thoroughly for mistakes. The core judicial function, he insists, is wrestling with complex, novel problems and making nuanced decisions—the “white-page problem”—which AI cannot and should not replace. He warns that relying on AI models to decide who receives custody of children or is eligible for bail threatens the very integrity of the justice system.
As AI becomes increasingly embedded in legal processes, the balance between efficiency and accuracy, innovation and accountability, grows ever more delicate. The experiences of judges navigating this new terrain—as detailed by MIT Technology Review—highlight the urgent need for careful guidelines, transparency, and a sober understanding of AI’s limitations before it reshapes the courts in ways that may prove irreversible.
