Some weeks ago, I published an article in this journal on Metacognition and AI – How to Improve Managerial Competence (https://slguardian.org/metacognition-and-ai/). Broadly speaking, metacognition is “thinking about thinking” or fact checking and prudent judgement.
A recent development on this subject occurred in the case of Nippon Life Insurance Company of America v. OpenAI where AI was sued for giving false and non-existing facts and information which managers must take note of. I acknowledge with appreciation that some of the text of this article is adapted from a recent post on LinkedIn by Mike Streich, a personal injury trial lawyer, .
The Case
The rapid diffusion of artificial intelligence into professional disciplines has begun to generate legal controversies that are as intellectually provocative as they are practically consequential. The recent action instituted by Nippon Life Insurance Company of America against OpenAI provides an illustrative example of the legal tensions that arise when emerging technological tools intersect with the established boundaries of the legal profession. The complaint, filed only recently, advances three principal allegations: tortious interference with contractual relations, abuse of process, and the unauthorized practice of law. Each of these claims reflects not merely a dispute between litigants but a broader anxiety within the legal system regarding the role of algorithmic systems in activities traditionally reserved for trained advocates.
The factual matrix underlying the dispute is, at first glance, deceptively simple. The matter arises from litigation involving a self-represented, or pro se, litigant who relied extensively on generative artificial intelligence—specifically the conversational system known as ChatGPT—to prepare motions, undertake legal research, and challenge a settlement agreement that had already been concluded and subsequently upheld by the court. The litigant, acting without the benefit of legal counsel, deployed the system in a manner that effectively transformed the tool into a surrogate legal assistant. What followed, however, reveals the epistemic limitations inherent in such reliance.
The Falsehood
Among the documents submitted to the court by the litigant was a pleading that cited a purported judicial precedent styled Carr v. Gateway, Inc., reported as 944 F. Supp. 2d 602 (D.S.C. 2013). The citation was offered as authority for the proposition that a court had declined to compel arbitration in a dispute involving the Employee Retirement Income Security Act—more commonly referred to as Employee Retirement Income Security Act (ERISA). The difficulty, however, lay in a simple but profound reality: the case did not exist. No such judicial decision had ever been rendered. The citation, the facts, and the holding were entirely fictitious.
More troubling still was the sequence of events that followed. When queried directly as to the existence and relevance of the case, the artificial intelligence system reportedly reaffirmed the citation, reproduced the same fabricated reference, and offered a summary of the purported legal reasoning contained therein. The litigant, lacking the training and instinctive skepticism that typically characterize professional legal practice, relied upon the representation in good faith. In doing so, she unwittingly introduced into the judicial record an authority that was entirely the product of algorithmic generation.
Other Cases
n Mata v. Avianca, Inc., a case arising from an alleged in-flight injury to a passenger during international carriage governed by the Montreal Convention of 1999, the United States District Court for the Southern District of New York confronted an unusual and troubling procedural irregularity involving the submission of fictitious legal authorities generated by the artificial intelligence platform ChatGPT. Counsel for the plaintiff filed an affirmation opposing the carrier’s motion to dismiss that cited several judicial decisions which, as later revealed, did not exist in any recognized body of jurisprudence.
The attorney had relied on ChatGPT to conduct legal research and draft the submission, operating under the mistaken belief that the system could not fabricate case law. The court found that counsel acted in subjective bad faith by signing and filing the document without verifying the authorities cited or undertaking even minimal inquiry to confirm their authenticity. This lapse was aggravated by the filing of a sworn affidavit attaching purported excerpts of the fabricated decisions and by a misrepresentation made to the court in seeking additional time. In invoking sanctions under Rule 11, the court emphasized that the use of artificial intelligence does not absolve legal practitioners of their professional duty to verify the accuracy of legal authorities placed before the court.
In another instance, in a case heard in 2024, a solicitor in the Australian state of Victoria became the first practitioner in that jurisdiction to face professional sanctions for submitting to the court authorities generated by artificial intelligence that contained fabricated legal citations. The matter arose when the lawyer, representing a party in a matrimonial dispute, presented a list of purported precedents requested by the presiding judge. Upon scrutiny, the court discovered that the cited cases were untraceable in any recognized legal repository.
The practitioner subsequently conceded that the research had been produced through AI-enabled legal software and acknowledged that he had failed in his professional duty to independently verify the accuracy of the material before placing it before the court. Although he tendered an unconditional apology and compensated the opposing party for the costs of the aborted hearing, the court deemed it necessary, in the broader public interest, to refer the matter to the regulatory authority. Following investigation, the Victorian Legal Services Board imposed disciplinary measures restricting the lawyer from practicing as a principal and requiring a period of supervised practice, thereby affirming the fundamental principle that technological convenience can never supplant the lawyer’s overriding duty of candor and diligence to the court
The Fallout
These cases underscore a critical jurisprudential issue: the distinction between a technological instrument and a professional decision-maker. Artificial intelligence systems, particularly those built upon large language models, possess a remarkable capacity to generate text that bears the outward form of legal reasoning. They can replicate citation styles, emulate doctrinal argumentation, and even construct narratives that resemble judicial analysis. Yet this facility with linguistic structure does not equate to epistemic awareness. The system cannot discern whether a proposition is grounded in verifiable authority or whether it is simply an internally plausible construction assembled from patterns in training data.
The trained lawyer, by contrast, operates within a disciplined intellectual framework shaped by legal education, professional ethics, and practical experience. A practitioner confronted with an unfamiliar case citation would instinctively verify its authenticity—often within seconds—through established legal research databases or institutional memory. The pro se litigant, however, lacks this protective layer of professional skepticism. For such individuals, the polished confidence of machine-generated language may create an illusion of reliability that is difficult to penetrate.
The consequences of this epistemic asymmetry are significant. Courts operate on the premise that authorities cited by parties are real, verifiable, and relevant. When fictitious precedents enter the judicial process, they not only undermine the credibility of the party submitting them but also consume judicial resources and threaten the integrity of adjudication itself. It is therefore unsurprising that the present litigation has attracted considerable attention among legal practitioners, scholars, and technologists alike.
In the Nippon Life Insurance Co. case, from the perspective of the plaintiff insurer, the matter is framed as a form of interference with an established contractual arrangement. The settlement agreement that the litigant sought to challenge had already been concluded and affirmed by the court. By assisting the litigant in generating filings designed to reopen or undermine that agreement, the argument goes, the artificial intelligence system facilitated conduct that interfered with the insurer’s contractual rights. Coupled with allegations of abuse of process and unauthorized practice of law, the complaint attempts to situate the conduct of the technology provider within doctrinal categories traditionally applied to human actors.
The allegation of unauthorized practice of law is perhaps the most conceptually provocative of the three. Legal systems in many jurisdictions maintain strict regulatory boundaries around the provision of legal advice and representation. These restrictions are intended to protect the public from unqualified practitioners and to preserve the professional standards associated with legal advocacy. If a technological system is deemed to have effectively provided legal advice—particularly when relied upon by a litigant in active proceedings—the question arises whether the entity responsible for that system has crossed into a domain regulated by professional licensing regimes.
Such a determination would have far-reaching implications. The burgeoning field of legal technology has, over the past decade, produced an array of tools designed to assist lawyers in research, drafting, and case analysis. These systems have generally been marketed as instruments that augment professional work rather than replace it. A judicial finding that generative artificial intelligence constitutes the unauthorized practice of law could compel developers to fundamentally reconsider how such tools are designed, marketed, and deployed.
Yet it is equally important to avoid conflating the capabilities of technology with the responsibilities of professional judgment. Artificial intelligence systems can undoubtedly enhance the efficiency of legal practice. They can accelerate research, organize information, and generate preliminary drafts with remarkable speed. Many practitioners now employ such tools routinely as part of their professional workflow. Used responsibly, they can make lawyers more thorough, more prepared, and more responsive to the complex informational demands of modern litigation.
What these systems cannot replicate, however, is judgment. Legal judgment is not merely a matter of assembling authorities or constructing arguments. It involves strategic evaluation, contextual awareness, and an appreciation of the subtleties that govern judicial decision-making. A seasoned practitioner understands the predispositions of a particular court, the practical consequences of a procedural step, and the circumstances in which a technically sound argument may nevertheless be strategically ill-advised. Sometimes the most prudent course of action in litigation is not to file a motion at all—a conclusion that emerges not from algorithmic reasoning but from human experience.
Artificial intelligence, by its very architecture, lacks this contextual awareness. It does not know the personalities of judges, the culture of specific jurisdictions, or the unwritten conventions that shape courtroom advocacy. Nor can it assess whether a particular argument, though doctrinally plausible, is likely to be dismissed by the court as irrelevant or opportunistic. In this sense, the technology functions best as a sophisticated drafting and research assistant rather than as a substitute for professional counsel.
My Take
While the utility of artificial intelligence in assisting research, drafting and preliminary analysis cannot be gainsaid, it must be recognized that these systems are not, in their present form, repositories of verified jurisprudence comparable to authoritative legal research platforms such as Westlaw or LexisNexis. The architecture of generative language models is premised on probabilistic prediction of linguistic patterns rather than on the retrieval of authoritative legal sources. Consequently, while such systems may generate citations that appear plausible and structurally correct, they may at times cite cases that are inaccurately described or entirely fictitious.
In principle, the possibility of litigation against developers of artificial intelligence systems cannot be entirely excluded. Claims grounded in negligence, misrepresentation, or even unauthorized practice of law might theoretically be advanced. However, such claims encounter formidable legal obstacles, not least because the providers of these technologies explicitly disclaim the provision of legal advice and caution users that outputs may contain inaccuracies. In addition, the nature of the technology itself militates against any guarantee of accuracy. A system such as ChatGPT does not operate as a real-time legal database nor does it possess the inherent capability to authenticate the existence, precedential value, or procedural history of judicial decisions across multiple jurisdictions. Rather, it generates responses based on statistical correlations derived from vast corpora of text, which may reproduce the form and structure of legal citations without necessarily corresponding to actual authorities.
For these reasons, the notion that generative artificial intelligence could offer a legally enforceable guarantee of accuracy remains untenable. The law is itself a dynamic and evolving system, constantly reshaped by new judicial decisions, legislative enactments and doctrinal developments. Any technological system purporting to provide definitive legal authority would necessarily require continuous verification against authoritative databases and editorial oversight. Until such integration becomes structurally embedded within these tools, their role must remain that of an auxiliary research instrument rather than an authoritative source of legal precedent. Accordingly, courts and professional regulators have increasingly made it clear that the ultimate responsibility for the accuracy of legal submissions continues to reside squarely with the lawyer who signs and files them.
The litigation initiated by Nippon Life therefore represents more than an isolated dispute. It is emblematic of a broader transitional moment in which legal institutions are grappling with the integration of powerful digital tools into professional practice. Courts, regulators, and practitioners must collectively determine where the boundary lies between permissible technological assistance and impermissible substitution for licensed expertise.
In the final analysis, the lesson emerging from this controversy is both simple and enduring. Technology can amplify human capability, but it cannot replace human responsibility. Artificial intelligence, when used thoughtfully, can be an invaluable aid to the modern lawyer. Yet reliance upon it without verification or professional oversight carries risks that the legal system is only beginning to confront.
The prudent course, therefore, is neither uncritical enthusiasm nor reflexive hostility toward technological innovation. Rather, it is the cultivation of disciplined usage: employing the tools where they enhance understanding, while maintaining the vigilance and skepticism that lie at the heart of legal reasoning.
In the interim, the message for practitioners and litigants alike is clear. Embrace the tools that modern technology provides, but never surrender the critical faculties that define the practice of law. Tools exist to serve the user; they should never become the unseen author of decisions that demand the careful exercise of human judgment.
