With AI becoming an easily accessible and affordable tool, many have rushed to use AI in court cases. Both self-represented people and lawyers are tempted to use AI as easy tool to find case law and other precedents to cite in the courtroom. Researching case law can take hours which could detract time from other important tasks. Using AI to research case law provides lawyers the opportunity to work on cases quicker and at a faster rate. For self-represented people, researching case law can be an impossible task especially when they are not familiar or do not have access to case law databases. AI removes that issue and can provide case law instantly often without cost.
However, many people fail to realize that AI can often provide inaccurate information especially when it comes to case law. AI is a tool that aims to provide the exact answer that a person is searching for and will sometimes make it up if necessary. When lawyers and self-represented people use false case law and cite it the judge could rule in favour of someone who is relying on law that does not exist. This can cause major issues regarding fairness in the legal system. As well, when judges do realize the case law is fake, it can lead to fines or other similar punishments for lawyers. For self-represented people it can be embarrassing in open court to think you are saying truthful information but end up misleading the court. Judges take the action of misleading courts very seriously and this could impact whether the case is ruled in your favour.
How Do AI Hallucinations Happen in Criminal Law?
Here are top 3 reasons why AI hallucinations are happening:
- Unreliable Sources: AI does not only look at case law databases when recommending cases. Instead, AI will scour the whole internet, including blog posts and forums. Sometimes these posts can cite stuff that is purely someone’s opinion and not actual valid case law. AI may believe it is valid caselaw and mislead you.
- Outdated information: AI sometimes do give proper case law, but it is out of date. So, there is a risk that even if it is real case law, it is no longer relevant or available. AI does not go the extra mile to see if a case law is not overturned or no longer in use. So, it will be up to a lawyer/self-represented person to check on secondary sources to see if the law is current and can be used in their case. If the law has been overturned, the judge will likely not be able to rely on it or rule in your favour.
- No supervision regarding citations: AI is not built with a tool that assesses legal citations. As a result, they will mischaracterize cases or say a case applies to your matter even when it doesn’t. So, there is a risk that the case law is real but AI wrongly characterizes that it applies in your case. When a case does not apply, the judge will likely not accept it as precedent or use it. AI can also change the ruling of the case. AI can often wrongly state that a case ruled a certain way when it did not. However, the worst offence remains that AI will often just completely make up citations that do not exist. The lack of oversight when it comes to AI citing cases has allowed AI to completely make up cases that “magically” apply to the facts that someone has provided. This is dangerous because people may believe that there is valid case law to back up their legal argument and when they argue it in court it is proven to be incorrect. Citing false cases in court can lead to consequences for the person using them.
In terms of the consequences in Zhang v Chen, Chen’s counsel cited two AI hallucinated cases. The other counsel representing Zhang noticed through research that Chen’s cases were ChatGPT hallucinations and sought for the judge to award special costs based on this action. Chen’s counsel apologized and did not intend to purposely mislead the court. Chen’s counsel provided that she was unaware of the harms of ChatGPT. Based on this the judge did not award special costs. However, Chen’s counsel still had to pay tariff costs to Zhang’s counsel. This could have been avoided if Chen’s counsel was aware of the potential for AI to hallucinate cases and the need to double check every case that AI provides. Furthermore, there are several cases in the U.S where lawyers have been forced to pay fines of up to $12,000 for citing AI hallucinations. To read further about that case follow this link: https://www.reuters.com/legal/litigation/judge-fines-lawyers-12000-over-ai-generated-submissions-patent-case-2026-02-03/
Based on this, you might be wondering: should lawyers and self-represented people in court completely avoid AI? The answer is not necessarily. AI can still be a useful tool in searching for cases and can be a good starting block. However, you should always double-check on legal databases whether the cases provided are valid/current law, what the ruling was in the case, and whether the case exists. People should take extra pre-cautions to ensure the accuracy of any case law that they plan to use in any court case. If this is not done, people can face the consequences of costs or dismissals. As well, there could be grave consequences to people’s confidence in the legal system if lawyers are able to use fake case law to win a case.
Overall, when deciding to use AI for case law research or for an argument in court, always proceed with caution. Always double-check your work before arguing it in a court of law or submitting documents to the court.
For further information regarding AI’s impact on court cases check out the following links: https://legal.thomsonreuters.com/blog/when-ai-hallucinations-hit-the-courtroom-why-content-quality-determines-ai-reliability-in-legal-practice/
