Assessing the Accuracy and Quality of Answers from Your Legal AI Tool

By Simon Weierman

Lawyers must rely on the accuracy and quality of their legal research when providing counsel to clients, drafting business agreements or writing legal documents for submission to the court. Inaccurate or non-responsive results from a search of legal databases can lead a lawyer down the wrong path, jeopardizing their legal strategy and credibility.

Lawyers must rely on accurate legal research when drafting contracts and agreements. Consulting legal precedent ensures alignment with established interpretations, reducing litigation risks and providing predictability. Reviewing current regulations ensures compliance with the latest standards, crucial in dynamic fields with frequent changes. Non-compliance can render contracts unenforceable or result in penalties. Reflecting precedent and regulations creates robust, enforceable documents protecting all parties' interests. This principle has been reinforced by the emergence of generative AI tools, with some producing "hallucinated" answers, prompting law firms to adopt specialized Legal AI trained for the legal profession.           

This basic axiom has been brought into renewed focus with the emergence of generative artificial intelligence (Gen AI) tools. Last year, a number of lawyers began to discover the risks of using general-purpose AI tools for legal research, and discovered the potential of Gen AI to produce “hallucinated” answers. This has caused a growing number of law firms and in-house legal departments to turn their attention to Legal AI, Gen AI tools trained for the legal profession.

Lawyers must carefully consider the quality of a Legal AI tool before selecting one they plan to integrate into their workflows. To help them with this due diligence, we ­­recently published “The Definitive Guide to Choosing a Gen AI Legal Research Solution,” a free buyer’s guide that details what law firms and in-house legal departments should look for so they can select the most appropriate solution for their organization.

One of the pillars that we unpack in the guide is Answer Quality.

3 Key Measures

 When evaluating a Legal AI solution, you should ask is the solution powered by a fine-tuned LLM alone, or does it have access to an authoritative content collection and utilize RAG or other approaches to draw its substance from that collection? If your legal AI solution is going to require knowledge of the law, it needs to be backed by global authoritative content.  This technology model is necessary to surface the most comprehensive, accurate set of results that are responsive to your query.

There are a few important measures that you can use to understand the inner workings of a Legal AI solution, helping you to assess the accuracy and quality of its answers:

1. Comprehensiveness of results

Legal Research & Drafting are conducted across a wide spectrum of client matters, practice areas, & jurisdictions. Any Legal AI solution touting "end-to-end" support must have access to and utilize a large repository of accurate and up to date legal content to serve each practitioner's wide-ranging needs. This serves as the grounding insights for any large language models involved in producing the output, so that it is not relying upon its own pre-training involving non-legal data which can lead to inaccurate and hallucinated responses.

2. Semantic search

Semantic search can understand the underlying message of your search query, reading between the lines of the words you typed to grasp your intent, and then matching your query to related concepts. This is distinct from keyword search, which simply retrieves answers that match the text entered in the search box. Semantic search is a superior model for a Legal AI solution because it increases the precision of your results, delivering answers that are more relevant and saving you the time required to wade through extraneous information.

3. Citation validation and grounding

Legal industry observers are by now very familiar with the risk posed by “open web” Gen AI tools that infamously hallucinated various case citations that didn’t even exist. These general-purpose AI models struggle with legal use cases because the underlying content feeding the models may be dated, lack citation authority, and are otherwise prone to factual and conceptual hallucinations. A trusted Legal AI solution should be grounded in authoritative legal content, check all linked legal citations against a reliable citation service to ensure validation, and provide results backed by verifiable authority.

Questions to Ask About the Accuracy and Quality of Answers

When evaluating a Legal AI solution, here are some questions to pose to the provider about the accuracy and quality of answers produced by their product:

• What is the size of the primary and secondary law database that your solution accesses to surface authoritative legal content in response to search queries?
• Does your tool require a separate subscription to access those primary and secondary sources or is it all integrated under one product experience?
• What options are available for combining keyword and semantic search techniques while conducting research?
• Does the Legal AI output provide in-line citations and links back to the original source material used for creating its answers?
• What steps do you take to minimize hallucination risks?

Evaluate Lexis+ AI

Lexis+ AI is our breakthrough Gen AI solution that we believe will transform legal work by providing a suite of legal research, drafting, and summarization tools that delivers on the potential of Gen AI technology.

Lexis+ AI pairs our unsurpassed legal content with breakthrough Gen AI technology in a way that could redefine the way that legal research is conducted and legal work product is created. Its answers are grounded in the world’s largest repository of accurate and exclusive legal content from LexisNexis with industry-leading data security and attention to privacy. Click here to request a free trial.

To download a free copy of The Definitive Guide to Choosing a Gen AI Legal Research Solution, please click here.