Downloads · 30 days
0
dcrodriguez/virginia-legal-rag-lexva
virginia-legal-rag-lexva is a machine learning model from dcrodriguez. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-4.0.
LexVA is a Retrieval-Augmented Generation (RAG) pipeline designed to answer questions grounded in the Code of Virginia. This project addresses a core challenge in legal AI: modern LLMs often hallucinate legal requirem…
Downloads · 30 days
0
Access
Public
Updated Dec 4, 2025
Repo size
—
Likes
1
Public
Click a slice to open those files.
.py41.1 KB · 74%
From the Hugging Face model README
LexVA is a Retrieval-Augmented Generation (RAG) pipeline designed to answer questions grounded in the Code of Virginia. This project addresses a core challenge in legal AI: modern LLMs often hallucinate legal requirements, misinterpret statutes, or fail to retrieve the specific text needed for accurate reasoning.
By combining Qwen3-14B for generation with statute-level embeddings generated using the Qwen3 Embedding Model 0.6B, LexVA provides statute-grounded legal responses with improved citation reliability.
LexVA is intended for:
It is not intended for production legal advice or case outcome prediction.
LexVA relies on two public datasets:
Source statutes were collected via the official LIS API (October 2025).
Dataset link: https://huggingface.co/datasets/dcrodriguez/Code-Of-Virginia
This dataset was synthetically generated by prompting Qwen3-14B to create real-world legal questions and grounded answers. Each QA example is tied to a single statute. The dataset is used for pipeline benchmarking.
3,000 random statutes were sampled → statutes graded by Mistral-7B → 277 retained and used for QA pair generation.
Dataset link:
https://huggingface.co/datasets/dcrodriguez/Virginia-Statute-QA
No train/validation/test split was required for the RAG pipeline, but the 277-question QA set was used as held-out evaluation cases.
LexVA uses a retrieval-augmented generation (RAG) architecture designed for Virginia-specific legal question answering.
Core Components
Objective
To create a consistent, explainable, citation-grounded legal information system supporting Virginians navigating legal processes.
LexVA follows a pure RAG approach without model finetuning:
Chosen for strong reasoning, thinking, and citation capability, as well as multilingual and long-context strengths.
No finetuning was performed; the evaluation tests the effectiveness of retrieval + prompting.
Please note that prompts and answers are cached in the SQLite DB to speed up benchmarking tasks.
| Model / Pipeline | Recall@3 | Recall@5 | Faithfulness (DeepEval) | GEval (DeepEval) | Cosine Similarity |
|---|---|---|---|---|---|
| Qwen3 Embedding 0.6B | 84.32% | 89.73% | NA | NA | NA |
| Qwen3-14B (no RAG) | NA | NA | 0.923 | 0.257 | 0.716 |
| Qwen3-14B (with RAG) | NA | NA | 0.950 | 0.705 | 0.719 |
| Mistral 7B (no RAG) | NA | NA | 0.855 | 0.224 | 0.717 |
| Mistral 7B (with RAG) | NA | NA | 0.946 | 0.664 | 0.790 |
For information retrieval, Qwen3 Embedding 0.6B performs extremely well. Part of this may be due to using higher quality synthetic QA pairs generated from a curated set of graded statutes.
Cosine similarity measures how semantically similar two answers are by generating two embedding vectors. While it's quick to calculate, it ended up being a poor comparison metric for legal Q&A. You can see all the similarity values are within 8 percent. Answers generated without RAG were semantically similar, but contained many hallucinations and false statements.
The DeepEval benchmarks both use an LLM as a judge. I used GPT-5 Mini for these benchmarks as the judge model. Smaller local models had trouble with reasoning, and scores seemed inflated.
Faithfulness measures how correct the pipeline output is when given the relevant statute. The input is an actual output, and expected output, and retrieved context documents. It is a more standardized metric compared to GEval, but I think it struggled with RAG and non-RAG answers being similar despite citations being made up for non-RAG answers. Medium size models like Qwen 14B do a surprisingly good job answering these questions without RAG, but once again they make up most of their references, which makes it much less useful as a legal research tool. I think this is why Faithfulness metrics are as close as they are. Qwen3 14B scored higher than Mistral 7B which is expected.
GEval was by far the most useful metric. The input is an actual output and an expected output. GEval supports custom grading criteria which was necessary to properly benchmark this pipeline. The evaluation steps are below.
evaluation_steps=[
"Check whether the facts in 'actual output' contradict any facts in 'expected output'.",
"Heavily penalize omission of important details from the expected output.",
"If a specific statute is referenced in 'expected output' but not 'actual output' lower the score, but numerically close references are ok. For example, 55.1-1243.2 is close to 55.1-1244.1, but is far from 19.2-308.",
"Additional statute references in actual output are ok",
"Minor stylistic differences or changes in wording are acceptable.",
],
This benchmark was significantly more successful at identifying hallucinated citations. It evaluates both the claims made in the answer, as well as examines if the expected statute citations were used. You can see the largest delta between RAG and non-RAG answers using this benchmark.
git clone https://huggingface.co/dcrodriguez/virginia-legal-rag-lexvainstall.shpython setup.py. This will download the dataset, load it into a SQLite DB, and generate all the document embeddings.python example.pyNotes:
.clear_history() method.setup.py to point to your drive, and then instantiate the chat engine like this:sqlite_path = "/content/drive/MyDrive/LexVA/va_code.db"
chat_engine = LLMChatEngine(sqlite_path)
response = chat_engine.chat("<Question>", rag=True)
print(response)
Question:
"Is it illegal to offer someone money to vote a certain way in an election?"
Answer with RAG:
In Virginia, it is illegal to offer someone money or anything of value to influence how they vote in an election. This is explicitly prohibited by Virginia Code § 24.2-1007, which states:
"No person shall solicit or accept directly or indirectly any money or any thing of value to influence his or another's vote in any election."
Violating this law is a Class 1 misdemeanor, which carries potential penalties such as fines or imprisonment. The statute applies to all elections, including those for political party nominations or delegate selection.
This law is designed to prevent bribery or corruption in the electoral process. Offering money (or anything of value) to sway a vote is a criminal offense under Virginia law.
Note: This is general information and does not constitute legal advice. For specific cases, consult a qualified attorney.
LexVA is a retrieval-augmented legal question-answering system designed to help users better understand Virginia statutes, court forms, procedures, and publicly available legal information. It is not a lawyer, does not provide legal advice, and cannot replace professional legal counsel.
Users:
At least 16GB of VRAM is recommended
If you use LexVA in research or development, please cite:
BibTeX:
@misc{cuevasrodriguez2025lexva,
title = {LexVA: Retrieval-Augmented Legal Question Answering for Virginia},
author = {Cuevas Rodriguez, Dalila},
year = {2025},
howpublished = {Hugging Face Hub},
note = {An access-to-justice RAG system grounded in Virginia legal texts}
}
APA:
Cuevas Rodriguez, D. (2025). LexVA: Retrieval-augmented legal question answering for Virginia. Hugging Face. https://huggingface.co/virginia-legal-rag-lexva