insignis-eb1a-lora-comparison

Insignis -- EB-1A Petition Intelligence

LoRA fine-tuned models and a terminal CLI for analyzing EB-1A extraordinary ability visa petitions. Three adapters trained on 1,467 curated AAO appellate decisions, plus a full-featured terminal interface for case management, evidence analysis, and AI-powered criteria evaluation grounded in USCIS policy.

Pre-publication research artifact. Part of the alphaXiv x marimo paper "How Much Does Base Model Choice Matter?" Not independently evaluated for production legal use. Don't rely on outputs as legal advice.

Author

Josue Diaz Flores

• GitHub: github.com/josuediazflores
• LinkedIn: linkedin.com/in/josuediazfl

---

Models

This repository hosts three LoRA adapters trained on the same EB-1A dataset with identical hyperparameters, so that the only experimental variable is the base model.

josuediazflores/insignis-eb1a-lora-comparison/
├── gemma-4-E4B/         Gemma 4 E4B Instruct + LoRA r16  (best performer)
├── qwen-2.5-7b/         Qwen 2.5 7B Instruct + LoRA r16
├── llama-3.1-8b/        Llama 3.1 8B Instruct + LoRA r16
└── cli/                 Insignis Terminal UI source code

Gemma 4 E4B + LoRA (recommended)

Best overall performer. 64.2% outcome prediction accuracy on 260 held-out AAO cases -- a +55 point lift over the 9.2% base model accuracy. This is the default model used by the Insignis CLI.

Qwen 2.5 7B + LoRA

Strong base model performance (52.3% accuracy without fine-tuning) but LoRA training degraded outcome prediction to 36.5%. Criteria analysis remains competitive.

Llama 3.1 8B + LoRA

Modest +6.1 point lift (15.4% to 21.5%). Lowest validation loss of the three (0.8176), but benchmark accuracy trails Gemma significantly.

---

Benchmark Results

Evaluated on 260 held-out AAO decisions never seen during training, scoring, or deduplication. Extraction performed independently with Claude Opus.

Outcome Prediction Accuracy

Criteria Identification (Weighted F1)

Training Metrics

Note: Gemma's higher absolute loss is due to differences in loss computation (HF Trainer includes padding tokens in denominator). Qualitative output quality is strong across all three.

---

Insignis Terminal UI

A full terminal interface for EB-1A petition analysis, built with Ink (React for CLI) and TypeScript.

<!-- Screenshot placeholder: run `npm run dev` and capture the splash screen -->

Features

• Case management -- create, browse, and track EB-1A petition cases
• Evidence upload -- attach and organize supporting documents per criterion
• AI-powered criteria analysis -- evaluate petitioner evidence against all 10 EB-1A criteria at 8 CFR 204.5(h)(3)
• Conversational follow-up -- ask questions about analysis results, request deeper dives into specific criteria
• RAG-grounded policy citations -- responses cite the USCIS Policy Manual directly, not just model knowledge
• Streaming inference -- token-by-token output from local Ollama or remote endpoints

Quick Start

1. Install Ollama

# macOS
brew install ollama
ollama serve

2. Create the Insignis model with the LoRA adapter

Create a Modelfile in your working directory:

FROM gemma-4-E4B-it
ADAPTER ./gemma-4-E4B

Then build it:

ollama create insignis-eb1a -f Modelfile

3. Run the CLI

cd cli
npm install
npm run dev

The CLI defaults to local Ollama at localhost:11434 with model insignis-eb1a.

Environment Variables

Examples:

# Use the Qwen adapter instead
INSIGNIS_MODEL=insignis-qwen npm run dev

# Point at a remote GPU server
INSIGNIS_API_URL=http://your-gpu-server:8000 npm run dev

Slash Commands

Stack

• Ink 7 + React 19 for the terminal UI
• TypeScript strict mode, ESM
• Ollama as the inference backend
• RAG over USCIS Policy Manual for grounded citations

---

Dataset

All three adapters are trained on the same curated dataset.

Pipeline

4,643 raw AAO decisions
  -> quality scoring + filtering
1,467 curated cases
  -> 4 task types x 10 criteria
13,083 ChatML training examples

Details

• Source: Publicly available AAO EB-1A appellate decisions
• Curation: Quality scoring rubric applied to each decision; duplicates removed at 0.95 cosine similarity threshold
• Task types: criterion identification, outcome prediction, evidence summarization, reasoning extraction
• Criteria covered: All 10 evidentiary criteria at 8 CFR 204.5(h)(3) (awards, membership, published material, judging, original contributions, scholarly articles, exhibitions, leading/critical role, high salary, commercial success)
• Split: 10,466 train / 1,308 validation / 260 held-out test (independently extracted)
• Dataset on the Hub: josuediazflores/aao-eb1a-chatml

---

Training

LoRA Configuration

Trained with HuggingFace PEFT. Qwen and Llama used a custom PyTorch + torch_xla 2.5 loop on TPU v4-32. Gemma used the HuggingFace Trainer on AMD MI300X due to torch_xla incompatibilities with its multimodal architecture.

Loading an Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

BASE_MODEL = "google/gemma-4-E4B-it"
ADAPTER_SUBFOLDER = "gemma-4-E4B"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(
    base,
    "josuediazflores/insignis-eb1a-lora-comparison",
    subfolder=ADAPTER_SUBFOLDER,
)
model.eval()

---

Limitations

• Domain narrowness. Trained only on EB-1A AAO decisions. Performance on adjacent categories (O-1A, EB-2 NIW) is unknown.
• Appellate bias. AAO decisions over-represent contested or denied petitions.
• Not legal advice. Outputs are pattern matches over historical decisions, not attorney judgment.
• Single seed, single recipe. No rank ablations or recipe sweeps.
• Base model biases inherited. LoRA at r=16 modifies a small subspace of base model behavior.

---

License

MIT

---

Citation

@misc{diazflores2026insignis,
  title  = {How Much Does Base Model Choice Matter? A Three-Way Comparison of LoRA Fine-Tuning on Domain-Specific Legal Reasoning},
  author = {Diaz Flores, Josue},
  year   = {2026},
  note   = {alphaXiv x marimo competition entry},
  url    = {https://github.com/josuediazflores/ProjectGreenlight}
}

Acknowledgements

• Google TPU Research Cloud grant for v4-32 compute (Qwen + Llama)
• DigitalOcean AMD MI300X GPU credits (Gemma)
• Alibaba Cloud / Qwen team, Meta / Llama team, and Google / Gemma team for releasing their base models
• HuggingFace for hosting the dataset and adapters