Harvest-IQ

HarvestIQ - Hunting Intelligence Assistant

1. Introduction

Hunters planning a season must reconcile two disparate sources of information: dense regulatory PDFs specifying season dates, bag limits, and weapon rules, and separate harvest-report spreadsheets indicating which units actually produce animals. General-purpose LLMs have encountered state hunting regulations during training, but they blend details across states and years, producing season dates or bag limits that do not correspond to the specific unit a hunter is asking about, since nothing constrains the model to ground its answer in the relevant document. HarvestIQ addresses this problem for two states, Virginia and Idaho, by combining three retrieval paths, a structured harvest database, a dense vector store over regulation and harvest-report text, and an in-memory knowledge graph, behind a query optimizer that routes each question to the appropriate path and a synthesis step that requires a citation for every claim. Idaho harvest statistics (6,868 records across 311 hunt units, 2020 to 2025) are fully ingested, while Virginia is currently represented in the regulation corpus only, as this is a proof-of-concept build and Virginia harvest CSVs were not sourced in time for this submission. Benchmarked against a pre-RAG baseline of the same generator model, the pipeline raises numeric-fact accuracy on held-out harvest questions from 3 to 7 percent to 84 to 98 percent, depending on the generator, which constitutes the central result of this submission.

2. Data

The RAG pipeline draws on two data sources. Harvest statistics originate from the Idaho Fish and Game hunt planner, downloaded as per-species, per-weapon CSVs and normalized into a single HarvestRecord schema (state, season year, species, weapon type, hunt unit, harvest counts, success rate) so that Idaho's hunt-unit granularity and Virginia's county granularity can eventually be represented uniformly. A Virginia parser exists in the ingestion code for this purpose, but no Virginia harvest CSVs were ingested for this submission, so the structured database currently covers Idaho only. Regulation text comes from PDF digests published by the Virginia DWR and Idaho Fish and Game, extracted page by page with PyMuPDF, cleaned of font-substitution artifacts, and chunked at section boundaries into 1,017 chunks tagged by state and species (turkey, whitetail deer, black bear, elk, and others) using a heading-detection state machine, since the source PDFs contain no structure that can otherwise be exploited for chunking. As no public benchmark exists for this domain, four test sets were constructed directly from the ingested data to evaluate the pipeline: a 150-question testing split of single-year Idaho harvest-fact lookups held out from harvest_normalized.csv, a 25-question harvest-trend set requiring multi-year, year-over-year reasoning, a 9-question regulation-lookup set spanning both states, and a 4-question cross-state synthesis set requiring Virginia and Idaho regulation text to be combined in a single answer.

3. Methodology

Each question is routed through an LLM-powered query optimizer (Claude) that classifies intent and selects among four retrieval paths: the structured harvest database for exact counts and rankings, a dense vector store for regulation text and harvest-report narratives, or the knowledge graph for multi-year trends and cross-unit comparisons. Text is embedded locally with BAAI/bge-large-en-v1.5 via sentence-transformers and indexed in ChromaDB using cosine similarity. This configuration was selected after comparing three embedding/metric combinations on a 39-question retrieval-hit benchmark: bge-large-en-v1.5 with cosine similarity (hit@3 = 79.5 percent) and with L2 distance (also 79.5 percent, since BGE embeddings are already L2-normalized) against bge-small-en-v1.5 with cosine similarity (hit@3 = 38.5 percent). The large embedding model was selected on this basis, as embedding capacity, rather than distance metric, determined retrieval quality. The knowledge graph is a networkx MultiDiGraph constructed from the harvest CSV at startup, connecting states, hunt units, species, weapon types, and seasons so that trend and cross-unit questions can be answered without a vector search. For the generation step, four models were compared under identical retrieval conditions: microsoft/Phi-3.5-mini-instruct, Qwen/Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, all run locally on an RTX 5090, and Claude Sonnet 5, accessed through the Anthropic API, all under a synthesis prompt requiring every factual claim to cite its source and prohibiting answers drawn from outside knowledge. The Llama weights were loaded from NousResearch/Meta-Llama-3.1-8B-Instruct, an unmodified, ungated mirror of meta-llama/Llama-3.1-8B-Instruct. The deployed Space uses Claude for both the query optimizer and synthesis, as it led on most benchmark tasks reported below, though the three local models proved competitive or better on specific tasks, discussed in Evaluation.

4. Evaluation

Four benchmark tasks were constructed from the project's own ingested data, described in the Data section above, rather than drawn from an existing public RAG benchmark, as none was identified that covers state-level hunting regulations and harvest statistics. The tasks are a testing split of exact harvest-fact lookups, a harvest-trend task requiring multi-year reasoning, a regulation-lookup task spanning both states, and a cross-state synthesis task. Five conditions were compared on each task: Claude Sonnet 5 with RAG (the model deployed in this Space, in its claude-sonnet-4-6 form), Claude Sonnet 5 without RAG (the base, no-retrieval ablation of the same model), Phi-3.5-mini-instruct with RAG, Qwen2.5-7B-Instruct with RAG, and Llama-3.1-8B-Instruct with RAG. Phi-3.5-mini-instruct, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct were chosen as the three comparison models because all three are open-weight, instruction-tuned generators run locally on the same RTX 5090 rather than through a hosted API, giving a lower-cost, self-hosted point of comparison against the API-based Claude deployment, at a range of parameter counts (3.8B to 8B) similar to or smaller than typical open generator baselines.

The results confirm that retrieval augmentation achieves its intended effect across all four generators: numeric accuracy on the testing split rises from single digits to the 70s, 80s, and 90s once retrieval is added, and citation accuracy rises from 0 percent to near perfect, as each model is provided source text to cite rather than relying on unsupported recall. With retrieval held constant across generators, the remaining gap reflects how well each model reasons over correctly retrieved context. Claude leads on the testing split and ties or leads on harvest trend, most notably on testing-split numeric accuracy (98.0 percent versus 84.0, 86.0, and 74.7 percent) and start-year numeric accuracy (100 percent versus 72.0, 68.0, and 80.0 percent), which motivated the choice to deploy Claude in this Space despite the three local models being the lower-cost, self-hosted option. The comparison models are not uniformly weaker, however: Llama-3.1-8B-Instruct matches Claude's trend-direction accuracy exactly (80.0 percent), Qwen2.5-7B-Instruct records the best regulation-lookup citation accuracy (88.9 percent) and the best combined cross-state keyword score (50.0 percent) of the five conditions, and Phi-3.5-mini-instruct posts the best Virginia-specific cross-state keyword score (52.1 percent), so generator choice interacts with task type rather than one model dominating uniformly. Cross-state synthesis remains the weakest task for all four generators, with retrieval hit rate at 50 percent, because the query optimizer currently issues a single retrieval call per question rather than decomposing cross-state questions into one call per state. Addressing this retrieval limitation, rather than swapping generators, is the most direct path to improving performance on this task.

5. Usage and Intended Uses

HarvestIQ is intended for hunters planning a season in Virginia or Idaho who require harvest statistics, season dates, or bag limits grounded in official state documents rather than a model's general training knowledge. It is not intended as a substitute for consulting official regulations before hunting, and its scope is limited to two states, so questions about other states will not be grounded in retrieved text. The Space (app.py) runs standalone once the ANTHROPIC_API_KEY secret is configured. The snippet below demonstrates the same pipeline used directly in Python, loading the local embedding model through sentence-transformers (which wraps Hugging Face transformers) and querying the persisted ChromaDB index included in this repository.

import chromadb
from chromadb.utils import embedding_functions
import anthropic

# Local embedding model, loaded through sentence-transformers / HF transformers
embed_fn = embedding_functions.SentenceTransformerEmbeddingFunction(
    model_name="BAAI/bge-large-en-v1.5"
)

# Persisted ChromaDB index included in this repository (data/embeddings/chroma)
client = chromadb.PersistentClient(path="data/embeddings/chroma")
regulations = client.get_collection("regulations", embedding_function=embed_fn)

# Retrieve, then synthesize under citation-required prompting; see app.py for
# the full query optimizer, retrieval dispatcher, and synthesis pipeline
hits = regulations.query(query_texts=["turkey season dates Virginia"], n_results=5,
                          where={"state": "VA"})

llm = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
response = llm.messages.create(
    model="claude-sonnet-4-6", max_tokens=1024,
    system="Answer only from the provided context and cite every claim.",
    messages=[{"role": "user", "content": f"Context:\n{hits}\n\nQuestion: turkey season dates Virginia"}],
)
print(response.content[0].text)

Prompt Format

The synthesis prompt is a fixed system message with retrieved context inserted, together with rules specifying a citation format per source type and prohibiting answers outside the retrieved context. The user turn is the rewritten, entity-expanded version of the hunter's question rather than the raw text as typed.

You are HarvestIQ, a hunting intelligence assistant for Virginia and Idaho
(MVP scope - NC deferred). Help hunters plan successful hunts using state
wildlife harvest data and hunting regulations.

RULES:
1. Every factual claim MUST come from the retrieved context below. No outside facts.
2. CITATIONS ARE REQUIRED for every claim:
   - Harvest data  -> cite as  (State, Unit, Year)  e.g.  (ID Unit 32A, 2024)
   - Regulations   -> cite as  [filename, p.N]  using the exact source_file from context
   - Multiple sources for one claim -> cite all.
3. If context lacks the answer, say so - do not guess.
4. This season = 2025 unless specified.
5. Be direct and practical - hunters are planning real trips.
6. End every response with a **Sources** section listing each cited source.

RETRIEVED CONTEXT:
{context}

Expected Output Format

A response consists of a direct answer followed by a Sources section, with every factual claim carrying an inline citation in the format defined by the prompt above.

Idaho's archery elk season in Unit 32A ran October 1 through October 14 in
the most recent regulations on file, with a general tag required
(ID Unit 32A regulations, p.14).

**Sources**
- Idaho seasons and rules big game 2026, p.14

Limitations

The most significant limitation is scope: Virginia harvest statistics are not yet ingested, so the structured database and knowledge graph cover Idaho only, while Virginia support exists solely in the regulation vector store. This reflects the current state of a proof-of-concept build rather than a modeling failure. Cross-state synthesis is the weakest capability reported in the evaluation above, with retrieval hit rate at 50 percent for both generators, since the query optimizer issues a single retrieval call per question rather than one call per state when a question spans Virginia and Idaho. Regulation-lookup keyword recall and citation accuracy (61 to 78 percent depending on generator) remain well below the near-perfect testing-split figures, because regulation text is unstructured natural language matched by embedding similarity, unlike harvest facts, which are resolved by exact entity match once the query optimizer extracts a state, unit, species, and year. The harvest-report narrative vector store is currently empty, as no harvest-report PDFs have been ingested, so questions specifically about narrative population context beyond the raw numbers will not retrieve any results. Finally, both the query optimizer and the synthesis step call the Anthropic API on every turn, so the assistant requires a live API key and network connection, incurs per-message latency and cost, and cannot operate fully offline in the way the local embedding model can.