Llama-AuditLM-GGUF

Llama-AuditLM — GGUF (Q4_K_M)

A Llama 3.1-8B fine-tune for US external audit (PCAOB / SEC / GAGAS), distributed as GGUF Q4_K_M for local inference via llama.cpp or Ollama.

Built with Llama. Fine-tuned from Meta Llama 3.1-8B under the Llama 3.1 Community License (full text in LLAMA_LICENSE). The "Llama" name prefix is required by that license. Affiliation: Independent Researcher.

---

⚠️ Critical: the model alone is NOT the verified system

These are the raw fine-tuned weights. The trust properties this project is known for — verified citations, zero fabricated citations, and honest GROUNDED / PARTIAL / GENERAL-KNOWLEDGE / DEFER labeling — come from the verification layer (deterministic code in the GitHub repo), not from these weights.

If you download and run this GGUF alone (e.g. in Ollama) without the verification layer, you get the base fine-tuned model — which can produce plausible-but-unverified or outright incorrect citations, exactly like any other capable LLM. Nothing about the raw weights guarantees a cited standard exists or is the right one.

To get the trustworthy behavior, run this model with the verification layer from https://github.com/vimalnakrani08/auditlm. The model recommends; the verification layer is what makes it safe. This is recommend-and-verify, not "trust the model."

---

1. What it is

Llama-AuditLM is a Llama 3.1-8B-Instruct fine-tune (QLoRA, "run-2") specialized for US external audit — PCAOB auditing standards, SEC regulations/filings, and GAGAS (Yellow Book). It is packaged here as a single GGUF, quantized to Q4_K_M (~4.6 GB), for CPU/GPU-friendly local inference through llama.cpp or Ollama. It is designed to be the generation component of the verified recommender system, and holds skills + judgment while a comprehensive RAG corpus holds the facts.

• Base model: meta-llama/Llama-3.1-8B-Instruct
• Method: QLoRA fine-tune → fused to f16 → converted + quantized to GGUF Q4_K_M
• File: Llama-AuditLM-Q4_K_M.gguf
• Context: 8192 tokens (as evaluated); chat template: Llama 3.1; recommended sampling: greedy / temperature=0, stops <|eot_id|> (matches the reference evaluation)

2. How it was built (brief)

Base Llama 3.1-8B-Instruct + QLoRA fine-tune on audit-domain instruction data, then the adapter was fused into the base and the result converted to GGUF and quantized to Q4_K_M. The domain data and RAG corpus are built entirely from public sources — no licensed standard text:

• ~12k public passages: PCAOB Auditing Standards (AS), SEC regulations (Reg S-X / S-K) + filings + Staff Accounting Bulletins, and GAO GAGAS (Yellow Book).
• FASB ASC appears as topic stubs only (no licensed Codification prose) — this is the deliberate open-corpus GAAP boundary: GAAP disclosure-text questions are out of scope and are declined.

Coverage is honestly scoped as complete public coverage.

3. The finding (the intellectual contribution)

Citation trustworthiness decomposes into two independent properties:

The verification layer guarantees existence (every shown citation is real or is stripped and flagged — re-parse any answer and find zero ungrounded citations). It cannot guarantee correctness: the model can cite a real standard for the wrong topic, or describe a real standard inaccurately, and the layer will still show it as a real citation. Hence recommend-and-verify — the tool makes checking fast and fabrication-free; the auditor confirms the claim against the cited source.

4. How to use

(a) With the full verified system — recommended

Clone the source repo, which runs this model behind the retrieval + verification layer (RAG → generate → parse → verify → confidence label):

git clone https://github.com/vimalnakrani08/auditlm
# follow the repo README: it fetches these weights and serves the verified recommender

This is the only configuration that delivers the verified-citation / zero-fabrication / calibrated-DEFER behavior.

(b) Standalone via Ollama / llama.cpp — unverified base model

Fast to try, but remember the ⚠️ warning above: this is the raw fine-tuned model, without verification.

# Ollama (pulls the GGUF straight from this repo)
ollama run hf.co/vimalnakrani/Llama-AuditLM-GGUF

# llama.cpp
llama-cli -hf vimalnakrani/Llama-AuditLM-GGUF -p "Which PCAOB standard governs the auditor's consideration of fraud?"

Or download the file directly:

from huggingface_hub import hf_hub_download
path = hf_hub_download("vimalnakrani/Llama-AuditLM-GGUF", "Llama-AuditLM-Q4_K_M.gguf")

Use the Llama 3.1 chat template, temperature=0, and stop on <|eot_id|> to match the reference setup.

5. Benchmarks

Evaluated on AssuranceBench (held-out test split, judge-scored), measured on the verified system with the reference Ollama setup — these numbers describe the verified system, not the raw weights run alone:

• In-scope (PCAOB/SEC) citation lookups: ~0.88. Blended benchmark figure 0.71, because the out-of-scope FASB/ASC items are correctly declined rather than answered.
• Safety gate: PASS — 0 dangerous conclusions across the safety suite; every hard professional- judgment call defers (independence, fraud, legal, novel-transaction, tax).
• 0 fabricated citations shown — machine-checked: re-parsing every shown answer yields zero ungrounded citations.

6. Links — the three artifacts

• Source + verification layer: https://github.com/vimalnakrani08/auditlm
• Benchmark (AssuranceBench): https://github.com/vimalnakrani08/assurancebench
• Live demo (Space): https://huggingface.co/spaces/vimalnakrani/Llama-AuditLM

7. License & attribution — Built with Llama

Built with Llama. Llama-AuditLM is fine-tuned from Meta Llama 3.1-8B and is governed by the Llama 3.1 Community License (Copyright © Meta Platforms, Inc.). A full copy of the agreement is included in this repository as LLAMA_LICENSE, as the license requires when redistributing Llama Materials. Per the license, distributed Llama derivatives carry the "Llama" name prefix — hence Llama-AuditLM.

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Corpus/data sourcing: public, free-to-use sources only (PCAOB, SEC EDGAR + regulations + SABs, GAO GAGAS, FASB ASC topic stubs). No firm-proprietary methodology. Affiliation: Independent Researcher.

8. Not professional advice

This is a recommend-and-verify tool to help auditors find and check citations quickly against public sources. It is not a substitute for professional judgment, and existence ≠ correctness — always read the cited passages and confirm the claims yourself. Guardrail zones (independence, fraud, legal, novel-transaction accounting, specific tax positions) are answered by explaining the framework and deferring the conclusion to a qualified professional, never by concluding.