vlm-lora-agentic-rag

🎯 VLM + LoRA + Agentic RAG: Enterprise Document Structuring

A production-ready Vision Language Model with LoRA adaptation and Agentic Retrieval-Augmented Generation for intelligent document analysis and structured extraction.

🏆 Architecture Overview

This model combines three complementary technologies:

1️⃣ Vision Language Model (VLM)

• Base: LLaVA-1.5-7B
• Capability: Multi-modal document understanding
• Purpose: Understand document layout, tables, charts, and visual elements
• Quantization: 4-bit (bitsandbytes) for T4 GPU efficiency

2️⃣ LoRA Adaptation Layer

• Method: Low-Rank Adaptation with r=64, alpha=128
• Trainable Parameters: 0.1% of base model (lightweight)
• Task: Fine-tuned for structured output accuracy
• Efficiency: 8GB VRAM (T4 compatible)

3️⃣ Agentic RAG Engine

• Strategy: Multi-strategy document retrieval
  • Keyword-based search
  • Semantic similarity search
  • Hybrid search with result verification
• Purpose: Intelligent answer verification and re-search if needed
• Embeddings: all-MiniLM-L6-v2 (sentence transformers)
• Indexing: FAISS (vector search)

---

📊 Technical Details

Model Specifications

Training Configuration

• Objective: Structured output accuracy (JSON/YAML/XML)
• Task: Extract business documents → Structured JSON
• Loss: Task-specific loss on final assistant output
• Chain-of-Thought: Preserved (intermediate reasoning masked during loss)

---

🎯 Use Cases

Financial Document Analysis

• Quarterly reports → Structured financials
• Income statements → Parsed key metrics
• Cash flow statements → Quantified outputs

Technical Documentation

• System architecture docs → Structured schemas
• API specs → Parsed endpoints
• Requirements → Extracted constraints

Business Intelligence

• Market analysis → Competitive summaries
• Risk assessments → Structured risk factors
• Strategic plans → Actionable metrics

---

📦 Installation & Usage

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model_id = "liuhaotian/llava-v1.5-7b"
adapter_id = "Shion1124/vlm-lora-agentic-rag"

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    load_in_4bit=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

Document Analysis Pipeline

from PIL import Image
import json

def analyze_document(image_path: str) -> dict:
    """Analyze document and extract structured JSON"""
    
    image = Image.open(image_path).convert("RGB")
    prompt = """Analyze this document and output ONLY valid JSON:
    {
      "title": "...",
      "summary": "...",
      "key_data": [...],
      "insights": "..."
    }"""
    
    # Process with VLM + LoRA
    # ... implementation ...
    
    return structured_json

Agentic RAG Search

from sentence_transformers import SentenceTransformer
import faiss

# Build document index
embedder = SentenceTransformer('all-MiniLM-L6-v2')
index = faiss.IndexFlatL2(vector_dim)
index.add(embeddings)

# Multi-strategy search with verification
def agentic_search(query: str, max_iterations: int = 3):
    for iteration in range(max_iterations):
        if iteration == 1:
            results = keyword_search(query)
        elif iteration == 2:
            results = semantic_search(query)
        else:
            results = hybrid_search(query)
        
        if verify_results(results):
            break
    
    return results

---

📈 Performance Metrics

---

🔧 Advanced Configuration

LoRA Parameters

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=64,                              # Rank
    lora_alpha=128,                    # Scaling
    target_modules=["q_proj", "v_proj"],  # Attention heads
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Agentic RAG Strategies

• Keyword Search: Fast, lexical matching
• Semantic Search: Slow, semantic matching
• Hybrid Search: Combined approach
• Verification Loop: Confidence-based re-search

---

📚 Training Details

Dataset Sources

• Structured output datasets (JSON/YAML/XML)
• Financial documents with labeled extractions
• Technical documentation with parsed schemas
• Business reports with key metric annotations

Training Objective

Maximize accuracy of structured extraction while preserving:

• Chain-of-thought reasoning
• Intermediate explanation quality
• Hallucination prevention through Agentic verification

---

⚠️ Limitations & Future Work

Current Limitations

• Requires labeled training data for domain adaptation
• OCR quality dependent on PDF preprocessing
• Multi-page document coherence needs improvement
• Multilingual support under development

Roadmap

• Multilingual variant (EN, JA, ZH)
• Larger model variant (13B, 70B)
• Fine-grained attribute extraction
• Real-time streaming inference
• Knowledge graph generation

---

📄 License

Apache 2.0

---

🙏 Acknowledgments

• LLaVA: Vision Language Model foundation
• PEFT: LoRA implementation
• Sentence Transformers: Embedding models
• FAISS: Vector search infrastructure

---

📧 Contact & Support

For questions, issues, or collaborations:

• GitHub: https://github.com/Shion1124/vlm-lora-agentic-rag
• HuggingFace: Shion1124/vlm-lora-agentic-rag

---

Updated: 2026-03-20
Status: Production-Ready