local-ai-toolkit

<p align="center"> <img src="https://img.shields.io/badge/README-AI_Generated-FF6B6B?style=flat-square&logo=openai&logoColor=white" alt="AI Generated"> </p>

⚠️ Note: This README was generated and is maintained by AI.

<div align="center">

🧠 Local AI Toolkit

A Curated Collection of Locally-Runnable AI Models

Optimized for RTX 3060 12GB VRAM + 32GB RAM

<p> <img src="https://img.shields.io/badge/GPU-RTX_3060_12GB-76B900?style=for-the-badge&logo=nvidia&logoColor=white" alt="GPU"> <img src="https://img.shields.io/badge/RAM-32GB_DDR4-4A90D9?style=for-the-badge&logo=memory&logoColor=white" alt="RAM"> <img src="https://img.shields.io/badge/Models-37+-FF6B6B?style=for-the-badge&logo=huggingface&logoColor=white" alt="Models"> <img src="https://img.shields.io/badge/Categories-8-F59E0B?style=for-the-badge&logo=databricks&logoColor=white" alt="Categories"> <img src="https://img.shields.io/badge/Format-GGUF_&_SafeTensors-8B5CF6?style=for-the-badge&logo=archive&logoColor=white" alt="Format"> </p> <p> <img src="https://img.shields.io/badge/🤖 LLMs-✅-success" alt="LLMs"> <img src="https://img.shields.io/badge/💻 Coding-✅-success" alt="Coding"> <img src="https://img.shields.io/badge/🔗 RAG-✅-success" alt="RAG"> <img src="https://img.shields.io/badge/🎙️ TTS-✅-success" alt="TTS"> <img src="https://img.shields.io/badge/🎤 STT-✅-success" alt="STT"> <img src="https://img.shields.io/badge/👁️ Multimodal-✅-success" alt="Multimodal"> <img src="https://img.shields.io/badge/🔗 Agents-✅-success" alt="Agents"> <img src="https://img.shields.io/badge/🖼️ Image_Gen-🔜-yellow" alt="Image Gen"> <img src="https://img.shields.io/badge/🎬 Video_Gen-🔜-yellow" alt="Video Gen"> </p>

<i>AI-assisted model selection · Quantized for consumer hardware · A suggested starting point</i>

</div>

---

📖 About This Toolkit

This repository is a suggested toolkit of open-source AI models assembled to run entirely locally on consumer-grade hardware — specifically an NVIDIA RTX 3060 (12GB VRAM) with 32GB system RAM. Each model was chosen to fit within these constraints while offering strong output quality for the VRAM budget.

The selection process was AI-assisted — I consulted several AI assistants to weigh benchmarks, quantization trade-offs, and model capabilities, then settled on the common recommendations you see here. This is not a definitively optimal combination: better models may already exist, and stronger ones are released regularly. Treat it as a well-reasoned starting point rather than a final answer — your own testing on your own hardware is the real benchmark.

🔄 Updates & Longevity

This toolkit is periodically refreshed as stronger or more efficient models are released. When a model gets clearly outclassed, it gets swapped out — so the lineup you see today may differ from a few months down the line.

🎯 Design Philosophy

Each category follows a dual-tier architecture. Both tiers target the same 12GB VRAM + 32GB RAM machine — the difference is how that budget is used:

Note on Coding models: The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants — they simply trade some reasoning depth for significantly faster token generation.

⚠️ Work In Progress

This toolkit is not yet complete. Notably, Image Generation and Video Generation models are still missing and will be added in future updates. The current collection focuses on text, speech, and multimodal understanding capabilities. Stay tuned for expansions.

---

📂 Repository Structure & Category Guide

local-ai-toolkit/
├── 1-Universal_Foundation_LLMs/      🧠 General-purpose language models
├── 2-Coding_Agents/                  💻 Specialized code generation & debugging
├── 3-Agent_Orchestration_and_Routing/ 🔗 Multi-agent system coordination
├── 4-Document_Analysis_and_RAG/      📄 Embeddings, rerankers, and retrieval
├── 5-TTS/                            🎙️ Text-to-speech synthesis
├── 6-STT/                            🎤 Speech-to-text recognition
├── 7-Multimodal/                     👁️ Vision + language understanding
└── 8-Uncensored_Models/              🔓 Unfiltered model variants

---

1️⃣ Universal Foundation LLMs

<div align="center"> <i>The backbone of any local AI setup — general-purpose models for conversation, analysis, writing, and reasoning.</i> </div>

These are your go-to models for everyday tasks: answering questions, drafting emails, summarizing documents, brainstorming ideas, creative writing, and general knowledge retrieval. They serve as the foundation upon which more specialized workflows are built.

Why these models? The Qwen 3.x series represents a significant leap in efficient architecture. The 35B-A3B MoE variant is particularly special — it gives you the reasoning depth of a much larger model while only activating 3 billion parameters at a time, making it one of the most VRAM-efficient high-quality models available. The 9B speed variant offers near-instant responses for situations where latency matters more than depth.

<details> <summary>📁 Folder Structure</summary>

1-Universal_Foundation_LLMs/
├── Quality/
│   └── Qwen3.6-35B-A3B-UD-Q4_K_XL/
│       ├── Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
│       └── mmproj-BF16.gguf
└── Speed/
    └── Qwen3.5-9B-Q5_K_M/
        ├── Qwen3.5-9B-Q5_K_M.gguf
        └── mmproj-BF16.gguf

</details>

---

2️⃣ Coding Agents

<div align="center"> <i>Specialized models for writing, debugging, and architecting code — the core of any AI-assisted development workflow.</i> </div>

This category contains models specifically fine-tuned for software engineering tasks. They are organized into five distinct roles that mirror a professional development team structure, from high-level architecture down to real-time inline suggestions.

Why this structure? Modern AI-assisted coding works best when you use specialized models for specialized tasks. The Qwen3-Coder excels at writing new code and understanding project structure, while the DeepSeek-R1 distill variant uses its chain-of-thought training to methodically trace through bugs. The 7B autocomplete model is lightweight enough to run alongside your IDE for real-time suggestions without competing for VRAM with your main coding model.

Pro tip: Use the "speed" variants for quick prototyping and test-driven development cycles where you need rapid iteration. They're still high-quality coders — they just generate tokens faster and use less memory, leaving headroom for other tools in your pipeline.

<details> <summary>📁 Folder Structure</summary>

2-Coding_Agents/
├── 1-Architect_and_Executor/
│   └── Qwen3-Coder-30B-A3B-Instruct-Q5_K_M/
│       └── Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf
├── 2-Detective_and_Debugger/
│   └── DeepSeek-R1-Distill-Qwen-32B-Q4_K_M/
│       └── DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf
├── 3-Inline_Autocomplete/
│   └── qwen2.5-coder-7b-instruct-q5_k_m/
│       └── qwen2.5-coder-7b-instruct-q5_k_m.gguf
├── 4-Architect_and_Executor_speed/
│   └── Qwen2.5_Coder_14B_Instruct_Q4_K_M/
│       └── Qwen2.5_Coder_14B_Instruct_Q4_K_M.gguf
└── 5-Detective_and_Debugger_speed/
    └── DeepSeek_R1_Distill_Qwen_14B_Q4_K_M/
        └── DeepSeek_R1_Distill_Qwen_14B_Q4_K_M.gguf

</details>

---

3️⃣ Agent Orchestration and Routing

<div align="center"> <i>The nervous system of multi-agent AI systems — models that coordinate, route, guard, and execute complex agent workflows.</i> </div>

When you move beyond single-model interactions into multi-agent architectures (where different specialized models collaborate on complex tasks), you need orchestrators that can plan, delegate, monitor, and validate. This category provides the complete infrastructure for building sophisticated agentic systems locally.

🎯 Orchestrators & Managers

The "brains" of your agent network. These models decide which worker to call, when to call it, and how to synthesize results.

⚙️ Agentic Workers

Models optimized to be called by orchestrators — they receive specific tasks, execute them, and return structured results.

🛡️ Guardrails & Safety

Models that monitor and filter agent inputs/outputs to prevent harmful, off-topic, or undesired behavior.

<details> <summary>📁 Folder Structure</summary>

3-Agent_Orchestration_and_Routing/
├── 1-Orchestrators_and_Managers/
│   ├── Agents_A1_35B_MoE_Calibrated_CEO/
│   │   └── Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated.gguf
│   ├── GLM-4.7-Flash-UD-Q4_K_XL/
│   │   └── GLM-4.7-Flash-UD-Q4_K_XL.gguf
│   └── Hermes_4_3_36B_Quality_Orchestrator/
│       └── hermes-4_3_36b-Q4_K_M.gguf
├── 2-Agentic_Workers/
│   └── Gemma4_12B_Agentic_Worker_v2/
│       └── Gemma4_12B_Agentic_Worker_v2.gguf
└── 3-Guardrails_and_Safety/
    ├── Hermes_4_14B_Speed_Guardrail/
    │   └── Hermes-4-14B-Q4_K_M.gguf
    └── Llama_Guard_3_8B_i1_Firewall/
        └── Llama-Guard-3-8B.i1-Q4_K_M.gguf

</details>

---

4️⃣ Document Analysis and RAG

<div align="center"> <i>The complete pipeline for building local Retrieval-Augmented Generation systems — from embedding to generation.</i> </div>

RAG (Retrieval-Augmented Generation) is how you give your AI access to your own documents, knowledge bases, and data. This category provides every component needed to build a production-quality local RAG pipeline — no cloud APIs required.

Pipeline Overview

📄 Documents → 🔤 Embedding → 📊 Vector Store → 🔄 Reranking → 🧠 LLM Generation
                  BGE-M3           (your DB)        BGE-Reranker     Gemma 4 12B
                                                              ↕
                                              🕸️ GraphRAG Extractor (NuExtract)
                                                              ↕
                                              🌐 Web Distiller (Jina Reader)

🔤 1. Embeddings — BGE-M3

The foundation of any RAG system. BGE-M3 is a state-of-the-art multilingual embedding model that supports three retrieval modes simultaneously:

• Dense retrieval — traditional semantic similarity search
• Sparse retrieval — BM25-style keyword matching (great for exact term matches)
• ColBERT retrieval — late interaction for fine-grained token-level relevance scoring

This multi-modal retrieval approach means you combine semantic understanding and keyword precision in a single model. It supports 100+ languages out of the box.

🔄 2. Reranker — BGE-Reranker-V2-M3

After initial retrieval returns candidate documents, the reranker re-scores and re-orders them for maximum relevance. This two-stage approach (retrieve broadly, then rerank precisely) dramatically improves RAG accuracy compared to single-stage retrieval. The M3 variant is specifically designed to complement BGE-M3 embeddings.

🕸️ 3. GraphRAG Extractor — NuExtract v1.5

Takes unstructured text and extracts structured knowledge graphs — entities, relationships, and attributes. This enables GraphRAG pipelines that can reason over the connections between facts, not just individual document chunks. Particularly powerful for research, legal, and technical domains where relationships matter as much as the facts themselves.

🌐 4. Web Distiller — Jina Reader LM 1.5B

A tiny but remarkably effective model that cleans and structures raw web content into clean, readable text suitable for embedding. Instead of embedding HTML noise, navigation bars, and ads, this model extracts the actual article content. At only 1.5B parameters, it runs almost instantly.

🧠 5. LLM Generator — Gemma 4 12B

The final stage of the RAG pipeline — a vision-capable LLM that synthesizes retrieved context into coherent, accurate answers. Includes speculative decoding via MTP (Multi-Token Prediction) for faster generation, and a multimodal projector for handling documents with images and diagrams.

<details> <summary>📁 Folder Structure</summary>

4-Document_Analysis_and_RAG/
├── 1-Embeddings/
│   └── BGE_M3_Core_Embedding/
│       ├── config.json, tokenizer files, model weights...
│       └── colbert_linear.pt, sparse_linear.pt
├── 2-Rerankers/
│   └── BGE_Reranker_V2_M3_Core/
│       ├── config.json, model.safetensors
│       └── tokenizer files
├── 3-GraphRAG_Extractors/
│   └── NuExtract_1_5_GraphRAG_Extractor/
│       └── NuExtract-v1.5-Q8_0.gguf
├── 4-Web_Distillers/
│   └── Jina_Reader_LM_1_5B_Web_Distiller/
│       └── reader-lm-1.5b-Q8_0.gguf
└── 5-LLM_Generators/
    └── gemma-4-12b-it-UD-Q4_K_XL/
        ├── gemma-4-12b-it-UD-Q4_K_XL.gguf
        ├── mmproj-BF16.gguf
        └── MTP/
            └── mtp-gemma-4-12b-it-Q8_0.gguf

</details>

---

5️⃣ Text-to-Speech (TTS)

<div align="center"> <i>Convert text to natural-sounding speech — from multi-speaker dialogue to voice cloning and custom voice design.</i> </div>

This category covers four distinct TTS paradigms, each suited to different use cases. Whether you need a single consistent narrator, a full cast of characters, or the ability to clone any voice from a short sample, there's a model here for it.

Why so many TTS models? Different tasks demand different approaches. Voice Design is perfect for creating brand voices or fictional characters. Zero-shot cloning excels when you need to match an existing voice. Multi-speaker Dia handles scripts with dialogue natively. And the Persian-specific models ensure high-quality Farsi output that general multilingual TTS models often struggle with.

<details> <summary>📁 Folder Structure</summary>

5-TTS/
├── MULTI_SPEAKER/
│   └── Dia2_2B_Dialogue/
├── TTS_PERSIAN/
│   ├── Chatterbox_TTS_Persian/
│   │   └── t3_fa.safetensors
│   └── F5_TTS_Persian/
├── VOICE_DESIGN/
│   └── Qwen3_TTS_12Hz_VoiceDesign/
│       ├── model.safetensors
│       └── speech_tokenizer/
└── ZERO_SHOT_CLONING/
    └── F5_TTS_v1_Base/
        └── F5TTS_v1_Base/

</details>

---

6️⃣ Speech-to-Text (STT)

<div align="center"> <i>Transcribe, translate, and preprocess audio — from studio-quality multi-language recognition to specialized Persian ASR.</i> </div>

A complete speech understanding pipeline that starts with audio preprocessing (cleaning and segmentation) and ends with either transcription or translation. Like other categories, it follows the Quality vs. Speed dual-tier design.

🔧 Audio Preprocessing Filters

🏆 Quality Tier — High Accuracy

⚡ Speed Tier — Fastest Processing

<details> <summary>📁 Folder Structure</summary>

6-STT/
├── Audio_Preprocessing_Filters/
│   ├── Studio_Vocal_Isolation_DemucsV4/
│   │   └── Studio_Vocal_Isolation_DemucsV4.onnx
│   └── Universal_Silence_Noise_Cutter_Silero/
│       └── Universal_Silence_Noise_Cutter_Silero.onnx
├── Quality/
│   ├── Multi_Language/
│   │   └── Whisper_Large_V3_Global_FP16/
│   ├── Persian_ASR/
│   │   ├── Anti_Hallucination_FastConformer_Fa/
│   │   │   └── stt_fa_fastconformer_hybrid_large.nemo
│   │   └── Whisper_Persian_V4_FP16/
│   └── Translation/
│       └── Seamless_M4T_V2_Large_Translation/
└── Speed/
    ├── Multi_Language/
    │   └── Whisper_Turbo_Global_Speed/
    └── Persian_ASR/
        └── Whisper_Turbo_Persian_Speed/

</details>

---

7️⃣ Multimodal

<div align="center"> <i>Models that see and hear — process images, audio, and text together for rich understanding.</i> </div>

Multimodal models are the frontier of local AI — they can analyze images, describe screenshots, read documents with visual layouts, and process audio alongside text. This enables workflows like "take a photo of this whiteboard and summarize it" or "analyze this chart and explain the trends."

⚠️ CRITICAL — Read before relying on "multimodal": The models shipped here are GGUF quantized files. In their current state, tooling only supports the vision (image) modality from these quantized files — audio input does not work on the quantized versions.

True text + image + audio multimodality is currently only available from the original, uncompressed (FP16/BF16) models — not from the quantized GGUF files in this toolkit. The quantized variants effectively behave as vision-language models (VLM): they can see images and read text, but they cannot process audio.

This is a tooling limitation, not a model defect — current inference engines (llama.cpp, Ollama, LM Studio, etc.) cannot yet route audio through quantized GGUF multimodal pipelines. If you need genuine audio understanding, download and run the original uncompressed weights from the source repos linked below.

I learned this the hard way after downloading — the quantized files simply won't run as full multimodal. Hence this warning.

Why Qwen3-Omni for quality? It's one of the very few models that genuinely handles text + image + audio in a unified architecture. Most "multimodal" models only handle text and images — Qwen3-Omni adds audio understanding, making it a true all-in-one model for multimedia tasks.

<details> <summary>📁 Folder Structure</summary>

7-Multimodal/
├── Quality/
│   └── Qwen3_Omni_30B_A3B_Quality/
│       ├── Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf
│       └── mmproj-Qwen3-Omni-30B-A3B-Instruct-Q8_0.gguf
└── Speed/
    └── Gemma4_E4B_Speed_Speculative/
        ├── gemma-4-E4B-it-UD-Q4_K_XL.gguf
        ├── mmproj-F16.gguf
        └── MTP/
            └── gemma-4-E4B-it-F16-MTP.gguf

</details>

---

8️⃣ Uncensored Models

<div align="center"> <i>Unfiltered variants for research, creative writing, and use cases requiring unrestricted model outputs.</i> </div>

These are abliterated (uncensored) versions of the models found in other categories. They have had their refusal mechanisms and output filters removed through fine-tuning. They produce the same quality outputs as their censored counterparts but without refusing requests.

⚠️ Disclaimer: These models are provided for legitimate research, creative, and educational purposes. Users are responsible for ensuring their use complies with applicable laws and ethical guidelines.

<details> <summary>📁 Folder Structure</summary>

8-Uncensored_Models/
├── Quality/
│   ├── Code/
│   │   └── Huihui_Qwen3_VL_30B_Uncensored_Q4_K_M/
│   │       └── Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated.Q4_K_M.gguf
│   └── General/
│       └── Huihui_Qwen3.6_35B_Claude4.7_Opus_MTP_Uncensored_Q4_K/
│           ├── Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-ggml-model-Q4_K.gguf
│           └── mmproj-model-f16.gguf
└── Speed/
    ├── Code/
    │   └── Huihui_Qwen3_Coder_30B_Uncensored_i1_Q3_K_M/
    │       └── Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated.i1-Q3_K_M.gguf
    └── General/
        └── Huihui_Qwen3_VL_8B_Uncensored_Q8_0/
            ├── Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf
            └── Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf

</details>

---

🔗 Model Source Links

All models in this toolkit are sourced from Hugging Face. During the initial download process, direct URLs were not recorded. The original sources were recovered by matching SHA-256 hashes against Hugging Face repositories. While some links may point to slightly different file versions within the same quantization family, the hash-verified content is identical to what is hosted in this toolkit.

1 — Universal Foundation LLMs

2 — Coding Agents

3 — Agent Orchestration and Routing

4 — Document Analysis and RAG

5 — Text-to-Speech

6 — Speech-to-Text

7 — Multimodal

8 — Uncensored Models

---

🛠️ Recommended Tools for Running These Models

---

📋 Hardware Requirements

💡 MoE Advantage: Many models in this toolkit use Mixture-of-Experts (MoE) architecture, which activates only a fraction of total parameters during inference. This means you get the quality of much larger models while staying within 12GB VRAM constraints.

---

<div align="center">

⭐ Acknowledgments

All models in this toolkit are the work of their respective creators and are used in accordance with their original licenses. Special thanks to:

Model Families: Qwen · DeepSeek · Gemma · Hermes/NousResearch · Whisper/OpenAI · Llama/Meta · GLM · BGE/BAAI

Quantization: unsloth · bartowski · mradermacher · ggml-org · Chungulus

Uncensored Variants: huihui-ai · mradermacher

---

<i>Assembled with AI-assisted model selection · Curated for RTX 3060 12GB · Updated as stronger models arrive</i>

</div>