Youtu-LLM-2B-GGUF

<span style="color: #7FFF7F;">Youtu-LLM-2B GGUF Models</span>

<span style="color: #7F7FFF;">Model Generation Details</span>

This model was generated using llama.cpp at commit ced765be4.

---

<a href="https://readyforquantum.com/huggingface_gguf_selection_guide.html" style="color: #7FFF7F;"> Click here to get info on choosing the right GGUF model format </a>

---

<!--Begin Original Model Card--> <div align="center">

<img src="assets/youtu-llm-logo.png" alt="Youtu-LLM Logo" height="100px">

📃 License • 💻 Code • 📑 Technical Report • 📊 Benchmarks • 🚀 Getting Started • 💡 Highlights

</div>

🎯 Brief Introduction

Youtu-LLM is a new, small, yet powerful LLM, contains only 1.96B parameters, supports 128k long context, and has native agentic talents. On general evaluations, Youtu-LLM significantly outperforms SOTA LLMs of similar size in terms of Commonsense, STEM, Coding and Long Context capabilities; in agent-related testing, Youtu-LLM surpasses larger-sized leaders and is truly capable of completing multiple end2end agent tasks.

Youtu-LLM has the following features:

• Type: Autoregressive Causal Language Models with Dense MLA
• Release versions: Base and Instruct
• Number of Parameters: 1.96B
• Number of Layers: 32
• Number of Attention Heads (MLA): 16 for Q/K/V
• MLA Rank: 1,536 for Q, 512 for K/V
• MLA Dim: 128 for QK Nope, 64 for QK Rope, and 128 for V
• Context Length: 131,072
• Vocabulary Size: 128,256

<a id="benchmarks"></a>

📊 Performance Comparisons

Instruct Model

<img src="assets/general_agentic_instruct.png" alt="Comparison between Youtu-LLM-2B and baselines" height="260px">

General Benchmarks

Agentic Benchmarks

<a id="quickstart"></a>

🚀 Quick Start

This guide will help you quickly deploy and invoke the Youtu-LLM-2B model. This model supports "Reasoning Mode", enabling it to generate higher-quality responses through Chain of Thought (CoT).

---

1. Environment Preparation

Ensure your Python environment has the transformers library installed and that the version meets the requirements.

pip install "transformers>=4.56" torch accelerate

---

2. Core Code Example

The following example demonstrates how to load the model, enable Reasoning Mode, and use the re module to parse the "Thought Process" and the "Final Answer" from the output.

import re
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# 1. Configure Model
model_id = "tencent/Youtu-LLM-2B"

# 2. Initialize Tokenizer and Model
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True
)

# 3. Construct Dialogue Input
prompt = "Hello"
messages = [{"role": "user", "content": prompt}]

# Use apply_chat_template to construct input; set enable_thinking=True to activate Reasoning Mode
input_ids = tokenizer.apply_chat_template(
    messages, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt",
    enable_thinking=True
).to(model.device)

# 4. Generate Response
outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    do_sample=True,
    temperature=1.0,
    top_p=0.95,
    repetition_penalty=1.05
)

# 5. Parse Results
full_response = tokenizer.decode(outputs[0], skip_special_tokens=True)

def parse_reasoning(text):
    """Extract thought process within <think> tags and the subsequent answer content"""
    thought_pattern = r"<think>(.*?)</think>"
    match = re.search(thought_pattern, text, re.DOTALL)
    
    if match:
        thought = match.group(1).strip()
        answer = text.split("</think>")[-1].strip()
    else:
        thought = "(No explicit thought process generated)"
        answer = text
    return thought, answer

thought, final_answer = parse_reasoning(full_response)

print(f"\n{'='*20} Thought Process {'='*20}\n{thought}")
print(f"\n{'='*20} Final Answer {'='*20}\n{final_answer}")

---

3. Key Configuration Details

Reasoning Mode Toggle

Controlled via the enable_thinking parameter in the apply_chat_template method:

• True (Recommended Default): Activates Chain of Thought; ideal for complex logic and reasoning tasks.
• False: Outputs results directly; faster response time, suitable for simple conversations.

Recommended Decoding Parameters

Depending on your use case, we suggest adjusting the following hyperparameters for optimal generation:

Tip: When using Reasoning Mode, a higher temperature helps the model perform deeper, more divergent thinking.

---

4. vLLM Deployment

We provide support for deploying the model using vLLM 0.10.2. The recommended Docker image is vllm/vllm-openai:v0.10.2.

Integration Steps

First, execute the following commands to integrate the Youtu-LLM model files into the vLLM framework. Note: Please extract our provided modified vllm zip file first. Then, replace <local_modified_vllm_path> with the path to the extracted vllm directory, and replace <vllm_path> with the installation path of vLLM.

cp <local_modified_vllm_path>/0_10_2_official/youtu_llm.py <vllm_path>/vllm/model_executor/models/youtu_llm.py
cp <local_modified_vllm_path>/0_10_2_official/configuration_youtu.py <vllm_path>/vllm/model_executor/models/configuration_youtu.py
cp <local_modified_vllm_path>/0_10_2_official/__init__.py <vllm_path>/vllm/config/__init__.py
cp <local_modified_vllm_path>/0_10_2_official/registry.py <vllm_path>/vllm/model_executor/models/registry.py

Service Startup

Once integrated, you can deploy the model using the following command:

vllm serve <model_path> --trust-remote-code

Tool Call Support: To enable tool calling capabilities, please append the following arguments to the startup command:

--enable-auto-tool-choice --tool-call-parser hermes

---

<a id="highlights"></a>

💡 Highlights

Showcase of General Tasks

Showcase of Specialized Tasks

Note: For specialized tasks, in-domain post-training is further applied.

<!--End Original Model Card-->

---

<span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>

Help me test my AI-Powered Quantum Network Monitor Assistant with quantum-ready security checks:

👉 Quantum Network Monitor

The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder

💬 How to test:
Choose an AI assistant type:

• TurboLLM (GPT-4.1-mini)
• HugLLM (Hugginface Open-source models)
• TestLLM (Experimental CPU-only)

What I’m Testing

I’m pushing the limits of small open-source models for AI network monitoring, specifically:

• Function calling against live network services
• How small can a model go while still handling:
  • Automated Nmap security scans
  • Quantum-readiness checks
  • Network Monitoring tasks

🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):

• ✅ Zero-configuration setup
• ⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
• 🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!

Other Assistants

🟢 TurboLLM – Uses gpt-4.1-mini :

• **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
• Create custom cmd processors to run .net code on Quantum Network Monitor Agents
• Real-time network diagnostics and monitoring
• Security Audits
• Penetration testing (Nmap/Metasploit)

🔵 HugLLM – Latest Open-source models:

• 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.

💡 Example commands you could test:

1. "Give me info on my websites SSL certificate"
2. "Check if my server is using quantum safe encyption for communication"
3. "Run a comprehensive security audit on my server"
4. '"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code on. This is a very flexible and powerful feature. Use with caution!

Final Word

I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.

If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.

I'm also open to job opportunities or sponsorship.

Thank you! 😊