robotics-llm

Robotics-LLM

A domain-specific Large Language Model fine-tuned for robotics and无人机 (UAV) engineering — combining Retrieval-Augmented Generation (RAG), speculative decoding, and a complete Teacher-Student distillation pipeline.

---

Overview

Robotics-LLM is a Qwen3-14B model fine-tuned via RAFT (Retrieval-Augmented Fine-Tuning) distillation from a Qwen3-30B-A3B Teacher model. It is purpose-built for:

• PX4 / ArduPilot drone configuration and troubleshooting
• ROS2 development and debugging
• Computer Vision for robotics (YOLO, SAM, calibration)
• EKF / SLAM navigation stack
• Reinforcement Learning simulation (MuJoCo, Gazebo, Isaac Lab)

The system integrates a RAG Proxy Server (OpenAI-compatible API) with hybrid retrieval (BM25 + BGE semantic search) over a curated knowledge base of robotics documentation, academic papers, and engineering guides.

---

Architecture

User (Cherry Studio / Open WebUI)
        │
        ▼
┌─────────────────────────────┐
│     RAG Proxy (:5000)       │  ← OpenAI-compatible API
│  ┌───────────────────────┐  │
│  │  Hybrid Searcher      │  │  ← BM25 + BGE (CPU)
│  │  - BM25 (jieba)       │  │
│  │  - BGE-small-zh-v1.5  │  │
│  └───────────────────────┘  │
│              │              │
│              ▼              │
│  ┌───────────────────────┐  │
│  │  vLLM Engine (:8000)   │  │  ← 14B main + 1.7B draft
│  │  - tensor-parallel=2   │  │
│  │  - Speculative Decode  │  │
│  └───────────────────────┘  │
└─────────────────────────────┘

---

Training Pipeline: RAFT Distillation

Stage 1: Data Generation (Teacher → Student)

Stage 2: LoRA SFT

Stage 3: Model Merging

After SFT, the LoRA adapter is merged into the base model using merge_and_unload() to create the final inference-ready model.

---

Inference: Speculative Decoding

To accelerate generation, speculative decoding pairs the 14B main model with a Qwen3-1.7B draft model:

Performance

Measured on 2 × H100 80GB with tensor-parallel-size=2

Note: With RAG retrieval overhead (hybrid search over 1080 documents), end-to-end latency increases by approximately 1–2 seconds per query. Actual throughput varies with prompt length and batch size.

---

Retrieval-Augmented Generation

Knowledge Base

Hybrid Search Configuration

---

RAG Proxy Server

The proxy exposes an OpenAI-compatible API at /v1/chat/completions, compatible with Cherry Studio, Open WebUI, and any OpenAI SDK client.

System prompt structure:

• <think> analysis with critical literature review
• Precision technical solutions (code / commands)
• ⚠️ Safety warnings before physical operations
• 🔍 Verification methods (ros2 topic echo, jtop, etc.)

Monitored metrics (appended to each response):

• Tokens per second
• End-to-end latency (ms)
• GPU memory utilization (%)

---

Quick Start

Prerequisites

• 2 × NVIDIA H100 80GB (or equivalent)
• CUDA 13.0+, PyTorch 2.11+
• Python 3.11

Installation

git clone https://github.com/stevensun2024/robotics-llm
cd robotics-llm/rag_teacher
pip install -r requirements.txt

Start vLLM (with speculative decoding)

bash start_vllm.sh

Start RAG Proxy

bash start_proxy.sh

Connect Client

Configure your OpenAI-compatible client:

API Endpoint: http://localhost:5000/v1
API Key: sk-rag-prod
Model: rag-14b

---

File Structure

robotics-llm/rag_teacher/
├── rag_proxy_server.py     # OpenAI-compatible proxy server
├── hybrid_search.py        # BM25 + BGE hybrid retriever
├── kb_loader.py            # YAML knowledge base loader
├── load_advanced_kb.py     # Multi-format knowledge base loader
├── config.py               # Central configuration
├── run_lora_sft.py         # LoRA SFT training script
├── generate_raft_data.py   # RAFT data generation
├── generate_dpo_pairs.py   # DPO pair generation
├── clean_raft_dataset.py   # Data cleaning and deduplication
├── gpu_burn.py             # GPU memory stress test
├── start_vllm.sh           # vLLM startup script
├── start_proxy.sh          # RAG Proxy startup script
├── deploy.sh               # Deployment script
└── requirements.txt        # Python dependencies

---

License

This project is released under the Apache License 2.0. The base model Qwen3-14B follows its original license terms.

---

Citation

If you use Robotics-LLM in your research, please cite:

@misc{robotics-llm-2026,
  author = {Sun, Steven and Contributors},
  title = {Robotics-LLM: Domain-Specific Language Model for Robotics Engineering},
  year = {2026},
  publisher = {Hugging Face / ModelScope},
  url = {https://huggingface.co/stevensun2024/robotics-llm}
}