CyberGong-Qwen2.5-1.5B-1.0

Personal Style Language Model Inference and Management System

A local-runnable personal style language model inference project based on Qwen2.5-1.5B and LoRA adapter.

Objectives

• ✅ Support loading base model from local directory
• ✅ Support 4-bit quantization loading (RTX 3060 6GB optimized)
• ✅ Support LoRA adapter mounting and inference
• ✅ Support model merging and independent export
• ✅ Support interactive chat and single-turn inference
• ✅ Fully usable in offline mode

Project Structure

CyberGong/
├── configs/
│   └── infer.yaml              # Inference configuration file
├── models/
│   ├── Qwen2.5-1.5B/           # Base model directory
│   ├── your-persona-lora/      # LoRA adapter directory
│   └── merged-persona-model/   # Merged model
├── src/
│   ├── download_model.py       # Model download script
│   ├── load_model.py           # Model loading script
│   ├── chat.py                 # Chat/inference script
│   ├── merge_lora.py           # LoRA merging script
│   └── utils.py                # Utility functions
├── data/
│   └── prompts.jsonl           # Test prompts
└── README.md                   # This file

Install Dependencies

pip install transformers peft bitsandbytes torch huggingface-hub pyyaml

Key Libraries Description

• transformers: Provides model loading and generation interfaces
• peft: Provides LoRA adapter support
• bitsandbytes: Provides 4-bit quantization support
• torch: PyTorch deep learning framework
• huggingface-hub: HuggingFace model download tool

Quick Start

1. Download Model

python src/download_model.py \
    --model-id Qwen/Qwen2.5-1.5B \
    --output-dir ./models/Qwen2.5-1.5B

Options:

• --model-id: HuggingFace model ID (default: Qwen/Qwen2.5-1.5B)
• --output-dir: Local output directory
• --cache-dir: Optional cache directory
• --max-workers: Concurrent download workers (default: 4)
• --use-cli: Use huggingface-cli to download

2. Load and Inference (Base Model Only)

python src/chat.py --model-dir ./models/Qwen2.5-1.5B

Options:

• --model-dir: Model directory
• --no-4bit: Disable 4-bit quantization
• --device-map: Device allocation (default: auto)

3. Load and Inference (Base + LoRA Adapter)

First prepare LoRA adapter directory, which should contain:

• adapter_config.json
• adapter_model.safetensors

python src/chat.py \
    --model-dir ./models/Qwen2.5-1.5B \
    --adapter ./models/your-persona-lora

4. Merge Models

Merge adapter with base model into a standalone model:

python src/merge_lora.py \
    --model-dir ./models/Qwen2.5-1.5B \
    --adapter-dir ./models/your-persona-lora \
    --output-dir ./models/merged-persona-model

After merging, it can be directly loaded and used:

python src/chat.py --model-dir ./models/merged-persona-model

Usage

Interactive Chat

python src/chat.py --model-dir ./models/Qwen2.5-1.5B

Commands:

• Type any text to get a reply
• exit: Exit chat
• clear: Clear chat history
• save: Save conversation to conversation.json

Single-turn Inference

python src/chat.py \
    --model-dir ./models/Qwen2.5-1.5B \
    --prompt "Hello, please introduce yourself"

Using Configuration File

Edit configs/infer.yaml to configure parameters:

model:
  base_model_dir: ./models/Qwen2.5-1.5B
  adapter_dir: ./models/your-persona-lora

generation:
  max_new_tokens: 256
  temperature: 0.7
  top_p: 0.9

chat:
  system_prompt: "You are a helpful assistant."
  max_history_length: 10

Then run:

python src/chat.py --config ./configs/infer.yaml

Core Features Explanation

Model Loading (load_model.py)

Load Base Model:

from load_model import load_base_model

model, tokenizer = load_base_model(
    model_dir="./models/Qwen2.5-1.5B",
    use_4bit=True,
    device_map="auto"
)

Load LoRA Adapter:

from load_model import load_persona_adapter

model_with_adapter = load_persona_adapter(
    base_model=model,
    adapter_dir="./models/your-persona-lora"
)

Chat Management (chat.py)

from chat import PersonaChat
from load_model import load_base_model

model, tokenizer = load_base_model("./models/Qwen2.5-1.5B")
chat = PersonaChat(model, tokenizer)

# Single-turn conversation
response = chat.generate("Hello")

# Get history
history = chat.get_history()

# Save conversation
chat.save_conversation("output.json")

Model Merging (merge_lora.py)

from merge_lora import merge_and_unload

success = merge_and_unload(
    base_model_dir="./models/Qwen2.5-1.5B",
    adapter_dir="./models/your-persona-lora",
    output_dir="./models/merged-persona-model"
)

Configuration Parameters Guide

Quantization Configuration

quantization:
  use_4bit: true              # Enable 4-bit quantization
  compute_dtype: bfloat16     # Compute precision
  quant_type: nf4             # Quantization type

4-bit Quantization Benefits:

• VRAM usage reduced from 3GB to 1-2GB
• Inference speed essentially lossless
• Supports RTX 3060 6GB

Generation Parameters

generation:
  max_new_tokens: 256         # Maximum tokens to generate
  temperature: 0.7            # Temperature (higher = more random)
  top_p: 0.9                  # Top-P sampling
  top_k: 50                   # Top-K sampling
  repetition_penalty: 1.1     # Repetition penalty
  do_sample: true             # Whether to use sampling
  num_beams: 1                # Beam search (1 = disabled)

Troubleshooting

Out of VRAM

Problem: torch.cuda.OutOfMemoryError

Solution:

1. Enable 4-bit quantization: Don't use --no-4bit
2. Reduce max_new_tokens
3. Reduce max_history_length

Missing Model Files

Problem: FileNotFoundError: Model directory does not exist

Solution:

1. Make sure model is downloaded: python src/download_model.py
2. Check if path is correct
3. Check log file model_inference.log

Adapter Loading Failed

Problem: Invalid adapter directory

Solution:

1. Check if adapter_config.json exists
2. Check if adapter_model.safetensors exists
3. Check if adapter matches base model

Performance Metrics (RTX 3060 6GB)

Logging

All operations are logged to model_inference.log:

2024-01-15 10:30:45,123 - __main__ - INFO - Start loading base model
2024-01-15 10:30:46,456 - load_model - INFO - ✓ Tokenizer loaded successfully
...

Offline Usage

All operations support offline mode, provided model files are completely downloaded:

1. Model fully downloaded via download_model.py
2. Use device_map="cpu" or "auto" instead of "cuda:0"
3. Transformers will automatically use local cache

Recommended Workflow

Development Environment

# 1. Download model (first time, requires network)
python src/download_model.py

# 2. Test base model
python src/chat.py --prompt "test"

# 3. Develop adapter
# Use external tools to train adapter...

# 4. Test adapter
python src/chat.py --adapter ./models/your-persona-lora

# 5. Merge model
python src/merge_lora.py

Production Environment

# Directly load merged model
python src/chat.py --model-dir ./models/merged-persona-model

Extension Suggestions

Add Gradio Interface

pip install gradio

# Create src/gradio_app.py

Add API Service

pip install fastapi uvicorn

# Create src/api_server.py

Model Quantization Optimization

• Use GPTQ or AWQ quantization
• Use int8 or int4 precision
• Model pruning

FAQ

Q: Can I use other models?
A: Yes, modify hf_model_id in config.yaml. Just ensure it's compatible with AutoModelForCausalLM.

Q: How to support multiple adapters?
A: PeftModel can load multiple adapters simultaneously, see peft official documentation.

Q: Can I continue loading adapters after merging?
A: The merged model is a complete standalone model and can be used as a new base model to load other adapters.

Q: How to optimize inference speed?
A:

• Use num_beams=1 (disable beam search)
• Use smaller max_new_tokens
• Enable 4-bit quantization
• Use device_map="auto" for automatic allocation

References

• Transformers Official Documentation
• PEFT LoRA Documentation
• Bitsandbytes Quantization
• Qwen2.5 Model Card

License

MIT License

Changelog

v1.0.0 (2024-01-15)

• ✅ Initial release
• ✅ Support 4-bit quantization
• ✅ Support LoRA adapter
• ✅ Support model merging
• ✅ Support interactive chat
• ✅ Complete error handling and logging

---

Feedback and suggestions are welcome! <img width="1024" height="1536" alt="2c78bf7267ec02e21286fa5a1a4d5078" src="https://github.com/user-attachments/assets/7ff31cf2-6ee8-4a37-9896-44bb9b6cba9b" />