qwen-alpaca-gguf

Qwen Alpaca GGUF

Model: ciphermosaic/qwen-alpaca-gguf

A fine tuned version of Qwen2.5-0.5B-Instruct trained on the tatsu-lab/alpaca instruction dataset using Unsloth. This project was built as a learning exercise to understand the complete LLM fine tuning workflow, from training and LoRA adaptation to model merging, GGUF conversion, local inference with Ollama, and deployment on Hugging Face.

Model Details

• Model Name: ciphermosaic/qwen-alpaca-gguf
• Base Model: Qwen2.5-0.5B-Instruct
• Model Type: Causal Language Model
• Primary Purpose: Chat Assistant
• Language: English
• License: Apache 2.0

Training Dataset

The model was fine tuned using the tatsu-lab/alpaca dataset, which consists of instruction, input, and output pairs designed for instruction following tasks.

The objective was to improve the model's ability to follow natural language instructions and generate coherent conversational responses.

Fine Tuning

The model was fine tuned using Unsloth with LoRA.

Training Configuration

After training:

1. The LoRA adapter was merged into the base model.
2. The merged model was exported.
3. The model was converted to GGUF format for efficient local inference.

Quantization

The released model uses the following GGUF file:

Qwen2.5-0.5B-Instruct.Q4_K_M.gguf

The Q4_K_M quantization provides a good balance between model size, inference speed, and response quality, making it suitable for running efficiently on consumer hardware.

Local Inference with Ollama

The model can be run locally using Ollama after creating a Modelfile that references:

Qwen2.5-0.5B-Instruct.Q4_K_M.gguf

Then create and run the model:

ollama create qwen-alpaca -f Modelfile
ollama run qwen-alpaca

This enables fast local inference without requiring a cloud API.

Deployment

The project has also been deployed on Hugging Face Spaces, allowing users to interact with the model through a web interface in addition to local inference with Ollama.

Intended Uses

This model is suitable for:

• General chat
• Instruction following
• Learning and experimentation
• Prompt engineering practice
• Local LLM inference
• Educational demonstrations of LLM fine tuning

Limitations

• Trained for only two epochs.
• No quantitative evaluation metrics were collected.
• Responses may occasionally contain factual inaccuracies or hallucinations.
• Not intended for production or high risk applications.
• Performance is limited by the 0.5B parameter size.

Example

Prompt

Explain what a Transformer is in simple terms.

Response

A Transformer is a neural network architecture designed to process sequences efficiently using self attention. Instead of reading text one word at a time, it looks at all relevant words together, allowing it to understand context better and generate more accurate responses.

Learning Objectives

This project was created to gain hands on experience with the complete LLM development pipeline, including:

• Instruction fine tuning
• LoRA adaptation
• Model merging
• GGUF conversion
• Quantization
• Local inference with Ollama
• Hugging Face model hosting
• Hugging Face Spaces deployment

Local Inference with Ollama

The model was successfully tested locally using Ollama.

Acknowledgements

• Qwen Team for the Qwen2.5 base model.
• Unsloth for efficient LoRA fine tuning.
• tatsu-lab for the Alpaca instruction dataset.
• Ollama for local LLM inference.
• Hugging Face for model hosting and Spaces deployment.

Live Demo

Try the model in your hugging face Space:

ByteBot: https://huggingface.co/spaces/ciphermosaic/ByteBot