llama3.1-8B-writer-Q8

llama3.1-8B-writer-Q8

An 8-bit MLX quantization of a fine-tuned Llama 3.1 8B Instruct model, optimized for creative writing on Apple Silicon.

Model Details

Training

This model was fine-tuned from meta-llama/Llama-3.1-8B-Instruct using LoRA on a 2× NVIDIA RTX 3090 cluster. The fine-tuning focused on improving creative and narrative writing quality. After training, the LoRA adapter was merged back into the base model and the result was converted to MLX 8-bit quantized format.

Training Details

• Method: LoRA (Low-Rank Adaptation)
• Hardware: 2× NVIDIA RTX 3090 (48 GB VRAM total)
• Framework: Unsloth
• Dataset: Curated creative writing examples
• Quantization tool: mlx_lm.convert

Intended Use

This model is designed for creative and narrative writing tasks — storytelling, prose generation, dialogue, and similar text generation. It follows the Llama 3.1 Instruct chat template.

How to Use

With MLX (Apple Silicon)

from mlx_lm import load, generate

model, tokenizer = load("warrenm/llama3.1-8B-writer-Q8")
response = generate(model, tokenizer, prompt="Write the opening paragraph of a noir detective story.", max_tokens=512)
print(response)

With Cogitae (macOS)

This model can be loaded directly in Cogitae via its built-in MLX engine. Place the model files in ~/.cogitae/models/warrenm/llama3.1-8B-writer-Q8/ and load from the MLX provider.

Files

Limitations

• Fine-tuned primarily for creative writing; may underperform on reasoning, coding, or factual Q&A compared to the base instruct model.
• 8-bit quantization introduces minor quality loss compared to the full-precision model.
• Inherits all limitations and biases of the base Llama 3.1 8B Instruct model.