Veltha-14B-StoryCraft-v1

Veltha-14B-StoryCraft-v1 🎭✨

A specialized 14B parameter language model fine-tuned for creative fantasy and science fiction novel writing.

Model Description

Veltha-14B-StoryCraft-v1 is a fine-tuned version of djuna/Q2.5-Veltha-14B-0.5 (a Qwen 2.5 14B merged model) optimized specifically for long-form creative writing in fantasy and science fiction genres.

Key Features

✨ No Repetition Issues - Conservative single-stage training prevents catastrophic overfitting 📚 Long-Form Coherence - Maintains consistent narrative voice across 1000+ word generations 🎨 Vivid Descriptions - Trained on high-quality fantasy and creative writing datasets 🧙 Genre Expertise - Specialized in fantasy and science fiction storytelling ⚡ Efficient - 14B parameters, runs on 24GB+ VRAM with 4-bit quantization

Training Details

Training Approach

Single-Stage Conservative Fine-Tuning:

• Duration: 3.77 hours (1 epoch)
• Final Loss: 1.9706
• Token Accuracy: 55%
• Tokens Processed: 17.8M

This conservative approach was specifically designed to avoid the repetition issues that plague aggressive multi-stage training.

Training Configuration

Model: djuna/Q2.5-Veltha-14B-0.5
Architecture: Qwen 2.5 14B (merged model)
Fine-tuning Method: LoRA (Low-Rank Adaptation)

LoRA Configuration:
  - Rank (r): 32
  - Alpha: 16
  - Dropout: 0.05
  - Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  - Trainable Parameters: 137.6M (1.66% of total)

Training Hyperparameters:
  - Learning Rate: 5e-5 (cosine schedule)
  - Batch Size: 2 (per device)
  - Gradient Accumulation: 8 steps
  - Effective Batch Size: 16
  - Epochs: 1
  - Optimizer: AdamW 8-bit
  - Weight Decay: 0.01
  - Warmup Ratio: 0.1
  - Mixed Precision: BF16

Hardware:
  - GPU: NVIDIA L40S (48GB)
  - Framework: Unsloth + TRL
  - Quantization: 4-bit NF4

Datasets

Combined Training Set (19,304 examples):

1. VellumK2-Fantasy-DPO-Large-01 (10,222 examples)

   • High-quality fantasy writing samples
   • DPO format (prompt/chosen pairs)
   • Source: lemon07r/VellumK2-Fantasy-DPO-Large-01

2. Creative_Writing_Multiturn (9,082 examples)

   • Long-form creative conversations
   • Multi-turn dialogue format
   • Source: Dampfinchen/Creative_Writing_Multiturn

Evaluation Results

Comprehensive testing across 4 creative writing tasks:

Total Output: 2,334 words with zero catastrophic repetition issues.

Sample Outputs

<details> <summary><b>Fantasy Opening Scene (541 words)</b></summary>

Prompt: Write the opening scene of a fantasy novel. A young apprentice discovers an ancient artifact in the forbidden archives that reveals a hidden truth about their kingdom's founding.

Output:

"Master Alaric," whispered Elara as she slipped into the shadows between two towering bookcases, her breath misting in the chill air. The moonlight filtered through stained glass windows, casting eerie patterns across rows upon rows of dusty tomes. She had been ordered not to come here; the Forbidden Archives were off-limits to apprentices like herself.

But curiosity was her greatest flaw. When Master Alaric left for his nightly meeting with Lord Reginald, Elara couldn't resist exploring this mysterious part of the library where whispers echoed instead of footsteps...

[Full output demonstrates strong worldbuilding, tension, and character development with no repetition]

</details> <details> <summary><b>Sci-Fi First Contact (533 words)</b></summary>

Prompt: Write a science fiction scene where humanity makes first contact with an alien species that communicates through colors and bioluminescence on a generation ship.

Output:

The Starlight Voyager had sailed the void of space for two hundred years, its passengers living in cramped quarters that smelled of dust and hope. They were a ragged band—scientists, dreamers, criminals—and they'd made it this far by necessity rather than virtue...

Colors bled across metal walls, through glass, into skin. Crew members screamed or laughed; some fell to their knees while others stretched out their hands as if embracing light itself...

[Demonstrates creative concept execution, vivid descriptions, and emotional depth]

</details>

Usage

Basic Usage

from unsloth import FastLanguageModel

# Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="writetoasik/Veltha-14B-StoryCraft-v1",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,  # For 24GB VRAM GPUs
)

# Prepare for inference
FastLanguageModel.for_inference(model)

# Create a writing prompt
messages = [
    {"role": "system", "content": "You are a creative novel writer specializing in fantasy and science fiction. Write engaging, coherent narratives with vivid descriptions and strong plot structure."},
    {"role": "user", "content": "Write the opening scene of a fantasy novel about a dragon rider who discovers their dragon can speak."}
]

# Format and generate
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs,
    max_new_tokens=800,
    temperature=0.8,
    top_p=0.9,
    do_sample=True,
    repetition_penalty=1.1,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Advanced Usage with Custom Parameters

# For longer outputs (up to 2048 tokens)
outputs = model.generate(
    input_ids=inputs,
    max_new_tokens=1500,
    temperature=0.85,      # Higher for more creativity
    top_p=0.92,            # Slightly higher for more diversity
    top_k=50,
    do_sample=True,
    repetition_penalty=1.15,  # Stronger penalty if needed
    no_repeat_ngram_size=3,   # Prevent 3-gram repetition
)

Recommended Generation Parameters

For best creative writing results:

generation_config = {
    "temperature": 0.8,           # Balanced creativity
    "top_p": 0.9,                 # Nucleus sampling
    "repetition_penalty": 1.1,    # Gentle repetition prevention
    "max_new_tokens": 800,        # ~600 words
    "do_sample": True,            # Enable sampling
}

Hardware Requirements

Recommended: 4-bit quantization on 24GB+ VRAM GPU (RTX 3090/4090, A5000, L40S)

Limitations

• System/User Prompts Visible: Occasionally includes chat template markers in output (post-processing recommended)
• Word Count Targets: May stop before reaching requested word counts if it finds a natural ending
• Genre Specialization: Optimized for fantasy/sci-fi; performance on other genres may vary
• Base Model Constraints: Inherits any limitations from Q2.5-Veltha-14B-0.5

Ethical Considerations

• Creative Content: This model generates fictional creative content and should not be used for factual information
• Attribution: Generated content should be reviewed and edited by human authors
• Bias: May reflect biases present in training data (fantasy/sci-fi tropes, narrative conventions)

Training Journey

This model represents the successful conclusion of an iterative fine-tuning process:

1. Qwen 2.5 7B (3-stage) → ❌ Failed: Repetition issues
2. Gemma 3 27B (3-stage) → ❌ Failed: Catastrophic overfitting (9 epochs total)
3. Gemma 3 27B (single-stage) → ❌ Failed: Out of memory
4. Q2.5-Veltha-14B (single-stage) → ✅ SUCCESS!

Key Learnings:

• Conservative single-stage training prevents overfitting
• 1 epoch with low learning rate (5e-5) is sufficient
• Smaller models (14B) can outperform larger ones (27B) with proper training
• Skip literary classics datasets to avoid archaic language patterns

Citation

If you use this model in your research or applications, please cite:

@misc{veltha-storycraft-v1,
  title={Veltha-14B-StoryCraft-v1: A Fine-Tuned Model for Fantasy and Science Fiction Writing},
  author={writetoasik},
  year={2025},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/writetoasik/Veltha-14B-StoryCraft-v1}},
}

Acknowledgments

• Base Model: djuna/Q2.5-Veltha-14B-0.5
• Framework: Unsloth for efficient training
• Datasets: VellumK2 and Creative Writing Multiturn contributors
• Training: AWS EC2 with NVIDIA L40S

License

Apache 2.0 (inherits from base model)

Contact

For questions, issues, or collaboration:

• 🤗 HuggingFace: @writetoasik
• 📧 Issues: Model Discussion

---

Model Version: 1.0 Release Date: January 2025 Training Completed: January 16, 2025 Status: Production Ready ✅