spoomplesmaxx-base-qwen3-14b

SpoomplesMaxx Base — Qwen3-14B CPT

A continued pre-training (CPT) of Qwen3-14B-Base on a curated mix of fiction, character knowledge, prose, and domain-specific corpora. This is the base model — further SFT and DPO stages follow.

Model Description

This model is part of the SpoomplesMaxx training pipeline: CPT → SFT → DPO

The CPT stage teaches the model general language patterns, domain knowledge, and writing styles by training on raw text corpora without chat templates. It grounds the model in character knowledge, narrative prose, multilingual content, and uncensored language before instruction tuning.

Training Data

CPT Curriculum (3 phases)

The prepared dataset (aimeri/spoomplesmaxx-cpt-small-Qwen3-14B-Base) was assembled from three curriculum phases, each with different repeat factors to control emphasis.

Total training samples: 50,000 (after dataset_limit, repacked to 3072 tokens)

The prepared dataset is pre-tokenized and publicly available on HuggingFace. Private data (custom character cards) is included in the tokenized form only.

Training Configuration

Hardware

• GPU: 4× NVIDIA H100
• Training Time: ~14 hours

Metrics

Intended Use

This CPT base model is intended as a foundation for the SpoomplesMaxx pipeline:

• Next step: SFT with chat/instruction data and persona injection
• Final step: DPO alignment with preference pairs

Use cases after full pipeline:

• Creative writing and fiction generation
• Character roleplay with consistent personas
• Uncensored conversational AI
• Multilingual content (English, Portuguese, some Italian/Spanish)

Not Recommended For:

• Direct use as a chat assistant (this is a base model — no instruction tuning yet)
• Factual Q&A or knowledge retrieval (CPT emphasizes narrative over factuality)
• Production safety-critical applications

Limitations

• This is a base model without instruction following. It will continue text, not answer questions.
• Domain knowledge is biased toward fiction, character knowledge, and creative writing.
• Contains uncensored/NSFW training data — outputs may include explicit content.
• Multilingual content is weighted toward English with some Portuguese/Brazilian content.
• 3072 token context window during CPT (base model supports 128K; longer contexts untested post-CPT).

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "aimeri/spoomplesmaxx-base-qwen3-14b",
    dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("aimeri/spoomplesmaxx-base-qwen3-14b")

# Text continuation (base model — no chat template)
inputs = tokenizer("The castle stood silent against the darkening sky, its towers reaching toward clouds that promised rain. Inside,", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.8, top_p=0.95)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model, please cite the base model and datasets:

@misc{qwen3-14b-base,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2025},
  url={https://huggingface.co/Qwen/Qwen3-14B-Base}
}

Acknowledgments

• Qwen Team for the Qwen3-14B-Base model
• PocketDoc for the Dans-Prosemaxx, DanChat format and related datasets
• PJMixers-Dev for curated fiction and RP datasets
• HuggingFace for the Transformers and Accelerate libraries
• DeepSpeed for ZeRO-3 distributed training

---