qwen2.5-72b-cpt

Qwen2.5-72B — Text Continual Pretraining (CPT) with LoRA Adapters

Model type: Causal Language Model
Base model: Qwen/Qwen2.5-72B
License: Apache 2.0
Framework: Axolotl + DeepSpeed ZeRO-1

---

Overview

qwen2.5-72b-cpt is a continual-pretrained version of Qwen 2.5-72B, trained using LoRA adapters in 4-bit NF4 quantization for efficient large-scale domain adaptation.
This release contains only the LoRA adapters and training configuration, allowing users to load them on top of the official Qwen 2.5-72B base model.

The CPT (Continual Pretraining) objective broadens the model’s scientific, governance, and energy-domain knowledge while preserving Qwen’s strong general and reasoning capabilities.
Training was performed on the Leonardo EuroHPC supercomputer using Axolotl 0.6 + DeepSpeed ZeRO-1 optimization with bfloat16 computation.

---

Training Setup

---

Dataset

Public text corpora from open scientific, governmental, and encyclopedic sources:

---

Hyperparameters

---

Tokenizer

Tokenizer type: AutoTokenizer
Special token: <|end_of_text|> as pad_token

---

Files Included

This repository hosts LoRA adapters and Axolotl metadata only.

Contents:

• adapter_config.json
• adapter_model.safetensors
• config.json
• special_tokens_map.json
• tokenizer_config.json
• tokenizer.json
• README.md

---

Usage — Load and Apply the Adapters

To use this CPT variant in Python:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = "Qwen/Qwen2.5-72B"
adapter_repo = "ubitech-edg/qwen2.5-72b-cpt"

# Load base and tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    device_map="auto",
    torch_dtype="bfloat16"
)

# Load LoRA adapters
model = PeftModel.from_pretrained(model, adapter_repo)
model.eval()

prompt = "Explain how renewable energy contributes to grid stability."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))