scugnizz-llama-training

Scugnizz Llama-PCS — training & eval package

Start here for teammates: HANDOFF.md
Project timeline: HISTORY.md

Weights live in ProjectScugnizz/scugnizz-llama-pcs.
This repo is the trainer + launchers + Gradio source.

One-command examples

# Train (HF Jobs)
bash examples/train-hf-extractive-harden.sh

# Train (RunPod PRO 6000)
CLOUD_TYPE=COMMUNITY bash examples/train-runpod-extractive-harden.sh

# Eval gate (HF Jobs)
FLAVOR=t4-small bash examples/eval-hf-tool-context.sh

# Eval gate (RunPod)
CLOUD_TYPE=COMMUNITY bash examples/eval-runpod-tool-context.sh

Legacy pretrain quick start

export HF_TOKEN=hf_...
export OUT_DIR=/mnt/rope-v2-training-results/runs/my-run
export TARGET_TOKENS=1000000000
export FLAVOR=a100x4
export TOKENS_PER_STEP=262144
export MODEL_SIZE=1.7b
export HUB_REPO_ID=ProjectScugnizz/scugnizz-llama-pcs
export HUB_PATH=training-runs/my-run
export SCRIPT_SOURCE=hub:ProjectScugnizz/scugnizz-llama-training
bash run-hf-job.sh

Model sizes

Architecture must match the starting weights.

Starting weights

Priority:

1. OUT_DIR/checkpoint_resume.pt or checkpoint_last.pt — full resume with optimizer
2. INIT_FROM — bucket/local path to model_final.pt or full checkpoint
3. INIT_HUB_REPO + INIT_HUB_FILE — download from Hub
4. OUT_DIR/model_final.pt — weights-only fallback

Examples:

# Continue an existing bucket run (automatic if checkpoints exist)
export OUT_DIR=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b

# Start from Hub weights
export INIT_HUB_REPO=ProjectScugnizz/scugnizz-llama-pcs
export INIT_HUB_FILE=training-runs/pretrain-fineweb-llama-pcs-550m/model_final.pt
export INIT_STEP=2100

# Start from bucket file
export INIT_FROM=/mnt/rope-v2-training-results/runs/pretrain-fineweb-llama-pcs-1.7b/model_final.pt
export INIT_STEP=0
export NO_RESUME=1

Multi-GPU (DDP)

Uses torchrun automatically when the job has >1 GPU. Set global throughput with:

export TOKENS_PER_STEP=262144   # auto-computes GRAD_ACCUM per GPU count
# or set GRAD_ACCUM manually per GPU

Monitor GPUs:

hf jobs stats ProjectScugnizz/<job_id>

Checkpoints

Keep (optimizer resume): checkpoint_resume.pt, checkpoint_last.pt

Safe to delete: checkpoint_weights_last.pt

I/O performance: checkpoints write to /tmp/scugnizz-ckpts first; only resume/last copied to bucket. Defaults: WEIGHTS_SAVE_INTERVAL=0, SAVE_INTERVAL=100. Logs show (N inst) instantaneous tok/s.

export CKPT_LOCAL_DIR=/tmp/scugnizz-ckpts
export WEIGHTS_SAVE_INTERVAL=0
export SAVE_INTERVAL=100
bash cleanup-checkpoints.sh hf://buckets/ProjectScugnizz/rope-v2-training-results/runs/my-run

Files