InstructPythia-410M-LoRA-v1

Pythia-410M Dolly LoRA on Modal

A compact instruction-tuning project that fine-tunes the standard, non-deduplicated EleutherAI/pythia-410m model on databricks/databricks-dolly-15k using 16-bit LoRA and Modal GPU compute.

Project summary

The project saves checkpoints, the final PEFT adapter, tokenizer files, and training metadata to a persistent Modal Volume named pythia-410m-dolly-output.

Results

Observed test generations:

Simple explanation

Prompt

Explain artificial intelligence in simple terms.

Response

Artificial Intelligence (AI) is the term for a new generation of computer
systems that are capable of performing tasks without human assistance. This can
be done by combining artificial neural networks with computational intelligence
algorithms to automate and improve the efficiency of our everyday life. AI
technology has already been used in areas such as robotics, video games,
healthcare, and manufacturing.

Closed factual question

Prompt

What is the capital of France?

Response

Paris.

Creative-writing limitation

Prompt

Write a short poem about a robot exploring the Moon.

Response

I love to explore, it's the best way to meet new people.

The model performs well on simple explanations and basic factual questions, but creative constraint-following remains unreliable at this model size.

Project layout

pythia-410m-dolly/
├── train.py
├── test_model.py
├── README.md
└── .venv/

Persistent files inside the Modal Volume:

/data/
├── cache/
│   └── huggingface/
└── outputs/
    └── pythia-410m-dolly-lora/
        ├── checkpoint-*/
        ├── final-adapter/
        │   ├── adapter_config.json
        │   ├── adapter_model.safetensors
        │   ├── tokenizer.json
        │   ├── tokenizer_config.json
        │   └── special_tokens_map.json
        ├── trainer_state.json
        └── training_summary.json

Requirements

• Python 3.11 or newer
• A Modal account
• Modal CLI authentication
• Internet access for the initial package and model downloads

Local environment

mkdir -p ~/pythia-410m-dolly
cd ~/pythia-410m-dolly

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
pip install --upgrade modal

modal setup

Training configuration

The training script uses approximately the following settings:

MODEL_NAME = "EleutherAI/pythia-410m"
DATASET_NAME = "databricks/databricks-dolly-15k"

MAX_LENGTH = 768
NUM_TRAIN_EPOCHS = 3
LEARNING_RATE = 2e-4

PER_DEVICE_TRAIN_BATCH_SIZE = 4
GRADIENT_ACCUMULATION_STEPS = 4

LORA_R = 16
LORA_ALPHA = 32
LORA_DROPOUT = 0.05

The effective training batch size is 16 examples per optimizer step.

The LoRA adapter targets Pythia/GPT-NeoX projection modules:

target_modules = [
    "query_key_value",
    "dense",
]

Training labels mask the instruction and context tokens with -100, so loss is calculated only over response tokens.

Run training

Check the script before launching a paid GPU container:

python -m py_compile train.py

Start a detached Modal run:

modal run --detach train.py

Modal's --detach option keeps the remote application running if the local process exits or disconnects.

Persistent storage

The same Modal Volume cannot be mounted at two different paths in one function. This project mounts it once:

VOLUME_ROOT = "/data"

volumes = {
    VOLUME_ROOT: output_volume,
}

Outputs and caches then use separate subdirectories:

OUTPUT_DIR = "/data/outputs/pythia-410m-dolly-lora"
CACHE_DIR = "/data/cache/huggingface"

Test the adapter on Modal

Run the default prompt:

modal run test_model.py

Run a custom prompt:

modal run test_model.py \
  --prompt "Explain artificial intelligence in simple terms."

Run deterministic greedy decoding:

modal run test_model.py \
  --prompt "What is the capital of France?" \
  --temperature 0 \
  --max-new-tokens 80

Run sampled decoding:

modal run test_model.py \
  --prompt "Write exactly four short lines about a robot on the Moon." \
  --temperature 0.45 \
  --top-p 0.85 \
  --top-k 30 \
  --repetition-penalty 1.12 \
  --max-new-tokens 100

At inference time, the script loads the original base model and then attaches the saved LoRA weights:

base_model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/pythia-410m",
    torch_dtype=torch.float16,
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_DIR,
    is_trainable=False,
)

Download the trained adapter

mkdir -p downloaded-output

modal volume get \
  pythia-410m-dolly-output \
  outputs/pythia-410m-dolly-lora/final-adapter \
  downloaded-output/

Inspect downloaded files:

find downloaded-output -type f -printf '%p  %k KB\n'

Training metrics

The completed run reported:

train_runtime: 3160.6921 seconds
train_samples_per_second: 13.962
train_steps: 2760
epoch: 3.0

Late-run training losses were mostly around 1.90 to 2.14. Gradient norms were generally stable, mostly near 0.7 to 1.3, with no obvious sign of gradient explosion in the supplied logs.

Limitations

• The model is small and may ignore multi-part or creative instructions.
• It can generate inaccurate or outdated factual claims.
• Dolly 15K is English-only and does not make the model reliably multilingual.
• LoRA instruction tuning does not replace the base model's knowledge.
• The model has not undergone dedicated safety alignment.
• Outputs should be reviewed before use in consequential settings.

Dataset and model licensing

• Pythia-410M is published under the Apache 2.0 license.
• Dolly 15K is published under CC BY-SA 3.0.
• Review both upstream licenses before redistributing a merged model, adapter, or derivative dataset.
• Final model license: CC BY-SA 3.0.

This repository should not imply endorsement by EleutherAI, Databricks, Hugging Face, or Modal.

References

• EleutherAI Pythia-410M model card
• Databricks Dolly 15K dataset card
• Modal run CLI documentation
• Modal Volume documentation
• Modal GPU documentation
• Hugging Face PEFT model documentation