continue_sft_bitd_lora_full_data_k30

continue_sft_bitd_lora_full_data_k30

LoRA adapter trained on top of tuongvy2603/BITD_baseline — a second-stage continued SFT on the full_data / k=30 data pool. Loading base + this adapter gives a model that has been SFT'd twice.

This adapter is part of a sweep over the per-prompt sample budget k used to build the continued-SFT training set from the full data pool.

Usage

Install

pip install transformers peft torch

Load and run

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# 1. Load base model
base = AutoModelForCausalLM.from_pretrained(
    "tuongvy2603/BITD_baseline",
    dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Attach LoRA adapter
model = PeftModel.from_pretrained(base, "tuongvy2603/continue_sft_bitd_lora_full_data_k30")

# 3. Tokenizer (chat template lives here)
tok = AutoTokenizer.from_pretrained("tuongvy2603/continue_sft_bitd_lora_full_data_k30")

# 4. Generate
messages = [{"role": "user", "content": "Pick open-minded or close-minded."}]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
out = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

No merging required — PeftModel injects the LoRA weights at runtime. Forward results are mathematically identical to a merged model.

Optional: merge for standalone use

If you want a single self-contained checkpoint (e.g. for llama.cpp / GGUF conversion, or serving stacks that don't load adapters):

merged = model.merge_and_unload()
merged.save_pretrained("continue_sft_bitd_lora_full_data_k30_merged")
tok.save_pretrained("continue_sft_bitd_lora_full_data_k30_merged")

Training details

Full resolved config: see run_config.json in this repo.

Framework versions

• PEFT 0.19.1
• TRL 1.4.0
• Transformers 5.8.1
• PyTorch 2.12.0
• Datasets 4.8.5
• Tokenizers 0.22.2

Citation

@software{vonwerra2020trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2020}
}