openllama-13b-instruct-human-feedback

LoRA OpenLLaMA 13B (HH-RLHF)

Lightweight LoRA adapter for openlm-research/open_llama_13b trained on Anthropic HH-RLHF preference data to encourage helpful, harmless text generation.

Model Details

• Developed by: Sagnik Das
• Shared by: Sagnik Das
• Model type: Causal LM LoRA adapter
• Language(s): English
• License: MIT
• Finetuned from: openlm-research/open_llama_13b
• Training data: Anthropic/hh-rlhf (chosen/rejected pairs converted to quality labels)

Model Sources

• Repository: https://github.com/sagnik-sudo/LLM-LoRA-Implementation (training code)
• Demo: Load with PeftModel.from_pretrained (see below)
• Paper: None (work in progress)

Uses

Direct Use

Attach this adapter to the base model for instruction-style text generation, quality-sensitive completions, and safety-aligned responses.

Downstream Use

Can be further tuned or evaluated for safety/quality scoring tasks using the same prompt/quality format.

Out-of-Scope Use

Avoid deployment in safety-critical, high-stakes, or fully autonomous settings without thorough alignment, red-teaming, and monitoring.

Bias, Risks, and Limitations

• Trained on HH-RLHF; inherits its coverage and biases.
• No safety finetuning beyond dataset intent; harmful or incorrect generations remain possible.
• Single-GPU LoRA run; not benchmarked for robustness.

Recommendations

Add task-specific filtering/guardrails and evaluate on your downstream data before production use.

How to Get Started

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "openlm-research/open_llama_13b"
adapter = "sagnik-das/lora-openllama-13b-hh-rlhf"

tokenizer = AutoTokenizer.from_pretrained(base, use_fast=False, legacy=False)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

prompt = "Explain why good documentation matters."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Details

• Objective: Supervised LM with labels derived from HH-RLHF chosen/rejected pairs ("Quality: good" vs "Quality: bad").
• LoRA config: r=8, alpha=16, dropout=0.05; target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj; bias=none.
• Optimization: AdamW (Trainer defaults), lr=2e-4, weight decay=0.01, warmup=100 steps, max_steps=7000, gradient_accumulation_steps=16.
• Batching: per_device_train_batch_size=1, per_device_eval_batch_size=1.
• Precision: bf16 (fallback to fp32 on CPU).
• Seq length: 256 tokens, pad to max length.
• Checkpoints: saved every 1500 steps; final adapter saved after step 7000.
• Hardware: single NVIDIA A40 GPU.

Evaluation

No formal eval recorded; a simple internal win-rate metric over a subset of HH-RLHF was printed during training. Please benchmark on your target tasks before deployment.

Environmental Impact

Not measured; single A40 session, <1 epoch over HH-RLHF with max 7000 steps. Estimate with MLCO2 if required.

Model Card Authors

• Sagnik Das