Downloads · 30 days
4
16% of all-time downloads
yujackein/onereason-8b-lora-item32k-user75-rec50-worldclean1601-all1-lr2e4-r32a32-seq10k-gbs4-1epoch
onereason-8b-lora-item32k-user75-rec50-worldclean1601-all1-lr2e4-r32a32-seq10k-gbs4-1epoch is a text generation model from yujackein. Use it when you need the model to write or continue text. It is set up for peft.
This is an experimental LoRA adapter for OpenOneRec/OneReason-8B-pretrain-competition, trained for the OneReason recommendation competition.
Downloads · 30 days
4
16% of all-time downloads
All-time downloads
25
Public
Repo size
365 MB
Likes
0
Public
Click a slice to open those files.
.safetensors349 MB · 96%
From the Hugging Face model README
This is an experimental LoRA adapter for
OpenOneRec/OneReason-8B-pretrain-competition,
trained for the OneReason recommendation competition.
The source mixture is the Item32K / User75 / Rec50 / WorldClean1601 recipe:
| Category | Source rows | Source rendered tokens |
|---|---|---|
| R0 perception | 32,000 | 5,801,468 |
| R2 evolution | 24,636 | 119,130,968 |
| R3 recommendation | 24,134 | 34,445,219 |
| Clean C-Eval world data | 1,601 | 195,253 |
| Total | 82,371 | 159,572,908 |
For the 10,000-token run, 15 R2_evolution rows longer than the cutoff were
removed because head truncation would eliminate all assistant supervision.
The resulting training set has 82,356 rows and 159,413,369 rendered tokens.
The exact four-rank packing plan consumes 99.0516% of these tokens in one safe
pass. All assistant loss weights were normalized to 1.0.
r=32, alpha=32, dropout 0.05)q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj2e-4, cosine schedule, 3% warmupAdapter SHA-256:
ec65db55748eeb8181591b53ba1d2eb7f3dfea702e5e32552f93e6198f45a776
The closest existing run is
yujackein/onereason-8b-lora-item32k-user75-rec50-worldclean1601-all1-lr2e4-r32a32-step323,
the one-packing-pass checkpoint from the 32,768-context r32/a32 LR2e-4 run.
This is not a strict sequence-length-only ablation. Relative to that step-323 checkpoint, this run also changes global batch size from 16 to 4, uses a standalone one-epoch cosine schedule rather than the first half of a two-epoch schedule, and removes 15 overlength rows. The data recipe, base model, LoRA configuration, optimizer, peak learning rate, precision, and seed are otherwise the same.
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "OpenOneRec/OneReason-8B-pretrain-competition"
adapter_id = "yujackein/onereason-8b-lora-item32k-user75-rec50-worldclean1601-all1-lr2e4-r32a32-seq10k-gbs4-1epoch"
tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
base_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)
Formal evaluation results have not yet been added.