Downloads · 30 days
9
2% of all-time downloads
violetxi/hparam-92m-block-mtp-truncated-peak_lr-3e-4-wu-0p01-s42
hparam-92m-block-mtp-truncated-peak_lr-3e-4-wu-0p01-s42 is a text generation model from violetxi. Use it when you need the model to write or continue text. It is set up for transformers.
Condition: 92m-blktrunc-lr3e4-wu01-s42. This branch contains ready-to-load model weights, config, tokenizer and the exact shared-core implementation. Each checkpoint has its own step-N branch. Install the packages in…
Downloads · 30 days
9
2% of all-time downloads
All-time downloads
449
Public
Parameters
92.1M
2.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors369 MB · 100%
From the Hugging Face model README
Condition: 92m-blktrunc-lr3e4-wu01-s42. This branch contains ready-to-load model weights,
config, tokenizer and the exact shared-core implementation. Each checkpoint
has its own step-N branch. Install the packages in requirements.txt.
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
repo = "violetxi/hparam-92m-block-mtp-truncated-peak_lr-3e-4-wu-0p01-s42"
revision = "step-781"
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, revision=revision, trust_remote_code=True, dtype=torch.bfloat16,
).to("cuda" if torch.cuda.is_available() else "cpu").eval()
inputs = tokenizer("Pe2e4 Pe7e5 Ng1f3", return_tensors="pt").to(model.device)
generated = model.generate(**inputs, max_new_tokens=20, do_sample=True,
temperature=1.0, top_k=0, top_p=1.0)
print(tokenizer.decode(generated[0], skip_special_tokens=True))
Three recurrent passes score three fixed, causally ordered query slots.
generate() commits that three-token block before the next recurrent block;
the mask token (ID 81) cannot be generated. Sampling and greedy generation,
padding and num_return_sequences are supported; KV caching and beam search
are not supported. Context is 1,024 logical tokens including the query block.
model.block_logits(input_ids) returns [batch, 1, 3, vocab] for the next block.
model(input_ids, labels=input_ids) uses the registered dense non-overlapping
block loss: logits[:, t] scores token t+1, with every group of three seeing
only the prefix before that group. It is not ordinary teacher-forced NTP.
Benchmark inference is not performed by publication. Condition selection
remains governed by the sweep's validation and collapse rules.
Safetensors values preserve the native model tensors exactly. Optimizer and
RNG recovery state remains in the native archive; it is not needed for HF
inference. training_summary.json binds this export to the native checksum
and source versions. Adapted core code licenses are included alongside it.