Downloads · 30 days
8
16% of all-time downloads
hanspeterlyngsoeraaschoujensen
hanspeterlyngsoeraaschoujensen/qwen3-0.6b-wikihop-crossbatch-sft-nb4-nv16
qwen3-0.6b-wikihop-crossbatch-sft-nb4-nv16 is a machine learning model from hanspeterlyngsoeraaschoujensen. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft.
This repository contains PEFT/LoRA adapter checkpoints for the WikiHop split SFT cross-sequence attention experiment.
Downloads · 30 days
8
16% of all-time downloads
All-time downloads
49
Public
Repo size
4.1 GB
Likes
0
Public
Click a slice to open those files.
.safetensors3.5 GB · 69%
From the Hugging Face model README
This repository contains PEFT/LoRA adapter checkpoints for the WikiHop split SFT cross-sequence attention experiment.
Configuration:
Qwen/Qwen3-0.6BQwen3BatchParScaleForCausalLMqangaroo/wikihopwikihop_splitn_batches=4: three support/evidence rows and one target/query rowseq_len=512n_virtual_tokens=1616 / 512 = 3.125%, one virtual token per 32 ordinary positionsuse_virtual_qkv=True, virtual_qkv_mode=qkv, virtual_qkv_init=copyr=64, alpha=128max_steps=200Validation on the first 100 WikiHop validation examples with candidate log-prob scoring:
accuracy_mean=0.26accuracy_sum=0.18correct_probability_mean=0.17784384276711085nll_mean=2.7201295478595555Reference baselines on the same slice:
0.047accuracy_mean=0.28accuracy_mean=0.37W&B run: https://wandb.ai/hprjdk/qwen3-crossbatch-finetune/runs/xm16x79l