Downloads · 30 days
50
7% of all-time downloads
1jamesthompson1/Qwen3.8-27B-nz-wvs-sampled_response-overall
Qwen3.8-27B-nz-wvs-sampled_response-overall is a text generation model from 1jamesthompson1. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as cc-by-sa-4.0.
This model is a LoRA fine-tune of Qwen/Qwen3.8-27B as part of the AIML589 project.
Downloads · 30 days
50
7% of all-time downloads
All-time downloads
711
Public
Repo size
19.1 GB
Likes
0
Public
Click a slice to open those files.
.pt2.6 GB · 66%
From the Hugging Face model README
This model is a LoRA fine-tune of Qwen/Qwen3.8-27B as part of the AIML589 project.
This adapter is licensed under CC BY-SA 4.0.
Fine-tuned on the sampled_response config of the wvs-nz-value-alignment dataset, overall subpopulation.
Part of the wvs-nz-value-alignment collection.
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition · Training time: 24m 58s
The exact command used to produce this adapter (also saved in
finetune_config.json):
finetune.py --model Qwen/Qwen3.8-27B --dataset sampled_response --subpopulation overall --upload-to-hf --lora-r 64 --lora-alpha 128
| Parameter | Value |
|---|---|
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0.05 |
| DoRA | False |
| Learning rate | 0.0002 |
| Batch size | 4 |
| Gradient accumulation | 4 |
| Epochs | 3 |
| Max seq length | 1024 |
| Warmup ratio | |
| Dtype | bf16 |
| Trainable parameters (LoRA) | 318,767,104 |
| Adapter size on disk | 1,275,137,464 bytes |
{"loss": 0.5106740951538086, "grad_norm": 1.8516865968704224, "learning_rate": 5.4545454545454546e-05, "entropy": 0.37830601055175067, "mean_token_accuracy": 0.8137880250811577, "num_tokens": 24270.0, "epoch": 0.09324009324009325, "step": 10}
{"eval_loss": 0.4490979313850403, "eval_runtime": 14.5137, "eval_samples_per_second": 20.188, "eval_steps_per_second": 5.099, "eval_entropy": 0.4208357319160289, "eval_mean_token_accuracy": 0.8256926053279156, "eval_num_tokens": 24270.0, "epoch": 0.09324009324009325, "step": 10}
{"loss": 0.35553340911865233, "grad_norm": 2.989311933517456, "learning_rate": 0.00011515151515151516, "entropy": 0.38502076622098685, "mean_token_accuracy": 0.8531894400715828, "num_tokens": 48304.0, "epoch": 0.1864801864801865, "step": 20}
{"eval_loss": 0.4185604453086853, "eval_runtime": 14.7408, "eval_samples_per_second": 19.877, "eval_steps_per_second": 5.02, "eval_entropy": 0.28698493043044143, "eval_mean_token_accuracy": 0.8556888933117325, "eval_num_tokens": 48304.0, "epoch": 0.1864801864801865, "step": 20}
{"loss": 0.33315315246582033, "grad_norm": 1.9446290731430054, "learning_rate": 0.00017575757575757578, "entropy": 0.25919492114335296, "mean_token_accuracy": 0.8733079850673675, "num_tokens": 72443.0, "epoch": 0.27972027972027974, "step": 30}
{"eval_loss": 0.4062638282775879, "eval_runtime": 14.9093, "eval_samples_per_second": 19.652, "eval_steps_per_second": 4.963, "eval_entropy": 0.40955733638760206, "eval_mean_token_accuracy": 0.8540234855703406, "eval_num_tokens": 72443.0, "epoch": 0.27972027972027974, "step": 30}
{"loss": 0.3867509365081787, "grad_norm": 1.9571096897125244, "learning_rate": 0.00019587628865979381, "entropy": 0.3357708005234599, "mean_token_accuracy": 0.8640182197093964, "num_tokens": 96458.0, "epoch": 0.372960372960373, "step": 40}
{"eval_loss": 0.4087931513786316, "eval_runtime": 14.8149, "eval_samples_per_second": 19.777, "eval_steps_per_second": 4.995, "eval_entropy": 0.2313703329497445, "eval_mean_token_accuracy": 0.8648878730632164, "eval_num_tokens": 96458.0, "epoch": 0.372960372960373, "step": 40}
{"loss": 0.29026432037353517, "grad_norm": 1.5169621706008911, "learning_rate": 0.00018900343642611685, "entropy": 0.22618941459804773, "mean_token_accuracy": 0.896461908519268, "num_tokens": 121019.0, "epoch": 0.4662004662004662, "step": 50}
{"eval_loss": 0.42036154866218567, "eval_runtime": 14.8036, "eval_samples_per_second": 19.792, "eval_steps_per_second": 4.999, "eval_entropy": 0.1946412901444411, "eval_mean_token_accuracy": 0.877482291008975, "eval_num_tokens": 121019.0, "epoch": 0.4662004662004662, "step": 50}
{"loss": 0.287019157409668, "grad_norm": 0.8279165625572205, "learning_rate": 0.00018213058419243986, "entropy": 0.2640441291965544, "mean_token_accuracy": 0.886538790166378, "num_tokens": 145913.0, "epoch": 0.5594405594405595, "step": 60}
{"eval_loss": 0.3168315291404724, "eval_runtime": 14.7866, "eval_samples_per_second": 19.815, "eval_steps_per_second": 5.005, "eval_entropy": 0.30197702997640985, "eval_mean_token_accuracy": 0.8974980510569908, "eval_num_tokens": 145913.0, "epoch": 0.5594405594405595, "step": 60}
{"loss": 0.21008532047271727, "grad_norm": 0.9767346978187561, "learning_rate": 0.0001752577319587629, "entropy": 0.20392372934147715, "mean_token_accuracy": 0.9273755297064781, "num_tokens": 170346.0, "epoch": 0.6526806526806527, "step": 70}
{"eval_loss": 0.31017637252807617, "eval_runtime": 14.883, "eval_samples_per_second": 19.687, "eval_steps_per_second": 4.972, "eval_entropy": 0.17668589960934744, "eval_mean_token_accuracy": 0.930160991243414, "eval_num_tokens": 170346.0, "epoch": 0.6526806526806527, "step": 70}
{"loss": 0.16980583667755128, "grad_norm": 0.9451003074645996, "learning_rate": 0.00016838487972508593, "entropy": 0.15016571674495935, "mean_token_accuracy": 0.9454508692026138, "num_tokens": 194088.0, "epoch": 0.745920745920746, "step": 80}
{"eval_loss": 0.3198226988315582, "eval_runtime": 14.8113, "eval_samples_per_second": 19.782, "eval_steps_per_second": 4.996, "eval_entropy": 0.16870883526280522, "eval_mean_token_accuracy": 0.9278164965075415, "eval_num_tokens": 194088.0, "epoch": 0.745920745920746, "step": 80}
{"loss": 0.19974710941314697, "grad_norm": 3.1258599758148193, "learning_rate": 0.00016151202749140894, "entropy": 0.1542603643145412, "mean_token_accuracy": 0.937090827524662, "num_tokens": 218492.0, "epoch": 0.8391608391608392, "step": 90}
{"eval_loss": 0.3854536712169647, "eval_runtime": 14.82, "eval_samples_per_second": 19.771, "eval_steps_per_second": 4.993, "eval_entropy": 0.09073794528061675, "eval_mean_token_accuracy": 0.944564961903804, "eval_num_tokens": 218492.0, "epoch": 0.8391608391608392, "step": 90}
{"loss": 0.13551597595214843, "grad_norm": 0.7995715737342834, "learning_rate": 0.00015463917525773197, "entropy": 0.1371931635774672, "mean_token_accuracy": 0.9686492070555687, "num_tokens": 242686.0, "epoch": 0.9324009324009324, "step": 100}
{"eval_loss": 0.2562425136566162, "eval_runtime": 14.8191, "eval_samples_per_second": 19.772, "eval_steps_per_second": 4.994, "eval_entropy": 0.12093184881755528, "eval_mean_token_accuracy": 0.9481284167315509, "eval_num_tokens": 242686.0, "epoch": 0.9324009324009324, "step": 100}
{"loss": 0.10475178956985473, "grad_norm": 1.8877465724945068, "learning_rate": 0.000147766323024055, "entropy": 0.07703856018850126, "mean_token_accuracy": 0.9767271022538881, "num_tokens": 264588.0, "epoch": 1.0186480186480187, "step": 110}
{"eval_loss": 0.3074987828731537, "eval_runtime": 14.7718, "eval_samples_per_second": 19.835, "eval_steps_per_second": 5.01, "eval_entropy": 0.050238378292478215, "eval_mean_token_accuracy": 0.9503478577008119, "eval_num_tokens": 264588.0, "epoch": 1.0186480186480187, "step": 110}
{"loss": 0.09583389163017272, "grad_norm": 1.6458184719085693, "learning_rate": 0.00014089347079037802, "entropy": 0.055893574841320515, "mean_token_accuracy": 0.9735625043511391, "num_tokens": 288874.0, "epoch": 1.1118881118881119, "step": 120}
{"eval_loss": 0.32906386256217957, "eval_runtime": 14.8547, "eval_samples_per_second": 19.724, "eval_steps_per_second": 4.982, "eval_entropy": 0.07678984788661772, "eval_mean_token_accuracy": 0.940730202842403, "eval_num_tokens": 288874.0, "epoch": 1.1118881118881119, "step": 120}
{"loss": 0.0568264901638031, "grad_norm": 0.6258577704429626, "learning_rate": 0.00013402061855670103, "entropy": 0.07421652502380312, "mean_token_accuracy": 0.9796055227518081, "num_tokens": 313305.0, "epoch": 1.205128205128205, "step": 130}
{"eval_loss": 0.2880438566207886, "eval_runtime": 14.8513, "eval_samples_per_second": 19.729, "eval_steps_per_second": 4.983, "eval_entropy": 0.08602434895127206, "eval_mean_token_accuracy": 0.9458578003419412, "eval_num_tokens": 313305.0, "epoch": 1.205128205128205, "step": 130}
{"loss": 0.05998890995979309, "grad_norm": 0.3214556574821472, "learning_rate": 0.00012714776632302406, "entropy": 0.053056696022395045, "mean_token_accuracy": 0.9839832872152329, "num_tokens": 337254.0, "epoch": 1.2983682983682985, "step": 140}
{"eval_loss": 0.32322099804878235, "eval_runtime": 14.7903, "eval_samples_per_second": 19.81, "eval_steps_per_second": 5.003, "eval_entropy": 0.04204791218570962, "eval_mean_token_accuracy": 0.9487286500028662, "eval_num_tokens": 337254.0, "epoch": 1.2983682983682985, "step": 140}
{"loss": 0.05625233054161072, "grad_norm": 0.3622892200946808, "learning_rate": 0.00012027491408934708, "entropy": 0.05295453425496817, "mean_token_accuracy": 0.980704988539219, "num_tokens": 361336.0, "epoch": 1.3916083916083917, "step": 150}
{"eval_loss": 0.24802015721797943, "eval_runtime": 14.7684, "eval_samples_per_second": 19.84, "eval_steps_per_second": 5.011, "eval_entropy": 0.07800518003342724, "eval_mean_token_accuracy": 0.9608748522964684, "eval_num_tokens": 361336.0, "epoch": 1.3916083916083917, "step": 150}
{"loss": 0.0391657292842865, "grad_norm": 0.720223605632782, "learning_rate": 0.0001134020618556701, "entropy": 0.06143993827572558, "mean_token_accuracy": 0.985044564306736, "num_tokens": 385802.0, "epoch": 1.4848484848484849, "step": 160}
{"eval_loss": 0.3093143105506897, "eval_runtime": 14.8478, "eval_samples_per_second": 19.734, "eval_steps_per_second": 4.984, "eval_entropy": 0.038675374506400094, "eval_mean_token_accuracy": 0.9596509385753322, "eval_num_tokens": 385802.0, "epoch": 1.4848484848484849, "step": 160}
{"loss": 0.059267479181289676, "grad_norm": 0.9854781627655029, "learning_rate": 0.00010652920962199313, "entropy": 0.02847423480998259, "mean_token_accuracy": 0.9850253105163574, "num_tokens": 409572.0, "epoch": 1.578088578088578, "step": 170}
{"eval_loss": 0.30544501543045044, "eval_runtime": 14.7905, "eval_samples_per_second": 19.81, "eval_steps_per_second": 5.003, "eval_entropy": 0.0350318578358211, "eval_mean_token_accuracy": 0.960938129876111, "eval_num_tokens": 409572.0, "epoch": 1.578088578088578, "step": 170}
{"loss": 0.0478329598903656, "grad_norm": 0.13517993688583374, "learning_rate": 9.965635738831616e-05, "entropy": 0.03786912175710313, "mean_token_accuracy": 0.9916273951530457, "num_tokens": 434206.0, "epoch": 1.6713286713286712, "step": 180}
{"eval_loss": 0.25706955790519714, "eval_runtime": 14.7743, "eval_samples_per_second": 19.832, "eval_steps_per_second": 5.009, "eval_entropy": 0.04643009695841151, "eval_mean_token_accuracy": 0.9605980982651582, "eval_num_tokens": 434206.0, "epoch": 1.6713286713286712, "step": 180}
{"loss": 0.010271672159433365, "grad_norm": 0.2743983864784241, "learning_rate": 9.278350515463918e-05, "entropy": 0.03550994105171412, "mean_token_accuracy": 0.9980769231915474, "num_tokens": 458497.0, "epoch": 1.7645687645687644, "step": 190}
{"eval_loss": 0.2718251645565033, "eval_runtime": 14.7882, "eval_samples_per_second": 19.813, "eval_steps_per_second": 5.004, "eval_entropy": 0.028795436087593034, "eval_mean_token_accuracy": 0.9613220361438958, "eval_num_tokens": 458497.0, "epoch": 1.7645687645687644, "step": 190}
{"loss": 0.0035189926624298097, "grad_norm": 0.005196348764002323, "learning_rate": 8.591065292096219e-05, "entropy": 0.01757364360964857, "mean_token_accuracy": 1.0, "num_tokens": 482875.0, "epoch": 1.8578088578088578, "step": 200}
{"eval_loss": 0.2962808907032013, "eval_runtime": 14.7948, "eval_samples_per_second": 19.804, "eval_steps_per_second": 5.002, "eval_entropy": 0.01331983832642436, "eval_mean_token_accuracy": 0.9613220361438958, "eval_num_tokens": 482875.0, "epoch": 1.8578088578088578, "step": 200}
{"loss": 0.0013927843421697617, "grad_norm": 0.010021727532148361, "learning_rate": 7.903780068728523e-05, "entropy": 0.009183780805324204, "mean_token_accuracy": 1.0, "num_tokens": 507236.0, "epoch": 1.951048951048951, "step": 210}
{"eval_loss": 0.31927573680877686, "eval_runtime": 14.7674, "eval_samples_per_second": 19.841, "eval_steps_per_second": 5.011, "eval_entropy": 0.0074244596679990706, "eval_mean_token_accuracy": 0.9613220361438958, "eval_num_tokens": 507236.0, "epoch": 1.951048951048951, "step": 210}
{"loss": 0.01904604434967041, "grad_norm": 0.006378693040460348, "learning_rate": 7.216494845360825e-05, "entropy": 0.005078906492279792, "mean_token_accuracy": 0.994152806900643, "num_tokens": 529634.0, "epoch": 2.0372960372960374, "step": 220}
{"eval_loss": 0.3192271888256073, "eval_runtime": 14.7984, "eval_samples_per_second": 19.799, "eval_steps_per_second": 5.001, "eval_entropy": 0.00774541417352314, "eval_mean_token_accuracy": 0.9613220361438958, "eval_num_tokens": 529634.0, "epoch": 2.0372960372960374, "step": 220}
{"loss": 0.00627545565366745, "grad_norm": 0.25158587098121643, "learning_rate": 6.529209621993127e-05, "entropy": 0.00744854452641448, "mean_token_accuracy": 0.9970833331346511, "num_tokens": 553736.0, "epoch": 2.1305361305361306, "step": 230}
{"eval_loss": 0.32524561882019043, "eval_runtime": 14.7985, "eval_samples_per_second": 19.799, "eval_steps_per_second": 5.001, "eval_entropy": 0.006779258040321135, "eval_mean_token_accuracy": 0.9613220361438958, "eval_num_tokens": 553736.0, "epoch": 2.1305361305361306, "step": 230}
{"loss": 0.017633379995822908, "grad_norm": 0.0020345281809568405, "learning_rate": 5.84192439862543e-05, "entropy": 0.004252842843561666, "mean_token_accuracy": 0.9972463771700859, "num_tokens": 578069.0, "epoch": 2.2237762237762237, "step": 240}
{"eval_loss": 0.2943844795227051, "eval_runtime": 14.8027, "eval_samples_per_second": 19.794, "eval_steps_per_second": 4.999, "eval_entropy": 0.008602032205089927, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 578069.0, "epoch": 2.2237762237762237, "step": 240}
{"loss": 0.0023283913731575014, "grad_norm": 0.18994489312171936, "learning_rate": 5.1546391752577315e-05, "entropy": 0.006335099332500249, "mean_token_accuracy": 0.9982142850756646, "num_tokens": 602281.0, "epoch": 2.317016317016317, "step": 250}
{"eval_loss": 0.2887563407421112, "eval_runtime": 14.811, "eval_samples_per_second": 19.783, "eval_steps_per_second": 4.996, "eval_entropy": 0.008193284760833391, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 602281.0, "epoch": 2.317016317016317, "step": 250}
{"loss": 0.004103139415383339, "grad_norm": 0.003211820963770151, "learning_rate": 4.467353951890035e-05, "entropy": 0.006146625852852594, "mean_token_accuracy": 0.9988095238804817, "num_tokens": 626574.0, "epoch": 2.41025641025641, "step": 260}
{"eval_loss": 0.29113122820854187, "eval_runtime": 15.0181, "eval_samples_per_second": 19.51, "eval_steps_per_second": 4.927, "eval_entropy": 0.0076493095367442115, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 626574.0, "epoch": 2.41025641025641, "step": 260}
{"loss": 0.00815114751458168, "grad_norm": 0.15567170083522797, "learning_rate": 3.7800687285223366e-05, "entropy": 0.007643775080214255, "mean_token_accuracy": 0.993711519241333, "num_tokens": 651202.0, "epoch": 2.5034965034965033, "step": 270}
{"eval_loss": 0.29471999406814575, "eval_runtime": 14.7985, "eval_samples_per_second": 19.799, "eval_steps_per_second": 5.001, "eval_entropy": 0.0072461218192443454, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 651202.0, "epoch": 2.5034965034965033, "step": 270}
{"loss": 0.0038479555398225783, "grad_norm": 0.0905834510922432, "learning_rate": 3.0927835051546395e-05, "entropy": 0.006777999275800539, "mean_token_accuracy": 0.9980769231915474, "num_tokens": 675541.0, "epoch": 2.596736596736597, "step": 280}
{"eval_loss": 0.2980296015739441, "eval_runtime": 14.7948, "eval_samples_per_second": 19.804, "eval_steps_per_second": 5.002, "eval_entropy": 0.006837818584785596, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 675541.0, "epoch": 2.596736596736597, "step": 280}
{"loss": 0.0002809752942994237, "grad_norm": 0.0019444398349151015, "learning_rate": 2.4054982817869417e-05, "entropy": 0.0026021723504527474, "mean_token_accuracy": 1.0, "num_tokens": 699189.0, "epoch": 2.6899766899766897, "step": 290}
{"eval_loss": 0.30105510354042053, "eval_runtime": 14.8014, "eval_samples_per_second": 19.795, "eval_steps_per_second": 5.0, "eval_entropy": 0.006540719075584659, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 699189.0, "epoch": 2.6899766899766897, "step": 290}
{"loss": 0.006621778756380081, "grad_norm": 0.004040197003632784, "learning_rate": 1.7182130584192442e-05, "entropy": 0.00866860578971682, "mean_token_accuracy": 0.9939772740006447, "num_tokens": 723617.0, "epoch": 2.7832167832167833, "step": 300}
{"eval_loss": 0.3028668463230133, "eval_runtime": 14.7672, "eval_samples_per_second": 19.841, "eval_steps_per_second": 5.011, "eval_entropy": 0.006431365003368842, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 723617.0, "epoch": 2.7832167832167833, "step": 300}
{"loss": 0.003116745501756668, "grad_norm": 0.005858445540070534, "learning_rate": 1.0309278350515464e-05, "entropy": 0.004151920105505269, "mean_token_accuracy": 0.9978758156299591, "num_tokens": 747712.0, "epoch": 2.8764568764568765, "step": 310}
{"eval_loss": 0.30419182777404785, "eval_runtime": 14.8547, "eval_samples_per_second": 19.724, "eval_steps_per_second": 4.982, "eval_entropy": 0.006277887287511918, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 747712.0, "epoch": 2.8764568764568765, "step": 310}
{"loss": 0.004541835561394691, "grad_norm": 0.0031163643579930067, "learning_rate": 3.4364261168384886e-06, "entropy": 0.0065833306376589466, "mean_token_accuracy": 1.0, "num_tokens": 772216.0, "epoch": 2.9696969696969697, "step": 320}
{"eval_loss": 0.30400803685188293, "eval_runtime": 14.7578, "eval_samples_per_second": 19.854, "eval_steps_per_second": 5.014, "eval_entropy": 0.006217476139411431, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 772216.0, "epoch": 2.9696969696969697, "step": 320}
{"eval_loss": 0.3050905466079712, "eval_runtime": 14.8448, "eval_samples_per_second": 19.738, "eval_steps_per_second": 4.985, "eval_entropy": 0.006229618625602106, "eval_mean_token_accuracy": 0.9622872874543473, "eval_num_tokens": 779835.0, "epoch": 3.0, "step": 324}
{"train_runtime": 1498.5878, "train_samples_per_second": 3.433, "train_steps_per_second": 0.216, "total_flos": 1.4256559975035802e+17, "train_loss": 0.1077416844753675, "epoch": 3.0, "step": 324}
| Package | Version |
|---|---|
| torch | 2.14.0 |
| transformers | 5.17.0 |
| trl | 1.13.0 |
| datasets | 5.0.1 |
| accelerate | 1.15.0 |
| python-dotenv | 1.2.3 |
| peft | 0.21.0 |
| bitsandbytes | 0.50.2 |
| huggingface-hub | ? |
| jinja2 | ? |
| torchvision | 0.29.0 |
| pillow | 12.3.0 |
This adapter is intended for research purposes only as part of the AIML589 project, which investigates value alignment of LLMs with New Zealand population distributions from the World Values Survey.
This model has not been safety-tuned for general-purpose deployment. It should not be used in production systems, for making decisions about people, or in contexts where reliability and safety are critical.