Downloads · 30 days
6
15% of all-time downloads
1jamesthompson1/Qwen3.6-27B-nz-wvs-distributional-overall
Qwen3.6-27B-nz-wvs-distributional-overall is a text generation model from 1jamesthompson1. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as cc-by-sa-4.0.
This model is a LoRA fine-tune of Qwen/Qwen3.6-27B as part of the AIML589 project.
Downloads · 30 days
6
15% of all-time downloads
All-time downloads
41
Public
Repo size
339 MB
Likes
0
Public
Click a slice to open those files.
.safetensors319 MB · 94%
From the Hugging Face model README
This model is a LoRA fine-tune of Qwen/Qwen3.6-27B as part of the AIML589 project.
This adapter is licensed under CC BY-SA 4.0.
Fine-tuned on the distributional config of the wvs-nz-value-alignment dataset, overall subpopulation.
Part of the wvs-nz-value-alignment collection.
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition · Training time: 40m 32s
| Parameter | Value |
|---|---|
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| DoRA | False |
| Learning rate | 0.0002 |
| Batch size | 1 |
| Gradient accumulation | 4 |
| Epochs | 1 |
| Max seq length | 1024 |
| Warmup ratio | 0.1 |
| Dtype | bf16 |
{"loss": 12.719064331054687, "grad_norm": 237.13966369628906, "learning_rate": 6e-05, "epoch": 0.03347280334728033, "step": 10}
{"eval_loss": 2.796537160873413, "eval_runtime": 19.5157, "eval_samples_per_second": 3.074, "eval_steps_per_second": 3.074, "epoch": 0.03347280334728033, "step": 10}
{"loss": 9.476187133789063, "grad_norm": 69.21638488769531, "learning_rate": 0.00012666666666666666, "epoch": 0.06694560669456066, "step": 20}
{"eval_loss": 2.093045473098755, "eval_runtime": 19.8803, "eval_samples_per_second": 3.018, "eval_steps_per_second": 3.018, "epoch": 0.06694560669456066, "step": 20}
{"loss": 8.532209777832032, "grad_norm": 81.49916076660156, "learning_rate": 0.00019333333333333333, "epoch": 0.100418410041841, "step": 30}
{"eval_loss": 1.7366498708724976, "eval_runtime": 19.9356, "eval_samples_per_second": 3.01, "eval_steps_per_second": 3.01, "epoch": 0.100418410041841, "step": 30}
{"loss": 7.217524719238281, "grad_norm": 48.18870544433594, "learning_rate": 0.0001933085501858736, "epoch": 0.13389121338912133, "step": 40}
{"eval_loss": 1.5172970294952393, "eval_runtime": 20.0397, "eval_samples_per_second": 2.994, "eval_steps_per_second": 2.994, "epoch": 0.13389121338912133, "step": 40}
{"loss": 6.041807556152344, "grad_norm": 12.789166450500488, "learning_rate": 0.0001858736059479554, "epoch": 0.16736401673640167, "step": 50}
{"eval_loss": 1.4546949863433838, "eval_runtime": 19.9829, "eval_samples_per_second": 3.003, "eval_steps_per_second": 3.003, "epoch": 0.16736401673640167, "step": 50}
{"loss": 5.424972915649414, "grad_norm": 3.4070775508880615, "learning_rate": 0.0001784386617100372, "epoch": 0.200836820083682, "step": 60}
{"eval_loss": 1.685111403465271, "eval_runtime": 19.9721, "eval_samples_per_second": 3.004, "eval_steps_per_second": 3.004, "epoch": 0.200836820083682, "step": 60}
{"loss": 5.7734016418457035, "grad_norm": 9.858166694641113, "learning_rate": 0.00017100371747211896, "epoch": 0.23430962343096234, "step": 70}
{"eval_loss": 1.5116097927093506, "eval_runtime": 19.9987, "eval_samples_per_second": 3.0, "eval_steps_per_second": 3.0, "epoch": 0.23430962343096234, "step": 70}
{"loss": 6.098352432250977, "grad_norm": 8.922426223754883, "learning_rate": 0.00016356877323420074, "epoch": 0.26778242677824265, "step": 80}
{"eval_loss": 1.3551506996154785, "eval_runtime": 19.9828, "eval_samples_per_second": 3.003, "eval_steps_per_second": 3.003, "epoch": 0.26778242677824265, "step": 80}
{"loss": 5.898929214477539, "grad_norm": 43.54411697387695, "learning_rate": 0.00015613382899628254, "epoch": 0.301255230125523, "step": 90}
{"eval_loss": 1.4195972681045532, "eval_runtime": 20.0115, "eval_samples_per_second": 2.998, "eval_steps_per_second": 2.998, "epoch": 0.301255230125523, "step": 90}
{"loss": 5.286152267456055, "grad_norm": 6.386129856109619, "learning_rate": 0.00014869888475836432, "epoch": 0.33472803347280333, "step": 100}
{"eval_loss": 1.411828875541687, "eval_runtime": 19.9415, "eval_samples_per_second": 3.009, "eval_steps_per_second": 3.009, "epoch": 0.33472803347280333, "step": 100}
{"loss": 4.903972625732422, "grad_norm": 3.99894380569458, "learning_rate": 0.00014126394052044612, "epoch": 0.3682008368200837, "step": 110}
{"eval_loss": 1.3494559526443481, "eval_runtime": 19.9446, "eval_samples_per_second": 3.008, "eval_steps_per_second": 3.008, "epoch": 0.3682008368200837, "step": 110}
{"loss": 5.159200286865234, "grad_norm": 8.32929515838623, "learning_rate": 0.0001338289962825279, "epoch": 0.401673640167364, "step": 120}
{"eval_loss": 1.3670872449874878, "eval_runtime": 19.9839, "eval_samples_per_second": 3.002, "eval_steps_per_second": 3.002, "epoch": 0.401673640167364, "step": 120}
{"loss": 5.456905364990234, "grad_norm": 6.040786266326904, "learning_rate": 0.00012639405204460967, "epoch": 0.4351464435146444, "step": 130}
{"eval_loss": 1.3503260612487793, "eval_runtime": 19.9969, "eval_samples_per_second": 3.0, "eval_steps_per_second": 3.0, "epoch": 0.4351464435146444, "step": 130}
{"loss": 5.228645706176758, "grad_norm": 7.370626449584961, "learning_rate": 0.00011895910780669145, "epoch": 0.4686192468619247, "step": 140}
{"eval_loss": 1.3124669790267944, "eval_runtime": 19.9737, "eval_samples_per_second": 3.004, "eval_steps_per_second": 3.004, "epoch": 0.4686192468619247, "step": 140}
{"loss": 4.8088031768798825, "grad_norm": 6.18451452255249, "learning_rate": 0.00011152416356877324, "epoch": 0.502092050209205, "step": 150}
{"eval_loss": 1.327501654624939, "eval_runtime": 19.9367, "eval_samples_per_second": 3.01, "eval_steps_per_second": 3.01, "epoch": 0.502092050209205, "step": 150}
{"loss": 4.949464416503906, "grad_norm": 3.5054779052734375, "learning_rate": 0.00010408921933085503, "epoch": 0.5355648535564853, "step": 160}
{"eval_loss": 1.3091278076171875, "eval_runtime": 19.989, "eval_samples_per_second": 3.002, "eval_steps_per_second": 3.002, "epoch": 0.5355648535564853, "step": 160}
{"loss": 5.220246505737305, "grad_norm": 14.068228721618652, "learning_rate": 9.66542750929368e-05, "epoch": 0.5690376569037657, "step": 170}
{"eval_loss": 1.340369701385498, "eval_runtime": 19.977, "eval_samples_per_second": 3.003, "eval_steps_per_second": 3.003, "epoch": 0.5690376569037657, "step": 170}
{"loss": 4.70911979675293, "grad_norm": 9.41586971282959, "learning_rate": 8.92193308550186e-05, "epoch": 0.602510460251046, "step": 180}
{"eval_loss": 1.313641905784607, "eval_runtime": 19.9696, "eval_samples_per_second": 3.005, "eval_steps_per_second": 3.005, "epoch": 0.602510460251046, "step": 180}
{"loss": 4.994095993041992, "grad_norm": 3.8128387928009033, "learning_rate": 8.178438661710037e-05, "epoch": 0.6359832635983264, "step": 190}
{"eval_loss": 1.289259672164917, "eval_runtime": 19.9782, "eval_samples_per_second": 3.003, "eval_steps_per_second": 3.003, "epoch": 0.6359832635983264, "step": 190}
{"loss": 4.66058464050293, "grad_norm": 5.796242713928223, "learning_rate": 7.434944237918216e-05, "epoch": 0.6694560669456067, "step": 200}
{"eval_loss": 1.309515118598938, "eval_runtime": 19.9232, "eval_samples_per_second": 3.012, "eval_steps_per_second": 3.012, "epoch": 0.6694560669456067, "step": 200}
{"loss": 4.188463973999023, "grad_norm": 2.148908853530884, "learning_rate": 6.691449814126395e-05, "epoch": 0.702928870292887, "step": 210}
{"eval_loss": 1.3056697845458984, "eval_runtime": 19.9293, "eval_samples_per_second": 3.011, "eval_steps_per_second": 3.011, "epoch": 0.702928870292887, "step": 210}
{"loss": 4.79797248840332, "grad_norm": 11.551894187927246, "learning_rate": 5.947955390334573e-05, "epoch": 0.7364016736401674, "step": 220}
{"eval_loss": 1.3100857734680176, "eval_runtime": 19.9855, "eval_samples_per_second": 3.002, "eval_steps_per_second": 3.002, "epoch": 0.7364016736401674, "step": 220}
{"loss": 4.88032341003418, "grad_norm": 2.8422751426696777, "learning_rate": 5.2044609665427516e-05, "epoch": 0.7698744769874477, "step": 230}
{"eval_loss": 1.3162628412246704, "eval_runtime": 19.9525, "eval_samples_per_second": 3.007, "eval_steps_per_second": 3.007, "epoch": 0.7698744769874477, "step": 230}
{"loss": 4.761834335327149, "grad_norm": 3.423215866088867, "learning_rate": 4.46096654275093e-05, "epoch": 0.803347280334728, "step": 240}
{"eval_loss": 1.3069977760314941, "eval_runtime": 19.9842, "eval_samples_per_second": 3.002, "eval_steps_per_second": 3.002, "epoch": 0.803347280334728, "step": 240}
{"loss": 5.531649017333985, "grad_norm": 3.182314395904541, "learning_rate": 3.717472118959108e-05, "epoch": 0.8368200836820083, "step": 250}
{"eval_loss": 1.3062748908996582, "eval_runtime": 19.9961, "eval_samples_per_second": 3.001, "eval_steps_per_second": 3.001, "epoch": 0.8368200836820083, "step": 250}
{"loss": 4.2216796875, "grad_norm": 2.8161354064941406, "learning_rate": 2.9739776951672864e-05, "epoch": 0.8702928870292888, "step": 260}
{"eval_loss": 1.2968310117721558, "eval_runtime": 19.9441, "eval_samples_per_second": 3.008, "eval_steps_per_second": 3.008, "epoch": 0.8702928870292888, "step": 260}
{"loss": 4.926880645751953, "grad_norm": 3.4710750579833984, "learning_rate": 2.230483271375465e-05, "epoch": 0.9037656903765691, "step": 270}
{"eval_loss": 1.2930675745010376, "eval_runtime": 19.9583, "eval_samples_per_second": 3.006, "eval_steps_per_second": 3.006, "epoch": 0.9037656903765691, "step": 270}
{"loss": 4.571000671386718, "grad_norm": 2.3111588954925537, "learning_rate": 1.4869888475836432e-05, "epoch": 0.9372384937238494, "step": 280}
{"eval_loss": 1.2904328107833862, "eval_runtime": 19.9924, "eval_samples_per_second": 3.001, "eval_steps_per_second": 3.001, "epoch": 0.9372384937238494, "step": 280}
{"loss": 4.576707077026367, "grad_norm": 3.230638027191162, "learning_rate": 7.434944237918216e-06, "epoch": 0.9707112970711297, "step": 290}
{"eval_loss": 1.2909356355667114, "eval_runtime": 19.965, "eval_samples_per_second": 3.005, "eval_steps_per_second": 3.005, "epoch": 0.9707112970711297, "step": 290}
{"eval_loss": 1.289475679397583, "eval_runtime": 19.9421, "eval_samples_per_second": 3.009, "eval_steps_per_second": 3.009, "epoch": 1.0, "step": 299}
{"train_runtime": 2431.9778, "train_samples_per_second": 0.491, "train_steps_per_second": 0.123, "total_flos": 2.1335991385826304e+17, "train_loss": 5.653244146136536, "epoch": 1.0, "step": 299}
| Package | Version |
|---|---|
| torch | 2.13.0 |
| transformers | 5.14.1 |
| trl | 1.9.2 |
| datasets | 5.0.1 |
| accelerate | 1.14.0 |
| python-dotenv | 1.2.2 |
| peft | 0.20.0 |
| bitsandbytes | 0.50.0 |
| huggingface-hub | ? |
| jinja2 | ? |
| torchvision | 0.28.0 |
| pillow | 12.3.0 |
This adapter is intended for research purposes only as part of the AIML589 project, which investigates value alignment of LLMs with New Zealand population distributions from the World Values Survey.
This model has not been safety-tuned for general-purpose deployment. It should not be used in production systems, for making decisions about people, or in contexts where reliability and safety are critical.