Downloads · 30 days
56
30% of all-time downloads
1jamesthompson1/Qwen3.8-27B-nz-wvs-modal_response-overall
Qwen3.8-27B-nz-wvs-modal_response-overall is a text generation model from 1jamesthompson1. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as cc-by-sa-4.0.
This model is a LoRA fine-tune of Qwen/Qwen3.8-27B as part of the AIML589 project.
Downloads · 30 days
56
30% of all-time downloads
All-time downloads
184
Public
Repo size
19.1 GB
Likes
0
Public
Click a slice to open those files.
.pt2.6 GB · 66%
From the Hugging Face model README
This model is a LoRA fine-tune of Qwen/Qwen3.8-27B as part of the AIML589 project.
This adapter is licensed under CC BY-SA 4.0.
Fine-tuned on the modal_response config of the wvs-nz-value-alignment dataset, overall subpopulation.
Part of the wvs-nz-value-alignment collection.
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition · Training time: 24m 59s
The exact command used to produce this adapter (also saved in
finetune_config.json):
finetune.py --model Qwen/Qwen3.8-27B --dataset modal_response --subpopulation overall --upload-to-hf --lora-r 64 --lora-alpha 128
| Parameter | Value |
|---|---|
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA dropout | 0.05 |
| DoRA | False |
| Learning rate | 0.0002 |
| Batch size | 4 |
| Gradient accumulation | 4 |
| Epochs | 3 |
| Max seq length | 1024 |
| Warmup ratio | |
| Dtype | bf16 |
| Trainable parameters (LoRA) | 318,767,104 |
| Adapter size on disk | 1,275,137,464 bytes |
{"loss": 0.311386251449585, "grad_norm": 2.521930456161499, "learning_rate": 5.4545454545454546e-05, "entropy": 0.29465388171374796, "mean_token_accuracy": 0.897952838242054, "num_tokens": 24259.0, "epoch": 0.09324009324009325, "step": 10}
{"eval_loss": 0.2637631595134735, "eval_runtime": 14.4616, "eval_samples_per_second": 20.26, "eval_steps_per_second": 5.117, "eval_entropy": 0.24888743528139753, "eval_mean_token_accuracy": 0.8988941545422012, "eval_num_tokens": 24259.0, "epoch": 0.09324009324009325, "step": 10}
{"loss": 0.19836851358413696, "grad_norm": 1.3360661268234253, "learning_rate": 0.00011515151515151516, "entropy": 0.18984935861080884, "mean_token_accuracy": 0.9322423577308655, "num_tokens": 48291.0, "epoch": 0.1864801864801865, "step": 20}
{"eval_loss": 0.15858271718025208, "eval_runtime": 14.6914, "eval_samples_per_second": 19.944, "eval_steps_per_second": 5.037, "eval_entropy": 0.1868036614126853, "eval_mean_token_accuracy": 0.9399007985720763, "eval_num_tokens": 48291.0, "epoch": 0.1864801864801865, "step": 20}
{"loss": 0.19112024307250977, "grad_norm": 2.3528544902801514, "learning_rate": 0.00017575757575757578, "entropy": 0.1536058599129319, "mean_token_accuracy": 0.9442664995789528, "num_tokens": 72409.0, "epoch": 0.27972027972027974, "step": 30}
{"eval_loss": 0.24914467334747314, "eval_runtime": 14.7916, "eval_samples_per_second": 19.809, "eval_steps_per_second": 5.003, "eval_entropy": 0.2438809067967373, "eval_mean_token_accuracy": 0.8973690247213518, "eval_num_tokens": 72409.0, "epoch": 0.27972027972027974, "step": 30}
{"loss": 0.21157667636871338, "grad_norm": 1.2557505369186401, "learning_rate": 0.00019587628865979381, "entropy": 0.1947059162892401, "mean_token_accuracy": 0.9232663199305534, "num_tokens": 96423.0, "epoch": 0.372960372960373, "step": 40}
{"eval_loss": 0.18457981944084167, "eval_runtime": 14.8444, "eval_samples_per_second": 19.738, "eval_steps_per_second": 4.985, "eval_entropy": 0.143733036691776, "eval_mean_token_accuracy": 0.9382097568060901, "eval_num_tokens": 96423.0, "epoch": 0.372960372960373, "step": 40}
{"loss": 0.1572788119316101, "grad_norm": 1.0724738836288452, "learning_rate": 0.00018900343642611685, "entropy": 0.05951508327125339, "mean_token_accuracy": 0.9632913634181023, "num_tokens": 120955.0, "epoch": 0.4662004662004662, "step": 50}
{"eval_loss": 0.26251375675201416, "eval_runtime": 14.7761, "eval_samples_per_second": 19.829, "eval_steps_per_second": 5.008, "eval_entropy": 0.07560718972259516, "eval_mean_token_accuracy": 0.9333748591912759, "eval_num_tokens": 120955.0, "epoch": 0.4662004662004662, "step": 50}
{"loss": 0.1685749888420105, "grad_norm": 0.214590921998024, "learning_rate": 0.00018213058419243986, "entropy": 0.14221295416355134, "mean_token_accuracy": 0.9503799468278885, "num_tokens": 145839.0, "epoch": 0.5594405594405595, "step": 60}
{"eval_loss": 0.28627634048461914, "eval_runtime": 14.8095, "eval_samples_per_second": 19.785, "eval_steps_per_second": 4.997, "eval_entropy": 0.10762930513560973, "eval_mean_token_accuracy": 0.9256700788800781, "eval_num_tokens": 145839.0, "epoch": 0.5594405594405595, "step": 60}
{"loss": 0.16093504428863525, "grad_norm": 2.0018250942230225, "learning_rate": 0.0001752577319587629, "entropy": 0.09020606367848813, "mean_token_accuracy": 0.9607993304729462, "num_tokens": 170269.0, "epoch": 0.6526806526806527, "step": 70}
{"eval_loss": 0.28439241647720337, "eval_runtime": 14.7706, "eval_samples_per_second": 19.837, "eval_steps_per_second": 5.01, "eval_entropy": 0.07406007137615234, "eval_mean_token_accuracy": 0.9511243628489005, "eval_num_tokens": 170269.0, "epoch": 0.6526806526806527, "step": 70}
{"loss": 0.18680245876312257, "grad_norm": 1.0350223779678345, "learning_rate": 0.00016838487972508593, "entropy": 0.146643881755881, "mean_token_accuracy": 0.9481542229652404, "num_tokens": 193998.0, "epoch": 0.745920745920746, "step": 80}
{"eval_loss": 0.1855505257844925, "eval_runtime": 14.7621, "eval_samples_per_second": 19.848, "eval_steps_per_second": 5.013, "eval_entropy": 0.1913612814670479, "eval_mean_token_accuracy": 0.9328578074236173, "eval_num_tokens": 193998.0, "epoch": 0.745920745920746, "step": 80}
{"loss": 0.08664531707763672, "grad_norm": 2.7835447788238525, "learning_rate": 0.00016151202749140894, "entropy": 0.07072625445434824, "mean_token_accuracy": 0.9753257662057877, "num_tokens": 218387.0, "epoch": 0.8391608391608392, "step": 90}
{"eval_loss": 0.19302597641944885, "eval_runtime": 14.7697, "eval_samples_per_second": 19.838, "eval_steps_per_second": 5.01, "eval_entropy": 0.05069159566443665, "eval_mean_token_accuracy": 0.9460081301025443, "eval_num_tokens": 218387.0, "epoch": 0.8391608391608392, "step": 90}
{"loss": 0.04949827790260315, "grad_norm": 0.3366875946521759, "learning_rate": 0.00015463917525773197, "entropy": 0.02573756533674896, "mean_token_accuracy": 0.9858695268630981, "num_tokens": 242551.0, "epoch": 0.9324009324009324, "step": 100}
{"eval_loss": 0.17919547855854034, "eval_runtime": 14.7624, "eval_samples_per_second": 19.848, "eval_steps_per_second": 5.013, "eval_entropy": 0.04190207606439504, "eval_mean_token_accuracy": 0.9704701095014006, "eval_num_tokens": 242551.0, "epoch": 0.9324009324009324, "step": 100}
{"loss": 0.09908731579780579, "grad_norm": 1.3010486364364624, "learning_rate": 0.000147766323024055, "entropy": 0.05294840840144536, "mean_token_accuracy": 0.9763124279073767, "num_tokens": 264446.0, "epoch": 1.0186480186480187, "step": 110}
{"eval_loss": 0.12856638431549072, "eval_runtime": 14.7534, "eval_samples_per_second": 19.86, "eval_steps_per_second": 5.016, "eval_entropy": 0.09720448612839587, "eval_mean_token_accuracy": 0.961386025757403, "eval_num_tokens": 264446.0, "epoch": 1.0186480186480187, "step": 110}
{"loss": 0.04380224645137787, "grad_norm": 1.1750184297561646, "learning_rate": 0.00014089347079037802, "entropy": 0.05420322470599785, "mean_token_accuracy": 0.9894749253988266, "num_tokens": 288715.0, "epoch": 1.1118881118881119, "step": 120}
{"eval_loss": 0.1058899462223053, "eval_runtime": 14.7558, "eval_samples_per_second": 19.857, "eval_steps_per_second": 5.015, "eval_entropy": 0.07736008003907832, "eval_mean_token_accuracy": 0.9744524053625159, "eval_num_tokens": 288715.0, "epoch": 1.1118881118881119, "step": 120}
{"loss": 0.06110140085220337, "grad_norm": 0.17783814668655396, "learning_rate": 0.00013402061855670103, "entropy": 0.046813498996198175, "mean_token_accuracy": 0.9883611395955085, "num_tokens": 313121.0, "epoch": 1.205128205128205, "step": 130}
{"eval_loss": 0.09259921312332153, "eval_runtime": 14.7402, "eval_samples_per_second": 19.878, "eval_steps_per_second": 5.02, "eval_entropy": 0.0653646263023335, "eval_mean_token_accuracy": 0.9791147741111549, "eval_num_tokens": 313121.0, "epoch": 1.205128205128205, "step": 130}
{"loss": 0.028206560015678405, "grad_norm": 0.8817558288574219, "learning_rate": 0.00012714776632302406, "entropy": 0.03069150827359408, "mean_token_accuracy": 0.9905599728226662, "num_tokens": 337076.0, "epoch": 1.2983682983682985, "step": 140}
{"eval_loss": 0.09077504277229309, "eval_runtime": 14.8897, "eval_samples_per_second": 19.678, "eval_steps_per_second": 4.97, "eval_entropy": 0.06336608863391636, "eval_mean_token_accuracy": 0.9658661758577501, "eval_num_tokens": 337076.0, "epoch": 1.2983682983682985, "step": 140}
{"loss": 0.046619457006454465, "grad_norm": 1.1646716594696045, "learning_rate": 0.00012027491408934708, "entropy": 0.05501706254435703, "mean_token_accuracy": 0.9831309720873833, "num_tokens": 361147.0, "epoch": 1.3916083916083917, "step": 150}
{"eval_loss": 0.06325710564851761, "eval_runtime": 14.8168, "eval_samples_per_second": 19.775, "eval_steps_per_second": 4.994, "eval_entropy": 0.04740549160114087, "eval_mean_token_accuracy": 0.9821525272485372, "eval_num_tokens": 361147.0, "epoch": 1.3916083916083917, "step": 150}
{"loss": 0.0544526219367981, "grad_norm": 0.0034125158563256264, "learning_rate": 0.0001134020618556701, "entropy": 0.030800834832916735, "mean_token_accuracy": 0.9834724962711334, "num_tokens": 385615.0, "epoch": 1.4848484848484849, "step": 160}
{"eval_loss": 0.03976129740476608, "eval_runtime": 14.7742, "eval_samples_per_second": 19.832, "eval_steps_per_second": 5.009, "eval_entropy": 0.03702763763037699, "eval_mean_token_accuracy": 0.9849799790897885, "eval_num_tokens": 385615.0, "epoch": 1.4848484848484849, "step": 160}
{"loss": 0.02616368532180786, "grad_norm": 2.506659984588623, "learning_rate": 0.00010652920962199313, "entropy": 0.026338954567472685, "mean_token_accuracy": 0.991689133644104, "num_tokens": 409361.0, "epoch": 1.578088578088578, "step": 170}
{"eval_loss": 0.055628545582294464, "eval_runtime": 14.7619, "eval_samples_per_second": 19.848, "eval_steps_per_second": 5.013, "eval_entropy": 0.022926484297558263, "eval_mean_token_accuracy": 0.9858710363104537, "eval_num_tokens": 409361.0, "epoch": 1.578088578088578, "step": 170}
{"loss": 0.03147937953472137, "grad_norm": 0.03703927993774414, "learning_rate": 9.965635738831616e-05, "entropy": 0.01485139998694649, "mean_token_accuracy": 0.9912788599729538, "num_tokens": 433983.0, "epoch": 1.6713286713286712, "step": 180}
{"eval_loss": 0.026821041479706764, "eval_runtime": 14.7833, "eval_samples_per_second": 19.82, "eval_steps_per_second": 5.006, "eval_entropy": 0.031676766934578084, "eval_mean_token_accuracy": 0.9943264867808368, "eval_num_tokens": 433983.0, "epoch": 1.6713286713286712, "step": 180}
{"loss": 0.0382485568523407, "grad_norm": 0.06816185265779495, "learning_rate": 9.278350515463918e-05, "entropy": 0.03301460822112858, "mean_token_accuracy": 0.9893269240856171, "num_tokens": 458256.0, "epoch": 1.7645687645687644, "step": 190}
{"eval_loss": 0.029211528599262238, "eval_runtime": 14.7516, "eval_samples_per_second": 19.862, "eval_steps_per_second": 5.016, "eval_entropy": 0.053183448749493704, "eval_mean_token_accuracy": 0.9891645892246349, "eval_num_tokens": 458256.0, "epoch": 1.7645687645687644, "step": 190}
{"loss": 0.011761382222175598, "grad_norm": 0.21739348769187927, "learning_rate": 8.591065292096219e-05, "entropy": 0.03190114335156977, "mean_token_accuracy": 0.9983387798070907, "num_tokens": 482601.0, "epoch": 1.8578088578088578, "step": 200}
{"eval_loss": 0.03959992900490761, "eval_runtime": 14.7408, "eval_samples_per_second": 19.877, "eval_steps_per_second": 5.02, "eval_entropy": 0.038160975339721785, "eval_mean_token_accuracy": 0.9817164065064611, "eval_num_tokens": 482601.0, "epoch": 1.8578088578088578, "step": 200}
{"loss": 0.01502387374639511, "grad_norm": 0.05332792550325394, "learning_rate": 7.903780068728523e-05, "entropy": 0.017657328562927434, "mean_token_accuracy": 0.9952500000596046, "num_tokens": 506953.0, "epoch": 1.951048951048951, "step": 210}
{"eval_loss": 0.03309974819421768, "eval_runtime": 14.7513, "eval_samples_per_second": 19.863, "eval_steps_per_second": 5.017, "eval_entropy": 0.03378042710018407, "eval_mean_token_accuracy": 0.9900552517658955, "eval_num_tokens": 506953.0, "epoch": 1.951048951048951, "step": 210}
{"loss": 0.027877017855644226, "grad_norm": 0.00380733422935009, "learning_rate": 7.216494845360825e-05, "entropy": 0.01409773326107628, "mean_token_accuracy": 0.9893930151655868, "num_tokens": 529342.0, "epoch": 2.0372960372960374, "step": 220}
{"eval_loss": 0.03955734893679619, "eval_runtime": 14.7173, "eval_samples_per_second": 19.909, "eval_steps_per_second": 5.028, "eval_entropy": 0.02965550131763873, "eval_mean_token_accuracy": 0.9881783747995222, "eval_num_tokens": 529342.0, "epoch": 2.0372960372960374, "step": 220}
{"loss": 0.014272943139076233, "grad_norm": 0.14554783701896667, "learning_rate": 6.529209621993127e-05, "entropy": 0.012182935852615628, "mean_token_accuracy": 0.9939772710204124, "num_tokens": 553447.0, "epoch": 2.1305361305361306, "step": 230}
{"eval_loss": 0.06179143488407135, "eval_runtime": 14.8359, "eval_samples_per_second": 19.749, "eval_steps_per_second": 4.988, "eval_entropy": 0.023730937492989657, "eval_mean_token_accuracy": 0.985062272967519, "eval_num_tokens": 553447.0, "epoch": 2.1305361305361306, "step": 230}
{"loss": 0.0033284474164247515, "grad_norm": 0.00803847424685955, "learning_rate": 5.84192439862543e-05, "entropy": 0.006925990225863643, "mean_token_accuracy": 0.9984375, "num_tokens": 577761.0, "epoch": 2.2237762237762237, "step": 240}
{"eval_loss": 0.07463129609823227, "eval_runtime": 14.7729, "eval_samples_per_second": 19.834, "eval_steps_per_second": 5.009, "eval_entropy": 0.021965690223402595, "eval_mean_token_accuracy": 0.9816838945891406, "eval_num_tokens": 577761.0, "epoch": 2.2237762237762237, "step": 240}
{"loss": 0.005613026395440102, "grad_norm": 0.310444712638855, "learning_rate": 5.1546391752577315e-05, "entropy": 0.00608841121684236, "mean_token_accuracy": 0.9970486104488373, "num_tokens": 601968.0, "epoch": 2.317016317016317, "step": 250}
{"eval_loss": 0.07936923205852509, "eval_runtime": 14.7612, "eval_samples_per_second": 19.849, "eval_steps_per_second": 5.013, "eval_entropy": 0.02082288997386255, "eval_mean_token_accuracy": 0.9793292670636564, "eval_num_tokens": 601968.0, "epoch": 2.317016317016317, "step": 250}
{"loss": 0.009035173803567886, "grad_norm": 0.1260085254907608, "learning_rate": 4.467353951890035e-05, "entropy": 0.008992704201227752, "mean_token_accuracy": 0.9931347891688347, "num_tokens": 626233.0, "epoch": 2.41025641025641, "step": 260}
{"eval_loss": 0.08104822039604187, "eval_runtime": 14.7724, "eval_samples_per_second": 19.834, "eval_steps_per_second": 5.009, "eval_entropy": 0.020790740055098147, "eval_mean_token_accuracy": 0.9793292670636564, "eval_num_tokens": 626233.0, "epoch": 2.41025641025641, "step": 260}
{"loss": 0.008879392594099044, "grad_norm": 0.1097167581319809, "learning_rate": 3.7800687285223366e-05, "entropy": 0.009547054303402547, "mean_token_accuracy": 0.9924206346273422, "num_tokens": 650832.0, "epoch": 2.5034965034965033, "step": 270}
{"eval_loss": 0.0815596878528595, "eval_runtime": 14.7466, "eval_samples_per_second": 19.869, "eval_steps_per_second": 5.018, "eval_entropy": 0.020388323040852148, "eval_mean_token_accuracy": 0.9793292670636564, "eval_num_tokens": 650832.0, "epoch": 2.5034965034965033, "step": 270}
{"loss": 0.005615378171205521, "grad_norm": 0.18375687301158905, "learning_rate": 3.0927835051546395e-05, "entropy": 0.006499038789479527, "mean_token_accuracy": 0.9971840664744377, "num_tokens": 675152.0, "epoch": 2.596736596736597, "step": 280}
{"eval_loss": 0.08162806928157806, "eval_runtime": 14.7615, "eval_samples_per_second": 19.849, "eval_steps_per_second": 5.013, "eval_entropy": 0.01972499680905949, "eval_mean_token_accuracy": 0.9816838945891406, "eval_num_tokens": 675152.0, "epoch": 2.596736596736597, "step": 280}
{"loss": 0.004639983549714089, "grad_norm": 0.0033270390704274178, "learning_rate": 2.4054982817869417e-05, "entropy": 0.005059999070363119, "mean_token_accuracy": 0.9981125354766845, "num_tokens": 698771.0, "epoch": 2.6899766899766897, "step": 290}
{"eval_loss": 0.08127416670322418, "eval_runtime": 14.7663, "eval_samples_per_second": 19.842, "eval_steps_per_second": 5.011, "eval_entropy": 0.019653764144931227, "eval_mean_token_accuracy": 0.9804553929212931, "eval_num_tokens": 698771.0, "epoch": 2.6899766899766897, "step": 290}
{"loss": 0.007612786442041397, "grad_norm": 0.11668085306882858, "learning_rate": 1.7182130584192442e-05, "entropy": 0.007699808984762057, "mean_token_accuracy": 0.9923629865050316, "num_tokens": 723209.0, "epoch": 2.7832167832167833, "step": 300}
{"eval_loss": 0.08156909793615341, "eval_runtime": 14.7539, "eval_samples_per_second": 19.859, "eval_steps_per_second": 5.016, "eval_entropy": 0.01949014219851076, "eval_mean_token_accuracy": 0.9815815187789299, "eval_num_tokens": 723209.0, "epoch": 2.7832167832167833, "step": 300}
{"loss": 0.0028498508036136627, "grad_norm": 0.00328033696860075, "learning_rate": 1.0309278350515464e-05, "entropy": 0.0036801236816245363, "mean_token_accuracy": 0.9978968247771263, "num_tokens": 747300.0, "epoch": 2.8764568764568765, "step": 310}
{"eval_loss": 0.0816095694899559, "eval_runtime": 14.845, "eval_samples_per_second": 19.737, "eval_steps_per_second": 4.985, "eval_entropy": 0.01938906139646959, "eval_mean_token_accuracy": 0.9804553929212931, "eval_num_tokens": 747300.0, "epoch": 2.8764568764568765, "step": 310}
{"loss": 0.00743260383605957, "grad_norm": 0.0012304404517635703, "learning_rate": 3.4364261168384886e-06, "entropy": 0.00903272690338781, "mean_token_accuracy": 0.9980769231915474, "num_tokens": 771787.0, "epoch": 2.9696969696969697, "step": 320}
{"eval_loss": 0.08165378868579865, "eval_runtime": 14.7635, "eval_samples_per_second": 19.846, "eval_steps_per_second": 5.012, "eval_entropy": 0.019268741207417554, "eval_mean_token_accuracy": 0.9815815187789299, "eval_num_tokens": 771787.0, "epoch": 2.9696969696969697, "step": 320}
{"eval_loss": 0.08126534521579742, "eval_runtime": 14.7275, "eval_samples_per_second": 19.895, "eval_steps_per_second": 5.025, "eval_entropy": 0.019338999524267354, "eval_mean_token_accuracy": 0.9815815187789299, "eval_num_tokens": 779394.0, "epoch": 3.0, "step": 324}
{"train_runtime": 1499.3056, "train_samples_per_second": 3.432, "train_steps_per_second": 0.216, "total_flos": 1.4236152922421453e+17, "train_loss": 0.07029568819030199, "epoch": 3.0, "step": 324}
| Package | Version |
|---|---|
| torch | 2.14.0 |
| transformers | 5.17.0 |
| trl | 1.13.0 |
| datasets | 5.0.1 |
| accelerate | 1.15.0 |
| python-dotenv | 1.2.3 |
| peft | 0.21.0 |
| bitsandbytes | 0.50.2 |
| huggingface-hub | ? |
| jinja2 | ? |
| torchvision | 0.29.0 |
| pillow | 12.3.0 |
This adapter is intended for research purposes only as part of the AIML589 project, which investigates value alignment of LLMs with New Zealand population distributions from the World Values Survey.
This model has not been safety-tuned for general-purpose deployment. It should not be used in production systems, for making decisions about people, or in contexts where reliability and safety are critical.