Downloads · 30 days
0
pythia-6.9b-goodwiki-lora-r64
spadeMIA/pythia-6.9b-goodwiki-lora-r64
pythia-6.9b-goodwiki-lora-r64 is a machine learning model from spadeMIA. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
LoRA-fine-tuned checkpoint produced for membership inference research at the SPADE Lab, Koç University.
Access
Public
Updated Sep 27, 2026
Repo size
537 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors537 MB · 99%
At a glance
- Access
- Public
- Created
- Aug 17, 2026
- Updated
- Sep 27, 2026
- SHA
- 4cd96481
Model card
From the Hugging Face model README
adapter
LoRA-fine-tuned checkpoint produced for membership inference research at the SPADE Lab, Koç University.
Base model
- Model:
EleutherAI/pythia-6.9b - PEFT method: LoRA
Training data
- Dataset:
spadeMIA/GoodWiki_Corpus_1024_2040 - Split used:
train(10,000 rows, all membership-positive) - Stored text token range: 1024-2039 tokens (Pythia tokenizer family), plus one appended EOS token
- Max training sequence length: 2048 tokens
- Observed token stats: min=1024, mean=1840.7, median=1926.0, max=2039
LoRA configuration
| Hyperparameter | Value |
|---|---|
| Rank (r) | 64 |
| Alpha | 128 |
| Dropout | 0.05 |
| Target modules | query_key_value, dense, dense_h_to_4h, dense_4h_to_h |
Training procedure
| Hyperparameter | Value |
|---|---|
| Epochs | 2 |
| Learning rate | 0.0002 (constant) |
| LR scheduler | constant_with_warmup |
| Warmup ratio | 0.01 |
| Weight decay | 0.0 |
| Max grad norm | 1.0 |
| Per-device batch size | 1 |
| Gradient accumulation steps | 32 |
| Effective batch size | 32 |
| Precision | bf16 |
| TF32 matmul | True |
| Gradient checkpointing | True |
| Optimizer | adamw_torch |
| Seed | 42 |
| Checkpointing | none during training; only the final adapter is saved |
Notes
- The corresponding held-out
testsplit was not loaded or trained on. - Eval-loss logging (monitoring only, no checkpoint selection) used
train[:50].
Project and contributors
This checkpoint was trained as part of an ongoing membership inference research project at the SPADE Lab, Koç University, with contributions from Batu Koray Masak (AI & Data Engineering and Computer Science, Özyeğin University) and Melda Paksoy (Computer Science, Abdullah Gül University).
Generated automatically by run_lora.py on 2026-08-17.
- Created
- Aug 17, 2026
- Updated
- Sep 27, 2026