Downloads · 30 days
10
2% of all-time downloads
mega-ar-small-4096-sw_minipile
pszemraj/mega-ar-small-4096-sw_minipile
mega-ar-small-4096-sw_minipile is a text generation model from pszemraj. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
should probably proofread and complete it, then remove this comment. --
All-time downloads
538
Public
Parameters
64.8M
16.6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin259 MB · 50%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- mega
- Access
- Public
- Created
- Sep 10, 2023
- Updated
- Dec 29, 2025
- SHA
- 4ea45be0
Try a prompt
Base models
Model card
From the Hugging Face model README
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
mega-ar-small-4096-NC-minipile-v1
65M parameter MEGA autoregressive model initialized from scratch and trained on:
pszemraj/simple_wikipedia_LMJeanKaddour/minipile
It achieves the following results on the evaluation set:
- Loss: 3.7502
- Accuracy: 0.3650
eval
initial 'get the feet wet':
hf-causal-experimental (pretrained=pszemraj/mega-ar-small-4096-sw_minipile,revision=main,trust_remote_code=True,dtype='float'), limit: None, provide_description: False, num_fewshot: 0, batch_size: 16
| Task | Version | Metric | Value | Stderr | |
|---|---|---|---|---|---|
| arc_easy | 0 | acc | 0.3173 | ± | 0.0096 |
| acc_norm | 0.3022 | ± | 0.0094 | ||
| boolq | 1 | acc | 0.4107 | ± | 0.0086 |
| lambada_openai | 0 | ppl | 6843.1824 | ± | 295.0792 |
| acc | 0.0155 | ± | 0.0017 | ||
| openbookqa | 0 | acc | 0.1220 | ± | 0.0147 |
| acc_norm | 0.2480 | ± | 0.0193 | ||
| piqa | 0 | acc | 0.5609 | ± | 0.0116 |
| acc_norm | 0.5566 | ± | 0.0116 | ||
| winogrande | 0 | acc | 0.5059 | ± | 0.0141 |
still some ways to go.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 1
- eval_batch_size: 1
- seed: 80085
- gradient_accumulation_steps: 64
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-07
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 1.0
Framework versions
- Transformers 4.33.1
- Pytorch 2.2.0.dev20230907+cu118
- Datasets 2.14.5
- Tokenizers 0.13.3
- Task
- Text Generation
- Library
- transformers
- Type
- mega
- License
- apache-2.0
- Created
- Sep 10, 2023
- Updated
- Dec 29, 2025