Downloads · 30 days
0
pythia-1B-feature-extraction-MIMIC-III
mia-project-2025/pythia-1B-feature-extraction-MIMIC-III
pythia-1B-feature-extraction-MIMIC-III is a machine learning model from mia-project-2025. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This repository contains a Pythia-1B model used in a feature-based transfer learning setup on the MIMIC-III dataset for clinical text generation. The model uses a frozen base Pythia-1B encoder with a custom LM decoder…
Access
Public
Updated Aug 30, 2025
Repo size
14.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8.1 GB · 57%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 30, 2025
- Updated
- Aug 30, 2025
- SHA
- 9ab433c1
Model card
From the Hugging Face model README
Pythia-Feature-Mimic-III
This repository contains a Pythia-1B model used in a feature-based transfer learning setup on the MIMIC-III dataset for clinical text generation. The model uses a frozen base Pythia-1B encoder with a custom LM decoder trained from scratch to predict the impressions field from patient notes.
Model Details
- Base model:
EleutherAI/pythia-1b(frozen) - Dataset: hejazizo/mimic-iii
- Task: Causal language modeling (predict impressions)
- Tokenizer: AutoTokenizer from
EleutherAI/pythia-1b - Custom LM Head: Linear projection from hidden states to vocabulary size
- Maximum sequence length: 512
- Block size for LM: 1024
- Fine-tuning strategy: Feature-based transfer learning (frozen base model)
- Precision: FP16 (mixed precision)
- Batch size: 8 per device, gradient accumulation steps 2
- Learning rate: 2e-4
- Epochs: 15
- Weight decay: 0.01
- Early stopping: Patience = 2 epochs
results
- final perplexity 12.90
- License
- apache-2.0
- Created
- Aug 30, 2025
- Updated
- Aug 30, 2025