Downloads · 30 days
35
4% of all-time downloads
Sasando-1-7M
afrizalha/Sasando-1-7M
Sasando-1-7M is a text generation model from afrizalha. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
<!DOCTYPE html <html lang="en" <head <meta charset="UTF-8" <meta name="viewport" content="width=device-width, initial-scale=1.0" <titleDocument Title</title <style h1 { font-size: 32px; color: navy; font-family: 'Taho…
All-time downloads
820
Public
Parameters
7.3M
29.4 MB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors29.4 MB · 99%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- phi3
- Access
- Public
- Created
- Jun 8, 2024
- Updated
- Feb 13, 2025
- SHA
- 486a2333
Spaces
Model card
From the Hugging Face model README
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Document Title</title>
<style>
h1 {
font-size: 32px;
color: navy;
font-family: 'Tahoma';
text-align: center;
}
</style>
</head>
<body>
<h1>Descaling low-resource language models</h1>
</body>
</html>
<center>
<img src="https://i.imgur.com/z9ey830.png" alt="Sasando" width="500" height="250">
<p><em>Sasando-1 is a tiny, highly experimental short-sequence text generator built using the Phi-3 architecture.</em></p>
<p><strong><a href="https://huggingface.co/spaces/afrizalha/Sasando-1" style="color: blue; font-family: Tahoma;">❕Go straight to the gradio demo❕</a></strong></p>
<p><em style="color: black; font-weight: bold;">This repo contains the 7M version</em></p>
<p><em style="color: black; font-weight: bold;">Preliminary research preview</em></p>
</center>
🎻 Welcome!
Sasando-1 is a tiny, highly experimental Indonesian text generator built using the Phi-3 architecture. It comes with two variations of microscopic sizes: 7M and 25M parameters. It is trained on a tightly-controlled Indo4B dataset filtered to only have 18000 unique words. The method is inspired by Microsoft's TinyStories paper which demonstrates that a tiny language model can produce fluent text when trained on tightly-controlled dataset.
🇮🇩 Context
Indonesia has +700 languages, and many of them are dying at an alarming rate. Language technologies like generative AI can play a massive role in language preservation. However, Indonesia has several contextual issues:
- Many languages, including those with millions of speakers, have low-volume digital resources
- Running large models can be costly, while Indonesia is a middle-income country with little funding
Overcoming these challenges require developers to work with what little data and money that they have. Sasando-1 is a prototypical demonstration that thinly-available resources can potentially still be leveraged to develop generative models with cheap compute.
✨ Specs
- Comes with 7M and 25M parameters
- Based on Phi-3 architecture
- Embedding vocab 4096
- Trained on ~257M tokens * 4 epoch
🔭 Out-of-Scope Use
This is a research preview base model. It is not intruction-tuned and has minimal safety curation. It is not intended for commercial or practical applications.
You are also not allowed to use this model without having fun.
Acknowledgments
- Developed by: Afrizal Hasbi Azizy
- License: MIT
Training log
<right> <img src="https://imgur.com/32NFAKm.png" alt="Training log" width="500" height="250"> </right>- Task
- Text Generation
- Library
- transformers
- Type
- phi3
- License
- mit
- Languages
- id
- Created
- Jun 8, 2024
- Updated
- Feb 13, 2025