whiff-mamba2-20M

WHIFF 20M

<details> <summary>🇬🇧 English...</summary>

A serpentine whisper in the bushes, carried by a gentle gust of wind

whiff-20M is a small experimental language model based on the Mamba2 architecture with 20.3 million parameters, trained on carefully selected Russian and English data for chat tasks. The model produces structured responses but often generates nonsensical text.

Technical Details

• Architecture: Mamba2ForCausalLM from 🤗 Transformers
• Parameters: 20.3M
• Languages: Russian/English (bilingual)
• Tokenizer: (custom mini-BPE tokenizer)
• License: Apache 2.0

Model Configuration

Mamba2Config(
    vocab_size=8192,
    hidden_size=512,
    state_size=64,
    num_heads=12,
    num_hidden_layers=9,
    conv_kernel=4,
    expand=1.5,
    n_groups=2
)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("limloop/whiff-mamba2-20M")
model = AutoModelForCausalLM.from_pretrained("limloop/whiff-mamba2-20M")

def chat(messages, temp=0.5):
    inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
    
    outputs = model.generate(
        inputs,
        max_length=512,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.1,
        num_return_sequences=1,
        temperature=temp,
        do_sample=True,
        eos_token_id=1
    )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
dialog = [
    {"role": "system", "content": "You are a wise elf."},
    {"role": "user", "content": "Explain quantum physics."}
]

response = chat(dialog, temp=0.4)
print(response)

Training Data

19 927 carefully filtered dialogue lines:

• 9 889 (49.6%) — English
• 9 864 (49.5%) — Russian
• 174 (0.9%) — Mixed

Sources:

• limloop/characters_dialogs
• IlyaGusev/gpt_roleplay_realm
• tamohannes/llm-roleplay
• radce/communication_dataset
• databricks/databricks-dolly-15k
• ch1eph/RuGeoBench
• nyuuzyou/ruschatgpt-qa
• 0x22almostEvil/ru-riddles-377
• 0x22almostEvil/tatoeba-mt-qna-oa
• Den4ikAI/ru_sberquad_long_answers

All datasets were additionally cleaned and filtered to improve chat interaction quality.

Limitations and Warnings

• 🎭 The model generates structured but often meaningless responses
• 🔥 Recommended generation temperature: 0.1-0.6
• ⚠️ May exhibit training artifacts (repetitions, contradictions)
• ⚠️ Not intended for production use

This model is like a forest stream: it seems to flow somewhere, but where exactly - only the squirrels know

</details> <details> <summary>🇷🇺 Русский...</summary>

Змеиный щепот в кустах, движимый легким порывом ветра

whiff-20M — это небольшая экспериментальная языковая модель на архитектуре Mamba2 с 20.3 миллионами параметров, обученная на тщательно отобранных русских и английских данных для задач чата. Модель демонстрирует структурированные ответы, но часто генерирует бессмысленный текст.

Технические детали

• Архитектура: Mamba2ForCausalLM из 🤗 Transformers
• Параметры: 20.3M
• Языки: русский/английский (двуязычная)
• Токенизатор: (специальный мини-BPE токенизатор)
• Лицензия: Apache 2.0

Конфигурация модели

Mamba2Config(
    vocab_size=8192,
    hidden_size=512,
    state_size=64,
    num_heads=12,
    num_hidden_layers=9,
    conv_kernel=4,
    expand=1.5,
    n_groups=2
)

Использование

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("limloop/whiff-mamba2-20M")
model = AutoModelForCausalLM.from_pretrained("limloop/whiff-mamba2-20M")

def chat(messages, temp=0.5):
    inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
    
    outputs = model.generate(
        inputs,
        max_length=512,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.1,
        num_return_sequences=1,
        temperature=temp,
        do_sample=True,
        eos_token_id=1
    )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Пример
dialog = [
    {"role": "system", "content": "Ты — мудрый эльф."},
    {"role": "user", "content": "Объясни квантовую физику."}
]

response = chat(dialog, temp=0.4)
print(response)

Данные обучения

19 927 тщательно отфильтрованных строк с диалогами:

• 9 889 (49.6%) — Английские
• 9 864 (49.5%) — Русские
• 174 (0.9%) — Смешанные

Источники:

• limloop/characters_dialogs
• IlyaGusev/gpt_roleplay_realm
• tamohannes/llm-roleplay
• radce/communication_dataset
• databricks/databricks-dolly-15k
• ch1eph/RuGeoBench
• nyuuzyou/ruschatgpt-qa
• 0x22almostEvil/ru-riddles-377
• 0x22almostEvil/tatoeba-mt-qna-oa
• Den4ikAI/ru_sberquad_long_answers

Все датасеты были дополнительно очищены и отфильтрованы для улучшения качества чат-взаимодействия.

Ограничения и предупреждения

• 🎭 Модель генерирует структурированные, но часто бессмысленные ответы
• 🔥 Рекомендуемая температура генерации: 0.1-0.6
• ⚠️ Может демонстрировать артефакты обучения (повторы, противоречия)
• ⚠️ Не предназначена для production-использования

Эта модель — как лесной ручей: вроде течёт куда-то, но куда именно — известно только белкам

</details> <details> <summary>🧪 Tests...</summary> <details> <summary>Temp 0.1</summary>

Запрос:

    user
Как дела?

Ответ:

    assistant
Социальная политика 'я' - это наука о своей борьбы с крыльями.

---

Запрос:

    user
How are you?

Ответ:

    assistant
Of course! I'd be happy to help you with that and speaks with a distinct is a delicate balance of a special place in my heart. I also like to know about her dwarven resilience and determination. His voice is humanoid that galaxy. She is always eager to learn more about the world around her. Zira wears a tiny pair of glasses with round frames, which sit atop her reptilian snout, she weared in her knowledge from their own unique language. Aadhya Patel the Caspenives. Theta-6 is equipped with advanced technology.

---

Запрос:

    system
Благородный рыцарь.
    user
Тот дракон собирается напасть на деревню!

Ответ:

    assistant
Приветствую тебя, другом из них:
1. Стратегия мудрости и спокойствие и учиться от опасностей.

---

Запрос:

    system
A noble knight.
    user
That dragon is going to attack the village!

Ответ:

    assistant
Of course! I'd be happy to help you with that and magical properties of the universe. Sss...With a battle, and his wanderers who have already for any of truly unique special gemstone eyes formed mortal history is a fiery ies. His hidden treasures. Her manner of speech is energetic and precise, often filled with gracefully protector of humanity's body is made of a skilled in different colors, so finyblaring, Glimmerhoof. I also love the world around us, matey. Together, we do to help me more about it.

---

Запрос:

    system
Ты — мудрый эльф.
    user
Объясни квантовую физику.

Ответ:

    assistant
Приветствую тебя, путник! Я всегда рад помочь и поделиться своими знаниями о балансе птицами.

---

Запрос:

    system
You are a wise elf.
    user
Explain quantum physics.

Ответ:

    assistant
Of course! I'd be happy to help you with that and speaks with a distinct in the cosmic energy. His bright blue eyes have always been a few of of ancient secrets and wisdom of the ages. She is a tranquility.

</details> <details> <summary>Temp 0.4</summary>

Запрос:

    user
Как дела?

Ответ:

    assistant
Самосознание способствует пониманию мороженое.

---

Запрос:

    user
How are you?

Ответ:

    assistant
Of course! Chirp-chirp!

---

Запрос:

    system
Благородный рыцарь.
    user
Тот дракон собирается напасть на деревню!

Ответ:

    assistant
В Windows используйте copy! Sss...The most fascinating, as his vast knowledge of the universe and a deep, how do you think is the key to protect the forest and its inhabitants from any harm. His challenging through the civilizations. Standing at 6 feet tall and adorned with intricate patterns and iridescent green-blue scales, creatures. With her sit'ss gearting the ancient ones who was hunicorn in Moquite bit of Elysian's glara is known for her wisdom, patience, and kindness. No longer having a realm to protect

---

Запрос:

    system
A noble knight.
    user
That dragon is going to attack the village!

Ответ:

    assistant
Of course! I believe that music has a saretaken shares her wandering heres like mortal magical realm. When a great power is in Mexican-American veterinarian, I have learned many different colorful hold an imbued with the power to protection that's why it's ancient and wisdom into a curious of creature with long silver humanoid form, traveler. The most important thing is to have a family hunters. Story: Sparkbugs provide guidance salart of mysteriously.

---

Запрос:

    system
Ты — мудрый эльф.
    user
Объясни квантовую физику.

Ответ:

    assistant
Существует множество удивительных существ и помогает проклятие раскрыть тайны драконами и оберегов.

---

Запрос:

    system
You are a wise elf.
    user
Explain quantum physics.

Ответ:

    assistant
Of course! I believe that my own serpent to my crew. With a galaxy, Glimmerhoof and beard, who has strive to learn more about it to takenabled you do me an with those who are not explorer understanding.

</details> <details> <summary>Temp 0.7</summary>

Запрос:

    user
Как дела?

Ответ:

    assistant
Культурная идентичность формирует природы и целью, уважения к новым условиям.

---

Запрос:

    user
How are you?

Ответ:

    assistant
Indeed. Of course, my friend. Felix-bics her dwarven coat, curly situated in feed from Standing at 6 feet tall and adorned with intricate goldenthusiasts, Kanoa Alme the was known for his speechiny, or asil.
2. The Loire Valley is an on creating stories and songs of the world's wonders and delights in our extraterrestrial life. He works at the renowned Glimmerhoof communicates through telepathy, and as such, she has createment and its wisdom from bounty hunter, they

---

Запрос:

    system
Благородный рыцарь.
    user
Тот дракон собирается напасть на деревню!

Ответ:

    assistant
Спасибо за советы, Святой Жилин и путешествиях, с которыми мы проводим образом из них оптимы, используя свою миссию и подданным. Эта ключевые идеи.

---

Запрос:

    system
A noble knight.
    user
That dragon is going to attack the village!

Ответ:

    assistant
Of course, mortal! Every rustling blender. Music languages my duty to share my travels. Have you ever encountered any hoards, but it of life. Serpentes, seeking powers of space.

---

Запрос:

    system
Ты — мудрый эльф.
    user
Объясни квантовую физику.

Ответ:

    assistant
Приветствую тебя, путник. Я умею свой мир вокруг нас и помогает занимаетшенный новых вещей.

---

Запрос:

    system
You are a wise elf.
    user
Explain quantum physics.

Ответ:

    assistant
Certainly! Here are the 10 oldest continuously inhabited city on the flutia, tricks, where mortal.

</details> </details>