Downloads · 30 days
0
SemantikaEU/Micka-gen3
Micka-gen3 is a machine learning model from SemantikaEU. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
0
Access
Public
Updated Apr 5, 2025
Repo size
3.9 GB
Likes
0
Public
Click a slice to open those files.
.bz23.9 GB · 100%
From the Hugging Face model README
Author: Semantika Research
Micka Gen3 is a specialized language model based on the Microsoft RetNet architecture, fine-tuned for Retrieval-Augmented Generation (RAG) usage in Slovenian Cultural Heritage Domain. It leverages an efficient retention mechanism, and should be used as baseline and in combination with the GAMS series of models.
A standalone series of models, based on the GaMS model will also be released.
The model was trained from scratch on:
The model underwent 20 epochs of training on the above datasets.
The final stage involved finetuning on 10,000 culturally relevant samples prepared specifically for the Povejmo Project, focusing on cultural heritage content.
This model uses the following tokenizer:
The tokenizer shares the same foundational training data, with additional cultural heritage samples included for domain specificity.
The Micka-Gen3 is based on the Microsoft RetNet architecture with the following detailed layers:
embedding.weight)out.weight, out.bias)The architecture is optimized for long-context document retrieval and generation tasks in combination with large Generative AI models.
Designed specifically for Retrieval-Augmented Generation (RAG), Micka-Gen3 performs well in:
The development of the Micka Tokenizer was partially funded by the PoVeJMo project, which aims to develop large language models for the Slovenian language.
The project PoVeJMo is cofinanced by:

This tokenizer is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0). This license allows for sharing and adaptation, provided appropriate credit is given and any derivatives are distributed under the same license.
Please cite the following if you use Micka-Gen3:
@misc{micka-gen3,
author = {Semantika Research},
title = {Micka-Gen3: A RetNet-based Slovenian Language Model for RAG tasks},
year = {2024},
publisher = {Hugging Face},
url = {https://huggingface.co/klokedm/micka-gen3}
}
For more information, please contact: