Downloads · 30 days
74
4% of all-time downloads
RichardErkhov/yanolja_-_KoSOLAR-10.7B-v0.1-deprecated-gguf
yanolja_-_KoSOLAR-10.7B-v0.1-deprecated-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
74
4% of all-time downloads
All-time downloads
2K
Public
Repo size
123 GB
Likes
0
Public
Click a slice to open those files.
.gguf143 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
KoSOLAR-10.7B-v0.1-deprecated - GGUF
license: apache-2.0 base_model: upstage/SOLAR-10.7B-v1.0 tags:
If you're passionate about the field of Large Language Models and wish to exchange knowledge and insights, we warmly invite you to join our Discord server. It's worth noting that Korean is the primary language used in this server. The landscape of LLM is evolving rapidly, and without active sharing, our collective knowledge risks becoming outdated swiftly. Let's collaborate and drive greater impact together! Join us here: https://discord.gg/b27bAHg95m.
This model is DEPRECATED due to an issue with the tokenizer. A new, corrected version will be uploaded shortly. We strongly advise against fine-tuning this model until the updated version is available. Details for the new version will be provided in a separate model card.
This model is a Korean vocabulary-extended version of upstage/SOLAR-10.7B-v1.0, specifically pre-trained on various Korean web-crawled datasets available on HuggingFace. Our approach was to expand the model's understanding of Korean by pre-training the embeddings for new tokens while preserving the original parameters of the base model.
Most parameters of upstage/SOLAR-10.7B-v1.0 were kept frozen during our training process. Only the embeddings for the newly added Korean tokens in the embed_tokens layer and the lm_head layer were pre-trained. This approach allowed us to enhance the model's performance in Korean while maintaining its original English capabilities.
No instruction tuning has been performed on this model. We recommend further training for specific purposes with caution, as it was primarily enhanced for Korean language understanding.
The model was pre-trained on various Korean web-crawled datasets openly available on HuggingFace.
It's essential to understand what "pre-trained" means in the context of this model. While the base model was already pre-trained on a broad, non-task-specific corpus of data, we further pre-trained only the embeddings for the expanded Korean vocabulary. This means that we did not alter the other existing parameters from the base model at all. This approach ensures a robust understanding of both English and Korean.
The following hyperparameters were used during training:
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| kmmlu | N/A | none | 0 | acc | 0.3004 | ± | 0.0528 |
| gsm8k | Yaml | get-answer | 5 | exact_match | 0.5625 | ± | 0.0137 |
| hellaswag | Yaml | none | 0 | acc | 0.6393 | ± | 0.0048 |
| mmlu | N/A | none | 0 | acc | 0.6305 | ± | 0.1452 |
| truthfulqa | N/A | none | 0 | acc | 0.4096 | ± | 0.0467 |
| winogrande | Yaml | none | 0 | acc | 0.7443 | ± | 0.0123 |
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| kmmlu | N/A | none | 0 | acc | 0.2948 | ± | 0.0537 |
| gsm8k | Yaml | get-answer | 5 | exact_match | 0.5527 | ± | 0.0137 |
| hellaswag | Yaml | none | 0 | acc | 0.6392 | ± | 0.0048 |
| mmlu | N/A | none | 0 | acc | 0.6303 | ± | 0.1411 |
| truthfulqa | N/A | none | 0 | acc | 0.3618 | ± | 0.0472 |
| winogrande | Yaml | none | 0 | acc | 0.7459 | ± | 0.0122 |