KONI-4B-instruct-20250901

KISTI-KONI/KONI-4B-instruct-20250901

Model Description

KONI (KISTI Open Neural Intelligence) is a large language model developed by the Korea Institute of Science and Technology Information (KISTI). Designed specifically for the scientific and technological domains, KONI excels in both Korean and English, making it an ideal tool for tasks requiring specialized knowledge in these areas.

---

Key Features

• Bilingual Model: Supports both Korean and English, with a focus on scientific and technical texts.
• Post-training: The model undergoes post-training via instruction tuning (IT) and direct preference optimization (DPO) using a filtered, high-quality bilingual dataset that includes scientific data and publicly available resources. This ensures adaptability to evolving scientific and technological content.
• Base Model: Built upon KISTI-KONI/KONI-4B-base-20250819, KONI-4B-instruct undergoes post-training for superior performance on both general and scientific benchmarks.
• Training Environment: Trained on 24 H200 GPUs at the KISTI supercomputer, optimizing both speed and quality during development.
• Dataset: Utilizes a high-quality and balanced dataset of 9 billion instruction-following pairs, comprising scientific texts as well as publicly available bilingual data.
• Data Optimization: The post-training process involved testing a variety of data distributions (balanced, reasoning-enhanced, knowledge-enhanced, minimal Korean settings, etc.) and selecting the optimal combination for training.
• Enhanced Performance: KONI-4B-instruct, developed through instruction tuning of the KONI-4B-base model, delivers superior performance compared to other similarly-sized models.

---

Model Performance

KONI-4B-instruct has demonstrated strong performance on a variety of scientific benchmarks, outperforming several other 4B-sized pretrained models. Here is a comparison of KONI-4B-instruct’s performance across various benchmarks including scientific and technological benchmarks:

As shown, KISTI-KONI/KONI-4B-instruct-20250901 is the top-performing model in the 4B-size instruction-tuned model category, outperforming google/gemma-3-4b-it and KISTI-KONI/KONI-4B-base-20250819.

---

Strengths & Use Cases

• Domain-Specific Excellence: KONI-4B-instruct excels at tasks involving scientific literature, technological content, and complex reasoning. It is ideal for research, academic analysis, and specialized problem-solving.
• Bilingual Advantage: The model’s bilingual nature enables handling diverse datasets and generating high-quality responses in both English and Korean, especially in bilingual scientific collaborations.
• Benchmark Performance: KONI-4B-instruct has shown superior performance in benchmarks such as KMMLU, kormedmcqa, and ScholarBench-MC, proving its robustness in knowledge-intensive tasks.

---

Usage

$ pip install -U transformers

from transformers import pipeline
import torch

pipe = pipeline("text-generation", model="KISTI-KONI/KONI-4B-instruct-20250901", device="cuda", torch_dtype=torch.bfloat16)

messages = [
    [
        {
            "role": "system",
            "content": [{"type": "text", "text": "You are a helpful assistant."},]
        },
        {
            "role": "user",
            "content": [{"type": "text", "text": "슈퍼컴퓨터에 대해서 설명해줘."},]
        },
    ],
]

output = pipe(
    messages,
    max_new_tokens=512,
    eos_token_id=[pipe.tokenizer.eos_token_id, pipe.tokenizer.convert_tokens_to_ids("<end_of_turn>")]
)

Citation

If you use this model in your work, please cite it as follows:

@article{KISTI-KONI/KONI-4B-instruct-20250901,
  title={KISTI-KONI/KONI-4B-instruct-20250901},
  author={KISTI},
  year={2025},
  url={https://huggingface.co/KISTI-KONI/KONI-4B-instruct-20250901}
}

---

Acknowledgements

• This research was supported by the Korea Institute of Science and Technology Information (KISTI) in 2025 (No. (KISTI) K25L1M1C1), aimed at developing KONI (KISTI Open Neural Intelligence), a large language model specialized in science and technology.
• This work also benefited from the resources and technical support provided by the National Supercomputing Center (KISTI).

---

References

• https://huggingface.co/KISTI-KONI/KONI-4B-base-20250819