Downloads · 30 days
29
5% of all-time downloads
mkreynolds/K-EXAONE-236B-A23B-GGUF-4bit
K-EXAONE-236B-A23B-GGUF-4bit is a text generation model from mkreynolds. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as other.
Downloads · 30 days
29
5% of all-time downloads
All-time downloads
580
Public
Repo size
143 GB
Likes
1
Public
Click a slice to open those files.
.gguf143 GB · 100%
From the Hugging Face model README
This is a Q4_K_M quantization of LG's K-EXAONE-236B-A23B model. Run it with <a href="https://github.com/Aim-Highest/llama.cpp">this llama.cpp repo</a>. No rights claimed by me. Subject to <a href="https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B/blob/main/LICENSE">LG's custom license</a>. This distribution is not for commercial purposes. <br> <br> Original model card:
<br> <br> <p align="center"> <img src="assets/K-EXAONE_Symbol_3d.png" width="400"> <br> <!-- <p align="center"> 🤗 <a href="https://huggingface.co/collections/LGAI-EXAONE/k-exaone">Hugging Face</a>   |   📝 <a href="#"> Blog</a>   |   📑 <a href="#"> Technical Report </a>--> <br> <br> <div align="center"> <a href="https://huggingface.co/collections/LGAI-EXAONE/k-exaone" style="text-decoration: none;"> <img src="https://img.shields.io/badge/🤗-Huggingface-FC926C?style=for-the-badge" alt="Huggingface"> </a> <a href="#" style="text-decoration: none;"> <img src="https://img.shields.io/badge/📝-Blog_(TBD)-E343BD?style=for-the-badge" alt="Blog"> </a> <a href="#" style="text-decoration: none;"> <img src="https://img.shields.io/badge/📑-Technical_Report_(TBD)-684CF4?style=for-the-badge" alt="Technical Report"> </a> </div> <br>We introduce K-EXAONE, a large-scale multilingual language model developed by LG AI Research. Built using a Mixture-of-Experts architecture, K-EXAONE features 236 billion total parameters, with 23 billion active during inference. Performance evaluations across various benchmarks demonstrate that K-EXAONE excels in reasoning, agentic capabilities, general knowledge, multilingual understanding, and long-context processing.
For more details, please refer to the technical report.
The following table shows the evaluation results of the K-EXAONE model in reasoning mode, compared to our previous model, EXAONE-4.0, and other competing models. The evaluation details can be found in the technical report.
<table> <tr> <th> </th> <th>K-EXAONE (Reasoning)</th> <th>EXAONE 4.0 (Reasoning)</th> <th>GPT-OSS (Reasoning: High)</th> <th>Qwen3-Thinking-2507</th> <th>DeepSeek-V3.2 (Reasoning)</th> </tr> <tr> <td align="center">Architecture</td> <td align="center">MoE</td> <td align="center">Dense</td> <td align="center">MoE</td> <td align="center">MoE</td> <td align="center">MoE</td> </tr> <tr> <td align="center">Total Params</td> <td align="center">236B</td> <td align="center">32B</td> <td align="center">117B</td> <td align="center">235B</td> <td align="center">671B</td> </tr> <tr> <td align="center">Active Params</td> <td align="center">23B</td> <td align="center">32B</td> <td align="center">5.1B</td> <td align="center">22B</td> <td align="center">37B</td> </tr> <tr> <td align="center" colspan='7'><i>World Knowledge</i></td> </tr> <tr> <td align="center">MMLU-Pro</td> <td align="center">83.9</td> <td align="center">81.8</td> <td align="center">80.7</td> <td align="center">84.4</td> <td align="center">85.0</td> </tr> <tr> <td align="center">GPQA-Diamond</td> <td align="center">80.0</td> <td align="center">75.4</td> <td align="center">80.1</td> <td align="center">81.1</td> <td align="center">82.4</td> </tr> <tr> <td align="center">Humanity's Last Exam</td> <td align="center">13.8</td> <td align="center">10.6</td> <td align="center">14.9</td> <td align="center">18.2</td> <td align="center">25.1</td> </tr> <tr> <td align="center" colspan='7'><i>Math</i></td> </tr> <tr> <td align="center">AIME 2025</td> <td align="center">92.6</td> <td align="center">85.3</td> <td align="center">92.5</td> <td align="center">92.3</td> <td align="center">93.1</td> </tr> <tr> <td align="center" colspan='7'><i>Coding</i></td> </tr> <tr> <td align="center">LiveCodeBench v6</td> <td align="center">81.1</td> <td align="center">66.7</td> <td align="center">81.9</td> <td align="center">74.1</td> <td align="center">79.4</td> </tr> <tr> <td align="center" colspan='7'><i>Agentic Tool Use</i></td> </tr> <tr> <td align="center">τ<sup>2</sup>-Bench (Telecom)</td> <td align="center">71.9</td> <td align="center">23.7</td> <td align="center">60.3</td> <td align="center">45.6</td> <td align="center">85.8</td> </tr> <tr> <td align="center" colspan='7'><i>Instruction Following</i></td> </tr> <tr> <td align="center">IFBench</td> <td align="center">67.4</td> <td align="center">36.0</td> <td align="center">69.5</td> <td align="center">52.6</td> <td align="center">62.5</td> </tr> <tr> <td align="center">IFEval</td> <td align="center">89.8</td> <td align="center">84.7</td> <td align="center">89.5</td> <td align="center">87.8</td> <td align="center">92.6</td> </tr> <tr> <td align="center" colspan='7'><i>Long Context Understanding</i></td> </tr> <tr> <td align="center">AA-LCR</td> <td align="center">53.5</td> <td align="center">14.0</td> <td align="center">50.7</td> <td align="center">67.0</td> <td align="center">65.0</td> </tr> <tr> <td align="center" colspan='7'><i>Korean Knowledge & Math</i></td> </tr> <tr> <td align="center">KMMLU-Pro</td> <td align="center">67.3</td> <td align="center">67.7</td> <td align="center">62.4</td> <td align="center">71.6</td> <td align="center">72.1</td> </tr> <tr> <td align="center">HRM8K</td> <td align="center">90.7</td> <td align="center">89.4</td> <td align="center">91.6</td> <td align="center">92.0</td> <td align="center">90.6</td> </tr> <tr> <td align="center">CLIcK</td> <td align="center">83.9</td> <td align="center">78.8</td> <td align="center">74.6</td> <td align="center">81.3</td> <td align="center">86.3</td> </tr> <tr> <td align="center">KoBALT</td> <td align="center">61.8</td> <td align="center">25.4</td> <td align="center">54.3</td> <td align="center">56.1</td> <td align="center">62.7</td> </tr> </table>Until the libraries officially support K-EXAONE, you need to install the requirements in our version with the EXAONE-MoE implementations. We will announce when these libraries are updated to support the K-EXAONE model.
You can install the latest version of Transformers with support for EXAONE-MoE architecture from this repository.
The base version of Transformers is 5.0.0rc1, so it might be helpful to check the migration guide from the Transformers library.
You should install both Transformers and vLLM to use K-EXAONE model on vLLM server. You can install the latest version of vLLM with support for EXAONE-MoE architecture from this repository.
You should install both Transformers and SGLang to use K-EXAONE model on SGLang server. You can install the latest version of SGLang with support for EXAONE-MoE architecture from this repository.
You can install the latest version of llama.cpp with support for EXAONE-MoE architecture from this repository. Please refer to the official build guide for details.
You can use the K-EXAONE model with the Transformers library. For better quality, you should check the usage guideline section.
For tasks that require accurate results, you can run the K-EXAONE model in reasoning mode as below.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "LGAI-EXAONE/K-EXAONE-236B-A23B"
model = AutoModelForCausalLM.from_pretrained(
model_name,
dtype="bfloat16",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
messages = [
{"role": "system", "content": "You are K-EXAONE, a large language model developed by LG AI Research in South Korea, built to serve as a helpful and reliable assistant."},
{"role": "user", "content": "Which one is bigger, 3.9 vs 3.12?"}
]
input_ids = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
enable_thinking=True, # skippable (default: True)
)
generated_ids = model.generate(
**input_ids.to(model.device),
max_new_tokens=16384,
temperature=1.0,
top_p=0.95,
)
output_ids = generated_ids[0][input_ids['input_ids'].shape[-1]:]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
For tasks where latency matters more than accuracy, you can run the K-EXAONE model in non-reasoning mode as below.
messages = [
{"role": "system", "content": "You are K-EXAONE, a large language model developed by LG AI Research in South Korea, built to serve as a helpful and reliable assistant."},
{"role": "user", "content": "Explain how wonderful you are"}
]
input_ids = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
enable_thinking=False,
)
generated_ids = model.generate(
**input_ids.to(model.device),
max_new_tokens=1024,
temperature=1.0,
top_p=0.95,
)
output_ids = generated_ids[0][input_ids['input_ids'].shape[-1]:]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
For your AI-powered agent, you can integrate external functionalities in the following OpenAI-style tool description format. Here is an example of creating a tool description from a Python function.
Please check the example file for an example of a search agent conversation using K-EXAONE.
from transformers.utils import get_json_schema
def roll_dice(max_num: int):
"""
Roll a dice with the number 1 to N. User can select the number N.
Args:
max_num: The maximum number on the dice.
"""
return random.randint(1, max_num)
tool_schema = get_json_schema(roll_dice)
tools = [tool_schema]
messages = [
{"role": "system", "content": "You are K-EXAONE, a large language model developed by LG AI Research in South Korea, built to serve as a helpful and reliable assistant."},
{"role": "user", "content": "Roll a D20 twice and sum the results."}
]
input_ids = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
tools=tools,
)
generated_ids = model.generate(
**input_ids.to(model.device),
max_new_tokens=16384,
temperature=1.0,
top_p=0.95,
)
output_ids = generated_ids[0][input_ids['input_ids'].shape[-1]:]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
[!IMPORTANT] To achieve the expected performance, we recommend using the following configurations:
- We strongly recommend to use
temperature=1.0,top_p=0.95,presence_penalty=0.0for best performance.- Different from EXAONE-4.0, K-EXAONE uses
enable_thinking=Trueas default. Thus, you need to setenable_thinking=Falsewhen you want to use non-reasoning mode.
TensorRT-LLM support for the K-EXAONE model is being prepared. Please refer to the EXAONE-MoE PR on TensorRT-LLM repository for details.
We support the K-EXAONE model on vLLM. You need to install our fork of the vLLM library to use the K-EXAONE model. Please check the requirements section. Practically, you can serve the model with a 256K context length using tensor parallel on 4 H200 GPUs.
After you install the vLLM library with an EXAONE-MoE implementation, you can run the vLLM server by following command:
vllm serve LGAI-EXAONE/K-EXAONE-236B-A23B \
--reasoning-parser deepseek_v3 \
--enable-auto-tool-choice \
--tool-call-parser hermes
An OpenAI-compatible API server will be available at http://localhost:8000/v1.
You can test the vLLM server by sending a chat completion request as below:
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "LGAI-EXAONE/K-EXAONE-236B-A23B",
"messages": [
{"role": "user", "content": "How many r'\''s in \"strawberry\"?"}
],
"max_tokens": 16384,
"temperature": 1.0,
"top_p": 0.95,
"chat_template_kwargs": {"enable_thinking": true}
}'
If you are interested in using MTP weights for speculative decoding, add according options as below.
vllm serve LGAI-EXAONE/K-EXAONE-236B-A23B \
--reasoning-parser deepseek_v3 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--no-enable-prefix-caching \
--speculative_config '{
"method": "mtp",
"num_speculative_tokens": 2
}'
We support the K-EXAONE model on SGLang. You need to install our fork of the SGLang library to use the K-EXAONE model. Please check the requirements section. Practically, you can serve the model with a 256K context length using tensor parallel on 4 H200 GPUs.
python -m sglang.launch_server \
--model LGAI-EXAONE/K-EXAONE-236B-A23B \
--reasoning-parser qwen3 \
--disable-hybrid-swa-memory
A SGLang server will be available at http://localhost:30000.
[!NOTE] Currently, using the OpenAI-compatible server is incompatible with the
transformers>=5.0.0rc0, so you need to use SGLang native API for now. For native API, please refer to the official documentation.Once the issue is resolved, we will update this section accordingly.
You can test the SGLang server by sending a request as below:
from transformers import AutoTokenizer
import requests
model_name = "LGAI-EXAONE/K-EXAONE-236B-A23B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
messages = [
{"role": "user", "content": "How many r'\''s in \"strawberry\"?"}
]
input_text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
return_tensors="pt",
)
response = requests.post(
f"http://localhost:30000/generate",
json={
"text": input_text,
"sampling_params": {
"temperature": 1.0,
"top_p": 0.95,
"max_new_tokens": 16384,
},
},
)
print(response.json()['text'])
If you are interested in in using MTP weights for speculative decoding, add according options as below.
python -m sglang.launch_server \
--model LGAI-EXAONE/K-EXAONE-236B-A23B \
--reasoning-parser qwen3 \
--disable-hybrid-swa-memory \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4
The K-EXAONE language model has certain limitations and may occasionally generate inappropriate responses. The language model generates responses based on the output probability of tokens, and it is determined during learning from training data. While we have made every effort to exclude personal, harmful, and biased information from the training data, some problematic content may still be included, potentially leading to undesirable responses. Please note that the text generated by K-EXAONE language model does not reflect the views of LG AI Research.
LG AI Research strives to reduce potential risks that may arise from K-EXAONE language models. Users are not allowed to engage in any malicious activities (e.g., keying in illegal information) that may induce the creation of inappropriate outputs violating LG AI's ethical principles when using K-EXAONE language models.
The model is licensed under K-EXAONE AI Model License Agreement
@article{k-exaone,
title={K-EXAONE Technical Report},
author={{LG AI Research}},
journal={arXiv preprint arXiv:TBD},
year={2025}
}
LG AI Research Technical Support: [email protected]