GigaChat3-10B-A1.8B-bf16-gguf

GigaChat3-10B-A1.8B-bf16 Quantized Model (GGUF format)

Overview

This repository contains the quantized versions of GigaChat3-10B-A1.8B model in GGUF format made from original BF16 precision, ready for deployment and inference.

Files

• GigaChat3-10B-A1.8B-QX_X_X.gguf: The main quantized model files in GGUF format.
• chat_template.jinja: Original template file for chat interaction, located in the repository.
• chat_template_lmstudio.jinja: Custom template file with LM Studio support for chat interaction, located in the repository.

Special Requirements

To run this model, you need the b7127 (or higher) release of llama.cpp containing a PR with a patch by ubergarm to correctly detect the model architecture, which can be found here: https://github.com/ggml-org/llama.cpp/releases/tag/b7127

Usage Examples

Bash (Linux)

# server start

CUDA_VISIBLE_DEVICES=0 ./llama.cpp/build/bin/llama-server   --model ./models/GigaChat3-10B-A1.8B-Q4_K_M.gguf   --ctx-size 2000   --parallel 1   --threads 8   --host 0.0.0.0   --port 8088   -cmoe   --jinja   --chat-template-file ./templates/chat_template.jinja

###

# sending a request

curl -X POST http://localhost:8088/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Что такое GigaChat?"}],
  "max_tokens": 200,
  "temperature": 0.5,
  "top_p": 0.95,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "n": 1
}'

PowerShell (Windows)

# server start

$env:CUDA_VISIBLE_DEVICES = "0"
$env:model = ".\models\GigaChat3-10B-A1.8B-Q4_K_M.gguf"

.\llama.cpp\build\bin\Release\llama-server.exe `
  --model $env:model `
  --ctx-size 2000 `
  --parallel 1 `
  --threads 8 `
  --host 0.0.0.0 `
  --port 8088 `
  -cmoe `
  --jinja `
  --chat-template-file .\templates\chat_template.jinja

###

# sending a request

[Console]::OutputEncoding = [System.Text.Encoding]::UTF8

$response = Invoke-RestMethod -Method POST -Uri http://localhost:8088/v1/chat/completions `
  -ContentType "application/json" `
  -Body '{
    "messages": [ { "role": "user", "content": "Что такое GigaChat?" } ],
    "max_tokens": 200,
    "temperature": 0.5,
    "top_p": 0.95,
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "n": 1
  }'

$response | Format-List -Force

$response.choices[0].message.content

###

# output

GigaChat — это искусственный интеллект российского производства, разработанный Сбером. Он может отвечать на вопросы, помогать в написании текстов, программировании и других задачах.

Если у вас есть вопрос, можете его задать!

**Ответ:** GigaChat — это российская нейросеть, созданная командой Сбера для общения с людьми, генерации текстов, помощи в программировании и других задачах.