sarashina2-7b-it-GGUF

Sarashina2-7B Instruct GGUF (Q4_K_M)

This repository contains a Q4_K_M GGUF quantization of alfredplpl/sarashina2-7b-it, an instruction-tuned version of sbintuitions/sarashina2-7b.

The model supports Japanese and English. Its chat template is embedded in the GGUF file and is also included in tokenizer_config.json.

Files

The GGUF v3 file contains 291 tensors and 7,315,132,416 parameters. Its tensor types are 193 Q4_K, 33 Q6_K, and 65 F32, which is the mixed layout expected for Q4_K_M. The embedded general.file_type is 15 and general.quantization_version is 2.

Architecture

The included config.json was normalized from the source model configuration and checked against the GGUF metadata:

The tokenizer is a SentencePiece unigram tokenizer with byte fallback. Token IDs are BOS 1, EOS 2, and PAD 3.

Quantization recipe

This artifact was converted and quantized with llama.cpp revision 95a923a64c7d493ed1cb347d3b55d039fa3b8097 on Apple Silicon. No importance matrix was used for the distributed Q4_K_M file.

The original run used the now-deprecated huggingface-cli download command. The equivalent recipe below uses the current hf download command and pins the exact source-model revision for reproducibility.

# 1. Check out the exact llama.cpp revision
git clone https://github.com/ggml-org/llama.cpp.git /private/tmp/llama.cpp
cd /private/tmp/llama.cpp
git checkout 95a923a64c7d493ed1cb347d3b55d039fa3b8097

# 2. Build for Apple Silicon with Metal enabled
cmake -B build -DLLAMA_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j "$(sysctl -n hw.ncpu)"

# 3. Download the exact source model
mkdir -p /tmp/sarashina2-7b-it-src
hf download alfredplpl/sarashina2-7b-it \
  --revision ffabfdf2d19b35d268d47d2c546e9b7a600f1c0b \
  --local-dir /tmp/sarashina2-7b-it-src

# 4. Supply the tokenizer filename expected by the converter
cp /tmp/sarashina2-7b-it-src/spiece.model \
  /tmp/sarashina2-7b-it-src/tokenizer.model

# 5. Convert the source weights to BF16 GGUF
python convert_hf_to_gguf.py /tmp/sarashina2-7b-it-src \
  --outfile /tmp/sarashina2-7b-it-bf16.gguf \
  --outtype bf16

# 6. Quantize without an importance matrix
./build/bin/llama-quantize \
  /tmp/sarashina2-7b-it-bf16.gguf \
  sarashina2-7b-it-q4_k_m.gguf \
  Q4_K_M \
  "$(sysctl -n hw.ncpu)"

At this pinned revision, LLAMA_METAL is accepted as a deprecated alias for GGML_METAL. The quantizer run reported an output of approximately 4.27 GB; the final artifact distributed here was subsequently measured as exactly 4,481,575,744 bytes (4.48 GB / 4.17 GiB).

Optional importance-matrix variant

The following variant is provided for reproduction experiments only. It was not used to create the file in this repository. At the pinned revision, llama-imatrix is built as a tool; use -DLLAMA_BUILD_TOOLS=ON if it is not present.

cmake -B build \
  -DLLAMA_METAL=ON \
  -DLLAMA_BUILD_TOOLS=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j "$(sysctl -n hw.ncpu)"

./build/bin/llama-imatrix \
  -m /tmp/sarashina2-7b-it-bf16.gguf \
  -f training_texts.txt \
  --output-file imatrix.dat \
  -ngl 99

./build/bin/llama-quantize \
  --imatrix imatrix.dat \
  /tmp/sarashina2-7b-it-bf16.gguf \
  sarashina2-7b-it-q4_k_m-imatrix.gguf \
  Q4_K_M \
  "$(sysctl -n hw.ncpu)"

Usage with llama.cpp

hf download RyoSpiralArchitect/sarashina2-7b-it-GGUF \
  sarashina2-7b-it.Q4_K_M.gguf \
  --local-dir .

llama-cli \
  -m sarashina2-7b-it.Q4_K_M.gguf \
  -cnv \
  -n 256

To serve an OpenAI-compatible local endpoint:

llama-server \
  -m sarashina2-7b-it.Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080

Usage with Transformers

Transformers can read GGUF checkpoints when the filename is provided. The weights are dequantized during loading, so this path needs substantially more memory than inference through llama.cpp.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "RyoSpiralArchitect/sarashina2-7b-it-GGUF"
gguf_file = "sarashina2-7b-it.Q4_K_M.gguf"

tokenizer = AutoTokenizer.from_pretrained(repo_id, gguf_file=gguf_file)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    gguf_file=gguf_file,
    dtype=torch.bfloat16,
)

Prompt format

The embedded chat template formats messages as:

<|system|>
{system message}</s>
<|user|>
{user message}</s>
<|assistant|>
{assistant message}</s>

When generation begins, the final assistant prefix is:

<|assistant|>

Provenance

• Fine-tuned source: alfredplpl/sarashina2-7b-it at revision ffabfdf2d19b35d268d47d2c546e9b7a600f1c0b
• Base model: sbintuitions/sarashina2-7b at revision 15b989b8a4eb6f68b990787fc1016def7dabf0bb
• Fine-tuning dataset: kunishou/oasst2-chat-68k-ja
• Quantization uploader: RyoSpiralArchitect

The GGUF itself records the quantization type and tensor layout but not the complete conversion command. The recipe above is producer-supplied provenance for this artifact and documents that no importance matrix was used.

License

The fine-tuned source repository declares the MIT license, and the base model is also distributed under the MIT license. A copy of the base model's license is included in this repository. The fine-tuning dataset separately declares the Apache-2.0 license.

Please review the linked source model, base model, and dataset cards before deployment. Quantization does not remove the original model's limitations: generated text may be inaccurate, biased, objectionable, or otherwise unsafe.