GPT-OSS-Swallow-120B-RL-v0.1-GGUF

GPT-OSS-Swallow-120B-RL-v0.1 (GGUF)

GGUF quantizations for tokyotech-llm/GPT-OSS-Swallow-120B-RL-v0.1.

This repository focuses on reproducibility and practical usage (llama.cpp / LM Studio / Ollama).

Source model

• Base model: tokyotech-llm/GPT-OSS-Swallow-120B-RL-v0.1
• License: Apache-2.0 (inherits from the source model)

Files

Hardware notes (rough)

• These GGUF files run on CPU-only as well, but GPU offload is recommended for speed.
• If you see OOM, try:
  • smaller quant (Q4_K_M)
  • lower context length
  • fewer GPU layers (-ngl)

Reproducibility

• llama.cpp commit: de5be8619609a72c15105044625dc623627789bc
• Steps (high level):
  • Convert HF → GGUF (F16): convert_hf_to_gguf.py --outtype f16
  • Quantize: llama-quantize to Q8_0 / Q6_K / Q5_K_M / Q4_K_M

Quick start (llama.cpp)

CPU-only

llama-cli -m GPT-OSS-Swallow-120B-RL-v0.1.Q5_K_M.gguf \
  -ngl 0 \
  -p "日本語と英語で自己紹介してください。" -n 128

### GPU offload (example)
```bash
llama-cli -m GPT-OSS-Swallow-120B-RL-v0.1.Q5_K_M.gguf \
  -ngl 40 \
  -p "日本語と英語で自己紹介してください。" -n 128

Tip: Tune -ngl depending on VRAM. Start from 10, 20, 40... and increase until stable.

## LM Studio

Download a GGUF file from this repo.

Open LM Studio → “Local Models” → load the GGUF.

Start with Q5_K_M, then move to Q6_K/Q8_0 if you have enough memory.

## Ollama (optional)

Ollama support may depend on build/runtime. If it fails, try LM Studio or llama.cpp.

Example Modelfile:

FROM ./GPT-OSS-Swallow-120B-RL-v0.1.Q5_K_M.gguf
PARAMETER temperature 0.7
PARAMETER num_ctx 4096
TEMPLATE """{{ .Prompt }}"""
## Acknowledgements

Original model: tokyotech-llm/GPT-OSS-Swallow-120B-RL-v0.1

GGUF tooling: ggml-org/llama.cpp

## Contact

Maintainer: nobikko