vibe-thinker-3b-gguf-q8-0

VibeThinker 3B - GGUF (Q8_0)

This model is a quantized gguf-format export of WeiboAI/VibeThinker-3B for ease of use in edge devices and CPU-based inference environments. The original model is transformed into gguf with Q8_0 tensors by the script convert_hf_to_gguf.py.

Usage

This is an example of usage based on the libraries llama-cpp-python, transformers and huggingface_hub.

Requirements:

pip install huggingface_hub transformers llama-cpp-python

Example:

import os
import glob
from llama_cpp import Llama
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download


class VibeThinkerGGUF:
    def __init__(self, model_source: str = "fdemelo/vibe-thinker-3b-gguf-q8-0") -> None:
        """
        Args:
            model_source (str): A Hugging Face repo ID (e.g., "fdemelo/vibe-thinker-3b-gguf-q8-0") 
                OR a path to a local directory (e.g., "./my_model_folder").
        """
        if os.path.isdir(model_source):
            print(f"Found local directory. Loading from: {model_source}")
            self.model_folder = model_source
        else:
            print(f"'{model_source}' is not a local directory. Assuming Hugging Face Repo ID...")

            # Create a sensible local folder name based on the repo ID
            # e.g., "fdemelo/vibe-thinker-3b-gguf-q8-0" -> "./vibe-thinker-3b-gguf-q8-0"
            repo_name = model_source.split("/")[-1]
            self.model_folder = f"./{repo_name}"

            print(f"Downloading from repo '{model_source}' to '{self.model_folder}'...")
            snapshot_download(
                repo_id=model_source,
                local_dir=self.model_folder,
                local_dir_use_symlinks=False
            )

        gguf_files = glob.glob(os.path.join(self.model_folder, "*.gguf"))
        if not gguf_files:
            raise FileNotFoundError(f"No .gguf file was found in {self.model_folder}.")

        model_path = gguf_files[0]

        self.model = Llama(
            model_path=model_path,
            n_ctx=4096,
            n_threads=4,        
            verbose=False       
        )

        self.tokenizer = AutoTokenizer.from_pretrained(
            self.model_folder,
            trust_remote_code=True,
        )

    def infer_text(self, prompt: str) -> str:
        messages = [{"role": "user", "content": prompt}]
        text = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True,
        )

        response = self.model(
            prompt=text,
            max_tokens=4096,
            temperature=1.0,
            top_p=0.95,
            top_k=50,
            repeat_penalty=1.0, 
            stop=["<|im_end|>", "<|endoftext|>"] 
        )

        return response["choices"][0]["text"]


if __name__ == "__main__":
    model = VibeThinkerGGUF("fdemelo/vibe-thinker-3b-gguf-q8-0")
    
    prompt = "Tell me a joke"
    print(model.infer_text(prompt))

Refer to https://github.com/WeiboAI/VibeThinker for more details.

Licensing

The license is derived from the original model: MIT. For more details, please refer to WeiboAI/VibeThinker-3B.