Nanbeige4.1-3B-MLX-8bit

Nanbeige4.1-3B-MLX-8bit (8-bit Quantized)

This is the Nanbeige4.1-3B model converted to MLX format with 8-bit quantization (affine, group_size=64) for Apple Silicon. This variant offers a good balance between quality and efficiency — near-original precision at roughly half the size.

Other variants:

• andrevp/Nanbeige4.1-3B-MLX-4bit — Smallest & fastest (~2.06 GB)
• andrevp/Nanbeige4.1-3B-MLX-BF16 — Full precision (~7.3 GB)

Original Model

• Source: Nanbeige/Nanbeige4.1-3B
• Developer: Nanbeige (BOSS Zhipin)
• Architecture: LlamaForCausalLM (4B parameters)
• License: Apache 2.0 (same as the original model)

Conversion Details

Performance on Apple Silicon

All Variants Compared

Performance

Quality Comparison (Head-to-Head, Identical Prompts, temp=0)

All three variants were tested with the same prompts under deterministic settings (temperature=0) to evaluate quality differences:

Key findings:

• 8-bit vs BF16: Produced word-for-word identical reasoning and answers in the majority of tests. Essentially zero quality loss.
• 4-bit vs BF16: Sometimes takes slightly different reasoning paths, but arrives at the same correct answers. Tool calling output is 100% identical across all variants.
• Recommendation: 4-bit is best for speed and memory. 8-bit is the sweet spot for quality. BF16 is for research and benchmarking where exact reproduction matters.

Capabilities

This model retains all capabilities of the original Nanbeige4.1-3B:

• Reasoning/Thinking: Uses <think>...</think> tags for chain-of-thought reasoning
• Tool/Function Calling: Generates structured <tool_call>...</tool_call> JSON output
• Multi-turn Conversation: Supports multi-turn chat with context tracking
• Multilingual: Strong performance in both English and Chinese
• Code Generation: Capable of writing and explaining code
• Deep-Search Agent: Supports deep-search tasks with 500+ rounds of tool invocations (using tokenizer_config_search.json)

Quickstart

Installation

pip install mlx-lm

CLI Usage

mlx_lm generate \
  --model andrevp/Nanbeige4.1-3B-MLX-8bit \
  --prompt "Explain quantum computing in simple terms." \
  --max-tokens 512 \
  --temp 0.6 \
  --top-p 0.95

Python - Chat

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("andrevp/Nanbeige4.1-3B-MLX-8bit")
sampler = make_sampler(temp=0.6, top_p=0.95)

messages = [
    {"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}
]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False
)
response = generate(
    model, tokenizer, prompt=prompt, max_tokens=512, sampler=sampler
)
print(response)

Python - Tool Calling

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("andrevp/Nanbeige4.1-3B-MLX-8bit")
sampler = make_sampler(temp=0.6, top_p=0.95)

messages = [
    {"role": "user", "content": "What is the weather in Tokyo?"}
]
tools = [
    {
        "type": "function",
        "function": {
            "name": "SearchWeather",
            "description": "Find the current weather in a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City name"
                    }
                },
                "required": ["location"]
            }
        }
    }
]
prompt = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, tokenize=False
)
response = generate(
    model, tokenizer, prompt=prompt, max_tokens=256, sampler=sampler
)
print(response)

Python - Multi-turn with Tool Responses

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("andrevp/Nanbeige4.1-3B-MLX-8bit")
sampler = make_sampler(temp=0.6, top_p=0.95)

tools = [
    {
        "type": "function",
        "function": {
            "name": "SearchWeather",
            "description": "Find the current weather in a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"}
                },
                "required": ["location"]
            }
        }
    }
]

messages = [
    {"role": "user", "content": "What's the weather like in Paris?"},
    {"role": "assistant", "content": "", "tool_calls": [
        {"function": {"name": "SearchWeather", "arguments": '{"location": "Paris"}'}}
    ]},
    {"role": "tool", "content": '{"temperature": "18°C", "condition": "Partly cloudy"}'}
]

prompt = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, tokenize=False
)
response = generate(
    model, tokenizer, prompt=prompt, max_tokens=300, sampler=sampler
)
print(response)

Recommended Inference Hyperparameters

Benchmarks (Original Model)

General Reasoning Tasks

Deep Search Benchmarks

Files Included

Special Tokens

Deep-Search Mode

For deep-search agent capabilities, switch to tokenizer_config_search.json and use the miroflow-framework for inference. See the original model card for detailed deep-search setup instructions.

Important: Extended Thinking Behavior

Nanbeige4.1-3B is a reasoning model that generates chain-of-thought inside <think>...</think> tags before producing a final answer. This is by design — and it affects all variants equally (4-bit, 8-bit, BF16).

What to expect

For complex tasks (translation, creative writing, open-ended questions), the model may spend 3,000–5,000+ tokens reasoning before delivering an answer. If max_tokens is too low, the output will be truncated mid-thinking and no final answer will appear — this is not a failure, the model simply needs more tokens to finish its thought process.

Workarounds

1. Increase max_tokens (recommended)

# For complex tasks, use high max_tokens
response = generate(
    model, tokenizer, prompt=prompt,
    max_tokens=8192,  # or higher for very complex tasks
    sampler=sampler
)

2. Skip thinking (experimental)

You can pre-fill an empty thinking block to force the model to answer directly. This does not always work — the model may re-enter thinking mode — but it can help for simpler open-ended tasks:

prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False
)
prompt += '<think>\n\n</think>\n\n'  # Force empty thinking block
response = generate(
    model, tokenizer, prompt=prompt,
    max_tokens=512, sampler=sampler
)

3. Post-process to extract the answer

if '</think>' in response:
    answer = response.split('</think>')[-1].strip()
else:
    answer = response  # Still in thinking — increase max_tokens

Limitations

While safety was emphasized during training, the model may still generate unexpected outputs due to its size and probabilistic nature. Users should not propagate harmful content. The developers assume no responsibility for consequences from dissemination of inappropriate content.

License

This model is released under the Apache 2.0 License, the same license as the original Nanbeige4.1-3B model.

Credits

• Original model: Nanbeige/Nanbeige4.1-3B by Nanbeige (BOSS Zhipin)
• MLX framework: Apple MLX
• Conversion: mlx-lm
• Contact (original model): [email protected]