Qwimi-3.6-27B-Coder-MTP-mlx-8Bit

Qwimi-3.6-27B-Coder-MTP-MLX-8bit

A 8-bit MLX quantization of trjxter/Qwimi-3.6-27B-Coder-MTP-BF16, a coding-focused SFT of Qwen 3.6 27B.

The model was trained in one mixed SFT run across coding, Qwen-native XML-style tool calling, and agentic software-engineering trajectories. This repository is intended for local inference on Apple silicon using mlx-lm.

This is an MLX repository, not GGUF. It is not intended for llama.cpp, CUDA inference, or a GGUF-only Ollama workflow.

Quant position: highest-fidelity quantized MLX release.
Primary tradeoff: smallest quantization loss of the Q4/Q6/Q8 set.
Recommendation: Best when quality is the priority and the model fits comfortably.

---

Contents

1. Source and quantization
2. Which MLX quant should I use?
3. Unified-memory guidance
4. Installation
5. How to run
6. Tool calling
7. Training summary
8. Benchmarks
9. Known limitations
10. Verification and reproducibility

---

1. Source and quantization

• Merged source: trjxter/Qwimi-3.6-27B-Coder-MTP-BF16
• Base architecture: Qwen 3.6 27B, approximately 27.8B parameters
• Architecture family: dense hybrid transformer with standard attention and GatedDeltaNet layers
• Specialization: text-only coding SFT; vision tower frozen
• Format: MLX model weights for Apple silicon
• Nominal weight precision: 8-bit
• Expected conversion flag: --q-bits 8
• Validated SFT sequence length: 16,384 tokens

The uploaded repository's config.json is the source of truth for quantization mode, group size, and tensor metadata. This README intentionally does not invent a group size that may differ from the actual conversion.

The source name includes MTP. Do not assume that conversion to MLX automatically preserves or activates the same MTP runtime path used by a specific llama.cpp build.

---

2. Which MLX quant should I use?

Choose Q8 when you want the closest of these three quantized releases to BF16 behavior and your Mac has enough unified memory.

Prefer Q6 for a better size/quality balance or Q4 when memory and context headroom dominate.

Exact quality and memory behavior should be measured with the uploaded artifacts and your own workload.

---

3. Unified-memory guidance

• Practical target: 64 GB unified memory or more.
• 48 GB Macs: may work with controlled context and limited competing memory use.
• 96 GB+ Macs: provide generous headroom for long prompts, cache, IDEs, and agents.

These are rough planning targets, not guarantees. Actual memory use depends on the exact file size, group size, prompt length, generation length, cache configuration, macOS version, and other applications using unified memory.

---

4. Installation

MLX requires Apple silicon.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip mlx-lm

Qwen 3.6 / qwen3_5 support is relatively new, so update mlx-lm before diagnosing model, cache, server, or tool-parser problems.

---

5. How to run

One-shot generation

mlx_lm.generate \
  --model trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit \
  --prompt "Write a Python function that reverses a linked list and include tests."

Interactive chat

mlx_lm.chat --model trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit

Python API

from mlx_lm import generate, load

MODEL_ID = "trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit"
model, tokenizer = load(MODEL_ID)

messages = [
    {
        "role": "user",
        "content": "Implement an LRU cache in Python and explain the complexity.",
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)

response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=1024,
    verbose=True,
)

print(response)

Local OpenAI-style server

mlx_lm.server \
  --model trjxter/Qwimi-3.6-27B-Coder-MTP-MLX-8bit \
  --host 127.0.0.1 \
  --port 8080

mlx_lm.server is most appropriate as a local development endpoint, not a hardened public production service.

Context guidance

The base architecture may expose a larger native context, but Qwimi was trained and validated at 16,384 tokens. Longer-context behavior should be tested separately.

---

6. Tool calling

Qwimi was trained on Qwen 3.6's native XML-style tool-call format rather than the older plain JSON-object style.

<tool_call>
<function=search_web>
<parameter=query>latest MLX documentation</parameter>
</function>
</tool_call>

Recommendations:

1. Apply the repository's tokenizer chat template.
2. Test the exact mlx-lm version you plan to deploy.
3. Validate multi-turn tool calls, inserted tool results, and stop behavior.
4. Do not assume that working text generation guarantees correct server-side tool parsing.
5. Keep direct XML validation or a fallback parser for agent workflows.

---

7. Training summary

• Base: unsloth/Qwen3.6-27B
• Model size: approximately 27.8B parameters
• Method: 4-bit QLoRA during SFT, rank 64, alpha 64
• Trainable parameters: 466,911,232, approximately 1.68%
• Training style: one mixed SFT run, not three sequential stages
• Total training rows: 22,359
• Training tokens: approximately 72.5M
• Maximum sequence length: 16,384
• Effective batch size: 16
• Epochs: 1
• Optimizer steps: 1,398
• Hardware: 1× A100-SXM4-80GB
• Vision tower: frozen
• Final eval loss: coding 0.4761; tool-calling 0.0208; agentic 0.2892
• All three eval curves improved monotonically through the run with no observed overfitting signal.

trjxter/Kimi-K2.7-CodingTraces-9000x was used in full: 9,014 rows.

8. Benchmarks

Qwimi was compared against the base Qwen 3.6 27B model using equivalent Q6 GGUF builds with MTP enabled and the same evaluation harness.

Important: these results were produced by the Q6 GGUF / llama.cpp builds. They are included as evidence about the underlying Qwimi model, not as direct benchmarks of this MLX quant. MLX Q4, Q6, and Q8 should be benchmarked separately before making backend- or precision-specific claims.

8.1 Fully auto-scored 300-task custom benchmark

Category results

Pairwise outcomes

Among the 42 decisive tasks where only one model passed, Qwimi won 23/42 = 54.8%.

300-task verdict: a slight overall quality win for Qwimi, driven by stronger coding and structured tool-use performance, with a large reduction in end-to-end task time. Agentic accuracy remains the clearest weakness.

8.2 Mixed 200-task benchmark

A separate 200-task run produced 400 raw answer rows. Categories with incomplete scorer or export integration were omitted rather than counted as zero.

On the 146 tasks with a valid paired auto-score:

Among the nine decisive paired tasks, Qwimi won 7/9 = 77.8%.

8.3 Interpreting the latency results

The measured advantage is task-level wall time, not proof of faster kernels or higher per-token throughput.

Task time ≈ prefill time + generated tokens × decode latency per token

SFT can reduce task time by producing shorter, more direct, better-formatted outputs and reaching the stop condition sooner. A controlled comparison of prompt throughput, generation throughput, and generated-token counts is required to claim a per-token speed advantage.

8.4 Benchmark limitations

• The published comparison used Q6 GGUF under llama.cpp, not MLX.
• MLX Q4, Q6, and Q8 were not benchmarked against one another.
• Wall time varies with Mac hardware, mlx-lm version, context length, cache behavior, generation length, and sampling settings.
• The custom suite is useful internal evidence, not a replacement for broad public benchmarks.
• Agentic auto-scoring cannot capture every aspect of repository-level engineering quality.

9. Known limitations

• Text-only specialization: the vision tower was frozen during SFT and was not improved.
• Validated context: training and evaluation used a maximum sequence length of 16,384 tokens. Behavior beyond that length is unvalidated.
• Agentic coverage: agentic data was much smaller than the coding portion, and the custom benchmark showed a four-point agentic accuracy regression against the base model.
• Quantization loss: this MLX conversion adds quantization error on top of BF16.
• Apple Silicon only: MLX is designed for Apple silicon and is not a CUDA/Windows runtime.
• Bleeding-edge architecture: Qwen 3.6 uses the qwen3_5 hybrid architecture with GatedDeltaNet layers. Use a current mlx-lm build and verify generation locally.
• MTP caveat: the name retains MTP from the source model, but this does not guarantee that MLX preserves or actively uses the same MTP acceleration path as the GGUF benchmark.
• Tool-calling caveat: XML-style tool output and server-side parsing can be version-sensitive. Validate multi-turn tools before relying on them in an agent workflow.
• Server caveat: mlx_lm.server is intended primarily as a local endpoint and is not a hardened production service.

---

10. Verification and reproducibility

Record the exact:

• mlx-lm version;
• source BF16 commit SHA;
• conversion command;
• quantization mode, bit width, and group size;
• output repository commit SHA;
• uploaded size and weight-shard checksums;
• smoke-test prompt and structural result.

Representative conversion command:

mlx_lm.convert \
  --hf-path trjxter/Qwimi-3.6-27B-Coder-MTP-BF16 \
  --mlx-path ./Qwimi-3.6-27B-Coder-MTP-MLX-8bit \
  -q \
  --q-bits 8

Add the actual --q-group-size, quantization mode, and upload arguments used during conversion instead of assuming defaults.

Suggested smoke tests

1. Plain coding generation.
2. Exact <think>...</think> formatting.
3. A single XML-style tool call.
4. A multi-turn tool call with a tool result.
5. A 12K-16K-token prompt to check memory and cache behavior.
6. Repeated generation to detect truncation or cache regressions.

---

License

Apache 2.0, inherited from the base and merged source model.