Macaron-V1-Tall-l2-coding

Macaron-V1-Tall (GGUF Pre-Merged Specialists)

This repository contains the GGUF-quantized, pre-merged specialist models for Macaron-V1-Tall, optimized for local deployment on consumer hardware.

🏆 Acknowledgments and Credits

All foundational research, training, and architecture design belong to the original authors. This repository solely provides a community-driven quantization and deployment format.

• Original Authors: mindlab-research/Macaron-V1-Tall
• Original Repository: mindlab-research/Macaron-V1-Tall
• Base Architecture: Qwen 3.6 (35B Base + 4x 3.7B LoRA Specialists)

🧩 The Architectural Pivot (Why this repository exists)

The Macaron-V1-Tall architecture utilizes a Mixture-of-LoRA (MoE) pattern over a Grouped-Query Attention (GQA) base model. Currently, the standard llama.cpp ecosystem encounters tensor reshaping limitations (NotImplementedError) when attempting to dynamically apply these specific LoRA adapters at runtime.

To bypass this mathematical constraint and enable seamless local execution, this repository utilizes a Pre-Merge Strategy. Each LoRA specialist has been physically fused into the 35B base model using CPU-RAM computation (bypassing VRAM bottlenecks) prior to GGUF conversion.

The Result: Four independent, fully fused GGUF models. Instead of dynamically swapping LoRAs in VRAM, developers can leverage operating system Page Caching to swiftly alternate between these ~21GB files in RAM, enabling rapid intent-based routing without out-of-memory (OOM) errors.

📦 Available Specialists (Q4_K_M Quantization)

The model have been quantized to Q4_K_M to balance perplexity and memory footprint, reducing the required storage from ~70GB (FP16) to approximately 21GB per specialist.

• macaron-l2-coding-35b-q4_k_m.gguf (Coding Specialist)

Note: The experimental mtp_num_hidden_layers (Multi-Token Prediction) metadata has been sanitized from the config to ensure strict compatibility with the llama.cpp loader.

🚀 Deployment & Usage (llama.cpp)

These models are heavily optimized for hybrid VRAM/RAM offloading. If you are running a consumer GPU (e.g., 16GB VRAM) backed by substantial system RAM, you must carefully balance the layers to prevent VRAM saturation.

Example of a Launch Command (llama-server)

The following command demonstrates how to load the model while explicitly offloading the heaviest MoE layers to the CPU, freeing up your VRAM for the KV cache and attention heads.

./llama-server \
  -m "macaron-l2-coding-35b-q4_k_m.gguf" \
  --host 127.0.0.1 \
  --port 9091 \
  -ngl 40 \
  -ncmoe 10 \
  --ctx-size 16384 \
  -b 512 \
  -fa on \
  -ctk q4_0 \
  -ctv q4_0 \
  -t 16 \
  --reasoning-preserve \
  --no-context-shift \
  -sps 0.0 \
  --ctx-checkpoints 0 \
  --models-max 1 \
  --parallel 1 \
  --no-mmap \
  --metrics