lemmy

<!-- This content is subject to the European Union Public Licence (EUPL-1.2). For full licence details, please refer to: https://huggingface.co/lthn/lemmy/tree/main/LICENSE Origin URL: https://huggingface.co/lthn/lemmy/tree/main -->

Lemmy — Gemma 4 26B A4B MoE (GGUF)

The Mixture-of-Experts member of the Lemma model family by Lethean. An EUPL-1.2 fork of Gemma 4 26B A4B with the Lethean Ethical Kernel (LEK) merged into the weights — consent-based reasoning baked into the attention projections via LoRA finetune, then merged so inference uses a single standalone model with no PEFT runtime required. 25.2B total parameters, 3.8B active per forward pass (8 experts active + 1 shared out of 128 total) — delivering near-4B inference speed with the knowledge capacity of a 26B model.

This repo ships the GGUF multi-quant build for Ollama, llama.cpp, LM Studio, and other gguf-compatible runners. The unmodified Gemma 4 26B A4B fork lives at LetheanNetwork/lemmy for users who want the raw Google weights without the LEK shift.

Looking for MLX? The native Apple Silicon builds live in sibling repos: lthn/lemmy-mlx (4-bit default) | lthn/lemmy-mlx-8bit | lthn/lemmy-mlx-bf16 (full precision)

A lemma is "something assumed" — an intermediate theorem on the path to a larger proof, or a heading that signals the subject of what follows. The Lemma model family is named for that role: each variant is a stepping stone between raw capability and ethical application.

Why Lemmy for Local Inference

Lemmy is the MoE member of the Lemma family, and it is uniquely positioned for local execution-agent workloads:

• ~4B inference speed (only 3.8B active params per token) means generation is fast even on modest hardware
• 26B knowledge capacity accessible via expert routing — much larger than a same-speed dense 4B model
• 256K context — double the dense Lemma family variants
• Consent-grounded attractor from LEK training — maintains posture under instruction-following workloads at rates that compete with frontier models on strict instruction following

If you're replacing paid API calls with local inference for structured execution work (code review, agent dispatch, instruction following, spec-driven implementation), lemmy is the target.

Repo Files

GGUF variants are published separately at lthn/lemmy-gguf — bf16, q8_0, q6_k, q5_k_m, q4_k_m. This split is a temporary artefact of our build pipeline and may be consolidated into this repo in a future update.

Quick Start

Apps & CLI

<details> <summary>Ollama (via lemmy-gguf)</summary>

ollama run hf.co/lthn/lemmy-gguf:Q4_K_M

</details> <details> <summary>llama.cpp (via lemmy-gguf)</summary>

Install via brew (macOS/Linux), winget (Windows), or build from source:

brew install llama.cpp        # macOS/Linux
winget install llama.cpp      # Windows

# Start a local OpenAI-compatible server with a web UI:
llama-server -hf lthn/lemmy-gguf:Q4_K_M

# Run inference directly in the terminal:
llama-cli -hf lthn/lemmy-gguf:Q4_K_M

</details> <details> <summary>MLX (Apple Silicon native)</summary>

uv tool install mlx-lm
mlx_lm.chat --model lthn/lemmy
mlx_lm.generate --model lthn/lemmy --prompt "Hello, how are you?"

</details>

Python Libraries

<details> <summary>llama-cpp-python (via lemmy-gguf)</summary>

uv pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="lthn/lemmy-gguf",
    filename="lemmy-q4_k_m.gguf",
)

# Text
llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)

</details> <details> <summary>mlx-vlm (vision)</summary>

uv tool install mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("lthn/lemmy")
config = load_config("lthn/lemmy")

image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

output = generate(model, processor, formatted_prompt, image)
print(output.text)

</details>

Servers (OpenAI-compatible API)

<details> <summary>MLX Server</summary>

lemmy is multimodal (text + image), so use mlx_vlm.server — the vision-aware variant. The text-only mlx_lm.server does not correctly route multimodal tensors for Gemma 4.

mlx_vlm.server --model lthn/lemmy

curl -X POST "http://localhost:8080/v1/chat/completions" \
    -H "Content-Type: application/json" \
    --data '{
        "model": "lthn/lemmy",
        "messages": [{"role": "user", "content": "Hello, how are you?"}],
        "max_tokens": 200
    }'

Works with any OpenAI-compatible client at http://localhost:8080/v1.

</details> <details> <summary>vLLM</summary>

vLLM requires the original (non-quantised) safetensors weights from LetheanNetwork/lemmy — it does not load GGUF or MLX-quantised safetensors. Linux + NVIDIA GPU with adequate VRAM for a 26B MoE model.

uv pip install vllm
vllm serve "LetheanNetwork/lemmy"

</details>

Model Details

The Lemma Family

Capabilities

• Configurable thinking mode (<|think|> token in system prompt enables it; off by default in our examples via enable_thinking=False)
• Native function calling and system prompt support
• Variable aspect ratio image understanding
• Multilingual support (140+ languages)
• Sparse MoE attention with 8+1 expert routing per layer
• Long context (256K tokens) for document-scale reasoning

Roadmap

This release of lemmy is Gemma 4 26B A4B MoE with the Lethean Ethical Kernel (LEK) merged in — axiom-based reasoning baked into the attention weights via LoRA finetune, then merged into the base so inference uses a single standalone model with no PEFT runtime required. The unmodified Gemma 4 26B A4B fork lives at LetheanNetwork/lemmy for users who want the raw Google weights without the LEK shift.

Observed during LEK training: lemmy converged to a training-loss floor on par with the much larger 31B dense variant despite having ~4× fewer LoRA parameters. Hypothesis: MoE routing multiplicity provides implicit LoRA replication across expert paths, amplifying the effective gradient signal per trainable parameter. This is a testable prediction being verified via the 8-PAC evaluation pipeline.

The LEK axioms are public domain and published at Snider/ai-ethics. Track research progress at LetheanNetwork and the LEM-research dataset.

Why EUPL-1.2

Lemmy is licensed under the European Union Public Licence v1.2 — not Apache 2.0 or MIT. This is a deliberate choice:

• 23 official languages, one legal meaning. EUPL is the only OSS licence designed by lawmakers across multiple legal systems. "Derivative work" means the same thing in German, French, Estonian, and Maltese law.
• Copyleft with compatibility. Modifications must be shared back, but the licence plays cleanly with GPL, LGPL, MPL, and other major OSS licences. No accidental relicensing.
• No proprietary capture. Anyone can use lemmy commercially — but they cannot fork it, train a competitor model on it, and close-source the result. The ethical layer stays in the open.
• Built for institutions. Government, research, and enterprise users get a licence designed for cross-border compliance, not a US-centric one.

Recommended Sampling

Use Google's standardised settings across all use cases:

Gemma 4 is calibrated for temperature: 1.0 — this is not the same as the typical 0.7 default for other models. Lower values reduce diversity without improving quality. These defaults are pre-configured in generation_config.json.

Variable Image Resolution

Gemma 4 supports a configurable visual token budget that controls how many tokens represent each image. Higher = more detail, lower = faster inference.

For multimodal prompts, place image content before text for best results.

The default budget (280) is set in processor_config.json via image_seq_length and max_soft_tokens.

Benchmarks

Live evaluation results published to the LEM-benchmarks dataset. The lemmy-specific results live at LEM-benchmarks/results/lemmy.

The 8-PAC eval pipeline runs continuously on our homelab and publishes results as they complete. Categories: ethics, reasoning, instruction-following, coding, multilingual, safety, knowledge, creativity.

Resources

About Lethean

Lethean is a social enterprise building ethical AI infrastructure. The Lemma model family is part of the LEM (Lethean Ethical Model) project — training protocol and tooling for intrinsic ethical alignment of language models.

• Website: lthn.ai
• GitHub: LetheanNetwork
• Licence: EUPL-1.2