jev-solomon-MLX-8-bit

Solomon MLX — 8-bit

Solomon is an open-weights decision model: a language model that does not write text. You hand it a document once and ask typed questions against it; every answer comes back as a calibrated probability over the options you listed — never a token outside them, never a chat turn, never a sample. The same document and the same question return the same numbers every time. This repository is an 8-bit derivative of the BF16 build for 48 GB Macs, packaged for Apple Silicon with the MLX runtime of the upstream release, DoccyHealth/Solomon (release 1.1.0, Apache-2.0).

The three builds

All three share the same adapter, heads and pins; they differ only in the backbone's precision.

Latency is the same at BF16 and at 8-bit, because it is set by the runtime rather than by the backbone's precision. One document with questions costs about 17 s of prefill (≈2,000 tokens at ≈120 tokens/s) plus about 2 s per candidate branch (≈110 tokens each), so a document with 7–10 candidates takes 30–35 s on either build. A microbenchmark of a 5120×5120 Linear layer puts MLX's affine 8-bit matmul at BF16 speed for prefill-sized inputs (8-bit 17.6 ms vs BF16 17.4 ms at 2,048 tokens; 2.8 vs 2.4 ms at 128 tokens) and 5× faster at one token, so precision is not the bottleneck: the per-token paths of the pinned runtime are — its linear-attention state-history path and its left-padded attention path step one token at a time. A speed patch to those paths is planned.

What it answers

It is for turning documents — contracts, letters, forms, tickets, listings, records — into structured, machine-readable answers with a number attached to each, where determinism and a refusal to drift matter more than fluency. It is not for open-ended question answering, chat, generation or summarisation.

Quickstart

Runtime patch. The upstream runtime accepts only its BF16 quality profile. runtime-patch-ac4f9cc.diff in this repository adds the quality-q8 and quality-q4 profiles (api.py) and a QuantizedLinear-aware LoRA shape check (engine.py), and carries the licence-text tolerance described under Provenance. Apply it to the pinned checkout with git apply runtime-patch-ac4f9cc.diff before loading this build.

git clone https://huggingface.co/DoccyHealth/Solomon && cd Solomon && git checkout ac4f9ccf
cd mlx && uv sync                      # Python 3.13, mlx 0.32.2, mlx-vlm 0.7.1 (pinned by the release)
hf download FrenchCastle/Solomon-MLX-8bit --local-dir ../models/quality-q8

import mlx.core as mx
from solomon_mlx import Solomon

mx.set_cache_limit(4 << 30)                  # cap MLX's buffer cache at 4 GB before loading (see below)
model = Solomon.load("../models/quality-q8", profile="quality-q8")

letter = """Rookwood Ltd confirms that its ISO 9001 certification is current until March 2028 and that
Meridian Logistics acts as its appointed carrier for all EU shipments. Payment terms are 30 days net."""

with model.prefill(letter) as state:
    result = model.decide(state=state, questions={
        "certified": {"type": "noul", "instructions": "Does the document establish that Rookwood Ltd holds a current certification?"},
        "terms": {"type": "choice", "instructions": "What payment terms does the document state?",
                  "options": ["prepayment", "30 days net", "60 days net", "not stated"]},
        "carrier": {"type": "noul", "instructions": "Is {candidate} the appointed carrier?",
                    "candidates": ["Meridian Logistics", "Rookwood Ltd", "Northgate Freight"]},
        "topics": {"type": "noul", "instructions": "Which of these does the document address?",
                   "candidates": ["certification", "payment terms", "warranty", "delivery schedule"],
                   "candidate_kind": "label"},
    })

print(result["answers"]["certified"]["noul"])            # e.g. 0.98
print(result["answers"]["terms"]["probabilities"])       # {"30 days net": 0.99, ...}
print(result["answers"]["carrier"]["candidates"])        # one probability per candidate
print(result["usage"])                                   # branches, input tokens, evidence calls

Keep the two mlx.core lines. MLX keeps every freed Metal buffer in a process-local cache, and the runtime's own memory guard reads only active memory, so a sequence of multi-branch questions can grow that cache by tens of gigabytes (measured: 90 GB free before one ten-branch question set, 14 GB free after) and push even a 128 GB Mac into swap. Cap the cache before loading, as above, and call mx.clear_cache() between documents; with the cap in place, memory stayed flat across many-branch documents.

Documents accept text, JSON objects, or ordered {"text": ...} / {"image": path} parts (page images go through the Qwen3.8 vision tower; there is no PDF renderer or OCR). evidence="support" adds text spans that point to where the support probably is. States can be saved and replayed. Hardware: about 36 GB of unified memory (a 48 GB Mac); the context ceiling is 40,960 tokens, prefilled in 2,048-token chunks.

How it works

• Base: Qwen/Qwen3.8-27B (Alibaba Cloud, Apache-2.0), pinned at 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
• Adaptation: a rank-64 LoRA applied on the question side only — the adapter is off while the document is read and on from the question onward — plus ten trained linear heads that read structured letter logits at a fixed position. Nothing is generated and parsed.
• Calibration: one positive temperature per answer type, fitted by the upstream authors and carried inside the runtime binding (yes/no 1.95 · single 1.0 · ordered 1.51 · multi-label 1.69 · entity 2.18); the MLX API serves at T = 1 unless you pass the calibration artifact.
• Execution: the document is prefilled once into a cached state; each question is an isolated branch off that prefix, so question order does not change answers, and cached answers match a full forward pass to within 1e-3.

Quantization

The quantization is post-training, round-to-nearest, applied to the verified BF16 build; binding.json names the profile (quality-q8), the quantization, the upstream pins, the sha256 of every file here and, under hub_preparation.derived_from, the BF16 binding it was derived from. Its effect on answer accuracy has not yet been measured; until it is, expect its probabilities to differ slightly from the BF16 model's, and use the BF16 build where the calibration figures below matter. It runs at the same speed as the BF16 build (see the latency note under The three builds) and needs about 36 GB of unified memory instead of about 60 GB.

Evaluation (upstream, BF16 model)

Measured by the upstream authors on their held-out synthetic panel (documents of the same generator as the training data):

On the one panel of real documents they report (40 documents, 198 questions, adjudicated agent labels, not human-verified), whole-question agreement is 0.81 (yes/no 0.98, single 1.00, ordered 0.95, multi-label 0.70, entity 0.44). With no document at all, on general-knowledge benchmarks, it tracks its base model (ARC-Challenge 97.2%, MMLU 82.4%, MMLU-Pro 60.5%) while being more overconfident than it. Figures for the quantized builds will be added as they are measured.

Limitations

• The synthetic-to-real gap is real and concentrated in entity and multi-label questions. The dominant error is a confident "No" on candidates the document never mentions, where "not established" would be right. If your questions ask about things that may simply be absent, this will hurt you.
• Probability magnitudes do not transfer out of domain, and there is no abstention: every question is answered; any threshold is your policy on your population.
• The rolled-up score of a multi-candidate question is a product of per-candidate probabilities — an ordering, not a validated joint probability. Read the per-candidate values.
• Yes/no answers above 0.99 are the least trustworthy high-confidence signal.
• Images are accepted but were not benchmarked; treat image answers as functional and uncalibrated.
• Nothing here is a certified error rate.

Provenance and verification

binding.json pins everything: the base revision, the Solomon revision, the sha256 of every backbone shard, the adapter (2addaf841ecc5882…) and the heads (126a9b5487dca937…), the converter identity and the runtime contract (solomon-mlx-binding-v1). The runtime re-hashes every file at load and refuses a package that does not match. The BF16 build was assembled with the release's own converter (solomon-mlx-hub prepare), and re-verified with solomon-mlx-hub verify.

One deliberate deviation: at revision ac4f9ccf the upstream mlx/bf16/NOTICE and MODIFICATIONS.md were edited (commit 5c0a4a8) without release.json being refreshed, so the release's own check of those two licence texts cannot pass on any copy of the repository. The loader was patched to tolerate a mismatch on those two files only, with a warning; every other check was enforced. The texts are included here verbatim, and binding.json records their actual hashes.

License and attribution

Apache-2.0 throughout. The backbone is a derivative of Qwen/Qwen3.8-27B, Copyright 2026 Alibaba Cloud (Apache-2.0); the adapter, heads and runtime are Copyright 2026 Doccy Pty Ltd (Apache-2.0), see LICENSE, NOTICE and MODIFICATIONS.md. "Qwen" and "Alibaba Cloud" are used nominatively to identify the base model; no endorsement or affiliation is implied. This conversion adds no restrictions.

Citation

@misc{solomon2026,
  title        = {Solomon: a decision model that returns calibrated probabilities over typed questions},
  author       = {{Doccy Pty Ltd}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DoccyHealth/Solomon}},
  note         = {release 1.1.0, revision ac4f9ccf; MLX packaging by FrenchCastle}
}