MacJev-322M-4K-Laya-GGUF

MacJev-322M-4K-Laya-GGUF

<!-- jevstyle:project-links:start -->

Local classification, routing and fixed-choice scoring with JevStyle.

MacJev-322M-4K-Laya is an earlier, Laya-based decision model for local Mac agents (2026-09-23), maintained by Chaoliang Yan and kept as an archive; it used no Jev outputs and runs only through the runtime shipped in this repository (the jev-style package does not load it). The current recommended release is 2B v4 (text and images; trained in part on soft labels distilled from TypeSafe Jev 1.14 through its public API) and, for the smallest text model, 0.8B v3. JevStyle is independently maintained by Chaoliang Yan, with no TypeSafe affiliation or endorsement. Training provenance is release-specific.

Official local installation of the current releases — no hosted API key or agent simulation: python -m pip install -U "jev-style[torch]" and jev-style serve --release 2b (2B v4; --release 0.8b for the 0.8B v3), or python -m pip install -U "jev-style[mlx]" on Apple silicon for text. The package runs downloaded JevStyle weights on your machine. Step-by-step installation and verification. TypeSafe/OpenRouter Jev access and skills that simulate decisions with another agent model are separate execution paths.

Hosted service is separate from local inference. Hosted API: operational. It serves the 2B v4 (text only; 8,192-token context, inputs up to 8,000 tokens). There is currently no fixed per-account quota: no daily request count and no per-minute cap. Capacity is shared, so a busy server answers 429 with a Retry-After header and clients should retry. Check current service status or live health when using the hosted API.

Choose a workflow: Python text classification, local routing tools and Apple silicon, receipt or screenshot yes/no decisions, reproducible routing comparison. These guides distinguish suitable alternatives and published evidence; they do not establish universal superiority.

<!-- jevstyle:project-links:end -->

Website: jevstyle.com/models — all JevStyle decision models, benchmarks and quickstart in one place.

MacJev on stock llama.cpp: an F16 encoder GGUF, a decision head, and a small runtime. No patched binaries.

MacJev-322M-4K-Laya is a compact decision model for local Mac agents. You give it an observed state and one typed question with candidate answers. In a single forward pass it returns a probability for every candidate. Use it to rank candidate actions, route tool calls and check task state.

This release follows the route used by community GGUF builds of Laya:

1. The mmBERT/ModernBERT encoder is a standard modern-bert GGUF, converted with the unmodified llama.cpp tools.
2. The decision head ships alongside it as MacJev-322M-4K-Laya-head.safetensors, in FP32.
3. macjev_gguf.py sends token ids to llama-server, receives one hidden state per token, and runs the head in NumPy. It needs only numpy, tokenizers and safetensors.

Highlights

Compared with the Laya multilingual checkpoint it was trained from, with the same 4096-token input budget, on held-out test sets:

• Reads long observations. Yes/no task-state checks on 2K–4K-token inputs rise from 31% to 89% accuracy.
• Probabilities you can threshold. Calibration error on long inputs is about 8 times lower, so stated confidence closely tracks actual accuracy.
• Better on public benchmarks too. Across 11,600 public decisions from typed decisions, Emotion and AG News, accuracy rises by 1.5 points, with a 95% interval of 1.1 to 1.8.
• Reproducible. The long-input and public-benchmark gains appear in all three independently trained seeds. This release is the seed chosen in advance on development data.

Quickstart

You need llama.cpp's llama-server and Python 3.10 or newer. On a Mac, brew install llama.cpp provides it. On other systems, use a llama.cpp release.

hf download chaoliangUNSW/MacJev-322M-4K-Laya-GGUF --local-dir MacJev-GGUF
cd MacJev-GGUF
python -m pip install -r requirements.txt

Start the encoder server. The encoder reads the whole input at once, so keep these flags:

llama-server -m MacJev-322M-4K-Laya-F16.gguf --host 127.0.0.1 --port 8080 \
  --embeddings --pooling none -c 4096 -b 4096 -ub 4096 -np 1 -fa on

Then, from another terminal in the same folder:

from macjev_gguf import MacJevGGUF

model = MacJevGGUF(".", server_url="http://127.0.0.1:8080")
result = model.decide(
    state={
        "request": "打开浏览器",
        "available_actions": ["open_browser", "copy_file", "ask_user"],
    },
    question={
        "t": "choice",
        "ins": "Which available action best matches the user's request?",
        "crit": {
            "open_browser": "Open the user's browser",
            "copy_file": "Copy a file inside the approved folder",
            "ask_user": "Ask for clarification",
        },
    },
)
print(result["answer"], result["probabilities"])  # A decision only. Nothing is executed.

Or let the script start and stop the server for you, with one JSON object per line:

echo '{"state": {"request": "open the browser"}, "question": {"t": "noul", "ins": "Does the user want a browser opened?"}}' \
  | python macjev_gguf.py --start-server

Other question shapes:

score = {"t": "score", "ins": "How complete is the observed task?",
         "crit": ["Not started", "Partly complete", "Complete and verified"]}
boolean = {"t": "noul", "ins": "Does the observation prove that the requested file exists?"}

The result contains answer, probabilities, top_probability, input_tokens, latency_ms, and actions_executed: False. Score answers are ordered string indices; Noul answers are "false" and "true".

Inputs can use up to 4096 tokens in total and 1024 for the question and options. Anything larger raises InputBudgetError instead of being truncated. The runtime tokenizes with the shipped Hugging Face tokenizer and sends token ids directly.

LM Studio users

LM Studio ships its own llama-server, so you may not need to install llama.cpp at all. On a Mac it is inside ~/.lmstudio/extensions/backends/llama.cpp-*/. The binary from LM Studio's llama.cpp backend 2.41.0 serves this file with the flags above and returns the same hidden states as the Homebrew build. Use it in place of llama-server in the command above.

The GGUF file holds the encoder, and the decision head lives in the separate safetensors file, so run MacJev through macjev_gguf.py. To let an LM Studio chat model use it, expose decide() as a tool or MCP server.

Validation

• Same decisions as the FP32 model where it counts. The GGUF picks the same top answer on every clear-cut decision: 4,964 of 4,964 validation decisions where the FP32 answer leads by at least 0.06.
• Same accuracy. Development accuracy is identical to the FP32 model.
• Exact head. Given the same encoder states, the NumPy head matches the PyTorch head to within 0.000005 in logits.

Full per-file measurements, conversion details and llama.cpp flags are in validation.json. Tested with llama.cpp build b10964 (Homebrew 0.4.1, Metal). On an M1 Max, a full request took about 0.11 s up to 1024 tokens, 0.64 s up to 2048 and 1.7 s up to 4096, including transferring the hidden states over HTTP.

Files

Other formats

• PyTorch FP32, with training details
• MLX for Apple silicon

License and attribution

Apache-2.0, inherited from Laya; see LICENSE and NOTICE. The mmBERT-base backbone is MIT-licensed; see MMBERT_LICENSE. The GGUF was produced with unmodified llama.cpp tools. MacJev is an independent project. It is not affiliated with or endorsed by JEV or the Laya authors, and it contains no JEV weights.

• Laya model and source
• mmBERT-base

Community

Interested in compact decision models, local agents, and practical macOS automation? Questions and ideas go to GitHub Discussions.

如果你也对轻量决策模型、本地 Agent 和 macOS 自动化感兴趣，欢迎到 GitHub Discussions 交流。