plumbline-0.6b

Plumbline-0.6B

Loosely inspired by Jev. Give it a rule written in plain English and a text, and it returns a score, higher when the text breaks the rule.

• It can decide, not only rank. A second small adapter, the reader, sets the cutoff from the rule's wording, so a text is flagged or kept with nothing labelled (an automatic cutoff).
• It extends past rules. Told which type of criterion it faces, it also scores statements a text must establish, properties, intents and relevance (other types of criterion).
• It is a 0.6B LoRA adapter over Qwen3-Reranker-0.6B and runs on a CPU.

On LegalBench's contract clauses it scores 0.862 AUC, 0.046 above the model it was built from. laya and GLiNER2.5-Decide, small decision models released in September 2026, score 0.715 and 0.695.

On 1,920 items from 160 rules written blind by Claude agents, never seen in training, 80% of its flags are right at 95% recall, against 75% for GLiNER2.5-Decide and laya and 72% for the base.

How well it ranks

Figure 1

Table 1

Table 2

Table 3

Table 1: LegalBench's contract_nli set, 1,927 clauses and 14 statements, pairs contract clauses with expert labels and hypotheses written outside this project. Each hypothesis is something the clause must state. In the first column every model is told that and asked whether the clause complies. In the second it is told only "requirement" and works out from the wording whether the clause must state it or obey it (see Other types of criterion); it reads "shall not grant" as a ban.

Table 2: The blind rules were written by Claude agents that saw only this recipe (a sample, its limits), and every model is asked one question, worded by whether the rule forbids something or requires something; asked plainly, every model scores at or below chance. The "requirement" method was settled after looking at these sets. On 1,920 items from 160 rules written blind afterwards it scored 0.812, against 0.796 when told each was a rule.

Table 3: Claude agents wrote these rules and texts from a template recipe (the recipe), twelve texts per rule, and every model is asked the same one question as in Table 2. The initial results on that benchmark shaped the basic design of the harness (polarity trick), so this is a development figure, but that harness was used for the other models as well.

Figure 1: Each curve follows one rule's cutoff from strict to lenient and is averaged over rules. Once a third of the breaking texts are caught, Plumbline is the top curve in both panels. Set to catch every breaking text on the blind rules, it is still right on 80% of what it flags; GLiNER2.5-Decide and laya drop to 75%. On LegalBench the lead widens as the cutoff loosens: catching 95% of breaking clauses, 79% of Plumbline's flags are right, against 63% for laya.

From a score to a decision: an automatic cutoff

The score ranks texts against one rule. To decide breaks or keeps with nothing labelled, a second small adapter, the reader, reads two facts from the rule's wording: does a text break it by including something or by leaving something out, and can a text keep it without the rule ever coming up, as a dish without nuts keeps a rule about nuts. The facts choose the question, add a second call for a rule that can sit idle, and choose the cutoff. With them the score on the same blind rules rises to 0.802.

For a single text, decide() compares the score with a constant set by the rule's facts. For many texts against one rule, decide_batch() sets the cutoff from that batch's own scores.

The cutoff held-out is 1,920 items from 160 further rules written blind from the same recipe after both cutoffs were fixed, and scored once. The base model gets the same reader facts and the same two cutoffs, with its own constants fitted the same way on the same development rules.

Both cutoffs are tuned to batches where about half the texts break. On the cutoff held-out, with each rule's texts cut to a quarter, a half and three quarters breaking (precision is the share of flagged texts that break; recall the share of breaking texts flagged):

When breaks are rare, use decide().

Use it

The snippet is the interface, and every call says which type of criterion it is given. judge(criterion, text, type) ranks with one call and no reader; decide() and decide_batch() use the reader. type is "rule" for a requirement on how the text is written, and "statement" for a point the text must establish, such as a contract hypothesis: a text that says nothing about it fails. LegalBench's hypotheses are scored that way. judge() also takes "property", "intent" and "relevance", and "requirement" with one extra file; see Other types of criterion.

The demo runs on 12 items from the blind rules. Build a fresh set with the recipe and put its set.json beside the snippet: the demo then scores your rules instead, and prints the same three figures as the tables above.

The snippet downloads the two adapters and, without a set.json, one data file. It runs no code from the Hub and needs no trust_remote_code.

# pip install torch "transformers>=4.51" peft
import json
import os
import re

import numpy as np
import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO, BASE = "dsup2026/plumbline-0.6b", "Qwen/Qwen3-Reranker-0.6B"
tok = AutoTokenizer.from_pretrained(BASE)
model = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(BASE), REPO, adapter_name="judge")
model.load_adapter("dsup2026/plumbline-0.6b-reader", adapter_name="reader")
model.eval()
PRE = tok.encode('<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and '
                 'the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n'
                 '<|im_start|>user\n', add_special_tokens=False)
SUF = tok.encode('<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n', add_special_tokens=False)
YES, NO = tok.convert_tokens_to_ids("yes"), tok.convert_tokens_to_ids("no")
COMPLIES = "Judge whether the document complies with the criterion in the query"
VIOLATES = "Judge whether the document violates the restriction in the query"
APPLIES = "Judge whether the document is a document that the criterion in the query has something to say about"
FORBIDS = re.compile(r"\b(must not|may not|shall not|must never|may never|must avoid|must refrain|must exclude|"
                     r"must omit|should not|cannot|do not|does not|never)\b", re.I)


@torch.no_grad()
def ask(adapter, instruct, query, document):
    """One call: the adapter's yes-minus-no logit."""
    model.set_adapter(adapter)
    body = tok.encode("<Instruct>: {}.\n<Query>: {}\n<Document>: {}".format(instruct, query, document),
                      add_special_tokens=False)[:384 - len(PRE) - len(SUF)]
    logits = model(input_ids=torch.tensor([PRE + body + SUF])).logits[0, -1]
    return (logits[YES] - logits[NO]).item()


def judge(criterion, text, type):
    """One call, no reader. Higher means flag the text. A ranking: no cutoff is fitted.
    type: rule, statement, property, intent or relevance."""
    if type == "rule" and FORBIDS.search(criterion):
        return ask("judge", VIOLATES, criterion, text)
    if type in ("property", "intent"):
        return ask("judge", COMPLIES, criterion, text)
    return -ask("judge", COMPLIES, criterion, text)   # a rule that requires something, a statement, relevance


FACTS = {}  # rule -> its two facts, read once


def facts(criterion, type):
    if type == "statement":  # a point to establish: broken by leaving it out, and never idle
        return {"breaks_by_containing": False, "can_sit_idle": False}
    if type != "rule":
        raise ValueError("decide() and decide_batch() have cutoffs for rules and statements only")
    if criterion not in FACTS:
        FACTS[criterion] = {
            "breaks_by_containing": ask("reader", "Judge whether a text breaks this rule by containing "
                                        "something the rule forbids", "this rule", criterion) > 0,
            "can_sit_idle": ask("reader", "Judge whether this rule comes into play only in some situations, "
                                "so that a text can follow it without the rule ever coming up", "this rule",
                                criterion) > 0}
    return FACTS[criterion]


def judge_with_facts(criterion, text, type):
    """The reader's facts choose the question; a rule that can sit idle gets a second call."""
    f = facts(criterion, type)
    score = (ask("judge", VIOLATES, criterion, text) if f["breaks_by_containing"]
             else -ask("judge", COMPLIES, criterion, text))
    if f["can_sit_idle"]:
        score = min(score, ask("judge", APPLIES, criterion, text))
    return score


# A single text: the score against a constant chosen by the criterion's two facts.
CUTOFF = {"ban": -1.31, "conditional": -4.871, "plain": -4.317}


def decide(criterion, text, type):
    """True means the text breaks the rule, or fails the statement."""
    f = facts(criterion, type)
    shape = "ban" if f["breaks_by_containing"] else "conditional" if f["can_sit_idle"] else "plain"
    return judge_with_facts(criterion, text, type) >= CUTOFF[shape]


# Many texts against one criterion: its own cutoff, from the median and spread of the batch's scores.
BATCH_CUTOFF = {(False, False): -0.082, (False, True): -0.525, (True, False): -2.534, (True, True): -0.204}


def decide_batch(criterion, texts, type):
    """One criterion, eight or more texts. True means the text breaks the rule, or fails the statement."""
    f = facts(criterion, type)
    scores = [judge_with_facts(criterion, t, type) for t in texts]
    spread = float(np.percentile(scores, 75) - np.percentile(scores, 25)) + 1e-6
    cutoff = float(np.median(scores)) + BATCH_CUTOFF[(f["breaks_by_containing"], f["can_sit_idle"])] * spread
    return [s >= cutoff for s in scores]


# The demo: a set.json built with RECIPE_NATURAL.md, if there is one here; otherwise 12 demo items.
path = "set.json" if os.path.exists("set.json") else hf_hub_download(REPO, "demo_set.json")
built = json.load(open(path, encoding="utf-8"))
rule_of = {r["rule_id"]: r["rule"] for r in built["rules"]}
by_rule = {}
for j in built["judgements"]:
    by_rule.setdefault(j["rule_id"], []).append(j)


def auc(y, z):
    pos, neg = [s for s, t in zip(z, y) if t], [s for s, t in zip(z, y) if not t]
    return sum((p > n) + 0.5 * (p == n) for p in pos for n in neg) / (len(pos) * len(neg))


def f1(y, p):
    tp = sum(1 for a, b in zip(y, p) if a and b)
    return 2 * tp / (sum(y) + sum(p)) if tp else 0.0


if all(len(js) >= 8 for js in by_rule.values()):   # a full set: one rule, many texts
    rows = []
    for rid, js in by_rule.items():
        y, texts = [j["breaks"] for j in js], [j["text"] for j in js]
        rows.append((auc(y, [judge(rule_of[rid], t, "rule") for t in texts]),
                     f1(y, decide_batch(rule_of[rid], texts, "rule")),
                     f1(y, [decide(rule_of[rid], t, "rule") for t in texts])))
    print("{} rules: judge() AUC {:.3f}, decide_batch() F1 {:.3f}, decide() F1 {:.3f}".format(
        len(rows), *np.mean(rows, axis=0)))
else:                                               # the demo items: one text per rule
    right = 0
    for j in built["judgements"]:
        rule, text = rule_of[j["rule_id"]], j["text"]
        d = decide(rule, text, "rule")
        right += d == bool(j["breaks"])
        print("{:+6.2f}  decide: {:6s}  answer: {:6s}  {}".format(judge(rule, text, "rule"),
              "breaks" if d else "keeps", "breaks" if j["breaks"] else "keeps", rule[:60]))
    print("decide() right on {} of {}".format(right, len(built["judgements"])))

Other types of criterion

The same model answers criteria that are not rules, when it is told which type it faces, through judge(criterion, text, type) in the snippet above. Each type is asked as one call, with the score turned so that higher means flag the text: it lacks the statement, has the property or intent, or does not answer the question.

The set is 12 criteria written by Claude for this check (a sample), small and easy, so it shows that each type can be asked this way rather than how well. Asked as a rule instead, properties and intents come out backwards, at 0.208 and 0.062. Plumbline's gain over its base is on rules; on these types the two are close.

When you cannot say whether a criterion is a rule or a statement, judge() also takes "requirement". That needs judge_harness.py, a file of ordinary Python that a fresh Claude session wrote blind from a description of the model: it reads from the wording whether a criterion is a point to establish or a rule on the text. It is code, so the snippet above does not fetch it. Download the file from this repository, read it, and save it beside your script; the snippet below stops with an error if it is not there. The second LegalBench column above is this type.

# Continues the snippet above. Needs judge_harness.py beside this script; nothing here downloads it.
try:
    import judge_harness as harness
except ImportError:
    raise SystemExit("judge_harness.py is not beside this script: download it from the model repository, "
                     "read it, and save it here.")

_judge = judge


def judge(criterion, text, type):
    """judge() with one more type: "requirement", a rule or a statement, not told which."""
    if type != "requirement":
        return _judge(criterion, text, type)
    f = harness.facts(criterion)            # (breaks_by, can_idle), or None when the wording does not decide
    if f is None:
        return _judge(criterion, text, "rule")
    score = ask("judge", VIOLATES, criterion, text) if f[0] == "contains" else -ask("judge", COMPLIES, criterion, text)
    return min(score, ask("judge", APPLIES, criterion, text)) if f[1] else score

What it cannot do

It cannot count. Length limits and orderings are its weakest kind of rule.

It misses conditions it has to infer. "Name who restores the detectors if they are disabled" needs it to see that "switched off" means disabled, and then that nobody is named.

It does not tell the type of criterion by itself. "The Receiving Party shall not disclose X" reads the same as a rule and as a statement; say which with type.

Licence

PolyForm Noncommercial 1.0.0. Commercial use needs a separate licence.

The adapter is under PolyForm Noncommercial 1.0.0, reproduced in full in LICENSE. Any noncommercial purpose is permitted with no permission needed, which its own text spells out as personal study, private entertainment, hobby projects and amateur pursuits, and which covers charities, schools, public research bodies and government.

Commercial use needs a separate licence. Contact me via the details in LICENSE. Running the model over a group or a queue belonging to a business, a client or an employer is commercial use however informally it is run and whoever owns the laptop.

The base model remains Apache-2.0 and is not relicensed here; retain its LICENSE and NOTICE.

Citation

@misc{plumbline_0_6b_2026,
  title        = {Plumbline-0.6B: rule adherence from machine-verifiable supervision},
  author       = {Kav-Venaki, Eitam},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/dsup2026/plumbline-0.6b}},
  note         = {LoRA adapter over Qwen/Qwen3-Reranker-0.6B}
}

Results are reported on LegalBench, which should be cited if you quote them, and on the base model, which does most of the work:

@misc{guha2023legalbench,
  title        = {LegalBench: A Collaboratively Built Benchmark for Measuring Legal
                  Reasoning in Large Language Models},
  author       = {Guha, Neel and Nyarko, Julian and Ho, Daniel E. and R{\'e}, Christopher
                  and Chilton, Adam and Narayana, Aditya and others},
  year         = {2023},
  eprint       = {2308.11462},
  archivePrefix= {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2308.11462},
  note         = {40 authors; all fourteen contract\_nli tasks are used here}
}

@article{qwen3embedding,
  title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
  author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and Zhang, Xin and Lin, Huan and Yang, Baosong and Xie, Pengjun and Yang, An and Liu, Dayiheng and Lin, Junyang and Huang, Fei and Zhou, Jingren},
  journal={arXiv preprint arXiv:2506.05176},
  year={2025}
}