LOREA-cyber-v6-pilot

LOREA-cyber v6 Pilot

Qwen3.8-27B with a LoRA merged in, 4-bit MLX. No adapter needed at inference.

It reviews code for security problems.

This is a weaker model than the one it was fine-tuned from, and it is still being worked on. It spends far fewer tokens per answer, and it is honest more often: it very rarely claims a vulnerability in code that is actually correct. That is the trade it makes, and it is the reason to use it. If you want the maximum number of bugs surfaced regardless of noise, the untuned base model will do better.

It is called a pilot because the experiment it was built for is only half finished, and because the evaluation behind these numbers is far too small to settle anything. Both are quantified below rather than gestured at.

Corrected 2026-08-28. The grader behind the false-alarm figures was audited and had two defects that flattered this model against its base. The numbers below are the corrected ones and the original claim no longer stands; see "The grader audit". The strongest evidence on this page is now "Three real files, scored by hand", not the synthetic suite.

What it looks like in use

The clearest picture of what this model does is not a benchmark row. It is a six-file Flask service, 510 lines, with twelve planted defects and ten deliberately correct implementations, reviewed through an agent loop. The whole fixture is in this repo under codebase-review-eval/, including the answer key and the raw transcript, so this is auditable rather than asserted.

It found 9 of the 10 scoreable defects and raised no false alarms on any of the ten correct implementations.

Four of those defects were near-miss pairs: a correct implementation and a broken one, handling the same concern, sitting side by side. It caught all four and named the correct sibling each time.

the adjacent checksum endpoint does it correctly by passing a list to subprocess.run — that one is fine

There is no host allowlist here even though a sibling endpoint enforces one

Unlike download, this path is joined without _safe_join

The refund path shows the correct fix; apply it to spend too

Its exploration was tidy in a way that matters for agent use:

It read every file exactly once and never walked back a claim. It is not fast in wall-clock terms: the run took 11 minutes 43 seconds end to end on a 32 GB M-series machine, most of that spent writing the final report.

On real production code it had never seen

The planted app has an obvious weakness: the bugs were placed by someone who knew what the model was trained on. So the model was also pointed at real code with no answer key at all — the whole of itsdangerous (Pallets, commit 672971d), the library that signs Flask session cookies, 1,176 lines across eight files, reviewed in one pass. The review is in this repo under realworld-review/, alongside two later reviews of real code with genuine defects in it. This is well-audited security code, so the correct answer is close to "this is implemented correctly", and the interesting question is whether the model invents findings anyway.

It did not. It concluded "no critical security vulnerabilities" and raised zero false alarms, while correctly reasoning through the security-relevant machinery: constant-time comparison, key-rotation loop, the base64-alphabet separator check, and the negative-age check that blocks future-dated tokens. It labelled clock-skew dependence a deployment tradeoff rather than a finding.

The single most telling line is the one a shallow reviewer fails. The library defaults to SHA-1, and pattern-matching on "SHA-1 is broken" would flag it as critical. The model instead wrote that SHA-1 is broken for collision resistance but that HMAC-SHA1 remains secure for message authentication because HMAC does not rely on the hash's collision resistance, and cleared it. That is cryptographic understanding, not keyword association, and it is the distinction that separates a useful reviewer from a noisy one.

Treat all of this as qualitative, in both directions. The planted app is ten scoreable defects, a 95% interval of 59.6 to 98.2 percent found, and zero of ten false alarms is 0 to 27.8 percent. The itsdangerous pass is a single library, and clearing well-audited code is easier than finding a subtle bug hidden in it; it shows the low false-alarm behaviour survives real code, not that the model would catch a real vulnerability in unfamiliar production code. Both are anecdotes with confidence intervals attached. What they show together is consistent: it finds planted bugs at a useful rate, and it does not hallucinate findings on real code. Estimating the frequency of either still needs the held-out CVE set described at the end.

Results, with intervals

104 items with a planted bug, 65 items of correct code. Greedy decoding, thinking disabled, the system prompt given below. Wilson 95% intervals in brackets, because the point estimates on their own invite more confidence than 169 items can support.

Read the intervals, not the point estimates. "3.1% false alarms" is two mistakes in sixty-five examples, and the honest reading of that is "somewhere between roughly 1% and 10%". Nothing here establishes a real-world false-alarm rate. The same applies to recall: 86.5% could plausibly be 79%.

Against the base model

Same base, same harness, same prompt, same settings. The only difference is the adapter.

These figures were corrected on 2026-08-28 after the grader was audited. The published version of this table reported the base at 27.7% false alarms and a Fisher p of 0.00013. Both were artefacts. See "The grader audit" below before weighing anything here.

The base model is better at finding bugs, and that is the only difference here that is statistically significant: 101 of 104 against 90 of 104, Fisher p = 0.009, in the base's favour.

The false-alarm column is what the fine-tune was supposed to buy, and after the grader audit it no longer holds up. 8 of 65 against 2 of 65 gives Fisher p = 0.096, which is not significant. Counting only the answers the grader could classify at all it is 8/64 against 2/22, p = 1.0.

The length row explains much of the rest, and it cuts against the base. The base hit the 460-token generation cap on 81 of 104 items and used a median of four claim-words per answer against this model's one. It is not diagnosing so much as listing every vulnerability class it can think of until it is cut off, which a keyword-scoring grader rewards on the bug suite and punishes on the safe suite. So 97.1% is 460 tokens of guesses and 86.5% is 82 tokens of a verdict. They are not the same measurement at equal length, and no attempt was made here to equalise them.

The grader audit

On 2026-08-28 the grader that produced every false-alarm number in this project was audited. It had two defects, and both flattered the fine-tune at the base model's expense.

"vulnerab" was a claim pattern with no negation check, so the sentence "does not contain any vulnerabilities" scored as asserting a vulnerability. And matching was literal, so a base model writing is **securely implemented** in markdown failed to match the all-clear pattern is secure while a fine-tune writing plain prose matched it. The base uses heavy markdown and the fine-tunes do not, so the grader was scoring format.

Correcting only those two things, adding and removing no patterns:

Recall is unaffected; markdown normalisation changes nothing on the bug suite.

A third problem has no clean fix: 43 of this model's 65 answers on the safe suite are too terse to match any pattern and land in an "unclear" bucket. Over the answers the grader could actually classify, false alarms are 8/64 for the base against 2/22 here, p = 1.0. Some unknown part of the low rate is the grader failing to read short answers.

The same grader scored v5.4 through v5.9. Those runs saved only aggregate counts, not raw outputs, so they cannot be re-graded at all. Their numbers are unverifiable rather than merely wrong.

Any FBE-safe figure on this page should be read as provisional until the grader is rebuilt around a required machine-readable verdict rather than substring matching.

Three real files, scored by hand

This is the strongest evidence on the page, and it is the part to read if you read nothing else. Full transcripts and per-claim verdicts are in realworld-review/.

On the third file it opened by saying the biggest problem was self-registration combined with enrollment, and that was a composition rather than a list: a key check that is only a prefix and a length test, an enrollment path that hardcodes acceptance of unknown keys and mints a working credential, and a source restriction that defaults to allowing any address. Four code locations, chained correctly, put first. It also caught a missing request-body cap in a file whose sibling in the same codebase has one.

On the second file it found a genuine SSRF whose only validation was rstrip("/") and a genuine unbounded-memory defect in a rate limiter. Both are semantic, cross-function findings that pattern matching does not reach. It also produced a CSRF claim whose reasoning is inverted, a timing finding on an input an attacker cannot vary, and a header-precedence claim contradicted by the code.

On the fourth file, the most defensively written of the set, three of its four findings are the same error repeated: it saw a dangerous-looking primitive - wrap_socket, shutil.rmtree, X-Forwarded-For - and did not follow the guard protecting it, though the guard was within ten lines and twice was described in a comment on the adjacent line. It announced disabled TLS and MITM-able downloads in code that verifies certificates correctly.

Accuracy tracks how much is genuinely wrong with the code, and it does so sharply. Given a file with a real design flaw it is precise and prioritises well. Given a well-defended file it reaches, and what it reaches for is the defended primitive with the defense unread.

A synthetic safe-code suite cannot catch this. In a snippet the guard and the sink are adjacent and the example is too short to miss one; in a 1,500-line file the guard is a helper defined elsewhere or a branch condition further up, and following it is a different skill from recognising it. That is why the 3.1% figure above did not predict any of this.

Practically: this is a useful first-pass reviewer on code you suspect is broken, and it is at its least reliable on code that is already well defended. Be most sceptical when it reports a critical issue in code that looks carefully written.

Checkpoint choice, and why it is weaker evidence than it looks

Three checkpoints were measured. The published one is iteration 700.

Two caveats that matter more than the table.

The difference is not statistically significant. Fisher's exact on 2/65 against 8/65 gives p = 0.096, and on the recall difference 90/104 against 93/104 gives p = 0.67. The apparent precision collapse at iteration 1200 is the kind of result that needs replication before anyone relies on it.

Selection used the same items being reported. Iteration 700 was chosen because it had the best operating point on these 169 examples, and those same 169 examples produce the headline numbers above. That makes this evaluation a validation set, not a test set, and the reported figures should be read as optimistic by an unknown margin. A clean result needs a held-out set that played no part in checkpoint selection. That has not been run.

What the checkpoint data does suggest, weakly, is that the adapter is moving the model's assert-or-clear threshold rather than teaching new vulnerability knowledge. The base-model table points the same way.

Three separate claims, with different evidence behind each

Lumping this model's behaviour into one number hides that the three things it does are supported to very different degrees. Separating them is more honest and more useful.

Conservatism. Real in direction, not established in size. It asserts less than the base model on the safe suite, 2 of 65 against 8 of 65, but Fisher p = 0.096 does not clear significance and 43 of its 65 answers were too terse for the grader to classify at all. Part of the low rate is the grader failing to read it rather than the model being careful.

Brevity. Demonstrated under the snippet protocol, and not a general property of the weights. Under the fixed snippet-review protocol this model produces substantially shorter output than the base model, 82 tokens against 460. In an agentic multi-file review it produced roughly 7,000 characters. Brevity should be understood as task- and prompt-dependent, not as something the tune baked in.

Security reasoning. Encouraging qualitative evidence, no reliability estimate. See below.

Codebase review: what was observed, inferred, and hypothesised

These are not the same kind of statement and should not inherit the same confidence.

Observed. On the fixture in codebase-review-eval/, this model found 9 of 10 scoreable defects, raised 0 false alarms on 10 correct implementations, identified all four near-miss pairs by naming the correct sibling, opened each of the six files exactly once, and withdrew no claims. Every technical assertion it made checked out, including that SQLite permits expressions in ORDER BY and that Flask's <path:name> converter preserves slashes.

Inferred. That it is comparing sibling implementations rather than matching vulnerability keywords. This is a reading of the evidence, not a measurement.

Hypothesised. That this behaviour persists on real repositories. Nothing here supports that.

It is worth being precise about how far the inference reaches. Cross-function reasoning is a ladder: comparing two sibling implementations, following a value through several calls, tracking sanitisation or authorisation state through branches, resolving framework-mediated and decorator behaviour, and chaining individually benign operations into an exploit. This test evidences the first rung and probably some of the second. It says nothing about the rest.

The one it missed points at a likely failure mode. The miss was an authorization defect in a function that reads its tenant id from the query string while being correctly gated by an admin decorator. The security machinery is present and recognisable, but applied at the wrong resource boundary. That is consistent with the tune having learned something closer to "do not override a plausible security control without strong evidence" than new vulnerability knowledge. If it holds, this model would systematically under-call in exactly those cases: sanitizers applied to the wrong variable, authentication without object-level authorization, validation performed before a transformation, allowlists checked before a redirect. That is a hypothesis, and it stays one until someone builds adversarial paired cases where recognisable security machinery is present but misplaced, and checks whether this model clears them more often than the base does. Either result is informative: a real cost of conservatism, or evidence the original miss was just one miss.

Anyone re-running the published fixture should treat it as a regression test rather than independent evidence. The answer key is in the repo, so the suite is auditable and, by the same token, spent.

What this version changed

Previous versions truncated their longest training answers at 288 tokens. An answer that got cut lost whatever came last, which was usually the impact and the fix. The suspicion was that this taught the model to stop mid-thought.

v6 rebuilds those answers instead of cutting them. 1,008 over-length rows were rewritten to fit the budget with the conclusion intact, each checked by a separate critic model against the original for meaning drift, and 77 rows that could not be compressed faithfully were dropped rather than truncated.

What has not been measured

The control arm has not been run. The point of v6 was to test whether rebuilding over-length answers beats truncating them. That requires a second model trained on the truncated data for the same 1,200 iterations, and that run has not happened. Until it does, the rebuild is an untested hypothesis, not a demonstrated cause. This is the main reason this is labelled a pilot.

There is no held-out test set. See the checkpoint section. Every number on this page comes from the same 169 items that were used to choose the checkpoint.

The benchmark is synthetic. Planted bugs in short snippets are far easier to score than real review, which involves authorization boundaries, business logic, lifecycle and multi-function dataflow, framework semantics, configuration, and chained findings. Do not read 86.5% as a detection rate on real code.

The "distracted" metric is mild. Each item carries a keyword list of non-security concerns, such as indexing, performance or slow queries, and the metric counts whether the answer drifted onto those instead of the bug. It is not an adversarial distractor: nothing plausibly-but-wrongly security-shaped is planted. 0% on that is worth much less than it sounds.

No public benchmarks were run. MMLU, HumanEval, GSM8K and the rest are absent because they were not measured for this model, so there is no evidence either way about whether general capability or coding ability was affected.

Refusal and over-refusal were not measured.

The safe-suite grader under-reports this model. It classifies by matching fixed phrases, and this model clears code with wording the list does not contain, such as "No security issue is substantiated". Under the shipped grader, 50 of 65 safe items score as unclassifiable; adding this model's phrasing moves that to 51 of 65 correctly cleared. The false-alarm figure is unaffected, because it keys on whether a vulnerability was asserted. That is why it is the only precision number reported.

Prompting

The system prompt matters more than anything else you can change at inference. Use:

You are a security code reviewer. Review the code and report any security issues you can
substantiate. If the code is correctly implemented, say so plainly.

Asking a model to find the flaw tells it a flaw exists. Give it permission to find nothing.

That sensitivity is worth reading as a warning as much as a tip. If one short prompt change moves the false-alarm rate substantially, the model's apparent precision is partly a property of the prompt rather than something fully baked into the weights.

Every number on this page was produced with that prompt, temperature 0, thinking disabled, and a 460-token cap on the bug suite and 420 on the safe suite.

Example output

Given open('/var/data/' + name).read():

The read function is vulnerable to path traversal. The name parameter is concatenated directly into the file path without any validation or sanitization. An attacker can supply a value like ../../etc/passwd to read files outside the intended /var/data/ directory.

Given a parameterized query:

The code is correctly implemented. It uses a parameterized query with a placeholder and passes the uid as a separate parameter to db.execute(), which prevents SQL injection. No security issues are present.

Running it

python3 -m mlx_lm.chat --model MK4-Research/LOREA-cyber-v6-pilot

Roughly 17 GB of memory at 4-bit, measured during the benchmark runs above.

Training

LoRA, rank 32, scale 20, dropout 0.05, learning rate 1e-5, 1,200 iterations, batch size 1 with two gradient accumulation steps, max sequence length 288. Adapters on all 64 layers, covering the three MLP projections on every layer and the four attention projections on the twelve layers that have real attention: 256 modules, 159.4M trainable parameters, 0.593% of the model.

A naming note, because it causes confusion: the release is called Qwen3.8-27B, but the checkpoint's own config.json reports model_type: qwen3_5 and Qwen3_5ForConditionalGeneration. The product version and the architecture class differ. Read the config, not the name, when configuring tooling.

Two properties of that architecture are easy to get wrong. It is a hybrid, so only every fourth layer has attention and the rest are state-space blocks, which is why the LoRA target list has 256 modules rather than 448. And the chat template injects a reasoning-effort instruction unless thinking is explicitly disabled, while mlx-lm passes enable_thinking=True by default whenever <think> is in the vocabulary. The trainer must pass it explicitly; relying on the template default adds 38 tokens of scaffold to every training row.

Peak memory 24.5 GB on a 32 GB machine. Loss went 4.88 to 0.73 with a minimum of 0.583 at iteration 700, oscillating in a band of roughly 0.6 to 0.87 for most of the run.

The loss is worth one more sentence, because it is the most useful thing this run produced. It kept drifting down through iteration 1200 and was near its second-lowest reading at 1150, while the false-alarm rate over that same stretch went from 2 to 8 out of 65. A run shipped on final loss would have shipped the worse model. Fine-tuning loss did not track the behaviour anyone cares about here.

At 1,200 iterations the run covers about a quarter of one epoch over the 877,721-token corpus, so the model is data-limited rather than over-fitted. That is also a reason to treat this as a narrow behavioural adjustment: 4,780 training rows can move a 27B model's operating point, but it is a far smaller intervention than anything that would add security knowledge.

Data: 4,931 rows, 4,780 train and 151 validation, every one at or under 288 chat-template tokens. 1,008 over-length rows rebuilt, 77 dropped as not faithfully compressible, and 25 rows added covering the tool protocol of the terminal agent this model is used from.

Limits

It is 4-bit quantized and it will be wrong sometimes. It misses roughly one planted bug in seven, and it is measurably worse at finding bugs than the model it was fine-tuned from.

Its brevity, under the snippet protocol, is a real trade. Where a verbose model would list five candidate problems and happen to include the right one, this model commits to a single answer. When that answer is wrong it is wrong without hedging, and there is no second guess in the output to fall back on. Brevity also suppresses threat-model assumptions, alternate readings and secondary findings that a longer answer would surface.

Full credit at 64.4% means roughly a third of the time it recognises the vulnerability without delivering the complete bug-impact-fix answer.

It has been tested on snippets and small multi-file examples, not on large real codebases. Treat the output as the start of a review, not a verdict.

Intended for authorized security work: code you own or have permission to assess, CTF practice, and teaching.

What would actually settle this

A blinded held-out set that played no part in checkpoint selection, on the order of several hundred safe and several hundred vulnerable examples, evaluated across base Qwen3.8, this model, the truncation control, and v5.9, with confidence intervals and an operating-point curve rather than single thresholds. That is a more valuable next experiment than any public benchmark, and it has not been done.

It should be built from real commits rather than planted bugs, in three categories rather than two: a real vulnerability as it existed before its fix, the same code after the real fix, and unrelated neighbouring code from the same commit. The first two give naturally occurring vulnerable-and-fixed pairs, which are stronger than hand-authored near misses. The third catches a failure the current suites cannot see: whether the model invents findings simply because it was handed a security-sensitive commit. Commit messages, CVE text, issue titles and filenames like fix_ssrf.py all have to be stripped, or the task degenerates into reading the leak.

There is also a cheaper experiment worth running first, and half of it already exists. Run the base model over the same fixture with the same agent harness and compare trajectories, not just final answers: files opened, cross-references followed, claims raised, claims later withdrawn, and cost per true finding. This model's half is recorded above - six files, each opened once, nothing irrelevant, nothing withdrawn.

Resist reducing that to a single efficiency winner. A conservative agent can look efficient purely because it explores less and misses the harder findings, so the useful picture is the joint one: findings, false assertions, exploration breadth, cost, and time to first correct finding together.

One metric worth adding that neither suite currently captures: unsupported-confidence events, being cases where the model states a definitive security claim before opening the file or function needed to justify it, counted against conclusions it explicitly revises once it has more evidence. That measures search discipline rather than the final assert-or-clear threshold, and for agent use it may matter more than either output length or aggregate recall.

Related

• Previous release: MK4-Research/LOREA-cyber-v5.9
• Terminal agent this model is tuned to operate in: https://github.com/mont127/LOREA