prose-rewriter-4b-v1.2

prose-rewriter-4b-v1.2

A paragraph-level prose rewriter: it takes prose written by a large model and re-renders it to be more human, preserving the semantics it was given.

Qwen/Qwen3-4B-Base with a rank-32 LoRA merged in at strength 1.15.

This is the larger of two checkpoints trained on the same pool and pipeline. A smaller version here: 1.7B

Variants

VRAM

Measured on an RTX 3090 with llama-server -ngl 99 at stock settings (flash attention auto, f16 KV), read as the server process's resident VRAM after load and before the first request.

-c is the TOTAL context and llama.cpp divides it by --parallel. A -c 4096 --parallel 8 server gives each slot 512 tokens and silently truncates long paragraphs. Ask for slots x 1280: 512 source tokens is the documented input ceiling, the generation budget never exceeds 512, and the prompt's three blocks are a dozen more.

It is linear, so extrapolate freely: the weights are a fixed 4,444 MiB (Q8_0) or 2,742 MiB (Q4_K_M), and every 1280-token slot adds 180 MiB of KV cache on either quant. Only the total matters -- -c 2048 --parallel 1 and -c 2048 --parallel 2 allocate the same bytes.

Four slots is a sensible default. Eight reserves better than a gigabyte of KV before the first request arrives, which on a small card is the difference between fitting and not. The 1.7B runs the same eight slots in 3,202 MiB.

Prompt format

<|im_start|>source
{paragraph}<|im_end|>
<|im_start|>edit
match<|im_end|>
<|im_start|>rewrite

The chat template in this repo builds exactly that string, byte for byte, from two roles:

messages = [
    {"role": "source", "content": paragraph},
    {"role": "edit",   "content": "match"},
]
tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

It is not a chat model. The template rejects user / assistant / system, and an edit value outside the three modes, rather than quietly building a prompt the weights have never seen.

One paragraph per call. Generation stops on <|im_end|>, which is set as an eos_token_id in generation_config.json. Temperature is the only knob.

The edit block is mandatory

edit names which of three length transforms is being asked for. The values are the corruptor's, so they read backwards. They name what was done to build the input, not what the model should do to it:

match is the setting for "rewrite it, do not trim it". It's strongly recommended you use this mode.

Sending no block is the worst thing you can do to this checkpoint. It was trained with the block, so omitting it collapses the model onto its deletion-heaviest mode.

Serving recipe

Sampled at temperature=0.9, top_p=0.9.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "chartreuse-verte/prose-rewriter-4b-v1.2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="cuda").eval()

def rewrite(paragraph, mode="match"):
    text = tok.apply_chat_template(
        [{"role": "source", "content": paragraph},
         {"role": "edit",   "content": mode}],
        tokenize=False, add_generation_prompt=True,
    )
    ids = tok(text, return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=512, do_sample=True, temperature=0.9, top_p=0.9)
    return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()

Same thing under llama.cpp. The roles are source and edit, which no chat API models, so build the string yourself; <|im_end|> stops it:

llama-cli -m GGUF/prose-rewriter-4b-v1.2-Q8_0.gguf -st -n 512 --temp 0.9 --top-p 0.9 \
  -p '<|im_start|>source
{paragraph}<|im_end|>
<|im_start|>edit
match<|im_end|>
<|im_start|>rewrite
'

Input length

length ratio is median output words over input words. invention is the share of generations carrying a sentence the input does not entail (DeBERTa-MNLI, min entailment < 0.5). 3-gram self-repetition is the fraction of a generation's 3-grams that repeat inside it -- the degeneracy signal.

Nothing degenerates at any length. Self-repetition sits at noise across the whole range, and not one of these generations failed to stop or came back empty. What actually happens below ~15 words is that the model pads and invents: it stretches the line toward its learned length and roughly 70% of outputs add material the input never supported, against ~25% at paragraph length. Between 15 and 120 words nothing moves much.

So serve it on anything from a full sentence up. Below 80 bytes, pass the text through unchanged.

Evaluation

The 4B invents less and copies no more, at flat diversity, which is the direction scale was expected to move.

A known quirk is that human-written texts use less rare vocabulary than LLM texts. This is a trade-off that must be accepted.

Training

Corrupt forward, train backward. The human paragraph is the target; an on-policy LLM manufactures the input by slop-ifying it.

The target side is human prose: roughly 60/40 r/WritingPrompts (Mollymo/Human-to-AI-writing) and AO3 (midwestern-simulation-active/ao3_random_subset), with a sliver of fanfiction.net (atom-in-the-universe/fanfics-10k-10k).

The input side was generated by eight corruptor endpoints, weighted so no single model's tics dominate:

Pairs pass invariant gates before they reach the GPU: POV, tense, who is in the scene, grammatical correctness on the target side, and NLI entailment both ways. About 44k of 317k raw corruptions survive; the training pool is the seeded subset of those. This is the same pool the 1.7B was trained on, in the same order.

Loss on the target paragraph only. Everything before rewrite is masked.

The merge, and one thing not to do

Merged at strength 1.15. Rank 32 with alpha 32 is a LoRA scaling of 1.0, so the effective scaling is 1.15: W + (B @ A) * 1.15. Merged in float32, stored bfloat16.

lm_head is adapted, and Qwen3-4B-Base ties lm_head.weight to embed_tokens.weight. This checkpoint is untied: the merged output head is stored separately and the input embeddings are bit-identical to the base model's, which is what training assumed. config.json says tie_word_embeddings: false and it means it. Do not re-tie it, and if you convert to another format, check that the head survived.

Limitations

• Not an instruct model. It has one job and one prompt. There is nothing to ask it.
• Works on fictional prose only. May not work on technical documentation.
• One paragraph per call. Longer input degrades; split it.
• Will not pass AI detectors. Pangram and such will still know because this model preserves word choices and certain sentence structures.
• English only, narrative register (third and first person fiction, dialogue with quoted speech).
• Short input pads and invents. The floor is about 15 words, and below it the failure is fabrication rather than gibberish. See Input length.

License

The weights in this repository are released under the GNU Affero General Public License, version 3. The full text is in LICENSE.

This is a derivative of Qwen/Qwen3-4B-Base, which is licensed under Apache License 2.0. That license is preserved and its terms continue to apply to the base weights this model was built from; the AGPL covers the combined work as distributed here. Apache-2.0 is one-way compatible with AGPLv3, which is what makes this combination possible.

If you run a modified version of this model as a network service, AGPL section 13 requires you to offer the corresponding source of your modifications to its users.