vnpen-writer-2b-v0.1-preview

vnpen-writer-2b-v0.1-preview

English · 中文

⚠️ This is a preview checkpoint, not the v0.1 release. It exists to prove the training → merge → evaluation → publish pipeline on real hardware and to produce a comparable set of programmatic metrics. It is not representative of the quality the writer series is aiming for, and it has a known degeneration problem documented below.

⚠️ 这是 preview 权重，不是 v0.1 正式发布。 它的目的是把训练、合并、 评估、发布这条链路在真实硬件上跑通并给出一份可对照的程序指标， 不代表 writer 系列的目标质量，且存在下文记录的退化问题。

---

<a name="english"></a>

English

VNPen is MewBaka Studio's visual-novel model series. The writer edition is for script writing, de-AI rewriting, and generating example scenes from a mood brief. Output format is one script line per line: speaker：text, with narration written as 旁白：.

This model writes Simplified Chinese only. It was trained on a Chinese visual-novel corpus and its identity/replay data is Chinese-first; it is not an English writing model. A verbatim sample of its output is in the Chinese section below (一条真实样例) rather than here, because the sample is a Chinese script and nothing about it would read differently in translation.

How this preview differs from the planned v0.1

Text-only: the vision tower is not in these weights

The base Qwen/Qwen3.5-2B is multimodal. Its checkpoint carries 297 model.visual.* tensors (a depth-24 / hidden-1024 / patch-16 ViT) and 15 mtp.* tensors for multi-token prediction.

This project is text-only. In transformers, AutoModelForCausalLM on a qwen3_5 config builds Qwen3_5ForCausalLM over a Qwen3_5TextConfig — so the vision weights were never loaded at any point: not for training, not for merging, not for saving. Verified on the published weights:

• config.json: model_type: qwen3_5_text, architectures: ["Qwen3_5ForCausalLM"], no vision_config
• 320 tensors saved, 0 matching visual / vision / merger
• no preprocessor_config.json, no video_preprocessor_config.json

This is not the result of stripping a multimodal checkpoint after the fact; the vision half was simply never read.

Training data subset

Drawn from the full writer epoch-1 mix (314,342 rows) by per-task stratified sampling, seed 0. rewrite and identity are taken whole; the other three are sampled to a quota.

Rendered with the project's own training template (qwen35_train_nothink.jinja): token p50 = 1,174, p95 = 1,889. 12 rows exceeded seq_len 4096 and were discarded, not truncated.

The visual-novel corpus itself is not redistributed.

Training

On the target modules. Qwen3.5 is a hybrid stack: of its 24 layers only 6 are standard attention (q/k/v/o_proj); the other 18 are gated-delta-rule linear attention whose projections are named linear_attn.in_proj_* / out_proj. A target list of only q,k,v,o,gate,up,down — the conventional "all linear layers" list — therefore leaves the token mixing of 18 of 24 layers unadapted. The three linear-attention projections are included here. in_proj_a and in_proj_b are deliberately excluded: they are [16, 2048] per-head gates, so rank 64 is capped at rank 16 regardless, and they set the decay rate of the linear-attention state.

Training loss, one point per 10% of the run:

Final 1.7489, minimum 1.5461. Single-point loss swings with batch composition (a batch of continuation rows averages ~1,400 tokens, a batch of identity rows ~41), so the per-step figure is noisy by construction.

Programmatic evaluation

131 items from the project's v0.1 eval set, scored by metrics.py. The model judge was not run — these are the programmatic metrics only, and they measure format and surface statistics, not writing quality.

Both sides use the same template, the same decoding parameters (temperature=0.8, top_p=0.95, seed 0, empty thinking block) and the same stop token. The only difference is the weights.

Reading the numbers:

• Line format is essentially solved (91.7% → 100.0%) and line-count discipline improved a lot: the median line-count overshoot went from +61 to +13. The strict format_ok metric stays low because it demands an exact line count with zero stray lines.
• Less parroting of the input on rewrite (0.38 → 0.12) and lower adjective density (0.030 → 0.014), which is the direction the de-AI rewrite task asks for.
• Identity was learned: 0% → 60% of identity items contain both VNPen and MewBaka (per keyword: 80% / 60%).
• Two regressions. ≤6-char dialogue share overshoots the corpus baseline of 29.0% (37.8% → 52.0%), and the sentence-length distribution moved further from the corpus (JS 0.472 → 0.537). The model over-learned short lines.
• Proofread is 0% on both sides and that is expected: proofreading belongs to the realtime profile's mix, and the writer mix contains none of it. Neither model emits the required JSON-Lines answer format.

Known limitation: repetition / degeneration

On long generations both this preview and the base model collapse into repeating a short cycle of lines. Measured over the 57 script-writing items:

The preview has slightly fewer catastrophic cases but a markedly higher median repeated-line share. Worst observed case: a single line emitted 310 times. The base model does this too (155 times), so a large part of it is decoding defaults rather than fine-tuning — no repetition penalty was set on either side, because the evaluation protocol fixed the decoding parameters in advance and they were reported as specified rather than tuned for a better number.

If you use this model, set a repetition penalty. Something in the 1.05–1.15 range, and/or no_repeat_ngram_size, removes most of it.

Stop token

config.json inherits eos_token_id: 248044 (<|endoftext|>) from the base, but the chat template ends an assistant turn with <|im_end|> (248046). Left alone, generate() waits for a token that a chat prompt never produces and runs to max_new_tokens every time. generation_config.json in this repo is therefore set to eos_token_id: [248046, 248044]. If you build a prompt yourself, pass the stop token explicitly.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

m = "VNPen/vnpen-writer-2b-v0.1-preview"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16).cuda()

msgs = [
    {"role": "system", "content": "你是 VNPen，由 MewBaka 工作室发布的视觉小说专项模型，擅长视觉小说剧本的写作、改写与示例生成。输出剧本时使用「说话人：文本」的行格式，旁白写作「旁白：」，示例场景一般 30–50 行。使用简体中文。用户提出与写作无关的问题时正常回答。"},
    {"role": "user", "content": "写一段黄昏天台、两个人都没把话说开的场景，30 行左右。"},
]
ids = tok.apply_chat_template(msgs, return_tensors="pt",
                              add_generation_prompt=True).cuda()
out = model.generate(
    ids, max_new_tokens=1200, do_sample=True,
    temperature=0.8, top_p=0.95,
    repetition_penalty=1.1,                      # see the limitation above
    eos_token_id=[248046, 248044],
    pad_token_id=tok.pad_token_id,
)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

The chat template is stored in tokenizer_config.json and reproduces the training-time rendering byte for byte — verified on 50 real training rows, 50/50 identical, generation prompt identical.

Repository contents

License

Apache License 2.0, the same as the base model.

Built from Qwen3.5 (Qwen/Qwen3.5-2B), Copyright the Qwen team,
Alibaba Group, licensed under the Apache License, Version 2.0.
Modified by supervised fine-tuning (LoRA, merged) and by keeping
only the text decoder: the vision tower was never loaded.

The upstream Qwen/Qwen3.5-2B repository ships no NOTICE file (Apache-2.0 §4(d) only requires propagating one when the original has one), so the NOTICE here is written by this project. Per §4(b): these weights are modified relative to the base — supervised fine-tuning, and only the text decoder is kept.

The replay portion of the training data comes from public instruction sets: shibing624/alpaca-zh (CC BY 4.0), llm-jp/oasst1-21k-ja (Apache-2.0), yahma/alpaca-cleaned (CC BY 4.0), OpenAssistant/oasst1 (Apache-2.0). The visual-novel corpus is not redistributed.

Apache-2.0 grants no rights to the Qwen / Alibaba / Tongyi names or logos; they appear here only as descriptive attribution.

---

<a name="中文"></a>

中文

VNPen 是 MewBaka 工作室的视觉小说专项模型系列。writer 版本负责剧本写作、 去 AI 味改写、按氛围生成示例场景。输出格式为一行一句剧本行： 说话人：文本，旁白写作 旁白：。

与计划中的 v0.1 的差别

纯文本：视觉编码器不在这份权重里

基座 Qwen/Qwen3.5-2B 是多模态的，checkpoint 里带 297 个 model.visual.* 张量（一个 depth 24 / hidden 1024 / patch 16 的 ViT）和 15 个 mtp.* 张量。

本项目只用文本。transformers 里 AutoModelForCausalLM 作用于 qwen3_5 配置时构建的是 Qwen3_5ForCausalLM，它建立在 Qwen3_5TextConfig 之上 —— 视觉权重在训练、合并、保存的任何一步都没有被加载过。在发布的权重上已核验：

• config.json：model_type: qwen3_5_text、 architectures: ["Qwen3_5ForCausalLM"]、不含 vision_config
• 保存 320 个张量，匹配 visual / vision / merger 的 0 个
• 不含 preprocessor_config.json 与 video_preprocessor_config.json

这不是"事后从多模态权重里剥离"的结果，而是那一半从来没被读进来过。

训练数据子集

从完整的 writer epoch-1 混合（314,342 行）按任务分层随机抽取，种子 0。 rewrite 与 identity 取全部，其余三项按配额抽样。

用项目自己的训练模板 qwen35_train_nothink.jinja 渲染：token p50 = 1,174、 p95 = 1,889。12 行超过 seq_len 4096，按规则丢弃而非截断。

视觉小说语料本身不随模型发布。

训练

关于目标层。 Qwen3.5 是混合架构：24 层里只有 6 层是标准注意力 （q/k/v/o_proj），另外 18 层是 gated-delta-rule 线性注意力，投影名为 linear_attn.in_proj_* / out_proj。只写 q,k,v,o,gate,up,down 这份常规的"全线性层"清单，会让 18/24 层的 token 混合完全没有 adapter。 本次补上了那三个线性注意力投影。in_proj_a 与 in_proj_b 刻意排除： 它们是 [16, 2048] 的每头门控，r=64 的有效秩被 16 卡死， 且它们决定线性注意力状态的衰减率。

训练 loss，每 10% 一个点：

最终 1.7489，最低 1.5461。单点 loss 随批次组成波动很大 （一批 continuation 平均约 1,400 token，一批 identity 约 41）， 所以逐步数值天然有噪声。

程序指标评估

项目 v0.1 评估集的 131 道题，用 metrics.py 打分。没有跑模型 judge —— 以下只是程序指标，衡量的是格式与表层统计，不衡量文笔。

两侧使用同一模板、同一组解码参数（temperature=0.8、top_p=0.95、 种子 0、空思考块）和同一个停止符，唯一差别是权重。

怎么读这些数字：

• 行格式基本解决（91.7% → 100.0%），行数纪律大幅改善：行数超出的中位数 从 +61 降到 +13。严格的 format_ok 仍然很低，是因为它要求行数精确相等 且一个杂行都没有。
• 改写时更少照抄输入（0.38 → 0.12），形容词密度减半（0.030 → 0.014）， 正是去 AI 味改写想要的方向。
• 身份学会了：含 VNPen 与 MewBaka 两个词的身份题从 0% 升到 60% （分词命中 80% / 60%）。
• 两项退步。 ≤6 字对白占比 冲过了语料基线 29.0%（37.8% → 52.0%）， 句长分布反而离语料更远（JS 0.472 → 0.537）。模型把短句学过头了。
• 校对两侧都是 0%，这是预期内的：校对属于 realtime profile 的混合， writer 混合里一条都没有，两个模型都不会输出要求的 JSON Lines 格式。

一条真实样例

评估集第 brief_01 题，原样引用 —— 未编辑、未重抽，用上文所述参数生成。请求是：

深夜便利店那种不能说话的沉闷气氛，然后有人偷偷塞个暖手宝过来，这种画面能写一段吗 请给出正好30行剧本作为答案。 登场角色：明日海（自称“我”，敬语/幼态，口癖“那，那个”）、伊吕波（自称“我”，书面/古语，口癖“羽川原伊吕波”）、隆之介（自称“老爸”，平语，口癖“阿莱斯特1号”）

回答的前十行：

旁白：因为之前，我好像说过“想和伊吕波酱一起住”之类的话。
旁白：但是，这也没办法啊。
旁白：但是，我明明想一直在一起，却只能放弃。
旁白：因为，我，是笨蛋吗……
旁白：因为，伊吕波酱，不在了吗……
旁白：所以……
旁白：（这么说着，我摸了摸自己的胸口）
旁白：（伊吕波酱的心脏，还跳得很快吗）
明日海：……那个，伊吕波酱……
旁白：（虽然我想说话，但伊吕波酱却沉默不语）

以及同一条回答的最后三行：

伊吕波：（为什么，我……）
伊吕波：（明明，一直都在我的身边）
旁白：（我，想一直，和伊吕波酱在一起……）

这条输出共 38 行，其中有一行出现了 3 次。它是八条盲读样本里退化 最轻的一条；这里连尾巴一起贴出而不是截到第十行为止，因为截断会让模型显得 比实际更好。参见下一节。

已知缺陷：重复 / 退化

长生成时，本 preview 与底模都会塌缩成几行一组的循环。 在 57 道剧本写作题上测得：

preview 的灾难性个案略少，但中位重复率明显更高。观测到的最坏情况是 同一行出现 310 次。底模同样如此（155 次），说明其中很大一部分来自解码默认值 而非微调 —— 两侧都没有设 repetition penalty，因为评估协议事先固定了 解码参数，这里如实按协议报告，而不是调参调到好看为止。

如果你要用这个模型，请设 repetition penalty。 1.05–1.15 区间， 或配合 no_repeat_ngram_size，可以消除其中大部分。

停止符

config.json 从基座继承了 eos_token_id: 248044（<|endoftext|>）， 但聊天模板结束 assistant 轮用的是 <|im_end|>（248046）。不管的话， generate() 会一直等一个对话格式下永远不会出现的 token，每次都顶到 max_new_tokens。因此本仓库的 generation_config.json 已设为 eos_token_id: [248046, 248044]。若你自己拼 prompt，请显式传停止符。

用法

见上方 English 小节的代码示例（注意其中的 repetition_penalty=1.1）。

chat template 存放在 tokenizer_config.json 中，与训练时的渲染 逐字一致 —— 用 50 条真实训练样本双模板对照，50/50 完全相同， 生成提示词也相同。

仓库内容

许可证

Apache License 2.0，与基座一致。归属声明见上方 English 小节。 上游 Qwen/Qwen3.5-2B 仓库未附带 NOTICE 文件 （Apache-2.0 §4(d) 只要求在原作有 NOTICE 时传递），本仓库的 NOTICE 由本项目撰写。按 §4(b) 声明：本权重相对基座已被修改 —— 监督微调，且只保留文本解码器。

训练数据中的 replay 部分来自公开指令数据集：shibing624/alpaca-zh （CC BY 4.0）、llm-jp/oasst1-21k-ja（Apache-2.0）、yahma/alpaca-cleaned （CC BY 4.0）、OpenAssistant/oasst1（Apache-2.0）。视觉小说语料不随模型发布。

Apache-2.0 不授予 Qwen / Alibaba / Tongyi 的名称与标识权利， 此处仅作描述性归属使用。