Downloads · 30 days
198
17% of all-time downloads
VNPen/vnpen-writer-2b-v0.1-preview-GGUF
vnpen-writer-2b-v0.1-preview-GGUF is a text generation model from VNPen. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
GGUF / Ollama builds of VNPen/vnpen-writer-2b-v0.1-preview, for on-device deployment.
Downloads · 30 days
198
17% of all-time downloads
All-time downloads
1.2K
Public
Repo size
11.2 GB
Likes
0
Public
Click a slice to open those files.
.gguf11.2 GB · 100%
From the Hugging Face model README
GGUF / Ollama builds of
VNPen/vnpen-writer-2b-v0.1-preview,
for on-device deployment.
⚠️ These are quantisations of a preview checkpoint, not of a v0.1 release. Everything the source card says about quality, scope and known problems applies here unchanged. Read it first.
⚠️ 这些是 preview 权重的量化版本,不是 v0.1 正式发布。 源模型卡中关于质量、适用范围与已知问题的说明在此完全适用,请先阅读。
<a name="english"></a>
Perplexity is measured on a held-out slice of the project's own corpus that is disjoint from the importance-matrix calibration set — 300 samples / 336,872 tokens, rendered through the model's own chat template. The quant loss below is therefore measured, not memorised.
| File / 文件 | Size / 大小 | PPL | Δ vs BF16 | RAM (8k ctx) | Use / 用途 |
|---|---|---|---|---|---|
writer-2b-preview-BF16.gguf | 3.78 GB | 10.8465 | — | — | Conversion source; requantise from this. / 转换源,重新量化用。 |
writer-2b-preview-Q8_0.gguf | 2.01 GB | 10.8568 | +0.0103 (+0.09%) | ~2.4 GB | Reference quality, effectively lossless. / 参考质量,基本无损。 |
writer-2b-preview-Q6_K.gguf | 1.56 GB | 10.8992 | +0.0527 (+0.49%) | ~2.0 GB | High quality. / 高质量。 |
writer-2b-preview-Q5_K_M.gguf | 1.41 GB | 10.9559 | +0.1094 (+1.01%) | ~1.8 GB | Balanced. / 均衡。 |
writer-2b-preview-Q4_K_M.gguf | 1.27 GB | 11.0456 | +0.1991 (+1.84%) | ~1.7 GB | Recommended for edge. / 端侧推荐。 |
writer-2b-preview-IQ4_XS.gguf | 1.20 GB | 11.1793 | +0.3328 (+3.07%) | ~1.6 GB | Only if 1.27 GB will not fit. / 仅在放不下 1.27 GB 时用。 |
RAM figures are the model file plus an 8,192-token context and a little runtime overhead — roughly what you need free to run it.
Take Q4_K_M. It gives up 1.84% perplexity for a file 66% smaller than
BF16, and it is the only build here whose size/quality trade is not beaten by
a neighbour.
IQ4_XS is included for completeness but is usually the wrong pick: it saves
only 6% over Q4_K_M and pays 3.07% perplexity for it. The reason is that
this model ties its input and output embeddings, and token_embd.weight
(248,320 × 2,048 — 27% of all parameters) is held at Q6_K in every K-quant
below Q6_K. That one tensor dominates the file, so the low-bit formats have
little left to shrink.
Use the Modelfile. It is what carries the tuned defaults below, and this
model needs them:
hf download VNPen/vnpen-writer-2b-v0.1-preview-GGUF \
writer-2b-preview-Q4_K_M.gguf Modelfile --local-dir vnpen
cd vnpen && ollama create vnpen-writer-2b -f Modelfile
ollama run vnpen-writer-2b "写一段黄昏天台、两个人都没把话说开的场景,30 行左右。"
The one-liner works too:
ollama run hf.co/VNPen/vnpen-writer-2b-v0.1-preview-GGUF:Q4_K_M
but it does not pick up this repo's Modelfile — Ollama builds that model
from the GGUF alone, so it gets the embedded chat template and the stop token
but not the repetition settings. Set them in the session yourself, or the
degeneration described below will show up:
/set parameter repeat_penalty 1.1
/set parameter repeat_last_n 256
llama-cli -m writer-2b-preview-Q4_K_M.gguf --jinja \
-p "写一段黄昏天台、两个人都没把话说开的场景,30 行左右。" \
--temp 0.8 --top-p 0.95 --repeat-penalty 1.1 --repeat-last-n 256 -c 8192
The chat template is embedded in the GGUF, so --jinja is enough.
The shipped Modelfile deliberately overrides Ollama's defaults:
| Parameter | Value | Why |
|---|---|---|
repeat_penalty | 1.1 | The source card documents a degeneration problem — median repeated-line share 0.256 over the script items, worst observed case one line emitted 310 times — and asks for a 1.05–1.15 penalty. |
repeat_last_n | 256 | Ollama's default window is 64 tokens, shorter than one narration + dialogue pair, so it cannot see the cycle it is meant to break. |
stop | <|im_end|>, <|endoftext|> | config.json inherits eos 248044 (<|endoftext|>) from the base while the chat template ends an assistant turn with <|im_end|> (248046). Both are listed so a turn ends on whichever the model emits. |
temperature / top_p | 0.8 / 0.95 | The decoding parameters the source model was evaluated under. |
num_ctx | 8192 | A starting point, not a ceiling — see below. |
| template | forced empty <think> block | Every assistant turn in training opened with an empty <think> block. Dropping it moves the model off its training distribution. |
With these defaults a 31-line scene came back with 31/31 lines in the
speaker:text format, 31 unique lines, and done_reason: stop — the
repetition the source card warns about did not appear.
Qwen3.5 is a hybrid. Of the 24 layers only 6 run full attention (layers 3, 7, 11, 15, 19, 23); the other 18 are linear-attention (gated delta-net) layers whose recurrent state is a fixed size regardless of sequence length.
So the KV cache costs about 12 KB/token instead of the ~48 KB/token a dense 24-layer model of this shape would need:
| Context | KV cache | + fixed recurrent state |
|---|---|---|
| 8,192 | ~96 MB | ~18 MB |
| 32,768 | ~384 MB | ~18 MB |
| 131,072 | ~1.5 GB | ~18 MB |
Raise num_ctx freely — it is far cheaper here than the parameter count
suggests. The model declares 262,144.
llama-bench, Q4_K_M, llama.cpp build 60081bb:
| Backend | Prefill (pp512) | Generation (tg128) |
|---|---|---|
| RTX 5090, all layers offloaded | 25,941 t/s | 578 t/s |
| CPU, 8 threads, no offload | 2,925 t/s | 30.9 t/s |
The CPU row is a server CPU restricted to 8 threads, not a phone and not a Raspberry Pi — read it as an order of magnitude, not as a promise about any particular device. 30 t/s is comfortably faster than reading speed.
60081bb, convert_hf_to_gguf.py,
architecture qwen35.--no-mtp is required for this checkpoint. The base Qwen/Qwen3.5-2B
carries 15 mtp.* multi-token-prediction tensors and config.json keeps
mtp_num_hidden_layers: 1. The merged release is the text decoder only and
does not carry them, so a default conversion writes block_count = 25
against 24 blocks of weights and the result fails to load with
check_tensor_dims: tensor 'blk.24.attn_norm.weight' not found. --no-mtp
writes block_count = 24 and drops the NextN metadata. MTP only serves
speculative decoding, which is not used here.train_writer_preview.jsonl (117,077 rows) by seeded reservoir sampling and
rendered through the model's own chat template, at -c 2048. Calibrating on
the distribution the model is actually deployed into is why the 4-bit loss
stays under 2%. writer-2b-imatrix.gguf is included so you can requantise.A_log, dt_bias, ssm_norm, conv1d)
are kept at F32 in every build, which is what a hybrid model needs.Quantised from VNPen/vnpen-writer-2b-v0.1-preview, which is itself built
from Qwen/Qwen3.5-2B (Copyright the Qwen team, Alibaba Group, Apache-2.0) by
LoRA supervised fine-tuning, merged, with the vision tower never loaded.
Apache-2.0 applies to these files too. The visual-novel corpus is not
redistributed.
<a name="中文"></a>
困惑度在与 imatrix 校准集不相交的留出集上测量——300 条样本 / 336,872 token,按模型自身的 chat template 渲染。因此下表的量化损失是实测的, 而不是在校准数据上的自我复现。
| File / 文件 | Size / 大小 | PPL | Δ vs BF16 | RAM (8k ctx) | Use / 用途 |
|---|---|---|---|---|---|
writer-2b-preview-BF16.gguf | 3.78 GB | 10.8465 | — | — | Conversion source; requantise from this. / 转换源,重新量化用。 |
writer-2b-preview-Q8_0.gguf | 2.01 GB | 10.8568 | +0.0103 (+0.09%) | ~2.4 GB | Reference quality, effectively lossless. / 参考质量,基本无损。 |
writer-2b-preview-Q6_K.gguf | 1.56 GB | 10.8992 | +0.0527 (+0.49%) | ~2.0 GB | High quality. / 高质量。 |
writer-2b-preview-Q5_K_M.gguf | 1.41 GB | 10.9559 | +0.1094 (+1.01%) | ~1.8 GB | Balanced. / 均衡。 |
writer-2b-preview-Q4_K_M.gguf | 1.27 GB | 11.0456 | +0.1991 (+1.84%) | ~1.7 GB | Recommended for edge. / 端侧推荐。 |
writer-2b-preview-IQ4_XS.gguf | 1.20 GB | 11.1793 | +0.3328 (+3.07%) | ~1.6 GB | Only if 1.27 GB will not fit. / 仅在放不下 1.27 GB 时用。 |
内存一列是模型文件加 8,192 token 上下文和少量运行时开销,约等于运行所需的空闲内存。
选 Q4_K_M。 相比 BF16 体积小 66%,代价只有 1.84% 的困惑度,
是这组里唯一没有被相邻档位比下去的取舍点。
IQ4_XS 为完整性而提供,但通常是错误选择:它只比 Q4_K_M 小 6%,
却多付 3.07% 的困惑度。原因是本模型输入输出嵌入共享权重,
token_embd.weight(248,320 × 2,048,占全部参数的 27%)在所有低于 Q6_K 的
K-quant 里都被保留为 Q6_K。这一个张量主导了文件体积,低比特格式已无多少可压。
请用 Modelfile。 下述调好的默认参数是靠它带上的,而这个模型需要这些参数:
hf download VNPen/vnpen-writer-2b-v0.1-preview-GGUF \
writer-2b-preview-Q4_K_M.gguf Modelfile --local-dir vnpen
cd vnpen && ollama create vnpen-writer-2b -f Modelfile
ollama run vnpen-writer-2b "写一段黄昏天台、两个人都没把话说开的场景,30 行左右。"
一行命令也可以:
ollama run hf.co/VNPen/vnpen-writer-2b-v0.1-preview-GGUF:Q4_K_M
但这条命令不会读取本仓库的 Modelfile——Ollama 只从 GGUF 构建模型,
因此它能拿到内嵌的 chat template 和停止词,却拿不到重复惩罚设置。
请在会话里自行设置,否则下文描述的退化会出现:
/set parameter repeat_penalty 1.1
/set parameter repeat_last_n 256
llama-cli -m writer-2b-preview-Q4_K_M.gguf --jinja \
-p "写一段黄昏天台、两个人都没把话说开的场景,30 行左右。" \
--temp 0.8 --top-p 0.95 --repeat-penalty 1.1 --repeat-last-n 256 -c 8192
chat template 已嵌入 GGUF,加 --jinja 即可。
随附的 Modelfile 有意覆盖了 Ollama 的出厂默认:
| 参数 | 取值 | 原因 |
|---|---|---|
repeat_penalty | 1.1 | 源模型卡记录了退化问题——剧本条目重复行占比中位数 0.256,最坏一行重复 310 次——并要求设置 1.05–1.15 的惩罚。 |
repeat_last_n | 256 | Ollama 默认窗口 64 token,比一组「旁白 + 对白」还短,看不到它本应打断的循环。 |
stop | <|im_end|>、<|endoftext|> | config.json 从基座继承了 eos 248044(<|endoftext|>),而 chat template 以 <|im_end|>(248046)结束助手轮。两个都列上,模型吐出哪个都能停。 |
temperature / top_p | 0.8 / 0.95 | 源模型评测时所用的解码参数。 |
num_ctx | 8192 | 起点而非上限,见下。 |
| template | 强制空 <think> 块 | 训练时每个助手轮都以空 <think> 块开头,去掉会让模型偏离训练分布。 |
用这组默认参数实测:一个 31 行的场景返回 31/31 行符合「说话人:文本」格式、
31 行全部唯一、done_reason: stop——源模型卡警告的重复退化没有出现。
Qwen3.5 是混合架构:24 层中只有 6 层是全注意力(第 3、7、11、15、19、23 层),其余 18 层是线性注意力(gated delta-net),其循环状态是与序列长度无关的 固定大小。
因此 KV cache 约为 12 KB/token,而不是同规模稠密 24 层模型所需的约 48 KB/token:
| 上下文 | KV cache | + 固定循环状态 |
|---|---|---|
| 8,192 | ~96 MB | ~18 MB |
| 32,768 | ~384 MB | ~18 MB |
| 131,072 | ~1.5 GB | ~18 MB |
可以放心调大 num_ctx,它比参数量给人的直觉便宜得多。模型声明支持 262,144。
llama-bench,Q4_K_M,llama.cpp 构建 60081bb:
| 后端 | 预填充 (pp512) | 生成 (tg128) |
|---|---|---|
| RTX 5090,全层 offload | 25,941 t/s | 578 t/s |
| CPU,8 线程,不 offload | 2,925 t/s | 30.9 t/s |
CPU 一行是限制到 8 线程的服务器 CPU,不是手机也不是树莓派——只能当数量级参考, 不构成对任何具体设备的承诺。30 t/s 已经明显快过阅读速度。
60081bb 的 convert_hf_to_gguf.py 转换,架构 qwen35。--no-mtp。 基座 Qwen/Qwen3.5-2B 带有 15 个 mtp.*
多 token 预测张量,且 config.json 保留了 mtp_num_hidden_layers: 1。
合并发布的权重只有文本解码器、不含这些张量,因此默认转换会写入
block_count = 25 而实际只有 24 块权重,加载时报
check_tensor_dims: tensor 'blk.24.attn_norm.weight' not found。
--no-mtp 会写入 block_count = 24 并移除 NextN 元数据。MTP 只服务于
投机解码,此处并不使用。train_writer_preview.jsonl(117,077 行)以定种子蓄水池采样抽取 1,200 条 /
1,373,121 token,按模型自身 chat template 渲染,-c 2048。
在模型实际部署的同一分布上做校准,正是 4-bit 损失能压在 2% 以内的原因。
仓库内附 writer-2b-imatrix.gguf,可用于自行重新量化。A_log、dt_bias、ssm_norm、conv1d)在所有
版本中均保持 F32,这是混合架构所必需的。量化自 VNPen/vnpen-writer-2b-v0.1-preview;后者由 Qwen/Qwen3.5-2B
(版权归 Qwen 团队 / 阿里巴巴集团,Apache-2.0)经 LoRA 监督微调并合并而来,
视觉塔从未被加载。本仓库文件同样适用 Apache-2.0。视觉小说语料不随之分发。