Qwen3.8-27B-Palimpsest

Qwen3.8-27B-Palimpsest

Palimpsest is an experimental BF16 fine-tune of Qwen3.8-27B focused on literary prose, continuity, structured tool use, and position-aware long-context behavior. It retains the base model's vision encoder and merges two small, architecture-aware LoRA stages into the language model.

The name reflects the model's goal: preserving and revising information across layers of a long working context. The training mix uses the openly licensed and project-generated sources documented below.

Release status: preliminary. The local behavior and positional gates below passed, but the full public capability suite and the derived GGUF quant ladder are still being evaluated. This card will be updated with reproduced benchmark results rather than copying scores from the base model.

What changed

The fine-tune used two stages:

1. Behavior and prose refinement. A 1,152-example assistant-loss-only mix trained LoRA modules across the model's full-attention projections, linear attention projections, and MLPs. The selected checkpoint was step 64.
2. Position-aware long-context repair. A 180-example PoSE-style curriculum trained only Q/K LoRA modules in the 16 full-attention layers for 80 steps. Physical sequences were 956-1,127 tokens long, with monotonically mapped virtual positions at 32K, 128K, 262K, 524K, and 1M.

The final long-context stage updates 1,507,328 parameters in 64 LoRA tensors. Both adapters were merged into the original BF16 weights; this repository does not require PEFT adapters at inference time.

Training data

The behavior mixture contained:

The behavior set contained 1,407,028 rendered tokens and 796,433 supervised assistant tokens. Named-author labels, uncensored profiles, and quality-judge outputs were excluded from the Novelist slice.

The long-context stage used six deterministic families: single retrieval, multi-hop joins, last-write chronology, unanswerable negatives, distractor collisions, and ordered joins. Forty-five of its 180 rows incorporated Apache-2.0 natural-text distractors. A separate 60-row fixture with disjoint seeds and templates was held out for evaluation.

Dataset revisions used for reproducibility:

• Dxniz/Novelist: 308c139e59687e6037394bbbefda57302b898fdf
• nchapman/figaro-creative-writing: 23b980e6a6dcc8d7a4709841ffbcccca7607ba4b
• microsoft/orca-agentinstruct-1M-v1: 86d609183249ff8037eae33d76ebca3af9390ea8

Local evaluation

These are small frozen development gates, not replacements for standard public benchmarks. Scores compare the locally loaded NF4 base/parent adapter and the selected adapter under the same deterministic evaluator.

The earlier behavior-only checkpoint improved its corresponding base control from 75/92 to 80/92. The second stage preserved tool formatting and added one continuity pass while improving the positional gate.

Derived-GGUF deployment qualification

The importance-calibrated MIX-IQ3KT GGUF converted from this merged checkpoint passed a five-needle, near-full-window audit at every native tier. These are GGUF deployment measurements, not BF16 throughput claims. Context shifting was disabled and every response had to reproduce all five codes in order under an exact one-line output contract.

Peak model-process swap was zero at all native tiers. The measured interactive recommendation is 131,072 tokens; the measured native strict/batch maximum is 262,144 tokens. YaRN-scaled 512K–1M testing is still in progress and is not inferred from the native passes.

At a 524,288-token allocation, a separate derived-GGUF speed tune using a fixed 9,011-token prompt plus 128 generated tokens improved prompt throughput from 316.52 to 467.11 tok/s and decode throughput from 2.44 to 4.49 tok/s on the local 8 GB + 8 GB + 10 GB rig. It used CPU Q4 KV, static YaRN factor 2, an Ampere-heavy 10/4/12 layer split, micro-batch 512, eight CPU threads, and native MTP. This deployment measurement does not replace the pending fully populated 512K retrieval gate and is not a BF16 throughput claim.

Context-length caveat

The underlying configuration remains 262,144 tokens natively. The 1M result above tests learned behavior at virtual token positions; it does not prove that this repository can process a resident one-million-token prompt on ordinary hardware.

On the local 8 GB + 8 GB + 10 GB GPU rig, a 1M-token FP16 KV payload for the 16 full-attention layers is about 65.5 GB before model weights and runtime buffers. Even a 4-bit KV payload is about 16.4 GB. Practical 1M use therefore requires a compatible RoPE/YaRN configuration plus approximately 2-bit KV, substantial CPU offload, or a different runtime/hardware setup. For the derived mixed GGUF, fully materialized accuracy is established through the native 262K window above. Fully materialized BF16 and YaRN-scaled 512K–1M accuracy have not yet been established.

Usage

Use a recent Transformers release with Qwen3.5/Qwen3.8 support and trust the repository's processor/chat template. A minimal text-only example is:

from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

repo = "xero0000/Qwen3.8-27B-Palimpsest"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    repo,
    dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(repo)

messages = [
    {"role": "user", "content": "Write a restrained scene where an old promise becomes newly relevant."}
]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Start with the base model's sampling guidance:

• thinking: temperature 1.0, top-p 0.95, top-k 20;
• non-thinking: temperature 0.7, top-p 0.80, top-k 20, presence penalty 1.5.

For deterministic tool/schema checks, use greedy decoding. The upcoming GGUF repository will document runtime-specific parameters and tested quant tiers.

Intended use

Palimpsest is intended for research and local experimentation involving creative drafting, continuity editing, structured tool calls, and long-context position generalization. It is not validated for medical, legal, financial, or other high-stakes decisions. Users remain responsible for checking generated facts, code, tool arguments, and safety-sensitive outputs.

Limitations

• Standard public capability benchmarks and human-blind prose comparisons are still pending.
• The current long-context test is teacher-forced and uses short physical sequences mapped to long virtual positions.
• English dominates the fine-tuning data; multilingual regressions have not been measured.
• Vision weights were retained but were not updated or re-evaluated during this text-focused fine-tune.
• Fine-tuning can introduce new failure modes even when small local gates improve. Do not infer base-model benchmark parity from the results above.

Quantized release

The quantized release repository xero0000/Qwen3.8-27B-Palimpsest-GGUF contains the developing quant ladder and an importance-calibrated mixed build of exactly 11,702,780,800 bytes. Quantized scores are reported separately from BF16 results, and both cards will be updated as the remaining gates finish.

License and attribution

This derivative is released under Apache-2.0, following Qwen/Qwen3.8-27B. Review the upstream model card and the individual dataset licenses before redistribution or commercial use.

Citation

@misc{qwen38palimpsest2026,
  title        = {Qwen3.8-27B-Palimpsest},
  author       = {xero0000},
  year         = {2026},
  howpublished = {Hugging Face model release},
  url          = {https://huggingface.co/xero0000/Qwen3.8-27B-Palimpsest}
}