qwen3.5-4b-chunky-FP8

p4b/qwen3.5-4b-chunky-FP8

Bitext chunk-alignment model. Given a source/target pair pre-segmented with numbered split markers [|i|], it predicts the cumulative split points at which the two sides stay mutually aligned — and omits spans that are one-sided (untranslated / noise).

FP8 (W8A8) language model, quantized with llm-compressor using static, calibrated activation scales.

Input / output format

Input:

<src>First sentence.[|1|]Second sentence.[|2|]Third.[|3|]</src><tgt>첫 문장.[|1|]두 번째 문장.[|2|]세 번째.[|3|]</tgt>

Output — cumulative SourceIndex-TargetIndex boundary pairs:

<answer>1-1, 2-2, 3-3</answer>

This is a text model. It is packaged in the Qwen3_5ForConditionalGeneration layout (the original vision tower is bundled unused) so it loads in both Hugging Face Transformers and vLLM; no images are involved.

Serving (vLLM)

vllm serve p4b/qwen3.5-4b-chunky-FP8 --language-model-only --mamba-cache-mode=align

--language-model-only skips the (unused) vision tower; --mamba-cache-mode=align is required by the Qwen3.5 hybrid linear-attention layers.

Usage (Transformers)

from transformers import AutoModelForImageTextToText, AutoTokenizer

tok = AutoTokenizer.from_pretrained("p4b/qwen3.5-4b-chunky-FP8")
model = AutoModelForImageTextToText.from_pretrained("p4b/qwen3.5-4b-chunky-FP8", device_map="auto")

msgs = [{"role": "user", "content": PROMPT + text}]  # text = the <src>...</tgt> block
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(input_ids=enc["input_ids"].to(model.device), max_new_tokens=128)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

License

Apache-2.0.