qwen-3.5-80b-post-cpt-base

Qwen 3.5 80B Post-CPT Base

This repository contains the selected repaired base checkpoint for our 80B-class Qwen 3.5-derived vision-language model.

Summary

This release is a repair / recovery continued-pretraining (CPT) checkpoint, not a final instruct model and not a posttrained agent model.

The published checkpoint is:

• v7-20260415-234214/checkpoint-100

selected from several checkpoint-100 candidates after a short multimodal repair-CPT run intended to stabilize a custom depth-upscaled 80B-class merged model and recover coherent base-model behavior.

Important note on the source checkpoint

At construction time, the upstream raw 27B base checkpoint was not available to us in the environment we were working from. The practical upstream starting point was the Qwen 3.5 27B instruct checkpoint.

This matters for interpretation:

• this 80B artifact should be treated as a repaired base candidate, not as a clean continuation of a pristine raw-base lineage
• some degradation or drift in instruction-format adherence, response style, and formatting behavior is expected during repair CPT
• additional posttraining repair and posttraining proper are expected and planned as the next stage

So this release is intentionally framed as the best current repaired base candidate from the available source path, not the end of the pipeline.

Construction pipeline

The model was built through a staged pipeline:

1. Abliteration / de-alignment pass on the available Qwen 3.5 27B instruct source checkpoint.
2. Custom scripted end-to-end depth upscaling / merge producing an ~80B-class dense multimodal model. The conceptual template is mergekit-style passthrough depth stacking, but the actual published artifact was produced through our own merge / graft pipeline written in-house after mergekit-based tooling paths did not work cleanly for this specific VLM setup.
3. Short repair CPT to stabilize the merged model, recover coherent text-side behavior, and preserve functional multimodal capability.

Conceptually, the goals were:

• preserve the first-class VLM nature of the Qwen 3.5 family
• produce a larger dense multimodal base candidate using depth upscaling instead of training a new 80B-class model from scratch
• then use a short repair CPT pass to recover coherence before moving into posttraining

Merge / architecture rationale

Rather than training an 80B model from scratch, we chose depth upscaling from a smaller pretrained source. The rationale is that repeating the pretrained stack end-to-end produces a deeper network that still inherits useful pretrained structure, and a short repair CPT pass can then smooth the new transitions. The merged artifact is not expected to be strong immediately out of merge; the repair CPT is an intentional part of the process, not an optional cleanup step.

Repair CPT setup

The selected checkpoint came from a short multimodal repair-CPT pilot using:

• backend: MS-Swift + DeepSpeed ZeRO-3
• nodes: 4 (32 B200 GPUs total)
• dtype: bf16
• mode: full-parameter training
• max sequence length: 8192
• save cadence: every 100 steps
• gradient checkpointing: enabled

The first multimodal pilot had to normalize the training data format before use, because the raw FineVision-derived parquet metadata did not cleanly flow through the trainer's dataset path.

Repair data mix

The pilot used a conservative mixture intended to recover general text-side behavior while keeping multimodal / document / OCR competence alive:

• 75% text backbone: cached C4 / en
• 25% multimodal / doc mix, normalized into a training-friendly format from FineVision-derived sources:
  • olmOCR-mix-0225-documents
  • olmOCR-mix-0225-books
  • docvqa
  • arxivqa
  • ocrvqa

Why checkpoint-100 was selected

This was explicitly treated as circuit repair CPT, not as a long-horizon capability training run, so the lowest absolute loss is not the primary criterion.

The checkpoint was chosen based on:

1. coherent text generation
2. absence of obvious merge-path regressions
3. clearly functional document / image understanding in qualitative checks
4. preference for an early, conservative checkpoint to reduce risk of overfitting to the repair data mix

Tiny checkpoint-selection validation

Only small validation subsets were used at this stage. These were run purely to identify the best repair checkpoint and to confirm that the model was recoverable. They should not be interpreted as final benchmark claims.

Plain Transformers smoke

Both the raw merged 80B checkpoint and checkpoint-100 were verified to:

• load in plain transformers
• generate tokens successfully

In a trivial prompt smoke test, the selected checkpoint produced a clean coherent response, while the raw merged checkpoint remained visibly degraded.

Tiny GSM8K subset (n=5)

A tiny direct transformers GSM8K slice was used for quick text-side checkpoint comparison:

• v6-20260415-234215/checkpoint-100: 1 / 5
• v7-20260415-164214/checkpoint-100: 4 / 5
• v7-20260415-234214/checkpoint-100: 4 / 5

The published checkpoint was chosen from the strongest candidates in this tiny sweep.

Tiny multimodal qualitative check

A small document / OCR-style qualitative check showed that the selected checkpoint is clearly able to:

• read document pages
• extract structure and content
• produce semantically faithful long-form document responses

A simple exact-match metric used in that check understated the practical multimodal capability, because the model frequently returned structured, explanatory or partially reformatted transcriptions instead of exact verbatim text. So the small multimodal checks were useful as sanity checks, but not for rigorous ranking.

What this release is for

Use this model as a base candidate for the next stages of work:

• stronger multimodal repair CPT, if needed
• posttraining repair of instruction behavior / formatting / style
• supervised posttraining
• preference / constitutional alignment
• downstream agent-style posttraining

What this release is not

This repository should not be treated as:

• a final instruction-tuned model
• a finished OCR-specialized model
• a final benchmarked public release with comprehensive evaluation

It is the selected post-CPT repaired base checkpoint from the current pipeline stage.

Expected next phase

The next stage is expected to focus on posttraining, with additional attention to:

• cleaning up instruction behavior inherited from the instruct-source starting point
• improving multimodal answer style and exactness where needed
• running broader, more task-appropriate evaluations beyond the tiny checkpoint-selection subsets used here
• optional additional repair CPT with a stronger multimodal focus if evaluation indicates it is needed

Provenance

Published from:

• source checkpoint: v7-20260415-234214/checkpoint-100
• repository: artivus-ai/qwen-3.5-80b-post-cpt-base