NCP_ArchPreview_dolma3_8.9B_Stage2_v2

NCP-ArchPreview 8.9B - Stage 2

Model collection | Technical report (arXiv) | HF Papers | Training code (coming soon) | Evaluation code

NCP-ArchPreview Stage 2 continues the Stage 1 model on Dolma 3 Dolmino, following the second stage of the OLMo-3 data curriculum. It retains joint Next Token Prediction (NTP) and Next Concept Prediction (NCP), allowing the model to refine token-level capabilities and its learned concept space during continued pretraining.

Developed by The NCP Team at Shanghai AI Lab and LUMIA Lab, Shanghai Jiao Tong University, the model has approximately 8.94B parameters and an 8,192-token training context. Stage 2 remains a base-model release for completion, evaluation, and further adaptation.

Highlights

• Continued training in token and concept space. Stage 2 preserves the 16-layer Encoder, 8-layer Concept Module, and 16-layer Decoder from Stage 1.
• Improved main Stage 2 average. Table 1 reports 57.57 Overall AVG, compared with 56.98 for the corresponding OLMo-3-7B model, with 83.02 GSM8K and 68.48 MMLU.
• Three matched-budget data recipes. Appendix C compares V1, V2, and V3 under a common inference and sampling setup. V1 leads on the reported code, mathematics, academic-STEM, and logical-reasoning tasks; V2/V3 lead on HellaSwag.
• A Target for concept-conditioned drafting. Stage 2 V1 is the paired Target for the separate NCPFlash draft model.

The report also finds that Stage 2 reaches the OLMo-3-7B final training loss with 66.2% of the training tokens, corresponding to 1.51x convergence in token budget. This is a training-loss comparison, not an inference throughput measurement.

Stage 2 Releases

These labels identify alternative Stage 2 data recipes. Higher version numbers do not imply a stronger model: although aggregate training loss decreases from V1 to V3, several downstream capabilities decline. The quick-start example uses V1, which is the strongest of these recipes across the free-form capabilities evaluated in Appendix C.

Evaluation

Main Stage 2 Comparison

The following values are the report's main Stage 2 comparison with OLMo-3-7B and should not be assigned to all three released variants. Scores are percentages; deltas are absolute percentage points.

<details> <summary>Results by domain and likelihood evaluation</summary>

Likelihood is reported separately in bits per UTF-8 byte (BPB), where lower is better.

</details>

The main Stage 2 model improves the overall average while showing a mixed profile across individual tasks, including lower HumanEval and ARC-Challenge scores. The report discusses a possible mismatch between the continued-training mixture and downstream domains; lower aggregate loss alone does not establish better performance on every task.

V1 / V2 / V3 Recipe Comparison

All three Stage 2 variants share the same inference and sampling configuration within this comparison.

For recipe selection, the report treats V2 and V3 as effectively tied on HellaSwag because they differ by only 0.05 percentage points. Its held-out proxy analysis supports the V1 > V2 > V3 ordering on four free-form capability measurements. This evidence compares complete recipes and does not isolate the contribution of an individual data source.

Protocol and Aggregation

The main comparison follows the OLMo evaluation protocol in OLMo-Core. Appendix E and Table 13 specify the per-task settings, including:

• GSM8K: 8-shot, one completion per problem, pass@1.
• MATH-500: 4-shot, 32 completions per problem, pass@1.
• HumanEval and MBPP: 3-shot, 32 completions per problem, execution pass@1.

The sample count is the number of generated completions per problem, not the inference batch size. Overall AVG is the unweighted mean of 26 constituent benchmark scores, excluding the aggregate MMLU row and domain averages. The ten-benchmark BPB AVG is calculated separately and is excluded from Overall AVG. See the evaluation configuration for the corresponding release assets and complete runtime settings.

Architecture

NCP-ArchPreview processes text through three modules:

1. A 16-layer Token Encoder produces contextual token states. Mean pooling over each group of four states forms a continuous concept representation.
2. An 8-layer Concept Module predicts the next concept. Product quantization defines the concept vocabulary using 32 codebooks. Predictions are differentiable weighted combinations of codewords.
3. A 16-layer Token Decoder receives token states and causally aligned concept predictions, then produces the next-token distribution.

Intra-module residual connections mix states across depths. Cross-module residual connections connect Encoder to Concept Module, Encoder to Decoder, and Concept Module to Decoder. Concept feedback is shifted and repeated at token resolution to preserve causality.

tokens -> Token Encoder -> mean pooling -> Concept Module -> concept prediction
                 |                                                |
                 +--------------> Token Decoder <-----------------+
                                       |
                                next-token logits

Architecture details follow Sections 2-3 and Appendix B, Table 10, of the technical report.

Training and Data

Stage 2 continues the Stage 1 checkpoint with the Dolma 3 Dolmino data curriculum described in Section 4.1. The underlying data family is documented in the Dolma 3 Dolmino pool. The pool describes available source data; the exact sampling mixture belongs to each Stage 2 recipe.

The Encoder, Concept Module, Decoder, and learned concept vocabulary continue to train jointly using NTP, NCP, and VQ objectives. The report describes Muon for matrix-valued parameters and AdamW for other parameter groups, with a cosine learning-rate schedule.

Quick Start

The checkpoint includes custom Transformers model code. This example uses a single prompt on one CUDA GPU with sufficient memory for the BF16 weights, activations, and cache.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_v1"
device = "cuda"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).eval().to(device)

prompt = "The role of hierarchical representations in language modeling is"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        use_cache=True,
        pad_token_id=tokenizer.eos_token_id,
    )

continuation = output_ids[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(continuation, skip_special_tokens=True))

Use plain completion or few-shot prompts for this base model. The example is a loading and generation example; benchmark reproduction requires the prompts, sampling configuration, and scorers described in the report. For reproducible runs, pin the model and tokenizer to the same Hub commit with revision. See the inference guide for supported runtime versions and optimized serving.

Using Stage 2 V1 with NCPFlash

The separate NCPFlash repository contains a roughly 1.1B-parameter DFlash2-style drafter paired with Stage 2 V1. It uses Target hidden states and a causal concept signal to propose blocks for speculative verification.

Section 5.3 reports an increase in mean accepted length from 5.933 to 6.180 (+4.17%) relative to the matched drafter without concept conditioning. This measures tokens committed per verification round. End-to-end throughput requires a separate measurement with the paired models and serving runtime. See the NCPFlash inference guide for use of the two checkpoints together.

Intended Use and Limitations

The Stage 2 releases support research on continued pretraining, latent-space modeling, data-recipe selection, benchmark evaluation, and further adaptation. Stage 2 V1 also serves as the Target for the paired NCPFlash drafter.

• These are base models, without conversational alignment. They may produce inaccurate, biased, or harmful content.
• A lower training loss or a higher version number is not a reliable capability ranking. Select a recipe using evaluations relevant to the intended task.
• The reported maximum training context is 8,192 tokens. Longer-context capability requires separate evaluation.
• Results from one variant or evaluation setup should not be attributed to another. The VQ adaptation and 3B-scale MTP studies in the report concern separate experimental checkpoints.

Citation

@misc{ncpteam2026ncparchpreviewtechnicalreportmoving,
      title={NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction}, 
      author={NCP Team and Jiaqi Cao and Chiyu Chen and Shuang Cheng and Xu Cheng and Beiya Dai and Yufan Feng and Kewen Ge and Ruijun Ge and Jiayi Huang and Yang Jiao and Dahua Lin and Zhouhan Lin and Yifan Liu and Yuliang Liu and Biqing Qi and Mowen Ruan and Junzhe Shen and Yunchong Song and Hao Sun and Zhongbo Tian and Yixuan Wang and Rubin Wei and Jiaxin Xiong and Kangyu Yang and Qian Yao and Qi Zhang and Bowen Zhou},
      year={2026},
      eprint={2609.10715},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.10715}, 
}

License

The model weights are released under the Apache License 2.0.

Acknowledgements

We thank the OLMo and Dolma teams and the contributors to the open datasets, training libraries, and evaluation tools used in this work.

<!-- Editorial source: Concept_Olmo_7B.pdf, report dated 2026-09-04, SHA-256 acf619d01c0ad46e4012b9dacf1af7c2b21501a510008e6dad57b035fd1eaf9e. Main comparison: Table 1. Recipe comparison: Appendix C / Table 11. Architecture: Table 10. Evaluation protocol: Appendix E / Table 13. NCPFlash evidence: Section 5.3 / Table 9. Target pairing: release owner confirmed v1. -->