DenseK3-4B

DenseK3-4B

A Qwen3.5-initialized Dense K3-style 4B Language Model

GitHub · 中文

DenseK3-4B is an independent 4.23B-parameter, text-only base causal language model initialized from Qwen/Qwen3.5-4B-Base and migrated to a Dense K3-style hybrid architecture.

The model replaces the donor's 24 Gated DeltaNet + 8 Gated Attention backbone with:

24 × Kimi Delta Attention (KDA)
 8 × strict-NoPE Multi-head Latent Attention (MLA)

and further integrates Block AttnRes, Dense SiTU-GLU, and True Latent Cache.

DenseK3-4B is not an official Moonshot AI / Kimi model, does not reproduce the complete Kimi K3 architecture, and is not instruction-tuned.

---

🚀 Quick Start

DenseK3-4B is released as a standalone Hugging Face checkpoint. The Qwen donor checkpoint is not required for inference.

Installation

Install a CUDA-compatible PyTorch build, then the minimal validated runtime dependencies:

python -m pip install \
  "transformers==5.12.1" \
  "accelerate>=1.14" \
  "safetensors>=0.8" \
  "flash-linear-attention==0.5.2"

For exact research-environment reproduction, use the full dependency snapshot in the DenseK3 GitHub repository.

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "qinfu19/DenseK3-4B"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="cuda",
).eval()

inputs = tokenizer(
    "DenseK3 preserves the following invariant:",
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=32,
    do_sample=False,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The validated release path is Hugging Face Transformers with the custom model implementation bundled in this repository.

DenseK3-4B is a base language model rather than a conversational assistant. Prompts should therefore be supplied as ordinary text rather than chat messages unless the model is separately fine-tuned and validated for such use.

Compatibility with vLLM, SGLang, Docker Model Runner, or other serving backends is not part of the current release validation, even if Hugging Face automatically displays generic integration examples for those runtimes.

Because DenseK3-4B uses custom Hub code, review the bundled Python implementation before enabling trust_remote_code=True. For immutable or security-sensitive deployments, pin a specific repository revision.

---

🏗️ Model Details

Architecture Summary

The KDA layers use:

Q/K/V heads    16 / 16 / 32
Head dimension 128
Conv kernel    4
Decay rank     128

The MLA layers use:

Attention heads       16
KV latent rank        512
QK NoPE head dim      256
Value head dim        256
RoPE component        disabled

DenseK3-4B intentionally retains a dense FFN rather than adopting Kimi K3's LatentMoE.

Detailed architecture equations, tensor shapes, normalization semantics, and Qwen → DenseK3 parameter mappings are maintained in the GitHub architecture documentation.

---

🎯 Intended Uses

DenseK3-4B is primarily intended for research on:

• pretrained-model architecture migration;
• hybrid recurrent/attention language models;
• KDA and MLA architectures;
• Block AttnRes and SiTU-style model design;
• latent-cache and long-context memory behavior;
• model recovery and knowledge distillation;
• downstream fine-tuning experiments;
• controlled base-model text generation.

DenseK3-4B is not an instruction-tuned chat model and should not be assumed to follow natural-language instructions reliably without additional adaptation.

---

🔄 Training and Migration

DenseK3-4B was not pretrained from scratch.

It inherits its initial pretrained state from Qwen3.5-4B-Base and is progressively migrated through:

Qwen3.5-4B-Base
        ↓
Gated DeltaNet → KDA
        ↓
24 KDA + 8 Gated Attention
        ↓
strict-NoPE MLA migration
        ↓
True Latent Cache
        ↓
Block AttnRes
        ↓
Dense SiTU-GLU
        ↓
Joint Recovery
        ↓
Capability Distillation
        ↓
DenseK3-4B

The final recovery pipeline includes:

• Qwen exact-token online policy distillation (OPD) in the shared tokenizer space;
• Kimi text-space corrective distillation using externally generated corrective teacher text.

The Hugging Face export itself performs no additional training. It materializes the frozen canonical model into a self-contained DenseK3ForCausalLM checkpoint.

The complete migration methodology, stage contracts, recovery settings, and distillation procedure are documented in the DenseK3 GitHub repository.

---

📚 Training Data

DenseK3-4B inherits its initial pretrained knowledge from Qwen3.5-4B-Base. The original Qwen pretraining corpus was not reproduced or redistributed as part of DenseK3.

The public release artifacts verify the following post-migration data components.

Recovery / Distillation Prompt Pool

The general prompt pool contains 1,024 records:

The materialization manifest records no held-out benchmark access for this pool.

Recorded provenance for the general prompt pool references publicly available sources including:

• cais/mmlu
• allenai/ai2_arc
• facebook/belebele
• google/IFEval

Qwen teacher targets are generated in the shared tokenizer space.

Kimi teacher data consists of API-generated text-space corrective trajectories. API credentials, private responses, and teacher caches are not distributed with the model.

Auxiliary Raw-LM Probe

The release also records a train-only FineWeb-Edu raw-language-model probe corpus:

976 sequences × 2,048 tokens
= 1,998,848 effective tokens

This is the size of the materialized probe corpus only and should not be interpreted as the total number of training tokens used to produce DenseK3-4B.

Training-Volume Disclosure

The public release artifacts do not expose a single auditable total record count or token count for the complete final post-selection distillation mixture.

For that reason, this Model Card does not infer or report a total training-token figure beyond the explicitly auditable components above.

Evaluation datasets are treated separately from this training-data disclosure and are not distributed in this model repository.

---

📊 Evaluation

The following values are frozen project results and were not rerun as part of the Hugging Face export.

General Capability

DenseK3-4B retains approximately 92.6% of the donor's MMLU 5-shot accuracy after the architecture migration.

LongBench-v2

The reported LongBench-v2 result uses the project's controlled MC-LL Accuracy metric over 298 eligible examples with tokenized context length ≤ 131,072.

This is a conditional-likelihood comparison metric and not the official LongBench-v2 generation leaderboard metric.

Reduced RULER

The released RULER results use a fixed reduced suite:

13 tasks × 4 cases per task
= 52 cases per context length

This is not the full official RULER leaderboard configuration.

DenseK3-4B does not claim to outperform its Qwen donor on general capability or long-context task quality.

For complete benchmark protocols, case manifests, and aggregate results, see docs/RESULTS.md.

---

💾 Memory and Context

Persistent Attention-State Scaling

For the eight attention layers under a two-byte cache representation:

Qwen-equivalent GQA
≈ 32 KiB/token

DenseK3 MLA latent cache
≈ 8 KiB/token

This corresponds to a derived 75% reduction in sequence-growing persistent attention-state payload.

This comparison applies only to the persistent attention state that grows with sequence length.

It does not include:

• model weights;
• KDA recurrent/convolution state;
• temporary activations;
• allocator fragmentation;
• tokenizer buffers;
• framework/runtime overhead.

Exported Context Configuration

The released configuration contains a maximum-position/runtime configuration of 262,144 tokens.

This is a runtime configuration value and should not be interpreted as validated 256K task quality.

Standardized public task-quality evaluation for DenseK3-4B currently stops at 128K.

512K Runtime Probe

A separate exact-semantics runtime probe on the pre-distillation DenseK3 parent architecture completed:

Context length             524,288 tokens
Full prefill               PASS
Continued decode           PASS

Peak allocated             15,762,103,296 bytes
                           ≈ 15.76 GB
                           ≈ 14.68 GiB

Peak reserved              15,994,978,304 bytes

Persistent latent state     4,294,967,296 bytes
                           = 4 GiB

Persistent expanded K                    0
Persistent expanded V                    0

The probe used True Latent Cache with no sliding window and no approximate attention.

It validates:

512K runtime feasibility and True Latent Cache semantics

It does not establish:

512K effective-context retrieval or reasoning quality

---

✅ Standalone Export Validation

DenseK3-4B is distributed as a materialized standalone Hugging Face checkpoint.

The exported model was compared against the frozen canonical research reference and passed:

• structural parity;
• numerical logit parity;
• top-1 / top-5 prediction parity;
• latent-cache parity;
• deterministic greedy-generation parity;
• Safetensors save/load round-trip;
• clean-room AutoModelForCausalLM loading.

The maximum observed absolute last-token logit difference in the frozen parity suite was:

0.017578125

while the validated prediction sets, cache behavior, deterministic generation, and save/load behavior remained consistent with the reference model.

Machine-readable validation evidence is available in PARITY_SUMMARY.json.

The standalone conversion is a packaging and implementation refactor, not a new training stage or a newly selected checkpoint.

---

⚠️ Limitations, Safety, and Bias

DenseK3-4B has several important limitations:

• It is a base causal language model, not an instruction-tuned assistant.
• It has not undergone dedicated safety alignment.
• DenseK3-4B trails Qwen3.5-4B-Base on the reported aggregate general-capability and long-context benchmarks.
• No throughput advantage over the Qwen donor is claimed.
• The 512K runtime probe does not establish 512K retrieval, reasoning, or effective-context quality.
• Standardized task-quality evidence currently stops at 128K.
• This release does not report dedicated bias, toxicity, hallucination, misuse, or red-team evaluations.
• Outputs may be inaccurate, biased, unsafe, offensive, or otherwise unsuitable for a target application.
• The model may inherit undesirable behavior or biases from the Qwen donor and from the recovery/distillation data.
• The tokenizer inherits upstream special tokens, including tokens originating from multimodal Qwen infrastructure; DenseK3-4B itself is text-only and does not provide image, audio, or video understanding.
• The validated inference path is Hugging Face Transformers with the bundled custom model implementation. Other serving backends are not covered by the release validation unless explicitly documented.

Users should perform application-specific capability and safety evaluation before deployment, particularly in medical, legal, financial, safety-critical, or other consequential settings.

---

🧬 Provenance and Reproducibility

The public model identity is:

DenseK3-4B

Its frozen internal research provenance corresponds to the project stage historically identified as:

P11.6

P11.6 is retained only for research traceability and is not a second public model name.

The donor model is:

Qwen/Qwen3.5-4B-Base
revision:
1001bb4d826a52d1f399e183466143f4da7b741b

Source code, architecture contracts, migration implementation, training documentation, benchmark protocols, environment snapshots, and release evidence are available at:

https://github.com/yigu666/DenseK3

No donor weights, datasets, optimizer states, teacher responses, API credentials, or private server paths are included in this model repository.

---

🙏 Acknowledgements

DenseK3 builds on models, ideas, and open-source work from:

• Qwen / Qwen3.5
• Kimi / Kimi K3 / Kimi Linear
• Attention Residuals
• Flash Linear Attention
• Hugging Face Transformers
• the authors and maintainers of the evaluation benchmarks used in this project

DenseK3-4B is an independent research release and is not affiliated with or endorsed by Moonshot AI.

See NOTICE and the GitHub repository's THIRD_PARTY_NOTICES.md for attribution and upstream licensing information.

---

📖 Citation

If DenseK3 is useful in your research, please cite:

@software{densek3_2026,
  author = {yigu666},
  title  = {DenseK3: A Qwen3.5-Initialized Dense K3-Style Language Model},
  year   = {2026},
  url    = {https://github.com/yigu666/DenseK3}
}

---

📄 License

DenseK3-4B is released under the Apache License 2.0, subject to the applicable terms of the donor model and third-party dependencies.

Please review:

• LICENSE
• NOTICE
• THIRD_PARTY_NOTICES.md