qwen3_5-full-attn-only-14

nineninesix/qwen3_5-full-attn-only-14

qwen3_5-full-attn-only-14 is a machine learning model from nineninesix. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

Note: This model was assembled with qwen35-rebuild. Embedding and output projection weights are copied from the source model. Transformer blocks are freshly initialized (random, std=0.02). The model requires fine-tuni…

Downloads · 30 days

395

3% of all-time downloads

All-time downloads

14.3K

Public

Parameters

511M

1 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors1 GB · 98%

At a glance

License: apache-2.0
Model type: qwen3_5_text
Access: Public
Created: Mar 19, 2026
Updated: Jul 6, 2026
SHA: a26dff91

Base models

Qwen/Qwen3.5-0.8B-Base

Model card

From the Hugging Face model README

Rebuilt Qwen3.5 — `qwen3_5-full-attn-only`

Note: This model was assembled with qwen3_5-rebuild. Embedding and output projection weights are copied from the source model. Transformer blocks are freshly initialized (random, std=0.02). The model requires fine-tuning before it can generate coherent text.

Source model

KaniTTS-research-team/qwen-3.5-prepare-0.6b

Architecture

Parameter	Value
`model_type`	`qwen3_5_text`
`hidden_size`	1024
`vocab_size`	248320
`num_hidden_layers`	14
`linear_attention` layers	0
`full_attention` layers	14

Parameter budget

Component	Parameters	Notes
`embed_tokens` (text emb)	254.28M	✅ copied from source
`lm_head`	tied	✅ shared with `embed_tokens`
transformer blocks	256.94M	🔄 freshly initialized
final `norm`	0.001M	🔄 freshly initialized
Total	511.22M	vs 586.36M source

embed_tokens represents 49.7% of total parameters. It is the dominant component — 254M params encode 248,320 token embeddings × 1024 dims.

Layer-by-layer breakdown

Layer	Type	Attn params	MLP params	Total
0	`full_attention`	7.341M	11.010M	18.353M
1	`full_attention`	7.341M	11.010M	18.353M
2	`full_attention`	7.341M	11.010M	18.353M
3	`full_attention`	7.341M	11.010M	18.353M
4	`full_attention`	7.341M	11.010M	18.353M
5	`full_attention`	7.341M	11.010M	18.353M
6	`full_attention`	7.341M	11.010M	18.353M
7	`full_attention`	7.341M	11.010M	18.353M
8	`full_attention`	7.341M	11.010M	18.353M
9	`full_attention`	7.341M	11.010M	18.353M
10	`full_attention`	7.341M	11.010M	18.353M
11	`full_attention`	7.341M	11.010M	18.353M
12	`full_attention`	7.341M	11.010M	18.353M
13	`full_attention`	7.341M	11.010M	18.353M

Layer type sequence

   0: full_attention
   1: full_attention
   2: full_attention
   3: full_attention
   4: full_attention
   5: full_attention
   6: full_attention
   7: full_attention
   8: full_attention
   9: full_attention
  10: full_attention
  11: full_attention
  12: full_attention
  13: full_attention

Initialization

Embeddings: copied as-is from KaniTTS-research-team/qwen-3.5-prepare-0.6b
Transformer blocks: random init, std=0.02
dtype: bfloat16
Seed: 42

Type: qwen3_5_text
License: apache-2.0
Created: Mar 19, 2026
Updated: Jul 6, 2026