IsoFormer

IsoFormer

IsoFormer is a Chinese language model trained on large-scale Chinese novel corpora.
The model adopts a symmetric-dimension Transformer architecture, where word embeddings, attention projections (Q/K/V), and feed-forward layers share the same dimensionality.
Perhaps the current transformer model contains the largest Q, K, V Dim, HAHA.

Model Details

Qwen3ForCausalLM type
Model Description
Model type: Transformer Decoder Language Model
Architecture: Unified-Dimension Transformer
Training data: Chinese novels and narrative text corpora
Language: Chinese
License: (Apache-2.0)
Key Features
Unified dimensionality architecture
Simplified transformer design
Optimized for Chinese narrative generation
Reduced parameter redundancy

Unlike conventional transformers where the feed-forward layer expands the hidden size (e.g., 4× expansion), IsoFormer keeps all core modules at the same dimensionality.

Architecture

IsoFormer introduces a symmetric Transformer architecture.

Typical Transformer:

Embedding: d
Attention: d
MLP: 4d → d

IsoFormer:

Embedding: d
Q projection: d
K projection: d
V projection: d
MLP: d → d

This unified design provides:

Maintaining equal dimensions, without any dimensional changes, is more conducive to model training and reduces the additional accuracy loss caused by dimensionality changes. With Q, K, V, and Word embeddings all within the same dimension in MLPs, reducing the number of Head elements to 1 and increasing the Q, K, and V dim values ​​to 1536 enhances expressive power. We will then verify whether this approach is more effective.
structural simplicity
reduced parameter count
more regular compute patterns

which may improve efficiency on certain hardware.

Training Details

Training Device

4060ti 16G
one epoch
It consumed approximately 10 kilowatt-hours of electricity.
loss to 4.0

Training Data

The model was trained primarily on Chinese fiction datasets, including:

about 1000  web novels, about 2G, so you should talk with it by Chinese

only pretraing stage, No dialogue mode. 

Training objective:

Causal Language Modeling (CLM)

Optimization:

AdamW optimizer
cosine learning rate schedule
mixed precision training

Tokenization:

BPE / SentencePiece tokenizer
Intended Uses

IsoFormer is designed for:

Chinese story generation
novel writing assistance
narrative text modeling

Example applications:

AI novel writing
roleplay chatbots
storytelling tools
Limitations

The model has several limitations:

Primarily trained on fictional text
May produce hallucinated information
Not optimized for factual QA or technical knowledge
May reflect biases present in online fiction datasets

Users should verify outputs when using the model in production environments.

Example Usage

Using HuggingFace Transformers:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Terminator-of-AI/IsoFormer")
model = AutoModelForCausalLM.from_pretrained("Terminator-of-AI/IsoFormer")

prompt = "少女抬头看向远方的天空"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=100)

print(tokenizer.decode(outputs[0]))

Others

Feel free to leave a comment if you're interested; a good job is also welcome.