Aether-7B-5Attn

📱 Run it on your phone or a GPU-less PC → POCKET · 🚀 Try it live (CPU chat)

VIDRAFT's on-device family: a 35B model that runs on iPhone and on CPU with no GPU — stock llama.cpp, no fork.

⚠️ Use batch_size=1. Some attention branches (NSA family) do not consume a padding mask, so batching padded sequences can silently corrupt results. Run inference one sequence at a time (batch_size=1).

Aether-7B-5Attn — Base

<!-- AETHER-FAMILY-LINKS -->

Aether family —

<!-- /AETHER-FAMILY-LINKS -->

Which Aether model should I use?

All four load the same way:

AutoModelForCausalLM.from_pretrained(MODEL, trust_remote_code=True, dtype=torch.bfloat16)

📚 Part of the Aether Foundation Model collection — base, instruction-tuned, and checkpoints in one place.

🧩 Intermediate checkpoints (110k · 115k · 162k) are released as a dataset: Aether-7B-5Attn-checkpoints.

🇰🇷 The first Korean foundation model to open its training data and training code

A 6.59B MoE language model that arranges its attention across 49 layers on a 7×7 Latin square. Weights · training data recipe · training code · complete training logs · full architecture source — all released.

Trained from scratch on 16 × NVIDIA B200 (2-node FSDP) over ~46 days, 144.2B tokens total. Apache-2.0

---

1. Why release everything

Most "open" LLMs today give you weights only. Weights alone let you neither understand a model, nor verify it, nor rebuild it. A model whose diet is secret is a black box, and you cannot build national or institutional AI sovereignty on a black box.

The Allen Institute's OLMo reset the bar for what "open" means — that only by releasing the training data, the training code, and the logs alongside the weights does a model become science. Aether follows that bar.

What we release

The recipe in §5 alone is enough to reconstruct our training data byte-for-byte. Every source is a public repository, so identical tokenizer + EOS + mixing weights produce identical binaries. Please verify it. That is what it is for.

A first for Korean foundation models

Korea has built foundation models from scratch before. None of them released their training data. To our knowledge, Aether is the first.

This is what sovereign AI actually means. Any country, any company, any lab should be able to rebuild its own foundation model from scratch using nothing but this repository. Downloading someone else's weights is not sovereignty.

---

Open-source fully-open LLMs — 6-country comparison

Across six sovereign fully-open LLMs, disclosure is comparable; Aether is the only single AI startup, with the most attention types (5) in a Latin-square layout.

2. Design — why should every layer use the same attention?

2.1 Nothing says one attention must serve all layers

Since GPT, nearly every transformer uses the same attention at every layer. A 49-layer model runs the same operation 49 times.

But layers do different jobs. Early layers pick up local morphology and grammar; middle layers handle syntactic structure; late layers work across the whole document. If the job is different, why must the operation be identical?

This is a question about a design assumption, not about performance. "Every layer, same attention" is not a proven conclusion — it is a convention that was never seriously questioned. Aether relaxes it.

2.2 Each attention carries a different inductive bias

None of them dominates the others. Each is good at something different and pays a different price. Which means picking one and repeating it 49 times also repeats its weakness 49 times.

2.3 The Latin square is a control, not decoration

"Mixing should help" is an easy intuition, but mixing carelessly makes the result impossible to attribute. If full happens to cluster in the late layers, the model did well "because its late layers are full" — not "because it is heterogeneous."

That is why the 49 layers form a 7 × 7 Latin square. A Latin square is a combinatorial arrangement that guarantees each type appears exactly once in each position, structurally preventing any type from concentrating at any depth.

The Latin square is therefore a control mechanism: it lets the question "what does heterogeneous placement do?" be asked without placement bias.

To our knowledge, no released model places heterogeneous attention in a Latin square.

---

3. Attention composition — a 7×7 Latin square over 49 layers

Seven attention labels are placed across 49 layers along the Latin square, exactly 7 layers each (7 × 7 = 49).

Where the name 5Attn comes from. The seven labels reduce to five distinct mechanisms: compress runs the full path, and linear is SDPA plus a gate. The model is named for the number of mechanisms (5), not the number of labels (7).

The placement rule and every implementation are in aether_pkg/. See LATIN_SQUARE_7x7 for the layer↔label mapping.

3.1 Measured cost profile per mechanism

Each of the five mechanisms was measured as a single standalone layer: prefill latency and peak memory. An attention's compute and memory cost is a property of the architecture, independent of trained weight values, so this table is reproducible by anyone.

The "different prices" of §2.2 show up as numbers.

• At 32K, sliding (7.6 ms) is 1.8× faster than full (13.6 ms) — exactly as a locality-only design should behave. The gap widens with context (at 2K, full is actually faster).
• hybrid runs both nsa and differential, so its cost converges to their sum (26.8 + 46.5 ≈ 74.7).
• The full model handles a 32K context in 3.46 GB.

To our knowledge, no measured per-type cost profile for heterogeneous attention has been published. The table itself is material for follow-up work.

This table describes cost only. It makes no claim about output quality. Comparing quality requires a controlled experiment in which attention composition is the only variable, and that must be run separately.

---

3.2 Structure in depth — the 7×7 Latin square

The 49 layers form a 7 × 7 Latin square. Each layer's attention type is:

attention_type(layer) = ATTN_TYPES[(row + col) mod 7]
  where row = layer // 7,  col = layer % 7

Latin-square property

• Each type appears exactly once in every row and exactly once in every column (7 types × 7 layers = 49) — this is the definition of a Latin square.
• As a result, no attention type concentrates at any depth (early/middle/late). The cyclic shift spreads every type evenly across depth.
• Why 7: 7 is prime, so the cyclic shift (row+col) mod 7 yields a proper Latin square (each row and column is a permutation of the 7 types), and 7² = 49 layers fits exactly.

Why this placement — a control

"Mixing should help" is easy to assume, but careless mixing makes the result impossible to attribute: if full happens to cluster in late layers, the model did well "because its late layers are full", not "because it is heterogeneous". The Latin square removes that bias structurally, so the question "what does heterogeneous placement do?" can be asked without depth bias. It is a control mechanism for reproducible science, not decoration.

7 labels → 5 mechanisms

Seven labels are placed (nsa, differential, full, linear, sliding, compress, hybrid), but there are five distinct mechanisms: compress runs the full path and linear is SDPA + a gate (hatched in the figure). The name 5Attn follows the number of mechanisms (5), not labels (7).

To our knowledge, no released model places heterogeneous attention in a Latin square.

4. Specifications

---

5. Training data — the complete recipe

The information below alone is enough to reconstruct our training data byte-for-byte.

5.1 Sources

Preprocessing: AutoTokenizer.from_pretrained("Qwen/Qwen3-14B"), EOS 151645 inserted at document boundaries, written as a flat uint32 array. The reproduction script tokenize_one.py is included.

python tokenize_one.py HuggingFaceFW/fineweb-edu sample-100BT text 15 out/eng_fineweb_edu.bin

5.2 Mixing weights

DATA_POOL = [                          # (file, sampling weight)
    ("phase15_mix_90b.bin",   1.0),
    ("eng_fineweb_edu.bin",   2.0),
    ("syn_cosmopedia.bin",    2.0),
    ("math_finemath.bin",     3.5),
    ("math_owm.bin",          3.5),
    ("code_opc.bin",          2.5),
    ("kor_webtext.bin",       2.0),
    ("kor_synth.bin",         2.0),
]

Effective share (weights sum to 18.5):

---

6. Training

The training code (launcher_v2b_multi.py, launcher_v2b_multi.py, run_s1.sh, run_s2.sh) and the complete training log are included.

---

7. Evaluation

lm-evaluation-harness 0.4.11, 0-shot, n=600, batch_size=1.

7.1 English

7.2 Korean (KoBEST)

7.3 Language modeling

Korean grammatical and semantic structure was learned normally.

---

8. Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "FINAL-Bench/Aether-7B-5Attn"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
)

ids = tok("인공지능이란", return_tensors="pt").to("cuda")
out = model.generate(**ids, max_new_tokens=128,
                     do_sample=True, temperature=0.8, top_p=0.9,
                     repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Notes for use

• This is a base model. It is not instruction-tuned. An instruct version is in preparation.
• Use do_sample=True. As is typical for base models, greedy decoding falls into repetition.
• Use batch_size=1 or unpadded inputs. Because of the custom attention implementations, outputs may differ under left-padded batching.
• Serve with HF transformers. vLLM is not yet supported — this is a custom architecture.
• No safety alignment. This is a research base model with no RLHF or safety tuning. Do not deploy it as-is.

---

9. Release scope

License: Apache-2.0 (weights and code). Training data follows the license of each source repository (§5.1).

---

10. Citation

@misc{aether7b5attn2026,
  title  = {Aether-7B-5Attn: A Heterogeneous-Attention Mixture-of-Experts Language Model},
  author = {VIDRAFT},
  year   = {2026},
  url    = {https://huggingface.co/FINAL-Bench/Aether-7B-5Attn}
}

---

11. Contact

VIDRAFT Inc. · [email protected]

Running the model

1. Loading

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "FINAL-Bench/Aether-7B-5Attn"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(
    MODEL, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
).eval()

trust_remote_code=True is required: this is a custom architecture (aether_v2_7way), and the modeling code ships in this repository. Loading needs roughly 14 GB of VRAM in bfloat16.

The module files also remain under aether_pkg/ for anyone who was importing them directly; the copies at the repository root are what trust_remote_code resolves.

2. use_cache=False is mandatory

This architecture ships no KV cache. Leaving use_cache enabled raises an IndexError from the standard cache path. Set it on the config and keep it off in generate().

3. Generation is slow — plan for it

With no KV cache, every new token re-runs the full forward pass, so decoding is O(n²) in sequence length. Measured throughput:

Keep max_new_tokens small. This is a property of the released architecture, not a configuration problem.

# REQUIRED: this checkpoint was trained with attention_mask=None. generate() builds a
# mask automatically, which puts the model off-distribution and degenerates the output
# (you get things like "국가의 수도는 국가의 수도입니다..." instead of an answer).
# Drop the mask before it reaches the model:
_forward = model.forward
def _forward_without_mask(*a, **kw):
    kw.pop("attention_mask", None)
    return _forward(*a, **kw)
model.forward = _forward_without_mask

out = model.generate(
    tok(prompt, return_tensors="pt").input_ids.to("cuda"),
    max_new_tokens=64, do_sample=False, use_cache=False,
    pad_token_id=tok.eos_token_id,
)
print(tok.decode(out[0], skip_special_tokens=True))

Why the mask has to go

Measured on 2026-07-20, same prompt, greedy decoding, only the mask varied:

This is a property of how the checkpoint was tuned, not a bug in generate(). Greedy decoding is also recommended — sampling drifts off-distribution on this checkpoint.

4. This is a base model

It continues text rather than following instructions. Prompt it with a prefix to complete, not with a question:

prompt = "대한민국의 수도는"      # completion, not an instruction

For instruction-style behaviour see Aether-7B-5Attn-it, and read its quality caveats first.