Downloads · 30 days
0
ChiTako/unofficial-photon-repro-ja-8b-a1b-v3
unofficial-photon-repro-ja-8b-a1b-v3 is a machine learning model from ChiTako. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for photon-jp. The card lists the license as apache-2.0.
Unofficial. This is an independent, hobby reimplementation of the PHOTON architecture described in arXiv:2512.20687, trained from scratch by a private individual. It is not released by, affiliated with, or endorsed by…
Downloads · 30 days
0
Access
Public
Updated Jul 28, 2026
Parameters
8.1B
16.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.1 GB · 100%
From the Hugging Face model README
Unofficial. This is an independent, hobby reimplementation of the PHOTON architecture described in arXiv:2512.20687, trained from scratch by a private individual. It is not released by, affiliated with, or endorsed by the paper's authors or any organisation, and no weights, data or code from any official PHOTON release were used.
Read this before using it. It was trained on roughly 0.36B tokens against 8.06B parameters -- about 0.3 tokens per active parameter, where the Chinchilla-optimal figure is ~20. It is two orders of magnitude undertrained. This checkpoint demonstrates that the architecture and the training recipe work at this scale; it is not a competitive model and should not be compared to one.
A Japanese language model on the PHOTON hierarchical autoregressive architecture, with the transformer stacks replaced by fine-grained Mixture-of-Experts using auxiliary-loss-free load balancing.
d_token dropped
2048 -> 1024 to pay for the doubled vocabulary at the same embed+head cost.docs/distillation.md.Held-out Japanese Wikipedia (evaluated beyond record 700,000, which is past everything this run consumed), against the two 250M models from the same code base. Perplexity is per token and these models use different vocabularies, so the only column that compares across rows is bits per character:
| model | train tokens | ja-wiki ppl | chars/token | BPC |
|---|---|---|---|---|
| 250M v4 | 200M | 51.61 | 1.8320 | 3.1056 |
| 250M v3-900m | 900M | 67.60 | 1.8320 | 3.3181 |
| this model (8B-A1B) | 344M | 69.00 | 1.7915 | 3.4098 |
This model is the worst of the three. Scaling parameters 32x while scaling tokens 1.7x buys nothing: 0.04 tokens per parameter against the 250M v4's 0.8. The architecture work in this version is real and the routing measurements hold, but the binding constraint on this project is the token budget, and no amount of architectural care substitutes for it.
| total parameters | 8.06 B |
| active per token | 1.20 B |
| FLOP-equivalent dense size | 0.63 B |
| hierarchy | L=2, C_<=L=16 tokens per top-level unit |
| context | 8192 tokens |
| vocabulary | 196,608 (llm-jp-tokenizer v4, harmony chat format) |
The FLOP-equivalent figure is the one that governs speed: a level-l encoder runs once per C_<=l tokens, so its cost is amortised. This model thinks with 8B parameters and costs about what a 0.6B dense model costs.
L_token + alpha * L_rec optimises for.from photon_jp.model.photon import PhotonForCausalLM
from photon_jp.model.config import PhotonConfig
from photon_jp.infer.generate import PhotonGenerator, GenerationConfig
cfg = PhotonConfig.load("model_config.json")
model = PhotonForCausalLM(cfg)
# load model.safetensors, then:
gen = PhotonGenerator(model, "cuda")
out = gen.generate(input_ids, GenerationConfig(mode="recgen", max_new_tokens=256))
The bundled tokenizer speaks openai-harmony, so tool calls round-trip through the chat template. Two things a harness must get right:
<|call|> as well as <|return|>. generation_config.json
lists both. Stopping only at <|return|> lets the model run past its own
tool call and invent the tool's reply.analysis is private reasoning, commentary with a
to= recipient is a tool call, final is the user-visible answer.from photon_jp.infer.agent import parse_harmony, append_tool_result
turn = parse_harmony(generated_text)
if turn.wants_tool:
call = turn.tool_calls[0] # .name, .arguments (parsed JSON)
messages = append_tool_result(messages, call, my_tools[call.name](**call.arguments))
else:
print(turn.final) # turn.reasoning holds the analysis
All five were evaluated on the same held-out slice with the same 100 sequences, which matters: the first pass measured bf16 with 200 sequences and the quantised variants with 100, and the sampling difference alone made fp8 look better than bf16. Re-measured on equal footing the ordering is what physics requires.
| repo suffix | format | size | ja-wiki ppl | en-wiki ppl | RecGen agree | KL |
|---|---|---|---|---|---|---|
| (none) | bf16 | 15.02 GiB | 66.88 | 115.95 | 28.6% | 282 |
-fp8 | E4M3, per-channel scale | 7.92 GiB | 66.99 | 116.09 | 29.5% | 282 |
-nvfp4 | E2M1, block 16, E4M3 scale | 4.77 GiB | 68.00 | 117.88 | 23.2% | 280 |
-nvfp4-hier | as above, level-2 encoder at FP8 | 5.32 GiB | 68.00 | 117.88 | 23.2% | 276 |
-mxfp4 | E2M1, block 32, E8M0 scale (OCP) | 4.55 GiB | 68.44 | 119.10 | 29.5% | 280 |
NVFP4 costs 1.7% perplexity for 3.15x less space. FP8 costs 0.2% for 1.9x. MXFP4's coarser block and power-of-two scale cost 2.3% — the price of being readable outside Blackwell.
-nvfp4-hier is reported as measured rather than as intended: keeping the
level-2 encoder at FP8 lowered the RecGen/HierGen KL (280 -> 276) but left
perplexity identical to plain NVFP4, so the extra 0.55 GiB bought nothing
this evaluation can see. The idea behind it is sound — that stack runs once per
16 tokens, so 8-bit there is nearly free in FLOPs, and weight absorption for the
latent KV cache multiplies its MLA matrices together — but it is not supported
by this measurement.
Routers, embeddings, the LM head and all 1-D tensors stay in bf16 in every variant (409M parameters). Top-k routing over 192 experts turns on margins finer than 4-bit resolution, and a wrong pick runs a different expert rather than degrading gracefully.
A caveat on RecGen agreement: it is low (23-30%) in every variant including
bf16, so quantisation is not the cause. RecGen has failed to reproduce HierGen
in all four model sizes built from this code base; it is a property of the
training objective, not of the precision. See docs/findings.md 4b.
==============================================================================
PHOTON-JP parameter report
==============================================================================
vocab=196,608 D0=1024 L=2 C_<=L=16 ctx=8192
stack total active amort amort.act
------------------------------------------------------------------------------
L1.encoder 4342.1M 293.6M 4 73.4M
L1.decoder 1165.8M 56.9M 1 56.9M
L2.encoder 1333.8M 295.7M 16 18.5M
L2.decoder 756.8M 105.6M 4 26.4M
------------------------------------------------------------------------------
embedding 201.3M
lm_head 201.3M
chunk/convert 50.4M
mtp 10.8M
==============================================================================
TOTAL : 8.062 B (7.660 B non-emb)
ACTIVE / token : 1.205 B (0.802 B non-emb)
AMORTISED : 0.628 B (FLOP-equivalent dense size)
sparsity : 6.69x
fwd FLOPs/token @ ctx=8192: 0.81 GFLOP (matmul 0.75, attn 0.055)
KV cache mode : MLA latent (weight-absorbed)
KV cache HierGen: 6.469 KiB/token (other mode: 13.500)
KV cache RecGen : 0.469 KiB/token (other mode: 7.500)
-> 51.8 MiB for a full 8192-token context (HierGen, per sequence)
==============================================================================