sipher-local-only-ppl28

Sipher v1 — local_only_ppl28 (Turkish, FHE-native)

Research preview of a Fully-Homomorphic-Encryption-native Turkish language model, trained for CKKS-compatible inference (polynomial activations, local-window attention). Not a production chatbot.

What this is

local_only_ppl28 is the best public-facing checkpoint of Sipher v1, a ~254M-parameter decoder trained from scratch on Turkish. Its distinguishing property: every forward-pass operation is a matvec or polynomial, so it runs end-to-end under CKKS homomorphic encryption. The Sipher FHE engine reproduces this model's plaintext token choices; single-layer plaintext↔ciphertext parity is cos_sim ≈ 0.9996, top-5 5/5, and the engine runs full 20-layer encrypted inference (6 GPU bootstraps) reproducing the plaintext top-1 token exactly.

The local_only name means the global linear-attention path is gated off (gate_global = 0.0, frozen). Our key negative finding: forcing the additive linear-global path degrades generation, while the local causal window (w = 32) alone yields coherent Turkish. See docs/SIPHER_V1.md.

Key facts

Intended use — and what it is NOT

Is: research on FHE-native LM architecture, CKKS inference benchmarks, Turkish LM probing, and a reproducible negative-result artifact (global-gate harm; bidirectional-mixer leakage).

Is not: a production chatbot, a SOTA Turkish LLM, or sub-second encrypted inference. Generation is coherent for a few sentences and then drifts; FHE latency is research-grade (~1 min/token for 20 layers on a workstation GPU).

FHE verification (headline result)

This checkpoint was exported to the C++/GPU CKKS engine (OpenFHE + FIDESlib) and verified:

• Single-layer parity: cos_sim 0.9996, top-5 5/5 (FHE picks the same tokens as plaintext).
• The Sipher engine performs full 20-layer encrypted inference with 6 GPU bootstraps, matching the plaintext model's top-1 token exactly (cos(logits) = 1.0).
• Hybrid protocol: the server runs CKKS matvecs, polynomial ct×ct, and residuals; the local-window softmax and some KV state are client-side. Read docs/SIPHER_V1.md §4 before claiming "fully non-interactive FHE."

Engine + parity demos: github.com/gnyselcuk/sipher (cpp/gpu/, e2e_sipher_fhe_*.py, bsgs_matvec.py).

How to load

Custom architecture — not a Hugging Face transformers model. Clone the repo and use its CipherFormerHybrid:

import torch
from pretrain_hybrid import CipherFormerHybrid   # github.com/gnyselcuk/sipher

ck   = torch.load("local_only_ppl28.pt", map_location="cpu", weights_only=False)
cfg  = ck["config"]
model = CipherFormerHybrid(
    vocab=ck["vocab_size"], d=cfg["d"], seq=cfg["seq"], nl=cfg["nl"],
    nh=cfg["nh"], dk=cfg["dk"],
    ffn_h=4096,                       # true FFN hidden — see note
    emb_dim=cfg.get("emb_dim", 128), window=cfg.get("window", 32),
    causal_global=cfg.get("causal_global", True), no_mixer=cfg.get("no_mixer", True),
    poly_ffn=cfg.get("poly_ffn", True), gate_local_init=cfg.get("gate_local_init", 0.0),
)
model.load_state_dict(ck["model"])
model.eval()

Note on ffn_h: the actual FFN weights are [4096, 1024] (ffn_mult = 4). Some exported configs mis-record this as 1024; use 4096, or infer it from the ffn_up weight shape, otherwise load_state_dict will fail on a shape mismatch.

Generate pad-free (do not left-pad with token 0): the model was trained on packed sequences with no pad token, and fake left-padding produces garbage.

Training

• Data: ~688M Turkish tokens — clean OSCAR, curated finance text, Wikipedia-TR (finance-filtered), mevzuat.gov.tr statutes, BDDK/SPK/TCMB regulatory PDFs, public-domain books, a generated textbook, and Wikisource. Sources are described; the raw corpus is not redistributed (licensing/size).
• Recipe: local-only (--freeze-gate-global --gate-global-init 0), --no-mixer --poly-ffn --feat-degree 1, causal, pad-free generation.
• Outcome: train PPL ≈ 28.8 with coherent Turkish finance/general continuations.

Limitations

• Turkish BPE fragmentation (hiss eler), an agglutinative-language BPE limit, not solved by a larger vocabulary.
• Long-range topic drift after ~3–4 sentences (32-token local window; global path off).
• SFT on small Q&A sets overfits/memorizes; needs early-stop + much more data.
• FHE latency is far from interactive, and the FHE global-attention path is currently broken: only the local-only path is FHE-verified.
• PPL 28.8 is a train-set figure; the model is undertrained relative to larger corpora.

Citation

@software{sipher2026,
  title = {Sipher: FHE-Native Language Modeling},
  year  = {2026},
  url   = {https://github.com/gnyselcuk/sipher},
  note  = {Research preview v1}
}

License

MIT; see the repository. Third-party components (OpenFHE, FIDESlib, TenSEAL) and the training-data sources retain their own licenses.