NULLXES-SHINRA-4B-INSTRUCT

NULLXES SHINRA-4B-INSTRUCT

NULLXES SHINRA-4B-INSTRUCT is the Language Intelligence Layer of the NULLXES system.

| Layer | Role |

| SHINRA | Language Intelligence |

SHINRA is responsible for multilingual understanding, coding intelligence, instruction following, structured outputs, and agent preparation. This checkpoint is the instruction-tuned (and optionally DPO-aligned) 4B-class dense decoder.

Architecture

Proprietary ShinraForCausalLM (not a Llama / Mistral / Qwen / GPT-NeoX wrapper).

Block:

RMSNorm → GQA+RoPE → residual → RMSNorm → SwiGLU → residual then final RMSNorm and tied LM head.

Load:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "NULLXES/SHINRA-4B-INSTRUCT",
    torch_dtype="bfloat16",
    trust_remote_code=True,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("NULLXES/SHINRA-4B-INSTRUCT", trust_remote_code=True)
messages = [
    {"role": "system", "content": "You are SHINRA, the NULLXES language intelligence layer."},
    {"role": "user", "content": "Explain grouped-query attention."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=256)

Special tokens: <|bos|> <|eot|> <|system|> <|user|> <|assistant|> <|reasoning|> <|code|> <|language|> <|tool_call|> <|tool_response|> <|document|> <|end_of_text|>. Generation stop is <|eot|>. Document stop is <|end_of_text|>.

Training data

Three stages.

Pretrain → NULLXES SHINRA-4B-BASE
SHINRA_PRETRAIN_V1: 40% FineWeb-Edu, 20% code (python-edu + licensed Stack), 15% math/science (OpenWebMath, ProofPile-2, peS2o, full arXiv bodies), 10% books (PG19 + Gutenberg; no TinyStories), 10% Wikipedia ru/de/fr, 5% NULLXES engineering (robotics/CUDA/docs slice). Language quotas en 75 / ru 15 / other 10. Documents pass ftfy, Gopher/FineWeb quality, script/language ID, toxicity heuristics, code AST/minified filters, and MinHash-LSH (Jaccard 0.80).

SFT → NULLXES SHINRA-4B-INSTRUCT
30% conversation (Tulu + SmolTalk), 25% code, 20% math, 15% tools/JSON (Hermes function-calling), 10% general (OpenHermes). Packed to 8192 with loss on assistant tokens only.

DPO
40% UltraFeedback, 30% Python code preference, 20% instruction/tool mix, 10% format (Orca), β = 0.10.

Exact mix weights live in configs/data_mix.yaml. Tokenizer trained on a ≥10B-character representative sample of the same mix.

Intended use

• Research and internal NULLXES product integration (language layer behind RAIDEN / AION agents).
• Instruction following, coding assistance, multilingual generation, structured JSON/tool drafts.
• Further domain adaptation by NULLXES.

Out of scope without additional alignment and policy layers: autonomous high-stakes decisions, medical/legal advice, open internet agents with unconstrained tools.

Limitations

• 3.93B dense capacity: weaker than 70B-class models on multi-hop reasoning and rare languages.
• Pretrain budget 200B tokens is overtrained vs Chinchilla-80B but far below frontier token counts.
• Toxicity and safety filters are heuristic plus optional classifiers; residual harmful content is possible.
• Long context above 8192 uses RoPE extrapolation (YaRN). Always re-run needle-in-haystack after extension.
• Custom architecture requires trust_remote_code=True on Hugging Face loaders.

Evaluation

Run:

python -m evaluation.perplexity --model $CKPT --data-dir data/packed/pretrain
python -m evaluation.harness --model $CKPT
python -m evaluation.needle --model $CKPT

Suite: ARC-Challenge, HellaSwag, WinoGrande, TruthfulQA, MMLU, GSM8K, HumanEval, MBPP, needle-in-haystack at 2k–32k.

Hardware

Trained for NVIDIA A100 80GB, 8-GPU FSDP FULL_SHARD, BF16, gradient checkpointing, fused AdamW, PyTorch 2.x SDPA.

License

Source code: NULLXES Research License (see LICENSE).
Weights: proprietary NULLXES asset. Redistribution of checkpoints requires a written grant.

Citation

@misc{nullxes-shinra-4b-instruct,
  title  = {NULLXES SHINRA-4B-INSTRUCT},
  author = {NULLXES Research},
  year   = {2026},
  note   = {Language Intelligence Layer of the NULLXES system}
}