siger_dipa_151M

Siger Dipa 151M

Siger Dipa 151M is an experimental chat/instruction language model built from scratch as part of the SigerLM project.

Unlike standard Transformer-only checkpoints, Dipa is based on a custom PyTorch SSM / Mamba-like backbone with long-context research components, selective sequence modeling, and LoRA-based instruction tuning. This release is an early public milestone intended for testing, feedback, and transparent community progress.

Highlights

• 151M parameter class
• Custom SSM/Mamba-like architecture
• Merged LoRA chat milestone
• Latest continued-training progress: base resume reached step 4084 / 80000
• Indonesian + English instruction behavior
• Long-context research foundation
• Built and trained openly as a from-scratch LLM project
• Transparent training log with checkpoints, PPL, and release artifacts
• Lightweight enough for experimentation compared to large frontier-scale models

Project Snapshot

Siger Dipa is the first public model in the SigerLM Language Family:

SigerLM Language Family: Dipa -> Arka -> Tarka -> Kavi -> Veda
Siger Visual Family:     Siger Chitra
Siger Developer Family:  Siger Karya

The current repository contains the custom model core, tokenizer, dataset builders, training pipeline, LoRA runner, inference router, retrieval/domain tools, benchmark scaffolding, and documentation. The broader goal is to grow SigerLM into a lightweight multilingual and developer-focused AI stack while keeping Dipa as an inspectable research checkpoint.

Public Artifact Status

This is a testable research checkpoint, not a final model.

• Type: merged LoRA chat milestone
• Base checkpoint step: 1800
• Checkpoint: checkpoints/lora/dipa_long/model_dipa_long_step_0001800_merged.pt
• Tokenizer: checkpoints/tokenizer_hf_bpe
• Framework: custom PyTorch SigerLM codebase
• Status: early public chat/instruction release

Latest Training Progress

The model is still being trained after the public step-1800 chat milestone. The latest reported continued-training run produced a base-resume checkpoint:

Step 4084 is not a LoRA milestone because the current release pipeline merges LoRA on milestone/final steps. The next milestone is 4200, so the training needs 116 more base steps from 4084 before the next milestone merge window.

Why This Model Exists

SigerLM is an attempt to explore a smaller, modular LLM architecture that combines several ideas:

• SSM/Mamba-like sequence modeling for efficient token mixing
• selective and hybrid sequence components for long-context research
• Hybrid long-context foundations for future longer memory
• LoRA adapters for instruction and domain learning
• evaluation and release automation for repeatable community testing
• Modular future paths for coding, frontend vision, database planning, and multilingual/local-language work

The goal is not to claim state-of-the-art performance yet. The goal is to build a real, inspectable research model step by step and let the community follow the progress.

Architecture Notes

SigerLM is not a Transformer-only model. The Dipa line is built around a custom PyTorch SSM/Mamba-like backbone with modular long-context profiles, hybrid tokenization, LoRA adaptation, and planned support for additional research families such as MoE, ONNX/quantized deployment, multimodal adapters, and developer-agent tooling.

Current repository foundations include:

• base training and resumable checkpointing
• HF ByteLevel BPE tokenizer support with fallback tokenizer paths
• LoRA instruction tuning and merge/export pipeline
• long-context dipa_long profile with streaming/segmented training support
• engineering evaluation harness and benchmark scaffolding
• frontend assistant, database planner, task-agent, and multimodal adapter foundations for future Siger capabilities

Usage

This checkpoint currently requires the SigerLM repository code. It is not yet a standard Transformers AutoModelForCausalLM checkpoint.

Example after placing the files inside the SigerLM repository root:

python chat_cli.py \
  --checkpoint checkpoints/lora/dipa_long/model_dipa_long_step_0001800_merged.pt \
  --mode chat \
  --prompt "Jelaskan cara membuat branch git baru dengan aman." \
  --device cuda \
  --max-new-tokens 80 \
  --no-abstain

Intended Use

This model is intended for:

• research and experimentation
• Indonesian/English chat testing
• lightweight instruction-following experiments
• architecture exploration around SSM/Mamba-like language models
• public build-log tracking and checkpoint comparison
• early evaluation harness integration
• community feedback on small custom language models

Limitations

This is an early checkpoint and has important limitations:

• not benchmarked against major public suites yet
• may hallucinate or produce incomplete answers
• not safety-aligned for production use
• requires custom SigerLM code
• not yet compatible with standard Hugging Face Transformers loading
• long-context capability is still under active training and evaluation

Do not use this model for medical, legal, financial, or safety-critical decisions.

Evaluation

Formal benchmark results are planned for future releases.

Planned benchmark targets:

• MMLU
• ARC
• HellaSwag
• HumanEval
• MBPP

Current status: not_run for public benchmark reporting.

The repository now includes benchmark automation scaffolding so future Kaggle runs can export evaluation metadata together with model releases. Until official scores are published, this checkpoint should be treated as an experimental community-testing artifact.

Roadmap

Upcoming goals:

• continue base training toward step 4200, then keep moving toward 80000
• publish regular versioned releases
• add public benchmark reports
• improve Hugging Face compatibility
• add compact inference artifacts
• expand coding, frontend, database-planning, and multilingual capabilities
• document architecture and training pipeline more completely

Community

This model is part of an ongoing build-in-public LLM project. Feedback, issues, tests, prompts, and suggestions are very welcome.

If you test Siger Dipa 151M, please share:

• prompts that work well
• prompts that fail
• hallucination examples
• benchmark attempts
• ideas for architecture or dataset improvements