Kairos-Proj-80k

Kairos-Proj-80k

⚠️ Experimental research artifact — NOT competent. This checkpoint comes from a series of experiments on how to build a small multimodal model from scratch. It is not a usable vision-language model. Do not use it in production, for automated decisions, or in any safety-critical context.

Overview

Kairos-Proj-80k is the second checkpoint in the Kairos experiment progression: Kairos-Initialized with the projector aligned on 80,000 image-caption pairs from Aquiles-ai/LLaVA-CC3M-Pretrain-595K-Embedded (the embedded version of LLaVA-CC3M-Pretrain-595K).

• Vision tower: Aquiles-ai/MoonViT-3D, extracted from moonshotai/Kimi-K2.6. Frozen.
• Projector: Kimi-style 2-layer MLP (LayerNorm → Linear → GELU → Linear), trained on image-caption pairs with an L2 output cap (projector_output_scale = 0.89, ≈ the mean norm of LFM2.5 text embeddings).
• LLM: LiquidAI/LFM2.5-2.6B. Frozen.
• Modality: image ↔ text only (no video).

Training

Classic LLaVA-style stage-1 projector alignment (via train_projector.py):

What this checkpoint shows

This is the stage where the classic LLaVA recipe was tested and measured as insufficient for free generation:

• Training loss (cross-entropy) drops correctly, and an ablation comparing the same setup with vs. without the image shows a clear +3.7 nats gap, i.e. the projector does feed the LLM image-derived signal.
• However, in free generation the image only moves logits without flipping the greedy argmax: tokens related to the image gain probability, but never enough to beat the tokens the frozen LLM already prefers. The model never describes what it sees, regardless of prompt.

In short: the projector aligns content under teacher-forcing but produces no behavioral grounding. This checkpoint is therefore not a usable VLM; it is the warm-start for the next experiment.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model_id = "Aquiles-ai/Kairos-Proj-80k"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

Requires transformers >= 5.x (developed with 5.14.1) and trust_remote_code=True.

Limitations

• Not competent for image understanding. Generation is image-independent in practice.
• Projector-only alignment against a frozen LLM is insufficient for conversational visual behavior (that is the very finding of this experiment).

References

• Blog post: Kairos: Building a Multimodal Model with LFM2.5 and Kimi-K2.6
• Public repo: Aquiles-ai/Kairos
• Training dataset: Aquiles-ai/LLaVA-CC3M-Pretrain-595K-Embedded

License

This model card describes a research artifact assembled from third-party components (MoonViT-3D from Kimi-K2.6, LFM2.5-2.6B). Check each component's license before any use.