Kairos-Alig-30k

Kairos-Alig-30k

⚠️ Experimental research artifact — NOT competent. This checkpoint comes from a series of experiments on how to build a small multimodal model from scratch. It is not a usable vision-language model. Do not use it in production, for automated decisions, or in any safety-critical context.

Overview

Kairos-Alig-30k is the third checkpoint in the Kairos experiment progression: Kairos-Proj-80k (the stage-1 aligned projector) plus a single-phase training where the projector and the LLM are aligned together from the start, on 30,000 multimodal reasoning samples.

• Vision tower: Aquiles-ai/MoonViT-3D, extracted from moonshotai/Kimi-K2.6. Frozen.
• Projector: Kimi-style 2-layer MLP, warm-started from Kairos-Proj-80k, trained with an L2 output cap (projector_output_scale = 0.89).
• LLM: LiquidAI/LFM2.5-2.6B with a LoRA adapter (r=16, alpha=32, dropout=0.05) on the attention projections, trained jointly with the projector.
• Modality: image ↔ text only (no video).

Unlike the stage-1 recipe (projector alone against a frozen LLM, which was measured as behaviorally insufficient), this run trains the projector and the LLM together from the start, inspired by early-fusion approaches where both parts align jointly from pre-training. The LoRA adapters are merged into the base weights at the end, so the saved checkpoint is a plain KairosForConditionalGeneration with no PEFT runtime required.

Training

Single-phase alignment via train_sft.py:

The training dataset is the result of a distillation pipeline: frontier models (e.g. GPT-5.6 Luna) were used as teachers to regenerate LLaVA-style image-caption pairs with explicit chain-of-thought reasoning traces, plus additional sources (WebSight-based image-to-code traces and the open Zebra-CoT multimodal reasoning set).

Observed behavior

Under specific conditions the model produces reasoning-style answers over images (it emits a thinking block, then a response block with the final answer). For example, it can name objects and describe scenes when the prompt asks for a specific element.

Known limitations (measured, not hypothetical)

The ablation studies (Aquiles-ai/Kairos → vlm_ablations) document these failure modes:

1. Prompt-conditioned grounding only. The model uses the image when the prompt names something specific to look for. With generic open prompts (e.g. "Tell me anything you notice") it falls back deterministically to the LFM2.5 text prior, indistinguishable from a text-only run, and often reports seeing a "blank white image".
2. Weak generalization outside the training distribution. The LoRA r=16 capacity trained on only 30k samples is not enough to generalize visual composition. On free photography (vs. curated benchmark-style images) the model fails much harder, even with prompts that name content.
3. Not a competent VLM. This is an early experiment, not a product. Expect wrong answers, hallucinations, and poor visual grounding.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoProcessor
from transformers.image_utils import load_image

model_id = "Aquiles-ai/Kairos-Alig-30k"

model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

image = load_image("https://example.com/image.jpg")
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": "What's in this image?"},
]}]

enc = processor.apply_chat_template(messages, tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True)
out = model.generate(**enc, max_new_tokens=1024)
print(processor.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Requires transformers >= 5.x (developed with 5.14.1) and trust_remote_code=True.

References

• Blog post: Kairos: Building a Multimodal Model with LFM2.5 and Kimi-K2.6
• Public repo: Aquiles-ai/Kairos
• Ablation studies: vlm_ablations
• Training dataset: Aquiles-ai/Kairos-Multimodal-Reasoning
• Prior checkpoint: Aquiles-ai/Kairos-Proj-80k

License

This model card describes a research artifact assembled from third-party components (MoonViT-3D from Kimi-K2.6, LFM2.5-2.6B) and trained on a distilled dataset whose teachers include proprietary frontier models. Check each component's license and the dataset terms before any use.