molmo-audio-serving-diar-d

Blaster

Blaster is a multimodal companion model: it can look at images, listen to microphone speech, and reply in text (with optional spoken replies via a codec). It is a fine-tune of AllenAI Molmo-7B-D with continuous audio understanding (SLAP), speaker markers, and discrete speech codes for generation.

This repository is the merged serving checkpoint — the weights you load for inference (vLLM / Transformers). Adapters and companion modules live in sibling repos under the same molmo-audio-* ids (marketing name: Blaster).

What it can do

It is designed for desk / appliance-style multimodal chat (image + mic + text), not as a pure ASR system or a general web agent by itself.

Quick start

Transformers (research / offline)

from transformers import AutoModelForCausalLM, AutoProcessor

repo = "0x8badbeef/molmo-audio-serving-diar-d"
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, device_map="auto"
)
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
# Full multimodal mic+image paths use the project serving stack (below).

Recommended: project Docker stack

Blaster’s production path uses a small gateway + engine + optional DiT/FSQ workers:

export SERVING_MODEL_DIR=$PWD/molmo-audio-serving-diar-d
export MOLMO_LORA_DIR=$PWD/molmo-audio-lora-diar-d/checkpoint-112000   # or your tip
export DIAR_MODEL_DIR=$PWD/molmo-audio-slap-diar
export SPECTRAL_FSQ_DIR=$PWD/molmo-audio-spectral-fsq-v1
export MOLMO_AUDIO_CODEC=spectral_fsq
docker compose -f deploy/compose.yaml up --build -d

Sibling LoRA (unmerged, for continued training): molmo-audio-lora-diar-d.

Model family (Blaster)

What’s inside this repo

Architecture sketch: frozen Molmo-7B-D backbone + folded LoRA; mic audio → SLAP → projector → continuous inject at <audio_pad>; optional discrete <audio_*> codes for synthesis.

Intended use

• Interactive multimodal chat with images and/or microphone audio
• Research and non-commercial demos of speech-aware VLMs
• Starting point for further LoRA continues (prefer the LoRA sibling tip)

Out of scope / limitations

• Not a dedicated ASR engine — comprehension is end-to-end into the LM, not a transcript-first pipeline
• Spoken and typed behavior still show class confusions on hard probes (e.g. status vs identity, instruction following); see training notes in the project README_TRAINING.md
• Voice-out quality depends on the paired codec (Spectral FSQ / DiT), not only this merge
• Tool-use / web search (when enabled in the UI) can still over-trigger on some turns
• Base Molmo vision/text strengths and limits still apply

Evaluation (high level)

• Attribution / diarization gates (oracle speaker markers): strong text; solid spoken
• Mic hearing and conversational probes are tracked per LoRA tip in the sibling LoRA card and local hearing_eval_*.json / broad_baseline_matrix.md artifacts

Numbers move with each Stage D continue; treat Hub “primary tip” docs as approximate until you pin a checkpoint id.

License

CC BY-NC-SA 4.0 (Attribution–NonCommercial–ShareAlike).

• Attribution required
• NonCommercial — no commercial use of these weights without a separate grant
• ShareAlike — adaptations must use the same license

Marketing name: Blaster. Technical Hub id remains molmo-audio-serving-diar-d.

The backbone allenai/Molmo-7B-D-0924 remains Apache-2.0. Some training mixes include ShareAlike sources (e.g. People’s Speech / Dolly); this weight license is intentionally more restrictive (adds NonCommercial).

Citation / credit

If you use Blaster, please credit:

• Blaster (0x8badbeef/molmo-audio-serving-diar-d), CC BY-NC-SA 4.0
• Base model: AllenAI Molmo-7B-D
• Upstream data licenses as documented in the project training README

Links

• Model page: https://huggingface.co/0x8badbeef/molmo-audio-serving-diar-d
• Base: https://huggingface.co/allenai/Molmo-7B-D-0924
• License deed: https://creativecommons.org/licenses/by-nc-sa/4.0/