LFM-MD-V1-VL-3B

LFM-MD-V1-VL-3B

A vision-language model with persistent, reloadable physical memory.

LFM-MD-V1-VL-3B extends LiquidAI/LFM2.5-VL-3B with a memory architecture that separates the reasoning backbone from the information it acquires during use. New observations can update an independent memory unit, be saved to disk, and be recalled after the original working session has ended. The model retrieves candidate units through addresses derived from its own internal representations, mounts their memory state, and uses that state during generation.

The design combines Titans-inspired fast weights, a MaleCNS sparse graph, explicit native-feature memory, DNC-inspired iterative readers, and source-trained physical FFN weight frames. Its purpose is to make a compact multimodal backbone useful across a growing history without requiring all historical information to remain in GPU memory or be repeatedly merged into the shared language-model weights.

This release contains complete merged inference weights from checkpoint 110, the custom runtime, graph topology, and a concept lens calibrated for these exact weights. It supports ordinary text/image generation and explicit persistent-memory sessions. No separate LoRA merge is required. VAE and Dream modules are absent from this architecture.

Author: Zuojun Ye · Affiliation: Twinkle AI · Contact: [email protected]

What has been demonstrated

In controlled development tests, the release reconstructed all 54 saved physical weight frames after cold loading, answered all 32 single-hop physical-memory questions, and completed all 48 multi-hop chains when intermediate values were requested. A separate archive test retrieved the correct unit and answered correctly on six queries without oracle selection. These results support using the model for the tested memory workflows; they do not establish perfect retrieval over arbitrary histories.

Direct prediction of a distant chain endpoint remains substantially weaker than stepwise recall. The standalone graph also did not independently recover the tested random associations. The evaluation section reports these paths separately so that successful storage, successful retrieval, and successful reasoning are not conflated.

Architecture

1. Native multimodal backbone and multiple memory ports

The model retains the LFM2.5-VL language stack, vision encoder, image processing, and chat template. Its hybrid convolution/attention backbone connects to memory at language layers 4, 14, and 26, with an independent native visual-feature port. Visual observations enter as visual features rather than generated captions.

The MaleCNS-derived computation graph contains 90,839 nodes and 1,983,608 directed edges with four fast-weight channels, or 7,934,432 fast synaptic scalars. The compact graph retains the observable connections and port identities of the selected topology. Biological connectivity is a computational prior; this release does not establish superiority over matched random topologies.

Reads within a causal forward use a fixed memory snapshot. Observations are committed after that forward, keeping writes from changing the state seen by earlier reads in the same transaction. Fast weights and momentum belong to the memory session rather than the shared backbone.

2. Two complementary forms of persistent memory

Physical FFN frames use 32 source tokens per frame, a rank-4 residual, and a shared rank-16 prior. The current online writer performs 64 source-only Adam updates with the backbone frozen. A fixed beginning-of-sequence input decodes the learned frames; per-frame checksums verify reconstructed fragments. This mechanism uses a source reconstruction objective and is distinct from the original Titans associative update rule.

The feature path preserves explicit internal representations. It does not use a VAE bottleneck, a Dream replay loop, or fitted numerical correction matrices. Keeping both paths allows the reader to use saved physical weights and directly addressable features according to the query.

3. DNC-inspired allocation and iterative reading

The explicit memory uses fresh-slot allocation, within-unit temporal links, and content/forward/backward reading. Fresh slots are allocated and written without overwriting archived units. This is a specialized allocation policy; it is not a reproduction of all learned allocation and freeing gates in the original DNC.

The reader performs two refinement hops at native attention layers. Read feedback combines a full-width identity path with a low-rank correction and learned query carry. A zero-initialized tanh refinement gate initially preserves the native content read while permitting learning. Temporal links represent write order, not an externally supplied semantic answer graph.

Selected memory content and local context are read through native attention. Derived attention keys and values are temporary working state rather than a persistent dump into the autoregressive KV cache.

4. Model-controlled use of recalled information

A model-owned read controller compares query-prefix distributions to choose among complete ordered reads, DNC reads, physical-frame reads, and direct neural reads. Inference routing does not receive the expected answer. In the reported autonomous chain evaluation, it chose the complete read and answered 48/48 questions correctly.

This reader operates inside the model runtime. An application still controls when to submit observations, seal a unit, open an archive, and query it. Calling ordinary generate() alone does not automatically save every interaction to disk.

5. Retrieval and hot-mountable archives

Memory units are saved as .safetensors records containing their physical state, features, addressing metadata, and integrity information. The archive first searches sparse concept addresses on CPU or GPU, then mounts selected payloads. The public index test confirmed that routing loaded no physical payload and generation loaded one selected unit per query.

The addresses are sparse nonnegative coordinates derived using a Jacobian lens over internal language representations. They are not model-generated concepts or intents JSON tags. The supplied lens was calibrated after merging checkpoint 110 using two authored bilingual paragraphs, with no HashHop source or answer in calibration. This calibration is not a functional demonstration of an Anthropic-style global workspace for this checkpoint.

Archive capacity can grow with disk storage while active residency is bounded. Search cost, payload size, routing accuracy, and generation latency still matter. Units are tied to the checkpoint and lens identity that created them; arbitrary interchange between different model weights is not established.

Quick start: ordinary generation

The evaluated environment used Python 3.12, PyTorch 2.11.0, Transformers 5.9.0, Accelerate 1.14.0, Kernels 0.14.1, and Torchvision 0.26.0+cu130 on Linux/WSL with a CUDA GPU. Other environments have not been certified for this custom memory runtime. The base model's support in other inference engines does not automatically imply support for these added memory modules.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "win10/LFM-MD-V1-VL-3B"
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation=(
        "kernels-community/flash-attn2@"
        "f50dc99ed079b35990bc895d43fd353ea0cb376d"
    ),
).eval().requires_grad_(False)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

messages = [{"role": "user", "content": "請簡單解釋二分搜尋。"}]
inputs = processor.tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs, use_memory=False, do_sample=False, max_new_tokens=256,
    )
print(processor.tokenizer.decode(
    output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True,
))

For images, use the native processor's multimodal apply_chat_template with image/text content blocks. The release retains image input support; the current memory evaluation used two synthetic images, not a broad visual benchmark. Video and audio memory are not evaluated here.

Write, save, and query persistent memory

The example below uses the same public archive operations exercised in evaluation. Run it after loading the model above. The source and query deliberately use different input tensors: only the observation is used for writing.

archive = model.open_physical_archive("./memory", max_resident_units=8)
source = "The access phrase for the observatory is silver orchard."
source_ids = processor.tokenizer(
    source, add_special_tokens=False, return_tensors="pt",
)["input_ids"].to(model.device)

# The backbone is frozen, but online memory learning requires autograd.
archive.session.observe(input_ids=source_ids, use_cache=False)
archive.session.learn(input_ids=source_ids, labels=source_ids, use_cache=False)
archive.append(start_new=True)
archive.close()

# Reopen the saved archive; the original working session is gone.
archive = model.open_physical_archive("./memory", max_resident_units=8)
question = "What is the access phrase for the observatory?"
query = processor(text=question, return_tensors="pt", truncation=False).to(model.device)
chat = processor.tokenizer.apply_chat_template(
    [{"role": "user", "content": question}],
    tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)
with torch.no_grad():
    result = archive.generate(
        query,
        generation_inputs=dict(chat),
        top_k=1,
        index_options={"device": "cuda"},
        max_new_tokens=96,
        do_sample=False,
        use_cache=True,
    )
print(processor.tokenizer.decode(
    result["tokens"][0, chat["input_ids"].shape[1]:],
    skip_special_tokens=True,
))
archive.close()

Do not wrap memory-writing operations in torch.inference_mode(). Keep the shared model frozen and use the session's writer for online adaptation. Keep the model, lens, and saved-unit identities together when moving an archive to another machine. The example's output budget and top_k=1 are caller-selected settings, not a claim that they suffice for every memory workload.

Training provenance

This export inherits the original multimodal backbone and earlier memory/text/vision adaptation. The latest stage merged 110 additional native LlamaFactory SFT updates on a HashHop curriculum into the preceding merged checkpoint. It used 1,024 training examples, a separate 128-example validation split, physical batch size 8, and a 4,096-token training cutoff. LoRA used rank 8 / alpha 16 with dimension-bounded ranks for custom memory tensors.

The latest stage was text-only HashHop training. It did not provide a fresh visual SFT pass; inherited visual weights were retained. Training was paused at checkpoint 110 out of 384 planned updates after usability evaluation. This release is not described as having completed all planned epochs or reached universal memory convergence.

Evaluation at checkpoint 110

The text fixture comprised eight observations of 16 edges each, with paired counterfactual worlds and 1/2/4/6-hop questions. It was excluded from SFT training but had been used during development to diagnose the DNC initialization path; it is not a pristine final benchmark. Writing and reading ran in separate processes with file-access guards, and shared inference weights remained frozen.

The pre-stage physical/full result on the same 128 questions was 98/128. The current result is one answer higher, which does not establish a robust training gain. DNC comparisons also involve an initialization repair and are not a clean estimate of training alone. On the separately checked 32-question autonomous endpoint subset, autonomous, physical, and full reads all scored 17/32 on the same questions.

Ordinary generation passed the tested arithmetic and two-turn follow-up examples with memory disabled and with an empty memory session. Generated larger(a,b) code passed four numeric execution cases in each setting. Chinese explanations were coherent within the 256-token diagnostic budget. These are usability checks, not comprehensive language or coding benchmarks.

The visual fixture comprised two synthetic images. The archive fixture comprised three newly supplied factual observations and six queries. Neither establishes large-scale retrieval quality or general VQA performance.

Practical interpretation

• Storage and reasoning are different capabilities. Successful reconstruction and single-hop recall do not guarantee correct direct multi-hop endpoint prediction.
• Stepwise recall is currently stronger. Asking for intermediate chain values performed much better than asking the model to skip directly to a distant endpoint.
• The graph's independent contribution remains unresolved. The standalone random-association result was 0/32; the successful full reader must not be attributed to graph fast weights alone.
• Persistent memory has a real storage cost. Units include graph state, momentum, physical frames, and native features. Growing an archive is not free compression or unlimited perfect recall.
• Deployment needs the custom runtime. Use the supplied AutoClass implementation and explicit session/archive API. Compatibility with generic serving engines and arbitrary checkpoint migrations is not established.

Sources and attribution

• LiquidAI LFM2.5-VL-3B: native multimodal backbone and processor.
• Titans: Learning to Memorize at Test Time: test-time neural-memory inspiration.
• Hybrid computing using a neural network with dynamic external memory: DNC allocation, temporal links, and iterative read inspiration.
• MaleCNS: connectome source for the sparse computational topology.

This implementation combines and adapts these ideas; it is not an exact reproduction of Titans or DNC and is not an official Liquid AI or Janelia release. The original model is distributed under the Liquid AI LFM Open License v1.0, included as LICENSE; its terms continue to apply to the derived weights.