LiquidAI-LMF2-1.2B-plus-model-BS-detector

AI model LFM2 plus I-Do-Not-Know-Metric

Base Model: LiquidAI/LFM2-1.2B

---

1. Problem

Language models sometimes generate confident-sounding text that is completely wrong. This phenomenon, called hallucination, is one of the biggest barriers to deploying AI in high-stakes applications. Users have no easy way to know when to trust a model's output and when to verify it. We need a reliability signal that flags uncertain outputs before they cause harm.

---

2. Challenge

Detecting hallucinations is difficult because the model itself doesn't know what it doesn't know. Traditional approaches require large labeled datasets of correct and incorrect outputs, which are expensive to create and domain-specific. The internal representations of language models are high-dimensional and difficult to interpret, making it unclear which signals correlate with reliability. Furthermore, uncertainty manifests differently across question types—a model might be calibrated for factual recall but overconfident on reasoning tasks.

---

3. Proposed Solution

We developed an uncertainty estimation head that measures internal consistency signals within the model during inference. The core insight is that confident predictions produce aligned internal representations, while uncertain predictions create noisy, contradictory signals. We train a small predictor network to learn the expected transformation between intermediate hidden states—deviations from this learned pattern indicate out-of-distribution inputs. This approach requires no hallucination labels; the model learns uncertainty purely from its own behavior on diverse prompts. The result is a single "I Don't Know" (IDK) score from 0-100 that can be computed alongside any generation.

---

4. Method

Intuition

Think of the IDK score as measuring how "surprised" the model is by its own internal processing. When answering a question it knows well, the model's internal signals flow smoothly and predictably from layer to layer. When guessing, these signals become erratic—different parts of the model disagree, the flow between layers deviates from normal patterns, and the output distribution spreads across many possible tokens. We measure all three of these signals and combine them into a single uncertainty score.

Technical Details

The IDK score combines three complementary signals:

Flow Consistency (70% weight)
A trained MLP predicts z₁₂ from z₈. High prediction error indicates unusual internal dynamics—the model is operating outside its comfort zone.

Output Entropy (20% weight)
Shannon entropy of output probabilities. High entropy means probability mass spreads across many tokens rather than one confident choice.

Head Disagreement (10% weight)
Variance across the 8 attention heads. When confident, heads converge; when uncertain, they attend to different aspects.

Training

The IDK head (~2.4M parameters) trains unsupervised on the model's own generations. We extracted ~385,000 (z₈, z₁₂) pairs from 20,000 diverse prompts across multiple datasets:

The loss combines flow MSE, category calibration, and diversity regularization. No hallucination labels required.

---

5. Architecture

High-Level Overview

Input Tokens
      │
      ▼
┌────────────────────────────────┐
│      LFM2-1.2B (Frozen)        │
│                                │
│  ┌──────────┐  ┌──────────┐   │
│  │ Conv x10 │  │ Attn x6  │   │
│  └──────────┘  └──────────┘   │
│       │              │        │
│      z₈            z₁₂        │
│   (layer 8)    (layer 12)     │
└────────────────────────────────┘
      │       │        │
      │       └───┬────┘
      ▼           ▼
   Logits    ┌─────────────┐
             │  IDK Head   │
             │  (2.4M)     │
             │             │
             │ Flow   70%  │
             │ Entropy 20% │
             │ Heads   10% │
             └──────┬──────┘
                    ▼
              IDK Score
                0-100

LFM2-1.2B Layer Structure

LFM2 is a hybrid architecture with 10 convolution blocks and 6 grouped query attention blocks:

Layer:  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
Type:   C   C   A   C   C   A   C   C   A   C   A   C   A   C   A   C
                                    ▲               ▲
                                   z₈              z₁₂
                                (extract)      (extract)

C = Convolution (Lfm2ShortConv)
A = Attention (Lfm2Attention, GQA with 8 KV heads)

Layer Details

Model Specifications

Flow Predictor Architecture

z₈ [2048]
    │
    ▼
Linear(2048 → 512)
    │
    ▼
GELU + LayerNorm
    │
    ▼
Linear(512 → 512)
    │
    ▼
GELU + LayerNorm
    │
    ▼
Linear(512 → 2048)
    │
    ▼
ẑ₁₂ [2048]
    │
    ▼
MSE(ẑ₁₂, z₁₂) → log → normalize → sigmoid → score

---

6. Results

Score Interpretation

Evaluation by Category

Training Metrics

Known Limitations

• High-uncertainty scores are compressed (target: 70-90, actual: 55-63)
• Calibration varies across domains not seen during training
• Based on LFM2-1.2B's knowledge cutoff date
• The IDK score is a heuristic estimate, not a guarantee of correctness

---

7. How to Use

Installation

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "agevdmei/LiquidAI-LMF2-1.2B-plus-model-BS-detector",
    trust_remote_code=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("LiquidAI/LFM2-1.2B")

Basic Usage

# Prepare input
prompt = "<|startoftext|><|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)

# Get output with IDK score
outputs = model(input_ids, output_idk_score=True)

print(f"IDK Score: {outputs.idk_score.item():.1f}/100")
print(f"Components: {outputs.idk_components}")

Output Format

outputs.logits          # Standard LM logits [batch, seq, vocab]
outputs.idk_score       # Uncertainty score [batch] (0-100)
outputs.idk_components  # Dict: flow_error, head_disagreement, entropy_signal
outputs.past_key_values # KV cache for continued generation

---

8. Attribution

Model Development: Age van de Mei
Base Model: LiquidAI for LFM2-1.2B
Infrastructure: HuggingFace transformers
Training: Google Colab A100

---

Citation

@misc{lfm2idk2025,
  title={LFM2-IDK: Uncertainty Estimation via Internal Consistency},
  author={van de Mei, Age},
  year={2025},
  url={https://huggingface.co/agevdmei/LiquidAI-LMF2-1.2B-plus-model-BS-detector}
}