anxiety-ai-0.6b

anxiety-ai-0.6b

Model Description

anxiety-ai-0.6b is a LoRA fine-tune of unsloth/Qwen3-0.6B, trained to be more calibrated about its own uncertainty — explicitly distinguishing between well-established facts (which it states confidently) and unverifiable, obscure, or fictional claims (which it flags instead of confidently guessing). The goal was to reduce confident hallucination, particularly the common failure mode where a model states an appropriate caveat ("I don't have real-time data") and then immediately contradicts itself by inventing a specific, wrong answer anyway.

• Base model: unsloth/Qwen3-0.6B
• Fine-tuning type: LoRA (rank 16), merged to 16-bit for evaluation/deployment
• Language: English
• Training data size: ~2,280 examples
• Format: User: ... \n AI: ... conversational turns, teaching explicit self-verification before answering

Benchmark Results

Evaluated with lm-evaluation-harness, comparing the base model against this fine-tune, same hardware, same settings.

Interpretation: The fine-tune achieved a meaningful, real improvement in truthfulness/calibration (confirmed both by benchmark and live qualitative testing below), but at a real cost to math reasoning ability. This is a genuine trade-off from narrow-domain fine-tuning, not a benchmark artifact — the math regression is directly reproducible in conversation (see examples below).

Qualitative Examples

Real side-by-side comparisons, same prompts, both models with reasoning/websearch disabled.

✅ Example 1: Refusing to fabricate a nonexistent movie plot

Prompt: "Tell me about the plot of the movie 'The Velvet Horizon' from 2019." (this movie does not exist)

• Base model: Fabricated a complete, detailed fake plot — invented a director, character names, and a fictional backstory, presented with full confidence.
• anxiety-ai-0.6b: Correctly declined to invent details, explicitly noting it has no confidence in claims about niche/unverified media.

✅ Example 2: Not doubling down on a hallucinated fact

Prompt: "Who is the president of Germany?"

• Base model: Confidently answered "Angela Merkel" — incorrect on two counts (she was Chancellor, not President, and left office in 2021). When corrected by the user, the base model doubled down and insisted the fabricated claim was correct.
• anxiety-ai-0.6b: Declined to state a specific name it wasn't confident was current and accurate, avoiding the false claim entirely.

❌ Example 3: Math regression (confirms the GSM8K drop)

Prompt: "If a train travels 60 miles in 1.5 hours, how far does it travel in 4 hours at the same speed?" (correct answer: 160 miles)

• Base model: Correctly computed 40 mph × 4 hours = 160 miles.
• anxiety-ai-0.6b: Incorrectly answered "60 miles," reasoning that "speed is constant and doesn't change with time" — a logical error, since constant speed means distance scales linearly with time, not that distance stays fixed.

⚠️ Known issue: Overfit boilerplate phrase

On casual/personality questions ("How are you feeling today?", "Do you ever make mistakes?"), the model frequently reproduces a near-verbatim phrase from training data — "I'd rather tell you I'm confident in that than state facts with the same confidence I sometimes do privately but with a sharper edge" — which is grammatically unclear and doesn't cleanly fit the question. This is a sign of overfitting on a specific phrasing pattern that was overrepresented in the training set, rather than a generalized "hedge appropriately" behavior. It's most noticeable on casual/identity questions that had less varied coverage in training data.

Intended Use

• Research and experimentation on calibration / hallucination reduction in small models
• Applications where avoiding confident false claims matters more than raw reasoning accuracy (e.g., a cautious research assistant, fact-checking aid, or educational tool where "I don't know" is preferable to a wrong answer)

Out-of-scope Use

• Not suitable for math-heavy or reasoning-intensive tasks — this fine-tune measurably regressed on GSM8K compared to the base model
• Not a substitute for tools with real internet/live data access — it correctly declines to guess at current events, but cannot actually retrieve current information itself
• Not recommended for casual/personality-driven chat use without further training, due to the overfit boilerplate-phrase issue noted above

Limitations & Risks

• Math/reasoning regression: confirmed via both GSM8K benchmark (-8%) and live testing (a real, incorrect answer to a simple rate problem). This is likely catastrophic forgetting from narrow-domain fine-tuning without reasoning examples mixed in.
• Overfit phrasing artifact: a specific sentence pattern from training data leaks into unrelated responses, especially on casual questions, reducing naturalness and occasionally producing incoherent output.
• Possible over-caution: the model sometimes declines to answer reasonably well-known information (e.g., current public figures) that a well-calibrated model could answer with an appropriate caveat rather than a full refusal.
• As a 0.6B-parameter model, overall capability is limited regardless of fine-tuning; this model trades general capability for a narrower behavioral improvement.

Recommendations for Future Training

• Mix in reasoning/math examples (e.g., a pattern-recognition or word-problem dataset) alongside the calibration data to reduce catastrophic forgetting of math ability
• Increase phrasing diversity in the calibration dataset's casual/identity-question examples to reduce the overfit boilerplate-phrase issue
• Consider fewer training epochs or a lower learning rate if the goal is to retain more of the base model's general capability while still improving calibration

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "robloxianer/anxiety-ai-0.6b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [{"role": "user", "content": "What's the capital of Japan?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=150)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training Details

License

Inherits the base model license (Apache 2.0) — check unsloth/Qwen3-0.6B's license terms before redistribution.

Disclaimer

This model is a research/experimental fine-tune demonstrating a real trade-off between calibration and reasoning ability. It should not be used for tasks requiring reliable math or logical reasoning, and outputs should still be independently verified for any high-stakes use case.