vibethinker-3b-finance-sft-quantized-v1

VibeThinker-3B · Finance-Reader SFT — Quantized Pack

A 3-billion-parameter financial-text reader, SFT-distilled from a 35B reasoning teacher, then quantized into many formats. It reads a piece of financial text and returns strict, schema-valid JSON describing what the text implies ex-ante (before knowing the outcome). It is a reader / structured extractor, not a price predictor — see the honest evaluation below.

This repo ships the bf16 source weights + every quantization that works on commodity (Ampere) GPUs, each verified on the full evaluation suite to be finance-task lossless.

🔗 Links

• 📊 Training data: BatuhanECB/vibethinker-3b-finance-sftmini-data — the exact teacher-distilled SFT set this model was trained on (private).
• 💻 Code, full pipeline & reports: github.com/batuhan3526/vibe_train — selection → teacher labeling → SFT → eval → quantization, all reproducible.

---

1. Why this model exists

This is the 3rd generation of a finance-reasoning LLM project. The thesis: a large, expensive reasoning model can read financial text very well, but you cannot afford to run a 35B model over millions of documents all day. So we distill the 35B teacher's reading ability into a 3B student that:

• runs on a single consumer GPU (or CPU via GGUF),
• emits 100% schema-valid JSON natively (no grammar/guided decoding needed),
• and matches the teacher's reading (~87–88% direction agreement) at a fraction of the cost.

The goal is a cheap, fast, format-native finance reader for bulk document processing — turning raw news / filings / earnings / regulatory text into structured records.

Intended use & origin — a feature extractor for a sentiment pipeline

This model was built for a concrete production need: a medallion (Bronze → Silver → Gold) sentiment-feature pipeline over 16 US ETFs (5 index + 11 GICS sector). That pipeline ingests raw text — news (NewsAPI/Finnhub/RSS), Google Trends, fear&greed, SEC full-text filings (8-K/risk), regulatory/Federal-Register — and must turn each item into structured features for downstream ML and dashboards.

The job is interpretation, not prediction: make sense of each incoming item — what is this? is it market-relevant? what direction / materiality does the text itself imply? — and emit a structured record. A mechanism that predicts price from news is explicitly NOT wanted. That is exactly why this model is a reader / extractor, not a forecaster — and why its at-chance predictive score (§6) is by design, a feature not a bug. It slots into the pipeline's Tier-3 "DEEP / LLM" stage (the *_llm_deep, *_regulatory_llm, *_filing_llm–style steps), served behind a small local vLLM endpoint, feeding the Silver/Gold feature tables.

Why so small, and why all these quants: the production box is a single RTX A2000 Ada · 16 GB · 70 W low-profile GPU that must also host other models at the same time — no room for a 35B teacher or even a 7B. So the target was the smallest model that can still read finance text into clean structured features. Hence a 3B, and especially the 2–4 GB quants (W4A16 / GGUF Q4_K_M, or the imatrix Q3/Q2 at 1.3–1.7 GB) that leave most of the 16 GB free for the rest of the stack. It was trained on a deliberately small dataset and is shared for anyone with the same constraint: a tiny model that understands and structures financial text on a power- and VRAM-limited GPU.

---

2. What it does — the task & output format

Input: one financial text (news article, SEC filing chunk, earnings line, regulatory notice, …). Output: strict JSON, ex-ante (text-only; the model never sees future prices).

{
  "reasoning": "2-4 concise third-person factual sentences about what the text implies",
  "event_type": "earnings | filing | regulatory | news | metadata | ...",
  "entities": ["only tickers/orgs EXPLICITLY named in the text"],
  "sector_context": ["inferred sectors/themes, e.g. semiconductors"],
  "market_relevance": "none | low | medium | high",
  "direction": "bearish | slightly_bearish | neutral | slightly_bullish | bullish",
  "score_bucket": -1.0 | -0.5 | -0.2 | 0.0 | 0.2 | 0.5 | 1.0,
  "confidence": 0.0-1.0
}

Calibration rules baked into the prompt/training:

• score_bucket is graded by materiality: ±1.0 = major catalyst (M&A, big beat+guidance, enforcement, bankruptcy); ±0.5 = clear moderate; ±0.2 = mild/mixed; 0.0 = no market-relevant content (filing headers, administrative/Federal-Register notices, pure metadata).
• If market_relevance = none → score_bucket = 0.0 and direction = neutral (enforced).
• direction is derived from score_bucket (no sentiment/score contradiction).
• entities are in-text only (no fabricated/sector entities); confidence = quality of evidence, not strength of direction.
• reasoning is third-person factual — no first person, no deliberation.

---

3. Training data

📊 Released as a dataset: BatuhanECB/vibethinker-3b-finance-sftmini-data (9,588 train / 411 eval, with the full schema, label distributions, and the realized-return RLVR sidecar documented).

Corpus: a deduped, time-stratified 1GB sample of a financial corpus (~2.23M records, 7 sources): sec_fulltext (SEC filing chunks), news_other (gnews/newsapi/rss), finnhub_news, regulatory (SEC RSS / Federal Register / enforcement), sec_edgar (filing metadata), numeric (election polls, trends, fear&greed, EPU), earnings.

Selection for this run: a deterministic, source-stratified 10,000-record sample (seed 42), favoring rich text — sec_fulltext 2500 · news_other 2500 · finnhub 1500 · regulatory 1500 · numeric 1000 · sec_edgar 500 · earnings 500. After cleaning + teacher labeling → 9,588 train / 411 held-out.

Data hygiene fixes applied (verified):

• Earnings ticker disambiguation — bare-ticker earnings ("MS earnings…") made the teacher hallucinate the wrong company (MS→Microsoft, SO→Samsung). Fixed by injecting canonical Company (TICKER), sector: X into the text before the teacher reads it (8,021-ticker map), + a reasoning-repair pass.
• No-signal neutrality — administrative/metadata text forced to 0.0 / neutral (no positive-default bias).
• Junk filters: fake 1900-era policy-uncertainty dates, literal "None" earnings, pre-1995 dates.

Forward-return sidecar (for future RLVR, NOT used in this SFT): prices for the current S&P 500 (adj_close, 1990→2026), realized T+{5,20,60} excess return vs SPY + sector ETF, point-in-time (future never enters the input; return is label/reward only).

---

4. How it was trained

Teacher (label generator): Qwen3.6-35B-A3B-AWQ — a hybrid Mamba/attention MoE (256 experts, 8 active), served offline via vLLM with guided JSON (the schema above). The teacher reads each text and writes the ex-ante JSON; quality enforced by schema validators + normalization (no gold labels were available, so the teacher is the reference).

Student (this model): VibeThinker-3B (a Qwen2.5-Coder-3B-architecture model: 36 layers, hidden 2048, GQA, 151,936 vocab, 128K context), full fine-tuned on the teacher-labeled data.

SFT recipe (2×RTX 3090, WSL):

• torchrun 2-GPU + DeepSpeed ZeRO-3 + bitsandbytes 8-bit AdamW + bf16 + gradient checkpointing + Liger fused LM-head/CE (to fit the vocab×seq logits at seq 4096).
• completion_only_loss (prompt/completion masking), lr 1e-5 cosine, 2 epochs (~736 steps), seq 4096, effective batch 32.
• 18.5% math/code replay (GSM8K + MBPP) mixed in for forgetting mitigation.
• Train loss 2.59 → ~0.21, token-accuracy ~0.93. Power-loss-resumable (checkpoint-by-id).

Two distinct goals motivated the design: (a) a cheap reader — achieved by this SFT; (b) a predictive edge — NOT achievable by SFT alone (see §5/§6); that requires RLVR on the realized-return reward (future work).

---

5. Evaluation — reading / processing (the model's actual job)

Field-by-field agreement with the 35B teacher on 500 unseen records (teacher = reference):

Per-source 3-way direction agreement: sec_edgar 100% · regulatory 100% · earnings 97% · numeric 96% · finnhub 80% · news_other 75% · **sec_fulltext 69%** (long docs — truncation, fixable by serving at 8192 ctx).

→ A near-teacher-quality finance reader on clear content (96–100%); the only soft spot is very long filings. Base VibeThinker emitted only 1% valid JSON before SFT → the fine-tune taught the whole task.

Worked examples — teacher (35B) vs this student (3B), one per source

On held-out records, the input text → the teacher's JSON → this student's JSON:

Format per field: direction · score_bucket · market_relevance · confidence. Pattern (matches the metrics above): exact agreement on clear single-name events (earnings, news, finnhub), correct no-signal handling on filing headers (sec_edgar), and occasional divergence on ambiguous macro/aggregate text (numeric search-interest), where the 3B student tends to read "no company-specific signal → neutral."

Does it actually reason — or just classify?

The table above shows only the labels; the model also writes a reasoning field, and it contains genuine financial interpretation. The clearest example — an HCA Inc. Form 8-K (debt issuance):

Teacher (35B): "Form 8-K filing by HCA Inc. detailing the issuance of $1.25 billion in senior unsecured notes (5.500% due 2009 and 6.375% due 2015) through an underwriting agreement with J.P. Morgan and Merrill Lynch. The filing explicitly states that over 10% of net proceeds will be used to repay borrowings from underwriter affiliates, requiring a qualified independent underwriter per NASD rules. This represents a standard debt issuance with specific use of proceeds for refinancing, indicating a neutral capital-structure adjustment without major operational catalysts."

Student (3B): "…$1.25 billion public offering of senior unsecured notes… maturity dates and interest-payment schedules for the 5.500% Notes due 2009 and 6.375% Notes due 2015. This represents a standard debt-issuance event with no immediate earnings or operational-guidance implications."

There is real financial reasoning here — not just a label. The model:

• identifies the event type (earnings beat / debt issuance / enforcement);
• extracts material figures & terms (EPS + surprise %; note size, coupon, maturity; "no guidance");
• applies domain knowledge — debt issuance → refinancing / capital structure; the NASD qualified-underwriter rule; "no operational catalysts → neutral";
• forms a calibrated judgment (direction + materiality) derived from that reasoning.

Crucially, the reasoning field comes before direction / score_bucket in the JSON — a short "think-then-label" chain, so the label is the output of the reasoning, not a standalone guess.

But — honestly — this is reading-depth, not analyst-depth. By design it does NOT:

• synthesize across multiple documents (one text, single pass);
• do valuation, scenario, or deep multi-step chain-of-thought (no long <think> like math);
• run time-series / cross-sectional analysis.

Why concise? (1) The teacher target is deliberately 2–4 factual, ex-ante sentences (no deliberation / <think>). (2) The base model's deep <think> math reasoning was traded away during SFT — that is the measured forgetting (GSM8K 94→67). We built a reader, not an analyst — on purpose.

This depth is exactly right for a bulk reader (raw text → structured record; 87–88% teacher agreement; 100% valid JSON). And deeper reasoning would not help prediction anyway — the 35B teacher and the 12B model are also at chance (the ceiling is the signal, not the reasoning). For deeper financial reasoning you would train a different recipe (keep the <think> CoT, a multi-step / multi-document target, or reasoning-distillation) — a possible next generation.

---

6. Evaluation — capability & the honest negative result

General capability (apples-to-apples, 150 GSM8K / 80 MBPP / robustness 40×5):

The student traded some general math (94→67, full-FT erosion) for finance specialization; code intact.

🔴 Predictive test (the real finance question): NO EDGE — and that's the finding. On a 147-record walk-forward holdout with realized T+20 returns (chance = 50%):

All three — a 35B, a 12B, and this 3B — are at chance. The ceiling is the label/signal (sentiment ≠ alpha), not model size or intelligence. Confidence does not predict returns either (flat calibration). Use this model for structured reading/extraction, not as a trading signal. A real predictive edge would need RLVR grounded on realized returns (future work), not more SFT.

Training status — SFT complete; RLVR (GRPO) attempted but NOT feasible on this hardware

This model is the SFT deliverable (a finance reader). The intended next stage — RLVR / GRPO grounded on the realized-return sidecar (reward = does the ex-ante call match the realized T+20 excess move?) — was attempted but could not be completed on the training box (WSL · 2× RTX 3090 / 48 GB):

• recurring WSL CUDA "device not ready" / out-of-memory failures — online RL with K-rollouts per prompt is far heavier than SFT or inference, and 48 GB was not enough;
• the runs that did start did not learn: most GRPO groups had zero reward variance (frac_reward_zero_std ≈ 70%), i.e. the realized-return reward was too faint to produce a gradient — consistent with the label-ceiling result above.

So RLVR is future work for a bigger box (A100 / native Linux) with a stronger reward design. The reward function, the realized-return data, and the GRPO scripts are all prepared — only the compute is the blocker. The shipped SFT reader is complete and fully evaluated.

---

7. Quantization — every type tested (same eval suite)

All variants were run through the identical student suite. "Finance-lossless" = valid-JSON 100% and direction/score within bf16 noise.

✅ Working (shipped here)

⚠️ Q2_K (2-bit) is finance-lossless (valid-JSON 100%, dir-3way 87%, sign-acc 40%) but general code collapses (MBPP 41→10%) and math drops (67→51%) — ship it for the finance-reading task only, not as a general model. It is the smallest usable variant at just 1.27 GB.

Also lossless but not shipped as a file (runtime-only): bitsandbytes NF4 (vLLM --quantization bitsandbytes).

❌ Did NOT work on this hardware (documented, not shipped)

Key quant findings:

1. 4-bit is enough for the finance task — AWQ-W4A16, GGUF-Q4_K_M, bnb-NF4 are all lossless.
2. Low-bit GGUF (≤3-bit) only works with an imatrix (importance matrix from in-domain finance calibration). Without it, Q3/Q2 produce pure garbage (0% valid). With an imatrix, even 2-bit (1.27 GB) is finance-lossless — though at 2-bit general code ability collapses, so Q2 is finance-reading-only.
3. FP8 is a Hopper/Ada format — don't use it on Ampere.
4. Quantization does not speed up this small model at batch (Q4-Q8 TP1 ≈ 900–1175 tok/s vs bf16 1175) — the win is VRAM/density, not per-stream speed. For max throughput on 2 GPUs use data-parallel TP=1 copies, not TP=2 (TP=2 only ~1.2–1.5× for a 3B). See QUANT_COMPARISON.md.

---

8. Usage

vLLM (compressed-tensors):

vllm serve <repo>/w4a16          # or w8a16 — quantization auto-detected

vLLM (GGUF):

vllm serve <repo>/gguf/student-Q4_K_M.gguf --tokenizer <base-or-this-repo>

llama.cpp (GGUF):

llama-cli -m student-Q4_K_M.gguf -p "<chat-formatted prompt>"

Prompt the model exactly as it was trained: "You are a financial analyst. Read ONLY the text below and return STRICT JSON … (ex-ante, no future prices)" — then the text. Use the model's chat template. Serve at 8192 context for long filings (its only weak spot is truncation on very long sec_fulltext).

---

9. Limitations

• Not a return/price predictor — at chance on realized returns (so is the 35B teacher and Gemma-12B). Use for reading.
• Long-document truncation — sec_fulltext agreement drops to ~69%; serve at 8192 ctx to mitigate.
• Survivorship / short news window in the underlying labels (current S&P 500 only; news ~3–4 months) — affects any predictive use, not the reading task.
• Quality ceiling = the teacher's labels (no human gold). FP8 / GPTQ-4bit / non-imatrix ≤Q3 quants are broken (above).

---

10. Files in this repo

• bf16/ — full-precision SFT source weights (6.2 GB, 2 safetensors shards) — the reference model every quant is derived from; load directly with transformers/vLLM.
• w4a16/, w8a16/ — compressed-tensors (vLLM, auto-detected).
• gguf/student-Q4_K_M · Q5_K_M · Q6_K · Q8_0 — standard K-quants (lossless).
• gguf/student-Q3_K_M-imat · Q3_K_L-imat — imatrix 3-bit (finance-lossless).
• gguf/student-Q2_K-imat — imatrix 2-bit, 1.27 GB (finance-lossless; smallest — but general code degraded, finance-only).

Base: VibeThinker-3B (license inherits from the base model). Teacher: Qwen3.6-35B-A3B. Finance-SFT, distillation, and quantization: 2026-06. Quantized with in-domain finance calibration (llmcompressor / llama.cpp imatrix).

---

Training data: BatuhanECB/vibethinker-3b-finance-sftmini-data · Code & reports: github.com/batuhan3526/vibe_train