FinSenti-Qwen3-0.6B-GGUF

FinSenti-Qwen3-0.6B - GGUF

GGUF builds of FinSenti-Qwen3-0.6B for use with Ollama, llama.cpp, LM Studio, KoboldCpp, and other GGUF-compatible runtimes.

This is the same model as the SafeTensors repo, just converted and quantized so you can run it on a CPU or a small GPU without pulling in PyTorch.

Files in this repo

If you're not sure which to pick: start with Q4_K_M. It's the smallest file, it runs everywhere, and the quality drop versus the original bf16 weights is small for a model this size.

Quick start (llama.cpp)

# Download the Q4_K_M file (or pick a different quant from the table above)
huggingface-cli download Ayansk11/FinSenti-Qwen3-0.6B-GGUF FinSenti-Qwen3-0.6B.Q4_K_M.gguf --local-dir .

# Run it
./llama-cli -m FinSenti-Qwen3-0.6B.Q4_K_M.gguf \
  --system "You are a financial sentiment analyst. For each headline you receive, write a short reasoning chain inside <reasoning>...</reasoning> tags, then give a single label inside <answer>...</answer> tags. The label must be exactly one of: positive, negative, neutral." \
  -p "Apple beats Q4 estimates as iPhone sales jump 12% year over year." \
  -n 256

Quick start (Ollama)

This repo ships a Modelfile for each quant. To register the Q4_K_M build under the name finsenti-qwen3-0-6b:

huggingface-cli download Ayansk11/FinSenti-Qwen3-0.6B-GGUF \
  FinSenti-Qwen3-0.6B.Q4_K_M.gguf Modelfile.Q4_K_M --local-dir ./finsenti-tmp
cd finsenti-tmp
ollama create finsenti-qwen3-0-6b -f Modelfile.Q4_K_M

# Then chat with it
ollama run finsenti-qwen3-0-6b "Apple beats Q4 estimates as iPhone sales jump 12% year over year."

You should see output like:

<reasoning>
Beating estimates is a positive earnings surprise. A 12% YoY iPhone sales jump in the company's biggest product line points to demand strength. Both signals push the read positive.
</reasoning>
<answer>positive</answer>

Quick start (Python via llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="./FinSenti-Qwen3-0.6B.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=8,
)

system = (
    "You are a financial sentiment analyst. For each headline you receive, "
    "write a short reasoning chain inside <reasoning>...</reasoning> tags, "
    "then give a single label inside <answer>...</answer> tags. The label "
    "must be exactly one of: positive, negative, neutral."
)

resp = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": system},
        {"role": "user", "content": "Apple beats Q4 estimates as iPhone sales jump 12% year over year."},
    ],
    max_tokens=256,
    temperature=0.0,
)
print(resp["choices"][0]["message"]["content"])

Hardware

The Q4_K_M build is about 0.37 GB on disk and needs roughly 1 GB of free RAM at runtime. On a modern laptop CPU you should see 15-40 tokens per second depending on the size of the model and your core count. Throwing it on a small GPU (Apple Silicon, a 6-8 GB NVIDIA card) gets you considerably faster generation.

If you need more headroom, the Q5_K_M and Q8_0 files are progressively closer to the original bf16 quality at the cost of size.

Picking a quant

• Q4_K_M (0.37 GB): the default for laptops and small servers. Mild quality dip versus full precision but fits almost anywhere.
• Q5_K_M (0.41 GB): a step up if you have the RAM. Most people won't notice the difference from Q8.
• Q8_0 (0.60 GB): closest to the bf16 weights. Use this if you want the cleanest output and have the disk space.

Run it on your phone

This model is small enough to run entirely on-device. The Q4_K_M build is 0.37 GB on disk and needs roughly 1.0 GB of free RAM during inference, so it fits on most phones with 4 GB+ RAM (roughly any Android flagship from 2020 onward, or iPhone 11 and newer).

iOS

The easiest path is PocketPal AI (free, App Store):

1. Install PocketPal AI from the App Store.
2. Open the app and go to Models -> + -> Add from Hugging Face.
3. Search for Ayansk11/FinSenti-Qwen3-0.6B-GGUF and select FinSenti-Qwen3-0.6B.Q4_K_M.gguf.
4. Tap download; the file is 0.37 GB.
5. Once downloaded, tap the model to load it. Open the chat tab.
6. Set the system prompt (gear icon) to:

   You are a financial sentiment analyst. For each headline you receive, write a short reasoning chain inside <reasoning>...</reasoning> tags, then give a single label inside <answer>...</answer> tags. The label must be exactly one of: positive, negative, neutral.

7. Send a headline like "Apple beats Q4 estimates as iPhone sales jump 12% YoY" and you'll get back the reasoning chain plus the label.

LLMFarm and Private LLM work too if you already use them.

Android

PocketPal AI is on Google Play as well, with the same flow as the iOS version.

If you'd rather avoid the Play Store, ChatterUI is a free, open-source client. Install the APK from the GitHub Releases page, then add the model from Hugging Face inside the app.

Tips for phone usage

• Keep max output tokens around 256. A reasoning chain plus an answer rarely needs more than that.
• Inference is fully offline once the model is downloaded. No data leaves your phone.
• Heat and battery: one classification finishes in a few seconds, but running hundreds in a loop will warm the device up. Charge while batching.
• Stick with Q4_K_M on phones. The quality difference vs Q5/Q8 for sentiment labels is small, and the smaller file leaves more headroom for the OS.

Prompt format

Same as the base model. Use the system prompt verbatim, put the headline or short snippet in the user turn, and parse the <answer>...</answer> block for the label.

Limitations

GGUF is a faithful conversion of the base model, so the same caveats apply:

• English only
• Short text only (training context was 2048 tokens)
• Three labels: positive, negative, neutral
• It explains its read but it isn't doing finance research; don't use the reasoning chain as investment advice

Quantization adds a small extra error on top of the base model. For Q4_K_M on a model this size you'll see occasional disagreement with the bf16 model on borderline headlines, usually neutral-vs-positive flips.

Related FinSenti models

Other sizes and bases trained with the same recipe:

• Qwen3: Qwen3-1.7B, Qwen3-4B, Qwen3-8B
• Qwen3.5: Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B
• DeepSeek: DeepSeek-R1-1.5B
• MobileLLM: MobileLLM-R1-950M
• Tiny-LLM: Tiny-LLM-10M
• Llama-3: Llama-3.2-1B
• SmolLM: SmolLM-1.7B

The full-precision SafeTensors version of this model is at Ayansk11/FinSenti-Qwen3-0.6B, and the training data is at Ayansk11/FinSenti-Dataset.

Citation

@misc{shaikh2026finsenti,
  title  = {FinSenti: Small Language Models for Financial Sentiment with Chain-of-Thought Reasoning},
  author = {Shaikh, Ayan},
  year   = {2026},
  url    = {https://huggingface.co/collections/Ayansk11/finsenti},
  note   = {Indiana University}
}

License

Apache 2.0.