shunya-0.5b-instruct

Shunya-0.5B-Instruct

Shunya-0.5B-Instruct is an instruction-tuned version of Shunya-0.5B-Base, fine-tuned using ORPO on a curated preference dataset mix. It supports multi-turn chat, system prompts, and tool calling via a custom chat template.

Model Details

Special tokens: <|system|>, <|user|>, <|assistant|>, <|endofturn|>, <tool_call>, </tool_call>, <tool_response>, </tool_response>

Training

• Base model: vivekmarakana/shunya-0.5b-base
• Method: SFT (Supervised Fine Tuning) & ORPO (Odds Ratio Preference Optimization)
• Fine-tuning datasets:
  • HuggingFaceTB/smoltalk2
  • NousResearch/hermes-function-calling-v1
  • lmsys/lmsys-chat-1m
  • argilla/ultrafeedback-multi-binarized-preferences-cleaned
  • mlabonne/orpo-dpo-mix-40k
• License: MIT

Usage

Chat

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("vivekmarakana/shunya-0.5b-instruct")
model = AutoModelForCausalLM.from_pretrained(
    "vivekmarakana/shunya-0.5b-instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Explain the difference between supervised and unsupervised learning."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Tool Calling

The model supports structured tool calls using <tool_call> / <tool_response> tags. Pass a list of tools to apply_chat_template:

tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a location.",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"]
        }
    }
]

messages = [{"role": "user", "content": "What's the weather in Mumbai?"}]
inputs = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=False))

Benchmarks

All evaluations were run with lm-evaluation-harness. Results use normalized accuracy (acc_norm) for completion tasks (ARC, PIQA, HellaSwag) and acc for classification tasks.

Qwen3-0.6B-Instruct and Gemma3-1B-IT are included as reference points — both have more parameters and/or larger training budgets.

Notable results:

• On GPQA Diamond (graduate-level science), Shunya-0.5B-Instruct (28.79%) outperforms both Qwen3-0.6B-Instruct (20.20%) and Gemma3-1B-IT (26.77%).
• On ARC-Easy and BoolQ, Shunya-0.5B-Instruct outperforms Qwen3-0.6B-Instruct.

License

MIT