andthattoo_-_subquery-SmolLM-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

subquery-SmolLM - GGUF

• Model creator: https://huggingface.co/andthattoo/
• Original model: https://huggingface.co/andthattoo/subquery-SmolLM/

Original model description:

base_model: nisten/Biggie-SmoLlm-0.15B-Base license: mit datasets:

• LDJnr/Capybara
• andthattoo/subqueries pipeline_tag: text-generation tags:
• llama

---

Fine-tuned Biggie-SmoLlm-0.15B-Base for generating subqueries

This dude is trained for boosting the performance of your RAG based question answering app

My motivation was to tackle a core problem of RAG with an extremely lightweight, but capable model.

If queries are

• multi-hop logic, break into simpler subqueries that focuses on a different step
• vague, ask follow up questions
• multiple sub questions, generate multiple queries for each of them

Training data was generated with Dria: A decentralized p2p network for synthetic data. Join discord to help decentralized data generation.

Heads up: Ollama version works 160 tps on 1 CPU core. No GPU? No worries. This little dude’s got you.

Use the model:

from transformers import AutoModel, AutoConfig, AutoTokenizer, AutoModelForCausalLM

config = AutoConfig.from_pretrained("andthattoo/subquery-SmolLM")
tokenizer = AutoTokenizer.from_pretrained("andthattoo/subquery-SmolLM")
model = AutoModelForCausalLM.from_pretrained("andthattoo/subquery-SmolLM", torch_dtype=torch.bfloat16)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
    model.config.pad_token_id = model.config.eos_token_id

input_data = "Generate subqueries for a given question. <question>What is this?</question>"
inputs = tokenizer(input_data, return_tensors='pt')
output = model.generate(**inputs, max_new_tokens=100)
decoded_output = tokenizer.decode(output[0], skip_special_tokens=True)

Also created a python package for ease of use

pip install subquery

from subquery import TransformersSubqueryGenerator

# Using the Transformers backend
generator = TransformersSubqueryGenerator()
result = generator.generate("What is this?")

print("Follow-up questions:", result.follow_up)
print("Subqueries:", result.subquery)

or

from subquery import OllamaSubqueryGenerator
# Using the Ollama backend
generator = OllamaSubqueryGenerator()
result = generator.generate("Are the Indiana Harbor and Ship Canal and the Folsom South Canal in the same state?")

print("Follow-up questions:", result.follow_up)
print("Subqueries:", result.subquery)