Llama3-2-3B-Instruct-text-to-sql-20K_GGUF_Q6_K

Llama3-2-3B fine-tuned with 20K lines of text-to-sql dataset with russian prompts localization

Info

• Developed by: Tvisterious
• License: mit
• Finetuned from model : unsloth/llama-3.2-3b-instruct-unsloth-bnb-4bit

Model discription and purpose

This small model was trained and quantized to generate SQL commands quickly, even on slow hardware:

• The speed using CUDA with T4 GPU is 2-5 sec.
• The speed on Intel N100 CPU with 16Gb RAM is about 30-50 sec.

Model was fine-tuned with the 20К lines of Tvisterious/gretelai_synthetic_text_to_sql_russian_prompts_localization dataset. It contains more than 80К lines with russian prompts, data base contexts and sql-commands. This is machine-translated origial gretelai/synthetic_text_to_sql dataset, including translation of the database content and filtering parts of sql-commands and containing only SELECT queries. Note that alpaca-prompt was used for fine-tuning. The model has not been tested with prompts in English or other languages, so it may be unstable.

Usage

For using this model you can follow the usage example below for CPU or you can download the gguf-file and use standard llama_cpp functional:

1. CPU

from huggingface_hub import hf_hub_download
from llama_cpp import Llama
model_path = hf_hub_download(
    repo_id="Tvisterious/Llama3-2-3B-Instruct-text-to-sql-20K_GGUF_Q6_K",
    filename="Llama3-2-3B-Instruct-text-to-sql-20K_GGUF_Q6_K.gguf",
    cache_dir="./models"
)
llm = Llama(
    model_path=model_path,
    n_ctx=1024,
    n_threads=8
)
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
        ### Instruction:
        SQL Prompt: {}
        ### Input:
        Company database: {}
        ### Response:
        SQL: {}
        """
response = llm(
    alpaca_prompt.format(
                "Сколько есть работников с красными машинами?", # instruction 'How many workers have red cars?'
                "T_Workers(worker_id, name, age, id_car), T_Cars(car_id, mark, type, color)", # input with DB context
                "", # output - leave this blank for generation!
                ),
    max_tokens=256,
    temperature=0.7
)
print(response['choices'][0]['text'])