sci-rus-tiny4-GGUF

Description

This model was converted to GGUF format from mlsa-iai-msu-lab/sci-rus-tiny4 using llama.cpp.

For more information go to here.

Test:

import numpy as np
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
import openai

# ./llama.cpp/build/bin/llama-server --models-dir sci-rus-tiny4-GGUF/ --port 8081 --embeddings
openai_client = openai.OpenAI(
    base_url="http://127.0.0.1:8081/v1",
    api_key="sk-no-key-required",
)

# embedding get
def get_embedding(text: str, limit_tokens: int=2048, model="embedding") -> list[float]:
    response = openai_client.embeddings.create(
        input=text[:limit_tokens],
        model=model,
    )
    return response.data[0].embedding

model = SentenceTransformer(
    "mlsa-iai-msu-lab/sci-rus-tiny4",
)

text = """Текст для математики. Пусть у нас есть функция f(x) = x^2 + 3x + 2. Найдите производную этой функции и определите ее критические точки."""

embed1 = model.encode(text)

for quant in ["Q8_0", "F16", "F32"]:
    embed2 = np.array(get_embedding(text, model=f"sci-rus-tiny4-{quant}"), dtype=np.float32)
    print(f"Cosine Similarity with {quant}: {cos_sim(embed1, embed2).item()}")

Output:

Cosine Similarity with Q8_0: 0.9999872446060181
Cosine Similarity with F16: 0.9999996423721313
Cosine Similarity with F32: 0.9999997615814209

Converting

To get the GGUF file, you have to:

1. Patch llama.cpp/conversion/base.py to add the new model:

# (after     res = "modern-bert")
# To get this hash, just run `./llama.cpp/convert_hf_to_gguf.py sci-rus-tiny4` to show the hash in the output.
        if chkhsh == "762dda4b8f4cebbb9a1e702c6994236675d3acd5cf50cd3d27dcd47bb7b3f599":
            # ref: https://huggingface.co/mlsa-iai-msu-lab/sci-rus-tiny4
            res = "modern-bert"

1. And run

./llama.cpp/convert_hf_to_gguf.py sci-rus-tiny4 --outtype q8_0 --outfile sci-rus-tiny4-GGUF/sci-rus-tiny4-Q8_0.gguf
./llama.cpp/convert_hf_to_gguf.py sci-rus-tiny4 --outtype f16 --outfile sci-rus-tiny4-GGUF/sci-rus-tiny4-F16.gguf
./llama.cpp/convert_hf_to_gguf.py sci-rus-tiny4 --outtype f32 --outfile sci-rus-tiny4-GGUF/sci-rus-tiny4-F32.gguf