meno-lite-0.1-gguf

Meno-Lite-0.1 GGUF

This repository contains quantized GGUF versions of Meno-Lite-0.1. All variants were produced using an importance matrix computed on the train split of the ru_llm_calibration dataset, and are intended to be run with llama.cpp.

Available Formats

Quality Evaluation

Quality was measured on the test split of the Ru LLM Calibration dataset using the llama-perplexity utility. The original FP16 model served as the reference.

↑ – higher is better; ↓ – lower is better

How to interpret these metrics:

• Mean PPL (Q): Lower is better. Shows the average perplexity of the quantized model.
• PPL correlation: Closer to 100% indicates the quantized model behaves almost identically to FP16. Values above 99.5% are considered excellent.
• Mean KLD: Measures the divergence between the output probability distributions. Lower is better; 0 means identical distributions.
• Same top p: The percentage of tokens where the quantized model's top prediction matches the FP16 model. Higher is better – it reflects how often the model's first-choice token remains unchanged.

Usage

1. Install llama.cpp

Follow the official build instructions.

2. Run the model

# CLI
./llama-cli -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf -p "Привет, как дела?"

# Server with WebUI (default http://127.0.0.1:8080)
./llama-server -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf --host 0.0.0.0 --port 8080

For more details on available parameters, see the llama.cpp documentation.

About Meno-Lite-0.1

Meno-Lite-0.1 is a 7B model based on Qwen2.5, fine-tuned for RAG, document QA, information extraction, and knowledge graph construction. Read more about its capabilities, training procedure, and limitations in the main model card.

License

All quantized variants inherit the license of the original model (Apache 2.0).