Downloads · 30 days
31
11% of all-time downloads
niushuai1991/all-MiniLM-L6-v2-f32-GGUF
all-MiniLM-L6-v2-f32-GGUF is a feature extraction model from niushuai1991. Use it when you need embeddings to search or compare text. It is set up for llama.cpp. The card lists the license as apache-2.0.
- Sentence-BERT embedding model, converted from sentence-transformers/all-MiniLM-L6-v2 - Architecture: BERT (BertModel), 6 layers, 384-dim embeddings - Max context length: 512 tokens - Format: GGUF F32 (full precision…
Downloads · 30 days
31
11% of all-time downloads
All-time downloads
282
Public
Repo size
90.9 MB
Likes
0
Public
Click a slice to open those files.
.gguf90.9 MB · 100%
From the Hugging Face model README
Converted using llama.cpp convert_hf_to_gguf.py:
python convert_hf_to_gguf.py --outfile all-MiniLM-L6-v2-f32.gguf --outtype f32 ./all-MiniLM-L6-v2
llama-server \
--model all-MiniLM-L6-v2-f32.gguf \
--embeddings \
--host 0.0.0.0 --port 8080 \
-c 512 \
-fa on
Then call POST /v1/embeddings:
curl -X POST http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-d '{"input": "hello world"}'
Response (384-dim vector):
{
"object": "list",
"data": [{"object": "embedding", "embedding": [...], "index": 0}],
"model": "all-MiniLM-L6-v2-f32.gguf",
"usage": {"prompt_tokens": 5, "total_tokens": 5}
}
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.embeddings.create(
model="all-MiniLM-L6-v2-f32.gguf",
input="hello world"
)
print(len(resp.data[0].embedding)) # 384
Ready-to-use Docker setup with multi-stage CUDA build, auto model download, and benchmark scripts:
llm-docker/models/all-minilm-l6-v2-f32
cd models/all-minilm-l6-v2-f32
cp .env.example .env
./download-model.sh
docker compose up -d --build
all-MiniLM-L6-v2-f32.gguf — F32 GGUF model (~90 MB)