Downloads · 30 days
5
8% of all-time downloads
gocharlie-ai/charlie-micro-ov
charlie-micro-ov is a text generation model from gocharlie-ai. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Charlie Micro is a 250M-parameter, ultra-lightweight large language model designed for on-device, offline, and low-power inference. Built on a compact Mistral-style transformer architecture, it is optimized for edge d…
Downloads · 30 days
5
8% of all-time downloads
All-time downloads
63
Public
Repo size
160 MB
Likes
0
Public
Click a slice to open those files.
.bin160 MB · 97%
From the Hugging Face model README
Charlie Micro is a 250M-parameter, ultra-lightweight large language model designed for on-device, offline, and low-power inference. Built on a compact Mistral-style transformer architecture, it is optimized for edge deployments where privacy, latency, and energy efficiency are critical.
The model is optimized using Intel OpenVINO mixed INT8 / INT4 quantization, enabling extremely fast inference on Intel CPUs and NPUs with minimal accuracy degradation. This makes Charlie Micro well-suited for consumer devices, IoT products, vehicles, and environments where cloud connectivity is unavailable or undesirable.
| Attribute | Description |
|---|---|
| Model Name | Charlie Micro |
| Parameters | ~250M |
| Architecture | Mistral-style decoder-only transformer |
| Layers | 12 |
| Hidden Size | 1,024 |
| FFN Size | 4,096 |
| Attention Heads | 32 |
| KV Heads | 8 |
| Context Window | 4,096 tokens |
| Vocabulary Size | 32,003 |
| Quantization | Intel OpenVINO (INT8 / INT4 mixed) |
| Model Size | ~155 MB |
| License | Apache License 2.0 |
Charlie Micro offers a rare combination of small model size, strong reasoning performance, and hardware-specific optimization, outperforming many models in its size class.
| Benchmark | Score |
|---|---|
| MMLU | 34 |
| MMLU-Pro | 18 |
| ARC | 38 |
| HellaSwag | 53 |
CPU:
Intel NPU:
Consumer GPU (local):
Performance may vary depending on hardware, runtime configuration, and prompt length.
Charlie Micro is intended for:
The model is provided as-is and should be independently evaluated before use in high-risk or safety-critical applications.
This approach uses the OpenVINO GenAI LLM pipeline, providing the lowest-level, fastest, and most power-efficient execution path. It is recommended for embedded, IoT, automotive, and NPU-based deployments.
pip install openvino-genai huggingface_hub
import openvino_genai as ov_genai
from huggingface_hub import snapshot_download
model_id = "gocharlie-ai/charlie-micro-ov"
local_dir = "charlie-micro-ov"
# Download model locally (one-time)
snapshot_download(
repo_id=model_id,
local_dir=local_dir,
)
device = "CPU" # or "NPU" if supported
config = ov_genai.GenerationConfig(
temperature=0.1,
max_new_tokens=100
)
pipe = ov_genai.LLMPipeline(
models_path=local_dir,
device=device
)
prompt = "How will AI impact humanity?"
output = pipe.generate(prompt, config=config)
print(output)
Why choose this path
This approach integrates OpenVINO with the Hugging Face ecosystem and is ideal for experimentation, prototyping, and chat-style usage.
pip install transformers optimum[openvino] openvino
from transformers import AutoTokenizer
from optimum.intel.openvino import OVModelForCausalLM
model_id = "gocharlie-ai/charlie-micro-ov"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(
model_id,
device="CPU", # or "NPU" if supported
ov_config={
"PERFORMANCE_HINT": "LATENCY"
}
)
messages = [
{"role": "system", "content": "You are an assistant. Be precise."},
{"role": "user", "content": "How AI will impact society?"}
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_new_tokens=256,
temperature=0.1,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Why choose this path
The model is released freely and without restriction under the Apache License 2.0. There are no restrictions on downstream usage beyond those stated in the license.
If you use Charlie Micro in research or commercial applications, please attribute it to the original Gaudium AI development team.