Downloads · 30 days
14
10% of all-time downloads
Echo9Zulu/Cydonia-24B-v2.1-int4_asym-awq-wqe-se-ov
Cydonia-24B-v2.1-int4_asym-awq-wqe-se-ov is a text generation model from Echo9Zulu. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Downloads · 30 days
14
10% of all-time downloads
All-time downloads
144
Public
Repo size
12.9 GB
Likes
0
Public
Click a slice to open those files.
.bin12.9 GB · 100%
From the Hugging Face model README
With OpenVINO GenAI:
import openvino_genai as ov_genai
model_dir = "path-to-your-converted-model"
pipe = ov_genai.LLMPipeline(
model_dir, # Path to the model directory
"CPU", # Define the device to use
)
generation_config = ov_genai.GenerationConfig(
max_new_tokens=128
)
prompt = "We don't even have a chat template so strap in and let it ride!"
result = pipe.generate([prompt], generation_config=generation_config)
perf_metrics = result.perf_metrics
print(f'Load time: {perf_metrics.get_load_time() / 1000:.2f} s')
print(f'Time to first token: {perf_metrics.get_ttft().mean / 1000:.2f} s')
print(f'Time per token: {perf_metrics.get_tpot().mean:.2f} ms/token')
print(f'Throughput: {perf_metrics.get_throughput().mean:.2f} tokens/s')
print(f'Generate duration: {perf_metrics.get_generate_duration().mean / 1000:.2f} s')
print(result)
And with Optimum-Intel, which is an OpenVINO integration for Transformers.