Qwen2.5-14B-Instruct-1M-FP8-Dynamic

This is a quantization of the Qwen2.5-14B-Instruct-1M.

Qwen2.5-14B-Instruct-1M, developed by Alibaba Cloud, is a standout model in the world of large language models due to its exceptional capability to handle ultra-long contexts, supporting up to 1 million tokens. This feature makes it significantly more effective for long-context tasks compared to previous versions, while still maintaining strong performance on shorter tasks. With an architecture incorporating advanced techniques like RoPE, SwiGLU, and RMSNorm, Qwen2.5-1M offers a balanced blend of sophistication and efficiency. It is designed as a causal language model and demonstrates considerable prowess in generating coherent and contextually aware text, marking a substantial advancement in handling complex language generation tasks.

Evaluations

This model provides an accuracy recovery of 100.09%.

We did not check for data contamination. Evaluation was done using Eval. Harness with limit=1000.

Usage

Install vLLM and run the server:

python -m vllm.entrypoints.openai.api_server --model cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic --max-model-len 42000 --gpu-memory-utilization 0.95

Access the model:

curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic",
        "prompt": "San Francisco is a"
    } '

⚡ This model is optimized to handle heavy workloads providing a total throughput of ️4497 tokens per second using one NVIDIA L40S ⚡