Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits

Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits: Optimized for SiMa.ai Modalix Accuracy Benchmarking

Overview

This repository contains the Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits model, optimized and compiled for the SiMa.ai Modalix platform.

This model is compiled with --return_logits so that it can be used for Modalix accuracy benchmarking without recompiling the source model.

• Model Architecture: Llama 2 (7B parameters)
• Quantization: A16W4 (16-bit activations, 4-bit weights)
• Maximum context length: 4096
• Source Model: meta-llama/Llama-2-7b-chat-hf
• Return logits: Enabled

Accuracy Benchmark Results

The table below compares accuracy results produced by SiMa.ai on the Modalix backend against Hugging Face backend reference results.

Evaluation Sample Counts

Prerequisites

To benchmark accuracy with this model, you need:

1. SiMa.ai Modalix Device
2. SiMa.ai CLI: Installed on your Modalix device.
3. SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
4. LLiMa benchmark CLI: Installed on the host machine used to launch accuracy benchmarking.
5. Hugging Face CLI: Optional, for downloading the model on a host before copying it to Modalix.

Installation & Deployment

Follow these steps to deploy the model to your Modalix device.

1. Install or Update Neat Runtime

Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download. Follow the SiMa.ai Neat getting started guide to install or update the Neat Library on your Modalix device. The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.

2. Download the Model

Download the compiled model assets from this repository directly to your device.

# Download the model to a local directory
llima pull Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits

Alternatively, you can download the compiled model to a Host and copy it to the Modalix device:

hf download simaai/Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits --local-dir Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits
scp -r Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address of your Modalix device. Expected Directory Structure:

/media/nvme/llima/
`-- models/
    `-- Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits/   # The compiled model

Usage

Modalix Backend Accuracy

Run the benchmark from a host machine. The positional model_id must be the source Hugging Face model because MOLE uses its tokenizer and configuration; --board_model selects this compiled artifact on Modalix.

MODALIX_IP="<modalix-ip>"
llima-benchmark accuracy meta-llama/Llama-2-7b-chat-hf \
    --backend modalix \
    --output ./results/modalix \
    --board_ip "$MODALIX_IP" \
    --board_model Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits \
    --task hellaswag piqa triviaqa wikitext winogrande

The compiled --board_model must already exist on Modalix and contain both devkit/ and elf_files/. The CLI starts the benchmark server by default. Add --board_venv_path /path/to/venv when llima is installed in a non-default virtual environment, or --no-board_start_server when connecting to a server that is already running.

Hugging Face Backend Accuracy

Run the same task suite against the source model on the host:

llima-benchmark accuracy meta-llama/Llama-2-7b-chat-hf \
    --backend hf \
    --output ./results/hf \
    --task hellaswag piqa triviaqa wikitext winogrande

See MOLE accuracy benchmarking for the full accuracy benchmarking workflow.

Limitations

• Quantization: This model is quantized (A16W4/A16W4) for Modalix execution. Minor deviations from the full-precision source model may occur.
• Return logits: This model is compiled with --return_logits for accuracy benchmarking workflows. For standard text generation deployments, use the corresponding SiMa.ai model from the Large Language Models collection.

Troubleshooting

• sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
• llima-benchmark not found: Ensure that the LLiMa benchmark CLI is installed on the host machine.
• llima not found: Install or update the Neat Library. See Getting Started.
• Modalix accuracy benchmark fails: Verify that the model was compiled with --return_logits and that the model directory contains both devkit/ and elf_files/.
• Model can't be found on Modalix: Verify the model directory is exactly inside /media/nvme/llima/models/ and not nested (e.g., /media/nvme/llima/models/Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits/Llama-2-7b-chat-hf-GPTQ-a16w4_return_logits).
• Permission Denied: Ensure you have read/write permissions for the /media/nvme directory.

Resources

• GenAI with LLiMa
• LLiMa compilation
• MOLE accuracy benchmarking