Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4

Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4: Optimized for SiMa.ai Modalix

Overview

This repository contains the Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4 model, optimized and compiled for the SiMa.ai Modalix platform.

• Model Architecture: Llama 3.1 Swallow (8B parameters)
• Languages: Japanese and English
• Quantization: A16W4 (BF16 activations, INT4 language weights), with mixed-precision exceptions retained from the prequantized checkpoint.
• Pre-quantization: The source checkpoint was pre-quantized with GPTQ before compilation.
• Maximum context length: 4096 tokens
• Source Model: tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.5
• Quantized Checkpoint: simaai/Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-Safetensors

Llama 3.1 Swallow extends Meta Llama 3.1 with enhanced Japanese-language capabilities while retaining English-language capabilities. This repository contains SiMa.ai compiled deployment artifacts and is not a Transformers checkpoint.

Performance

The following measurements characterize text generation on Modalix.

Measured with MoLE using batch size 1, four samples per input length, and max_new_tokens=128. Values are arithmetic means. TTFT includes language-model prefill and the first generated token; generation rate is measured after the first token.

Prerequisites

To run this model, you need:

1. SiMa.ai Modalix Device
2. SiMa.ai CLI: Installed on your Modalix device.
3. SiMa.ai Neat Runtime: Install or update the Neat Library on Modalix. The LLiMa runtime is installed as part of the Neat runtime.
4. Hugging Face CLI: Optional, for downloading the model on a host before copying it to Modalix.

Installation & Deployment

Follow these steps to deploy the model to your Modalix device.

1. Install or Update Neat Runtime

Note: This is a one-time setup. If the Neat Library is already installed on your Modalix device, you can skip this step and continue with model download.

Follow the SiMa.ai Neat getting started guide to install or update the Neat Library on your Modalix device.

The llima CLI is available on Modalix after the Neat runtime is installed. It manages precompiled GenAI models under /media/nvme/llima/models by default. Set LLIMA_MODELS_PATH to use a different model directory.

2. Download the Model

Download the compiled model assets from this repository directly to your device.

hf download simaai/Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4 --local-dir /media/nvme/llima/models/Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4

Alternatively, download the compiled model to a host and copy it to the Modalix device:

hf download simaai/Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4 \
  --local-dir Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4

scp -r Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4 \
  sima@<modalix-ip>:/media/nvme/llima/models/

Replace <modalix-ip> with the IP address of your Modalix device.

Expected Directory Structure:

/media/nvme/llima/
└── models/
    └── Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4/   # The compiled model

Usage

Validate with LLiMa CLI

Run the model directly on Modalix:

llima run Llama-3.1-Swallow-8B-Instruct-v0.5-GPTQ-a16w4

For all runtime options, run:

llima run -h

GenAI Demo Application

The GenAI demo application is separate from LLiMa installation. Use the GenAI Multimodal Assistant page to install and run the demo app. Once installed, the demo app can use precompiled models such as this one.

API Usage

To serve this model with OpenAI- or Ollama-compatible APIs and send requests to it, use the GenAI server workflow in Serve GenAI Models.

For direct LLM calls without setting up a server, see Run an LLM.

Recompilation

To build for an 8,192-token context, recompile the matching source checkpoint with this repository's compile.sh in a compatible SiMa Model Compiler and LLiMa compilation environment. Change --max_num_tokens 4096 to --max_num_tokens 8192 in the script, then follow its source-checkpoint and output-directory instructions.

The context budget includes input and generated tokens. A larger context increases KV-cache memory requirements and must fit the target device. The downloadable artifacts use a 4,096-token context; any performance measurements in this card refer to that build.

Limitations

• Quantization: A16W4 (BF16 activations, INT4 language weights), with mixed-precision exceptions retained from the prequantized checkpoint.
• Platform: The compiled artifacts in this repository are intended for SiMa.ai Modalix and cannot be loaded directly with Hugging Face Transformers.
• Language and task coverage: Validate quality with representative Japanese and English prompts from the intended deployment workload.
• License and acceptable use: Use is subject to the source model's Llama 3.3 and Gemma license terms, intended-use guidance, and restrictions. Review the source model card and included license materials before use or redistribution.

Troubleshooting

• sima-cli not found: Ensure that sima-cli is installed on your Modalix device.
• llima not found: Install or update the Neat Library. See Getting Started.
• Model can't be run: Verify that the model directory is directly inside /media/nvme/llima/models/ and not nested inside another directory with the same name.
• Permission denied: Ensure that you have read/write permissions for /media/nvme.

Resources

• Source model
• Pre-LLiMa quantized checkpoint
• GenAI with LLiMa
• Serve GenAI Models
• Run an LLM
• GenAI Multimodal Assistant