Llama-3-8B-LLM2Vec-ARDY-INT8

Llama-3-8B-LLM2Vec-ARDY-INT8

Built with Meta Llama 3.

An 8-bit build of the LLM2Vec Llama-3-8B text encoder used by NVIDIA ARDY for text-to-motion generation, packaged for Voxta.

ARDY conditions on a single 4096-dimensional pooled vector per prompt, produced by LLM2Vec: Meta-Llama-3-8B-Instruct with bidirectional attention, the McGill-NLP MNTP adapter, and the McGill-NLP supervised adapter. Assembling that at runtime means a 16GB download, a 15.4GB resident encoder, and a two-stage adapter merge on every launch. This repo is that same encoder with both adapters already merged and the final weights quantized to 8-bit.

This is the higher-fidelity option. For the smallest footprint, see the NF4 build.

Why a merged build

The adapters were trained against fp16 base weights. Merging them into weights that have already been quantized compounds the error badly: measured cosine similarity against the bf16 reference drops to 0.77. This build merges first at bf16 and quantizes the final weights, which is the correct order.

Measured against the bf16 reference

Twelve motion prompts, encoded through the full LLM2Vec pipeline, compared to the unquantized bf16 assembly:

Relative geometry is preserved: across the prompt set, maximum pairwise-similarity delta is 0.010 and every prompt still lands nearest its own reference vector.

Note that a diffusion model is chaotic with respect to its conditioning. For a fixed seed this encoder produces a different, equally valid motion rather than a degraded one, so seeds are not comparable across precisions. As a control, switching bf16 to fp16 (a change nobody would call a quality loss) already shifts joints by 6.5mm on average.

Usage

Requires bitsandbytes and a CUDA device.

from ardy.model.llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained("voxta/Llama-3-8B-LLM2Vec-ARDY-INT8", device_map={"": "cuda"})

# LLM2Vec selects the Llama-3 instruction template from this field, and transformers 5 does not
# serialize it. Set it or the pooled vectors will not match the reference.
model.model.config._name_or_path = "meta-llama/Meta-Llama-3-8B-Instruct"
model.model = model.model.eval()

vector = model.encode(["A person walks forward and waves."], batch_size=1)

llm2vec_config.json ships in this repo, so pooling behaviour (mean pooling over the prompt tokens, instruction skipped) is restored automatically.

Provenance and licensing

• Base weights: meta-llama/Meta-Llama-3-8B-Instruct, under the Meta Llama 3 Community License and its Acceptable Use Policy.
• Adapters: McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp and McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised, MIT licensed. See LLM2Vec.
• Consumer: NVIDIA ARDY, Apache-2.0 code with NVIDIA Open Model checkpoints.

Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.