qwen2.5-coder-7b-it-dynamic_wi4_afp32

Qwen2.5-Coder-7B-Instruct LiteRT-LM

This repository contains a LiteRT-LM converted version of Qwen2.5-Coder-7B-Instruct for efficient on-device code generation and text generation.

The model was converted using the official LiteRT-LM exporter and has been verified to run successfully with LiteRT-LM.

Base Model

• Model: Qwen/Qwen2.5-Coder-7B-Instruct
• Architecture: Qwen2.5
• Task: Text Generation / Code Generation
• Original License: Apache-2.0 :contentReference[oaicite:0]{index=0}

Conversion Environment

Conversion Configuration

Export Configuration

task                              : text_generation
quantization_recipe               : dynamic_wi4_afp32
batch_size                        : 1
cache_length                      : 4096
prefill_lengths                   : [128]
cache_implementation              : LiteRTLMCache
bundle_litert_lm                  : True
externalize_embedder              : True
externalize_rope                  : False
enable_dynamic_shape              : False
enable_gpu_dynamic_cache          : False
enable_gpu_dynamic_prefill        : False
experimental_use_fp16             : False
experimental_use_mixed_precision  : False
use_jinja_template                : True
stateful_after                    : -1

Features

• Official Qwen2.5-Coder-7B-Instruct converted for LiteRT-LM
• Dynamic 4-bit weight quantization (dynamic_wi4_afp32)
• FP32 activation precision
• LiteRT-LM optimized
• Externalized embedding layer
• Original tokenizer and chat template preserved
• GPU backend supported by LiteRT-LM
• Ready for edge-device deployment

Usage

import litert_lm

engine = litert_lm.Engine(
    "PATH_TO_MODEL",
    backend=litert_lm.Backend.GPU(),
)

response = engine.generate(
    "Write a Python implementation of quicksort."
)

print(response)

Performance

The model uses:

• Weight Quantization: Dynamic Int4 (dynamic_wi4_afp32)
• Activation Precision: FP32
• KV Cache: 4096 tokens
• Prefill Length: 128 tokens

Actual inference speed and memory usage depend on the LiteRT-LM backend and target hardware.

Compatibility

This model has been tested with:

• LiteRT-LM 0.14.0
• Transformers 5.14.1

Other LiteRT-LM versions may also work but have not been verified.

Notes

This repository contains only the LiteRT-LM converted model.

It is not intended to be loaded using transformers or AutoModelForCausalLM.from_pretrained(). Instead, use the LiteRT-LM runtime.

Acknowledgements

• Alibaba Cloud Qwen Team
• Google LiteRT-LM

License

This repository contains a converted version of the original model and does not modify the model weights beyond the LiteRT-LM conversion process.

Please refer to the original model repository for licensing details:

https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct