MegaBeam-Mistral-7B-300k-AWQ

MegaBeam-Mistral-7B-300k-AWQ Model

MegaBeam-Mistral-7B-300k-AWQ is a version of the MegaBeam-Mistral-7B-300k model that was quantized using the AWQ method developed by Lin et al. (2023). The MegaBeam-Mistral-7B-300k-AWQ models are approximately 70% smaller than those of MegaBeam-Mistral-7B-300k whilst maintaining comparable performance.

Please refer to the original MegaBeam-Mistral-7B-300k model card for details about the model preparation and training processes.

MegaBeam-Mistral-7B-300k Variants

Dependencies

• autoawq==0.2.5 – AutoAWQ was used to quantize the MegaBeam-Mistral-7B-300k model.
• vllm==0.4.2 – vLLM was used to host models for benchmarking.

Evaluations

InfiniteBench

This benchmark was developed by Zhang et al. (2024), available from https://github.com/OpenBMB/InfiniteBench.

See the original MegaBeam-Mistral-7B-300k model card for more details.

Long Context

The following benchmark results are shown as accuracy (%) values, unless stated otherwise.

Topic Retrieval

See https://lmsys.org/blog/2023-06-29-longchat/

Line Retrieval

See https://lmsys.org/blog/2023-06-29-longchat/#longeval-results

Pass Key Retrieval

See https://github.com/epfml/landmark-attention/blob/main/llama/run_test.py#L101

QuALITY (Question Answering with Long Input Texts, Yes!)

See https://nyu-mll.github.io/quality/

Usage

Inference via vLLM HTTP Host

Launch Host

python -m vllm.entrypoints.openai.api_server \
    --model aws-prototyping/MegaBeam-Mistral-7B-300k-AWQ \
    --quantization awq

Query Host

curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{ "model": "aws-prototyping/MegaBeam-Mistral-7B-300k-AWQ",
          "prompt": "<|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>",
          "temperature": 0,
          "echo": false
    }'

Inference via vLLM Offline Inference

from vllm import LLM, SamplingParams

prompts = [
   "<|prompter|>What are the main challenges to support a long context for LLM?</s><|assistant|>",
]
sampling_params = SamplingParams(temperature=0, max_tokens=100)

llm = LLM(model="aws-prototyping/MegaBeam-Mistral-7B-300k-AWQ")

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

License

Apache 2.0

Limitations

Before using the MegaBeam-Mistral-7B-300k-AWQ model, it is important to perform your own independent assessment, and take measures to ensure that your use would comply with your own specific quality control practices and standards, and that your use would comply with the local rules, laws, regulations, licenses and terms that apply to you, and your content.