edgerunner-ai_-_EdgeRunner-Tactical-7B-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

EdgeRunner-Tactical-7B - GGUF

• Model creator: https://huggingface.co/edgerunner-ai/
• Original model: https://huggingface.co/edgerunner-ai/EdgeRunner-Tactical-7B/

Original model description:

library_name: transformers license: apache-2.0 language:

• en

---

EdgeRunner-Tactical-7B

Introduction

EdgeRunner-Tactical-7B is a powerful and efficient language model for the edge. Our mission is to build Generative AI for the edge that is safe, secure, and transparent. To that end, the EdgeRunner team is proud to release EdgeRunner-Tactical-7B, the most powerful language model for its size to date.

EdgeRunner-Tactical-7B is a 7 billion parameter language model that delivers powerful performance while demonstrating the potential of running state-of-the-art (SOTA) models at the edge.

Highlights

• 7 billion parameters that balance power and efficiency
• SOTA performance within the 7B model range
• Initialized from Qwen2-Instruct, leveraging prior advancements
• Self-Play Preference Optimization (SPPO) applied for continuous training and alignment
• Competitive performance on several benchmarks with Meta’s Llama-3-70B, Mixtral 8x7B, and Yi 34B
• Context length of 128K tokens, ideal for extensive conversations and large-scale text tasks

Quickstart

Below is a code snippet to show you how to load the tokenizer and model, and how to generate contents.

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "edgerunner-ai/EdgeRunner-Tactical-7B",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("edgerunner-ai/EdgeRunner-Tactical-7B")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Example Outputs

Create a Quantum Future:

<img src="https://cdn-uploads.huggingface.co/production/uploads/633fe629f81b9d10135fefda/3b00jTWhIV_5OWxtW6zFI.png" width="95%">

Ask for a structured JSON output:

<img src="https://cdn-uploads.huggingface.co/production/uploads/633fe629f81b9d10135fefda/CzW5qUh9tAkZV8k8Xs4nm.png" width="95%">

Evaluation

In this section, we report the results for EdgeRunner-Tactical-7B models on standard automatic benchmarks. Below are the results.

Arena-Hard Benchmark

InfiniteBench

GSM@ZeroEval

MMLU-REDUX@ZeroEval

WildBench

AlpacaEval 2.0