Felladrin_-_TinyMistral-248M-Chat-v2-awq

Quantization made by Richard Erkhov.

Github

Discord

Request more models

TinyMistral-248M-Chat-v2 - AWQ

• Model creator: https://huggingface.co/Felladrin/
• Original model: https://huggingface.co/Felladrin/TinyMistral-248M-Chat-v2/

Original model description:

language:

• en license: apache-2.0 base_model: Locutusque/TinyMistral-248M datasets:
• HuggingFaceH4/ultrachat_200k
• Felladrin/ChatML-ultrachat_200k
• Open-Orca/OpenOrca
• Felladrin/ChatML-OpenOrca
• hkust-nlp/deita-10k-v0
• Felladrin/ChatML-deita-10k-v0
• LDJnr/Capybara
• Felladrin/ChatML-Capybara
• databricks/databricks-dolly-15k
• Felladrin/ChatML-databricks-dolly-15k
• euclaise/reddit-instruct-curated
• Felladrin/ChatML-reddit-instruct-curated
• CohereForAI/aya_dataset
• Felladrin/ChatML-aya_dataset pipeline_tag: text-generation widget:
• messages:
  • role: system content: You are a highly knowledgeable and friendly assistant. Your goal is to understand and respond to user inquiries with clarity. Your interactions are always respectful, helpful, and focused on delivering the most accurate information to the user.
  • role: user content: Hey! Got a question for you!
  • role: assistant content: Sure! What's it?
  • role: user content: What are some potential applications for quantum computing?
• messages:
  • role: user content: Heya!
  • role: assistant content: Hi! How may I help you?
  • role: user content: I'm interested in developing a career in software engineering. What would you recommend me to do?
• messages:
  • role: user content: Morning!
  • role: assistant content: Good morning! How can I help you today?
  • role: user content: Could you give me some tips for becoming a healthier person?
• messages:
  • role: system content: You are a very creative assistant. User will give you a task, which you should complete with all your knowledge.
  • role: user content: Hello! Can you please elaborate a background story of an RPG game about wizards and dragons in a sci-fi world? inference: parameters: max_new_tokens: 250 penalty_alpha: 0.5 top_k: 5 model-index:
• name: TinyMistral-248M-Chat-v2 results:
  • task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
    • type: acc_norm value: 23.29 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/TinyMistral-248M-Chat-v2 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
    • type: acc_norm value: 27.39 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/TinyMistral-248M-Chat-v2 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
    • type: acc value: 23.52 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/TinyMistral-248M-Chat-v2 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
    • type: mc2 value: 41.32 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/TinyMistral-248M-Chat-v2 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
    • type: acc value: 49.01 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/TinyMistral-248M-Chat-v2 name: Open LLM Leaderboard
  • task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
    • type: acc value: 0.0 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Felladrin/TinyMistral-248M-Chat-v2 name: Open LLM Leaderboard

---

Locutusque's TinyMistral-248M trained on chat datasets

• Base model: Locutusque/TinyMistral-248M with two additional special tokens (<|im_start|> and <|im_end|>)
• Datasets:
  • [ChatML] HuggingFaceH4/ultrachat_200k
  • [ChatML] Open-Orca/OpenOrca
  • [ChatML] hkust-nlp/deita-10k-v0
  • [ChatML] LDJnr/Capybara
  • [ChatML] databricks/databricks-dolly-15k
  • [ChatML] euclaise/reddit-instruct-curated
  • [ChatML] CohereForAI/aya_dataset
• License: Apache License 2.0
• Availability in other ML formats:
  • GGUF: Felladrin/gguf-TinyMistral-248M-Chat-v2
  • ONNX: Felladrin/onnx-TinyMistral-248M-Chat-v2

Recommended Prompt Format

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant

Recommended Inference Parameters

penalty_alpha: 0.5
top_k: 5

Usage Example

from transformers import pipeline

generate = pipeline("text-generation", "Felladrin/TinyMistral-248M-Chat-v2")

messages = [
    {
        "role": "system",
        "content": "You are a highly knowledgeable and friendly assistant. Your goal is to understand and respond to user inquiries with clarity. Your interactions are always respectful, helpful, and focused on delivering the most accurate information to the user.",
    },
    {
        "role": "user",
        "content": "Hey! Got a question for you!",
    },
    {
        "role": "assistant",
        "content": "Sure! What's it?",
    },
    {
        "role": "user",
        "content": "What are some potential applications for quantum computing?",
    },
]

prompt = generate.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

output = generate(
    prompt,
    max_new_tokens=256,
    penalty_alpha=0.5,
    top_k=5,
)

print(output[0]["generated_text"])

How it was trained

This model was trained with SFTTrainer using the following settings:

Open LLM Leaderboard Evaluation Results

Detailed results can be found here