CarrotAI_-_Rabbit-Ko-15B-Instruct-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Rabbit-Ko-15B-Instruct - GGUF

• Model creator: https://huggingface.co/CarrotAI/
• Original model: https://huggingface.co/CarrotAI/Rabbit-Ko-15B-Instruct/

Original model description:

license: mit datasets:

• CarrotAI/ko-instruction-dataset
• CarrotAI/Amazing-Instructions
• CarrotAI/kommlu language:
• ko base_model:
• Qwen/Qwen2-7B-Instruct pipeline_tag: text-generation tags:
• Carrot
• Korea
• mergekit

---

Model Detail

I learned only from Singleton data

Quickstart

Here provides a code snippet with apply_chat_template to show you how to load the tokenizer and model and how to generate contents.

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "CarrotAI/Rabbit-Ko-15B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("CarrotAI/Rabbit-Ko-15B-Instruct")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Processing Long Texts

1. Install vLLM: You can install vLLM by running the following command.

pip install "vllm>=0.4.3"

Or you can install vLLM from source.

1. Configure Model Settings: After downloading the model weights, modify the config.json file by including the below snippet:

       {
           "architectures": [
               "Qwen2ForCausalLM"
           ],
           // ...
           "vocab_size": 152064,
   
           // adding the following snippets
           "rope_scaling": {
               "factor": 4.0,
               "original_max_position_embeddings": 32768,
               "type": "yarn"
           }
       }
   

   This snippet enable YARN to support longer contexts.

2. Model Deployment: Utilize vLLM to deploy your model. For instance, you can set up an openAI-like server using the command:

   python -m vllm.entrypoints.openai.api_server --served-model-name CarrotAI/Rabbit-Ko-15B-Instruct --model path/to/weights
   

   Then you can access the Chat API by:

   curl http://localhost:8000/v1/chat/completions \
       -H "Content-Type: application/json" \
       -d '{
       "model": "CarrotAI/Rabbit-Ko-15B-Instruct",
       "messages": [
           {"role": "system", "content": "You are a helpful assistant."},
           {"role": "user", "content": "Your Long Input Here."}
       ]
       }'
   

Applications

This fine-tuned model is particularly suited for [mention applications, e.g., chatbots, question-answering systems, etc.]. Its enhanced capabilities ensure more accurate and contextually appropriate responses in these domains.

Limitations and Considerations

While our fine-tuning process has optimized the model for specific tasks, it's important to acknowledge potential limitations. The model's performance can still vary based on the complexity of the task and the specificities of the input data. Users are encouraged to evaluate the model thoroughly in their specific context to ensure it meets their requirements.

If you liked this model, please use the card below

@article{RabbitKo15BInstruct,
  title={CarrotAI/Rabbit-Ko-15B-Instruct Card},
  author={CarrotAI (L, GEUN)},
  year={2024},
  url = {https://huggingface.co/CarrotAI/Rabbit-Ko-15B-Instruct}
}