llama-3.2-Korean-Bllossom-3B-GGUF

<!-- header start --> <!-- 200823 --> <div style="width: auto; margin-left: auto; margin-right: auto"> <img src="https://github.com/LlamaEdge/LlamaEdge/raw/dev/assets/logo.svg" style="width: 100%; min-width: 400px; display: block; margin: auto;"> </div> <hr style="margin-top: 1.0em; margin-bottom: 1.0em;"> <!-- header end -->

llama-3.2-Korean-Bllossom-3B-GGUF

Original Model

Bllossom/llama-3.2-Korean-Bllossom-3B

Run with LlamaEdge

• LlamaEdge version: v0.16.9 and above

• Prompt template

  • Prompt type for chat: llama-3-chat

    • Prompt string

      <|begin_of_text|><|start_header_id|>system<|end_header_id|>
      
      {{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>
      
      {{ user_message_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
      
      {{ model_answer_1 }}<|eot_id|><|start_header_id|>user<|end_header_id|>
      
      {{ user_message_2 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
      

  • Prompt type for tool use: llama-3-tool

    • Prompt string

      <|begin_of_text|><|start_header_id|>system<|end_header_id|>
      
      {system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>
      
      Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.
      
      Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.
      
      [{"type":"function","function":{"name":"get_current_weather","description":"Get the current weather in a given location","parameters":{"type":"object","properties":{"location":{"type":"string","description":"The city and state, e.g. San Francisco, CA"},"unit":{"type":"string","description":"The temperature unit to use. Infer this from the users location.","enum":["celsius","fahrenheit"]}},"required":["location","unit"]}}}]
      
      Question: {user_message}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
      

• Context size: 128000

• Run as LlamaEdge service

  • Chat

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-3.2-Korean-Bllossom-3B-Q5_K_M.gguf \
        llama-api-server.wasm \
        --prompt-template llama-3-chat \
        --ctx-size 128000 \
        --model-name llama-3.2-Korean
    

  • Tool use

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-3.2-Korean-Bllossom-3B-Q5_K_M.gguf \
        llama-api-server.wasm \
        --prompt-template llama-3-tool \
        --ctx-size 128000 \
        --model-name llama-3.2-Korean
    

• Run as LlamaEdge command app

  wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-3.2-Korean-Bllossom-3B-Q5_K_M.gguf \
    llama-chat.wasm \
    --prompt-template llama-3-chat \
    --ctx-size 128000
  

Quantized GGUF Models

Quantized with llama.cpp b4831