Seed-OSS-36B-Instruct-GGUF

<!-- header start --> <!-- 200823 --> <div style="width: auto; margin-left: auto; margin-right: auto"> <img src="https://github.com/LlamaEdge/LlamaEdge/raw/dev/assets/logo.svg" style="width: 100%; min-width: 400px; display: block; margin: auto;"> </div> <hr style="margin-top: 1.0em; margin-bottom: 1.0em;"> <!-- header end -->

Seed-OSS-36B-Instruct-GGUF

Original Model

ByteDance-Seed/Seed-OSS-36B-Instruct

Run with LlamaEdge

• LlamaEdge version: coming soon

<!-- - LlamaEdge version: [v0.25.1](https://github.com/LlamaEdge/LlamaEdge/releases/tag/0.25.1) and above -->

• Prompt template

  • Prompt type:

    • seed-oss-think for think mode
    • seed-oss-no-think for no think mode

  • Prompt string

    • Thinking mode

      <seed:bos>system
      You are Doubao, a helpful AI assistant.
      <seed:eos>
      
      <seed:bos>user
      {user_message_1}
      <seed:eos>
      
      <seed:bos>assistant
      <seed:think>{thinking_content}</seed:think>
      {assistant_message_1}
      <seed:eos>
      
      <seed:bos>user
      {user_message_2}
      <seed:eos>
      
      <seed:bos>assistant
      

    • No-thinking mode

      <seed:bos>system
      You are Doubao, a helpful AI assistant.
      <seed:eos>
      
      <seed:bos>system
      You are an intelligent assistant that can answer questions in one step without the need for reasoning and thinking, that is, your thinking budget is 0. Next, please skip       the thinking process and directly start answering the user's questions.
      <seed:eos>
      
      <seed:bos>user
      {user_message_1}
      <seed:eos>
      
      <seed:bos>assistant
      {assistant_message_1}
      <seed:eos>
      
      <seed:bos>user
      {user_message_2}
      <seed:eos>
      
      <seed:bos>assistant
      

• Context size: 512000

• Run as LlamaEdge service

  wasmedge --dir .:. \
    --nn-preload default:GGML:AUTO:Seed-OSS-36B-Instruct-Q5_K_M.gguf \
    llama-api-server.wasm \
    --prompt-template seed-oss-no-think \
    --ctx-size 512000 \
    --model-name seed-oss
  

Quantized GGUF Models

Quantized with llama.cpp b6301.