Qwen2.5-Coder-7B-Instruct-Q8_0-GGUF

smcleod/Qwen2.5-Coder-7B-Instruct-Q8_0-GGUF

This model was converted to GGUF format from Qwen/Qwen2.5-Coder-7B-Instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Ollama Modelfile (draft/beta!)

#  ollama create qwen2.5-coder-7b-instruct:q8_0 -f modelfiles/Modelfile-qwen2.5-coder

FROM ../qwen2.5-coder-7b-instruct-q8_0.gguf

# This is Sam's hacked up template 2024-09-19
TEMPLATE """
{{- $fim_prefix := .FIMPrefix -}}
{{- $fim_suffix := .FIMSuffix -}}
{{- $repo_name := .RepoName -}}
{{- $files := .Files -}}
{{- $has_tools := gt (len .Tools) 0 -}}
{{- if $fim_prefix -}}
<|fim_prefix|>{{ $fim_prefix }}<|fim_suffix|>{{ $fim_suffix }}<|fim_middle|>
{{- else if $repo_name -}}
<|repo_name|>{{ $repo_name }}
{{- range $files }}
<|file_sep|>{{ .Path }}
{{ .Content }}
{{- end }}
{{- else -}}
{{- if or .System $has_tools -}}
<|im_start|>system
{{- if .System }}
{{ .System }}
{{- end }}
{{- if $has_tools }}

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{{- range .Tools }}
{"type": "function", "function": {{ .Function }}}
{{- end }}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
{{- end }}
<|im_end|>
{{- end }}
{{- if .Messages }}
{{- range $i, $message := .Messages }}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{- else if eq .Role "assistant" }}<|im_start|>assistant
{{- if .Content }}{{ .Content }}
{{- else if .ToolCalls }}<tool_call>
{{- range .ToolCalls }}
{"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}}
{{- end }}
</tool_call>
{{- end }}<|im_end|>
{{- else if eq .Role "tool" }}<|im_start|>user
<tool_response>
{{ .Content }}
</tool_response><|im_end|>
{{- end }}
{{- end }}
{{- else if .Prompt -}}
<|im_start|>user
{{ .Prompt }}<|im_end|>
{{- end -}}
<|im_start|>assistant
{{ .Response }}
{{- end -}}
"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|fim_prefix|>"
PARAMETER stop "<|fim_suffix|>"
PARAMETER stop "<|fim_middle|>"
PARAMETER stop "<|repo_name|>"
PARAMETER stop "<|file_sep|>"

### Tuning ##
PARAMETER num_ctx 16384
PARAMETER temperature 0.3
PARAMETER top_p 0.8

#  PARAMETER num_batch 1024
#  PARAMETER num_keep 512
#  PARAMETER presence_penalty 0.2
#  PARAMETER frequency_penalty 0.2
#  PARAMETER repeat_last_n 50

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo smcleod/Qwen2.5-Coder-7B-Instruct-Q8_0-GGUF --hf-file qwen2.5-coder-7b-instruct-q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo smcleod/Qwen2.5-Coder-7B-Instruct-Q8_0-GGUF --hf-file qwen2.5-coder-7b-instruct-q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo smcleod/Qwen2.5-Coder-7B-Instruct-Q8_0-GGUF --hf-file qwen2.5-coder-7b-instruct-q8_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo smcleod/Qwen2.5-Coder-7B-Instruct-Q8_0-GGUF --hf-file qwen2.5-coder-7b-instruct-q8_0.gguf -c 2048