qwen3-event-extraction-exp5.1-GGUF

qwen3-event-extraction-exp5.1-GGUF

Q8_0 GGUF quantization of ahalt/qwen3-event-extraction-exp5.1, the attribute-extraction model for NGEC (Next Generation Event Coder). See that repo's model card for what the model does, its evaluation numbers, and the full prompt format.

Built with convert_hf_to_gguf.py --outtype bf16 followed by llama-quantize ... Q8_0 (8.5 bits/weight). No importance matrix was used — at Q8_0 the quantization is close enough to lossless that it isn't needed; see llama.cpp's quantize docs.

Usage with llama.cpp

llama-server -m qwen3-event-extraction-exp5.1-Q8_0.gguf --port 8080 -c 8192

The chat template is embedded in the GGUF, so llama-server's /v1/chat/completions endpoint applies it the same way tokenizer .apply_chat_template(..., enable_thinking=False) does on the Python side. Use the same system prompt and message format documented on the base model's card — the ## Document: … ## Event Type: <definition> v5 format, not the format the older ahalt/event-attribute-extractor GGUFs use:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "Given the event type definition below, find all instances of that event in the document and extract their attributes as JSON. ..."},
      {"role": "user", "content": "## Document: ... \n\n## Event Type: ## Event: **PROTEST**: ..."}
    ],
    "temperature": 0.5,
    "top_p": 0.8,
    "top_k": 20,
    "presence_penalty": 1.5,
    "max_tokens": 2048
  }'

Within the NGEC package, ngec.attribute_model.AttributeModel(backend="llamacpp") builds this request for you — point llamacpp_url at a running llama-server and model_name/NGEC_ATTRIBUTE_MODEL at ahalt/qwen3-event-extraction-exp5.1 so the prompt format is picked correctly (the GGUF supplies weights; the Python side still needs the tokenizer/prompt-format pairing). See demo/deploy/README.md for a supervised deployment of llama-server on a CPU host.

Quality

Q8_0 does not reproduce the unquantized model byte-for-byte, but on the demo's curated example set it produces valid JSON on every prompt and matches the reference (bf16, served via vLLM) closely — see demo/DESIGN.md's "Two backends" section in the NGEC repo for the comparison methodology.