Downloads · 30 days
162
3% of all-time downloads
megashuckle/Qwen3.8-27B-Heretic-ARA-NVFP4-NInfer
Qwen3.8-27B-Heretic-ARA-NVFP4-NInfer is a machine learning model from megashuckle. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for ninfer. The card lists the license as apache-2.0.
这是 Qwen3.8-27B Heretic ARA 的 NInfer NVFP4 artifact,identity 为 qwen3.8-27b-heretic-ara / nvfp4。
Downloads · 30 days
162
3% of all-time downloads
All-time downloads
5.7K
Public
Repo size
24.9 GB
Likes
6
Public
Click a slice to open those files.
.ninfer23.7 GB · 98%
From the Hugging Face model README
这是 Qwen3.8-27B Heretic ARA 的 NInfer NVFP4 artifact,identity 为 qwen3.8-27b-heretic-ara / nvfp4。
模型权重来自 Heretic ARA。DFlash2 companion 来自 z-lab/Qwen3.8-27B-DFlash2,固定 revision 为 50307d4c4cde6860d4eee73e2547cd786fe8e8a4。
本 artifact 使用官方 Qwen/Qwen3.8-27B frontend,固定 revision 为 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0,包含 tokenizer、tokenizer 配置、chat template、generation 配置、image preprocessor 和 video preprocessor 六个资源。部分 Unicode 文本在官方 Qwen frontend 与 Heretic 源仓库资源中的 tokenization 不同。
Heretic ARA 原始权重没有沿用普通 Qwen3.8 checkpoint 的运行 identity。本项目为它建立了独立的 NInfer identity,并将官方 Qwen frontend 作为明确的 frontend 输入;converter wrapper 对 Heretic 权重、frontend 资源和 DFlash2 companion 分开校验,runtime registry 和 binder 负责识别并加载该独立 identity。这样可以在保留 Heretic 权重的同时,使用 NInfer 已有的 Qwen3.8 执行路径、MTP、DFlash2 和 Vision 接口。
qwen3_8_27b_heretic_ara_nvfp4.ninfer265336f230bce2e1e82ae08a196d6390c0e066de46c94bbd88205398d8826b87Windows portable runtime 包含已编译的 ninfer.exe、ninfer-serve.exe 和所需 DLL。可直接下载 ninfer-windows-x64-heretic.zip。用户需要安装匹配的 NVIDIA 驱动和 Microsoft Visual C++ 2015–2022 Redistributable x64;不需要安装 CUDA Toolkit。
普通运行:
.\ninfer.exe C:\models\qwen3_8_27b_heretic_ara_nvfp4.ninfer --prompt "编写一个 Python 脚本。" --max-context 8192 --max-new 512 --kv-dtype fp8
MTP3:
.\ninfer.exe C:\models\qwen3_8_27b_heretic_ara_nvfp4.ninfer --prompt "编写一个 Python 脚本。" --max-context 8192 --max-new 2048 --kv-dtype fp8 --spec mtp --draft-tokens 3 --lm-head-draft
DFlash2-7:
.\ninfer.exe C:\models\qwen3_8_27b_heretic_ara_nvfp4.ninfer --prompt "编写一个 Python 脚本。" --max-context 8192 --max-new 2048 --kv-dtype fp8 --spec dflash2 --draft-tokens 7 --lm-head-draft
HTTP 服务使用 ninfer-serve.exe,接口为 OpenAI-compatible /v1;图像或视频请求需要添加 --vision。
测试环境为 Windows 11 x64、GeForce RTX 5090、driver 616.56、FP8 KV、greedy decoding、单请求。每个工况使用独立进程,服务就绪后执行 1 次预热请求,再执行 5 次正式请求。固定任务要求生成一个较完整的 Python 工具,输出长度为 2,048 token。
| 模式 | 8K capacity | 32K capacity | 128K capacity |
|---|---|---|---|
| Base | 70.67 tok/s | 70.73 tok/s | 70.71 tok/s |
| MTP3 | 189.10 tok/s | 190.40 tok/s | 191.51 tok/s |
| DFlash2-7 | 292.00 tok/s | 291.57 tok/s | 292.84 tok/s |
同一任务下,llama.cpp 2.29.1 的 Base 为 71.99 / 71.18 / 71.23 tok/s,MTP3 为 104.08 / 104.11 / 103.50 tok/s。对照模型为 dawncr0w/Qwen3.8-Heretic-NVFP4/Qwen3.8-27B-Heretic-ARA-ModelOpt-NVFP4.gguf,MTP draft 为 dawncr0w/Qwen3.8-Heretic-NVFP4/Qwen3.8-27B-MTP-ONLY-Q4_K_M.gguf。
已运行文本生成、中文、多轮对话、代码生成、翻译、thinking、Vision、MTP3、DFlash2-7、OpenAI-compatible HTTP API,以及 28,672-token context 启动与生成。
This is an NInfer NVFP4 artifact for Qwen3.8-27B Heretic ARA with identity qwen3.8-27b-heretic-ara / nvfp4.
The model weights are from Heretic ARA. The DFlash2 companion is from z-lab/Qwen3.8-27B-DFlash2, pinned to revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4.
The artifact uses the official Qwen/Qwen3.8-27B frontend pinned to revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, containing tokenizer, tokenizer configuration, chat template, generation configuration, image preprocessor, and video preprocessor resources. Some Unicode text tokenizes differently between the official Qwen frontend and the resources in the Heretic source repository.
The original Heretic ARA weights do not use the ordinary Qwen3.8 runtime identity. This work gives them a separate NInfer identity and selects the official Qwen frontend as an explicit frontend input. The converter wrapper validates the Heretic weights, frontend resources, and DFlash2 companion separately; the runtime registry and binder recognize and load the separate identity. This keeps the Heretic weights while using NInfer's existing Qwen3.8 execution path, MTP, DFlash2, and Vision interfaces.
qwen3_8_27b_heretic_ara_nvfp4.ninfer265336f230bce2e1e82ae08a196d6390c0e066de46c94bbd88205398d8826b87The Windows portable runtime contains the compiled ninfer.exe, ninfer-serve.exe, and required DLLs. Download ninfer-windows-x64-heretic.zip. Install a compatible NVIDIA driver and Microsoft Visual C++ 2015–2022 Redistributable x64. CUDA Toolkit is not required.
Normal execution:
.\ninfer.exe C:\models\qwen3_8_27b_heretic_ara_nvfp4.ninfer --prompt "Write a Python script." --max-context 8192 --max-new 512 --kv-dtype fp8
MTP3:
.\ninfer.exe C:\models\qwen3_8_27b_heretic_ara_nvfp4.ninfer --prompt "Write a Python script." --max-context 8192 --max-new 2048 --kv-dtype fp8 --spec mtp --draft-tokens 3 --lm-head-draft
DFlash2-7:
.\ninfer.exe C:\models\qwen3_8_27b_heretic_ara_nvfp4.ninfer --prompt "Write a Python script." --max-context 8192 --max-new 2048 --kv-dtype fp8 --spec dflash2 --draft-tokens 7 --lm-head-draft
The HTTP server is ninfer-serve.exe with an OpenAI-compatible /v1 API. Add --vision for image or video requests.
The test machine was Windows 11 x64 with a GeForce RTX 5090 and driver 616.56. Tests used FP8 KV, greedy decoding, and one request per process. Each condition used one warmup request followed by five measured requests. The fixed workload generated a substantial Python utility with a 2,048-token output.
| Mode | 8K capacity | 32K capacity | 128K capacity |
|---|---|---|---|
| Base | 70.67 tok/s | 70.73 tok/s | 70.71 tok/s |
| MTP3 | 189.10 tok/s | 190.40 tok/s | 191.51 tok/s |
| DFlash2-7 | 292.00 tok/s | 291.57 tok/s | 292.84 tok/s |
On the same task, llama.cpp 2.29.1 measured 71.99 / 71.18 / 71.23 tok/s for Base and 104.08 / 104.11 / 103.50 tok/s for MTP3. The comparison model was dawncr0w/Qwen3.8-Heretic-NVFP4/Qwen3.8-27B-Heretic-ARA-ModelOpt-NVFP4.gguf; the MTP draft was dawncr0w/Qwen3.8-Heretic-NVFP4/Qwen3.8-27B-MTP-ONLY-Q4_K_M.gguf.
The following paths were run: text generation, Chinese text, multi-turn chat, code generation, translation, thinking, Vision, MTP3, DFlash2-7, the OpenAI-compatible HTTP API, and 28,672-token context startup and generation.