comfyui-workflowgenerator-models

ComfyUI-WorkflowGenerator Models

This repository contains the quantized GGUF models required for ComfyUI-WorkflowGenerator, a custom node implementation that generates ComfyUI workflows from natural language descriptions.

Models Included

1. Workflow Generator Model (Required)

• File: workflow-generator-q8_0.gguf
• Tokenizer: workflow-generator/ directory
• Purpose: Generates workflow diagrams from natural language instructions
• Base Model: Qwen2.5-14B
• Training: Fine-tuned from Qwen2.5-14B using LLaMA-Factory (see original ComfyGPT repository)
• Status: Required - This model is always needed

2. Embedding Model (Required)

• Directory: paraphrase-multilingual-MiniLM-L12-v2/
• Purpose: Semantic search for node name matching and validation
• Base Model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Status: Required - Always needed for semantic search in NodeValidator

3. Node Validator Model (Optional)

• File: Qwen2.5-7B-Instruct-q8_0.gguf
• Tokenizer: Qwen2.5-7B-Instruct/ directory
• Purpose: Refines and corrects node names in workflow diagrams (LLM refinement mode)
• Base Model: Qwen2.5-7B-Instruct (base model, not fine-tuned)
• Status: Optional - Only needed if using LLM refinement (use_llm_refinement=True)

Model Training Information

Workflow Generator Model

The workflow-generator-q8_0.gguf model was trained based on the ComfyGPT research methodology:

• Original Model Source: xiatianzs/resources - Original fine-tuned model from ComfyGPT research team
• Base Model: Qwen/Qwen2.5-14B from HuggingFace
• Training Method: Full fine-tuning (Supervised Fine-Tuning / SFT)
• Training Framework: LLaMA-Factory
• Training Dataset: FlowDataset.json - Contains instruction-input-output pairs where:
  • Instruction: "Based on the description I provided, generate a JSON example of the required ComfyUi workflow."
  • Input: Natural language workflow descriptions
  • Output: JSON diagrams (list of edges representing workflow connections)
• Training Hyperparameters:
  • Learning rate: 1.0e-5
  • Epochs: 3.0
  • Batch size: 1 per device (gradient accumulation: 4 steps)
  • LR scheduler: Cosine with 0.1 warmup ratio
  • Precision: bf16
  • Cutoff length: 8,192 tokens (training cutoff; model architecture supports up to 131,072 tokens)
  • DeepSpeed: ZeRO-3 optimization
• Quantization: q8_0 (8-bit quantization for efficient inference)

For more details on the training process, see the original ComfyGPT repository and training configuration.

Embedding Model

The paraphrase-multilingual-MiniLM-L12-v2 model is a SentenceTransformer model used for semantic search in the NodeValidator. It encodes node names into embeddings and finds the most similar nodes when correcting invalid node names in workflow diagrams.

• Original Model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Type: SentenceTransformer (HuggingFace format)
• Size: ~420 MB
• Dimensions: 384 (embedding vector size)
• Use Case: Semantic similarity search for node name matching

Node Validator Model

The Qwen2.5-7B-Instruct-q8_0.gguf model is the base Qwen2.5-7B-Instruct model (not fine-tuned), used for its built-in instruction-following capabilities to select the best node from semantic search candidates.

Quick Download

Download all models at once:

huggingface-cli download DanielPFlorian/comfyui-workflowgenerator-models \
    --local-dir ./ComfyUI/models/LLM/

Or download specific models:

# Download only the required models (workflow-generator + embedding model)
huggingface-cli download DanielPFlorian/comfyui-workflowgenerator-models \
    --include "workflow-generator-q8_0.gguf" "workflow-generator/*" "paraphrase-multilingual-MiniLM-L12-v2/*" \
    --local-dir ./ComfyUI/models/LLM/

Installation and Setup

Step 1: Download Models

Download the models to your ComfyUI models directory:

# Navigate to ComfyUI directory
cd /path/to/ComfyUI

# Download all models
huggingface-cli download DanielPFlorian/comfyui-workflowgenerator-models \
    --local-dir ./models/LLM/

Step 2: Organize Files in LLM Directory

After downloading, organize the files in ComfyUI/models/LLM/ as follows:

ComfyUI/models/LLM/
├── workflow-generator-q8_0.gguf          # Main model (required)
├── workflow-generator/                   # Main tokenizer (required)
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── vocab.json
│   ├── merges.txt
│   ├── special_tokens_map.json
│   ├── added_tokens.json
│   ├── config.json
│   ├── generation_config.json
│   └── model.safetensors.index.json
├── paraphrase-multilingual-MiniLM-L12-v2/  # Embedding model (required)
│   ├── config.json
│   ├── model.safetensors
│   ├── modules.json
│   ├── sentence_bert_config.json
│   ├── config_sentence_transformers.json
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── special_tokens_map.json
│   ├── sentencepiece.bpe.model
│   ├── unigram.json
│   └── 1_Pooling/
│       └── config.json
├── Qwen2.5-7B-Instruct-q8_0.gguf         # NodeValidator model (optional)
└── Qwen2.5-7B-Instruct/                   # NodeValidator tokenizer (optional)
    ├── tokenizer.json
    ├── tokenizer_config.json
    ├── vocab.json
    ├── merges.txt
    ├── config.json
    └── generation_config.json

Important: The tokenizer directory name must match the model name (without .gguf extension and quantization suffix). The auto-detection code looks for:

• workflow-generator-q8_0.gguf → workflow-generator/ tokenizer
• Qwen2.5-7B-Instruct-q8_0.gguf → Qwen2.5-7B-Instruct/ tokenizer

Usage

Required Models

• workflow-generator-q8_0.gguf + workflow-generator/ tokenizer - Always needed
• paraphrase-multilingual-MiniLM-L12-v2/ - Always needed for semantic search

Optional Models

• Qwen2.5-7B-Instruct-q8_0.gguf + Qwen2.5-7B-Instruct/ tokenizer - Only needed if using LLM refinement (use_llm_refinement=True)

Model Usage in ComfyUI-WorkflowGenerator

1. WorkflowGenerator Node: Uses workflow-generator-q8_0.gguf to generate workflow diagrams from natural language
2. NodeValidator Node:
   • Uses paraphrase-multilingual-MiniLM-L12-v2 for semantic search (always)
   • Uses Qwen2.5-7B-Instruct-q8_0.gguf for LLM refinement (optional, when use_llm_refinement=True)
3. WorkflowBuilder Node: No models needed (deterministic code)

Model Specifications

Workflow Generator Model

• Format: GGUF (q8_0 quantization)
• Base: Qwen2.5-14B
• Size: ~8-9 GB (quantized)
• Context Window: 131,072 tokens (128K) - Model architecture supports up to 131K tokens, though training used 16,384 token cutoff
• Quantization: q8_0 (8-bit, good balance of quality and size)

Embedding Model

• Format: SentenceTransformer (HuggingFace format)
• Base: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Size: ~420 MB
• Dimensions: 384 (embedding vector size)

Node Validator Model

• Format: GGUF (q8_0 quantization)
• Base: Qwen2.5-7B-Instruct
• Size: ~4-5 GB (quantized)
• Context Window: 32,768 tokens
• Quantization: q8_0 (8-bit, good balance of quality and size)

System Requirements

• VRAM:
  • Minimum: 8 GB (for workflow-generator with CPU offloading)
  • Recommended: 16+ GB (for both models on GPU)
• RAM: 16+ GB recommended
• Storage: ~15-16 GB for all models, tokenizers, and embedding model

Performance Tips

1. Use GGUF models: Smaller size and better VRAM efficiency than HuggingFace models
2. GPU Layers: Use "auto" setting for optimal GPU layer allocation
3. LLM Refinement: Only enable if you need higher accuracy (slower but more accurate)
4. Semantic Search Only: Faster execution, deterministic results (recommended for most use cases)

Troubleshooting

Model Not Found

• Verify models are in ComfyUI/models/LLM/ directory
• Check tokenizer directory name matches model name (without .gguf and quantization suffix)
• Restart ComfyUI after moving files

Tokenizer Not Found

• Ensure tokenizer directory exists with the correct name
• Verify tokenizer files (tokenizer.json, tokenizer_config.json, vocab.json) are present
• Check directory structure matches the expected format

Out of Memory

• Reduce n_gpu_layers (try "auto" or lower number)
• Use smaller quantization (q4_0 instead of q8_0) - note: you'll need to re-quantize
• Set device_preference to "cpu" for some operations

Related Resources

• ComfyUI-WorkflowGenerator: GitHub Repository
• Original ComfyGPT Research: GitHub Repository
• Research Paper: arXiv:2503.17671
• Project Website: https://comfygpt.github.io/

Citation

If you use these models in your research, please cite the original ComfyGPT paper:

@article{huang2025comfygpt,
  title={ComfyGPT: A Self-Optimizing Multi-Agent System for Comprehensive ComfyUI Workflow Generation},
  author={Huang, Oucheng and Ma, Yuhang and Zhao, Zeng and Wu, Mingrui and Ji, Jiayi and Zhang, Rongsheng and Hu, Zhipeng and Sun, Xiaoshuai and Ji, Rongrong},
  journal={arXiv preprint arXiv:2503.17671},
  year={2025}
}

License

These models are provided for use with ComfyUI-WorkflowGenerator. Please refer to:

• Original ComfyGPT repository for model training details and licensing
• Qwen2.5 model licenses from HuggingFace
• ComfyUI-WorkflowGenerator repository for usage terms

Support

For issues, questions, or contributions:

• Issues: ComfyUI-WorkflowGenerator Issues
• Documentation: ComfyUI-WorkflowGenerator Wiki