Skip to content

About

Explore DiffSynth in ComfyUI

Resources

Stars

13 stars

Watchers

0 watching

Forks

Repository files navigation

DiffSynth-ComfyUI

A ComfyUI custom node plugin for the open-source framework DiffSynth-Studio. It integrates DiffSynth-Studio's model configuration, quantization, LoRA, and inference pipeline capabilities into ComfyUI, enabling node-based image, video, and audio generation and editing.

DiffSynth-ComfyUI

Installation

Option 1: ComfyUI Registry

ComfyUI Desktop / ComfyUI-Manager:

  • Open the sidebar Manage Extensions or Manager
  • Search for DiffSynth-ComfyUI
  • Click Install, then restart ComfyUI

Option 2: Manual git clone

  1. Install ComfyUI
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
pip install -r requirements.txt
  1. Install DiffSynth-Studio
pip install git+https://github.com/modelscope/DiffSynth-Studio.git
  1. Install this plugin
cd custom_nodes
git clone https://github.com/modelscope/DiffSynth-ComfyUI.git
cd ..
python main.py

If you need quantization features, install the quantization extras:

pip install "diffsynth[quant] @ git+https://github.com/modelscope/DiffSynth-Studio.git"

Once started, open the ComfyUI address shown in the terminal (typically http://127.0.0.1:8188) in your browser.

Quick Start

Using Preset Workflows

Open the ComfyUI interface and load a preset using either method:

  • Template: Click the Template tab in the left sidebar and select a DiffSynth template.
  • JSON file: Choose a workflow from the Workflow Index. Use the .json file from your local example_workflows/ directory, or download it from GitHub using Download raw file on the file page, then drag it onto the ComfyUI canvas.

Modify parameters such as prompt and seed in the Inference node, provide any required input images, video, or audio, then click Run.

Building a Workflow from Scratch

Right-click on a blank canvas to add nodes, then connect them in the following order:

VRAMConfig → ModelConfig → MergeModelConfigs ───┐
                                                ├──→ Loader → Inference → SaveImage
                              VRAMLimit ────────┘

Example: Qwen-Image Text-to-Image

  1. Add a VRAM Config node to configure VRAM strategy for each stage (optional)
  2. Add 3 ModelConfig nodes with the following settings:
    • model_id: Qwen/Qwen-Image, origin_file_pattern: transformer/diffusion_pytorch_model*.safetensors
    • model_id: Qwen/Qwen-Image, origin_file_pattern: text_encoder/model*.safetensors
    • model_id: Qwen/Qwen-Image, origin_file_pattern: vae/diffusion_pytorch_model.safetensors
  3. Connect the VRAM Config output to the vram_config input of all 3 ModelConfig nodes
  4. Add a Merge ModelConfigs node and connect the outputs of the 3 ModelConfig nodes
  5. Add a VRAM Limit node (optional)
  6. Add a Qwen Image Loader node and connect model_configs and vram_limit
  7. Add a Qwen Image Inference node and connect the pipe output from the Loader
  8. Add a built-in SaveImage node and connect the image output from the Inference node
  9. Fill in the prompt in the Inference node and click Run

Workflow Index

All workflow JSON files are in example_workflows/. Both general_* and featured_* files can be imported using the steps above.

Image Generation & Editing

Model Workflows
Anima Preview
Boogu Image Base · Turbo · Edit
ERNIE Image Standard · Turbo
Fibo 2.0 Dev
FLUX.1 Dev · FLEX.2 Preview
FLUX.2 Dev · Klein 4B · Klein 9B
FLUX.3 Klein · EntroPack
HiDream O1 Image
Ideogram 4 FP8
JoyAI Image Edit
Krea 2 Raw · Turbo
MiniMax H3 Image Text-to-Image · Edit · EntroPack
MOSS Image Base · SFT
Qwen Image Standard · 2512 · Edit · Edit 2511 Lightning
Qwen Image 2.1 Featured · EntroPack
SenseNova U1.5 MoT
Stable Diffusion v1.5 · XL Base 1.0
Z-Image Standard · Turbo · Turbo Featured · Turbo EntroPack

Video & Audio-Video Generation

Model Workflows
LingBot Video Text-to-Video
LTX-2 Two-Stage · 2.3 Two-Stage
MiniMax H3 FL2VA · FL2VA EntroPack · Ref2VA · Ref2VA EntroPack · Pruned NF4
MOVA 360P Image-to-Audio-Video
Qwen Video Edit Video Editing
Wan Video Text-to-Video 1.3B · Text-to-Video 14B · Fun A14B Control

Audio Generation

Model Workflows
ACE-Step 1.5 Standard · XL SFT
MiniMax Music 3 Music Generation
YuE2 Music Generation

Node Reference

All nodes are located under the DiffSynth category in the ComfyUI node menu, organized into 4 subcategories:

DiffSynth/config — Model & VRAM Configuration

Node Function
VRAM Config Configures DiffSynth-Studio's four-level VRAM management strategy (offload / onload / preparing / computation), specifying device and data dtype for each stage. Connect the output to the vram_config input of ModelConfig nodes.
VRAM Limit Limits the available VRAM during inference, reserving buffer_size GB for the system. Connect the output to the vram_limit input of the Loader.
ModelConfig Declares a model file's download source (model_id) and file path (origin_file_pattern).
Merge ModelConfigs Merges multiple ModelConfig nodes into a single list for the Loader's model_configs input.
Quantization Config Configures the model quantization scheme. method selects the quantization method (supports bitsandbytes NF4/FP4, etc.), mode selects the quantization mode. Optionally use target_modules and exclude_modules to control the quantization scope. Connect the output to the quant_config input of ModelConfig.
Mixed Quantize Config Combines multiple Quantization Configs to apply different quantization strategies to different modules. Outputs a merged quant_config after connecting multiple Quantization Config inputs.

DiffSynth/loader — Pipeline Loaders

Each pipeline has a corresponding Loader node that loads model weights, applies VRAM management strategy, and returns a pipe object.

Inputs:

  • model_configs (required): Merged configuration from Merge ModelConfigs
  • torch_dtype: Model precision
  • device: Inference device
  • vram_limit (optional): From a VRAM Limit node
  • Other optional parameters: Vary by pipeline

DiffSynth/inference — Pipeline Inference

Each pipeline has a corresponding Inference node that receives a pipe object and performs inference.

Parameters fall into two categories:

  • Widgets: Parameters such as prompt, seed, num_inference_steps, height, and width that can be edited directly on the node
  • Connection inputs: Parameters such as input_image and edit_image that are passed in via connections

DiffSynth/LoRA — LoRA Management

Node Function
LoRA Clear Clears all loaded LoRA weights on the pipeline. Usage: connect the pipe output from the Loader, then chain to LoRA Load after clearing old weights
LoRA Load Loads a LoRA weight to a specified module. lora_config connects to a ModelConfig (declaring the LoRA file source), module specifies the target module, and alpha controls the strength. Multiple LoRA Load nodes can be chained to stack multiple LoRAs: Loader → LoRA Clear → LoRA Load → LoRA Load → Inference

About

Explore DiffSynth in ComfyUI

Resources

Stars

13 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages