-
Notifications
You must be signed in to change notification settings - Fork 6.8k
[Modular] update the auto pipeline blocks doc #13148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
yiyixuxu
wants to merge
2
commits into
main
Choose a base branch
from
auto-blocks-doc
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+73
−4
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -121,16 +121,16 @@ from diffusers.modular_pipelines import AutoPipelineBlocks | |
|
|
||
| class AutoImageBlocks(AutoPipelineBlocks): | ||
| # List of sub-block classes to choose from | ||
| block_classes = [block_inpaint_cls, block_i2i_cls, block_t2i_cls] | ||
| block_classes = [InpaintBlock, ImageToImageBlock, TextToImageBlock] | ||
| # Names for each block in the same order | ||
| block_names = ["inpaint", "img2img", "text2img"] | ||
| # Trigger inputs that determine which block to run | ||
| # - "mask" triggers inpaint workflow | ||
| # - "image" triggers img2img workflow (but only if mask is not provided) | ||
| # - if none of above, runs the text2img workflow (default) | ||
| block_trigger_inputs = ["mask", "image", None] | ||
| # Description is extremely important for AutoPipelineBlocks | ||
|
|
||
| @property | ||
| def description(self): | ||
| return ( | ||
| "Pipeline generates images given different types of conditions!\n" | ||
|
|
@@ -141,7 +141,7 @@ class AutoImageBlocks(AutoPipelineBlocks): | |
| ) | ||
| ``` | ||
|
|
||
| It is **very** important to include a `description` to avoid any confusion over how to run a block and what inputs are required. While [`~modular_pipelines.AutoPipelineBlocks`] are convenient, it's conditional logic may be difficult to figure out if it isn't properly explained. | ||
| It is **very** important to include a `description` to avoid any confusion over how to run a block and what inputs are required. While [`~modular_pipelines.AutoPipelineBlocks`] are convenient, its conditional logic may be difficult to figure out if it isn't properly explained. | ||
|
|
||
| Create an instance of `AutoImageBlocks`. | ||
|
|
||
|
|
@@ -152,5 +152,74 @@ auto_blocks = AutoImageBlocks() | |
| For more complex compositions, such as nested [`~modular_pipelines.AutoPipelineBlocks`] blocks when they're used as sub-blocks in larger pipelines, use the [`~modular_pipelines.SequentialPipelineBlocks.get_execution_blocks`] method to extract the a block that is actually run based on your input. | ||
|
|
||
| ```py | ||
| auto_blocks.get_execution_blocks("mask") | ||
| auto_blocks.get_execution_blocks(mask=True) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Out of curiosity. What was the reasoning behind shifting towards a flag based configuration as opposed to strings ( |
||
| ``` | ||
|
|
||
| ## ConditionalPipelineBlocks | ||
|
|
||
| [`~modular_pipelines.AutoPipelineBlocks`] is a special case of [`~modular_pipelines.ConditionalPipelineBlocks`]. While [`~modular_pipelines.AutoPipelineBlocks`] selects blocks based on whether a trigger input is provided or not, [`~modular_pipelines.ConditionalPipelineBlocks`] lets you implement a `select_block` method with any custom selection logic. | ||
|
|
||
| Here is the same example written using [`~modular_pipelines.ConditionalPipelineBlocks`] directly: | ||
|
|
||
| ```py | ||
| from diffusers.modular_pipelines import ConditionalPipelineBlocks | ||
|
|
||
| class AutoImageBlocks(ConditionalPipelineBlocks): | ||
| block_classes = [InpaintBlock, ImageToImageBlock, TextToImageBlock] | ||
| block_names = ["inpaint", "img2img", "text2img"] | ||
| block_trigger_inputs = ["mask", "image"] | ||
| default_block_name = "text2img" | ||
|
|
||
| @property | ||
| def description(self): | ||
| return ( | ||
| "Pipeline generates images given different types of conditions!\n" | ||
| + "This is an auto pipeline block that works for text2img, img2img and inpainting tasks.\n" | ||
| + " - inpaint workflow is run when `mask` is provided.\n" | ||
| + " - img2img workflow is run when `image` is provided (but only when `mask` is not provided).\n" | ||
| + " - text2img workflow is run when neither `image` nor `mask` is provided.\n" | ||
| ) | ||
|
|
||
| def select_block(self, mask=None, image=None) -> str | None: | ||
| if mask is not None: | ||
| return "inpaint" | ||
| if image is not None: | ||
| return "img2img" | ||
| return None # falls back to default_block_name ("text2img") | ||
| ``` | ||
|
|
||
| The inputs listed in `block_trigger_inputs` are passed as keyword arguments to `select_block()`. When `select_block` returns `None`, it falls back to `default_block_name`. If `default_block_name` is also `None`, the entire conditional block is skipped — this is useful for optional processing steps that should only run when specific inputs are provided. | ||
|
|
||
| ## Workflows | ||
|
|
||
| Pipelines that contain conditional blocks ([~modular_pipelines.AutoPipelineBlocks] or [~modular_pipelines.ConditionalPipelineBlocks]) can support multiple workflows — for example, our SDXL modular pipeline supports a dozen workflows all in one pipeline. But this also means it can be confusing for users to know what workflows are supported and how to run them. For pipeline builders, it's useful to be able to extract only the blocks relevant to a specific workflow. | ||
|
|
||
| We recommend defining a `_workflow_map` to give each workflow a name and explicitly list the inputs it requires. | ||
|
|
||
| ```py | ||
| from diffusers.modular_pipelines import SequentialPipelineBlocks | ||
|
|
||
| class MyPipelineBlocks(SequentialPipelineBlocks): | ||
| block_classes = [TextEncoderBlock, AutoImageBlocks, DecodeBlock] | ||
| block_names = ["text_encoder", "auto_image", "decode"] | ||
|
|
||
| _workflow_map = { | ||
| "text2image": {"prompt": True}, | ||
| "image2image": {"image": True, "prompt": True}, | ||
| "inpaint": {"mask": True, "image": True, "prompt": True}, | ||
| } | ||
| ``` | ||
|
|
||
| All of our built-in modular pipelines come with workflows defined. You can list them all supported workflows like this: | ||
|
|
||
| ```py | ||
| pipeline_blocks = MyPipelineBlocks() | ||
| pipeline_blocks.available_workflows | ||
| # ['text2image', 'image2image', 'inpaint'] | ||
| ``` | ||
|
|
||
| And retrieve the execution blocks for a specific workflow. This returns a [`~modular_pipelines.SequentialPipelineBlocks`] containing only the blocks that would actually execute for that workflow, which is useful for inspecting and debugging. | ||
|
|
||
| ```py | ||
| pipeline_blocks.get_workflow("inpaint") | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we consider providing some kinda of templating in this case? DOesn't have to be in this PR, though.