Add support for Chroma Radiance #9682
Open
+813
−8
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
blepping
force-pushed
the
feat_support_chroma_radiance
branch
2 times, most recently
from
70a17b1 to
85bfc39
Compare
|
Can you show some generation examples? |
|
The image is not loading |
Oh it loaded now |
blepping
force-pushed
the
feat_support_chroma_radiance
branch
from
7e4045e to
747abae
Compare
|
blepping
force-pushed
the
feat_support_chroma_radiance
branch
from
b64cc0b to
19bd9b8
Compare
blepping
requested review from
yoland68,
robinjhuang,
pythongosssss,
ltdrdata,
Kosinkadink,
webfiltered,
christian-byrne,
guill and
comfyanonymous
as code owners
blepping
force-pushed
the
feat_support_chroma_radiance
branch
from
19bd9b8 to
6e5f987
Compare
Add batch dimension and fix channels when necessary in ChromaRadianceImageToLatent node
Bump Radiance nerf tile size to 32 Support EasyCache/LazyCache on Radiance (maybe)
…be in line with existing VAE behavior
Cleanups/refactoring to reduce code duplication with Chroma.
Minor code cleanups and tooltip improvements
blepping
force-pushed
the
feat_support_chroma_radiance
branch
from
6e5f987 to
ca5d3ff
Compare
|
This merges but probably needs to be updated for #9639 to work properly. I will try to do that today. Not very likely it would have been merged in the next few hours, but mentioning this just in case. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds support for Chroma Radiance, a pixel-space model adapted from Chroma, which is showing very promising results (although it is relatively early in training). The performance is quite close to Chroma.
I've tried to reuse existing code as much as possible so this is fairly self-contained. A pretty good chunk of it is copied from the implementation at https://github.com/lodestone-rock/flow - I can't take any credit for those parts.
The model can be downloaded at: https://huggingface.co/lodestones/chroma-debug-development-only/tree/main/radiance
No difference from a normal Chroma workflow except you need to use the
ChromaRadianceStubVAEnode instead of loading a VAE. It's not a real VAE, it just moves the channel dimension and scales the input image values to -1..1 for encode and the reverse for decode. There are alsoChromaRadianceLatentToImageandChromaRadianceImageToLatentnodes that do the same thing which could be removed. Lodestone Rock (Chroma and Radiance creator) said he preferred leaving them, I don't have a preference.There is also a
ChromaRadianceOptionsnode that allows overriding some parameters. The main one of interest is likelynerf_tile_size: setting it as high as possible without OOMing is beneficial for performance. I've been told that Blackwell benefits from having it set to 64, the default is currently 32.I have a few more things I want to do before I'd consider it ready to merge (mostly minor stuff like better tooltips for the nodes) so this is starting out as a draft pull. Any comments/feedback are very welcome!
edit: Going to move this out of draft status now. Possible things to resolve before merging:
Chromamodule instead of just copying the code. I try to use code when possible, but this does mean future modifications to Chroma need to take Radiance using it into account. Also, I don't think there is an existing precedent for doing it this way in the existing code. I can change this if necessary.ChromaRadianceLatentToImageandChromaRadianceImageToLatentnodes to clean up the node list. They aren't doing anything unique, on the other hand they are just wrappers around the existing conversion code so maintaining them doesn't add much of a burden and removing them would possibly break existing workflows.ChromaRadianceStubVAE- I'm not good at naming stuff so this may not be the best name. It would be kind of a pain to change now since it would break the existing workflows (though changing the display name would be free). If necessary this can be changed.Let me know if anything needs to be changed to make this mergeable. General comments/feedback are also welcome, of course.