diff --git a/README.md b/README.md index 1337c5b92f..db0548d092 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,7 @@ API and command-line option may change frequently.*** - [Ming-Image Design](./docs/ming_image.md) - [PixArt](./docs/pixart.md) - [Iris-3B](./docs/iris.md) + - [P-H3-Image](./docs/p_h3_image.md) - [Image Edit Models](./docs/edit.md) - [FLUX.1-Kontext-dev](./docs/kontext.md) - [Qwen Image Edit series](./docs/qwen_image_edit.md) @@ -81,6 +82,7 @@ API and command-line option may change frequently.*** - [Mage-Flow-Edit](./docs/mage_flow.md#image-editing) - [LLaDA-Image Edit](./docs/llada_image.md#image-editing) - [Fibo 2.0 Dev / Edit Turbo](./docs/fibo2.md#editing-and-turbo) + - [P-H3-Image](./docs/p_h3_image.md#image-editing) - Video Models - [Wan2.1/Wan2.2](./docs/wan.md) - [MiniMax-H3](./docs/minimax_h3.md) diff --git a/docs/p_h3_image.md b/docs/p_h3_image.md new file mode 100644 index 0000000000..ff16da199c --- /dev/null +++ b/docs/p_h3_image.md @@ -0,0 +1,64 @@ +# P-H3-Image + +[P-H3-Image](https://huggingface.co/PrunaAI/P-H3-Image) is Pruna AI's image model built on +[MiniMax-H3](./minimax_h3.md): MiniMax-H3's 33B transformer generating a single frame, a +Qwen3-VL-4B text encoder whose hidden states a text connector maps to the transformer's text +input, and a VAE with a single-frame image decoder. It does text-to-image and edits with one to +eight images. It is guidance-distilled: there is no CFG and no negative prompt. + +## Model files + +Pass the components separately, in MiniMax-H3's original layout: + +- `--diffusion-model`: the MiniMax-H3 transformer +- `--llm`: the Qwen3-VL-4B text encoder with the text connector's tensors under `text_connector.` +- `--vae`: the VAE with the single-frame image decoder + +The text connector is what identifies P-H3-Image: the same transformer without it loads as +MiniMax-H3. + +The diffusers checkpoint in PrunaAI/P-H3-Image does not load directly. + +### Download weights + +- gguf: https://huggingface.co/SearchingMan/P-H3-Image-GGUF/tree/main + - `p_h3_image-Q4_K_M.gguf`, `p_h3_image_text_encoder-Q8_0.gguf`, `p_h3_image_vae-F16.gguf` + +## Text-to-image + +```sh +.\bin\Release\sd-cli.exe --diffusion-model ..\models\p_h3_image-Q4_K_M.gguf --llm ..\models\p_h3_image_text_encoder-Q8_0.gguf --vae ..\models\p_h3_image_vae-F16.gguf -p "A German Expressionist oil painting of a Berlin street at night in the manner of Ernst Ludwig Kirchner, with gaunt figures hurrying past a glowing gas lamp." -W 2048 -H 2048 --diffusion-fa -v +``` + +The model works best at 8 megapixels (2912x2912, 3360x2496, 3872x2176); 2 and 4 megapixels +show more artifacts in faces, hands and small details. Width and height are rounded to +multiples of 32. + +Sampling uses the checkpoint's 8-step sigma grid, `1.0, 0.984753, 0.967905, 0.951688, 0.925372, +0.881923, 0.810083, 0.528718, 0.0`, whatever `--steps` says; pass `--sigmas` to sample another +schedule. + +## Image editing + +Pass the image to edit, then up to seven more references, with `-r`. The prompt is the edit +instruction; it is encoded after `Edit . ` unless it already starts with that or with +`subject_definitions:`. Refer to the other images as `` and so on. + +```sh +.\bin\Release\sd-cli.exe --diffusion-model ..\models\p_h3_image-Q4_K_M.gguf --llm ..\models\p_h3_image_text_encoder-Q8_0.gguf --vae ..\models\p_h3_image_vae-F16.gguf -r street.png -p "Make it a snowy winter night: heavy snow falls across the whole scene, the pavement and rooftops are covered in fresh white snow, and the figures wear snow on their hats and shoulders. Keep the painting style, composition, figures and lamp unchanged." -W 2048 -H 2048 --diffusion-fa -v +``` + +Each image keeps its own size, scaled into 1 to 8 megapixels; the text encoder sees it at about +one megapixel. `--ref-image-args` size settings do not apply. Edits work best at 4 or 8 megapixels and with an instruction that describes the +result, including what to keep: a bare "let it snow" on a 1 megapixel canvas comes out +oversaturated with little snow, while a full description on a 2048x2048 canvas edits cleanly. + +## Notes + +- The reference pipeline rewrites every prompt with a Qwen3.5-9B prompt upsampler first. sd.cpp + renders the prompt as given, so detailed prompts work best. +- `generate_image()` runs P-H3-Image through the MiniMax-H3 video pipeline with one frame; + `generate_video()` with `--video-frames 1` does the same. +- P-H3-Image is licensed under the MiniMax H3 Community License, which excludes the European + Union, the United Kingdom, the Republic of Korea and the United States unless MiniMax grants a + separate license. Its Qwen-derived parts are under Apache 2.0. diff --git a/examples/server/recipes/defaults.md b/examples/server/recipes/defaults.md index e20a2cd783..b00f8d9e2d 100644 --- a/examples/server/recipes/defaults.md +++ b/examples/server/recipes/defaults.md @@ -73,6 +73,7 @@ For community edit adapters and other optional extensions, these are the base-mo | `moss-image-sft` | 3 | 20 | 1024 × 1024 | [Source](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT) | | `nitrosd-realism` | 1 | 1 | 1024 × 1024 | [Source](https://huggingface.co/ChenDY/NitroFusion) | | `ovis-image` | 5 | 20 | 1024 × 1024 | [Source](https://huggingface.co/ATH-MaaS/Ovis-Image-7B) | +| `p-h3-image` | 1 | 8 | 1024 × 1024 | [Source](https://huggingface.co/PrunaAI/P-H3-Image) Guidance-distilled; its fixed 8-step sigma grid is applied on model load. The source recommends 8 MP (default 2912 × 2912); edits work best at 4 or 8 MP. | | `pid` | 1 | 4 | 2048 × 2048 | [Source](https://huggingface.co/nvidia/PiD) Standalone PiD decoding at 2K; guidance is handled by the upstream generator. | | `pid-15` | 1 | 4 | 2048 × 2048 | [Source](https://huggingface.co/nvidia/PiD) Standalone PiD decoding at 2K; guidance is handled by the upstream generator. | | `pid-15-flux2` | 1 | 4 | 2048 × 2048 | [Source](https://huggingface.co/nvidia/PiD) Standalone PiD decoding at 2K; guidance is handled by the upstream generator. | diff --git a/examples/server/recipes/p-h3-image.json b/examples/server/recipes/p-h3-image.json new file mode 100644 index 0000000000..183ec1e06a --- /dev/null +++ b/examples/server/recipes/p-h3-image.json @@ -0,0 +1,38 @@ +{ + "schema_version": 1, + "id": "p-h3-image", + "name": "P-H3-Image", + "size_multiple": 32, + "description": "Text-to-image, or editing with one to eight reference images; the first is the image to edit. Uses the text encoder with its text connector.", + "model_url": "https://huggingface.co/PrunaAI/P-H3-Image", + "slots": [ + { + "id": "diffusion_model", + "required": true + }, + { + "id": "llm", + "required": true, + "label": "P-H3-Image Qwen 3 VL 4B Text Encoder" + }, + { + "id": "vae", + "required": true, + "label": "P-H3-Image VAE" + } + ], + "defaults": { + "cfg_scale": 1, + "steps": 8, + "width": 1024, + "height": 1024, + "sampling_method": "euler" + }, + "features": [ + "ref_images" + ], + "disabled_features": [ + "img2img", + "mask_image" + ] +} diff --git a/src/conditioning/conditioner.hpp b/src/conditioning/conditioner.hpp index e3832a1d8e..ed473150e1 100644 --- a/src/conditioning/conditioner.hpp +++ b/src/conditioning/conditioner.hpp @@ -1971,11 +1971,102 @@ struct AnimaConditioner : public Conditioner { } }; +struct PH3ImageTextConnectorConfig { + int64_t in_channels = 5 * 2560; + int64_t out_channels = 5120; + + static PH3ImageTextConnectorConfig detect_from_weights(const String2TensorStorage& tensor_storage_map, + const std::string& prefix) { + PH3ImageTextConnectorConfig config; + auto proj = tensor_storage_map.find(prefix + ".proj.weight"); + if (proj == tensor_storage_map.end()) { + return config; + } + config.in_channels = proj->second.ne[0]; + config.out_channels = proj->second.ne[1]; + LOG_VERBOSE("p_h3_image text connector: in_channels = %" PRId64 ", out_channels = %" PRId64, + config.in_channels, + config.out_channels); + return config; + } +}; + +// P-H3-Image's map from five Qwen3-VL hidden states to the channels MiniMax-H3's transformer +// reads: standardize, project, destandardize. Ref: diffusers PH3ImageTextConnector. +struct PH3ImageTextConnector : public GGMLBlock { + PH3ImageTextConnectorConfig config; + + explicit PH3ImageTextConnector(const PH3ImageTextConnectorConfig& config) + : config(config) { + blocks["proj"] = std::make_shared(config.in_channels, config.out_channels, false); + } + + void init_params(ggml_context* ctx, + const String2TensorStorage& tensor_storage_map = {}, + const std::string prefix = "") override { + SD_UNUSED(tensor_storage_map); + SD_UNUSED(prefix); + params["input_mean"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, config.in_channels); + params["input_std"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, config.in_channels); + params["output_mean"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, config.out_channels); + params["output_std"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, config.out_channels); + params["sink_embedding"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, config.out_channels); + } + + ggml_tensor* forward(GGMLRunnerContext* ctx, ggml_tensor* x) { + auto proj = std::dynamic_pointer_cast(blocks["proj"]); + x = ggml_div(ctx->ggml_ctx, ggml_sub(ctx->ggml_ctx, x, params["input_mean"]), params["input_std"]); + x = proj->forward(ctx, x); + x = ggml_add(ctx->ggml_ctx, ggml_mul(ctx->ggml_ctx, x, params["output_std"]), params["output_mean"]); + // The first token is an attention sink, conditioned the same for every prompt. + auto sink = ggml_reshape_2d(ctx->ggml_ctx, params["sink_embedding"], config.out_channels, 1); + if (x->ne[1] == 1) { + return sink; + } + return ggml_concat(ctx->ggml_ctx, sink, ggml_ext_slice(ctx->ggml_ctx, x, 1, 1, x->ne[1]), 1); + } +}; + +struct PH3ImageTextConnectorRunner : public GGMLRunner { + PH3ImageTextConnectorConfig config; + PH3ImageTextConnector model; + + PH3ImageTextConnectorRunner(ggml_backend_t backend, + const String2TensorStorage& tensor_storage_map = {}, + const std::string& prefix = "", + std::shared_ptr weight_manager = nullptr) + : GGMLRunner(backend, weight_manager), + config(PH3ImageTextConnectorConfig::detect_from_weights(tensor_storage_map, prefix)), + model(config) { + model.init(params_ctx, tensor_storage_map, prefix); + } + + std::string get_desc() override { + return "p_h3_image_text_connector"; + } + + void get_param_tensors(std::map& tensors, const std::string& prefix) { + model.get_param_tensors(tensors, prefix); + } + + sd::Tensor compute(int n_threads, const sd::Tensor& hidden_states) { + auto get_graph = [&]() -> ggml_cgraph* { + ggml_cgraph* gf = ggml_new_graph(compute_ctx); + auto runner_ctx = get_context(); + ggml_build_forward_expand(gf, model.forward(&runner_ctx, make_input(hidden_states))); + return gf; + }; + return restore_trailing_singleton_dims(GGMLRunner::compute(get_graph, n_threads, true), + hidden_states.dim()); + } +}; + struct LLMEmbedder : public Conditioner { SDVersion version; std::shared_ptr tokenizer; std::shared_ptr llm; std::shared_ptr byt5; + std::shared_ptr text_connector; LLMEmbedder(ggml_backend_t backend, const String2TensorStorage& tensor_storage_map = {}, @@ -2046,6 +2137,12 @@ struct LLMEmbedder : public Conditioner { tokenizer = std::make_shared(); } } + if (sd_version_is_p_h3_image(version)) { + text_connector = std::make_shared(backend, + tensor_storage_map, + "text_encoders.llm.text_connector", + weight_manager); + } if (sd_version_is_hunyuan_video(version)) { const std::string byt5_prefix = "text_encoders.t5xxl.transformer"; for (const auto& [name, _] : tensor_storage_map) { @@ -2066,6 +2163,9 @@ struct LLMEmbedder : public Conditioner { if (byt5) { byt5->get_param_tensors(tensors, "text_encoders.t5xxl.transformer"); } + if (text_connector) { + text_connector->get_param_tensors(tensors, "text_encoders.llm.text_connector"); + } } void get_param_tensor_ops(std::map& tensor_ops) override { @@ -2077,6 +2177,9 @@ struct LLMEmbedder : public Conditioner { if (byt5) { byt5->set_max_graph_vram_bytes(max_vram_bytes); } + if (text_connector) { + text_connector->set_max_graph_vram_bytes(max_vram_bytes); + } } void set_runtime_backends(const std::vector& backends) override { @@ -2141,6 +2244,9 @@ struct LLMEmbedder : public Conditioner { if (byt5) { byt5->runner_end(); } + if (text_connector) { + text_connector->runner_end(); + } } std::tuple, std::vector, std::vector> tokenize(std::string text, @@ -2459,6 +2565,10 @@ struct LLMEmbedder : public Conditioner { prompt_template_encode_start_idx = 0; out_layers = {50}; prompt_attn_range = {0, 0}; + if (text_connector) { + // hidden_states 13, 19, 25, 31 and the final normed one, as the connector was trained on. + out_layers = {13, 19, 25, 31, static_cast(llm->config.num_layers) + 1}; + } if (llm->enable_vision) { const int patch_size = llm->config.vision.patch_size; @@ -2469,14 +2579,30 @@ struct LLMEmbedder : public Conditioner { int width = static_cast(image.shape()[0]); int h_bar = std::max(factor, static_cast(std::round(static_cast(height) / factor)) * factor); int w_bar = std::max(factor, static_cast(std::round(static_cast(width) / factor)) * factor); - resize_image_dims(height, - width, - h_bar, - w_bar, - factor, - 3136, - 12845056, - RefImageResizeMode::AREA); + if (sd_version_is_p_h3_image(version)) { + // About one megapixel at the image's aspect ratio, as P-H3-Image's vision_pixels; + // a size already on the grid and within 20% of that is kept. + constexpr double vision_pixels = 1000000.0; + double area = static_cast(width) * height; + if (width % factor != 0 || height % factor != 0 || + area < 0.8 * vision_pixels || area > 1.2 * vision_pixels) { + double ratio = static_cast(width) / height; + w_bar = std::max(factor, static_cast(std::round(std::sqrt(vision_pixels * ratio) / factor)) * factor); + h_bar = std::max(factor, static_cast(std::round(std::sqrt(vision_pixels / ratio) / factor)) * factor); + } else { + w_bar = width; + h_bar = height; + } + } else { + resize_image_dims(height, + width, + h_bar, + w_bar, + factor, + 3136, + 12845056, + RefImageResizeMode::AREA); + } auto resized = sd::ops::interpolate( image, std::vector{w_bar, h_bar, image.shape()[2], image.shape()[3]}); @@ -2564,6 +2690,15 @@ struct LLMEmbedder : public Conditioner { } } } + if (sd_version_is_p_h3_image(version) && !image_embeds.empty()) { + // An edit names the image it changes, unless the prompt already does (as the + // reference prompt upsampler's edit briefs do). + const size_t start = conditioner_params.text.find_first_not_of(" \t\n\r"); + const auto text = start == std::string::npos ? std::string() : conditioner_params.text.substr(start); + if (!starts_with(text, "Edit .") && !starts_with(text, "subject_definitions:")) { + prompt += "Edit . "; + } + } prompt += conditioner_params.text; } else if (sd_version_is_hunyuan_video(version)) { prompt_template_encode_start_idx = 98; @@ -3227,6 +3362,12 @@ struct LLMEmbedder : public Conditioner { if (hidden_states.empty()) { return {}; } + if (text_connector) { + hidden_states = text_connector->compute(n_threads, hidden_states); + if (hidden_states.empty()) { + return {}; + } + } std::vector> extra_hidden_states_vec; if (sd_version_is_hunyuan_video(version) && byt5) { std::vector quoted_texts; diff --git a/src/model.h b/src/model.h index 12bc22e0f0..de302ec47e 100644 --- a/src/model.h +++ b/src/model.h @@ -70,6 +70,7 @@ enum SDVersion { VERSION_MOSS_IMAGE, VERSION_FIBO2, VERSION_CHROMA, + VERSION_P_H3_IMAGE, VERSION_COUNT, }; @@ -165,8 +166,13 @@ static inline bool sd_version_is_ltxav(SDVersion version) { return false; } +// P-H3-Image is MiniMax-H3 generating one frame, so it takes every MiniMax-H3 path. static inline bool sd_version_is_minimax_h3(SDVersion version) { - return version == VERSION_MINIMAX_H3; + return version == VERSION_MINIMAX_H3 || version == VERSION_P_H3_IMAGE; +} + +static inline bool sd_version_is_p_h3_image(SDVersion version) { + return version == VERSION_P_H3_IMAGE; } static inline bool sd_version_is_wan(SDVersion version) { @@ -303,7 +309,7 @@ static inline bool sd_version_supports_video_generation(SDVersion version) { } static inline bool sd_version_supports_image_generation(SDVersion version) { - return !sd_version_supports_video_generation(version); + return !sd_version_supports_video_generation(version) || sd_version_is_p_h3_image(version); } static inline bool sd_version_uses_flux_vae(SDVersion version) { diff --git a/src/model/integrations/minimax_h3.cpp b/src/model/integrations/minimax_h3.cpp index 2569b626ea..c5b27efb7b 100644 --- a/src/model/integrations/minimax_h3.cpp +++ b/src/model/integrations/minimax_h3.cpp @@ -40,6 +40,12 @@ const std::vector& minimax_h3_definitions() { definition.build = build_minimax_h3; result.push_back(std::move(definition)); } + { + ModelDefinition definition = result.back(); + definition.version = VERSION_P_H3_IMAGE; + definition.name = "P-H3-Image"; + result.push_back(std::move(definition)); + } return result; }(); return definitions; diff --git a/src/model/vae/minimax_h3_vae.hpp b/src/model/vae/minimax_h3_vae.hpp index 1e06558a19..065906a7e7 100644 --- a/src/model/vae/minimax_h3_vae.hpp +++ b/src/model/vae/minimax_h3_vae.hpp @@ -365,9 +365,12 @@ namespace MiniMaxH3VAE { static constexpr int num_layers = 36; static constexpr int num_register_tokens = 4; static constexpr int patch_size = 16; - static constexpr int patch_size_t = 4; + // P-H3-Image's decoder turns one latent frame into one image instead of four frames. + const bool image; + const int patch_size_t; - Decoder() { + explicit Decoder(bool image = false) + : image(image), patch_size_t(image ? 1 : 4) { blocks["x_embedder"] = std::make_shared(24, dim, true); for (int i = 0; i < num_layers; ++i) { blocks["transformer_blocks." + std::to_string(i)] = @@ -389,7 +392,9 @@ namespace MiniMaxH3VAE { GGML_TYPE_F32, dim, num_register_tokens); - params["mask_token"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, dim); + if (!image) { + params["mask_token"] = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, dim); + } } ggml_tensor* forward(GGMLRunnerContext* ctx, @@ -443,7 +448,7 @@ namespace MiniMaxH3VAE { }; struct MiniMaxH3VideoVAE : public GGMLBlock { - MiniMaxH3VideoVAE() { + explicit MiniMaxH3VideoVAE(bool image_decoder = false) { blocks["encoder"] = std::make_shared(); blocks["quant_conv"] = std::make_shared(48, 48, @@ -451,7 +456,7 @@ namespace MiniMaxH3VAE { blocks["post_quant_conv"] = std::make_shared(24, 24, std::tuple{1, 1, 1}); - blocks["decoder"] = std::make_shared(); + blocks["decoder"] = std::make_shared(image_decoder); } ggml_tensor* encode(GGMLRunnerContext* ctx, @@ -483,6 +488,7 @@ namespace MiniMaxH3VAE { }; struct MiniMaxH3VideoVAERunner : public VAE { + bool image_decoder; MiniMaxH3VideoVAE model; sd::Tensor pixel_mean; sd::Tensor pixel_std; @@ -490,11 +496,20 @@ namespace MiniMaxH3VAE { sd::Tensor latents_std; sd::Tensor rope_cache; + static bool detect_image_decoder(const String2TensorStorage& tensor_storage_map, + const std::string& prefix) { + auto proj_out = tensor_storage_map.find(prefix + ".decoder.proj_out.weight"); + return proj_out != tensor_storage_map.end() && + proj_out->second.ne[1] == 3 * Decoder::patch_size * Decoder::patch_size; + } + MiniMaxH3VideoVAERunner(ggml_backend_t backend, const String2TensorStorage& tensor_storage_map, const std::string& prefix = "first_stage_model", std::shared_ptr weight_manager = nullptr) : VAE(VERSION_MINIMAX_H3, backend, prefix, weight_manager), + image_decoder(detect_image_decoder(tensor_storage_map, prefix)), + model(image_decoder), pixel_mean({1, 1, 1, 3}, {0.485f, 0.456f, 0.406f}), pixel_std({1, 1, 1, 3}, {0.229f, 0.224f, 0.225f}), latents_mean({1, 1, 1, 24}, @@ -752,7 +767,10 @@ namespace MiniMaxH3VAE { ids.reserve(static_cast(width * height * num_frames + 5)); constexpr float two_pi = 6.28318530717958647692f; for (int64_t t = 0; t < num_frames; ++t) { - float pt = (2.f * ((t + 0.5f) / num_frames) - 1.f) * two_pi; + // The image decoder places its frame where a video clip's first latent frame of + // seven would be. + float pt = image_decoder ? (2.f * (0.5f / 7.f) - 1.f) * two_pi + : (2.f * ((t + 0.5f) / num_frames) - 1.f) * two_pi; for (int64_t h = 0; h < height; ++h) { float ph = (2.f * ((h + 0.5f) / height) - 1.f) * two_pi; for (int64_t w = 0; w < width; ++w) { diff --git a/src/model_loader.cpp b/src/model_loader.cpp index 5576067409..7ca8f5f068 100644 --- a/src/model_loader.cpp +++ b/src/model_loader.cpp @@ -578,6 +578,10 @@ SDVersion ModelLoader::get_sd_version() const { } if (tensor_storage.name.find("model.diffusion_model.video_patch_proj.weight") != std::string::npos && tensor_storage_map.find("model.diffusion_model.audio_patch_proj.weight") != tensor_storage_map.end()) { + // P-H3-Image keeps MiniMax-H3's transformer and adds a text connector to its text encoder. + if (tensor_storage_map.find("text_encoders.llm.text_connector.proj.weight") != tensor_storage_map.end()) { + return VERSION_P_H3_IMAGE; + } return VERSION_MINIMAX_H3; } if (name == "model.diffusion_model.y_embedder.layer_pool.weight" && diff --git a/src/pipeline/diffusion_engine.cpp b/src/pipeline/diffusion_engine.cpp index d161d9e5be..1912c66821 100644 --- a/src/pipeline/diffusion_engine.cpp +++ b/src/pipeline/diffusion_engine.cpp @@ -1491,7 +1491,10 @@ bool StableDiffusionGGML::build_denoiser() { } case H3_AV_FLOW_PRED: { LOG_INFO("running in MiniMax H3 AV FLOW mode"); - denoiser = std::make_shared(default_flow_shift, 3.f, get_latent_channel()); + // P-H3-Image steps its audio rows on the image's schedule (0 follows the video shift). + denoiser = std::make_shared(default_flow_shift, + sd_version_is_p_h3_image(version) ? 0.f : 3.f, + get_latent_channel()); break; } case MING_IMAGE_FLOW_PRED: { @@ -2589,14 +2592,15 @@ sd::Tensor StableDiffusionGGML::sample(const std::shared_ptr StableDiffusionGGML::generate_init_latent(int width, int StableDiffusionGGML::video_frames_to_latent_frames(int frames) { int latent_frames = frames; - if (sd_version_is_ltxav(version)) { + if (sd_version_is_p_h3_image(version)) { + latent_frames = 1; + } else if (sd_version_is_ltxav(version)) { latent_frames = ((frames - 1) / 8) + 1; } else if (sd_version_is_minimax_h3(version)) { latent_frames = frames <= 5 ? 2 : ((frames - 5) / 17) * 5 + 2; @@ -2960,6 +2966,9 @@ int StableDiffusionGGML::latent_frames_to_video_frames(int latent_frames) { if (latent_frames <= 0) { return latent_frames; } + if (sd_version_is_p_h3_image(version)) { + return 1; + } if (sd_version_is_ltxav(version)) { return (latent_frames - 1) * 8 + 1; } @@ -2973,6 +2982,9 @@ int StableDiffusionGGML::latent_frames_to_video_frames(int latent_frames) { } int StableDiffusionGGML::align_video_frames(int frames) { + if (sd_version_is_p_h3_image(version)) { + return 1; + } if (sd_version_is_minimax_h3(version)) { frames = std::max(frames, 5); while (frames % 17 != 5) { diff --git a/src/pipeline/image.cpp b/src/pipeline/image.cpp index 607ec18c5e..d141075193 100644 --- a/src/pipeline/image.cpp +++ b/src/pipeline/image.cpp @@ -831,6 +831,75 @@ namespace sd::pipeline { return {}; } + // P-H3-Image is MiniMax-H3 generating a single frame, so each image of a batch is one run of + // the video pipeline; edit images are its reference images. + static bool generate_p_h3_image(StableDiffusionGGML* sd, + const sd_img_gen_params_t* params, + sd_image_t** images_out, + int* num_images_out, + std::vector* infos_out) { + if (params->init_image.data != nullptr || params->mask_image.data != nullptr) { + LOG_ERROR("P-H3-Image edits the images passed as reference images (--ref-image), not an init image"); + return false; + } + if (params->control_image.data != nullptr || params->ip_adapter_image.data != nullptr || + params->pm_params.id_images_count > 0 || + (params->pulid_params.id_embedding_path != nullptr && *params->pulid_params.id_embedding_path) || + params->hires.enabled) { + LOG_ERROR("P-H3-Image supports neither control, IP-Adapter, PhotoMaker, PuLID nor hires fix"); + return false; + } + const int64_t seed = resolve_seed(params->seed); + sd_vid_gen_params_t video; + sd_vid_gen_params_init(&video); + video.loras = params->loras; + video.lora_count = params->lora_count; + video.prompt = params->prompt; + video.negative_prompt = params->negative_prompt; + video.clip_skip = params->clip_skip; + video.ref_images = params->ref_images; + video.ref_images_count = params->ref_images_count; + video.ref_image_args = params->ref_image_args; + video.width = params->width; + video.height = params->height; + video.sample_params = params->sample_params; + video.video_frames = 1; + video.fps = 24; + video.vae_tiling_params = params->vae_tiling_params; + video.cache = params->cache; + video.circular_x = params->circular_x; + video.circular_y = params->circular_y; + video.image_preprocess = params->image_preprocess; + + std::vector images; + for (int b = 0; b < params->batch_count; ++b) { + if (b > 0 && sd->get_cancel_flag() != SD_CANCEL_RESET) { + break; + } + video.seed = static_cast((static_cast(seed) + b) & INT64_MAX); + sd_image_t* frames = nullptr; + int count = 0; + nlohmann::json info; + if (!generate_video(sd, &video, &frames, &count, nullptr, nullptr, &info) || count < 1) { + free_sd_images(frames, count); + free_sd_images(images.data(), static_cast(images.size())); + return false; + } + images.push_back(frames[0]); + for (int i = 1; i < count; ++i) { + free(frames[i].data); + } + free(frames); + if (infos_out != nullptr) { + infos_out->push_back(std::move(info)); + } + } + *images_out = static_cast(malloc(images.size() * sizeof(sd_image_t))); + std::copy(images.begin(), images.end(), *images_out); + *num_images_out = static_cast(images.size()); + return true; + } + bool generate_image(StableDiffusionGGML* sd, const sd_img_gen_params_t* sd_img_gen_params, sd_image_t** images_out, @@ -853,6 +922,9 @@ namespace sd::pipeline { sd_version_name(sd->version)); return false; } + if (sd_version_is_p_h3_image(sd->version)) { + return generate_p_h3_image(sd, sd_img_gen_params, images_out, num_images_out, infos_out); + } sd->reset_cancel_flag(); diff --git a/src/pipeline/request.cpp b/src/pipeline/request.cpp index d333fbb8be..399fec40c8 100644 --- a/src/pipeline/request.cpp +++ b/src/pipeline/request.cpp @@ -310,6 +310,11 @@ namespace sd::pipeline { if (!img_cfg_was_set) { guidance->img_cfg = 1.f; } + if (sd_version_is_p_h3_image(sd->version) && (guidance->txt_cfg != 1.f || guidance->img_cfg != 1.f)) { + LOG_WARN("P-H3-Image is guidance-distilled; ignoring CFG %.2f", guidance->txt_cfg); + guidance->txt_cfg = 1.f; + guidance->img_cfg = 1.f; + } if (!sd_version_supports_img_cfg(sd->version, has_ref_images)) { if (img_cfg_was_set && guidance->img_cfg != 1.f) { @@ -444,6 +449,14 @@ namespace sd::pipeline { LOG_WARN("total_steps != custom_sigmas_count - 1, set high_noise_sample_steps to %d", high_noise_sample_steps); } } + } else if (sd_version_is_p_h3_image(sd->version)) { + // The checkpoint is distilled to these eight steps; pass custom sigmas for another schedule. + sigmas = {1.f, 0.984753f, 0.967905f, 0.951688f, 0.925372f, 0.881923f, 0.810083f, 0.528718f, 0.f}; + if (sample_steps != 8) { + LOG_WARN("P-H3-Image samples its 8-step grid; ignoring %d steps (pass custom sigmas to change it)", + sample_steps); + } + sample_steps = total_steps = 8; } else { scheduler = resolve_scheduler(sd, sample_params->scheduler, diff --git a/src/pipeline/video.cpp b/src/pipeline/video.cpp index a07b93175c..3089ddf069 100644 --- a/src/pipeline/video.cpp +++ b/src/pipeline/video.cpp @@ -488,6 +488,19 @@ namespace sd::pipeline { return {align32(source_w * scale), align32(source_h * scale)}; } + // P-H3-Image keeps each edit image's own size, scaled into 1 to 8 megapixels at its aspect ratio, + // with both sides rounded to multiples of 32. + static std::pair p_h3_image_reference_size(int source_w, int source_h) { + constexpr double kMinPixels = 1024.0 * 1024.0; + constexpr double kMaxPixels = 8.0 * 1024.0 * 1024.0; + double area = static_cast(source_w) * source_h; + double scale = std::sqrt(std::clamp(area, kMinPixels, kMaxPixels) / area); + auto align32 = [](double value) { + return std::max(32, static_cast(std::round(value / 32.0)) * 32); + }; + return {align32(source_w * scale), align32(source_h * scale)}; + } + static std::optional prepare_video_generation_latents(StableDiffusionGGML* sd, const sd_vid_gen_params_t* sd_vid_gen_params, GenerationRequest* request, @@ -616,7 +629,9 @@ namespace sd::pipeline { int source_w = static_cast(image.shape()[0]); int source_h = static_cast(image.shape()[1]); auto [width, height] = - minimax_h3_reference_size(source_w, source_h, request->width, request->height, ref_image_params.ref_image_size); + sd_version_is_p_h3_image(sd->version) + ? p_h3_image_reference_size(source_w, source_h) + : minimax_h3_reference_size(source_w, source_h, request->width, request->height, ref_image_params.ref_image_size); LOG_INFO("MiniMax-H3 reference image %d: %dx%d -> %dx%d", i + 1, source_w, diff --git a/src/runtime/denoiser.hpp b/src/runtime/denoiser.hpp index 6fbcb43df3..70714025a0 100644 --- a/src/runtime/denoiser.hpp +++ b/src/runtime/denoiser.hpp @@ -1407,21 +1407,25 @@ struct MingImageFlowDenoiser : DiscreteFlowDenoiser { struct H3AVFlowDenoiser : public DiscreteFlowDenoiser { int64_t video_channels; - float audio_shift; + float audio_shift; // 0 steps the audio on the video's shift H3AVFlowDenoiser(float shift, float audio_shift, int64_t video_channels) : DiscreteFlowDenoiser(shift), video_channels(video_channels), audio_shift(audio_shift) { - GGML_ASSERT(shift > 0.f && audio_shift > 0.f && video_channels > 0); + GGML_ASSERT(shift > 0.f && audio_shift >= 0.f && video_channels > 0); + } + + float effective_audio_shift() const { + return audio_shift > 0.f ? audio_shift : shift; } sd::Tensor process_latent_in(const sd::Tensor& latent) override { - return scale_audio(latent, shift / audio_shift); + return scale_audio(latent, shift / effective_audio_shift()); } sd::Tensor process_latent_out(sd::Tensor latent) override { - auto transformed = scale_audio(latent, audio_shift / shift); + auto transformed = scale_audio(latent, effective_audio_shift() / shift); if (transformed.empty()) { return latent; }