[Feature] Tiled Diffusion

[Green-Sky]

... is often used for upscaling purposes.

Detailed Description

Often, the best upscaler is the diffusion model itself!

Whenever you want large images, you run into issues: not enough memory, numerical instability or simply the fact that the diffusion model was not trained on high res and shows sd stretch-marks or breaks down completely.

Also, since tiles are generally smaller, a larger quality quant can be used on restricted hardware.

I have heard about this now pretty often and is apparently common usage.

example for comfy: https://comfyui.dev/docs/guides/nodes/ultimate-sd-upscale/
and https://github.com/shiimizu/ComfyUI-TiledDiffusion
and https://multidiffusion.github.io/
and https://github.com/albarji/mixture-of-diffusers (!)
also this sdxl controlnet https://huggingface.co/xinsir/controlnet-tile-sdxl-1.0
flux1-dev controlnet https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Upscaler

[stduhpf]

It's not too hard to implement in a naive way, I've done it a couple times with older versions of sd.cpp already, but I felt it had too many issues to be worth making a PR.

For example when prompting for a normal portrait, you get things like that:

(of course it's less noticable when doing img2img, but still there)

Maybe there's a way to offset the positional encodings to mitigate these issues though, but I haven't figured that out.

[Green-Sky]

It's not too hard to implement in a naive way, I've done it a couple times with older versions of sd.cpp already, but I felt it had too many issues to be worth making a PR.

q.q

Maybe there's a way to offset the positional encodings to mitigate these issues though, but I haven't figured that out.

Rope scaling? Maybe, but we would get "sd stretch-marks" again.

---

I hacked together a bashscript using image-magick that tiles the image in 2x2 and upscales it. It (tries to) use a mask where there is overlap to keep the image aligned. I have only really tried sdxl yet.
I have yet to achieve an actual improvement using sdxl. But it might just be the model or parameters.

#! /usr/bin/env bash
set -e

tile_pad=128

# input image size
#W=1024
W=$(magick identify -format '%w' $1)
#H=1024
H=$(magick identify -format '%h' $1)

echo "W:$W H:$H tile_pad:$tile_pad"

# copy working copy
cp $1 tmp_input.png

#scale=2

## HACK: first downscale to 50%, to hide vae/diffusion/quant artifacts
## actually better to do upscale first, then downscale back to og after
#magick tmp_input.png -resize 50% tmp_input.png
#(( W/=2 ))
#(( H/=2 ))

# 2x
magick tmp_input.png -resize 200% tmp_input.png
# w and h are now tile size (without pad)


# splint into 2x2 tiles
magick tmp_input.png -crop 2x2+$((tile_pad*2))+$((tile_pad*2))@ tmp_tile.png

# TODO: add overlap in addition to padding

# gen masks
# keep in mind that w/h are already 0.5x

# mask off right+bottom
magick -size $((W))x$((H)) canvas:white -background black -gravity Southeast -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-0.png
# mask off left+bottom
magick -size $((W))x$((H)) canvas:white -background black -gravity Southwest -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-1.png
# mask off right+top
magick -size $((W))x$((H)) canvas:white -background black -gravity Northeast -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-2.png
# mask off left+top
magick -size $((W))x$((H)) canvas:white -background black -gravity Northwest -splice $((tile_pad))x$((tile_pad)) tmp_tile_mask-3.png

echo "tiles and masks generated."

sd(){
	result/bin/sd-cli -m models/cyberrealisticXL_v80-q8_0.gguf --cfg-scale 3 --steps 36 --scheduler exponential --sampling-method dpm++2m -n "lowres, bad quality, worst quality, artistic error, wrong hand, wrong foot, bad anatomy, bad feet, bad reflection, bad perspective, bad arm, bad proportions, bad hands, bad leg, watermark, signature, dated, artist name, copyright name, sketch, censored, facial mark, scan" --diffusion-fa --offload-to-cpu --vae-tiling --vae-tile-size 3x3 --preview proj "$@"
}

prompt='An ultra-realistic 8k HDR editorial photograph. Sharp focus, 8K resolution, detailed texture.'
strength="0.35"

sd -p "$prompt" --strength $strength -i tmp_tile-0.png --mask tmp_tile_mask-0.png -o tmp_tile_diff-0.png
echo "-----------------------"
sd -p "$prompt" --strength $strength -i tmp_tile-1.png --mask tmp_tile_mask-1.png -o tmp_tile_diff-1.png
echo "-----------------------"
sd -p "$prompt" --strength $strength -i tmp_tile-2.png --mask tmp_tile_mask-2.png -o tmp_tile_diff-2.png
echo "-----------------------"
sd -p "$prompt" --strength $strength -i tmp_tile-3.png --mask tmp_tile_mask-3.png -o tmp_tile_diff-3.png
echo "-----------------------"

echo "diffusion done, merging..."

# TODO: blended merging
#magick \( tmp_tile_diff-0.png tmp_tile_diff-1.png +smush -$((tile_pad*2)) \) \( tmp_tile_diff-2.png tmp_tile_diff-3.png +smush -$((tile_pad*2)) \) -smush -$((tile_pad*2)) result.png

# alt crop
magick tmp_tile_diff-0.png -gravity Northwest -extent $((W))x$((H)) +repage tmp_tile_diff_crop-0.png
magick tmp_tile_diff-1.png -gravity Northeast -extent $((W))x$((H)) +repage tmp_tile_diff_crop-1.png
magick tmp_tile_diff-2.png -gravity Southwest -extent $((W))x$((H)) +repage tmp_tile_diff_crop-2.png
magick tmp_tile_diff-3.png -gravity Southeast -extent $((W))x$((H)) +repage tmp_tile_diff_crop-3.png

magick \( tmp_tile_diff_crop-0.png tmp_tile_diff_crop-1.png +smush 0 \) \( tmp_tile_diff_crop-2.png tmp_tile_diff_crop-3.png +smush 0 \) -smush 0 result.png

Also there is no blending, so the seam is very visible.

input:

(ernie-image is great)

output: