Document the image processing limits - #1208
Open
ajwild wants to merge 19 commits into
Open
Conversation
Covers the new processing.timeout and processing.maxConcurrentResizes options, the raised upload timeout, and the one-thread-per-image change.
Both limits and the thread pool they share have to be set together, and none of them can be derived from the host's core count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The legacy transform runs while the result is streamed to storage, outside the maxConcurrentProcesses queue, so the thread pool is the only limit on it — which makes a large thread pool a memory risk rather than a free win. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The thread pool, not the two concurrency options, is what decides how many images are processed at once, so the sizing advice follows from both together. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Link both configuration sections instead of restating how the limits interact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-frame limit and the frame count multiplied out to far more than either suggests, which is what the new option bounds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-image figure was measured at roughly 200MB. It is about 4 bytes per pixel of every frame, so a 200 megapixel animation takes 800MB, and the sizing guidance was recommending combinations that exceed a container's memory by an order of magnitude.
Still images stream, so their cost follows the variant being produced rather than the source. Only one of the two paths decodes anything: variants with use2025Behavior, uploads without it, never both.
maxTotalResolution bounds raster work to well under 20 seconds, so the hour-long processing the timeout was described as preventing is no longer reachable. SVG rendering cost is not measured by any pixel limit.
The headroom under the timeout is a function of maxTotalResolution, which is meant to be raised. Processing time scales with it linearly, so the timeout is what still holds at a high or unbounded limit.
The legacy transform now writes its result to uploadProcessingDirectory as well, so the volume holds two files per in-flight upload rather than one.
Sizing the concurrency limits from the memory table forces the count to the largest image the resolution limit allows; the budget states the limit directly.
The memory bound reads the container's limit at startup and the variant limit follows the thread pool, so neither is a number an operator has to translate from a container size any more. The sizing guidance shrinks to the two choices that are left.
The memory budget took over the decode, so lowering the limit no longer relieves memory pressure. What it still bounds is temp disk and concurrent writes to storage.
JordiVM
reviewed
Sep 1, 2026
JordiVM
reviewed
Sep 4, 2026
|
|
||
| Image processing runs on Node.js' thread pool, which is also what every file, DNS and compression operation in the process uses. Its default size is **4 threads**. An image takes a thread once one is free and then holds it for the whole operation — several seconds for a large animated image. | ||
|
|
||
| Keep `UV_THREADPOOL_SIZE` a little above the container's CPU limit, which `os.availableParallelism()` reports. Do not use `os.cpus().length`: it reports the host's cores and ignores the container's CPU limit. The headroom stops file and DNS operations from queueing behind image processing, while a much larger pool mostly buys memory — for a screenful of uncached animated thumbnails, raising it from `4` to `16` took the time from 12.8s to 8.8s and the peak from 2.6GB to 8.2GB. The memory reservation is what holds that second figure down now, but the pool is still what decides whether the extra threads are worth having. |
Contributor
There was a problem hiding this comment.
Just tested it, it reports the same with both commands :)
> os.availableParallelism()
2
> os.cpus().length
2
It shows the Container CPU limit
Contributor
Author
There was a problem hiding this comment.
I expect that Kubernetes handles things better (some kind of system level isolation) than the testing I did with local docker containers.
Contributor
|
In summary, we have to define a sensible |
Contributor
Author
|
It's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the image processing limits added in these server PRs:
Based on #1207, which introduces the section it extends — review that one first.
mediaLibrary.images.processing.timeoutandmediaLibrary.images.processing.maxConcurrentResizesoptions.os.availableParallelism()rather thanos.cpus().lengthis the number to size against, and recommended values per container CPU limit.maxConcurrentProcessesas covering an upload end to end, which is what it does as of server PR #9865.uploadProcessingDirectorynow holds every transformed upload rather than only large animated ones, and corrects the volume sizing guidance: the worst case is 2GB rather than 300MB for themaxFileSize: 100mbthat customers commonly configure.