Offload canvas drawing operations to a worker - #20729
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #20729 +/- ##
==========================================
- Coverage 90.01% 89.18% -0.84%
==========================================
Files 264 265 +1
Lines 66893 67263 +370
==========================================
- Hits 60214 59987 -227
- Misses 6679 7276 +597
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
/botio test |
From: Bot.io (Windows)ReceivedCommand cmd_test from @nicolo-ribaudo received. Current queue size: 0 Live output at: http://54.193.163.58:8877/e0227fb9789b076/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @nicolo-ribaudo received. Current queue size: 0 Live output at: http://54.241.84.105:8877/b6e4acc21cb49a8/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/b6e4acc21cb49a8/output.txt Total script time: 46.91 mins
Image differences available at: http://54.241.84.105:8877/b6e4acc21cb49a8/reftest-analyzer.html#web=eq.log |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/e0227fb9789b076/output.txt Total script time: 90.47 mins
Image differences available at: http://54.193.163.58:8877/e0227fb9789b076/reftest-analyzer.html#web=eq.log |
03ad6c3 to
1afc3d4
Compare
|
I was looking through the reftest failures, it seems like the only real ones (the others are minor pixel differences due to the different rendering pipeline, but not visible to humans) are:
|
1d6de6c to
43a73d7
Compare
There's also gradient difference in 17069, also there are differences in 19022, same as issue8092 |
|
/botio test |
From: Bot.io (Windows)ReceivedCommand cmd_test from @nicolo-ribaudo received. Current queue size: 0 Live output at: http://54.193.163.58:8877/832219967f9b55d/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @nicolo-ribaudo received. Current queue size: 0 Live output at: http://54.241.84.105:8877/659eb95d2835e52/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/659eb95d2835e52/output.txt Total script time: 46.69 mins
Image differences available at: http://54.241.84.105:8877/659eb95d2835e52/reftest-analyzer.html#web=eq.log |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/832219967f9b55d/output.txt Total script time: 78.06 mins
Image differences available at: http://54.193.163.58:8877/832219967f9b55d/reftest-analyzer.html#web=eq.log |
134955c to
ba0fa2c
Compare
|
/botio test |
From: Bot.io (Windows)ReceivedCommand cmd_test from @nicolo-ribaudo received. Current queue size: 0 Live output at: http://54.193.163.58:8877/528a274cb665c1e/output.txt |
From: Bot.io (Linux m4)ReceivedCommand cmd_test from @nicolo-ribaudo received. Current queue size: 0 Live output at: http://54.241.84.105:8877/67316c0b179c2dc/output.txt |
From: Bot.io (Linux m4)FailedFull output at http://54.241.84.105:8877/67316c0b179c2dc/output.txt Total script time: 46.49 mins
Image differences available at: http://54.241.84.105:8877/67316c0b179c2dc/reftest-analyzer.html#web=eq.log |
From: Bot.io (Windows)FailedFull output at http://54.193.163.58:8877/528a274cb665c1e/output.txt Total script time: 75.41 mins
Image differences available at: http://54.193.163.58:8877/528a274cb665c1e/reftest-analyzer.html#web=eq.log |
96dda63 to
cca0198
Compare
GitHub Actions
Inactive
| * @param {RendererWorkerParameters} params - The worker initialization | ||
| * parameters. | ||
| */ | ||
| class RendererWorker { |
There was a problem hiding this comment.
It doesn't appear that the verbosity parameter is sent to the worker-thread, compare with the existing PDFWorker implementation, why not?
There was a problem hiding this comment.
This now gets sent to renderer worker.
|
The new |
| ? "resource://pdf.js/build/pdf.renderer.mjs" | ||
| : "../build/pdf.renderer.mjs"; | ||
| } | ||
|
|
There was a problem hiding this comment.
Why is this code placed here, since it really ought to be handled in the same way as the workerSrc instead?
Lines 553 to 563 in e9a946e
There was a problem hiding this comment.
Removed this block with build-target defaults was removed from api.js; RendererWorker.rendererSrc now throws if GlobalWorkerOptions.rendererSrc is unset, mirroring PDFWorker.workerSrc.
If rendererSrc is unset, the throw is caught, the worker capability rejects, and just disables worker rendering with a warning.
There was a problem hiding this comment.
As mentioned in #20729 (comment) the RendererWorker must report coverage data; note how #20729 (comment) reports significantly reduced coverage with this patch.
Edit: Also, the commit history should be cleaned-up before this lands by folding any "fixup" commits into their parent commits, in order to keep the commit history clean.
| renderTaskId: this._renderTaskId, | ||
| enableHWA: this._enableHWA, | ||
| enableWebGPU: this._enableWebGPU, | ||
| optionalContentConfig: optionalContentConfig?.serializable ?? null, |
There was a problem hiding this comment.
The optionalContentConfig must be available here, otherwise there's a bug somewhere else.
| optionalContentConfig: optionalContentConfig?.serializable ?? null, | |
| optionalContentConfig: optionalContentConfig.serializable, |
| const optionalContentConfig = data.optionalContentConfig | ||
| ? OptionalContentConfig.fromSerializable(data.optionalContentConfig) | ||
| : new OptionalContentConfig(null); |
There was a problem hiding this comment.
Again, it shouldn't be possible for the optionalContentConfig-data to be undefined here.
| const optionalContentConfig = data.optionalContentConfig | |
| ? OptionalContentConfig.fromSerializable(data.optionalContentConfig) | |
| : new OptionalContentConfig(null); | |
| const optionalContentConfig = | |
| OptionalContentConfig.fromSerializable(data.optionalContentConfig); |
| } | ||
| internalRenderTask.initializeGraphics({ | ||
| const { transparency, hasCanvasFilters = false } = | ||
| typeof renderPageData === "object" && renderPageData !== null |
There was a problem hiding this comment.
Nit: This can be shortened.
| typeof renderPageData === "object" && renderPageData !== null | |
| renderPageData && typeof renderPageData === "object" |
I'll do it in a follow-up. |
| static #appendOperatorList(renderTaskState, fnArray, argsArray, lastChunk) { | ||
| const { operatorList } = renderTaskState; | ||
| if (fnArray) { | ||
| operatorList.fnArray.push(...fnArray); |
There was a problem hiding this comment.
It could fails with a large array, so it should be a little more robust.
| #worker = null; | ||
|
|
||
| #rendererHandler = null; | ||
|
|
||
| #capability = Promise.withResolvers(); |
There was a problem hiding this comment.
Why are these fields not sorted alphabetically?
| class RendererWorker { | ||
| #worker = null; | ||
|
|
||
| #rendererHandler = null; |
There was a problem hiding this comment.
Why is this not called #messageHandler, since that's what it'll contain?
| try { | ||
| const { rendererSrc } = RendererWorker; | ||
| const worker = new Worker(rendererSrc, { type: "module" }); |
There was a problem hiding this comment.
Given that this code looks like it was just copied from the existing PDFWorker implementation, why isn't this fully consistent with the following code?
| try { | |
| const { rendererSrc } = RendererWorker; | |
| const worker = new Worker(rendererSrc, { type: "module" }); | |
| let { rendererSrc } = RendererWorker; | |
| try { | |
| // Wraps rendererSrc path into blob URL, if the former does not belong | |
| // to the same origin. | |
| if ( | |
| typeof PDFJSDev !== "undefined" && | |
| PDFJSDev.test("GENERIC") && | |
| !PDFWorker._isSameOrigin(window.location, rendererSrc) | |
| ) { | |
| rendererSrc = PDFWorker._createCDNWrapper( | |
| new URL(rendererSrc, window.location).href | |
| ); | |
| } | |
| const worker = new Worker(rendererSrc, { type: "module" }); |
|
The bug and the patch for updating the number of prefs: https://bugzilla.mozilla.org/show_bug.cgi?id=2056102 |
| let initPromise = null; | ||
| if (useWorkerRendering) { | ||
| try { | ||
| const offscreen = this._canvas.transferControlToOffscreen(); |
There was a problem hiding this comment.
Once the control is transferred to the offscreen canvas it cannot be done again so I think it could be break the possibility of reusing the same canvas but in such a case we can reuse the associated offscreen one.
| processed.put(graphicState.objId); | ||
| } | ||
| try { | ||
| if (this._hasTransferMaps(graphicState.get("TR"))) { |
There was a problem hiding this comment.
I recently added support for TR2 too
| } | ||
| } | ||
|
|
||
| const xObjects = node.get("XObject"); |
There was a problem hiding this comment.
You only visit XObject but a TR/TR2 can be in a tiling pattern too.
Snuffleupagus
left a comment
Snuffleupagus
left a comment
There was a problem hiding this comment.
Given that this PR consists of a number of commits, please make sure that every single one of them works correctly on their own. Hence ensure, by testing locally, that all tests pass when run against each commit.
This is imperative to make sure that it's possible to bisect any future regressions to an exact commit, since the total size of the PR would otherwise make that really difficult.
|
@Snuffleupagus I have checked by testing locally, that all tests pass when run against each commit. |
Considering all of the "fixup" commits here, which must be folded into their appropriate parent commits, it sounds very surprising that every single commits works! |
Move the commonobj/obj resolution logic from WorkerTransport.setupMessageHandler into a reusable ObjectHandler class. This enables sharing the object resolution logic between the main thread (WorkerTransport) and the renderer worker. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Introduce the RendererWorker class for offloading canvas rendering to a dedicated Web Worker. Alongside, it adds RendererMessageHandler, GlobalWorkerOptions.rendererSrc configuration, entrypoints for pdf.renderer.js bundle and build targets in gulpfile. No rendering changes are introduced in this commit, this is a setup for later commits that wire-up graphics execution and object forwarding. The `disableWorkerRendering` option defaults to disabled and is flipped in the final commit of this series. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Add a hasCanvasFilters method to PartialEvaluator that walks the page-level ExtGState dictionaries, and those of Form XObject and tiling pattern resources, to detect transfer functions (TR/TR2) that require DOM SVG filters. Such filters are unavailable on OffscreenCanvas, so detecting them up front lets the display layer fall back to main-thread rendering for affected pages. The flag rides along on the existing StartRenderPage message down to initializeGraphics; the rendering decision that consumes it is added later in this series. SMask rendering already has a pixel-buffer fallback in canvas.js, and TR inside Type3 glyph streams or annotation appearance streams is rare enough in practice that walking those sub-resources (and gating first paint on annotation parsing) isn't worth it. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
The renderer worker has no PDFPageProxy objects, only per-page PDFObjects instances, so accept a page cache holding either and create missing entries on demand behind `shouldCreatePageObjs`. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Forward the commonobj/obj messages that WorkerTransport receives on to the renderer worker, so that it builds up the same commonObjs and per-page objs as the main thread. CopyLocalImage is handled separately, since the core worker sends only an image reference: the renderer worker is asked to resolve it from its own objects first, and only when it can't does the main thread send the image data it just found. A forwarded object that cannot be delivered is reported back as `objFailed`, so the renderer worker rejects it rather than waiting forever on a dependency that will never arrive. The renderer worker's object stores are released by the cleanupPage and Cleanup handlers added later in this series. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Since operator lists will be posted across threads, Path2D objects can no longer be materialized into argsArray; they are cached in a pathCache map on the operator list instead, keeping it structured-cloneable. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Look up the annotation canvas in `annotationCanvasMap` before creating a new one, and resize it in place when it is already there. This lets a canvas that was created (and possibly transferred) elsewhere be drawn into, rather than being replaced by a fresh one. Track the canvas name on the object as well as in the DOM attribute, so that the same matching works for canvases without a DOM node. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Add the handlers that let the renderer worker initialize graphics and execute an operator list against a transferred OffscreenCanvas: InitializeGraphics, ExecuteOperatorList, UpdateAnnotationCanvases, CleanupRenderTask, ReleaseCanvas, cleanupPage, restorePage and Cleanup, along with the per-render-task state they operate on. Also adds the OffscreenCanvas and worker-side filter factories, and lets CanvasGraphics.executeOperatorList report a failed object dependency through an errorCallback, so a rejected object aborts the render instead of hanging it. Nothing sends these messages yet: src/pdf.renderer.js is the only importer of this file, so no main-thread code path reaches it. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Decide whether to use worker rendering based on hasCanvasFilters, pageColors and the debug-recording path, transfer the canvas via transferControlToOffscreen and send the operator list to the renderer worker in chunks, along with any annotation canvases the list refers to. The recorded bounding boxes and image coordinates now come back from the worker in the final ExecuteOperatorList response, so the trackers are built inside initializeGraphics rather than by the caller. Since a canvas can only be transferred once, the OffscreenCanvas is tracked per canvas id and reused when the same canvas is re-rendered. The gate added here is off by default, so nothing takes this path yet; it is flipped in the final commit of this series. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
A canvas whose control has been transferred to the renderer worker can no longer be resized or re-acquired on the main thread, so releasing it has to go through the worker instead of setting width/height to zero, and it cannot be reused as the source for a thumbnail. This is inert while the gate is off, since resetWorkerCanvas is never set and releaseCanvas behaves exactly as the previous width = height = 0. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Render into a separate canvas in the reftest driver and copy the result back, since a canvas can only be transferred once, and read pixels through createImageBitmap in the integration helpers for canvases whose context can no longer be acquired. Nothing here enables worker rendering; GlobalWorkerOptions.rendererSrc is set in the final commit of this series, together with the library and viewer defaults. Every change in this commit behaves identically on the main thread. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Thread the enableWebGPU flag from getDocument() through WorkerTransport and InternalRenderTask to the renderer worker's InitializeGraphics handler, where it triggers GPU device initialization. The main thread already waits for InitializeGraphics to resolve before sending any operators, so the GPU device is ready by the time drawing starts. This commit is a part of the renderer-worker series, the worker rendering stays disabled until the final commit in this series.
Flip disableWorkerRendering to opt-out in the API and to false in the
viewer, and point the reftest driver and the unit tests at the renderer
worker bundle, so the whole suite exercises this path from here on.
This changes the default for API consumers: the canvas passed to
render() has its control transferred to the renderer worker, so calling
getContext("2d") on it afterwards will throw. Pass
disableWorkerRendering: true to opt out.
Getting worker rendering also requires GlobalWorkerOptions.rendererSrc to
point at the pdf.renderer.mjs bundle. The viewer and the pdfjs-dist
webpack entry set it automatically, but integrators who bundle the library
themselves must set it as well; when it's unset, or the renderer worker
fails to start, rendering falls back to the main-thread with a warning
rather than failing.
Thumbnails keep rendering on the main thread by passing a canvasContext
rather than a canvas. Note that this also means InternalRenderTask no
longer sees the canvas, so the "same canvas during multiple render()
operations" guard does not cover thumbnails.
This commit is the final commit of the renderer-worker series, it enables
the worker rendering that the previous commits kept disabled.
Mirror the existing nonBlendModesSet with a nonCanvasFiltersSet, so that resources already proven filter-free aren't walked again on subsequent pages. A separate set is needed since hasBlendModes descends into Form XObjects only, while hasCanvasFilters also walks tiling patterns. Replace _hasTransferMaps with _getTransferFunctions, now shared with handleTransferFunction, since the two only differed in whether the 256-entry transfer maps get built. This commit is a part of the renderer-worker series.
|
@Snuffleupagus Anyway, to address the problem I mentioned earlier that there was a disparity between when feature was enabled vs when we actually test the feature, I have moved around things a little bit, the branch tip stays the same as before, any differences are cosmetic, and the content is close to what you reviewed, with one new commit at the tip Now, the second-to-last commit is the one that actually enables worker-rendering and all commits before that are setting the stage to be able to enable that. Until the 12th commit, the main thread rendering works as expected and on enabling the worker rendering in the 13th commit, we actually test the feature: The tests that have been run locally on all 14-commits branch (worker rendering is off until commit 13):
(Note: They were run only on Firefox locally on my MacOS 26.5.1) Bisecting is a little complex but the above approach does achieve the following: A few preparatory commits modify code the existing main-thread renderer runs today:
Those are refactors of current code paths. Their tests passing proves they didn't regress current rendering. Commit 13 is then the only place where behaviour changes at all, which is a clean thing to review. Coming to the newer commit division, there are 14 commits, all commits have a "This commit is a part of the renderer-worker series." to make them easier to search, I wasn't sure if to tag them differently. I will update the PR description with the following details: Commit Details
I believe all the review comments should be addressed now. |
| transform, | ||
| viewport, | ||
| transparency, | ||
| background, |
There was a problem hiding this comment.
According to the jsdoc, background could be a pattern or a gradient which are neither clonable nor transferable.
8 participants
Commit descriptions
Note that commit hashes might change in the future due to commit edits later
1. 34ab2b6: Extract ObjectHandler from WorkerTransport …
Notable changes:
this.shouldCreatePageObjsis added in object handler, it does not affect the main-thread rendering, it is set to false, but for worker-rendering, the renderer worker keeps only Map<pageIndex, PDFObjects>, not PDFPageProxy instances. So object_handler.js (line 126) has to handle both shapes.The
shouldCreatePageObjspart exists because renderer-worker obj messages can arrive before InitializeGraphics has called#getPageObjs(pageIndex). In that case the renderer still must cache the image/pattern object, otherwise later CanvasGraphics will hit an unresolved dependency while executing the operator list.Relevent code:
Open questions:
ObjectHandlerclass?2. 741ca8d: Adds RendererWorker class for offloading canvas …
Notable changes
This commit just sets up the renderer worker while following the same pattern as PDFWorker setup.
disableWorkerRenderingfor disabling the worker-rendering. Note that the flag is only checked once to check whether we should set up theRendererHandler, all the further decisions to use the worker-rendering are deferred to whetherRendererHandleris notnull.WorkerAPI,OffScreenCanvasis not supported, a customownerDocumentis present, since custom ownerDocument can be iframe document etc. which cannot be transferred to a worker and the worker cannot create DOM nodes inside it. Same forstyleElement, it is used byFontLoader, and is also a DOM element.In the renderer worker, fonts would instead be loaded via the FontFace API (self.fonts.add()). When a custom styleElement is provided (testing scenarios), it forces CSS-based font loading instead of the FontFace API. Since CSS font loading doesn't work in a worker context, the renderer worker can't render fonts correctly if styleElement is in use.
3. c28992d: Add canvas filter detection in core layer …
Due to bug https://bugzilla.mozilla.org/show_bug.cgi?id=2011237, since there's no DOM access from a worker the filter has to be defined with an external URL which is not currently supported in
OffscreenCanvasRenderingContext2DNotable changes:
hasCanvasFiltersmethod is very similar tohasBlendModeswhere both of these methods traverse the resource graph to check the presence of canvas filters.hasCanvasFiltershowever returnstrueconservatively. I considererd reusing some of the code fromhasBlendModesbut that made the patch more complex and hard to read.4. ffe8e9a: Add object forwarding between main thread and renderer worker …
Open questions
At present the way renderer works is we use the main thread for forwarding the objs/commonObjs and font fallback instead of PDFworker directly sending it to the renderer worker. So
objs/commonObjsare duplicated and also adds an additional hop.The commonobjs/objs are still being duplicated for sending to each of main-thread and renderer, which can be fixed in the following two ways:
a. Use SAB
b. Move the InternalRenderTask entirely to the renderer worker, I think this is possible and this is the approach I would prefer, to not have main thread deal with objs/commonObjs at all, but this is a much larger refactor which I am not sure should be a part of this patch.
While fixing a. appears to be somewhat easy, I have tried with SAB and the current version of sending to both main and renderer is not the best approach because there is a race-condition that it introduces that causes browser tests to almost always timeout, I've tried fixing several but so far, the tests still timeout, if we want to remove the forwarding, I can spend more time looking into forwarding, however I think if we remove the dependency on main thread entirely, it should fix both issues.
For now there's a TODO comment to remove the forwarding in the future.
5. 9ddbd41: Add graphics initialization and operator list execution in renderer worker …
Notable changes
a.
keepRendererCanvas: This flag is added to to still clear normal page state, but ask the renderer worker to keep the transferred canvas alive. This lets PDF.js free operator lists, page objects, fonts/images tied to the page, etc., without making already-rendered visible pages go blank when scrolling etc.b. In main-thread rendering, CanvasGraphics gets the actual OptionalContentConfig instance directly. In renderer-worker rendering, that object cannot be sent as-is: it has class methods/private fields. There is a change to pass the plain data we receive from PDFWorker and rebuilding the object on the worker side.
c. We also need to transfer the annotation canvases, so we find the annotations with
hasOwnCanvas, and send it to renderer worker. It also caches the transferred canvases in_transferredAnnotationCanvasIds.d. Worker rendering is disabled when we have canvas filters and page colors for reasons described above and it falls back to main-thread rendering. It is also disabled when dependencyTracker and imagesTracker are present, this would require making them transferable, which is not too complex and can be done in future iterations.
e. Operator list is sent to renderer worker in chunks. Sending the full growing operator list every time would be very expensive. So the main thread only sends the new tail of the operator list: from the last sent index to the current length. After the renderer worker receives it, it appends that partition to its own worker-local operator list.
7. f085c07: Adapt viewer and tests for OffscreenCanvas renderer worker …
Notable changes
getImageDataon the temporary canvas.