Bundling with Webpack

[Bessonov]

Possible install-time or require-time problem

• I have read and understood all of the documentation relating to installation.
• I have searched for known bugs relating to this problem in my choice of package manager.

You must confirm both of these before continuing.

Are you using the latest version of sharp?

I first tried with the latest version (0.34.1), but saw here some current issues, so now I'm trying 0.33.5 instead.

Are you using a supported runtime?

• I am using Node.js with a version that satisfies ^18.17.0 || ^20.3.0 || >=21.0.0: v22.14.0

Are you using a supported package manager and installing optional dependencies?

• I am using pnpm >= 7.1.0 with --no-optional=false: 10.6.5

What is the complete error message, including the full stack trace?

What is the complete output of running npm install --verbose --foreground-scripts sharp in an empty directory?

What is the output of running npx envinfo --binaries --system --npmPackages=sharp --npmGlobalPackages=sharp?

First of all, thank you for the amazing library!

I’m packaging my app using Webpack and generate a single server.mjs file along with the necessary *.node binaries. This helps significantly reduce the size of my Docker image (few MBs vs. ~700MB). This approach previously worked well with sharp, even in Lambda environments.

However, I've noticed some changes that seem to interfere with this setup.

I’ve read through the documentation on pnpm and webpack. However, when using:

externals: {
  'sharp': 'commonjs sharp'
}

...this assumes the library is discoverable at runtime, typically via node_modules. But in my setup, after bundling, that’s not the case.

From what I understand, this limitation stems from the way prebuilt binaries are resolved:

const paths = [
  `../src/build/Release/sharp-${runtimePlatform}.node`,
  '../src/build/Release/sharp-wasm32.node',
  `@img/sharp-${runtimePlatform}/sharp.node`,
  '@img/sharp-wasm32/sharp.node'
];

let sharp;
const errors = [];
for (const path of paths) {
  try {
    sharp = require(path);

This dynamic resolution makes it difficult for bundlers to include the relevant .node binaries, as these paths aren’t statically analyzable.

A possible workaround would be to hardcode the potential paths to make them discoverable during bundling, for example:

let sharp = undefined;
try { sharp = require(`../src/build/Release/sharp-linux-x64.node`,); } catch {...}
if (!sharp) try { sharp = require(`../src/build/Release/sharp-linux-arm.node`,); } catch {...}
if (!sharp) try { sharp = require(`../src/build/Release/sharp-linux-arm64.node`,); } catch {...}
if (!sharp) try { sharp = require(`@img/sharp-linux-x64/sharp.node`,); } catch {...}
[...]

Yes, it's tedious and not very elegant, but I haven’t found a better solution so far. I think it's acceptable to start with an initial list and expand it as needed.

Additionally, we could fall back to the current dynamic require logic at the end, preserving backward compatibility.

Does this approach make sense? Am I missing a better alternative?

Thanks again for the great work on sharp!

[lovell]

The relative paths between shared library binaries stored within node_modules are important, so even with the suggested changes above, bundling all the .node files together is not guaranteed to work.

What we could do is partly implement this by require-ing the @img/sharp-wasm32 WebAssembly package using a non-dynamic location, as that is a good fallback for almost all possible environments, plus should support whatever file system layout a bundler can throw at it.

[Bessonov]

@lovell

The relative paths between shared library binaries stored within node_modules are important, so even with the suggested changes above, bundling all the .node files together is not guaranteed to work.

Thanks! So you mean that the binaries communicate directly with each other and expect a relative - but exact - location? That's unexpected. I also see that libvips is included in the same way. Where can I learn more about this issue? I think I can configure Webpack to store the binaries at a specific path.

What we could do is partly implement this by require-ing the @img/sharp-wasm32 WebAssembly package using a non-dynamic location, as that is a good fallback for almost all possible environments, plus should support whatever file system layout a bundler can throw at it.

I'm fine with that approach. Does it have any drawbacks on Node.js, like performance degradation?

[Bessonov]

I have deployed my state to a test environment so it can be tested end to end. It seems to be working - at least for my use case with version 0.33.5. To replace the dynamic require with a hardcoded value, I'm using the following loader:

	module: {
		rules: [
			{
				test: /sharp\.js$/,
				loader: 'string-replace-loader',
				options: {
					search: 'sharp = require(path)',
					replace: "sharp = require('@img/sharp-linux-x64/sharp.node')",
				},
			},

[lovell]

Are you able to apply and test the following diff to see if that works for the scenario you (and hopefully others) are using? Thank you.

--- a/lib/sharp.js
+++ b/lib/sharp.js
@@ -13,8 +13,7 @@ const runtimePlatform = runtimePlatformArch();
 const paths = [
   `../src/build/Release/sharp-${runtimePlatform}.node`,
   '../src/build/Release/sharp-wasm32.node',
-  `@img/sharp-${runtimePlatform}/sharp.node`,
-  '@img/sharp-wasm32/sharp.node'
+  `@img/sharp-${runtimePlatform}/sharp.node`
 ];
 
 let path, sharp;
@@ -29,6 +28,15 @@ for (path of paths) {
   }
 }
 
+/* istanbul ignore next */
+if (!sharp) {
+  try {
+    sharp = require('@img/sharp-wasm32/sharp.node');
+  } catch (err) {
+    errors.push(err);
+  }
+}
+
 /* istanbul ignore next */
 if (sharp && path.startsWith('@img/sharp-linux-x64') && !sharp._isUsingX64V2()) {
   const err = new Error('Prebuilt binaries for linux-x64 require v2 microarchitecture');

[lovell]

@Bessonov Were you able to test the suggested change above?

[Bessonov]

@lovell Oh, I'm very sorry - it seems I missed the notification. I hope I can try it tomorrow.

[Bessonov]

@lovell It doesn't work out of the box.

First, I get the following error:

main-backend-1  | file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/sharp.js:131
main-backend-1  |   throw new Error(help.join('\n'));
main-backend-1  | ^
main-backend-1  | Error: Could not load the "sharp" module using the linux-x64 runtime
main-backend-1  | ENOENT: ENOENT: no such file or directory, open '/home/dev/app/packages/main-backend/dist/sharp-wasm32.node.wasm'

This happens because sharp-wasm32.node.js uses a plain path:

function findWasmBinary(){return locateFile("sharp-wasm32.node.wasm")}

If I copy the file manually, then I get this error:

main-backend-1  | file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/sharp.js:131
main-backend-1  |   throw new Error(help.join('\n'));
main-backend-1  | ^
main-backend-1  | Error: Could not load the "sharp" module using the linux-x64 runtime
main-backend-1  | Unsupported CPU: Prebuilt binaries for linux-x64 require v2 microarchitecture
[...]

This is due to an outdated path value and the following check:

if (sharp && path.startsWith('@img/sharp-linux-x64') && !sharp._isUsingX64V2()) {

After fixing that, I'm facing the following issue:

const logo = sharp(await uploadedFile.bytes()) // takes ~1ms, same as with sharp-linux-x64
const metadata = await logo.metadata() // not finished after >10 minutes; ~2ms with sharp-linux-x64
await logo.resize(...).toFile(...) // not finished after >10 minutes; ~24ms with sharp-linux-x64

There's nothing unusual in top or similar tools - no sign of high CPU or memory usage ¯\(ツ)/¯

Any ideas?

[Bessonov]

I have deployed my state to a test environment so it can be tested end to end. It seems to be working - at least for my use case with version 0.33.5. To replace the dynamic require with a hardcoded value, I'm using the following loader:

BTW, the workaround doesn't work with 0.34.2:

main-backend-1  | file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/utility.js:14
main-backend-1  | const libvipsVersion = sharp.libvipsVersion();
main-backend-1  | ^
main-backend-1  | TypeError: sharp.libvipsVersion is not a function
main-backend-1  |     at Object.../../.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/utility.js (file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/utility.js:14:1)
main-backend-1  |     at __webpack_require__ (file:///home/dev/app/packages/main-backend/dist/file:/webpack/bootstrap:19:1)
main-backend-1  |     at Object.../../.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/index.js (file:///home/dev/app/packages/main-backend/dist/file:/home/dev/app/.cache/pnpm/virtual-store-dir/sharp@0.34.2/node_modules/sharp/lib/index.js:14:1)
main-backend-1  |     at __webpack_require__ (file:///home/dev/app/packages/main-backend/dist/file:/webpack/bootstrap:19:1)
[...]

[lovell]

Thanks for the research.

First, I get the following error... This happens because sharp-wasm32.node.js uses a plain path:

This looks like it might relate to emscripten-core/emscripten#22140 and suggests we need to try the emscripten EXPORT_ES6 flag, although there's a follow-up issue at emscripten-core/emscripten#22508 that remains open.

[Bessonov]

I haven't tried it yet, but I think I can work around it. However, I have no idea why it hangs...

[Rush]

Also experiencing this problem and the workarounds above don't help.

[Abraham-Flutterwave]

Getting this today!

[Rush]

We're using this workaround:

        {
          test: /sharp\/lib\/sharp\.js$/,
          loader: 'string-replace-loader',
          options: {
            search: 'sharp = require(path)',
            replace:  existsSync(resolveModuleDir('sharp', `./src/build/Release/sharp-${process.platform}-${process.arch}.node`))
              ? `sharp = require('../src/build/Release/sharp-${process.platform}-${process.arch}.node')` :
                `sharp = require('@img/sharp-${process.platform}-${process.arch}/sharp.node')`,
          },
        },

[Bessonov]

@Rush @Abraham-Flutterwave Don't forget to vote on this issue. I'm not expecting it to make a big difference, but not voting definitely won't help.