Fix integer overflow in quantize_pngquant()

[myagmartseren]

Integer Overflow → Heap Buffer Overflow in QuantPngQuant.c

Summary

quantize_pngquant() allocates memory using width * height (both unsigned int) without checking for integer overflow. When the product exceeds UINT_MAX, it wraps to a small value, causing an undersized malloc(). Subsequent liq_write_remapped_image_rows() writes the real size into the small buffer → heap buffer overflow.

Affected Code

src/libImaging/QuantPngQuant.c, lines 80, 96:

charMatrix = malloc(width * height);                          // overflows
*quantizedPixels = malloc(sizeof(uint32_t) * width * height); // overflows

Example

• width=65537, height=65537 → product wraps to 131,073 (should be 4.3 billion)
• malloc(131073) allocates 128 KB; decoder writes 4 GB → heap corruption

Fix

• Add overflow check: if (height != 0 && (size_t)width > SIZE_MAX / (size_t)height)
• Use size_t total_pixels for all size calculations
• Also replaces sprintf → snprintf (line 129), consistent with CVE-2024-28219

Classification

• CWE-190 (Integer Overflow) → CWE-122 (Heap Buffer Overflow)
• CVSS 3.1: 8.1 (High) — AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
• Prerequisite: libimagequant support compiled in

[EricSoroos]

Note: Security policy: https://github.com/python-pillow/Pillow/blob/main/.github/SECURITY.md

This function is only called in one location:

Pillow/src/libImaging/Quant.c

Line 1814 in 3a44ba1

result = quantize_pngquant(

#ifdef HAVE_LIBIMAGEQUANT
            result = quantize_pngquant(
                p,
                im->xsize,
                im->ysize,
                colors,
                &palette,
                &paletteLength,
                &newData,
                withAlpha
            );

And previously in that function: (https://github.com/python-pillow/Pillow/blob/3a44ba1c75d72e7f375be9c6b7c71e2aef5e35e7/src/libImaging/Quant.c#L1697C5-L1699C6)

if (im->xsize > INT_MAX / im->ysize) {
        return ImagingError_MemoryError();
    }

[radarhere]

And one level up from that, we also check that the height is not zero.

Pillow/src/_imaging.c

Lines 1733 to 1737 in 3a44ba1

	if (!self->image->xsize || !self->image->ysize) {
	/* no content; return an empty image */
	return PyImagingNew(
	ImagingNew(IMAGING_MODE_P, self->image->xsize, self->image->ysize)
	);

[radarhere]

This change is included in #9476. See that issue for comments.