Describe the bug
When publishUpdate produces a compressed vulnerability proto larger than maxPubSubMessageSize, it logs a warning and publishes the update with Data = nil while leaving deleted=false.
The Go worker's parseVuln treats empty message data as a valid nil vulnerability (documented there as expected for delete requests). handleMessage then constructs a TaskUpdate, and Engine.handleUpdate returns vuln not provided. The subscriber NACKs the message, so the same oversized update can be retried without a path to successful processing.
Relevant paths on current master:
go/internal/importer/importer.go (publishUpdate)go/internal/worker/subscriber.go (parseVuln, handleMessage)go/internal/worker/engine.go (handleUpdate)
To Reproduce
- Construct a non-deleted vulnerability whose zstd-compressed protobuf exceeds
maxPubSubMessageSize.
- Pass it through
publishUpdate.
- Observe that the published message has empty data and
deleted=false.
- Deliver that message to the worker subscriber.
parseVuln returns nil, nil, the task is classified as TaskUpdate, handleUpdate returns vuln not provided, and the message is NACKed.
Expected behaviour
Oversized vulnerability updates should have an explicit recoverable or terminal handling path rather than becoming an update message that deterministically fails and is retried.
A small fix could either avoid publishing this invalid update shape or make the worker/importer coordinate an explicit oversized-record path. The exact approach is probably best decided by maintainers because the importer comment currently says to let the worker decide what to do.
Additional context
I searched open/closed issues and PRs for the warning/error strings and maxPubSubMessageSize and did not find an existing report. I have not started implementation, per the repository's issue-first/assignment requirement. I am happy to work on a narrowly scoped fix and regression test if this issue is assigned to me.
Describe the bug
When
publishUpdateproduces a compressed vulnerability proto larger thanmaxPubSubMessageSize, it logs a warning and publishes the update withData = nilwhile leavingdeleted=false.The Go worker's
parseVulntreats empty message data as a valid nil vulnerability (documented there as expected for delete requests).handleMessagethen constructs aTaskUpdate, andEngine.handleUpdatereturnsvuln not provided. The subscriber NACKs the message, so the same oversized update can be retried without a path to successful processing.Relevant paths on current
master:go/internal/importer/importer.go(publishUpdate)go/internal/worker/subscriber.go(parseVuln,handleMessage)go/internal/worker/engine.go(handleUpdate)To Reproduce
maxPubSubMessageSize.publishUpdate.deleted=false.parseVulnreturnsnil, nil, the task is classified asTaskUpdate,handleUpdatereturnsvuln not provided, and the message is NACKed.Expected behaviour
Oversized vulnerability updates should have an explicit recoverable or terminal handling path rather than becoming an update message that deterministically fails and is retried.
A small fix could either avoid publishing this invalid update shape or make the worker/importer coordinate an explicit oversized-record path. The exact approach is probably best decided by maintainers because the importer comment currently says to let the worker decide what to do.
Additional context
I searched open/closed issues and PRs for the warning/error strings and
maxPubSubMessageSizeand did not find an existing report. I have not started implementation, per the repository's issue-first/assignment requirement. I am happy to work on a narrowly scoped fix and regression test if this issue is assigned to me.