Skip to content

bug: no column projection should still persist row count#4444

Open
coderfender wants to merge 3 commits into
apache:mainfrom
coderfender:zero_column_projection_should_still_return_row_count
Open

bug: no column projection should still persist row count#4444
coderfender wants to merge 3 commits into
apache:mainfrom
coderfender:zero_column_projection_should_still_return_row_count

Conversation

@coderfender
Copy link
Copy Markdown
Contributor

@coderfender coderfender commented May 27, 2026

Which issue does this PR close?

Closes : #4445 discovered while implementing native BroadcastNestedLoopJoin.

Rationale for this change

Utils.serializeBatches and Utils.coalesceBroadcastBatches lose the row
count when the input batch has zero columns, so a column-pruned broadcast
relation arrives at the receiver as empty.

Both functions build an Arrow VectorSchemaRoot from the batch's field
vectors. With zero columns the VSR has nothing to infer rowCount from and
defaults to 0; IPC then encodes length = 0.

Latent until now because every
prior caller of CometBroadcastExchangeExec was BHJ, whose build side
always carries join keys. Native BroadcastNestedLoopJoin implementation exposed this bug when build side that can broadcast a 0-column side (e.g. SELECT count(*) FROM big, small), resulting in 0 rows

What changes are included in this PR?

  • Utils.serializeBatches: root.setRowCount(batch.numRows()) after VSR
    construction.
  • Utils.coalesceBroadcastBatches: targetRoot.setRowCount(totalRows.toInt)

How are these changes tested?

New UtilsSuite tests for 0-column batches.

@coderfender coderfender changed the title bug: Zero column projection should still return row count bug: no column projection should still persist row count May 27, 2026
Copy link
Copy Markdown
Contributor

@mbutrovich mbutrovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice find, thanks @coderfender!

A couple of suggestions before merging.

CI: check-missing-suites. dev/ci/check-suites.py walks both pr_build_linux.yml and pr_build_macos.yml, so UtilsSuite also needs to be added to .github/workflows/pr_build_macos.yml.

Consider gating the setRowCount calls on the no-column case. The condition itself documents why the call exists, and the non-empty path stays byte-identical to today.

val root = new VectorSchemaRoot(fieldVectors.asJava)
if (fieldVectors.isEmpty) {
  // VSR cannot infer rowCount without field vectors
  root.setRowCount(batch.numRows())
}
if (targetRoot.getSchema.getFields.isEmpty) {
  // VSRAppender does not update rowCount with no columns
  targetRoot.setRowCount(totalRows.toInt)
}

Follow-up. A SQL-level test like SELECT count(*) FROM big CROSS JOIN small with a broadcast hint would have caught this pre-fix. Worth leaving a TODO pointing at the native BNLJ PR so it lands there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug zero column projection does not persist row count across IPC

2 participants