gpfdist supports reading LZO-compressed files.

[ZTE-EBASE]

Fixes #ISSUE_Number

What does this PR do?

This PR adds LZO compressed file format read/write support to the gpfdist utility in Apache Cloudberry.

Type of Change

• New feature (non-breaking change)

Breaking Changes

Test Plan

• Unit tests added/updated
• Integration tests added/updated
• Passed make installcheck
• Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

Scenario 1: High-Volume Data Loading

LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio.

Why this matters: For time-critical ETL jobs, LZO can reduce load times by 2-5x compared to heavier compression algorithms, while still saving ~50% storage space compared to uncompressed files.

Scenario 2: Hadoop Ecosystem Integration

Many Hadoop ecosystem tools natively support LZO compression:

1. Hive: Supports LZO-compressed text and sequence files
2. Spark: Can write LZO-compressed output (via hadoop-lzo library)
3. Sqoop: Commonly uses LZO for bulk imports from RDBMS to HDFS

Scenario 3: Storage Efficiency with Parallel Performance

LZO is splittable, meaning multiple gpfdist instances (or segments) can read different portions of the same file simultaneously. This enables near-linear scalability with additional segments.

User-facing changes:

example

The --with-lzo configure option must be specified at compile time.
./configure --with-lzo --prefix=/home/cdblzo

Dependencies:

#1886 (reply in thread)

Checklist

• Followed contribution guide
• Added/updated documentation
• Reviewed code for security implications
• This PR contains AI-assisted code generation
• Requested review from cloudberry committers

References

Technical Documentation

• LZO Official Website
• LZOP File Format Specification
• Hadoop LZO Compression Guide

Cloudberry Documentation

• GPFDIST Official Docs
• External Tables Guide

Related Projects

• Apache Cloudberry GitHub
• Greenplum Database (upstream)

Additional Context

CI Skip Instructions

---

[tuhaihe]

Before moving forward with the current PR, would you have the bandwidth to take another look at apache/cloudberry#1226?

The two PRs follow a similar operational model, and the previous PR has already been open for quite some time. It might be better to bring that one to completion first and then proceed with this PR. This would present a more consistent development process and help build confidence among the contributors and reviewers.

[ZTE-EBASE]

I validated this feature in the development environment previously and confirmed it works properly. Back then, Cloudberry’s CI dependencies were undergoing adjustments. Later on, I was occupied with urgent work tasks. Currently, there are assessment requirements for community contributions, hence I intend to contribute the newly developed features to Cloudberry.
Support for LZO compression constitutes a standalone new feature with no logical overlaps or dependencies on the earlier PR, meaning the two can be advanced independently.

[ZTE-EBASE]

The CI of another candidate item I reviewed also shows a failure. Please help review it again. Follow-up work will include gradual optimization to support exporting table data as LZO files via gpfdist.