[Feature Request]: OpenLineage extension for the Java SDK

[rahul-madaan]

What would you like to happen?

Beam pipelines should be able to emit OpenLineage events describing the datasets they read and write, so OpenLineage consumers (Marquez, Atlan, and others) can build lineage graphs for Beam jobs on any runner. This continues #33981 and builds on the pluggable lineage mechanism from #36790.

Proposed: a new sdks/java/extensions/openlineage module that follows the architecture of the official OpenLineage Spark and Flink integrations:

• Configuration extends io.openlineage.client.OpenLineageConfig and resolves with the Spark integration's precedence: pipeline options over openlineage.yml over OPENLINEAGE__ environment variables. Transport, facets, dataset and circuit-breaker settings reuse the standard openlineage-java config classes.
• OpenLineageRunner wraps any runner (--runner=OpenLineageRunner --openLineageDelegateRunner=FlinkRunner), extracts datasets from the pipeline graph via classpath-guarded per-IO visitors (PubsubIO, IcebergIO; extensible through a public LineageProvider interface), mints a UUIDv7 run id at submission and propagates it via serialized options, and emits START and terminal COMPLETE/ABORT/FAIL events.
• A job tracker emits periodic RUNNING events on the Flink integration's trackingIntervalInSeconds cadence (default 60s) and sweeps lineage reported through the Beam Lineage metrics API.
• OpenLineageLineage implements the LineageBase plugin (--lineageType) to capture lineage live in worker JVMs for long-running streaming jobs, teeing FQNs back into the metrics store so runner-native consumers keep working.
• Dataset naming strictly follows the OpenLineage naming conventions; Iceberg datasets use the physical table location with the catalog identity as a TABLE symlink, matching the Spark integration's IcebergHandler.

Building on - #33981

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Java SDK
• Component: Python SDK