Skip to content

docs(blog): add the InfoQ article on computing time in queue for Hudi Kafka pipelines - #19795

Open
rangareddy wants to merge 1 commit into
apache:asf-sitefrom
rangareddy:blog-beyond-offset-lag-kafka-hudi
Open

docs(blog): add the InfoQ article on computing time in queue for Hudi Kafka pipelines#19795
rangareddy wants to merge 1 commit into
apache:asf-sitefrom
rangareddy:blog-beyond-offset-lag-kafka-hudi

Conversation

@rangareddy

@rangareddy rangareddy commented Aug 30, 2026

Copy link
Copy Markdown
Collaborator

Describe the issue this Pull Request addresses

The blog index has no entry for Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake
Pipelines at Petabyte Scale
, published
on InfoQ on 26 August 2026 by Srikanth Mamidala.

It is a production case study worth surfacing from hudi.apache.org. Twilio's data lake ingests five trillion
records a month through Hudi Streamer, and the article is about why Kafka offset lag is the wrong freshness
signal for those pipelines: offset lag says how far behind a consumer is, not how old the data is. It derives
a time-in-queue metric instead, by reading the Kafka checkpoint out of the latest .hoodie commit file,
seeking to that offset in the topic, and measuring the timestamp delta against now, so it works off the
timeline Hudi Streamer already writes with no instrumentation change to the pipeline. It also covers the edge
cases that showed up in production: commits carrying no checkpoint metadata (walk backward through the commit
history), clock skew, multi-partition topics, multiple writers to one table, and epoch timestamp defaults.

That makes it directly useful to anyone running Hudi Streamer against Kafka, and it is the kind of real-world
adoption story the case-study category exists for.

Summary and Changelog

  • website/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.mdx — a
    redirect stub in the established shape for externally hosted articles: frontmatter plus
    import Redirect from '@site/src/components/Redirect'; and a <Redirect url="...">, the same as
    2024-06-18-how-to-use-apache-hudi-with-databricks.mdx and the other external posts. The author is declared
    inline (authors: / - name: Srikanth Mamidala), which is what the other external-author posts do rather
    than adding an authors.yml entry for a one-off contributor. The filename date matches the article's
    publication date.
  • website/static/assets/images/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.png
    — a purpose-drawn hero at the required 1200x600, in the site's own brand palette
    (--ifm-color-primary-text #0DB1F9 on --ifm-color-primary-darkest). It is not a screenshot of the
    article: it draws the distinction the article is built on, with one Kafka partition, the checkpoint recorded
    in the latest .hoodie commit, and the same span measured twice — as an offset count above and as elapsed
    time below. 266 KB, which sits between the median (180 KB) and p90 (662 KB) of the existing blog images.
  • .github/scripts/validate-blog.py — adds twilio to ALLOWED_TAGS.

One thing for the reviewer

The new twilio tag needs your approval. validate-blog.py rejects it otherwise, and its own error
message says to add it to the allowlist in the same PR and ask the reviewer. It follows the existing
company-tag convention: uber, walmart, robinhood, peloton, upstox, metica, penn interactive,
southwest airlines, zupee, yuno and grab are already in that set. Happy to drop the tag and the
allowlist line if you would rather not grow it — the other three tags (apache kafka, observability,
hudi streamer) carry the post on their own.

Unrelated to this post, but noticed while writing it: README's blog section asks for a source: prefixed tag
(source:medium, source:linkedin, source:onehouse). No post in website/blog/ uses one and
ALLOWED_TAGS contains none, so source:infoq would fail CI. Left it out. The README and the validator have
drifted apart there and it is worth reconciling in a separate PR. The same README says image is required,
and validate-blog.py does list it in REQUIRED_FIELDS, but its missing-field branch only ever raises for
author, so a post with no image passes silently today.

Verification

python3 .github/scripts/validate-blog.py website/blog/2026-08-26-beyond-offset-lag-...-pipelines.mdx
  -> All 1 blog(s) passed validation

cd website && npm run build
  -> [SUCCESS] Generated static files in "build"

curl -I https://www.infoq.com/articles/beyond-offset-lag-kafka-apache-hudi/
  -> HTTP 200

The build emits /blog/2026/08/26/beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines/ with
the redirect target present in the rendered HTML. content/ is untouched, per the asf-site.ci.yml guard.

Screenshot 2026-08-31 at 9 22 06 AM

… Kafka pipelines

"Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at
Petabyte Scale", by Srikanth Mamidala, InfoQ, 26 August 2026.

Twilio ingests five trillion records a month through Hudi Streamer, and the article is
about why Kafka offset lag is the wrong freshness signal for those pipelines: it says
how far behind a consumer is, not how old the data is. It derives a time-in-queue metric
by reading the Kafka checkpoint out of the latest .hoodie commit file, seeking to that
offset in the topic, and measuring the timestamp delta against now, so it runs off the
timeline Hudi Streamer already writes with no change to the pipeline.

Added as a redirect stub, the established shape for externally hosted articles, with the
author declared inline rather than in authors.yml since it is a one-off contributor.

The hero is drawn for this post at the required 1200x600 in the site's brand palette,
rather than copied from the article. It shows one Kafka partition with the checkpoint
recorded in the latest commit, and the same span measured twice: as an offset count
above and as elapsed time below.

validate-blog.py rejects the twilio tag, so it is added to ALLOWED_TAGS here, as the
script's own error message directs. It follows the existing company-tag convention
alongside uber, walmart, robinhood, upstox, metica and southwest airlines, and needs a
reviewer's approval.
@github-actions github-actions Bot added docs size:S PR with lines of changes in (10, 100] labels Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs size:S PR with lines of changes in (10, 100]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant