docs(blog): add the InfoQ article on computing time in queue for Hudi Kafka pipelines - #19795
Open
rangareddy wants to merge 1 commit into
Open
docs(blog): add the InfoQ article on computing time in queue for Hudi Kafka pipelines#19795rangareddy wants to merge 1 commit into
rangareddy wants to merge 1 commit into
Conversation
… Kafka pipelines "Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake Pipelines at Petabyte Scale", by Srikanth Mamidala, InfoQ, 26 August 2026. Twilio ingests five trillion records a month through Hudi Streamer, and the article is about why Kafka offset lag is the wrong freshness signal for those pipelines: it says how far behind a consumer is, not how old the data is. It derives a time-in-queue metric by reading the Kafka checkpoint out of the latest .hoodie commit file, seeking to that offset in the topic, and measuring the timestamp delta against now, so it runs off the timeline Hudi Streamer already writes with no change to the pipeline. Added as a redirect stub, the established shape for externally hosted articles, with the author declared inline rather than in authors.yml since it is a one-off contributor. The hero is drawn for this post at the required 1200x600 in the site's brand palette, rather than copied from the article. It shows one Kafka partition with the checkpoint recorded in the latest commit, and the same span measured twice: as an offset count above and as elapsed time below. validate-blog.py rejects the twilio tag, so it is added to ALLOWED_TAGS here, as the script's own error message directs. It follows the existing company-tag convention alongside uber, walmart, robinhood, upstox, metica and southwest airlines, and needs a reviewer's approval.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
The blog index has no entry for Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake
Pipelines at Petabyte Scale, published
on InfoQ on 26 August 2026 by Srikanth Mamidala.
It is a production case study worth surfacing from hudi.apache.org. Twilio's data lake ingests five trillion
records a month through Hudi Streamer, and the article is about why Kafka offset lag is the wrong freshness
signal for those pipelines: offset lag says how far behind a consumer is, not how old the data is. It derives
a time-in-queue metric instead, by reading the Kafka checkpoint out of the latest
.hoodiecommit file,seeking to that offset in the topic, and measuring the timestamp delta against now, so it works off the
timeline Hudi Streamer already writes with no instrumentation change to the pipeline. It also covers the edge
cases that showed up in production: commits carrying no checkpoint metadata (walk backward through the commit
history), clock skew, multi-partition topics, multiple writers to one table, and epoch timestamp defaults.
That makes it directly useful to anyone running Hudi Streamer against Kafka, and it is the kind of real-world
adoption story the
case-studycategory exists for.Summary and Changelog
website/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.mdx— aredirect stub in the established shape for externally hosted articles: frontmatter plus
import Redirect from '@site/src/components/Redirect';and a<Redirect url="...">, the same as2024-06-18-how-to-use-apache-hudi-with-databricks.mdxand the other external posts. The author is declaredinline (
authors:/- name: Srikanth Mamidala), which is what the other external-author posts do ratherthan adding an
authors.ymlentry for a one-off contributor. The filename date matches the article'spublication date.
website/static/assets/images/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.png— a purpose-drawn hero at the required 1200x600, in the site's own brand palette
(
--ifm-color-primary-text#0DB1F9on--ifm-color-primary-darkest). It is not a screenshot of thearticle: it draws the distinction the article is built on, with one Kafka partition, the checkpoint recorded
in the latest
.hoodiecommit, and the same span measured twice — as an offset count above and as elapsedtime below. 266 KB, which sits between the median (180 KB) and p90 (662 KB) of the existing blog images.
.github/scripts/validate-blog.py— addstwiliotoALLOWED_TAGS.One thing for the reviewer
The new
twiliotag needs your approval.validate-blog.pyrejects it otherwise, and its own errormessage says to add it to the allowlist in the same PR and ask the reviewer. It follows the existing
company-tag convention:
uber,walmart,robinhood,peloton,upstox,metica,penn interactive,southwest airlines,zupee,yunoandgrabare already in that set. Happy to drop the tag and theallowlist line if you would rather not grow it — the other three tags (
apache kafka,observability,hudi streamer) carry the post on their own.Unrelated to this post, but noticed while writing it: README's blog section asks for a
source:prefixed tag(
source:medium,source:linkedin,source:onehouse). No post inwebsite/blog/uses one andALLOWED_TAGScontains none, sosource:infoqwould fail CI. Left it out. The README and the validator havedrifted apart there and it is worth reconciling in a separate PR. The same README says
imageis required,and
validate-blog.pydoes list it inREQUIRED_FIELDS, but its missing-field branch only ever raises forauthor, so a post with noimagepasses silently today.Verification
The build emits
/blog/2026/08/26/beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines/withthe redirect target present in the rendered HTML.
content/is untouched, per theasf-site.ci.ymlguard.