Implementation of custom bytetrack demo for real time stream analysis [GSoC'2026] - #4420
Open
Vishwa2684 wants to merge 8 commits into
Open
Implementation of custom bytetrack demo for real time stream analysis [GSoC'2026]#4420Vishwa2684 wants to merge 8 commits into
Vishwa2684 wants to merge 8 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new MediaPipe ByteTrack demo that integrates OVMS + the existing real-time stream analysis Python client to run object detection + multi-object tracking on RTSP streams. It also updates the MediaPipe build integration and adjusts streaming timestamp handling in the KFS graph executor.
Changes:
- Adds a new
demos/mediapipe/bytetrackdemo (graph, OVMS config, and setup README). - Extends the real-time stream analysis Python demo CLI to pass FFmpeg output dimensions through to
StreamClient. - Updates MediaPipe/Bazel integration (new ByteTrack calculators dependency; changes MediaPipe git_repository source) and adds stricter timestamp monotonicity checks for streaming.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
WORKSPACE |
Switches the MediaPipe repository source (currently to a fork/branch). |
third_party/mediapipe_calculators/BUILD |
Adds ByteTrack calculators target to the OVMS MediaPipe calculators aggregation library. |
src/kfs_frontend/kfs_graph_executor_impl.cpp |
Changes timestamp deserialization/defaulting and adds stream-level monotonic timestamp enforcement. |
demos/real_time_stream_analysis/python/client.py |
Adds CLI args for FFmpeg output width/height and forwards them to StreamClient. |
demos/mediapipe/bytetrack/README.md |
Introduces demo setup/run instructions for ByteTrack RTSP streaming demo. |
demos/mediapipe/bytetrack/config.json |
Adds OVMS config for the YOLOX model + ByteTrack MediaPipe graph. |
demos/mediapipe/bytetrack/bytetrack_ovms.pbtxt |
Adds the MediaPipe graph wiring OpenVINO/OVMS inference + ByteTrack tracking + overlay. |
Suppressed comments (1)
src/kfs_frontend/kfs_graph_executor_impl.cpp:1147
- The inline comment "// this method" is not descriptive. Either remove it or replace with a comment that explains what the function does.
// this method
Status createAndPushPacketsImpl(
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+111
to
+112
| #BYTETRACK CALCULATORS (GSoC'2026 - Vishwa2684) | ||
| "@mediapipe//mediapipe/graphs/bytetrack/calculators:bytetrack_calculators", |
Comment on lines
+29
to
+33
| # Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a | ||
| # vector of tensors representing, for instance, detection boxes/keypoints and | ||
| # scores. | ||
| node { | ||
| calculator: "OpenVINOModelServerSessionCalculator" |
Comment on lines
175
to
182
| ################################### Official/forked mediapipe repository ######### | ||
| #### Will be used on feature release | ||
| git_repository( | ||
| name = "mediapipe", | ||
| remote = "https://github.com/openvinotoolkit/mediapipe", | ||
| commit = "12e8d511cfbc5f471c498278a65a02dd250963e8", # top of mediapipe main branch as of 26.11.2025 | ||
| remote = "https://github.com/Vishwa2684/mediapipe_ovms", | ||
| # commit = "12e8d511cfbc5f471c498278a65a02dd250963e8", # top of mediapipe main branch as of 26.11.2025 | ||
| branch = "custom_bytetrack_graph" | ||
| ) |
Comment on lines
+1051
to
1062
| // steady_clock never jumps backward; system_clock can | ||
| auto now = std::chrono::steady_clock::now().time_since_epoch(); | ||
| int64_t candidate = std::chrono::duration_cast<std::chrono::microseconds>(now).count(); | ||
| timestamp = ::mediapipe::Timestamp::CreateNoErrorChecking(candidate); | ||
| } | ||
|
|
||
| // Enforce strict monotonicity regardless of source | ||
| if (lastPushedTimestamp != ::mediapipe::Timestamp::Unset() && timestamp <= lastPushedTimestamp) { | ||
| SPDLOG_DEBUG("Non-monotonic timestamp detected: new={} <= last={}; for request to: {}", | ||
| timestamp.DebugString(), lastPushedTimestamp.DebugString(), request.model_name()); | ||
| return Status(StatusCode::MEDIAPIPE_INVALID_TIMESTAMP, "Timestamp did not increase relative to previous packet in stream"); | ||
| } |
Comment on lines
+1030
to
1031
| // this method | ||
| static Status deserializeTimestampIfAvailable( |
Comment on lines
+1057
to
+1061
| // Enforce strict monotonicity regardless of source | ||
| if (lastPushedTimestamp != ::mediapipe::Timestamp::Unset() && timestamp <= lastPushedTimestamp) { | ||
| SPDLOG_DEBUG("Non-monotonic timestamp detected: new={} <= last={}; for request to: {}", | ||
| timestamp.DebugString(), lastPushedTimestamp.DebugString(), request.model_name()); | ||
| return Status(StatusCode::MEDIAPIPE_INVALID_TIMESTAMP, "Timestamp did not increase relative to previous packet in stream"); |
Comment on lines
+11
to
+24
| ## 2. Convert the ONNX Model to TensorFlow Lite | ||
|
|
||
| Open a Google Colab notebook and: | ||
|
|
||
| 1. Install `onnx2tf`. | ||
| 2. Upload `yolox_tiny.onnx` to the notebook. | ||
| 3. Run: | ||
|
|
||
| ```bash | ||
| !onnx2tf -i yolox_tiny.onnx -o yolox_tiny | ||
| ``` | ||
|
|
||
| This generates the TensorFlow Lite model. | ||
|
|
dtrawins
reviewed
Aug 11, 2026
|
|
||
| --- | ||
|
|
||
| ## 2. Convert the ONNX Model to TensorFlow Lite |
Collaborator
There was a problem hiding this comment.
why this conversion is added? Onnx model can be imported directly. There is also a collection of yolo models in IR format in https://huggingface.co/OpenVINO/models?search=yolo
config.json also includes the classes.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Collaborator
|
make sdl-check is failing with error: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@atobiszei @dtrawins
Summary
This PR adds a ByteTrack Real-Time Stream Analysis demo for OpenVINO Model Server.
The demo reuses the MediaPipe graph from
demos/mediapipe/bytetrackwithyolox_tinymodel and integrates it with thereal_time_stream_analysisPython client, allowing users to perform object detection and multi-object tracking on RTSP streams through OVMS.What's included
demos/mediapipe/bytetrack)client.pyexample for running inference on video and RTSP input and output streams