[FEAT] added demuxer and file_functions module
#1662
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
@steel-bucket Is it still WIP? |
Contributor
Author
Yes, I'm done with the hard part though, file_functions module is fully tested and ready. And the demuxer module just needs a couple more tests. Then I just have the gxf one to do. It won't be long though. Sorry to be late with it, I had some exams which are cleared out now. |
steel-bucket
changed the title
demuxer moduledemuxer module
prateekmedia
changed the title
demuxer moduledemuxer module
Contributor
Author
|
Hi, @prateekmedia the builds and tests are all working(other than regression). Please review it if time permits. Also should I squash the commits together? |
steel-bucket
changed the title
demuxer moduledemuxer and file_functions module
steel-bucket
force-pushed
the
migration-demuxer-module
branch
from
5268a7d to
bfbe1d6
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR ports the C demuxer and file_functions modules to Rust, adds corresponding FFI bindings for MXF/GXF, and updates build configuration.
- Introduced new Rust modules (
demuxer,file_functions) and added them to the crate root. - Extended the C wrapper (
wrapper.h) andextern "C"block inlib.rsfor MXF/GXF and demuxer functions. - Updated C source (
ccx_gxf.*,ccx_demuxer_mxf.*,ccx_demuxer.c) to remove duplicate definitions and conditionally call Rust implementations.
Reviewed Changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/rust/wrapper.h | Added headers for GXF and MXF demuxer |
| src/rust/src/parser.rs | Tightened cast to usize for input file capacity check |
| src/rust/src/libccxr_exports/mod.rs | Exported the new demuxer module |
| src/rust/src/lib.rs | Declared new Rust modules and FFI externs |
| src/rust/src/file_functions/mod.rs | Added file_functions module with documentation stub |
| src/rust/src/demuxer/common_structs.rs | New demuxer data structures and defaults |
| src/lib_ccx/ccx_gxf.h | Defined ccx_gxf struct for GXF support |
| src/lib_ccx/ccx_gxf.c | Removed duplicate struct; left stray comments |
| src/lib_ccx/ccx_demuxer_mxf.h | Added MXF context and type definitions |
| src/lib_ccx/ccx_demuxer_mxf.c | Cleaned up duplicate MXF definitions |
| src/lib_ccx/ccx_demuxer.c | Wrapped demuxer calls under DISABLE_RUST |
| docs/CHANGES.TXT | Updated changelog with new entries |
Comments suppressed due to low confidence (5)
src/lib_ccx/ccx_demuxer_mxf.c:1
- This file uses
uint8_tand other fixed-width types but does not include<stdint.h>. Add the proper include to avoid compilation errors.
#define IS_KLV_KEY(x, y) (!memcmp(x, y, sizeof(y)))
docs/CHANGES.TXT:44
- The leading
--appears duplicated in this changelog entry. Use a single-to match the existing formatting.
-- Fix: Unit Test Rust failing due to changes in Rust Version 1.86.0
src/rust/src/lib.rs:43
- [nitpick] The imported types
c_uchar,c_ulong, andc_voidare not used elsewhere in this file; consider removing this import.
use std::os::raw::{c_uchar, c_ulong, c_void};
src/lib_ccx/ccx_gxf.h:9
- Ensure this header has proper include guards or
#pragma onceto prevent duplicate definitions when included multiple times.
struct ccx_gxf
src/lib_ccx/ccx_demuxer_mxf.h:6
- Add include guards or
#pragma onceto this header to prevent multiple inclusion, and verify that<stdint.h>is included foruint8_t.
typedef uint8_t UID[16];
src/rust/src/file_functions/mod.rs
Outdated
Comment on lines
4
to
7
| * @param ctx ccx_demuxer context properly initilaized ccx_demuxer with some input | ||
| * Not to be NULL, since ctx is derefrenced inside this function | ||
| * | ||
| * @param buffer if buffer then it must be allocated to at;east bytes len as |
There was a problem hiding this comment.
Fix typos in the documentation: change "initilaized" to "initialized", "derefrenced" to "dereferenced", and "at;east" to "at least".
Suggested change
| * @param ctx ccx_demuxer context properly initilaized ccx_demuxer with some input | |
| * Not to be NULL, since ctx is derefrenced inside this function | |
| * | |
| * @param buffer if buffer then it must be allocated to at;east bytes len as | |
| * @param ctx ccx_demuxer context properly initialized ccx_demuxer with some input | |
| * Not to be NULL, since ctx is dereferenced inside this function | |
| * | |
| * @param buffer if buffer then it must be allocated to at least bytes len as |
Comment on lines
119
to
127
| #[allow(unused)] // used in tests | ||
| pub(crate) fn default() -> PSIBuffer { | ||
| PSIBuffer { | ||
| prev_ccounter: 0, | ||
| buffer: Box::into_raw(Box::new(0u8)), | ||
| buffer_length: 0, | ||
| ccounter: 0, | ||
| } | ||
| } |
There was a problem hiding this comment.
You have both an impl Default for PSIBuffer and an inherent default() method; consider removing the duplicate inherent method or merging them to avoid confusion.
Suggested change
| #[allow(unused)] // used in tests | |
| pub(crate) fn default() -> PSIBuffer { | |
| PSIBuffer { | |
| prev_ccounter: 0, | |
| buffer: Box::into_raw(Box::new(0u8)), | |
| buffer_length: 0, | |
| ccounter: 0, | |
| } | |
| } |
src/lib_ccx/ccx_gxf.c
Outdated
| if (result != len) | ||
| ret = CCX_EOF; | ||
| return ret; | ||
| // #endif |
There was a problem hiding this comment.
[nitpick] Remove stray commented // #endif lines inside function bodies to clean up leftover conditional compilation markers.
Suggested change
| // #endif |
prateekmedia
approved these changes
Nov 2, 2025
Member
|
The windows compiling is separate and can be resolved separately. |
Member
|
@steel-bucket Can you merge main? |
Contributor
Author
|
@prateekmedia Done, I think we might have to find an alternative solution to the windows problem. |
Collaborator
CCExtractor CI platform finished running the test files on linux. Below is a summary of the test results, when compared to test for commit 810e02f...:
Your PR breaks these cases:
It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you). Check the result page for more info. |
Collaborator
CCExtractor CI platform finished running the test files on windows. Below is a summary of the test results, when compared to test for commit 810e02f...:
Your PR breaks these cases:
It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you). Check the result page for more info. |
x15sr71
pushed a commit
to x15sr71/ccextractor
that referenced
this pull request
Dec 9, 2025
* feat: added demuxer module * Cargo Lock Update * Completed file_functions and demuxer * Completed file_functions and demuxer * written extern functions for demuxer * Removed libc completely, added tests for gxf and ported gxf to C * Hardsubx error fixed * Fixing format issues * clippy errors fixed * fixing format issues * fixing format issues * Windows failing tests * Windows failing tests * demuxer: added demuxer data transfer functions and removed some structs * made Demuxer and File Functions * Minor formatting changes * Minor Rebasing changes * demuxer: format rust and unit test rust checks * C formatting * Windows Failing test * Windows Failing test * Update CHANGES.TXT * Update CHANGES.TXT * Windows Failing Tests * Windows Failing Tests * Problem in Copy to Rust and some typos that copilot review suggested * Minor Formatting Error * Windows Failing Regressions * Windows Failing Regressions * Minor Comment Change * Data transfer module for DemuxerData added and more rustlike syntax to ctorust.rs * Minor Formatting Changes * demuxer: Rebase and a few tweaks to file_functions * demuxer: Minor Formatting Error * [FIX] 134 Codes in XDS and General Tests (CCExtractor#1708) * Made pointers valid in Unit Tests of Decoder * fix: test_do_cb * Copilot Suggestions * Suggestions about Redundancy * Suggestions about Redundancy * [FEAT] Add `bitstream` module in `lib_ccxr` (CCExtractor#1649) * feat: Add bitstream module * run code formatters * Run cargo clippy --fix * Run cargo fmt --all * refactor: remove rust pointer from C struct * feat: Add bitstream module * run code formatters * Run cargo clippy --fix * Run cargo fmt --all * refactor: remove rust pointer from C struct * Added Bitstream to libccxr_exports * Minor Formatting Issue * Bitstream: Removed redundant CType * bitstream: recommended changes for is_byte_aligned * bitstream: recommended changes for long comments * bitstream: comment fix * bitstream: removed redundant comparism comments --------- Co-authored-by: Deepnarayan Sett <depnra1@gmail.com> Co-authored-by: Deepnarayan Sett <71217129+steel-bucket@users.noreply.github.com> * demuxer: minor formatting changes * Demuxer: Changes to mistakes in CHANGES.txt * Demuxer: Removed extra newline in ccextractor.c * Demuxer: Changes to Encoding resolved * Demuxer: Moved CCX_NOPTS to common structs and some changes to Demuxer Data regd. MPEG_CLOCK_FREQ * some refactoring to CCX_NOPTS * Demuxer: Minor Mistake regarding CHANGES.txt * Demuxer: Unit test rust failing because of CCX_NOPTS * Demuxer: changed common_structs to common_types * Demuxer: Removed redundant libraries from Cargo.toml and moved tempfile to dev-dependencies * Demuxer: Removed to_vec function and renamed PSIBuffer/PMTEntry from_ctype functions * Demuxer: Renamed Stream_Type, improved Time complexity of the default() function and removed redundant comments * Demuxer: Removed two repeated code blocks and removed redundant comments * Demuxer: Removed two code blocks * Demuxer: Review Changes * Demuxer: Removed redundant tests * Update src/rust/src/demuxer/demux.rs Co-authored-by: Prateek Sunal <prtksunal@gmail.com> * Demuxer: Errors due to Rebase * Demuxer: Removed get_stream_mode * Demuxer: Errors due to rebasing and removing redundant CType Functions * Demuxer: Failing ES regressions * Demuxer: MythTV failing regression * Demuxer: Removed redundant comments * Demuxer: Unplugged ES for now * Demuxer: Replugged in ES * Demuxer: Formatting error * Demuxer: Windows failing CI * Demuxer: Windows failing CI * Demuxer: Windows failing Regressions * Demuxer: Formatting * Demuxer: Minor Cargo Clippy change * Demuxer: running regressions again * Demuxer: Cargo Lockfile Change * Demuxer: running regressions again * Demuxer: running regressions again --------- Co-authored-by: Swastik Patel <swastikpatel29@gmail.com> Co-authored-by: Prateek Sunal <prtksunal@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In raising this pull request, I confirm the following (please check boxes):
My familiarity with the project is as follows (check one):
In this PR, I have attempted to port the large
demuxermodule to Rust, the primary logic of the heavily interconnected C librariesfile_functions.cand,ccx_demuxer.cand their corresponding header files has aleady been implemented here.This PR was inspired by the ones done for the 708 Decoder in CCextractor.
The part of the codebase that the demuxer part of this PR migrates to Rust is the part that Opens a File(ccx_demuxer_open), points the codebase towards that file, detects the stream type and some other parameters like myth, and then closes the file or gets the file size.
The file_functions part of this PR is tested locally, and in unit tests, but integrating it into C made the codebase really slow, due to the constant copying back and forth C and Rust, so it was left to be used in future Rust Libraries like MythTV, MXF, GXF, etc.
Any criticism or suggestion is wholeheartedly welcome.