Add streaming parsers for all seven implementations - #203
Conversation
Adding CLAUDE.md with task information for AI processing. This file will be removed when the task is complete. Issue: #197
…sages This commit implements a streaming parser for both Rust and JavaScript implementations of links-notation, addressing issue #197. The streaming parser enables incremental parsing of Links Notation data, which is essential for handling large messages without loading everything into memory. **Features:** Rust Implementation (src/stream_parser.rs): - StreamParser struct with callback-based API - on_link() callback for processing parsed links as they arrive - on_error() callback with detailed error location information - write() method for feeding data chunks incrementally - finish() method to complete parsing and return all links - ErrorLocation type with line, column, and offset information - Comprehensive test suite with 6 tests covering various scenarios JavaScript Implementation (src/StreamParser.js): - EventEmitter-based StreamParser class - 'link' event emitted for each parsed link - 'error' event with line/column location information - write() method for incremental data feeding - end() method to finalize parsing - position() method for tracking parse progress - Configurable maxInputSize and maxDepth options - Comprehensive test suite with 17 tests **Examples:** - examples/rust_streaming_parser.rs - 7 examples demonstrating various use cases - examples/js_streaming_parser.js - 6 examples showing streaming capabilities **Documentation:** - Updated rust/README.md with streaming parser section and API reference - Updated js/README.md with streaming parser section and API reference - Added inline documentation and usage examples in code **Use Cases:** 1. Memory efficiency - Process large messages without loading everything 2. Latency reduction - Start processing before full message arrives 3. Network integration - Natural fit for TCP/HTTP streaming 4. Real-time processing - Handle data as it becomes available **Testing:** - All 39 existing Rust tests pass - All 205 existing JavaScript tests pass (188 + 17 new streaming tests) - No regressions introduced Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Apply rustfmt to Rust files - Apply prettier to README files - No functional changes Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace len() >= 1 with !is_empty() for better idiomatic Rust code. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
konard
changed the title
This reverts commit ed06731.
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
Now working session is ended, feel free to review and add any feedback on the solution draft. |
|
We need to make sure streaming parser is fully supported in all the languages, meaning we can parse links notation with symbol by symbol, line by line and so on streaming. So it is possible to have continued access to real time data, or just start processing as soon as it works. We should support in all our languages all best practices - even driven approach, enumarators and so on, so every language feature is used at the fullest potential and everything is done perfectly in all languages, so most possible use cases are covered. We also should try to integrate best of #202 if there is something useful and close 202 once we took everything from there. |
|
🤖 AI Work Session Started Starting automated work session at 2026-09-15T13:28:35.134Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback. Runtime: solve |
# Conflicts: # README.md # js/README.md # js/src/index.js # rust/Cargo.toml # rust/README.md # rust/links-notation/src/stream_parser.rs # rust/src/lib.rs
Add runnable JavaScript and Rust examples plus a seven-language parity experiment. Document each native streaming API in English and Russian, align shared contract tests, regenerate the comparison matrix, and bump all maintained packages for the streaming release.
konard
changed the title
|
Implemented the requested streaming parity across all seven maintained languages and incorporated the strongest parts of #202. The new parsers support arbitrary symbol/line chunks, canonical syntax validation, native push and lazy APIs, bounded unresolved-record memory, position/drain/reset controls, documentation, examples, and coordinated 0.20.0 release metadata. The branch now includes current |
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
📊 Context and tokens usage:
Total: (634.6K + 20.0M cached) input tokens, 117.8K output tokens, $24.595521 cost 🤖 Models used:
📎 Log file uploaded as Gist (8461KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
1 participant
Closes #197.
Summary
Adds production-ready incremental parsing to every maintained Links Notation implementation. The parsers accept arbitrary chunks—including one symbol or one line at a time—and emit a top-level record only after indentation, multiline quotes, parentheses, and comments are structurally complete. Every completed segment is validated by the existing canonical parser, so streaming keeps the same syntax and output as whole-document parsing.
This incorporates the useful event/callback and position-tracking ideas from #202, replaces its newline/parenthesis heuristic with canonical record validation, extends the feature from JavaScript and Rust to all seven languages, and supersedes #202.
Native APIs
LinkParsedeventIEnumerable/IAsyncEnumerablelink,error, andendeventsIteratorio.Writeriter.Seq2Consumer<Link>Stream<Link>GeneratorAll implementations also expose finish, absolute position, reset/drain, opt-out collection, and a maximum unresolved-record buffer. Disabling collection keeps long-lived callback/iterator streams bounded to the unresolved record.
Reproduction and verification
Before this change, the repository had no supported streaming API on
main; the earlier draft handled only JavaScript and Rust and could split an indented record from children or treat newlines inside quotes as record boundaries.The new shared contract feeds the same document symbol-by-symbol and line-by-line in every language and compares the result with that language's canonical parser. The fixture includes comments, indented children, multiline quoted references, multiline parentheses, and a final record without a trailing newline. It also checks callbacks/events, native lazy adapters, finish behavior, drain/reset, position tracking, and bounded unresolved-record memory.
Run the focused cross-language verification with:
Local full gates passed:
Release and documentation