feat: add reproducible JavaScript and Rust benchmark suites - #117
Conversation
Adding CLAUDE.md with task information for AI processing. This file will be removed when the task is complete. Issue: #29
Implements a complete benchmarking solution comparing command-stream against: - execa (98M+ downloads) - cross-spawn (409M+ downloads) - ShellJS (35M+ downloads) - zx (4.2M+ downloads) - Bun.$ (built-in) ## Features Added: ### 📦 Bundle Size Analysis - Compare installed sizes and gzipped estimates - Dependency footprint analysis - Memory usage tracking ### ⚡ Performance Benchmarks - Process spawning speed tests - Streaming vs buffering throughput - Pipeline performance comparison - Concurrent execution scaling - Error handling performance ### 🧪 Feature Completeness Tests - Template literal support validation - Real-time streaming capabilities - Async iteration compatibility - EventEmitter pattern support - Built-in commands availability - Pipeline support verification ### 🌍 Real-World Use Cases - CI/CD pipeline simulation - Log processing benchmarks - File operations testing - Development workflow optimization ### 📊 Reporting & Visualization - Comprehensive HTML reports - Interactive performance charts - Feature compatibility matrix - JSON data export - Quick demo script ## Package Updates: - Version bump to 0.8.0 for new benchmarking capabilities - Added benchmark npm scripts: - `npm run benchmark` - Full comprehensive suite - `npm run benchmark:quick` - Fast subset - `npm run benchmark:demo` - Quick demonstration ## Usage: ```bash npm run benchmark # Complete suite (~5-10 minutes) npm run benchmark:demo # Quick demo (~30 seconds) npm run benchmark:quick # Essential benchmarks only ``` Reports generated in `benchmarks/results/` with HTML visualizations. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
konard
changed the title
Includes: - CI-INTEGRATION.md with setup instructions - benchmarks.yml workflow template for manual installation - Explains OAuth permission requirements for workflow files 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
|
We should add full benchmarks and tests suite, to exceed all competitors in features, performance and so on. Yet keep the most intuitive and simple API contract possible. |
|
🤖 AI Work Session Started Starting automated work session at 2026-09-15T18:47:30.532Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback. Runtime: solve |
konard
changed the title
Working session summaryImplemented and pushed the comprehensive benchmark suite. PR #117 is ready for review, mergeable, and based on current Highlights:
Verification:
Rest of the working session summary (41 characters)This summary was automatically extracted from the AI working session output. |
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
📊 Context and tokens usage:
Total: (580.5K + 21.6M cached) input tokens, 89.4K output tokens, $24.569521 cost 🤖 Models used:
📎 Log file uploaded as Gist (6942KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
|
We should support all programming languages equally, and make sure this parity is guaranteed by CI/CD, so it is not possible to change JavaScript code without changing Rust code and vice versa. |
|
🤖 AI Work Session Started Starting automated work session at 2026-09-15T20:21:58.454Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback. Runtime: solve |
konard
changed the title
|
Addressed in commits a174407, b857630, and ad3aa69: Rust now has an equivalent competitor benchmark playground and CI job, source and benchmark parity are enforced independently in both directions, and both Rust lockfiles are audited. The PR description now documents the two-language scope and verification. |
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
📊 Context and tokens usage:
Total: (628.8K + 25.3M cached) input tokens, 88.7K output tokens, $27.912957 cost 🤖 Models used:
📎 Log file uploaded as Gist (8160KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
🎉 Auto-mergedThis pull request has been automatically merged by hive-mind.
Auto-merged by hive-mind with --auto-merge flag |
1 participant
Summary
main, weekly, and on demandparity-exemptlabelProblem and reproduction
The former JavaScript suite was rooted outside the package after the monorepo migration. Running
bun benchmarks/run-all-benchmarks.mjsfailed while importing the deleted rootsrctree. It also estimated some results, simulated competitor workloads, swallowed failed measurements, and was not exercised by active CI. Rust had no corresponding competitor benchmark suite.The replacements pin competitors, validate warmups and measured outputs, rotate adapter order, record runtime/package versions, and keep timing regressions informational because shared CI hosts are noisy.
Run the smoke profiles locally:
Verification
rust/benchmarksis excluded from the publishedcommand-streamcrateThe generated JavaScript and Rust reports are uploaded as workflow artifacts. Pull requests use small smoke/base-head profiles;
main, weekly, and manual runs use the full profile.Fixes #29