Replace quirc and BC-UR libraries with lighter pure-C implementations - #303
Replace quirc and BC-UR libraries with lighter pure-C implementations#303odudex wants to merge 6 commits into
Conversation
|
Hi @odudex awesome stuff! I have been irked by bc-ur for a long time (we also use it in gdk) - .the c++ impl is inefficient and unwieldy to work with, and the upstream maintainers are resistant to any kind of changes. Its long been a dream of mine to have time to rewrite it as clean C, so its great to see you've done exactly that :) The code size reduction and performance improvements are also very welcome, so I'm very interested in merging these changes. Swapping from our current impl would require us to fully review/test both libraries, which will take a while, and we'd almost certainly do this a library at a time, starting with bc-ur. So please bear with us as we review cUR first and then we will look at migrating to it in due course. |
|
Note I've cherry-picked the libjade change into master so you don't have to keep carrying it. |
|
Great to hear there's a demand for cUR! Your review and feedback are really welcome, and you can count on me to make adjustments if needed. |
|
I'd say for the purposes of keeping this more easily rebase-able, it would be a good idea to structure it as the following:
I'd leave the removal of the existing components out for now since nuking them after the migration is trivial, and rebasing with them deleted will be painful if there are changes there while this is under review. |
|
Commits structured as requested. |
|
Done: cUR:
Jade:
|
Perfect, this is exactly what I envisioned with the state based API :) |
|
cUR had a fix and improvements on recent push. |
|
@odudex I'm dubious about the cUR optimizations added TBH. e.g.:
IMO cUR is too young a project to be complicating the impl with micro-optimizations when you are still making high level changes like design optimizations and API cleanups, and still finding bugs. At the very least these changes should be (1) separate re-implementations of functions, not ifdef'd code complicating the existing functions (2) default to a reasonable size/speed tradeoff and (3) be gated by cpu defines only (so builders don't have to iterate the source files trying to determine which set of platform defines they need to set). I also think you might want to hold micro-optimizations back until the library has been completely tested/reviewed, although that last one is just IMO. Finally, please note that the github user greenaddress is no longer associated with Blockstream or Blockstream Green in any way. |
Fair points, most of them taken.
For Jade, |
|
cUR + k_quirc on the right: Jade_r.mp4 |
Add cUR (https://github.com/odudex/cUR), a pure-C implementation of BCR-2020-005 UR encoding, pinned at the head of its main branch. No other changes - the source migration follows in the next commit.
Move the UR transport layer from the C++20 bc-ur library (and its C shim with placement-new sizing constants) to cUR, a pure-C implementation of BCR-2020-005, used as the UR transport envelope only - all payload CBOR remains TinyCBOR in bcur.c. - main/bcur.c and main/selfcheck.c moved to the cUR API: heap encoder/decoder handles, results borrowed from the decoder, fragments freed with free(). Encoder output is always uppercase. - collect_any_bcur() now replaces the decoder via qr_data->ctx on hard failure, and bcur_scan_qr() re-reads it after scanning. - selfcheck: a duplicate part is deduped before being counted, so the 'processed parts' expectation differs when the same part is presented twice. - Build with UR_ENVELOPE_ONLY to skip cUR's payload-type codecs; libjade links the same component's host static lib (bundled SHA-256, no mbedcrypto/wally dependencies). The esp32_bc-ur component is left in place (now unused) so it can be removed separately once the migration has settled.
Drive the animated-QR progress bar with ur_decoder_estimated_percent_complete_weighted() instead of the pure received/expected fragment counts. The weighted estimate also credits mixed fountain parts that have not yet resolved a pure fragment, so progress keeps moving where the count-based bar would stall at 'almost done' until the final reconstruction. Capped at 99% until the decode actually completes.
Add k_quirc (https://github.com/odudex/k_quirc), a rewritten ESP32-optimized QR decoder (bilinear/adaptive thresholding, span-based flood fill, SPIRAM-aware allocation), pinned at the head of its master branch. No other changes - the source migration follows in the next commit.
Rewrite the thin wrapper in main/qrscan.c to the k_quirc_* API, folding quirc_extract + quirc_decode into a single k_quirc_decode() and dropping the caller-allocated datastream scratch buffer (now managed internally). k_quirc_end() is called with find_inverted=false to preserve current behavior. Downstream callers use the qr_data_t / jade_camera_scan_qr / scan_qr wrapper API and are unaffected. Also update the three build references: main/CMakeLists.txt PRIV_REQUIRES, libjade include dirs, and the libjade.c amalgamated .c includes. The vendored esp32-quirc component is left in place (now unused) so it can be removed separately once the migration has settled.
- qrscan: test data_type as a bitmask rather than with ==, so codes containing a Kanji segment are rejected (PR303-007) - bcur: make UR_DECODER_ERROR_UNSUPPORTED_SIZE stop the scan, reported from bcur_scan_qr rather than the camera callback; move the decoder into a stable context struct, removing the refetch-qr_data->ctx hazard; show the decoder reset after a checksum failure (PR303-008) - bcur/qrmode: add bcur_can_create_qr_icons() and check it before encoding, escalating QR density or erroring instead of emitting a stream the decoder rejects (PR303-008) - libjade: define K_QUIRC_ADAPTIVE_THRESHOLD and K_QUIRC_BILINEAR_THRESHOLD to match the firmware build (PR303-016) - selfcheck: use strcmp for the bc-ur result type and encoded parts 1-2; keep part 3 a prefix check but require more than its header (PR303-025)
This PR proposes, for your evaluation, replacing two libraries in the QR scan/display path:
components/esp32-quirc→ k_quirc submodule — QR decodingcomponents/esp32_bc-ur(C++) → cUR (C) submodule — BC-UR transport (bytewords, fountain codes, multi-part assembly)Both were originally developed for Kern and Krux projects. cUR is used envelope-only: all payload CBOR stays in TinyCBOR in
main/bcur.c, andmain/bcur.his unchanged, so no callers needed modification. A third small commit makes the libjadeESP_LOGmocks accept non-literal tags (k_quirc logs with aTAGvariable).Measured impact (esp32s3 dev config, vs master)
jade.binlibstdc++qr_qvga_*camera fixtures (desktop benchmark feeding the same 220×220 scan input to both decoders at-O2; per-image speedups 1.9–3.2×, decode parity 13/13).Compatibility and verification
UR:CRYPTO-PSBTparts, wire-format parity with bc-ur. Fountain fragments are additionally cross-validated in cUR's own test suite against foundation-ur-py output.processed_partsstays at 1 when the same part is presented twice (bc-ur counted 2).test_jade.pypasses via libjade, including the multi-partUR:CRYPTO-PSBTcamera-scan fixture.--psram --unamalgamated).Caveats
Motivation
If all goes well, give back from DIY community to the Jade project and at the same time get reviews to help harden these tools.