Repository navigation
fix(ingest): run on Cloudflare Workers - #587
Merged
Merged
Conversation
Workers bundles pick turndown's browser build, which parses an HTML string with the page's document. A Worker has no document, so every page failed with "document is not defined". Parse with domino (the parser turndown's Node build already uses) and hand turndown the element.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@chatcops/ingestdidn't work inside a Cloudflare Worker. Bundlers targeting Workers (wrangler, esbuild with theworkerd/browserconditions) pick turndown's browser build, and that build parses an HTML string with the globaldocument. A Worker has none, so every page failed withdocument is not defined.The fix parses the fragment with
@mixmark-io/dominoand passes turndown the element. domino is the parser turndown's Node build already uses, so Markdown output on Node doesn't change, and the existing extract tests pass as they were.The
node:fs/node:pathimports fordirectorysources turned out not to be a problem. They load fine undernodejs_compat, so they stay as they are.How I checked
tests/extract-workers.test.tsswaps in turndown's browser build and converts the pricing fixture (a table included) with no globaldocument. On dev it fails withReferenceError: document is not defined, and with this change it passes.ingest()ofhttps://chat.codercops.com/sitemap-index.xmlin Miniflare (workerd), bundled the way a Worker would be (esbuild, platform browser, conditionsworkerd,worker,browser,import,nodejs_compat), with a stub embedder and memory stores. On dev all pages failed. With this change: 30 pages indexed, 262 chunks, 0 failed, in 14.5 s.This is what the hosted crawl needs to run in a Workflow.