From 7ae229830ad8eded6dda200cfab9c236e74333e1 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Thu, 13 Aug 2026 12:01:48 +0200 Subject: [PATCH 1/5] docs(infrastructure): botasaurus-first auto strategy for gem 0.24.0 --- .../guides/handling-dynamic-content.mdx | 4 +-- .../docs/ruby-gem/reference/cli-reference.mdx | 4 +-- .../docs/ruby-gem/reference/strategy.mdx | 28 +++++++++++-------- 3 files changed, 20 insertions(+), 16 deletions(-) diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx index c2f34468..fbe522f4 100644 --- a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx +++ b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx @@ -9,9 +9,9 @@ Some websites load their content dynamically using JavaScript. Static fetch path ## Solution -Use a [browser-based extraction strategy](/ruby-gem/reference/strategy/) when JavaScript-heavy pages do not work with default static fetching. +Default `strategy: auto` already tries `faraday` then `botasaurus` (when `BOTASAURUS_SCRAPER_URL` is configured). That covers many JS-rendered listing pages without pinning a strategy. -`browserless` is common for this workflow, and `botasaurus` is an alternate browser-based strategy when you run a Botasaurus scrape API. +Pin [`browserless`](/ruby-gem/reference/strategy/#browserless) when you need headless Chrome with preload (wait, click, scroll) or other Browserless-only controls — Browserless is not part of the `auto` chain. Keep the strategy at the top level and put request-specific options under `request`: diff --git a/src/content/docs/ruby-gem/reference/cli-reference.mdx b/src/content/docs/ruby-gem/reference/cli-reference.mdx index 961c1845..7d4a9d17 100644 --- a/src/content/docs/ruby-gem/reference/cli-reference.mdx +++ b/src/content/docs/ruby-gem/reference/cli-reference.mdx @@ -36,7 +36,7 @@ Command: `html2rss auto [URL]` Available options: -- `--strategy`: Optional request strategy (`auto`, `faraday`, `browserless`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus` -> `browserless`. +- `--strategy`: Optional request strategy (`auto`, `faraday`, `browserless`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`. Pin `browserless` explicitly when you need headless Chrome (preload/interaction). - `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`. - `--items_selector`: Optional CSS selector hint for item extraction. - `--max-redirects`: Maximum redirects to follow per request. @@ -77,7 +77,7 @@ If all fallback tiers run but still extract zero items, html2rss raises: - `No RSS feed items extracted after auto fallback ...` -If failures continue after URL/surface fixes, retry with an explicit browser-based override (`--strategy browserless`), or `--strategy botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. +If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so the `auto` Botasaurus tier can run, or pin `--strategy browserless` when you need Browserless preload/interaction. Start by changing the input URL to a direct listing/update page, then move to explicit selectors if needed. diff --git a/src/content/docs/ruby-gem/reference/strategy.mdx b/src/content/docs/ruby-gem/reference/strategy.mdx index 921baaa5..a90ac644 100644 --- a/src/content/docs/ruby-gem/reference/strategy.mdx +++ b/src/content/docs/ruby-gem/reference/strategy.mdx @@ -7,25 +7,27 @@ import { Code } from "@astrojs/starlight/components"; The `strategy` key defines how `html2rss` fetches a website's content. -- **`auto`** (default): Tries concrete strategies in order: `faraday` -> `botasaurus` -> `browserless`. +- **`auto`** (default): Tries concrete strategies in order: `faraday` -> `botasaurus`. - **`faraday`**: Makes a direct HTTP request. It is fast but does not execute JavaScript. -- **`browserless`**: Renders the website in a headless Chrome browser, which is necessary for JavaScript-heavy sites. -- **`botasaurus`**: Delegates fetching to a Botasaurus scrape API. This is opt-in and requires `BOTASAURUS_SCRAPER_URL`. +- **`botasaurus`**: Delegates fetching to a Botasaurus scrape API. Included in the `auto` chain; requires `BOTASAURUS_SCRAPER_URL` when that tier runs (or when you pin `strategy: botasaurus`). +- **`browserless`**: Renders the website in a headless Chrome browser. **Explicit only** — set `strategy: browserless` or `--strategy browserless` (not part of `auto`). - **`local_file`**: Reads HTML content directly from a local file on disk without making network requests. `strategy` is a top-level config key. Request-specific controls live under `request`. -`auto` falls back to the next strategy when the current attempt errors or extracts zero items. Use explicit `--strategy ...` only when you need to force a specific transport for troubleshooting or reproducibility. +`auto` falls back to the next strategy when the current attempt errors or extracts zero items. Pin a concrete strategy when you need a specific transport (for example Browserless preload/interaction, or a forced Botasaurus-only run). ## `auto` (default) The default strategy chain is: -`faraday` -> `botasaurus` -> `browserless` +`faraday` -> `botasaurus` + +`browserless` is not in this chain. Pin it when you need headless Chrome with preload (wait/click/scroll) or other Browserless-only controls. Auto fallback shares one request budget across all strategy attempts. For pagination-heavy or dynamic pages, increase `request.max_requests` (or `--max-requests`) when retries exhaust the budget. -Auto fallback decisions are hidden at the default `LOG_LEVEL=warn`; run with `LOG_LEVEL=info` to include them in CLI output. +Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `selected_strategy`, `attempt_count`, and `strategy_attempts` (see [Managing Feed Configs](/ruby-gem/guides/managing-feed-configs/#ruby-api-feedresult)). Auto fallback decisions are also visible at `LOG_LEVEL=info` (hidden at the default `LOG_LEVEL=warn`). ## `faraday` @@ -204,12 +206,14 @@ For custom Browserless websocket endpoints, `BROWSERLESS_IO_API_TOKEN` is mandat ## `botasaurus` -`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. This strategy is explicit opt-in and requires: +`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of `auto`, or when you pin `strategy: botasaurus`. + +Requirements: -- `strategy: botasaurus` - `BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`) +- pin `strategy: botasaurus` only when you want to skip Faraday and force this transport -html2rss still enforces local request policy preflight and timeout budget. Botasaurus handles browser navigation/rendering internals, so some policy details are delegated to upstream execution. +html2rss still enforces local request policy preflight and timeout budget. When a total request timeout remains, Botasaurus `max_retries` and `wait_timeout_seconds` are clamped so upstream work fits the remaining budget. Botasaurus handles browser navigation/rendering internals, so some policy details are delegated to upstream execution. ### Configuration @@ -222,7 +226,7 @@ html2rss still enforces local request policy preflight and timeout budget. Botas request: botasaurus: navigation_mode: auto - max_retries: 2 + max_retries: 1 headless: false `} lang="yml" @@ -231,7 +235,7 @@ html2rss still enforces local request policy preflight and timeout budget. Botas Supported `request.botasaurus` options: - `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`; default `auto`) -- `max_retries` (`0..3`; default `2`) +- `max_retries` (`0..3`; default `1`) - `wait_for_selector` (string) - `wait_timeout_seconds` (integer) - `block_images` (boolean) @@ -252,7 +256,7 @@ Example scrape-API payload shape: { "url": "https://example.com", "navigation_mode": "auto", - "max_retries": 2, + "max_retries": 1, "headless": false } `} From 49d723646e32d10091d057672e898f01a74cb73d Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Thu, 13 Aug 2026 12:01:48 +0200 Subject: [PATCH 2/5] docs(adapters): FeedResult status and enclosure RSS rules for 0.24.0 --- .../ruby-gem/guides/managing-feed-configs.mdx | 32 +++++++++++++++++++ .../docs/ruby-gem/reference/selectors.mdx | 4 ++- 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index a28968bf..07f0446d 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -69,6 +69,38 @@ puts rss lang="ruby" /> +### Ruby API: FeedResult + +Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON Feed, or when you Marshal-cache the scrape and render later. `Html2rss.feed` and `Html2rss.json_feed` remain convenience wrappers over the same path. + +`FeedResult` is an opaque handle. Public surface: + +- `empty?` — whether the scrape produced items +- `channel_title` — channel title string only +- `to_rss` / `to_json_feed(feed_url:)` — render formats +- `status` — scrape telemetry (`Html2rss::Status`) + +`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`). + + { version: "...", dedup_dropped: 0, selected_strategy: :botasaurus, ... } + +`} +lang="ruby" +/> + +Depth and method contracts: [YARD for `html2rss`](https://www.rubydoc.info/gems/html2rss). + ### Command Line ` uses the first **non-image** enclosure. Images stay on the item description (and on JSON Feed `image` / attachments) — they are not promoted into RSS ``. For podcast/media RSS, select a non-image resource with `enclosure`. Date: Thu, 13 Aug 2026 12:01:49 +0200 Subject: [PATCH 3/5] docs(use-cases): 0.24.0 migration notes for auto and enclosures --- .../guides/backward-compatibility.mdx | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 2757205a..6ab8e9d3 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -7,6 +7,14 @@ import { Code } from "@astrojs/starlight/components"; This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations. +## 0.24.0 + +Upgrade notes for configs and integrators targeting gem **0.24.0**: + +1. **`auto` strategy chain** — Default `strategy: auto` is `faraday` → `botasaurus` only. Pin `strategy: browserless` (or `--strategy browserless`) when you need Browserless preload/interaction; Browserless is not a fallback tier. +2. **RSS enclosures** — Configure media with the `enclosure` selector. RSS does not promote `image` into ``; images stay on the description / JSON Feed `image`. +3. **Ruby dual-format / telemetry** — Prefer `Html2rss.feed_result` for one scrape that must render RSS and JSON Feed (or be Marshal-cached). Read `result.status.to_h` for scrape telemetry (`selected_strategy`, `attempt_count`, `strategy_attempts` under `auto`). + ## Removed Legacy Selector Aliases In previous versions, `html2rss` accepted legacy selector names with a deprecation warning. These shims have been removed: @@ -30,8 +38,8 @@ Update any occurrences of `pubDate` or `updated` in your `selectors` block to `p # Current & Required selectors: - published_at: - selector: ".date" +published_at: +selector: ".date" `} lang="yaml" /> @@ -53,9 +61,9 @@ lang="yaml" strategy: browserless headers: - User-Agent: "CustomAgent/1.0" +User-Agent: "CustomAgent/1.0" channel: - url: "https://example.com/articles" +url: "https://example.com/articles" `} lang="yaml" /> @@ -66,4 +74,7 @@ When upgrading to modern `html2rss` releases: 1. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`. 2. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file. -3. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. +3. **Pin Browserless when needed**: If you relied on `auto` falling through to Browserless, set `strategy: browserless` (or `--strategy browserless`). +4. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS ``; select podcast/media URLs with `enclosure`. +5. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`. +6. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. From 6e10aa123ca5e09cddda2d65619d6a7f35f3b362 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Thu, 13 Aug 2026 13:39:30 +0200 Subject: [PATCH 4/5] docs: fix Code sample indentation Prettier was stripping Use string-concat Code props so YAML nesting and Ruby samples survive prettier --write. --- .../guides/backward-compatibility.mdx | 60 +++++++++---------- .../ruby-gem/guides/managing-feed-configs.mdx | 27 ++++----- 2 files changed, 43 insertions(+), 44 deletions(-) diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 6ab8e9d3..1eda3318 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -29,19 +29,19 @@ In previous versions, `html2rss` accepted legacy selector names with a deprecati Update any occurrences of `pubDate` or `updated` in your `selectors` block to `published_at`: ## Removed Channel Attributes @@ -49,23 +49,23 @@ lang="yaml" `strategy` and `headers` were previously tolerated inside the `channel` block. These legacy shims have been removed. Both keys must now be placed at the top level of your feed configuration: ## Migration Checklist diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index 07f0446d..70763c15 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -83,20 +83,19 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F `status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`). { version: "...", dedup_dropped: 0, selected_strategy: :botasaurus, ... } - -`} -lang="ruby" + code={ + "require 'html2rss'\n" + + "\n" + + "config = Html2rss.config_from_yaml_file('feeds.yml', 'my-first-feed')\n" + + "result = Html2rss.feed_result(config)\n" + + "\n" + + "result.to_rss\n" + + "result.to_json_feed(feed_url: 'https://example.com/feeds/my-first-feed.json')\n" + + "result.status.to_h\n" + + "\n" + + '# => { version: "...", dedup_dropped: 0, selected_strategy: :botasaurus, ... }' + } + lang="ruby" /> Depth and method contracts: [YARD for `html2rss`](https://www.rubydoc.info/gems/html2rss). From 38736e7cd76b98432fec127a486bff8ec0543634 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Thu, 13 Aug 2026 13:51:21 +0200 Subject: [PATCH 5/5] docs: prettier-safe Code sample indentation Encode the no-blank-line template rule in AGENTS.md and restore readable multiline Code samples that survive Prettier MDX. --- AGENTS.md | 4 ++ .../guides/backward-compatibility.mdx | 52 +++++++++---------- .../ruby-gem/guides/managing-feed-configs.mdx | 43 +++++++-------- 3 files changed, 46 insertions(+), 53 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 16d702d8..ae1250b8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,6 +92,10 @@ In docs content (`src/content/docs/**`) and docs-supporting components: `import { Code } from '@astrojs/starlight/components';` - Do not use: `import Code from "astro/components/Code.astro";` +- Prefer multiline template literals: `code={\`...\`}`. +- Give every content line the same 2-space base indent (Starlight strips the common indent on render). +- Do not put blank lines inside the template — Prettier MDX strips indentation after blank lines and corrupts nested YAML/Ruby. Separate sections with `#` comment lines instead. +- Do not use `"...\n" +` string concat for snippets unless a concrete Prettier conflict remains after following the no-blank-line rule (should be rare). ### Accuracy Rules diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 1eda3318..885da47f 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -29,18 +29,16 @@ In previous versions, `html2rss` accepted legacy selector names with a deprecati Update any occurrences of `pubDate` or `updated` in your `selectors` block to `published_at`: @@ -49,22 +47,20 @@ Update any occurrences of `pubDate` or `updated` in your `selectors` block to `p `strategy` and `headers` were previously tolerated inside the `channel` block. These legacy shims have been removed. Both keys must now be placed at the top level of your feed configuration: diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index 70763c15..d346afa0 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -53,20 +53,16 @@ You can define global settings that apply to all feeds, and then define individu ### Ruby API: FeedResult @@ -83,18 +79,15 @@ Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON F `status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`). { version: "...", dedup_dropped: 0, selected_strategy: :botasaurus, ... }' - } + code={` + require 'html2rss' + config = Html2rss.config_from_yaml_file('feeds.yml', 'my-first-feed') + result = Html2rss.feed_result(config) + result.to_rss + result.to_json_feed(feed_url: 'https://example.com/feeds/my-first-feed.json') + result.status.to_h + # => { version: "...", dedup_dropped: 0, selected_strategy: :botasaurus, ... } +`} lang="ruby" />