diff --git a/AGENTS.md b/AGENTS.md index 16d702d8..ae1250b8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,6 +92,10 @@ In docs content (`src/content/docs/**`) and docs-supporting components: `import { Code } from '@astrojs/starlight/components';` - Do not use: `import Code from "astro/components/Code.astro";` +- Prefer multiline template literals: `code={\`...\`}`. +- Give every content line the same 2-space base indent (Starlight strips the common indent on render). +- Do not put blank lines inside the template — Prettier MDX strips indentation after blank lines and corrupts nested YAML/Ruby. Separate sections with `#` comment lines instead. +- Do not use `"...\n" +` string concat for snippets unless a concrete Prettier conflict remains after following the no-blank-line rule (should be rare). ### Accuracy Rules diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 2757205a..885da47f 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -7,6 +7,14 @@ import { Code } from "@astrojs/starlight/components"; This page outlines recent breaking changes, purged legacy aliases, and migration steps for older feed configurations. +## 0.24.0 + +Upgrade notes for configs and integrators targeting gem **0.24.0**: + +1. **`auto` strategy chain** — Default `strategy: auto` is `faraday` → `botasaurus` only. Pin `strategy: browserless` (or `--strategy browserless`) when you need Browserless preload/interaction; Browserless is not a fallback tier. +2. **RSS enclosures** — Configure media with the `enclosure` selector. RSS does not promote `image` into ``; images stay on the description / JSON Feed `image`. +3. **Ruby dual-format / telemetry** — Prefer `Html2rss.feed_result` for one scrape that must render RSS and JSON Feed (or be Marshal-cached). Read `result.status.to_h` for scrape telemetry (`selected_strategy`, `attempt_count`, `strategy_attempts` under `auto`). + ## Removed Legacy Selector Aliases In previous versions, `html2rss` accepted legacy selector names with a deprecation warning. These shims have been removed: @@ -26,14 +34,12 @@ Update any occurrences of `pubDate` or `updated` in your `selectors` block to `p selectors: updated: selector: ".date" - -# Current & Required - -selectors: - published_at: - selector: ".date" + # Current & Required + selectors: + published_at: + selector: ".date" `} -lang="yaml" + lang="yaml" /> ## Removed Channel Attributes @@ -48,16 +54,14 @@ lang="yaml" strategy: browserless headers: User-Agent: "CustomAgent/1.0" - -# Current & Required - -strategy: browserless -headers: - User-Agent: "CustomAgent/1.0" -channel: - url: "https://example.com/articles" + # Current & Required + strategy: browserless + headers: + User-Agent: "CustomAgent/1.0" + channel: + url: "https://example.com/articles" `} -lang="yaml" + lang="yaml" /> ## Migration Checklist @@ -66,4 +70,7 @@ When upgrading to modern `html2rss` releases: 1. **Rename date selectors**: Ensure date selectors use `published_at` rather than `updated` or `pubDate`. 2. **Move channel-level transport keys**: Ensure `strategy` and `headers` are defined at the top level of the YAML file. -3. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. +3. **Pin Browserless when needed**: If you relied on `auto` falling through to Browserless, set `strategy: browserless` (or `--strategy browserless`). +4. **Use `enclosure` for RSS media**: Do not rely on `image` becoming an RSS ``; select podcast/media URLs with `enclosure`. +5. **Prefer `feed_result` for dual-format / cache**: Integrators that need RSS + JSON Feed from one scrape (or Marshal caching) should use `Html2rss.feed_result` and `status.to_h`. +6. **Validate configurations**: Run `html2rss validate config.yml` to ensure your YAML conforms to the current schema. diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx index c2f34468..fbe522f4 100644 --- a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx +++ b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx @@ -9,9 +9,9 @@ Some websites load their content dynamically using JavaScript. Static fetch path ## Solution -Use a [browser-based extraction strategy](/ruby-gem/reference/strategy/) when JavaScript-heavy pages do not work with default static fetching. +Default `strategy: auto` already tries `faraday` then `botasaurus` (when `BOTASAURUS_SCRAPER_URL` is configured). That covers many JS-rendered listing pages without pinning a strategy. -`browserless` is common for this workflow, and `botasaurus` is an alternate browser-based strategy when you run a Botasaurus scrape API. +Pin [`browserless`](/ruby-gem/reference/strategy/#browserless) when you need headless Chrome with preload (wait, click, scroll) or other Browserless-only controls — Browserless is not part of the `auto` chain. Keep the strategy at the top level and put request-specific options under `request`: diff --git a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx index a28968bf..d346afa0 100644 --- a/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/managing-feed-configs.mdx @@ -53,22 +53,46 @@ You can define global settings that apply to all feeds, and then define individu + +### Ruby API: FeedResult + +Prefer `Html2rss.feed_result` when one scrape must render as both RSS and JSON Feed, or when you Marshal-cache the scrape and render later. `Html2rss.feed` and `Html2rss.json_feed` remain convenience wrappers over the same path. -# Build a specific feed from the YAML file +`FeedResult` is an opaque handle. Public surface: -my_feed_config = Html2rss.config_from_yaml_file('feeds.yml', 'my-first-feed') -rss = Html2rss.feed(my_feed_config) -puts rss +- `empty?` — whether the scrape produced items +- `channel_title` — channel title string only +- `to_rss` / `to_json_feed(feed_url:)` — render formats +- `status` — scrape telemetry (`Html2rss::Status`) -# If the YAML file contains only one feed, you can omit the feed name +`status.to_h` is the stable observability payload. Always includes `version` and `dedup_dropped`. When present, it may also include `scraper_tallies`, `selected_strategy`, `attempt_count`, and `strategy_attempts` (auto-fallback attempts; empty outside `strategy: auto`). -single_feed_config = Html2rss.config_from_yaml_file('single.yml') -rss = Html2rss.feed(single_feed_config) -puts rss + { version: "...", dedup_dropped: 0, selected_strategy: :botasaurus, ... } `} -lang="ruby" + lang="ruby" /> +Depth and method contracts: [YARD for `html2rss`](https://www.rubydoc.info/gems/html2rss). + ### Command Line `botasaurus` -> `browserless`. +- `--strategy`: Optional request strategy (`auto`, `faraday`, `browserless`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`. Pin `browserless` explicitly when you need headless Chrome (preload/interaction). - `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`. - `--items_selector`: Optional CSS selector hint for item extraction. - `--max-redirects`: Maximum redirects to follow per request. @@ -77,7 +77,7 @@ If all fallback tiers run but still extract zero items, html2rss raises: - `No RSS feed items extracted after auto fallback ...` -If failures continue after URL/surface fixes, retry with an explicit browser-based override (`--strategy browserless`), or `--strategy botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. +If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so the `auto` Botasaurus tier can run, or pin `--strategy browserless` when you need Browserless preload/interaction. Start by changing the input URL to a direct listing/update page, then move to explicit selectors if needed. diff --git a/src/content/docs/ruby-gem/reference/selectors.mdx b/src/content/docs/ruby-gem/reference/selectors.mdx index 9b4e5578..616d33c1 100644 --- a/src/content/docs/ruby-gem/reference/selectors.mdx +++ b/src/content/docs/ruby-gem/reference/selectors.mdx @@ -288,7 +288,9 @@ To create a custom GUID for an item, provide a list of selector names to the `gu ### Enclosures -To add an enclosure (e.g., an image, audio, or video file) to an item, use the `enclosure` selector to specify the URL of the file. +Use the `enclosure` selector to attach media (audio, video, or other non-image files) to an item. The selector is wired into each article and rendered into the feed. + +RSS `` uses the first **non-image** enclosure. Images stay on the item description (and on JSON Feed `image` / attachments) — they are not promoted into RSS ``. For podcast/media RSS, select a non-image resource with `enclosure`. `botasaurus` -> `browserless`. +- **`auto`** (default): Tries concrete strategies in order: `faraday` -> `botasaurus`. - **`faraday`**: Makes a direct HTTP request. It is fast but does not execute JavaScript. -- **`browserless`**: Renders the website in a headless Chrome browser, which is necessary for JavaScript-heavy sites. -- **`botasaurus`**: Delegates fetching to a Botasaurus scrape API. This is opt-in and requires `BOTASAURUS_SCRAPER_URL`. +- **`botasaurus`**: Delegates fetching to a Botasaurus scrape API. Included in the `auto` chain; requires `BOTASAURUS_SCRAPER_URL` when that tier runs (or when you pin `strategy: botasaurus`). +- **`browserless`**: Renders the website in a headless Chrome browser. **Explicit only** — set `strategy: browserless` or `--strategy browserless` (not part of `auto`). - **`local_file`**: Reads HTML content directly from a local file on disk without making network requests. `strategy` is a top-level config key. Request-specific controls live under `request`. -`auto` falls back to the next strategy when the current attempt errors or extracts zero items. Use explicit `--strategy ...` only when you need to force a specific transport for troubleshooting or reproducibility. +`auto` falls back to the next strategy when the current attempt errors or extracts zero items. Pin a concrete strategy when you need a specific transport (for example Browserless preload/interaction, or a forced Botasaurus-only run). ## `auto` (default) The default strategy chain is: -`faraday` -> `botasaurus` -> `browserless` +`faraday` -> `botasaurus` + +`browserless` is not in this chain. Pin it when you need headless Chrome with preload (wait/click/scroll) or other Browserless-only controls. Auto fallback shares one request budget across all strategy attempts. For pagination-heavy or dynamic pages, increase `request.max_requests` (or `--max-requests`) when retries exhaust the budget. -Auto fallback decisions are hidden at the default `LOG_LEVEL=warn`; run with `LOG_LEVEL=info` to include them in CLI output. +Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `selected_strategy`, `attempt_count`, and `strategy_attempts` (see [Managing Feed Configs](/ruby-gem/guides/managing-feed-configs/#ruby-api-feedresult)). Auto fallback decisions are also visible at `LOG_LEVEL=info` (hidden at the default `LOG_LEVEL=warn`). ## `faraday` @@ -204,12 +206,14 @@ For custom Browserless websocket endpoints, `BROWSERLESS_IO_API_TOKEN` is mandat ## `botasaurus` -`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. This strategy is explicit opt-in and requires: +`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of `auto`, or when you pin `strategy: botasaurus`. + +Requirements: -- `strategy: botasaurus` - `BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`) +- pin `strategy: botasaurus` only when you want to skip Faraday and force this transport -html2rss still enforces local request policy preflight and timeout budget. Botasaurus handles browser navigation/rendering internals, so some policy details are delegated to upstream execution. +html2rss still enforces local request policy preflight and timeout budget. When a total request timeout remains, Botasaurus `max_retries` and `wait_timeout_seconds` are clamped so upstream work fits the remaining budget. Botasaurus handles browser navigation/rendering internals, so some policy details are delegated to upstream execution. ### Configuration @@ -222,7 +226,7 @@ html2rss still enforces local request policy preflight and timeout budget. Botas request: botasaurus: navigation_mode: auto - max_retries: 2 + max_retries: 1 headless: false `} lang="yml" @@ -231,7 +235,7 @@ html2rss still enforces local request policy preflight and timeout budget. Botas Supported `request.botasaurus` options: - `navigation_mode` (`auto`, `get`, `google_get`, `google_get_bypass`; default `auto`) -- `max_retries` (`0..3`; default `2`) +- `max_retries` (`0..3`; default `1`) - `wait_for_selector` (string) - `wait_timeout_seconds` (integer) - `block_images` (boolean) @@ -252,7 +256,7 @@ Example scrape-API payload shape: { "url": "https://example.com", "navigation_mode": "auto", - "max_retries": 2, + "max_retries": 1, "headless": false } `}