Skip to content

Cleaner article capture: strip in-body clutter, keep section breaks and captions - #608

Open
fredrivett wants to merge 4 commits into
mainfrom
fredrivett/dropcaps-scraping-improvement
Open

fredrivett wants to merge 4 commits into
mainfrom
fredrivett/dropcaps-scraping-improvement

Conversation

@fredrivett

@fredrivett fredrivett commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

What & why

Long-form articles were being captured with publisher furniture mid-body ("Read More" cards, subscribe prompts, podcast "Listen on" widgets), with section dividers dropped so sections ran together, and with photo captions rendered as ordinary paragraphs. A new pre-Readability pass (article-cleanup.ts) removes blocks whose class names match a short generic clutter list — guarded so anything holding prose, a large share of the page's text, the article itself or a tweet marker is kept — turns SVG/empty dividers and * * * paragraphs into <hr>, and turns classed caption divs into <figcaption> (now kept in the markdown). The reader gives the paragraph after each section break a drop cap (our own typography, since source drop caps are mostly CSS-only) and sets captions small, muted and tight under their image.

Checklist

  • User-facing change → added/updated a PostHog event (and a funnel/insight for a new flow), or N/A — N/A, improves existing capture/reader rendering, no new action
  • User-facing or service change → docs updated (README features/roadmap/services, .env.example, in-app help, CONTRIBUTING.md, onboarding checklist), or N/A — N/A, no new feature or service
  • New error paths report via captureServerException / the error boundary, or N/A — N/A, runs inside the existing never-throw extraction
  • New behavior is covered by tests, or N/A — unit tests for matching, guards, dividers, captions and break tidying; a synthetic sectioned-magazine fixture; and a check that the real article fixtures come through unchanged
  • If this fixes a recurring defect, considered a guardrail via the learn skill, or N/A — N/A

Notes

Only new captures and re-captures pick this up; existing items keep their stored content. The fixture copies a real magazine layout's structure and class names with placeholder prose. Adds an Article/ArticleDetailView story with a Sectioned variant.

🤖 Generated with Claude Code


Summary by cubic

Improves long-form article capture by removing in-body publisher furniture, preserving section breaks, and keeping photo captions.

A pre-Readability cleanup pass now strips "Read More" cards, subscribe prompts, podcast widgets, and other clutter whose class names match a short generic list — guarded so prose, wrappers holding a large share of the page's text, the article itself, and tweet markers are never removed. Empty or SVG-only dividers and * * * paragraphs become <hr> section breaks, and classed caption divs become <figcaption> elements that survive into the markdown. The reader gives the paragraph after each break a drop cap and renders captions as small, muted text directly under their image.

  • Guards apply to the matched element itself, so a clutter class on an articleBody wrapper, or a divider/caption class on an image, no longer slips through.
  • A Storybook play test asserts the rendered structure the reader's sibling selectors rely on: dropped caps after breaks and the caption sitting tight under its image.
  • Only new captures and re-captures pick this up; existing stored content is unchanged.
  • Real article fixtures are verified to pass through unchanged, and a synthetic magazine fixture covers the new paths.

Written for commit 32f8097. Summary will update on new commits.

Review in cubic

fredrivett and others added 2 commits October 5, 2026 21:30
…rticles

Readability lets publisher furniture inside the article body through
("Read More" cards, subscribe prompts, podcast players), drops SVG-only
section dividers as contentless, and flattens classed caption divs into
body paragraphs. A pre-Readability pass now removes blocks whose class
names match a short generic clutter list (guarded so prose, wrappers, the
article itself and tweet markers are never removed), turns dividers and
asterism paragraphs into <hr>, and turns caption divs into <figcaption>,
which the markdown now keeps.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The paragraph after each section break gets a drop cap, and captions sit
just under their image in smaller, muted type. Adds an ArticleDetailView
story covering a sectioned article.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
abode Ignored Ignored Preview Oct 5, 2026 8:38pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread app/src/lib/article-cleanup.ts Outdated
Comment thread app/src/lib/article-cleanup.ts Outdated
Comment thread app/src/components/article/article-prose.ts
fredrivett and others added 2 commits October 5, 2026 21:37
A clutter-classed articleBody wrapper, or a divider/caption class on an
image itself, slipped past guards that only checked descendants.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reader's sibling selectors only match while captured markdown still
renders breaks and kept captions as paragraph siblings; assert the drop
caps and the tight caption gap in the rendered reader.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant