browser-api-scraper

Browser API for Automation and Web Scraping — Free, Open-Source Tool

A reusable toolkit for scraping tables hidden behind dynamic filters — where the filter changes the page content, the URL never changes, and a plain HTTP request gets back an empty shell with nothing in it. Built and documented against one real target (gname.com/dropcatch) as a demonstration of the methodology, not a one-off scraper.

The code here is free and MIT-licensed. It drives a remote, CDP-controllable browser via 2Captcha's Browser API — see the Browser API product page and its API docs. That underlying service is a paid, quota-metered account (see Requirements) — this repository doesn't make that part free, only the tool itself.

Learning project. This repository shows a technique — intercepting the network response a page already makes, instead of parsing rendered HTML — using a cloud browser reached over CDP. See Responsible use before pointing it at anything.

Русская версия — README.ru.md

What this solves

Some sites render their results table entirely client-side, from an AJAX call the page's own JavaScript makes after you touch a filter. The URL stays the same regardless of what's selected, and a plain fetch/curl against that URL returns the server-rendered skeleton — an empty <table>, no rows, no data. You need an actual browser to trigger the request, but you don't need to parse the DOM it renders: the response the page fetches is already structured JSON. This project drives a real (cloud, CDP-controlled) browser to fill the filter form, click search, and capture that JSON response directly — bypassing the DOM entirely for the data itself.

Requirements

• Node.js 20+
• A 2Captcha account with Browser API access (this is what supplies the remote, CDP-controllable Chrome instance — see 2captcha.com, the Browser API product page, and its API reference)
• A residential proxy configured on your Browser API profile. Not optional: the demo target sits behind Cloudflare, which rejects datacenter IPs outright (HTTP 403, a "Just a moment..." challenge page). Without a residential proxy on the profile, the page never loads at all.

Installation

git clone <this-repo>
cd browser-api-scraper
npm install
cp .env.example .env

Fill in .env:

TWOCAPTCHA_API_KEY=your_key_here
TWOCAPTCHA_ACCOUNT_ID=your_account_id
TWOCAPTCHA_PROFILE_ID=your_profile_id
# BROWSER_CONNECTION_URI is optional — leave empty unless you already have a
# still-valid connectionUri from a previous run you want to reuse.

Quick start

Try it without touching the network or your 2Captcha quota at all — runs the full pipeline (parse → validate → export) against a captured sample response:

npx tsx src/cli.ts run --filters filters/basic.json --from-fixture

That writes output/domains.csv. Once your .env is filled in, drop --from-fixture to hit the live site:

npx tsx src/cli.ts run --filters filters/basic.json

Console output is deliberately honest about coverage, not just a row count — see Limits for why:

Найдено по фильтру: 8299
Получено: 50
Показано: 0.6% — сайт отдаёт максимум 50 строк за запрос
Сохранено: output/domains.csv
Сузь фильтр, чтобы увидеть больше: уменьши диапазон длины, возьми одну дату или поставь пороги по метрикам.

Filters file format

A filters file is JSON, validated against a schema before anything is sent anywhere. You don't need to know the target's internal field names (ymhz, ymcd_1, ...) — the schema exposes readable names and maps them internally.

{
  "tlds": ["com"],
  "domainLength": { "min": 4, "max": 5 },
  "dropDate": "2026-08-06",
  "include": { "values": ["shop"] },
  "exclude": { "values": ["free"] },
  "deletionTypes": [1, 23],
  "sorting": "highDomainAuthority",
  "metrics": {
    "domainAuthority": { "min": 20 },
    "referringDomains": { "min": 50 }
  },
  "maxRequests": 50,
  "delayMs": 3000
}

Field	Type	Notes
tlds	string[], max 1 item	e.g. "com", "co.uk". No fixed whitelist — validated by format only.
domainLength	{ min?, max? }	Both optional, min <= max.
include	{ values: string[], position? }	Substring match, OR across values. position is accepted but rejected at validation — see What we learned.
exclude	same shape as include	Domain must contain none of the values.
dropDate	string, YYYY-MM-DD	Exact date match.
deletionTypes	number[]	Site-specific codes, no fixed whitelist.
freeBackorder	boolean	Unverified in practice — see below.
sorting	one of the named presets below	Not raw sort codes.
metrics	object of { min?, max? } per metric	domainAuthority, pageAuthority, historicalAuthority, trustFlow, citationFlow, backlinks, referringDomains, googleIndex, bingIndex.
maxRequests	number, 1–1000, default 200	Not yet enforced by a request loop (see Limits) — reserved for the slicing feature.
delayMs	number, ≥2000, default 3000	Overridable per-run with --delay.

sorting presets (named, not raw site codes — confirmed live against the target):

• highHistoricalAuthority
• highDomainAuthority
• highPageAuthority
• highAge
• highValuation

Two working examples are in filters/: basic.json (TLD + length + date) and seo.json (adds DA/referring-domains thresholds). Both are deliberately narrow — see Limits for why a wide filter is a bad idea here.

CLI

npx tsx src/cli.ts run --filters <path> [options]

Flag	Default	Description
--filters <path>	(required)	Path to a filters JSON file, validated against the schema above.
--format <csv|json|jsonl>	csv	Output format.
--out <path>	output/domains.<format>	Output file path.
--from-fixture	off	Run the whole pipeline against the bundled fixture, no browser/.env needed — the main way to develop and debug without spending Browser API quota.
--delay <ms>	—	Overrides delayMs from the filters file for this run (still validated, ≥2000).

Output formats:

• CSV — named Row fields only (raw is deliberately excluded — it's ~60 mostly unconfirmed fields, including a large embedded JSON blob). UTF-8 with a BOM, so Excel doesn't mangle it.
• JSON — { count, retrieved, rows }, rows keep the full raw object.
• JSONL — one Row object per line (with raw), no wrapping object — the streaming-friendly format.

Limits

• 50 rows per request, hard. The target ignores any pageSize-equivalent parameter for anonymous users and always returns exactly 50 rows (or fewer, if fewer match).
• Pagination requires login and is not supported. Past the first page, the site expects an authenticated session; this project deliberately does not automate login, so it only ever sees page 1.
• This tool gives you a slice by filter, not a full export. If your filter matches more than 50 results, you're only getting the first 50 the site would show — narrow the filter (tighter length range, a single date, metric thresholds) rather than treating the output as complete. The CLI's console summary makes this explicit every run (see Quick start).
• A recursive slicing feature to work around the 50-row cap by splitting a wide filter into many narrow ones is designed but not yet built — maxRequests/delayMs exist in the filters schema for it already, but today's CLI makes exactly one request per run.

Adding an adapter for another site

This isn't (yet) a plugin system with a formal extension interface — the target's specifics are threaded directly through a handful of files. To point this methodology at a different site, you'd change:

1. docs/recon.md — redo the recon from scratch for the new target: the hidden form (if any), the actual network endpoint, request/response shape, required headers. Don't skip this or guess — see What we learned for why guessing the wrong thing here is expensive.
2. src/types.ts / src/parse.ts — new Row shape and a new parseResponse matching the new target's response format.
3. src/filters.ts — new FiltersSchema (whatever filters the new target actually supports) and a new buildFormFields/validateRows pair for its field names.
4. src/harvest.ts — new selectors (form ID, submit button, endpoint path) and whatever quirks the new target has (does it read its own hidden form for every field, or does some subset silently defer to a visible widget instead? Don't assume — verify per field).
5. src/browser.ts / src/captcha.ts stay as-is — those talk to the Browser API, not the target site.

Browser API limits

• One profile = one active CDP session. A second concurrent connection on the same profile gets profile_locked — don't run two instances against the same profile.
• 30 minutes per session, hard cap. Long-running jobs need to reconnect partway through (not yet implemented here — today's harvest is a single request per run, well under the limit).
• connectionUri is always requested via POST https://api.2captcha.com/browser/connection — never assembled by hand. Set BROWSER_CONNECTION_URI in .env to reuse an already-issued one instead of requesting a fresh one (useful across quick dev restarts; the caller is responsible for it still being valid).

Troubleshooting

profile_locked — another CDP session is already open on this profile (a second terminal, a crashed process that didn't disconnect cleanly, etc.). Close it, or wait; this project does not retry on profile_locked — retrying immediately just repeats the same rejection and wastes a request.

Cloudflare 403 / "Just a moment..." page — the Browser API profile doesn't have a residential proxy configured, or the proxy dropped. Datacenter IPs get rejected outright. Not retried automatically — check the profile's proxy settings.

"#cha-form did not accept the expected values after writing them" — the FormData verification step (see What we learned) caught a field that didn't take the value it was given. This is a real defect worth investigating, not a transient error — it isn't retried automatically either.

Slider verification appears — across every live run so far, a residential proxy alone was enough and this never came up. If it does, src/captcha.ts wraps the Browser API's Captcha CDP domain defensively, but it has never been exercised against a real captcha — treat it as best-effort, not a guarantee.

Responsible use

• Check robots.txt and the target site's Terms of Service before scraping anything — this is the first step, not a footnote at the end.
• Look for an official API before scraping. gname has one: POST /api/dropcatch/list, documented at https://www.gname.com/domain/api/dropcatch/list — but it requires an approved APPID, so it's not available to an anonymous visitor without going through gname first. There's also a "Deletion List" export link visible directly on the page (/download/expired) — not verified here to cover the same filters or to work without login, but worth checking before scraping if it fits your use case.
• This project only ever touches the public first page. The target requires login past page one; logging in is deliberately not automated anywhere in this codebase.
• Respect the pauses. Default delay between requests is 3000ms, floor is 2000ms (enforced by the filters schema); default request budget is bounded (maxRequests, 1–1000, default 200).
• This is a learning project, built to demonstrate a scraping methodology, not a product aimed at any specific site. Responsibility for complying with a target site's own rules — whichever site you point this at — is entirely on whoever runs it.

What we learned in practice

This section is arguably the most useful part of this README — real findings from actually building this against a live target, not assumptions.

• The hidden form is a real contract for filters — but not a universal one. Filling a target's hidden form and reading it back afterward looked like a solid, generic approach right up until it wasn't.
• Text "contains" fields are read from their visible widget, not the hidden form — and there's no way to tell from the hidden form alone. Writing a value straight into the hidden field, confirmed present via FormData immediately before clicking search, still returned the fully unfiltered result. The search handler for that field never reads the hidden form at all; it reads the widget the user actually sees.
• <input type="text"> strips line breaks from its own value — by HTML5 spec, not site behavior. Assigning field.value = "a\nb" on a single-line text input silently collapses to "ab" in every standards-compliant browser. A field that needs to accept multiple newline-separated values structurally cannot be a single-line hidden input; it has to be a real <textarea>, and the multi-value technique has to target that element specifically.
• Confirming a write is not confirming it was applied. FormData verification only proves a value landed in a form field — not that anything downstream used it. The only real proof a filter took effect is checking the results against what was asked for (validateRows in this codebase): does every returned row actually satisfy the filter? Two identically "successful" write-and-verify sequences produced completely different outcomes here — one that worked and one that silently returned everything.
• Slider/captcha verification never appeared once across 8 live runs. A residential proxy on the Browser API profile was sufficient on its own. Captcha handling in this project (src/captcha.ts) is defensive code for if it ever shows up, not something this project actually had to lean on.
• Blocking analytics/ad requests cut per-run traffic to roughly a third of the original. A session without blocking transferred close to 8 MB per page load; with only known analytics/ad hosts blocked (nothing else — images, fonts, scripts, and iframes are left alone, since blocking those breaks the slider verification), a full run transferred about 3 MB. On a quota-metered Browser API, that's not a rounding error.

Stack

TypeScript, Node 20, Playwright (chromium.connectOverCDP), zod, commander, vitest. No database — output is CSV/JSON/JSONL to disk.

An HTML-fragment parsing fallback (for a target whose response isn't JSON) was considered and deliberately not built — see src/parse.ts: it would need a real HTML sample with actual column-header text to map headers to fields without guessing, and no such sample exists for this target (its response has only ever been observed as JSON). A non-JSON response fails loudly instead of being silently mis-parsed.

License

MIT — see LICENSE.