An MLS scraper API collects property listings, prices and agent details from real-estate portals without you running browsers or rotating IPs yourself. This repository is about the part most guides skip: MLS-class portals are among the hardest targets on the public web, and the difference between a working pipeline and a stalled one is how you handle failure and cost.
Every parameter and cost figure below was checked against the live ScrapingBee API on 2026-08-25.
- Why MLS data is hard to collect
- Never pay for a blocked request
- The escalation ladder
- Extracting listing fields
- When selectors rot
- Cost model for a listings pipeline
- Legal and licensing footing
Multiple Listing Service data reaches the public through portal front-ends, and those portals defend hard. Three things happen at once:
- Listing content is rendered client-side. The HTML that arrives from a plain HTTP request frequently has no listings in it at all, only an application shell.
- Anti-bot systems fingerprint the client. Datacenter IPs get challenged, then residential IPs get challenged, and the response is often a 403 or a silent empty result rather than an obvious block.
- Markup churns. Portals ship redesigns constantly, and any scraper pinned to CSS classes breaks without warning.
Point three is not theoretical. A published set of selectors for a major portal, result-card
and its pc-price and pc-meta-beds children, returned zero rows when tested against the live
site on 2026-08-25. Any MLS scraper you build needs to detect that condition rather than write
empty rows into a database.
This is the single most useful fact when working against a hostile target:
A request that fails with HTTP 500 costs 0 credits. Retrying is free.
Auto-Mode extends the idea. With mode=auto, ScrapingBee tries configurations from cheapest to
most expensive, charges only for the one that succeeds, and charges nothing at all when every
configuration fails:
curl -sS -G "https://app.scrapingbee.com/api/v1" \
-H "Authorization: Bearer YOUR-API-KEY" \
--data-urlencode "url=https://example-portal.com/listings" \
--data-urlencode "mode=auto" \
--data-urlencode "max_cost=75" \
-D - -o body.html | grep -i "^spb-auto-cost"The Spb-auto-cost header reports what was actually billed. A 0 there means every tier was
tried and none worked, and your balance is untouched. On a target that blocks you, that turns a
budgeting problem into an availability problem, which is a much easier thing to manage.
Five configurations, cheapest to most expensive:
| Configuration | Credits | When it is enough |
|---|---|---|
| Rotating proxy, no JavaScript | 1 | Server-rendered listing pages |
| Rotating proxy with JavaScript | 5 | Client-rendered content, no IP reputation checks |
| Premium proxy, no JavaScript | 10 | Static pages behind IP reputation filtering |
| Premium proxy with JavaScript | 25 | The common answer for portal search results |
| Stealth proxy with JavaScript | 75 | Aggressive anti-bot stacks |
premium_proxy=true is also what unlocks country_code, which matters because portals localise
listings and pricing.
Set max_cost to whatever a single record is worth to you. max_cost=25 refuses to spend 75
credits on a page that may not parse anyway.
extract_rules turns a page into JSON in the same request, so nothing has to come back as HTML.
The shape is a named list plus a per-item output map:
{
"listings": {
"selector": "SELECTOR-FOR-ONE-CARD",
"type": "list",
"output": {
"price": { "selector": "SELECTOR-FOR-PRICE" },
"beds": { "selector": "SELECTOR-FOR-BEDS" },
"baths": { "selector": "SELECTOR-FOR-BATHS" },
"address": { "selector": "SELECTOR-FOR-ADDRESS" },
"url": { "selector": "a", "output": "@href" }
}
},
"next_page": { "selector": "a[aria-label='Go to next page']", "output": "@href" }
}Pulling next_page in the same call is what makes pagination cheap: one request yields both the
rows and the pointer to the following page, so you never spend a request just to discover where
to go next. The full syntax is documented under
data extraction.
Confirm your selectors against the live page before wiring them into a pipeline. Selectors copied from any tutorial, including this one, have a short shelf life on portal sites.
Because portal markup changes so often, an MLS scraper needs a fallback that does not depend on
class names at all. ai_query describes the fields in plain language and costs 5 credits on top
of the base request:
curl -G "https://app.scrapingbee.com/api/v1" \
-H "Authorization: Bearer YOUR-API-KEY" \
--data-urlencode "url=https://example-portal.com/listings" \
--data-urlencode "mode=auto" \
--data-urlencode "ai_query=every property listing with its price, bed count, bath count and full address"A sound production pattern is to run selectors first, count the rows, and fall back to
AI extraction only when the count is
zero. You get selector economics on the normal path and resilience on the day a redesign lands.
Pair it with ai_selector to narrow the region the model reads, which keeps the call faster.
Assume search pages resolve at 25 credits and detail pages at 25.
| Job | Requests | Credits |
|---|---|---|
One search page, 20 listings, with next_page |
1 | 25 |
| 50 search pages | 50 | 1,250 |
| 1,000 detail pages | 1,000 | 25,000 |
| Same 1,000 with AI fallback on 5 percent | 1,000 | 26,250 |
Blocked attempts add nothing to those totals, because failures are free. Monitor consumption programmatically rather than guessing:
curl "https://app.scrapingbee.com/api/v1/usage" -H "Authorization: Bearer YOUR-API-KEY"That endpoint is free and allows 6 calls a minute. Credit allowances per plan are listed at scrapingbee.com/pricing.
MLS data is licensed data. Public portal pages are public, but the underlying records are governed by agreements between brokerages, associations and the portals themselves, and IDX rules vary by region.
Two rules that are not negotiable here:
- Public, pre-login pages only. Scraping behind login credentials is prohibited by the ScrapingBee terms. MLS member portals sit behind exactly such a login, and are out of bounds.
- Check the licence before you build on the data. Collecting a page and redistributing listing records are different acts with different obligations. This repository is engineering guidance, not legal advice.
Data extraction rules . AI web scraping . API reference . Zillow scraper . MLS scraper API
MIT. See LICENSE.