Examples of popular Python tools for web scraping, using the same Goodreads Quotes task for comparison.
Each example extracts:
text
author
tags
likes
The examples cover static scraping, browser automation, crawling frameworks, and managed scraping services.
| Tool | Best for |
|---|---|
| Requests + Beautiful Soup | Beginners and simple scripts |
| Requests + lxml | Fast parsing and XPath |
| HTTPX + Selectolax | Fast async scraping |
| Playwright | JavaScript-heavy websites |
| Selenium | Browser automation |
| Crawl4AI | Browser crawling and AI workflows |
| Scrapy | Large, repeatable crawls |
| Crawlee | Async crawling |
| MechanicalSoup | Forms and sessions without JavaScript |
| SerpApi | Managed scraping services |
python-web-scraping-libraries/
├── libraries/
│ ├── common.py
│ ├── static/
│ ├── dynamic/
│ ├── frameworks/
│ ├── stateful/
│ └── enterprise/
├── .gitignore
├── requirements.txt
└── README.md
Create and activate a virtual environment:
python3 -m venv .venv
source .venv/bin/activateInstall dependencies:
pip install -r requirements.txtSome browser tools require additional setup.
Playwright:
playwright installCrawl4AI:
crawl4ai-setupRun each scraper from the repository root:
python3 -m libraries.static.beautiful_soup_requests
python3 -m libraries.static.lxml_requests
python3 -m libraries.static.httpx_selectolax
python3 -m libraries.dynamic.playwright
python3 -m libraries.dynamic.selenium
python3 -m libraries.dynamic.crawl4ai
python3 -m libraries.frameworks.scrapy
python3 -m libraries.frameworks.crawlee
python3 -m libraries.stateful.mechanical_soup
python3 -m libraries.enterprise.serpapiThe examples save their results as JSON files.
Shared settings are in libraries/common.py.
BASE_URL = "https://www.goodreads.com/quotes"
DELAY_SECONDS = 1
MAX_PAGES = 10The examples use the same pages, fields, and output format so the tools can be compared directly.
Start with the simplest tool that fits the page:
- Static HTML: Requests + Beautiful Soup
- Fast parsing: Requests + lxml
- Async scraping: HTTPX + Selectolax
- JavaScript: Playwright or Selenium
- Larger crawls: Scrapy or Crawlee
- Forms without JavaScript: MechanicalSoup
- Managed scraping: SerpApi
If the site provides an API, use it instead of scraping when possible.
This repository accompanies the article Getting Started with Web Scraping in Python, which explains how to identify the type of data a website serves and choose an appropriate scraping tool.
Scrape responsibly. Check the site's robots.txt and terms of service, use reasonable request rates, and avoid unnecessary concurrency.