diff --git a/deploy/docker/.llm.env.example b/deploy/docker/.llm.env.example index 012435d83..2d7a2ea25 100644 --- a/deploy/docker/.llm.env.example +++ b/deploy/docker/.llm.env.example @@ -1,3 +1,14 @@ +# REQUIRED for a reachable server: API token for the Docker server (0.9.0+). +# Without it the server binds loopback inside the container and the published +# port answers with "connection reset". Any non-empty value works, but treat it +# as a password — use a long random string (e.g. from: openssl rand -hex 32). +# Note: with docker compose, the token MUST be set here — exporting it in your +# shell does not reach the container. +CRAWL4AI_API_TOKEN= + +# Optional: enable declarative hooks support (disabled by default) +# CRAWL4AI_HOOKS_ENABLED=true + # LLM Provider Keys OPENAI_API_KEY=your_openai_key_here DEEPSEEK_API_KEY=your_deepseek_key_here diff --git a/deploy/docker/MIGRATION.md b/deploy/docker/MIGRATION.md index 753823060..b895ae588 100644 --- a/deploy/docker/MIGRATION.md +++ b/deploy/docker/MIGRATION.md @@ -25,11 +25,23 @@ loopback by default and will not expose itself without a credential. export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" ``` +> ⚠️ **Docker Compose users:** `export` alone does **not** work — the shipped +> `docker-compose.yml` does not forward host environment variables. Set the +> token in the `.llm.env` file at the project root instead (the example file +> ships an empty `CRAWL4AI_API_TOKEN=` line — fill it in). +> +> For plain `docker run`, pass it explicitly: +> `-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN"` (the value-less shorthand +> `-e CRAWL4AI_API_TOKEN` silently passes empty from a shell where the variable +> isn't set). + - With a token set, you may expose the server (put a TLS-terminating reverse proxy in front) and must send `Authorization: Bearer ` on every request except `GET /health`. -- With **no** token set, the server binds `127.0.0.1` only and prints a one-off - token at startup for local use. +- With **no** token set, the server binds `127.0.0.1` only (the **container's** + loopback — published ports answer with *connection reset* even though the + container reports healthy) and prints a one-off token at startup for + in-container use. WebSocket clients (MCP, monitor) that can't set headers may pass `?token=...`. @@ -60,6 +72,10 @@ safe maximums. ### Hooks: declarative actions instead of code +Hooks are now **disabled by default** — enable them with +`CRAWL4AI_HOOKS_ENABLED=true` in the container environment, or any request +containing `hooks` returns HTTP 403. + `hooks.code` (Python strings) is replaced by a fixed set of declarative actions: ```jsonc @@ -77,6 +93,13 @@ Available actions: `block_resources`, `add_cookies`, `set_headers`, `scroll_to_bottom`, `wait_for_timeout`. Call `GET /hooks/info` for the parameter schemas. Arbitrary hook code is available in a self-hosted in-process build. +> ⚠️ **Legacy `hooks.code` requests fail silently.** With hooks enabled, a +> request in the old format returns HTTP 200 with +> `"hooks": {"status": "success", "attached": []}` — the inline code is +> dropped without error. If `attached` is empty, your hooks did not run. +> (With hooks disabled, the same request returns the generic 403, whose +> "enable hooks" hint will not make code hooks work either.) + ### Screenshot / PDF: artifact id instead of `output_path` `output_path` is removed. The server stores the result and returns an id + URL: @@ -89,6 +112,10 @@ schemas. Arbitrary hook code is available in a self-hosted in-process build. Fetch the file with `GET /artifacts/{artifact_id}` (authenticated). Artifacts have a TTL and a storage quota. +> ⚠️ A request that still includes `output_path` is **silently ignored** — it +> returns `success: true` with an artifact id, but no file is written to the +> requested path. Update your code to fetch from `/artifacts/{artifact_id}`. + ### LLM endpoints: provider by name `base_url` is removed from `/md`, `/llm`, and `/llm/job`. Select a provider by diff --git a/docs/md_v2/core/self-hosting.md b/docs/md_v2/core/self-hosting.md index 1420774d1..966672f04 100644 --- a/docs/md_v2/core/self-hosting.md +++ b/docs/md_v2/core/self-hosting.md @@ -1,13 +1,17 @@ # Self-Hosting Crawl4AI 🚀 -> **🔐 0.9.0 is secure-by-default (breaking changes).** The self-hosted Docker -> server now requires authentication by default, binds to loopback unless you +> **🔐 0.9.0+ is secure-by-default (breaking changes).** The self-hosted Docker +> server requires authentication by default, binds to loopback unless you > set a token, validates request bodies against a strict trust boundary, uses > declarative hooks instead of inline Python, and returns artifact ids for > screenshot/pdf. If you are upgrading from 0.8.x, read the > [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) -> first. Some examples below are being updated for 0.9.0; the migration guide is -> the authoritative reference for the new defaults. +> first. +> +> **The single most important thing to know:** without a `CRAWL4AI_API_TOKEN`, +> the server binds loopback **inside the container** — published ports will +> answer with *connection reset* even though the container reports healthy. +> Every quickstart below therefore starts by setting a token. **Take Control of Your Web Crawling Infrastructure** @@ -76,13 +80,13 @@ Pull and run images directly from Docker Hub without building locally. #### 1. Pull the Image -Our latest release is `0.8.0`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system. +Our latest release is `0.9.2`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system. -> 💡 **Note**: The `latest` tag points to the stable `0.8.0` version. +> 💡 **Note**: The `latest` tag points to the most recent stable version. ```bash # Pull the latest version -docker pull unclecode/crawl4ai:0.8.0 +docker pull unclecode/crawl4ai:0.9.2 # Or pull using the latest tag docker pull unclecode/crawl4ai:latest @@ -121,7 +125,23 @@ EOL ``` > 🔑 **Note**: Keep your API keys secure! Never commit `.llm.env` to version control. -#### 3. Run the Container +#### 3. Set an API Token (Required to Reach the Server) + +Since 0.9.0 the server is secure-by-default: **without a token it binds +loopback inside the container**, so the published port answers with +*connection reset* — even though `docker ps` shows the container healthy and +the port mapped. Generate a token first: + +```bash +export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" +``` + +> ⚠️ Use the explicit form `-e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN"` below. +> The shorthand `-e CRAWL4AI_API_TOKEN` (no value) forwards the variable from +> the *current shell* — in a shell where it isn't set, it silently passes an +> empty value and the server starts in loopback mode with no host-side warning. + +#### 4. Run the Container * **Basic run:** ```bash @@ -129,6 +149,7 @@ EOL -p 11235:11235 \ --name crawl4ai \ --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ unclecode/crawl4ai:latest ``` @@ -140,12 +161,30 @@ EOL --name crawl4ai \ --env-file .llm.env \ --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ unclecode/crawl4ai:latest ``` -> The server will be available at `http://localhost:11235`. Visit `/playground` to access the interactive testing interface. +The server will be available at `http://localhost:11235` after a short startup +(allow ~10 seconds; the healthcheck allows up to 40). Verify: + +```bash +curl http://localhost:11235/health # no auth needed for /health +``` + +Every other endpoint requires `Authorization: Bearer $CRAWL4AI_API_TOKEN`. +Visit `/playground` for the interactive testing interface and `/dashboard` for +the monitoring UI — both have an **API token** bar in the top navigation; paste +your token there and click **Set**. + +> 💡 **Troubleshooting "connection reset":** during the ~10s startup window the +> published port also answers with connection reset — the same symptom as the +> missing-token failure. If resets persist after 20s, check +> `docker logs crawl4ai` for `binding loopback only`: that means no token +> reached the container (note the `127.0.0.1` in that log line is the +> *container's* loopback, not your host's — the port mapping cannot reach it). -#### 4. Stopping the Container +#### 5. Stopping the Container ```bash docker stop crawl4ai && docker rm crawl4ai @@ -154,7 +193,7 @@ docker stop crawl4ai && docker rm crawl4ai #### Docker Hub Versioning Explained * **Image Name:** `unclecode/crawl4ai` -* **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.8.0`) +* **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.9.2`) * `LIBRARY_VERSION`: The semantic version of the core `crawl4ai` Python library * `SUFFIX`: Optional tag for release candidates (``) and revisions (`r1`) * **`latest` Tag:** Points to the most recent stable version @@ -171,17 +210,32 @@ git clone https://github.com/unclecode/crawl4ai.git cd crawl4ai ``` -#### 2. Environment Setup (API Keys) +#### 2. Environment Setup (Required) -If you plan to use LLMs, copy the example environment file and add your API keys. This file should be in the **project root directory**. +The compose file loads `.llm.env` from the **project root directory** — the +file must exist even if you don't use LLMs, or compose will fail with +"env file .llm.env not found". Create it from the example and add an API token: ```bash # Make sure you are in the 'crawl4ai' root directory cp deploy/docker/.llm.env.example .llm.env +``` + +Then open `.llm.env` and fill in the `CRAWL4AI_API_TOKEN=` line at the top — +**required**, or the server will be unreachable (loopback-only). Any long +random string works, e.g. from `openssl rand -hex 32`. One-liner: -# Now edit .llm.env and add your API keys +```bash +sed -i.bak "s|^CRAWL4AI_API_TOKEN=.*|CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)|" .llm.env && rm .llm.env.bak ``` +Optionally add your LLM API keys in the same file. + +> ⚠️ **The token must go inside `.llm.env`.** `export CRAWL4AI_API_TOKEN=...` +> in your shell does **not** work with compose — the compose file does not +> forward host environment variables, and the server silently starts in +> loopback-only mode (published port → connection reset). + **Flexible LLM Provider Configuration:** The Docker setup now supports flexible LLM provider configuration through a hierarchical system: @@ -249,7 +303,9 @@ The `docker-compose.yml` file in the project root provides a simplified approach ENABLE_GPU=true docker compose up --build -d ``` -> The server will be available at `http://localhost:11235`. +> The server will be available at `http://localhost:11235` (allow ~10 seconds +> for startup). All endpoints except `GET /health` require +> `Authorization: Bearer `. #### 4. Stopping the Service @@ -287,12 +343,20 @@ docker buildx build \ #### 3. Run the Container +Set a token first (required — without it the server is loopback-only and the +published port answers with connection reset): + +```bash +export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)" +``` + * **Basic run (no LLM support):** ```bash docker run -d \ -p 11235:11235 \ --name crawl4ai-standalone \ --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ crawl4ai-local:latest ``` @@ -304,10 +368,13 @@ docker buildx build \ --name crawl4ai-standalone \ --env-file .llm.env \ --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ crawl4ai-local:latest ``` -> The server will be available at `http://localhost:11235`. +> The server will be available at `http://localhost:11235` (allow ~10 seconds +> for startup). All endpoints except `GET /health` require +> `Authorization: Bearer $CRAWL4AI_API_TOKEN`. #### 4. Stopping the Manual Container @@ -402,13 +469,32 @@ Captures a full-page PNG screenshot of the specified URL. ```json { "url": "https://example.com", - "screenshot_wait_for": 2, - "output_path": "/path/to/save/screenshot.png" + "screenshot_wait_for": 2 } ``` - `screenshot_wait_for`: Optional delay in seconds before capture (default: 2) -- `output_path`: Optional path to save the screenshot (recommended) + +The response contains the image inline (base64) **and** an artifact id you can +fetch later: + +```json +{"success": true, "screenshot": "", + "artifact_id": "…", "url": "/artifacts/…", "mime": "image/png", "size": 16668} +``` + +Download the stored file with an authenticated request (artifacts have a TTL +and a storage quota): + +```bash +curl -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \ + -o screenshot.png http://localhost:11235/artifacts/ +``` + +> **⚠️ Changed in 0.9.0:** `output_path` was removed (server-side file writes +> were a path-traversal risk). A request that still includes `output_path` +> currently returns `success: true` but **silently ignores the field** — no +> file is written. Use the artifact flow above instead. ### PDF Export Endpoint @@ -420,12 +506,13 @@ Generates a PDF document of the specified URL. ```json { - "url": "https://example.com", - "output_path": "/path/to/save/document.pdf" + "url": "https://example.com" } ``` -- `output_path`: Optional path to save the PDF (recommended) +Like `/screenshot`, the response returns the document plus an `artifact_id`; +fetch the file via `GET /artifacts/{artifact_id}` with your Bearer token. +`output_path` was removed in 0.9.0 and is silently ignored if sent. ### JavaScript Execution Endpoint @@ -449,684 +536,112 @@ Executes JavaScript snippets on the specified URL and returns the full crawl res --- -## User-Provided Hooks API - -> **⚠️ Changed in 0.9.0.** The inline-Python hook API described below was removed. -> Sending arbitrary Python code to the server is no longer accepted (it was an -> unauthenticated code-execution surface). 0.9.0 replaces it with **declarative -> hooks**: a fixed set of safe, server-validated actions (for example -> `add_cookies`, `set_headers`, `block_resources`) supplied as JSON, with no code -> execution. See the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) -> for the declarative hook format. The inline-code examples in this section apply -> to 0.8.x only and are kept for reference until this page is fully rewritten. +## Hooks (Declarative Actions) -The Docker API supports user-provided hook functions, allowing you to customize the crawling behavior by injecting your own Python code at specific points in the crawling pipeline. This powerful feature enables authentication, performance optimization, and custom content extraction without modifying the server code. +> **⚠️ Changed in 0.9.0.** The previous hooks API — sending Python code strings in +> `hooks.code` — was removed. It was an unauthenticated remote-code-execution +> surface. The server now accepts **declarative hooks**: a fixed set of safe, +> server-validated actions supplied as JSON. No code execution. If you need +> arbitrary hook code, use the in-process Python SDK (`AsyncWebCrawler`), where +> you keep full control. -> ⚠️ **IMPORTANT SECURITY WARNING**: -> - **Never use hooks with untrusted code or on untrusted websites** -> - **Be extremely careful when crawling sites that might be phishing or malicious** -> - **Hook code has access to page context and can interact with the website** -> - **Always validate and sanitize any data extracted through hooks** -> - **Never expose credentials or sensitive data in hook code** -> - **Consider running the Docker container in an isolated network when testing** - -### Hook Information Endpoint - -``` -GET /hooks/info -``` +### Enabling hooks -Returns information about available hook points and their signatures: +Hooks are **disabled by default**. Enable them with an environment variable when +starting the container: ```bash -curl http://localhost:11235/hooks/info +docker run -d \ + -p 11235:11235 \ + --name crawl4ai \ + --shm-size=1g \ + -e CRAWL4AI_API_TOKEN="$CRAWL4AI_API_TOKEN" \ + -e CRAWL4AI_HOOKS_ENABLED=true \ + unclecode/crawl4ai:latest ``` -### Available Hook Points - -The API supports 8 hook points that match the local SDK: - -| Hook Point | Parameters | Description | Best Use Cases | -|------------|------------|-------------|----------------| -| `on_browser_created` | `browser` | After browser instance creation | Light setup tasks | -| `on_page_context_created` | `page, context` | After page/context creation | **Authentication, cookies, route blocking** | -| `before_goto` | `page, context, url` | Before navigating to URL | Custom headers, logging | -| `after_goto` | `page, context, url, response` | After navigation completes | Verification, waiting for elements | -| `on_user_agent_updated` | `page, context, user_agent` | When user agent changes | UA-specific logic | -| `on_execution_started` | `page, context` | When JS execution begins | JS-related setup | -| `before_retrieve_html` | `page, context` | Before getting final HTML | **Scrolling, lazy loading** | -| `before_return_html` | `page, context, html` | Before returning HTML | Final modifications, metrics | - -### Using Hooks in Requests - -Add hooks to any crawl request by including the `hooks` parameter: +Without the flag, any request containing `hooks` returns: ```json -{ - "urls": ["https://httpbin.org/html"], - "hooks": { - "code": { - "hook_point_name": "async def hook(...): ...", - "another_hook": "async def hook(...): ..." - }, - "timeout": 30 // Optional, default 30 seconds (max 120) - } -} +{"detail": "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable."} ``` -### Hook Examples with Real URLs +### Available actions -#### 1. Authentication with Cookies (GitHub) +| Action | Hook point | Description | +|--------|-----------|-------------| +| `block_resources` | `on_page_context_created` | Abort matching resource types (`image`, `stylesheet`, `font`, `media`) | +| `add_cookies` | `on_page_context_created` | Add cookies to the browser context before navigation (auth) | +| `set_headers` | `before_goto` | Set extra HTTP request headers before navigating | +| `scroll_to_bottom` | `before_retrieve_html` | Scroll to the page bottom in bounded steps (lazy-load), `max_steps` 1–50, `delay_ms` 0–5000 | +| `wait_for_timeout` | `before_retrieve_html` | Wait a bounded number of milliseconds (0–60000) before retrieving HTML | -```python -import requests +Get the full parameter schemas at runtime: -# Example: Setting GitHub session cookie (use your actual session) -hooks_code = { - "on_page_context_created": """ -async def hook(page, context, **kwargs): - # Add authentication cookies for GitHub - # WARNING: Never hardcode real credentials! - await context.add_cookies([ - { - 'name': 'user_session', - 'value': 'your_github_session_token', # Replace with actual token - 'domain': '.github.com', - 'path': '/', - 'httpOnly': True, - 'secure': True, - 'sameSite': 'Lax' - } - ]) - return page -""" -} - -response = requests.post("http://localhost:11235/crawl", json={ - "urls": ["https://github.com/settings/profile"], # Protected page - "hooks": {"code": hooks_code, "timeout": 30} -}) -``` - -#### 2. Basic Authentication (httpbin.org for testing) - -```python -# Safe testing with httpbin.org (a service designed for HTTP testing) -hooks_code = { - "before_goto": """ -async def hook(page, context, url, **kwargs): - import base64 - # httpbin.org/basic-auth expects username="user" and password="passwd" - credentials = base64.b64encode(b"user:passwd").decode('ascii') - - await page.set_extra_http_headers({ - 'Authorization': f'Basic {credentials}' - }) - return page -""" -} - -response = requests.post("http://localhost:11235/crawl", json={ - "urls": ["https://httpbin.org/basic-auth/user/passwd"], - "hooks": {"code": hooks_code, "timeout": 15} -}) -``` - -#### 3. Performance Optimization (News Sites) - -```python -# Example: Optimizing crawling of news sites like CNN or BBC -hooks_code = { - "on_page_context_created": """ -async def hook(page, context, **kwargs): - # Block images, fonts, and media to speed up crawling - await context.route("**/*.{png,jpg,jpeg,gif,webp,svg,ico}", lambda route: route.abort()) - await context.route("**/*.{woff,woff2,ttf,otf,eot}", lambda route: route.abort()) - await context.route("**/*.{mp4,webm,ogg,mp3,wav,flac}", lambda route: route.abort()) - - # Block common tracking and ad domains - await context.route("**/googletagmanager.com/*", lambda route: route.abort()) - await context.route("**/google-analytics.com/*", lambda route: route.abort()) - await context.route("**/doubleclick.net/*", lambda route: route.abort()) - await context.route("**/facebook.com/tr/*", lambda route: route.abort()) - await context.route("**/amazon-adsystem.com/*", lambda route: route.abort()) - - # Disable CSS animations for faster rendering - await page.add_style_tag(content=''' - *, *::before, *::after { - animation-duration: 0s !important; - transition-duration: 0s !important; - } - ''') - - return page -""" -} - -response = requests.post("http://localhost:11235/crawl", json={ - "urls": ["https://www.bbc.com/news"], # Heavy news site - "hooks": {"code": hooks_code, "timeout": 30} -}) +```bash +curl -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \ + http://localhost:11235/hooks/info ``` -#### 4. Handling Infinite Scroll (Twitter/X) +### Using hooks in a request -```python -# Example: Scrolling on Twitter/X (requires authentication) -hooks_code = { - "before_retrieve_html": """ -async def hook(page, context, **kwargs): - # Scroll to load more tweets - previous_height = 0 - for i in range(5): # Limit scrolls to avoid infinite loop - current_height = await page.evaluate("document.body.scrollHeight") - if current_height == previous_height: - break # No more content to load - - await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - await page.wait_for_timeout(2000) # Wait for content to load - previous_height = current_height - - return page -""" -} +Add a `hooks` object with a list of actions (maximum 10 per request): -# Note: Twitter requires authentication for most content -response = requests.post("http://localhost:11235/crawl", json={ - "urls": ["https://twitter.com/nasa"], # Public profile - "hooks": {"code": hooks_code, "timeout": 30} -}) -``` - -#### 5. E-commerce Login (Example Pattern) - -```python -# SECURITY WARNING: This is a pattern example. -# Never use real credentials in code! -# Always use environment variables or secure vaults. - -hooks_code = { - "on_page_context_created": """ -async def hook(page, context, **kwargs): - # Example pattern for e-commerce sites - # DO NOT use real credentials here! - - # Navigate to login page first - await page.goto("https://example-shop.com/login") - - # Wait for login form to load - await page.wait_for_selector("#email", timeout=5000) - - # Fill login form (use environment variables in production!) - await page.fill("#email", "test@example.com") # Never use real email - await page.fill("#password", "test_password") # Never use real password - - # Handle "Remember Me" checkbox if present - try: - await page.uncheck("#remember_me") # Don't remember on shared systems - except: - pass - - # Submit form - await page.click("button[type='submit']") - - # Wait for redirect after login - await page.wait_for_url("**/account/**", timeout=10000) - - return page -""" -} +```bash +curl -X POST http://localhost:11235/crawl \ + -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{ + "urls": ["https://example.com"], + "hooks": { + "hooks": [ + {"action": "block_resources", "params": {"resource_types": ["image", "font"]}}, + {"action": "scroll_to_bottom", "params": {"max_steps": 10, "delay_ms": 500}} + ] + } + }' ``` -#### 6. Extracting Structured Data (Wikipedia) - -```python -# Safe example using Wikipedia -hooks_code = { - "after_goto": """ -async def hook(page, context, url, response, **kwargs): - # Wait for Wikipedia content to load - await page.wait_for_selector("#content", timeout=5000) - return page -""", - - "before_retrieve_html": """ -async def hook(page, context, **kwargs): - # Extract structured data from Wikipedia infobox - metadata = await page.evaluate('''() => { - const infobox = document.querySelector('.infobox'); - if (!infobox) return null; - - const data = {}; - const rows = infobox.querySelectorAll('tr'); - - rows.forEach(row => { - const header = row.querySelector('th'); - const value = row.querySelector('td'); - if (header && value) { - data[header.innerText.trim()] = value.innerText.trim(); - } - }); - - return data; - }''') - - if metadata: - print("Extracted metadata:", metadata) - - return page -""" -} +The response reports what was attached and executed: -response = requests.post("http://localhost:11235/crawl", json={ - "urls": ["https://en.wikipedia.org/wiki/Python_(programming_language)"], - "hooks": {"code": hooks_code, "timeout": 20} -}) +```json +{"success": true, "results": [...], "hooks": {"status": "success", "attached": ["before_retrieve_html"]}} ``` -### Security Best Practices - -> 🔒 **Critical Security Guidelines**: - -1. **Never Trust User Input**: If accepting hook code from users, always validate and sandbox it -2. **Avoid Phishing Sites**: Never use hooks on suspicious or unverified websites -3. **Protect Credentials**: - - Never hardcode passwords, tokens, or API keys in hook code - - Use environment variables or secure secret management - - Rotate credentials regularly -4. **Network Isolation**: Run the Docker container in an isolated network when testing -5. **Audit Hook Code**: Always review hook code before execution -6. **Limit Permissions**: Use the least privileged access needed -7. **Monitor Execution**: Check hook execution logs for suspicious behavior -8. **Timeout Protection**: Always set reasonable timeouts (default 30s) - -### Hook Response Information - -When hooks are used, the response includes detailed execution information: +Cookie example (authentication): ```json { - "success": true, - "results": [...], + "urls": ["https://example.com/account"], "hooks": { - "status": { - "status": "success", // or "partial" or "failed" - "attached_hooks": ["on_page_context_created", "before_retrieve_html"], - "validation_errors": [], - "successfully_attached": 2, - "failed_validation": 0 - }, - "execution_log": [ - { - "hook_point": "on_page_context_created", - "status": "success", - "execution_time": 0.523, - "timestamp": 1234567890.123 - } - ], - "errors": [], // Any runtime errors - "summary": { - "total_executions": 2, - "successful": 2, - "failed": 0, - "timed_out": 0, - "success_rate": 100.0 - } + "hooks": [ + {"action": "add_cookies", "params": {"cookies": [ + {"name": "session", "value": "your-session-token", + "domain": ".example.com", "path": "/", "secure": true} + ]}}, + {"action": "set_headers", "params": {"headers": {"Accept-Language": "en-US"}}} + ] } } ``` -### Error Handling - -The hooks system is designed to be resilient: - -1. **Validation Errors**: Caught before execution (syntax errors, wrong parameters) -2. **Runtime Errors**: Handled gracefully - crawl continues with original page object -3. **Timeout Protection**: Hooks automatically terminated after timeout (configurable 1-120s) - -### Complete Example: Safe Multi-Hook Crawling - -```python -import requests -import json -import os - -# Safe example using httpbin.org for testing -hooks_code = { - "on_page_context_created": """ -async def hook(page, context, **kwargs): - # Set viewport and test cookies - await page.set_viewport_size({"width": 1920, "height": 1080}) - await context.add_cookies([ - {"name": "test_cookie", "value": "test_value", "domain": ".httpbin.org", "path": "/"} - ]) - - # Block unnecessary resources for httpbin - await context.route("**/*.{png,jpg,jpeg}", lambda route: route.abort()) - return page -""", - - "before_goto": """ -async def hook(page, context, url, **kwargs): - # Add custom headers for testing - await page.set_extra_http_headers({ - "X-Test-Header": "crawl4ai-test", - "Accept-Language": "en-US,en;q=0.9" - }) - print(f"[HOOK] Navigating to: {url}") - return page -""", - - "before_retrieve_html": """ -async def hook(page, context, **kwargs): - # Simple scroll for any lazy-loaded content - await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - await page.wait_for_timeout(1000) - return page -""" -} - -# Make the request to safe testing endpoints -response = requests.post("http://localhost:11235/crawl", json={ - "urls": [ - "https://httpbin.org/html", - "https://httpbin.org/json" - ], - "hooks": { - "code": hooks_code, - "timeout": 30 - }, - "crawler_config": { - "cache_mode": "bypass" - } -}) - -# Check results -if response.status_code == 200: - data = response.json() - - # Check hook execution - if data['hooks']['status']['status'] == 'success': - print(f"✅ All {len(data['hooks']['status']['attached_hooks'])} hooks executed successfully") - print(f"Execution stats: {data['hooks']['summary']}") - - # Process crawl results - for result in data['results']: - print(f"Crawled: {result['url']} - Success: {result['success']}") -else: - print(f"Error: {response.status_code}") -``` - -> 💡 **Remember**: Always test your hooks on safe, known websites first before using them on production sites. Never crawl sites that you don't have permission to access or that might be malicious. - -### Hooks Utility: Function-Based Approach (Python) - -For Python developers, Crawl4AI provides a more convenient way to work with hooks using the `hooks_to_string()` utility function and Docker client integration. - -#### Why Use Function-Based Hooks? - -**String-Based Approach (shown above)**: -```python -hooks_code = { - "on_page_context_created": """ -async def hook(page, context, **kwargs): - await page.set_viewport_size({"width": 1920, "height": 1080}) - return page -""" -} -``` - -**Function-Based Approach (recommended for Python)**: -```python -from crawl4ai import Crawl4aiDockerClient - -async def my_hook(page, context, **kwargs): - await page.set_viewport_size({"width": 1920, "height": 1080}) - return page - -async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client: - result = await client.crawl( - ["https://example.com"], - hooks={"on_page_context_created": my_hook} - ) -``` - -**Benefits**: -- ✅ Write hooks as regular Python functions -- ✅ Full IDE support (autocomplete, syntax highlighting, type checking) -- ✅ Easy to test and debug -- ✅ Reusable hook libraries -- ✅ Automatic conversion to API format - -#### Using the Hooks Utility - -The `hooks_to_string()` utility converts Python function objects to the string format required by the API: - -```python -from crawl4ai import hooks_to_string - -# Define your hooks as functions -async def setup_hook(page, context, **kwargs): - await page.set_viewport_size({"width": 1920, "height": 1080}) - await context.add_cookies([{ - "name": "session", - "value": "token", - "domain": ".example.com" - }]) - return page - -async def scroll_hook(page, context, **kwargs): - await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - return page - -# Convert to string format -hooks_dict = { - "on_page_context_created": setup_hook, - "before_retrieve_html": scroll_hook -} -hooks_string = hooks_to_string(hooks_dict) - -# Now use with REST API or Docker client -# hooks_string contains the string representations -``` - -#### Docker Client with Automatic Conversion - -The Docker client automatically detects and converts function objects: - -```python -from crawl4ai import Crawl4aiDockerClient - -async def auth_hook(page, context, **kwargs): - """Add authentication cookies""" - await context.add_cookies([{ - "name": "auth_token", - "value": "your_token", - "domain": ".example.com" - }]) - return page - -async def performance_hook(page, context, **kwargs): - """Block unnecessary resources""" - await context.route("**/*.{png,jpg,gif}", lambda r: r.abort()) - await context.route("**/analytics/*", lambda r: r.abort()) - return page - -async with Crawl4aiDockerClient(base_url="http://localhost:11235") as client: - # Pass functions directly - automatic conversion! - result = await client.crawl( - ["https://example.com"], - hooks={ - "on_page_context_created": performance_hook, - "before_goto": auth_hook - }, - hooks_timeout=30 # Optional timeout in seconds (1-120) - ) - - print(f"Success: {result.success}") - print(f"HTML: {len(result.html)} chars") -``` - -#### Creating Reusable Hook Libraries - -Build collections of reusable hooks: - -```python -# hooks_library.py -class CrawlHooks: - """Reusable hook collection for common crawling tasks""" - - @staticmethod - async def block_images(page, context, **kwargs): - """Block all images to speed up crawling""" - await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda r: r.abort()) - return page - - @staticmethod - async def block_analytics(page, context, **kwargs): - """Block analytics and tracking scripts""" - tracking_domains = [ - "**/google-analytics.com/*", - "**/googletagmanager.com/*", - "**/facebook.com/tr/*", - "**/doubleclick.net/*" - ] - for domain in tracking_domains: - await context.route(domain, lambda r: r.abort()) - return page - - @staticmethod - async def scroll_infinite(page, context, **kwargs): - """Handle infinite scroll to load more content""" - previous_height = 0 - for i in range(5): # Max 5 scrolls - current_height = await page.evaluate("document.body.scrollHeight") - if current_height == previous_height: - break - await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - await page.wait_for_timeout(1000) - previous_height = current_height - return page - - @staticmethod - async def wait_for_dynamic_content(page, context, url, response, **kwargs): - """Wait for dynamic content to load""" - await page.wait_for_timeout(2000) - try: - # Click "Load More" if present - load_more = await page.query_selector('[class*="load-more"]') - if load_more: - await load_more.click() - await page.wait_for_timeout(1000) - except: - pass - return page - -# Use in your application -from hooks_library import CrawlHooks -from crawl4ai import Crawl4aiDockerClient - -async def crawl_with_optimizations(url): - async with Crawl4aiDockerClient() as client: - result = await client.crawl( - [url], - hooks={ - "on_page_context_created": CrawlHooks.block_images, - "before_retrieve_html": CrawlHooks.scroll_infinite - } - ) - return result -``` - -#### Choosing the Right Approach +### Migrating from 0.8.x `hooks.code` -| Approach | Best For | IDE Support | Language | -|----------|----------|-------------|----------| -| **String-based** | Non-Python clients, REST APIs, other languages | ❌ None | Any | -| **Function-based** | Python applications, local development | ✅ Full | Python only | -| **Docker Client** | Python apps with automatic conversion | ✅ Full | Python only | +Requests using the removed `hooks.code` (Python strings) format are **not +executed**. Be aware of the current behavior: -**Recommendation**: -- **Python applications**: Use Docker client with function objects (easiest) -- **Non-Python or REST API**: Use string-based hooks (most flexible) -- **Manual control**: Use `hooks_to_string()` utility (middle ground) +- With hooks disabled (default): HTTP 403 "Hooks are disabled…" — note that + enabling hooks will *not* make code hooks work. +- With hooks enabled: the request succeeds (HTTP 200) but the inline code is + **silently ignored** — the response shows `"hooks": {"status": "success", + "attached": []}`. If you see an empty `attached` list, your hooks did not run. -#### Complete Example with Function Hooks +Map your old hook code to declarative actions where possible (resource blocking, +cookies, headers, scrolling, waits). For anything beyond the fixed action set — +custom JavaScript, form logins, conditional logic — use the in-process Python +SDK, which retains the full 8-hook-point API described in the +[hooks documentation](../advanced/hooks-auth.md). -```python -from crawl4ai import Crawl4aiDockerClient, BrowserConfig, CrawlerRunConfig, CacheMode - -# Define hooks as regular Python functions -async def setup_environment(page, context, **kwargs): - """Setup crawling environment""" - # Set viewport - await page.set_viewport_size({"width": 1920, "height": 1080}) - - # Block resources for speed - await context.route("**/*.{png,jpg,gif}", lambda r: r.abort()) - - # Add custom headers - await page.set_extra_http_headers({ - "Accept-Language": "en-US", - "X-Custom-Header": "Crawl4AI" - }) - - print("[HOOK] Environment configured") - return page - -async def extract_content(page, context, **kwargs): - """Extract and prepare content""" - # Scroll to load lazy content - await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") - await page.wait_for_timeout(1000) - - # Extract metadata - metadata = await page.evaluate('''() => ({ - title: document.title, - links: document.links.length, - images: document.images.length - })''') - - print(f"[HOOK] Page metadata: {metadata}") - return page - -async def main(): - async with Crawl4aiDockerClient(base_url="http://localhost:11235", verbose=True) as client: - # Configure crawl - browser_config = BrowserConfig(headless=True) - crawler_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS) - - # Crawl with hooks - result = await client.crawl( - ["https://httpbin.org/html"], - browser_config=browser_config, - crawler_config=crawler_config, - hooks={ - "on_page_context_created": setup_environment, - "before_retrieve_html": extract_content - }, - hooks_timeout=30 - ) - - if result.success: - print(f"✅ Crawl successful!") - print(f" URL: {result.url}") - print(f" HTML: {len(result.html)} chars") - print(f" Markdown: {len(result.markdown)} chars") - else: - print(f"❌ Crawl failed: {result.error_message}") - -if __name__ == "__main__": - import asyncio - asyncio.run(main()) -``` - -#### Additional Resources - -- **Comprehensive Examples**: See `/docs/examples/hooks_docker_client_example.py` for Python function-based examples -- **REST API Examples**: See `/docs/examples/hooks_rest_api_example.py` for string-based examples -- **Comparison Guide**: See `/docs/examples/README_HOOKS.md` for detailed comparison -- **Utility Documentation**: See `/docs/hooks-utility-guide.md` for complete guide --- @@ -1634,7 +1149,14 @@ Communicate with the running Docker server via its REST API (defaulting to `http ### Playground Interface -A built-in web playground is available at `http://localhost:11235/playground` for testing and generating API requests. The playground allows you to: +A built-in web playground is available at `http://localhost:11235/playground` for testing and generating API requests. + +> 🔑 Before running requests, paste your API token into the **API token** bar in +> the top navigation and click **Set** — otherwise every request returns +> `{"detail": "Authentication required"}` (note: the status banner may still +> show "Success" for such error responses; check the response body). + +The playground allows you to: 1. Configure `CrawlerRunConfig` and `BrowserConfig` using the main library's Python syntax 2. Test crawling operations directly from the interface @@ -1646,7 +1168,13 @@ This is the easiest way to translate Python configuration to JSON requests when Install the SDK: `pip install crawl4ai` -The Python SDK provides a convenient way to interact with the Docker API, including **automatic hook conversion** when using function objects. +The Python SDK provides a convenient way to interact with the Docker API. + +> **⚠️ Changed in 0.9.0:** the SDK's function-based hooks (`hooks={...}` with +> Python functions) no longer work against the Docker server — they were +> converted to code strings server-side, and request-supplied hook code was +> removed. Use [declarative hooks](#hooks-declarative-actions) over the REST +> API, or run the in-process SDK (`AsyncWebCrawler`) for full hook support. ```python import asyncio @@ -1687,25 +1215,6 @@ async def main(): except Exception as e: print(f"Streaming crawl failed: {e}") - # Example with hooks (Python function objects) - print("\n--- Crawl with Hooks ---") - - async def my_hook(page, context, **kwargs): - """Custom hook to optimize performance""" - await page.set_viewport_size({"width": 1920, "height": 1080}) - await context.route("**/*.{png,jpg}", lambda r: r.abort()) - print("[HOOK] Page optimized") - return page - - result = await client.crawl( - ["https://httpbin.org/html"], - browser_config=BrowserConfig(headless=True), - crawler_config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS), - hooks={"on_page_context_created": my_hook}, # Pass function directly! - hooks_timeout=30 - ) - print(f"Crawl with hooks success: {result.success}") - # Example Get schema print("\n--- Getting Schema ---") schema = await client.get_schema() @@ -1730,8 +1239,6 @@ The Docker client supports the following parameters: - `urls` (List[str]): List of URLs to crawl - `browser_config` (Optional[BrowserConfig]): Browser configuration - `crawler_config` (Optional[CrawlerRunConfig]): Crawler configuration -- `hooks` (Optional[Dict]): Hook functions or strings - **automatically converts function objects!** -- `hooks_timeout` (int): Timeout for each hook execution in seconds (default: 30) **Returns**: - Single URL: `CrawlResult` object @@ -1978,9 +1485,15 @@ One of the key advantages of self-hosting is complete visibility into your infra Access the **built-in real-time monitoring dashboard** for complete operational visibility: ``` -http://localhost:11235/monitor +http://localhost:11235/dashboard ``` +> ⚠️ The dashboard UI lives at `/dashboard` — **not** `/monitor`, which is the +> API namespace (`/monitor/health`, `/monitor/ws`, …) and returns +> `{"detail": "Authentication required"}` in a browser. On the dashboard, paste +> your API token into the **API token** bar (top right) and click **Set**; the +> WebSocket then connects and live stats appear. + ![Monitoring Dashboard](https://via.placeholder.com/800x400?text=Crawl4AI+Monitoring+Dashboard) **Dashboard Features:** @@ -2425,7 +1938,7 @@ Returns: ```json { "status": "healthy", - "version": "0.7.4" + "version": "0.9.2" } ``` @@ -2622,14 +2135,14 @@ By self-hosting Crawl4AI, you: **Next Steps:** 1. **Start Simple**: Deploy with Docker Hub image and test with the playground -2. **Monitor Everything**: Open `http://localhost:11235/monitor` to watch your server +2. **Monitor Everything**: Open `http://localhost:11235/dashboard` to watch your server 3. **Integrate**: Connect your applications using the Python SDK or REST API 4. **Scale Smart**: Use the monitoring data to optimize your deployment 5. **Go Production**: Set up alerting, log aggregation, and automated cleanup **Key Resources:** - 🎮 **Playground**: `http://localhost:11235/playground` - Interactive testing -- 📊 **Monitor Dashboard**: `http://localhost:11235/monitor` - Real-time visibility +- 📊 **Monitor Dashboard**: `http://localhost:11235/dashboard` - Real-time visibility - 📖 **Architecture Docs**: `deploy/docker/ARCHITECTURE.md` - Deep technical dive - 💬 **Discord Community**: Get help and share experiences - ⭐ **GitHub**: Report issues, contribute, show support