Skip to content

feat(cli): add lightweight remote crawling with trusted proxy configuration#2101

Open
nnWhisperer wants to merge 2 commits into
unclecode:mainfrom
nnWhisperer:crwl-remote
Open

feat(cli): add lightweight remote crawling with trusted proxy configuration#2101
nnWhisperer wants to merge 2 commits into
unclecode:mainfrom
nnWhisperer:crwl-remote

Conversation

@nnWhisperer

Copy link
Copy Markdown

Summary

Adds a lightweight crwl-remote command for crawling through a remote Crawl4AI
API without installing local browser dependencies, which can be used instead of
crwl command, which requires a local setup.

The Docker API now uses the existing configuration provenance mechanism to
trust browser and crawler configurations authenticated with the static operator
CRAWL4AI_API_TOKEN. This enables supported settings such as authenticated
upstream proxies while retaining restricted validation for other callers.

Also adds global remote credential storage, documentation, tests, and an agent
skill compatible with either local crwl or alias crwl='crwl-remote'.

List of files changed and why

  • .agents/skills/crawl-with-crwl/SKILL.md - Adds guidance for agents to crawl
    exclusively through the backend-agnostic crwl command.
  • .agents/skills/crawl-with-crwl/agents/openai.yaml - Adds metadata for the
    crawl skill.
  • README.md - Documents crwl-remote, global credentials, lightweight
    installation, shell aliases, browser configuration, and proxy usage.
  • crwl_remote.py - Implements the dependency-light remote API client and
    global credential configuration.
  • deploy/docker/api.py - Loads browser, crawler, streaming, and per-URL
    configurations using the caller's established Provenance.
  • deploy/docker/server.py - Selects Provenance.TRUSTED for the static
    operator API token and passes it consistently through regular and streaming
    crawl paths.
  • deploy/docker/tests/test_admin_proxy_config.py - Verifies trusted operator
    configuration, authenticated proxies, typed proxy configuration, streaming
    behavior, and restricted handling for other callers.
  • docs/codebase/remote-cli-requirements.md - Records the feature requirements,
    acceptance criteria, and focused-change principle.
  • pyproject.toml - Registers crwl-remote and the empty crwl-remote
    installation extra while preserving the original local crwl entry point.
  • tests/cli/test_crwl_remote.py - Tests request serialization, credentials,
    output, failures, and deep-crawl options.

How Has This Been Tested?

Tests were run inside an ephemeral Podman container to avoid installing
Playwright or browser dependencies on the host:

  • Remote CLI suite: 5 passed
  • Docker security suite: 321 passed
  • Expected failures: 1 pre-existing xfail
  • Unexpected failures: 0

The modified backend was also exercised through a running Podman container
using an authenticated rotating proxy. Two separate crwl requests to
https://ipv4.webshare.io returned different public IP addresses, verifying
that the proxy configuration was correctly applied to the browser inside the
container.

This confirmed that the proxy configuration reached Chromium inside the
container and that proxy rotation worked.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my
    feature works
  • New and existing unit tests pass locally with my changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant