Skip to content

feat: Add CrewAI integration for autonomous web crawling #2102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
saturnabitsick wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: Add CrewAI integration for autonomous web crawling #2102

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3d63dec
feat: Add CrewAI integration for autonomous web crawling
invalid-email-address Jul 26, 2026
80c8a1e
docs: Add integration summary and status
invalid-email-address Jul 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
350 changes: 350 additions & 0 deletions CREWAI_INTEGRATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,350 @@
# CrewAI Integration - Implementation Summary

## Overview

This integration allows CrewAI agents to use Crawl4AI as a web scraping tool. The integration provides a seamless way for autonomous AI agents to browse websites, extract data, and gather information as part of their tasks.

## What Was Created

### 1. Core Integration Module: `crawl4ai/crewai_tool.py`

This module provides:

**Four Main Tools:**

1. **crawl_website(url, cache_mode, include_links, include_images, js_enabled, timeout, extraction_strategy)**
- Crawls a single website and returns markdown content
- Supports JavaScript rendering
- Can include links and images metadata
- Configurable caching strategies

2. **extract_data_from_url(url, extraction_schema, js_enabled, timeout)**
- Extracts structured data from a website based on a schema
- Returns both raw content and extracted data

3. **crawl_multiple_urls(urls, cache_mode, js_enabled, timeout)**
- Efficiently crawls multiple URLs in parallel
- Returns aggregated results with success/failure counts

4. **search_and_crawl(query, num_results, include_links)**
- Placeholder for search functionality
- Currently recommends using specific URLs with crawl_website

**Utility Functions:**

- `create_crawl4ai_tools()` - Returns a list of CrewAI-compatible tools
- `_get_crawler()` - Manages global crawler instance for efficiency
- `_sync_wrapper()` - Converts async functions to sync for CrewAI compatibility
- `cleanup_crawler()` - Clean up resources when done

### 2. Updated Package Init: `crawl4ai/__init__.py`

- Added optional imports for CrewAI tools (gracefully handled if CrewAI not installed)
- Updated `__all__` exports to include all new tools
- Maintains backward compatibility

### 3. Documentation: `docs/crewai_integration.md`

Comprehensive guide including:
- Installation instructions
- Quick start examples
- Detailed tool documentation
- 4+ use cases (market research, content analysis, data collection, research assistant)
- Multi-agent workflows
- Best practices
- Performance tips
- Troubleshooting guide
- Advanced examples

### 4. Example Scripts: `examples/`

Four practical examples:

1. **crewai_example_1_basic.py** - Simple website crawling
2. **crewai_example_2_multi_agent.py** - Multi-agent research workflow
3. **crewai_example_3_competitive_analysis.py** - Competitive intelligence gathering
4. **crewai_example_4_data_extraction.py** - Structured data extraction

### 5. Examples README: `examples/README.md`

- Overview of all examples
- Setup instructions
- How to build custom examples
- Tips for success
- Troubleshooting guide

## How It Works

### Architecture

```
┌─────────────────────────────────────────┐
│ CrewAI Agent │
│ (role, goal, tools, llm) │
└────────────────┬────────────────────────┘
│ calls
▼
┌─────────────────────────────────────────┐
│ CrewAI Tool (from Crawl4AI) │
│ - crawl_website() │
│ - extract_data_from_url() │
│ - crawl_multiple_urls() │
│ - search_and_crawl() │
└────────────────┬────────────────────────┘
│ uses
▼
┌─────────────────────────────────────────┐
│ AsyncWebCrawler (Crawl4AI) │
│ - Playwright browser automation │
│ - Content extraction │
│ - Markdown generation │
│ - Caching support │
└─────────────────────────────────────────┘
```

### Usage Flow

1. **Agent Definition**
```python
agent = Agent(
role="...",
goal="...",
tools=create_crawl4ai_tools(),
llm="gpt-4"
)
```

2. **Task Creation**
```python
task = Task(
description="Crawl website and extract...",
agent=agent,
expected_output="..."
)
```

3. **Crew Execution**
```python
crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()
```

4. **Agent Decides to Use Tool**
- Agent determines it needs to crawl a website
- Calls appropriate tool (crawl_website, extract_data_from_url, etc.)
- Receives results
- Processes and returns findings

## Key Features

### 1. CrewAI Compatibility
- Tools decorated with `@tool` decorator for CrewAI
- Sync wrapper handles async/sync conversion
- Returns structured dictionaries that LLMs can understand

### 2. Efficient Resource Management
- Global crawler instance avoids repetitive initialization
- Parallel crawling with `crawl_multiple_urls`
- Configurable cache modes (BYPASS, CACHED, CACHE_FIRST)

### 3. Flexible Configuration
- JavaScript rendering support for dynamic sites
- Configurable timeouts and browser settings
- Multiple extraction strategies
- Link and image inclusion options

### 4. Error Handling
- Graceful error handling with informative messages
- Returns errors in structured format
- Agent can see and respond to failures

### 5. Backward Compatibility
- CrewAI import is optional (try/except)
- Doesn't break existing Crawl4AI functionality
- Adds no required dependencies

## Integration Points

### Optional Dependency
CrewAI is treated as an optional dependency:
```python
try:
from .crewai_tool import ...
except ImportError:
pass # CrewAI not installed
```

This means:
- Existing Crawl4AI users are unaffected
- CrewAI support is added when needed
- No version conflicts

### Extensibility

The integration can be extended with:

```python
# Custom tool for specific use case
@tool
async def custom_crawl_task(url: str) -> dict:
"""Custom crawling logic specific to your domain"""
crawler = await _get_crawler()
# Custom implementation
return result

# Add to tools list
def create_crawl4ai_tools(custom_config=None):
tools = [...]
if custom_config and custom_config.get("include_custom"):
tools.append(custom_crawl_task)
return tools
```

## Usage Examples

### Simple Research Task
```python
from crewai import Agent, Task, Crew
from crawl4ai import create_crawl4ai_tools

agent = Agent(
role="Researcher",
goal="Find information",
tools=create_crawl4ai_tools(),
llm="gpt-4"
)

task = Task(description="Research Python", agent=agent)
crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()
```

### Multi-URL Competitive Analysis
```python
task = Task(
description="""
Visit these competitor websites and compare their pricing:
- competitor1.com
- competitor2.com
- competitor3.com
""",
agent=intelligence_agent,
expected_output="Competitive pricing analysis"
)
```

### Data Extraction Pipeline
```python
task = Task(
description="Extract product names and prices from the catalog",
agent=extraction_agent,
expected_output="JSON with products: [{name, price, description}]"
)
```

## Performance Considerations

### Optimization Tips

1. **Parallel Processing**
- Use `crawl_multiple_urls()` for multiple sites
- Agents can process results while crawler works

2. **Caching**
- Use CACHE_FIRST for stable content
- BYPASS for real-time data
- CACHED for balance

3. **JavaScript**
- Only enable when needed
- Doubles processing time
- Mention in task description if needed

4. **Timeouts**
- Set reasonable timeouts (default 30s)
- Prevent hanging on slow sites
- Adjust based on target sites

## Testing

The implementation includes:
- Type hints for better IDE support
- Docstrings for all functions
- Error handling for edge cases
- Example files for validation

To test:
```bash
# Install dependencies
pip install crawl4ai crewai

# Run example
python examples/crewai_example_1_basic.py
```

## Future Enhancements

Potential additions:
1. Search API integration for `search_and_crawl`
2. Custom extraction strategies
3. Real-time result streaming
4. Webhook support for event notifications
5. Browser pool management
6. Advanced caching strategies

## Files Changed

```
crawl4ai/
├── crewai_tool.py (NEW) # Main integration module
└── __init__.py (MODIFIED) # Added imports and exports

docs/
└── crewai_integration.md (NEW) # Comprehensive documentation

examples/
├── README.md (NEW) # Examples guide
├── crewai_example_1_basic.py (NEW)
├── crewai_example_2_multi_agent.py (NEW)
├── crewai_example_3_competitive_analysis.py (NEW)
└── crewai_example_4_data_extraction.py (NEW)
```

## Installation & Setup

1. **Install requirements:**
```bash
pip install crawl4ai crewai
```

2. **Set up API keys:**
```bash
export OPENAI_API_KEY="your-key-here"
```

3. **Import and use:**
```python
from crawl4ai import create_crawl4ai_tools
from crewai import Agent, Task, Crew

tools = create_crawl4ai_tools()
agent = Agent(..., tools=tools, ...)
```

## Support & Documentation

- **Full Documentation**: `docs/crewai_integration.md`
- **Examples**: `examples/` directory
- **Crawl4AI**: https://github.com/unclecode/crawl4ai
- **CrewAI**: https://github.com/joaomdmoura/crewAI

## Summary

The CrewAI integration transforms Crawl4AI into a powerful web scraping tool that autonomous agents can use. It's:

✅ **Easy to use** - Simple one-line integration
✅ **Flexible** - Supports multiple use cases
✅ **Efficient** - Global instance, parallel crawling
✅ **Reliable** - Proper error handling
✅ **Compatible** - Backward compatible, optional dependency
✅ **Well-documented** - Guides and examples included
Loading