CrawlGraph

Crawl any website and visualize its link structure as a force-directed graph. Optionally overlay Google Analytics traffic data to see which pages actually matter.

Two view modes:

• Structure — node size reflects inbound links, color reflects content section
• Traffic — node size reflects sessions, color reflects traffic volume (requires GA4)

Quick Start

# Install dependencies
pip install -r requirements.txt

# Crawl your site
python3 crawl.py https://yoursite.com

# Open the viewer
python3 -m http.server 8080
# Visit http://localhost:8080

How It Works

1. Crawl

python3 crawl.py https://yoursite.com              # default: 600 pages max
python3 crawl.py https://yoursite.com --limit 1000  # crawl more
python3 crawl.py https://yoursite.com -o graph.json # custom output

The crawler follows internal links, deduplicates edges, classifies pages into content groups (blog, docs, product, etc.), and outputs graph.json.

2. View

Serve the directory over HTTP (the viewer loads JSON via fetch):

python3 -m http.server 8080

Open http://localhost:8080 in your browser. You'll see a force-directed graph with:

• Sidebar — collapsible URL tree, search, depth filter, category toggles
• Hub-aware edges — the top 50 most-linked pages (nav, footer, sidebar targets) are classified as hubs. Their edges are excluded from rendering to keep the graph readable. Only meaningful content-to-content edges are shown.
• Zero edges at rest — edges appear on hover/click, not all at once
• Dim-on-select — click a node to highlight its neighborhood and dim everything else. Use the depth slider (0–4 hops) to expand the visible neighborhood via BFS.
• Category toggles — show/hide content groups (blog, docs, product, etc.)

3. Add Traffic Data (Optional)

If you have Google Analytics 4, you can overlay real traffic data:

python3 ga4.py --property-id YOUR_PROPERTY_ID --credentials service_account.json

This pulls yearly per-page traffic and landing page channel breakdowns into traffic.json. The viewer automatically detects this file and enables Traffic mode.

To set up GA4 access:

1. Create a service account in Google Cloud Console
2. Enable the Google Analytics Data API
3. Add the service account email as a viewer on your GA4 property
4. Download the service account JSON key file

In Traffic mode:

• Node size = sessions (sqrt scale)
• Node color = blue→orange heat map (log scale, colorblind-safe)
• Year slider to browse historical data
• Toggle between "All traffic" and "Landing only" (entry-point sessions)
• Tooltips show channel breakdown (Organic Search, Direct, Referral, etc.)

Customizing Content Groups

The crawler classifies pages into groups by URL path patterns. Edit the classify_group() function in crawl.py to match your site's structure:

def classify_group(path):
    if "/blog" in path:
        return "blog"
    if "/docs" in path:
        return "docs"
    # ... add your own patterns
    return "other"

The viewer maps groups to colors using the Wong colorblind-safe palette.

Performance

The viewer handles large sites efficiently:

• Canvas + SVG hybrid — edges drawn on HTML canvas (no DOM overhead), nodes as SVG circles (interactive)
• Hub-aware edge strategy — for a site with 800 pages and 34K edges, ~25K are "chrome" edges (to/from nav hubs) that are never rendered. Only ~9K content edges are used for layout and display.
• On-demand edge rendering — edges only appear when hovering or selecting a node
• Seeded force layout — initial positions are set by content group (angular) and URL depth (radial) for fast convergence

Credits

Interaction patterns (sidebar tree, depth filter, dim-on-select) inspired by GitNexus.

Colorblind-safe palette from Wong, B. "Points of view: Color blindness." Nature Methods 8, 441 (2011).

License

MIT