Adblock Compiler (currently in code freeze at v0.79.4 for e2e testing and refactoring)

Compiler-as-a-Service for adblock filter lists. Transform, optimize, and combine filter lists from multiple sources with real-time progress tracking.

🌐 Try the Admin Dashboard | 🔧 Compiler UI | 🚀 API Endpoint | 📚 Documentation | WIP

✨ Features

• 🎯 Multi-Source Compilation - Combine filter lists from URLs, files, or inline rules
• ⚡ Performance - Gzip compression (70-80% cache reduction), request deduplication, smart caching
• 🔄 Circuit Breaker - Automatic retry with exponential backoff for unreliable sources
• 📊 Visual Diff - See what changed between compilations
• 🎪 Batch Processing - Compile up to 10 lists in parallel
• 📡 Real-time Updates - Server-Sent Events (SSE) and WebSocket support
• 🔔 Async Notifications - Get notified when background jobs complete
• 🌐 Admin Dashboard - Monitor metrics, queue depth, and system health
• 📖 OpenAPI 3.0 Specification - Full API documentation with contract tests
• 🌍 Universal - Works in Deno, Node.js, Cloudflare Workers, browsers
• 🖥️ Angular 21 SPA - Production frontend with zoneless change detection, Material Design 3, and SSR
• 🎨 11 Transformations - Deduplicate, compress, validate, and more
• 📝 Structured Logging - Production-ready JSON logs for observability (CloudWatch, Datadog, Splunk)
• 🚨 Error Reporting - Centralized error tracking with Sentry and Cloudflare Analytics Engine
• ✅ Zod Validation - Runtime schema validation for all configurations and API inputs
• 🧩 AST Rule Parsing - Full abstract syntax tree parsing via @adguard/agtree
• ⚙️ Cloudflare Workflows - Durable execution for long-running and background compilations

📚 Documentation

Full documentation is available in mdBook format at this link and in the /docs directory.

🚀 Quick Start

Installation

Run directly without installation:

deno run --allow-read --allow-write --allow-net jsr:@jk-com/adblock-compiler -c config.json -o output.txt

Or install globally:

deno install --allow-read --allow-write --allow-net -n adblock-compiler jsr:@jk-com/adblock-compiler/cli

Or run with Docker:

docker compose up -d

Access the web UI at http://localhost:8787

📚 Quick Start Guide | 📚 Docker Deployment Guide

Basic Usage

Quick hosts conversion

adblock-compiler -i hosts.txt -i hosts2.txt -o output.txt

Build a configurable blocklist from multiple sources

adblock-compiler -c configuration.json -o output.txt

CLI Options

Usage: adblock-compiler [options]

General:
  -c, --config <file>          Path to the compiler configuration file
  -i, --input <source>         URL or file path to convert (repeatable)
  -t, --input-type <type>      Input format: hosts|adblock [default: hosts]
  -v, --verbose                Enable verbose logging
  -b, --benchmark              Show performance benchmark report
  -q, --use-queue              Use asynchronous queue-based compilation
      --priority <level>       Queue priority: standard|high [default: standard]
      --version                Show version number
  -h, --help                   Show help

Output:
  -o, --output <file>          Path to the output file [required unless --stdout]
      --stdout                 Write output to stdout instead of a file
      --append                 Append to output file instead of overwriting
      --format <format>        Output format
      --name <file>            Compare output against existing file and print diff
      --max-rules <n>          Truncate output to at most <n> rules

Transformations:
      --no-deduplicate         Skip the Deduplicate transformation
      --no-validate            Skip the Validate transformation
      --no-compress            Skip the Compress transformation
      --no-comments            Skip the RemoveComments transformation
      --invert-allow           Apply the InvertAllow transformation
      --remove-modifiers       Apply the RemoveModifiers transformation
      --allow-ip               Use ValidateAllowIp instead of Validate
      --convert-to-ascii       Apply the ConvertToAscii transformation
      --transformation <name>  Explicit transformation pipeline (repeatable,
                               overrides all other transformation flags)

Filtering:
      --exclude <pattern>      Exclude rules matching pattern (repeatable)
      --exclude-from <file>    Load exclusions from file (repeatable)
      --include <pattern>      Include only rules matching pattern (repeatable)
      --include-from <file>    Load inclusions from file (repeatable)

Networking:
      --timeout <ms>           HTTP request timeout in milliseconds
      --retries <n>            Number of HTTP retry attempts
      --user-agent <string>    Custom HTTP User-Agent header

Authentication (when using --use-queue with a remote API):
      --api-key <key>          API key (abc_...) for authenticated requests
      --bearer-token <token>   Clerk JWT Bearer token for authenticated requests
      --api-url <url>          Worker API base URL [default: http://localhost:8787]

Examples:
  adblock-compiler -c config.json -o output.txt
      compile a blocklist and write the output to output.txt

  adblock-compiler -i hosts.txt -i hosts2.txt --stdout
      compile from multiple inputs and stream to stdout

  adblock-compiler -c config.json -o output.txt --no-compress --allow-ip
      compile without compression, keeping IP-address rules

  adblock-compiler -i https://example.org/hosts.txt -o output.txt \
      --transformation RemoveComments --transformation Deduplicate
      compile with an explicit transformation pipeline

📚 See the CLI Reference for the full flag reference and advanced examples.

📖 Further Reading

Topic	Doc
CLI reference	docs/usage/CLI.md
Configuration reference	docs/usage/CONFIGURATION.md
Transformations reference	docs/usage/TRANSFORMATIONS.md
TypeScript API & Zod validation	docs/api/README.md
OpenAPI specification	docs/api/OPENAPI_TOOLING.md
Platform support & custom fetchers	docs/api/PLATFORM_SUPPORT.md
Extensibility	docs/development/EXTENSIBILITY.md
Structured logging & OpenTelemetry	docs/development/LOGGING.md
Error reporting	docs/development/ERROR_REPORTING.md
Docker deployment	docs/deployment/docker.md
Cloudflare Workers deployment	docs/deployment/cloudflare-pages.md
Angular frontend	frontend/README.md

🏗️ Tech Stack

Layer	Technology	Notes
Runtime	Deno 2.x, Cloudflare Workers, Typescript 6.0	Edge-first execution, Deno runtime, Typescript 6.x
Framework	Hono, tRPC	Lightweight web framework for Workers
Database	Neon PostgreSQL (primary), Cloudflare D1 (edge cache)	Neon via Hyperdrive connection pooling
ORM	Prisma 7.x	@prisma/adapter-pg for PostgreSQL, @prisma/adapter-d1 for D1
Authentication	Better Auth (primary)	Better Auth for new sessions; Clerk for legacy tokens during migration
Storage	Cloudflare R2, KV	R2 for compiled outputs, KV for caching & rate limits
Frontend	Angular 21, Vite, TailwindCSS	Zoneless change detection, Material Design 3, SSR on Workers
Validation	Zod	Runtime schema validation at all trust boundaries
API	OpenAPI 3.0, Postman	OpenAPI 3.0 specification
Routing	Hono Router	Hono router for Typescript
Testing	Deno 2.x, Vitest, Postman, Playwright	Deno test for backend, Vitest for frontend, Postman for API, Playwright for e2e
Security	Full ZTA, JWT	Zero-trust architecture at every layer/API call via JWT + Better Auth
Observability	Sentry, Prometheus, Cloudflare Observatory	Prometheus, Sentry, and Cloudflare Observatory log pushing
DevOps	Github, CodeQL, Lighthouse	DevOps hosted on Github with CodeQL and Lighthouse integration
Architecture	Edge First, Workers Modules, Cloud Native, ZTA, Microservices	Cloudlfare-native microservices hosted in Workers, ZTA at every boundary

🔧 Development

Prerequisites

• Deno 2.0 or later

Available tasks

Note: Always use deno task test instead of deno test directly. The tests require specific permissions (--allow-read, --allow-write, --allow-net, --allow-env) that are configured in the task.

# Run in development mode with watch
deno task dev

# Run the compiler
deno task compile

# Build standalone executable
deno task build

# Run tests (all tests co-located with source files in src/)
deno task test

# Run tests in watch mode
deno task test:watch

# Run tests with coverage
deno task test:coverage

# Run specific test file (with required permissions)
deno test --allow-read --allow-write --allow-net --allow-env src/cli/ArgumentParser.test.ts

# Lint code
deno task lint

# Format code
deno task fmt

# Check formatting
deno task fmt:check

# Type check
deno task check

# Cache dependencies
deno task cache

# Generate the TypeScript API reference (into book/api-reference/)
deno task docs:api

# Build the full mdBook site + API reference (requires mdBook CLI installed)
deno task docs:build

# Live-preview the mdBook (does not include API reference; requires mdBook CLI installed)
deno task docs:serve

Angular Frontend Development

The primary frontend is an Angular 21 SPA in frontend/. It uses:

• Angular 21 with zoneless change detection, signals, rxResource, linkedSignal
• Angular Material 3 for UI components and theming
• SSR via @angular/ssr on Cloudflare Workers
• Vitest for unit testing

# Development server (http://localhost:4200)
pnpm --filter adblock-frontend run start

# Production build
pnpm --filter adblock-frontend run build

# Run tests
pnpm --filter adblock-frontend run test

For full-stack local development:

deno task wrangler:dev        # Worker on :8787
pnpm --filter adblock-frontend run start  # Angular on :4200 → proxies /api to :8787

Pages:

• Dashboard — live metrics from /api/metrics and /api/health
• Compiler — filter list compilation with JSON and SSE streaming modes, drag-and-drop
• Performance — real-time compilation latency and throughput
• Validation — AGTree-powered filter rule validation
• API Docs — HTTP endpoint reference
• Admin — KV/R2/D1 storage management (requires admin key)

The public/ directory contains the original vanilla HTML frontend. It will be removed once the Angular migration is complete.

📄 SPA Benefits Analysis - Analysis of SPA benefits for this application

Project structure

mindmap
  root((Project structure))
    src["src/"]
      cli["cli/ — Command-line interface (with *.test.ts files)"]
      compiler["compiler/ — Core compilation logic (with *.test.ts files)"]
      configuration["configuration/ — Configuration validation (with *.test.ts files)"]
      downloader["downloader/ — Filter list downloading (with *.test.ts files)"]
      platform["platform/ — Platform abstraction layer (with *.test.ts files)"]
      transformations["transformations/ — Rule transformations (with *.test.ts files)"]
      types["types/ — TypeScript type definitions"]
      utils["utils/ — Utility functions (with *.test.ts files)"]
      indexTs["index.ts — Main library exports"]
      modTs["mod.ts — Deno module exports"]
    worker["worker/ — Cloudflare Worker implementation (production-ready)"]
      workerTs["worker.ts — Main worker with API endpoints"]
      htmlTs["html.ts — Fallback HTML templates"]
    frontend["frontend/ — Angular 21 SPA (replaces public/)"]
      app["src/app/ — Components, services, guards, interceptors"]
      indexHtml["src/index.html — Root HTML with Cloudflare analytics"]
      angularJson["angular.json — Build configuration (SSR + browser)"]
    public["public/ — Legacy static web UI (to be removed)"]
    examples["examples/"]
      cloudflareWorker["cloudflare-worker/ — Legacy deployment reference"]

Loading

Publishing to JSR

# Dry run to verify everything is correct
deno publish --dry-run

# Publish to JSR
deno publish

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Quick Start for Contributors

1. Fork and clone the repository
2. Make your changes following our commit message guidelines
3. Use conventional commits: feat:, fix:, docs:, etc.
4. Submit a pull request

Automatic Version Bumping: When your PR is merged, the version will be automatically bumped based on your commit messages. See docs/reference/AUTO_VERSION_BUMP.md for details.

📄 License

MIT - See LICENSE for details.