A modular and scalable search engine built to process, index, and rank web data using modern Big Data technologies.
This project is structured into four main modules:
Collects raw web data from multiple sources concurrently, storing it for later processing.
Tags and structures the collected content, converting it to Avro format for efficient storage and compatibility with distributed systems.
Computes document relevance through algorithms based on content and link analysis, preparing optimized data for search queries.
Processes user queries and retrieves ranked results based on computed relevance scores.
| Component | Purpose |
|---|---|
| HDFS | Distributed storage system for raw and processed data |
| Kafka | Real-time data ingestion and communication between modules |
| Spark | Distributed data processing and indexing engine |
Below is the full system architecture and data flow diagram:
Please review our community guidelines in CODE_OF_CONDUCT.md.
- Distributed crawling and indexing
- Data serialization using Avro
- Scalable and fault-tolerant architecture
- Modular design for experimentation and extension
- Compatible with large-scale web data pipelines
- Clone the repository:
git clone https://github.com/Marioxuloh/Aether.git cd Aether - Launch System:
docker compose up -d
- Aether: port 5050
- HDFS: port 9870
- Spark: port 8080
- Ranking API: port 8000 with postman collection in /docs