Declarative Data Pipeline

A multi-module Spark-based data pipeline application built with Scala and Java.

Modules

• core - Core declarative data pipeline framework
• examples - Example implementations and sample jobs

Build

# Build all modules
sbt compile

# Build specific module
sbt core/compile
sbt examples/compile

Test

# Test all modules
sbt test

# Test specific module
sbt core/test

Run Examples

# Run the simple example
sbt examples/run

Dependencies

• Apache Spark 3.5.4
• Scala 2.12.19
• Java 17+

Package Structure

• core/ - Core framework with com.amazonaws.declarative.data.pipeline package
• examples/ - Sample jobs demonstrating the declarative data pipeline pattern
• Supports both Scala and Java components
• Includes unit tests for both languages