Skip to content

Conversation

@ag-ramachandran
Copy link
Contributor

Initial work on the IngestV2 API

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces the initial work on the IngestV2 API, implementing a new Kotlin-based ingestion service alongside the existing Java implementation. The major change is upgrading the project from Java 8 to Java 11 and updating various dependencies to support the new ingest-v2 module.

  • Project upgraded from Java 8 to Java 11 with dependency version updates
  • New ingest-v2 module written in Kotlin with modern async capabilities
  • Centralized dependency management moved to root POM for consistency

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
pom.xml Upgraded Java version, updated dependencies, added ingest-v2 module, centralized dependency management
ingest/pom.xml Removed duplicate dependency management, cleaned up version specifications
data/pom.xml Removed duplicate dependency management, cleaned up version specifications
ingest-v2/pom.xml Complete new module configuration with Kotlin, Ktor, and OpenAPI generation
ingest-v2/src/main/resources/openapi.yaml OpenAPI specification for the new ingest REST API
ingest-v2/src/main/kotlin/... Core Kotlin implementation files for authentication, retry policies, data sources, and API clients
ingest-v2/src/test/kotlin/... Unit tests for retry policies and configuration API

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@github-actions
Copy link

github-actions bot commented Sep 9, 2025

Test Results

530 tests  ±0   521 ✅ ±0   3m 27s ⏱️ ±0s
 31 suites ±0     9 💤 ±0 
 31 files   ±0     0 ❌ ±0 

Results for commit 51c96be. ± Comparison against base commit b4dd2cf.

♻️ This comment has been updated with latest results.

@AsafMah AsafMah marked this pull request as ready for review September 18, 2025 09:12
tanmaya-panda1 and others added 7 commits December 29, 2025 16:04
* Add IngestionSource Blob type and QueuedIngest
* Merge code with IngestV2 branch
* Reorganize code and remove DTO's
* * Fix some review comments

---------

Co-authored-by: ag-ramachandran <ramacg@microsoft.com>
* * Add boilerplate
* Merge code with IngestV2 branch
* * Add support for non-blob sources
* * Add support for non-blob sources
* * Add support for File and Stream uploads
* managed streaming ingest client
* * Minor edits - Make constants for ManagedStreamingIngestPolicy
* Add a method called size for determining size in the AbstractSourceInfo interface
---------

Co-authored-by: Ramachandran A G <ramacg@microsoft.com>

* * Fix some formatting

* * Remove unused config

---------

Co-authored-by: ag-ramachandran <ramacg@microsoft.com>
* added private link access context
* Some refactor and add changes for ONE_LAKE_UPLOAD test
* Fix uploader using DFS
* Add additional test context for failures
* Add error details
* Add some info on ClientDetails

---------

Co-authored-by: Tanmaya Panda <tanmayapanda@microsoft.com>
* * Refactor to add GZIP Compression
* Add a simple streaming sample
* Fix issue with compression on Streaming data ingest
* Fix copilot review comments
* Fix imports
* Add assertions

---------

Co-authored-by: Tanmaya Panda <tanmayapanda@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants