-
Notifications
You must be signed in to change notification settings - Fork 44
Feature/ingest v2 #439
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Feature/ingest v2 #439
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces the initial work on the IngestV2 API, implementing a new Kotlin-based ingestion service alongside the existing Java implementation. The major change is upgrading the project from Java 8 to Java 11 and updating various dependencies to support the new ingest-v2 module.
- Project upgraded from Java 8 to Java 11 with dependency version updates
- New ingest-v2 module written in Kotlin with modern async capabilities
- Centralized dependency management moved to root POM for consistency
Reviewed Changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| pom.xml | Upgraded Java version, updated dependencies, added ingest-v2 module, centralized dependency management |
| ingest/pom.xml | Removed duplicate dependency management, cleaned up version specifications |
| data/pom.xml | Removed duplicate dependency management, cleaned up version specifications |
| ingest-v2/pom.xml | Complete new module configuration with Kotlin, Ktor, and OpenAPI generation |
| ingest-v2/src/main/resources/openapi.yaml | OpenAPI specification for the new ingest REST API |
| ingest-v2/src/main/kotlin/... | Core Kotlin implementation files for authentication, retry policies, data sources, and API clients |
| ingest-v2/src/test/kotlin/... | Unit tests for retry policies and configuration API |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
...main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/auth/AzCliTokenCredentialsProvider.kt
Outdated
Show resolved
Hide resolved
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/BlobSource.kt
Outdated
Show resolved
Hide resolved
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/source/LocalSource.kt
Outdated
Show resolved
Hide resolved
ingest-v2/src/main/kotlin/com/microsoft/azure/kusto/ingest/v2/common/utils/PathUtils.kt
Outdated
Show resolved
Hide resolved
* Add IngestionSource Blob type and QueuedIngest * Merge code with IngestV2 branch * Reorganize code and remove DTO's * * Fix some review comments --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>
* * Add boilerplate * Merge code with IngestV2 branch * * Add support for non-blob sources * * Add support for non-blob sources * * Add support for File and Stream uploads * managed streaming ingest client * * Minor edits - Make constants for ManagedStreamingIngestPolicy * Add a method called size for determining size in the AbstractSourceInfo interface --------- Co-authored-by: Ramachandran A G <ramacg@microsoft.com> * * Fix some formatting * * Remove unused config --------- Co-authored-by: ag-ramachandran <ramacg@microsoft.com>
* added private link access context * Some refactor and add changes for ONE_LAKE_UPLOAD test * Fix uploader using DFS * Add additional test context for failures * Add error details * Add some info on ClientDetails --------- Co-authored-by: Tanmaya Panda <tanmayapanda@microsoft.com>
* * Refactor to add GZIP Compression * Add a simple streaming sample * Fix issue with compression on Streaming data ingest * Fix copilot review comments * Fix imports * Add assertions --------- Co-authored-by: Tanmaya Panda <tanmayapanda@microsoft.com>
* Remove lints
3529091 to
41bc087
Compare
Initial work on the IngestV2 API