Commit 77a0d7e
authored
Refactored DataFrame JDBC API plus DataSource handling (#1487)
* Refactored DataFrame JDBC API for enhanced DataSource handling
This commit introduces new DataFrame JDBC extension functions with `DataSource` support, removing redundant duplicate utilities. It includes revisions to streamline table/schema reading and validation features, delegating reusable connection-handling logic to a dedicated utility class. Also refactor file structure for better organization of DB-related code.
* Refactored schema extraction to use `readSqlTable` and `readSqlQuery` for consistency and improved readability.
* Refactored and modularized schema extraction utilities into a dedicated file `readDataFrameSchema.kt`, improving organization and code clarity. Converted several helper functions to `internal` visibility for encapsulation.
* Refactor: Replace `DataFrame` with `DataFrameSchema` for schema-related methods in tests and main codebase
* Update logging levels in validation utilities to debug and minor schema interface cleanup
* Refactor: support custom `PreparedStatement` configuration, unify query limits, and standardize identifier quoting for various database dialects
* Refactor: enhance `DbType` with batch size and query timeout properties, improve result set processing, and streamline table metadata handling
* Refactor: centralize `makeCommonSqlToKTypeMapping` in `DbType`, streamline SQL type handling and improve readability
* Refactored query execution logic by introducing `readDataFrameFromDatabase` utility for improved code reuse and clarity.
* Refactored ResultSet-processing utilities to use mutable lists for improved post-processing efficiency and reduced copying. Adjusted related functions to accept mutable lists accordingly.
* Add `configureStatement` missed parameters
* Refactored JDBC utilities: added comprehensive error handling in `readDataFrameFromDatabase`, converted data classes to classes with equality and hashCode implementations, added validation for database interaction methods.
* Update the exception type in the ` read from non-existing table` test for accuracy
* Renamed schema extraction functions from `getSchemaFor*` to `from*` for consistency and clarity across the DataFrame JDBC API. Updated all usages and tests accordingly.
* Rename `fromSqlTable` and `fromSqlQuery` to `readSqlTable` and `readSqlQuery` for consistency with updated naming conventions.
* Update `GenerateDataSchemaTask` to use `DataFrameSchema` methods for SQL table and query schema generation
* Refactor: improve code consistency, update parameter documentation, streamline SQL query-related methods, and enhance table schema handling
* Replace `DEFAULT_LIMIT` with nullable `limit` parameter, defaulting to `null` for unlimited row fetching. Updated all related methods and documentation for clarity.
* Add `validateLimit` utility to ensure limit parameter is null or positive; removed redundant exception handling and updated query-building logic.
* Add `validateLimit` calls across all JDBC read methods to enforce limit validation.
* Clarify "limit" parameter documentation and rename `readDataFrameFromDatabase` to `executeQueryAndBuildDataFrame` across JDBC methods for improved readability and consistency.
* Refactor JDBC data handling: relocate and centralize `buildSchemaByTableColumns` and `buildDataColumn` functions, streamline column post-processing logic, and enhance modularity across schema and SQL utilities.
* Ktlint with Junie
* Linter with Junie, part 2
* Refactor and enhance JDBC: update references for improved consistency, streamline `TableMetadata` class with compact constructor and copy method, and remove unused `columnMetadata` parameter from `buildDataColumn`.
* Add `DataFrameSchema.Companion` class to core API1 parent a27e415 commit 77a0d7e
File tree
33 files changed
+2009
-1124
lines changed- core
- api
- src/main/kotlin/org/jetbrains/kotlinx/dataframe/schema
- dataframe-jdbc
- api
- src
- main/kotlin/org/jetbrains/kotlinx/dataframe/io
- db
- test/kotlin/org/jetbrains/kotlinx/dataframe/io
- h2
- local
- plugins
- dataframe-gradle-plugin/src/main/kotlin/org/jetbrains/dataframe/gradle
- symbol-processor/src/main/kotlin/org/jetbrains/dataframe/ksp
33 files changed
+2009
-1124
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6810 | 6810 | | |
6811 | 6811 | | |
6812 | 6812 | | |
| 6813 | + | |
6813 | 6814 | | |
6814 | 6815 | | |
6815 | 6816 | | |
6816 | 6817 | | |
6817 | 6818 | | |
| 6819 | + | |
| 6820 | + | |
| 6821 | + | |
6818 | 6822 | | |
6819 | 6823 | | |
6820 | 6824 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
Large diffs are not rendered by default.
Lines changed: 93 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
0 commit comments