protocol: add v2 protocol design

.github/workflows/check_json_schemas.yml

@@ -0,0 +1,25 @@
+name: Check v2 JSON schemas
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+jobs:
+  validate:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Setup Python & Poetry Environment
+        uses: exasol/python-toolbox/.github/actions/python-environment@v9
+        with:
+          python-version: "3.10"
+          poetry-version: "2.3.0"
+
+      - name: Install Poetry dependencies
+        run: poetry install --with dev
+
+      - name: Validate v2 JSON schemas
+        run: poetry run nox -s validate-json-schemas

doc/design/v2/README.md

@@ -0,0 +1,19 @@
+# UDF Protocol v2 Design Documents
+
+This directory contains the split protocol design for the new UDF protocol.
+
+- [protocol/design_draft.md](protocol/design_draft.md) is the umbrella design draft.
+- [protocol/low_level/protocol.md](protocol/low_level/protocol.md) describes the wire-level rules, generic call lifecycle, and control
+  stream.
+- [protocol/high_level/calls.md](protocol/high_level/calls.md) describes `Run`, Function operations,
+  `get_connection`, `get_script`, and DB/UDFRunner scheduling policy.
+- [protocol/high_level/payloads.md](protocol/high_level/payloads.md) defines their named string and JSON
+  payload contracts.
+- [protocol/high_level/type_mapping.md](protocol/high_level/type_mapping.md) is the high-level Exasol-to-Arrow
+  column conversion contract, including physical types, parameters, and extension metadata.
+
+Mermaid sources and rendered SVGs use matching names and scopes so the textual and visual material stays aligned.
+The low-level schema defines reusable Arrow-compatible physical type capabilities. Exasol type selection and
+logical/extension metadata are defined by `high_level/type_mapping.md`.
+
+The JSON schemas and external examples can be validated with `poetry run nox -s validate-json-schemas`.

doc/design/v2/protocol/high_level/call_model.mmd

@@ -0,0 +1,13 @@
+flowchart TD
+    Call["Call\n(CallOpen / first peer call-scoped message / CallClose)"]
+    Call -->|opened by DB, or by UDFRunner only when nested| Opener{{"DB opens top-level calls"}}
+    Call -->|optional, at most one| DataStream["Bidirectional Data Stream\n(independent schema per direction)"]
+    Call -->|optional| Nested["Nested Call(s)\n(started by nested CallOpen traffic)"]
+
+    Run["Run (DB-opened)"] -.->|instance of| Call
+    Function["Function operation (DB-opened)"] -.->|instance of| Call
+    GetConnection["get_connection (UDFRunner-opened)"] -.->|instance of| Call
+    GetScript["get_script (UDFRunner-opened)"] -.->|instance of| Call
+
+    Run --> RunData["Input/Output Data Stream"]
+    Function --> FunctionKinds["default output columns / virtual schema / import SQL / export SQL"]

doc/design/v2/protocol/high_level/calls.md

@@ -0,0 +1,161 @@
+# UDF Protocol v2: High-Level Calls
+
+This document describes the high-level protocol calls built on top of the generic call and data-stream mechanisms.
+
+## Scope
+
+This document covers:
+
+- `Run` and Function operations
+- `get_connection` and `get_script`
+- which calls carry data streams
+- representative message sequences
+- DB/UDFRunner scheduling policy
+
+Related diagrams:
+
+- [call_model.svg](call_model.svg)
+- [nested_calls.svg](nested_calls.svg)
+- [run_sequence.svg](run_sequence.svg)
+- [endpoint_scheduling.svg](endpoint_scheduling.svg)
+
+## Call Families
+
+### `DB`-opened calls
+
+| Call | Data stream | Notes |
+| --- | --- | --- |
+| `Run` | Yes, bidirectional | Each direction carries group and row correlation in the data itself. |
+| Function operation | No | One of `default_output_columns`, `virtual_schema_adapter`, `generate_sql_for_import_spec`, or `generate_sql_for_export_spec`. |
+
+### Nested `UDFRunner`-opened calls
+
+| Call | Data stream | Notes |
+| --- | --- | --- |
+| `get_connection` | No | Returns connection information. |
+| `get_script` | No | Returns script content. |
+
+These `UDFRunner`-opened calls are ordinary nested calls, not a separate callback transport. `UDFRunner` does not open
+top-level calls while idle; it can open them only while handling an active DB call.
+
+## Call-Specific Result Payloads
+
+Each high-level call defines the names and bodies of its own result payloads, carried by `Payloads(...)`. A result
+payload may be sent while the call remains active or together with `CloseCall` in the same composite
+`StreamMessage`. Calls that have no result payload may still close normally.
+
+## Typical Semantics
+
+### `Run`
+
+- opened by `DB`
+- uses `call_metadata` and `column_metadata`, sent before any call, between calls, or with the opening message
+- may carry the first input batch together with the opening message
+- may stay active while nested calls such as `get_script` or `get_connection` execute
+- group and row correlation belong in the data, not in `Next(...)`
+
+#### Group and Row Correlation
+
+Each `Run` direction may combine multiple logical groups in one `DataRecordBatch`. Its `DataSchema` sets both
+`has_group_id` and `has_row_id` to `true`, adding an ordered reserved prefix before user data columns:
+
+| Position | Column | Purpose |
+| --- | --- | --- |
+| `0` | Group ID | Identifies the logical input group. |
+| `1` | Row ID | Identifies the input row to which an output row maps. |
+| `2+` | User data | Input or output columns defined by the call. |
+
+Groups may span multiple rows. In particular, a `SET ... EMITS` UDF may receive multiple input rows in one group.
+The group ID and row ID columns are correlation fields identified only by this prefix layout, not by field names.
+
+`DataRecordBatch.is_end_of_group` marks whether the group identified by the batch's final row is complete. It is
+defined only for a `Run` direction whose schema sets `has_group_id` to `true`:
+
+- `true` means no later batch in that direction contains the trailing group.
+- `false` means the trailing group continues in a later batch.
+- a change in group ID still delimits each non-trailing group within the same batch.
+- an empty batch does not complete a group.
+
+This is a group-boundary marker, not an end-of-stream marker. Generic stream-completion semantics remain a
+low-level open question and are not encoded in `DataRecordBatch` metadata.
+
+The preferred encodings are:
+
+| Column and direction | Preferred encoding | Compatible fallback |
+| --- | --- | --- |
+| Group ID, either direction | `RunEndEncoded` over unsigned 64-bit IDs when groups contain repeated rows. | Plain unsigned 64-bit. |
+| Row ID, `DB` to `UDFRunner` | `exasol.udf.range_run` extension array. | Plain unsigned 64-bit. |
+| Row ID, `UDFRunner` to `DB`, `RETURNS` UDF | `exasol.udf.range_run` extension array. | Plain unsigned 64-bit. |
+| Row ID, `UDFRunner` to `DB`, `EMITS` UDF | `RunEndEncoded` over unsigned 64-bit IDs. | Plain unsigned 64-bit. |
+
+`exasol.udf.range_run` uses `RunEndEncoded` as its Arrow storage type. Its `run_ends` child is a signed 64-bit
+integer array and its unsigned 64-bit `values` child stores the first row ID for each run; each following logical
+value in that run increases by one. The field sets `ARROW:extension:name` to `exasol.udf.range_run`; no extension
+metadata is required in version 1.
+
+### Function Operations
+
+- opened by `DB` with one of the Function operation names
+- each call uses `call_metadata` and `column_metadata`, sent before any call, between calls, or with the opening message
+- has no attached data stream in the current model
+- has the operation-specific request and result payloads defined in
+  [payloads.md](payloads.md)
+
+### `get_connection`
+
+- opened by `UDFRunner`
+- returns `Payloads(connection_information)`
+- may still carry additional named payload traffic while active
+
+### `get_script`
+
+- opened by `UDFRunner`
+- returns `Payloads(script)`
+- may still carry additional named payload traffic while active
+
+## Payload Contracts
+
+The complete call-metadata, script-metadata, Function, and nested-call payload contracts are defined in
+[payloads.md](payloads.md). `StringPayload` is used directly for
+scalar strings; JSON is used only where a payload has structured fields.
+
+## Representative Sequences
+
+The current design keeps the high-level sequences intentionally simple:
+
+- nested callback-style calls execute while a parent `Run` or Function call remains active
+- `Run` combines `OpenCall`, `call_metadata`, input schema announcement, and the first input batch when practical
+
+See [nested_calls.svg](nested_calls.svg) and
+[run_sequence.svg](run_sequence.svg).
+
+## Scheduling Policy
+
+The source material implies the following DB/UDFRunner deadlock-avoidance rules. These rules govern high-level
+call orchestration and do not alter the generic Client/Server stream rules in the low-level protocol.
+
+### `UDFRunner`
+
+1. run socket handling and user-code execution as independently wakeable activities
+2. wait for either DB socket activity or user-code activity; do not block solely on socket receive
+3. use `Next(...)` byte budgets to bound data in flight; do not impose a message-count limit
+4. send regular `KeepAlive` messages so `DB` can continue housekeeping
+
+### `DB`
+
+1. prioritize nested-call responses before data-stream work
+2. if nothing is ready to send, block waiting for new incoming messages
+3. monitor peer liveness and terminate unhealthy sessions when needed
+
+See [endpoint_scheduling.svg](endpoint_scheduling.svg).
+
+## Forward-Looking Ideas Still Open
+<>
+- `ExecuteScript` and `execute_query` call shapes and data streams
+- whether `UDFRunner` may open its own pquery-style call to `DB`
+- whether table-prefetch-like declarations should be added for future call setup
+
+## Relationship To Other Docs
+
+- low-level protocol lives in [../low_level/protocol.md](../low_level/protocol.md)
+- high-level payload contracts live in [payloads.md](payloads.md)

doc/design/v2/protocol/high_level/endpoint_scheduling.mmd

@@ -0,0 +1,22 @@
+stateDiagram-v2
+    state "UDFRunner" as UDF {
+        [*] --> UdfIdle
+        UdfIdle --> UdfInitializing: Accept
+        UdfInitializing --> UdfActive: Capabilities
+        UdfActive --> UdfWaitingForActivity: Wait for DB or user code
+        UdfWaitingForActivity --> UdfSendingData: Next credit
+        UdfSendingData --> UdfActive: Transfer done
+        UdfWaitingForActivity --> UdfActive: Socket or user-code event
+    }
+
+    state "DB" as DB {
+        [*] --> DbIdle
+        DbIdle --> DbInitializing: Connect
+        DbInitializing --> DbActive: Capabilities
+        DbActive --> DbHandlingNestedCall: Call
+        DbHandlingNestedCall --> DbActive: Reply
+        DbActive --> DbWaitingForUdfMessage: Wait
+        DbWaitingForUdfMessage --> DbActive: Message
+        DbActive --> DbSendingData: Data transfer
+        DbSendingData --> DbActive: Transfer done
+    }

doc/design/v2/protocol/high_level/examples/call_metadata.json

@@ -0,0 +1,14 @@
+{
+  "database_name": "EXASOL",
+  "database_version": "8.0",
+  "session_id": "42",
+  "statement_id": 1,
+  "node_count": 1,
+  "node_id": 0,
+  "vm_id": "7",
+  "maximal_memory_limit": "1073741824",
+  "script_schema": "SYS",
+  "input_iter_type": "EXACTLY_ONCE",
+  "output_iter_type": "EXACTLY_ONCE",
+  "single_call_mode": false
+}

doc/design/v2/protocol/high_level/examples/column_definitions.json

@@ -0,0 +1,35 @@
+[
+  {
+    "name": "NAME",
+    "type": "VARCHAR",
+    "type_name": "VARCHAR(128)",
+    "size": 128,
+    "character_set": "UTF8"
+  },
+  {
+    "name": "DIGEST",
+    "type": "HASHTYPE",
+    "type_name": "HASHTYPE(32 BYTE)",
+    "size": 32,
+    "size_unit": "BYTE"
+  },
+  {
+    "name": "LOCATION",
+    "type": "GEOMETRY",
+    "type_name": "GEOMETRY(4326)",
+    "srid": 4326
+  },
+  {
+    "name": "AGE",
+    "type": "INTERVAL YEAR TO MONTH",
+    "type_name": "INTERVAL YEAR(4) TO MONTH",
+    "precision": 4
+  },
+  {
+    "name": "ELAPSED",
+    "type": "INTERVAL DAY TO SECOND",
+    "type_name": "INTERVAL DAY(6) TO SECOND(9)",
+    "precision": 6,
+    "fractional_second_precision": 9
+  }
+]

doc/design/v2/protocol/high_level/examples/column_metadata.json

@@ -0,0 +1,12 @@
+{
+  "input_columns": [
+    {
+      "name": "AMOUNT",
+      "type": "DECIMAL",
+      "type_name": "DECIMAL(12,2)",
+      "precision": 12,
+      "scale": 2
+    }
+  ],
+  "output_columns": []
+}

doc/design/v2/protocol/high_level/examples/connection_information.json

@@ -0,0 +1,6 @@
+{
+  "kind": "JDBC",
+  "address": "jdbc:example://host/database",
+  "user": "user",
+  "password": "secret"
+}

doc/design/v2/protocol/high_level/examples/day_time_interval_field_metadata.json

@@ -0,0 +1,5 @@
+{
+  "name": "ELAPSED",
+  "nullable": true,
+  "arrow_storage_type": "Interval(MonthDayNano)"
+}

doc/design/v2/protocol/high_level/examples/decimal_field_metadata.json

@@ -0,0 +1,5 @@
+{
+  "name": "AMOUNT",
+  "nullable": true,
+  "arrow_storage_type": "Decimal(64)"
+}

doc/design/v2/protocol/high_level/examples/export_specification.json

@@ -0,0 +1,6 @@
+{
+  "has_truncate": true,
+  "has_replace": false,
+  "source_column_names": ["ID", "NAME"],
+  "connection_name": "REMOTE_CONNECTION"
+}

doc/design/v2/protocol/high_level/examples/geometry_extension_metadata.json

@@ -0,0 +1,4 @@
+{
+  "ARROW:extension:name": "geoarrow.wkb",
+  "ARROW:extension:metadata": "{\"crs\":\"4326\",\"crs_type\":\"srid\"}"
+}

doc/design/v2/protocol/high_level/examples/hashtype_field_metadata.json

@@ -0,0 +1,5 @@
+{
+  "name": "DIGEST",
+  "nullable": true,
+  "arrow_storage_type": "FixedSizeBinary(32)"
+}

doc/design/v2/protocol/high_level/examples/import_specification.json

@@ -0,0 +1,14 @@
+{
+  "is_subselect": true,
+  "connection_name": "REMOTE_CONNECTION",
+  "subselect_column_specification": [
+    {
+      "name": "ID",
+      "type": "DECIMAL",
+      "type_name": "DECIMAL(18,0)",
+      "precision": 18,
+      "scale": 0
+    }
+  ],
+  "parameters": [{ "key": "encoding", "value": "UTF-8" }]
+}

doc/design/v2/protocol/high_level/examples/timestamp_field_metadata.json

@@ -0,0 +1,5 @@
+{
+  "name": "CREATED_AT",
+  "nullable": false,
+  "arrow_storage_type": "Timestamp(Microsecond, UTC)"
+}

doc/design/v2/protocol/high_level/examples/year_month_interval_field_metadata.json

@@ -0,0 +1,9 @@
+{
+  "name": "AGE",
+  "nullable": true,
+  "arrow_storage_type": "Int64",
+  "metadata": {
+    "ARROW:extension:name": "exasol.interval.year_month",
+    "ARROW:extension:metadata": "{\"layout\":\"signed_total_months\"}"
+  }
+}

doc/design/v2/protocol/high_level/nested_calls.mmd

@@ -0,0 +1,12 @@
+sequenceDiagram
+    participant UDFRunner
+    participant DB
+
+    Note over UDFRunner,DB: Nested calls opened by UDFRunner while a parent DB-opened call remains active
+
+    UDFRunner->>DB: CallOpen(get_connection) + Payloads(connection_name)
+    DB-->>UDFRunner: Payloads(connection_information)
+    UDFRunner->>DB: CloseCall
+
+    UDFRunner->>DB: CallOpen(get_script) + Payloads(script_name)
+    DB-->>UDFRunner: Payloads(script) + CloseCall