[KYUUBI #7458] Add server-side statement interceptor SPI by wangzhigang1999 · Pull Request #7530 · apache/kyuubi

wangzhigang1999 · 2026-06-29T08:13:18Z

Why are the changes needed?

First half of #7458 (the statement hook; the result-set hook remains follow-up work). Kyuubi lacks a gateway-level, per-statement hook before a statement is routed to an engine; the existing Spark authorization path is Spark-specific. This adds a server-side StatementInterceptor SPI for interactive statements before operation creation and routing. Interceptors can inspect, rewrite, reject, or tune per-statement configuration, providing one stable hook for authorization, risky-SQL interception, rewriting, and execution tuning.

The public SPI lives in the dependency-free kyuubi-server-plugin module and exposes JDK types only; session credentials and mutable session configuration are not exposed.
kyuubi.operation.statement.interceptors has server audience and is immutable to session users. Plugins initialize eagerly, run in configured order, and fail closed on exceptions or invalid results.
Configuration deltas accumulate across the interceptor chain. Their effect is engine-dependent; Spark SQL execution and plan-only paths consume the supported per-statement overlay without cloning SQLConf when the overlay is empty.
statementId() is allocated before interception and, when execution proceeds, matches the client operation handle for later correlation.

The SPI currently covers interactive statements only.

How was this patch tested?

Coverage includes unit tests (StatementInterceptionSuite, PluginLoaderSuite) and the JDBC-to-server-to-engine StatementInterceptorSuite for proceed, rewrite, reject, executeOnServer, fail-closed behavior, concurrency, statement correlation, and configuration tuning.

After rebasing onto current apache/master, the refresh was validated with:

dev/reformat
dev/gen/gen_all_config_docs.sh (AllKyuubiConfiguration: 1/1)
build/mvn -Pflink-provided,hive-provided,spark-provided test -pl kyuubi-server -am -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.StatementInterceptionSuite (18/18)

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: Claude Opus 4.8 and Codex GPT-5

wangzhigang1999 · 2026-06-30T06:41:40Z

Design note — statementId

StatementInterceptContext#statementId() is the operation-handle UUID, pre-allocated before interception and reused for the operation (instead of being self-generated afterwards). So one stable id spans intercept time → operation → result set → the handle the client gets. That's the correlation/join key for end-to-end audit and lineage, and for the planned result-set transform hook (second half of #7458). It's threaded down via a defaulted ctor param through AbstractOperation/KyuubiOperation/ExecuteStatement/ExecutedCommandExec, so no engine ripple.

Also manually verified on a Kubernetes deployment: REWRITE/REJECT over Beeline (reject surfaces SQLState 42000), statementId matches the operation handle, and neither batch nor a session-level override bypasses the serverOnly interceptor.

pan3793 · 2026-07-14T03:12:54Z

Before taking a detailed look, another common use case, in addition to REWRITE/REJECT, is to tune the config.

wangzhigang1999 · 2026-07-14T03:31:40Z

Before taking a detailed look, another common use case, in addition to REWRITE/REJECT, is to tune the config.

Good point. The current SPI exposes the statement-level conf overlay as read-only, so an interceptor cannot tune it.

I am considering letting StatementInterceptResult carry a config-overlay delta. The interceptor chain would accumulate those updates in order, each following interceptor would see the updated overlay, and later values would win. The final overlay would then be passed to the engine operation together with the intercepted statement.

I prefer an explicit result delta over making StatementInterceptContext mutable, since config tuning can also be combined with REWRITE and the chain semantics remain clear.

Does that match the use case you have in mind?

Copilot

Pull request overview

Adds a gateway-level, server-side StatementInterceptor SPI that runs on each interactive statement before operation creation/routing, enabling statement inspection/rewrite/reject and per-statement config tuning, while preserving a stable statementId correlated with the resulting operation handle.

Changes:

Introduces StatementInterceptor / StatementInterceptContext / StatementInterceptResult SPI in kyuubi-server-plugin, plus server-side execution and lifecycle management (eager init, fail-closed, close on shutdown).
Wires interception into the interactive execution path and pre-allocates OperationHandle so the interceptor observes the same id returned to clients.
Propagates per-statement config overlays into Spark engine statement execution and plan-only paths; adds unit + end-to-end coverage and user documentation.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
kyuubi-server/src/test/scala/org/apache/kyuubi/plugin/StatementInterceptionSuite.scala	Unit tests for interceptor chaining semantics, fail-closed behavior, config overlay accumulation, and thread-safety.
kyuubi-server/src/test/scala/org/apache/kyuubi/plugin/PluginLoaderSuite.scala	Tests loading/ordering/error handling for statement interceptors via `PluginLoader`.
kyuubi-server/src/test/scala/org/apache/kyuubi/operation/StatementInterceptorSuite.scala	End-to-end JDBC→server→engine tests validating proceed/rewrite/reject/tune plus `statementId` and impersonation behavior.
kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala	Eagerly loads/initializes/closes interceptors; adds `interceptStatement` entry point used by sessions.
kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala	Runs interception before parsing/routing; pre-allocates and reuses an `OperationHandle` for stable `statementId`.
kyuubi-server/src/main/scala/org/apache/kyuubi/plugin/StatementInterception.scala	Implements interceptor initialization lifecycle + per-statement execution chain and fail-closed semantics.
kyuubi-server/src/main/scala/org/apache/kyuubi/plugin/PluginLoader.scala	Adds `loadStatementInterceptors` to instantiate configured interceptor classes.
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/KyuubiOperationManager.scala	Adds overloads allowing server-side injection of a pre-allocated `OperationHandle`.
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/KyuubiOperation.scala	Makes operation handle injectable via constructor parameter to preserve statement id during superclass init.
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/ExecuteStatement.scala	Accepts injected `OperationHandle` for statement operations.
kyuubi-server/src/main/scala/org/apache/kyuubi/operation/ExecutedCommandExec.scala	Accepts injected `OperationHandle` for server-side command operations.
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala	Adds `kyuubi.operation.statement.interceptors` server-only immutable config entry.
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/SparkSQLOperationManager.scala	Passes per-statement overlay into Spark operations (execute + plan-only).
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/PlanOnlyStatement.scala	Applies per-statement overlay via an operation-scoped `SQLConf`.
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecuteStatement.scala	Applies per-statement overlay via an operation-scoped `SQLConf` for Spark statement execution.
extensions/server/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/StatementInterceptResult.java	Defines immutable interceptor decision object (proceed/rewrite/reject) plus config overlay delta.
extensions/server/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/StatementInterceptor.java	Defines the SPI interface and lifecycle (`initialize`/`beforeExecuteStatement`/`close`).
extensions/server/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/StatementInterceptContext.java	Defines stable gateway context (JDK types only) including `statementId`.
docs/extensions/server/statement_interceptor.rst	Documents SPI contract, config, semantics, and examples/notes.
docs/extensions/server/index.rst	Adds statement interceptor docs page to server extensions index.
docs/configuration/settings.md	Adds generated settings entry for `kyuubi.operation.statement.interceptors`.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Add a user-facing English doc for the statement interceptor SPI under docs/extensions/server and register it in the server extensions toctree.

…ptor Pre-allocate the operation handle before interception and reuse it for the operation instead of letting the operation self-generate one, and add StatementInterceptContext#statementId so an interceptor can correlate the intercepted statement with its later operation and result set. The id equals the operation handle the client receives.

…cted statements Expose StatementInterceptContext#realUser, the authenticated user before any impersonation, alongside user() which is documented as the effective (proxy) user the statement runs as, so an interceptor can authorize and audit both the effective and the real identity. Throw the REJECT error with SQLState 42000 (access rule violation) so JDBC and BI clients can tell a policy rejection from a syntax error programmatically rather than by parsing the message text. Claude-Session: https://claude.ai/code/session_01Mco2CEWmiF95hJFVy77usj

…ommon unchanged Inject the pre-allocated operation handle through KyuubiOperation's `override protected val handle` constructor param and revert the AbstractOperation constructor in kyuubi-common to its original single-arg form. The handle stays a constructor paramaccessor so AbstractOperation's statementId reads it during superclass initialization. Other refinements: - Reduce StatementInterceptContextImpl to `override val` constructor params. - Drop the redundant defensive copies of the already read-only confOverlay and server conf maps; normalize null ipAddress/engineType with Option. - Mark statementInterceptors @volatile. - Document that engineType() returns the upper-cased kyuubi.engine.type enum names such as SPARK_SQL, and that batch and Data Agent REST paths are not intercepted.

…tion handling

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

github-actions Bot added kind:documentation Documentation is a feature! module:server module:common module:extensions labels Jun 29, 2026

wangzhigang1999 changed the title ~~[WIP][KYUUBI #7458] Add server-side statement interceptor SPI~~ [KYUUBI #7458] Add server-side statement interceptor SPI Jun 30, 2026

wangzhigang1999 marked this pull request as ready for review June 30, 2026 06:02

wangzhigang1999 marked this pull request as draft July 3, 2026 02:42

wangzhigang1999 marked this pull request as ready for review July 10, 2026 06:04

wangzhigang1999 marked this pull request as draft July 20, 2026 07:02

pan3793 force-pushed the master branch from 77f4fa7 to 14322e8 Compare July 21, 2026 05:50

github-actions Bot added the module:spark label Jul 28, 2026

wangzhigang1999 force-pushed the kyuubi-statement-interceptors branch from f6088e5 to e463209 Compare July 28, 2026 09:18

wangzhigang1999 requested a lite review from Copilot August 30, 2026 14:47

Copilot started reviewing on behalf of wangzhigang1999 August 30, 2026 14:48 View session

Copilot AI reviewed Aug 30, 2026

View reviewed changes

wangzhigang1999 added 11 commits August 31, 2026 14:54

[KYUUBI apache#7458] Add server-side statement interceptor SPI

e60aae6

[KYUUBI apache#7458] Add English doc for statement interceptor

5edd2cc

Add a user-facing English doc for the statement interceptor SPI under docs/extensions/server and register it in the server extensions toctree.

[KYUUBI apache#7458] Harden statement interceptor lifecycle and rejec…

f80b717

…tion handling

[KYUUBI apache#7458] Fix statement interceptor test JDBC prefix

ed59112

[KYUUBI apache#7458] Support per-statement config overlay

b5320f8

[KYUUBI apache#7458] Adopt config audience for statement interceptors

1d4053b

[KYUUBI apache#7458] Avoid unnecessary SQLConf clones

e779246

[KYUUBI apache#7458] Align statement interceptor docs with 1.13

5c7440f

wangzhigang1999 force-pushed the kyuubi-statement-interceptors branch from 0730404 to 5c7440f Compare August 31, 2026 07:00

wangzhigang1999 requested a lite review from Copilot August 31, 2026 07:09

Copilot started reviewing on behalf of wangzhigang1999 August 31, 2026 07:12 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Comment thread kyuubi-server/src/main/scala/org/apache/kyuubi/operation/KyuubiOperation.scala Outdated

Comment thread ...er/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/StatementInterceptResult.java

wangzhigang1999 self-assigned this Aug 31, 2026

[KYUUBI apache#7458] Avoid empty interceptor config copies

99a196a

wangzhigang1999 marked this pull request as ready for review August 31, 2026 08:40

Navigation Menu

Uh oh!

[KYUUBI #7458] Add server-side statement interceptor SPI - #7530

[KYUUBI #7458] Add server-side statement interceptor SPI#7530
wangzhigang1999 wants to merge 12 commits into
apache:masterfrom
wangzhigang1999:kyuubi-statement-interceptors

wangzhigang1999 commented Jun 29, 2026 •

edited

Loading

Uh oh!

wangzhigang1999 commented Jun 30, 2026

Uh oh!

pan3793 commented Jul 14, 2026

Uh oh!

wangzhigang1999 commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wangzhigang1999 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

wangzhigang1999 commented Jun 30, 2026

Uh oh!

pan3793 commented Jul 14, 2026

Uh oh!

wangzhigang1999 commented Jul 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wangzhigang1999 commented Jun 29, 2026 •

edited

Loading