[1.9.0 release] [PES][EC][publicservice][spark] feat: ResultSet field processing, engine switch, task diagnosis and more by aiceflower · Pull Request #5356 · apache/linkis

aiceflower · 2026-03-26T06:35:54Z

What is the purpose of the change

This PR is the comprehensive feature release for Apache Linkis 1.9.0, including the following core functionalities:

ResultSet Enhancement:
- Field truncation: Supports truncating oversized fields to prevent memory overflow
- Sensitive field masking: Supports data masking for sensitive information to enhance security
StarRocks Engine Switch:
- Implements hot engine switch mechanism for AISQL accessing StarRocks
- Supports task-level engine retry and failover
Task Diagnosis and Monitoring Enhancement:
- Spark task timeout diagnosis
- Fine-grained log return improvement
- System user login restriction for security
EngineConn Plugin Enhancement:
- HBase plugin improvement (multi-version support)
- Impala, JDBC, Trino connector optimizations
- Python, Shell engine improvements
Entrance Scheduling Optimization:
- Entrance FIFO user consumer optimization
- Parallel consumer manager improvement
- Task retry interceptor

Related issues/PRs

Related issues: close #590 close #591
Related pr: #591

Brief change log

Core Module Changes:

linkis-commons/linkis-storage: Added field truncation and sensitive field masking entities and utilities
linkis-computation-governance/linkis-entrance: Added task retry interceptor, optimized scheduling consumers
linkis-computation-governance/linkis-engineconn: Engine connection management and executor optimization
linkis-engineconn-plugins: Multi-engine plugin enhancement (HBase/StarRocks/Impala/JDBC, etc.)

Documentation:

Added 1.9.0 design docs: StarRocks engine switch, result set truncation, sensitive field masking
Added 1.9.0 requirement docs
Added AI-assisted development docs

Database Changes:

Added 1.9.0 MySQL/PostgreSQL upgrade scripts

Configuration Updates:

Updated GitHub Actions build configuration
Updated component configuration files

Checklist

I have read the Contributing Guidelines on pull requests.
I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing
list first)
If this is a code change: I have written unit tests to fully verify the new behavior.

Dev 1.12.0 webank

* code optimization * code optimization * add aisql type * code optimization

* bug fix * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

…inkis into dev-1.12.0-webank

* bug fix * bug fix * Security work order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 * revent spring-webflux --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 * revent spring-webflux * Security work order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 * revent spring-webflux * Security work order repair * add spark conf dir * bug fix * velocity exclusion --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

[Feature] Merge WeDataSphere enhancements and improvements

Co-authored-by: Casion <casionone@gmail.com>

- Change \${mockito.version} to \${mockito-bom.version} for mockito dependencies - This fixes the build error where mockito.version was undefined #AI COMMIT#

Consolidate multi-line logger.error calls into single-line format in DefaultEngineCreateService and DefaultEngineReuseService. This improves code readability while maintaining the same logging functionality. Files changed: - DefaultEngineCreateService.scala - DefaultEngineReuseService.scala

…tion (#5362) * #AI commit# feat: 集成 bes 应用服务器相关依赖 * #AI commit# feat: 集成 bes 应用服务器相关依赖 --------- Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

Co-authored-by: v-kkhuang <420895376@qq.com>

…tion (#5364) * #AI commit# feat: 集成 bes 应用服务器相关依赖 * #AI commit# feat: 集成 bes 应用服务器相关依赖 --------- Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

Co-authored-by: v-kkhuang <420895376@qq.com>

…by type and proxy user (#5370) * #AI commit# feat: 添加按类型和代理用户获取已发布数据源功能 * docs: Add PR template with Chinese version * docs: Update PR template - title must be all English * #AI commit# docs: Update PR template to English only * #AI commit# docs: Update PR template - move Checklist to the end --------- Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: kinghao <kinghao@apache.org>

Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

…ce by type and proxy user (#5376) * #AI commit# feat: 添加日志级别过滤功 - 在 openLog 接口添加 logLevel 参数支持日志级别过滤 - 实现多级别组合支持，允许使用逗号分隔的级别组合如 error,info,warn * #AI commit# feat: 提交添加日志级别过滤文档 --------- Co-authored-by: v-kkhuang <420895376@qq.com>

- 在 AMUtils 中引入复用线程池，将线程数设置为 5 来提升指标更新效率 - 修改 DefaultEngineAskEngineService 中的条件逻辑结构，优化引擎复用节点的处理流程 - 将 DefaultEngineCreateService 中的日志级别从 info 提升到 warn 并添加异常堆栈信息 - 在 DefaultEngineCreateService 和 DefaultEngineReuseService 中添加空值检查来避免 NPE - 使用 tryCatch 包装指标更新逻辑以提高系统稳定性 - 从引擎复用过滤条件中移除节点状态解锁检查以提升复用率 Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

Co-authored-by: v-kkhuang <420895376@qq.com>

* #AI commit# feat: linkis 结果集字段屏蔽和字段截取功能支持 * #AI commit# 开发阶段：优化结果集获取逻辑 --------- Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

- 在EntranceConfiguration中新增任务诊断配置项，包括启用开关、引擎类型、超时时间等 - 添加DOCTOR_REALTIME_DIAGNOSE_URL配置项和延长诊断请求超时时间至300秒 - 在AI SQL支持创建者列表中增加nodeexecution选项 Co-authored-by: v-kkhuang <420895376@qq.com>

- 引入ComputationEngineUtils工具类，提供向entrance发送消息的功能 - 添加配置项支持上下文语句和JDBC SET语句的错误索引调整 - 实现任务重试逻辑，支持JDBC和aiSQL类型的断点续跑 - 优化结果集别名数量管理，确保重试时状态正确恢复 - 新增TaskRetryInterceptor拦截器，动态为任务添加重试开关 - 增强错误处理机制，支持多种错误场景的正则匹配重试 - 修复任务重试时结果目录重复创建的问题 - 扩展ResponseTaskStatusWithExecuteCodeIndex协议，增加aliasNum字段 - 优化JDBC任务重试时的SQL索引调整逻辑，处理SET语句场景 - 增加StarRocks引擎类型支持和相关配置选项 Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

Co-authored-by: v-kkhuang <420895376@qq.com>

casionone

LGTM.

修复 import 顺序和格式问题 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

移除以下 bes 依赖: - bes-lite-spring-boot-2.x-starter - bes-gmssl - bes-jasper - bes-jdbcra - bes-websocket - bes-websocket-support - bes.version 版本属性 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- 将矩阵从 spark × hadoop 组合改为 4 个具体的 profile 场景 - default: Spark 2.4.3 + Hadoop 2.7.2 (无 profile) - spark-3: Spark 3.4.4 + Hadoop 2.7.2 - hadoop-3.3: Spark 2.4.3 + Hadoop 3.3.1 - spark-3-hadoop-3.3: Spark 3.4.4 + Hadoop 3.3.1 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

casionone

LGTM.

mayinrain and others added 30 commits March 11, 2025 10:34

fix: 调整执行阶段

adeade1

Merge pull request #769 from mayinrain/dev-1.12.0-webank

06f0d29

Dev 1.12.0 webank

code optimization (#770)

a701cfa

* code optimization * code optimization * add aisql type * code optimization

update python version

3a25381

update ldap expire time to 20m

9d7dccf

update conf

c6a252d

code optimization

ba3f275

update progress

df91ac5

update code

e8b62ce

update progress

323920c

code optimization

2c33baa

code optimization

e957b7a

Code optimization (#771)

0bc7f7b

* bug fix * bug fix --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

print dynamic param

0032dec

Merge branch 'dev-1.12.0-webank' of https://github.com/WeDataSphere/l…

e667ddd

…inkis into dev-1.12.0-webank

code review fix

ebe21ec

code optimization

d55988f

code optimization

c39bc26

code optimization

7fa2a9f

Security work order repair (#772)

012ee23

* bug fix * bug fix * Security work order repair --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

code optimization

a087ade

code optimization

315e76d

update max executors

25c8e19

Add Spark 3 default (#774)

727aab3

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

bug fix (#775)

1819803

* bug fix * bug fix * Security work order repair * Add Spark 3 default * 优化MaxMetaspaceSize * 新增spark 扩展配置，预防spark 默认配置没有用户设置的参数，导致参数无法回滚 --------- Co-authored-by: “v_kkhuang” <“420895376@qq.com”>

aiceflower and others added 23 commits March 26, 2026 09:30

Merge pull request #5345 from WeDataSphere/origin/dev-1.9.0

d8dce3f

[Feature] Merge WeDataSphere enhancements and improvements

Fix and update metrics information exception bug (#5296)

744290a

The result set supports field truncation (#5297)

2759fdc

Provide an interface for accessing available data sources (#5295)

e540d6e

Co-authored-by: Casion <casionone@gmail.com>

Merge branch 'master' into dev-1.9.0

86f88c1

Fix mockito version property reference in pom.xml

9847911

- Change \${mockito.version} to \${mockito-bom.version} for mockito dependencies - This fixes the build error where mockito.version was undefined #AI COMMIT#

#AI commit# feat: 优化bmlClient连接池参数 (#5359)

833afdc

#AI commit# feat: AISQL支持starrocks切换 (#5363)

b245569

Co-authored-by: v-kkhuang <420895376@qq.com>

#AI commit# feat: 优化bmlClient连接池参数 (#5366)

8e196ad

Co-authored-by: v-kkhuang <420895376@qq.com>

#AI commit# 开发阶段：优化jdbc堆栈日志不全，monitor 清理定时任务时间优化，manager线程池优化 (#5368)

3c0d6bf

Co-authored-by: v-kkhuang <420895376@qq.com>

#AI commit# 优化：解决引擎创建和复用时指标更新的空指针异常 (#5372)

2937222

Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

#AI commit# 优化： AI规约文档提交，Linkis日志打印优化 (#5374)

c6e38ac

Co-authored-by: v-kkhuang <420895376@qq.com> Co-authored-by: Casion <casionone@gmail.com>

#AI commit# feat: hadoop用户禁止登录管理台 (#5380)

cf2cbf3

Co-authored-by: v-kkhuang <420895376@qq.com>

#AI commit# feat: 添加批量队列资源获取功能 (#5388)

03e86df

Co-authored-by: v-kkhuang <420895376@qq.com>

casionone previously approved these changes Mar 26, 2026

View reviewed changes

aiceflower and others added 2 commits March 26, 2026 18:08

#AI COMMIT# refactor: 代码格式化调整

f7a1822

修复 import 顺序和格式问题 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

#AI COMMIT# refactor: 移除 bes 相关依赖包

a9ef58d

移除以下 bes 依赖: - bes-lite-spring-boot-2.x-starter - bes-gmssl - bes-jasper - bes-jdbcra - bes-websocket - bes-websocket-support - bes.version 版本属性 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

aiceflower dismissed casionone’s stale review via a9ef58d March 26, 2026 10:10

casionone approved these changes Mar 26, 2026

View reviewed changes

aiceflower changed the title ~~Dev 1.9.0~~ [1.9.0 release] [PES][EC][publicservice][spark] feat: ResultSet field processing, engine switch, task diagnosis and more Mar 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[1.9.0 release] [PES][EC][publicservice][spark] feat: ResultSet field processing, engine switch, task diagnosis and more#5356

[1.9.0 release] [PES][EC][publicservice][spark] feat: ResultSet field processing, engine switch, task diagnosis and more#5356
aiceflower wants to merge 1534 commits intomasterfrom
dev-1.9.0

aiceflower commented Mar 26, 2026 •

edited

Loading

Uh oh!

casionone left a comment

Uh oh!

casionone left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

aiceflower commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Related issues/PRs

Brief change log

Checklist

Uh oh!

casionone left a comment

Choose a reason for hiding this comment

Uh oh!

casionone left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

aiceflower commented Mar 26, 2026 •

edited

Loading