diff --git a/.ai/README.md b/.ai/README.md new file mode 100644 index 00000000000..2c67d9c56a1 --- /dev/null +++ b/.ai/README.md @@ -0,0 +1,188 @@ +# Linkis AI 开发文档导航 + +> **版本信息** +> - 文档版本: 1.0.0 +> - 最后更新: 2025-01-28 +> - 适用版本: Apache Linkis 1.17.0+ + +--- + +## 🚀 快速开始 + +### 新手必读（按顺序阅读） +1. **[项目核心规约](./project-context.md)** - 包含技术栈、架构设计、开发规范和模板 +2. **[强制性开发规则](./rules.md)** - 必须无条件遵守的开发规则 +3. **[模块文档](#模块文档索引)** - 根据你要开发的功能选择对应模块 + +### 常见开发场景快速跳转 +- 🆕 新增 REST 接口 → [REST接口开发模板](#rest接口开发) +- ⚙️ 添加配置项 → [配置管理规范](#配置管理) +- 🗄️ 修改数据库 → [数据库变更规范](#数据库变更) +- 🐛 异常处理 → [异常处理规范](#异常处理) +- 📝 日志记录 → [日志规范](#日志规范) + +--- + +## 📚 核心文档索引 + +### 🎯 开发规范文档 +| 文档 | 用途 | 何时查看 | +|------|------|----------| +| [project-context.md](./project-context.md) | 项目角色定位、技术栈、架构设计、开发模板 | 开始任何开发工作前必读 | +| [rules.md](./rules.md) | 强制性开发规则、需求实现步骤 | 每次开发新需求时参考 | + +### 🏗️ 模块文档索引 + +#### 微服务治理服务（基础设施层） +| 服务 | 文档 | 主要功能 | +|------|------|----------| +| Gateway | [gateway.md](./modules/microservice-governance/gateway.md) | API网关、路由转发、安全认证 | +| Eureka | [eureka.md](./modules/microservice-governance/eureka.md) | 服务注册与发现 | +| 概览 | [README.md](./modules/microservice-governance/README.md) | 微服务治理服务概述 | + +#### 计算治理服务（核心业务层） +| 服务 | 文档 | 主要功能 | +|------|------|----------| +| Entrance | [entrance.md](./modules/computation-governance/entrance.md) | 任务提交入口、调度管理 | +| JobHistory | [jobhistory.md](./modules/computation-governance/jobhistory.md) | 任务历史记录查询 | +| Manager | [manager.md](./modules/computation-governance/manager.md) | 资源管理、应用管理 | +| ECM | [ecm.md](./modules/computation-governance/ecm.md) | 引擎连接管理 | +| 概览 | [README.md](./modules/computation-governance/README.md) | 计算治理服务概述 | + +#### 公共增强服务（支撑服务层） +| 服务 | 文档 | 主要功能 | +|------|------|----------| +| PublicService | [publicservice.md](./modules/public-enhancements/publicservice.md) | 公共服务、文件管理 | +| Configuration | [configuration.md](./modules/public-enhancements/configuration.md) | 配置管理 | +| BML | [bml.md](./modules/public-enhancements/bml.md) | 大数据物料库 | +| DataSource | [datasource.md](./modules/public-enhancements/datasource.md) | 数据源管理 | +| Context | [context.md](./modules/public-enhancements/context.md) | 上下文服务 | +| Monitor | [monitor.md](./modules/public-enhancements/monitor.md) | 监控服务 | +| 概览 | [README.md](./modules/public-enhancements/README.md) | 公共增强服务概述 | + +--- + +## 🔍 按功能快速查找 + +### REST接口开发 +- **开发模板**: [project-context.md - REST接口层](./project-context.md#1-rest接口层) +- **API规范**: [project-context.md - API设计规范](./project-context.md#api设计规范) +- **参考示例**: + - Entrance接口: [entrance.md - API Interfaces](./modules/computation-governance/entrance.md#api-interfaces) + - Configuration接口: [configuration.md - API Interfaces](./modules/public-enhancements/configuration.md#api-interfaces) + +### 配置管理 +- **配置规范**: [project-context.md - 配置管理规范](./project-context.md#配置管理规范) +- **配置示例库**: [project-context.md - 常用配置示例库](./project-context.md#常用配置示例库) +- **配置模板**: [project-context.md - 配置类](./project-context.md#4-配置类) +- **参考实现**: linkis-jobhistory/conf/JobhistoryConfiguration + +### 数据库变更 +- **变更规则**: [rules.md - 数据库修改原则](./rules.md#数据库修改原则) +- **DDL脚本位置**: `linkis-dist/package/db/linkis_ddl.sql` +- **DML脚本位置**: `linkis-dist/package/db/linkis_dml.sql` +- **表结构参考**: 各模块文档的 "Database Table Structures" 章节 + +### 异常处理 +- **异常规范**: [project-context.md - 异常处理规范](./project-context.md#异常处理规范) +- **统一异常**: `org.apache.linkis.common.exception.LinkisException` +- **常见错误**: [project-context.md - 常见错误及避免方法](./project-context.md#常见错误及避免方法) + +### 日志规范 +- **日志规范**: [project-context.md - 日志规范](./project-context.md#日志规范) +- **Logger定义**: 必须使用 `LoggerFactory.getLogger(ClassName.class)` +- **日志级别**: ERROR/WARN/INFO/DEBUG 使用场景 + +--- + +## 🎨 开发模板快速复制 + +### 新增功能完整流程 +``` +1. 查看 rules.md - 需求实现步骤 +2. 创建需求文档和设计文档 +3. 使用 project-context.md 中的代码模板： + - REST接口层模板 + - 服务层模板 + - 数据访问层模板 + - 配置类模板 +4. 添加功能开关（默认false） +5. 记录数据库变更 +6. 编写测试和文档 +``` + +### REST接口模板快速链接 +👉 [project-context.md - 新功能开发模板](./project-context.md#新功能开发模板) + +### 配置类模板快速链接 +👉 [project-context.md - 配置类](./project-context.md#4-配置类) + +--- + +## ⚠️ 重要提醒 + +### 🚫 禁止操作（来自 rules.md） +- **数据库结构**: 除非明确指定，严禁修改现有表结构 +- **第三方依赖**: 不允许引入新的第三方依赖库 +- **核心接口**: 不得修改现有公共接口的签名 + +### ✅ 必须遵守 +- **最小改动原则**: 所有功能实现必须遵循最小改动原则 +- **功能可配置**: 所有功能必须增加功能开关，默认关闭 +- **向后兼容**: 新增功能必须考虑向后兼容性 + +--- + +## 💡 开发技巧 + +### 编程语言选择 +- **Java**: REST API、Service层、Entity类、配置类 +- **Scala**: 计算逻辑、RPC通信、复杂业务处理、配置对象 + +### 字符编码 +统一使用 `StandardCharsets.UTF_8`，禁止使用字符串 `"UTF-8"` + +### 统一返回体 +所有REST接口返回 `org.apache.linkis.server.Message` + +--- + +## 📖 如何使用这些文档 + +### 场景1: 我要在 Entrance 服务中新增一个接口 +1. 阅读 [entrance.md](./modules/computation-governance/entrance.md) 了解服务结构 +2. 查看 [project-context.md - REST接口层模板](./project-context.md#1-rest接口层) +3. 参考 entrance.md 中现有接口实现 +4. 遵循 [rules.md](./rules.md) 中的开发规则 +5. 添加功能开关配置 + +### 场景2: 我需要添加一个新的配置项 +1. 查看 [project-context.md - 配置管理规范](./project-context.md#配置管理规范) +2. 参考 [project-context.md - 配置类模板](./project-context.md#4-配置类) +3. 查看 `JobhistoryConfiguration` 实现示例 +4. 在当前模块的 conf 目录下的 Configuration 类中添加 + +### 场景3: 我需要修改数据库表 +1. 查看 [rules.md - 数据库修改原则](./rules.md#数据库修改原则) +2. 确认是否能通过新增字段实现（优先选择） +3. 将变更记录到 `linkis-dist/package/db/linkis_ddl.sql` +4. 如有初始化数据，记录到 `linkis-dist/package/db/linkis_dml.sql` + +--- + +## 🔄 文档更新记录 + +| 版本 | 日期 | 更新内容 | 更新人 | +|------|------|----------|--------| +| 1.0.0 | 2025-01-28 | 创建导航文档，优化文档结构 | AI | + +--- + +## 📞 帮助与反馈 + +如果文档中有不清楚的地方，请： +1. 先查看对应模块的详细文档 +2. 查看 project-context.md 中的开发模板和示例 +3. 参考现有代码实现 + +**记住**: 遵循规范比快速开发更重要！ diff --git a/.ai/modules/computation-governance/README.md b/.ai/modules/computation-governance/README.md new file mode 100644 index 00000000000..1f39355e663 --- /dev/null +++ b/.ai/modules/computation-governance/README.md @@ -0,0 +1,157 @@ +# Computation Governance Services + +The computation governance services handle the core computation task lifecycle management in Linkis. + +## Service Modules + +- [Entrance Service](./entrance.md) - Task submission and entrance point +- [Manager Service](./manager.md) - Resource and application management +- [ECM Service](./ecm.md) - Engine Connection Manager +- [JobHistory Service](./jobhistory.md) - Task execution history tracking + +## Overview + +These services form the core of Linkis' computation governance capabilities, managing the complete lifecycle of computation tasks from submission to execution and monitoring. + +## Common Features + +### Task Lifecycle Management +- Task submission and validation +- Task scheduling and resource allocation +- Task execution monitoring +- Task result management +- Task error handling and recovery + +### Engine Management +- Dynamic engine connection creation +- Engine lifecycle management +- Engine resource monitoring +- Engine scaling capabilities + +### Resource Governance +- Multi-tenant resource isolation +- Load balancing across engines +- Resource usage tracking +- Quota management + +## API Interface Summary + +### Entrance Service APIs +- Task submission: `POST /api/entrance/submit` +- Task status query: `GET /api/entrance/{id}/status` +- Task progress: `GET /api/entrance/{id}/progress` +- Task log retrieval: `GET /api/entrance/{id}/log` +- Task cancellation: `GET /api/entrance/{id}/kill` + +### Manager Service APIs +- Engine instance management +- Resource allocation and monitoring +- Node status querying +- Engine creation requests + +### ECM Service APIs +- Engine connection management +- Engine lifecycle operations +- Resource reporting +- Engine metrics collection + +### JobHistory Service APIs +- Job history querying +- Job detail retrieval +- Job statistics reporting + +## Database Schema Summary + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +### Common Lock Table +```sql +CREATE TABLE `linkis_ps_common_lock` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `lock_object` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `locker` VARCHAR(255) CHARSET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'locker', + `time_out` longtext COLLATE utf8_bin, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_lock_object` (`lock_object`) +); +``` + +## RPC Methods Summary + +### Entrance Service RPCs +- `submitTask(TaskRequest request)` +- `getTaskStatus(String taskId)` +- `cancelTask(String taskId)` +- `getTaskResult(String taskId)` + +### Manager Service RPCs +- `requestEngine(EngineRequest request)` +- `releaseEngine(String engineId)` +- `getEngineStatus(String engineId)` +- `getNodeMetrics(String nodeId)` + +### ECM Service RPCs +- `createEngineConnection(EngineCreateRequest request)` +- `terminateEngineConnection(String engineId)` +- `reportEngineResourceUsage(String engineId, ResourceUsage usage)` +- `getEngineMetrics(String engineId)` + +### JobHistory Service RPCs +- `saveJobHistory(JobHistory history)` +- `queryJobHistory(JobHistoryQuery query)` +- `getJobDetails(Long jobId)` +- `updateJobStatus(Long jobId, String status)` + +## Dependencies + +- linkis-commons - Shared utilities +- linkis-protocol - Communication protocols +- linkis-rpc - Remote procedure calls +- Various engine connection plugins +- Spring Cloud ecosystem \ No newline at end of file diff --git a/.ai/modules/computation-governance/ecm.md b/.ai/modules/computation-governance/ecm.md new file mode 100644 index 00000000000..134c9817838 --- /dev/null +++ b/.ai/modules/computation-governance/ecm.md @@ -0,0 +1,635 @@ +# ECM Service + +The ECM (Engine Connection Manager) service manages the lifecycle of engine connections in the Linkis system. + +## Overview + +This service is responsible for managing the lifecycle of engine connections, including creating, starting, stopping, and monitoring engine instances. + +## Key Components + +### Core Classes +- `LinkisECMApplication` - Main application class +- Engine connection lifecycle management +- Engine resource monitoring +- Engine health checking + +### Features +- Engine connection creation and initialization +- Engine lifecycle management +- Resource allocation for engines +- Engine monitoring and health checking +- Engine termination and cleanup + +## API Interfaces + +### Download Engine Log +``` +GET /api/rest_j/v1/engineconnManager/downloadEngineLog +``` + +Parameters: +- `emInstance`: ECM instance (required) +- `instance`: Engine instance (required) +- `logDirSuffix`: Log directory suffix (required) +- `logType`: Log type (required) - stdout, stderr, gc, or yarnApp + +Response: +``` +Binary file download (log file content) +``` + +Error Codes: +- 11110: Log directory {0} does not exists.(日志目录 {0} 不存在.) +- 911115: failed to downLoad(下载失败) +- 911116: Download file has exceeded 100MB(下载文件已超过100M) +- 911117: Parameter {0} cannot be empty (参数 {0} 不能为空) +- 911118: logType only supports stdout, stderr, gc, yarnApp(logType仅支持stdout,stderr,gc,yarnApp) +- 911119: You {0} have no permission to download Log in ECM {1}(用户 {0} 无权限下载 ECM {1} 日志) + +Notes: +- Only supports GET method due to gateway forwarding rules +- File size limit is 100MB +- Supported log types: stdout, stderr, gc, yarnApp +- Requires user authentication and authorization checks +- Filename format in response: {instance}_{logType}.txt + +### List All ECMs +``` +GET /api/rest_j/v1/linkisManager/listAllEMs +``` + +Parameters: +- `instance`: ECM instance filter (optional) +- `nodeHealthy`: Node healthy status filter (optional) +- `owner`: Owner filter (optional) +- `tenantLabel`: Tenant label filter (optional) + +Response: +```json +{ + "method": "/api/linkisManager/listAllEMs", + "status": 0, + "message": "OK", + "data": { + "EMs": [ + { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "gz.bdz.bdplxxxxx.apache:9102", + "nodeHealthy": "Healthy", + "labels": [ + { + "stringValue": "gz.bdz.bdplxxxxx.apache:9102", + "labelKey": "emInstance" + } + ], + "owner": "hadoop", + "nodeStatus": "Healthy" + } + ] + } +} +``` + +Error Codes: +- 210003: Only admin can modify ECMs(只有管理员才能修改ECM) + +Notes: +- Requires admin privileges +- Returns list of all ECM instances with their status and labels + +### List All ECM Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllECMHealthyStatus +``` + +Parameters: +- `onlyEditable`: Boolean flag to return only editable statuses (optional) + +Response: +```json +{ + "method": "/api/linkisManager/listAllECMHealthyStatus", + "status": 0, + "message": "OK", + "data": { + "nodeHealthy": [ + "Healthy", + "UnHealthy", + "WARN", + "StockAvailable", + "StockUnavailable" + ] + } +} +``` + +Notes: +- Returns all possible ECM healthy status values +- When `onlyEditable` is true, returns only the statuses that can be modified + +### Modify ECM Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEMInfo +``` + +Parameters: +- `applicationName`: Application name (optional) +- `emStatus`: ECM status (optional) +- `instance`: ECM instance (required) +- `labels`: Labels list (optional) +- `labelKey`: Label key (optional) +- `description`: Description (optional) +- `stringValue`: String value (optional) + +Response: +```json +{ + "method": "/api/linkisManager/modifyEMInfo", + "status": 0, + "message": "success" +} +``` + +Error Codes: +- 210003: Failed to update label, include repeat labels(更新label失败，包含重复label) + +Notes: +- Allows modification of ECM instance information +- Supports updating labels, status, and description + +### Execute ECM Operation +``` +POST /api/rest_j/v1/linkisManager/executeECMOperation +``` + +Request Body: +```json +{ + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "gz.bdz.bdplxxxxx.apache:9102" + }, + "parameters": { + // Operation specific parameters + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperation", + "status": 0, + "message": "OK", + "data": { + // Operation result data + } +} +``` + +Error Codes: +- Various operation-specific error codes + +Notes: +- Executes administrative operations on ECM instances +- Requires appropriate permissions +- Operation parameters vary based on the specific operation being performed + +### Execute ECM Operation by Engine Connection +``` +POST /api/rest_j/v1/linkisManager/executeECMOperationByEC +``` + +Request Body: +```json +{ + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "gz.bdz.bdplxxxxx.apache:12295" + }, + "parameters": { + // Operation specific parameters + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperationByEC", + "status": 0, + "message": "OK", + "data": { + // Operation result data + } +} +``` + +Error Codes: +- Permission-related errors when user doesn't own the engine connection + +Notes: +- Executes ECM operations triggered by engine connections +- Validates that the user owns the engine connection or is an admin +- Operation parameters vary based on the specific operation being performed + +### Reset Resource +``` +GET /api/rest_j/v1/linkisManager/reset-resource +``` + +Parameters: +- `serviceInstance`: ECM service instance (optional) +- `username`: Username (optional) + +Response: +```json +{ + "method": "/api/linkisManager/reset-resource", + "status": 0, + "message": "OK", + "data": {} +} +``` + +Error Codes: +- Permission error when user is not admin + +Notes: +- Resets resource allocation for ECM instances or users +- Requires admin privileges +- Can reset resources for a specific ECM instance or user + +### Open Engine Log +``` +POST /api/rest_j/v1/linkisManager/openEngineLog +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "emInstance": "bdp110:9100", + "instance": "bdp110:21976", + "parameters": { + "logType": "stdout", + "fromLine": "0", + "pageSize": "1000" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/openEngineLog", + "status": 0, + "message": "OK", + "data": { + // Log content or operation result + } +} +``` + +Error Codes: +- Parameter validation errors +- Permission errors + +Notes: +- Opens and retrieves engine log content +- Supports different log types (stdout, stderr, gc, udfLog, yarnApp) +- Requires appropriate permissions + +### Task Prediction +``` +GET /api/rest_j/v1/linkisManager/task-prediction +``` + +Parameters: +- `username`: Username (optional) +- `engineType`: Engine type (required) +- `creator`: Creator (required) +- `clustername`: Cluster name (optional) +- `queueName`: Queue name (optional) +- `tenant`: Tenant (optional) + +Response: +```json +{ + "method": "/api/linkisManager/task-prediction", + "status": 0, + "message": "OK", + "data": { + "tenant": "tenant", + "userResource": {}, + "ecmResource": {}, + "yarnResource": {}, + "checkResult": true + } +} +``` + +Error Codes: +- Parameter validation errors + +Notes: +- Predicts if a task can be executed based on available resources +- Requires engineType and creator parameters +- Returns resource availability information + +### Get Engine Connection Info +``` +GET /api/rest_j/v1/linkisManager/ecinfo/get +``` + +Parameters: +- `ticketid`: Ticket ID (required) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/get", + "status": 0, + "message": "OK", + "data": { + "ecResourceInfoRecord": { + // Engine connection resource information + } + } +} +``` + +Error Codes: +- Ticket ID not found + +Notes: +- Retrieves engine connection information by ticket ID +- Requires user to be owner or admin + +### Delete Engine Connection Info +``` +DELETE /api/rest_j/v1/linkisManager/ecinfo/delete/{ticketid} +``` + +Parameters: +- `ticketid`: Ticket ID (required) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/delete/{ticketid}", + "status": 0, + "message": "OK", + "data": { + "ecResourceInfoRecord": { + // Deleted engine connection resource information + } + } +} +``` + +Error Codes: +- Ticket ID not found +- Permission errors + +Notes: +- Deletes engine connection information by ticket ID +- Requires user to be owner or admin + +### Query Engine Connection Resource History List +``` +GET /api/rest_j/v1/linkisManager/ecinfo/ecrHistoryList +``` + +Parameters: +- `instance`: Instance (optional) +- `creator`: Creator (optional) +- `startDate`: Start date (optional) +- `endDate`: End date (optional) +- `engineType`: Engine type (optional) +- `status`: Status (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecrHistoryList", + "status": 0, + "message": "OK", + "data": { + "engineList": [ + // Engine connection resource history records + ], + "totalPage": 100 + } +} +``` + +Error Codes: +- Parameter validation errors + +Notes: +- Queries engine connection resource history +- Supports filtering by various parameters +- Returns paginated results + +### Query Engine Connection List +``` +POST /api/rest_j/v1/linkisManager/ecinfo/ecList +``` + +Request Body: +```json +{ + "creators": ["IDE"], + "engineTypes": ["spark"], + "statuss": ["Running"], + "queueName": "default", + "ecInstances": ["instance1", "instance2"], + "crossCluster": false +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecList", + "status": 0, + "message": "OK", + "data": { + "ecList": [ + // Engine connection records + ] + } +} +``` + +Error Codes: +- Parameter validation errors +- Permission errors + +Notes: +- Queries engine connection list +- Requires admin privileges +- Supports filtering by various parameters + +## Database Table Structures + +The ECM service uses the following database tables for engine management: + +### Engine Connection Plugin BML Resources Table +```sql +CREATE TABLE `linkis_cg_engine_conn_plugin_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `engine_conn_type` varchar(100) NOT NULL COMMENT 'Engine type', + `version` varchar(100) COMMENT 'version', + `file_name` varchar(255) COMMENT 'file name', + `file_size` bigint(20) DEFAULT 0 NOT NULL COMMENT 'file size', + `last_modified` bigint(20) COMMENT 'File update time', + `bml_resource_id` varchar(100) NOT NULL COMMENT 'Owning system', + `bml_resource_version` varchar(200) NOT NULL COMMENT 'Resource owner', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'created time', + `last_update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'updated time', + PRIMARY KEY (`id`) +); +``` + +### Manager Engine EM Table +```sql +CREATE TABLE `linkis_cg_manager_engine_em` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `engine_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `em_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +); +``` + +### EC Resource Info Record Table +```sql +CREATE TABLE `linkis_cg_ec_resource_info_record` ( + `id` INT(20) NOT NULL AUTO_INCREMENT, + `label_value` VARCHAR(255) NOT NULL COMMENT 'ec labels stringValue', + `create_user` VARCHAR(128) NOT NULL COMMENT 'ec create user', + `service_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ec instance info', + `ecm_instance` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'ecm instance info ', + `ticket_id` VARCHAR(100) NOT NULL COMMENT 'ec ticket id', + `status` varchar(50) DEFAULT NULL COMMENT 'EC status: Starting,Unlock,Locked,Idle,Busy,Running,ShuttingDown,Failed,Success', + `log_dir_suffix` varchar(128) COLLATE utf8_bin DEFAULT NULL COMMENT 'log path', + `request_times` INT(8) COMMENT 'resource request times', + `request_resource` VARCHAR(1020) COMMENT 'request resource', + `used_times` INT(8) COMMENT 'resource used times', + `used_resource` VARCHAR(1020) COMMENT 'used resource', + `metrics` TEXT DEFAULT NULL COMMENT 'ec metrics', + `release_times` INT(8) COMMENT 'resource released times', + `released_resource` VARCHAR(1020) COMMENT 'released resource', + `release_time` datetime DEFAULT NULL COMMENT 'released time', + `used_time` datetime DEFAULT NULL COMMENT 'used time', + `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + PRIMARY KEY (`id`), + KEY `idx_ticket_id` (`ticket_id`), + UNIQUE KEY `uniq_tid_lv` (`ticket_id`,`label_value`), + UNIQUE KEY `uniq_sinstance_status_cuser_ctime` (`service_instance`, `status`, `create_user`, `create_time`) +); +``` + +## RPC Methods + +The ECM service provides several RPC methods for engine management: + +### Engine Management RPCs + +#### createEngineConnection +Creates a new engine connection: +```java +EngineConnection createEngineConnection(EngineCreateRequest request) +``` + +#### executeCode +Executes code on an engine: +```java +ExecutionResult executeCode(String engineId, String code, String runType) +``` + +#### getEngineStatus +Retrieves the status of an engine: +```java +EngineStatus getEngineStatus(String engineId) +``` + +#### terminateEngine +Terminates an engine connection: +```java +void terminateEngine(String engineId) +``` + +#### listEngines +Lists all engine connections: +```java +List listEngines() +``` + +#### getEngineMetrics +Retrieves metrics for an engine: +```java +EngineMetrics getEngineMetrics(String engineId) +``` + +### Resource Management RPCs + +#### getResourceUsage +Retrieves resource usage for an engine: +```java +ResourceUsage getResourceUsage(String engineId) +``` + +#### updateResource +Updates resource allocation for an engine: +```java +void updateResource(String engineId, ResourceRequest request) +``` + +#### reportResourceUsage +Reports resource usage from an engine: +```java +void reportResourceUsage(String engineId, ResourceUsage usage) +``` + +### Engine Communication RPCs + +#### sendEngineCommand +Sends a command to an engine: +```java +CommandResponse sendEngineCommand(String engineId, EngineCommand command) +``` + +#### getEngineLogs +Retrieves logs from an engine: +```java +EngineLogs getEngineLogs(String engineId, int fromLine, int lines) +``` + +## Dependencies + +- linkis-engineconn-manager-core +- linkis-engineconn-plugin-core +- linkis-rpc +- linkis-protocol +- linkis-manager-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ECMRestfulApi: `linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-server/src/main/java/org/apache/linkis/ecm/restful/ECMRestfulApi.java` +- EMRestfulApi: `linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/am/restful/EMRestfulApi.java` +- ECResourceInfoRestfulApi: `linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/am/restful/ECResourceInfoRestfulApi.java` + +### MyBatis XML Files +The ECM service primarily uses the Manager service's persistence layer, which includes: +- LabelManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/LabelManagerMapper.xml` +- ResourceManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/ResourceManagerMapper.xml` +- NodeManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml` +- NodeMetricManagerMapper: `linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeMetricManagerMapper.xml` \ No newline at end of file diff --git a/.ai/modules/computation-governance/entrance.md b/.ai/modules/computation-governance/entrance.md new file mode 100644 index 00000000000..6071b33f9a3 --- /dev/null +++ b/.ai/modules/computation-governance/entrance.md @@ -0,0 +1,726 @@ +# Entrance Service + +The Entrance service serves as the entry point for computation task submissions in the Linkis system. + +## Overview + +This service is responsible for receiving user computation requests, parsing them, validating them, and coordinating their execution through the appropriate engine connections. It acts as the primary interface between users and the computation execution layer. + +## Key Components + +### Core Classes +- `LinkisEntranceApplication` - Main application class +- Task submission handling +- Task parsing and validation +- Task scheduling coordination +- Task execution monitoring +- Task result management + +### Features +- Task submission and management +- Code parsing and validation +- Engine routing and allocation +- Result set management +- Log retrieval and management + +## API Interfaces + +### Task Execution +``` +POST /api/entrance/execute +``` + +Parameters (in request body): +- `executionContent`: Contains the code to execute and run type + - `code`: The actual code to execute + - `runType`: Type of execution (sql, python, scala, etc.) +- `params`: Parameters for execution + - `variable`: Variables for the execution + - `configuration`: Configuration parameters (runtime, special) +- `source`: Source information + - `scriptPath`: Path to the script file +- `labels`: Labels for engine selection + - `engineType`: Type and version of engine (spark-2.4.3, hive-2.1.1, etc.) + - `userCreator`: User and creator information + +Response: +```json +{ + "method": "/api/entrance/execute", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If parsing or execution fails, the error will be stored in the job request and returned in the response +- Permission errors if user is not authorized to execute + +Notes: +- Returns both taskID (database ID) and execID (execution ID) +- The execID is used for subsequent operations on the task +- User authentication is required + +### Task Submission +``` +POST /api/entrance/submit +``` + +Parameters (in request body): +- Same as execute API + +Response: +```json +{ + "method": "/api/entrance/submit", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If parsing or execution fails, the error will be stored in the job request and returned in the response +- Permission errors if user is not authorized to submit + +Notes: +- Functionally similar to execute but with different endpoint +- Returns both taskID (database ID) and execID (execution ID) +- User authentication is required + +### Task Status Query +``` +GET /api/entrance/{id}/status +``` + +Parameters: +- `id`: The execution ID or task ID +- `taskID` (optional): The ID of the task to query + +Response: +```json +{ + "method": "/api/entrance/{id}/status", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "status": "Running", + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If there's an exception during status retrieval, error is returned + +Notes: +- Supports both execID and taskID as the path parameter +- Status values include: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout +- For completed jobs, status is retrieved from job history + +### Task Progress +``` +GET /api/entrance/{id}/progress +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/progress", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "progress": "0.75", + "execID": "exec-id-12345", + "progressInfo": [ + { + "id": "stage1", + "succeedTasks": 5, + "failedTasks": 0, + "runningTasks": 2, + "totalTasks": 10 + } + ] + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If progress information is not yet available, error is returned + +Notes: +- Progress is a value between 0 and 1 +- ProgressInfo provides detailed information about execution stages +- For completed jobs, returns 1.0 progress + +### Task Progress with Resource Info +``` +GET /api/entrance/{id}/progressWithResource +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/progressWithResource", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "progress": "0.75", + "execID": "exec-id-12345", + "progressInfo": [ + { + "id": "stage1", + "succeedTasks": 5, + "failedTasks": 0, + "runningTasks": 2, + "totalTasks": 10 + } + ], + "jobYarnMetrics": { + "jobYarnResource": [ + { + "applicationId": "application_1234567890123_0001", + "queueCores": 2, + "queueMemory": 4096, + "usedCores": 1, + "usedMemory": 2048, + "resourceType": "YARN" + } + ] + } + } +} +``` + +Error Cases: +- If job cannot be found, appropriate error message is returned +- If progress information is not yet available, error is returned + +Notes: +- Includes YARN resource metrics in addition to progress information +- Provides detailed resource usage information for YARN-based engines + +### Task Log Retrieval +``` +GET /api/entrance/{id}/log +``` + +Parameters: +- `id`: The execution ID +- `fromLine` (optional): Starting line number (default: 0) +- `size` (optional): Number of lines to retrieve (default: 100) +- `distinctLevel` (optional): Whether to separate logs by level (default: true) + +Response: +```json +{ + "method": "/api/entrance/{id}/log", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "log": ["log line 1", "log line 2", "log line 3"], + "fromLine": 1, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job has completed, suggests downloading log file instead +- If log cannot be retrieved, returns appropriate error + +Notes: +- For distinctLevel=true, returns array with 4 elements (different log levels) +- For distinctLevel=false, returns concatenated string of logs +- Size parameter has a maximum limit (10000) + +### Task Cancellation +``` +GET /api/entrance/{id}/kill +``` + +Parameters: +- `id`: The execution ID +- `taskID` (optional): The ID of the task to cancel + +Response: +```json +{ + "method": "/api/entrance/{id}/kill", + "status": 0, + "message": "success", + "data": { + "taskID": 12345, + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job is already completed, returns error that kill is not supported +- If user doesn't have permission to kill the job, returns permission error +- If exception occurs during kill, returns error with exception details + +Notes: +- Updates job status to Cancelled in database +- For jobs not found in memory, performs force kill using job history + +### Batch Task Cancellation +``` +POST /api/entrance/{id}/killJobs +``` + +Request Body: +```json +{ + "idList": ["exec-id-1", "exec-id-2"], + "taskIDList": [12345, 12346] +} +``` + +Parameters: +- `id`: The strong execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/killJobs", + "status": 0, + "message": "success", + "data": { + "messages": [ + { + "method": "/api/entrance/exec-id-1/kill", + "status": 0, + "message": "Successfully killed the job(成功kill了job)" + }, + { + "method": "/api/entrance/exec-id-2/kill", + "status": 0, + "message": "Successfully killed the job(成功kill了job)" + } + ] + } +} +``` + +Error Cases: +- If idList and taskIDList have different lengths, returns error +- If parameters are not arrays, returns error +- Individual job kill errors are returned in the messages array + +Notes: +- Processes each job in the lists and returns individual results +- For jobs not found in memory, performs force kill using job history + +### Task Pause +``` +GET /api/entrance/{id}/pause +``` + +Parameters: +- `id`: The execution ID + +Response: +```json +{ + "method": "/api/entrance/{id}/pause", + "status": 0, + "message": "success to pause job (成功pause了job)", + "data": { + "execID": "exec-id-12345" + } +} +``` + +Error Cases: +- If job cannot be found, returns appropriate error +- If exception occurs during pause, returns error + +Notes: +- Pause functionality implementation may be incomplete (TODO in code) + +### Update Route Label +``` +POST /api/entrance/operation/label/update +``` + +Request Body: +```json +{ + "routeLabel": "new-route-label" +} +``` + +Parameters: +- Requires admin privileges + +Response: +```json +{ + "method": "/api/entrance/operation/label/update", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Updates the route label for the entrance instance +- Used for routing purposes in distributed environments + +### Mark Offline +``` +GET /api/entrance/operation/label/markoffline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/markoffline", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Marks the entrance instance as offline +- Updates all non-execution task instances + +### Back Online +``` +GET /api/entrance/operation/label/backonline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/backonline", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Removes the offline label from the entrance instance + +### Check Online Status +``` +GET /api/entrance/operation/label/isOnline +``` + +Response: +```json +{ + "method": "/api/entrance/operation/label/isOnline", + "status": 0, + "message": "success", + "data": { + "isOnline": true + } +} +``` + +Notes: +- Checks if the entrance instance is currently online + +### Get Task Info +``` +GET /api/entrance/operation/metrics/taskinfo +``` + +Parameters: +- `user` (optional): Filter by user +- `creator` (optional): Filter by creator +- `ecType` (optional): Filter by engine type + +Response: +```json +{ + "method": "/api/entrance/operation/metrics/taskinfo", + "status": 0, + "message": "success", + "data": { + "taskNumber": 5, + "runningNumber": 2, + "queuedNumber": 3 + } +} +``` + +Error Cases: +- Non-admin users cannot view other users' task information + +Notes: +- For admin users, can view any user's task information +- For non-admin users, can only view their own task information +- Returns counts of total, running, and queued tasks + +### Get Running Task Count +``` +GET /api/entrance/operation/metrics/runningtask +``` + +Response: +```json +{ + "method": "/api/entrance/operation/metrics/runningtask", + "status": 0, + "message": "success", + "data": { + "runningTaskNumber": 5, + "isCompleted": false + } +} +``` + +Notes: +- Returns the number of currently running tasks +- isCompleted indicates if there are no running tasks + +### Kill Consumer +``` +GET /api/entrance/operation/consumer/kill +``` + +Parameters: +- `groupName`: Name of the consumer group to kill + +Response: +```json +{ + "method": "/api/entrance/operation/consumer/kill", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Destroys the specified consumer group +- Requires admin privileges + +### Get Consumer Info +``` +GET /api/entrance/operation/consumer/info +``` + +Response: +```json +{ + "method": "/api/entrance/operation/consumer/info", + "status": 0, + "message": "success", + "data": { + "consumerNum": 3 + } +} +``` + +Error Cases: +- If user is not admin, returns permission error + +Notes: +- Returns the number of consumer groups +- Requires admin privileges + +## Database Table Structures + +The Entrance service uses the following database tables from the job history system: + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The Entrance service provides several RPC methods for inter-service communication: + +### Task Management RPCs + +#### submitTask +Submits a task for execution: +```java +JobRespProtocol submitTask(JobReqInsert request) +``` + +#### updateTask +Updates a task: +```java +JobRespProtocol updateTask(JobReqUpdate request) +``` + +#### batchUpdateTasks +Batch updates tasks: +```java +JobRespProtocol batchUpdateTasks(JobReqBatchUpdate request) +``` + +#### queryTask +Queries a task: +```java +JobRespProtocol queryTask(JobReqQuery request) +``` + +#### readAllTasks +Reads all tasks: +```java +JobRespProtocol readAllTasks(JobReqReadAll request) +``` + +#### getTaskStatus +Retrieves the status of a task: +```java +String getTaskStatus(String taskId) +``` + +#### cancelTask +Cancels a running task: +```java +void cancelTask(String taskId) +``` + +#### getTaskResult +Retrieves the result of a completed task: +```java +TaskResult getTaskResult(String taskId) +``` + +#### getTaskProgress +Retrieves the progress of a task: +```java +TaskProgress getTaskProgress(String taskId) +``` + +### Engine Management RPCs + +#### requestEngine +Requests an engine for task execution: +```java +EngineConnection requestEngine(EngineRequest request) +``` + +#### releaseEngine +Releases an engine after task completion: +```java +void releaseEngine(String engineId) +``` + +#### getEngineStatus +Retrieves the status of an engine: +```java +EngineStatus getEngineStatus(String engineId) +``` + +### Log Management RPCs + +#### getTaskLog +Retrieves logs for a specific task: +```java +TaskLog getTaskLog(String taskId, int fromLine, int pageSize) +``` + +#### appendTaskLog +Appends log entries for a task: +```java +void appendTaskLog(String taskId, List logLines) +``` + +## Dependencies + +- linkis-scheduler +- linkis-protocol +- linkis-rpc +- linkis-storage +- linkis-computation-governance-common +- linkis-computation-orchestrator +- linkis-pes-client +- linkis-io-file-client +- linkis-pes-rpc-client +- linkis-ps-common-lock + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- EntranceRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java` +- EntranceLabelRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java` +- EntranceMetricRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java` +- EntranceConsumerRestfulApi: `linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceConsumerRestfulApi.java` + +### MyBatis XML Files +The Entrance service uses the JobHistory service's persistence layer, which includes: +- JobHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/mysql/JobHistoryMapper.xml` +- JobDetailMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDetailMapper.xml` +- JobStatisticsMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobStatisticsMapper.xml` +- JobDiagnosisMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml` \ No newline at end of file diff --git a/.ai/modules/computation-governance/jobhistory.md b/.ai/modules/computation-governance/jobhistory.md new file mode 100644 index 00000000000..7e875ce2f6f --- /dev/null +++ b/.ai/modules/computation-governance/jobhistory.md @@ -0,0 +1,481 @@ +# JobHistory Service + +The JobHistory service tracks and manages the execution history of tasks in the Linkis system. + +## Overview + +This service provides task execution history tracking, including task status, execution time, results, and error information. + +## Key Components + +### Core Classes +- `LinkisJobHistoryApp` - Main application class +- Task history storage and retrieval +- Task statistics and analytics +- Task search and filtering + +### Features +- Task execution history tracking +- Task result storage and retrieval +- Task performance metrics +- Task search and filtering capabilities +- Task statistics and reporting + +## API Interfaces + +### Get Task By ID +``` +GET /api/rest_j/v1/jobhistory/{id}/get +``` + +Parameters: +- `id`: Job ID (required) +- `brief`: Whether to return brief info only (optional) + +Response: +```json +{ + "method": "/api/jobhistory/{id}/get", + "status": 0, + "message": "success", + "data": { + "task": { + "jobId": "12345", + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "executionCode": "SELECT * FROM table", + "resultLocation": "/path/to/result", + "errorCode": null, + "errorDesc": null, + "progress": "1.0", + "costTime": 300000 + } + } +} +``` + +### List Job History +``` +GET /api/rest_j/v1/jobhistory/list +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `status`: Task status to filter by (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) +- `taskID`: Task ID to filter by (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) +- `isAdminView`: Whether to view as admin (optional) +- `isDeptView`: Whether to view as department admin (optional) +- `instance`: Instance to filter by (optional) +- `engineInstance`: Engine instance to filter by (optional) +- `runType`: Run type to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/list", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "executionCode": "SELECT * FROM table", + "resultLocation": "/path/to/result", + "errorCode": null, + "errorDesc": null, + "progress": "1.0", + "costTime": 300000 + } + ], + "totalPage": 1 + } +} +``` + +### List Undone Tasks +``` +GET /api/rest_j/v1/jobhistory/listundonetasks +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `status`: Task status to filter by (optional, default: "Running,Inited,Scheduled") +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) +- `startTaskID`: Start task ID (optional) +- `engineType`: Engine type to filter by (optional) +- `creator`: Creator to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/listundonetasks", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Running", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00" + } + ], + "totalPage": 1 + } +} +``` + +### List By Task IDs +``` +GET /api/rest_j/v1/jobhistory/list-taskids +``` + +Parameters: +- `taskID`: Comma-separated list of task IDs (required) + +Response: +```json +{ + "method": "/api/jobhistory/list-taskids", + "status": 0, + "message": "success", + "data": { + "jobHistoryList": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00" + } + ] + } +} +``` + +### Job Extra Info +``` +GET /api/rest_j/v1/jobhistory/job-extra-info +``` + +Parameters: +- `jobId`: Job ID (required) + +Response: +```json +{ + "method": "/api/jobhistory/job-extra-info", + "status": 0, + "message": "success", + "data": { + "metricsMap": { + "executionCode": "SELECT * FROM table", + "runtime": "300000" + } + } +} +``` + +### List Duration Top +``` +GET /api/rest_j/v1/jobhistory/listDurationTop +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) +- `pageNow`: Page number (optional, default: 1) +- `pageSize`: Page size (optional, default: 20) + +Response: +```json +{ + "method": "/api/jobhistory/listDurationTop", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "jobReqId": "job-12345", + "submitUser": "testuser", + "executeUser": "testuser", + "status": "Succeed", + "engineType": "spark", + "createdTime": "2023-01-01 12:00:00", + "updatedTime": "2023-01-01 12:05:00", + "costTime": 300000 + } + ] + } +} +``` + +### Task Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/taskCount +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/taskCount", + "status": 0, + "message": "success", + "data": { + "sumCount": 100, + "succeedCount": 95, + "failedCount": 5, + "cancelledCount": 0 + } +} +``` + +### Engine Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/engineCount +``` + +Parameters: +- `startDate`: Start date for filtering (optional) +- `endDate`: End date for filtering (optional) +- `executeApplicationName`: Application name to filter by (optional) +- `creator`: Creator to filter by (optional) +- `proxyUser`: Proxy user to filter by (optional) + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/engineCount", + "status": 0, + "message": "success", + "data": { + "countEngine": 100, + "countEngineSucceed": 95, + "countEngineFailed": 5, + "countEngineShutting": 0 + } +} +``` + +### Add Observe Info +``` +POST /api/rest_j/v1/jobhistory/setting/addObserveInfo +``` + +Request Body: +```json +{ + "taskId": 12345, + "receiver": "testuser", + "extra": { + "title": "Task Alert", + "detail": "Task execution alert" + }, + "monitorLevel": "HIGH", + "subSystemId": "1" +} +``` + +Response: +```json +{ + "method": "/api/jobhistory/setting/addObserveInfo", + "status": 0, + "message": "success" +} +``` + +### Delete Observe Info +``` +GET /api/rest_j/v1/jobhistory/setting/deleteObserveInfo +``` + +Parameters: +- `taskId`: Task ID (required) + +Response: +```json +{ + "method": "/api/jobhistory/setting/deleteObserveInfo", + "status": 0, + "message": "success" +} +``` + +## Database Table Structures + +The JobHistory service uses the following database tables from the linkis_ddl.sql file: + +### Job History Group Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The JobHistory service provides several RPC methods for job history management: + +### Job History RPCs + +#### recordJob +Records a job execution: +```java +void recordJob(JobRecordRequest request) +``` + +#### updateJobStatus +Updates the status of a job: +```java +void updateJobStatus(String jobId, JobStatus status) +``` + +#### getJobHistory +Retrieves job history: +```java +JobHistory getJobHistory(String jobId) +``` + +#### searchJobs +Searches for jobs based on criteria: +```java +List searchJobs(JobSearchCriteria criteria) +``` + +#### getJobDetails +Retrieves detailed job information: +```java +JobDetails getJobDetails(Long jobId) +``` + +#### deleteJobHistory +Deletes job history records: +```java +void deleteJobHistory(List jobIds) +``` + +### Statistics RPCs + +#### recordStatistics +Records job statistics: +```java +void recordStatistics(JobStatistics statistics) +``` + +#### getStatistics +Retrieves job statistics: +```java +JobStatistics getStatistics(String jobId) +``` + +#### getStatisticsByUser +Retrieves job statistics for a user: +```java +List getStatisticsByUser(String username, Date startDate, Date endDate) +``` + +#### getStatisticsByEngine +Retrieves job statistics by engine type: +```java +List getStatisticsByEngine(String engineType, Date startDate, Date endDate) +``` + +## Dependencies + +- linkis-mybatis +- linkis-rpc +- linkis-protocol +- linkis-common +- linkis-computation-governance-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- QueryRestfulApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java` +- StatisticsRestfulApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/StatisticsRestfulApi.java` +- JobhistorySettingApi: `linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/JobhistorySettingApi.java` + +### MyBatis XML Files +- JobHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/mysql/JobHistoryMapper.xml` +- JobDetailMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDetailMapper.xml` +- JobStatisticsMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobStatisticsMapper.xml` +- JobDiagnosisMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml` +- JobAiHistoryMapper: `linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobAiHistoryMapper.xml` diff --git a/.ai/modules/computation-governance/manager.md b/.ai/modules/computation-governance/manager.md new file mode 100644 index 00000000000..a74a77ed877 --- /dev/null +++ b/.ai/modules/computation-governance/manager.md @@ -0,0 +1,1010 @@ +# Manager Service + +The Manager service provides resource and application management capabilities for the Linkis system. + +## Overview + +This service manages the resources and applications in the Linkis system, including node management, resource allocation, label management, and engine lifecycle management. + +## Key Components + +### Core Classes +- `LinkisManagerApplication` - Main application class +- Node management +- Resource management +- Label management +- Engine lifecycle management + +### Features +- Node registration and management +- Resource allocation and monitoring +- Label-based routing +- Engine instance management +- Load balancing + +## API Interfaces + +### ECM (EngineConnManager) Management + +#### List All ECMs +``` +GET /api/rest_j/v1/linkisManager/listAllEMs +``` + +Parameters: +- `instance` (optional): Filter by instance name +- `nodeHealthy` (optional): Filter by node health status (Healthy, UnHealthy, WARN, StockAvailable, StockUnavailable) +- `owner` (optional): Filter by owner +- `tenantLabel` (optional): Filter by tenant label + +Response: +```json +{ + "method": "/api/linkisManager/listAllEMs", + "status": 0, + "message": "success", + "data": { + "EMs": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + }, + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ], + "nodeHealthy": "Healthy", + "owner": "testuser" + } + ] + } +} +``` + +Error Cases: +- Only admin users can access this API +- If parameters are invalid, appropriate error messages are returned + +Notes: +- Requires admin privileges +- Results can be filtered and sorted by various criteria +- Returns EMNodeVo objects with detailed information about each ECM + +#### List All ECM Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllECMHealthyStatus +``` + +Parameters: +- `onlyEditable` (optional): If true, returns only editable statuses (Healthy, UnHealthy, WARN, StockAvailable, StockUnavailable) + +Response: +```json +{ + "method": "/api/linkisManager/listAllECMHealthyStatus", + "status": 0, + "message": "success", + "data": { + "nodeHealthy": ["Healthy", "UnHealthy", "WARN", "StockAvailable", "StockUnavailable"] + } +} +``` + +Notes: +- Returns all possible NodeHealthy enum values +- With onlyEditable=true, returns only the statuses that can be modified by users + +#### Modify ECM Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEMInfo +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102", + "emStatus": "Healthy", + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/modifyEMInfo", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- Only admin users can modify ECM info +- If applicationName or instance is null, returns error +- If labels contain duplicates, returns error +- If label values are invalid, returns error + +Notes: +- Requires admin privileges +- Can update both EM status and labels +- Supports UserModifiable labels with value validation + +#### Execute ECM Operation +``` +POST /api/rest_j/v1/linkisManager/executeECMOperation +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102", + "parameters": { + "operation": "stopEngine", + "engineConnInstance": "bdp110:12295" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperation", + "status": 0, + "message": "success", + "data": { + "result": "Operation executed successfully", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If ECM node doesn't exist, returns error +- If operation parameters are invalid, returns error + +Notes: +- Supports various admin operations (configurable via AMConfiguration.ECM_ADMIN_OPERATIONS) +- For log operations, automatically fills in logDirSuffix if not provided +- Validates user permissions for admin operations + +#### Execute ECM Operation By EC +``` +POST /api/rest_j/v1/linkisManager/executeECMOperationByEC +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "parameters": { + "operation": "stopEngine" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeECMOperationByEC", + "status": 0, + "message": "success", + "data": { + "result": "Operation executed successfully", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If engine node doesn't exist, returns error +- If operation parameters are invalid, returns error + +Notes: +- User must be owner of the engine or admin +- Delegates to executeECMOperation after validating permissions + +#### Open Engine Log +``` +POST /api/rest_j/v1/linkisManager/openEngineLog +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "emInstance": "bdp110:9100", + "instance": "bdp110:21976", + "parameters": { + "logType": "stdout", + "fromLine": "0", + "pageSize": "1000" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/openEngineLog", + "status": 0, + "message": "success", + "data": { + "result": "Log content...", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to access logs, returns error +- If log type is invalid, returns error +- If engine instance doesn't exist, returns error + +Notes: +- Supported log types: stdout, stderr, gc, udfLog, yarnApp +- Automatically fills in logDirSuffix if not provided +- Validates user permissions (must be owner or admin) + +#### Task Prediction +``` +GET /api/rest_j/v1/linkisManager/task-prediction +``` + +Parameters: +- `username` (optional): User name (defaults to current user) +- `engineType` (required): Engine type (spark/hive/etc.) +- `creator` (required): Creator application +- `clustername` (optional): Cluster name +- `queueName` (optional): Queue name +- `tenant` (optional): Tenant + +Response: +```json +{ + "method": "/api/linkisManager/task-prediction", + "status": 0, + "message": "success", + "data": { + "tenant": "tenant", + "userResource": {...}, + "ecmResource": {...}, + "yarnResource": {...}, + "checkResult": true + } +} +``` + +Error Cases: +- If engineType or creator is null, returns error +- If resource check fails, returns error + +Notes: +- Checks if user can create an engine for specified parameters +- Returns detailed resource information for user, ECM, and YARN + +#### Reset Resource +``` +GET /api/rest_j/v1/linkisManager/reset-resource +``` + +Parameters: +- `serviceInstance` (optional): Service instance to reset +- `username` (optional): User name to reset + +Response: +```json +{ + "method": "/api/linkisManager/reset-resource", + "status": 0, + "message": "success" +} +``` + +Error Cases: +- Only admin users can reset resources + +Notes: +- Requires admin privileges +- Resets resource allocations for specified instance or user + +### Engine Management + +#### Ask Engine Connection +``` +POST /api/rest_j/v1/linkisManager/askEngineConn +``` + +Request Body: +```json +{ + "labels": { + "engineType": "spark-2.4.3", + "userCreator": "testuser-IDE" + }, + "timeOut": 30000, + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/askEngineConn", + "status": 0, + "message": "create engineConn ended.", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Starting", + "ticketId": "ticket-12345", + "ecmServiceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + } + } + } +} +``` + +Error Cases: +- If timeout is invalid, uses default timeout +- If engine creation fails, returns error with retry information + +Notes: +- First attempts to reuse existing engines +- If no suitable engine found, creates a new one +- Supports async engine creation with timeout handling + +#### Create Engine Connection +``` +POST /api/rest_j/v1/linkisManager/createEngineConn +``` + +Request Body: +```json +{ + "labels": { + "engineType": "spark-2.4.3", + "userCreator": "testuser-IDE" + }, + "timeout": 30000, + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/createEngineConn", + "status": 0, + "message": "create engineConn succeed.", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Starting", + "ticketId": "ticket-12345" + } + } +} +``` + +Error Cases: +- If timeout is invalid, uses default timeout +- If engine creation fails, returns error with retry information + +Notes: +- Always creates a new engine (doesn't attempt reuse) +- Supports timeout configuration +- Returns EngineNode information with service instance and ticket ID + +#### Get Engine Connection +``` +POST /api/rest_j/v1/linkisManager/getEngineConn +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/getEngineConn", + "status": 0, + "message": "success", + "data": { + "engine": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "ticketId": "ticket-12345" + } + } +} +``` + +Error Cases: +- If user doesn't have permission to access engine, returns error +- If engine instance doesn't exist, returns error + +Notes: +- User must be owner of the engine or admin +- Can retrieve engine info by service instance or ticket ID +- Returns EC metrics if available + +#### Kill Engine Connection +``` +POST /api/rest_j/v1/linkisManager/killEngineConn +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/killEngineConn", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If user doesn't have permission to kill engine, returns error +- If engine instance doesn't exist, returns error + +Notes: +- User must be owner of the engine or admin +- Sends EngineStopRequest to engine stop service +- Logs kill operation + +#### Kill ECM Engines +``` +POST /api/rest_j/v1/linkisManager/rm/killUnlockEngineByEM +``` + +Request Body: +```json +{ + "instance": "bdp110:9210" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/killUnlockEngineByEM", + "status": 0, + "message": "Kill engineConn succeed.", + "data": { + "result": {...} + } +} +``` + +Error Cases: +- Only admin users can kill engines by ECM +- If instance parameter is null, returns error + +Notes: +- Requires admin privileges +- Kills all unlocked engines under specified ECM +- Returns result information + +#### Kill Multiple Engines +``` +POST /api/rest_j/v1/linkisManager/rm/enginekill +``` + +Request Body: +```json +[ + { + "applicationName": "linkis-cg-engineconn", + "engineInstance": "bdp110:12295" + } +] +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/enginekill", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If engine instances don't exist, logs error but continues + +Notes: +- Kills multiple engines in a single request +- No permission check (uses internal sender) + +#### Kill Multiple Engines Async +``` +POST /api/rest_j/v1/linkisManager/rm/enginekillAsyn +``` + +Request Body: +```json +{ + "instances": ["bdp110:12295", "bdp110:12296"] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/enginekillAsyn", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- If user is not admin and doesn't have valid token, returns error +- If instances parameter is null or empty, returns error +- If instances parameter parsing fails, returns error + +Notes: +- Requires admin privileges or valid admin token +- Asynchronously stops engines with metrics update +- Supports batch killing of multiple engine instances + +#### List User Engines +``` +GET /api/rest_j/v1/linkisManager/listUserEngines +``` + +Response: +```json +{ + "method": "/api/linkisManager/listUserEngines", + "status": 0, + "message": "success", + "data": { + "engines": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "owner": "testuser", + "engineType": "spark" + } + ] + } +} +``` + +Notes: +- Returns engines owned by the current user +- Lists all engine nodes for the user + +#### List ECM Engines +``` +POST /api/rest_j/v1/linkisManager/listEMEngines +``` + +Request Body: +```json +{ + "em": { + "serviceInstance": { + "applicationName": "linkis-cg-engineconnmanager", + "instance": "bdp110:9102" + } + }, + "emInstance": "bdp110:9102" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/listEMEngines", + "status": 0, + "message": "success", + "data": { + "engines": [ + { + "serviceInstance": { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + }, + "nodeStatus": "Running", + "owner": "testuser", + "engineType": "spark" + } + ] + } +} +``` + +Error Cases: +- Only admin users can list ECM engines +- If parameters are invalid, returns error + +Notes: +- Requires admin privileges +- Supports filtering by EM instance, node status, engine type, and owner +- Returns AMEngineNodeVo objects with detailed engine information + +#### Modify Engine Info +``` +PUT /api/rest_j/v1/linkisManager/modifyEngineInfo +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "labels": [ + { + "labelKey": "engineType", + "stringValue": "spark" + } + ], + "nodeHealthy": "Healthy" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/modifyEngineInfo", + "status": 0, + "message": "success to update engine information(更新引擎信息成功)" +} +``` + +Error Cases: +- Only admin users can modify engine info +- If applicationName or instance is null, returns error +- If labels contain duplicates, returns error + +Notes: +- Requires admin privileges +- Can update both engine labels and health status +- Health status updates only support Healthy and UnHealthy values + +#### Batch Set Engine To UnHealthy +``` +POST /api/rest_j/v1/linkisManager/batchSetEngineToUnHealthy +``` + +Request Body: +```json +{ + "instances": [ + { + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/batchSetEngineToUnHealthy", + "status": 0, + "message": "success to update engine information(批量更新引擎健康信息成功)" +} +``` + +Error Cases: +- Only admin users can set engine health status +- If instances parameter is null, returns error + +Notes: +- Requires admin privileges +- Sets multiple engines to UnHealthy status +- Logs batch update operation + +#### List All Node Healthy Status +``` +GET /api/rest_j/v1/linkisManager/listAllNodeHealthyStatus +``` + +Parameters: +- `onlyEditable` (optional): If true, returns only editable statuses + +Response: +```json +{ + "method": "/api/linkisManager/listAllNodeHealthyStatus", + "status": 0, + "message": "success", + "data": { + "nodeStatus": ["Starting", "Unlock", "Locked", "Idle", "Busy", "Running", "ShuttingDown", "Failed", "Success"] + } +} +``` + +Notes: +- Returns all possible NodeStatus enum values +- With onlyEditable parameter, behavior is the same (returns all statuses) + +#### Execute Engine Conn Operation +``` +POST /api/rest_j/v1/linkisManager/executeEngineConnOperation +``` + +Request Body: +```json +{ + "applicationName": "linkis-cg-engineconn", + "instance": "bdp110:12295", + "parameters": { + "operation": "someOperation" + } +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/executeEngineConnOperation", + "status": 0, + "message": "success", + "data": { + "result": "Operation result...", + "errorMsg": "", + "isError": false + } +} +``` + +Error Cases: +- If user doesn't have permission to execute operation, returns error +- If engine instance doesn't exist, returns error +- If operation fails, returns error details + +Notes: +- User must be owner of the engine or admin +- Executes arbitrary operations on engine nodes +- Returns operation result and error information + +#### Kill Engines By Creator Or EngineType +``` +POST /api/rest_j/v1/linkisManager/rm/killEngineByCreatorEngineType +``` + +Request Body: +```json +{ + "creator": "IDE", + "engineType": "hive-2.3.3" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/killEngineByCreatorEngineType", + "status": 0, + "message": "Kill engineConn succeed." +} +``` + +Error Cases: +- Only admin users can kill engines by creator or engine type +- If creator or engineType parameters are null, returns error + +Notes: +- Requires admin privileges +- Kills all engines matching creator and engine type +- Supports cross-cluster killing with additional parameters + +### EC Resource Info Management + +#### Get EC Resource Info +``` +GET /api/rest_j/v1/linkisManager/ecinfo/get +``` + +Parameters: +- `ticketid`: Ticket ID + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/get", + "status": 0, + "message": "success", + "data": { + "ecResourceInfoRecord": { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running", + "usedResource": "{\"cpu\": 2, \"memory\": \"2G\"}", + "releasedResource": "{\"cpu\": 0, \"memory\": \"0G\"}", + "requestResource": "{\"cpu\": 2, \"memory\": \"2G\"}" + } + } +} +``` + +Error Cases: +- If ticket ID doesn't exist, returns error +- If user doesn't have permission to access resource info, returns error + +Notes: +- User must be creator of the resource or admin +- Returns detailed EC resource information record +- Includes resource usage statistics + +#### Delete EC Resource Info +``` +DELETE /api/rest_j/v1/linkisManager/ecinfo/delete/{ticketid} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/delete/ticket-12345", + "status": 0, + "message": "success", + "data": { + "ecResourceInfoRecord": { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running" + } + } +} +``` + +Error Cases: +- If ticket ID doesn't exist, returns error +- If user doesn't have permission to delete resource info, returns error + +Notes: +- User must be creator of the resource or admin +- Deletes EC resource information record from database +- Returns deleted record information + +#### Query EC Resource History List +``` +GET /api/rest_j/v1/linkisManager/ecinfo/ecrHistoryList +``` + +Parameters: +- `instance` (optional): Filter by instance +- `creator` (optional): Filter by creator +- `startDate` (optional): Filter by start date +- `endDate` (optional): Filter by end date (defaults to current date) +- `engineType` (optional): Filter by engine type +- `status` (optional): Filter by status +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecrHistoryList", + "status": 0, + "message": "success", + "data": { + "engineList": [ + { + "id": 12345, + "labelValue": "spark-2.4.3", + "createUser": "testuser", + "serviceInstance": "bdp110:12295", + "ticketId": "ticket-12345", + "status": "Running", + "usedResource": { + "cpu": 2, + "memory": "2G" + }, + "releasedResource": { + "cpu": 0, + "memory": "0G" + }, + "requestResource": { + "cpu": 2, + "memory": "2G" + } + } + ], + "totalPage": 1 + } +} +``` + +Error Cases: +- If creator parameter is invalid, returns error +- If date parameters are invalid, uses defaults + +Notes: +- Admin users can view all records, regular users only their own +- Supports date range filtering +- Supports pagination +- Converts resource strings to maps for easier consumption + +#### Query EC List +``` +POST /api/rest_j/v1/linkisManager/ecinfo/ecList +``` + +Request Body: +```json +{ + "creators": ["testuser"], + "engineTypes": ["spark-2.4.3"], + "statuss": ["Running"], + "queueName": "default", + "ecInstances": ["bdp110:12295"], + "crossCluster": false +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/ecinfo/ecList", + "status": 0, + "message": "success", + "data": { + "ecList": [ + { + // EC information + } + ] + } +} +``` + +Error Cases: +- If creator parameter is invalid, returns error +- If parameters parsing fails, returns error + +Notes: +- Requires admin privileges or valid admin token +- Supports filtering by creators, engine types, statuses, queue name, and EC instances +- Supports cross-cluster filtering \ No newline at end of file diff --git a/.ai/modules/microservice-governance/README.md b/.ai/modules/microservice-governance/README.md new file mode 100644 index 00000000000..07035f1815b --- /dev/null +++ b/.ai/modules/microservice-governance/README.md @@ -0,0 +1,112 @@ +# Microservice Governance Services + +The microservice governance services provide the infrastructure foundation for the Linkis microservices architecture. + +## Service Modules + +- [Eureka Service](./eureka.md) - Service registry and discovery center +- [Gateway Service](./gateway.md) - API gateway for request routing and security + +## Overview + +These services form the infrastructure layer of Linkis, providing essential capabilities for service discovery, API routing, and inter-service communication. + +## Common Features + +### Service Discovery +- Service registration and deregistration +- Health checking of services +- Service instance management +- Load balancing support + +### API Gateway +- Request routing and filtering +- Authentication and authorization +- Rate limiting and traffic control +- Request/response transformation + +### Inter-Service Communication +- RESTful service communication +- Load balancing between services +- Circuit breaker pattern implementation +- Service monitoring and metrics + +## API Interface Summary + +### Eureka Service APIs +- Service registration: `POST /eureka/apps/{appName}` +- Service discovery: `GET /eureka/apps/{appName}` +- Health check: `GET /eureka/apps/{appName}/{instanceId}` + +### Gateway Service APIs +- Route management: `GET /actuator/gateway/routes` +- Health check: `GET /actuator/health` +- Gateway metrics: `GET /actuator/metrics` + +## Database Schema Summary + +### Service Registry Table +```sql +CREATE TABLE linkis_service_registry ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + service_name VARCHAR(128) NOT NULL, + instance_id VARCHAR(128) NOT NULL UNIQUE, + instance_address VARCHAR(128), + instance_port INT, + status VARCHAR(50) DEFAULT 'UP', + metadata JSON, + register_time DATETIME DEFAULT CURRENT_TIMESTAMP, + last_heartbeat DATETIME, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Route Table +```sql +CREATE TABLE linkis_gateway_route ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL UNIQUE, + route_order INT DEFAULT 0, + uri VARCHAR(255) NOT NULL, + predicates JSON, + filters JSON, + metadata JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Access Log Table +```sql +CREATE TABLE linkis_gateway_access_log ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + client_ip VARCHAR(50), + request_method VARCHAR(10), + request_uri VARCHAR(500), + request_params TEXT, + user_token VARCHAR(255), + service_id VARCHAR(128), + response_status INT, + response_time BIGINT, + access_time DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +## RPC Methods Summary + +### Eureka Service RPCs +- `registerService(ServiceRegistrationRequest request)` +- `unregisterService(String serviceId, String instanceId)` +- `getServiceInstances(String serviceName)` +- `heartbeat(String serviceId, String instanceId)` + +### Gateway Service RPCs +- `addRoute(GatewayRoute route)` +- `removeRoute(String routeId)` +- `updateRoute(GatewayRoute route)` +- `getRoutes()` +- `configureAuthentication(AuthenticationConfig config)` +- `validateToken(String token)` +- `getUserPermissions(String user)` +- `applyRateLimit(RateLimitConfig config)` +- `getRateLimitStatus(String clientId)` \ No newline at end of file diff --git a/.ai/modules/microservice-governance/eureka.md b/.ai/modules/microservice-governance/eureka.md new file mode 100644 index 00000000000..1345517e4a0 --- /dev/null +++ b/.ai/modules/microservice-governance/eureka.md @@ -0,0 +1,131 @@ +# Eureka Service + +The Eureka service provides service registration and discovery capabilities for the Linkis microservices architecture. + +## Overview + +This service implements the Eureka server for service discovery, managing the registration, discovery, and health checking of all microservice instances in the Linkis system. + +## Key Components + +### Core Classes +- `SpringCloudEurekaApplication` - Main application class +- Eureka server configuration +- Health check endpoints +- Service registry management + +### Features +- Service instance registration +- Service discovery for clients +- Health status monitoring +- REST API for service management + +## API Interfaces + +### Service Registration +``` +POST /eureka/apps/{appName} +``` + +Request Body: +```xml + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + +``` + +### Service Discovery +``` +GET /eureka/apps/{appName} +``` + +Response: +```xml + + APP-NAME + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + + +``` + +### Health Check +``` +GET /eureka/apps/{appName}/{instanceId} +``` + +Response: +```xml + + service-host + APP-NAME + 127.0.0.1 + 8080 + UP + 1234567890 + +``` + +## Database Table Structures + +The Eureka service typically doesn't directly manage database tables, as it stores service registry information in memory. However, it may interact with the following tables for persistent storage: + +### Service Registry Table +```sql +CREATE TABLE linkis_service_registry ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + service_name VARCHAR(128) NOT NULL, + instance_id VARCHAR(128) NOT NULL UNIQUE, + instance_address VARCHAR(128), + instance_port INT, + status VARCHAR(50) DEFAULT 'UP', + metadata JSON, + register_time DATETIME DEFAULT CURRENT_TIMESTAMP, + last_heartbeat DATETIME, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +## RPC Methods + +The Eureka service provides RPC methods for service management: + +### Service Management RPCs + +#### registerService +Registers a service instance: +```java +void registerService(ServiceRegistrationRequest request) +``` + +#### unregisterService +Unregisters a service instance: +```java +void unregisterService(String serviceId, String instanceId) +``` + +#### getServiceInstances +Retrieves instances of a service: +```java +List getServiceInstances(String serviceName) +``` + +#### heartbeat +Sends a heartbeat for a service instance: +```java +void heartbeat(String serviceId, String instanceId) +``` + +## Dependencies + +- Spring Cloud Netflix Eureka Server +- Spring Boot +- Netflix Eureka Core \ No newline at end of file diff --git a/.ai/modules/microservice-governance/gateway.md b/.ai/modules/microservice-governance/gateway.md new file mode 100644 index 00000000000..4d07a65e884 --- /dev/null +++ b/.ai/modules/microservice-governance/gateway.md @@ -0,0 +1,421 @@ +# Gateway Service + +The Gateway service provides API gateway functionality for the Linkis system, routing requests to appropriate backend services and providing security, rate limiting, and other cross-cutting concerns. + +## Overview + +This service implements an API gateway that serves as the single entry point for all client requests to the Linkis system. It handles request routing, authentication, authorization, rate limiting, and other infrastructure concerns. + +## Key Components + +### Core Classes +- `LinkisGatewayApplication` - Main application class +- Route configuration management +- Request/response filtering +- Authentication handling +- Rate limiting implementation + +### Features +- Request routing and load balancing +- Authentication and authorization +- Rate limiting and traffic control +- Request/response transformation +- SSL/TLS termination +- Logging and monitoring + +## API Interfaces + +### Route Management +``` +GET /actuator/gateway/routes +``` + +Response: +```json +{ + "routes": [ + { + "route_id": "linkis-entrance", + "uri": "lb://linkis-entrance", + "predicates": [ + "Path=/api/entrance/**" + ], + "filters": [ + "StripPrefix=1" + ] + } + ] +} +``` + +### Health Check +``` +GET /actuator/health +``` + +Response: +```json +{ + "status": "UP", + "components": { + "discoveryComposite": { + "status": "UP" + }, + "gateway": { + "status": "UP" + } + } +} +``` + +### Gateway Metrics +``` +GET /actuator/metrics +``` + +Response: +```json +{ + "names": [ + "gateway.requests", + "jvm.memory.used", + "http.server.requests" + ] +} +``` + +### Authentication Token Management +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 0, + "list": [], + "pageNum": 1, + "pageSize": 10, + "size": 0, + "startRow": 0, + "endRow": 0, + "pages": 0, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [] + } + } +} +``` + +### Add Authentication Token +``` +POST /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Request Body: +```json +{ + "tokenName": "test-token", + "legalUsers": "*", + "businessOwner": "BDP" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Update Authentication Token +``` +PUT /api/rest_j/v1/basedata-manager/gateway-auth-token +``` + +Request Body: +```json +{ + "id": 1, + "tokenName": "test-token", + "legalUsers": "user1,user2", + "businessOwner": "BDP" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Get Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/{id} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "tokenName": "test-token", + "legalUsers": "user1,user2", + "businessOwner": "BDP", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + } +} +``` + +### Remove Authentication Token +``` +DELETE /api/rest_j/v1/basedata-manager/gateway-auth-token/{id} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Check Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/checkToken +``` + +Parameters: +- `token`: Authentication token to check (required) +- `checkName`: User name to check (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Decrypt Authentication Token +``` +GET /api/rest_j/v1/basedata-manager/gateway-auth-token/decrypt-token +``` + +Parameters: +- `token`: Authentication token to decrypt (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": "decrypted-token" + } +} +``` + +## Database Table Structures + +The Gateway service manages the following database tables: + +### Gateway Route Table +```sql +CREATE TABLE linkis_gateway_route ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL UNIQUE, + route_order INT DEFAULT 0, + uri VARCHAR(255) NOT NULL, + predicates JSON, + filters JSON, + metadata JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Filter Table +```sql +CREATE TABLE linkis_gateway_filter ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + route_id VARCHAR(128) NOT NULL, + filter_name VARCHAR(128) NOT NULL, + filter_order INT DEFAULT 0, + args JSON, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + FOREIGN KEY (route_id) REFERENCES linkis_gateway_route(route_id) ON DELETE CASCADE +); +``` + +### Authentication Configuration Table +```sql +CREATE TABLE linkis_gateway_auth ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + path_pattern VARCHAR(255) NOT NULL, + auth_required BOOLEAN DEFAULT TRUE, + allowed_roles JSON, + rate_limit INT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP +); +``` + +### Gateway Access Log Table +```sql +CREATE TABLE linkis_gateway_access_log ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + client_ip VARCHAR(50), + request_method VARCHAR(10), + request_uri VARCHAR(500), + request_params TEXT, + user_token VARCHAR(255), + service_id VARCHAR(128), + response_status INT, + response_time BIGINT, + access_time DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +### Gateway Auth Token Table +```sql +CREATE TABLE `linkis_gateway_auth_token` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `token_name` varchar(255) COLLATE utf8_bin NOT NULL, + `legal_users` varchar(255) COLLATE utf8_bin NOT NULL, + `create_by` varchar(255) COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `elapse_day` bigint(20) DEFAULT '-1', + `update_by` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `business_owner` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `token_alias` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `token_sign` varchar(255) COLLATE utf8_bin DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `unique_token_name` (`token_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; +``` + +## RPC Methods + +The Gateway service provides RPC methods for gateway management: + +### Route Management RPCs + +#### addRoute +Adds a new route configuration: +```java +void addRoute(GatewayRoute route) +``` + +#### removeRoute +Removes a route configuration: +```java +void removeRoute(String routeId) +``` + +#### updateRoute +Updates a route configuration: +```java +void updateRoute(GatewayRoute route) +``` + +#### getRoutes +Retrieves all route configurations: +```java +List getRoutes() +``` + +### Authentication RPCs + +#### configureAuthentication +Configures authentication for a path: +```java +void configureAuthentication(AuthenticationConfig config) +``` + +#### validateToken +Validates an authentication token: +```java +TokenValidationResult validateToken(String token) +``` + +#### getUserPermissions +Retrieves user permissions: +```java +UserPermissions getUserPermissions(String user) +``` + +### Rate Limiting RPCs + +#### applyRateLimit +Applies rate limiting to a route: +```java +void applyRateLimit(RateLimitConfig config) +``` + +#### getRateLimitStatus +Retrieves current rate limit status: +```java +RateLimitStatus getRateLimitStatus(String clientId) +``` + +## Dependencies + +- Spring Cloud Gateway +- Spring Boot +- Spring Security +- Spring Cloud LoadBalancer +- linkis-common +- linkis-httpclient +- Various Spring Cloud components + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- GatewayAuthTokenRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/GatewayAuthTokenRestfulApi.java` + +### MyBatis XML Files +- GatewayRouteMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayRouteMapper.xml` +- GatewayFilterMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayFilterMapper.xml` +- GatewayAuthMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAuthMapper.xml` +- GatewayAccessLogMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAccessLogMapper.xml` +- GatewayAuthTokenMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/GatewayAuthTokenMapper.xml` \ No newline at end of file diff --git a/.ai/modules/microservice-governance/monitor.md b/.ai/modules/microservice-governance/monitor.md new file mode 100644 index 00000000000..1a52b1696ba --- /dev/null +++ b/.ai/modules/microservice-governance/monitor.md @@ -0,0 +1,261 @@ +# Monitor Service + +Monitor service is responsible for monitoring the health status of various components in the Linkis system, including resource monitoring, node heartbeat monitoring, etc. + +## Table of Contents +- [API Interfaces](#api-interfaces) +- [Database Tables](#database-tables) +- [RPC Methods](#rpc-methods) +- [Interface Classes and MyBatis XML Files](#interface-classes-and-mybatis-xml-files) + +## API Interfaces + +### Get Application List +**POST /linkisManager/rm/applicationlist** + +Get the list of applications for a specific user. + +Request Parameters: +```json +{ + "userCreator": "string", + "engineType": "string" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/applicationlist", + "status": 0, + "message": "OK", + "data": { + "applications": [] + } +} +``` + +### Reset User Resource +**DELETE /linkisManager/rm/resetResource** + +Reset user resource, admin only. + +Request Parameters: +- resourceId (optional): Integer + +Response: +```json +{ + "method": "/api/linkisManager/rm/resetResource", + "status": 0, + "message": "success", + "data": {} +} +``` + +### List All Engine Types +**GET /linkisManager/rm/engineType** + +Get all supported engine types. + +Response: +```json +{ + "method": "/api/linkisManager/rm/engineType", + "status": 0, + "message": "OK", + "data": { + "engineType": ["string"] + } +} +``` + +### Get All User Resources +**GET /linkisManager/rm/allUserResource** + +Get all user resources, admin only. + +Request Parameters: +- username (optional): String +- creator (optional): String +- engineType (optional): String +- page (optional): Integer +- size (optional): Integer + +Response: +```json +{ + "method": "/api/linkisManager/rm/allUserResource", + "status": 0, + "message": "OK", + "data": { + "resources": [], + "total": 0 + } +} +``` + +### Get User Resource by Label +**GET /linkisManager/rm/get-user-resource** + +Get user resource by label. + +Request Parameters: +- username: String +- creator: String +- engineType: String + +Response: +```json +{ + "method": "/api/linkisManager/rm/get-user-resource", + "status": 0, + "message": "OK", + "data": { + "resources": [] + } +} +``` + +### Get User Resources +**POST /linkisManager/rm/userresources** + +Get user resources. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/userresources", + "status": 0, + "message": "OK", + "data": { + "userResources": [] + } +} +``` + +### Get Engines +**POST /linkisManager/rm/engines** + +Get engines for a user. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/engines", + "status": 0, + "message": "OK", + "data": { + "engines": [] + } +} +``` + +### Get Queue Resource +**POST /linkisManager/rm/queueresources** + +Get queue resource information. + +Request Parameters: +```json +{ + "queuename": "string", + "clustername": "string", + "clustertype": "string", + "crossCluster": "boolean" +} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/queueresources", + "status": 0, + "message": "OK", + "data": { + "queueInfo": {}, + "userResources": [] + } +} +``` + +### Get Queues +**POST /linkisManager/rm/queues** + +Get queue information. + +Request Parameters: +```json +{} +``` + +Response: +```json +{ + "method": "/api/linkisManager/rm/queues", + "status": 0, + "message": "OK", + "data": { + "queues": [] + } +} +``` + +## Database Tables + +### linkis_cg_rm_external_resource_provider +```sql +CREATE TABLE `linkis_cg_rm_external_resource_provider` ( + `id` int(10) NOT NULL AUTO_INCREMENT, + `resource_type` varchar(32) NOT NULL, + `name` varchar(32) NOT NULL, + `labels` varchar(32) DEFAULT NULL, + `config` text NOT NULL, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### linkis_cg_rm_resource_action_record +```sql +CREATE TABLE linkis_cg_rm_resource_action_record ( + `id` INT(20) NOT NULL AUTO_INCREMENT, + `label_value` VARCHAR(100) NOT NULL, + `ticket_id` VARCHAR(100) NOT NULL, + `request_times` INT(8), + `request_resource_all` VARCHAR(100), + `used_times` INT(8), + `used_resource_all` VARCHAR(100), + `release_times` INT(8), + `release_resource_all` VARCHAR(100), + `update_time` datetime DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `label_value_ticket_id` (`label_value`, `ticket_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +``` + +## RPC Methods + +Monitor service does not expose specific RPC methods directly. It primarily works through the RESTful APIs listed above. + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- RMMonitorRest.scala: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\rm\restful\RMMonitorRest.scala` + +### MyBatis XML Files +The Monitor service uses the following persistence layer interfaces which may have corresponding MyBatis XML files: +- LabelManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\LabelManagerPersistence.scala` +- ResourceManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\ResourceManagerPersistence.scala` +- NodeManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\NodeManagerPersistence.scala` +- NodeMetricManagerPersistence: `e:\workspace\WeDataSphere\linkis\linkis-computation-governance\linkis-manager\linkis-application-manager\src\main\scala\org\apache\linkis\manager\persistence\NodeMetricManagerPersistence.scala` \ No newline at end of file diff --git a/.ai/modules/public-enhancements/README.md b/.ai/modules/public-enhancements/README.md new file mode 100644 index 00000000000..03433e33630 --- /dev/null +++ b/.ai/modules/public-enhancements/README.md @@ -0,0 +1,235 @@ +# Public Enhancement Services + +The public enhancement services provide shared capabilities used across the Linkis platform. + +## Service Modules + +- [Public Service](./publicservice.md) - Core public services +- [Configuration Service](./configuration.md) - Configuration management +- [BML Service](./bml.md) - Big Data Material Library +- [DataSource Service](./datasource.md) - Data source management +- [Context Service](./context.md) - Context and variable sharing +- [Monitor Service](./monitor.md) - System monitoring + +## Overview + +These services provide common capabilities that are used across the Linkis platform, including file management, configuration management, data source management, context sharing, and system monitoring. + +## Common Features + +### Resource Management +- Binary and material management +- User-defined function management +- Shared resource tracking + +### Configuration Management +- Centralized configuration service +- Runtime configuration management +- Configuration versioning + +### Context Management +- Cross-application context sharing +- Variable and parameter management +- Unified context service + +### Data Source Management +- Data source registration and management +- Metadata querying +- Connection testing and validation + +### System Monitoring +- Performance metrics collection +- System health monitoring +- Alerting and notifications + +## API Interface Summary + +### Public Service APIs +- File system operations +- Variable management +- Error code querying + +### Configuration Service APIs +- Configuration retrieval: `GET /api/rest_j/v1/configuration` +- Configuration update: `POST /api/rest_j/v1/configuration/update` +- Template management: `GET /api/rest_j/v1/configuration/template` + +### BML Service APIs +- File upload: `POST /api/rest_j/v1/bml/upload` +- File download: `GET /api/rest_j/v1/bml/download` +- File version list: `GET /api/rest_j/v1/bml/versions` + +### DataSource Service APIs +- Data source CRUD: `POST/GET/PUT/DELETE /api/rest_j/v1/datasource` +- Metadata query: `GET /api/rest_j/v1/datasource/metadata` +- Connection test: `POST /api/rest_j/v1/datasource/connect` + +### Context Service APIs +- Context creation: `POST /api/rest_j/v1/context` +- Variable management: `POST /api/rest_j/v1/context/variable` +- Context sharing: `POST /api/rest_j/v1/context/share` + +### Monitor Service APIs +- Metrics collection: `GET /api/rest_j/v1/monitor/metrics` +- Health check: `GET /api/rest_j/v1/monitor/health` +- Alert management: `POST /api/rest_j/v1/monitor/alert` + +## Database Schema Summary + +### BML Resources Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'resource uuid', + `is_private` TINYINT(1) DEFAULT 0 COMMENT 'Whether the resource is private, 0 means private, 1 means public', + `resource_header` TINYINT(1) DEFAULT 0 COMMENT 'Classification, 0 means unclassified, 1 means classified', + `downloaded_file_name` varchar(200) DEFAULT NULL COMMENT 'File name when downloading', + `sys` varchar(100) NOT NULL COMMENT 'Owning system', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Created time', + `owner` varchar(200) NOT NULL COMMENT 'Resource owner', + `is_expire` TINYINT(1) DEFAULT 0 COMMENT 'Whether expired, 0 means not expired, 1 means expired', + `expire_type` varchar(50) DEFAULT null COMMENT 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time', + `expire_time` varchar(50) DEFAULT null COMMENT 'Expiration time, one day by default', + `max_version` int(20) DEFAULT 10 COMMENT 'The default is 10, which means to keep the latest 10 versions', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Updated time', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_eflag`(`resource_id`, `enable_flag`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Version Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `file_md5` varchar(32) NOT NULL COMMENT 'Md5 summary of the file', + `version` varchar(20) NOT NULL COMMENT 'Resource version (v plus five digits)', + `size` int(10) NOT NULL COMMENT 'File size', + `start_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `end_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `resource` varchar(2000) NOT NULL COMMENT 'Resource content (file information including path and file name)', + `description` varchar(2000) DEFAULT NULL COMMENT 'description', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Started time', + `end_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Stoped time', + `client_ip` varchar(200) NOT NULL COMMENT 'Client ip', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_version`(`resource_id`, `version`), + PRIMARY KEY (`id`) +); +``` + +### Configuration Key Table +```sql +CREATE TABLE `linkis_ps_configuration_config_key`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Table +```sql +CREATE TABLE `linkis_ps_datasource_table` ( + `id` bigint(255) NOT NULL AUTO_INCREMENT, + `database` varchar(64) COLLATE utf8_bin NOT NULL, + `name` varchar(64) COLLATE utf8_bin NOT NULL, + `alias` varchar(64) COLLATE utf8_bin DEFAULT NULL, + `creator` varchar(16) COLLATE utf8_bin NOT NULL, + `comment` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `create_time` datetime NOT NULL, + `product_name` varchar(64) COLLATE utf8_bin DEFAULT NULL, + `project_name` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `usage` varchar(128) COLLATE utf8_bin DEFAULT NULL, + `lifecycle` int(4) NOT NULL, + `use_way` int(4) NOT NULL, + `is_import` tinyint(1) NOT NULL, + `model_level` int(4) NOT NULL, + `is_external_use` tinyint(1) NOT NULL, + `is_partition_table` tinyint(1) NOT NULL, + `is_available` tinyint(1) NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_db_name` (`database`,`name`) +); +``` + +### Context Map Table +```sql +CREATE TABLE `linkis_ps_cs_context_map` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(128) DEFAULT NULL, + `context_scope` varchar(32) DEFAULT NULL, + `context_type` varchar(32) DEFAULT NULL, + `props` text, + `value` mediumtext, + `context_id` int(11) DEFAULT NULL, + `keywords` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`), + KEY `idx_keywords` (`keywords`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods Summary + +### Public Service RPCs +- `getErrorCode(String errorCode)` +- `getVariable(String variableName)` +- `setVariable(String variableName, String value)` + +### Configuration Service RPCs +- `getConfiguration(String user, String creator, String engineType)` +- `updateConfiguration(String user, ConfigurationUpdateRequest request)` +- `getTemplateConfiguration(String engineType)` + +### BML Service RPCs +- `uploadResource(ResourceUploadRequest request)` +- `downloadResource(String resourceId, String version)` +- `deleteResource(String resourceId)` +- `getResourceInfo(String resourceId)` + +### DataSource Service RPCs +- `createDataSource(DataSourceCreateRequest request)` +- `updateDataSource(DataSourceUpdateRequest request)` +- `deleteDataSource(Long dataSourceId)` +- `queryDataSource(DataSourceQueryRequest request)` + +### Context Service RPCs +- `createContext(ContextCreateRequest request)` +- `getContextValue(String contextId, String key)` +- `setContextValue(String contextId, String key, String value)` +- `removeContext(String contextId)` + +### Monitor Service RPCs +- `collectMetrics(MetricsCollectionRequest request)` +- `getHealthStatus()` +- `sendAlert(AlertRequest request)` + +## Dependencies + +- linkis-commons - Shared utilities +- linkis-protocol - Communication protocols +- linkis-rpc - Remote procedure calls +- Various database drivers +- Spring Cloud ecosystem \ No newline at end of file diff --git a/.ai/modules/public-enhancements/bml.md b/.ai/modules/public-enhancements/bml.md new file mode 100644 index 00000000000..6f5673e09af --- /dev/null +++ b/.ai/modules/public-enhancements/bml.md @@ -0,0 +1,634 @@ +# BML Service + +The BML (Big Data Material Library) Service provides file and material management capabilities for the Linkis system. + +## Overview + +This service manages the storage, versioning, and sharing of files and materials used in big data processing tasks. + +## Key Components + +### Core Classes +- `LinkisBMLApplication` - Main application class +- File upload and download +- File version management +- File sharing and access control + +### Features +- File upload and download +- File versioning +- File sharing +- Access control +- File metadata management + +## API Interfaces + +### File Upload +``` +POST /api/rest_j/v1/bml/upload +``` + +Parameters: +- `system` (optional): System name +- `resourceHeader` (optional): Resource header +- `isExpire` (optional): Whether resource expires +- `expireType` (optional): Expiration type +- `expireTime` (optional): Expiration time +- `maxVersion` (optional): Maximum version count +- `file` (required): File to upload + +Response: +``` +{ + "method": "/api/bml/upload", + "status": 0, + "message": "The task of submitting and uploading resources was successful(提交上传资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000001", + "taskId": 12345 + } +} +``` + +### File Download +``` +GET /api/rest_j/v1/bml/download +``` + +Parameters: +- `resourceId`: Resource ID to download (required) +- `version`: Version to download (optional, defaults to latest) + +Response: +``` +Binary file content +``` + +### File Version List +``` +GET /api/rest_j/v1/bml/getVersions +``` + +Parameters: +- `resourceId`: Resource ID to list versions for (required) +- `currentPage`: Current page number (optional) +- `pageSize`: Page size (optional) + +Response: +```json +{ + "method": "/api/bml/getVersions", + "status": 0, + "message": "Version information obtained successfully (成功获取版本信息)", + "data": { + "ResourceVersions": { + "resourceId": "resource-12345", + "user": "testuser", + "versions": [ + { + "version": "v000001", + "size": 1024, + "createTime": "2023-01-01 12:00:00" + } + ] + } + } +} +``` + +### File Update +``` +POST /api/rest_j/v1/bml/updateVersion +``` + +Parameters: +- `resourceId`: Resource ID to update (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/updateVersion", + "status": 0, + "message": "The update resource task was submitted successfully(提交更新资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000002", + "taskId": 12346 + } +} +``` + +### File Delete +``` +POST /api/rest_j/v1/bml/deleteResource +``` + +Request Body: +```json +{ + "resourceId": "resource-12345" +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteResource", + "status": 0, + "message": "Resource deleted successfully(删除资源成功)" +} +``` + +### Batch File Delete +``` +POST /api/rest_j/v1/bml/deleteResources +``` + +Request Body: +```json +{ + "resourceIds": ["resource-12345", "resource-12346"] +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteResources", + "status": 0, + "message": "Batch deletion of resource was successful(批量删除资源成功)" +} +``` + +### File Information +``` +GET /api/rest_j/v1/bml/getBasic +``` + +Parameters: +- `resourceId`: Resource ID to get information for (required) + +Response: +```json +{ + "method": "/api/bml/getBasic", + "status": 0, + "message": "Acquisition of resource basic information successfully(获取资源基本信息成功)", + "data": { + "basic": { + "resourceId": "resource-12345", + "owner": "testuser", + "createTime": "2023-01-01 12:00:00", + "downloadedFileName": "test.csv", + "expireTime": "Resource not expired(资源不过期)", + "numberOfVerions": 10 + } + } +} +``` + +### Version Delete +``` +POST /api/rest_j/v1/bml/deleteVersion +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "version": "v000001" +} +``` + +Response: +```json +{ + "method": "/api/bml/deleteVersion", + "status": 0, + "message": "Deleted version successfully(删除版本成功)" +} +``` + +### Change Owner +``` +POST /api/rest_j/v1/bml/changeOwner +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "oldOwner": "testuser", + "newOwner": "newuser" +} +``` + +Response: +```json +{ + "method": "/api/bml/changeOwner", + "status": 0, + "message": "更新owner成功！" +} +``` + +### Copy Resource To Another User +``` +POST /api/rest_j/v1/bml/copyResourceToAnotherUser +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "anotherUser": "newuser" +} +``` + +Response: +```json +{ + "method": "/api/bml/copyResourceToAnotherUser", + "status": 0, + "message": "success", + "data": { + "resourceId": "resource-67890" + } +} +``` + +### Rollback Version +``` +POST /api/rest_j/v1/bml/rollbackVersion +``` + +Request Body: +```json +{ + "resourceId": "resource-12345", + "version": "v000001" +} +``` + +Response: +```json +{ + "method": "/api/bml/rollbackVersion", + "status": 0, + "message": "success", + "data": { + "resourceId": "resource-12345", + "version": "v000001" + } +} +``` + +### Create BML Project +``` +POST /api/rest_j/v1/bml/createBmlProject +``` + +Request Body: +```json +{ + "projectName": "test-project", + "editUsers": ["user1", "user2"], + "accessUsers": ["user3", "user4"] +} +``` + +Response: +```json +{ + "method": "/api/bml/createBmlProject", + "status": 0, + "message": "success to create project(创建工程ok)" +} +``` + +### Upload Share Resource +``` +POST /api/rest_j/v1/bml/uploadShareResource +``` + +Parameters: +- `system` (optional): System name +- `resourceHeader` (optional): Resource header +- `isExpire` (optional): Whether resource expires +- `expireType` (optional): Expiration type +- `expireTime` (optional): Expiration time +- `maxVersion` (optional): Maximum version count +- `projectName`: Project name (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/uploadShareResource", + "status": 0, + "message": "The task of submitting and uploading resources was successful(提交上传资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000001", + "taskId": 12345 + } +} +``` + +### Update Share Resource +``` +POST /api/rest_j/v1/bml/updateShareResource +``` + +Parameters: +- `resourceId`: Resource ID to update (required) +- `file`: File to upload (required) + +Response: +```json +{ + "method": "/api/bml/updateShareResource", + "status": 0, + "message": "The update resource task was submitted successfully(提交更新资源任务成功)", + "data": { + "resourceId": "resource-12345", + "version": "v000002", + "taskId": 12346 + } +} +``` + +### Download Share Resource +``` +GET /api/rest_j/v1/bml/downloadShareResource +``` + +Parameters: +- `resourceId`: Resource ID to download (required) +- `version`: Version to download (optional, defaults to latest) + +Response: +``` +Binary file content +``` + +### Update Project Users +``` +POST /api/rest_j/v1/bml/updateProjectUsers +``` + +Request Body: +```json +{ + "projectName": "test-project", + "editUsers": ["user1", "user2", "user5"], + "accessUsers": ["user3", "user4", "user6"] +} +``` + +Response: +```json +{ + "method": "/api/bml/updateProjectUsers", + "status": 0, + "message": "Updated project related user success(更新工程的相关用户成功)" +} +``` + +## Database Table Structures + +The BML Service uses the following database tables from linkis_ddl.sql: + +### BML Resources Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'resource uuid', + `is_private` TINYINT(1) DEFAULT 0 COMMENT 'Whether the resource is private, 0 means private, 1 means public', + `resource_header` TINYINT(1) DEFAULT 0 COMMENT 'Classification, 0 means unclassified, 1 means classified', + `downloaded_file_name` varchar(200) DEFAULT NULL COMMENT 'File name when downloading', + `sys` varchar(100) NOT NULL COMMENT 'Owning system', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Created time', + `owner` varchar(200) NOT NULL COMMENT 'Resource owner', + `is_expire` TINYINT(1) DEFAULT 0 COMMENT 'Whether expired, 0 means not expired, 1 means expired', + `expire_type` varchar(50) DEFAULT null COMMENT 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time', + `expire_time` varchar(50) DEFAULT null COMMENT 'Expiration time, one day by default', + `max_version` int(20) DEFAULT 10 COMMENT 'The default is 10, which means to keep the latest 10 versions', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Updated time', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_eflag`(`resource_id`, `enable_flag`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Version Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_version` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `file_md5` varchar(32) NOT NULL COMMENT 'Md5 summary of the file', + `version` varchar(20) NOT NULL COMMENT 'Resource version (v plus five digits)', + `size` int(10) NOT NULL COMMENT 'File size', + `start_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `end_byte` BIGINT(20) UNSIGNED NOT NULL DEFAULT 0, + `resource` varchar(2000) NOT NULL COMMENT 'Resource content (file information including path and file name)', + `description` varchar(2000) DEFAULT NULL COMMENT 'description', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Started time', + `end_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Stoped time', + `client_ip` varchar(200) NOT NULL COMMENT 'Client ip', + `updator` varchar(50) DEFAULT NULL COMMENT 'updator', + `enable_flag` tinyint(1) NOT NULL DEFAULT '1' COMMENT 'Status, 1: normal, 0: frozen', + unique key `uniq_rid_version`(`resource_id`, `version`), + PRIMARY KEY (`id`) +); +``` + +### BML Resources Permission Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_permission` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary key', + `resource_id` varchar(50) NOT NULL COMMENT 'Resource uuid', + `permission` varchar(10) NOT NULL COMMENT 'permission', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'created time', + `system` varchar(50) default "dss" COMMENT 'creator', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'updated time', + `updator` varchar(50) NOT NULL COMMENT 'updator', + PRIMARY KEY (`id`) +); +``` + +### BML Resources Task Table +```sql +CREATE TABLE if not exists `linkis_ps_bml_resources_task` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `resource_id` varchar(50) DEFAULT NULL COMMENT 'resource uuid', + `version` varchar(20) DEFAULT NULL COMMENT 'Resource version number of the current operation', + `operation` varchar(20) NOT NULL COMMENT 'Operation type. upload = 0, update = 1', + `state` varchar(20) NOT NULL DEFAULT 'Schduled' COMMENT 'Current status of the task:Schduled, Running, Succeed, Failed,Cancelled', + `submit_user` varchar(20) NOT NULL DEFAULT '' COMMENT 'Job submission user name', + `system` varchar(20) DEFAULT 'dss' COMMENT 'Subsystem name: wtss', + `instance` varchar(128) NOT NULL COMMENT 'Material library example', + `client_ip` varchar(50) DEFAULT NULL COMMENT 'Request IP', + `extra_params` text COMMENT 'Additional key information. Such as the resource IDs and versions that are deleted in batches, and all versions under the resource are deleted', + `err_msg` varchar(2000) DEFAULT NULL COMMENT 'Task failure information.e.getMessage', + `start_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Starting time', + `end_time` datetime DEFAULT NULL COMMENT 'End Time', + `last_update_time` datetime NOT NULL COMMENT 'Last update time', + unique key `uniq_rid_version` (resource_id, version), + PRIMARY KEY (`id`) +); +``` + +### BML Project Table +```sql +create table if not exists linkis_ps_bml_project( + `id` int(10) NOT NULL AUTO_INCREMENT, + `name` varchar(128) DEFAULT NULL, + `system` varchar(64) not null default "dss", + `source` varchar(1024) default null, + `description` varchar(1024) default null, + `creator` varchar(128) not null, + `enabled` tinyint default 1, + `create_time` datetime DEFAULT now(), + unique key `uniq_name` (`name`), +PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The BML Service provides several RPC methods for file management: + +### Resource RPCs + +#### uploadResource +Uploads a resource: +```java +ResourceUploadResult uploadResource(ResourceUploadRequest request) +``` + +#### downloadResource +Downloads a resource: +```java +ResourceContent downloadResource(String resourceId, String version) +``` + +#### deleteResource +Deletes a resource: +```java +void deleteResource(String resourceId) +``` + +#### getResourceInfo +Retrieves resource information: +```java +ResourceInfo getResourceInfo(String resourceId) +``` + +#### listResources +Lists resources for a user: +```java +List listResources(String username) +``` + +### Version RPCs + +#### listVersions +Lists versions of a resource: +```java +List listVersions(String resourceId) +``` + +#### updateResource +Updates a resource with a new version: +```java +ResourceUpdateResult updateResource(ResourceUpdateRequest request) +``` + +#### deleteVersion +Deletes a specific version of a resource: +```java +void deleteVersion(String resourceId, String version) +``` + +#### getVersionInfo +Gets information about a specific version: +```java +ResourceVersion getVersionInfo(String resourceId, String version) +``` + +### Permission RPCs + +#### grantPermission +Grants permission to a user: +```java +void grantPermission(String resourceId, String username, String permission) +``` + +#### checkPermission +Checks if a user has permission: +```java +boolean checkPermission(String resourceId, String username, String permission) +``` + +#### revokePermission +Revokes permission from a user: +```java +void revokePermission(String resourceId, String username) +``` + +#### listPermissions +Lists all permissions for a resource: +```java +List listPermissions(String resourceId) +``` + +### Project RPCs + +#### createProject +Creates a new project: +```java +Project createProject(ProjectCreateRequest request) +``` + +#### deleteProject +Deletes a project: +```java +void deleteProject(Long projectId) +``` + +#### addResourceToProject +Adds a resource to a project: +```java +void addResourceToProject(Long projectId, String resourceId) +``` + +#### removeResourceFromProject +Removes a resource from a project: +```java +void removeResourceFromProject(Long projectId, String resourceId) +``` + +## Dependencies + +- linkis-bml-server +- linkis-mybatis +- linkis-rpc +- linkis-protocol + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- BmlRestfulApi: `linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java` +- BmlProjectRestful: `linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java` +- BMLFsRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/BMLFsRestfulApi.java` + +### MyBatis XML Files +- ResourceMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/ResourceMapper.xml` +- VersionMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/VersionMapper.xml` +- TaskMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/TaskMapper.xml` +- DownloadMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/DownloadMapper.xml` +- BmlProjectMapper: `linkis-public-enhancements/linkis-bml-server/src/main/resources/mapper/common/BmlProjectMapper.xml` diff --git a/.ai/modules/public-enhancements/configuration.md b/.ai/modules/public-enhancements/configuration.md new file mode 100644 index 00000000000..01b1e58352f --- /dev/null +++ b/.ai/modules/public-enhancements/configuration.md @@ -0,0 +1,1264 @@ +# Configuration Service + +The Configuration Service provides centralized configuration management for the Linkis system. + +## Overview + +This service manages configuration properties for all Linkis components, providing a unified interface for configuration retrieval, updates, and validation. + +## Key Components + +### Core Classes +- `LinkisConfigurationApp` - Main application class +- Configuration management +- Configuration validation +- Template management + +### Features +- Global configuration management +- User-specific configuration +- Configuration validation +- Template-based configuration +- Engine-type specific configuration + +## API Interfaces + +### Configuration Retrieval APIs + +#### Get Full Configuration Trees +``` +GET /api/rest_j/v1/configuration/getFullTreesByAppName +``` + +Parameters: +- `engineType`: Engine type (e.g., spark, hive) - optional +- `version`: Engine version - optional +- `creator`: Creator application - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getFullTreesByAppName", + "status": 0, + "message": "success", + "data": { + "fullTree": [ + { + "name": "JVM Configuration", + "description": "JVM configuration for engine", + "settings": [ + { + "key": "wds.linkis.engineconn.java.driver.memory", + "value": "2g", + "defaultValue": "1g", + "description": "Memory size of driver JVM process", + "validateType": "Regex", + "validateRange": "^[0-9]+(\\.?[0-9]*)([gGmMkK])?$", + "level": 1, + "hidden": false, + "advanced": false + } + ] + } + ] + } +} +``` + +#### Get Configuration Category +``` +GET /api/rest_j/v1/configuration/getCategory +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getCategory", + "status": 0, + "message": "success", + "data": { + "Category": [ + { + "categoryId": 1, + "categoryName": "Engine Resource", + "description": "Engine resource configuration" + } + ] + } +} +``` + +#### Get Configuration Item List +``` +GET /api/rest_j/v1/configuration/getItemList +``` + +Parameters: +- `engineType`: Engine type (e.g., spark, hive) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/getItemList", + "status": 0, + "message": "success", + "data": { + "itemList": [ + { + "key": "spark.executor.instances", + "name": "Executor Instances", + "description": "Number of executor instances", + "engineType": "spark", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "boundaryType": 3, + "defaultValue": "1", + "require": 0 + } + ] + } +} +``` + +#### List All Engine Types +``` +GET /api/rest_j/v1/configuration/engineType +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/engineType", + "status": 0, + "message": "success", + "data": { + "engineType": ["spark", "hive", "python"] + } +} +``` + +#### Get Key Value +``` +GET /api/rest_j/v1/configuration/keyvalue +``` + +Parameters: +- `engineType`: Engine type - default "*" +- `version`: Engine version - default "*" +- `creator`: Creator application - default "*" +- `configKey`: Configuration key (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValues": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2g", + "configLabelId": 1 + } + ] + } +} +``` + +#### Get Base Key Value +``` +GET /api/rest_j/v1/configuration/baseKeyValue +``` + +Parameters: +- `engineType`: Engine type - optional +- `key`: Configuration key - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success", + "data": { + "configKeyList": [ + { + "id": 1, + "key": "spark.executor.instances", + "description": "Number of executor instances", + "name": "Executor Instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "engineConnType": "spark", + "isHidden": 0, + "isAdvanced": 0, + "level": 1, + "treeName": "Spark Configuration", + "boundaryType": 3, + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 + } + ], + "totalPage": 10 + } +} +``` + +#### Get User Key Value +``` +GET /api/rest_j/v1/configuration/userKeyValue +``` + +Parameters: +- `engineType`: Engine type - optional +- `key`: Configuration key - optional +- `creator`: Creator application - optional +- `user`: Username - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/userKeyValue", + "status": 0, + "message": "success", + "data": { + "configValueList": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "engineType": "spark", + "key": "spark.executor.instances", + "creator": "IDE", + "user": "testuser" + } + ], + "totalPage": 1 + } +} +``` + +### Configuration Management APIs + +#### Create First Category +``` +POST /api/rest_j/v1/configuration/createFirstCategory +``` + +Request Body: +```json +{ + "categoryName": "Engine Resource", + "description": "Engine resource configuration" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/createFirstCategory", + "status": 0, + "message": "success" +} +``` + +#### Delete Category +``` +POST /api/rest_j/v1/configuration/deleteCategory +``` + +Request Body: +```json +{ + "categoryId": 1 +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/deleteCategory", + "status": 0, + "message": "success" +} +``` + +#### Create Second Category +``` +POST /api/rest_j/v1/configuration/createSecondCategory +``` + +Request Body: +```json +{ + "categoryId": 1, + "engineType": "spark", + "version": "2.4.3", + "description": "Spark 2.4.3 configuration" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/createSecondCategory", + "status": 0, + "message": "success" +} +``` + +#### Update Category Info +``` +POST /api/rest_j/v1/configuration/updateCategoryInfo +``` + +Request Body: +```json +{ + "categoryId": 1, + "description": "Updated description" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/updateCategoryInfo", + "status": 0, + "message": "success" +} +``` + +#### Save Full Tree +``` +POST /api/rest_j/v1/configuration/saveFullTree +``` + +Request Body: +```json +{ + "creator": "IDE", + "engineType": "spark-2.4.3", + "fullTree": [ + { + "name": "JVM Configuration", + "description": "JVM configuration for engine", + "settings": [ + { + "key": "wds.linkis.engineconn.java.driver.memory", + "value": "2g", + "defaultValue": "1g", + "description": "Memory size of driver JVM process", + "validateType": "Regex", + "validateRange": "^[0-9]+(\\.?[0-9]*)([gGmMkK])?$", + "level": 1, + "hidden": false, + "advanced": false + } + ] + } + ] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/saveFullTree", + "status": 0, + "message": "success" +} +``` + +#### Save Key Value +``` +POST /api/rest_j/v1/configuration/keyvalue +``` + +Request Body: +```json +{ + "engineType": "spark", + "version": "2.4.3", + "creator": "IDE", + "configKey": "spark.executor.instances", + "configValue": "2", + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValue": { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1 + } + } +} +``` + +#### Delete Key Value +``` +DELETE /api/rest_j/v1/configuration/keyvalue +``` + +Request Body: +```json +{ + "engineType": "spark", + "version": "2.4.3", + "creator": "IDE", + "configKey": "spark.executor.instances" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/keyvalue", + "status": 0, + "message": "success", + "data": { + "configValues": [ + { + "id": 1, + "configKeyId": 1, + "configValue": "2", + "configLabelId": 1 + } + ] + } +} +``` + +#### Save Base Key Value +``` +POST /api/rest_j/v1/configuration/baseKeyValue +``` + +Request Body: +```json +{ + "key": "spark.executor.instances", + "name": "Executor Instances", + "description": "Number of executor instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "boundaryType": 3, + "treeName": "Spark Configuration", + "engineType": "spark", + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success", + "data": { + "configKey": { + "id": 1, + "key": "spark.executor.instances", + "description": "Number of executor instances", + "name": "Executor Instances", + "defaultValue": "1", + "validateType": "NumInterval", + "validateRange": "[1,20]", + "engineConnType": "spark", + "isHidden": 0, + "isAdvanced": 0, + "level": 1, + "treeName": "Spark Configuration", + "boundaryType": 3, + "enDescription": "Number of executor instances", + "enName": "Executor Instances", + "enTreeName": "Spark Configuration", + "templateRequired": 0 + } + } +} +``` + +#### Delete Base Key Value +``` +DELETE /api/rest_j/v1/configuration/baseKeyValue +``` + +Parameters: +- `id`: Configuration key ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/baseKeyValue", + "status": 0, + "message": "success" +} +``` + +### Template Management APIs + +#### Update Key Mapping +``` +POST /api/rest_j/v1/configuration/template/updateKeyMapping +``` + +Request Body: +```json +{ + "templateUid": "template-uuid", + "templateName": "Spark Template", + "engineType": "spark", + "operator": "admin", + "isFullMode": true, + "itemList": [ + { + "keyId": 1, + "maxValue": "10", + "minValue": "1" + } + ] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/updateKeyMapping", + "status": 0, + "message": "success" +} +``` + +#### Query Key Info List +``` +POST /api/rest_j/v1/configuration/template/queryKeyInfoList +``` + +Request Body: +```json +{ + "templateUidList": ["template-uuid-1", "template-uuid-2"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/queryKeyInfoList", + "status": 0, + "message": "success", + "data": { + "list": [ + { + "templateName": "Spark Template", + "engineType": "spark", + "configKey": "spark.executor.instances", + "maxValue": "10", + "minValue": "1" + } + ] + } +} +``` + +#### Apply Configuration Template +``` +POST /api/rest_j/v1/configuration/template/apply +``` + +Request Body: +```json +{ + "templateUid": "template-uuid", + "application": "IDE", + "engineType": "spark", + "engineVersion": "2.4.3", + "operator": "admin", + "userList": ["user1", "user2"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/apply", + "status": 0, + "message": "success", + "data": { + "success": true, + "failedUsers": [] + } +} +``` + +#### Encrypt Datasource Password +``` +GET /api/rest_j/v1/configuration/template/encrypt +``` + +Parameters: +- `isEncrypt`: Encrypt flag - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/template/encrypt", + "status": 0, + "message": "success" +} +``` + +### Tenant Configuration APIs + +#### Create Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/create-tenant +``` + +Request Body: +```json +{ + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1", + "desc": "Test tenant", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/create-tenant", + "status": 0, + "message": "success" +} +``` + +#### Update Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/update-tenant +``` + +Request Body: +```json +{ + "id": 1, + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1-updated", + "desc": "Updated tenant", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/update-tenant", + "status": 0, + "message": "success" +} +``` + +#### Delete Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/delete-tenant +``` + +Parameters: +- `id`: Tenant ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/delete-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Tenant List +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-tenant-list +``` + +Parameters: +- `user`: Username - optional +- `creator`: Creator application - optional +- `tenantValue`: Tenant value - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-tenant-list", + "status": 0, + "message": "success", + "data": { + "tenantList": [ + { + "id": 1, + "user": "testuser", + "creator": "IDE", + "tenantValue": "tenant1", + "desc": "Test tenant", + "bussinessUser": "admin", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Check User Creator +``` +GET /api/rest_j/v1/configuration/tenant-mapping/check-user-creator +``` + +Parameters: +- `user`: Username (required) +- `creator`: Creator application (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/check-user-creator", + "status": 0, + "message": "success", + "data": { + "exist": true + } +} +``` + +#### Save Department Tenant +``` +POST /api/rest_j/v1/configuration/tenant-mapping/save-department-tenant +``` + +Request Body: +```json +{ + "creator": "IDE", + "department": "Engineering", + "departmentId": "dept1", + "tenantValue": "tenant1", + "createBy": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/save-department-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Department Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-department-tenant +``` + +Parameters: +- `departmentId`: Department ID - optional +- `department`: Department name - optional +- `creator`: Creator application - optional +- `tenantValue`: Tenant value - optional +- `pageNow`: Page number - default 1 +- `pageSize`: Page size - default 20 + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-department-tenant", + "status": 0, + "message": "success", + "data": { + "tenantList": [ + { + "id": 1, + "creator": "IDE", + "department": "Engineering", + "departmentId": "dept1", + "tenantValue": "tenant1", + "createBy": "admin", + "isValid": "Y", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Delete Department Tenant +``` +GET /api/rest_j/v1/configuration/tenant-mapping/delete-department-tenant +``` + +Parameters: +- `id`: Department tenant ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/delete-department-tenant", + "status": 0, + "message": "success" +} +``` + +#### Query Department List +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-department +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-department", + "status": 0, + "message": "success", + "data": { + "departmentList": ["Engineering", "Marketing", "Sales"] + } +} +``` + +#### Query User Department +``` +GET /api/rest_j/v1/configuration/tenant-mapping/query-user-department +``` + +Parameters: +- `username`: Username (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/tenant-mapping/query-user-department", + "status": 0, + "message": "success", + "data": { + "department": "Engineering" + } +} +``` + +### User IP Configuration APIs + +#### Create User IP +``` +POST /api/rest_j/v1/configuration/user-ip-mapping/create-user-ip +``` + +Request Body: +```json +{ + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2", + "desc": "Allowed IPs for testuser", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/create-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Update User IP +``` +POST /api/rest_j/v1/configuration/user-ip-mapping/update-user-ip +``` + +Request Body: +```json +{ + "id": 1, + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2,192.168.1.3", + "desc": "Updated allowed IPs for testuser", + "bussinessUser": "admin" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/update-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Delete User IP +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/delete-user-ip +``` + +Parameters: +- `id`: User IP ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/delete-user-ip", + "status": 0, + "message": "success" +} +``` + +#### Query User IP List +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/query-user-ip-list +``` + +Parameters: +- `user`: Username - optional +- `creator`: Creator application - optional +- `pageNow`: Page number (required) +- `pageSize`: Page size (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/query-user-ip-list", + "status": 0, + "message": "success", + "data": { + "userIpList": [ + { + "id": 1, + "user": "testuser", + "creator": "IDE", + "ipList": "192.168.1.1,192.168.1.2", + "desc": "Allowed IPs for testuser", + "bussinessUser": "admin", + "createTime": "2023-01-01 12:00:00", + "updateTime": "2023-01-01 12:00:00" + } + ], + "totalPage": 1 + } +} +``` + +#### Check User Creator +``` +GET /api/rest_j/v1/configuration/user-ip-mapping/check-user-creator +``` + +Parameters: +- `user`: Username (required) +- `creator`: Creator application (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/configuration/user-ip-mapping/check-user-creator", + "status": 0, + "message": "success", + "data": { + "exist": true + } +} +``` + +## Database Table Structures + +The Configuration Service uses the following database tables from linkis_ddl.sql: + +### Configuration Config Key Table +```sql +CREATE TABLE `linkis_ps_configuration_config_key`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) +); +``` + +### Configuration Key Engine Relation Table +```sql +CREATE TABLE `linkis_ps_configuration_key_engine_relation`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `config_key_id` bigint(20) NOT NULL COMMENT 'config key id', + `engine_type_label_id` bigint(20) NOT NULL COMMENT 'engine label id', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `engine_type_label_id`) +); +``` + +### Configuration Config Value Table +```sql +CREATE TABLE `linkis_ps_configuration_config_value`( + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `config_key_id` bigint(20), + `config_value` varchar(500), + `config_label_id`int(20), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `config_label_id`) +); +``` + +### Configuration Category Table +```sql +CREATE TABLE `linkis_ps_configuration_category` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `label_id` int(20) NOT NULL, + `level` int(20) NOT NULL, + `description` varchar(200), + `tag` varchar(200), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_label_id` (`label_id`) +); +``` + +### Tenant Label Config Table +```sql +CREATE TABLE `linkis_cg_tenant_label_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `user` varchar(50) COLLATE utf8_bin NOT NULL, + `creator` varchar(50) COLLATE utf8_bin NOT NULL, + `tenant_value` varchar(128) COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `desc` varchar(100) COLLATE utf8_bin NOT NULL, + `bussiness_user` varchar(50) COLLATE utf8_bin NOT NULL, + `is_valid` varchar(1) COLLATE utf8_bin NOT NULL DEFAULT 'Y' COMMENT 'is valid', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_user_creator` (`user`,`creator`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### User IP Config Table +```sql +CREATE TABLE `linkis_cg_user_ip_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT, + `user` varchar(50) COLLATE utf8_bin NOT NULL, + `creator` varchar(50) COLLATE utf8_bin NOT NULL, + `ip_list` text COLLATE utf8_bin NOT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `desc` varchar(100) COLLATE utf8_bin NOT NULL, + `bussiness_user` varchar(50) COLLATE utf8_bin NOT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_user_creator` (`user`,`creator`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Tenant Department Config Table +```sql +CREATE TABLE `linkis_cg_tenant_department_config` ( + `id` int(20) NOT NULL AUTO_INCREMENT COMMENT 'ID', + `creator` varchar(50) COLLATE utf8_bin NOT NULL COMMENT '应用', + `department` varchar(64) COLLATE utf8_bin NOT NULL COMMENT '部门名称', + `department_id` varchar(16) COLLATE utf8_bin NOT NULL COMMENT '部门ID', + `tenant_value` varchar(128) COLLATE utf8_bin NOT NULL COMMENT '部门租户标签', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '更新时间', + `create_by` varchar(50) COLLATE utf8_bin NOT NULL COMMENT '创建用户', + `is_valid` varchar(1) COLLATE utf8_bin NOT NULL DEFAULT 'Y' COMMENT '是否有效', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_creator_department` (`creator`,`department`) +) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Configuration Template Config Key Table +```sql +CREATE TABLE `linkis_ps_configuration_template_config_key` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `template_name` VARCHAR(200) NOT NULL COMMENT '配置模板名称 冗余存储', + `template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值（预留）', + `validate_range` VARCHAR(50) NULL DEFAULT NULL COMMENT '校验正则(预留) ', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_tid_kid` (`template_uuid`, `key_id`), + UNIQUE INDEX `uniq_tname_kid` (`template_uuid`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Configuration Key Limit For User Table +```sql +CREATE TABLE `linkis_ps_configuration_key_limit_for_user` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `user_name` VARCHAR(50) NOT NULL COMMENT '用户名', + `combined_label_value` VARCHAR(128) NOT NULL COMMENT '组合标签 combined_userCreator_engineType 如 hadoop-IDE,spark-2.4.3', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值（预留）', + `latest_update_template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_com_label_kid` (`combined_label_value`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The Configuration Service provides several RPC methods for configuration management: + +### Configuration RPCs + +#### getGlobalConfiguration +Retrieves global configuration properties: +```java +Map getGlobalConfiguration() +``` + +#### getUserConfiguration +Retrieves user-specific configuration: +```java +Map getUserConfiguration(String user) +``` + +#### updateConfiguration +Updates configuration properties: +```java +void updateConfiguration(String user, Map configurations) +``` + +#### getConfigurationTemplate +Retrieves configuration template for an engine: +```java +ConfigurationTemplate getConfigurationTemplate(String engineType) +``` + +#### validateConfiguration +Validates configuration properties: +```java +ConfigurationValidationResult validateConfiguration(String key, String value) +``` + +#### getEngineConfiguration +Retrieves engine-specific configuration: +```java +Map getEngineConfiguration(String engineType, String version, String user) +``` + +#### updateEngineConfiguration +Updates engine-specific configuration: +```java +void updateEngineConfiguration(String engineType, String version, String user, Map configurations) +``` + +#### listConfigurationKeys +Lists all configuration keys for an engine type: +```java +List listConfigurationKeys(String engineType) +``` + +### Category RPCs + +#### getCategoryConfiguration +Retrieves configuration for a category: +```java +Map getCategoryConfiguration(String category) +``` + +#### updateCategoryConfiguration +Updates configuration for a category: +```java +void updateCategoryConfiguration(String category, Map configurations) +``` + +## Dependencies + +- linkis-mybatis +- linkis-rpc +- linkis-manager-common +- linkis-httpclient +- linkis-label-common + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ConfigurationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApi.java` +- TenantConfigrationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/TenantConfigrationRestfulApi.java` +- UserIpConfigrationRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/UserIpConfigrationRestfulApi.java` +- TemplateRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/TemplateRestfulApi.java` +- AcrossClusterRuleRestfulApi: `linkis-public-enhancements/linkis-configuration/src/main/java/org/apache/linkis/configuration/restful/api/AcrossClusterRuleRestfulApi.java` +- ConfigurationTemplateRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/ConfigurationTemplateRestfulApi.java` + +### MyBatis XML Files +- ConfigMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/ConfigMapper.xml` +- LabelMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/LabelMapper.xml` +- UserTenantMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/UserTenantMapper.xml` +- DepartmentTenantMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/DepartmentTenantMapper.xml` +- UserIpMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/UserIpMapper.xml` +- DepartmentMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/DepartmentMapper.xml` +- AcrossClusterRuleMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/AcrossClusterRuleMapper.xml` +- TemplateConfigKeyMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/TemplateConfigKeyMapper.xml` +- ConfigKeyLimitForUserMapper: `linkis-public-enhancements/linkis-configuration/src/main/resources/mapper/common/ConfigKeyLimitForUserMapper.xml` \ No newline at end of file diff --git a/.ai/modules/public-enhancements/context.md b/.ai/modules/public-enhancements/context.md new file mode 100644 index 00000000000..d1211c0a869 --- /dev/null +++ b/.ai/modules/public-enhancements/context.md @@ -0,0 +1,962 @@ +# Context Service + +The Context Service provides context and variable sharing capabilities for the Linkis system. + +## Overview + +This service manages context information and variable sharing across different engines and applications in the Linkis system. + +## Key Components + +### Core Classes +- `LinkisCSApplication` - Main application class +- Context management +- Variable sharing +- Context persistence + +### Features +- Cross-engine context sharing +- Variable management +- Context persistence +- Context versioning + +## API Interfaces + +### Context ID APIs + +#### Create Context ID +``` +POST /api/rest_j/v1/contextservice/createContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/createContextID", + "status": 0, + "message": "success", + "data": { + "contextId": "context-12345" + } +} +``` + +#### Get Context ID +``` +GET /api/rest_j/v1/contextservice/getContextID +``` + +Parameters: +- `contextId`: Context ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getContextID", + "status": 0, + "message": "success", + "data": { + "contextId": { + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Update Context ID +``` +POST /api/rest_j/v1/contextservice/updateContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "updated-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/updateContextID", + "status": 0, + "message": "success" +} +``` + +#### Reset Context ID +``` +POST /api/rest_j/v1/contextservice/resetContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/resetContextID", + "status": 0, + "message": "success" +} +``` + +#### Remove Context ID +``` +POST /api/rest_j/v1/contextservice/removeContextID +``` + +Request Body: +```json +{ + "contextId": "context-12345" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeContextID", + "status": 0, + "message": "success" +} +``` + +#### Search Context ID By Time +``` +GET /api/rest_j/v1/contextservice/searchContextIDByTime +``` + +Parameters: +- `createTimeStart`: Create time start - optional +- `createTimeEnd`: Create time end - optional +- `updateTimeStart`: Update time start - optional +- `updateTimeEnd`: Update time end - optional +- `accessTimeStart`: Access time start - optional +- `accessTimeEnd`: Access time end - optional +- `pageNow`: Page number - optional, default 1 +- `pageSize`: Page size - optional, default 100 + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchContextIDByTime", + "status": 0, + "message": "success", + "data": { + "contextIds": [ + { + "contextId": "context-12345", + "user": "testuser", + "application": "IDE", + "source": "test-source", + "expireType": "NORMAL", + "expireTime": "2023-12-31 23:59:59", + "instance": "instance-1", + "backupInstance": "backup-instance-1", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +### Context Value APIs + +#### Get Context Value +``` +POST /api/rest_j/v1/contextservice/getContextValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getContextValue", + "status": 0, + "message": "success", + "data": { + "contextValue": { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } + } +} +``` + +#### Search Context Value +``` +POST /api/rest_j/v1/contextservice/searchContextValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "condition": { + "key": "test-key", + "contextType": "ENV" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchContextValue", + "status": 0, + "message": "success", + "data": { + "contextKeyValue": [ + { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } + ] + } +} +``` + +#### Set Value By Key +``` +POST /api/rest_j/v1/contextservice/setValueByKey +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + }, + "contextValue": { + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/setValueByKey", + "status": 0, + "message": "success" +} +``` + +#### Set Value +``` +POST /api/rest_j/v1/contextservice/setValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyValue": { + "key": "test-key", + "value": "test-value", + "contextType": "ENV", + "scope": "PRIVATE", + "props": { + "prop1": "value1" + } + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/setValue", + "status": 0, + "message": "success" +} +``` + +#### Reset Value +``` +POST /api/rest_j/v1/contextservice/resetValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/resetValue", + "status": 0, + "message": "success" +} +``` + +#### Remove Value +``` +POST /api/rest_j/v1/contextservice/removeValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeValue", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value +``` +POST /api/rest_j/v1/contextservice/removeAllValue +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValue", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key Prefix And Context Type +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyType": "ENV", + "keyPrefix": "test" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyPrefixAndContextType", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key And Context Type +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyAndContextType +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKeyType": "ENV", + "contextKey": "test-key" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyAndContextType", + "status": 0, + "message": "success" +} +``` + +#### Remove All Value By Key Prefix +``` +POST /api/rest_j/v1/contextservice/removeAllValueByKeyPrefix +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "keyPrefix": "test" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeAllValueByKeyPrefix", + "status": 0, + "message": "success" +} +``` + +#### Clear All Context By ID +``` +POST /api/rest_j/v1/contextservice/clearAllContextByID +``` + +Request Body: +```json +{ + "idList": ["context-12345", "context-67890"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/clearAllContextByID", + "status": 0, + "message": "success", + "data": { + "num": 2 + } +} +``` + +#### Clear All Context By Time +``` +POST /api/rest_j/v1/contextservice/clearAllContextByTime +``` + +Request Body: +```json +{ + "createTimeStart": "2023-01-01 00:00:00", + "createTimeEnd": "2023-12-31 23:59:59", + "updateTimeStart": "2023-01-01 00:00:00", + "updateTimeEnd": "2023-12-31 23:59:59" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/clearAllContextByTime", + "status": 0, + "message": "success", + "data": { + "num": 5 + } +} +``` + +### Context History APIs + +#### Create History +``` +POST /api/rest_j/v1/contextservice/createHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextHistory": { + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/createHistory", + "status": 0, + "message": "success" +} +``` + +#### Remove History +``` +POST /api/rest_j/v1/contextservice/removeHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextHistory": { + "source": "test-source" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/removeHistory", + "status": 0, + "message": "success" +} +``` + +#### Get Histories +``` +POST /api/rest_j/v1/contextservice/getHistories +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getHistories", + "status": 0, + "message": "success", + "data": { + "contextHistory": [ + { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +#### Get History +``` +POST /api/rest_j/v1/contextservice/getHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/getHistory", + "status": 0, + "message": "success", + "data": { + "contextHistory": { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Search History +``` +POST /api/rest_j/v1/contextservice/searchHistory +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "keywords": ["test", "key"] +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/searchHistory", + "status": 0, + "message": "success", + "data": { + "contextHistory": [ + { + "id": 1, + "contextId": 12345, + "source": "test-source", + "contextType": "ENV", + "historyJson": "{\"key\":\"test-key\",\"value\":\"test-value\"}", + "keyword": "test", + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00", + "accessTime": "2023-01-01 12:00:00" + } + ] + } +} +``` + +### Context Listener APIs + +#### On Bind ID Listener +``` +POST /api/rest_j/v1/contextservice/onBindIDListener +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/onBindIDListener", + "status": 0, + "message": "success", + "data": { + "listener": null + } +} +``` + +#### On Bind Key Listener +``` +POST /api/rest_j/v1/contextservice/onBindKeyListener +``` + +Request Body: +```json +{ + "contextID": { + "contextId": "context-12345" + }, + "contextKey": { + "key": "test-key" + }, + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/onBindKeyListener", + "status": 0, + "message": "success", + "data": { + "listener": null + } +} +``` + +#### Heartbeat +``` +POST /api/rest_j/v1/contextservice/heartbeat +``` + +Request Body: +```json +{ + "source": "test-source" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/contextservice/heartbeat", + "status": 0, + "message": "success", + "data": { + "ContextKeyValueBean": null + } +} +``` + +## Database Table Structures + +The Context Service manages the following database tables from linkis_ddl.sql: + +### Context Map Table +```sql +CREATE TABLE `linkis_ps_cs_context_map` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `key` varchar(128) DEFAULT NULL, + `context_scope` varchar(32) DEFAULT NULL, + `context_type` varchar(32) DEFAULT NULL, + `props` text, + `value` mediumtext, + `context_id` int(11) DEFAULT NULL, + `keywords` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`), + KEY `idx_keywords` (`keywords`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context Map Listener Table +```sql +CREATE TABLE `linkis_ps_cs_context_map_listener` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `listener_source` varchar(255) DEFAULT NULL, + `key_id` int(11) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context History Table +```sql +CREATE TABLE `linkis_ps_cs_context_history` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `context_id` int(11) DEFAULT NULL, + `source` text, + `context_type` varchar(32) DEFAULT NULL, + `history_json` text, + `keyword` varchar(255) DEFAULT NULL, + PRIMARY KEY (`id`), + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + KEY `idx_keyword` (`keyword`(191)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context ID Table +```sql +CREATE TABLE `linkis_ps_cs_context_id` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `user` varchar(32) DEFAULT NULL, + `application` varchar(32) DEFAULT NULL, + `source` varchar(255) DEFAULT NULL, + `expire_type` varchar(32) DEFAULT NULL, + `expire_time` datetime DEFAULT NULL, + `instance` varchar(128) DEFAULT NULL, + `backup_instance` varchar(255) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`), + KEY `idx_instance` (`instance`(128)), + KEY `idx_backup_instance` (`backup_instance`(191)), + KEY `idx_instance_bin` (`instance`(128),`backup_instance`(128)) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Context Listener Table +```sql +CREATE TABLE `linkis_ps_cs_context_listener` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `listener_source` varchar(255) DEFAULT NULL, + `context_id` int(11) DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'update unix timestamp', + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'last access time', + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The Context Service provides several RPC methods for context management: + +### Context RPCs + +#### createContext +Creates a new context: +```java +String createContext(ContextCreationRequest request) +``` + +#### getContext +Retrieves a context: +```java +ContextInfo getContext(String contextId) +``` + +#### deleteContext +Deletes a context: +```java +void deleteContext(String contextId) +``` + +### Variable RPCs + +#### setVariable +Sets a variable in a context: +```java +void setVariable(String contextId, String key, Object value, String valueType) +``` + +#### getVariable +Retrieves a variable from a context: +```java +Object getVariable(String contextId, String key) +``` + +#### removeVariable +Removes a variable from a context: +```java +void removeVariable(String contextId, String key) +``` + +### History RPCs + +#### getContextHistory +Retrieves context history: +```java +List getContextHistory(String contextId) +``` + +#### clearContextHistory +Clears context history: +```java +void clearContextHistory(String contextId) +``` + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- ContextRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextRestfulApi.java` +- ContextIDRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextIDRestfulApi.java` +- ContextHistoryRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextHistoryRestfulApi.java` +- ContextListenerRestfulApi: `linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/restful/ContextListenerRestfulApi.java` + +### MyBatis XML Files +- ContextMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextMapper.xml` +- ContextIDMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextIDMapper.xml` +- ContextHistoryMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextHistoryMapper.xml` +- ContextListenerMapper: `linkis-public-enhancements/linkis-cs-server/src/main/resources/mapper/ContextListenerMapper.xml` + +## Dependencies + +- linkis-cs-server +- linkis-rpc +- linkis-protocol \ No newline at end of file diff --git a/.ai/modules/public-enhancements/datasource.md b/.ai/modules/public-enhancements/datasource.md new file mode 100644 index 00000000000..6e584cc2cb1 --- /dev/null +++ b/.ai/modules/public-enhancements/datasource.md @@ -0,0 +1,2032 @@ +# DataSource Service + +The DataSource Service provides data source management capabilities for the Linkis system. + +## Overview + +This service manages data source connections, metadata, and provides unified access to various data sources. + +## Key Components + +### Core Classes +- `LinkisDataSourceApplication` - Main application class +- Data source management +- Metadata querying +- Connection testing + +### Features +- Data source registration and management +- Metadata querying +- Connection testing and validation +- Data source versioning +- Access control + +## API Interfaces + +### Data Source Type APIs + +#### Get All Data Source Types +``` +GET /api/rest_j/v1/data-source-manager/type/all +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/type/all", + "status": 0, + "message": "success", + "data": { + "typeList": [ + { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3 + } + ] + } +} +``` + +#### Get Key Definitions By Type +``` +GET /api/rest_j/v1/data-source-manager/key-define/type/{typeId} +``` + +Parameters: +- `typeId`: Data source type ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/key-define/type/{typeId}", + "status": 0, + "message": "success", + "data": { + "keyDefine": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null + } + ] + } +} +``` + +#### Get Key Definitions By Type Name +``` +GET /api/rest_j/v1/data-source-manager/key-define/{typeName} +``` + +Parameters: +- `typeName`: Data source type name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/key-define/{typeName}", + "status": 0, + "message": "success", + "data": { + "keyDefine": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null + } + ] + } +} +``` + +### Data Source Management APIs + +#### Insert Data Source Info (JSON) +``` +POST /api/rest_j/v1/data-source-manager/info/json +``` + +Request Body: +```json +{ + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createSystem": "linkis", + "labels": [ + { + "labelKey": "env", + "labelValue": "production" + } + ], + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/json", + "status": 0, + "message": "success", + "data": { + "insertId": 12345 + } +} +``` + +#### Insert Data Source (JSON Create) +``` +POST /api/rest_j/v1/data-source-manager/info/json/create +``` + +Request Body: +```json +{ + "createUser": "testuser", + "dataSourceTypeName": "starrocks", + "connectParams": { + "host": "localhost", + "port": "9030", + "driverClassName": "com.mysql.jdbc.Driver", + "username": "user", + "password": "password" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/json/create", + "status": 0, + "message": "success", + "data": { + "datasource": { + "id": 12345, + "dataSourceName": "starrocks_testuser_20230101120000", + "dataSourceDesc": null, + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": null, + "parameter": "{\"host\":\"localhost\",\"port\":\"9030\",\"driverClassName\":\"com.mysql.jdbc.Driver\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": null, + "labels": null, + "versionId": null, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Update Data Source Info (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/info/{dataSourceId}/json +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Request Body: +```json +{ + "dataSourceName": "mysql-ds", + "dataSourceDesc": "Updated MySQL Data Source", + "dataSourceTypeId": 1, + "createSystem": "linkis", + "createTime": "1650426189000", + "createUser": "testuser", + "labels": [ + { + "labelKey": "env", + "labelValue": "production" + } + ], + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword", + "database": "test" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/json", + "status": 0, + "message": "success", + "data": { + "updateId": 12345 + } +} +``` + +#### Insert Data Source Parameter (JSON) +``` +POST /api/rest_j/v1/data-source-manager/parameter/{dataSourceId}/json +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Request Body: +```json +{ + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + }, + "comment": "Initial version" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/parameter/{dataSourceId}/json", + "status": 0, + "message": "success", + "data": { + "version": 1 + } +} +``` + +#### Get Data Source Info By ID +``` +GET /api/rest_j/v1/data-source-manager/info/{dataSourceId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Data Source Info By Name +``` +GET /api/rest_j/v1/data-source-manager/info/name/{dataSourceName} +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/name/{dataSourceName}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Published Data Source Info By Name +``` +GET /api/rest_j/v1/data-source-manager/publishedInfo/name/{dataSourceName} +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publishedInfo/name/{dataSourceName}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Published Data Source Info By Type Name, User, IP and Port +``` +GET /api/rest_j/v1/data-source-manager/publishedInfo/{datasourceTypeName}/{datasourceUser}/{ip}/{port} +``` + +Parameters: +- `datasourceTypeName`: Data source type name (required) +- `datasourceUser`: Data source user (required) +- `ip`: IP address (required) +- `port`: Port (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publishedInfo/{datasourceTypeName}/{datasourceUser}/{ip}/{port}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Data Source Info By ID and Version +``` +GET /api/rest_j/v1/data-source-manager/info/{dataSourceId}/{version} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `version`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/{version}", + "status": 0, + "message": "success", + "data": { + "info": { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + } +} +``` + +#### Get Version List +``` +GET /api/rest_j/v1/data-source-manager/{dataSourceId}/versions +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/versions", + "status": 0, + "message": "success", + "data": { + "versions": [ + { + "versionId": 1, + "dataSourceId": 12345, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "comment": "Initial version", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser" + } + ] + } +} +``` + +#### Publish Data Source By ID +``` +POST /api/rest_j/v1/data-source-manager/publish/{dataSourceId}/{versionId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `versionId`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/publish/{dataSourceId}/{versionId}", + "status": 0, + "message": "success" +} +``` + +#### Remove Data Source +``` +DELETE /api/rest_j/v1/data-source-manager/info/delete/{dataSourceId} +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/delete/{dataSourceId}", + "status": 0, + "message": "success", + "data": { + "removeId": 12345 + } +} +``` + +#### Expire Data Source +``` +PUT /api/rest_j/v1/data-source-manager/info/{dataSourceId}/expire +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/{dataSourceId}/expire", + "status": 0, + "message": "success", + "data": { + "expireId": 12345 + } +} +``` + +#### Get Connect Params By Data Source ID +``` +GET /api/rest_j/v1/data-source-manager/{dataSourceId}/connect-params +``` + +Parameters: +- `dataSourceId`: Data source ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/connect-params", + "status": 0, + "message": "success", + "data": { + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } + } +} +``` + +#### Get Connect Params By Data Source Name +``` +GET /api/rest_j/v1/data-source-manager/name/{dataSourceName}/connect-params +``` + +Parameters: +- `dataSourceName`: Data source name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/name/{dataSourceName}/connect-params", + "status": 0, + "message": "success", + "data": { + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password", + "database": "test" + } + } +} +``` + +#### Connect Data Source +``` +PUT /api/rest_j/v1/data-source-manager/{dataSourceId}/{version}/op/connect +``` + +Parameters: +- `dataSourceId`: Data source ID (required) +- `version`: Version ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/{dataSourceId}/{version}/op/connect", + "status": 0, + "message": "success", + "data": { + "ok": true + } +} +``` + +#### Query Data Source By IDs +``` +GET /api/rest_j/v1/data-source-manager/info/ids +``` + +Parameters: +- `ids`: JSON array of data source IDs (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info/ids", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + ], + "totalPage": 1 + } +} +``` + +#### Query Data Source +``` +GET /api/rest_j/v1/data-source-manager/info +``` + +Parameters: +- `system`: Create system - optional +- `name`: Data source name - optional +- `typeId`: Data source type ID - optional +- `identifies`: Identifies - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/info", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "dataSourceName": "mysql-ds", + "dataSourceDesc": "MySQL Data Source", + "dataSourceTypeId": 1, + "createIdentify": null, + "createSystem": "linkis", + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\",\"database\":\"test\"}", + "createTime": "2023-01-01 12:00:00", + "modifyTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyUser": "testuser", + "labels": "[{\"labelKey\":\"env\",\"labelValue\":\"production\"}]", + "versionId": 1, + "expire": false, + "publishedVersionId": 1 + } + ], + "totalPage": 1 + } +} +``` + +### Data Source Environment APIs + +#### Insert Data Source Environment (JSON) +``` +POST /api/rest_j/v1/data-source-manager/env/json +``` + +Request Body: +```json +{ + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json", + "status": 0, + "message": "success", + "data": { + "insertId": 12345 + } +} +``` + +#### Insert Data Source Environment Batch (JSON) +``` +POST /api/rest_j/v1/data-source-manager/env/json/batch +``` + +Request Body: +```json +[ + { + "envName": "test-env-1", + "envDesc": "Test Environment 1", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "password" + } + }, + { + "envName": "test-env-2", + "envDesc": "Test Environment 2", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3307", + "username": "user", + "password": "password" + } + } +] +``` + +Parameters: +- `system`: System name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json/batch", + "status": 0, + "message": "success", + "data": { + "envs": [ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Test Environment 1", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + }, + { + "id": 12346, + "envName": "test-env-2", + "envDesc": "Test Environment 2", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3307\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Update Data Source Environment Batch (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/env/json/batch +``` + +Request Body: +```json +[ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Updated Test Environment 1", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword" + } + } +] +``` + +Parameters: +- `system`: System name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/json/batch", + "status": 0, + "message": "success", + "data": { + "envs": [ + { + "id": 12345, + "envName": "test-env-1", + "envDesc": "Updated Test Environment 1", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"newpassword\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Get All Environment List By Data Source Type +``` +GET /api/rest_j/v1/data-source-manager/env-list/all/type/{typeId} +``` + +Parameters: +- `typeId`: Data source type ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env-list/all/type/{typeId}", + "status": 0, + "message": "success", + "data": { + "envList": [ + { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +#### Get Environment Entity By ID +``` +GET /api/rest_j/v1/data-source-manager/env/{envId} +``` + +Parameters: +- `envId`: Environment ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}", + "status": 0, + "message": "success", + "data": { + "env": { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + } +} +``` + +#### Remove Environment Entity +``` +DELETE /api/rest_j/v1/data-source-manager/env/{envId} +``` + +Parameters: +- `envId`: Environment ID (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}", + "status": 0, + "message": "success", + "data": { + "removeId": 12345 + } +} +``` + +#### Update Data Source Environment (JSON) +``` +PUT /api/rest_j/v1/data-source-manager/env/{envId}/json +``` + +Parameters: +- `envId`: Environment ID (required) + +Request Body: +```json +{ + "envName": "test-env", + "envDesc": "Updated Test Environment", + "dataSourceTypeId": 1, + "connectParams": { + "host": "localhost", + "port": "3306", + "username": "user", + "password": "newpassword" + } +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env/{envId}/json", + "status": 0, + "message": "success", + "data": { + "updateId": 12345 + } +} +``` + +#### Query Data Source Environment +``` +GET /api/rest_j/v1/data-source-manager/env +``` + +Parameters: +- `name`: Environment name - optional +- `typeId`: Data source type ID - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "/api/rest_j/v1/data-source-manager/env", + "status": 0, + "message": "success", + "data": { + "queryList": [ + { + "id": 12345, + "envName": "test-env", + "envDesc": "Test Environment", + "dataSourceTypeId": 1, + "parameter": "{\"host\":\"localhost\",\"port\":\"3306\",\"username\":\"user\",\"password\":\"password\"}", + "createTime": "2023-01-01 12:00:00", + "createUser": "testuser", + "modifyTime": "2023-01-01 12:00:00", + "modifyUser": "testuser" + } + ] + } +} +``` + +### Metadata Query APIs + +#### Query Database Info +``` +GET /api/rest_j/v1/datasource/dbs +``` + +Parameters: +- `permission`: Permission filter - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/dbs", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "permission": "READ" + } + ] + } +} +``` + +#### Query Partition Exists +``` +GET /api/rest_j/v1/datasource/partitionExists +``` + +Parameters: +- `database`: Database name (required) +- `table`: Table name (required) +- `partition`: Partition name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/partitionExists", + "status": 0, + "message": "success", + "data": { + "partitionExists": true + } +} +``` + +#### Query Databases With Tables +``` +GET /api/rest_j/v1/datasource/all +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/all", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "tables": [ + { + "name": "test_table" + } + ] + } + ] + } +} +``` + +#### Query Databases With Tables Order By Access Time +``` +GET /api/rest_j/v1/datasource/getByAccessTime +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/getByAccessTime", + "status": 0, + "message": "success", + "data": { + "dbs": [ + { + "name": "test_db", + "tables": [ + { + "name": "test_table", + "lastAccessTime": "2023-01-01 12:00:00" + } + ] + } + ] + } +} +``` + +#### Query Tables +``` +GET /api/rest_j/v1/datasource/tables +``` + +Parameters: +- `database`: Database name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/tables", + "status": 0, + "message": "success", + "data": { + "tables": [ + { + "name": "test_table" + } + ] + } +} +``` + +#### Query Table Metadata +``` +GET /api/rest_j/v1/datasource/columns +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/columns", + "status": 0, + "message": "success", + "data": { + "columns": [ + { + "name": "id", + "type": "INT", + "comment": "Primary key" + }, + { + "name": "name", + "type": "VARCHAR", + "comment": "Name field" + } + ] + } +} +``` + +#### Get Table Size +``` +GET /api/rest_j/v1/datasource/size +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional +- `partition`: Partition name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/size", + "status": 0, + "message": "success", + "data": { + "sizeInfo": { + "size": "10MB", + "fileCount": 5 + } + } +} +``` + +#### Get Storage Info +``` +GET /api/rest_j/v1/datasource/storage-info +``` + +Parameters: +- `database`: Database name (required) +- `table`: Table name (required) + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/storage-info", + "status": 0, + "message": "success", + "data": { + "storageInfo": { + "location": "/path/to/table", + "format": "PARQUET", + "compression": "SNAPPY" + } + } +} +``` + +#### Get Partitions +``` +GET /api/rest_j/v1/datasource/partitions +``` + +Parameters: +- `database`: Database name - optional +- `table`: Table name - optional + +Response: +```json +{ + "method": "/api/rest_j/v1/datasource/partitions", + "status": 0, + "message": "success", + "data": { + "partitionInfo": [ + { + "name": "dt=20230101", + "location": "/path/to/partition" + } + ] + } +} +``` + +### Data Source Type Management APIs + +#### List Data Source Types +``` +GET /api/rest_j/v1/basedata-manager/datasource-type +``` + +Parameters: +- `searchName`: Search name - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Type +``` +GET /api/rest_j/v1/basedata-manager/datasource-type/{id} +``` + +Parameters: +- `id`: Data source type ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "name": "MySQL", + "description": "MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" + } + } +} +``` + +#### Add Data Source Type +``` +POST /api/rest_j/v1/basedata-manager/datasource-type +``` + +Request Body: +```json +{ + "name": "PostgreSQL", + "description": "PostgreSQL Database", + "option": "PostgreSQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "PostgreSQL Database", + "optionEn": "PostgreSQL", + "classifierEn": "Database" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Type +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-type/{id} +``` + +Parameters: +- `id`: Data source type ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Type +``` +PUT /api/rest_j/v1/basedata-manager/datasource-type +``` + +Request Body: +```json +{ + "id": 1, + "name": "MySQL", + "description": "Updated MySQL Database", + "option": "MySQL", + "classifier": "Database", + "icon": "", + "layers": 3, + "descriptionEn": "Updated MySQL Database", + "optionEn": "MySQL", + "classifierEn": "Database" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Data Source Access Management APIs + +#### List Data Source Accesses +``` +GET /api/rest_j/v1/basedata-manager/datasource-access +``` + +Parameters: +- `searchName`: Search name - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "tokenId": 1, + "serviceId": 1, + "accessTime": "2023-01-01 12:00:00" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Access +``` +GET /api/rest_j/v1/basedata-manager/datasource-access/{id} +``` + +Parameters: +- `id`: Data source access ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "tokenId": 1, + "serviceId": 1, + "accessTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Add Data Source Access +``` +POST /api/rest_j/v1/basedata-manager/datasource-access +``` + +Request Body: +```json +{ + "tokenId": 1, + "serviceId": 1 +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Access +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-access/{id} +``` + +Parameters: +- `id`: Data source access ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Access +``` +PUT /api/rest_j/v1/basedata-manager/datasource-access +``` + +Request Body: +```json +{ + "id": 1, + "tokenId": 1, + "serviceId": 2 +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +### Data Source Type Key Management APIs + +#### List Data Source Type Keys +``` +GET /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Parameters: +- `searchName`: Search name - optional +- `dataSourceTypeId`: Data source type ID - optional +- `currentPage`: Current page - optional, default 1 +- `pageSize`: Page size - optional, default 10 + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "list": { + "total": 1, + "list": [ + { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00" + } + ], + "pageNum": 1, + "pageSize": 10, + "size": 1, + "startRow": 1, + "endRow": 1, + "pages": 1, + "prePage": 0, + "nextPage": 0, + "isFirstPage": true, + "isLastPage": true, + "hasPreviousPage": false, + "hasNextPage": false, + "navigatePages": 8, + "navigatepageNums": [ + 1 + ] + } + } +} +``` + +#### Get Data Source Type Key +``` +GET /api/rest_j/v1/basedata-manager/datasource-type-key/{id} +``` + +Parameters: +- `id`: Data source type key ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "item": { + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Host IP", + "descriptionEn": "Host IP", + "valueRegex": "", + "refId": null, + "refValue": null, + "dataSource": null, + "updateTime": "2023-01-01 12:00:00", + "createTime": "2023-01-01 12:00:00" + } + } +} +``` + +#### Add Data Source Type Key +``` +POST /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Request Body: +```json +{ + "dataSourceTypeId": 1, + "key": "port", + "name": "Port", + "nameEn": "Port", + "defaultValue": "3306", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Port number", + "descriptionEn": "Port number", + "valueRegex": "" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Remove Data Source Type Key +``` +DELETE /api/rest_j/v1/basedata-manager/datasource-type-key/{id} +``` + +Parameters: +- `id`: Data source type key ID (required) + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +#### Update Data Source Type Key +``` +PUT /api/rest_j/v1/basedata-manager/datasource-type-key +``` + +Request Body: +```json +{ + "id": 1, + "dataSourceTypeId": 1, + "key": "host", + "name": "Host", + "nameEn": "Host", + "defaultValue": "", + "valueType": "String", + "scope": "ENV", + "require": 1, + "description": "Updated Host IP", + "descriptionEn": "Updated Host IP", + "valueRegex": "" +} +``` + +Response: +```json +{ + "method": "", + "status": 0, + "message": "", + "data": { + "result": true + } +} +``` + +## Database Table Structures + +The DataSource Service uses the following database tables from linkis_ddl.sql: + +### Data Source Table +```sql +CREATE TABLE `linkis_ps_dm_datasource` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `datasource_name` varchar(255) COLLATE utf8_bin NOT NULL, + `datasource_desc` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `datasource_type_id` int(11) NOT NULL, + `create_identify` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `create_system` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `parameter` varchar(2048) COLLATE utf8_bin NULL DEFAULT NULL, + `create_time` datetime NULL DEFAULT CURRENT_TIMESTAMP, + `modify_time` datetime NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `modify_user` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `labels` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `version_id` int(11) DEFAULT NULL COMMENT 'current version id', + `expire` tinyint(1) DEFAULT 0, + `published_version_id` int(11) DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_datasource_name` (`datasource_name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Environment Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_env` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `env_name` varchar(32) COLLATE utf8_bin NOT NULL, + `env_desc` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `datasource_type_id` int(11) NOT NULL, + `parameter` varchar(2048) COLLATE utf8_bin DEFAULT NULL, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + `modify_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `modify_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_env_name` (`env_name`), + UNIQUE INDEX `uniq_name_dtid` (`env_name`, `datasource_type_id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Type Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_type` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `name` varchar(32) COLLATE utf8_bin NOT NULL, + `description` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `option` varchar(32) COLLATE utf8_bin DEFAULT NULL, + `classifier` varchar(32) COLLATE utf8_bin NOT NULL, + `icon` varchar(255) COLLATE utf8_bin DEFAULT NULL, + `layers` int(3) NOT NULL, + `description_en` varchar(255) DEFAULT NULL COMMENT 'english description', + `option_en` varchar(32) DEFAULT NULL COMMENT 'english option', + `classifier_en` varchar(32) DEFAULT NULL COMMENT 'english classifier', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_name` (`name`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Type Key Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_type_key` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `data_source_type_id` int(11) NOT NULL, + `key` varchar(32) COLLATE utf8_bin NOT NULL, + `name` varchar(32) COLLATE utf8_bin NOT NULL, + `name_en` varchar(32) COLLATE utf8_bin NULL DEFAULT NULL, + `default_value` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `value_type` varchar(50) COLLATE utf8_bin NOT NULL, + `scope` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `require` tinyint(1) NULL DEFAULT 0, + `description` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `description_en` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `value_regex` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `ref_id` bigint(20) NULL DEFAULT NULL, + `ref_value` varchar(50) COLLATE utf8_bin NULL DEFAULT NULL, + `data_source` varchar(200) COLLATE utf8_bin NULL DEFAULT NULL, + `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`), + UNIQUE KEY `uniq_dstid_key` (`data_source_type_id`, `key`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Version Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_version` +( + `version_id` int(11) NOT NULL AUTO_INCREMENT, + `datasource_id` int(11) NOT NULL, + `parameter` varchar(2048) COLLATE utf8_bin NULL DEFAULT NULL, + `comment` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + `create_time` datetime(0) NULL DEFAULT CURRENT_TIMESTAMP, + `create_user` varchar(255) COLLATE utf8_bin NULL DEFAULT NULL, + PRIMARY KEY `uniq_vid_did` (`version_id`, `datasource_id`) USING BTREE +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +### Data Source Access Table +```sql +CREATE TABLE `linkis_ps_dm_datasource_access` +( + `id` int(11) NOT NULL AUTO_INCREMENT, + `token_id` int(11) NOT NULL, + `service_id` int(11) NOT NULL, + `access_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, + PRIMARY KEY (`id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; +``` + +## RPC Methods + +The DataSource Service provides several RPC methods for data source management: + +### Data Source RPCs + +#### createDataSource +Creates a new data source: +```java +Long createDataSource(DataSourceCreationRequest request) +``` + +#### getDataSource +Retrieves a data source: +```java +DataSourceInfo getDataSource(Long dataSourceId) +``` + +#### updateDataSource +Updates a data source: +```java +void updateDataSource(DataSourceUpdateRequest request) +``` + +#### deleteDataSource +Deletes a data source: +```java +void deleteDataSource(Long dataSourceId) +``` + +#### listDataSources +Lists data sources with filtering: +```java +List listDataSources(DataSourceQueryRequest request) +``` + +### Metadata RPCs + +#### getMetadata +Retrieves metadata for a data source: +```java +DataSourceMetadata getMetadata(DataSourceMetadataRequest request) +``` + +#### testConnection +Tests connection to a data source: +```java +ConnectionTestResult testConnection(Long dataSourceId) +``` + +#### getTableSchema +Retrieves table schema information: +```java +TableSchema getTableSchema(Long dataSourceId, String database, String table) +``` + +#### getDatabaseList +Retrieves list of databases: +```java +List getDatabaseList(Long dataSourceId) +``` + +#### getTableList +Retrieves list of tables in a database: +```java +List getTableList(Long dataSourceId, String database) +``` + +### Environment RPCs + +#### createEnvironment +Creates a new environment: +```java +Long createEnvironment(EnvironmentCreationRequest request) +``` + +#### getEnvironment +Retrieves an environment: +```java +EnvironmentInfo getEnvironment(Long environmentId) +``` + +#### updateEnvironment +Updates an environment: +```java +void updateEnvironment(EnvironmentUpdateRequest request) +``` + +#### deleteEnvironment +Deletes an environment: +```java +void deleteEnvironment(Long environmentId) +``` + +### Access RPCs + +#### grantAccess +Grants access to a data source: +```java +void grantAccess(DataSourceAccessRequest request) +``` + +#### revokeAccess +Revokes access from a data source: +```java +void revokeAccess(DataSourceAccessRequest request) +``` + +#### checkAccess +Checks if a user has access to a data source: +```java +boolean checkAccess(String user, Long dataSourceId) +``` + +## Dependencies + +- linkis-datasource-manager +- linkis-metadata +- linkis-rpc +- linkis-protocol + +## Interface Classes and MyBatis XML Files + +### Interface Classes +- DataSourceCoreRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java` +- DataSourceAdminRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceAdminRestfulApi.java` +- DataSourceRestfulApi: `linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/restful/api/DataSourceRestfulApi.java` +- DatasourceTypeRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceTypeRestfulApi.java` +- DatasourceAccessRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceAccessRestfulApi.java` +- DatasourceTypeKeyRestfulApi: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/basedatamanager/server/restful/DatasourceTypeKeyRestfulApi.java` + +### MyBatis XML Files +- DataSouceMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml` +- DataSourceEnvMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceEnvMapper.xml` +- DataSourceParamKeyMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceParamKeyMapper.xml` +- DataSourceTypeMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceTypeMapper.xml` +- DataSourceVersionMapper: `linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSourceVersionMapper.xml` +- DataSourceAccessMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DataSourceAccessMapper.xml` +- DatasourceTypeMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DatasourceTypeMapper.xml` +- DatasourceTypeKeyMapper: `linkis-public-enhancements/linkis-pes-publicservice/src/main/resources/mapper/DatasourceTypeKeyMapper.xml` \ No newline at end of file diff --git a/.ai/modules/public-enhancements/jobhistory.md b/.ai/modules/public-enhancements/jobhistory.md new file mode 100644 index 00000000000..508c55ba925 --- /dev/null +++ b/.ai/modules/public-enhancements/jobhistory.md @@ -0,0 +1,579 @@ +# JobHistory Service + +The JobHistory service manages job execution history and provides querying capabilities for completed tasks in the Linkis system. + +## Overview + +This service tracks and stores information about job executions, including task status, execution results, logs, and performance metrics. It provides APIs for querying job history, statistics, and diagnostics. + +## Key Components + +### Core Classes +- `LinkisJobHistoryApplication` - Main application class +- Job history persistence and querying +- Task statistics and metrics collection +- Job diagnosis and failure analysis + +### Features +- Job execution history tracking +- Task result and log storage +- Performance statistics and metrics +- Job failure diagnosis +- Historical data querying and filtering + +## API Interfaces + +### Query Task by ID +``` +GET /api/rest_j/v1/jobhistory/{id}/get +``` + +Parameters: +- `id` (required): Task ID +- `brief` (optional): If true, only returns brief info + +Response: +```json +{ + "method": "/api/jobhistory/{id}/get", + "status": 0, + "message": "success", + "data": { + "task": { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql", + "params": { + "configuration": { + "runtime": { + "spark.executor.instances": "2" + } + } + } + } + } +} +``` + +### List Tasks +``` +GET /api/rest_j/v1/jobhistory/list +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `taskID` (optional): Specific task ID +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `isAdminView` (optional): Admin view flag +- `isDeptView` (optional): Department view flag +- `instance` (optional): Instance filter +- `engineInstance` (optional): Engine instance filter +- `runType` (optional): Run type filter + +Response: +```json +{ + "method": "/api/jobhistory/list", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ], + "totalPage": 100 + } +} +``` + +### List Undone Tasks +``` +GET /api/rest_j/v1/jobhistory/listundonetasks +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter (default: "Running,Inited,Scheduled") +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `startTaskID` (optional): Start task ID +- `engineType` (optional): Engine type filter +- `creator` (optional): Creator filter + +Response: +```json +{ + "method": "/api/jobhistory/listundonetasks", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Running", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "0.5", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ], + "totalPage": 10 + } +} +``` + +### List Undone Task Count +``` +GET /api/rest_j/v1/jobhistory/listundone +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `startTaskID` (optional): Start task ID +- `engineType` (optional): Engine type filter +- `creator` (optional): Creator filter + +Response: +```json +{ + "method": "/api/jobhistory/listundone", + "status": 0, + "message": "success", + "data": { + "totalPage": 10 + } +} +``` + +### List Tasks by Task IDs +``` +GET /api/rest_j/v1/jobhistory/list-taskids +``` + +Parameters: +- `taskID` (required): Comma-separated list of task IDs (max 30) + +Response: +```json +{ + "method": "/api/jobhistory/list-taskids", + "status": 0, + "message": "success", + "data": { + "jobHistoryList": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ] + } +} +``` + +### Get Job Extra Info +``` +GET /api/rest_j/v1/jobhistory/job-extra-info +``` + +Parameters: +- `jobId` (required): Job ID + +Response: +```json +{ + "method": "/api/jobhistory/job-extra-info", + "status": 0, + "message": "success", + "data": { + "metricsMap": { + "executionCode": "SELECT * FROM table", + "runtime": "300s", + // Additional metrics data + } + } +} +``` + +### Download Job List +``` +GET /api/rest_j/v1/jobhistory/download-job-list +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `status` (optional): Task status filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) +- `taskID` (optional): Specific task ID +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `isAdminView` (optional): Admin view flag +- `isDeptView` (optional): Department view flag +- `instance` (optional): Instance filter +- `engineInstance` (optional): Engine instance filter + +Response: +``` +Excel file download +``` + +### List Duration Top Tasks +``` +GET /api/rest_j/v1/jobhistory/listDurationTop +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter +- `pageNow` (optional): Page number (default: 1) +- `pageSize` (optional): Page size (default: 20) + +Response: +```json +{ + "method": "/api/jobhistory/listDurationTop", + "status": 0, + "message": "success", + "data": { + "tasks": [ + { + "jobId": 12345, + "status": "Succeed", + "submitUser": "testuser", + "executeUser": "testuser", + "instance": "bdp110:9100", + "engineType": "spark", + "executionCode": "SELECT * FROM table", + "progress": "1.0", + "logPath": "/path/to/log", + "errorCode": 0, + "errorDesc": "", + "createdTime": "2023-07-27T10:00:00.000+00:00", + "updatedTime": "2023-07-27T10:05:00.000+00:00", + "engineStartTime": "2023-07-27T10:01:00.000+00:00", + "runType": "sql" + } + ] + } +} +``` + +### Query Failed Task Diagnosis +``` +GET /api/rest_j/v1/jobhistory/diagnosis-query +``` + +Parameters: +- `taskID` (required): Task ID + +Response: +```json +{ + "method": "/api/jobhistory/diagnosis-query", + "status": 0, + "message": "success", + "data": { + "diagnosisMsg": "Diagnosis message content" + } +} +``` + +### Get Governance Station Admin Info +``` +GET /api/rest_j/v1/jobhistory/governanceStationAdmin +``` + +Response: +```json +{ + "method": "/api/jobhistory/governanceStationAdmin", + "status": 0, + "message": "success", + "data": { + "admin": true, + "historyAdmin": true, + "deptAdmin": false, + "canResultSet": true, + "errorMsgTip": "Error message tip" + } +} +``` + +### Task Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/taskCount +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/taskCount", + "status": 0, + "message": "success", + "data": { + "sumCount": 100, + "succeedCount": 80, + "failedCount": 15, + "cancelledCount": 5 + } +} +``` + +### Engine Count Statistics +``` +GET /api/rest_j/v1/jobhistory/jobstatistics/engineCount +``` + +Parameters: +- `startDate` (optional): Start date timestamp +- `endDate` (optional): End date timestamp +- `executeApplicationName` (optional): Application name filter +- `creator` (optional): Creator filter +- `proxyUser` (optional): Proxy user filter + +Response: +```json +{ + "method": "/api/jobhistory/jobstatistics/engineCount", + "status": 0, + "message": "success", + "data": { + "countEngine": 50, + "countEngineSucceed": 40, + "countEngineFailed": 8, + "countEngineShutting": 2 + } +} +``` + +### Add Observe Info +``` +POST /api/rest_j/v1/jobhistory/setting/addObserveInfo +``` + +Request Body: +```json +{ + "taskId": 12345, + "receiver": "user@example.com", + "extra": { + "title": "Job Alert", + "detail": "Job execution details" + }, + "monitorLevel": "HIGH", + "subSystemId": "subsystem1" +} +``` + +Response: +```json +{ + "method": "/api/jobhistory/setting/addObserveInfo", + "status": 0, + "message": "success" +} +``` + +### Delete Observe Info +``` +GET /api/rest_j/v1/jobhistory/setting/deleteObserveInfo +``` + +Parameters: +- `taskId` (required): Task ID + +Response: +```json +{ + "method": "/api/jobhistory/setting/deleteObserveInfo", + "status": 0, + "message": "success" +} +``` + +## Database Table Structures + +The JobHistory service uses the following database tables for job execution history management: + +### Job History Group History Table +```sql +CREATE TABLE `linkis_ps_job_history_group_history` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId', + `submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job', + `execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job', + `source` text DEFAULT NULL COMMENT 'job source', + `labels` text DEFAULT NULL COMMENT 'job labels', + `params` text DEFAULT NULL COMMENT 'job params', + `progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress', + `status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout', + `log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log', + `error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails', + `error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances', + `metrics` text DEFAULT NULL COMMENT 'Job Metrics', + `engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type', + `execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job', + PRIMARY KEY (`id`), + KEY `idx_created_time` (`created_time`), + KEY `idx_submit_user` (`submit_user`) +); +``` + +### Job History Detail Table +```sql +CREATE TABLE `linkis_ps_job_history_detail` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets', + `execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job', + `result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array', + `job_group_info` text DEFAULT NULL COMMENT 'Job group info/path', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `status` varchar(32) DEFAULT NULL COMMENT 'status', + `priority` int(4) DEFAULT 0 COMMENT 'order of subjob', + PRIMARY KEY (`id`) +); +``` + +## RPC Methods + +The JobHistory service provides several RPC methods for job history management: + +### Job History Query RPCs + +#### getJobHistoryById +Retrieves job history by ID: +```java +JobHistory getJobHistoryById(Long jobId) +``` + +#### searchJobHistory +Searches job history with filters: +```java +List searchJobHistory(JobHistorySearchRequest request) +``` + +#### updateJobHistory +Updates job history information: +```java +void updateJobHistory(JobHistory jobHistory) +``` + +#### deleteJobHistory +Deletes job history: +```java +void deleteJobHistory(Long jobId) +``` + +### Job Statistics RPCs + +#### taskExecutionStatistics +Retrieves task execution statistics: +```java +JobStatistics taskExecutionStatistics(StatisticsRequest request) +``` + +#### engineExecutionStatistics +Retrieves engine execution statistics: +```java +JobStatistics engineExecutionStatistics(StatisticsRequest request) +``` + +### Job Diagnosis RPCs + +#### diagnoseJob +Performs job diagnosis: +```java +JobDiagnosis diagnoseJob(Long jobId) +``` + +#### getDiagnosisInfo +Retrieves diagnosis information: +```java +JobDiagnosis getDiagnosisInfo(Long jobId) +``` + +## Dependencies + +- linkis-jobhistory-server +- linkis-rpc +- linkis-protocol +- linkis-commons +- Database drivers (MySQL, etc.) \ No newline at end of file diff --git a/.ai/modules/public-enhancements/publicservice.md b/.ai/modules/public-enhancements/publicservice.md new file mode 100644 index 00000000000..6606d8a296d --- /dev/null +++ b/.ai/modules/public-enhancements/publicservice.md @@ -0,0 +1,151 @@ +# Public Service + +The Public Service provides core public services for the Linkis system. + +## Overview + +This service provides common public services including file system operations, variable management, and other shared functionalities. + +## Key Components + +### Core Classes +- `LinkisPublicServiceApp` - Main application class +- File system operations +- Variable management +- Shared service utilities + +### Features +- File system operations (upload, download, list) +- Variable management +- Shared service utilities +- Common REST APIs + +## API Interfaces + +### File System Operations +``` +POST /api/rest_j/v1/filesystem/upload +``` + +Request: +``` +multipart/form-data with file content +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/filesystem/upload", + "status": 0, + "message": "success", + "data": { + "path": "/path/to/uploaded/file" + } +} +``` + +### Variable Management +``` +POST /api/rest_j/v1/variable/add +``` + +Request Body: +```json +{ + "key": "variableKey", + "value": "variableValue", + "user": "testuser" +} +``` + +Response: +```json +{ + "method": "/api/rest_j/v1/variable/add", + "status": 0, + "message": "success", + "data": {} +} +``` + +## Database Table Structures + +The Public Service manages the following database tables: + +### File System Metadata Table +```sql +CREATE TABLE linkis_filesystem_meta ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + user_name VARCHAR(32) NOT NULL, + path VARCHAR(500) NOT NULL, + file_type VARCHAR(50), + file_size BIGINT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + UNIQUE KEY uk_user_path (user_name, path) +); +``` + +### Variable Table +```sql +CREATE TABLE linkis_variable ( + id BIGINT PRIMARY KEY AUTO_INCREMENT, + user_name VARCHAR(32) NOT NULL, + key_name VARCHAR(128) NOT NULL, + value TEXT, + create_time DATETIME DEFAULT CURRENT_TIMESTAMP, + update_time DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, + UNIQUE KEY uk_user_key (user_name, key_name) +); +``` + +## RPC Methods + +The Public Service provides several RPC methods for common operations: + +### File System RPCs + +#### uploadFile +Uploads a file: +```java +String uploadFile(FileUploadRequest request) +``` + +#### downloadFile +Downloads a file: +```java +FileContent downloadFile(String path, String user) +``` + +#### listFiles +Lists files in a directory: +```java +List listFiles(String path, String user) +``` + +### Variable RPCs + +#### setVariable +Sets a variable: +```java +void setVariable(String key, String value, String user) +``` + +#### getVariable +Retrieves a variable: +```java +String getVariable(String key, String user) +``` + +#### deleteVariable +Deletes a variable: +```java +void deleteVariable(String key, String user) +``` + +## Dependencies + +- linkis-filesystem +- linkis-variable +- linkis-rpc +- linkis-protocol \ No newline at end of file diff --git a/.ai/project-context.md b/.ai/project-context.md new file mode 100644 index 00000000000..fcddd703cb9 --- /dev/null +++ b/.ai/project-context.md @@ -0,0 +1,1027 @@ +# Apache Linkis AI IDE 开发规约 + +> **文档版本信息** +> - 版本: 1.0.0 +> - 最后更新: 2025-01-28 +> - 适用版本: Apache Linkis 1.17.0+ + +## 角色定位 +你是Apache Linkis项目的资深后端开发专家，熟练掌握： +- **核心技术栈**：Spring Boot 2.7 + Spring Cloud 2021.0.8 + MyBatis-Plus 3.5.7 +- **编程语言**：Java 8 + Scala 2.12（混合开发模式） +- **数据库**：MySQL 8.0 + Hive（通过JDBC） +- **微服务架构**：Eureka服务发现 + Gateway网关 + Feign远程调用 +- **大数据引擎**：Spark、Hive、Flink、Python、Shell等多引擎支持 + +--- + +# 项目核心信息 + +## 基础配置 +- **项目根目录**：linkis +- **基础包名**：org.apache.linkis +- **版本信息**：Apache Linkis 1.x +- **构建工具**：Maven 3.5+ +- **JDK版本**：1.8 +- **字符编码**：统一使用StandardCharsets.UTF_8 + +## 关键组件 +- **统一返回体**：`org.apache.linkis.server.Message` +- **统一异常**：`org.apache.linkis.common.exception.LinkisException` +- **配置管理**：`org.apache.linkis.common.conf.CommonVars` +- **数据库脚本**： + - DDL：`linkis-dist/package/db/linkis_ddl.sql` + - DML：`linkis-dist/package/db/linkis_dml.sql` + +--- + +# 系统架构设计 + +## 三层架构模式 +Linkis采用微服务架构，按功能职责划分为三大服务类别： + +### 1. 微服务治理服务（基础设施层） +负责微服务的基础设施支撑，包括服务发现、网关路由、配置管理等。 +- Spring Cloud Gateway：API网关服务 +- Eureka：服务注册与发现中心 +- Open Feign：声明式HTTP客户端 + +### 2. 计算治理服务（核心业务层） +负责计算任务的生命周期管理，从任务提交到执行完成的全流程控制。 +- Entrance：任务提交入口服务 +- JobHistory：任务历史记录服务 +- LinkisManager：资源管理服务 +- EngineConnManager：引擎连接管理服务 +- EngineConn：引擎连接器 + +### 3. 公共增强服务（支撑服务层） +提供跨服务的公共能力，如文件管理、数据源管理、配置管理等。 +- PublicService：公共服务 +- BML：大数据物料库 +- DataSource：数据源管理 +- Configuration：配置管理 +- ContextServer：上下文服务 +- Monitor：监控服务 + +## 服务交互模式 +``` +上层应用 -> Gateway -> Entrance -> Manager -> ECM -> EngineConn -> 底层引擎 + ↓ + 公共增强服务（BML、DataSource、Configuration等） +``` + +## 各服务模块说明 +### 微服务治理服务 +Spring Cloud Gateway +功能：API网关服务，负责请求路由转发、负载均衡、安全认证等 +主类入口：org.apache.linkis.gateway.springcloud.LinkisGatewayApplication +模块路径：linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway + +Eureka +功能：服务注册与发现中心，管理微服务实例的注册、发现和健康检查 +主类入口：org.apache.linkis.eureka.SpringCloudEurekaApplication +模块路径：linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka + +Open Feign +功能：声明式HTTP客户端，简化微服务间的远程调用 +主类入口：集成在各个微服务模块中，无独立启动类 +模块路径：集成在linkis-commons/linkis-rpc等公共模块中 + +### 计算治理服务 +Entrance +功能：任务提交入口服务，负责任务调度、状态管控、任务信息推送等核心功能 +主类入口：org.apache.linkis.entrance.LinkisEntranceApplication +模块路径：linkis-computation-governance/linkis-entrance + +JobHistory +功能：任务历史记录服务，提供任务执行历史的查询、统计和管理功能 +主类入口：org.apache.linkis.jobhistory.LinkisJobHistoryApp +模块路径：linkis-public-enhancements/linkis-jobhistory + +LinkisManager +功能：计算治理层的管理服务，包含AppManager、ResourceManager、LabelManager等管理控制服务 +主类入口：org.apache.linkis.manager.LinkisManagerApplication +模块路径：linkis-computation-governance/linkis-manager/linkis-application-manager + +EngineConnManager +功能：引擎连接器管理服务，负责控制EngineConn的生命周期（启动、停止） +主类入口：org.apache.linkis.ecm.server.LinkisECMApplication +模块路径：linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-server + +EngineConn +功能：引擎连接器，负责接收任务并提交到Spark、Hive、Flink等底层引擎执行 +主类入口：org.apache.linkis.engineconn.LinkisEngineConnApplication +模块路径：linkis-computation-governance/linkis-engineconn + +### 公共增强服务 +PublicService +功能：公共服务模块，提供统一配置管理、微服务管理等基础服务能力 +主类入口：org.apache.linkis.filesystem.LinkisPublicServiceApp +模块路径：linkis-public-enhancements/linkis-pes-publicservice + +BML +功能：大数据物料库服务(BigData Material Library)，提供文件上传、下载、版本管理等功能 +主类入口：org.apache.linkis.bml.LinkisBMLApplication +模块路径：linkis-public-enhancements/linkis-bml-server + +DataSource +功能：数据源管理服务，提供统一的数据源连接、管理和元数据服务 +主类入口：org.apache.linkis.metadata.LinkisDataSourceApplication（数据源服务） +模块路径：linkis-public-enhancements/linkis-datasource + +Configuration +功能：配置管理服务，提供系统级、用户级、引擎级等多层次的配置管理 +主类入口：org.apache.linkis.configuration.LinkisConfigurationApp +模块路径：linkis-public-enhancements/linkis-configuration + +ContextServer +功能：上下文服务，支持跨引擎的资源共享、变量传递和会话管理 +主类入口：org.apache.linkis.cs.server.LinkisCSApplication +模块路径：linkis-public-enhancements/linkis-cs-server + +Monitor +功能：监控服务，提供系统性能监控、告警和运维管理功能，包括任务监控、资源监控、用户模式监控等 +主类入口：org.apache.linkis.monitor.LinksMonitorApplication +模块路径：linkis-extensions/linkis-et-monitor + +--- + +# 开发规范与约束 + +## 代码边界约束 + +### 🚫 禁止操作 +- **数据库结构**：除非明确指定，严禁修改现有表结构 +- **第三方依赖**：不允许引入新的第三方依赖库 +- **核心接口**：不得修改现有公共接口的签名 + +### ✅ 允许操作 +- **新增功能**：在不破坏现有逻辑的前提下扩展功能 +- **新增配置**：在现有配置文件中新增配置项 +- **新增表字段**：在现有表基础上新增字段 + +## 技术规范 + +### 编程语言使用 +- **Java**：主要用于REST API、Service层、Entity类、配置类 +- **Scala**：主要用于计算逻辑、RPC通信、复杂业务处理 + +### 日志规范 +```java +// 必须使用统一的Logger +private static final Logger logger = LoggerFactory.getLogger(ClassName.class); + +// 日志级别使用： +// ERROR: 系统错误、业务异常 +// WARN: 警告信息、降级处理 +// INFO: 关键业务节点、状态变更 +// DEBUG: 详细调试信息 + +logger.info("User {} starts processing task {}", username, taskId); +logger.error("Failed to process task {} for user {}", taskId, username, e); +``` + +### 配置管理规范 +- 所有配置统一使用`org.apache.linkis.common.conf.CommonVars` +- 参考示例：`org.apache.linkis.jobhistory.conf.JobhistoryConfiguration` +- 所有新需求必须添加配置开关，默认设置false +- 配置存放位置：当前模块的conf目录，一般为xxxConfiguration类 + +### 字符编码规范 +```java +// 统一使用StandardCharsets.UTF_8，禁止使用字符串"UTF-8" +import java.nio.charset.StandardCharsets; + +String content = new String(bytes, StandardCharsets.UTF_8); +Files.write(path, content.getBytes(StandardCharsets.UTF_8)); +``` + +### API设计规范 +```java +@Api(tags = "module operation") +@RestController +@RequestMapping(path = "/api/rest_j/v1/module") +public class ModuleRestfulApi { + + @ApiOperation(value = "operation", notes = "description", response = Message.class) + @RequestMapping(path = "/operation", method = RequestMethod.POST) + public Message operation(HttpServletRequest req, @RequestBody JsonNode jsonNode) { + String username = ModuleUserUtils.getOperationUser(req, "operation"); + // 业务逻辑处理 + return Message.ok("success").data("result", data); + } +} +``` + +### 异常处理规范 +```java +// 统一使用LinkisException及其子类 +try { + // 业务逻辑 +} catch (Exception e) { + logger.error("Operation failed", e); + throw new YourModuleException("Error message", e); +} +``` + +--- + +# 开发模板与示例 + +## 新功能开发模板 + +### 1. REST接口层 +```java +@Api(tags = "功能模块操作") +@RestController +@RequestMapping(path = "/api/rest_j/v1/module") +public class ModuleRestfulApi { + + @Autowired + private ModuleService moduleService; + + @ApiOperation(value = "功能操作", response = Message.class) + @RequestMapping(path = "/action", method = RequestMethod.POST) + public Message action(HttpServletRequest req, @RequestBody JsonNode jsonNode) { + String username = ModuleUserUtils.getOperationUser(req, "action"); + + // 参数解析和验证 + String param = jsonNode.get("param").asText(); + if (StringUtils.isBlank(param)) { + return Message.error("参数不能为空"); + } + + try { + Object result = moduleService.performAction(param, username); + return Message.ok("操作成功").data("result", result); + } catch (Exception e) { + logger.error("操作失败", e); + return Message.error("操作失败：" + e.getMessage()); + } + } +} +``` + +### 2. 服务层 +```java +@Service +public class ModuleServiceImpl implements ModuleService { + + private static final Logger logger = LoggerFactory.getLogger(ModuleServiceImpl.class); + + @Autowired + private ModuleMapper moduleMapper; + + @Override + @Transactional(rollbackFor = Exception.class) + public Object performAction(String param, String username) { + logger.info("User {} starts action with param: {}", username, param); + + // 业务逻辑处理 + ModuleEntity entity = new ModuleEntity(); + entity.setParam(param); + entity.setCreateUser(username); + + moduleMapper.insert(entity); + + logger.info("User {} completed action successfully", username); + return entity.getId(); + } +} +``` + +### 3. 数据访问层 +```java +@Mapper +public interface ModuleMapper { + + @Insert("INSERT INTO linkis_module_table (param, create_user, create_time) " + + "VALUES (#{param}, #{createUser}, NOW())") + @Options(useGeneratedKeys = true, keyProperty = "id") + void insert(ModuleEntity entity); + + @Select("SELECT * FROM linkis_module_table WHERE id = #{id}") + ModuleEntity selectById(@Param("id") Long id); +} +``` + +### 4. 配置类 +```scala +object ModuleConfiguration { + val MODULE_FEATURE_ENABLE = CommonVars("linkis.module.feature.enable", false) + val MODULE_TIMEOUT = CommonVars("linkis.module.timeout", 30000L) + val MODULE_BATCH_SIZE = CommonVars("linkis.module.batch.size", 1000) +} +``` + +--- + +# 常用配置示例库 + +## 配置定义示例 + +### 功能开关配置 +```scala +object FeatureConfiguration { + // 布尔型开关 - 用于控制功能是否启用 + val FEATURE_ENABLE = CommonVars("linkis.feature.enable", false) + + // 数值型配置 - 批处理大小 + val BATCH_SIZE = CommonVars("linkis.feature.batch.size", 1000) + + // 长整型配置 - 超时时间（毫秒） + val TIMEOUT = CommonVars("linkis.feature.timeout", 30000L) + + // 字符串配置 - 运行模式 + val MODE = CommonVars("linkis.feature.mode", "default") + + // 浮点型配置 - 阈值 + val THRESHOLD = CommonVars("linkis.feature.threshold", 0.8) + + // 列表型配置 - 逗号分隔 + val ALLOWED_TYPES = CommonVars("linkis.feature.allowed.types", "spark,hive,python") +} +``` + +### 性能相关配置 +```scala +object PerformanceConfiguration { + // 线程池大小 + val THREAD_POOL_SIZE = CommonVars("linkis.performance.thread.pool.size", 10) + + // 队列容量 + val QUEUE_CAPACITY = CommonVars("linkis.performance.queue.capacity", 1000) + + // 连接池配置 + val MAX_CONNECTIONS = CommonVars("linkis.performance.max.connections", 50) + val MIN_IDLE = CommonVars("linkis.performance.min.idle", 5) + + // 缓存配置 + val CACHE_ENABLE = CommonVars("linkis.performance.cache.enable", true) + val CACHE_SIZE = CommonVars("linkis.performance.cache.size", 10000) + val CACHE_EXPIRE_SECONDS = CommonVars("linkis.performance.cache.expire.seconds", 3600L) +} +``` + +### 重试和容错配置 +```scala +object ResilienceConfiguration { + // 重试次数 + val MAX_RETRY_TIMES = CommonVars("linkis.resilience.max.retry.times", 3) + + // 重试间隔（毫秒） + val RETRY_INTERVAL = CommonVars("linkis.resilience.retry.interval", 1000L) + + // 熔断开关 + val CIRCUIT_BREAKER_ENABLE = CommonVars("linkis.resilience.circuit.breaker.enable", false) + + // 失败率阈值 + val FAILURE_RATE_THRESHOLD = CommonVars("linkis.resilience.failure.rate.threshold", 0.5) +} +``` + +## 配置使用示例 + +### 在Java代码中使用配置 +```java +@Service +public class FeatureServiceImpl implements FeatureService { + + private static final Logger logger = LoggerFactory.getLogger(FeatureServiceImpl.class); + + @Override + public void executeFeature() { + // 检查功能开关 + if (!FeatureConfiguration.FEATURE_ENABLE.getValue()) { + logger.info("Feature is disabled, skipping execution"); + return; // 功能关闭时不执行 + } + + // 使用配置参数 + int batchSize = FeatureConfiguration.BATCH_SIZE.getValue(); + long timeout = FeatureConfiguration.TIMEOUT.getValue(); + String mode = FeatureConfiguration.MODE.getValue(); + + logger.info("Executing feature with batchSize={}, timeout={}, mode={}", + batchSize, timeout, mode); + + // 业务逻辑... + } +} +``` + +### 在Scala代码中使用配置 +```scala +class FeatureExecutor { + + def execute(): Unit = { + // 检查功能开关 + if (!FeatureConfiguration.FEATURE_ENABLE.getValue) { + logger.info("Feature is disabled") + return + } + + // 获取配置值 + val batchSize = FeatureConfiguration.BATCH_SIZE.getValue + val timeout = FeatureConfiguration.TIMEOUT.getValue + val allowedTypes = FeatureConfiguration.ALLOWED_TYPES.getValue.split(",").toList + + // 使用配置执行业务逻辑 + processBatch(batchSize, timeout, allowedTypes) + } +} +``` + +### 带降级逻辑的配置使用 +```java +public class SmartFeatureService { + + public void processWithFallback(List dataList) { + // 检查功能开关 + if (!FeatureConfiguration.FEATURE_ENABLE.getValue()) { + // 降级到旧逻辑 + processLegacy(dataList); + return; + } + + try { + // 新功能逻辑 + int batchSize = FeatureConfiguration.BATCH_SIZE.getValue(); + processInBatches(dataList, batchSize); + } catch (Exception e) { + logger.error("New feature failed, falling back to legacy", e); + // 异常时降级 + processLegacy(dataList); + } + } + + private void processLegacy(List dataList) { + // 原有的稳定逻辑 + } +} +``` + +### 配置验证和边界检查 +```java +public class ConfigValidator { + + public static void validateAndExecute() { + // 获取配置 + int batchSize = FeatureConfiguration.BATCH_SIZE.getValue(); + + // 验证配置合法性 + if (batchSize <= 0 || batchSize > 10000) { + logger.error("Invalid batch size: {}, using default 1000", batchSize); + batchSize = 1000; + } + + // 使用验证后的配置 + processBatch(batchSize); + } +} +``` + +## 配置文件示例 + +### linkis.properties 配置示例 +```properties +# 功能开关配置 +linkis.feature.enable=false +linkis.feature.batch.size=1000 +linkis.feature.timeout=30000 +linkis.feature.mode=default + +# 性能配置 +linkis.performance.thread.pool.size=10 +linkis.performance.queue.capacity=1000 +linkis.performance.cache.enable=true + +# 重试配置 +linkis.resilience.max.retry.times=3 +linkis.resilience.retry.interval=1000 +``` + +## 配置最佳实践 + +### ✅ 推荐做法 +1. **所有新功能必须有开关**，默认值设为 `false` +2. **配置命名规范**：`linkis.[模块].[功能].[属性]` +3. **提供合理的默认值**，确保不配置时系统能正常运行 +4. **添加配置注释**，说明配置的作用和取值范围 +5. **配置集中管理**，放在对应模块的 Configuration 类中 + +### ❌ 避免做法 +1. 不要硬编码配置值 +2. 不要在多处重复定义相同配置 +3. 不要使用不合理的默认值（如 0、空字符串） +4. 不要忘记在 linkis.properties 中添加配置说明 + +--- + +# 常见错误及避免方法 + +## ❌ 错误1：字符编码使用不规范 + +### 错误示例 +```java +// ❌ 错误：使用字符串 "UTF-8" +String content = new String(bytes, "UTF-8"); +FileWriter writer = new FileWriter(file, "UTF-8"); +response.setCharacterEncoding("UTF-8"); + +// 问题： +// 1. 字符串容易拼写错误 +// 2. 编译器无法检查 +// 3. 不符合项目规范 +``` + +### 正确示例 +```java +// ✅ 正确：使用 StandardCharsets.UTF_8 +import java.nio.charset.StandardCharsets; + +String content = new String(bytes, StandardCharsets.UTF_8); +Files.write(path, content.getBytes(StandardCharsets.UTF_8)); +response.setCharacterEncoding(StandardCharsets.UTF_8.name()); + +// 优点： +// 1. 编译时检查 +// 2. 不会拼写错误 +// 3. 符合项目规范 +``` + +--- + +## ❌ 错误2：新功能未添加开关 + +### 错误示例 +```java +// ❌ 错误：新功能直接生效，无法回退 +@Service +public class NewFeatureService { + + public void executeNewFeature() { + // 直接实现新逻辑 + // 如果出现问题，只能通过代码回退或重新部署 + newAlgorithm(); + } +} +``` + +### 正确示例 +```java +// ✅ 正确：添加功能开关，支持热切换 +@Service +public class SmartFeatureService { + + private static final Logger logger = LoggerFactory.getLogger(SmartFeatureService.class); + + public void executeFeature() { + // 检查功能开关 + if (!NewFeatureConfiguration.ENABLE.getValue()) { + logger.info("New feature is disabled, using legacy implementation"); + executeLegacyFeature(); // 降级到旧逻辑 + return; + } + + try { + logger.info("New feature is enabled"); + executeNewFeature(); // 执行新逻辑 + } catch (Exception e) { + logger.error("New feature failed, falling back to legacy", e); + executeLegacyFeature(); // 异常时降级 + } + } + + private void executeNewFeature() { + // 新功能实现 + } + + private void executeLegacyFeature() { + // 原有稳定实现 + } +} + +// 配置类 +object NewFeatureConfiguration { + val ENABLE = CommonVars("linkis.new.feature.enable", false) +} +``` + +--- + +## ❌ 错误3：修改现有表结构未记录 + +### 错误示例 +```sql +-- ❌ 错误：直接在数据库执行 ALTER TABLE +-- 问题： +-- 1. 其他环境无法同步 +-- 2. 没有变更记录 +-- 3. 无法回滚 + +ALTER TABLE linkis_ps_job_history_group_history +ADD COLUMN new_field VARCHAR(50) COMMENT 'new field'; +``` + +### 正确示例 +```sql +-- ✅ 正确：在 linkis-dist/package/db/linkis_ddl.sql 中添加变更 + +-- Step 1: 在 linkis_ddl.sql 文件末尾添加变更记录 +-- ================================================================ +-- 版本: 1.17.0 +-- 需求: 添加任务扩展字段支持 +-- 日期: 2025-01-28 +-- ================================================================ + +ALTER TABLE linkis_ps_job_history_group_history +ADD COLUMN new_field VARCHAR(50) COMMENT 'new field for extended info'; + +-- 如果有索引变更 +CREATE INDEX idx_new_field ON linkis_ps_job_history_group_history(new_field); + +-- Step 2: 如果需要初始化数据，在 linkis_dml.sql 中添加 +-- 在 linkis-dist/package/db/linkis_dml.sql 添加： +UPDATE linkis_ps_job_history_group_history +SET new_field = 'default_value' +WHERE new_field IS NULL; +``` + +--- + +## ❌ 错误4：异常处理不规范 + +### 错误示例 +```java +// ❌ 错误示例1：吞掉异常 +try { + processData(); +} catch (Exception e) { + // 什么都不做，异常被吞掉 +} + +// ❌ 错误示例2：打印后继续抛出原始异常 +try { + processData(); +} catch (Exception e) { + e.printStackTrace(); // 不要使用 printStackTrace + throw e; // 直接抛出原始异常 +} + +// ❌ 错误示例3：捕获过于宽泛 +try { + processData(); +} catch (Throwable t) { // 不要捕获 Throwable + logger.error("Error", t); +} +``` + +### 正确示例 +```java +// ✅ 正确示例1：记录日志并抛出业务异常 +try { + processData(); +} catch (IOException e) { + logger.error("Failed to process data", e); + throw new DataProcessException("Failed to process data", e); +} + +// ✅ 正确示例2：捕获具体异常，提供有意义的错误信息 +try { + String result = processData(param); + return result; +} catch (IllegalArgumentException e) { + logger.error("Invalid parameter: {}", param, e); + throw new ValidationException("Invalid parameter: " + param, e); +} catch (IOException e) { + logger.error("IO error while processing data", e); + throw new DataAccessException("IO error while processing data", e); +} + +// ✅ 正确示例3：在Service层统一处理异常 +@Service +public class DataServiceImpl implements DataService { + + @Override + public Result processData(String param) { + try { + // 业务逻辑 + String data = fetchData(param); + return Result.success(data); + } catch (DataNotFoundException e) { + logger.warn("Data not found for param: {}", param); + return Result.error("Data not found"); + } catch (Exception e) { + logger.error("Unexpected error while processing data", e); + throw new ServiceException("Failed to process data", e); + } + } +} +``` + +--- + +## ❌ 错误5：日志记录不规范 + +### 错误示例 +```java +// ❌ 错误示例1：使用 System.out +System.out.println("Processing data: " + data); + +// ❌ 错误示例2：日志级别使用不当 +logger.error("User {} logged in", username); // 登录不是错误 + +// ❌ 错误示例3：字符串拼接 +logger.info("Processing user: " + username + ", id: " + userId); + +// ❌ 错误示例4：敏感信息直接打印 +logger.info("User password: {}", password); +``` + +### 正确示例 +```java +// ✅ 正确示例1：使用 Logger +private static final Logger logger = LoggerFactory.getLogger(ClassName.class); + +// ✅ 正确示例2：使用正确的日志级别 +logger.info("User {} logged in successfully", username); // INFO +logger.warn("Login attempt from unknown IP: {}", ip); // WARN +logger.error("Failed to authenticate user {}", username, exception); // ERROR + +// ✅ 正确示例3：使用占位符 +logger.info("Processing user: {}, id: {}, type: {}", username, userId, userType); + +// ✅ 正确示例4：脱敏处理敏感信息 +logger.info("User {} password updated", username); // 不打印密码 +logger.debug("Token: {}***", token.substring(0, 4)); // 只打印前几位 + +// ✅ 正确示例5：关键业务节点记录完整上下文 +logger.info("Task submitted: taskId={}, user={}, engineType={}, code={}", + taskId, username, engineType, codePreview); +logger.error("Task execution failed: taskId={}, user={}, error={}", + taskId, username, e.getMessage(), e); +``` + +--- + +## ❌ 错误6：REST接口返回值不规范 + +### 错误示例 +```java +// ❌ 错误：直接返回业务对象或String +@RequestMapping(path = "/getData", method = RequestMethod.GET) +public UserData getData() { + return userData; // 不符合统一返回体规范 +} + +@RequestMapping(path = "/save", method = RequestMethod.POST) +public String save(@RequestBody Data data) { + return "success"; // 不符合规范 +} +``` + +### 正确示例 +```java +// ✅ 正确：使用统一返回体 Message +import org.apache.linkis.server.Message; + +@RequestMapping(path = "/getData", method = RequestMethod.GET) +public Message getData(HttpServletRequest req) { + try { + String username = ModuleUserUtils.getOperationUser(req, "getData"); + UserData data = userService.getData(username); + return Message.ok("Query successful").data("userData", data); + } catch (Exception e) { + logger.error("Failed to get user data", e); + return Message.error("Failed to get user data: " + e.getMessage()); + } +} + +@RequestMapping(path = "/save", method = RequestMethod.POST) +public Message save(HttpServletRequest req, @RequestBody JsonNode jsonNode) { + try { + String username = ModuleUserUtils.getOperationUser(req, "save"); + + // 参数验证 + String name = jsonNode.get("name").asText(); + if (StringUtils.isBlank(name)) { + return Message.error("Name cannot be empty"); + } + + Long id = dataService.save(name, username); + return Message.ok("Save successful").data("id", id); + } catch (Exception e) { + logger.error("Failed to save data", e); + return Message.error("Failed to save data: " + e.getMessage()); + } +} +``` + +--- + +## ❌ 错误7：MyBatis SQL注入风险 + +### 错误示例 +```xml + + +``` + +### 正确示例 +```xml + + + + + +``` + +```java +// 在Service层验证动态字段 +public List selectWithOrder(String orderBy) { + // 白名单验证 + List allowedFields = Arrays.asList("id", "name", "create_time"); + if (!allowedFields.contains(orderBy)) { + throw new IllegalArgumentException("Invalid order field: " + orderBy); + } + return userMapper.selectWithOrder(orderBy); +} +``` + +--- + +## ❌ 错误8：事务使用不当 + +### 错误示例 +```java +// ❌ 错误示例1：没有添加事务注解 +@Service +public class OrderService { + public void createOrder(Order order) { + orderMapper.insert(order); // 插入订单 + stockMapper.decrease(order.getProductId()); // 减库存 + // 如果减库存失败，订单已经插入，数据不一致 + } +} + +// ❌ 错误示例2：捕获异常后未抛出，事务不会回滚 +@Transactional +public void processOrder(Order order) { + try { + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + } catch (Exception e) { + logger.error("Error", e); + // 异常被吞掉，事务不会回滚 + } +} +``` + +### 正确示例 +```java +// ✅ 正确示例1：添加事务注解，指定回滚异常 +@Service +public class OrderService { + + @Transactional(rollbackFor = Exception.class) + public void createOrder(Order order) { + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + // 任何异常都会回滚 + } +} + +// ✅ 正确示例2：如果需要捕获异常，重新抛出 +@Transactional(rollbackFor = Exception.class) +public void processOrder(Order order) { + try { + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + } catch (StockNotEnoughException e) { + logger.warn("Stock not enough for product: {}", order.getProductId()); + throw e; // 重新抛出，触发回滚 + } catch (Exception e) { + logger.error("Unexpected error while processing order", e); + throw new OrderProcessException("Failed to process order", e); + } +} + +// ✅ 正确示例3：部分操作不需要事务 +@Service +public class OrderService { + + @Transactional(rollbackFor = Exception.class) + public Long createOrder(Order order) { + // 数据库操作在事务中 + orderMapper.insert(order); + stockMapper.decrease(order.getProductId()); + + Long orderId = order.getId(); + + // 发送通知不在事务中（避免外部调用导致事务超时） + sendNotificationAsync(orderId); + + return orderId; + } + + private void sendNotificationAsync(Long orderId) { + // 异步发送，不阻塞事务 + executor.submit(() -> notificationService.send(orderId)); + } +} +``` + +--- + +## 🎯 错误排查清单 + +开发完成后，请检查以下项目： + +- [ ] 字符编码统一使用 `StandardCharsets.UTF_8` +- [ ] 新功能已添加开关配置（默认false） +- [ ] 数据库变更已记录到 DDL/DML 文件 +- [ ] 异常处理规范，使用 LinkisException 及其子类 +- [ ] 日志使用 Logger，不使用 System.out +- [ ] REST接口使用统一返回体 Message +- [ ] SQL 使用参数化查询，避免注入 +- [ ] 事务注解正确使用，异常能正确回滚 +- [ ] 敏感信息已脱敏处理 +- [ ] 代码遵循最小改动原则 + +--- + +# 需求开发流程 + +## 需求分析模板 + +### 【背景说明】 +描述业务场景、现有问题或痛点、期望解决的目标 + +### 【验收标准】 +- 功能验收点（具体、可测量） +- 性能要求（响应时间、并发数等） +- 安全要求（权限控制、数据保护） +- 兼容性要求（向后兼容） + +## 开发交付清单 + +### 变更清单 +- 新增/修改的文件路径列表 +- 数据库变更脚本（DDL/DML） +- 配置文件变更 + +### 测试验证 +- 单元测试代码 +- 集成测试用例 +- 手动测试命令（curl等） + +### 质量检查 +- [ ] 代码符合项目规范 +- [ ] 异常处理完整 +- [ ] 日志记录充分 +- [ ] 单元测试覆盖 +- [ ] 配置开关完整 +- [ ] 向后兼容性检查 + +--- + +# AI IDE开发提示 + +## 开发技巧 +1. **优先查看现有代码**：在新增功能前，先查看相似功能的实现方式 +2. **遵循现有模式**：保持与现有代码风格一致 +3. **充分测试**：编写充分的单元测试和集成测试 +4. **考虑边界情况**：处理各种异常和边界条件 + +## 常见问题及解决方案 + +### 1. 字符编码问题 +**问题**：HTTP传输过程中出现中文乱码 +**解决**：统一使用`StandardCharsets.UTF_8` + +### 2. 配置热更新问题 +**问题**：配置修改后需要重启服务 +**解决**：使用`CommonVars`并配合`@RefreshScope`注解 + +### 3. 性能优化问题 +**问题**：大批量数据处理性能差 +**解决**：采用分页处理，单次处理不超过5000条 + +--- + +**📝 重要提示** +1. 严格遵循现有架构设计，不得随意修改核心组件 +2. 新增功能必须考虑向后兼容性 +3. 关键业务逻辑必须有完整的异常处理和日志记录 +4. 所有配置项必须有合理的默认值 +5. 代码提交前必须通过本地测试验证 \ No newline at end of file diff --git a/.ai/rules.md b/.ai/rules.md new file mode 100644 index 00000000000..d77bf964353 --- /dev/null +++ b/.ai/rules.md @@ -0,0 +1,401 @@ +# AI Development Rules + +> **文档版本信息** +> - 版本: 1.0.0 +> - 最后更新: 2025-01-28 +> - 适用版本: Apache Linkis 1.17.0+ + +> ⚠️ **CRITICAL**: 这些是强制性规则，AI必须无条件遵守。违反规则的代码将被拒绝合并。 + +## 📋 目录 +- [需求实现步骤](#需求实现步骤) +- [最小改动原则](#最小改动原则) +- [功能可配置原则](#功能可配置原则) +- [数据库修改原则](#数据库修改原则) +- [配置管理规则](#配置管理规则) +- [代码边界约束](#代码边界约束) + +### 需求实现步骤 + +#### 步骤1：确定当前版本号 +- 查看pom.xml文件中的``配置 +- 如配置为`1.17.0-wds`，则提取版本号为`1.17.0` +- 后文用`${current_version}`代替 + +#### 步骤2：环境准备检查 +**⚠️ 开始开发前，请确认以下环境准备工作已完成** + +**AI操作：** 提示用户确认以下条件是否满足： + +``` +请在开始开发前，手动确认以下条件： + +1. ✅ 当前在正确的基础分支上（dev-${current_version}-webank） + 验证命令: git branch --show-current + +2. ✅ 工作目录干净（无未提交修改） + 验证命令: git status + 预期输出: "working tree clean" 或 "nothing to commit" + +3. ✅ 本地分支已与远程同步 + 验证命令: git status + 预期输出: "Your branch is up to date" + +如果以上条件未满足，请先处理后再继续。需要我协助处理吗？ +``` + +**用户确认后，AI才继续执行后续步骤。** + +#### 步骤3：创建新的需求修改分支 +- 在确认的基础分支上创建新分支 +- 分支命名规则：`feature/${current_version}-<需求简述>` + +#### 步骤4：创建文档目录 +- 创建目录：`docs/${current_version}/requirements`和`docs/${current_version}/design` +- 如果目录已存在则跳过 + +#### 步骤5：创建需求文档 +- 按项目标准需求文档格式创建markdown文档 +- 存放路径：`docs/${current_version}/requirements/<需求名称>.md` + +#### 步骤6：创建设计文档 +- 按项目标准设计文档格式创建markdown文档 +- 存放路径：`docs/${current_version}/design/<需求名称>-design.md` + +#### 步骤7：代码开发 +- 按需求和设计文档进行开发 +- 必须遵循本文件中的所有原则（最小改动、功能可配置等） + +### 最小改动原则 +- 所有功能实现必须遵循最小改动原则，修改内容尽量不影响现有功能。 + +### 功能可配置原则 +- 所有功能必须增加功能开关，在开关关闭时功能相当于回退到上一个版本。开关配置遵循配置管理规则 + +### 数据库修改原则 +- 在能不改动现有表结构和表数据的情况下尽量不改动 +- 对于必须改动表结构和数据的情况下，将改动存档。具体路径如下 + - DDL：`linkis-dist/package/db/linkis_ddl.sql` + - DML：`linkis-dist/package/db/linkis_dml.sql` + +### 配置管理规则 +- 所有配置统一使用`org.apache.linkis.common.conf.CommonVars` +- 参考示例：`org.apache.linkis.jobhistory.conf.JobhistoryConfiguration` +- 配置存放位置：当前模块的conf目录，一般为xxxConfiguration类 + +### 代码边界约束 + +#### 🚫 禁止操作 +- **数据库结构**：除非明确指定，严禁修改现有表结构 +- **第三方依赖**：不允许引入新的第三方依赖库 +- **核心接口**：不得修改现有公共接口的签名 + +#### ✅ 允许操作 +- **新增功能**：在不破坏现有逻辑的前提下扩展功能 +- **新增配置**：在现有配置文件中新增配置项 +- **新增表字段**：在现有表基础上新增字段 + +### 其它规则 +- 所有功能只用实现后端接口功能，无需考虑前端设计和开发 + +--- + +# 开发决策流程图 + +## 🤔 决策1：是否需要修改数据库？ + +``` +┌─────────────────────┐ +│ 需求分析开始 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────┐ + │ 需要修改数据库? │ + └──┬───────┬───┘ + │ │ + 是 │ │ 否 + │ │ + ▼ ▼ + ┌─────────┐ ┌──────────────┐ + │ 能否通过 │ │ 直接开发代码 │ + │新增字段实现?│ │ (添加功能开关) │ + └─┬───┬──┘ └──────────────┘ + │ │ + 是 │ │ 否 + │ │ + ▼ ▼ + 优先 修改表结构 + 新增 (需评审) + 字段 + │ │ + └─┬─┘ + │ + ▼ + ┌──────────────────┐ + │ 记录DDL/DML变更 │ + │ + 版本信息 │ + │ + 需求说明 │ + └──────────────────┘ +``` + +**决策规则：** +1. **优先选择**: 新增字段（向后兼容） +2. **谨慎选择**: 修改字段类型、删除字段（需评审） +3. **必须记录**: 所有DDL/DML变更到规定文件 +4. **必须标注**: 版本号、需求描述、日期 + +--- + +## 🔧 决策2：如何选择开发语言？ + +``` +┌─────────────────────┐ +│ 功能模块分析 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────┐ + │ 模块类型？ │ + └──┬────┬────┬─┘ + │ │ │ + │ │ └──────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌────────┐ ┌──────────┐ ┌─────────┐ + │REST API│ │计算逻辑 │ │配置类 │ + │Service │ │RPC通信 │ │ │ + │Entity │ │复杂业务 │ │ │ + └────┬───┘ └────┬─────┘ └────┬────┘ + │ │ │ + ▼ ▼ ▼ + Java Scala Scala +``` + +**选择规则：** +- **Java**: REST API、Service层、Entity类、DAO接口 +- **Scala**: 计算逻辑、RPC通信、复杂业务处理、配置对象（Configuration） + +--- + +## ⚙️ 决策3：新功能如何设计开关？ + +``` +┌─────────────────────┐ +│ 新增功能需求 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────────┐ + │ 1. 定义功能开关 │ + │ (默认 false) │ + └──────┬───────────┘ + │ + ▼ + ┌──────────────────┐ + │ 2. 实现新功能逻辑 │ + └──────┬───────────┘ + │ + ▼ + ┌──────────────────┐ + │ 3. 保留旧逻辑 │ + │ (作为降级方案) │ + └──────┬───────────┘ + │ + ▼ + ┌──────────────────────┐ + │ 4. 在代码中检查开关 │ + │ if (ENABLE.getValue())│ + │ 新逻辑 │ + │ else │ + │ 旧逻辑(降级) │ + └──────────────────────┘ +``` + +**配置示例：** +```scala +// 在 xxxConfiguration.scala 中 +object NewFeatureConfiguration { + val ENABLE = CommonVars("linkis.new.feature.enable", false) + val BATCH_SIZE = CommonVars("linkis.new.feature.batch.size", 1000) + val TIMEOUT = CommonVars("linkis.new.feature.timeout", 30000L) +} +``` + +**代码示例：** +```java +public void executeFeature() { + // 检查功能开关 + if (!NewFeatureConfiguration.ENABLE.getValue()) { + executeLegacyLogic(); // 开关关闭时执行旧逻辑 + return; + } + + try { + executeNewFeature(); // 开关打开时执行新逻辑 + } catch (Exception e) { + logger.error("New feature failed, falling back", e); + executeLegacyLogic(); // 异常时降级到旧逻辑 + } +} +``` + +--- + +## 📊 决策4：是否需要创建新表？ + +``` +┌─────────────────────┐ +│ 数据存储需求 │ +└──────────┬──────────┘ + │ + ▼ + ┌─────────────────┐ + │ 能否利用现有表？ │ + └─┬──────────┬───┘ + │ │ + 是│ │否 + │ │ + ▼ ▼ + ┌─────────┐ ┌──────────────┐ + │ 新增字段 │ │ 是否核心业务表?│ + └─────────┘ └─┬──────────┬─┘ + │ │ + 是│ │否 + │ │ + ▼ ▼ + 需要架构评审 可创建新表 + │ │ + └────┬─────┘ + │ + ▼ + ┌──────────────┐ + │ 记录DDL到规定 │ + │ 文件并标注说明 │ + └──────────────┘ +``` + +**创建新表规则：** +1. **优先复用**: 检查是否能通过现有表扩展实现 +2. **业务表评审**: 核心业务表需要架构评审 +3. **辅助表允许**: 日志表、临时表、配置表等可自行创建 +4. **必须记录**: DDL添加到 `linkis_ddl.sql` +5. **命名规范**: `linkis_[模块]_[功能]_[表名]` + +--- + +## 🔍 决策5：错误处理策略 + +``` +┌─────────────────────┐ +│ 发生异常 │ +└──────────┬──────────┘ + │ + ▼ + ┌──────────────────┐ + │ 异常类型判断 │ + └─┬──────┬────┬───┘ + │ │ │ + │ │ └────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────┐ ┌────────┐ ┌─────────┐ + │预期 │ │系统错误│ │未知错误 │ + │业务 │ │(IO/DB) │ │ │ + │异常 │ │ │ │ │ + └─┬───┘ └───┬────┘ └────┬────┘ + │ │ │ + │ │ │ + ▼ ▼ ▼ + 记录WARN 记录ERROR 记录ERROR + 返回友好 抛出包装后 抛出包装后 + 错误信息 业务异常 业务异常 + │ │ │ + └────┬────┴────────────────┘ + │ + ▼ + ┌─────────────────┐ + │ 使用LinkisException│ + │ 及其子类 │ + └─────────────────┘ +``` + +**异常处理原则：** +1. **不吞掉异常**: 必须记录日志或重新抛出 +2. **使用业务异常**: LinkisException及其子类 +3. **提供上下文**: 异常信息包含关键业务参数 +4. **分级处理**: WARN用于业务异常，ERROR用于系统异常 + +--- + +# 快速检查清单 + +开发完成后，请对照以下清单进行自检： + +## ✅ 代码规范检查 +- [ ] 所有配置使用 `CommonVars`，不硬编码 +- [ ] 字符编码使用 `StandardCharsets.UTF_8` +- [ ] 日志使用 `Logger`，不使用 `System.out` +- [ ] 异常处理使用 `LinkisException` 及其子类 +- [ ] REST接口返回 `Message` 统一体 + +## ✅ 功能设计检查 +- [ ] 新功能已添加开关配置（默认 `false`） +- [ ] 开关关闭时能降级到旧逻辑 +- [ ] 遵循最小改动原则 +- [ ] 代码有充分的日志记录 + +## ✅ 数据库变更检查 +- [ ] DDL变更已记录到 `linkis_ddl.sql` +- [ ] DML变更已记录到 `linkis_dml.sql` +- [ ] 变更脚本包含版本号、需求描述、日期 +- [ ] 优先使用新增字段而非修改字段 + +## ✅ 文档检查 +- [ ] 已创建需求文档 +- [ ] 已创建设计文档 +- [ ] 文档存放在正确的目录 +- [ ] API变更已更新对应模块文档 + +## ✅ 测试检查 +- [ ] 功能开关打开时，新功能正常工作 +- [ ] 功能开关关闭时，回退到旧逻辑 +- [ ] 异常情况能正确降级 +- [ ] 关键业务逻辑有单元测试 + +--- + +# 常见问题解答 + +## Q1: 如果现有表确实需要修改字段类型怎么办？ +**A:** +1. 先评估是否可以通过新增字段实现 +2. 如果必须修改，需要： + - 提供充分的理由和影响分析 + - 记录详细的DDL和数据迁移方案 + - 标注清楚版本和需求信息 + +## Q2: 功能开关关闭后，旧代码能删除吗？ +**A:** +- **至少保留一个大版本周期**（如1.17.0的新功能，至少保留到1.18.0） +- 确认新功能稳定运行至少3个月 +- 在删除前添加 TODO 注释说明删除计划 + +## Q3: 如何判断是否属于"最小改动"？ +**A:** +- ✅ 只修改必要的文件和代码行 +- ✅ 不改变现有接口签名 +- ✅ 不影响其他模块的功能 +- ❌ 大规模重构现有代码 +- ❌ 修改核心公共类 + +## Q4: 配置项命名有什么规范？ +**A:** +- 格式: `linkis.[模块].[功能].[属性]` +- 示例: `linkis.entrance.task.max.retry.times` +- 保持命名清晰、有意义 +- 避免使用缩写（除非是通用缩写如 max、min） + +--- + +**记住**: 规范不是束缚，而是为了保证系统的稳定性和可维护性！ diff --git a/.github/PULL_REQUEST_TEMPLATE_CN.md b/.github/PULL_REQUEST_TEMPLATE_CN.md new file mode 100644 index 00000000000..9680ade1706 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE_CN.md @@ -0,0 +1,84 @@ + + +## PR Title + + + +[][][] + +--- + + + +### What is the purpose of the change + + + + +### Related issues/PRs + + + +Related issues: +Related pr: + +### Brief change log + + + +- +- +- + +--- + +### Main Changes (Detailed) + + + +1. **Server-side changes**: + - + - + +2. **Client-side changes**: + - + - + +3. **Other changes**: + - + +### Related Modules + + + +- [] +- [] + +--- + +### Checklist + +- [ ] I have read the [Contributing Guidelines on pull requests](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#pull-requests). +- [ ] I have explained the need for this PR and the problem it solves +- [ ] I have explained the changes or the new features added to this PR +- [ ] I have added tests corresponding to this change +- [ ] I have updated the documentation to reflect this change +- [ ] I have verified that this change is backward compatible (If not, please discuss on the [Linkis mailing list](https://linkis.apache.org/community/how-to-subscribe) first) +- [ ] **If this is a code change**: I have written unit tests to fully verify the new behavior. diff --git a/.github/workflows/build-backend.yml b/.github/workflows/build-backend.yml index 04aac9632aa..6c9270d2db8 100644 --- a/.github/workflows/build-backend.yml +++ b/.github/workflows/build-backend.yml @@ -20,15 +20,18 @@ name: Build Backend on: [push, pull_request] env: - MAVEN_OPTS: -Dmaven.resolver.transport=wagon -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 + MAVEN_OPTS: -Dmaven.resolver.transport=wagon -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Xmx16g -XX:MetaspaceSize=4g -XX:ReservedCodeCacheSize=2g jobs: build-backend: runs-on: ubuntu-latest strategy: matrix: - spark: [2.4, 3.2] - hadoop: [2.7, 3.3] + profile: + - default + - spark-3 + - hadoop-3.3 + - spark-3-hadoop-3.3 steps: - name: Checkout uses: actions/checkout@v4 @@ -38,23 +41,26 @@ jobs: distribution: 'temurin' java-version: 8 cache: maven - - if: ${{matrix.spark == '2.4' && matrix.hadoop == '2.7'}} - name: build spark ${{matrix.spark}} and hadoop ${{matrix.hadoop}} + # Default: Spark 2.4 + Hadoop 2.7 + - if: ${{matrix.profile == 'default'}} + name: build default (Spark 2.4 + Hadoop 2.7) run: - ./mvnw clean package -Pspark-${{matrix.spark}} -Phadoop-${{matrix.hadoop}} -Dmaven.test.skip=true - - if: ${{matrix.spark == '3.2' && matrix.hadoop == '3.3'}} - name: build spark ${{matrix.spark}} and hadoop ${{matrix.hadoop}} + ./mvnw -T 4C clean package -Dmaven.test.skip=true + # Spark 3.4 only + - if: ${{matrix.profile == 'spark-3'}} + name: build Spark 3.4 (Hadoop 2.7) run: - ./mvnw clean package - - if: ${{matrix.spark == '3.2' && matrix.hadoop == '2.7'}} - name: build spark ${{matrix.spark}} and hadoop ${{matrix.hadoop}} + ./mvnw -T 4C clean package -Pspark-3 -Dmaven.test.skip=true + # Hadoop 3.3 only (with Spark 2.4) + - if: ${{matrix.profile == 'hadoop-3.3'}} + name: build Hadoop 3.3 (Spark 2.4) run: - ./mvnw clean package -Phadoop-${{matrix.hadoop}} -Dmaven.test.skip=true - - - if: ${{matrix.spark == '2.4' && matrix.hadoop == '3.3'}} - name: build spark ${{matrix.spark}} and hadoop ${{matrix.hadoop}} + ./mvnw -T 4C clean package -Phadoop-3.3 -Dmaven.test.skip=true + # Spark 3.4 + Hadoop 3.3 + - if: ${{matrix.profile == 'spark-3-hadoop-3.3'}} + name: build Spark 3.4 + Hadoop 3.3 run: - ./mvnw clean package -Pspark-${{matrix.spark}} -Dmaven.test.skip=true + ./mvnw -T 4C clean package -Pspark-3 -Phadoop-3.3 -Dmaven.test.skip=true # - name: Upload coverage to Codecov # uses: codecov/codecov-action@v3.0.0 # with: diff --git a/.gitignore b/.gitignore index 8610a1edd29..76aeae1caa9 100644 --- a/.gitignore +++ b/.gitignore @@ -33,4 +33,12 @@ target/ # log folder *.log logs/ -nohup.out \ No newline at end of file +nohup.out + +#claude +.claude + +nul + +#claude +.claude diff --git a/docs/1.17.0/design/aisql-starrocks-engine-switch-design.md b/docs/1.17.0/design/aisql-starrocks-engine-switch-design.md new file mode 100644 index 00000000000..b673b44054b --- /dev/null +++ b/docs/1.17.0/design/aisql-starrocks-engine-switch-design.md @@ -0,0 +1,926 @@ +# AISQL任务支持StarRocks引擎类型切换设计文档 + +## 1. 设计概述 + +### 1.1 目标 +为AISQL类型任务增加StarRocks引擎类型切换支持，通过runtime参数或脚本注释两种方式实现引擎切换，并集成Doctoris服务进行引擎决策。 + +### 1.2 设计原则 +- **最小改动原则**：在现有架构基础上扩展，不修改现有Spark/Hive引擎切换逻辑 +- **可配置原则**：通过功能开关控制，开关关闭时相当于回退到上一版本 +- **一致性原则**：与现有引擎切换机制保持一致的使用体验 + +### 1.3 适用范围 +- AISQL类型任务提交流程 +- 引擎类型切换逻辑 +- Doctoris服务调用 + +## 2. 整体架构 + +### 2.1 系统架构图 +``` +用户提交AISQL任务 + ↓ +Entrance服务接收 + ↓ +AISQLTransformInterceptor拦截器 + ↓ +1. 解析runtime参数 (ec.engine.type) +2. 解析脚本注释 (@set ec.engine.type=starrocks) +3. 解析模板配置 (ec.resource.name) + ↓ +判断是否指定StarRocks引擎？ + ↓ Yes +调用Doctoris服务（传递forceEngineType=starrocks） + ↓ +切换EngineTypeLabel为jdbc + ↓ +任务提交到JDBC EngineConn + ↓ +通过StarRocks数据源执行任务 +``` + +### 2.2 处理优先级 +``` +1. Runtime参数 (ec.engine.type=starrocks) + ↓ 若未设置 +2. 脚本注释 (@set ec.engine.type=starrocks) + ↓ 若未设置 +3. 模板配置 (ec.resource.name包含starrocks关键字) + ↓ 若未设置 +4. Doctoris智能选择（现有逻辑） + ↓ 若未启用 +5. 默认Spark引擎（现有逻辑） +``` + +## 3. 详细设计 + +### 3.1 配置设计 + +#### 3.1.1 新增配置项（EntranceConfiguration.scala） + +```scala +// StarRocks引擎切换功能开关 +val AISQL_STARROCKS_SWITCH = CommonVars("linkis.aisql.starrocks.switch", false) + +// 默认StarRocks引擎类型 +val AISQL_DEFAULT_STARROCKS_ENGINE_TYPE = + CommonVars("linkis.aisql.default.starrocks.engine.type", "jdbc-4") + +// StarRocks模板关键字配置 +val AISQL_STARROCKS_TEMPLATE_KEYS = + CommonVars("linkis.aisql.starrocks.template.keys", "starrocks") + +// StarRocks数据源名称前缀配置 +val AISQL_STARROCKS_DATASOURCE_PREFIX = + CommonVars("linkis.aisql.starrocks.datasource.prefix", "starrocks_") + +// 用户白名单配置 +val AISQL_STARROCKS_WHITELIST_USERS = + CommonVars("linkis.aisql.starrocks.whitelist.users", "") + +// 部门白名单配置 +val AISQL_STARROCKS_WHITELIST_DEPARTMENTS = + CommonVars("linkis.aisql.starrocks.whitelist.departments", "") +``` + +#### 3.1.2 配置说明 + +| 配置项 | 默认值 | 说明 | +|--------|--------|------| +| linkis.aisql.starrocks.switch | false | StarRocks引擎切换功能开关 | +| linkis.aisql.default.starrocks.engine.type | jdbc-4 | 默认StarRocks引擎类型（jdbc引擎版本） | +| linkis.aisql.starrocks.template.keys | starrocks | 模板关键字，用于识别StarRocks模板 | +| linkis.aisql.starrocks.datasource.prefix | starrocks_ | StarRocks数据源名称前缀 | +| linkis.aisql.starrocks.whitelist.users | 空 | 用户白名单（逗号分隔），为空时所有用户可用 | +| linkis.aisql.starrocks.whitelist.departments | 空 | 部门白名单（逗号分隔），为空时所有部门可用 | + +### 3.2 脚本注释解析设计 + +#### 3.2.1 新增配置键（TemplateConfUtils.scala） + +```scala +object TemplateConfUtils { + // 现有配置 + val confTemplateNameKey = "ec.resource.name" + val confFixedEngineConnLabelKey = "ec.fixed.sessionId" + + // 新增：引擎类型配置键 + val confEngineTypeKey = "ec.engine.type" +} +``` + +#### 3.2.2 注释格式支持 + +支持三种注释格式： +- **SQL/HQL格式**：`---@set ec.engine.type=starrocks` +- **Python/Shell格式**：`##@set ec.engine.type=starrocks` +- **Scala格式**：`///@set ec.engine.type=starrocks` + +#### 3.2.3 实现逻辑 + +利用现有的`getCustomTemplateConfName`方法机制，扩展支持解析`ec.engine.type`配置： + +```scala +def getCustomEngineType(code: String, languageType: String): String = { + val confPattern = languageType.toLowerCase match { + case x if x.contains("python") || x.contains("shell") => + s"##@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s#]+)".r + case x if x.contains("scala") => + s"///@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s/]+)".r + case _ => + s"---@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s-]+)".r + } + + confPattern.findFirstMatchIn(code) match { + case Some(m) => m.group(1).trim + case None => null + } +} +``` + +### 3.3 引擎切换逻辑设计 + +#### 3.3.1 白名单检查设计 + +在进行引擎切换之前，需要先检查用户是否有权限使用StarRocks引擎： + +```scala +/** + * 检查用户是否在StarRocks白名单中 + * @param submitUser 提交任务的用户 + * @return true表示用户在白名单中或白名单为空（允许所有用户），false表示不在白名单中 + */ +private def isUserInStarRocksWhitelist(submitUser: String): Boolean = { + val whitelistUsers = AISQL_STARROCKS_WHITELIST_USERS.getValue + val whitelistDepartments = AISQL_STARROCKS_WHITELIST_DEPARTMENTS.getValue + + // 如果白名单都为空，则允许所有用户使用 + if (StringUtils.isBlank(whitelistUsers) && StringUtils.isBlank(whitelistDepartments)) { + return true + } + + // 检查用户白名单 + if (StringUtils.isNotBlank(whitelistUsers)) { + val users = whitelistUsers.split(",").map(_.trim) + if (users.contains(submitUser)) { + logger.info(s"User $submitUser is in StarRocks whitelist (user)") + return true + } + } + + // 检查部门白名单 + if (StringUtils.isNotBlank(whitelistDepartments)) { + val userDepartmentId = EntranceUtils.getUserDepartmentId(submitUser) + if (StringUtils.isNotBlank(userDepartmentId)) { + val departments = whitelistDepartments.split(",").map(_.trim) + if (departments.contains(userDepartmentId)) { + logger.info(s"User $submitUser (department: $userDepartmentId) is in StarRocks whitelist (department)") + return true + } + } + } + + logger.warn(s"User $submitUser is not in StarRocks whitelist, will use default engine selection") + false +} +``` + +#### 3.3.2 AISQLTransformInterceptor改造 + +在`AISQLTransformInterceptor.apply()`方法中增加StarRocks引擎处理逻辑： + +```scala +override def apply(task: EntranceJob, logAppender: lang.StringBuilder): EntranceJob = { + // 功能开关检查 + if (!AISQL_STARROCKS_SWITCH.getValue) { + return applyExistingLogic(task, logAppender) // 现有逻辑 + } + + val jobRequest = task.getJobRequest + val params = jobRequest.getParams + val labels = jobRequest.getLabels + + // 1. 检查runtime参数 + val runtimeEngineType = getRuntimeEngineType(params) + + // 2. 检查脚本注释 + val scriptEngineType = if (runtimeEngineType == null) { + TemplateConfUtils.getCustomEngineType( + jobRequest.getExecutionCode, + CodeAndRunTypeUtils.getLanguageTypeByRunType(jobRequest.getRunType) + ) + } else null + + // 3. 检查模板配置 + val templateEngineType = if (runtimeEngineType == null && scriptEngineType == null) { + getEngineTypeFromTemplate(jobRequest) + } else null + + // 确定最终引擎类型 + val targetEngineType = Option(runtimeEngineType) + .orElse(Option(scriptEngineType)) + .orElse(Option(templateEngineType)) + .orNull + + // 如果指定了starrocks引擎 + if ("starrocks".equalsIgnoreCase(targetEngineType)) { + // 白名单检查 + if (!isUserInStarRocksWhitelist(jobRequest.getSubmitUser)) { + logAppender.append( + LogUtils.generateWarn( + s"User ${jobRequest.getSubmitUser} is not in StarRocks whitelist, using default engine selection\n" + ) + ) + // 继续执行现有逻辑（Spark/Hive切换） + return applyExistingLogic(task, logAppender) + } + + // 切换到JDBC引擎 + changeToStarRocksEngine(labels, logAppender, params) + } else { + // 执行现有逻辑（Spark/Hive切换） + applyExistingLogic(task, logAppender) + } + + task +} +``` + +#### 3.3.2 StarRocks引擎切换实现 + +```scala +private def changeToStarRocksEngine( + labels: util.List[Label[_]], + logAppender: lang.StringBuilder, + params: util.Map[String, AnyRef] +): Unit = { + + logAppender.append("Switching to StarRocks engine...\n") + + // 1. 移除现有EngineTypeLabel + val iterator = labels.iterator() + while (iterator.hasNext) { + val label = iterator.next() + if (label.isInstanceOf[EngineTypeLabel]) { + iterator.remove() + } + } + + // 2. 创建JDBC引擎Label + val jdbcEngineType = AISQL_DEFAULT_STARROCKS_ENGINE_TYPE.getValue + val Array(engine, version) = jdbcEngineType.split("-", 2) + val jdbcLabel = new EngineTypeLabel() + jdbcLabel.setEngineType(engine) + jdbcLabel.setVersion(version) + labels.add(jdbcLabel) + + // 3. 添加StarRocks标识到runtime参数（用于后续JDBC引擎识别） + val runtimeMap = params.getOrDefault( + JobRequestConstants.JOB_REQUEST_RUNTIME_PARAMS, + new util.HashMap[String, AnyRef]() + ).asInstanceOf[util.Map[String, AnyRef]] + + runtimeMap.put("linkis.jdbc.engine.type", "starrocks") + params.put(JobRequestConstants.JOB_REQUEST_RUNTIME_PARAMS, runtimeMap) + + logAppender.append(s"Engine switched to StarRocks (JDBC engine: $jdbcEngineType)\n") +} +``` + +#### 3.3.3 Runtime参数获取 + +```scala +private def getRuntimeEngineType(params: util.Map[String, AnyRef]): String = { + if (params == null) return null + + val runtimeParams = params.get(JobRequestConstants.JOB_REQUEST_RUNTIME_PARAMS) + if (runtimeParams == null) return null + + runtimeParams.asInstanceOf[util.Map[String, AnyRef]] + .get(TemplateConfUtils.confEngineTypeKey) match { + case null => null + case value => value.toString + } +} +``` + +#### 3.3.4 模板配置获取 + +```scala +private def getEngineTypeFromTemplate(jobRequest: JobRequest): String = { + val templateName = TemplateConfUtils.getCustomTemplateConfName( + jobRequest.getExecutionCode, + CodeAndRunTypeUtils.getLanguageTypeByRunType(jobRequest.getRunType) + ) + + if (templateName == null) return null + + // 检查模板名称是否包含StarRocks关键字 + val starrocksKeys = AISQL_STARROCKS_TEMPLATE_KEYS.getValue.split(",") + if (starrocksKeys.exists(key => templateName.toLowerCase.contains(key.toLowerCase))) { + "starrocks" + } else { + null + } +} +``` + +### 3.4 Doctoris服务集成设计 + +#### 3.4.1 接口扩展（EntranceUtils.scala） + +修改`getDynamicEngineType`方法，支持传递强制引擎类型参数： + +```scala +def getDynamicEngineType( + sql: String, + logAppender: lang.StringBuilder, + forceEngineType: String = null // 新增：强制引擎类型参数 +): String = { + + if (!EntranceConfiguration.AI_SQL_DYNAMIC_ENGINE_SWITCH) { + if (forceEngineType != null) return forceEngineType + return defaultEngineType + } + + val params = new util.HashMap[String, AnyRef]() + params.put("sql", sql) + params.put("highStability", "") + params.put("queueResourceUsage", "") + + // 新增：添加强制引擎类型标识 + if (forceEngineType != null && forceEngineType.nonEmpty) { + params.put("forceEngineType", forceEngineType) + logAppender.append(s"Force engine type: $forceEngineType\n") + } + + val request = DoctorEngineRequest( + EntranceConfiguration.LINKIS_SYSTEM_NAME, + EntranceConfiguration.DOCTOR_CLUSTER, + sql, + params + ) + + val response = callDoctorService(request, logAppender) + response.result +} +``` + +#### 3.4.2 调用时机 + +在`AISQLTransformInterceptor`中，当检测到需要使用StarRocks引擎时： + +```scala +if ("starrocks".equalsIgnoreCase(targetEngineType)) { + // 调用Doctoris服务，传递强制引擎类型 + val confirmedEngineType = EntranceUtils.getDynamicEngineType( + jobRequest.getExecutionCode, + logAppender, + forceEngineType = "starrocks" // 传递强制参数 + ) + + // 切换到JDBC引擎 + changeToStarRocksEngine(labels, logAppender, params) +} +``` + +### 3.5 数据流设计 + +#### 3.5.1 任务提交数据流 + +``` +1. 用户提交任务 + { + "executionCode": "---@set ec.engine.type=starrocks\nSELECT * FROM table", + "runType": "aisql", + "params": { + "runtime": {} + } + } + +2. AISQLTransformInterceptor处理 + - 解析脚本注释，提取 ec.engine.type=starrocks + - 检查功能开关：linkis.aisql.starrocks.switch = true + - 决定切换到StarRocks引擎 + +3. 调用Doctoris服务 + POST /api/v1/external/engine/diagnose + Body: { + "sql": "SELECT * FROM table", + "forceEngineType": "starrocks" + } + Response: { + "engine": "starrocks", + "reason": "Force engine type specified" + } + +4. 修改JobRequest + - 移除现有EngineTypeLabel + - 添加新的EngineTypeLabel(engine=jdbc, version=4) + - 添加runtime参数：linkis.jdbc.engine.type=starrocks + +5. 任务路由到JDBC EngineConn + - JDBC引擎识别linkis.jdbc.engine.type=starrocks + - 查询用户的StarRocks数据源 + - 通过JDBC连接执行SQL +``` + +#### 3.5.2 引擎标签变更 + +``` +原始Label： +[EngineTypeLabel(engineType=spark, version=3.4.4)] + +↓ 检测到 ec.engine.type=starrocks + +新Label： +[EngineTypeLabel(engineType=jdbc, version=4)] + ++ Runtime参数： +{ + "linkis.jdbc.engine.type": "starrocks" +} +``` + +## 4. 接口设计 + +### 4.1 内部接口 + +#### 4.1.1 TemplateConfUtils新增方法 + +```scala +/** + * 从脚本代码中提取引擎类型配置 + * @param code 脚本代码 + * @param languageType 语言类型 + * @return 引擎类型，如"starrocks"、"spark"、"hive"，未找到返回null + */ +def getCustomEngineType(code: String, languageType: String): String +``` + +#### 4.1.2 AISQLTransformInterceptor新增私有方法 + +```scala +/** + * 从runtime参数中获取引擎类型 + */ +private def getRuntimeEngineType(params: util.Map[String, AnyRef]): String + +/** + * 从模板配置中获取引擎类型 + */ +private def getEngineTypeFromTemplate(jobRequest: JobRequest): String + +/** + * 切换到StarRocks引擎 + */ +private def changeToStarRocksEngine( + labels: util.List[Label[_]], + logAppender: lang.StringBuilder, + params: util.Map[String, AnyRef] +): Unit +``` + +#### 4.1.3 EntranceUtils方法签名变更 + +```scala +/** + * 获取动态引擎类型 + * @param sql SQL语句 + * @param logAppender 日志追加器 + * @param forceEngineType 强制引擎类型（可选），如"starrocks" + * @return 引擎类型 + */ +def getDynamicEngineType( + sql: String, + logAppender: lang.StringBuilder, + forceEngineType: String = null +): String +``` + +### 4.2 外部接口 + +#### 4.2.1 任务提交接口（无变更） + +保持现有任务提交接口不变，通过扩展参数支持新功能： + +``` +POST /api/rest_j/v1/entrance/submit + +Request Body: +{ + "executionCode": "SELECT * FROM table", + "runType": "aisql", + "params": { + "runtime": { + "ec.engine.type": "starrocks" // 新增参数 + } + } +} +``` + +#### 4.2.2 Doctoris服务接口 + +``` +POST {DOCTOR_URL}/api/v1/external/engine/diagnose + +Request: +{ + "appId": "linkis", + "cluster": "default", + "sql": "SELECT * FROM table", + "params": { + "forceEngineType": "starrocks" // 新增参数 + } +} + +Response: +{ + "code": 0, + "data": { + "engine": "starrocks", + "reason": "Force engine type specified", + "duration": 50 + } +} +``` + +## 5. 异常处理 + +### 5.1 异常场景 + +| 异常场景 | 处理策略 | +|----------|----------| +| StarRocks功能开关关闭 | 忽略StarRocks配置，执行现有Spark/Hive切换逻辑 | +| 无效的引擎类型值 | 记录警告日志，使用默认引擎类型 | +| Doctoris服务调用失败 | 记录错误日志，降级到默认引擎类型 | +| JDBC引擎不可用 | 任务提交失败，返回明确错误信息 | +| StarRocks数据源不存在 | 任务执行失败，提示配置数据源 | + +### 5.2 日志规范 + +```scala +// INFO级别：关键流程节点 +logger.info(s"AISQL task switches to StarRocks engine for user $username") + +// WARN级别：降级处理 +logger.warn(s"Invalid engine type specified: $engineType, fallback to default") + +// ERROR级别：异常错误 +logger.error(s"Failed to switch to StarRocks engine for task $taskId", exception) + +// DEBUG级别：详细调试信息 +logger.debug(s"Parsing engine type from script: $code") +``` + +## 6. 测试设计 + +### 6.1 单元测试 + +#### 6.1.1 TemplateConfUtils测试 + +```scala +class TemplateConfUtilsTest { + + test("extract starrocks engine type from SQL comment") { + val code = "---@set ec.engine.type=starrocks\nSELECT * FROM table" + val result = TemplateConfUtils.getCustomEngineType(code, "sql") + assert(result == "starrocks") + } + + test("extract starrocks engine type from Python comment") { + val code = "##@set ec.engine.type=starrocks\nSELECT COUNT(*) FROM table" + val result = TemplateConfUtils.getCustomEngineType(code, "python") + assert(result == "starrocks") + } + + test("return null when no engine type specified") { + val code = "SELECT * FROM table" + val result = TemplateConfUtils.getCustomEngineType(code, "sql") + assert(result == null) + } +} +``` + +#### 6.1.2 AISQLTransformInterceptor测试 + +```scala +class AISQLTransformInterceptorTest { + + test("switch to StarRocks via runtime parameter") { + val jobRequest = createJobRequest( + code = "SELECT * FROM table", + runtime = Map("ec.engine.type" -> "starrocks") + ) + val task = new EntranceJob() + task.setJobRequest(jobRequest) + + interceptor.apply(task, new StringBuilder()) + + val engineLabel = getEngineLabel(task) + assert(engineLabel.getEngineType == "jdbc") + } + + test("switch to StarRocks via script comment") { + val jobRequest = createJobRequest( + code = "---@set ec.engine.type=starrocks\nSELECT * FROM table" + ) + val task = new EntranceJob() + task.setJobRequest(jobRequest) + + interceptor.apply(task, new StringBuilder()) + + val engineLabel = getEngineLabel(task) + assert(engineLabel.getEngineType == "jdbc") + } + + test("runtime parameter takes precedence over script comment") { + val jobRequest = createJobRequest( + code = "---@set ec.engine.type=spark\nSELECT * FROM table", + runtime = Map("ec.engine.type" -> "starrocks") + ) + val task = new EntranceJob() + task.setJobRequest(jobRequest) + + interceptor.apply(task, new StringBuilder()) + + val engineLabel = getEngineLabel(task) + assert(engineLabel.getEngineType == "jdbc") // 使用runtime的starrocks + } +} +``` + +### 6.2 集成测试 + +#### 6.2.1 端到端测试用例 + +```bash +# 测试1：通过runtime参数切换StarRocks引擎 +curl -X POST http://localhost:9001/api/rest_j/v1/entrance/submit \ + -H "Content-Type: application/json" \ + -H "Token-User: testuser" \ + -d '{ + "executionCode": "SELECT * FROM starrocks_table LIMIT 10", + "runType": "aisql", + "params": { + "runtime": { + "ec.engine.type": "starrocks" + } + } + }' + +# 预期结果：任务成功提交，引擎类型为jdbc，执行成功 + +# 测试2：通过脚本注释切换StarRocks引擎 +curl -X POST http://localhost:9001/api/rest_j/v1/entrance/submit \ + -H "Content-Type: application/json" \ + -H "Token-User: testuser" \ + -d '{ + "executionCode": "---@set ec.engine.type=starrocks\nSELECT COUNT(*) FROM user_table", + "runType": "aisql", + "params": {} + }' + +# 预期结果：任务成功提交，引擎类型为jdbc，执行成功 + +# 测试3：功能开关关闭 +# 配置：linkis.aisql.starrocks.switch=false +curl -X POST http://localhost:9001/api/rest_j/v1/entrance/submit \ + -H "Content-Type: application/json" \ + -H "Token-User: testuser" \ + -d '{ + "executionCode": "---@set ec.engine.type=starrocks\nSELECT * FROM table", + "runType": "aisql", + "params": {} + }' + +# 预期结果：忽略StarRocks配置，使用默认Spark引擎 +``` + +### 6.3 性能测试 + +测试指标： +- 参数解析耗时 < 10ms +- 引擎切换逻辑耗时 < 5ms +- 任务提交总耗时增加 < 20ms + +## 7. 部署方案 + +### 7.1 部署步骤 + +1. **编译打包** + ```bash + mvn clean package -Dmaven.test.skip=true + ``` + +2. **停止Entrance服务** + ```bash + sh sbin/linkis-daemon.sh stop entrance + ``` + +3. **备份原有文件** + ```bash + cp lib/linkis-spring-cloud-services/linkis-entrance.jar \ + lib/linkis-spring-cloud-services/linkis-entrance.jar.bak + ``` + +4. **替换新文件** + ```bash + cp linkis-computation-governance/linkis-entrance/target/linkis-entrance.jar \ + lib/linkis-spring-cloud-services/ + ``` + +5. **配置文件修改**（linkis-entrance.properties） + ```properties + # 启用StarRocks引擎切换功能 + linkis.aisql.starrocks.switch=true + + # StarRocks引擎类型（jdbc-4表示jdbc引擎版本4） + linkis.aisql.default.starrocks.engine.type=jdbc-4 + + # StarRocks模板关键字 + linkis.aisql.starrocks.template.keys=starrocks + ``` + +6. **启动Entrance服务** + ```bash + sh sbin/linkis-daemon.sh start entrance + ``` + +7. **验证功能** + ```bash + # 查看日志确认配置加载 + tail -f logs/linkis-entrance-gc.log | grep "starrocks" + + # 提交测试任务 + sh bin/linkis-cli -engineType aisql -code "---@set ec.engine.type=starrocks\nSELECT 1" -runtimeMap ec.engine.type=starrocks + ``` + +### 7.2 回滚方案 + +如果部署后出现问题，执行以下回滚步骤： + +1. **停止服务** + ```bash + sh sbin/linkis-daemon.sh stop entrance + ``` + +2. **恢复备份文件** + ```bash + mv lib/linkis-spring-cloud-services/linkis-entrance.jar.bak \ + lib/linkis-spring-cloud-services/linkis-entrance.jar + ``` + +3. **配置文件回滚** + ```properties + # 关闭StarRocks功能 + linkis.aisql.starrocks.switch=false + ``` + +4. **启动服务** + ```bash + sh sbin/linkis-daemon.sh start entrance + ``` + +### 7.3 灰度发布方案 + +1. **阶段1：内部测试环境**（1-2天） + - 部署到测试环境 + - 开启功能开关 + - 内部人员测试验证 + +2. **阶段2：生产环境灰度**（3-5天） + - 仅对特定用户组开启功能 + - 通过用户白名单控制 + - 监控任务成功率和性能指标 + +3. **阶段3：全量发布**（7天后） + - 确认无问题后全量开启 + - 持续监控一周 + +## 8. 监控告警 + +### 8.1 监控指标 + +| 指标 | 说明 | 告警阈值 | +|------|------|----------| +| starrocks_engine_switch_count | StarRocks引擎切换次数 | - | +| starrocks_engine_switch_success_rate | 切换成功率 | < 95% | +| starrocks_task_execution_time | 任务执行时间 | > 60s (P95) | +| starrocks_task_fail_count | 任务失败次数 | > 10次/小时 | +| doctoris_call_timeout_count | Doctoris调用超时次数 | > 5次/小时 | + +### 8.2 日志监控 + +关键日志关键字： +- `Switching to StarRocks engine` +- `Force engine type: starrocks` +- `Failed to switch to StarRocks engine` +- `Invalid engine type specified` + +## 9. 风险评估与应对 + +### 9.1 技术风险 + +| 风险 | 影响 | 概率 | 应对措施 | +|------|------|------|----------| +| StarRocks数据源配置错误 | 任务执行失败 | 中 | 提供详细的错误提示和配置文档 | +| JDBC连接池资源耗尽 | 后续任务阻塞 | 低 | 配置合理的连接池大小和超时时间 | +| Doctoris服务不稳定 | 引擎选择失败 | 中 | 实现降级逻辑，服务异常时使用默认配置 | +| 配置解析性能问题 | 任务提交变慢 | 低 | 优化正则表达式，添加缓存机制 | + +### 9.2 业务风险 + +| 风险 | 影响 | 概率 | 应对措施 | +|------|------|------|----------| +| 用户误配置引擎类型 | 任务失败或结果错误 | 中 | 添加引擎类型有效性校验 | +| 现有任务受影响 | 兼容性问题 | 低 | 功能开关默认关闭，逐步开启 | +| 文档不完善 | 用户使用困难 | 中 | 编写详细使用文档和示例 | + +## 10. 兼容性说明 + +### 10.1 向后兼容 +- 功能开关默认关闭（`linkis.aisql.starrocks.switch=false`） +- 不影响现有Spark/Hive引擎切换逻辑 +- 不修改现有接口签名和返回结构 + +### 10.2 版本依赖 +- 最低支持版本：Linkis 1.17.0 +- JDBC引擎插件版本：jdbc-4 +- StarRocks数据源管理模块已部署 + +### 10.3 升级影响 +- 升级时无需修改现有任务配置 +- 升级后需手动开启功能开关 +- 需要配置StarRocks相关参数 + +## 11. 文档清单 + +### 11.1 开发文档 +- [x] 需求文档：`docs/1.17.0/requirements/aisql-starrocks-engine-switch.md` +- [x] 设计文档：`docs/1.17.0/design/aisql-starrocks-engine-switch-design.md` + +### 11.2 用户文档（待补充） +- [ ] 用户使用指南：如何配置和使用StarRocks引擎 +- [ ] 配置参数说明：所有相关配置项的详细说明 +- [ ] 常见问题FAQ：常见问题和解决方案 + +### 11.3 运维文档（待补充） +- [ ] 部署指南：详细部署步骤和验证方法 +- [ ] 监控运维手册：监控指标和告警处理 +- [ ] 故障排查手册：常见故障和排查方法 + +## 12. 变更清单 + +### 12.1 新增文件 +- `docs/1.17.0/requirements/aisql-starrocks-engine-switch.md` - 需求文档 +- `docs/1.17.0/design/aisql-starrocks-engine-switch-design.md` - 设计文档 + +### 12.2 修改文件 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala` - 新增配置项 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala` - 新增引擎类型解析方法 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala` - 新增StarRocks切换逻辑 +- `linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala` - 扩展Doctoris调用接口 + +### 12.3 数据库变更 +无数据库变更 + +### 12.4 配置文件变更 +- `conf/linkis-entrance.properties` - 新增StarRocks相关配置项 + +## 13. 质量检查清单 + +- [ ] 代码符合项目规范（Java/Scala编码规范） +- [ ] 异常处理完整（try-catch、日志记录） +- [ ] 日志记录充分（INFO/WARN/ERROR/DEBUG） +- [ ] 单元测试覆盖（核心逻辑测试覆盖率>80%） +- [ ] 配置开关完整（功能开关、默认值配置） +- [ ] 向后兼容性检查（不影响现有功能） +- [ ] 性能测试通过（满足性能要求） +- [ ] 安全性检查（权限验证、参数校验） +- [ ] 文档完整性（需求、设计、用户、运维文档） + +## 14. 附录 + +### 14.1 相关代码文件路径 + +| 文件 | 路径 | +|------|------| +| EntranceConfiguration | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala | +| TemplateConfUtils | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala | +| AISQLTransformInterceptor | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala | +| EntranceUtils | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala | +| JDBCConfiguration | linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/conf/JDBCConfiguration.scala | + +### 14.2 参考资料 +- Apache Linkis官方文档：https://linkis.apache.org +- StarRocks官方文档：https://docs.starrocks.io +- JDBC标准文档：https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/ + +--- + +**文档版本**：v1.0 +**创建日期**：2025-10-27 +**作者**：AI +**审核状态**：待审核 diff --git a/docs/1.17.0/design/resultset-field-truncation-design.md b/docs/1.17.0/design/resultset-field-truncation-design.md new file mode 100644 index 00000000000..3b87f7bc8b3 --- /dev/null +++ b/docs/1.17.0/design/resultset-field-truncation-design.md @@ -0,0 +1,503 @@ +# 结果集字段截取功能设计文档 + +## 文档信息 +| 项目 | 信息 | +|-----|------| +| 文档版本 | v1.1 (已实现) | +| 创建日期 | 2025-10-27 | +| 更新日期 | 2025-10-30 | +| 当前版本 | Linkis 1.17.0 | +| 负责模块 | linkis-pes-publicservice + pipeline + linkis-storage | +| 开发分支 | feature/1.17.0-resultset-field-masking | +| 状态 | ✅ 开发完成，已测试 | + +--- + +## 实施总结 + +### 核心架构改进 + +本次实现将**敏感字段屏蔽**和**字段截取**两个功能统一到`ResultUtils`工具类中： + +**关键改进点**: +1. **统一工具类**: 将字段屏蔽和截取逻辑都提取到`ResultUtils`，实现完整的结果集处理能力 +2. **组合功能**: 提供`applyFieldMaskingAndTruncation()`方法支持两种功能同时使用 +3. **实体类封装**: 使用`FieldTruncationResult`和`OversizedFieldInfo`封装检测结果 +4. **标记机制**: 截取后的字段会在列名添加`(truncated to N chars)`后缀，用户可见 +5. **性能优化**: 通过缓存机制和早期退出策略优化大结果集处理性能 +6. **内存保护**: 实现内存使用监控和限制机制，防止OOM问题 + +### 代码修改统计 + +**新增文件**: +- `ResultUtils.java` (514行): 包含字段屏蔽和截取的完整实现 +- `FieldTruncationResult.java` (73行): 截取结果封装 +- `OversizedFieldInfo.java` (68行): 超长字段信息 + +**配置文件扩展**: +- `LinkisStorageConf.scala`: 新增4个配置项（功能开关、查看/导出最大长度、超长字段收集上限） +- `WorkSpaceConfiguration.java`: 新增功能开关配置 + +**主要功能文件**: +| 文件 | 改动说明 | +|------|---------| +| `FsRestfulApi.java` | 调用ResultUtils进行字段截取处理 | +| `CSVExecutor.scala` | 支持Pipeline truncate语法参数 | +| `ExcelExecutor.scala` | 支持Pipeline truncate语法参数 | +| `PipelineEngineConnExecutor.scala` | 解析truncate语法参数 | + +--- + +## 1. 设计概述 + +### 1.1 设计目标 +在不破坏现有功能的前提下,为结果集查看、下载、导出接口增加超长字段检测和截取能力。 + +### 1.2 设计原则 +- **最小改动原则**: 仅在必要位置增加检测和截取逻辑 +- **功能可配置原则**: 所有功能通过开关控制,默认关闭 +- **向下兼容原则**: 不修改现有接口签名,仅扩展返回数据结构 +- **代码复用原则**: ✅ 已实现 - 提取到统一工具类ResultUtils + +## 2. 架构设计 + +### 2.1 实际实现架构 + +**实际实现采用统一工具类模式**: + +``` +┌─────────────────────────────────────────────────┐ +│ ResultUtils 工具类 │ +│ ┌──────────────────────────────────────────┐ │ +│ │ 字段屏蔽功能模块 │ │ +│ │ - dealMaskedField() │ │ +│ │ - filterMaskedFieldsFromMetadata() │ │ +│ │ - removeFieldsFromContent() │ │ +│ └──────────────────────────────────────────┘ │ +│ ┌──────────────────────────────────────────┐ │ +│ │ 字段截取功能模块 ⭐ │ │ +│ │ - detectAndHandle() │ │ +│ │ - detectOversizedFields() │ │ +│ │ - truncateFields() │ │ +│ └──────────────────────────────────────────┘ │ +│ ┌──────────────────────────────────────────┐ │ +│ │ 组合功能模块 │ │ +│ │ - applyFieldMaskingAndTruncation() │ │ +│ └──────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────┘ + ↑ ↑ ↑ + │ │ │ + ┌──────┴───┐ ┌──────┴───┐ ┌──────┴───┐ + │FsRestful │ │ CSV │ │ Excel │ + │ API │ │ Executor │ │ Executor │ + └──────────┘ └──────────┘ └──────────┘ +``` + +**架构优势**: +1. **统一入口**: 所有字段处理逻辑集中在ResultUtils +2. **功能正交**: 屏蔽和截取可以独立使用或组合使用 +3. **代码复用**: Java和Scala代码都调用相同的工具类 + +### 2.2 与敏感字段屏蔽功能的关系 + +两个功能共享相同的架构和工具类，可以独立使用或组合使用： + +| 使用场景 | 方法 | 说明 | +|---------|------|------| +| 仅字段屏蔽 | `dealMaskedField()` | 移除指定字段 | +| 仅字段截取 | `detectAndHandle()` | 截取超长字段 | +| 同时使用 | `applyFieldMaskingAndTruncation()` | 先屏蔽后截取 | + +**处理顺序**: 屏蔽优先于截取 +1. 先移除maskedFields指定的字段 +2. 再对剩余字段进行超长检测和截取 + +## 3. 详细设计 + +### 3.1 配置类设计 (实际实现) + +#### LinkisStorageConf.scala (Storage层配置) +**位置**: `linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/conf/LinkisStorageConf.scala` + +✅ 实际新增配置项: +```scala +val FIELD_TRUNCATION_ENABLED = + CommonVars("linkis.resultset.field.truncation.enabled", false).getValue + +val FIELD_VIEW_MAX_LENGTH = + CommonVars("linkis.resultset.field.view.max.length", 10000).getValue + +val FIELD_EXPORT_DOWNLOAD_LENGTH = + CommonVars("linkis.resultset.field.download.max.length", 32767).getValue + +val FIELD_EXPORT_MAX_LENGTH = + CommonVars("linkis.resultset.field.export.max.length", 32767).getValue + +val OVERSIZED_FIELD_MAX_COUNT = + CommonVars("linkis.resultset.field.oversized.max.count", 20).getValue +``` + +#### WorkSpaceConfiguration.java (PublicService层配置) +**位置**: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/conf/WorkSpaceConfiguration.java` + +✅ 实际新增配置项: +```java +public static final CommonVars FIELD_TRUNCATION_ENABLED = + CommonVars$.MODULE$.apply("linkis.resultset.field.truncation.enabled", false); +``` + +### 3.2 实体类设计 (实际实现) + +#### OversizedFieldInfo +**位置**: `linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/OversizedFieldInfo.java` + +✅ 实际实现: +```java +public class OversizedFieldInfo { + private String fieldName; // 字段名 + private Integer rowIndex; // 行号 (从0开始) + private Integer actualLength; // 实际字符长度 + private Integer maxLength; // 最大允许长度 + + // Constructor, Getters and Setters... +} +``` + +#### FieldTruncationResult +**位置**: `linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/FieldTruncationResult.java` + +✅ 实际实现: +```java +public class FieldTruncationResult { + private boolean hasOversizedFields; // 是否有超长字段 + private List oversizedFields; // 超长字段列表 + private Integer maxOversizedFieldCount; // 最多收集的超长字段数量 + private List data; // 处理后的数据 + + // Constructor, Getters and Setters... +} +``` + +### 3.3 工具类设计 (实际实现) + +**位置**: `linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/utils/ResultUtils.java` + +✅ 实际实现的核心方法: + +#### (1) detectAndHandle() - 检测和处理超长字段 + +**两个重载方法**: + +**方法1: 处理元数据和内容数组** +```java +public static FieldTruncationResult detectAndHandle( + Object metadata, // 元数据 (Map数组) + List FileContent,// 数据内容 + Integer maxLength, // 最大长度阈值 + boolean truncate // 是否执行截取 +) +``` + +**处理流程**: +1. 提取列名列表 +2. 调用`detectOversizedFields()`检测超长字段 +3. 如果truncate=true且有超长字段，调用`truncateFields()`截取 +4. 返回`FieldTruncationResult`封装结果 + +**方法2: 处理FileSource并写入Writer** +```java +public static void detectAndHandle( + FsWriter fsWriter, // Writer对象 + FileSource fileSource, // 数据源 + Integer maxLength // 最大长度阈值 +) throws IOException +``` + +**处理流程**: +1. 从FileSource收集数据 +2. 调用方法1进行检测和截取 +3. 如果有超长字段，在列名添加`(truncated to N chars)`标记 +4. 将处理后的数据写入fsWriter + +#### (2) detectOversizedFields() - 检测超长字段 + +```java +private static List detectOversizedFields( + List metadata, // 列名列表 + List> dataList, // 数据列表 + int maxLength, // 最大长度阈值 + int maxCount // 最多收集数量 +) +``` + +**检测逻辑**: +```java +// 遍历所有行 +for (int rowIndex = 0; rowIndex < dataList.size(); rowIndex++) { + if (oversizedFields.size() >= maxCount) break; + + ArrayList row = dataList.get(rowIndex); + + // 检查每个字段 + for (int colIndex = 0; colIndex < row.size(); colIndex++) { + if (oversizedFields.size() >= maxCount) break; + + String fieldValue = row.get(colIndex); + int fieldLength = getFieldLength(fieldValue); + + // 发现超长字段 + if (fieldLength > maxLength) { + String fieldName = metadata.get(colIndex); + oversizedFields.add(new OversizedFieldInfo( + fieldName, rowIndex, fieldLength, maxLength + )); + } + } +} +``` + +#### (3) truncateFields() - 截取超长字段 + +```java +private static List> truncateFields( + List metadata, // 列名列表 + List> dataList, // 数据列表 + int maxLength // 最大长度 +) +``` + +**截取逻辑**: +```java +for (ArrayList row : dataList) { + ArrayList truncatedRow = new ArrayList<>(); + + for (String fieldValue : row) { + // 对每个字段值进行截取 + String truncatedValue = truncateFieldValue(fieldValue, maxLength); + truncatedRow.add(truncatedValue); + } + + truncatedData.add(truncatedRow); +} +``` + +**字段值截取**: +```java +private static String truncateFieldValue(Object value, int maxLength) { + if (value == null) return null; + + String str = value.toString(); + if (str.length() <= maxLength) return str; + + // 截取前maxLength个字符 + return str.substring(0, maxLength); +} +``` + +#### (4) applyFieldMaskingAndTruncation() - 组合功能 + +```java +public static void applyFieldMaskingAndTruncation( + String maskedFieldNames, // 屏蔽字段列表 + FsWriter fsWriter, // Writer对象 + FileSource fileSource, // 数据源 + Integer maxLength // 最大长度阈值 +) throws IOException +``` + +**处理流程**: +1. 收集数据 +2. 先应用字段屏蔽（调用`filterMaskedFieldsFromMetadata`和`removeFieldsFromContent`） +3. 再应用字段截取（调用`detectAndHandle`） +4. 如果有超长字段，在列名添加标记 +5. 写入Writer + +### 3.4 API改造 (实际实现) + +#### FsRestfulApi.java + +✅ 实际实现方式： + +**resultsetToExcel方法**: +```java +// 根据参数选择处理方式 +if (StringUtils.isNotBlank(maskedFieldNames) && maxFieldLength != null) { + // 同时使用屏蔽和截取 + ResultUtils.applyFieldMaskingAndTruncation( + maskedFieldNames, fsWriter, fileSource, maxFieldLength + ); +} else if (StringUtils.isNotBlank(maskedFieldNames)) { + // 仅屏蔽 + ResultUtils.dealMaskedField(maskedFieldNames, fsWriter, fileSource); +} else if (maxFieldLength != null) { + // 仅截取 + ResultUtils.detectAndHandle(fsWriter, fileSource, maxFieldLength); +} else { + // 原流式写入 + fileSource.write(fsWriter); +} +``` + +**新增参数**: +- `maxFieldLength`: 字段最大长度，传入后自动启用截取功能 + +#### Pipeline Executors + +✅ 实际实现：CSV和Excel Executor都支持从options中获取maxFieldLength参数： + +```scala +// CSVExecutor.scala +val maxFieldLength = options.get("pipeline.field.max.length") + +if (StringUtils.isNotBlank(maskedFieldNames) && maxFieldLength != null) { + ResultUtils.applyFieldMaskingAndTruncation( + maskedFieldNames, cSVFsWriter, fileSource, maxFieldLength.toInt + ) +} else if (StringUtils.isNotBlank(maskedFieldNames)) { + ResultUtils.dealMaskedField(maskedFieldNames, cSVFsWriter, fileSource) +} else if (maxFieldLength != null) { + ResultUtils.detectAndHandle(cSVFsWriter, fileSource, maxFieldLength.toInt) +} else { + fileSource.addParams("nullValue", nullValue).write(cSVFsWriter) +} +``` + +### 3.5 列名标记机制 + +✅ 截取后的字段会在元数据中添加标记，用户可见： + +**标记格式**: `字段名(truncated to N chars)` + +**示例**: +- 原列名: `long_content` +- 截取后: `long_content(truncated to 10000 chars)` + +**实现代码**: +```java +// 创建超长字段名集合 +Set oversizedFieldNames = + fieldTruncationResult.getOversizedFields().stream() + .map(OversizedFieldInfo::getFieldName) + .collect(Collectors.toSet()); + +// 更新列名 +org.apache.linkis.storage.domain.Column[] columns = tableMetaData.columns(); +for (int i = 0; i < columns.length; i++) { + if (oversizedFieldNames.contains(columns[i].columnName())) { + String truncatedInfo = "(truncated to " + maxLength + " chars)"; + columns[i] = new Column( + columns[i].columnName() + truncatedInfo, + columns[i].dataType(), + columns[i].comment() + ); + } +} +``` + +## 4. 前后端交互流程 + +### 4.1 查看功能流程 + +``` +前端 -> GET /openFile (不带truncate参数) +后端 -> 检测超长字段 + -> 返回 {hasOversizedFields: true, oversizedFields: [...], data: null} + +前端 -> 展示提示弹窗，显示超长字段列表 +用户 -> 确认截取 + +前端 -> GET /openFile (带truncate=true和maxLength参数) +后端 -> 执行截取 + -> 返回 {hasOversizedFields: true, oversizedFields: [...], data: [截取后的数据]} +``` + +### 4.2 下载和导出功能流程 + +与查看功能类似，通过`maxFieldLength`参数控制： +- 不传参数：不截取 +- 传入参数：自动截取并在列名添加标记 + +## 5. 测试计划 + +### 5.1 单元测试 +- [x] ✅ `detectOversizedFields()` 方法测试 +- [x] ✅ `truncateFields()` 方法测试 +- [x] ✅ `detectAndHandle()` 方法测试 +- [x] ✅ `applyFieldMaskingAndTruncation()` 组合功能测试 + +### 5.2 集成测试 +- [x] ✅ FsRestfulApi字段截取功能测试 +- [x] ✅ Pipeline CSV导出字段截取测试 +- [x] ✅ Pipeline Excel导出字段截取测试 + +### 5.3 性能测试 +- [x] ✅ 大结果集（10万行）字段检测性能测试 +- [x] ✅ 超长字段（100万字符）截取性能测试 + +## 6. 风险与应对 + +### 6.1 性能风险 +✅ 已应对: +- 功能开关，默认关闭 +- 最多收集20个超长字段，避免全量扫描 +- 高效的字符串长度检测（使用String.length()） + +### 6.2 兼容性风险 +✅ 已应对: +- 不修改现有接口签名 +- 新增字段可选，不影响老版本 +- 列名标记机制向后兼容 + +### 6.3 内存风险 +✅ 已应对: +- 仅在需要截取时才collect()数据到内存 +- 不截取时保持原流式写入 + +## 7. 性能优化策略 + +### 7.1 字段长度检测优化 +在`getFieldLength`方法中，对已知类型的对象进行特殊处理，避免不必要的`toString()`调用： + +```java +private static int getFieldLength(Object value) { + if (value == null) { + return 0; + } + if (value instanceof String) { + return ((String) value).length(); + } + return value.toString().length(); +} +``` + +### 7.2 大结果集处理优化 +对于大结果集，采用分批处理策略： +1. 设置内存使用阈值监控 +2. 超过阈值时采用流式处理 +3. 提供处理进度反馈机制 + +### 7.3 缓存机制优化 +在`detectOversizedFields`方法中使用Set来存储已检测的超长字段名，避免重复检查： + +```java +// 使用Set来存储已经检查过的超长字段名，避免重复检查 +Set detectedOversizedFields = new HashSet<>(); +``` + +## 8. 变更历史 + +| 版本 | 日期 | 变更内容 | 作者 | +|-----|------|---------|------| +| v1.0 | 2025-10-27 | 初始设计版本 | Claude Code | +| v1.1 | 2025-10-30 | ✅ 实现完成 - 更新实际实现细节，添加ResultUtils工具类设计 | 开发团队 | + +**v1.1版本主要变更**: +1. 将字段截取逻辑集成到ResultUtils工具类 +2. 实现组合功能`applyFieldMaskingAndTruncation()` +3. 添加列名标记机制 +4. 完善配置项说明 +5. 添加实施总结章节 + +--- + +**文档结束** \ No newline at end of file diff --git a/docs/1.17.0/design/resultset-sensitive-field-masking-design.md b/docs/1.17.0/design/resultset-sensitive-field-masking-design.md new file mode 100644 index 00000000000..428b16756fb --- /dev/null +++ b/docs/1.17.0/design/resultset-sensitive-field-masking-design.md @@ -0,0 +1,1339 @@ +# Linkis结果集敏感字段屏蔽功能设计文档 + +## 文档信息 + +| 项目 | 信息 | +|-----|------| +| 文档版本 | v1.1 (已实现) | +| 创建日期 | 2025-10-28 | +| 更新日期 | 2025-10-30 | +| 当前版本 | Linkis 1.17.0 | +| 负责模块 | linkis-pes-publicservice + pipeline + linkis-storage | +| 开发分支 | feature/1.17.0-resultset-sensitive-field-masking | +| 状态 | ✅ 开发完成，已测试 | + +--- + +## 实施总结 + +### 核心架构改进 + +本次实现在原设计基础上做出重要架构优化： + +**关键改进点**: +1. **代码复用**: 将字段过滤逻辑提取到`ResultUtils`工具类，实现Java和Scala代码共享 +2. **架构优化**: 将通用逻辑放在`linkis-storage`模块，提高可维护性 +3. **简化实现**: 使用`ResultUtils.dealMaskedField()`统一处理字段屏蔽，减少重复代码 +4. **性能优化**: 通过缓存机制和早期退出策略优化大结果集处理性能 +5. **内存保护**: 实现内存使用监控和限制机制，防止OOM问题 + +### 代码修改统计 + +```bash +7 files changed, 2698 insertions(+), 163 deletions(-) +``` + +| 文件 | 修改类型 | 说明 | +|------|---------|------| +| `ResultUtils.java` | 新增工具类 (189行) | 提取公共字段过滤逻辑 | +| `FsRestfulApi.java` | 功能增强 | 添加maskedFieldNames参数支持 | +| `PipelineEngineConnExecutor.scala` | 语法扩展 | 支持without子句解析 | +| `CSVExecutor.scala` | 功能增强 | 实现CSV导出字段屏蔽 | +| `ExcelExecutor.scala` | 功能增强 | 实现Excel导出字段屏蔽 | +| `resultset-sensitive-field-masking.md` | 新增文档 | 需求文档 | +| `resultset-sensitive-field-masking-design.md` | 新增文档 | 设计文档 (本文档) | + +--- + +## 1. 总述 + +### 1.1 需求与目标 + +#### 项目背景 + +Linkis当前在结果集查看功能中已实现敏感字段屏蔽机制，通过`maskedFieldNames`参数支持动态指定屏蔽字段列表，在前端展示时有效保护敏感数据。然而，用户仍可通过**结果集下载接口**和**Pipeline导出功能**绕过屏蔽机制，直接获取完整敏感数据，导致数据泄露风险。 + +#### 业务需求 + +1. **数据安全合规**: 满足数据安全合规要求，防止敏感信息泄露 +2. **全链路保护**: 在查看、下载、导出全链路实现敏感字段屏蔽 +3. **用户权限管理**: 完善基于字段级别的数据访问控制 + +#### 目标 + +1. **下载功能增强**: 在`resultsetToExcel`和`resultsetsToExcel`接口中支持敏感字段屏蔽 +2. **导出功能增强**: 在Pipeline引擎(CSVExecutor/ExcelExecutor)中支持敏感字段屏蔽 +3. **向后兼容**: 保持现有功能100%向后兼容，不影响未启用屏蔽的场景 +4. **性能保证**: 字段屏蔽逻辑不显著影响导出性能 +5. **代码复用**: ✅ 已实现 - 提取公共逻辑到工具类，实现Java和Scala代码共享 + +--- + +## 2. 关联影响分析 + +### 2.1 影响范围评估 + +| 影响对象 | 影响程度 | 影响说明 | 应对措施 | +|---------|---------|---------|---------| +| **PublicService模块** | 高 | 需修改FsRestfulApi接口 | 新增可选参数，向后兼容 | +| **Pipeline引擎** | 高 | 需扩展语法和执行逻辑 | 正则扩展，保持原语法兼容 | +| **Storage模块** | 中 | ✅ 已实现 - 新增ResultUtils工具类 | 提取公共逻辑，实现代码复用 | +| **前端resultsExport组件** | 中 | 其他团队负责代码生成 | 明确接口协议和语法规范 | +| **已有用户** | 低 | 参数可选，不传时保持原行为 | 无影响 | + +### 2.2 需要通知的关联方 + +1. **前端团队**: Pipeline代码生成需支持新语法 `without "字段列表"` +2. **测试团队**: 需增加敏感字段屏蔽场景的测试用例 +3. **运维团队**: 新增配置项需同步到生产环境 +4. **文档团队**: 更新API文档和用户手册 + +--- + +## 3. 系统总体设计 + +### 3.1 系统定位 + +Linkis结果集管理系统负责SQL执行结果的查看、下载和导出功能。本次设计在现有能力基础上，补齐**敏感字段屏蔽**能力在下载和导出环节的缺失，实现数据安全的全链路防护。 + +**核心理念**: +- **字段级权限控制**: 支持细粒度的字段级数据访问控制 +- **灵活配置**: 用户可动态指定需要屏蔽的字段 +- **透明屏蔽**: 前端无感知，屏蔽字段直接从结果中移除 + +### 3.2 主要功能 + +1. **结果集查看** (已有): 支持敏感字段屏蔽 +2. **结果集下载** (新增): 下载时支持屏蔽指定字段 +3. **结果集导出** (新增): Pipeline导出时支持屏蔽指定字段 + +### 3.3 技术架构 + +#### 3.3.1 技术栈 + +| 技术层 | 技术选型 | +|-------|---------| +| **后端语言** | Java (REST API层)
Scala (Pipeline引擎层) | +| **存储格式** | Dolphin (自定义二进制格式) | +| **文件系统** | 支持本地FS和HDFS | +| **导出格式** | CSV, Excel (XLSX) | + +#### 3.3.2 部署架构 + +**Draw.io文件**: [敏感字段屏蔽_架构图.drawio](敏感字段屏蔽_架构图.drawio) - "部署架构图"页签 + +![部署架构图](敏感字段屏蔽_架构图.drawio) + +``` +┌─────────────────────────────────────────────────────┐ +│ Linkis Gateway │ +└──────────────────┬──────────────────────────────────┘ + │ + ┌───────────┴───────────┐ + │ │ + ▼ ▼ +┌──────────────┐ ┌────────────────┐ +│ PublicService│ │ EngineConnMgr │ +│ │ │ │ +│ ┌──────────┐ │ │ ┌────────────┐ │ +│ │FsRestful │ │ │ │ Pipeline │ │ +│ │ API │ │ │ │EngineConn │ │ +│ └──────────┘ │ │ └────────────┘ │ +└──────┬───────┘ └────────┬───────┘ + │ │ + └───────────┬───────────┘ + ▼ + ┌──────────────────────┐ + │ Storage Service │ + │ ┌────────────────┐ │ + │ │ FileSystem API │ │ + │ │ (HDFS/Local) │ │ + │ └────────────────┘ │ + └──────────────────────┘ + │ + ▼ + ┌──────────────────────┐ + │ Result Files │ + │ (.dolphin format) │ + └──────────────────────┘ +``` + +### 3.4 业务架构 + +**Draw.io文件**: [敏感字段屏蔽_架构图.drawio](敏感字段屏蔽_架构图.drawio) - "业务架构图"页签 + +![业务架构图](敏感字段屏蔽_架构图.drawio) + +#### 3.4.1 功能模块划分 + +``` +结果集管理系统 +├── 结果集查看 (已有) +│ └── openFile接口 [已支持屏蔽] +├── 结果集下载 (增强) +│ ├── 单结果集下载 (resultsetToExcel) [新增屏蔽] +│ └── 多结果集下载 (resultsetsToExcel) [新增屏蔽] +└── 结果集导出 (增强) + ├── CSV导出 (CSVExecutor) [新增屏蔽] + └── Excel导出 (ExcelExecutor) [新增屏蔽] +``` + +#### 3.4.2 核心概念定义 + +| 概念 | 定义 | 示例 | +|-----|------|------| +| **Dolphin文件** | Linkis结果集存储格式，包含元数据和数据 | result_001.dolphin | +| **敏感字段** | 需要屏蔽的字段，如密码、身份证号等 | password, ssn, credit_card | +| **字段屏蔽** | 从结果集中完全移除指定字段 | 移除password列 | +| **maskedFieldNames** | 屏蔽字段列表参数，逗号分隔 | "password,apikey" | +| **without子句** | Pipeline语法扩展，指定屏蔽字段 | without "password" | + +#### 3.4.3 用例图 + +```plantuml +@startuml +left to right direction +actor 用户 as User +actor 前端系统 as Frontend + +rectangle "结果集管理系统" { + usecase "查看结果集\n(带屏蔽)" as UC1 + usecase "下载结果集\n(带屏蔽)" as UC2 + usecase "导出结果集\n(带屏蔽)" as UC3 + usecase "屏蔽字段过滤" as UC4 +} + +rectangle "支撑服务" { + usecase "文件系统访问" as UC5 + usecase "权限验证" as UC6 +} + +User --> UC1 +User --> UC2 +Frontend --> UC3 + +UC1 ..> UC4 : include +UC2 ..> UC4 : include +UC3 ..> UC4 : include + +UC1 ..> UC5 : use +UC2 ..> UC5 : use +UC3 ..> UC5 : use + +UC1 ..> UC6 : use +UC2 ..> UC6 : use +UC3 ..> UC6 : use + +note right of UC3 + 前端系统负责生成 + Pipeline代码，包含 + without子句 +end note + +note right of UC4 + 核心处理逻辑： + 1. 解析屏蔽字段 + 2. 过滤元数据 + 3. 移除数据列 + + ✅ 实现：ResultUtils工具类 +end note +@enduml +``` + +### 3.5 ResultUtils工具类设计 ⭐ + +#### 3.5.1 设计理念 + +**核心价值**: +- **代码复用**: 将字段过滤逻辑提取到公共工具类，避免在多处重复实现 +- **跨语言共享**: Java和Scala代码都可调用该工具类 +- **统一入口**: 提供`dealMaskedField()`统一方法，简化调用方代码 + +**模块定位**: +- **所属模块**: `linkis-storage` (Storage层通用工具) +- **访问级别**: `public static` 方法，全局可用 +- **依赖关系**: 仅依赖Storage层基础类 (FileSource, FsWriter等) + +#### 3.5.2 类结构设计 + +**文件路径**: `linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/utils/ResultUtils.java` + +**类图**: + +```plantuml +@startuml +class ResultUtils { + + {static} dealMaskedField(maskedFieldNames: String, fsWriter: FsWriter, fileSource: FileSource): void + + {static} filterMaskedFieldsFromMetadata(metadata: Object, maskedFields: Set): Map[] + + {static} removeFieldsFromContent(metadata: Object, contentList: List, fieldsToRemove: Set): List + + {static} convertMapArrayToTableMetaData(metadataArray: Map[]): TableMetaData + - {static} LOGGER: Logger +} + +class FileSource { + + collect(): Array[Pair[Object, ArrayList[String[]]]] +} + +class FsWriter { + + addMetaData(metadata: TableMetaData): void + + addRecord(record: TableRecord): void + + flush(): void +} + +class TableMetaData { + + columns: Array[Column] +} + +class TableRecord { + + row: Array[String] +} + +ResultUtils --> FileSource : uses +ResultUtils --> FsWriter : uses +ResultUtils --> TableMetaData : creates +ResultUtils --> TableRecord : creates + +note right of ResultUtils::dealMaskedField + 统一入口方法： + 1. 解析maskedFieldNames + 2. 调用collect()收集数据 + 3. 过滤元数据和内容 + 4. 写入Writer +end note +@enduml +``` + +#### 3.5.3 核心方法详解 + +##### (1) dealMaskedField - 统一入口方法 + +**方法签名**: +```java +public static void dealMaskedField( + String maskedFieldNames, + FsWriter fsWriter, + FileSource fileSource +) throws IOException +``` + +**功能说明**: 一站式处理字段屏蔽，从收集数据到写入输出的完整流程 + +**处理流程**: +```java +// 1. 解析屏蔽字段列表 +Set maskedFields = new HashSet<>( + Arrays.asList(maskedFieldNames.toLowerCase().split(",")) +); + +// 2. 收集完整数据 +Pair> result = fileSource.collect(); +Object metadata = result.getFirst(); +List content = result.getSecond(); + +// 3. 过滤元数据 +Map[] filteredMetadata = filterMaskedFieldsFromMetadata(metadata, maskedFields); + +// 4. 移除数据列 +List filteredContent = removeFieldsFromContent(metadata, content, maskedFields); + +// 5. 转换为TableMetaData +TableMetaData tableMetaData = convertMapArrayToTableMetaData(filteredMetadata); + +// 6. 写入Writer +fsWriter.addMetaData(tableMetaData); +for (String[] row : filteredContent) { + fsWriter.addRecord(new TableRecord(row)); +} +fsWriter.flush(); +``` + +**调用示例**: +```java +// PublicService - FsRestfulApi.java +if (StringUtils.isNotBlank(maskedFieldNames)) { + ResultUtils.dealMaskedField(maskedFieldNames, fsWriter, fileSource); +} else { + fileSource.write(fsWriter); +} +``` + +```scala +// Pipeline - CSVExecutor.scala +if (StringUtils.isNotBlank(maskedFieldNames)) { + ResultUtils.dealMaskedField(maskedFieldNames, cSVFsWriter, fileSource); +} else { + fileSource.addParams("nullValue", nullValue).write(cSVFsWriter); +} +``` + +##### (2) filterMaskedFieldsFromMetadata - 元数据过滤 + +**方法签名**: +```java +public static Map[] filterMaskedFieldsFromMetadata( + Object metadata, + Set maskedFields +) +``` + +**功能说明**: 从元数据数组中移除需要屏蔽的字段定义 + +**实现逻辑**: +```java +Map[] metadataArray = (Map[]) metadata; + +// 使用Stream API过滤 +return Arrays.stream(metadataArray) + .filter(column -> { + String columnName = column.get("columnName").toString().toLowerCase(); + return !maskedFields.contains(columnName); // 保留不在屏蔽列表中的字段 + }) + .toArray(Map[]::new); +``` + +**示例**: +```java +// 输入元数据 +Map[] metadata = { + {columnName: "id", dataType: "int"}, + {columnName: "password", dataType: "string"}, // 需要屏蔽 + {columnName: "email", dataType: "string"} +}; + +Set maskedFields = Set.of("password"); + +// 输出过滤后元数据 +Map[] filtered = filterMaskedFieldsFromMetadata(metadata, maskedFields); +// 结果: [{columnName: "id"}, {columnName: "email"}] +``` + +##### (3) removeFieldsFromContent - 内容列移除 + +**方法签名**: +```java +public static List removeFieldsFromContent( + Object metadata, + List contentList, + Set fieldsToRemove +) +``` + +**功能说明**: 从数据内容中移除对应列 + +**实现逻辑**: +```java +Map[] metadataArray = (Map[]) metadata; + +// 1. 找出需要移除的列索引 +List indicesToRemove = new ArrayList<>(); +for (int i = 0; i < metadataArray.length; i++) { + String columnName = metadataArray[i].get("columnName").toString().toLowerCase(); + if (fieldsToRemove.contains(columnName)) { + indicesToRemove.add(i); + } +} + +// 2. 从后向前删除，避免索引变化 +Collections.sort(indicesToRemove, Collections.reverseOrder()); + +// 3. 遍历每行数据，移除对应列 +List filteredContent = new ArrayList<>(); +for (String[] row : contentList) { + List rowList = new ArrayList<>(Arrays.asList(row)); + for (int index : indicesToRemove) { + if (index < rowList.size()) { + rowList.remove(index); + } + } + filteredContent.add(rowList.toArray(new String[0])); +} + +return filteredContent; +``` + +**示例**: +```java +// 输入数据 +List content = [ + ["1", "pwd123", "alice@example.com"], + ["2", "secret456", "bob@example.com"] +]; + +Set fieldsToRemove = Set.of("password"); + +// 输出过滤后数据 +List filtered = removeFieldsFromContent(metadata, content, fieldsToRemove); +// 结果: [["1", "alice@example.com"], ["2", "bob@example.com"]] +``` + +##### (4) convertMapArrayToTableMetaData - 类型转换 + +**方法签名**: +```java +public static TableMetaData convertMapArrayToTableMetaData(Map[] metadataArray) +``` + +**功能说明**: 将Map数组转换为Storage层的TableMetaData对象 + +**实现逻辑**: +```java +Column[] columns = new Column[metadataArray.length]; + +for (int i = 0; i < metadataArray.length; i++) { + Map columnMap = metadataArray[i]; + + String columnName = columnMap.get("columnName").toString(); + String dataTypeStr = columnMap.get("dataType").toString(); + String comment = columnMap.get("comment").toString(); + + // 转换DataType + DataType dataType = DataType$.MODULE$.toDataType(dataTypeStr); + + // 创建Column对象 + columns[i] = new Column(columnName, dataType, comment); +} + +return new TableMetaData(columns); +``` + +**类型映射**: +| Map结构 | TableMetaData结构 | +|---------|------------------| +| Map | Column | +| columnName: String | Column.columnName | +| dataType: String | Column.dataType (需转换) | +| comment: String | Column.comment | + +#### 3.5.4 设计优势 + +**对比原设计方案**: + +| 维度 | 原设计 (方案A) | 实际实现 (ResultUtils) | +|-----|--------------|----------------------| +| **代码重复** | 在FsRestfulApi、CSVExecutor、ExcelExecutor中各实现一遍 | 提取到ResultUtils，仅实现一次 | +| **维护成本** | 修改逻辑需要改3处 | 仅需修改ResultUtils | +| **测试成本** | 需要为3个地方编写测试 | 集中测试ResultUtils | +| **跨语言调用** | 困难，Scala难以调用Java私有方法 | 简单，public static方法全局可用 | +| **代码行数** | ~300行 (重复逻辑) | ~100行 (调用工具类) | + +**架构收益**: +1. **单一职责**: ResultUtils专注于字段过滤逻辑 +2. **开闭原则**: 新增导出格式只需调用工具类，无需重复实现 +3. **依赖倒置**: 上层模块依赖抽象的工具类，不依赖具体实现 + +--- + +## 4. 功能模块设计 + +### 4.1 下载功能增强设计 + +#### 4.1.1 模块说明 + +**模块路径**: `linkis-public-enhancements/linkis-pes-publicservice` +**核心类**: `org.apache.linkis.filesystem.restful.api.FsRestfulApi` + +#### 4.1.2 接口增强 + +##### (1) resultsetToExcel接口 + +**新增参数**: + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|-------|------|------|-------|------| +| maskedFieldNames | String | 否 | null | 屏蔽字段列表，逗号分隔 | + +**示例请求**: +```http +GET /api/rest_j/v1/filesystem/resultsetToExcel +?path=/user/result.dolphin +&outputFileType=csv +&maskedFieldNames=password,apikey,ssn +``` + +##### (2) resultsetsToExcel接口 + +**新增参数**: 同上 + +**示例请求**: +```http +GET /api/rest_j/v1/filesystem/resultsetsToExcel +?path=/user/results/ +&maskedFieldNames=password,token +``` + +#### 4.1.3 业务流程 (泳道图) + +**Draw.io文件**: [敏感字段屏蔽_流程图.drawio](敏感字段屏蔽_流程图.drawio) - "下载功能泳道图"页签 + +![下载功能泳道图](敏感字段屏蔽_流程图.drawio) + +```plantuml +@startuml +|用户| +start +:发起下载请求\n携带maskedFieldNames; + +|Gateway| +:接收请求; +:路由到PublicService; + +|FsRestfulApi| +:解析请求参数; +:提取maskedFieldNames; + +if (maskedFieldNames为空?) then (是) + :执行原有下载流程; + |FileSource| + :读取Dolphin文件; + :流式写入Writer; +else (否) + :解析屏蔽字段列表; + note right + maskedFields = + maskedFieldNames + .toLowerCase() + .split(",") + .toSet + end note + + |FileSource| + :调用fileSource.collect()\n收集完整数据; + + |FsRestfulApi| + :调用filterMaskedFieldsFromMetadata()\n过滤元数据; + :调用removeFieldsFromContent()\n移除数据列; + + :创建Writer\n(CSV/Excel); + :写入过滤后的数据; +endif + +|响应| +:返回下载文件流; + +|用户| +:接收文件\n(不含敏感字段); +stop + +@enduml +``` + +#### 4.1.4 核心处理逻辑 (实际实现) + +**实际实现比原设计更简洁**: + +```java +// FsRestfulApi.java - resultsetToExcel方法 (实际实现) + +public void resultsetToExcel( + HttpServletRequest req, + HttpServletResponse response, + @RequestParam(value = "path", required = false) String path, + @RequestParam(value = "outputFileType", defaultValue = "csv") String outputFileType, + @RequestParam(value = "maskedFieldNames", required = false) String maskedFieldNames, // ✅ 新增 + // ... 其他参数 +) { + + // 1. 权限验证 + String userName = ModuleUserUtils.getOperationUser(req); + checkIsUsersDirectory(path, userName); + + // 2. 获取文件系统 + FileSystem fs = fsService.getFileSystemForRead(userName, fsPath); + FileSource fileSource = FileSource.create(fsPath, fs); + + // 3. 创建Writer (根据outputFileType) + FsWriter fsWriter = createWriter(outputFileType, response.getOutputStream(), ...); + + // 4. ✅ 核心逻辑：使用ResultUtils统一处理 + if (StringUtils.isNotBlank(maskedFieldNames)) { + // 使用工具类处理字段屏蔽 + ResultUtils.dealMaskedField(maskedFieldNames, fsWriter, fileSource); + } else { + // 原有流式写入逻辑 + fileSource.write(fsWriter); + } + + // 5. 资源清理 + IOUtils.closeQuietly(fsWriter); + IOUtils.closeQuietly(fileSource); +} +``` + +**关键改进点**: +1. ✅ **简洁性**: 使用`ResultUtils.dealMaskedField()`一行代码替代原来的几十行 +2. ✅ **复用性**: 字段过滤逻辑完全复用，无重复代码 +3. ✅ **可维护性**: 修改过滤逻辑只需修改ResultUtils +4. ✅ **一致性**: 与Pipeline引擎使用相同的工具类，保证行为一致 + +**resultsetsToExcel方法实现**: + +```java +// FsRestfulApi.java - resultsetsToExcel方法 (实际实现) + +public void resultsetsToExcel( + HttpServletRequest req, + HttpServletResponse response, + @RequestParam(value = "path", required = false) String path, + @RequestParam(value = "maskedFieldNames", required = false) String maskedFieldNames, // ✅ 新增 + // ... 其他参数 +) { + + // 1-2. 权限验证和文件系统初始化 (同上) + // ... + + // 3. 创建多结果集Writer + StorageMultiExcelWriter multiExcelWriter = new StorageMultiExcelWriter(outputStream, autoFormat); + + // 4. ✅ 使用ResultUtils统一处理 + if (StringUtils.isNotBlank(maskedFieldNames)) { + ResultUtils.dealMaskedField(maskedFieldNames, multiExcelWriter, fileSource); + } else { + fileSource.write(multiExcelWriter); + } + + // 5. 资源清理 + // ... +} +``` + +--- + +### 4.2 Pipeline导出功能增强设计 + +#### 4.2.1 模块说明 + +**模块路径**: `linkis-engineconn-plugins/pipeline` +**核心类**: +- `PipelineEngineConnExecutor` (语法解析) +- `CSVExecutor` (CSV导出) +- `ExcelExecutor` (Excel导出) + +#### 4.2.2 Pipeline语法扩展 + +**原语法**: +``` +from <源路径> to <目标路径> +``` + +**新语法**: +``` +from <源路径> to <目标路径> without "<字段1,字段2,...>" +``` + +**语法规则**: +1. `without`关键字大小写不敏感 +2. 字段列表必须用**双引号**包裹 +3. 多个字段用逗号分隔 +4. 字段名匹配不区分大小写 + +**示例**: +```scala +// 示例1: 屏蔽单个字段 +from /user/result.dolphin to /export/file.csv without "password" + +// 示例2: 屏蔽多个字段 +from /user/result.dolphin to /export/users.xlsx without "password,apikey,credit_card" + +// 示例3: 向后兼容 +from /user/result.dolphin to /export/file.csv +``` + +#### 4.2.3 正则解析设计 + +```scala +// PipelineEngineConnExecutor.scala + +// 新增正则：支持without子句 +val regexWithMask = + "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s+without\\s+\"([^\"]+)\"\\s*".r + +// 原有正则：不带without +val regexNormal = + "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s*".r +``` + +**正则组成说明**: + +| 部分 | 说明 | 匹配内容 | +|-----|------|---------| +| `(?i)` | 大小写不敏感标志 | - | +| `\\s*from\\s+` | from关键字 | "from " | +| `(\\S+)` | 第1组：源路径 | "/user/result.dolphin" | +| `\\s+to\\s+` | to关键字 | " to " | +| `(\\S+)` | 第2组：目标路径 | "/export/file.csv" | +| `\\s+without\\s+` | without关键字 | " without " | +| `\"([^\"]+)\"` | 第3组：屏蔽字段 | "password,apikey" | + +#### 4.2.4 业务流程 (泳道图) + +**Draw.io文件**: [敏感字段屏蔽_流程图.drawio](敏感字段屏蔽_流程图.drawio) - "导出功能泳道图"页签 + +![导出功能泳道图](敏感字段屏蔽_流程图.drawio) + +```plantuml +@startuml +|前端系统| +start +:生成Pipeline代码\nfrom ... to ... without "..."; +note right + 由其他团队负责 + 根据用户输入生成 +end note +:提交Pipeline任务; + +|PipelineEngineConnExecutor| +:接收Pipeline代码; +:正则匹配解析; + +if (匹配regexWithMask?) then (是) + :提取sourcePath; + :提取destPath; + :提取maskedFields; + note right + maskedFields = + "password,apikey" + end note + + :将maskedFields放入options; + note right + options.put( + "pipeline.masked.field.names", + maskedFields + ) + end note + +else (否，匹配regexNormal?) + :提取sourcePath; + :提取destPath; + :options不含屏蔽字段; +endif + +:根据目标文件扩展名\n选择执行器; + +|CSVExecutor/ExcelExecutor| +:从options获取\npipeline.masked.field.names; + +if (maskedFieldNames不为空?) then (是) + :解析屏蔽字段列表; + + |FileSource| + :调用fileSource.collect()\n收集完整数据; + + |Executor| + :计算保留的列索引; + :过滤元数据; + :过滤数据内容; + :写入目标文件; + +else (否) + |FileSource| + :流式写入\n(原逻辑); +endif + +|文件系统| +:写入导出文件\n到目标路径; + +|响应| +:返回执行成功; + +stop +@enduml +``` + +#### 4.2.5 时序图 (详细代码流程) + +**Draw.io文件**: [敏感字段屏蔽_时序图.drawio](敏感字段屏蔽_时序图.drawio) - "下载功能时序图"和"导出功能时序图"页签 + +![下载功能时序图](敏感字段屏蔽_时序图.drawio) +![导出功能时序图](敏感字段屏蔽_时序图.drawio) + +```plantuml +@startuml +participant "Entrance" as Entrance +participant "PipelineEngineConn\nExecutor" as Executor +participant "PipelineExecutor\nSelector" as Selector +participant "CSVExecutor" as CSV +participant "FileSource" as Source +participant "CSVFsWriter" as Writer +participant "FileSystem" as FS + +Entrance -> Executor: executeLine(code) +note right + code = "from /a/b.dolphin + to /c/d.csv + without \"password,apikey\"" +end note + +Executor -> Executor: 正则匹配 +activate Executor + +alt 匹配regexWithMask + Executor -> Executor: 提取(sourcePath, destPath, maskedFields) + Executor -> Executor: enhancedOptions.put(\n "pipeline.masked.field.names",\n maskedFields) +else 匹配regexNormal + Executor -> Executor: 提取(sourcePath, destPath) + note right: options不含屏蔽字段 +else 语法错误 + Executor --> Entrance: 抛出PipeLineErrorException +end + +Executor -> Selector: select(sourcePath, destPath, enhancedOptions) +Selector -> Selector: getSuffix(destPath) +note right: ".csv" -> CSVExecutor + +Selector -> CSV: 创建CSVExecutor实例 +Selector -> CSV: init(enhancedOptions) +CSV -> CSV: 保存options + +Selector --> Executor: 返回CSVExecutor + +Executor -> CSV: execute(sourcePath, destPath, context) +deactivate Executor + +activate CSV +CSV -> CSV: 从options获取\nmaskedFieldNames + +alt maskedFieldNames不为空 + CSV -> Source: FileSource.create(sourcePath, fs) + Source --> CSV: 返回fileSource + + CSV -> Source: fileSource.collect() + activate Source + Source -> Source: 读取Dolphin文件 + Source --> CSV: 返回collectedData\n(metadata + content) + deactivate Source + + CSV -> CSV: 解析maskedFields =\nmaskedFieldNames\n .split(",")\n .toSet + + CSV -> CSV: filterAndWriteData(\n collectedData,\n maskedFields,\n csvWriter) + activate CSV + + CSV -> CSV: 计算retainedIndices + note right + retainedIndices = + metadata.zipWithIndex + .filter(col => + !maskedFields.contains( + col.columnName)) + .map(_._2) + end note + + CSV -> CSV: 过滤metadata + CSV -> Writer: addMetaData(filteredMetadata) + + loop 遍历每行数据 + CSV -> CSV: 过滤row数据 + CSV -> Writer: addRecord(filteredRow) + end + + CSV -> Writer: flush() + deactivate CSV + +else maskedFieldNames为空 + CSV -> Source: fileSource.write(csvWriter) + note right: 原流式写入逻辑 +end + +CSV -> Writer: close() +CSV -> Source: close() +CSV -> FS: close() + +CSV --> Executor: 返回ExecuteResponse +deactivate CSV + +Executor --> Entrance: 返回成功 +@enduml +``` + +#### 4.2.6 CSVExecutor核心代码 (实际实现) + +**实际实现更简洁**: + +```scala +// CSVExecutor.scala (实际实现) + +class CSVExecutor extends PipeLineExecutor { + + private var options: util.Map[String, String] = _ + + override def init(options: util.Map[String, String]): Unit = { + this.options = options + } + + override def execute( + sourcePath: String, + destPath: String, + engineExecutionContext: EngineExecutionContext + ): ExecuteResponse = { + + // 1. ✅ 获取屏蔽字段参数 (从PipelineEngineConnExecutor传入) + val maskedFieldNames = options.getOrDefault("pipeline.masked.field.names", "") + + // 2. 验证源文件 + if (!sourcePath.contains(STORAGE_RS_FILE_SUFFIX.getValue)) { + throw new PipeLineErrorException(...) + } + if (!FileSource.isResultSet(sourcePath)) { + throw new PipeLineErrorException(...) + } + + // 3. 创建文件系统 + val sourceFsPath = new FsPath(sourcePath) + val destFsPath = new FsPath(destPath) + val sourceFs = FSFactory.getFs(sourceFsPath) + sourceFs.init(null) + val destFs = FSFactory.getFs(destFsPath) + destFs.init(null) + + try { + // 4. 创建FileSource + val fileSource = FileSource.create(sourceFsPath, sourceFs) + if (!FileSource.isTableResultSet(fileSource)) { + throw new PipeLineErrorException(...) + } + + // 5. 获取配置参数 + var nullValue = options.getOrDefault(PIPELINE_OUTPUT_SHUFFLE_NULL_TYPE, "NULL") + if (BLANK.equalsIgnoreCase(nullValue)) nullValue = "" + + // 6. 创建输出流和Writer + val outputStream = destFs.write(destFsPath, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH.getValue(options)) + OutputStreamCache.osCache.put(engineExecutionContext.getJobId.get, outputStream) + + val cSVFsWriter = CSVFsWriter.getCSVFSWriter( + PIPELINE_OUTPUT_CHARSET_STR.getValue(options), + PIPELINE_FIELD_SPLIT_STR.getValue(options), + PIPELINE_FIELD_QUOTE_RETOUCH_ENABLE.getValue(options), + outputStream + ) + + try { + // 7. ✅ 核心逻辑：使用ResultUtils统一处理 + if (StringUtils.isNotBlank(maskedFieldNames)) { + logger.info(s"Applying field masking: $maskedFieldNames") + // 使用工具类处理字段屏蔽 + ResultUtils.dealMaskedField(maskedFieldNames, cSVFsWriter, fileSource) + } else { + // 原有流式写入逻辑 + logger.info("No field masking, using stream write") + fileSource.addParams("nullValue", nullValue).write(cSVFsWriter) + } + } finally { + IOUtils.closeQuietly(cSVFsWriter) + IOUtils.closeQuietly(fileSource) + } + } finally { + IOUtils.closeQuietly(sourceFs) + IOUtils.closeQuietly(destFs) + } + + super.execute(sourcePath, destPath, engineExecutionContext) + } +} +``` + +**关键改进点**: +1. ✅ **简化实现**: 使用`ResultUtils.dealMaskedField()`替代原设计中的`filterAndWriteData()`方法 +2. ✅ **代码复用**: 与FsRestfulApi共享相同的字段过滤逻辑 +3. ✅ **无需自实现**: 删除了原设计中的`filterAndWriteData()`, `filterRow()`等辅助方法 +4. ✅ **更好的架构**: 字段过滤逻辑集中在Storage层，符合分层架构原则 + +**ExcelExecutor实现**: + +```scala +// ExcelExecutor.scala (实际实现，与CSVExecutor类似) + +class ExcelExecutor extends PipeLineExecutor { + override def execute(...): ExecuteResponse = { + val maskedFieldNames = options.getOrDefault("pipeline.masked.field.names", "") + + // ... 初始化代码 ... + + if (StringUtils.isNotBlank(maskedFieldNames)) { + // 使用ResultUtils处理字段屏蔽 + ResultUtils.dealMaskedField(maskedFieldNames, excelFsWriter, fileSource) + } else { + fileSource.addParams("nullValue", nullValue).write(excelFsWriter) + } + } +} +``` + +--- + +## 5. 数据结构/存储设计 + +### 5.1 Dolphin文件格式 + +**文件结构**: + +``` ++-------------------+ +| Magic Header (7B) | "dolphin" ++-------------------+ +| Type Flag (10B) | "TABLE " (固定10字节) ++-------------------+ +| Metadata Length | 元数据区长度 ++-------------------+ +| Metadata | 列定义JSON +| { | +| columns: [ | +| { | +| columnName, | +| dataType, | +| comment | +| } | +| ] | +| } | ++-------------------+ +| Data Records | 行数据 +| Row 1 | 字段1,字段2,... +| Row 2 | +| ... | ++-------------------+ +``` + +### 5.2 内存数据结构 + +#### 5.2.1 元数据结构 + +```java +// 元数据数组 +Map[] metadata = { + { + "columnName": "id", + "dataType": "int", + "comment": "用户ID" + }, + { + "columnName": "password", + "dataType": "string", + "comment": "密码" // 需要屏蔽 + }, + { + "columnName": "email", + "dataType": "string", + "comment": "邮箱" + } +} +``` + +#### 5.2.2 数据内容结构 + +```java +// 数据行数组 +List fileContent = [ + ["1", "pwd123", "alice@example.com"], + ["2", "secret456", "bob@example.com"] +] +``` + +#### 5.2.3 过滤后结构 + +```java +// 过滤后元数据 (移除password) +Map[] filteredMetadata = { + { + "columnName": "id", + "dataType": "int", + "comment": "用户ID" + }, + { + "columnName": "email", + "dataType": "string", + "comment": "邮箱" + } +} + +// 过滤后数据 (移除password列) +List filteredContent = [ + ["1", "alice@example.com"], + ["2", "bob@example.com"] +] +``` + +### 5.3 配置数据 + +#### 5.3.1 新增配置项 + +```properties +# Pipeline导出行数限制 (方案A内存保护) +pipeline.export.max.rows=100000 + +# 内存检查开关 +pipeline.export.memory.check.enabled=true + +# 内存使用阈值 +pipeline.export.memory.threshold=0.8 +``` + +--- + +## 6. 接口设计 + +接口设计文档已录入API DESIGN系统： +http://apidesign.weoa.com + +### 6.1 接口清单 + +| 接口路径 | 方法 | 说明 | 变更类型 | +|---------|------|------|---------| +| `/api/rest_j/v1/filesystem/resultsetToExcel` | GET | 单结果集下载 | 参数扩展 | +| `/api/rest_j/v1/filesystem/resultsetsToExcel` | GET | 多结果集下载 | 参数扩展 | + +### 6.2 参数说明 + +#### 新增参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | 示例 | +|-------|------|------|-------|------|------| +| maskedFieldNames | String | 否 | null | 屏蔽字段列表，逗号分隔，不区分大小写 | password,apikey,ssn | + +#### 响应说明 + +**成功响应**: 返回文件流 (与原接口一致) + +**错误响应**: +```json +{ + "status": 1, + "message": "字段名格式错误", + "data": null +} +``` + +--- + +## 7. 专利点识别 + +### 7.1 潜在专利点 + +#### 专利点1: 基于Pipeline语法扩展的字段级数据脱敏方法 + +**技术特点**: +1. 通过扩展Pipeline DSL语法，实现声明式字段屏蔽 +2. 在数据导出过程中动态解析语法并应用屏蔽逻辑 +3. 保持向后兼容的同时提供灵活的字段控制能力 + +**创新点**: +- 使用正则匹配提取屏蔽字段，避免修改前端代码 +- 基于语法扩展的声明式安全控制 +- 职责分离：前端生成、后端执行 + +#### 专利点2: 基于内存感知的字段屏蔽策略选择方法 + +**技术特点**: +1. 根据结果集大小和内存情况动态选择处理策略 +2. 小结果集使用collect模式，大结果集使用流式模式或拒绝 +3. 内存阈值检查机制，防止OOM + +**创新点**: +- 自适应的字段屏蔽策略 +- 基于运行时内存监控的容量保护 +- 性能与安全的平衡 + +### 7.2 专利录入 + +专利信息已录入到"BDP 专利"文档： +http://docs.weoa.com/sheets/2wAlXOo1WBHwPrAP/zDmhC + +--- + +## 8. 附录 + +### 8.1 关键文件清单 + +| 文件路径 | 说明 | 修改类型 | +|---------|------|---------| +| `linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/utils/ResultUtils.java` | ✅ 工具类 (新增189行) | **核心改进** - 提取公共字段过滤逻辑 | +| `linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/FsRestfulApi.java` | REST API | 参数扩展+调用ResultUtils | +| `linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/PipelineEngineConnExecutor.scala` | Pipeline执行器 | 正则扩展 | +| `linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/CSVExecutor.scala` | CSV导出 | 调用ResultUtils | +| `linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/ExcelExecutor.scala` | Excel导出 | 调用ResultUtils | + +### 8.2 配置项清单 + +| 配置项 | 默认值 | 说明 | 模块 | +|-------|-------|------|------| +| `wds.linkis.workspace.resultset.download.maxsize.csv` | 5000 | CSV下载最大行数 | 下载 | +| `wds.linkis.workspace.resultset.download.maxsize.excel` | 5000 | Excel下载最大行数 | 下载 | +| `pipeline.export.max.rows` | 100000 | Pipeline导出最大行数 | 导出 | +| `pipeline.export.memory.check.enabled` | true | 是否启用内存检查 | 导出 | +| `pipeline.export.memory.threshold` | 0.8 | 内存使用阈值 | 导出 | + +### 8.3 测试用例清单 + +#### 功能测试 + +| 用例ID | 用例名称 | 优先级 | +|-------|---------|--------| +| TC001 | 下载单字段屏蔽-CSV | P0 | +| TC002 | 下载多字段屏蔽-Excel | P0 | +| TC003 | 导出Pipeline语法-单字段 | P0 | +| TC004 | 导出Pipeline语法-多字段 | P0 | +| TC005 | 向后兼容-不传参数 | P0 | +| TC006 | 字段名大小写不敏感 | P1 | +| TC007 | 屏蔽不存在字段 | P1 | +| TC008 | 屏蔽所有字段 | P2 | + +#### 性能测试 + +| 用例ID | 数据量 | 屏蔽字段数 | 期望性能 | +|-------|-------|----------|---------| +| PT001 | 1万行×10列 | 2 | <3秒 | +| PT002 | 5万行×50列 | 5 | <8秒 | +| PT003 | 10万行×100列 | 10 | <15秒 | + +--- + +## 9. 性能优化策略 + +### 9.1 字段长度检测优化 +在`getFieldLength`方法中，对已知类型的对象进行特殊处理，避免不必要的`toString()`调用： + +```java +private static int getFieldLength(Object value) { + if (value == null) { + return 0; + } + if (value instanceof String) { + return ((String) value).length(); + } + return value.toString().length(); +} +``` + +### 9.2 大结果集处理优化 +对于大结果集，采用分批处理策略： +1. 设置内存使用阈值监控 +2. 超过阈值时采用流式处理 +3. 提供处理进度反馈机制 + +### 9.3 缓存机制优化 +在字段处理过程中使用Set来存储已处理的字段名，避免重复处理： + +```java +// 使用Set来存储已经处理过的字段名，避免重复处理 +Set processedFields = new HashSet<>(); +``` + +--- + +## 10. 变更历史 + +| 版本 | 日期 | 变更内容 | 作者 | +|-----|------|---------|------| +| v1.0 | 2025-10-28 | 初始版本 - 完成系统设计 | Claude Code | +| v1.1 | 2025-10-30 | ✅ 实现完成 - 更新实际实现细节，添加ResultUtils工具类设计 | 开发团队 | + +**v1.1版本主要变更**: +1. 新增ResultUtils工具类设计章节 (3.5节) +2. 更新PublicService实现代码，反映实际使用ResultUtils的简化实现 (4.1.4节) +3. 更新Pipeline引擎实现代码，反映实际使用ResultUtils的简化实现 (4.2.6节) +4. 更新文件清单，突出ResultUtils核心地位 (9.1节) +5. 添加实施总结章节，说明架构改进点 + +--- + +**文档结束** \ No newline at end of file diff --git a/docs/1.17.0/requirements/aisql-starrocks-engine-switch.md b/docs/1.17.0/requirements/aisql-starrocks-engine-switch.md new file mode 100644 index 00000000000..878c5aaf369 --- /dev/null +++ b/docs/1.17.0/requirements/aisql-starrocks-engine-switch.md @@ -0,0 +1,179 @@ +# AISQL任务支持StarRocks引擎类型切换需求 + +## 需求概述 + +为AISQL类型任务增加StarRocks引擎类型切换支持，允许用户通过多种方式指定任务使用StarRocks引擎执行。 + +## 需求背景 + +### 当前现状 +1. AISQL任务目前支持Spark和Hive两种引擎类型的切换 +2. 引擎切换通过以下两种方式实现： + - 通过模板配置（ec.resource.name）中的关键字匹配自动选择引擎 + - 通过调用Doctoris服务进行智能引擎选择 +3. StarRocks引擎已通过JDBC引擎实现，可通过查询用户数据源名称的方式提交任务 + +### 存在问题 +- 当前AISQL任务无法切换到StarRocks引擎执行 +- 用户无法通过配置参数或脚本注释指定使用StarRocks引擎 + +### 业务价值 +- 支持用户灵活选择StarRocks引擎执行AISQL任务 +- 与现有Spark/Hive引擎切换机制保持一致 +- 扩展AISQL任务的引擎支持能力 + +## 功能需求 + +### 功能点1：Runtime参数方式指定StarRocks引擎 + +**需求描述**： +用户可以在任务提交时，通过runtime参数指定当前AISQL任务使用StarRocks引擎执行。 + +**实现方式**： +- 新增runtime参数键：`ec.engine.type` +- 参数值：`starrocks` +- 当检测到该参数时，优先切换到StarRocks引擎 + +**示例**： +```json +{ + "executionCode": "SELECT * FROM table", + "runType": "aisql", + "params": { + "runtime": { + "ec.engine.type": "starrocks" + } + } +} +``` + +### 功能点2：脚本注释方式指定StarRocks引擎 + +**需求描述**： +用户可以在AISQL脚本中通过注释配置参数，指定当前任务使用StarRocks引擎执行。 + +**实现方式**： +- 参考TemplateConfUtils中ec.resource.name的实现方式 +- 支持在脚本注释中添加`@set ec.engine.type=starrocks`配置 +- 支持多种注释格式（SQL/Python/Scala） + +**示例**： +```sql +---@set ec.engine.type=starrocks +SELECT * FROM starrocks_table WHERE dt = '2024-01-01' +``` + +```python +##@set ec.engine.type=starrocks +SELECT COUNT(*) FROM user_table +``` + +### 功能点3：Doctoris服务集成 + +**需求描述**： +当指定使用StarRocks引擎时，需要调用Doctoris服务，传递标识参数表明当前任务必须使用StarRocks引擎。 + +**实现要求**： +- 扩展现有Doctoris服务调用接口 +- 新增参数标识：`forceEngineType` 或 `fixedEngineType` +- 参数值：`starrocks` +- Doctoris服务根据该标识强制返回StarRocks引擎 + +## 验收标准 + +### 功能验收 +1. 通过runtime参数`ec.engine.type=starrocks`可成功切换到StarRocks引擎 +2. 通过脚本注释`@set ec.engine.type=starrocks`可成功切换到StarRocks引擎 +3. Runtime参数方式的优先级高于脚本注释方式 +4. 调用Doctoris服务时正确传递StarRocks标识参数 +5. 任务执行记录中正确记录使用的引擎类型为JDBC +6. 白名单功能正常工作： + - 白名单为空时，所有用户可以使用StarRocks引擎 + - 白名单配置用户后，只有白名单用户可以使用 + - 白名单配置部门后，只有白名单部门的用户可以使用 + - 不在白名单的用户指定StarRocks引擎时，系统忽略该配置并使用默认引擎 + +### 性能要求 +- 引擎切换逻辑不影响现有任务提交性能 +- 参数解析耗时不超过10ms + +### 兼容性要求 +- 不影响现有Spark和Hive引擎的切换功能 +- 功能开关关闭时，行为与上一版本保持一致 +- 向后兼容，不修改现有接口签名 + +### 安全要求 +- 验证用户是否有对应StarRocks数据源的访问权限 +- 引擎切换不绕过现有权限校验机制 + +## 功能点4：用户和部门白名单控制 + +**需求描述**： +为了更安全地控制StarRocks引擎切换功能的使用范围，需要增加用户和部门白名单机制。只有配置在白名单中的用户或部门才能使用StarRocks引擎切换功能。 + +**实现方式**： +- 新增配置项：`linkis.aisql.starrocks.whitelist.users`，配置允许使用的用户列表（逗号分隔） +- 新增配置项：`linkis.aisql.starrocks.whitelist.departments`，配置允许使用的部门ID列表（逗号分隔） +- 白名单检查逻辑： + - 如果白名单配置为空，则所有用户都可以使用（兼容现有行为） + - 如果白名单配置不为空，则只有白名单中的用户或部门才能使用 + - 用户检查：检查提交任务的用户是否在用户白名单中 + - 部门检查：获取提交任务用户的部门ID，检查是否在部门白名单中 + - 满足任一条件即可使用StarRocks引擎 + +**示例配置**： +```properties +# 允许使用StarRocks引擎的用户（逗号分隔） +linkis.aisql.starrocks.whitelist.users=user1,user2,admin + +# 允许使用StarRocks引擎的部门ID（逗号分隔） +linkis.aisql.starrocks.whitelist.departments=dept001,dept002 + +# 如果两个配置都为空，则所有用户都可以使用 +``` + +**行为说明**： +- 当用户不在白名单中时，即使指定了`ec.engine.type=starrocks`，系统也会忽略该配置，继续使用默认的Spark/Hive引擎选择逻辑 +- 日志中会记录白名单检查结果，方便问题排查 + +## 配置开关 + +新增配置项： +- `linkis.aisql.starrocks.switch`：StarRocks引擎切换功能开关，默认值：`false` +- `linkis.aisql.default.starrocks.engine.type`：默认StarRocks引擎类型，默认值：`jdbc-4` +- `linkis.aisql.starrocks.template.keys`：StarRocks模板关键字，默认值：`starrocks` +- `linkis.aisql.starrocks.whitelist.users`：用户白名单（逗号分隔），默认值：空（所有用户可用） +- `linkis.aisql.starrocks.whitelist.departments`：部门白名单（逗号分隔），默认值：空（所有部门可用） + +## 依赖关系 + +### 前置依赖 +- JDBC引擎插件已支持StarRocks +- StarRocks数据源管理功能已实现 + +### 影响模块 +- linkis-entrance：AISQL任务拦截器 +- linkis-entrance-conf：配置管理 +- linkis-computation-governance-common：任务协议 + +## 风险评估 + +### 技术风险 +- **风险**：StarRocks引擎可能不支持某些AISQL语法 +- **应对**：在文档中明确说明语法限制，引擎切换失败时给出明确提示 + +### 兼容性风险 +- **风险**：新增配置可能与现有配置冲突 +- **应对**：使用独立的配置键名，遵循现有配置命名规范 + +## 参考资料 + +- TemplateConfUtils实现：`linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala` +- AISQLTransformInterceptor实现：`linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala` +- JDBCConfiguration配置：`linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/conf/JDBCConfiguration.scala` + +## 更新记录 + +| 版本 | 日期 | 作者 | 变更说明 | +|------|------|------|----------| +| v1.0 | 2025-10-27 | AI | 初始版本 | diff --git a/docs/1.17.0/requirements/resultset-field-truncation.md b/docs/1.17.0/requirements/resultset-field-truncation.md new file mode 100644 index 00000000000..9ef5c0b0758 --- /dev/null +++ b/docs/1.17.0/requirements/resultset-field-truncation.md @@ -0,0 +1,230 @@ +# 结果集查看、下载和导出接口优化需求文档 + +## 文档信息 +| 项目 | 信息 | +|-----|------| +| 文档版本 | v1.1 (已实现) | +| 创建日期 | 2025-10-27 | +| 更新日期 | 2025-10-30 | +| 当前版本 | Linkis 1.17.0 | +| 负责模块 | linkis-pes-publicservice + pipeline + linkis-storage | +| 开发分支 | feature/1.17.0-resultset-field-masking | +| 状态 | ✅ 开发完成，已测试 | + +## 实施总结 + +### 代码修改统计 +本次开发包含**敏感字段屏蔽**和**字段截取**两个功能： + +```bash +15 files changed, 4166 insertions(+), 386 deletions(-) +``` + +### 新增文件 + +| 文件 | 行数 | 说明 | +|------|-----|------| +| `ResultUtils.java` | 514行 | 核心工具类，包含字段屏蔽和截取逻辑 | +| `FieldTruncationResult.java` | 73行 | 字段截取结果封装实体类 | +| `OversizedFieldInfo.java` | 68行 | 超长字段信息实体类 | + +### 修改文件 + +| 文件 | 修改类型 | 说明 | +|------|---------|------| +| `LinkisStorageConf.scala` | 配置扩展 (+11行) | 新增字段截取相关配置项 | +| `WorkSpaceConfiguration.java` | 配置扩展 (+4行) | 新增功能开关配置 | +| `FsRestfulApi.java` | 功能增强 (218改动) | 下载接口支持字段屏蔽和截取 | +| `PipelineEngineConnExecutor.scala` | 语法扩展 (+16改动) | 支持without和truncate子句 | +| `CSVExecutor.scala` | 功能增强 (70改动) | CSV导出支持屏蔽和截取 | +| `ExcelExecutor.scala` | 功能增强 (140改动) | Excel导出支持屏蔽和截取 | +| 文档 | 新增4份 | 需求和设计文档 | + +### 核心改进点 + +1. **统一工具类**: 将字段屏蔽和截取逻辑提取到`ResultUtils`工具类，实现代码复用 +2. **组合功能**: 支持字段屏蔽和字段截取同时使用（`applyFieldMaskingAndTruncation`方法） +3. **可配置化**: 所有阈值和开关都通过`CommonVars`配置管理 +4. **向后兼容**: 功能可选，不影响现有功能 +5. **标记机制**: 截取后的字段会在列名添加`(truncated to N chars)`后缀标记 + +### 实现的核心方法 + +**ResultUtils工具类方法**: +- `detectAndHandle()`: 检测并处理超长字段（主入口方法） +- `detectOversizedFields()`: 检测超长字段，返回超长字段列表 +- `truncateFields()`: 截取超长字段值 +- `applyFieldMaskingAndTruncation()`: 同时应用字段屏蔽和截取 + +--- + +## 1. 需求概述 + +### 1.1 需求主题 +结果集查看、下载和导出接口优化 - 超长字段截取功能 + +### 1.2 需求背景 +当前结果集查看功能存在以下问题： +- 当某一列字段内容超过10000字符时，会导致结果集无法正常查看 +- 缺少对超长字段的检测和处理机制 +- 用户无法获知哪些字段超长，也无法选择处理方式 + +### 1.3 需求目标 +为结果集查看、下载、导出功能增加超长字段检测和截取能力，提升系统稳定性和用户体验。 + +## 2. 功能需求 + +### 2.1 核心功能点 + +#### 2.1.1 结果集查看功能优化 +- **触发条件**：结果集中存在字段值长度超过10000字符 +- **处理逻辑**： + 1. 检测所有字段值长度 + 2. 收集超过10000字符的字段信息（字段名、行号、实际长度） + 3. 最多收集20个超长字段 + 4. 返回超长字段列表给前端，由用户确认是否截取 + 5. 若用户确认截取，则截取前10000个字符后返回结果集 + 6. 若用户取消，则返回原始数据（可能导致查看失败） + +#### 2.1.2 结果集下载功能优化 +- **触发条件**：结果集中存在字段值长度超过10000字符 +- **处理逻辑**：与查看功能相同 + 1. 检测所有字段值长度 + 2. 收集超过10000字符的字段信息 + 3. 最多收集20个超长字段 + 4. 返回超长字段列表给前端确认 + 5. 若用户确认截取，则截取前10000个字符后下载 + 6. 若用户取消，则下载原始数据 + +#### 2.1.3 结果集导出功能优化 +- **触发条件**：结果集中存在字段值长度超过32767字符 +- **处理逻辑**： + 1. 检测所有字段值长度 + 2. 收集超过32767字符的字段信息（字段名、行号、实际长度） + 3. 最多收集20个超长字段 + 4. 返回超长字段列表给前端确认 + 5. 若用户确认截取，则截取前32767个字符后导出 + 6. 若用户取消，则导出原始数据（可能导致导出失败） + +### 2.2 功能约束 + +#### 2.2.1 超长字段收集上限 +- 最多收集20个超长字段信息 +- 超过20个时，只返回前20个 + +#### 2.2.2 截取长度配置 +- 查看和下载：默认10000字符，可配置 +- 导出：默认32767字符，可配置 + +#### 2.2.3 功能开关 +- 必须提供功能总开关，关闭时相当于回退到原版本功能 +- 开关关闭时，不进行任何检测和截取 + +## 3. 非功能需求 + +### 3.1 性能要求 +- 字段长度检测不应显著增加接口响应时间 +- 对于大结果集，检测逻辑应高效执行 + +### 3.2 兼容性要求 +- 遵循最小改动原则，不影响现有功能 +- 功能开关关闭时，行为与原版本完全一致 + +### 3.3 可配置性要求 +- 所有阈值参数必须可配置 +- 配置必须使用 `CommonVars` 统一管理 +- 参考 `JobhistoryConfiguration` 的配置方式 + +## 4. 接口设计要求 + +### 4.1 返回数据结构 +需要在结果集相关接口的响应中增加以下信息： + +```json +{ + "hasOversizedFields": true, + "oversizedFields": [ + { + "fieldName": "column1", + "rowIndex": 0, + "actualLength": 15000, + "maxLength": 10000 + } + ], + "maxOversizedFieldCount": 20, + "data": "结果集数据" +} +``` + +### 4.2 前端交互流程 +1. 后端检测到超长字段，返回超长字段列表 +2. 前端展示提示弹窗，显示超长字段信息 +3. 用户选择是否截取 +4. 前端带着用户选择结果重新请求接口 +5. 后端根据用户选择返回截取或原始数据 + +## 5. 配置项清单 + +| 配置项名称 | 默认值 | 说明 | +|-----------|--------|------| +| `linkis.resultset.field.truncation.enabled` | `false` | 功能总开关 | +| `linkis.resultset.field.view.max.length` | `10000` | 查看功能字段最大长度 | +| `linkis.resultset.field.download.max.length` | `10000` | 下载功能字段最大长度 | +| `linkis.resultset.field.export.max.length` | `32767` | 导出功能字段最大长度 | +| `linkis.resultset.field.oversized.max.count` | `20` | 最多收集超长字段数量 | + +## 6. 实施范围 + +### 6.1 开发范围 +- 仅实现后端接口功能 +- 不涉及前端页面开发 + +### 6.2 代码边界 +- 不修改现有表结构 +- 不引入新的第三方依赖 +- 不修改现有公共接口签名（只扩展返回数据） + +## 7. 验收标准 + +### 7.1 功能验收 +- [x] ✅ 功能开关关闭时，行为与原版本一致 +- [x] ✅ 功能开关开启时，能正确检测超长字段 +- [x] ✅ 能返回正确的超长字段信息列表（通过FieldTruncationResult封装） +- [x] ✅ 用户选择截取时，能正确截取指定长度 +- [x] ✅ 超长字段超过20个时，只返回前20个 +- [x] ✅ 截取后的字段会在列名添加标记`(truncated to N chars)` + +### 7.2 配置验收 +- [x] ✅ 所有配置项使用 `CommonVars` 管理 +- [x] ✅ 配置项放在对应模块的 Configuration 类中（LinkisStorageConf和WorkSpaceConfiguration） +- [x] ✅ 配置项可以正确读取和生效 + +### 7.3 兼容性验收 +- [x] ✅ 不影响现有结果集查看功能 +- [x] ✅ 不影响现有结果集下载功能 +- [x] ✅ 不影响现有结果集导出功能 + +### 7.4 扩展功能验收 (新增) +- [x] ✅ 支持字段屏蔽和字段截取同时使用 +- [x] ✅ Pipeline语法支持truncate参数 +- [x] ✅ CSV和Excel导出都支持字段截取 + +## 8. 风险评估 + +### 8.1 技术风险 +- **性能影响**：字段长度检测可能影响性能，需要优化检测逻辑 +- **内存占用**：大结果集检测可能增加内存占用 + +### 8.2 兼容性风险 +- **前端兼容**：老版本前端不识别新增的返回字段，需要保证向下兼容 + +## 9. 参考资料 + +### 9.1 相关代码模块 +- 结果集查看相关代码 +- 结果集下载相关代码 +- 结果集导出相关代码 + +### 9.2 配置参考 +- `org.apache.linkis.jobhistory.conf.JobhistoryConfiguration` +- `org.apache.linkis.common.conf.CommonVars` diff --git a/docs/1.17.0/requirements/resultset-sensitive-field-masking.md b/docs/1.17.0/requirements/resultset-sensitive-field-masking.md new file mode 100644 index 00000000000..1c29f87b40a --- /dev/null +++ b/docs/1.17.0/requirements/resultset-sensitive-field-masking.md @@ -0,0 +1,1182 @@ +# Linkis结果集下载和导出功能支持敏感字段屏蔽需求文档 + +## 文档信息 + +| 项目 | 信息 | +|-----|------| +| 文档版本 | v1.1 (已实现) | +| 创建日期 | 2025-10-27 | +| 更新日期 | 2025-10-30 | +| 当前版本 | Linkis 1.17.0 | +| 负责模块 | linkis-pes-publicservice (Filesystem) + pipeline | +| 开发分支 | feature/1.17.0-resultset-sensitive-field-masking | +| 状态 | ✅ 开发完成，已测试 | + +--- + +## 实施总结 + +### 代码修改统计 + +```bash +7 files changed, 2698 insertions(+), 163 deletions(-) +``` + +| 文件 | 修改类型 | 说明 | +|------|---------|------| +| `FsRestfulApi.java` | 功能增强 | 添加maskedFieldNames参数支持 | +| `ResultUtils.java` | 新增工具类 | 提取公共字段过滤逻辑 | +| `PipelineEngineConnExecutor.scala` | 语法扩展 | 支持without子句解析 | +| `CSVExecutor.scala` | 功能增强 | 实现CSV导出字段屏蔽 | +| `ExcelExecutor.scala` | 功能增强 | 实现Excel导出字段屏蔽 | +| `resultset-sensitive-field-masking.md` | 新增文档 | 需求文档 | +| `resultset-sensitive-field-masking-design.md` | 新增文档 | 设计文档 | + +### 核心改进点 + +1. **代码复用**: 将字段过滤逻辑提取到`ResultUtils`工具类，实现Java和Scala代码共享 +2. **简化实现**: 使用`ResultUtils.dealMaskedField()`统一处理字段屏蔽，减少重复代码 +3. **更好的架构**: 将通用逻辑放在`linkis-storage`模块，提高可维护性 + +--- + +## 1. 需求背景 + +### 1.1 现状说明 + +Linkis当前在结果集查看功能(`/api/rest_j/v1/filesystem/openFile`)中已经实现了敏感字段屏蔽机制,通过`maskedFieldNames`参数支持动态指定需要屏蔽的字段列表。该功能在前端展示结果集时可以有效保护敏感数据。 + +### 1.2 存在的安全风险 + +虽然结果集查看时支持屏蔽敏感字段,但用户仍然可以通过以下接口**绕过屏蔽机制**获取完整的敏感数据: + +1. **单结果集下载接口**: `/api/rest_j/v1/filesystem/resultsetToExcel` +2. **多结果集下载接口**: `/api/rest_j/v1/filesystem/resultsetsToExcel` +3. **其他导出接口**(如存在) + +这导致敏感字段屏蔽功能形同虚设,存在数据泄露风险。 + +### 1.3 需求来源 + +- 数据安全合规要求 +- 敏感信息保护策略的全链路落地 +- 用户权限管理的完善 + +--- + +## 2. 功能现状分析 + +### 2.1 结果集查看功能 (已支持屏蔽) + +#### 接口信息 + +``` +接口路径: /api/rest_j/v1/filesystem/openFile +请求方法: GET +Controller类: org.apache.linkis.filesystem.restful.api.FsRestfulApi +实现方法: openFile() (行625-777) +``` + +#### 敏感字段屏蔽参数 + +| 参数名 | 类型 | 是否必填 | 说明 | 示例 | +|-------|------|---------|------|------| +| maskedFieldNames | String | 否 | 需要屏蔽的字段名,多个字段用逗号分隔(不区分大小写) | password,apikey,secret_token | + +#### 屏蔽实现机制 + +**实现位置**: FsRestfulApi.java 行735-858 + +```java +// 1. 解析屏蔽字段列表 +Set maskedFields = + new HashSet<>(Arrays.asList(maskedFieldNames.toLowerCase().split(","))); + +// 2. 过滤元数据 +Map[] metadata = filterMaskedFieldsFromMetadata(resultmap, maskedFields); + +// 3. 移除数据内容中的对应列 +List fileContent = + removeFieldsFromContent(resultmap, result.getSecond(), maskedFields); +``` + +**关键方法**: + +1. `filterMaskedFieldsFromMetadata()` (行841-858): 从元数据中过滤屏蔽字段 +2. `removeFieldsFromContent()` (行787-838): 从内容数据中移除屏蔽字段列 + +**特性**: +- 不区分大小写匹配字段名 +- 从后向前删除列索引,避免索引变化问题 +- 同时处理元数据和内容数据 + +--- + +### 2.2 单结果集下载功能 (不支持屏蔽) + +#### 接口信息 + +``` +接口路径: /api/rest_j/v1/filesystem/resultsetToExcel +请求方法: GET +Controller类: org.apache.linkis.filesystem.restful.api.FsRestfulApi +实现方法: resultsetToExcel() (行972-1084) +``` + +#### 核心参数 + +| 参数名 | 类型 | 默认值 | 说明 | +|-------|------|-------|------| +| path | String | - | 结果集文件路径(必填) | +| outputFileType | String | csv | 导出格式: csv 或 xlsx | +| csvSeparator | String | , | CSV分隔符 | +| outputFileName | String | downloadResultset | 输出文件名 | +| sheetName | String | result | Excel sheet名称 | +| nullValue | String | NULL | null值替换字符串 | +| limit | Integer | 0 | 行数限制(0表示使用配置值) | +| autoFormat | Boolean | false | 是否自动格式化 | +| keepNewline | Boolean | false | 是否保留换行符 | + +#### 实现流程 + +``` +用户请求 → 权限验证 → 文件系统操作 → 格式判断(CSV/XLSX) +→ Writer初始化 → 数据写入 → 响应流输出 +``` + +**问题**: 当前实现直接将完整的结果集数据写入输出流,**没有任何字段过滤或屏蔽逻辑**。 + +--- + +### 2.3 多结果集下载功能 (不支持屏蔽) + +#### 接口信息 + +``` +接口路径: /api/rest_j/v1/filesystem/resultsetsToExcel +请求方法: GET +Controller类: org.apache.linkis.filesystem.restful.api.FsRestfulApi +实现方法: resultsetsToExcel() (行1105-1189) +``` + +#### 核心参数 + +| 参数名 | 类型 | 默认值 | 说明 | +|-------|------|-------|------| +| path | String | - | 结果集目录路径(必填) | +| outputFileName | String | downloadResultset | 输出文件名 | +| nullValue | String | NULL | null值替换字符串 | +| limit | Integer | 0 | 每个结果集的行数限制 | +| autoFormat | Boolean | false | 是否自动格式化 | + +#### 特殊说明 + +- **仅支持XLSX格式** +- path参数为目录路径,包含多个结果集文件 +- 使用`StorageMultiExcelWriter`将多个结果集合并到单个Excel的不同Sheet +- 自动按文件序号排序: `ResultSetUtils.sortByNameNum()` + +**问题**: 与单结果集下载类似,**没有任何字段过滤或屏蔽逻辑**。 + +--- + +### 2.4 结果集导出功能 (不支持屏蔽) + +#### 功能说明 + +结果集导出功能与下载功能**不同**,它使用**Pipeline引擎**将dolphin结果集文件导出到服务器共享目录,而非直接下载到客户端。 + +#### 实现方式 + +**核心引擎**: Pipeline引擎 +**实现语言**: Scala +**代码路径**: `linkis-engineconn-plugins/pipeline/` + +#### 工作流程 + +``` +用户操作流程: +1. 用户在前端点击"导出"按钮 +2. 前端弹出导出配置对话框 (resultsExport.vue) +3. 用户选择: + - 导出文件名 + - 导出格式 (CSV/Excel) + - 目标路径 (服务器共享目录) + - 是否导出全部结果集 (多结果集时) +4. 前端生成Pipeline代码: from <源路径> to <目标路径> +5. 提交Pipeline脚本到引擎执行 +6. Pipeline引擎读取dolphin文件 → 转换格式 → 写入目标目录 +``` + +#### 前端实现 + +**文件路径**: `linkis-web/src/components/consoleComponent/resultsExport.vue` + +**导出配置参数**: + +| 参数 | 类型 | 说明 | 验证规则 | +|-----|------|------|---------| +| name | String | 导出文件名 | 1-200字符,仅支持英文/数字/中文 | +| path | String | 目标目录路径 | 必填,从目录树选择 | +| format | String | 导出格式 | 1=CSV, 2=Excel | +| isAll | Boolean | 是否导出全部结果集 | 仅多结果集且Excel格式时可选 | + +**Pipeline代码生成逻辑**: + +```javascript +// resultsExport.vue 导出确认方法 +exportConfirm() { + // 生成临时脚本名称 + const tabName = `new_stor_${Date.now()}.out`; + + // 确定源路径 + let temPath = this.currentPath; // 当前结果集路径 + if (this.isAll) { + // 导出全部时,源路径为目录(不带文件名) + temPath = temPath.substring(0, temPath.lastIndexOf('/')); + } + + // 根据格式添加扩展名 + const exportOptionName = this.exportOption.format === '2' + ? `${this.exportOption.name}.xlsx` + : `${this.exportOption.name}.csv`; + + // 生成Pipeline执行代码 + const code = `from ${temPath} to ${this.exportOption.path}/${exportOptionName}`; + + // 添加临时脚本并自动执行 + this.dispatch('Workbench:add', { id: md5Path, code, saveAs: true }, (f) => { + this.$nextTick(() => { + this.dispatch('Workbench:run', { id: md5Path }); + }); + }); +} +``` + +#### Pipeline引擎实现 + +**执行入口**: `PipelineEngineConnExecutor.scala` (行69-89) + +```scala +// 正则解析Pipeline语法 +val regex = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s?".r + +code match { + case regex(sourcePath, destPath) => + // 选择合适的执行器 + PipelineExecutorSelector + .select(sourcePath, destPath, options) + .execute(sourcePath, destPath, engineExecutorContext) +} +``` + +**执行器选择逻辑**: `PipelineExecutorSelector.scala` + +```scala +def select(sourcePath: String, destPath: String, options: Map[String, String]): PipeLineExecutor = { + // 根据目标文件扩展名选择执行器 + getSuffix(destPath) match { + case ".csv" => CSVExecutor // CSV导出 + case ".xlsx" => ExcelExecutor // Excel导出 + case _ if sameFileName => CopyExecutor // 文件复制 + case _ => throw UnsupportedOutputTypeException + } +} +``` + +#### 三大执行器实现 + +##### 1. CSVExecutor - CSV格式导出 + +**文件**: `CSVExecutor.scala` + +**执行流程**: + +```scala +override def execute(sourcePath: String, destPath: String, context: EngineExecutionContext): ExecuteResponse = { + // 1. 验证源文件是否为结果集 + if (!FileSource.isResultSet(sourcePath)) { + throw NotAResultSetFileException + } + + // 2. 创建文件系统 + val sourceFs = FSFactory.getFs(new FsPath(sourcePath)) + val destFs = FSFactory.getFs(new FsPath(destPath)) + + // 3. 创建FileSource读取结果集 + val fileSource = FileSource.create(new FsPath(sourcePath), sourceFs) + + // 4. 获取配置参数 + val nullValue = options.getOrDefault("pipeline.output.shuffle.null.type", "NULL") + val charset = options.getOrDefault("pipeline.output.charset", "UTF-8") + val separator = options.getOrDefault("pipeline.field.split", ",") + val quoteRetouchEnable = options.getOrDefault("pipeline.field.quote.retoch.enable", false) + + // 5. 创建CSV Writer + val outputStream = destFs.write(new FsPath(destPath), isOverwrite = true) + val csvWriter = CSVFsWriter.getCSVFSWriter(charset, separator, quoteRetouchEnable, outputStream) + + // 6. 写入数据 (仅处理nullValue参数) + fileSource.addParams("nullValue", nullValue).write(csvWriter) + + // 7. 清理资源 + IOUtils.closeQuietly(csvWriter) + IOUtils.closeQuietly(fileSource) +} +``` + +**问题**: ❌ **没有任何敏感字段屏蔽逻辑** + +##### 2. ExcelExecutor - Excel格式导出 + +**文件**: `ExcelExecutor.scala` + +**执行流程**: + +```scala +override def execute(sourcePath: String, destPath: String, context: EngineExecutionContext): ExecuteResponse = { + val sourceFs = FSFactory.getFs(new FsPath(sourcePath)) + val destFs = FSFactory.getFs(new FsPath(destPath)) + + val outputStream = destFs.write(new FsPath(destPath), isOverwrite = true) + + // 支持两种模式: + // 模式1: 单个结果集文件 (sourcePath包含".") + if (sourcePath.contains(".")) { + val fileSource = FileSource.create(new FsPath(sourcePath), sourceFs) + val excelWriter = ExcelFsWriter.getExcelFsWriter( + charset = "utf-8", + sheetName = "result", + dateFormat = "yyyy-MM-dd HH:mm:ss", + outputStream, + autoFormat = false + ) + fileSource.addParams("nullValue", nullValue).write(excelWriter) + } + // 模式2: 多个结果集 (sourcePath为目录) + else { + val fsPathList = sourceFs.listPathWithError(new FsPath(sourcePath)).getFsPaths + ResultSetUtils.sortByNameNum(fsPathList) // 按序号排序 + val fileSource = FileSource.create(fsPathList.toArray, sourceFs) + val multiExcelWriter = new StorageMultiExcelWriter(outputStream, autoFormat) + fileSource.addParams("nullValue", nullValue).write(multiExcelWriter) + } +} +``` + +**问题**: ❌ **同样没有敏感字段屏蔽逻辑** + +##### 3. CopyExecutor - 文件复制 + +**文件**: `CopyExecutor.scala` + +```scala +override def execute(sourcePath: String, destPath: String, context: EngineExecutionContext): ExecuteResponse = { + val sourceFs = FSFactory.getFs(new FsPath(sourcePath)) + val destFs = FSFactory.getFs(new FsPath(destPath)) + + val inputStream = sourceFs.read(new FsPath(sourcePath)) + val outputStream = destFs.write(new FsPath(destPath), isOverwrite = true) + + // 直接流复制,不做任何处理 + IOUtils.copy(inputStream, outputStream) +} +``` + +**问题**: ❌ **直接复制文件,完全绕过所有检查** + +#### 关键配置项 + +**文件**: `PipelineEngineConfiguration.scala` + +| 配置项 | 默认值 | 说明 | +|-------|-------|------| +| pipeline.output.charset | UTF-8 | 输出字符集 | +| pipeline.field.split | , | CSV字段分隔符 | +| pipeline.output.shuffle.null.type | NULL | 空值替换标记 | +| pipeline.field.quote.retoch.enable | false | 引号处理开关 | +| pipeline.output.isoverwrite | true | 是否覆盖已存在文件 | +| wds.linkis.pipeline.export.excel.auto_format.enable | false | Excel自动格式化 | + +#### Dolphin结果集文件格式 + +**文件**: `Dolphin.scala` + +```scala +object Dolphin { + val MAGIC = "dolphin" // 文件头魔数 (7字节) + val DOLPHIN_FILE_SUFFIX = ".dolphin" // 文件后缀 + val COL_SPLIT = "," // 列分隔符 + val NULL = "NULL" // 空值标记 + val INT_LEN = 10 // 整数字段长度(固定10字节) +} +``` + +**文件结构**: +1. 文件头 (7字节): "dolphin" +2. 类型标识 (10字节): TABLE/PICTURE/TEXT等 +3. 元数据区 (变长): 列名、数据类型、注释等 +4. 数据区 (变长): 按行存储的数据记录 + +#### 结果集读取流程 + +**FileSource.scala** → **ResultsetFileSource.scala** → **StorageCSVWriter.scala** + +```scala +// ResultsetFileSource.scala - 结果集字段处理 +class ResultsetFileSource(fileSplits: Array[FileSplit]) extends AbstractFileSource(fileSplits) { + // 应用shuffle变换 (仅处理NULL值和Double格式) + shuffle({ + case t: TableRecord => + new TableRecord(t.row.map { + case null | "NULL" => + val nullValue = getParams.getOrDefault("nullValue", "NULL") + nullValue + case value: Double => StorageUtils.doubleToString(value) + case rvalue => rvalue + }) + }) +} +``` + +**关键发现**: +- ✅ 有NULL值处理 +- ✅ 有数值格式化 +- ❌ **没有字段级别的过滤或屏蔽** +- ❌ **没有敏感字段检查** +- ❌ **没有数据脱敏处理** + +#### 导出目标路径 + +根据配置项 `wds.linkis.filesystem.root.path`: + +| 文件系统类型 | 默认根路径 | 说明 | +|------------|----------|------| +| 本地文件系统 | file:///tmp/linkis/ | LOCAL_USER_ROOT_PATH | +| HDFS | hdfs:///tmp/{user}/linkis/ | HDFS_USER_ROOT_PATH_PREFIX + user + SUFFIX | + +**用户可选择的导出路径**: +- 个人工作目录 +- 共享目录(如配置允许) +- 项目目录 + +#### 核心安全问题 + +##### 问题1: dolphin源文件包含所有字段 + +``` +执行SQL: SELECT name, password, email FROM users; + ↓ +生成结果集: /user/hadoop/linkis/result_001.dolphin + ↓ +dolphin文件内容: + - 元数据: [name, password, email] + - 数据: ["Alice", "pwd123", "alice@example.com"] + ["Bob", "secret456", "bob@example.com"] +``` + +**问题**: 结果集文件已包含所有敏感字段 + +##### 问题2: 导出时未进行字段屏蔽 + +``` +用户执行导出: + from /user/hadoop/linkis/result_001.dolphin + to /shared/exports/users.csv + ↓ +CSVExecutor.execute() 流程: + 1. FileSource.create() - 读取dolphin文件 + 2. fileSource.addParams("nullValue", "NULL") + 3. fileSource.write(csvWriter) + ↓ +输出文件 /shared/exports/users.csv: + name,password,email + Alice,pwd123,alice@example.com + Bob,secret456,bob@example.com +``` + +**问题**: ❌ **password字段未被屏蔽,直接导出** + +##### 问题3: 导出文件存在数据泄露风险 + +``` +导出后的文件位置: + - 服务器共享目录 (/shared/exports/) + - 其他用户可能有读权限 + - 文件未加密 + - 没有访问审计 +``` + +**风险**: +- 敏感数据以明文形式存储在共享目录 +- 可被其他有权限的用户访问 +- 可被复制或传播 +- 难以追踪数据流向 + +#### 与下载功能的对比 + +| 维度 | 下载功能 (resultsetToExcel) | 导出功能 (Pipeline) | +|-----|------------------------|------------------| +| **触发方式** | REST API调用 | Pipeline脚本执行 | +| **数据流向** | 服务器 → 客户端浏览器 | 服务器 → 服务器目录 | +| **格式转换** | FsRestfulApi中实现 | Pipeline执行器实现 | +| **敏感字段屏蔽** | ❌ 不支持 | ❌ 不支持 | +| **行数限制** | ✅ 支持 (默认5000) | ❌ 不限制 | +| **权限检查** | ✅ checkIsUsersDirectory() | ⚠️ 仅文件系统级别 | +| **审计日志** | ✅ 有日志记录 | ⚠️ 仅引擎执行日志 | +| **文件访问控制** | ✅ 单次下载后用户控制 | ⚠️ 服务器文件系统权限 | + +#### 完整的Pipeline导出执行链路 + +``` +前端 resultsExport.vue + ↓ [生成Pipeline代码] +from /user/hadoop/linkis/result.dolphin to /shared/export/file.csv + ↓ [提交到Workbench执行] +PipelineEngineConnExecutor.executeLine() + ↓ [正则解析] +sourcePath = /user/hadoop/linkis/result.dolphin +destPath = /shared/export/file.csv + ↓ [选择执行器] +PipelineExecutorSelector.select() → CSVExecutor + ↓ [执行导出] +CSVExecutor.execute() + ├─ FSFactory.getFs(sourcePath) + ├─ FileSource.create(sourcePath, fs) + │ └─ ResultSetFactory.getResultSetByPath() + │ └─ ResultSetReader.getResultSetReader() + │ └─ 读取dolphin文件 (含所有字段) + ├─ CSVFsWriter.getCSVFSWriter() + ├─ fileSource.addParams("nullValue", "NULL") + └─ fileSource.write(csvWriter) + ├─ ResultsetFileSource.shuffle() [仅NULL值处理] + ├─ StorageCSVWriter.addMetaData() [写入所有列名] + └─ StorageCSVWriter.addRecord() [写入所有数据] + ↓ +输出文件: /shared/export/file.csv (包含所有敏感字段) +``` + +**关键发现**: +- 整个链路中**没有任何一个环节**检查或过滤敏感字段 +- 所有字段从dolphin文件原样转换到目标格式 +- 用户可以轻松绕过任何前置的敏感数据检查 + +--- + +## 3. 需求详细说明 + +### 3.1 核心需求 + +**在结果集下载和导出时支持敏感字段屏蔽功能,与查看功能保持一致的安全策略,全面堵塞敏感数据泄露渠道。** + +**涉及的三个功能模块**: +1. ✅ **结果集查看** (`/api/rest_j/v1/filesystem/openFile`) - 已支持屏蔽 +2. ❌ **结果集下载** (`/api/rest_j/v1/filesystem/resultsetToExcel`, `resultsetsToExcel`) - 需要支持 +3. ❌ **结果集导出** (Pipeline引擎: `CSVExecutor`, `ExcelExecutor`) - 需要支持 + +### 3.2 功能要求 + +#### 3.2.1 参数设计 + +##### (1) 下载接口参数扩展 + +在`resultsetToExcel`和`resultsetsToExcel`两个接口中**新增可选参数**: + +| 参数名 | 类型 | 是否必填 | 默认值 | 说明 | +|-------|------|---------|-------|------| +| maskedFieldNames | String | 否 | null | 需要屏蔽的字段名,多个字段用逗号分隔 | + +**示例请求**: + +``` +GET /api/rest_j/v1/filesystem/resultsetToExcel?path=/user/result.dolphin&outputFileType=csv&maskedFieldNames=password,apikey,ssn +``` + +##### (2) Pipeline导出语法扩展 ⭐ + +**职责划分**: +- **其他团队**: 负责前端交互和Pipeline代码生成（包含屏蔽字段） +- **我们团队**: 负责Pipeline引擎执行（解析语法并应用屏蔽逻辑） + +**新增Pipeline语法**: + +``` +from <源路径> to <目标路径> without "<字段名1,字段名2,...>" +``` + +**语法规则**: +- `without` 关键字后跟屏蔽字段列表 +- 字段名用**双引号**包裹 +- 多个字段用**逗号分隔**（不区分大小写） +- 双引号内可包含空格 + +**示例**: + +```sql +-- 示例1: 屏蔽单个字段 +from /user/result.dolphin to /export/file.csv without "password" + +-- 示例2: 屏蔽多个字段 +from /user/result.dolphin to /export/users.xlsx without "password,apikey,credit_card" + +-- 示例3: 字段名包含空格 +from /user/result.dolphin to /export/data.csv without "user password, api key, credit card" + +-- 示例4: 不屏蔽（保持原语法） +from /user/result.dolphin to /export/file.csv +``` + +**语法兼容性**: +- ✅ 向后兼容：不使用`without`子句时，保持原有行为 +- ✅ 大小写不敏感：`WITHOUT`、`without`、`Without`均可 +- ✅ 空格容忍：关键字前后的空格会被自动处理 + +#### 3.2.2 屏蔽规则 + +1. **字段匹配** + - 不区分大小写 + - 精确匹配字段名(columnName) + - 支持多字段,使用逗号分隔 + +2. **屏蔽方式** + - 完全移除屏蔽字段列(而非替换为空值或掩码) + - 同时处理元数据(metadata)和数据内容(fileContent) + - 保持与`openFile`接口的一致性 + +3. **异常处理** + - 如果指定的屏蔽字段不存在,不报错,正常导出 + - 如果所有字段都被屏蔽,返回空结果集(仅包含结果集结构) + +#### 3.2.3 兼容性要求 + +1. **向后兼容**: 不传`maskedFieldNames`参数时,保持原有行为(导出完整数据) +2. **格式兼容**: 支持CSV和XLSX两种导出格式 +3. **性能要求**: 字段屏蔽逻辑不应显著影响导出性能 + +--- + +### 3.3 技术实现要求 + +#### 3.3.1 代码复用 + +- **复用现有方法**: 直接复用`openFile`中已实现的以下方法: + - `filterMaskedFieldsFromMetadata()` (FsRestfulApi.java 行841-858) + - `removeFieldsFromContent()` (FsRestfulApi.java 行787-838) + +- **考虑重构**: 如果方法访问级别不合适,建议将这两个方法: + - 从`private`修改为`protected`或提取到工具类 + - 或直接复制到下载方法和Pipeline执行器中 + +#### 3.3.2 实现位置 + +##### (1) 下载功能实现位置 + +**文件**: `FsRestfulApi.java` + +**修改方法**: +1. `resultsetToExcel()` (行972-1084) +2. `resultsetsToExcel()` (行1105-1189) + +**关键修改点**: + +```java +// 在fileSource.write(fsWriter)之前添加字段过滤逻辑 + +if (StringUtils.isNotBlank(maskedFieldNames)) { + Set maskedFields = + new HashSet<>(Arrays.asList(maskedFieldNames.toLowerCase().split(","))); + + // 获取元数据并过滤 + // 修改FileSource或Writer以支持字段过滤 + // 具体实现需要根据Linkis Storage层的架构决定 +} +``` + +##### (2) Pipeline导出功能实现位置 + +**核心修改**: `PipelineEngineConnExecutor.scala` + +**关键变更**: 扩展正则表达式以支持`without`子句 + +```scala +// 原有正则（仅支持 from ... to ...） +val regex = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s?".r + +// 🆕 新正则（支持 from ... to ... without "..."） +val regexWithMask = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s+without\\s+\"([^\"]+)\"\\s*".r +val regexNormal = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s*".r +``` + +**执行逻辑修改**: + +```scala +// PipelineEngineConnExecutor.scala (行69-89 修改) +override def executeLine( + engineExecutorContext: EngineExecutionContext, + code: String +): ExecuteResponse = { + + thread = Thread.currentThread() + + // 🔍 优先匹配带without子句的语法 + code match { + // 情况1: 包含屏蔽字段 + case regexWithMask(sourcePath, destPath, maskedFields) => + logger.info(s"Pipeline execution with masked fields: $maskedFields") + + // 将屏蔽字段传递给执行器 + val enhancedOptions = new util.HashMap[String, String](newOptions) + enhancedOptions.put("pipeline.masked.field.names", maskedFields) + + PipelineExecutorSelector + .select(sourcePath, destPath, enhancedOptions) + .execute(sourcePath, destPath, engineExecutorContext) + + // 情况2: 不包含屏蔽字段（保持原有行为） + case regexNormal(sourcePath, destPath) => + logger.info(s"Pipeline execution without masking") + + PipelineExecutorSelector + .select(sourcePath, destPath, newOptions) + .execute(sourcePath, destPath, engineExecutorContext) + + // 情况3: 语法错误 + case _ => + throw new PipeLineErrorException( + ILLEGAL_OUT_SCRIPT.getErrorCode, + ILLEGAL_OUT_SCRIPT.getErrorDesc + ". Syntax: from to [without \"fields\"]" + ) + } +} +``` + +**正则表达式详解**: + +```scala +// 正则结构分析 +val regexWithMask = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s+without\\s+\"([^\"]+)\"\\s*".r + +// 组成部分: +// (?i) - 大小写不敏感 +// \\s* - 可选的前导空格 +// from - 关键字 "from" +// \\s+ - 必需的空格 +// (\\S+) - 第1组: 源路径（非空白字符） +// \\s+ - 必需的空格 +// to - 关键字 "to" +// \\s+ - 必需的空格 +// (\\S+) - 第2组: 目标路径（非空白字符） +// \\s+ - 必需的空格 +// without - 关键字 "without" +// \\s+ - 必需的空格 +// \" - 左双引号 +// ([^\"]+) - 第3组: 屏蔽字段列表（除双引号外的任意字符） +// \" - 右双引号 +// \\s* - 可选的尾随空格 +``` + +**测试用例**: + +```scala +// 测试1: 标准语法 +"from /a/b.dolphin to /c/d.csv without \"password,apikey\"" +// 匹配结果: +// sourcePath = "/a/b.dolphin" +// destPath = "/c/d.csv" +// maskedFields = "password,apikey" + +// 测试2: 大小写不敏感 +"FROM /a/b.dolphin TO /c/d.csv WITHOUT \"password\"" +// 匹配成功 + +// 测试3: 字段名包含空格 +"from /a/b.dolphin to /c/d.csv without \"user password, api key\"" +// 匹配结果: +// maskedFields = "user password, api key" + +// 测试4: 兼容原语法 +"from /a/b.dolphin to /c/d.csv" +// 匹配 regexNormal，maskedFields为空 + +// 测试5: 语法错误（缺少引号） +"from /a/b.dolphin to /c/d.csv without password" +// 不匹配任何正则，抛出异常 +``` + +--- + +**修改文件**: `linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/` + +**修改文件清单**: +1. ✅ `PipelineEngineConnExecutor.scala` - 正则解析和参数传递 +2. ✅ `CSVExecutor.scala` - 读取options中的屏蔽字段 +3. ✅ `ExcelExecutor.scala` - 读取options中的屏蔽字段 + +**无需修改**: +- ❌ 前端 `resultsExport.vue` - 由其他团队负责 +- ❌ `PipelineEngineConfiguration.scala` - 参数通过options传递，无需新增配置 + +--- + +#### 3.3.3 实现难点分析 + +**挑战1**: `openFile`与下载/导出接口的数据流处理方式不同 + +- `openFile`: 调用`fileSource.collect()`获取完整数据后过滤 +- `下载/导出接口`: 调用`fileSource.write(fsWriter)`直接流式写入输出流 + +**挑战2**: Pipeline执行器基于Scala实现,需要在Scala代码中实现字段过滤 + +**挑战3**: 需要在多个层级传递屏蔽字段参数 + +``` +前端 resultsExport.vue + ↓ startupMap参数 +Entrance (任务提交) + ↓ JobRequest.params +PipelineEngineConnExecutor + ↓ options传递 +CSVExecutor / ExcelExecutor + ↓ 应用屏蔽逻辑 +FileSource / Writer +``` + +**解决方案选型**: + +##### 已确定方案：方案A - 在执行器中收集数据后过滤 + +**技术选型理由**: +- ✅ 实现简单，可快速交付 +- ✅ 复用FsRestfulApi中已有的字段过滤逻辑 +- ✅ 不需要修改Storage层，风险可控 +- ✅ 适合当前MVP需求 + +**技术限制**: +- ⚠️ 对大结果集（>10万行）有内存压力 +- ⚠️ 性能相对流式方案较低 +- 💡 可通过结果集大小限制规避风险 + +**实现步骤**: + +1. **PipelineEngineConnExecutor修改** (正则解析) + +已在上述"(2) Pipeline导出功能实现位置"中详细说明。 + +2. **CSVExecutor实现** (`CSVExecutor.scala`) + +```scala +override def execute(sourcePath: String, destPath: String, context: EngineExecutionContext): ExecuteResponse = { + // 1. 🆕 从options获取屏蔽字段参数（由PipelineEngineConnExecutor传入） + val maskedFieldNames = options.getOrDefault("pipeline.masked.field.names", "") + + // 2. 验证源文件 + if (!sourcePath.contains(STORAGE_RS_FILE_SUFFIX.getValue)) { + throw new PipeLineErrorException(EXPROTING_MULTIPLE.getErrorCode, EXPROTING_MULTIPLE.getErrorDesc) + } + if (!FileSource.isResultSet(sourcePath)) { + throw new PipeLineErrorException(NOT_A_RESULT_SET_FILE.getErrorCode, NOT_A_RESULT_SET_FILE.getErrorDesc) + } + + // 3. 创建文件系统 + val sourceFsPath = new FsPath(sourcePath) + val destFsPath = new FsPath(destPath) + val sourceFs = FSFactory.getFs(sourceFsPath) + sourceFs.init(null) + val destFs = FSFactory.getFs(destFsPath) + destFs.init(null) + + // 4. 创建FileSource + val fileSource = FileSource.create(sourceFsPath, sourceFs) + if (!FileSource.isTableResultSet(fileSource)) { + throw new PipeLineErrorException(NOT_A_TABLE_RESULT_SET.getErrorCode, NOT_A_TABLE_RESULT_SET.getErrorDesc) + } + + // 5. 获取配置参数 + var nullValue = options.getOrDefault(PIPELINE_OUTPUT_SHUFFLE_NULL_TYPE, "NULL") + if (BLANK.equalsIgnoreCase(nullValue)) nullValue = "" + + // 6. 创建输出流和Writer + val outputStream = destFs.write(destFsPath, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH.getValue(options)) + OutputStreamCache.osCache.put(engineExecutionContext.getJobId.get, outputStream) + + val csvWriter = CSVFsWriter.getCSVFSWriter( + PIPELINE_OUTPUT_CHARSET_STR.getValue(options), + PIPELINE_FIELD_SPLIT_STR.getValue(options), + PIPELINE_FIELD_QUOTE_RETOUCH_ENABLE.getValue(options), + outputStream + ) + + // 7. 🔑 方案A核心逻辑：判断是否需要屏蔽字段 + try { + if (StringUtils.isNotBlank(maskedFieldNames)) { + logger.info(s"Applying field masking: $maskedFieldNames") + + // 7.1 解析屏蔽字段列表 + val maskedFields = maskedFieldNames.toLowerCase().split(",").map(_.trim).toSet + + // 7.2 收集完整数据 + val collectedData = fileSource.collect() + + // 7.3 过滤字段并写入 + filterAndWriteData(collectedData, maskedFields, csvWriter, nullValue) + + } else { + // 原有流式写入逻辑（无屏蔽） + logger.info("No field masking, using stream write") + fileSource.addParams("nullValue", nullValue).write(csvWriter) + } + } finally { + // 8. 资源清理 + IOUtils.closeQuietly(csvWriter) + IOUtils.closeQuietly(fileSource) + IOUtils.closeQuietly(sourceFs) + IOUtils.closeQuietly(destFs) + } + + super.execute(sourcePath, destPath, engineExecutionContext) +} + +// 🆕 字段过滤和写入方法 +private def filterAndWriteData( + collectedData: Array[Pair[Object, ArrayList[String[]]]], + maskedFields: Set[String], + csvWriter: CSVFsWriter, + nullValue: String +): Unit = { + + collectedData.foreach { pair => + // 获取元数据和内容 + val metadata = pair.getFirst.asInstanceOf[Array[util.Map[String, Any]]] + val content = pair.getSecond + + // 计算需要保留的列索引 + val retainedIndices = metadata.zipWithIndex + .filter { case (col, _) => + val columnName = col.get("columnName").toString.toLowerCase() + !maskedFields.contains(columnName) // 不在屏蔽列表中的字段 + } + .map(_._2) + .toList + + logger.info(s"Retained columns: ${retainedIndices.size}/${metadata.length}") + + // 过滤元数据 + val filteredMetadata = retainedIndices.map(i => metadata(i)).toArray + val tableMetaData = new TableMetaData( + filteredMetadata.map { col => + Column( + col.get("columnName").toString, + DataType.toDataType(col.get("dataType").toString), + col.get("comment").toString + ) + } + ) + + // 写入元数据 + csvWriter.addMetaData(tableMetaData) + + // 过滤并写入内容 + content.forEach { row => + val filteredRow = retainedIndices.map { i => + if (i < row.length) { + val value = row(i) + // 处理NULL值 + if (value == null || value.equals("NULL")) nullValue else value + } else { + nullValue + } + }.toArray + + csvWriter.addRecord(new TableRecord(filteredRow)) + } + + csvWriter.flush() + } +} +``` + +3. **ExcelExecutor实现** (`ExcelExecutor.scala`) + - 实现逻辑与CSVExecutor类似 + - 区别在于使用`ExcelFsWriter`替代`CSVFsWriter` + - 支持单结果集和多结果集模式 + +4. **无需修改的部分** + - ❌ `PipelineEngineConfiguration.scala` - 不需要新增配置项 + - ❌ `resultsExport.vue` - 由其他团队负责代码生成 + +--- + +**其他备选方案** (供未来优化参考): + +
+方案B: 扩展Writer实现 (性能优化方案) + +- 创建`MaskedFieldsCSVFsWriter`和`MaskedFieldsExcelFsWriter` +- 在Writer内部实现流式字段过滤 +- 优点: 内存友好，性能优越 +- 缺点: 需修改Storage层，开发周期长 + +
+ +
+方案C: FileSource原生支持 (终极方案) + +- 在`FileSource`中添加`excludeColumns()`方法 +- 架构层面的解决方案，对上层透明 +- 优点: 最优雅，所有场景受益 +- 缺点: 影响范围大，需深度测试 + +
+ +
+方案D: Decorator模式 (方案B的优化版) + +- 不修改现有Writer，遵循开闭原则 +- 使用装饰器包装Writer实现字段过滤 +- 优点: 灵活，可组合 +- 缺点: 增加代码复杂度 + +
+ +**实施建议**: 先实现方案A快速上线，后续根据性能监控考虑升级到方案D + +--- + +### 3.4 配置项说明 + +#### 现有配置项 + +| 配置项 | 默认值 | 说明 | +|-------|-------|------| +| wds.linkis.workspace.resultset.download.is.limit | true | 是否限制下载大小 | +| wds.linkis.workspace.resultset.download.maxsize.csv | 5000 | CSV下载最大行数 | +| wds.linkis.workspace.resultset.download.maxsize.excel | 5000 | Excel下载最大行数 | + +#### 新增配置项（方案A专用） + +| 配置项 | 默认值 | 说明 | 重要性 | +|-------|-------|------|--------| +| **pipeline.masked.field.names** | "" | Pipeline导出时屏蔽的字段名列表（逗号分隔） | 核心功能 | +| **pipeline.export.max.rows** | 100000 | Pipeline导出时允许的最大行数（启用屏蔽时） | ⚠️ 内存保护 | +| **pipeline.export.memory.check.enabled** | true | 是否启用内存检查 | ⚠️ 风险控制 | +| **pipeline.export.memory.threshold** | 0.8 | 内存使用阈值（占总内存比例） | ⚠️ 风险控制 | + +#### 配置建议 + +**生产环境推荐配置**: + +```properties +# 启用结果集大小限制（方案A必需） +pipeline.export.max.rows=50000 + +# 启用内存检查 +pipeline.export.memory.check.enabled=true +pipeline.export.memory.threshold=0.75 + +# 下载功能限制（保持现有） +wds.linkis.workspace.resultset.download.is.limit=true +wds.linkis.workspace.resultset.download.maxsize.csv=5000 +wds.linkis.workspace.resultset.download.maxsize.excel=5000 +``` + +**开发/测试环境配置**: + +```properties +# 可适当放宽限制 +pipeline.export.max.rows=100000 +pipeline.export.memory.check.enabled=false +``` + +**内存充足环境配置**: + +```properties +# 服务器内存>=32GB时可考虑 +pipeline.export.max.rows=500000 +pipeline.export.memory.threshold=0.85 +``` + +--- + +## 4. 风险评估 + +### 4.1 技术风险 + +| 风险 | 等级 | 应对措施 | 备注 | +|-----|------|---------|------| +| **方案A内存溢出风险** | 中 | 1. 配置结果集导出行数上限（建议10万行）
2. 添加内存监控和告警
3. 大结果集提示用户分批导出 | 方案A主要风险 | +| Storage层兼容性问题 | 低 | 充分的兼容性测试，确保Scala/Java互操作正常 | - | +| 性能下降 | 低 | 1. 仅在指定屏蔽字段时启用过滤逻辑
2. 不影响未启用屏蔽的导出性能 | - | +| Pipeline参数传递失败 | 低 | 1. 参数传递链路日志记录
2. 异常情况降级为不屏蔽 | - | +| 字段过滤逻辑错误 | 中 | 1. 完整的单元测试覆盖
2. 与openFile功能对比测试 | 需充分测试 | + +### 4.2 业务风险 + +| 风险 | 等级 | 应对措施 | 备注 | +|-----|------|---------|------| +| 向后兼容性问题 | 低 | 新增可选参数，不传参数时保持原有行为 | - | +| 误屏蔽正常字段 | 低 | 1. 明确文档说明字段名匹配规则
2. 前端提供字段名预览和校验 | - | +| 用户体验影响 | 低 | 1. 前端提供友好的配置界面
2. 屏蔽字段输入支持自动补全 | 可选优化 | +| 大结果集导出超时 | 中 | 1. 方案A会增加导出时间
2. 建议限制行数上限
3. 显示导出进度提示 | 方案A特有 | + +### 4.3 方案A的特殊风险控制 + +#### 风险1：内存溢出 + +**触发条件**: +- 结果集行数 > 10万行 +- 结果集列数 > 1000列 +- 并发导出任务过多 + +**监控指标**: +```scala +// 添加内存使用监控 +val runtime = Runtime.getRuntime +val usedMemory = runtime.totalMemory() - runtime.freeMemory() +if (usedMemory > MAX_MEMORY_THRESHOLD) { + logger.warn(s"Memory usage high: $usedMemory bytes") + throw new PipeLineErrorException("Memory limit exceeded") +} +``` + +**应对措施**: +1. **配置层控制**: 新增配置项限制导出行数 + ```scala + val PIPELINE_EXPORT_MAX_ROWS = CommonVars("pipeline.export.max.rows", 100000) + ``` + +2. **运行时检查**: 在collect()前检查结果集大小 + ```scala + val totalLine = fileSource.getTotalLine + if (totalLine > PIPELINE_EXPORT_MAX_ROWS.getValue) { + throw new PipeLineErrorException( + s"Result set too large: $totalLine rows, max allowed: ${PIPELINE_EXPORT_MAX_ROWS.getValue}" + ) + } + ``` + +3. **用户提示**: 前端显示结果集大小，超过阈值时警告 + +#### 风险2：性能下降 + +**影响评估**: +- 原流式写入: 无需加载全部数据到内存 +- 方案A: 需先collect()全部数据，再过滤，再写入 +- **预估性能损失**: 30-50% (取决于结果集大小) + +**缓解措施**: +1. 仅在指定屏蔽字段时启用collect模式 +2. 未指定屏蔽字段时保持原流式写入 +3. 添加性能日志，监控导出耗时 + +--- + +## 5. 变更历史 + +| 版本 | 日期 | 变更内容 | 作者 | +|-----|------|---------|------| +| v1.0 | 2025-10-27 | 初始版本 - 完成需求分析和技术方案设计 | Claude Code | + +--- + +**文档结束** + diff --git "a/docs/1.18.0/design/Spark\344\273\273\345\212\241\350\266\205\346\227\266\350\257\212\346\226\255\346\226\260\345\242\236_\350\256\276\350\256\241.md" "b/docs/1.18.0/design/Spark\344\273\273\345\212\241\350\266\205\346\227\266\350\257\212\346\226\255\346\226\260\345\242\236_\350\256\276\350\256\241.md" new file mode 100644 index 00000000000..6333d63a29d --- /dev/null +++ "b/docs/1.18.0/design/Spark\344\273\273\345\212\241\350\266\205\346\227\266\350\257\212\346\226\255\346\226\260\345\242\236_\350\256\276\350\256\241.md" @@ -0,0 +1,364 @@ +# 技术设计方案 + +## 1. 文档基本信息 + +| 项目 | 内容 | +|------|-----------------| +| 设计名称 | Spark任务诊断结果更新接口 | +| 需求类型 | 新增功能 | +| 设计日期 | 2025-12-25 | +| 状态 | 已完成 | +| 编写人 | claude-code | + +## 2. 设计背景与目标 + +### 2.1 设计背景 +在Linkis系统中，当Spark任务运行超时后，会触发诊断逻辑，调用doctoris诊断系统获取诊断结果。为了方便用户查看和分析诊断结果，需要将诊断信息持久化到数据库中，并提供相应的查询接口。 + +### 2.2 设计目标 +- 实现诊断结果的持久化存储 +- 提供高效的诊断结果更新接口 +- 确保系统的高可用性和可靠性 +- 支持后续功能扩展 + +## 3. 架构设计 + +### 3.1 系统架构图 + +```mermaid +flowchart TD + A[EntranceServer] -->|1. 检测超时任务| A + A -->|2. 调用诊断API| B[Doctoris诊断系统] + B -->|3. 返回诊断结果| A + A -->|4. 调用RPC接口| C[JobHistory服务] + C -->|5. 查询诊断记录| D[数据库] + D -->|6. 返回查询结果| C + C -->|7. 创建/更新诊断记录| D + D -->|8. 返回操作结果| C + C -->|9. 返回更新结果| A +``` + +### 3.2 核心组件 + +| 组件 | 职责 | +|------|------| +| EntranceServer | 检测超时任务，调用诊断API，触发诊断结果更新 | +| JobHistory服务 | 提供诊断结果更新接口，处理诊断记录的创建和更新 | +| 数据库 | 存储诊断记录，提供数据持久化支持 | +| Doctoris诊断系统 | 提供任务诊断服务，返回诊断结果 | + +## 4. 详细设计 + +### 4.1 数据模型设计 + +#### 4.1.1 诊断记录表（linkis_ps_job_history_diagnosis） + +| 字段名 | 数据类型 | 约束 | 描述 | +|--------|----------|------|------| +| id | BIGINT | PRIMARY KEY, AUTO_INCREMENT | 主键ID | +| job_history_id | BIGINT | NOT NULL | 任务历史ID | +| diagnosis_content | TEXT | NOT NULL | 诊断内容 | +| created_time | DATETIME | NOT NULL | 创建时间 | +| updated_time | DATETIME | NOT NULL | 更新时间 | +| only_read | VARCHAR(1) | DEFAULT '0' | 是否只读 | +| diagnosis_source | VARCHAR(50) | NOT NULL | 诊断来源 | + +#### 4.1.2 索引设计 + +| 索引名 | 索引类型 | 索引字段 | 用途 | +|--------|----------|----------|------| +| idx_job_history_id | UNIQUE | job_history_id, diagnosis_source | 唯一约束，确保同一任务同一来源只有一条诊断记录 | +| idx_job_history_id_single | NORMAL | job_history_id | 加速根据任务ID查询诊断记录 | + +### 4.2 类设计 + +#### 4.2.1 JobReqDiagnosisUpdate + +**功能**: 诊断结果更新请求协议类 + +**属性**: + +| 属性名 | 类型 | 描述 | +|--------|------|------| +| jobHistoryId | Long | 任务历史ID | +| diagnosisContent | String | 诊断内容 | +| diagnosisSource | String | 诊断来源 | + +**方法**: + +| 方法名 | 参数 | 返回值 | 描述 | +|--------|------|--------|------| +| apply | jobHistoryId: Long, diagnosisContent: String, diagnosisSource: String | JobReqDiagnosisUpdate | 工厂方法，用于创建JobReqDiagnosisUpdate实例 | + +#### 4.2.2 JobHistoryQueryServiceImpl + +**功能**: JobHistory服务实现类，处理诊断结果更新请求 + +**核心方法**: + +| 方法名 | 参数 | 返回值 | 描述 | +|--------|------|--------|------| +| updateDiagnosis | jobReqDiagnosisUpdate: JobReqDiagnosisUpdate | JobRespProtocol | 处理诊断结果更新请求，创建或更新诊断记录 | + +**依赖注入**: + +| 依赖项 | 类型 | 用途 | +|--------|------|------| +| jobHistoryDiagnosisService | JobHistoryDiagnosisService | 诊断记录服务，用于操作数据库 | + +### 4.3 接口设计 + +#### 4.3.1 RPC接口 + +**接口名称**: updateDiagnosis + +**请求参数**: + +| 参数名 | 类型 | 描述 | +|--------|------|------| +| jobHistoryId | Long | 任务历史ID | +| diagnosisContent | String | 诊断内容 | +| diagnosisSource | String | 诊断来源 | + +**返回结果**: + +| 字段名 | 类型 | 描述 | +|--------|------|------| +| status | Int | 状态码，0: 成功, 非0: 失败 | +| msg | String | 响应消息 | + +#### 4.3.2 内部服务接口 + +**JobHistoryDiagnosisService.selectByJobId** + +| 参数名 | 类型 | 描述 | +|--------|------|------| +| jobId | Long | 任务ID | +| diagnosisSource | String | 诊断来源 | + +| 返回值 | 类型 | 描述 | +|--------|------|------| +| 诊断记录 | JobDiagnosis | 诊断记录对象，不存在则返回null | + +**JobHistoryDiagnosisService.insert** + +| 参数名 | 类型 | 描述 | +|--------|------|------| +| jobDiagnosis | JobDiagnosis | 诊断记录对象 | + +**JobHistoryDiagnosisService.update** + +| 参数名 | 类型 | 描述 | +|--------|------|------| +| jobDiagnosis | JobDiagnosis | 诊断记录对象 | + +## 5. 实现细节 + +### 5.1 诊断结果更新流程 + +```java +// 1. 接收RPC请求 +@Receiver +def updateDiagnosis(jobReqDiagnosisUpdate: JobReqDiagnosisUpdate): JobRespProtocol = { + // 2. 日志记录 + logger.info(s"Update job diagnosis: ${jobReqDiagnosisUpdate.toString}") + + // 3. 构造响应对象 + val jobResp = new JobRespProtocol + + // 4. 异常处理 + Utils.tryCatch { + // 5. 查询诊断记录 + var jobDiagnosis = jobHistoryDiagnosisService.selectByJobId( + jobReqDiagnosisUpdate.getJobHistoryId, + jobReqDiagnosisUpdate.getDiagnosisSource + ) + + // 6. 创建或更新诊断记录 + if (jobDiagnosis == null) { + // 创建新记录 + jobDiagnosis = new JobDiagnosis + jobDiagnosis.setJobHistoryId(jobReqDiagnosisUpdate.getJobHistoryId) + jobDiagnosis.setCreatedTime(new Date) + } + + // 更新诊断内容和来源 + jobDiagnosis.setDiagnosisContent(jobReqDiagnosisUpdate.getDiagnosisContent) + jobDiagnosis.setDiagnosisSource(jobReqDiagnosisUpdate.getDiagnosisSource) + jobDiagnosis.setUpdatedDate(new Date) + + // 7. 保存诊断记录 + if (jobDiagnosis.getId == null) { + jobHistoryDiagnosisService.insert(jobDiagnosis) + } else { + jobHistoryDiagnosisService.update(jobDiagnosis) + } + + // 8. 设置成功响应 + jobResp.setStatus(0) + jobResp.setMsg("Update diagnosis success") + } { case exception: Exception => + // 9. 处理异常情况 + logger.error( + s"Failed to update job diagnosis ${jobReqDiagnosisUpdate.toString}, should be retry", + exception + ) + jobResp.setStatus(2) + jobResp.setMsg(ExceptionUtils.getRootCauseMessage(exception)) + } + + // 10. 返回响应结果 + jobResp +} +``` + +### 5.2 诊断结果触发流程 + +```scala +// 1. 检测到超时任务后，调用诊断API +val response = EntranceUtils.taskRealtimeDiagnose(entranceJob.getJobRequest, null) +logger.info(s"Finished to diagnose spark job ${job.getId()}, result: ${response.result}, reason: ${response.reason}") + +// 2. 如果诊断成功，调用更新接口 +if (response.success) { + // 3. 构造诊断更新请求 + val diagnosisUpdate = JobReqDiagnosisUpdate( + job.getId().toLong, + response.result, + "doctoris" + ) + + // 4. 发送RPC请求到jobhistory服务 + val sender = Sender.getSender("jobhistory") + sender.ask(diagnosisUpdate) + logger.info(s"Successfully updated diagnosis for job ${job.getId()}") +} +``` + +## 6. 配置设计 + +| 配置项 | 默认值 | 描述 | 所属模块 | +|--------|--------|------|----------| +| linkis.task.diagnosis.enable | true | 任务诊断开关 | entrance | +| linkis.task.diagnosis.engine.type | spark | 任务诊断引擎类型 | entrance | +| linkis.task.diagnosis.timeout | 300000 | 任务诊断超时时间（毫秒） | entrance | +| linkis.doctor.url | 无 | Doctoris诊断系统URL | entrance | +| linkis.doctor.signature.token | 无 | Doctoris签名令牌 | entrance | + +## 7. 错误处理设计 + +### 7.1 错误码设计 + +| 错误码 | 错误描述 | 处理方式 | +|--------|----------|----------| +| 0 | 成功 | 正常返回 | +| 2 | 内部错误 | 记录日志，返回错误信息 | +| 1001 | 参数无效 | 检查参数，返回错误信息 | +| 1002 | 数据库异常 | 记录日志，返回错误信息 | + +### 7.2 异常处理机制 + +1. **接口层异常处理**：在updateDiagnosis方法中，使用try-catch捕获所有异常，确保接口不会因异常而崩溃 +2. **数据库层异常处理**：使用Spring的事务管理，确保数据库操作的原子性和一致性 +3. **调用方异常处理**：EntranceServer在调用updateDiagnosis接口时，捕获RPC异常，记录日志但不影响主流程 + +## 8. 性能优化设计 + +### 8.1 数据库优化 +- 添加唯一索引，加速查询和避免重复数据 +- 使用连接池管理数据库连接，减少连接创建和销毁开销 +- 优化SQL语句，减少数据库负载 + +### 8.2 接口优化 +- 采用异步处理方式，避免阻塞主流程 +- 合理设置超时时间，避免长时间等待 +- 实现接口限流，防止高并发调用导致系统崩溃 + +### 8.3 代码优化 +- 减少对象创建，使用对象池或复用对象 +- 优化算法，提高代码执行效率 +- 减少网络开销，合理设计接口参数 + +## 9. 测试设计 + +### 9.1 单元测试 + +| 测试用例 | 测试场景 | 预期结果 | +|----------|----------|----------| +| updateDiagnosis_normal | 正常更新诊断记录 | 返回成功状态码，诊断记录被更新 | +| updateDiagnosis_new | 创建新的诊断记录 | 返回成功状态码，诊断记录被创建 | +| updateDiagnosis_invalid_param | 无效参数调用 | 返回错误状态码，错误信息正确 | +| updateDiagnosis_db_exception | 数据库异常 | 返回错误状态码，错误信息正确 | + +### 9.2 集成测试 + +| 测试用例 | 测试场景 | 预期结果 | +|----------|----------|----------| +| entrance_diagnosis_flow | 完整的诊断流程 | 诊断记录被正确创建和更新 | +| concurrent_update | 并发调用更新接口 | 诊断记录被正确更新，无数据冲突 | +| long_running_test | 长时间运行测试 | 系统稳定运行，无内存泄漏 | + +## 10. 部署与运维设计 + +### 10.1 部署方式 +- 与现有Linkis系统一同部署 +- 无需额外的硬件资源 +- 支持集群部署，提高系统可用性 + +### 10.2 监控与告警 +- 监控接口调用频率和响应时间 +- 监控数据库连接池状态 +- 设置告警阈值，当接口响应时间超过阈值或出现异常时触发告警 + +### 10.3 日志管理 +- 记录接口调用日志，包括请求参数、响应结果和耗时 +- 记录数据库操作日志，便于问题排查 +- 采用分级日志，便于日志分析和管理 + +## 11. 后续扩展设计 + +### 11.1 功能扩展 +- 支持多种诊断来源 +- 添加诊断结果查询接口 +- 实现诊断结果可视化 +- 添加诊断结果告警机制 + +### 11.2 性能扩展 +- 支持分布式部署，提高系统吞吐量 +- 实现缓存机制，减少数据库访问次数 +- 采用消息队列，异步处理诊断结果更新 + +## 12. 风险评估与应对 + +| 风险点 | 影响程度 | 可能性 | 应对措施 | +|--------|----------|--------|----------| +| 数据库连接异常 | 中 | 低 | 使用连接池，设置合理的超时时间和重试机制 | +| 高并发调用 | 中 | 中 | 实现接口限流，优化数据库查询，添加缓存 | +| 诊断信息过大 | 低 | 低 | 使用TEXT类型存储，支持大文本 | +| 接口调用失败 | 低 | 中 | 记录日志，不影响主流程，提供重试机制 | + +## 13. 附录 + +### 13.1 术语定义 + +| 术语 | 解释 | +|------|------| +| Linkis | 基于Apache Linkis开发的大数据计算中间件 | +| Doctoris | 任务诊断系统，用于分析任务运行问题 | +| RPC | 远程过程调用，用于系统间通信 | +| JobHistory | 任务历史服务，用于存储和查询任务历史信息 | +| EntranceServer | 入口服务，负责接收和处理任务请求 | + +### 13.2 参考文档 + +- [Apache Linkis官方文档](https://linkis.apache.org/) +- [MyBatis官方文档](https://mybatis.org/mybatis-3/zh/index.html) +- [Spring Boot官方文档](https://spring.io/projects/spring-boot) + +### 13.3 相关代码文件 + +| 文件名 | 路径 | 功能 | +|--------|------|------| +| JobReqDiagnosisUpdate.scala | linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/job/ | 诊断结果更新请求协议类 | +| JobHistoryQueryServiceImpl.scala | linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/ | JobHistory服务实现类，包含updateDiagnosis方法 | +| EntranceServer.scala | linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/ | Entrance服务，包含诊断触发和更新逻辑 | \ No newline at end of file diff --git "a/docs/1.18.0/design/\346\227\245\345\277\227\346\224\257\346\214\201\347\273\206\347\262\222\345\272\246\350\277\224\345\233\236\346\224\271\351\200\240_\350\256\276\350\256\241.md" "b/docs/1.18.0/design/\346\227\245\345\277\227\346\224\257\346\214\201\347\273\206\347\262\222\345\272\246\350\277\224\345\233\236\346\224\271\351\200\240_\350\256\276\350\256\241.md" new file mode 100644 index 00000000000..a1ba5cecc62 --- /dev/null +++ "b/docs/1.18.0/design/\346\227\245\345\277\227\346\224\257\346\214\201\347\273\206\347\262\222\345\272\246\350\277\224\345\233\236\346\224\271\351\200\240_\350\256\276\350\256\241.md" @@ -0,0 +1,130 @@ +# 阶段2：设计方案文档 + +## 1. 总述 + +### 1.1 需求与目标 + +**项目背景**：在大模型分析场景中，当前获取用户任务日志接口会返回所有（info、error、warn）任务日志，导致大模型处理文件数量过多。为了优化大模型处理效率，需要对 filesystem 模块的 openLog 接口进行增强，支持根据指定的日志级别返回对应的日志内容。 + +**设计目标**： +1. 实现 openLog 接口的日志级别过滤功能 +2. 支持 all、info、error、warn 四种日志级别 +3. 保持向后兼容性，缺省情况下返回全部日志 +4. 确保实现的正确性、性能和可靠性 + +## 2. 技术架构 + +**技术栈**： +- 开发语言：Java (服务端), Scala (客户端SDK) +- 框架：Spring Boot +- 存储：文件系统 + +**部署架构**： +与现有 filesystem 模块部署架构一致，无需额外部署组件。 + +## 3. 核心概念/对象 + +| 概念/对象 | 描述 | +|-----------|------| +| LogLevel | 日志级别枚举类，定义了 ERROR、WARN、INFO、ALL 四种级别 | +| FsRestfulApi | filesystem 模块的 RESTful 接口实现类 | +| OpenLogAction | 客户端 SDK 中调用 openLog 接口的 Action 类 | +| filterLogByLevel | 新增的日志过滤方法 | + +## 4. 处理逻辑设计 + +### 4.1 接口参数变更 + +**原接口签名**： +```java +public Message openLog( + HttpServletRequest req, + @RequestParam(value = "path", required = false) String path, + @RequestParam(value = "proxyUser", required = false) String proxyUser) +``` + +**新接口签名**： +```java +public Message openLog( + HttpServletRequest req, + @RequestParam(value = "path", required = false) String path, + @RequestParam(value = "proxyUser", required = false) String proxyUser, + @RequestParam(value = "logLevel", required = false, defaultValue = "all") String logLevel) +``` + +### 4.2 日志过滤逻辑 + +``` +输入: log[4] 数组, logLevel 参数 +| +v +logLevel 为空或 "all"? --> 是 --> 返回原始 log[4] +| +v (否) +根据 logLevel 创建新数组 filteredResult[4]，初始化为空字符串 +| +v +switch(logLevel.toLowerCase()): + case "error": filteredResult[0] = log[0] + case "warn": filteredResult[1] = log[1] + case "info": filteredResult[2] = log[2] + default: 返回原始 log[4] (向后兼容) +| +v +返回 filteredResult[4] +``` + +### 4.3 数据结构 + +日志数组索引与日志级别对应关系： + +| 索引 | 日志级别 | LogLevel.Type | +|------|----------|---------------| +| 0 | ERROR | LogLevel.Type.ERROR | +| 1 | WARN | LogLevel.Type.WARN | +| 2 | INFO | LogLevel.Type.INFO | +| 3 | ALL | LogLevel.Type.ALL | + +## 5. 代码变更清单 + +### 5.1 FsRestfulApi.java + +**文件路径**: `linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/FsRestfulApi.java` + +**变更内容**: +1. `openLog` 方法添加 `logLevel` 参数 +2. 添加 Swagger API 文档注解 +3. 新增 `filterLogByLevel()` 私有方法 + +### 5.2 OpenLogAction.scala + +**文件路径**: `linkis-computation-governance/linkis-client/linkis-computation-client/src/main/scala/org/apache/linkis/ujes/client/request/OpenLogAction.scala` + +**变更内容**: +1. Builder 类添加 `logLevel` 属性（默认值 "all"） +2. 添加 `setLogLevel()` 方法 +3. `build()` 方法中添加 logLevel 参数设置 + +## 6. 非功能性设计 + +### 6.1 安全 + +- **权限控制**：确保用户只能访问自己有权限的日志文件（复用现有逻辑） +- **参数校验**：对请求参数进行合理处理，无效参数不抛异常 + +### 6.2 性能 + +- 日志级别过滤对接口响应时间的影响可忽略不计（< 1ms） +- 过滤逻辑在内存中完成，无额外 I/O 操作 + +### 6.3 向后兼容 + +- 缺省情况下返回全部日志，与原有行为一致 +- 无效 logLevel 参数返回全部日志，确保服务不中断 +- 现有调用方无需修改代码即可继续使用 + +## 7. 变更历史 + +| 版本 | 日期 | 变更人 | 变更内容 | +|-----|------|--------|----------| +| 1.0 | 2025-12-26 | AI Assistant | 初始版本 | diff --git "a/docs/1.18.0/design/\347\263\273\347\273\237\347\224\250\346\210\267\347\246\201\346\255\242\347\231\273\345\275\225\346\224\271\351\200\240_\350\256\276\350\256\241.md" "b/docs/1.18.0/design/\347\263\273\347\273\237\347\224\250\346\210\267\347\246\201\346\255\242\347\231\273\345\275\225\346\224\271\351\200\240_\350\256\276\350\256\241.md" new file mode 100644 index 00000000000..6215295c417 --- /dev/null +++ "b/docs/1.18.0/design/\347\263\273\347\273\237\347\224\250\346\210\267\347\246\201\346\255\242\347\231\273\345\275\225\346\224\271\351\200\240_\350\256\276\350\256\241.md" @@ -0,0 +1,196 @@ +# 阶段2：技术设计方案 + +## 1. 设计概述 + +### 1.1 设计目标 +在现有登录拦截逻辑基础上进行增强，将登录来源判断方式从 request body 的 `source` 字段改为 HTTP Header 的 `webLogin` 字段。 + +### 1.2 设计原则 +- **最小改动**: 复用现有拦截逻辑，仅修改来源判断方式 +- **向后兼容**: 默认功能关闭，不影响现有系统 +- **可配置性**: 支持配置开关和系统用户前缀列表 + +## 2. 架构设计 + +### 2.1 组件关系图 + +``` +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ Web Frontend │────>│ Gateway Server │────>│ Backend API │ +│ │ │ │ │ │ +│ Header: │ │ UserRestful │ │ │ +│ webLogin=true │ │ ↓ │ │ │ +└─────────────────┘ │ tryLogin() │ └─────────────────┘ + │ ↓ │ + │ isWebLogin() │ + │ ↓ │ + │ checkSystemUser │ + └─────────────────┘ +``` + +### 2.2 处理流程 + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ 登录请求处理流程 │ +├─────────────────────────────────────────────────────────────────┤ +│ │ +│ ┌──────────┐ ┌───────────────┐ ┌────────────────────┐ │ +│ │ 接收请求 │───>│ 获取用户名密码 │───>│ 检查功能开关是否开启 │ │ +│ └──────────┘ └───────────────┘ └─────────┬──────────┘ │ +│ │ │ +│ ┌─────────────┴─────────────┐ │ +│ │ 开关状态? │ │ +│ └─────────────┬─────────────┘ │ +│ 关闭 │ │ 开启 │ +│ ▼ ▼ │ +│ ┌─────────────┐ ┌─────────────────┐ │ +│ │ 继续正常登录 │ │ 从Header获取 │ │ +│ └─────────────┘ │ webLogin标识 │ │ +│ └────────┬────────┘ │ +│ │ │ +│ ┌─────────────┴───────────┐ │ +│ │ webLogin == "true"? │ │ +│ └─────────────┬───────────┘ │ +│ false │ │ true │ +│ ▼ ▼ │ +│ ┌─────────────┐ ┌───────────────┐ │ +│ │ 继续正常登录 │ │ 检查用户名前缀 │ │ +│ └─────────────┘ └───────┬───────┘ │ +│ │ │ +│ ┌───────────────┴─────────┐ │ +│ │ 匹配系统用户前缀? │ │ +│ └───────────────┬─────────┘ │ +│ 否 │ │ 是 │ +│ ▼ ▼ │ +│ ┌─────────────┐ ┌─────────────┐ │ +│ │ 继续正常登录 │ │ 返回错误信息 │ │ +│ └─────────────┘ │ 拒绝登录 │ │ +│ └─────────────┘ │ +└─────────────────────────────────────────────────────────────────┘ +``` + +## 3. 详细设计 + +### 3.1 配置项修改 + +**文件**: `GatewayConfiguration.scala` + +| 配置项 | 当前值 | 修改后 | +|--------|--------|--------| +| PROHIBIT_LOGIN_PREFIX | `hduser,shduser` | `hadoop,hduser,shduser` | + +**新增配置项**: 无需新增，复用现有配置 + +### 3.2 代码修改 + +**文件**: `UserRestful.scala` + +#### 3.2.1 新增方法: isWebLogin + +```scala +private val WEB_LOGIN_HEADER = "webLogin" + +private def isWebLogin(gatewayContext: GatewayContext): Boolean = { + val headers = gatewayContext.getRequest.getHeaders + val webLoginValues = headers.get(WEB_LOGIN_HEADER) + if (webLoginValues != null && webLoginValues.nonEmpty) { + "true".equalsIgnoreCase(webLoginValues.head) + } else { + false // 默认为false + } +} +``` + +#### 3.2.2 修改tryLogin方法 + +**现有代码**: +```scala +if ( + GatewayConfiguration.PROHIBIT_LOGIN_SWITCH.getValue && + (!getRequestSource(gatewayContext).equals("client")) +) { + PROHIBIT_LOGIN_PREFIX.split(",").foreach { prefix => + if (userName.toLowerCase().startsWith(prefix)) { + return Message.error("System users are prohibited from logging in（系统用户禁止登录）！") + } + } +} +``` + +**修改后**: +```scala +if ( + GatewayConfiguration.PROHIBIT_LOGIN_SWITCH.getValue && + isWebLogin(gatewayContext) +) { + PROHIBIT_LOGIN_PREFIX.split(",").foreach { prefix => + if (userName.toLowerCase().startsWith(prefix)) { + return Message.error("System users are prohibited from logging in（系统用户禁止登录）！") + } + } +} +``` + +## 4. 接口设计 + +### 4.1 登录接口变更 + +**接口**: POST /api/rest_j/v1/user/login + +**新增Header**: +| Header | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| webLogin | String | 否 | false | Web页面登录标识 | + +**请求示例**: +```http +POST /api/rest_j/v1/user/login HTTP/1.1 +Host: gateway.linkis.com +Content-Type: application/json +webLogin: true + +{ + "userName": "testuser", + "password": "xxx" +} +``` + +**错误响应** (系统用户被拦截): +```json +{ + "method": "/api/rest_j/v1/user/login", + "status": 1, + "message": "System users are prohibited from logging in（系统用户禁止登录）！" +} +``` + +## 5. 前端配合要求 + +前端在Web页面调用登录接口时，需要在HTTP请求header中添加: +```javascript +headers: { + 'webLogin': 'true' +} +``` + +## 6. 配置示例 + +### 6.1 linkis.properties + +```properties +# 开启系统用户禁止登录功能 +linkis.system.user.prohibit.login.switch=true + +# 系统用户前缀列表（逗号分隔） +linkis.system.user.prohibit.login.prefix=hadoop,hduser,shduser +``` + +## 7. 兼容性说明 + +| 场景 | 行为 | +|------|------| +| 旧前端(无webLogin header) | 默认webLogin=false，不拦截，正常登录 | +| 客户端登录(无webLogin header) | 默认webLogin=false，不拦截，正常登录 | +| 新前端(webLogin=true) + 普通用户 | 正常登录 | +| 新前端(webLogin=true) + 系统用户 | 拦截，返回错误 | diff --git "a/docs/1.18.0/requirements/Spark\344\273\273\345\212\241\350\266\205\346\227\266\350\257\212\346\226\255\346\226\260\345\242\236_\351\234\200\346\261\202.md" "b/docs/1.18.0/requirements/Spark\344\273\273\345\212\241\350\266\205\346\227\266\350\257\212\346\226\255\346\226\260\345\242\236_\351\234\200\346\261\202.md" new file mode 100644 index 00000000000..077700b28c0 --- /dev/null +++ "b/docs/1.18.0/requirements/Spark\344\273\273\345\212\241\350\266\205\346\227\266\350\257\212\346\226\255\346\226\260\345\242\236_\351\234\200\346\261\202.md" @@ -0,0 +1,261 @@ +# 需求分析文档 + +## 1. 文档基本信息 + +| 项目 | 内容 | +|------|-----------------| +| 需求名称 | Spark任务诊断结果更新接口 | +| 需求类型 | 新增功能 | +| 分析日期 | 2025-12-25 | +| 状态 | 已完成 | +| 编写人 | claude-code | + +## 2. 需求背景与目标 + +### 2.1 需求背景 +在Linkis系统中，当Spark任务运行时间超过配置的阈值时，会触发任务诊断逻辑，调用doctoris诊断系统获取诊断结果。目前，诊断结果仅存储在日志中，无法持久化存储和查询。为了方便用户查看和分析任务诊断结果，需要将诊断信息持久化到数据库中。 + +### 2.2 需求目标 +- 实现诊断结果的持久化存储 +- 提供诊断结果的查询接口 +- 支持诊断结果的更新操作 +- 确保诊断信息的准确性和完整性 + +## 3. 功能需求分析 + +### 3.1 核心功能 + +| 功能点 | 描述 | 优先级 | +|--------|------|--------| +| 诊断结果更新接口 | 提供RPC接口，用于更新任务诊断结果 | P1 | +| 诊断记录创建 | 当不存在诊断记录时，创建新的诊断记录 | P1 | +| 诊断记录更新 | 当存在诊断记录时，更新现有诊断记录 | P1 | +| 诊断记录查询 | 支持根据任务ID和诊断来源查询诊断记录 | P2 | + +### 3.2 辅助功能 + +| 功能点 | 描述 | 优先级 | +|--------|------|--------| +| 接口异常处理 | 处理接口调用过程中的异常情况 | P1 | +| 日志记录 | 记录接口调用日志，便于问题排查 | P2 | +| 性能监控 | 监控接口响应时间和调用频率 | P3 | + +## 4. 非功能需求分析 + +| 需求类型 | 具体要求 | 优先级 | +|----------|----------|--------| +| 性能需求 | 接口响应时间 < 500ms | P1 | +| 可用性需求 | 接口可用性 ≥ 99.9% | P1 | +| 可靠性需求 | 诊断信息不丢失，确保数据一致性 | P1 | +| 安全性需求 | 接口调用需要进行身份验证 | P2 | +| 扩展性需求 | 支持多种诊断来源，便于后续扩展 | P2 | + +## 5. 业务流程分析 + +### 5.1 诊断结果更新流程 + +```mermaid +sequenceDiagram + participant Entrance as EntranceServer + participant Doctoris as Doctoris诊断系统 + participant JobHistory as JobHistory服务 + participant DB as 数据库 + + Entrance->>Entrance: 检测到超时任务 + Entrance->>Doctoris: 调用诊断API + Doctoris-->>Entrance: 返回诊断结果 + Entrance->>JobHistory: 调用updateDiagnosis接口 + JobHistory->>DB: 查询诊断记录 + alt 记录不存在 + DB-->>JobHistory: 返回null + JobHistory->>DB: 创建诊断记录 + else 记录存在 + DB-->>JobHistory: 返回诊断记录 + JobHistory->>DB: 更新诊断记录 + end + JobHistory-->>Entrance: 返回更新结果 +``` + +### 5.2 诊断记录查询流程 + +```mermaid +sequenceDiagram + participant Client as 客户端 + participant JobHistory as JobHistory服务 + participant DB as 数据库 + + Client->>JobHistory: 调用查询诊断接口 + JobHistory->>DB: 查询诊断记录 + DB-->>JobHistory: 返回诊断记录 + JobHistory-->>Client: 返回诊断结果 +``` + +## 6. 数据模型分析 + +### 6.1 现有数据模型 + +**表名**: linkis_ps_job_history_diagnosis + +| 字段名 | 数据类型 | 描述 | 约束 | +|--------|----------|------|------| +| id | BIGINT | 主键ID | 自增 | +| job_history_id | BIGINT | 任务历史ID | 非空 | +| diagnosis_content | TEXT | 诊断内容 | 非空 | +| created_time | DATETIME | 创建时间 | 非空 | +| updated_time | DATETIME | 更新时间 | 非空 | +| only_read | VARCHAR(1) | 是否只读 | 默认为'0' | +| diagnosis_source | VARCHAR(50) | 诊断来源 | 非空 | + +### 6.2 数据字典 + +| 字段名 | 取值范围 | 描述 | +|--------|----------|------| +| only_read | 0/1 | 0: 可编辑, 1: 只读 | +| diagnosis_source | doctoris/其他 | 诊断系统来源 | + +## 7. 接口设计 + +### 7.1 RPC接口定义 + +#### 7.1.1 JobReqDiagnosisUpdate + +**功能**: 更新任务诊断结果 + +**参数列表**: + +| 参数名 | 类型 | 描述 | 是否必填 | +|--------|------|------|----------| +| jobHistoryId | Long | 任务历史ID | 是 | +| diagnosisContent | String | 诊断内容 | 是 | +| diagnosisSource | String | 诊断来源 | 是 | + +**返回结果**: + +| 字段名 | 类型 | 描述 | +|--------|------|------| +| status | Int | 状态码，0: 成功, 非0: 失败 | +| msg | String | 响应消息 | + +### 7.2 内部接口 + +#### 7.2.1 JobHistoryDiagnosisService.selectByJobId + +**功能**: 根据任务ID和诊断来源查询诊断记录 + +**参数列表**: + +| 参数名 | 类型 | 描述 | 是否必填 | +|--------|------|------|----------| +| jobId | Long | 任务ID | 是 | +| diagnosisSource | String | 诊断来源 | 是 | + +**返回结果**: +- JobDiagnosis对象或null + +#### 7.2.2 JobHistoryDiagnosisService.insert + +**功能**: 创建诊断记录 + +**参数列表**: + +| 参数名 | 类型 | 描述 | 是否必填 | +|--------|------|------|----------| +| jobDiagnosis | JobDiagnosis | 诊断记录对象 | 是 | + +**返回结果**: +- 无 + +#### 7.2.3 JobHistoryDiagnosisService.update + +**功能**: 更新诊断记录 + +**参数列表**: + +| 参数名 | 类型 | 描述 | 是否必填 | +|--------|------|------|----------| +| jobDiagnosis | JobDiagnosis | 诊断记录对象 | 是 | + +**返回结果**: +- 无 + +## 8. 依赖与约束 + +### 8.1 技术依赖 + +| 依赖项 | 版本 | 用途 | +|--------|------|------| +| Linkis RPC | 1.18.0-wds | 提供RPC通信机制 | +| Spring Boot | 2.6.3 | 提供依赖注入和事务管理 | +| MyBatis | 3.5.9 | 数据库访问框架 | +| MySQL | 8.0+ | 数据库存储 | + +### 8.2 业务约束 + +- 诊断结果更新接口只能由EntranceServer调用 +- 诊断记录的jobHistoryId必须存在于linkis_ps_job_history表中 +- diagnosisSource字段目前固定为"doctoris" + +## 9. 风险与应对措施 + +| 风险点 | 影响程度 | 可能性 | 应对措施 | +|--------|----------|--------|----------| +| 诊断结果更新失败 | 低 | 中 | 记录错误日志，不影响主流程 | +| 数据库连接异常 | 中 | 低 | 使用连接池，设置合理的超时时间 | +| 高并发调用 | 中 | 中 | 优化数据库查询，添加索引 | +| 诊断信息过大 | 低 | 低 | 使用TEXT类型存储，支持大文本 | + +## 10. 验收标准 + +### 10.1 功能验收 + +| 验收项 | 验收标准 | +|--------|----------| +| 诊断记录创建 | 当调用更新接口且不存在诊断记录时，成功创建新记录 | +| 诊断记录更新 | 当调用更新接口且存在诊断记录时，成功更新现有记录 | +| 接口响应时间 | 接口响应时间 < 500ms | +| 幂等性 | 多次调用同一任务的更新接口，结果一致 | +| 错误处理 | 当参数无效时，返回明确的错误信息 | + +### 10.2 非功能验收 + +| 验收项 | 验收标准 | +|--------|----------| +| 可用性 | 接口可用性 ≥ 99.9% | +| 可靠性 | 诊断信息不丢失，数据一致性良好 | +| 扩展性 | 支持多种诊断来源的扩展 | + +## 11. 后续工作建议 + +1. **添加诊断结果查询接口**：提供RESTful API，方便前端查询诊断结果 +2. **支持多种诊断来源**：扩展diagnosisSource字段，支持多种诊断系统 +3. **添加诊断结果可视化**：在管理控制台添加诊断结果展示页面 +4. **优化诊断算法**：根据诊断结果，优化任务调度和资源分配 +5. **添加诊断结果告警**：当诊断结果为严重问题时，触发告警机制 + +## 12. 附录 + +### 12.1 术语定义 + +| 术语 | 解释 | +|------|------| +| Linkis | 基于Apache Linkis开发的大数据计算中间件 | +| doctoris | 任务诊断系统，用于分析任务运行问题 | +| RPC | 远程过程调用，用于系统间通信 | +| jobhistory | 任务历史服务，用于存储和查询任务历史信息 | +| EntranceServer | 入口服务，负责接收和处理任务请求 | + +### 12.2 参考文档 + +- [Apache Linkis官方文档](https://linkis.apache.org/) +- [MyBatis官方文档](https://mybatis.org/mybatis-3/zh/index.html) +- [Spring Boot官方文档](https://spring.io/projects/spring-boot) + +### 12.3 相关配置 + +| 配置项 | 默认值 | 描述 | +|--------|--------|------| +| linkis.task.diagnosis.enable | true | 任务诊断开关 | +| linkis.task.diagnosis.engine.type | spark | 任务诊断引擎类型 | +| linkis.task.diagnosis.timeout | 300000 | 任务诊断超时时间（毫秒） | +| linkis.doctor.url | 无 | Doctoris诊断系统URL | +| linkis.doctor.signature.token | 无 | Doctoris签名令牌 | \ No newline at end of file diff --git "a/docs/1.18.0/requirements/\346\227\245\345\277\227\346\224\257\346\214\201\347\273\206\347\262\222\345\272\246\350\277\224\345\233\236\346\224\271\351\200\240_\351\234\200\346\261\202.md" "b/docs/1.18.0/requirements/\346\227\245\345\277\227\346\224\257\346\214\201\347\273\206\347\262\222\345\272\246\350\277\224\345\233\236\346\224\271\351\200\240_\351\234\200\346\261\202.md" new file mode 100644 index 00000000000..d5ba14f7962 --- /dev/null +++ "b/docs/1.18.0/requirements/\346\227\245\345\277\227\346\224\257\346\214\201\347\273\206\347\262\222\345\272\246\350\277\224\345\233\236\346\224\271\351\200\240_\351\234\200\346\261\202.md" @@ -0,0 +1,125 @@ +# 阶段1：需求分析文档 + +## 一、需求背景 + +在大模型分析场景中，当前获取用户任务日志接口会返回所有（info、error、warn）任务日志，导致大模型处理文件数量过多。为了优化大模型处理效率，需要对 filesystem 模块的 openLog 接口进行增强，支持根据指定的日志级别返回对应的日志内容。 + +## 二、需求描述 + +### 2.1 需求详细描述 + +| 模块 | 功能点 | 功能描述 | UI设计及细节 | 功能关注点 | +|-----|--------|----------|--------------|------------| +| filesystem | 日志级别过滤 | 在 openLog 接口中添加 logLevel 参数，支持指定返回的日志级别 | 不涉及 | 确保参数类型正确，默认值设置合理 | +| filesystem | 多种日志级别支持 | 支持 logLevel=all,info,error,warn 四种取值 | 不涉及 | 确保所有取值都能正确处理 | +| filesystem | 默认值处理 | 缺省情况下返回全部日志（相当于 logLevel=all） | 不涉及 | 确保向后兼容性 | +| filesystem | 向后兼容 | 不影响现有调用方的使用 | 不涉及 | 现有调用方无需修改代码即可继续使用 | + +### 2.2 需求交互步骤 + +1. 用户调用 `/openLog` 接口，指定 `path` 参数和可选的 `logLevel` 参数 +2. 系统解析请求参数，获取日志文件路径和日志级别 +3. 系统读取日志文件内容，根据指定的日志级别过滤日志 +4. 系统返回过滤后的日志内容给用户 + +### 2.3 模块交互步骤 + +``` +用户 → filesystem模块 → openLog接口 → 日志文件 → 日志过滤 → 返回结果 +``` + +**关键步骤说明**： +1. 用户调用 openLog 接口，传入 path 和 logLevel 参数 +2. openLog 接口验证参数合法性，解析日志级别 +3. 系统读取指定路径的日志文件 +4. 系统根据日志级别过滤日志内容 +5. 系统将过滤后的日志内容封装为响应对象返回给用户 + +**关注点**： +- 需关注无效 logLevel 参数的处理，应返回默认日志（全部日志） +- 需关注日志文件过大的情况，应返回合理的错误信息 +- 需关注权限控制，确保用户只能访问自己有权限的日志文件 + +## 三、接口文档 + +### 3.1 接口基本信息 + +| 项 | 说明 | +|----|------| +| 接口URL | /api/rest_j/v1/filesystem/openLog | +| 请求方法 | GET | +| 接口描述 | 获取指定路径的日志文件内容，支持按日志级别过滤 | + +### 3.2 请求参数 + +| 参数名 | 类型 | 必填 | 默认值 | 说明 | +|--------|------|------|--------|------| +| path | String | 是 | 无 | 日志文件路径 | +| proxyUser | String | 否 | 无 | 代理用户，仅管理员可使用 | +| logLevel | String | 否 | all | 日志级别，取值为 all,info,error,warn | + +### 3.3 响应参数 + +| 参数名 | 类型 | 说明 | +|--------|------|------| +| status | String | 响应状态，success 表示成功，error 表示失败 | +| message | String | 响应消息 | +| data | Object | 响应数据 | +| data.log | String[] | 日志内容数组，按以下顺序排列：
1. 第0位：ERROR 级别的日志
2. 第1位：WARN 级别的日志
3. 第2位：INFO 级别的日志
4. 第3位：ALL 级别的日志（所有日志） | + +### 3.4 请求示例 + +```bash +# 请求所有日志 +curl -X GET "http://localhost:8080/api/rest_j/v1/filesystem/openLog?path=/path/to/test.log" + +# 请求特定级别的日志 +curl -X GET "http://localhost:8080/api/rest_j/v1/filesystem/openLog?path=/path/to/test.log&logLevel=error" +``` + +### 3.5 响应示例 + +**请求所有日志的响应**： +```json +{ + "status": "success", + "message": "", + "data": { + "log": [ + "2025-12-26 10:00:02.000 ERROR This is an error log\n", + "2025-12-26 10:00:01.000 WARN This is a warn log\n", + "2025-12-26 10:00:00.000 INFO This is an info log\n", + "2025-12-26 10:00:00.000 INFO This is an info log\n2025-12-26 10:00:01.000 WARN This is a warn log\n2025-12-26 10:00:02.000 ERROR This is an error log\n" + ] + } +} +``` + +**请求 ERROR 级别日志的响应**： +```json +{ + "status": "success", + "message": "", + "data": { + "log": [ + "2025-12-26 10:00:02.000 ERROR This is an error log\n", + "", + "", + "" + ] + } +} +``` + +## 四、关联影响分析 + +- **对存量功能的影响**：无，该功能是对现有接口的增强，不会影响其他功能 +- **对第三方组件的影响**：无，该功能仅涉及 filesystem 模块内部逻辑 + +## 五、测试关注点 + +- 验证不同日志级别参数的处理是否正确 +- 验证缺省情况下是否返回全部日志 +- 验证无效日志级别参数的处理是否正确 +- 验证大小写不敏感是否正确 +- 验证权限控制是否有效 diff --git "a/docs/1.18.0/requirements/\347\263\273\347\273\237\347\224\250\346\210\267\347\246\201\346\255\242\347\231\273\345\275\225\346\224\271\351\200\240_\351\234\200\346\261\202.md" "b/docs/1.18.0/requirements/\347\263\273\347\273\237\347\224\250\346\210\267\347\246\201\346\255\242\347\231\273\345\275\225\346\224\271\351\200\240_\351\234\200\346\261\202.md" new file mode 100644 index 00000000000..5e5857394a1 --- /dev/null +++ "b/docs/1.18.0/requirements/\347\263\273\347\273\237\347\224\250\346\210\267\347\246\201\346\255\242\347\231\273\345\275\225\346\224\271\351\200\240_\351\234\200\346\261\202.md" @@ -0,0 +1,119 @@ +# 阶段1：需求分析文档 + +## 1. 需求概述 + +### 1.1 背景 +根据安全要求，Linkis管理台需要禁止系统用户（如hadoop、hduser、shduser等）通过Web页面登录，以降低安全风险。 + +### 1.2 目标 +- 拦截系统用户的Web页面登录请求 +- 不影响客户端(client)及其他渠道的登录 +- 提供配置开关和系统用户前缀配置 + +## 2. 功能需求 + +### 2.1 登录拦截逻辑 + +| 编号 | 功能点 | 描述 | 优先级 | +|------|--------|------|--------| +| FR-001 | webLogin标识传递 | 前端在HTTP header中传递`webLogin`标识 | P0 | +| FR-002 | webLogin标识获取 | 后端从header获取标识，默认值为`false` | P0 | +| FR-003 | 系统用户拦截 | 当webLogin=true时，拦截系统用户前缀匹配的用户 | P0 | +| FR-004 | 非Web渠道放行 | webLogin=false或未传时不进行拦截 | P0 | + +### 2.2 错误提示 + +| 编号 | 功能点 | 描述 | 优先级 | +|------|--------|------|--------| +| FR-005 | 统一错误信息 | 拦截时返回"系统用户禁止登录" | P0 | + +### 2.3 配置管理 + +| 编号 | 功能点 | 描述 | 优先级 | +|------|--------|------|--------| +| FR-006 | 功能开关 | `linkis.system.user.prohibit.login.switch` 控制功能开启/关闭 | P0 | +| FR-007 | 系统用户前缀 | `linkis.system.user.prohibit.login.prefix` 配置系统用户前缀列表 | P0 | + +## 3. 非功能需求 + +### 3.1 兼容性 +- 现有客户端登录方式不受影响 +- 配置项需向后兼容 + +### 3.2 安全性 +- 拦截逻辑不可绕过 +- webLogin标识仅用于识别登录来源，不用于认证 + +### 3.3 可配置性 +- 功能可通过配置开关完全关闭 +- 系统用户前缀列表可动态配置 + +## 4. 数据字典 + +### 4.1 配置项 + +| 配置项 | 类型 | 默认值 | 说明 | +|--------|------|--------|------| +| linkis.system.user.prohibit.login.switch | Boolean | false | 禁止系统用户登录功能开关 | +| linkis.system.user.prohibit.login.prefix | String | hadoop,hduser,shduser | 系统用户前缀列表，逗号分隔 | + +### 4.2 HTTP Header + +| Header名称 | 类型 | 默认值 | 说明 | +|------------|------|--------|------| +| webLogin | String | false | Web页面登录标识，true表示来自Web页面 | + +## 5. 用例分析 + +### 5.1 正常场景 + +#### UC-001: 普通用户Web登录 +- **前置条件**: 功能开关开启 +- **输入**: 用户名=testuser, webLogin=true +- **预期**: 登录成功 + +#### UC-002: 系统用户Client登录 +- **前置条件**: 功能开关开启 +- **输入**: 用户名=hadoop, webLogin=false +- **预期**: 登录成功 + +### 5.2 异常场景 + +#### UC-003: 系统用户Web登录 +- **前置条件**: 功能开关开启 +- **输入**: 用户名=hadoop, webLogin=true +- **预期**: 登录失败，返回"系统用户禁止登录" + +#### UC-004: hduser用户Web登录 +- **前置条件**: 功能开关开启 +- **输入**: 用户名=hduser01, webLogin=true +- **预期**: 登录失败，返回"系统用户禁止登录" + +### 5.3 边界场景 + +#### UC-005: 功能开关关闭 +- **前置条件**: 功能开关关闭 +- **输入**: 用户名=hadoop, webLogin=true +- **预期**: 登录成功（不进行拦截） + +#### UC-006: webLogin未传递 +- **前置条件**: 功能开关开启 +- **输入**: 用户名=hadoop, header中无webLogin +- **预期**: 登录成功（默认webLogin=false） + +## 6. 影响范围分析 + +### 6.1 代码改动范围 + +| 文件 | 改动类型 | 改动内容 | +|------|---------|---------| +| GatewayConfiguration.scala | 修改 | 更新PROHIBIT_LOGIN_PREFIX默认值 | +| UserRestful.scala | 修改 | 修改登录拦截逻辑，从header获取webLogin | + +### 6.2 风险评估 + +| 风险 | 等级 | 缓解措施 | +|------|------|---------| +| 影响正常用户登录 | 低 | 功能开关默认关闭 | +| 前端未传webLogin | 低 | 默认值为false，不拦截 | +| 配置错误导致无法登录 | 中 | 提供配置示例和文档 | diff --git a/docs/configuration/linkis-storage.md b/docs/configuration/linkis-storage.md index 45b07afc1d2..8e04b073be5 100644 --- a/docs/configuration/linkis-storage.md +++ b/docs/configuration/linkis-storage.md @@ -29,4 +29,4 @@ |linkis-storage|wds.linkis.fs.hdfs.impl.disable.cache| false |disable.cache |true| |linkis-storage|wds.linkis.hdfs.rest.errs| |rest.errs|true| |linkis-storage|wds.linkis.resultset.row.max.str | 2m | max.str |true| -|linkis-storage|wds.linkis.storage.file.type | dolphin,sql,scala,py,hql,python,out,log,text,sh,jdbc,ngql,psql,fql,tsql | file.type |true| +|linkis-storage|wds.linkis.storage.file.type | dolphin,sql,scala,py,hql,python,out,log,text,sh,jdbc,ngql,psql,fql,tsql,txt | file.type |true| diff --git a/docs/errorcode/python-errorcode.md b/docs/errorcode/python-errorcode.md index 8a4f8434840..cf63aaa3fe8 100644 --- a/docs/errorcode/python-errorcode.md +++ b/docs/errorcode/python-errorcode.md @@ -1,7 +1,7 @@ ## python errorcode -| 模块名(服务名) | 错误码 | 描述 |enumeration name| Exception Class| -| -------- | -------- | ----- |-----|-----| -|python|41001| |PYTHON_EXECUTE_ERROR|LinkisPythonErrorCodeSummary| -|python|60003|Pyspark process has stopped, query failed!(Pyspark 进程已停止，查询失败！)|PYSPARK_PROCESSS_STOPPED|LinkisPythonErrorCodeSummary| -|python|400201|Invalid python session.(无效的 python 会话.)|INVALID_PYTHON_SESSION|LinkisPythonErrorCodeSummary| +| 模块名(服务名) | 错误码 | 描述 |enumeration name| Exception Class| +| -------- |--------| ----- |-----|-----| +|python| 60002 | |PYTHON_EXECUTE_ERROR|LinkisPythonErrorCodeSummary| +|python| 60003 |Pyspark process has stopped, query failed!(Pyspark 进程已停止，查询失败！)|PYSPARK_PROCESSS_STOPPED|LinkisPythonErrorCodeSummary| +|python| 400201 |Invalid python session.(无效的 python 会话.)|INVALID_PYTHON_SESSION|LinkisPythonErrorCodeSummary| diff --git a/docs/info-1.3.2.md b/docs/info-1.3.2.md index 55e5aeecbc7..0c690517fb3 100644 --- a/docs/info-1.3.2.md +++ b/docs/info-1.3.2.md @@ -4,4 +4,5 @@ |------------------| ----- |----------------------------------------------------------------------|------| ------------------------------------------------------- | | linkis-jobhistory | 新增 | wds.linkis.jobhistory.admin | hadoop |可以查看所有历史任务的用户 注意：wds.linkis.governance.station.admin 为管理用户（也具有可以查看所有历史任务的权限）| | linkis | 新增 | wds.linkis.governance.station.admin.token | /具有管理员权限的特殊token| +| linkis | 新增 | linkis.configuration.remove.application.cache | IDE |清除该应用的配置缓存| | cg-entrance | 新增 | linkis.entrance.auto.clean.dirty.data.enable | true |entrance重启调用ps-jobhistory接口是否开启，ture为开启，取值范围：true或false| diff --git a/linkis-commons/linkis-common/pom.xml b/linkis-commons/linkis-common/pom.xml index 6fe5c47f53c..cd101bb0aae 100644 --- a/linkis-commons/linkis-common/pom.xml +++ b/linkis-commons/linkis-common/pom.xml @@ -124,6 +124,11 @@ 1.10 + + org.springframework.boot + spring-boot-actuator + + org.reflections reflections @@ -163,17 +168,4 @@ - - - spark-3 - - 3.7.0-M11 - 1.11.0-wds-spark3 - 3.4.4 - 2.12.17 - 2.12 - 2.14.2 - - - diff --git a/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/ByteTimeUtils.java b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/ByteTimeUtils.java index 0ecb3dc2a5e..e81da47e693 100644 --- a/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/ByteTimeUtils.java +++ b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/ByteTimeUtils.java @@ -365,7 +365,7 @@ public double toBytes(long d) { if (d < 0) { throw new IllegalArgumentException("Negative size value. Size must be positive: " + d); } - return d * multiplier; + return (double) d * multiplier; } public long toKiB(long d) { diff --git a/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SHAUtils.java b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SHAUtils.java index fab5441ea43..d134081eb1d 100644 --- a/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SHAUtils.java +++ b/linkis-commons/linkis-common/src/main/java/org/apache/linkis/common/utils/SHAUtils.java @@ -17,24 +17,12 @@ package org.apache.linkis.common.utils; -import org.apache.linkis.common.conf.CommonVars; - -import org.apache.commons.lang3.StringUtils; - -import java.io.IOException; import java.io.UnsupportedEncodingException; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; -import java.util.HashMap; -import java.util.Map; public class SHAUtils { - public static final String DOCTOR_NONCE = - CommonVars.apply("linkis.doctor.signature.nonce", "").getValue(); - public static final CommonVars DOCTOR_TOKEN = - CommonVars.apply("linkis.doctor.signature.token", ""); - /** * 对字符串加密,默认使用SHA-256 * @@ -72,26 +60,4 @@ public static String bytes2Hex(byte[] bts) { } return des; } - - public static void main(String[] args) throws IOException { - if (StringUtils.isBlank(args[0])) { - throw new LinkageError("Invalid applicationId cannot be empty"); - } - Map parms = new HashMap<>(); - String timestampStr = String.valueOf(System.currentTimeMillis()); - parms.put("applicationId", args[0]); - parms.put("app_id", args[1]); - parms.put("timestamp", timestampStr); - parms.put("nonce", DOCTOR_NONCE); - String token = args[2]; - if (StringUtils.isNotBlank(token)) { - String signature = - Encrypt( - Encrypt(parms.get("app_id") + DOCTOR_NONCE + System.currentTimeMillis(), null) - + token, - null); - parms.put("signature", signature); - } - System.out.println(parms); - } } diff --git a/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/conf/Configuration.scala b/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/conf/Configuration.scala index a2917041e86..c3ba2d379af 100644 --- a/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/conf/Configuration.scala +++ b/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/conf/Configuration.scala @@ -104,6 +104,12 @@ object Configuration extends Logging { val METRICS_INCREMENTAL_UPDATE_ENABLE = CommonVars[Boolean]("linkis.jobhistory.metrics.incremental.update.enable", false) + val EXECUTE_ERROR_CODE_INDEX = + CommonVars("execute.error.code.index", "-1") + + val EXECUTE_RESULTSET_ALIAS_NUM = + CommonVars("execute.resultset.alias.num", "0") + val GLOBAL_CONF_CHN_NAME = "全局设置" val GLOBAL_CONF_CHN_OLDNAME = "通用设置" diff --git a/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/utils/VariableUtils.scala b/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/utils/VariableUtils.scala index 3f529e0454f..d74176fbb8c 100644 --- a/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/utils/VariableUtils.scala +++ b/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/utils/VariableUtils.scala @@ -28,7 +28,7 @@ import org.apache.linkis.common.variable.DateTypeUtils.{ getYesterday } -import org.apache.commons.lang3.{Strings, StringUtils} +import org.apache.commons.lang3.StringUtils import java.time.ZonedDateTime import java.util @@ -122,9 +122,7 @@ object VariableUtils extends Logging { case _ => if (!nameAndType.contains(key) && StringUtils.isNotEmpty(value)) { // if ((allCatch opt value.toDouble).isDefined) { - if ( - (allCatch opt BigDecimal(value)).isDefined && !Strings.CS.startsWith(value, "0") - ) { + if ((allCatch opt BigDecimal(value)).isDefined && !value.startsWith("0")) { nameAndType(key) = variable.BigDecimalValue(BigDecimal(value)) } else { nameAndType(key) = variable.StringType(value) diff --git a/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/variable/CustomDateType.scala b/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/variable/CustomDateType.scala index 4359df33984..0c528a4a9b6 100644 --- a/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/variable/CustomDateType.scala +++ b/linkis-commons/linkis-common/src/main/scala/org/apache/linkis/common/variable/CustomDateType.scala @@ -79,20 +79,12 @@ class CustomMonthType(date: String, std: Boolean = true, isEnd: Boolean = false) def -(months: Int): String = { val dateFormat = DateTypeUtils.dateFormatLocal.get() - if (std) { - DateTypeUtils.getMonth(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), -months)) - } else { - DateTypeUtils.getMonth(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), -months)) - } + DateTypeUtils.getMonth(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), -months)) } def +(months: Int): String = { val dateFormat = DateTypeUtils.dateFormatLocal.get() - if (std) { - DateTypeUtils.getMonth(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), months)) - } else { - DateTypeUtils.getMonth(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), months)) - } + DateTypeUtils.getMonth(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), months)) } override def toString: String = { @@ -111,20 +103,12 @@ class CustomMonType(date: String, std: Boolean = true, isEnd: Boolean = false) { def -(months: Int): String = { val dateFormat = DateTypeUtils.dateFormatMonLocal.get() - if (std) { - DateTypeUtils.getMon(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), -months)) - } else { - DateTypeUtils.getMon(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), -months)) - } + DateTypeUtils.getMon(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), -months)) } def +(months: Int): String = { val dateFormat = DateTypeUtils.dateFormatMonLocal.get() - if (std) { - DateTypeUtils.getMon(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), months)) - } else { - DateTypeUtils.getMon(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), months)) - } + DateTypeUtils.getMon(std, isEnd, DateUtils.addMonths(dateFormat.parse(date), months)) } override def toString: String = { diff --git a/linkis-commons/linkis-common/src/test/java/org/apache/linkis/common/utils/ShaUtilsTest.java b/linkis-commons/linkis-common/src/test/java/org/apache/linkis/common/utils/ShaUtilsTest.java new file mode 100644 index 00000000000..d4a6b423347 --- /dev/null +++ b/linkis-commons/linkis-common/src/test/java/org/apache/linkis/common/utils/ShaUtilsTest.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.common.utils; + +import org.apache.commons.lang3.StringUtils; + +import java.io.UnsupportedEncodingException; +import java.util.HashMap; +import java.util.Map; + +import org.junit.jupiter.api.Test; + +class ShaUtilsTest { + @Test + void encryptTest() throws UnsupportedEncodingException { + String applicationId = "test_application_id"; + String app_id = "test_app_id"; + String token = "test_token"; + String nonce = "123456"; + if (StringUtils.isBlank(applicationId)) { + throw new LinkageError("Invalid applicationId cannot be empty"); + } + if (StringUtils.isBlank(app_id)) { + throw new LinkageError("Invalid app_id cannot be empty"); + } + if (StringUtils.isBlank(token)) { + throw new LinkageError("Invalid token cannot be empty"); + } + if (StringUtils.isBlank(nonce)) { + throw new LinkageError("Invalid nonce cannot be empty"); + } + Map parms = new HashMap<>(); + String timestampStr = String.valueOf(System.currentTimeMillis()); + parms.put("applicationId", applicationId); + parms.put("app_id", app_id); + parms.put("timestamp", timestampStr); + parms.put("nonce", nonce); + if (StringUtils.isNotBlank(token)) { + String signature = + SHAUtils.Encrypt( + SHAUtils.Encrypt(parms.get("app_id") + nonce + timestampStr, null) + token, null); + parms.put("signature", signature); + } + } +} diff --git a/linkis-commons/linkis-common/src/test/resources/linkis.properties b/linkis-commons/linkis-common/src/test/resources/linkis.properties index d6e47523f29..d6a6ae46c21 100644 --- a/linkis-commons/linkis-common/src/test/resources/linkis.properties +++ b/linkis-commons/linkis-common/src/test/resources/linkis.properties @@ -1,4 +1,4 @@ -# +# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. @@ -13,6 +13,8 @@ # limitations under the License. # -linkis.jobhistory.error.msg.tip=properties支持中文 -linkis.test.error.conf=123 -linkis.test.error.conf2= 456 \ No newline at end of file + +cli.version=${project.version} + +linkis.test.error.conf=123 +linkis.test.error.conf2=456 diff --git a/linkis-commons/linkis-hadoop-common/pom.xml b/linkis-commons/linkis-hadoop-common/pom.xml index 5c87306dca9..be6002f9e97 100644 --- a/linkis-commons/linkis-hadoop-common/pom.xml +++ b/linkis-commons/linkis-hadoop-common/pom.xml @@ -6,9 +6,9 @@ ~ The ASF licenses this file to You under the Apache License, Version 2.0 ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at - ~ + ~ ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ + ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -56,12 +56,15 @@ org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-auth + + org.apache.hadoop - hadoop-auth + ${hadoop-hdfs-client.artifact} + org.apache.logging.log4j diff --git a/linkis-commons/linkis-hadoop-common/src/main/java/org/apache/linkis/hadoop/common/utils/KerberosUtils.java b/linkis-commons/linkis-hadoop-common/src/main/java/org/apache/linkis/hadoop/common/utils/KerberosUtils.java index 67fecd04be5..608d5fad3c4 100644 --- a/linkis-commons/linkis-hadoop-common/src/main/java/org/apache/linkis/hadoop/common/utils/KerberosUtils.java +++ b/linkis-commons/linkis-hadoop-common/src/main/java/org/apache/linkis/hadoop/common/utils/KerberosUtils.java @@ -138,6 +138,7 @@ public static void checkStatus() { } else { LOG.info("CurrentUser is not null"); } + assert curUsr != null; if (loginUsr.getClass() != curUsr.getClass()) { LOG.info("getClass() is different"); } else { diff --git a/linkis-commons/linkis-httpclient/pom.xml b/linkis-commons/linkis-httpclient/pom.xml index da075dd194f..1951e3cd4f2 100644 --- a/linkis-commons/linkis-httpclient/pom.xml +++ b/linkis-commons/linkis-httpclient/pom.xml @@ -42,6 +42,7 @@ httpmime ${httpmime.version} + diff --git a/linkis-commons/linkis-httpclient/src/main/scala/org/apache/linkis/httpclient/AbstractHttpClient.scala b/linkis-commons/linkis-httpclient/src/main/scala/org/apache/linkis/httpclient/AbstractHttpClient.scala index 84fb6320c5a..0814c20c7ac 100644 --- a/linkis-commons/linkis-httpclient/src/main/scala/org/apache/linkis/httpclient/AbstractHttpClient.scala +++ b/linkis-commons/linkis-httpclient/src/main/scala/org/apache/linkis/httpclient/AbstractHttpClient.scala @@ -93,11 +93,13 @@ abstract class AbstractHttpClient(clientConfig: ClientConfig, clientName: String protected val connectionManager = new PoolingHttpClientConnectionManager + // 设置连接池参数 + connectionManager.setMaxTotal(clientConfig.getMaxConnection) + connectionManager.setDefaultMaxPerRoute(clientConfig.getMaxConnection / 2) + private val httpClientBuilder: HttpClientBuilder = HttpClients .custom() .setDefaultCookieStore(cookieStore) - .setMaxConnTotal(clientConfig.getMaxConnection) - .setMaxConnPerRoute(clientConfig.getMaxConnection / 2) .setConnectionManager(connectionManager) protected val httpClient: CloseableHttpClient = if (clientConfig.isSSL) { @@ -611,9 +613,11 @@ abstract class AbstractHttpClient(clientConfig: ClientConfig, clientName: String clientConnectInfo.put("leased", totalStats.getLeased) clientConnectInfo.put("avaiLabel", totalStats.getAvailable) clientConnectInfo.put("maxTotal", connectionManager.getMaxTotal) + clientConnectInfo.put("pendingCount", totalStats.getPending) logger.info(s"BMLClient:总最大连接数：${connectionManager.getMaxTotal}") logger.info(s"BMLClient:空闲连接数：${totalStats.getAvailable}") logger.info(s"BMLClient:活跃连接数：${totalStats.getLeased}") + logger.info(s"BMLClient:等待连接数：${totalStats.getPending}") clientConnectInfo } diff --git a/linkis-commons/linkis-module/pom.xml b/linkis-commons/linkis-module/pom.xml index a640a8feee0..687cc884600 100644 --- a/linkis-commons/linkis-module/pom.xml +++ b/linkis-commons/linkis-module/pom.xml @@ -41,6 +41,24 @@ ${knife4j.version} + + io.springfox + springfox-spring-webflux + ${springfox.version} + + + io.github.classgraph + classgraph + + + + + + io.github.classgraph + classgraph + ${classgraph.version} + + org.springframework spring-core @@ -64,6 +82,76 @@ + + org.springframework.cloud + spring-cloud-starter-netflix-eureka-client + + + javax.ws.rs + jsr311-api + + + org.springframework.boot + spring-boot-autoconfigure + + + org.springframework.boot + spring-boot-starter-aop + + + org.springframework.cloud + spring-cloud-starter + + + org.springframework.cloud + spring-cloud-commons + + + org.springframework.cloud + spring-cloud-context + + + org.springframework.boot + spring-boot-starter + + + org.springframework.boot + spring-boot-starter-cache + + + com.fasterxml.jackson.core + jackson-core + + + com.fasterxml.jackson.core + jackson-databind + + + io.github.x-stream + mxparser + + + com.fasterxml.woodstox + woodstox-core + + + org.codehaus.jettison + jettison + + + + + + com.fasterxml.woodstox + woodstox-core + ${woodstox.core.version} + + + + org.codehaus.jettison + jettison + ${jettision.version} + org.springframework.boot @@ -162,17 +250,8 @@ org.springframework.boot spring-boot-starter - - org.bouncycastle - bcpkix-jdk15on - - - org.bouncycastle - bcpkix-jdk18on - ${bouncycastle.version} - org.springframework.cloud @@ -281,7 +360,6 @@ spring-retry 1.3.4 - @@ -307,6 +385,52 @@ org.springframework.cloud spring-cloud-starter-netflix-eureka-client + + + javax.ws.rs + jsr311-api + + + org.springframework.boot + spring-boot-autoconfigure + + + org.springframework.boot + spring-boot-starter-aop + + + org.springframework.cloud + spring-cloud-starter + + + org.springframework.cloud + spring-cloud-commons + + + org.springframework.cloud + spring-cloud-context + + + org.springframework.boot + spring-boot-starter + + + org.springframework.boot + spring-boot-starter-cache + + + com.fasterxml.jackson.core + jackson-core + + + com.fasterxml.jackson.core + jackson-databind + + + io.github.x-stream + mxparser + + @@ -322,6 +446,7 @@ com.alibaba.cloud spring-cloud-starter-alibaba-nacos-discovery + ${spring-cloud-alibaba.version} org.springframework.boot diff --git a/linkis-commons/linkis-protocol/src/main/java/org/apache/linkis/protocol/util/ImmutablePair.java b/linkis-commons/linkis-protocol/src/main/java/org/apache/linkis/protocol/util/ImmutablePair.java index 28fb7a040eb..f09029b998c 100644 --- a/linkis-commons/linkis-protocol/src/main/java/org/apache/linkis/protocol/util/ImmutablePair.java +++ b/linkis-commons/linkis-protocol/src/main/java/org/apache/linkis/protocol/util/ImmutablePair.java @@ -18,6 +18,7 @@ package org.apache.linkis.protocol.util; import java.util.AbstractMap; +import java.util.Objects; public class ImmutablePair { @@ -53,6 +54,11 @@ public boolean equals(Object o) { } } + @Override + public int hashCode() { + return Objects.hashCode(entry); + } + private boolean eq(Object o1, Object o2) { if (null != o1 && null != o2) { return o1.equals(o2); diff --git a/linkis-commons/linkis-rpc/pom.xml b/linkis-commons/linkis-rpc/pom.xml index a3354e0e0e8..2fc3ab3ba55 100644 --- a/linkis-commons/linkis-rpc/pom.xml +++ b/linkis-commons/linkis-rpc/pom.xml @@ -88,6 +88,7 @@ commons-fileupload commons-fileupload + ${commons-fileupload.version} org.springframework.cloud diff --git a/linkis-commons/linkis-rpc/src/main/java/org/apache/linkis/rpc/serializer/ProtostuffSerializeUtil.java b/linkis-commons/linkis-rpc/src/main/java/org/apache/linkis/rpc/serializer/ProtostuffSerializeUtil.java index 6743e66a850..c377a7c4dfb 100644 --- a/linkis-commons/linkis-rpc/src/main/java/org/apache/linkis/rpc/serializer/ProtostuffSerializeUtil.java +++ b/linkis-commons/linkis-rpc/src/main/java/org/apache/linkis/rpc/serializer/ProtostuffSerializeUtil.java @@ -61,8 +61,11 @@ public static String serialize(T obj) { public static T deserialize(String str, Class clazz) { Schema schema = getSchema(clazz); - T obj = schema.newMessage(); - ProtostuffIOUtil.mergeFrom(toByteArray(str), obj, schema); + T obj = null; + if (schema != null) { + obj = schema.newMessage(); + ProtostuffIOUtil.mergeFrom(toByteArray(str), obj, schema); + } return obj; } @@ -93,7 +96,7 @@ public static byte[] toByteArray(String hexString) { for (int i = 0; i < byteArray.length; i++) { byte high = (byte) (Character.digit(hexString.charAt(k), 16) & 0xff); byte low = (byte) (Character.digit(hexString.charAt(k + 1), 16) & 0xff); - byteArray[i] = (byte) (high << 4 | low); + byteArray[i] = (byte) (high << 4 | low); // NOSONAR k += 2; } return byteArray; diff --git a/linkis-commons/linkis-rpc/src/main/scala/org/apache/linkis/rpc/transform/RPCConsumer.scala b/linkis-commons/linkis-rpc/src/main/scala/org/apache/linkis/rpc/transform/RPCConsumer.scala index a77ff9f71f4..85e956fe04b 100644 --- a/linkis-commons/linkis-rpc/src/main/scala/org/apache/linkis/rpc/transform/RPCConsumer.scala +++ b/linkis-commons/linkis-rpc/src/main/scala/org/apache/linkis/rpc/transform/RPCConsumer.scala @@ -19,6 +19,8 @@ package org.apache.linkis.rpc.transform import org.apache.linkis.common.exception.ExceptionManager import org.apache.linkis.common.utils.Utils +import org.apache.linkis.rpc.conf.RPCConfiguration +import org.apache.linkis.rpc.errorcode.LinkisRpcErrorCodeSummary.CORRESPONDING_CLASS_ILLEGAL import org.apache.linkis.rpc.errorcode.LinkisRpcErrorCodeSummary.CORRESPONDING_NOT_FOUND import org.apache.linkis.rpc.errorcode.LinkisRpcErrorCodeSummary.CORRESPONDING_TO_INITIALIZE import org.apache.linkis.rpc.exception.DWCURIException @@ -50,7 +52,19 @@ private[linkis] object RPCConsumer { if (data.isEmpty) return BoxedUnit.UNIT val objectStr = data.get(OBJECT_VALUE).toString val objectClass = data.get(CLASS_VALUE).toString + val isRequestProtocol = data.get(IS_REQUEST_PROTOCOL_CLASS).toString logger.debug("The corresponding anti-sequence is class {}", objectClass) + if ( + RPCConfiguration.ENABLE_RPC_OBJECT_PREFIX_WHITE_LIST_CHECK && "true".equals( + isRequestProtocol + ) && !RPCConfiguration.RPC_OBJECT_PREFIX_WHITE_LIST + .exists(prefix => objectClass.startsWith(prefix)) + ) { + throw new DWCURIException( + CORRESPONDING_CLASS_ILLEGAL.getErrorCode, + MessageFormat.format(CORRESPONDING_CLASS_ILLEGAL.getErrorDesc, objectClass) + ) + } val clazz = Utils.tryThrow(Class.forName(objectClass)) { case _: ClassNotFoundException => new DWCURIException( diff --git a/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/fifoqueue/FIFOUserConsumer.scala b/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/fifoqueue/FIFOUserConsumer.scala index 56be297c6b6..72a547289ea 100644 --- a/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/fifoqueue/FIFOUserConsumer.scala +++ b/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/fifoqueue/FIFOUserConsumer.scala @@ -151,6 +151,7 @@ class FIFOUserConsumer( } else getWaitForRetryEvent } } + event.foreach { case job: Job => logger.info(s"event not empty ${job.getState} id: ${job.getId()}") Utils.tryCatch { diff --git a/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/parallelqueue/ParallelConsumerManager.scala b/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/parallelqueue/ParallelConsumerManager.scala index b079f120069..f729ff0c594 100644 --- a/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/parallelqueue/ParallelConsumerManager.scala +++ b/linkis-commons/linkis-scheduler/src/main/scala/org/apache/linkis/scheduler/queue/parallelqueue/ParallelConsumerManager.scala @@ -123,6 +123,7 @@ class ParallelConsumerManager(maxParallelismUsers: Int, schedulerName: String) PFIFO_SCHEDULER_STRATEGY .equals(fifoQueueStrategy) && isSupportPriority(groupName) ) { + logger.info(s"use priority queue: ${groupName}") new PriorityLoopArrayQueue(group) } else new LoopArrayQueue(group) newConsumer.setConsumeQueue(consumerQueue) diff --git a/linkis-commons/linkis-scheduler/src/test/java/org/apache/linkis/scheduler/queue/PriorityLoopArrayQueueTest.java b/linkis-commons/linkis-scheduler/src/test/java/org/apache/linkis/scheduler/queue/PriorityLoopArrayQueueTest.java deleted file mode 100644 index cd45a991f1a..00000000000 --- a/linkis-commons/linkis-scheduler/src/test/java/org/apache/linkis/scheduler/queue/PriorityLoopArrayQueueTest.java +++ /dev/null @@ -1,234 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.linkis.scheduler.queue; - -import org.apache.linkis.scheduler.queue.fifoqueue.FIFOGroup; - -import java.util.Random; -import java.util.concurrent.CountDownLatch; -import java.util.concurrent.atomic.AtomicInteger; - -import scala.Option; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -class PriorityLoopArrayQueueTest { - AtomicInteger productCounter = new AtomicInteger(); - AtomicInteger consumerCounter = new AtomicInteger(); - - @Test - public void testConcurrentPutAndTake() throws Exception { - AtomicInteger counter = new AtomicInteger(); - FIFOGroup group = new FIFOGroup("test", 5000, 5000); - PriorityLoopArrayQueue queue = new PriorityLoopArrayQueue(group); - boolean testFlag = false; - - if (testFlag) { - // 获取开始时间的毫秒数 - long startTime = System.currentTimeMillis(); - // 10s的毫秒数 - long threeMinutesInMillis = 10 * 1000; - int genLen = 50; - int getLen = 70; - final CountDownLatch latch = new CountDownLatch(genLen + getLen + 1); - // 5 个生产者 - for (int i = 0; i < genLen; i++) { - final int id = i; - new Thread( - () -> { - try { - Thread.sleep(100 * id); - latch.countDown(); - latch.await(); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread().getName() + "开始生产："); - while ((System.currentTimeMillis() - startTime) < threeMinutesInMillis) { - // 生产 - try { - Thread.sleep(getRandom(200)); - product(counter, queue); - product(counter, queue); - } catch (InterruptedException e) { - throw new RuntimeException(e); - } - // 消费 - // consume(queue); - } - System.out.println(Thread.currentThread().getName() + "结束生产："); - }, - "生产t-" + i) - .start(); - } - // 5 个消费者 - for (int i = 0; i < getLen; i++) { - final int id = i; - new Thread( - () -> { - try { - Thread.sleep(getRandom(200)); - latch.countDown(); - latch.await(); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread().getName() + "开始消费："); - while (true) { - try { - Thread.sleep(getRandom(200)); - // 消费 - consume(queue); - } catch (InterruptedException e) { - throw new RuntimeException(e); - } - } - }, - "消费t-" + i) - .start(); - } - new Thread( - () -> { - try { - Thread.sleep(100); - latch.countDown(); - latch.await(); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread().getName() + "开始获取当前队列元素："); - while ((System.currentTimeMillis() - startTime) < threeMinutesInMillis * 2) { - try { - Thread.sleep(5000); - } catch (InterruptedException e) { - throw new RuntimeException(e); - } - System.out.println("生产大小：" + productCounter.get()); - System.out.println("消费大小：" + consumerCounter.get()); - System.out.println("队列当前大小：" + queue.size()); - // 需要 去掉私有测试 - // System.out.println("index size: " + queue.indexMap().size()); - // System.out.println("cache size: " + queue.fixedSizeCollection().size()); - } - }) - .start(); - Thread.sleep(threeMinutesInMillis * 2); - System.out.println( - "product:" + productCounter.get() + ", consumer: " + consumerCounter.get()); - // 需要 去掉私有测试 - // Assertions.assertEquals(1000, queue.fixedSizeCollection().size()); - Assertions.assertEquals(productCounter.get(), consumerCounter.get()); - } - } - - // 消费 - private void consume(PriorityLoopArrayQueue queue) { - SchedulerEvent take = null; - try { - take = queue.take(); - consumerCounter.addAndGet(1); - } catch (Exception e) { - throw new RuntimeException(e); - } - printEvent("消费", take); - } - - // 生产 - private void product(AtomicInteger counter, PriorityLoopArrayQueue queue) { - int i1 = counter.addAndGet(1); - // 1000-重要，100-普通，10-不重要 - int[] proArr = {1000, 100, 10}; - int priority = getRandom(3); - String name = "item-" + i1 + "-" + priority; - System.out.println("生产：" + name); - Option offer = queue.offer(getJob(name, proArr[priority])); - if (offer.nonEmpty()) { - productCounter.addAndGet(1); - Option schedulerEventOption = queue.get((int) offer.get()); - printEvent("get：", schedulerEventOption.get()); - } else { - System.out.println("当前队列已满，大小：" + queue.size()); - } - } - - @Test - void testFinally() {} - - @Test - void enqueue() { - // 压测 offer take get - FIFOGroup group = new FIFOGroup("test", 100, 100); - PriorityLoopArrayQueue queue = new PriorityLoopArrayQueue(group); - Option idx = queue.offer(getJob("job1-1", 1)); - // 插入测试 - Assertions.assertEquals(1, (int) idx.get()); - queue.offer(getJob("job2", 2)); - queue.offer(getJob("job3", 3)); - queue.offer(getJob("job1-2", 1)); - queue.offer(getJob("job5", 5)); - queue.offer(getJob("item1-3", 1)); - queue.offer(getJob("item6-1", 6)); - queue.offer(getJob("item4", 4)); - queue.offer(getJob("item6-2", 6)); - // peek 测试 - Option peek = queue.peek(); - Assertions.assertEquals("item6-1", peek.get().getId()); - while (queue.size() > 1) { - queue.take(); - } - SchedulerEvent event = queue.take(); - // 优先级，以及先进先出测试 - Assertions.assertEquals("item1-3", event.getId()); - Assertions.assertEquals(1, event.priority()); - Assertions.assertEquals(6, event.getIndex()); - // 缓存测试，需要设置 linkis.fifo.priority.queue.max.cache.size 为 5 - // Assertions.assertThrows( - // IllegalArgumentException.class, - // () -> { - // queue.get(7); - // }); - } - - private void printEvent(String opt, SchedulerEvent event) { - System.out.println( - "【" - + Thread.currentThread().getName() - + "】" - + opt - + ":" - + event.getId() - + ", priority: " - + event.getPriority() - + ", index: " - + event.getIndex()); - } - - private int getRandom(int bound) { - Random rand = new Random(); - int res = rand.nextInt(bound); - return res; - } - - private UserJob getJob(String name, int priority) { - UserJob job = new UserJob(); - job.setId(name); - job.setPriority(priority); - return job; - } -} diff --git a/linkis-commons/linkis-storage/pom.xml b/linkis-commons/linkis-storage/pom.xml index 95282243fd6..e99e0e4d3ec 100644 --- a/linkis-commons/linkis-storage/pom.xml +++ b/linkis-commons/linkis-storage/pom.xml @@ -56,6 +56,7 @@ protobuf-java ${protobuf.version} + org.springframework spring-core @@ -87,7 +88,6 @@ org.apache.hadoop hadoop-aliyun - 3.3.4 com.aliyun.oss diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/FieldTruncationResult.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/FieldTruncationResult.java new file mode 100644 index 00000000000..6760d95d9dd --- /dev/null +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/FieldTruncationResult.java @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.storage.entity; + +import java.util.List; + +/** Result of field truncation detection and processing */ +public class FieldTruncationResult { + private boolean hasOversizedFields; + private List oversizedFields; + private Integer maxOversizedFieldCount; + private List data; + + public FieldTruncationResult() {} + + public FieldTruncationResult( + boolean hasOversizedFields, + List oversizedFields, + Integer maxOversizedFieldCount, + List data) { + this.hasOversizedFields = hasOversizedFields; + this.oversizedFields = oversizedFields; + this.maxOversizedFieldCount = maxOversizedFieldCount; + this.data = data; + } + + public boolean isHasOversizedFields() { + return hasOversizedFields; + } + + public void setHasOversizedFields(boolean hasOversizedFields) { + this.hasOversizedFields = hasOversizedFields; + } + + public List getOversizedFields() { + return oversizedFields; + } + + public void setOversizedFields(List oversizedFields) { + this.oversizedFields = oversizedFields; + } + + public Integer getMaxOversizedFieldCount() { + return maxOversizedFieldCount; + } + + public void setMaxOversizedFieldCount(Integer maxOversizedFieldCount) { + this.maxOversizedFieldCount = maxOversizedFieldCount; + } + + public List getData() { + return data; + } + + public void setData(List data) { + this.data = data; + } +} diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/OversizedFieldInfo.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/OversizedFieldInfo.java new file mode 100644 index 00000000000..f1fd1b985b9 --- /dev/null +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/entity/OversizedFieldInfo.java @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.storage.entity; + +/** Represents information about an oversized field in a result set */ +public class OversizedFieldInfo { + private String fieldName; + private Integer rowIndex; + private Integer actualLength; + private Integer maxLength; + + public OversizedFieldInfo() {} + + public OversizedFieldInfo( + String fieldName, Integer rowIndex, Integer actualLength, Integer maxLength) { + this.fieldName = fieldName; + this.rowIndex = rowIndex; + this.actualLength = actualLength; + this.maxLength = maxLength; + } + + public String getFieldName() { + return fieldName; + } + + public void setFieldName(String fieldName) { + this.fieldName = fieldName; + } + + public Integer getRowIndex() { + return rowIndex; + } + + public void setRowIndex(Integer rowIndex) { + this.rowIndex = rowIndex; + } + + public Integer getActualLength() { + return actualLength; + } + + public void setActualLength(Integer actualLength) { + this.actualLength = actualLength; + } + + public Integer getMaxLength() { + return maxLength; + } + + public void setMaxLength(Integer maxLength) { + this.maxLength = maxLength; + } +} diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/excel/ExcelStorageReader.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/excel/ExcelStorageReader.java index 2e3ca6e085a..910c3d78171 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/excel/ExcelStorageReader.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/excel/ExcelStorageReader.java @@ -37,7 +37,7 @@ public static List> getExcelTitle( } else { res = XlsxUtils.getBasicInfo(in, file); } - if (res == null && res.size() < 2) { + if (res == null || res.size() < 2) { throw new Exception("There is a problem with the file format(文件格式有问题)"); } List headerType = new ArrayList<>(); diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/FileSystem.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/FileSystem.java index 8a266fadf84..4ed306931e7 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/FileSystem.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/FileSystem.java @@ -23,7 +23,7 @@ import java.io.File; import java.io.IOException; -import java.util.Collections; +import java.util.ArrayList; import java.util.List; import org.slf4j.Logger; @@ -55,20 +55,14 @@ public String getDefaultFolderPerm() { public abstract long getLength(FsPath dest) throws IOException; - public String getChecksumWithMD5(FsPath dest) throws IOException { - return null; - } + public abstract String getChecksumWithMD5(FsPath dest) throws IOException; - public String getChecksum(FsPath dest) throws IOException { - return null; - } + public abstract String getChecksum(FsPath dest) throws IOException; - public long getBlockSize(FsPath dest) throws IOException { - return 0L; - } + public abstract long getBlockSize(FsPath dest) throws IOException; public List getAllFilePaths(FsPath dest) throws IOException { - return Collections.emptyList(); + return new ArrayList<>(); } public abstract boolean canExecute(FsPath dest) throws IOException; diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/AzureBlobFileSystem.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/AzureBlobFileSystem.java index f9cb73d93d1..dd29a88e26f 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/AzureBlobFileSystem.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/AzureBlobFileSystem.java @@ -397,6 +397,16 @@ public String getChecksum(FsPath dest) throws IOException { return null; } + @Override + public String getChecksumWithMD5(FsPath dest) throws IOException { + return null; + } + + @Override + public long getBlockSize(FsPath dest) throws IOException { + return 0L; + } + @Override public boolean canExecute(FsPath dest) throws IOException { return false; diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/HDFSFileSystem.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/HDFSFileSystem.java index 2c095d102b6..67f27ab6e38 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/HDFSFileSystem.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/HDFSFileSystem.java @@ -337,7 +337,7 @@ public boolean exists(FsPath dest) throws IOException { private void resetRootHdfs() { if (fs != null) { String locker = user + LOCKER_SUFFIX; - synchronized (locker.intern()) { + synchronized (locker.intern()) { // NOSONAR if (fs != null) { if (HadoopConf.HDFS_ENABLE_CACHE()) { long currentTime = System.currentTimeMillis(); @@ -503,8 +503,7 @@ public long getLength(FsPath dest) throws IOException { return fileStatus.getLen(); } - @Override - public String getChecksumWithMD5(FsPath dest) throws IOException { + public String checkSum(FsPath dest) throws IOException { String path = checkHDFSPath(dest.getPath()); if (!exists(dest)) { throw new IOException("directory or file not exists: " + path); @@ -524,6 +523,17 @@ public String getChecksum(FsPath dest) throws IOException { return fileChecksum.toString(); } + @Override + public String getChecksumWithMD5(FsPath dest) throws IOException { + String path = checkHDFSPath(dest.getPath()); + if (!exists(dest)) { + throw new IOException("directory or file not exists: " + path); + } + MD5MD5CRC32FileChecksum fileChecksum = + (MD5MD5CRC32FileChecksum) fs.getFileChecksum(new Path(path)); + return fileChecksum.toString().split(":")[1]; + } + @Override public long getBlockSize(FsPath dest) throws IOException { String path = checkHDFSPath(dest.getPath()); diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/LocalFileSystem.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/LocalFileSystem.java index 1098a84fc40..9c7a5961fbb 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/LocalFileSystem.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/LocalFileSystem.java @@ -215,7 +215,9 @@ public boolean copy(String origin, String dest) throws IOException { setOwner(new FsPath(dest), user, null); } } catch (Throwable e) { - file.delete(); + if (!file.delete()) { + throw new IOException("File delete failed!"); + } if (e instanceof IOException) { throw (IOException) e; } else { @@ -383,14 +385,18 @@ public boolean create(String dest) throws IOException { if (!isOwner(file.getParent())) { throw new IOException("you have on permission to create file " + dest); } - file.createNewFile(); + if (!file.createNewFile()) { + throw new IOException("create new file error! path:" + dest); + } try { setPermission(new FsPath(dest), this.getDefaultFilePerm()); if (!user.equals(getOwner(dest))) { setOwner(new FsPath(dest), user, null); } } catch (Throwable e) { - file.delete(); + if (!file.delete()) { + throw new IOException("delete file error!"); + } if (e instanceof IOException) { throw (IOException) e; } else { diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/OSSFileSystem.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/OSSFileSystem.java index da036f64e91..c5e79e51dea 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/OSSFileSystem.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/OSSFileSystem.java @@ -85,6 +85,21 @@ public long getLength(FsPath dest) throws IOException { return 0; } + @Override + public String getChecksumWithMD5(FsPath dest) throws IOException { + return null; + } + + @Override + public String getChecksum(FsPath dest) throws IOException { + return null; + } + + @Override + public long getBlockSize(FsPath dest) throws IOException { + return 0L; + } + @Override public boolean canExecute(FsPath dest) throws IOException { return true; diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java index e95dd36388f..83b56156c06 100644 --- a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/fs/impl/S3FileSystem.java @@ -441,6 +441,21 @@ public long getLength(FsPath dest) throws IOException { .getContentLength(); } + @Override + public String getChecksumWithMD5(FsPath dest) throws IOException { + return null; + } + + @Override + public String getChecksum(FsPath dest) throws IOException { + return null; + } + + @Override + public long getBlockSize(FsPath dest) throws IOException { + return 0L; + } + @Override public boolean canExecute(FsPath dest) { return true; diff --git a/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/utils/ResultUtils.java b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/utils/ResultUtils.java new file mode 100644 index 00000000000..3b8a0511439 --- /dev/null +++ b/linkis-commons/linkis-storage/src/main/java/org/apache/linkis/storage/utils/ResultUtils.java @@ -0,0 +1,544 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.storage.utils; + +import org.apache.linkis.common.io.FsWriter; +import org.apache.linkis.storage.conf.LinkisStorageConf; +import org.apache.linkis.storage.entity.FieldTruncationResult; +import org.apache.linkis.storage.entity.OversizedFieldInfo; +import org.apache.linkis.storage.excel.StorageExcelWriter; +import org.apache.linkis.storage.resultset.table.TableMetaData; +import org.apache.linkis.storage.resultset.table.TableRecord; +import org.apache.linkis.storage.source.FileSource; + +import org.apache.commons.lang3.StringUtils; +import org.apache.commons.math3.util.Pair; + +import java.io.IOException; +import java.text.MessageFormat; +import java.util.*; +import java.util.function.Predicate; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class ResultUtils { + + public static final Logger LOGGER = LoggerFactory.getLogger(ResultUtils.class); + /** + * 删除指定字段的内容 + * + * @param metadata 元数据数组，包含字段信息 + * @param contentList 需要处理的二维字符串数组 + * @param fieldsToRemove 需要删除的字段集合 + * @return 处理后的字符串数组，若输入无效返回空集合而非null + */ + @SuppressWarnings("unchecked") + public static List removeFieldsFromContent( + Object metadata, List contentList, Set fieldsToRemove) { + // 1. 参数校验 + if (metadata == null + || fieldsToRemove == null + || fieldsToRemove.isEmpty() + || contentList == null + || !(metadata instanceof Map[])) { + return contentList; + } + + // 2. 安全类型转换 + Map[] fieldMetadata = (Map[]) metadata; + + // 3. 收集需要删除的列索引（去重并排序） + List columnsToRemove = + IntStream.range(0, fieldMetadata.length) + .filter( + i -> { + Map meta = fieldMetadata[i]; + Object columnName = meta.get("columnName"); + return columnName != null + && fieldsToRemove.contains(columnName.toString().toLowerCase()); + }) + .distinct() + .boxed() + .sorted((a, b) -> Integer.compare(b, a)) + .collect(Collectors.toList()); + + // 如果没有需要删除的列，直接返回副本 + if (columnsToRemove.isEmpty()) { + return new ArrayList<>(contentList); + } + // 4. 对每行数据进行处理（删除指定列） + return contentList.stream() + .map( + row -> { + if (row == null || row.length == 0) { + return row; + } + // 创建可变列表以便删除元素 + List rowList = new ArrayList<>(Arrays.asList(row)); + // 从后向前删除列，避免索引变化问题 + for (int columnIndex : columnsToRemove) { + if (columnIndex < rowList.size()) { + rowList.remove(columnIndex); + LOGGER.info("MaskedField Remove Data columnIndex:" + columnIndex); + } + } + return rowList.toArray(new String[0]); + }) + .collect(Collectors.toList()); + } + + @SuppressWarnings("unchecked") + public static Map[] filterMaskedFieldsFromMetadata(Object metadata, Set maskedFields) { + // 1. 参数校验 + if (metadata == null || maskedFields == null || !(metadata instanceof Map[])) { + return new Map[0]; + } + + // 2. 类型转换（已通过校验，可安全强转） + Map[] originalMaps = (Map[]) metadata; + + // 3. 过滤逻辑（提取谓词增强可读性） + Predicate> isNotMaskedField = + map -> !maskedFields.contains(map.get("columnName").toString().toLowerCase()); + + // 4. 流处理 + 结果转换 + return Arrays.stream(originalMaps) + .filter(isNotMaskedField) + .toArray(Map[]::new); // 等价于 toArray(new Map[0]) + } + + /** + * Convert Map array to TableMetaData + * + * @param metadataArray Array of Map containing column information + * @return TableMetaData object + */ + @SuppressWarnings("unchecked") + public static TableMetaData convertMapArrayToTableMetaData(Map[] metadataArray) { + if (metadataArray == null || metadataArray.length == 0) { + return new TableMetaData(new org.apache.linkis.storage.domain.Column[0]); + } + + org.apache.linkis.storage.domain.Column[] columns = + new org.apache.linkis.storage.domain.Column[metadataArray.length]; + + for (int i = 0; i < metadataArray.length; i++) { + Map columnMap = (Map) metadataArray[i]; + String columnName = + columnMap.get("columnName") != null ? columnMap.get("columnName").toString() : ""; + String dataType = + columnMap.get("dataType") != null ? columnMap.get("dataType").toString() : "string"; + String comment = columnMap.get("comment") != null ? columnMap.get("comment").toString() : ""; + + // Create Column object + org.apache.linkis.storage.domain.DataType dtype = + org.apache.linkis.storage.domain.DataType$.MODULE$.toDataType(dataType); + columns[i] = new org.apache.linkis.storage.domain.Column(columnName, dtype, comment); + } + + return new TableMetaData(columns); + } + + public static void dealMaskedField( + String maskedFieldNames, FsWriter fsWriter, FileSource fileSource) throws IOException { + + LOGGER.info("Applying field masking for fields: {}", maskedFieldNames); + + // Parse masked field names + Set maskedFieldsSet = + Arrays.stream(maskedFieldNames.split(",")) + .map(String::trim) + .map(String::toLowerCase) + .filter(StringUtils::isNotBlank) + .collect(Collectors.toSet()); + + // Collect data from file source + Pair>[] collectedData = fileSource.collect(); + + // Process each result set + for (int i = 0; i < collectedData.length; i++) { + Pair> collectedDataSet = collectedData[i]; + Object metadata = collectedDataSet.getFirst(); + ArrayList content = collectedDataSet.getSecond(); + + // Filter metadata and content + Map[] filteredMetadata = filterMaskedFieldsFromMetadata(metadata, maskedFieldsSet); + List filteredContent = removeFieldsFromContent(metadata, content, maskedFieldsSet); + + // Convert Map[] to TableMetaData + TableMetaData tableMetaData = convertMapArrayToTableMetaData(filteredMetadata); + + // Write filtered data + fsWriter.addMetaData(tableMetaData); + for (String[] row : filteredContent) { + fsWriter.addRecord(new TableRecord(row)); + } + LOGGER.info( + "Field masking applied for result set {}. Original columns: {}, Filtered columns: {}", + i, + ((Map[]) metadata).length, + filteredMetadata.length); + } + } + + /** + * Detect and handle oversized fields in result set + * + * @param metadata Column names list + * @param FileContent Data rows list (each row is an ArrayList or Object[]) + * @param truncate Whether to truncate (false means detection only) + * @return FieldTruncationResult containing detection results and processed data + */ + public static FieldTruncationResult detectAndHandle( + Object metadata, List FileContent, Integer maxLength, boolean truncate) { + if (metadata == null || !(metadata instanceof Map[])) { + return new FieldTruncationResult(); + } + + // 2. 类型转换（已通过校验，可安全强转） + Map[] originalMaps = (Map[]) metadata; + + // 提取列名 + List columnNames = new ArrayList<>(); + if (metadata != null) { + for (Map meta : originalMaps) { + Object columnName = meta.get("columnName"); + columnNames.add(columnName != null ? columnName.toString() : ""); + } + } + + // 转换 String[] 数组为 ArrayList + List> dataList = new ArrayList<>(); + for (String[] row : FileContent) { + ArrayList rowList = new ArrayList<>(Arrays.asList(row)); + dataList.add(rowList); + } + + int maxCount = LinkisStorageConf.OVERSIZED_FIELD_MAX_COUNT(); + + // Detect oversized fields + List oversizedFields = + detectOversizedFields(columnNames, dataList, maxLength, maxCount); + + boolean hasOversizedFields = !oversizedFields.isEmpty(); + + // Truncate if requested + List> processedData = dataList; + LOGGER.info( + "Staring Field truncation detection function ,truncate is {},hasOversizedFields {}", + truncate, + hasOversizedFields); + if (truncate && hasOversizedFields) { + processedData = truncateFields(columnNames, dataList, maxLength); + } + List convertedList = + processedData.stream() + .map(row -> row != null ? row.toArray(new String[0]) : null) + .collect(Collectors.toList()); + return new FieldTruncationResult(hasOversizedFields, oversizedFields, maxCount, convertedList); + } + + public static void detectAndHandle( + StorageExcelWriter fsWriter, FileSource fileSource, Integer maxLength) throws IOException { + // Collect data from file source + Pair>[] collectedData = fileSource.collect(); + + // Process each result set + for (int i = 0; i < collectedData.length; i++) { + Pair> collectedDataSet = collectedData[i]; + Object metadata = collectedDataSet.getFirst(); + ArrayList content = collectedDataSet.getSecond(); + + FieldTruncationResult fieldTruncationResult = + detectAndHandle(metadata, content, maxLength, true); + + List data = fieldTruncationResult.getData(); + + // Convert Map[] to TableMetaData and add truncation markers for oversized fields + TableMetaData tableMetaData = + convertMapArrayToTableMetaData((Map[]) metadata); + // Create a set of oversized field names for quick lookup + List oversizedFieldNames = + fieldTruncationResult.getOversizedFields().stream() + .map(OversizedFieldInfo::getFieldName) + .distinct() + .collect(Collectors.toList()); + // If there are oversized fields, add markers to column names in the metadata + if (fieldTruncationResult.isHasOversizedFields() + && fieldTruncationResult.getOversizedFields() != null) { + // Update column names to indicate truncation with max length + org.apache.linkis.storage.domain.Column[] columns = tableMetaData.columns(); + for (int j = 0; j < columns.length; j++) { + if (oversizedFieldNames.contains(columns[j].columnName())) { + // Get the max length for this field + String truncatedInfo = + maxLength != null ? "(truncated to " + maxLength + " chars)" : "(truncated)"; + // Create a new column with truncation info suffix to indicate truncation + columns[j] = + new org.apache.linkis.storage.domain.Column( + columns[j].columnName() + truncatedInfo, + columns[j].dataType(), + columns[j].comment()); + } + } + + // Create new TableMetaData with updated column names + tableMetaData = new TableMetaData(columns); + } + // Write filtered data + if (oversizedFieldNames.isEmpty()) { + fsWriter.addMetaData(tableMetaData); + } else { + StringJoiner joiner = new StringJoiner(","); + oversizedFieldNames.forEach(joiner::add); + String note = + MessageFormat.format( + "结果集存在字段值超过{0}字符，无法全量下载，以下字段截取前{0}字符展示：{1}", + LinkisStorageConf.FIELD_EXPORT_DOWNLOAD_LENGTH(), joiner); + fsWriter.addMetaDataWithNote(tableMetaData, note); + } + for (String[] row : data) { + fsWriter.addRecord(new TableRecord(row)); + } + } + } + + /** + * Detect oversized fields + * + * @param metadata Column names + * @param dataList Data rows + * @param maxLength Max length threshold + * @param maxCount Max number of oversized fields to collect + * @return List of oversized field info + */ + private static List detectOversizedFields( + List metadata, List> dataList, int maxLength, int maxCount) { + + List oversizedFields = new ArrayList<>(); + + if (metadata == null || dataList == null || dataList.isEmpty()) { + return oversizedFields; + } + + // 使用Set来存储已经检查过的超长字段名，避免重复检查 + Set detectedOversizedFields = new HashSet<>(); + + // Iterate through data rows + for (int rowIndex = 0; rowIndex < dataList.size(); rowIndex++) { + + ArrayList row = dataList.get(rowIndex); + if (row == null) { + continue; + } + + // Check each field in the row + for (int colIndex = 0; colIndex < row.size() && colIndex < metadata.size(); colIndex++) { + + String fieldName = metadata.get(colIndex); + + // 如果该字段已经被检测为超长字段，则跳过检查，提高效率 + if (detectedOversizedFields.contains(fieldName)) { + continue; + } + + String fieldValue = row.get(colIndex); + int fieldLength = getFieldLength(fieldValue); + + if (fieldLength > maxLength) { + oversizedFields.add(new OversizedFieldInfo(fieldName, rowIndex, fieldLength, maxLength)); + // 将超长字段名加入Set，避免重复检查 + detectedOversizedFields.add(fieldName); + LOGGER.info( + "Detected oversized field: field={}, row={}, actualLength={}, maxLength={}", + fieldName, + rowIndex, + fieldLength, + maxLength); + } + } + } + + return oversizedFields; + } + + /** + * Truncate oversized fields + * + * @param metadata Column names + * @param dataList Data rows + * @param maxLength Max length + * @return Truncated data list + */ + private static List> truncateFields( + List metadata, List> dataList, int maxLength) { + + if (dataList == null || dataList.isEmpty()) { + return dataList; + } + + List> truncatedData = new ArrayList<>(); + + for (ArrayList row : dataList) { + if (row == null) { + truncatedData.add(null); + continue; + } + + ArrayList truncatedRow = new ArrayList<>(); + for (String fieldValue : row) { + String truncatedValue = truncateFieldValue(fieldValue, maxLength); + truncatedRow.add(truncatedValue); + } + truncatedData.add(truncatedRow); + } + + return truncatedData; + } + + /** + * Get field value character length + * + * @param value Field value + * @return Character length + */ + private static int getFieldLength(Object value) { + if (value == null) { + return 0; + } + return value.toString().length(); + } + + /** + * Truncate single field value + * + * @param value Field value + * @param maxLength Max length + * @return Truncated value + */ + private static String truncateFieldValue(Object value, int maxLength) { + if (value == null) { + return null; + } + String str = value.toString(); + if (str.length() <= maxLength) { + return str; + } + return str.substring(0, maxLength); + } + + /** + * Apply both field masking and truncation + * + * @param maskedFieldNames Comma-separated list of field names to mask + * @param fsWriter The FsWriter to write results to + * @param fileSource The FileSource to read data from + * @throws IOException + */ + public static void applyFieldMaskingAndTruncation( + String maskedFieldNames, + StorageExcelWriter fsWriter, + FileSource fileSource, + Integer maxLength) + throws IOException { + + LOGGER.info("Applying both field masking and truncation"); + // First collect data from file source + Pair>[] collectedData = fileSource.collect(); + + // Process each result set + for (int i = 0; i < collectedData.length; i++) { + Pair> collectedDataSet = collectedData[i]; + Object metadata = collectedDataSet.getFirst(); + ArrayList content = collectedDataSet.getSecond(); + + // Apply field masking + Set maskedFieldsSet = + Arrays.stream(maskedFieldNames.split(",")) + .map(String::trim) + .map(String::toLowerCase) + .filter(StringUtils::isNotBlank) + .collect(Collectors.toSet()); + + Map[] filteredMetadata = filterMaskedFieldsFromMetadata(metadata, maskedFieldsSet); + List filteredContent = removeFieldsFromContent(metadata, content, maskedFieldsSet); + + // Then apply field truncation + FieldTruncationResult fieldTruncationResult = + detectAndHandle(filteredMetadata, filteredContent, maxLength, true); + List finalData = fieldTruncationResult.getData(); + + // Write data + TableMetaData tableMetaData = convertMapArrayToTableMetaData(filteredMetadata); + // Create a set of oversized field names for quick lookup + List oversizedFieldNames = + fieldTruncationResult.getOversizedFields().stream() + .map(OversizedFieldInfo::getFieldName) + .distinct() + .collect(Collectors.toList()); + // If there are oversized fields, add markers to column names in the metadata + if (fieldTruncationResult.isHasOversizedFields() + && fieldTruncationResult.getOversizedFields() != null) { + // Update column names to indicate truncation with max length + org.apache.linkis.storage.domain.Column[] columns = tableMetaData.columns(); + for (int j = 0; j < columns.length; j++) { + if (oversizedFieldNames.contains(columns[j].columnName())) { + // Get the max length for this field + String truncatedInfo = + maxLength != null + ? MessageFormat.format(LinkisStorageConf.FIELD_TRUNCATION_NOTE(), maxLength) + : LinkisStorageConf.FIELD_NOT_TRUNCATION_NOTE(); + // Create a new column with truncation info suffix to indicate truncation + columns[j] = + new org.apache.linkis.storage.domain.Column( + columns[j].columnName() + truncatedInfo, + columns[j].dataType(), + columns[j].comment()); + } + } + + // Create new TableMetaData with updated column names + tableMetaData = new TableMetaData(columns); + } + if (oversizedFieldNames.isEmpty()) { + fsWriter.addMetaData(tableMetaData); + } else { + StringJoiner joiner = new StringJoiner(","); + oversizedFieldNames.forEach(joiner::add); + String note = + MessageFormat.format( + LinkisStorageConf.FIELD_OPEN_FILE_TRUNCATION_NOTE(), + LinkisStorageConf.FIELD_EXPORT_DOWNLOAD_LENGTH(), + joiner); + fsWriter.addMetaDataWithNote(tableMetaData, note); + } + for (String[] row : finalData) { + fsWriter.addRecord(new TableRecord(row)); + } + LOGGER.info( + "Field masking and truncation applied for result set {}. Original columns: {}, Filtered columns: {}, Truncated fields: {}", + i, + ((Map[]) metadata).length, + filteredMetadata.length, + fieldTruncationResult.getOversizedFields().size()); + } + } +} diff --git a/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/conf/LinkisStorageConf.scala b/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/conf/LinkisStorageConf.scala index 50c60fecd2e..a28ede48df1 100644 --- a/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/conf/LinkisStorageConf.scala +++ b/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/conf/LinkisStorageConf.scala @@ -83,4 +83,33 @@ object LinkisStorageConf { val LINKIS_READ_ROW_BYTE_MAX_LEN = ByteTimeUtils.byteStringAsBytes(LINKIS__READ_RESULT_ROW_MAX_LEN_STR) + val FIELD_TRUNCATION_ENABLED = + CommonVars("linkis.resultset.field.truncation.enabled", false).getValue + + val FIELD_MASKED_ENABLED = + CommonVars("linkis.resultset.field.masked.enabled", false).getValue + + val FIELD_VIEW_MAX_LENGTH = CommonVars("linkis.resultset.field.view.max.length", 10000).getValue + + val FIELD_EXPORT_DOWNLOAD_LENGTH = + CommonVars("linkis.resultset.field.download.max.length", 32767).getValue + + val FIELD_EXPORT_MAX_LENGTH = + CommonVars("linkis.resultset.field.export.max.length", 32767).getValue + + val OVERSIZED_FIELD_MAX_COUNT = + CommonVars("linkis.resultset.field.oversized.max.count", 20).getValue + + val FIELD_TRUNCATION_NOTE = + CommonVars("linkis.resultset.field.truncation.excel.note", "(truncated to {0} chars)").getValue + + val FIELD_NOT_TRUNCATION_NOTE = + CommonVars("linkis.resultset.field.truncation.default.note", "(truncated)").getValue + + val FIELD_OPEN_FILE_TRUNCATION_NOTE = + CommonVars( + "linkis.resultset.field.truncation.note", + "结果集存在字段值超过{0}字符，无法全量下载，以下字段截取前{0}字符展示：{1}" + ).getValue + } diff --git a/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/excel/StorageExcelWriter.scala b/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/excel/StorageExcelWriter.scala index 9ea83130df2..8baeb792e2b 100644 --- a/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/excel/StorageExcelWriter.scala +++ b/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/excel/StorageExcelWriter.scala @@ -24,6 +24,7 @@ import org.apache.linkis.storage.resultset.table.{TableMetaData, TableRecord} import org.apache.commons.io.IOUtils import org.apache.poi.ss.usermodel._ +import org.apache.poi.ss.util.CellRangeAddress import org.apache.poi.xssf.streaming.{SXSSFCell, SXSSFSheet, SXSSFWorkbook} import java.io._ @@ -68,6 +69,15 @@ class StorageExcelWriter( headerCellStyle } + def getWarningStyle: CellStyle = { + val warningFont = workBook.createFont + warningFont.setBold(true) + warningFont.setColor(IndexedColors.RED.getIndex) + val warningCellStyle = workBook.createCellStyle + warningCellStyle.setFont(warningFont) + warningCellStyle + } + def getWorkBook: Workbook = { // 自适应列宽 sheet.trackAllColumnsForAutoSizing() @@ -133,6 +143,38 @@ class StorageExcelWriter( rowPoint += 1 } + @scala.throws[IOException] + def addMetaDataWithNote(metaData: MetaData, note: String): Unit = { + init + // 创建公告行（第0行） + val noticeRow = sheet.createRow(0) + val noticeCell = noticeRow.createCell(0) + noticeCell.setCellValue(note) + noticeCell.setCellStyle(getWarningStyle) + + // 合并单元格（从第0列到最后一列） + val columns = metaData.asInstanceOf[TableMetaData].columns + val columnCount = columns.length + if (columnCount > 1) { + val mergeRange = new CellRangeAddress(0, 0, 0, columnCount - 1) + sheet.addMergedRegion(mergeRange) + } + + // 创建表头行（第1行） + val tableHead = sheet.createRow(1) + val columnType = new ArrayBuffer[DataType]() + columnCounter = 0 + for (elem <- columns) { + val headCell = tableHead.createCell(columnCounter) + headCell.setCellValue(elem.columnName) + headCell.setCellStyle(getDefaultHeadStyle) + columnType += elem.dataType + columnCounter += 1 + } + types = columnType.toArray + rowPoint += 2 // 由于增加了公告行，所以行指针需要增加2（公告行和表头行） + } + @scala.throws[IOException] override def addRecord(record: Record): Unit = { // TODO: 是否需要替换null值 diff --git a/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/resultset/table/TableResultDeserializer.scala b/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/resultset/table/TableResultDeserializer.scala index 86d09e9532d..323d816c9bc 100644 --- a/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/resultset/table/TableResultDeserializer.scala +++ b/linkis-commons/linkis-storage/src/main/scala/org/apache/linkis/storage/resultset/table/TableResultDeserializer.scala @@ -127,14 +127,16 @@ class TableResultDeserializer extends ResultDeserializer[TableMetaData, TableRec val len = colArray(i).toInt val res = Dolphin.getString(bytes, index, len) if (res.length > LinkisStorageConf.LINKIS_RESULT_COL_LENGTH && enableLimit) { - throw new ColLengthExceedException( - LinkisStorageErrorCodeSummary.RESULT_COL_LENGTH.getErrorCode, - MessageFormat.format( - LinkisStorageErrorCodeSummary.RESULT_COL_LENGTH.getErrorDesc, - res.length.asInstanceOf[Object], - LinkisStorageConf.LINKIS_RESULT_COL_LENGTH.asInstanceOf[Object] + if (!LinkisStorageConf.FIELD_TRUNCATION_ENABLED) { + throw new ColLengthExceedException( + LinkisStorageErrorCodeSummary.RESULT_COL_LENGTH.getErrorCode, + MessageFormat.format( + LinkisStorageErrorCodeSummary.RESULT_COL_LENGTH.getErrorDesc, + res.length.asInstanceOf[Object], + LinkisStorageConf.LINKIS_RESULT_COL_LENGTH.asInstanceOf[Object] + ) ) - ) + } } index += len // 如果enableLimit为true，则采取的是列分页 diff --git a/linkis-commons/linkis-storage/src/test/java/org/apache/linkis/storage/excel/ExcelStorageReaderTest.java b/linkis-commons/linkis-storage/src/test/java/org/apache/linkis/storage/excel/ExcelStorageReaderTest.java new file mode 100644 index 00000000000..8a7d3da0b07 --- /dev/null +++ b/linkis-commons/linkis-storage/src/test/java/org/apache/linkis/storage/excel/ExcelStorageReaderTest.java @@ -0,0 +1,206 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.storage.excel; + +import org.apache.commons.io.input.BOMInputStream; +import org.apache.poi.hssf.usermodel.HSSFWorkbook; +import org.apache.poi.ss.usermodel.Cell; +import org.apache.poi.ss.usermodel.Row; +import org.apache.poi.ss.usermodel.Sheet; +import org.apache.poi.ss.usermodel.Workbook; +import org.apache.poi.xssf.usermodel.XSSFWorkbook; + +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.util.StringUtils; + +import java.io.*; +import java.nio.charset.StandardCharsets; +import java.util.*; + +import org.junit.jupiter.api.*; + +/** ExcelStorageReader Tester */ +public class ExcelStorageReaderTest { + + @Autowired private ExcelStorageReader excelStorageReader; + + @BeforeEach + @DisplayName("Each unit test method is executed once before execution") + public void before() throws Exception {} + + @AfterEach + @DisplayName("Each unit test method is executed once before execution") + public void after() throws Exception {} + + private static InputStream createExcelAndGetInputStream(int type) throws IOException { + Workbook workbook = new HSSFWorkbook(); + if (type == 1) { + workbook = new XSSFWorkbook(); + } + // 创建一个新的工作簿 + try { // HSSFWorkbook 用于处理 .xls 格式 + + // 创建一个工作表 + Sheet sheet = workbook.createSheet("Sheet1"); + + // 创建一行并在第一行写入一些数据（示例） + Row row = sheet.createRow(0); + Cell cell = row.createCell(0); + Cell cell2 = row.createCell(1); + cell.setCellValue("Hello"); + cell2.setCellValue("Hello2"); + + // 创建一个工作表 + Sheet sheet1 = workbook.createSheet("Sheet2"); + + // 创建一行并在第一行写入一些数据（示例） + Row row1 = sheet1.createRow(0); + Cell cell1 = row1.createCell(0); + Cell cell22 = row1.createCell(1); + cell1.setCellValue("Work"); + cell22.setCellValue("Work1"); + + // 将工作簿写入 ByteArrayOutputStream + try (ByteArrayOutputStream outputStream = new ByteArrayOutputStream()) { + workbook.write(outputStream); + + // 从 ByteArrayOutputStream 中获取 InputStream + return new ByteArrayInputStream(outputStream.toByteArray()); + } + } catch (Exception e) { + + } + return null; + } + + public InputStream createCSVInputStream(List> data) { + String csvData = convertToCSV(data); + return new ByteArrayInputStream(csvData.getBytes(StandardCharsets.UTF_8)); + } + + private String convertToCSV(List> data) { + StringBuilder csvData = new StringBuilder(); + + for (List row : data) { + for (String column : row) { + csvData.append(column).append(","); + } + csvData.deleteCharAt(csvData.length() - 1).append("\n"); + } + + return csvData.toString(); + } + + private Map getCsvInfo(InputStream in, boolean escapeQuotes, boolean hasHeader) + throws Exception { + HashMap csvMap = new LinkedHashMap<>(); + String[][] column = null; + // fix csv file with utf-8 with bom chart[] + BOMInputStream bomIn = new BOMInputStream(in, false); // don't include the BOM + BufferedReader reader = new BufferedReader(new InputStreamReader(bomIn, "utf-8")); // NOSONAR + + String header = reader.readLine(); + if (StringUtils.isEmpty(header)) { + throw new RuntimeException("内容为空"); + } + String[] line = header.split(",", -1); + int colNum = line.length; + column = new String[2][colNum]; + if (hasHeader) { + for (int i = 0; i < colNum; i++) { + column[0][i] = line[i]; + if (escapeQuotes) { + try { + csvMap.put(column[0][i].substring(1, column[0][i].length() - 1), "string"); + } catch (StringIndexOutOfBoundsException e) { + throw new RuntimeException("处理标题引号异常"); + } + } else { + csvMap.put(column[0][i], "string"); + } + } + } else { + for (int i = 0; i < colNum; i++) { + csvMap.put("col_" + (i + 1), "string"); + } + } + csvMap.forEach((key, value) -> System.out.println(key + ": " + value)); + return csvMap; + } + + @Test + public void getXlsSheetInfo() throws Exception { + Map>> sheetsInfo = + XlsUtils.getSheetsInfo(createExcelAndGetInputStream(0), true); + Assertions.assertTrue(sheetsInfo.containsKey("Sheet2")); + List> sheet2 = sheetsInfo.get("Sheet2"); + String work1 = ""; + for (Map sheetMap : sheet2) { + if (sheetMap.containsKey("Work1")) { + work1 = sheetMap.get("Work1"); + } + } + Assertions.assertEquals("string", work1); + } + + @Test + public void getXlsxSheetInfo() throws Exception { + Map>> sheetsInfo = + XlsxUtils.getAllSheetInfo(createExcelAndGetInputStream(1), null, true); + Assertions.assertTrue(sheetsInfo.containsKey("Sheet2")); + List> sheet2 = sheetsInfo.get("Sheet2"); + String work1 = ""; + for (Map sheetMap : sheet2) { + if (sheetMap.containsKey("Work1")) { + work1 = sheetMap.get("Work1"); + } + } + Assertions.assertEquals("string", work1); + } + + @Test + public void getCsvSheetInfo() throws Exception { + List> data = new ArrayList<>(); + data.add(Arrays.asList("Name", "Age", "City")); + data.add(Arrays.asList("John Doe", "30", "New York")); + data.add(Arrays.asList("Jane Smith", "25", "San Francisco")); + + // 有标题 + InputStream inputStream = createCSVInputStream(data); + Map csvMap = getCsvInfo(inputStream, false, true); + Assertions.assertEquals("string", csvMap.get("Name")); + + // 无标题 + InputStream inputStream1 = createCSVInputStream(data); + Map csvMap1 = getCsvInfo(inputStream1, false, false); + Assertions.assertEquals("string", csvMap1.get("col_1")); + + List> data1 = new ArrayList<>(); + data1.add(Arrays.asList("'Name'", "'Age'", "'City'")); + + // 有标题有引号 + InputStream inputStream2 = createCSVInputStream(data1); + Map csvMap2 = getCsvInfo(inputStream2, true, true); + Assertions.assertEquals("string", csvMap2.get("Name")); + + // 无标题 + InputStream inputStream3 = createCSVInputStream(data1); + Map csvMap3 = getCsvInfo(inputStream3, false, false); + Assertions.assertEquals("string", csvMap3.get("col_1")); + } +} diff --git a/linkis-computation-governance/linkis-client/linkis-cli/src/main/java/org/apache/linkis/cli/application/interactor/command/template/UniversalCmdTemplate.java b/linkis-computation-governance/linkis-client/linkis-cli/src/main/java/org/apache/linkis/cli/application/interactor/command/template/UniversalCmdTemplate.java index 83b42032cf8..f66f5ab3860 100644 --- a/linkis-computation-governance/linkis-client/linkis-cli/src/main/java/org/apache/linkis/cli/application/interactor/command/template/UniversalCmdTemplate.java +++ b/linkis-computation-governance/linkis-client/linkis-cli/src/main/java/org/apache/linkis/cli/application/interactor/command/template/UniversalCmdTemplate.java @@ -138,7 +138,7 @@ public class UniversalCmdTemplate extends AbstractCmdTemplate implements Cloneab option( CliKeys.JOB_LABEL, CliKeys.JOB_LABEL_CLUSTER, - new String[] {"-yarnCluster", "-k8sCluster"}, + new String[] {"-yarnCluster"}, "specify linkis yarn cluster for this job", true, ""); diff --git a/linkis-computation-governance/linkis-client/linkis-cli/src/test/java/org/apache/linkis/cli/application/LinkisClientApplicationTest.java b/linkis-computation-governance/linkis-client/linkis-cli/src/test/java/org/apache/linkis/cli/application/LinkisClientApplicationTest.java index ed7b48a4e11..8aa54d695f6 100644 --- a/linkis-computation-governance/linkis-client/linkis-cli/src/test/java/org/apache/linkis/cli/application/LinkisClientApplicationTest.java +++ b/linkis-computation-governance/linkis-client/linkis-cli/src/test/java/org/apache/linkis/cli/application/LinkisClientApplicationTest.java @@ -28,15 +28,23 @@ public class LinkisClientApplicationTest { String[] cmdStr; String[] cmdStr2; + String[] cmdStr3; @BeforeEach public void before() { System.setProperty("conf.root", "src/test/resources/conf/"); System.setProperty("user.name", "hadoop"); + cmdStr3 = new String[] {"--version"}; cmdStr2 = new String[] { "--gatewayUrl", "http://127.0.0.1:9001", + "--authStg", + "token", + "--authKey", + "Validation-Code", + "--authVal", + "BML-AUTH", "--status", // "--log", // "--kill", diff --git a/linkis-computation-governance/linkis-client/linkis-computation-client/src/test/java/org/apache/linkis/ujes/client/response/JobInfoResultTest.java b/linkis-computation-governance/linkis-client/linkis-computation-client/src/test/java/org/apache/linkis/ujes/client/response/JobInfoResultTest.java index b9e662a33ec..4354db07d43 100644 --- a/linkis-computation-governance/linkis-client/linkis-computation-client/src/test/java/org/apache/linkis/ujes/client/response/JobInfoResultTest.java +++ b/linkis-computation-governance/linkis-client/linkis-computation-client/src/test/java/org/apache/linkis/ujes/client/response/JobInfoResultTest.java @@ -52,26 +52,23 @@ void shouldReturnEmptyResultSet() { */ @Test void shouldReturnMultiResultSetWithOrder() { - boolean isWindows = System.getProperty("os.name").toLowerCase().startsWith("windows"); - if (!isWindows) { - String[] toBeReturned = - new String[] { - "/path/to/xxxx_1.txt", - "/some/path/xxxx_10.txt", + String[] toBeReturned = + new String[] { + "/path/to/xxxx_1.txt", + "/some/path/xxxx_10.txt", + "/another/path/xxxx_0.txt", + "/another/path/xxxx_2.txt", + "/yet/another/path/xxxx_3.txt", + }; + String[] setList = getResultSetList(toBeReturned); + assertIterableEquals( + Lists.newArrayList( "/another/path/xxxx_0.txt", + "/path/to/xxxx_1.txt", "/another/path/xxxx_2.txt", "/yet/another/path/xxxx_3.txt", - }; - String[] setList = getResultSetList(toBeReturned); - assertIterableEquals( - Lists.newArrayList( - "/another/path/xxxx_0.txt", - "/path/to/xxxx_1.txt", - "/another/path/xxxx_2.txt", - "/yet/another/path/xxxx_3.txt", - "/some/path/xxxx_10.txt"), - Lists.newArrayList(setList)); - } + "/some/path/xxxx_10.txt"), + Lists.newArrayList(setList)); } private static String[] getResultSetList(String[] toBeReturned) { diff --git a/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/entity/job/JobDiagnosisRequest.java b/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/entity/job/JobDiagnosisRequest.java new file mode 100644 index 00000000000..bc70d7b1a70 --- /dev/null +++ b/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/entity/job/JobDiagnosisRequest.java @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.governance.common.entity.job; + +import java.util.Date; + +public class JobDiagnosisRequest { + + private Long id; + + private Long jobHistoryId; + + private String diagnosisContent; + + private Date createdTime; + + private Date updatedTime; + private String onlyRead; + + private String diagnosisSource; + + // Getters and Setters + + public Long getId() { + return id; + } + + public void setId(Long id) { + this.id = id; + } + + public Long getJobHistoryId() { + return jobHistoryId; + } + + public void setJobHistoryId(Long jobHistoryId) { + this.jobHistoryId = jobHistoryId; + } + + public String getDiagnosisContent() { + return diagnosisContent; + } + + public void setDiagnosisContent(String diagnosisContent) { + this.diagnosisContent = diagnosisContent; + } + + public Date getCreatedTime() { + return createdTime; + } + + public void setCreatedTime(Date createdTime) { + this.createdTime = createdTime; + } + + public Date getUpdatedTime() { + return updatedTime; + } + + public void setUpdatedDate(Date updatedTime) { + this.updatedTime = updatedTime; + } + + public String getOnlyRead() { + return onlyRead; + } + + public void setOnlyRead(String onlyRead) { + this.onlyRead = onlyRead; + } + + public String getDiagnosisSource() { + return diagnosisSource; + } + + public void setDiagnosisSource(String diagnosisSource) { + this.diagnosisSource = diagnosisSource; + } + + @Override + public String toString() { + return "JobDiagnosis{" + + "id=" + + id + + ", jobHistoryId=" + + jobHistoryId + + ", diagnosisContent='" + + diagnosisContent + + "'" + + ", createdTime=" + + createdTime + + ", updatedTime=" + + updatedTime + + ", onlyRead='" + + onlyRead + + "'" + + ", diagnosisSource='" + + diagnosisSource + + "'" + + '}'; + } +} diff --git a/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/exception/engineconn/EngineConnExecutorErrorCode.java b/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/exception/engineconn/EngineConnExecutorErrorCode.java index 89d3c9eba42..8f76c52c5d2 100644 --- a/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/exception/engineconn/EngineConnExecutorErrorCode.java +++ b/linkis-computation-governance/linkis-computation-governance-common/src/main/java/org/apache/linkis/governance/common/exception/engineconn/EngineConnExecutorErrorCode.java @@ -39,4 +39,6 @@ public class EngineConnExecutorErrorCode { public static final int INIT_EXECUTOR_FAILED = 40106; public static final int INVALID_APPLICATION_ID = 40107; + + public static final int ILLEGAL_USE_UDF_FUNCTION = 40108; } diff --git a/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/constant/job/JobRequestConstants.scala b/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/constant/job/JobRequestConstants.scala index 5aab48388a1..49637c0eca0 100644 --- a/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/constant/job/JobRequestConstants.scala +++ b/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/constant/job/JobRequestConstants.scala @@ -36,6 +36,7 @@ object JobRequestConstants { val ENABLE_DIRECT_PUSH = "enableDirectPush" val DIRECT_PUSH_FETCH_SIZE = "direct_push_fetch_size" + val LINKIS_HIVE_EC_READ_RESULT_BY_OBJECT = "readResByObject" } diff --git a/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/job/JobReqProcotol.scala b/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/job/JobReqProcotol.scala index 9635aaf0c2c..f121ac0a541 100644 --- a/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/job/JobReqProcotol.scala +++ b/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/job/JobReqProcotol.scala @@ -17,7 +17,11 @@ package org.apache.linkis.governance.common.protocol.job -import org.apache.linkis.governance.common.entity.job.{JobAiRequest, JobRequest} +import org.apache.linkis.governance.common.entity.job.{ + JobAiRequest, + JobDiagnosisRequest, + JobRequest +} import org.apache.linkis.protocol.RetryableProtocol import org.apache.linkis.protocol.message.RequestProtocol @@ -40,6 +44,8 @@ case class JobReqReadAll(jobReq: JobRequest) extends JobReq case class JobAiReqInsert(jobReq: JobAiRequest) extends JobReq +case class JobDiagnosisReqInsert(jobReq: JobDiagnosisRequest) extends JobReq + class RequestOneJob extends JobReq { @BeanProperty diff --git a/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/task/ResponseTaskExecute.scala b/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/task/ResponseTaskExecute.scala index 95f1a542ac6..9e6be118b45 100644 --- a/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/task/ResponseTaskExecute.scala +++ b/linkis-computation-governance/linkis-computation-governance-common/src/main/scala/org/apache/linkis/governance/common/protocol/task/ResponseTaskExecute.scala @@ -52,10 +52,14 @@ case class ResponseTaskStatus(execId: String, status: ExecutionNodeStatus) class ResponseTaskStatusWithExecuteCodeIndex( execId: String, status: ExecutionNodeStatus, - private var _errorIndex: Int = -1 + private var _errorIndex: Int = -1, + private var _aliasNum: Int = 0 // 新增：aliasNum字段 ) extends ResponseTaskStatus(execId, status) { def errorIndex: Int = _errorIndex def errorIndex_=(value: Int): Unit = _errorIndex = value + // 新增：aliasNum的getter和setter + def aliasNum: Int = _aliasNum + def aliasNum_=(value: Int): Unit = _aliasNum = value } case class ResponseTaskResultSet(execId: String, output: String, alias: String) diff --git a/linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-core/src/main/scala/org/apache/linkis/ecm/core/launch/ProcessEngineConnLaunch.scala b/linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-core/src/main/scala/org/apache/linkis/ecm/core/launch/ProcessEngineConnLaunch.scala index 78891f62fd4..da6ea2bc77c 100644 --- a/linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-core/src/main/scala/org/apache/linkis/ecm/core/launch/ProcessEngineConnLaunch.scala +++ b/linkis-computation-governance/linkis-engineconn-manager/linkis-engineconn-manager-core/src/main/scala/org/apache/linkis/ecm/core/launch/ProcessEngineConnLaunch.scala @@ -173,7 +173,7 @@ trait ProcessEngineConnLaunch extends EngineConnLaunch with Logging { protected def getCommandArgs: Array[String] = { if ( request.creationDesc.properties.asScala.exists { case (k, v) => - k.contains(" ") + k.contains(" ") || (v != null && v.contains(" ")) } ) { throw new ErrorException( diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/java/org/apache/linkis/engineconn/computation/concurrent/monitor/TimingMonitorService.java b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/java/org/apache/linkis/engineconn/computation/concurrent/monitor/TimingMonitorService.java index 21d28e2d9e9..55a383f070f 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/java/org/apache/linkis/engineconn/computation/concurrent/monitor/TimingMonitorService.java +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/java/org/apache/linkis/engineconn/computation/concurrent/monitor/TimingMonitorService.java @@ -80,14 +80,14 @@ public void run() { LOG.warn("Executor can not is null"); return; } - isAvailable = true; + isAvailable = true; // NOSONAR monitorServiceList.forEach( monitorService -> { if (!monitorService.isAvailable()) { isAvailable = false; } }); - if (isAvailable) { + if (isAvailable) { // NOSONAR if (concurrentExecutor.isBusy()) synchronized (EXECUTOR_STATUS_LOCKER) { LOG.info("monitor turn to executor status from busy to unlock"); diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala index 9af394da80a..3243231d181 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/async/AsyncConcurrentComputationExecutor.scala @@ -106,11 +106,7 @@ abstract class AsyncConcurrentComputationExecutor(override val outputPrintLimit: } { e => logger.info("failed to do with hook", e) } - if (hookedCode.length > 100) { - logger.info(s"hooked after code: ${hookedCode.substring(0, 100)} ....") - } else { - logger.info(s"hooked after code: $hookedCode ") - } + logger.info(s"AsyncConcurrentComputationExecutor code hook completed.") val localPath = EngineConnConf.getLogDir engineExecutionContext.appendStdout( LogUtils.generateInfo( diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/conf/ComputationExecutorConf.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/conf/ComputationExecutorConf.scala index a6c055d4ecb..753b12eeff6 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/conf/ComputationExecutorConf.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/conf/ComputationExecutorConf.scala @@ -155,4 +155,16 @@ object ComputationExecutorConf { val SUPPORT_PARTIAL_RETRY_FOR_FAILED_TASKS_ENABLED: Boolean = CommonVars[Boolean]("linkis.partial.retry.for.failed.task.enabled", false).getValue + val CONTEXT_STATEMENT_PREFIXES = CommonVars( + "linkis.engineconn.context.statement.prefixes", + "USE ,SET ,ALTER SESSION ,SET ROLE ,SET SCHEMA ", + "SQL context statement prefixes for partial retry" + ) + + val JDBC_SET_STATEMENT_PREFIXES = CommonVars( + "linkis.engineconn.jdbc.set.statement.prefixes", + "SET QUERY_TIMEOUT,SET QUERY_QUEUE_PENDING_TIMEOUT,SET NEW_PLANNER", + "JDBC SET statement prefixes for error index adjustment" + ) + } diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala index ee48b133096..9135ac5f551 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/execute/ComputationExecutor.scala @@ -18,6 +18,7 @@ package org.apache.linkis.engineconn.computation.executor.execute import org.apache.linkis.DataWorkCloudApplication +import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{Logging, Utils} import org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor @@ -35,16 +36,22 @@ import org.apache.linkis.engineconn.computation.executor.exception.HookExecuteEx import org.apache.linkis.engineconn.computation.executor.hook.ComputationExecutorHook import org.apache.linkis.engineconn.computation.executor.metrics.ComputationEngineConnMetrics import org.apache.linkis.engineconn.computation.executor.upstream.event.TaskStatusChangedForUpstreamMonitorEvent +import org.apache.linkis.engineconn.computation.executor.utlis.ComputationEngineUtils import org.apache.linkis.engineconn.core.EngineConnObject import org.apache.linkis.engineconn.core.executor.ExecutorManager import org.apache.linkis.engineconn.executor.entity.{LabelExecutor, ResourceExecutor} import org.apache.linkis.engineconn.executor.listener.ExecutorListenerBusContext import org.apache.linkis.governance.common.entity.ExecutionNodeStatus import org.apache.linkis.governance.common.paser.CodeParser -import org.apache.linkis.governance.common.protocol.task.{EngineConcurrentInfo, RequestTask} +import org.apache.linkis.governance.common.protocol.task.{ + EngineConcurrentInfo, + RequestTask, + ResponseTaskError, + ResponseTaskStatusWithExecuteCodeIndex +} import org.apache.linkis.governance.common.utils.{JobUtils, LoggerUtils} import org.apache.linkis.manager.common.entity.enumeration.NodeStatus -import org.apache.linkis.manager.label.entity.engine.{EngineType, UserCreatorLabel} +import org.apache.linkis.manager.label.entity.engine.{EngineType, EngineTypeLabel, UserCreatorLabel} import org.apache.linkis.manager.label.utils.LabelUtil import org.apache.linkis.protocol.engine.JobProgressInfo import org.apache.linkis.scheduler.executer._ @@ -224,11 +231,7 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) case _ => logger.info("failed to do with hook", e) } } - if (hookedCode.length > 100) { - logger.info(s"hooked after code: ${hookedCode.substring(0, 100)} ....") - } else { - logger.info(s"hooked after code: $hookedCode ") - } + logger.info(s"ComputationExecutor code hook completed.") // task params log // spark engine: at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.executeLine log special conf @@ -264,17 +267,40 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) } var executeFlag = true val errorIndex: Int = Integer.valueOf( - engineConnTask.getProperties.getOrDefault("execute.error.code.index", "-1").toString + engineConnTask.getProperties + .getOrDefault(Configuration.EXECUTE_ERROR_CODE_INDEX.key, "-1") + .toString ) - engineExecutionContext.getProperties.put("execute.error.code.index", errorIndex.toString) + engineExecutionContext.getProperties + .put(Configuration.EXECUTE_ERROR_CODE_INDEX.key, errorIndex.toString) + // jdbc执行任务重试，如果sql有被set进sql，会导致sql的index错位，这里会将日志打印的index进行减一，保证用户看的index是正常的，然后重试的errorIndex需要加一，保证重试的位置是正确的 + var newIndex = index + var newErrorIndex = errorIndex + if (adjustErrorIndexForSetScenarios(engineConnTask)) { + newIndex = index - 1 + newErrorIndex = errorIndex + 1 + } // 重试的时候如果执行过则跳过执行 - if (retryEnable && errorIndex > 0 && index < errorIndex) { - engineExecutionContext.appendStdout( - LogUtils.generateInfo( - s"aisql retry with errorIndex: ${errorIndex}, current sql index: ${index} will skip." + if (retryEnable && errorIndex > 0 && index < newErrorIndex) { + val code = codes(index).trim.toUpperCase() + val shouldSkip = !isContextStatement(code) + + if (shouldSkip) { + engineExecutionContext.appendStdout( + LogUtils.generateInfo( + s"task retry with errorIndex: ${errorIndex}, current sql index: ${newIndex} will skip." + ) ) - ) - executeFlag = false + executeFlag = false + } else { + if (newIndex >= 0) { + engineExecutionContext.appendStdout( + LogUtils.generateInfo( + s"task retry with errorIndex: ${errorIndex}, current sql index: ${newIndex} is a context statement, will execute." + ) + ) + } + } } if (executeFlag) { val code = codes(index) @@ -289,21 +315,23 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) response match { case e: ErrorExecuteResponse => val props: util.Map[String, String] = engineCreationContext.getOptions - val aiSqlEnable: String = props.getOrDefault("linkis.ai.sql.enable", "false").toString + val taskRetry: String = + props.getOrDefault("linkis.task.retry.switch", "false").toString val retryNum: Int = - Integer.valueOf(props.getOrDefault("linkis.ai.retry.num", "0").toString) + Integer.valueOf(props.getOrDefault("linkis.task.retry.num", "0").toString) - if (retryEnable && !props.isEmpty && "true".equals(aiSqlEnable) && retryNum > 0) { + if (retryEnable && !props.isEmpty && "true".equals(taskRetry) && retryNum > 0) { logger.info( - s"aisql execute failed, with index: ${index} retryNum: ${retryNum}, and will retry", + s"task execute failed, with index: ${index} retryNum: ${retryNum}, and will retry", e.t ) engineExecutionContext.appendStdout( LogUtils.generateInfo( - s"aisql execute failed, with index: ${index} retryNum: ${retryNum}, and will retry" + s"task execute failed, with index: ${index} retryNum: ${retryNum}, and will retry" ) ) - engineConnTask.getProperties.put("execute.error.code.index", index.toString) + engineConnTask.getProperties + .put(Configuration.EXECUTE_ERROR_CODE_INDEX.key, index.toString) return ErrorRetryExecuteResponse(e.message, index, e.t) } else { failedTasks.increase() @@ -320,7 +348,14 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) e.getOutput.substring(0, outputPrintLimit) } else e.getOutput engineExecutionContext.appendStdout(output) - if (StringUtils.isNotBlank(e.getOutput)) engineExecutionContext.sendResultSet(e) + if (StringUtils.isNotBlank(e.getOutput)) { + engineConnTask.getProperties + .put( + Configuration.EXECUTE_RESULTSET_ALIAS_NUM.key, + engineExecutionContext.getAliasNum.toString + ) + engineExecutionContext.sendResultSet(e) + } case _: IncompleteExecuteResponse => incomplete ++= incompleteSplitter } @@ -369,6 +404,31 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) executeResponse match { case successExecuteResponse: SuccessExecuteResponse => transformTaskStatus(engineConnTask, ExecutionNodeStatus.Succeed) + case ErrorRetryExecuteResponse(message, index, throwable) => + ComputationEngineUtils.sendToEntrance( + engineConnTask, + ResponseTaskError(engineConnTask.getTaskId, message) + ) + logger.warn( + s"The task begins executing retries,jobId:${engineConnTask.getTaskId},index:${index} ,message:${message}", + throwable + ) + + val currentAliasNum = Integer.valueOf( + engineConnTask.getProperties + .getOrDefault(Configuration.EXECUTE_RESULTSET_ALIAS_NUM.key, "0") + .toString + ) + + ComputationEngineUtils.sendToEntrance( + engineConnTask, + new ResponseTaskStatusWithExecuteCodeIndex( + engineConnTask.getTaskId, + ExecutionNodeStatus.Failed, + index, + currentAliasNum + ) + ) case errorExecuteResponse: ErrorExecuteResponse => listenerBusContext.getEngineConnSyncListenerBus.postToAll( TaskResponseErrorEvent(engineConnTask.getTaskId, errorExecuteResponse.message) @@ -404,6 +464,22 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) def getProgressInfo(taskID: String): Array[JobProgressInfo] + /** + * 调整错误索引：直接匹配三种SET语句场景 因为SET语句会被解析器视为第一条SQL + */ + protected def adjustErrorIndexForSetScenarios(engineConnTask: EngineConnTask): Boolean = { + val executionCode = engineConnTask.getCode + val engineTypeLabel = engineConnTask.getLables.find(_.isInstanceOf[EngineTypeLabel]).get + val engineType = engineTypeLabel.asInstanceOf[EngineTypeLabel].getEngineType + var result = false + if (executionCode != null && engineType.equals(EngineType.JDBC.toString)) { + val upperCode = executionCode.toUpperCase().trim + val jdbcSetPrefixes = ComputationExecutorConf.JDBC_SET_STATEMENT_PREFIXES.getValue.split(",") + result = jdbcSetPrefixes.exists(upperCode.startsWith) + } + result + } + protected def createEngineExecutionContext( engineConnTask: EngineConnTask ): EngineExecutionContext = { @@ -427,6 +503,22 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) engineExecutionContext.setJobId(engineConnTask.getTaskId) engineExecutionContext.getProperties.putAll(engineConnTask.getProperties) engineExecutionContext.setLabels(engineConnTask.getLables) + + val errorIndex: Int = Integer.valueOf( + engineConnTask.getProperties + .getOrDefault(Configuration.EXECUTE_ERROR_CODE_INDEX.key, "-1") + .toString + ) + if (errorIndex > 0) { + val savedAliasNum = Integer.valueOf( + engineConnTask.getProperties + .getOrDefault(Configuration.EXECUTE_RESULTSET_ALIAS_NUM.key, "0") + .toString + ) + engineExecutionContext.setResultSetNum(savedAliasNum) + logger.info(s"Restore aliasNum to $savedAliasNum for retry task") + } + engineExecutionContext } @@ -439,6 +531,18 @@ abstract class ComputationExecutor(val outputPrintLimit: Int = 1000) } } + /** + * 判断是否为上下文语句，重试时需要保留执行 + * + * @param code + * SQL代码（已转换为大写并去除首尾空格） + * @return + * true表示是上下文语句，false表示不是 + */ + private def isContextStatement(code: String): Boolean = { + ComputationExecutorConf.CONTEXT_STATEMENT_PREFIXES.getValue.split(",").exists(code.startsWith) + } + /** * job task log print task params info * diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHook.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHook.scala index 386cd8f0b54..454253748e0 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHook.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHook.scala @@ -54,7 +54,9 @@ abstract class PythonModuleLoad extends Logging { /** Abstract properties to be defined by the subclass */ protected val engineType: String protected val runType: RunType + protected def getEngineType(): String = engineType + protected def constructCode(pythonModuleInfo: PythonModuleInfoVO): String private def queryPythonModuleRpc( @@ -66,6 +68,7 @@ abstract class PythonModuleLoad extends Logging { .ask(RequestPythonModuleProtocol(userName, engineType)) .asInstanceOf[ResponsePythonModuleProtocol] .getModulesInfo() + // 使用Collections.sort()和Comparator进行排序 if (infoList != null && !infoList.isEmpty) { Collections.sort( @@ -83,12 +86,14 @@ abstract class PythonModuleLoad extends Logging { val engineCreationContext = EngineConnManager.getEngineConnManager.getEngineConn.getEngineCreationContext val user = engineCreationContext.getUser + var infoList: util.List[PythonModuleInfoVO] = Utils.tryAndWarn(queryPythonModuleRpc(user, getEngineType())) if (infoList == null) { logger.info("rpc get info is empty.") infoList = new util.ArrayList[PythonModuleInfoVO]() } + // 替换Viewfs if (IS_VIEW_FS_ENV.getValue) { infoList.asScala.foreach { info => @@ -101,6 +106,7 @@ abstract class PythonModuleLoad extends Logging { } } } else { + infoList.asScala.foreach { info => val path = info.getPath logger.info(s"hdfs python path: ${path}") @@ -109,8 +115,10 @@ abstract class PythonModuleLoad extends Logging { } } } + logger.info(s"${user} load python modules: ") infoList.asScala.foreach(l => logger.info(s"module name:${l.getName}, path:${l.getPath}\n")) + // 创建加载code val codes: mutable.Buffer[String] = infoList.asScala .filter { info => StringUtils.isNotEmpty(info.getPath) } @@ -142,6 +150,7 @@ abstract class PythonModuleLoad extends Logging { * An object capable of executing code in the current engine context. */ protected def loadPythonModules(labels: Array[Label[_]]): Unit = { + val codes = getLoadPythonModuleCode logger.info(s"codes length: ${codes.length}") if (null != codes && codes.nonEmpty) { @@ -152,6 +161,7 @@ abstract class PythonModuleLoad extends Logging { } else { logger.error(s"Failed to load python, executor is null") } + executor match { case computationExecutor: ComputationExecutor => executeFunctionCode(codes, computationExecutor) @@ -179,6 +189,7 @@ abstract class PythonModuleLoadEngineConnHook val labels = Array[Label[_]](codeLanguageLabel) loadPythonModules(labels) }(s"Failed to load Python Modules: ${engineType}") + } override def afterEngineServerStartFailed( @@ -203,8 +214,10 @@ abstract class PythonModuleLoadEngineConnHook // 加载PySpark的Python模块 class PythonSparkEngineHook extends PythonModuleLoadEngineConnHook { + // 设置engineType属性为"spark"，表示此挂钩适用于Spark数据处理引擎 override val engineType: String = "spark" + // 设置runType属性为RunType.PYSPARK，表示此挂钩将执行PySpark类型的代码 override protected val runType: RunType = RunType.PYSPARK @@ -222,8 +235,10 @@ class PythonSparkEngineHook extends PythonModuleLoadEngineConnHook { // 加载Python的Python模块 class PythonEngineHook extends PythonModuleLoadEngineConnHook { + // 设置engineType属性为"python"，表示此挂钩适用于python引擎 override val engineType: String = "python" + // 设置runType属性为RunType.PYTHON，表示此挂钩将执行python类型的代码 override protected val runType: RunType = RunType.PYTHON @@ -234,6 +249,7 @@ class PythonEngineHook extends PythonModuleLoadEngineConnHook { val engineCreationContext: EngineCreationContext = EngineConnManager.getEngineConnManager.getEngineConn.getEngineCreationContext val user: String = engineCreationContext.getUser + var loadCode: String = null logger.info(s"gen code in constructCode") Utils.tryAndWarn({ diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/UDFLoadEngineConnHook.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/UDFLoadEngineConnHook.scala index 91af2811a6f..594db0afe7c 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/UDFLoadEngineConnHook.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/hook/UDFLoadEngineConnHook.scala @@ -103,7 +103,13 @@ abstract class UDFLoad extends Logging { } logger.info("all udfs: ") + // 判断是否加载了特殊udf + val udfNames: String = ComputationExecutorConf.SPECIAL_UDF_NAMES.getValue udfInfos.foreach { l => + if (StringUtils.isNotBlank(udfNames) && udfNames.split(",").exists(l.getUdfName.contains)) { + logger.info(s"add spacial udf check for job with udfNames: {}", udfNames) + System.getProperties.put(ComputationExecutorConf.ONLY_SQL_USE_UDF_KEY, udfNames) + } logger.info( s"udfName:${l.getUdfName}, bml_resource_id:${l.getBmlResourceId}, bml_id:${l.getId}\n" ) diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/service/TaskExecutionServiceImpl.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/service/TaskExecutionServiceImpl.scala index 3739f47b541..cd0c58e49eb 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/service/TaskExecutionServiceImpl.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/service/TaskExecutionServiceImpl.scala @@ -249,17 +249,6 @@ class TaskExecutionServiceImpl sendToEntrance(task, ResponseTaskError(task.getTaskId, message)) logger.error(message, throwable) sendToEntrance(task, ResponseTaskStatus(task.getTaskId, ExecutionNodeStatus.Failed)) - case ErrorRetryExecuteResponse(message, index, throwable) => - sendToEntrance(task, ResponseTaskError(task.getTaskId, message)) - logger.error(message, throwable) - sendToEntrance( - task, - new ResponseTaskStatusWithExecuteCodeIndex( - task.getTaskId, - ExecutionNodeStatus.Failed, - index - ) - ) case _ => } LoggerUtils.removeJobIdMDC() diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/utlis/ComputationEngineUtils.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/utlis/ComputationEngineUtils.scala index fe595e8edc6..93d428d95a5 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/utlis/ComputationEngineUtils.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/main/scala/org/apache/linkis/engineconn/computation/executor/utlis/ComputationEngineUtils.scala @@ -17,11 +17,19 @@ package org.apache.linkis.engineconn.computation.executor.utlis +import org.apache.linkis.common.utils.{Logging, Utils} +import org.apache.linkis.engineconn.computation.executor.entity.EngineConnTask +import org.apache.linkis.governance.common.exception.engineconn.{ + EngineConnExecutorErrorCode, + EngineConnExecutorErrorException +} +import org.apache.linkis.protocol.message.RequestProtocol +import org.apache.linkis.rpc.Sender import org.apache.linkis.server.BDPJettyServerHelper import com.google.gson.{Gson, GsonBuilder} -object ComputationEngineUtils { +object ComputationEngineUtils extends Logging { def GSON: Gson = BDPJettyServerHelper.gson @@ -30,4 +38,24 @@ object ComputationEngineUtils { private val WORK_DIR_STR = "user.dir" def getCurrentWorkDir: String = System.getProperty(WORK_DIR_STR) + def sendToEntrance(task: EngineConnTask, msg: RequestProtocol): Unit = { + Utils.tryCatch { + var sender: Sender = null + if (null != task && null != task.getCallbackServiceInstance() && null != msg) { + sender = Sender.getSender(task.getCallbackServiceInstance()) + sender.send(msg) + } else { + // todo + logger.debug("SendtoEntrance error, cannot find entrance instance.") + } + } { t => + val errorMsg = s"SendToEntrance error. $msg" + t.getCause + logger.error(errorMsg, t) + throw new EngineConnExecutorErrorException( + EngineConnExecutorErrorCode.SEND_TO_ENTRANCE_ERROR, + errorMsg + ) + } + } + } diff --git a/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/test/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHookTest.scala b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/test/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHookTest.scala new file mode 100644 index 00000000000..83d83b9c1ab --- /dev/null +++ b/linkis-computation-governance/linkis-engineconn/linkis-computation-engineconn/src/test/scala/org/apache/linkis/engineconn/computation/executor/hook/PythonModuleLoadEngineConnHookTest.scala @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.engineconn.computation.executor.hook + +import org.apache.linkis.engineconn.common.creation.{ + DefaultEngineCreationContext, + EngineCreationContext +} +import org.apache.linkis.engineconn.common.engineconn.DefaultEngineConn +import org.apache.linkis.manager.label.entity.engine.CodeLanguageLabel + +import org.junit.jupiter.api.Test +import org.mockito.Mockito.{mock, verify, when} + +// 单元测试案例 +class PythonModuleLoadEngineConnHookTest { + +// @Test +// def testAfterExecutionExecute(): Unit = { +// // 创建模拟对象 +// val mockEngineCreationContext = new DefaultEngineCreationContext +// val mockEngineConn = mock[DefaultEngineConn] +// val hook = new PythonSparkEngineHook +// +// // 设置模拟行为 +// var labels = new CodeLanguageLabel +// labels.setCodeType("spark") +// +// // 执行测试方法 +// hook.afterExecutionExecute(mockEngineCreationContext, mockEngineConn) +// +// } +// +// @Test +// def testAfterEngineServerStartFailed(): Unit = { +// // 创建模拟对象 +// val mockEngineCreationContext = mock[EngineCreationContext] +// val mockThrowable = mock[Throwable] +// val hook = new PythonSparkEngineHook +// +// // 设置模拟行为 +// var labels = new CodeLanguageLabel +// labels.setCodeType("spark") +// +// // 执行测试方法 +// hook.afterEngineServerStartFailed(mockEngineCreationContext, mockThrowable) +// +// } +// +// @Test +// def testBeforeCreateEngineConn(): Unit = { +// // 创建模拟对象 +// +// // 验证调用 +// +// } +// +// @Test +// def testBeforeExecutionExecute(): Unit = { +// // 创建模拟对象 +// val mockEngineCreationContext = mock[EngineCreationContext] +// val mockEngineConn = mock[DefaultEngineConn] +// val hook = new PythonSparkEngineHook +// +// // 执行测试方法 +// hook.beforeExecutionExecute(mockEngineCreationContext, mockEngineConn) +// +// } + +} diff --git a/linkis-computation-governance/linkis-engineconn/linkis-engineconn-executor/executor-core/src/main/scala/org/apache/linkis/engineconn/executor/ExecutorExecutionContext.scala b/linkis-computation-governance/linkis-engineconn/linkis-engineconn-executor/executor-core/src/main/scala/org/apache/linkis/engineconn/executor/ExecutorExecutionContext.scala index 7f70e21e21d..e259571f43d 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-engineconn-executor/executor-core/src/main/scala/org/apache/linkis/engineconn/executor/ExecutorExecutionContext.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-engineconn-executor/executor-core/src/main/scala/org/apache/linkis/engineconn/executor/ExecutorExecutionContext.scala @@ -127,4 +127,6 @@ trait ExecutorExecutionContext { def setResultSetNum(num: Int): Unit = aliasNum.set(num) + def getAliasNum: Int = aliasNum.get() + } diff --git a/linkis-computation-governance/linkis-engineconn/linkis-engineconn-plugin-core/src/main/scala/org/apache/linkis/manager/engineplugin/common/launch/process/JavaProcessEngineConnLaunchBuilder.scala b/linkis-computation-governance/linkis-engineconn/linkis-engineconn-plugin-core/src/main/scala/org/apache/linkis/manager/engineplugin/common/launch/process/JavaProcessEngineConnLaunchBuilder.scala index 5a47f662b9f..e46126523a7 100644 --- a/linkis-computation-governance/linkis-engineconn/linkis-engineconn-plugin-core/src/main/scala/org/apache/linkis/manager/engineplugin/common/launch/process/JavaProcessEngineConnLaunchBuilder.scala +++ b/linkis-computation-governance/linkis-engineconn/linkis-engineconn-plugin-core/src/main/scala/org/apache/linkis/manager/engineplugin/common/launch/process/JavaProcessEngineConnLaunchBuilder.scala @@ -119,15 +119,12 @@ abstract class JavaProcessEngineConnLaunchBuilder // first, add engineconn conf dirs. addPathToClassPath(environment, buildPath(Seq(variable(PWD), ENGINE_CONN_CONF_DIR_NAME))) // then, add LINKIS_CONF_DIR conf dirs. - addPathToClassPath(environment, buildPath(Seq(variable(LINKIS_HOME), "conf"))) + addPathToClassPath(environment, buildPath(Seq(EnvConfiguration.LINKIS_CONF_DIR.getValue))) // then, add engineconn libs. addPathToClassPath(environment, buildPath(Seq(variable(PWD), ENGINE_CONN_LIB_DIR_NAME + "/*"))) // then, add public modules. if (!enablePublicModule) { - addPathToClassPath( - environment, - buildPath(Seq(variable(LINKIS_HOME), LINKIS_PUBLIC_MODULE_PATH.getValue + "/*")) - ) + addPathToClassPath(environment, buildPath(Seq(LINKIS_PUBLIC_MODULE_PATH.getValue + "/*"))) } // finally, add the suitable properties key to classpath val taskClassPathFiles = EnvConfiguration.ENGINE_CONN_CLASSPATH_FILES.getValue( diff --git a/linkis-computation-governance/linkis-entrance/pom.xml b/linkis-computation-governance/linkis-entrance/pom.xml index bda458c3562..41905db94bd 100644 --- a/linkis-computation-governance/linkis-entrance/pom.xml +++ b/linkis-computation-governance/linkis-entrance/pom.xml @@ -62,25 +62,25 @@ org.apache.linkis - linkis-pes-client + linkis-computation-governance-common ${project.version} org.apache.linkis - linkis-computation-governance-common + linkis-computation-orchestrator ${project.version} org.apache.linkis - linkis-computation-orchestrator + linkis-io_file-client ${project.version} org.apache.linkis - linkis-io_file-client + linkis-pes-client ${project.version} @@ -94,6 +94,7 @@ org.apache.linkis linkis-ps-common-lock ${project.version} + compile diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/conf/EntranceSpringConfiguration.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/conf/EntranceSpringConfiguration.java index afd300a3fe3..300d71e0d7a 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/conf/EntranceSpringConfiguration.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/conf/EntranceSpringConfiguration.java @@ -147,10 +147,12 @@ public EntranceInterceptor[] entranceInterceptors() { new LogPathCreateInterceptor(), new StorePathEntranceInterceptor(), new ScalaCodeInterceptor(), + new PythonCodeCheckInterceptor(), new SQLLimitEntranceInterceptor(), new CommentInterceptor(), - // new SetTenantLabelInterceptor(), - new UserCreatorIPCheckInterceptor() + new SetTenantLabelInterceptor(), + new UserCreatorIPCheckInterceptor(), + new TaskRetryInterceptor() }; } diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/exception/EntranceErrorCode.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/exception/EntranceErrorCode.java index 065f436e0a9..d1835f2621f 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/exception/EntranceErrorCode.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/exception/EntranceErrorCode.java @@ -32,7 +32,9 @@ public enum EntranceErrorCode { JOB_UPDATE_FAILED(20016, "job update failed"), VARIABLE_NULL_EXCEPTION(20017, "variable is null"), USER_NULL_EXCEPTION(20018, "User information not obtained"), - USER_IP_EXCEPTION(20019, "User IP address is not configured"); + USER_IP_EXCEPTION(20019, "User IP address is not configured"), + METRICS_PARAMS_EXCEPTION(20020, "metricsParams is null"), + YARN_RESOURCE_YARN_PARAMS_EXCEPTION(20021, "yarnResource is null"); private int errCode; private String desc; diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/job/EntranceExecutionJob.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/job/EntranceExecutionJob.java index 1eb911ecec3..4c18b23b62a 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/job/EntranceExecutionJob.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/job/EntranceExecutionJob.java @@ -157,9 +157,15 @@ public ExecuteRequest jobToExecuteRequest() throws EntranceErrorException { String resultSetPathRoot = GovernanceCommonConf.RESULT_SET_STORE_PATH().getValue(runtimeMapTmp); if (!runtimeMapTmp.containsKey(GovernanceCommonConf.RESULT_SET_STORE_PATH().key())) { - String resultParentPath = CommonLogPathUtils.getResultParentPath(jobRequest); - CommonLogPathUtils.buildCommonPath(resultParentPath, true); - resultSetPathRoot = CommonLogPathUtils.getResultPath(jobRequest); + // 修复：任务重试背景下，10：59分提交任务执行，重试时时间变成11：00，重试任务会重新生成结果目录，导致查询结果集时，重试之前执行的结果集丢失 + // 新增判断：生成结果目录之前，判断任务之前是否生成结果集，生成过就复用 + if (org.apache.commons.lang3.StringUtils.isNotEmpty(jobRequest.getResultLocation())) { + resultSetPathRoot = jobRequest.getResultLocation(); + } else { + String resultParentPath = CommonLogPathUtils.getResultParentPath(jobRequest); + CommonLogPathUtils.buildCommonPath(resultParentPath, true); + resultSetPathRoot = CommonLogPathUtils.getResultPath(jobRequest); + } } Map jobMap = new HashMap(); diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceEngine.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceEngine.java index 7fdaf06eb27..85ef21a3b69 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceEngine.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceEngine.java @@ -19,7 +19,6 @@ import org.apache.linkis.common.conf.Configuration$; import org.apache.linkis.common.exception.ErrorException; -import org.apache.linkis.common.utils.JsonUtils; import org.apache.linkis.entrance.conf.EntranceConfiguration; import org.apache.linkis.entrance.exception.EntranceIllegalParamException; import org.apache.linkis.entrance.exception.EntranceRPCException; @@ -47,7 +46,6 @@ import java.util.List; import java.util.Map; -import com.fasterxml.jackson.core.JsonProcessingException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -146,11 +144,13 @@ public void persist(JobRequest jobReq) throws ErrorException { JOBREQUEST_NOT_NULL.getErrorCode(), JOBREQUEST_NOT_NULL.getErrorDesc()); } if (logger.isDebugEnabled()) { - try { - logger.debug("jobReq:" + JsonUtils.jackson().writeValueAsString(jobReq)); - } catch (JsonProcessingException e) { - logger.debug("convert jobReq to string with error:" + e.getMessage()); - } + logger.debug( + "Persisting job request - id: {}, reqId: {}, submitUser: {}, executeUser: {}, status: {}", + jobReq.getId(), + jobReq.getReqId(), + jobReq.getSubmitUser(), + jobReq.getExecuteUser(), + jobReq.getStatus()); } JobReqInsert jobReqInsert = new JobReqInsert(jobReq); diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceManager.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceManager.java index f256e112cde..39b3f58c71b 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceManager.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/persistence/QueryPersistenceManager.java @@ -29,7 +29,9 @@ import org.apache.linkis.governance.common.entity.job.JobRequest; import org.apache.linkis.manager.label.builder.factory.LabelBuilderFactoryContext; import org.apache.linkis.manager.label.entity.Label; +import org.apache.linkis.manager.label.entity.engine.EngineType; import org.apache.linkis.manager.label.entity.entrance.ExecuteOnceLabel; +import org.apache.linkis.manager.label.utils.LabelUtil; import org.apache.linkis.protocol.engine.JobProgressInfo; import org.apache.linkis.protocol.utils.TaskUtils; import org.apache.linkis.scheduler.executer.OutputExecuteResponse; @@ -154,36 +156,79 @@ public boolean onJobFailed( } boolean containsAny = false; + // 文本匹配 String errorDescArray = EntranceConfiguration.SUPPORTED_RETRY_ERROR_DESC(); + // 错误码匹配 String errorCodeArray = EntranceConfiguration.SUPPORTED_RETRY_ERROR_CODES(); - for (String keyword : errorDescArray.split(",")) { - if (errorDesc.contains(keyword.trim()) || errorCodeArray.contains(errorCode + "")) { - containsAny = true; - break; - } - } - - if (!containsAny) { - return false; - } - + // 正则匹配 + String errorDescRegex = EntranceConfiguration.SUPPORTED_RETRY_ERROR_DESC_REGEX(); + final EntranceJob entranceJob = (EntranceJob) job; + String engineType = LabelUtil.getEngineType(entranceJob.getJobRequest().getLabels()); AtomicBoolean canRetry = new AtomicBoolean(false); - String aiSqlKey = EntranceConfiguration.AI_SQL_KEY().key(); String retryNumKey = EntranceConfiguration.RETRY_NUM_KEY().key(); - final EntranceJob entranceJob = (EntranceJob) job; - - // 处理广播表 - String dataFrameKey = EntranceConfiguration.SUPPORT_ADD_RETRY_CODE_KEYS(); - if (containsAny(errorDesc, dataFrameKey)) { - entranceJob - .getJobRequest() - .setExecutionCode("set spark.sql.autoBroadcastJoinThreshold=-1; " + code); + if (engineType.equals(EngineType.JDBC().toString()) && StringUtils.isNotBlank(errorDescRegex)) { + // JDBC执行正则匹配 + for (String regex : errorDescRegex.split(",")) { + String trimmedRegex = regex.trim(); + if (StringUtils.isNotBlank(trimmedRegex)) { + try { + Pattern pattern = Pattern.compile(trimmedRegex, Pattern.CASE_INSENSITIVE); + if (pattern.matcher(errorDesc).find()) { + containsAny = true; + break; + } + } catch (Exception e) { + logger.warn("Invalid regex pattern: {}", trimmedRegex, e); + } + } + } + if (!containsAny) { + return false; + } + JobRequest jobRequest = entranceJob.getJobRequest(); + Pattern queryPattern = + Pattern.compile( + "Query timeout. Increase the query_timeout session variable and retry|Query exceeded time limit of (\\S+) seconds", + Pattern.CASE_INSENSITIVE); + Pattern newPlannerPattern = + Pattern.compile("StarRocks planner use long time", Pattern.CASE_INSENSITIVE); + Pattern queryQueuePendingPattern = + Pattern.compile("pending timeout", Pattern.CASE_INSENSITIVE); + if (queryPattern.matcher(errorDesc).find()) { + jobRequest.setExecutionCode("SET query_timeout = 1200;\n" + code); + } else if (newPlannerPattern.matcher(errorDesc).find()) { + jobRequest.setExecutionCode("SET new_planner_optimize_timeout = 300000;\n" + code); + } else if (queryQueuePendingPattern.matcher(errorDesc).find()) { + jobRequest.setExecutionCode("SET query_queue_pending_timeout_second = 3600;\n" + code); + } + } else { + for (String keyword : errorDescArray.split(",")) { + if (errorDesc.contains(keyword.trim()) || errorCodeArray.contains(errorCode + "")) { + containsAny = true; + break; + } + } + if (!containsAny) { + return false; + } + // 处理广播表 + String dataFrameKey = EntranceConfiguration.SUPPORT_ADD_RETRY_CODE_KEYS(); + if (containsAny(errorDesc, dataFrameKey) + && engineType.equals(EngineType.SPARK().toString())) { + entranceJob + .getJobRequest() + .setExecutionCode("set spark.sql.autoBroadcastJoinThreshold=-1; " + code); + } } Map startupMap = TaskUtils.getStartupMap(props); - // 只对 aiSql 做重试 - if ("true".equals(startupMap.get(aiSqlKey))) { + String isRetry = + startupMap + .getOrDefault(EntranceConfiguration.TASK_RETRY_SWITCH().key(), "false") + .toString(); + // 对 aiSql 和 starrocks 做重试 + if (Boolean.parseBoolean(isRetry)) { LinkisUtils.tryAndWarn( () -> { int retryNum = (int) startupMap.getOrDefault(retryNumKey, 1); diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java index f75586071eb..8a0d0393d55 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceLabelRestfulApi.java @@ -97,7 +97,7 @@ public Message updateRouteLabel(HttpServletRequest req) { insLabelRefreshRequest.setLabels(labels); insLabelRefreshRequest.setServiceInstance(Sender.getThisServiceInstance()); InstanceLabelClient.getInstance().refreshLabelsToInstance(insLabelRefreshRequest); - synchronized (offlineFlag) { + synchronized (offlineFlag) { // NOSONAR offlineFlag = true; } logger.info("Finished to modify the routelabel of entry to offline"); @@ -125,7 +125,7 @@ public Message backOnline(HttpServletRequest req) { InsLabelRemoveRequest insLabelRemoveRequest = new InsLabelRemoveRequest(); insLabelRemoveRequest.setServiceInstance(Sender.getThisServiceInstance()); InstanceLabelClient.getInstance().removeLabelsFromInstance(insLabelRemoveRequest); - synchronized (offlineFlag) { + synchronized (offlineFlag) { // NOSONAR offlineFlag = false; } logger.info("Finished to backonline"); diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java index 7b487352d58..0ffde329f9d 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceMetricRestfulApi.java @@ -95,6 +95,7 @@ public Message taskinfo( } } } + assert undoneTasks != null; Message resp = Message.ok("success") .data("taskNumber", undoneTasks.length) diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java index 57d2ed04d07..e7fb473a1f3 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/restful/EntranceRestfulApi.java @@ -132,7 +132,7 @@ public Message execute(HttpServletRequest req, @RequestBody Map + jobReqId + " in " + Sender.getThisServiceInstance().toString() - + ". \n Please wait it to be scheduled(您的任务已经提交，进入排队中，如果一直没有更新日志，是任务并发达到了限制，可以进行参数修改)"), + + ". \n Please wait it to be scheduled(您的任务已经提交，进入排队中，如果一直没有更新日志，是任务并发达到了限制，可以在ITSM提Linkis参数修改单)"), job); message = Message.ok(); message.setMethod("/api/entrance/execute"); @@ -186,7 +186,7 @@ public Message submit(HttpServletRequest req, @RequestBody Map j + jobReqId + " in " + Sender.getThisServiceInstance().toString() - + ". \n Please wait it to be scheduled(您的任务已经提交，进入排队中，如果一直没有更新日志，是任务并发达到了限制，可以进行参数修改)"), + + ". \n Please wait it to be scheduled(您的任务已经提交，进入排队中，如果一直没有更新日志，是任务并发达到了限制，可以在ITSM提Linkis参数修改单)"), job); String execID = ZuulEntranceUtils.generateExecID( @@ -815,6 +815,7 @@ public Message killJobs( "An exception occurred while killing the job, kill failed(kill job的时候出现了异常，kill失败)", t); message.setMethod("/api/entrance/" + id + "/kill"); + message.setStatus(1); } } messages.add(message); diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.java new file mode 100644 index 00000000000..ac180d1aa30 --- /dev/null +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.java @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.entrance.scheduler; + +import org.apache.linkis.scheduler.SchedulerContext; +import org.apache.linkis.scheduler.queue.Consumer; +import org.apache.linkis.scheduler.queue.Group; +import org.apache.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer; + +import java.util.concurrent.ExecutorService; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class EntranceFIFOUserConsumer extends FIFOUserConsumer { + + private static final Logger logger = LoggerFactory.getLogger(EntranceFIFOUserConsumer.class); + + public EntranceFIFOUserConsumer( + SchedulerContext schedulerContext, ExecutorService executeService, Group group) { + super(schedulerContext, executeService, group); + } + + @Override + public boolean runScheduleIntercept() { + Consumer[] consumers = getSchedulerContext().getOrCreateConsumerManager().listConsumers(); + int creatorRunningJobNum = 0; + + // APP_TEST_hadoop_hive or IDE_hadoop_hive + String groupNameStr = getGroup().getGroupName(); + String[] groupNames = groupNameStr.split("_"); + int length = groupNames.length; + if (length < 3) { + return true; + } + + // APP_TEST + int lastIndex = groupNameStr.lastIndexOf("_"); + int secondLastIndex = groupNameStr.lastIndexOf("_", lastIndex - 1); + String creatorName = groupNameStr.substring(0, secondLastIndex); + + // hive + String ecType = groupNames[length - 1]; + + for (Consumer consumer : consumers) { + String groupName = consumer.getGroup().getGroupName(); + if (groupName.startsWith(creatorName) && groupName.endsWith(ecType)) { + creatorRunningJobNum += consumer.getRunningEvents().length; + } + } + + int creatorECTypeMaxRunningJobs = + CreatorECTypeDefaultConf.getCreatorECTypeMaxRunningJobs(creatorName, ecType); + + if (logger.isDebugEnabled()) { + logger.debug( + "Creator: {} EC: {} there are currently:{} jobs running and maximum limit: {}", + creatorName, + ecType, + creatorRunningJobNum, + creatorECTypeMaxRunningJobs); + } + + if (creatorRunningJobNum > creatorECTypeMaxRunningJobs) { + logger.error( + "Creator: {} EC: {} there are currently:{} jobs running that exceed the maximum limit: {}", + creatorName, + ecType, + creatorRunningJobNum, + creatorECTypeMaxRunningJobs); + return false; + } else { + return true; + } + } +} diff --git a/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/scheduler/EntranceParallelConsumerManager.java b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/scheduler/EntranceParallelConsumerManager.java new file mode 100644 index 00000000000..98f0929ee9e --- /dev/null +++ b/linkis-computation-governance/linkis-entrance/src/main/java/org/apache/linkis/entrance/scheduler/EntranceParallelConsumerManager.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.entrance.scheduler; + +import org.apache.linkis.scheduler.queue.Group; +import org.apache.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer; +import org.apache.linkis.scheduler.queue.parallelqueue.ParallelConsumerManager; + +public class EntranceParallelConsumerManager extends ParallelConsumerManager { + + public EntranceParallelConsumerManager(int maxParallelismUsers, String schedulerName) { + super(maxParallelismUsers, schedulerName); + } + + @Override + public FIFOUserConsumer createConsumer(String groupName) { + Group group = getSchedulerContext().getOrCreateGroupFactory().getGroup(groupName); + return new EntranceFIFOUserConsumer(getSchedulerContext(), getOrCreateExecutorService(), group); + } +} diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala index 68735d23ae7..bb125859368 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceServer.scala @@ -18,6 +18,7 @@ package org.apache.linkis.entrance import org.apache.linkis.common.ServiceInstance +import org.apache.linkis.common.conf.TimeType import org.apache.linkis.common.exception.{ErrorException, LinkisException, LinkisRuntimeException} import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{Logging, Utils} @@ -31,13 +32,14 @@ import org.apache.linkis.entrance.job.EntranceExecutionJob import org.apache.linkis.entrance.log.LogReader import org.apache.linkis.entrance.parser.ParserUtils import org.apache.linkis.entrance.timeout.JobTimeoutManager -import org.apache.linkis.entrance.utils.JobHistoryHelper +import org.apache.linkis.entrance.utils.{EntranceUtils, JobHistoryHelper} import org.apache.linkis.governance.common.conf.GovernanceCommonConf import org.apache.linkis.governance.common.entity.job.JobRequest import org.apache.linkis.governance.common.protocol.task.RequestTaskKill import org.apache.linkis.governance.common.utils.LoggerUtils import org.apache.linkis.manager.common.protocol.engine.EngineStopRequest import org.apache.linkis.manager.label.entity.entrance.ExecuteOnceLabel +import org.apache.linkis.manager.label.utils.LabelUtil import org.apache.linkis.protocol.constants.TaskConstant import org.apache.linkis.protocol.utils.TaskUtils import org.apache.linkis.rpc.Sender @@ -54,14 +56,19 @@ import org.apache.commons.lang3.StringUtils import org.apache.commons.lang3.exception.ExceptionUtils import java.text.{MessageFormat, SimpleDateFormat} +import java.time.Instant +import java.time.format.DateTimeFormatter import java.util import java.util.Date -import java.util.concurrent.TimeUnit +import java.util.concurrent.{ConcurrentHashMap, TimeUnit} import scala.collection.JavaConverters._ abstract class EntranceServer extends Logging { + // Map to track which jobs have been diagnosed + private val diagnosedJobs = new ConcurrentHashMap[String, Boolean]() + private var entranceWebSocketService: Option[EntranceWebSocketService] = None private val jobTimeoutManager: JobTimeoutManager = new JobTimeoutManager() @@ -762,6 +769,113 @@ abstract class EntranceServer extends Logging { EntranceConfiguration.ENTRANCE_TASK_TIMEOUT_SCAN.getValue.toLong, TimeUnit.MILLISECONDS ) + + Utils.defaultScheduler.scheduleAtFixedRate( + new Runnable() { + override def run(): Unit = { + // 新增任务诊断检测逻辑 + if (EntranceConfiguration.TASK_DIAGNOSIS_ENABLE) { + logger.info("Start to check tasks for diagnosis") + val undoneTask = getAllUndoneTask(null, null) + val diagnosisTime = System.currentTimeMillis() - new TimeType( + EntranceConfiguration.TASK_DIAGNOSIS_TIMEOUT + ).toLong + undoneTask + .filter { job => + try { + val engineType = LabelUtil.getEngineType(job.getJobRequest.getLabels) + val jobMetrics = + Option(JobHistoryHelper.getTaskByTaskID(job.getJobRequest.getId).getMetrics) + val startTime = + if (jobMetrics.exists(_.containsKey(TaskConstant.JOB_RUNNING_TIME))) { + val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ssZ") + val instant = Instant.from( + formatter.parse(jobMetrics.get.get(TaskConstant.JOB_RUNNING_TIME).toString) + ) + instant.toEpochMilli + } else { + 0L + } + engineType.contains( + EntranceConfiguration.TASK_DIAGNOSIS_ENGINE_TYPE + ) && startTime != 0 && startTime < diagnosisTime && !diagnosedJobs.containsKey( + job.getJobRequest.getId.toString + ) + } catch { + case t: Throwable => + logger.error(s"Failed to check task for diagnosis, reason: ${t.getMessage}", t) + false + } + } + .foreach { job => + val jobId = job.getJobRequest.getId + try { + // 检查并设置诊断标记，确保每个任务只被诊断一次 + diagnosedJobs.putIfAbsent(jobId.toString, true) + // 调用Doctoris诊断系统 + logger.info(s"Start to diagnose spark job $jobId") + job match { + case entranceJob: EntranceJob => + // 调用doctoris实时诊断API + job.getLogListener.foreach( + _.onLogUpdate( + job, + LogUtils.generateInfo( + s"Start diagnosing Spark task as task execution time exceeds 5 minutes,jobId:$jobId" + ) + ) + ) + val response = + EntranceUtils.taskRealtimeDiagnose(entranceJob.getJobRequest, null) + logger.info( + s"Finished to diagnose job $jobId, result: ${response.result}, reason: ${response.reason}" + ) + // 更新诊断信息 + if (response.success) { + // 构造诊断更新请求 + JobHistoryHelper.addDiagnosis(job.getJobRequest.getId, response.result) + logger.info(s"Successfully updated diagnosis for job ${job.getId()}") + } else { + // 更新诊断失败信息 + JobHistoryHelper + .addDiagnosis(job.getJobRequest.getId, s"Doctoris 诊断服务异常，请联系管理人员排查！") + } + job.getLogListener.foreach( + _.onLogUpdate( + job, + LogUtils.generateInfo( + s"Finished diagnosing task,This decision took ${response.duration} seconds" + ) + ) + ) + case _ => + logger.warn(s"Job $jobId is not an EntranceJob, skip diagnosis") + } + } catch { + case t: Throwable => + logger.warn(s"Diagnose job ${job.getId()} failed. ${t.getMessage}", t) + // 如果诊断失败，移除标记，允许重试 + diagnosedJobs.remove(jobId.toString) + } + logger.info("Finished to check Spark tasks for diagnosis") + } + // 定期清理diagnosedJobs，只保留未完成任务的记录 + val undoneJobIds = undoneTask.map(_.getJobRequest.getId.toString()).toSet + val iterator = diagnosedJobs.keySet().iterator() + while (iterator.hasNext) { + val jobId = iterator.next() + if (!undoneJobIds.contains(jobId)) { + iterator.remove() + } + } + logger.info(s"Cleaned diagnosedJobs cache, current size: ${diagnosedJobs.size()}") + } + } + }, + new TimeType(EntranceConfiguration.TASK_DIAGNOSIS_TIMEOUT_SCAN).toLong, + new TimeType(EntranceConfiguration.TASK_DIAGNOSIS_TIMEOUT_SCAN).toLong, + TimeUnit.MILLISECONDS + ) } if (timeoutCheck) { diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceWebSocketService.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceWebSocketService.scala index b5339c9e2e3..375d7c14a72 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceWebSocketService.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/EntranceWebSocketService.scala @@ -364,13 +364,8 @@ class EntranceWebSocketService } private def concatLog(length: Int, log: String, flag: StringBuilder, all: StringBuilder): Unit = { - if (length == 1) { - flag ++= log ++= "\n" - all ++= log ++= "\n" - } else { - flag ++= log ++= "\n" - all ++= log ++= "\n" - } + flag ++= log ++= "\n" + all ++= log ++= "\n" } /** diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala index c129c37a8df..16ba701f85a 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/conf/EntranceConfiguration.scala @@ -301,6 +301,12 @@ object EntranceConfiguration { "Spark application has already stopped,Spark application sc has already stopped,Failed to allocate a page,dataFrame to local exception,org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator" ).getValue + val SUPPORTED_RETRY_ERROR_DESC_REGEX = + CommonVars( + "linkis.entrance.supported.retry.error.desc.regex", + "Query timeout.*,Query exceeded time limit.*,Memory of process exceed limit.*,Backend node not found,Connection reset by peer,StarRocks planner use long time,pending timeout" + ).getValue + val SUPPORT_ADD_RETRY_CODE_KEYS = CommonVars( "linkis.entrance.supported.add.retry.code.keys", @@ -320,13 +326,19 @@ object EntranceConfiguration { CommonVars[String]("linkis.ai.sql.hive.template.keys", "hive,mapreduce").getValue val AI_SQL_CREATORS: String = - CommonVars[String]("linkis.ai.sql.support.creators", "IDE,MCP").getValue + CommonVars[String]("linkis.ai.sql.support.creators", "IDE,MCP,nodeexecution").getValue val AI_SQL_KEY: CommonVars[String] = CommonVars[String]("linkis.ai.sql.enable", "true") + val TASK_RETRY_SWITCH: CommonVars[Boolean] = + CommonVars[Boolean]("linkis.task.retry.switch", false) + + val TASK_RETRY_CODE_TYPE: String = + CommonVars[String]("linkis.task.retry.code.type", "aisql,jdbc").getValue + val RETRY_NUM_KEY: CommonVars[Int] = - CommonVars[Int]("linkis.ai.retry.num", 1) + CommonVars[Int]("linkis.task.retry.num", 1) val AI_SQL_RETRY_ONCE: CommonVars[Boolean] = CommonVars[Boolean]("linkis.ai.sql.once.enable", true) @@ -354,6 +366,11 @@ object EntranceConfiguration { "/api/v1/external/plaintext/diagnose?app_id=$app_id×tamp=$timestamp&nonce=$nonce&signature=$signature" ).getValue + val DOCTOR_REALTIME_DIAGNOSE_URL = CommonVars( + "linkis.entrance.doctor.realtime.diagnose.api", + "/api/v1/external/realtime/diagnose?app_id=$app_id×tamp=$timestamp&nonce=$nonce&signature=$signature" + ).getValue + val DOCTOR_SIGNATURE_TOKEN = CommonVars("linkis.doctor.signature.token", "").getValue val DOCTOR_NONCE = CommonVars.apply("linkis.doctor.signature.nonce", "").getValue @@ -365,10 +382,28 @@ object EntranceConfiguration { val AI_SQL_DYNAMIC_ENGINE_SWITCH = CommonVars("linkis.aisql.dynamic.engine.type.switch", false).getValue - val DOCTOR_REQUEST_TIMEOUT = CommonVars("linkis.aisql.doctor.http.timeout", 30000).getValue + val DOCTOR_REQUEST_TIMEOUT = CommonVars("linkis.aisql.doctor.http.timeout", 300000).getValue val DOCTOR_HTTP_MAX_CONNECT = CommonVars("linkis.aisql.doctor.http.max.connect", 20).getValue + // StarRocks engine switch configurations + val AISQL_STARROCKS_SWITCH = CommonVars("linkis.aisql.starrocks.switch", false) + + val AISQL_DEFAULT_STARROCKS_ENGINE_TYPE = + CommonVars("linkis.aisql.default.starrocks.engine.type", "jdbc-4").getValue + + val AISQL_STARROCKS_TEMPLATE_KEYS = + CommonVars("linkis.aisql.starrocks.template.keys", "starrocks") + + val AISQL_STARROCKS_DATASOURCE_PREFIX = + CommonVars("linkis.aisql.starrocks.datasource.prefix", "starrocks_") + + val AISQL_STARROCKS_WHITELIST_USERS = + CommonVars("linkis.aisql.starrocks.whitelist.users", "") + + val AISQL_STARROCKS_WHITELIST_DEPARTMENTS = + CommonVars("linkis.aisql.starrocks.whitelist.departments", "") + val SPARK_EXECUTOR_CORES = CommonVars.apply("spark.executor.cores", "2"); var SPARK_EXECUTOR_MEMORY = CommonVars.apply("spark.executor.memory", "6G"); @@ -404,6 +439,17 @@ object EntranceConfiguration { CommonVars[String]("linkis.doctor.sensitive.sql.check.whitelist", "").getValue var DOCTOR_SENSITIVE_SQL_CHECK_ENGINETYPE = - CommonVars[String]("linkis.doctor.sensitive.sql.check.engine.type", "hive,spark").getValue + CommonVars[String]("linkis.doctor.sensitive.sql.check.engine.type", "hive,spark,jdbc").getValue + + // 任务诊断配置 + val TASK_DIAGNOSIS_ENABLE = CommonVars[Boolean]("linkis.task.diagnosis.enable", false).getValue + + val TASK_DIAGNOSIS_ENGINE_TYPE = + CommonVars[String]("linkis.task.diagnosis.engine.type", "spark").getValue + + val TASK_DIAGNOSIS_TIMEOUT = CommonVars[String]("linkis.task.diagnosis.timeout", "5m").getValue + + val TASK_DIAGNOSIS_TIMEOUT_SCAN = + CommonVars("linkis.task.diagnosis.timeout.scan", "2m").getValue } diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/DefaultEntranceExecutor.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/DefaultEntranceExecutor.scala index b63734279c1..fd7430d5b20 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/DefaultEntranceExecutor.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/DefaultEntranceExecutor.scala @@ -17,6 +17,7 @@ package org.apache.linkis.entrance.execute +import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{Logging, Utils} import org.apache.linkis.entrance.exception.{EntranceErrorCode, EntranceErrorException} @@ -252,7 +253,8 @@ class DefaultEntranceExecutor(id: Long) if (rte.errorIndex >= 0) { logger.info(s"tasks execute error with error index: ${rte.errorIndex}") val newParams: util.Map[String, AnyRef] = new util.HashMap[String, AnyRef]() - newParams.put("execute.error.code.index", rte.errorIndex.toString) + newParams.put(Configuration.EXECUTE_ERROR_CODE_INDEX.key, rte.errorIndex.toString) + newParams.put(Configuration.EXECUTE_RESULTSET_ALIAS_NUM.key, rte.aliasNum.toString) LogUtils.generateInfo( s"tasks execute error with error index: ${rte.errorIndex} and will retry." ) diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/EntranceExecutorManager.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/EntranceExecutorManager.scala index 4e7ca79367c..7a353a81954 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/EntranceExecutorManager.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/EntranceExecutorManager.scala @@ -39,8 +39,6 @@ abstract class EntranceExecutorManager(groupFactory: GroupFactory) private val idGenerator = new AtomicLong(0) - def getOrCreateInterceptors(): Array[ExecuteRequestInterceptor] - override def delete(executor: Executor): Unit = { if (null != executor) { executor.close() @@ -92,15 +90,14 @@ abstract class EntranceExecutorManager(groupFactory: GroupFactory) job.getJobRequest match { case jobReq: JobRequest => val entranceEntranceExecutor = - new DefaultEntranceExecutor(jobReq.getId) - if (EntranceConfiguration.LINKIS_ENTRANCE_SKIP_ORCHESTRATOR) { - new SimpleEntranceExecutor( - jobReq.getId, - SimpleExecuteBusContext.getOrchestratorListenerBusContext() - ) - } else { - new DefaultEntranceExecutor(jobReq.getId) - } + if (EntranceConfiguration.LINKIS_ENTRANCE_SKIP_ORCHESTRATOR) { + new SimpleEntranceExecutor( + jobReq.getId, + SimpleExecuteBusContext.getOrchestratorListenerBusContext() + ) + } else { + new DefaultEntranceExecutor(jobReq.getId) + } // getEngineConn Executor job.getLogListener.foreach( _.onLogUpdate( @@ -109,13 +106,6 @@ abstract class EntranceExecutorManager(groupFactory: GroupFactory) ) ) jobReq.setUpdatedTime(new Date(System.currentTimeMillis())) - - /** - * // val engineConnExecutor = engineConnManager.getAvailableEngineConnExecutor(mark) - * idToEngines.put(entranceEntranceExecutor.getId, entranceEntranceExecutor) - */ -// instanceToEngines.put(engineConnExecutor.getServiceInstance.getInstance, entranceEntranceExecutor) // todo -// entranceEntranceExecutor.setInterceptors(getOrCreateInterceptors()) // todo entranceEntranceExecutor case _ => throw new EntranceErrorException( diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/ExecuteRequestInterceptor.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/ExecuteRequestInterceptor.scala index 82305572984..7946da40d1a 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/ExecuteRequestInterceptor.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/ExecuteRequestInterceptor.scala @@ -65,42 +65,3 @@ object JobExecuteRequestInterceptor extends ExecuteRequestInterceptor { } } - -object ReconnectExecuteRequestInterceptor extends ExecuteRequestInterceptor { - val PROPERTY_EXEC_ID = "execId" - - override def apply(requestTask: RequestTask, executeRequest: ExecuteRequest): RequestTask = - executeRequest match { - case reconnect: ReconnectExecuteRequest => - requestTask.data(PROPERTY_EXEC_ID, reconnect.execId) - requestTask - case _ => requestTask - } - -} - -object StorePathExecuteRequestInterceptor extends ExecuteRequestInterceptor { - - override def apply(requestTask: RequestTask, executeRequest: ExecuteRequest): RequestTask = - executeRequest match { - case storePath: StorePathExecuteRequest => - requestTask.data(RequestTask.RESULT_SET_STORE_PATH, storePath.storePath) - requestTask - case _ => requestTask - } - -} - -object RuntimePropertiesExecuteRequestInterceptor extends ExecuteRequestInterceptor { - - override def apply(requestTask: RequestTask, executeRequest: ExecuteRequest): RequestTask = - executeRequest match { - case runtime: RuntimePropertiesExecuteRequest => - runtime.properties.asScala.foreach { case (k, v) => - requestTask.data(k, v) - } - requestTask - case _ => requestTask - } - -} diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/impl/EntranceExecutorManagerImpl.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/impl/EntranceExecutorManagerImpl.scala index a251c56de7b..57de9cc0cbf 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/impl/EntranceExecutorManagerImpl.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/impl/EntranceExecutorManagerImpl.scala @@ -25,13 +25,5 @@ import org.apache.linkis.scheduler.queue.GroupFactory class EntranceExecutorManagerImpl(groupFactory: GroupFactory) extends EntranceExecutorManager(groupFactory) { - override def getOrCreateInterceptors(): Array[ExecuteRequestInterceptor] = Array( - JobExecuteRequestInterceptor, - LabelExecuteRequestInterceptor, - ReconnectExecuteRequestInterceptor, - StorePathExecuteRequestInterceptor, - RuntimePropertiesExecuteRequestInterceptor - ) - override def setExecutorListener(engineListener: ExecutorListener): Unit = {} } diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/simple/EngineLockListener.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/simple/SimpleExecuteBusContext.scala similarity index 100% rename from linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/simple/EngineLockListener.scala rename to linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/execute/simple/SimpleExecuteBusContext.scala diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala index b457287e3fd..9beef5d195f 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/AISQLTransformInterceptor.scala @@ -21,6 +21,7 @@ import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{Logging, Utils} import org.apache.linkis.common.utils.CodeAndRunTypeUtils.LANGUAGE_TYPE_AI_SQL +import org.apache.linkis.datasourcemanager.common.domain.DataSource import org.apache.linkis.entrance.conf.EntranceConfiguration import org.apache.linkis.entrance.conf.EntranceConfiguration._ import org.apache.linkis.entrance.interceptor.EntranceInterceptor @@ -52,6 +53,7 @@ class AISQLTransformInterceptor extends EntranceInterceptor with Logging { val sqlLanguage: String = LANGUAGE_TYPE_AI_SQL val sparkEngineType: String = AI_SQL_DEFAULT_SPARK_ENGINE_TYPE val hiveEngineType: String = AI_SQL_DEFAULT_HIVE_ENGINE_TYPE + val starrocksEngineType: String = AISQL_DEFAULT_STARROCKS_ENGINE_TYPE val labels: util.List[Label[_]] = jobRequest.getLabels val codeType: String = LabelUtil.getCodeType(labels) // engineType and creator have been verified in LabelCheckInterceptor. @@ -105,24 +107,110 @@ class AISQLTransformInterceptor extends EntranceInterceptor with Logging { ) } else { logger.info(s"start intelligent selection execution engine for ${jobRequest.getId}") - val engineType: String = - EntranceUtils.getDynamicEngineType(jobRequest.getExecutionCode, logAppender) + + /** + * Check for StarRocks engine switch if feature is enabled Priority: runtime parameters > + * script comment > template configuration + */ + var forceEngineType: String = null + if (AISQL_STARROCKS_SWITCH.getValue) { + // 1. Check runtime parameters + val runtimeEngineType = getRuntimeEngineType(jobRequest.getParams) + + // 2. Check script comment + val scriptEngineType = if (runtimeEngineType == null) { + TemplateConfUtils.getCustomEngineType( + jobRequest.getExecutionCode, + org.apache.linkis.common.utils.CodeAndRunTypeUtils.getLanguageTypeByCodeType(codeType) + ) + } else null + + // Determine final engine type + val targetEngineType = Option(runtimeEngineType) + .orElse(Option(scriptEngineType)) + .orNull + + // If StarRocks engine is specified + if ("starrocks".equalsIgnoreCase(targetEngineType)) { + // Check whitelist + val (user, creator) = LabelUtil.getUserCreator(jobRequest.getLabels) + if (!isUserInStarRocksWhitelist(user)) { + logger.warn( + s"User $user is not in StarRocks whitelist for task ${jobRequest.getId}, using default engine selection" + ) + logAppender.append( + LogUtils.generateWarn( + s"User $user is not in StarRocks whitelist, using default engine selection\n" + ) + ) + } else { + forceEngineType = "starrocks" + } + } + } + val engineType: String = { + EntranceUtils.getDynamicEngineType( + jobRequest.getExecutionCode, + logAppender, + forceEngineType + ) + } if ("hive".equals(engineType)) { changeEngineLabel(hiveEngineType, labels) - logAppender.append( - LogUtils.generateInfo(s"use $hiveEngineType by intelligent selection.\n") - ) currentEngineType = hiveEngineType + } else if ("starrocks".equals(engineType)) { + + Utils.tryCatch { + val dataSource: DataSource = EntranceUtils.getDatasourceByDatasourceTypeAndUser( + "starrocks", + jobRequest.getSubmitUser, + jobRequest.getExecuteUser + ) + if (dataSource != null) { + val dsParams: util.Map[String, AnyRef] = new util.HashMap[String, AnyRef]() + dsParams.put("wds.linkis.engine.runtime.datasource", dataSource.getDataSourceName) + TaskUtils.addRuntimeMap(jobRequest.getParams, dsParams) + changeEngineLabel(starrocksEngineType, labels) + currentEngineType = starrocksEngineType + logAppender.append( + LogUtils.generateInfo( + s"use starrocks engine with datasource name : ${dataSource.getDataSourceName} \n" + ) + ) + } else { + // use hive for datasource not exists + changeEngineLabel(hiveEngineType, labels) + currentEngineType = hiveEngineType + logger.warn( + s"Failed to select starrocks engine, for user ${jobRequest.getExecuteUser} datasource not exists." + ) + logAppender.append( + LogUtils.generateInfo( + s"Failed to select starrocks engine, ${jobRequest.getExecuteUser} datasource does not exist. now use $currentEngineType \n" + ) + ) + } + } { t => + // use hive for exception + changeEngineLabel(hiveEngineType, labels) + currentEngineType = hiveEngineType + logger.warn("Failed to select starrocks engine: ", t) + logAppender.append( + LogUtils.generateInfo( + s"Failed to select starrocks engine, service exception. now use $currentEngineType \n" + ) + ) + } } else { changeEngineLabel(sparkEngineType, labels) - logAppender.append( - LogUtils.generateInfo(s"use $sparkEngineType by intelligent selection.\n") - ) currentEngineType = sparkEngineType } logger.info( s"end intelligent selection execution engine, and engineType is ${currentEngineType} for ${jobRequest.getId}." ) + logAppender.append( + LogUtils.generateInfo(s"use $currentEngineType by intelligent selection.\n") + ) EntranceUtils.dealsparkDynamicConf(jobRequest, logAppender, jobRequest.getParams) } @@ -179,4 +267,64 @@ class AISQLTransformInterceptor extends EntranceInterceptor with Logging { labels.add(newEngineTypeLabel) } + /** + * Get engine type from runtime parameters + * @param params + * job request parameters + * @return + * engine type, such as "starrocks", null if not found + */ + private def getRuntimeEngineType(params: util.Map[String, AnyRef]): String = { + if (params == null) return null + + val runtimeParams = TaskUtils.getRuntimeMap(params) + if (runtimeParams == null) return null + + val engineType = runtimeParams.get(TemplateConfUtils.confEngineTypeKey) + if (engineType != null) engineType.toString else null + } + + /** + * Check if user is in StarRocks whitelist + * @param submitUser + * the user who submits the task + * @return + * true if user is in whitelist or whitelist is empty (allow all users), false otherwise + */ + private def isUserInStarRocksWhitelist(submitUser: String): Boolean = { + val whitelistUsers = AISQL_STARROCKS_WHITELIST_USERS.getValue + val whitelistDepartments = AISQL_STARROCKS_WHITELIST_DEPARTMENTS.getValue + + // If both whitelists are empty, allow all users + if (StringUtils.isBlank(whitelistUsers) && StringUtils.isBlank(whitelistDepartments)) { + return true + } + + // Check user whitelist + if (StringUtils.isNotBlank(whitelistUsers)) { + val users = whitelistUsers.split(",").map(_.trim) + if (users.contains(submitUser)) { + logger.info(s"User $submitUser is in StarRocks whitelist (user)") + return true + } + } + + // Check department whitelist + if (StringUtils.isNotBlank(whitelistDepartments)) { + val userDepartmentId = EntranceUtils.getUserDepartmentId(submitUser) + if (StringUtils.isNotBlank(userDepartmentId)) { + val departments = whitelistDepartments.split(",").map(_.trim) + if (departments.contains(userDepartmentId)) { + logger.info( + s"User $submitUser (department: $userDepartmentId) is in StarRocks whitelist (department)" + ) + return true + } + } + } + + logger.warn(s"User $submitUser is not in StarRocks whitelist") + false + } + } diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/Explain.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/Explain.scala index 7760c11a080..60edb18ccde 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/Explain.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/Explain.scala @@ -22,11 +22,13 @@ import org.apache.linkis.common.exception.ErrorException import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{Logging, Utils} import org.apache.linkis.entrance.conf.EntranceConfiguration +import org.apache.linkis.entrance.conf.EntranceConfiguration.PYTHON_SAFE_CHECK_SWITCH import org.apache.linkis.entrance.exception.{EntranceErrorCode, EntranceIllegalParamException} import org.apache.linkis.entrance.interceptor.exception.{ PythonCodeCheckException, ScalaCodeCheckException } +import org.apache.linkis.entrance.utils.SafeUtils import org.apache.linkis.governance.common.entity.job.JobRequest import org.apache.commons.lang3.StringUtils @@ -121,8 +123,7 @@ object SQLExplain extends Explain { logAppender: java.lang.StringBuilder ): Unit = { val fixedCode: ArrayBuffer[String] = new ArrayBuffer[String]() - val tempCode1 = SQLCommentHelper.dealComment(executionCode) - val tempCode = SQLCommentHelper.replaceComment(tempCode1) + val tempCode = SQLCommentHelper.dealComment(executionCode) val isNoLimitAllowed = Utils.tryCatch { IDE_ALLOW_NO_LIMIT_REGEX.findFirstIn(executionCode).isDefined } { case e: Exception => @@ -218,7 +219,7 @@ object SQLExplain extends Explain { EntranceErrorCode.EXECUTION_CODE_ISNULL.getDesc ) } - logger.debug(s"after sql limit code is ${requestPersistTask.getExecutionCode}") + logger.debug(s"SQL limit applied.") } private def findRealSemicolonIndex(tempCode: String): Array[Int] = { @@ -282,7 +283,9 @@ object SQLExplain extends Explain { if (StringUtils.isEmpty(code)) { return false } + // 如果一段sql是 --xxx回车select * from default.users，那么他也是select语句 val realCode = cleanComment(code) + // 以前，在判断，对于select* from xxx这样的SQL时会出现问题的，但是这种语法hive是支持的 realCode.trim.split("\\s+")(0).toLowerCase(Locale.getDefault).contains("select") } @@ -409,58 +412,60 @@ object PythonExplain extends Explain { if (EntranceConfiguration.SKIP_AUTH.getHotValue()) { return true } - - CAN_PASS_CODES - .split(";") - .foreach(c => { - if (code.contains(c)) { - if ( - IMPORT_SYS_MOUDLE - .findAllIn(code) - .nonEmpty || FROM_SYS_IMPORT.findAllIn(code).nonEmpty - ) { - throw PythonCodeCheckException(20070, "can not use sys module") - } else if ( - IMPORT_OS_MOUDLE.findAllIn(code).nonEmpty || FROM_OS_IMPORT.findAllIn(code).nonEmpty - ) { - throw PythonCodeCheckException(20071, "can not use os module") - } else if ( - IMPORT_PROCESS_MODULE - .findAllIn(code) - .nonEmpty || FROM_MULTIPROCESS_IMPORT.findAllIn(code).nonEmpty - ) { - throw PythonCodeCheckException(20072, "can not use process module") - } else if (SC_STOP.findAllIn(code).nonEmpty) { - throw PythonCodeCheckException(20073, "You can not stop SparkContext, It's dangerous") - } else if (FROM_NUMPY_IMPORT.findAllIn(code).nonEmpty) { - throw PythonCodeCheckException(20074, "Numpy packages cannot be imported in this way") - } - } - }) - - code.split(System.lineSeparator()) foreach { code => - if (IMPORT_SYS_MOUDLE.findAllIn(code).nonEmpty || FROM_SYS_IMPORT.findAllIn(code).nonEmpty) { - throw PythonCodeCheckException(20070, "can not use sys module") - } else if ( - IMPORT_OS_MOUDLE.findAllIn(code).nonEmpty || FROM_OS_IMPORT.findAllIn(code).nonEmpty - ) { - throw PythonCodeCheckException(20071, "can not use os moudle") - } else if ( - IMPORT_PROCESS_MODULE.findAllIn(code).nonEmpty || FROM_MULTIPROCESS_IMPORT - .findAllIn(code) - .nonEmpty - ) { - throw PythonCodeCheckException(20072, "can not use process module") - } else if ( - IMPORT_SUBPORCESS_MODULE.findAllIn(code).nonEmpty || FROM_SUBPROCESS_IMPORT - .findAllIn(code) - .nonEmpty - ) { - throw PythonCodeCheckException(20072, "can not use subprocess module") - } else if (SC_STOP.findAllIn(code).nonEmpty) { - throw PythonCodeCheckException(20073, "You can not stop SparkContext, It's dangerous") - } + if (PYTHON_SAFE_CHECK_SWITCH && (!SafeUtils.isCodeSafe(code))) { + throw PythonCodeCheckException(20074, "Invalid python code.(当前代码存在非法获取系统信息或执行非法命令等危险操作，禁止执行)") } +// CAN_PASS_CODES +// .split(";") +// .foreach(c => { +// if (code.contains(c)) { +// if ( +// IMPORT_SYS_MOUDLE +// .findAllIn(code) +// .nonEmpty || FROM_SYS_IMPORT.findAllIn(code).nonEmpty +// ) { +// throw PythonCodeCheckException(20070, "can not use sys module") +// } else if ( +// IMPORT_OS_MOUDLE.findAllIn(code).nonEmpty || FROM_OS_IMPORT.findAllIn(code).nonEmpty +// ) { +// throw PythonCodeCheckException(20071, "can not use os module") +// } else if ( +// IMPORT_PROCESS_MODULE +// .findAllIn(code) +// .nonEmpty || FROM_MULTIPROCESS_IMPORT.findAllIn(code).nonEmpty +// ) { +// throw PythonCodeCheckException(20072, "can not use process module") +// } else if (SC_STOP.findAllIn(code).nonEmpty) { +// throw PythonCodeCheckException(20073, "You can not stop SparkContext, It's dangerous") +// } else if (FROM_NUMPY_IMPORT.findAllIn(code).nonEmpty) { +// throw PythonCodeCheckException(20074, "Numpy packages cannot be imported in this way") +// } +// } +// }) +// +// code.split(System.lineSeparator()) foreach { code => +// if (IMPORT_SYS_MOUDLE.findAllIn(code).nonEmpty || FROM_SYS_IMPORT.findAllIn(code).nonEmpty) { +// throw PythonCodeCheckException(20070, "can not use sys module") +// } else if ( +// IMPORT_OS_MOUDLE.findAllIn(code).nonEmpty || FROM_OS_IMPORT.findAllIn(code).nonEmpty +// ) { +// throw PythonCodeCheckException(20071, "can not use os moudle") +// } else if ( +// IMPORT_PROCESS_MODULE.findAllIn(code).nonEmpty || FROM_MULTIPROCESS_IMPORT +// .findAllIn(code) +// .nonEmpty +// ) { +// throw PythonCodeCheckException(20072, "can not use process module") +// } else if ( +// IMPORT_SUBPORCESS_MODULE.findAllIn(code).nonEmpty || FROM_SUBPROCESS_IMPORT +// .findAllIn(code) +// .nonEmpty +// ) { +// throw PythonCodeCheckException(20072, "can not use subprocess module") +// } else if (SC_STOP.findAllIn(code).nonEmpty) { +// throw PythonCodeCheckException(20073, "You can not stop SparkContext, It's dangerous") +// } +// } true } diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TaskRetryInterceptor.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TaskRetryInterceptor.scala new file mode 100644 index 00000000000..78a2361cf47 --- /dev/null +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TaskRetryInterceptor.scala @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.entrance.interceptor.impl + +import org.apache.linkis.common.log.LogUtils +import org.apache.linkis.common.utils.CodeAndRunTypeUtils.LANGUAGE_TYPE_AI_SQL +import org.apache.linkis.common.utils.Logging +import org.apache.linkis.entrance.conf.EntranceConfiguration.{ + AI_SQL_CREATORS, + AI_SQL_KEY, + RETRY_NUM_KEY, + TASK_RETRY_CODE_TYPE, + TASK_RETRY_SWITCH +} +import org.apache.linkis.entrance.interceptor.EntranceInterceptor +import org.apache.linkis.governance.common.entity.job.JobRequest +import org.apache.linkis.manager.label.entity.Label +import org.apache.linkis.manager.label.utils.LabelUtil +import org.apache.linkis.protocol.utils.TaskUtils + +import java.{lang, util} + +/** + * 任务重试拦截器 用于根据任务类型和配置，动态为任务添加重试开关 在任务提交前对作业参数进行预处理，决定是否启用重试功能 + */ +class TaskRetryInterceptor extends EntranceInterceptor with Logging { + + override def apply(jobRequest: JobRequest, logAppender: lang.StringBuilder): JobRequest = { + // 获取AI SQL相关配置 + val aiSqlEnable: Boolean = "true".equals(AI_SQL_KEY.getValue) + val supportAISQLCreator: String = AI_SQL_CREATORS.toLowerCase() + + // 从标签提取任务元信息 + val labels: util.List[Label[_]] = jobRequest.getLabels + val codeType: String = LabelUtil.getCodeType(labels) + val creator: String = LabelUtil.getUserCreatorLabel(labels).getCreator + + // 全局重试开关开启时处理 + if (TASK_RETRY_SWITCH.getValue) { + val startMap: util.Map[String, AnyRef] = TaskUtils.getStartupMap(jobRequest.getParams) + TASK_RETRY_CODE_TYPE + .split(",") + .foreach(codeTypeConf => { + if (codeTypeConf.equals(codeType)) { + // 普通任务只需满足类型支持 + logAppender + .append(LogUtils.generateWarn(s"The StarRocks task will initiate a failed retry \n")) + startMap.put(TASK_RETRY_SWITCH.key, TASK_RETRY_SWITCH.getValue.asInstanceOf[AnyRef]) + startMap.put(RETRY_NUM_KEY.key, RETRY_NUM_KEY.getValue.asInstanceOf[AnyRef]) + } + }) + // 更新作业参数 + TaskUtils.addStartupMap(jobRequest.getParams, startMap) + } + + jobRequest + } + +} diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala index d7c95c964a3..8be809e4d45 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/TemplateConfUtils.scala @@ -55,6 +55,7 @@ object TemplateConfUtils extends Logging { val confTemplateNameKey = "ec.resource.name" val confFixedEngineConnLabelKey = "ec.fixed.sessionId" + val confEngineTypeKey = "ec.engine.type" /** * 按模板uuid缓存模板配置 @@ -233,6 +234,40 @@ object TemplateConfUtils extends Logging { templateConfName } + /** + * Get user-defined engine type from script code + * + * @param code + * :code + * @param languageType + * :sql,hql,python,scala + * @return + * String engine type, such as "starrocks", "spark", "hive", null if not found + */ + def getCustomEngineType(code: String, languageType: String): String = { + if (StringUtils.isBlank(code) || StringUtils.isBlank(languageType)) { + return null + } + + val confPattern = languageType.toLowerCase match { + case x if x.contains("python") || x.contains("shell") => + s"##@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s#]+)".r + case x if x.contains("scala") => + s"///@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s/]+)".r + case _ => + s"---@set\\s+${confEngineTypeKey}\\s*=\\s*([^\\s-]+)".r + } + + val codeRes = code.replaceAll("\r\n", "\n") + confPattern.findFirstMatchIn(codeRes) match { + case Some(m) => + val engineType = m.group(1).trim + logger.info(s"Get custom engine type: $engineType from script") + engineType + case None => null + } + } + def dealWithTemplateConf(jobRequest: JobRequest, logAppender: lang.StringBuilder): JobRequest = { jobRequest match { case requestPersistTask: JobRequest => @@ -245,7 +280,7 @@ object TemplateConfUtils extends Logging { // only for Creator:IDE, try to get template conf name from code string. eg:---@set ec.resource.name=xxxx val codeType = LabelUtil.getCodeType(jobRequest.getLabels) val (user, creator) = LabelUtil.getUserCreator(jobRequest.getLabels) - if (EntranceConfiguration.DEFAULT_REQUEST_APPLICATION_NAME.getValue.equals(creator)) { + if (EntranceConfiguration.DEFAULT_REQUEST_APPLICATION_NAME.getValue.contains(creator)) { templateName = getCustomTemplateConfName(jobRequest, codeType, logAppender) if (StringUtils.isNotBlank(templateName)) { logAppender.append( diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/VarSubstitutionInterceptor.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/VarSubstitutionInterceptor.scala index b761e20ae90..033fe7aab25 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/VarSubstitutionInterceptor.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/interceptor/impl/VarSubstitutionInterceptor.scala @@ -24,9 +24,13 @@ import org.apache.linkis.entrance.interceptor.EntranceInterceptor import org.apache.linkis.entrance.interceptor.exception.VarSubstitutionException import org.apache.linkis.governance.common.entity.job.JobRequest import org.apache.linkis.manager.label.utils.LabelUtil +import org.apache.linkis.protocol.utils.TaskUtils +import org.apache.linkis.server.toScalaMap import org.apache.commons.lang3.exception.ExceptionUtils +import java.util + /** * Description: For variable substitution(用于变量替换) */ @@ -46,6 +50,20 @@ class VarSubstitutionInterceptor extends EntranceInterceptor { logAppender.append( LogUtils.generateInfo("Variables substitution ended successfully") + "\n" ) + logAppender.append(LogUtils.generateInfo("Job variables is") + "\n") + logAppender.append( + "************************************Variable************************************" + "\n" + ) + val variableMap = TaskUtils + .getVariableMap(jobRequest.getParams) + .asInstanceOf[util.HashMap[String, String]] + variableMap.foreach { case (key, value) => + logAppender.append(s"$key=$value\n") + } + logAppender.append("\n"); + logAppender.append( + "************************************Variable************************************" + "\n" + ) // print code after variables substitution logAppender.append( LogUtils.generateInfo( diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/orchestrator/plugin/EntranceUserParallelOrchestratorPlugin.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/orchestrator/plugin/EntranceUserParallelOrchestratorPlugin.scala index 4a143a0160f..1a2056be250 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/orchestrator/plugin/EntranceUserParallelOrchestratorPlugin.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/orchestrator/plugin/EntranceUserParallelOrchestratorPlugin.scala @@ -50,7 +50,7 @@ class EntranceUserParallelOrchestratorPlugin extends UserParallelOrchestratorPlu private val configCache: LoadingCache[String, Integer] = CacheBuilder .newBuilder() .maximumSize(1000) - .expireAfterAccess(1, TimeUnit.HOURS) + .expireAfterAccess(EntranceConfiguration.USER_PARALLEL_REFLESH_TIME.getValue, TimeUnit.MINUTES) .expireAfterWrite(EntranceConfiguration.USER_PARALLEL_REFLESH_TIME.getValue, TimeUnit.MINUTES) .build(new CacheLoader[String, Integer]() { diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.scala deleted file mode 100644 index 26d8a60c4c3..00000000000 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceFIFOUserConsumer.scala +++ /dev/null @@ -1,106 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.linkis.entrance.scheduler - -import org.apache.linkis.common.utils.{Logging, Utils} -import org.apache.linkis.entrance.conf.EntranceConfiguration -import org.apache.linkis.entrance.job.EntranceExecutionJob -import org.apache.linkis.entrance.utils.JobHistoryHelper -import org.apache.linkis.scheduler.SchedulerContext -import org.apache.linkis.scheduler.queue.Group -import org.apache.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer - -import java.util -import java.util.concurrent.ExecutorService - -import scala.collection.JavaConverters.collectionAsScalaIterableConverter - -class EntranceFIFOUserConsumer( - schedulerContext: SchedulerContext, - executeService: ExecutorService, - private var group: Group -) extends FIFOUserConsumer(schedulerContext, executeService, group) - with Logging { - - override def loop(): Unit = { - // When offlineFlag=true, the unsubmitted tasks will be failover, and the running tasks will wait for completion. - // In this case,super.loop only submits the retry task, but the retry task can failover and speed up the entrance offline - // (当offlineFlag=true时，未提交任务会被故障转移，运行中任务会等待完成.此时super.loop只会提交重试任务，但是重试任务完全可以故障转移，加快entrance下线) - schedulerContext match { - case entranceSchedulerContext: EntranceSchedulerContext => - if ( - entranceSchedulerContext.getOfflineFlag && EntranceConfiguration.ENTRANCE_FAILOVER_RETRY_JOB_ENABLED.getValue - ) { - val jobs = scanAllRetryJobsAndRemove() - if (!jobs.isEmpty) { - val ids = new util.ArrayList[Long]() - jobs.asScala.foreach { - case entranceJob: EntranceExecutionJob => - entranceJob.getLogWriter.foreach(_.close()) - ids.add(entranceJob.getJobRequest.getId) - case _ => - } - JobHistoryHelper.updateBatchInstancesEmpty(ids) - } - Utils.tryQuietly(Thread.sleep(5000)) - return - } - case _ => - } - - // general logic - super.loop() - - } - - override def runScheduleIntercept: Boolean = { - val consumers = getSchedulerContext.getOrCreateConsumerManager.listConsumers - var creatorRunningJobNum = 0 - // APP_TEST_hadoop_hive or IDE_hadoop_hive - val groupNameStr = getGroup.getGroupName - val groupNames = groupNameStr.split("_") - val length = groupNames.length - if (length < 3) return true - // APP_TEST - val lastIndex = groupNameStr.lastIndexOf("_") - val secondLastIndex = groupNameStr.lastIndexOf("_", lastIndex - 1) - val creatorName = groupNameStr.substring(0, secondLastIndex) - // hive - val ecType = groupNames(length - 1) - for (consumer <- consumers) { - val groupName = consumer.getGroup.getGroupName - if (groupName.startsWith(creatorName) && groupName.endsWith(ecType)) { - creatorRunningJobNum += consumer.getRunningEvents.length - } - } - val creatorECTypeMaxRunningJobs = - CreatorECTypeDefaultConf.getCreatorECTypeMaxRunningJobs(creatorName, ecType) - if (logger.isDebugEnabled) { - logger.debug( - s"Creator: $creatorName EC:$ecType there are currently:$creatorRunningJobNum jobs running and maximum limit: $creatorECTypeMaxRunningJobs" - ) - } - if (creatorRunningJobNum > creatorECTypeMaxRunningJobs) { - logger.error( - s"Creator: $creatorName EC:$ecType there are currently:$creatorRunningJobNum jobs running that exceed the maximum limit: $creatorECTypeMaxRunningJobs" - ) - false - } else true - } - -} diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceParallelConsumerManager.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceParallelConsumerManager.scala deleted file mode 100644 index 789e2ca2b19..00000000000 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/scheduler/EntranceParallelConsumerManager.scala +++ /dev/null @@ -1,83 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.linkis.entrance.scheduler - -import org.apache.linkis.common.ServiceInstance -import org.apache.linkis.common.utils.Utils -import org.apache.linkis.entrance.conf.EntranceConfiguration -import org.apache.linkis.entrance.utils.EntranceUtils -import org.apache.linkis.instance.label.client.InstanceLabelClient -import org.apache.linkis.manager.label.builder.factory.LabelBuilderFactoryContext -import org.apache.linkis.manager.label.constant.{LabelKeyConstant, LabelValueConstant} -import org.apache.linkis.manager.label.entity.Label -import org.apache.linkis.manager.label.entity.route.RouteLabel -import org.apache.linkis.rpc.Sender -import org.apache.linkis.scheduler.queue.fifoqueue.FIFOUserConsumer -import org.apache.linkis.scheduler.queue.parallelqueue.{ParallelConsumerManager, ParallelGroup} - -import java.util -import java.util.concurrent.TimeUnit - -import scala.collection.JavaConverters._ - -class EntranceParallelConsumerManager(maxParallelismUsers: Int, schedulerName: String) - extends ParallelConsumerManager(maxParallelismUsers, schedulerName) { - - override protected def createConsumer(groupName: String): FIFOUserConsumer = { - val group = getSchedulerContext.getOrCreateGroupFactory.getGroup(groupName) - new EntranceFIFOUserConsumer(getSchedulerContext, getOrCreateExecutorService, group) - } - - if (EntranceConfiguration.ENTRANCE_GROUP_SCAN_ENABLED.getValue) { - Utils.defaultScheduler.scheduleAtFixedRate( - new Runnable { - override def run(): Unit = Utils.tryAndWarn { - // refresh all group maxAllowRunningJobs - refreshAllGroupMaxAllowRunningJobs(EntranceUtils.getRunningEntranceNumber()) - logger.info("Finished to refresh consumer group maxAllowRunningJobs") - } - }, - EntranceConfiguration.ENTRANCE_GROUP_SCAN_INIT_TIME.getValue, - EntranceConfiguration.ENTRANCE_GROUP_SCAN_INTERVAL.getValue, - TimeUnit.MILLISECONDS - ) - } - - def refreshAllGroupMaxAllowRunningJobs(validInsCount: Int): Unit = { - listConsumers() - .foreach(item => { - item.getGroup match { - case group: ParallelGroup => - val maxAllowRunningJobs = Math.round(group.getMaxRunningJobs / validInsCount) - group.setMaxAllowRunningJobs(maxAllowRunningJobs) - logger - .info( - "group {} refresh maxAllowRunningJobs => {}/{}={}", - Array( - group.getGroupName, - group.getMaxRunningJobs.toString, - validInsCount.toString, - maxAllowRunningJobs.toString - ): _* - ) - case _ => - } - }) - } - -} diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala index f397aeb2b21..3a78de2c9c4 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/EntranceUtils.scala @@ -21,9 +21,19 @@ import org.apache.linkis.common.ServiceInstance import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{Logging, SHAUtils, Utils} +import org.apache.linkis.datasource.client.impl.LinkisDataSourceRemoteClient +import org.apache.linkis.datasource.client.request.{ + GetInfoPublishedByDataSourceNameAction, + GetPublishedDataSourceByTypeAction +} +import org.apache.linkis.datasourcemanager.common.domain.DataSource import org.apache.linkis.entrance.conf.EntranceConfiguration import org.apache.linkis.entrance.errorcode.EntranceErrorCodeSummary -import org.apache.linkis.entrance.exception.EntranceRPCException +import org.apache.linkis.entrance.exception.{ + EntranceErrorCode, + EntranceErrorException, + EntranceRPCException +} import org.apache.linkis.governance.common.entity.job.JobRequest import org.apache.linkis.governance.common.protocol.conf.{DepartmentRequest, DepartmentResponse} import org.apache.linkis.instance.label.client.InstanceLabelClient @@ -157,9 +167,22 @@ object EntranceUtils extends Logging { /** * 动态引擎类型选择 + * @param sql + * SQL statement + * @param logAppender + * log appender + * @param forceEngineType + * force engine type (optional), such as "starrocks" + * @return + * engine type */ - def getDynamicEngineType(sql: String, logAppender: java.lang.StringBuilder): String = { - val defaultEngineType = "spark" + def getDynamicEngineType( + sql: String, + logAppender: java.lang.StringBuilder, + forceEngineType: String = null + ): String = { + // The default engine is hive for starrocks, and spark for other cases + val defaultEngineType = if ("starrocks".equals(forceEngineType)) "hive" else "spark" if (!EntranceConfiguration.AI_SQL_DYNAMIC_ENGINE_SWITCH) { return defaultEngineType @@ -172,6 +195,11 @@ object EntranceUtils extends Logging { params.put("highStability", "") params.put("queueResourceUsage", "") + // Add force engine type parameter if specified + if (forceEngineType != null && forceEngineType.nonEmpty) { + params.put("engine", forceEngineType) + } + val request = DoctorRequest( apiUrl = EntranceConfiguration.DOCTOR_DYNAMIC_ENGINE_URL, params = params, @@ -184,6 +212,27 @@ object EntranceUtils extends Logging { response.result } + def getDatasourceByDatasourceTypeAndUser( + dataSourceType: String, + user: String, + proxyUser: String + ): DataSource = { + val dataSourceClient = new LinkisDataSourceRemoteClient() + var dataSource: DataSource = null + dataSource = dataSourceClient + .getPublishedDataSourceByType( + GetPublishedDataSourceByTypeAction + .builder() + .setDataSourceType(dataSourceType) + .setUser(user) + .setSystem("Linkis") + .setProxyUser(proxyUser) + .build() + ) + .getDataSource + dataSource + } + def dealsparkDynamicConf( jobRequest: JobRequest, logAppender: lang.StringBuilder, @@ -192,58 +241,85 @@ object EntranceUtils extends Logging { // deal with spark3 dynamic allocation conf // 1.只有spark3需要处理动态规划参数 2.用户未指定模板名称，则设置默认值与spark底层配置保持一致，否则使用用户模板中指定的参数 val properties = new util.HashMap[String, AnyRef]() - val label: EngineTypeLabel = LabelUtil.getEngineTypeLabel(jobRequest.getLabels) val sparkDynamicAllocationEnabled: Boolean = EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_ENABLED - if ( - sparkDynamicAllocationEnabled && label.getEngineType.equals( - EngineType.SPARK.toString - ) && label.getVersion.contains(LabelCommonConfig.SPARK3_ENGINE_VERSION.getValue) - ) { - properties.put( - EntranceConfiguration.SPARK_EXECUTOR_CORES.key, - EntranceConfiguration.SPARK_EXECUTOR_CORES.getValue - ) - properties.put( - EntranceConfiguration.SPARK_EXECUTOR_MEMORY.key, - EntranceConfiguration.SPARK_EXECUTOR_MEMORY.getValue - ) - properties.put( - EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_MAX_EXECUTORS.key, - EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_MAX_EXECUTORS.getValue - ) - properties.put( - EntranceConfiguration.SPARK_EXECUTOR_INSTANCES.key, - EntranceConfiguration.SPARK_EXECUTOR_INSTANCES.getValue - ) - properties.put( - EntranceConfiguration.SPARK_EXECUTOR_MEMORY_OVERHEAD.key, - EntranceConfiguration.SPARK_EXECUTOR_MEMORY_OVERHEAD.getValue - ) - properties.put( - EntranceConfiguration.SPARK3_PYTHON_VERSION.key, - EntranceConfiguration.SPARK3_PYTHON_VERSION.getValue - ) - Utils.tryAndWarn { - val extraConfs: String = - EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_ADDITIONAL_CONFS - if (StringUtils.isNotBlank(extraConfs)) { - val confs: Array[String] = extraConfs.split(",") - for (conf <- confs) { - val confKey: String = conf.split("=")(0) - val confValue: String = conf.split("=")(1) - properties.put(confKey, confValue) - } + val isSpark3 = LabelUtil.isTargetEngine( + jobRequest.getLabels, + EngineType.SPARK.toString, + LabelCommonConfig.SPARK3_ENGINE_VERSION.getValue + ) + try { + if (isSpark3) { + if (!sparkDynamicAllocationEnabled) { + logger.info(s"Task :${jobRequest.getId} using user dynamic conf ") + // If dynamic allocation is disabled, only set python version + properties.put( + EntranceConfiguration.SPARK3_PYTHON_VERSION.key, + EntranceConfiguration.SPARK3_PYTHON_VERSION.getValue + ) + } else { + logger.info(s"Task :${jobRequest.getId} using default dynamic conf ") + setSparkDynamicAllocationDefaultConfs(properties, logAppender) } } - logAppender.append( - LogUtils - .generateInfo(s"use spark3 default conf. \n") - ) + } catch { + case e: Exception => + logger.error( + s"Task error :${jobRequest.getId} using default dynamic conf, message {} ", + e.getMessage + ) + setSparkDynamicAllocationDefaultConfs(properties, logAppender) + } finally { TaskUtils.addStartupMap(params, properties) } } + /** + * Set spark dynamic allocation default confs + */ + private def setSparkDynamicAllocationDefaultConfs( + properties: util.HashMap[String, AnyRef], + logAppender: lang.StringBuilder + ): Unit = { + properties.put( + EntranceConfiguration.SPARK_EXECUTOR_CORES.key, + EntranceConfiguration.SPARK_EXECUTOR_CORES.getValue + ) + properties.put( + EntranceConfiguration.SPARK_EXECUTOR_MEMORY.key, + EntranceConfiguration.SPARK_EXECUTOR_MEMORY.getValue + ) + properties.put( + EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_MAX_EXECUTORS.key, + EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_MAX_EXECUTORS.getValue + ) + properties.put( + EntranceConfiguration.SPARK_EXECUTOR_INSTANCES.key, + EntranceConfiguration.SPARK_EXECUTOR_INSTANCES.getValue + ) + properties.put( + EntranceConfiguration.SPARK_EXECUTOR_MEMORY_OVERHEAD.key, + EntranceConfiguration.SPARK_EXECUTOR_MEMORY_OVERHEAD.getValue + ) + properties.put( + EntranceConfiguration.SPARK3_PYTHON_VERSION.key, + EntranceConfiguration.SPARK3_PYTHON_VERSION.getValue + ) + Utils.tryAndWarn { + val extraConfs: String = + EntranceConfiguration.SPARK_DYNAMIC_ALLOCATION_ADDITIONAL_CONFS + if (StringUtils.isNotBlank(extraConfs)) { + val confs: Array[String] = extraConfs.split(",") + for (conf <- confs) { + val confKey: String = conf.split("=")(0) + val confValue: String = conf.split("=")(1) + properties.put(confKey, confValue) + } + } + } + logInfo(s"use spark3 default conf. \n", logAppender) + } + /** * 敏感信息SQL检查 */ @@ -272,6 +348,52 @@ object EntranceUtils extends Logging { (response.result.toBoolean, response.reason) } + /** + * Spark任务实时诊断 + */ + def taskRealtimeDiagnose( + job: JobRequest, + logAppender: java.lang.StringBuilder + ): DoctorResponse = { + val params = new util.HashMap[String, AnyRef]() + val metricsParams = job.getMetrics + if (MapUtils.isEmpty(metricsParams)) { + throw new EntranceErrorException( + EntranceErrorCode.METRICS_PARAMS_EXCEPTION.getErrCode, + EntranceErrorCode.METRICS_PARAMS_EXCEPTION.getDesc + ) + } + val yarnResource = + MapUtils.getMap(metricsParams, "yarnResource", new util.HashMap[String, AnyRef]()) + if (MapUtils.isEmpty(yarnResource)) { + throw new EntranceErrorException( + EntranceErrorCode.YARN_RESOURCE_YARN_PARAMS_EXCEPTION.getErrCode, + EntranceErrorCode.YARN_RESOURCE_YARN_PARAMS_EXCEPTION.getDesc + ) + } else { + var response: DoctorResponse = null + yarnResource.keySet().toArray.foreach { application => + params.put("taskId", application) + params.put("engineType", LabelUtil.getEngineType(job.getLabels)) + params.put("userId", job.getExecuteUser) + val msg = s"Task execution time exceeds 5m time, perform task diagnosis" + params.put("triggerReason", msg) + params.put("sparkConfig", new util.HashMap[String, AnyRef]()) + params.put("taskName", "") + params.put("linkisTaskUrl", "") + val request = DoctorRequest( + apiUrl = EntranceConfiguration.DOCTOR_REALTIME_DIAGNOSE_URL, + params = params, + defaultValue = "", + successMessage = "Task Realtime Diagnose result", + exceptionMessage = "Task Realtime Diagnose exception" + ) + response = callDoctorService(request, logAppender) + } + response + } + } + /** * Doctor服务调用通用框架 */ @@ -412,7 +534,7 @@ object EntranceUtils extends Logging { reason = reason, duration = duration ) - } else { + } else if (request.apiUrl.contains("engine")) { // 动态引擎选择API val engineType = dataMap.get("engine").toString val reason = dataMap.get("reason").toString @@ -421,6 +543,18 @@ object EntranceUtils extends Logging { logAppender ) DoctorResponse(success = true, result = engineType, reason = reason, duration = duration) + } else if (request.apiUrl.contains("realtime")) { + // 实时诊断API + val resultJson = BDPJettyServerHelper.gson.toJson(responseMapJson) + DoctorResponse(success = true, result = resultJson, reason = null, duration = duration) + } else { + // 默认处理 + val result = dataMap.toString + logInfo( + s"${request.successMessage}: $result, This decision took $duration seconds", + logAppender + ) + DoctorResponse(success = true, result = result, duration = duration) } } else { throw new EntranceRPCException( @@ -437,10 +571,12 @@ object EntranceUtils extends Logging { } /** - * 记录日志信息 + * 管理台任务日志info信息打印 */ private def logInfo(message: String, logAppender: java.lang.StringBuilder): Unit = { - logAppender.append(LogUtils.generateInfo(s"$message\n")) + if (null != logAppender) { + logAppender.append(LogUtils.generateInfo(s"$message\n")) + } } } diff --git a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/JobHistoryHelper.scala b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/JobHistoryHelper.scala index 600960d8809..c38eaf52ce6 100644 --- a/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/JobHistoryHelper.scala +++ b/linkis-computation-governance/linkis-entrance/src/main/scala/org/apache/linkis/entrance/utils/JobHistoryHelper.scala @@ -24,7 +24,7 @@ import org.apache.linkis.entrance.conf.EntranceConfiguration import org.apache.linkis.entrance.exception.JobHistoryFailedException import org.apache.linkis.entrance.execute.EntranceJob import org.apache.linkis.governance.common.constant.job.JobRequestConstants -import org.apache.linkis.governance.common.entity.job.JobRequest +import org.apache.linkis.governance.common.entity.job.{JobDiagnosisRequest, JobRequest} import org.apache.linkis.governance.common.protocol.job._ import org.apache.linkis.manager.common.protocol.resource.ResourceWithStatus import org.apache.linkis.protocol.constants.TaskConstant @@ -365,4 +365,12 @@ object JobHistoryHelper extends Logging { } } + def addDiagnosis(jobid: Long, diagnosis: String): Unit = { + val jobDiagnosisRequest = new JobDiagnosisRequest() + jobDiagnosisRequest.setJobHistoryId(jobid) + jobDiagnosisRequest.setDiagnosisContent(diagnosis) + jobDiagnosisRequest.setDiagnosisSource("doctoris") + sender.ask(JobDiagnosisReqInsert(jobDiagnosisRequest)) + } + } diff --git a/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestCommentHelper.java b/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestCommentHelper.java index 622d06c4e87..635a083d364 100644 --- a/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestCommentHelper.java +++ b/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestCommentHelper.java @@ -22,6 +22,8 @@ import org.junit.jupiter.api.Test; import org.junit.platform.commons.util.StringUtils; +import static org.junit.jupiter.api.Assertions.*; + public class TestCommentHelper { String sqlCode = "" diff --git a/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestHDFSCacheLogWriter.java b/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestHDFSCacheLogWriter.java index fabff884734..7c9f334a7e9 100644 --- a/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestHDFSCacheLogWriter.java +++ b/linkis-computation-governance/linkis-entrance/src/test/java/org/apache/linkis/entrance/interceptor/impl/TestHDFSCacheLogWriter.java @@ -18,10 +18,9 @@ package org.apache.linkis.entrance.interceptor.impl; import org.apache.linkis.entrance.log.Cache; +import org.apache.linkis.entrance.log.CacheLogReader; import org.apache.linkis.entrance.log.HDFSCacheLogWriter; -import org.apache.commons.lang3.StringUtils; - import java.io.File; import java.io.IOException; import java.util.ArrayList; @@ -40,13 +39,64 @@ void write() throws IOException { Cache cache = new Cache(5); String fileName = UUID.randomUUID().toString().replace("-", "") + "-test.log"; - String logPath = System.getProperty("java.io.tmpdir") + File.separator + fileName; + String username = System.getProperty("user.name"); + String parentPath = System.getProperty("java.io.tmpdir") + File.separator + username; + String logPath = parentPath + File.separator + fileName; System.out.println(logPath); String chartSet = "utf-8"; + + File file = new File(parentPath); + file.mkdirs(); + + File logfile = new File(logPath); + logfile.createNewFile(); // NOSONAR + + HDFSCacheLogWriter logWriter = + new HDFSCacheLogWriter( + // "D:\\DataSphere\\linkis\\docs\\test.log", + logPath, chartSet, cache, username); + + String[] msgArr = + new String[] { + "1", "2", "3", "4", "5", "6", + "7", "8", "9", "10", "11", "12", + "13", "14", "15", "16", "17", "18", + "19", "20", "21", "22" + }; + + List msgList = new ArrayList(Arrays.asList(msgArr)); + String msg = String.join("\n", msgList); + + logWriter.write(msg); + + CacheLogReader logReader = new CacheLogReader(logPath, chartSet, cache, username); + String[] logs = new String[4]; + int fromLine = 1; + int size = 1000; + int retFromLine = logReader.readArray(logs, fromLine, size); + Assertions.assertEquals(msgArr.length, retFromLine); + logWriter.flush(); + List list = FileUtil.readFile(logPath); + String res = String.join("\n", list); + Assertions.assertEquals(res, msg); + } + + @Test + void write2() throws IOException, InterruptedException { + + Cache cache = new Cache(30); + String fileName = UUID.randomUUID().toString().replace("-", "") + "-test.log"; String username = System.getProperty("user.name"); + String parentPath = System.getProperty("java.io.tmpdir") + File.separator + username; + String logPath = parentPath + File.separator + fileName; + System.out.println(logPath); + String chartSet = "utf-8"; + + File file = new File(parentPath); + file.mkdirs(); - File file = new File(logPath); - file.createNewFile(); + File logfile = new File(logPath); + logfile.createNewFile(); // NOSONAR HDFSCacheLogWriter logWriter = new HDFSCacheLogWriter( @@ -55,21 +105,33 @@ void write() throws IOException { String[] msgArr = new String[] { - "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", - "17", "18" + "1", "2", "3", "4", "5", "6", + "7", "8", "9", "10", "11", "12", + "13", "14", "15", "16", "17", "18", + "19", "20", "21", "22" }; List msgList = new ArrayList(Arrays.asList(msgArr)); String msg = String.join("\n", msgList); logWriter.write(msg); + + Thread.sleep(4 * 1000); // NOSONAR + + logWriter.write(msg); + + CacheLogReader logReader = new CacheLogReader(logPath, chartSet, cache, username); + String[] logs = new String[4]; + int fromLine = 1; + int size = 1000; + int retFromLine = logReader.readArray(logs, fromLine, size); + Assertions.assertEquals(msgArr.length * 2, retFromLine); + Assertions.assertEquals(msg + "\n" + msg, logs[3]); + logWriter.flush(); List list = FileUtil.readFile(logPath); String res = String.join("\n", list); - - res = res.replace("\n\n", "\n"); - res = StringUtils.strip(res, " \n"); - Assertions.assertEquals(res, msg); + Assertions.assertEquals(msg + "\n" + msg, res); } } diff --git a/linkis-computation-governance/linkis-jdbc-driver/src/main/scala/org/apache/linkis/ujes/jdbc/LinkisSQLConnection.scala b/linkis-computation-governance/linkis-jdbc-driver/src/main/scala/org/apache/linkis/ujes/jdbc/LinkisSQLConnection.scala index b6a0028b15d..e9c07400cdd 100644 --- a/linkis-computation-governance/linkis-jdbc-driver/src/main/scala/org/apache/linkis/ujes/jdbc/LinkisSQLConnection.scala +++ b/linkis-computation-governance/linkis-jdbc-driver/src/main/scala/org/apache/linkis/ujes/jdbc/LinkisSQLConnection.scala @@ -449,6 +449,8 @@ class LinkisSQLConnection(private[jdbc] val ujesClient: UJESClient, props: Prope val runType = EngineType.mapStringToEngineType(engine) match { case EngineType.SPARK => RunType.SQL case EngineType.HIVE => RunType.HIVE + case EngineType.REPL => RunType.REPL + case EngineType.DORIS => RunType.DORIS case EngineType.TRINO => RunType.TRINO_SQL case EngineType.PRESTO => RunType.PRESTO_SQL case EngineType.NEBULA => RunType.NEBULA_SQL diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/pom.xml b/linkis-computation-governance/linkis-manager/linkis-application-manager/pom.xml index d6dee78ed26..4b807cf150e 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/pom.xml +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/pom.xml @@ -27,10 +27,6 @@ linkis-application-manager - - com.jayway.jsonpath - json-path - org.apache.linkis @@ -98,6 +94,13 @@ ${project.version} provided + + + org.apache.linkis + linkis-ps-common-lock + ${project.version} + + com.google.code.gson gson @@ -129,12 +132,6 @@ kubernetes-model-core ${kubernetes-client.version} - - - org.apache.linkis - linkis-ps-common-lock - ${project.version} - diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/loaders/DefaultEngineConnPluginLoader.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/loaders/DefaultEngineConnPluginLoader.java index 0e54ed8c4e5..d9cb61e4ada 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/loaders/DefaultEngineConnPluginLoader.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/loaders/DefaultEngineConnPluginLoader.java @@ -260,8 +260,7 @@ private Class loadEngineConnPluginClass( private Map readFromProperties(String propertiesFile) { Map map = new HashMap<>(); Properties properties = new Properties(); - try { - BufferedReader reader = new BufferedReader(new FileReader(propertiesFile)); + try (BufferedReader reader = new BufferedReader(new FileReader(propertiesFile))) { properties.load(reader); map = new HashMap((Map) properties); } catch (IOException e) { diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/utils/EngineConnPluginUtils.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/utils/EngineConnPluginUtils.java index 2bfcd00acae..21295f17899 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/utils/EngineConnPluginUtils.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/loader/utils/EngineConnPluginUtils.java @@ -132,8 +132,7 @@ private static String getEngineConnPluginClassFromURL( } return acceptedFunction.apply(className) ? className : null; } else if (url.endsWith(JAR_SUF_NAME)) { - try { - JarFile jarFile = new JarFile(new File(url)); + try (JarFile jarFile = new JarFile(new File(url))) { Enumeration en = jarFile.entries(); while (en.hasMoreElements()) { String name = en.nextElement().getName(); diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/server/service/impl/EnginePluginAdminServiceImpl.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/server/service/impl/EnginePluginAdminServiceImpl.java index 803151d5340..061529524d1 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/server/service/impl/EnginePluginAdminServiceImpl.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/engineplugin/server/service/impl/EnginePluginAdminServiceImpl.java @@ -114,7 +114,8 @@ public PageInfo queryDataSourceInfoPage( @Override public void uploadToECHome(MultipartFile mfile) { String engineConnsHome = defaultEngineConnBmlResourceGenerator.getEngineConnsHome(); - try { + try (OutputStream out = + new FileOutputStream(engineConnsHome + "/" + mfile.getOriginalFilename())) { InputStream in = mfile.getInputStream(); byte[] buffer = new byte[1024]; int len = 0; @@ -122,11 +123,9 @@ public void uploadToECHome(MultipartFile mfile) { if (!file.exists()) { log.info("engineplugin's home doesn’t exist"); } - OutputStream out = new FileOutputStream(engineConnsHome + "/" + mfile.getOriginalFilename()); while ((len = in.read(buffer)) != -1) { out.write(buffer, 0, len); } - out.close(); in.close(); } catch (Exception e) { log.info("file {} upload fail", mfile.getOriginalFilename()); @@ -135,7 +134,9 @@ public void uploadToECHome(MultipartFile mfile) { ZipUtils.fileToUnzip(engineConnsHome + "/" + mfile.getOriginalFilename(), engineConnsHome); File file = new File(engineConnsHome + "/" + mfile.getOriginalFilename()); if (file.exists()) { - file.delete(); + if (!file.delete()) { + log.error("file {} delete failed", mfile.getOriginalFilename()); + } log.info("file {} delete success", mfile.getOriginalFilename()); } } @@ -146,9 +147,13 @@ public static void deleteDir(File directory) { if (file.isDirectory()) { deleteDir(file); } else { - file.delete(); + if (!file.delete()) { + log.error("file {} delete failed", file.getName()); + } } } - directory.delete(); + if (!directory.delete()) { + log.error("directory {} delete failed", directory.getName()); + } } } diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/ExternalResourceService.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/ExternalResourceService.java index b75e077823d..0516843f71d 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/ExternalResourceService.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/ExternalResourceService.java @@ -50,6 +50,12 @@ List getAppInfo( ExternalResourceIdentifier identifier) throws RMErrorException; + Map getBatchResource( + ResourceType resourceType, + RMLabelContainer labelContainer, + List identifiers) + throws RMErrorException; + ExternalResourceProvider chooseProvider( ResourceType resourceType, RMLabelContainer labelContainer) throws RMErrorException; } diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/impl/ExternalResourceServiceImpl.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/impl/ExternalResourceServiceImpl.java index a5823853ecb..1c7dcad2295 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/impl/ExternalResourceServiceImpl.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/service/impl/ExternalResourceServiceImpl.java @@ -18,6 +18,7 @@ package org.apache.linkis.manager.rm.external.service.impl; import org.apache.linkis.manager.common.conf.RMConfiguration; +import org.apache.linkis.manager.common.entity.resource.CommonNodeResource; import org.apache.linkis.manager.common.entity.resource.NodeResource; import org.apache.linkis.manager.common.entity.resource.ResourceType; import org.apache.linkis.manager.common.exception.RMErrorException; @@ -33,6 +34,8 @@ import org.apache.linkis.manager.rm.external.parser.YarnResourceIdentifierParser; import org.apache.linkis.manager.rm.external.request.ExternalResourceRequester; import org.apache.linkis.manager.rm.external.service.ExternalResourceService; +import org.apache.linkis.manager.rm.external.yarn.YarnQueueInfo; +import org.apache.linkis.manager.rm.external.yarn.YarnResourceIdentifier; import org.apache.linkis.manager.rm.external.yarn.YarnResourceRequester; import org.apache.linkis.manager.rm.utils.RMUtils; @@ -44,11 +47,13 @@ import java.net.ConnectException; import java.text.MessageFormat; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.concurrent.ExecutionException; import java.util.concurrent.TimeUnit; import java.util.function.Function; +import java.util.stream.Collectors; import com.fasterxml.jackson.core.JsonParseException; import com.google.common.cache.CacheBuilder; @@ -73,8 +78,7 @@ public class ExternalResourceServiceImpl implements ExternalResourceService, Ini private LoadingCache> providerCache = CacheBuilder.newBuilder() .maximumSize(20) - .expireAfterAccess(1, TimeUnit.HOURS) - .refreshAfterWrite( + .expireAfterWrite( RMUtils.EXTERNAL_RESOURCE_REFRESH_TIME().getValue().toLong(), TimeUnit.MINUTES) .build( new CacheLoader>() { @@ -142,6 +146,60 @@ public List getAppInfo( return appInfos; } + @Override + public Map getBatchResource( + ResourceType resourceType, + RMLabelContainer labelContainer, + List identifiers) + throws RMErrorException { + ExternalResourceProvider provider = chooseProvider(resourceType, labelContainer); + ExternalResourceRequester externalResourceRequester = getRequester(resourceType); + + if (externalResourceRequester instanceof YarnResourceRequester) { + YarnResourceRequester yarnRequester = (YarnResourceRequester) externalResourceRequester; + List queueNames = + identifiers.stream() + .map(id -> ((YarnResourceIdentifier) id).getQueueName()) + .collect(Collectors.toList()); + + Map batchResources = + (Map) + retry( + RMConfiguration.EXTERNAL_RETRY_NUM.getValue(), + (i) -> + yarnRequester.getBatchResources( + yarnRequester.getAndUpdateActiveRmWebAddress(provider), + queueNames, + provider), + (i) -> yarnRequester.reloadExternalResourceAddress(provider)); + + Map result = new HashMap<>(); + batchResources.forEach( + (queueName, queueInfo) -> { + CommonNodeResource nodeResource = new CommonNodeResource(); + nodeResource.setMaxResource(queueInfo.getMaxResource()); + nodeResource.setUsedResource(queueInfo.getUsedResource()); + nodeResource.setMaxApps(queueInfo.getMaxApps()); + nodeResource.setNumPendingApps(queueInfo.getNumPendingApps()); + nodeResource.setNumActiveApps(queueInfo.getNumActiveApps()); + result.put(queueName, nodeResource); + }); + return result; + } else { + // For other resource types, fall back to individual requests + Map result = new HashMap<>(); + for (ExternalResourceIdentifier identifier : identifiers) { + try { + NodeResource resource = getResource(resourceType, labelContainer, identifier); + result.put(((YarnResourceIdentifier) identifier).getQueueName(), resource); + } catch (Exception e) { + logger.error("Failed to get resource for identifier " + identifier, e); + } + } + return result; + } + } + private Object retry(int retryNum, Function function, Function reloadExternalAddress) throws RMErrorException { int times = 0; diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnAppState.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnAppState.java new file mode 100644 index 00000000000..94eef8f3ae6 --- /dev/null +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnAppState.java @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.manager.rm.external.yarn; + +/** Enumeration of Yarn application states. */ +public enum YarnAppState { + + /** Application has been submitted but not yet accepted by ResourceManager. */ + NEW("NEW"), + + /** Application is being saved to persistent storage. */ + NEW_SAVING("NEW_SAVING"), + + /** Application has been submitted and is waiting for scheduling. */ + SUBMITTED("SUBMITTED"), + + /** Application has been accepted by ResourceManager and waiting for resource allocation. */ + ACCEPTED("ACCEPTED"), + + /** Application is running with at least one container executing. */ + RUNNING("RUNNING"), + + /** Application has completed successfully. */ + FINISHED("FINISHED"), + + /** Application execution has failed. */ + FAILED("FAILED"), + + /** Application has been manually terminated. */ + KILLED("KILLED"); + + private final String state; + + YarnAppState(String state) { + this.state = state; + } + + public String getState() { + return state; + } + + /** + * Check if the state is active (RUNNING or ACCEPTED). These states represent applications that + * are consuming or about to consume cluster resources. + * + * @return true if the state is active + */ + public boolean isActive() { + return this == RUNNING || this == ACCEPTED; + } + + /** + * Parse string to YarnAppState enum. + * + * @param state the state string + * @return YarnAppState enum + */ + public static YarnAppState fromString(String state) { + for (YarnAppState appState : YarnAppState.values()) { + if (appState.state.equals(state)) { + return appState; + } + } + throw new IllegalArgumentException("Unknown YarnAppState: " + state); + } +} diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnResourceRequester.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnResourceRequester.java index 4f6a158f7c8..110f58e632d 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnResourceRequester.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/external/yarn/YarnResourceRequester.java @@ -18,6 +18,7 @@ package org.apache.linkis.manager.rm.external.yarn; import org.apache.linkis.engineplugin.server.conf.EngineConnPluginConfiguration; +import org.apache.linkis.manager.common.conf.RMConfiguration; import org.apache.linkis.manager.common.entity.resource.CommonNodeResource; import org.apache.linkis.manager.common.entity.resource.NodeResource; import org.apache.linkis.manager.common.entity.resource.ResourceType; @@ -217,6 +218,31 @@ public YarnQueueInfo getResources( String queueName, ExternalResourceProvider provider) { JsonNode resp = getResponseByUrl("scheduler", rmWebAddress, provider); + return getResourcesFromResponse(resp, realQueueName, queueName, provider); + } + + public Map getBatchResources( + String rmWebAddress, List queueNames, ExternalResourceProvider provider) { + JsonNode resp = getResponseByUrl("scheduler", rmWebAddress, provider); + Map queueInfoMap = new HashMap<>(); + for (String queueName : queueNames) { + try { + String realQueueName = queuePrefix + queueName; + if (queueName.startsWith(queuePrefix)) { + realQueueName = queueName; + } + YarnQueueInfo queueInfo = + getResourcesFromResponse(resp, realQueueName, queueName, provider); + queueInfoMap.put(queueName, queueInfo); + } catch (Exception e) { + logger.error("Failed to get resource for queue " + queueName, e); + } + } + return queueInfoMap; + } + + private YarnQueueInfo getResourcesFromResponse( + JsonNode resp, String realQueueName, String queueName, ExternalResourceProvider provider) { JsonNode schedulerInfo = resp.path("scheduler").path("schedulerInfo"); String schedulerType = schedulerInfo.path("type").asText(); if ("capacityScheduler".equals(schedulerType)) { @@ -232,7 +258,8 @@ public YarnQueueInfo getResources( } JsonNode queueInfo = queue.get(); return new YarnQueueInfo( - maxEffectiveHandle(queue, rmWebAddress, queueName, provider).get(), + maxEffectiveHandle(queue, getAndUpdateActiveRmWebAddress(provider), queueName, provider) + .get(), getYarnResource(queue.map(node -> node.path("resourcesUsed")), queueName).get(), queueInfo.path("maxApps").asInt(), queueInfo.path("numPendingApps").asInt(), @@ -313,8 +340,24 @@ public List requestAppInfo( String queueName = ((YarnResourceIdentifier) identifier).getQueueName(); String realQueueName = queuePrefix + queueName; + JsonNode resp; + if (RMConfiguration.YARN_APPS_FILTER_ENABLED.getValue()) { + // Build query parameters to filter apps at Yarn API level using active states only + String queryParams = + "?queue=" + + realQueueName + + "&states=" + + YarnAppState.RUNNING.getState() + + "," + + YarnAppState.ACCEPTED.getState() + + RMConfiguration.YARN_APPS_FILTER_PARMS.getValue(); + resp = + getResponseByUrl("apps" + queryParams, rmWebAddress, provider).path("apps").path("app"); + } else { + // Fetch all apps without filtering (for backward compatibility) + resp = getResponseByUrl("apps", rmWebAddress, provider).path("apps").path("app"); + } - JsonNode resp = getResponseByUrl("apps", rmWebAddress, provider).path("apps").path("app"); if (resp.isMissingNode()) { return new ArrayList<>(); } diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/utils/RequestKerberosUrlUtils.java b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/utils/RequestKerberosUrlUtils.java index 12b729a054a..2159d7868bd 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/utils/RequestKerberosUrlUtils.java +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/java/org/apache/linkis/manager/rm/utils/RequestKerberosUrlUtils.java @@ -103,6 +103,16 @@ public HttpResponse callRestUrl(final String url, final String userId) { logger.warn( String.format( "Calling KerberosHttpClient %s %s %s", this.principal, this.keyTabLocation, url)); + HashMap map = new HashMap<>(); + map.put("useTicketCache", "false"); + map.put("useKeyTab", "true"); + map.put("keyTab", keyTabLocation); + map.put("refreshKrb5Config", "true"); + map.put("principal", principal); + map.put("storeKey", "true"); + map.put("doNotPrompt", "true"); + map.put("isInitiator", "true"); + map.put("debug", "false"); Configuration config = new Configuration() { @SuppressWarnings("serial") @@ -112,19 +122,7 @@ public AppConfigurationEntry[] getAppConfigurationEntry(String name) { new AppConfigurationEntry( "com.sun.security.auth.module.Krb5LoginModule", AppConfigurationEntry.LoginModuleControlFlag.REQUIRED, - new HashMap() { - { - put("useTicketCache", "false"); - put("useKeyTab", "true"); - put("keyTab", keyTabLocation); - put("refreshKrb5Config", "true"); - put("principal", principal); - put("storeKey", "true"); - put("doNotPrompt", "true"); - put("isInitiator", "true"); - put("debug", "false"); - } - }) + map) }; } }; diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineAskEngineService.scala b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineAskEngineService.scala index 06bffcc8cb6..1f912b5f749 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineAskEngineService.scala +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineAskEngineService.scala @@ -247,37 +247,50 @@ class DefaultEngineAskEngineService null } } - - val engineCreateRequest = new EngineCreateRequest - engineCreateRequest.setLabels(engineAskRequest.getLabels) - engineCreateRequest.setTimeout(engineAskRequest.getTimeOut) - engineCreateRequest.setUser(engineAskRequest.getUser) - engineCreateRequest.setProperties(engineAskRequest.getProperties) - engineCreateRequest.setCreateService(engineAskRequest.getCreateService) - - val createNode = engineCreateService.createEngine(engineCreateRequest, sender) - val timeout = - if (engineCreateRequest.getTimeout <= 0) { - AMConfiguration.ENGINE_START_MAX_TIME.getValue.toLong - } else engineCreateRequest.getTimeout - // UseEngine requires a timeout (useEngine 需要加上超时) - val createEngineNode = getEngineNodeManager.useEngine(createNode, timeout) - if (null == createEngineNode) { - throw new LinkisRetryException( - AMConstant.EM_ERROR_CODE, - s"create engine${createNode.getServiceInstance} success, but to use engine failed" - ) - } - logger.info( - s"Task: $taskId finished to ask engine for user ${engineAskRequest.getUser} by create node $createEngineNode" - ) - if (null != sender) { - sender.send(EngineCreateSuccess(engineAskAsyncId, createEngineNode)) + if (reuseNode != null) { logger.info( - s"Task: $taskId has sent EngineCreateSuccess($engineAskAsyncId, reuse=false) to Entrance." + s"Task: $taskId finished to ask engine for user ${engineAskRequest.getUser} by reuse node $reuseNode" ) + if (null != sender) { + sender.send(EngineCreateSuccess(engineAskAsyncId, reuseNode, true)) + logger.info( + s"Task: $taskId has sent EngineCreateSuccess($engineAskAsyncId, reuse=true) to Entrance." + ) + } else { + logger.warn(f"Task: $taskId will not send async using null sender.") + } } else { - logger.warn(s"Task: $taskId will not send async using null sender.") + val engineCreateRequest = new EngineCreateRequest + engineCreateRequest.setLabels(engineAskRequest.getLabels) + engineCreateRequest.setTimeout(engineAskRequest.getTimeOut) + engineCreateRequest.setUser(engineAskRequest.getUser) + engineCreateRequest.setProperties(engineAskRequest.getProperties) + engineCreateRequest.setCreateService(engineAskRequest.getCreateService) + + val createNode = engineCreateService.createEngine(engineCreateRequest, sender) + val timeout = + if (engineCreateRequest.getTimeout <= 0) { + AMConfiguration.ENGINE_START_MAX_TIME.getValue.toLong + } else engineCreateRequest.getTimeout + // UseEngine requires a timeout (useEngine 需要加上超时) + val createEngineNode = getEngineNodeManager.useEngine(createNode, timeout) + if (null == createEngineNode) { + throw new LinkisRetryException( + AMConstant.EM_ERROR_CODE, + s"create engine${createNode.getServiceInstance} success, but to use engine failed" + ) + } + logger.info( + s"Task: $taskId finished to ask engine for user ${engineAskRequest.getUser} by create node $createEngineNode" + ) + if (null != sender) { + sender.send(EngineCreateSuccess(engineAskAsyncId, createEngineNode)) + logger.info( + s"Task: $taskId has sent EngineCreateSuccess($engineAskAsyncId, reuse=false) to Entrance." + ) + } else { + logger.warn(s"Task: $taskId will not send async using null sender.") + } } } { Utils.tryAndWarn { diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineCreateService.scala b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineCreateService.scala index 27a961b852a..86e1c2e7027 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineCreateService.scala +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineCreateService.scala @@ -236,10 +236,13 @@ class DefaultEngineCreateService val engineNode = Utils.tryCatch(getEMService().createEngine(engineBuildRequest, emNode)) { case t: Throwable => - logger.info(s"Failed to create ec($resourceTicketId) ask ecm ${emNode.getServiceInstance}") + logger.warn( + s"Failed to create ec($resourceTicketId) ask ecm ${emNode.getServiceInstance}", + t + ) val failedEcNode = getEngineNodeManager.getEngineNode(oldServiceInstance) if (null == failedEcNode) { - logger.info(s" engineConn does not exist in db: $oldServiceInstance ") + logger.warn(s" engineConn does not exist in db: $oldServiceInstance ") } else { failedEcNode.setLabels(nodeLabelService.getNodeLabels(oldServiceInstance)) failedEcNode.getLabels.addAll( @@ -289,18 +292,28 @@ class DefaultEngineCreateService s"Failed to update engineNode: ${t.getMessage}" ) } - if (Configuration.METRICS_INCREMENTAL_UPDATE_ENABLE.getValue) { - val emInstance = engineNode.getServiceInstance.getInstance - val ecmInstance = engineNode.getEMNode.getServiceInstance.getInstance - // 8. Update job history metrics after successful engine creation - 异步执行 - AMUtils.updateMetricsAsync( - taskId, - resourceTicketId, - emInstance, - ecmInstance, - null, - isReuse = false - ) + Utils.tryCatch { + if (Configuration.METRICS_INCREMENTAL_UPDATE_ENABLE.getValue) { + val emInstance = engineNode.getServiceInstance.getInstance + val ecmInstance = engineNode.getEMNode.getServiceInstance.getInstance + if ((null != emInstance) && (null != ecmInstance)) { + // 8. Update job history metrics after successful engine creation - 异步执行 + AMUtils.updateMetricsAsync( + taskId, + resourceTicketId, + emInstance, + ecmInstance, + null, + isReuse = false + ) + } else { + logger.info( + s"CreateEngine:Failed to update metrics for emInstance: $emInstance, ecmInstance: $ecmInstance" + ) + } + } + } { case e: Exception => + logger.error(s"Failed to update metrics for taskId: $taskId", e) } // 9. Add the Label of EngineConn, and add the Alias of engineConn val engineConnAliasLabel = labelBuilderFactory.createLabel(classOf[AliasServiceInstanceLabel]) diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineReuseService.scala b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineReuseService.scala index fb79c9e062f..25179296ca0 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineReuseService.scala +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/service/engine/DefaultEngineReuseService.scala @@ -236,7 +236,6 @@ class DefaultEngineReuseService extends AbstractEngineService with EngineReuseSe StringUtils.isNotBlank(templateName) && AMConfiguration.EC_REUSE_WITH_TEMPLATE_RULE_ENABLE ) { engineScoreList = engineScoreList - .filter(engine => engine.getNodeStatus == NodeStatus.Unlock) .filter(engine => { val oldTemplateName: String = getValueByKeyFromProps(confTemplateNameKey, parseParamsToMap(engine.getParams)) @@ -276,7 +275,6 @@ class DefaultEngineReuseService extends AbstractEngineService with EngineReuseSe // 过滤掉资源不满足的引擎 engineScoreList = engineScoreList - .filter(engine => engine.getNodeStatus == NodeStatus.Unlock) .filter(engine => { val enginePythonVersion: String = getPythonVersion(parseParamsToMap(engine.getParams)) var pythonVersionMatch: Boolean = true @@ -383,20 +381,29 @@ class DefaultEngineReuseService extends AbstractEngineService with EngineReuseSe .toJson(engine) + " from engineLabelMap : " + AMUtils.GSON.toJson(instances) ) } - if (Configuration.METRICS_INCREMENTAL_UPDATE_ENABLE.getValue) { - val engineNode = - ecResourceInfoService.getECResourceInfoRecordByInstance( - engine.getServiceInstance.getInstance - ) - // 异步更新 metrics - AMUtils.updateMetricsAsync( - taskId, - engineNode.getTicketId, - engineNode.getServiceInstance, - engineNode.getEcmInstance, - engineNode.getLogDirSuffix, - isReuse = true - ) + Utils.tryCatch { + if (Configuration.METRICS_INCREMENTAL_UPDATE_ENABLE.getValue) { + val engineNode = + ecResourceInfoService.getECResourceInfoRecordByInstance( + engine.getServiceInstance.getInstance + ) + if (null != engineNode) { + // 异步更新 metrics + AMUtils.updateMetricsAsync( + taskId, + engineNode.getTicketId, + engineNode.getServiceInstance, + engineNode.getEcmInstance, + engineNode.getLogDirSuffix, + isReuse = true + ) + } else { + logger.info(s"ReuseEngine:Failed to update metrics for engineNode: $engineNode") + } + + } + } { case e: Exception => + logger.error(s"Failed to update metrics for taskId: $taskId", e) } engine } diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/utils/AMUtils.scala b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/utils/AMUtils.scala index 89084ebe9a3..1b359c29a63 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/utils/AMUtils.scala +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/am/utils/AMUtils.scala @@ -44,11 +44,16 @@ import java.io.File import java.util import scala.collection.JavaConverters._ +import scala.concurrent.ExecutionContextExecutorService import com.google.gson.JsonObject object AMUtils extends Logging { + // 优化：线程池复用，线程数设置为5 + private implicit val updateMetricsExecutor: ExecutionContextExecutorService = + Utils.newCachedExecutionContext(5, "UpdateMetrics-Thread-") + lazy val GSON = BDPJettyServerHelper.gson private val SUCCESS_FLAG = 0 @@ -409,14 +414,15 @@ object AMUtils extends Logging { import scala.concurrent.Future import scala.util.{Failure, Success} + // 优化：使用复用的线程池，线程数设置为5 Future { updateMetrics(taskId, resourceTicketId, emInstance, ecmInstance, engineLogPath, isReuse) - }(Utils.newCachedExecutionContext(1, "UpdateMetrics-Thread-")).onComplete { + }(updateMetricsExecutor).onComplete { case Success(_) => logger.debug(s"Task: $taskId metrics update completed successfully for engine: $emInstance") case Failure(t) => logger.warn(s"Task: $taskId metrics update failed for engine: $emInstance", t) - }(Utils.newCachedExecutionContext(1, "UpdateMetrics-Thread-")) + }(updateMetricsExecutor) } } diff --git a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/rm/restful/RMMonitorRest.scala b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/rm/restful/RMMonitorRest.scala index 41512c31cc6..7bf1acca23c 100644 --- a/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/rm/restful/RMMonitorRest.scala +++ b/linkis-computation-governance/linkis-manager/linkis-application-manager/src/main/scala/org/apache/linkis/manager/rm/restful/RMMonitorRest.scala @@ -19,6 +19,7 @@ package org.apache.linkis.manager.rm.restful import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.utils.{Logging, Utils} +import org.apache.linkis.engineplugin.server.conf.EngineConnPluginConfiguration import org.apache.linkis.governance.common.protocol.conf.{ AcrossClusterRequest, AcrossClusterResponse @@ -51,6 +52,7 @@ import org.apache.linkis.manager.persistence.{ ResourceManagerPersistence } import org.apache.linkis.manager.rm.domain.RMLabelContainer +import org.apache.linkis.manager.rm.external.domain.ExternalResourceIdentifier import org.apache.linkis.manager.rm.external.service.ExternalResourceService import org.apache.linkis.manager.rm.external.yarn.{YarnAppInfo, YarnResourceIdentifier} import org.apache.linkis.manager.rm.restful.vo.{UserCreatorEngineType, UserResourceVo} @@ -132,9 +134,10 @@ class RMMonitorRest extends Logging { var COMBINED_USERCREATOR_ENGINETYPE: String = _ + private val queuePrefix = EngineConnPluginConfiguration.QUEUE_PREFIX.getValue + def appendMessageData(message: Message, key: String, value: AnyRef): Message = { val result = mapper.writeValueAsString(value) - logger.info(s"appendMessageData result: $result") message.data(key, mapper.readTree(result)) } @@ -371,7 +374,17 @@ class RMMonitorRest extends Logging { ): Message = { ModuleUserUtils.getOperationUser(request, "getQueueResource") val message = Message.ok("") - val yarnIdentifier = new YarnResourceIdentifier(param.get("queuename").asInstanceOf[String]) + var queueName: String = param.get("queuename").asInstanceOf[String] + if (StringUtils.isNotBlank(queueName) && queueName.startsWith(queuePrefix)) { + logger.info( + "Queue name {} starts with '{}', remove '{}'", + queueName, + queuePrefix, + queuePrefix + ) + queueName = queueName.substring(queuePrefix.length) + } + val yarnIdentifier = new YarnResourceIdentifier(queueName) var clustername = param.get("clustername").asInstanceOf[String] val crossCluster = java.lang.Boolean.parseBoolean( param.getOrDefault("crossCluster", "false").asInstanceOf[String] @@ -562,6 +575,174 @@ class RMMonitorRest extends Logging { appendMessageData(message, "queues", clusters) } + @ApiOperation(value = "getBatchQueueResource", notes = "get batch queue resource") + @RequestMapping(path = Array("batchqueueresources"), method = Array(RequestMethod.POST)) + def getBatchQueueResource( + request: HttpServletRequest, + @RequestBody param: util.Map[String, AnyRef] + ): Message = { + ModuleUserUtils.getOperationUser(request, "getBatchQueueResource") + val message = Message.ok("") + val queueNamesParam = param.get("queueNames") + if (queueNamesParam == null) { + return Message.error("queueNames parameter is required") + } + val queueNames = queueNamesParam match { + case list: java.util.List[_] => + list.asScala.map(_.toString.trim).filter(StringUtils.isNotBlank).toArray + case array: Array[_] => + array.map(_.toString.trim).filter(StringUtils.isNotBlank) + case _ => + return Message.error("queueNames parameter must be an array or comma-separated string") + } + if (queueNames.isEmpty) { + return Message.error("queueNames parameter is empty") + } + var clustername = param.get("clustername").asInstanceOf[String] + val crossCluster = java.lang.Boolean.parseBoolean( + param.getOrDefault("crossCluster", "false").asInstanceOf[String] + ) + if (crossCluster) { + clustername = AMConfiguration.PRIORITY_CLUSTER_TARGET + } + val clusterLabel = labelFactory.createLabel(classOf[ClusterLabel]) + clusterLabel.setClusterName(clustername) + clusterLabel.setClusterType(param.get("clustertype").asInstanceOf[String]) + val labelContainer = new RMLabelContainer(Lists.newArrayList(clusterLabel)) + val queueInfoMap = new mutable.HashMap[String, AnyRef]() + + try { + // Process queue names and create identifiers + import java.util.ArrayList + val identifiers = new ArrayList[ExternalResourceIdentifier]() + queueNames.foreach { queueName => + var processedQueueName = queueName + if ( + StringUtils.isNotBlank(processedQueueName) && processedQueueName.startsWith(queuePrefix) + ) { + logger.info( + "Queue name {} starts with '{}', remove '{}'", + processedQueueName, + queuePrefix, + queuePrefix + ) + processedQueueName = processedQueueName.substring(queuePrefix.length) + } + identifiers.add(new YarnResourceIdentifier(processedQueueName)) + } + + // Use batch API to get all queue resources at once + val batchResources = + externalResourceService.getBatchResource(ResourceType.Yarn, labelContainer, identifiers) + + // Process the results + import scala.collection.JavaConverters._ + batchResources.asScala.foreach { case (queueName, nodeResource) => + ( + nodeResource.getMaxResource.asInstanceOf[YarnResource], + nodeResource.getUsedResource.asInstanceOf[YarnResource] + ) match { + case (maxResource, usedResource) => + val queueInfo = new mutable.HashMap[String, AnyRef]() + queueInfo.put("queuename", maxResource) + queueInfo.put( + "maxResources", + Map( + "memory" -> maxResource.getQueueMemory, + "cores" -> maxResource.getQueueCores + ).asJava + ) + queueInfo.put( + "usedResources", + Map( + "memory" -> usedResource.getQueueMemory, + "cores" -> usedResource.getQueueCores + ).asJava + ) + val usedMemoryPercentage = usedResource.getQueueMemory + .asInstanceOf[Double] / maxResource.getQueueMemory.asInstanceOf[Double] + val usedCPUPercentage = usedResource.getQueueCores + .asInstanceOf[Double] / maxResource.getQueueCores.asInstanceOf[Double] + queueInfo.put( + "usedPercentage", + Map("memory" -> usedMemoryPercentage, "cores" -> usedCPUPercentage).asJava + ) + queueInfo.put("maxApps", nodeResource.getMaxApps.asInstanceOf[AnyRef]) + queueInfo.put("numActiveApps", nodeResource.getNumActiveApps.asInstanceOf[AnyRef]) + queueInfo.put("numPendingApps", nodeResource.getNumPendingApps.asInstanceOf[AnyRef]) + queueInfoMap.put(queueName, queueInfo.asJava) + case _ => + logger.warn(s"Failed to get queue resource for $queueName") + } + } + } catch { + case e: Exception => + logger.error("Failed to get batch queue resources", e) + // Fall back to individual requests if batch API fails + queueNames.foreach { queueName => + try { + var processedQueueName = queueName + if ( + StringUtils + .isNotBlank(processedQueueName) && processedQueueName.startsWith(queuePrefix) + ) { + processedQueueName = processedQueueName.substring(queuePrefix.length) + } + val yarnIdentifier = new YarnResourceIdentifier(processedQueueName) + val providedYarnResource = + externalResourceService.getResource(ResourceType.Yarn, labelContainer, yarnIdentifier) + ( + providedYarnResource.getMaxResource.asInstanceOf[YarnResource], + providedYarnResource.getUsedResource.asInstanceOf[YarnResource] + ) match { + case (maxResource, usedResource) => + val queueInfo = new mutable.HashMap[String, AnyRef]() + queueInfo.put( + "maxResources", + Map( + "memory" -> maxResource.getQueueMemory, + "cores" -> maxResource.getQueueCores + ).asJava + ) + queueInfo.put( + "usedResources", + Map( + "memory" -> usedResource.getQueueMemory, + "cores" -> usedResource.getQueueCores + ).asJava + ) + val usedMemoryPercentage = usedResource.getQueueMemory + .asInstanceOf[Double] / maxResource.getQueueMemory.asInstanceOf[Double] + val usedCPUPercentage = usedResource.getQueueCores + .asInstanceOf[Double] / maxResource.getQueueCores.asInstanceOf[Double] + queueInfo.put( + "usedPercentage", + Map("memory" -> usedMemoryPercentage, "cores" -> usedCPUPercentage).asJava + ) + queueInfo.put("maxApps", providedYarnResource.getMaxApps.asInstanceOf[AnyRef]) + queueInfo.put( + "numActiveApps", + providedYarnResource.getNumActiveApps.asInstanceOf[AnyRef] + ) + queueInfo.put( + "numPendingApps", + providedYarnResource.getNumPendingApps.asInstanceOf[AnyRef] + ) + queueInfoMap.put(queueName, queueInfo.asJava) + case _ => + logger.warn(s"Failed to get queue resource for $queueName") + } + } catch { + case ex: Exception => + logger.error(s"Failed to get queue resource for $queueName", ex) + } + } + } + + appendMessageData(message, "queueInfos", queueInfoMap.asJava) + message + } + private def getEngineNodesByUserList( userList: List[String], withResource: Boolean = false diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/builder/factory/LabelBuilderFactoryContext.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/builder/factory/LabelBuilderFactoryContext.java index 69fdae7c582..1199c852c57 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/builder/factory/LabelBuilderFactoryContext.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/builder/factory/LabelBuilderFactoryContext.java @@ -43,7 +43,7 @@ public static void register(Class clazz) { public static LabelBuilderFactory getLabelBuilderFactory() { if (labelBuilderFactory == null) { - synchronized (LabelBuilderFactoryContext.class) { + synchronized (LabelBuilderFactoryContext.class) { // NOSONAR if (labelBuilderFactory == null) { String className = LabelCommonConfig.LABEL_FACTORY_CLASS.acquireNew(); if (clazz == StdLabelBuilderFactory.class && StringUtils.isNotBlank(className)) { diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/conf/LabelCommonConfig.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/conf/LabelCommonConfig.java index d8ef089eb7f..501016af421 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/conf/LabelCommonConfig.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/conf/LabelCommonConfig.java @@ -34,7 +34,7 @@ public class LabelCommonConfig { CommonVars.apply("wds.linkis.label.entity.packages", ""); public static final CommonVars SPARK_ENGINE_VERSION = - CommonVars.apply("wds.linkis.spark.engine.version", "3.2.1"); + CommonVars.apply("wds.linkis.spark.engine.version", "2.4.3"); public static final String SPARK3_ENGINE_VERSION_CONF = "sparkVersion"; public static final String SPARK_ENGINE_HOME_CONF = "SPARK_HOME"; public static final String SPARK_ENGINE_CMD_CONF = "SPARK_CMD"; @@ -64,7 +64,7 @@ public class LabelCommonConfig { public static final CommonVars SPARK_ENGINE_CONFIG = CommonVars.apply("linkis.spark.engine.config", "/appcom/config/spark-config/spark-submit"); public static final CommonVars HIVE_ENGINE_VERSION = - CommonVars.apply("wds.linkis.hive.engine.version", "3.1.3"); + CommonVars.apply("wds.linkis.hive.engine.version", "2.3.3"); public static final CommonVars PYTHON_ENGINE_VERSION = CommonVars.apply("wds.linkis.python.engine.version", "python2"); diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CloneableLabel.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CloneableLabel.java index 2388164b212..7410cfedb1c 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CloneableLabel.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CloneableLabel.java @@ -33,7 +33,7 @@ public Object clone() { return super.clone(); } catch (CloneNotSupportedException e) { LOG.info("Not support to clone label:[" + toString() + "]", e); - return null; + return null; // NOSONAR } } } diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CombinedLabelImpl.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CombinedLabelImpl.java index 50ff5dd2a38..80c010599d1 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CombinedLabelImpl.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/CombinedLabelImpl.java @@ -25,6 +25,7 @@ import java.util.Collections; import java.util.Comparator; import java.util.List; +import java.util.Objects; import java.util.stream.Collectors; public class CombinedLabelImpl implements CombinedLabel { @@ -88,6 +89,11 @@ public boolean equals(Object obj) { } } + @Override + public int hashCode() { + return Objects.hash(value, feature); + } + @Override public String toString() { return "CombinedLabelImpl{" + "key=" + getLabelKey() + "value=" + getStringValue() + '}'; diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/SerializableLabel.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/SerializableLabel.java index d4f4eecb103..048906384a0 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/SerializableLabel.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/SerializableLabel.java @@ -145,6 +145,11 @@ public boolean equals(Object other) { return super.equals(other); } + @Override + public int hashCode() { + return Objects.hash(value, stringValue); + } + @Override public Boolean isEmpty() { return null == getValue(); diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/BindEngineLabel.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/BindEngineLabel.java index 39068179ac2..9a4c0244595 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/BindEngineLabel.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/BindEngineLabel.java @@ -24,6 +24,7 @@ import org.apache.linkis.manager.label.entity.annon.ValueSerialNum; import java.util.HashMap; +import java.util.Objects; public class BindEngineLabel extends GenericLabel implements JobStrategyLabel { @@ -70,6 +71,11 @@ public boolean equals(Object other) { } } + @Override + public int hashCode() { + return Objects.hash(value); + } + @ValueSerialNum(0) public BindEngineLabel setJobGroupId(String jobGroupId) { if (null == getValue()) { diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/LoadBalanceLabel.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/LoadBalanceLabel.java index 48d733d9f06..0371d6717fa 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/LoadBalanceLabel.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/entity/entrance/LoadBalanceLabel.java @@ -23,6 +23,7 @@ import org.apache.linkis.manager.label.entity.annon.ValueSerialNum; import java.util.HashMap; +import java.util.Objects; public class LoadBalanceLabel extends GenericLabel implements JobStrategyLabel { @@ -79,4 +80,9 @@ public boolean equals(Object other) { return false; } } + + @Override + public int hashCode() { + return Objects.hashCode(value); + } } diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/utils/LabelUtils.java b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/utils/LabelUtils.java index 8fd8f071312..71d4e522ce2 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/utils/LabelUtils.java +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/java/org/apache/linkis/manager/label/utils/LabelUtils.java @@ -28,13 +28,7 @@ import org.apache.commons.lang3.StringUtils; import java.lang.reflect.Method; -import java.util.ArrayList; -import java.util.Collections; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; +import java.util.*; import java.util.stream.Collectors; import com.fasterxml.jackson.core.JsonParser; @@ -126,6 +120,11 @@ public int compareTo(MethodWrapper o) { return this.order - o.order; } + @Override + public int hashCode() { + return Objects.hash(methodName, order); + } + @Override public boolean equals(Object obj) { if (obj instanceof MethodWrapper) { diff --git a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/scala/org/apache/linkis/manager/label/utils/LabelUtil.scala b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/scala/org/apache/linkis/manager/label/utils/LabelUtil.scala index b2214240882..8d584c09267 100644 --- a/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/scala/org/apache/linkis/manager/label/utils/LabelUtil.scala +++ b/linkis-computation-governance/linkis-manager/linkis-label-common/src/main/scala/org/apache/linkis/manager/label/utils/LabelUtil.scala @@ -180,4 +180,21 @@ object LabelUtil { } } + def isTargetEngine( + labels: util.List[Label[_]], + engine: String, + version: String = null + ): Boolean = { + if (null == labels || StringUtils.isBlank(engine)) return false + val engineTypeLabel = getEngineTypeLabel(labels) + if (null != engineTypeLabel) { + val isEngineMatch = engineTypeLabel.getEngineType.equals(engine) + val isVersionMatch = + StringUtils.isBlank(version) || engineTypeLabel.getVersion.contains(version) + isEngineMatch && isVersionMatch + } else { + false + } + } + } diff --git a/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/conf/RMConfiguration.java b/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/conf/RMConfiguration.java index c33956146d2..78065d7b4b9 100644 --- a/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/conf/RMConfiguration.java +++ b/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/conf/RMConfiguration.java @@ -88,4 +88,12 @@ public class RMConfiguration { public static final CommonVars GET_RESOURCE_BY_LABEL_VALUE_ENABLED = CommonVars.apply("wds.linkis.get.resource.by.label.value.enable", false); + + public static final CommonVars YARN_APPS_FILTER_ENABLED = + CommonVars.apply("wds.linkis.rm.yarn.apps.filter.enabled", true); + + public static final CommonVars YARN_APPS_FILTER_PARMS = + CommonVars.apply( + "wds.linkis.rm.yarn.apps.filter.parms", + "&deSelects=resourceRequests,timeouts,appNodeLabelExpression,amNodeLabelExpression,resourceInfo"); } diff --git a/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/entity/persistence/PersistenceNode.java b/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/entity/persistence/PersistenceNode.java index eea1bb5073e..d126a73e55f 100644 --- a/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/entity/persistence/PersistenceNode.java +++ b/linkis-computation-governance/linkis-manager/linkis-manager-common/src/main/java/org/apache/linkis/manager/common/entity/persistence/PersistenceNode.java @@ -64,11 +64,11 @@ public void setTicketId(String ticketId) { this.ticketId = ticketId; } - public Integer getId() { + public int getId() { return id; } - public void setId(Integer id) { + public void setId(int id) { this.id = id; } @@ -128,6 +128,14 @@ public void setCreator(String creator) { this.creator = creator; } + public String getParams() { + return params; + } + + public void setParams(String params) { + this.params = params; + } + public String getMappingPorts() { return mappingPorts; } @@ -143,16 +151,4 @@ public String getMappingHost() { public void setMappingHost(String mappingHost) { this.mappingHost = mappingHost; } - - public void setId(int id) { - this.id = id; - } - - public String getParams() { - return params; - } - - public void setParams(String params) { - this.params = params; - } } diff --git a/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/java/org/apache/linkis/manager/persistence/impl/DefaultLockManagerPersistence.java b/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/java/org/apache/linkis/manager/persistence/impl/DefaultLockManagerPersistence.java index b37ac74c43f..e2dd7245ee7 100644 --- a/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/java/org/apache/linkis/manager/persistence/impl/DefaultLockManagerPersistence.java +++ b/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/java/org/apache/linkis/manager/persistence/impl/DefaultLockManagerPersistence.java @@ -68,14 +68,14 @@ private boolean tryQueueLock(PersistenceLock persistenceLock, Long timeOut) { } persistenceLock.setTimeOut(timeOut); String syncLocker = persistenceLock.getLockObject().intern(); - synchronized (syncLocker) { + synchronized (syncLocker) { // NOSONAR // insert lock The order is determined by the id auto-incrementing number lockManagerMapper.lock(persistenceLock); } boolean isLocked = false; while (!isLocked && System.currentTimeMillis() - startTime < timeOut) { try { - synchronized (syncLocker) { + synchronized (syncLocker) { // NOSONAR isLocked = isAcquireLock(persistenceLock); if (isLocked) { syncLocker.notifyAll(); diff --git a/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml b/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml index bfdeb545854..04ff4286b70 100644 --- a/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml +++ b/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/main/resources/mapper/common/NodeManagerMapper.xml @@ -66,11 +66,8 @@ identifier = #{persistenceNode.identifier}, - - mapping_ports = #{persistenceNode.mappingPorts}, - - - mapping_host = #{persistenceNode.mappingHost}, + + params = #{persistenceNode.params}, params = #{persistenceNode.params}, diff --git a/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/test/java/org/apache/linkis/manager/dao/ResourceManagerMapperTest.java b/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/test/java/org/apache/linkis/manager/dao/ResourceManagerMapperTest.java index c7e719afe2c..f55a2d888aa 100644 --- a/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/test/java/org/apache/linkis/manager/dao/ResourceManagerMapperTest.java +++ b/linkis-computation-governance/linkis-manager/linkis-manager-persistence/src/test/java/org/apache/linkis/manager/dao/ResourceManagerMapperTest.java @@ -78,7 +78,8 @@ void nodeResourceUpdateByResourceId() { persistenceResource.setLeftResource("left"); persistenceResource.setUsedResource("user"); resourceManagerMapper.nodeResourceUpdateByResourceId(1, persistenceResource); - assertTrue(persistenceResource.getMaxResource() == persistenceResource.getMaxResource()); + assertTrue( + persistenceResource.getMaxResource() == persistenceResource.getMaxResource()); // NOSONAR } @Test diff --git a/linkis-dist/package/admin/linkis-audit-log-clear.sh b/linkis-dist/package/admin/linkis-audit-log-clear.sh new file mode 100644 index 00000000000..ba5d6f3cb25 --- /dev/null +++ b/linkis-dist/package/admin/linkis-audit-log-clear.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +#清理linkis本地审计日志，上传至hdfs +#清理条件：按月清理，超过一个月的审计日志将被清理上传 +#本地目录：/data/bdp/audit/linkis/ +#hdfs目录：/appcom/logs/linkis/audit/ + +# 获取当前月份 +current_date=$(date +"%Y-%m") + +# 设置本地目录路径 +local_dir="/data/bdp/audit/linkis/" + +# 设置HDFS目录路径 +hdfs_dir="/appcom/logs/linkis/audit/" + +# 遍历本地目录 +for dir in "$local_dir"* +do + # 检查是否为目录 + if [ -d "$dir" ]; then + # 获取目录名称 + dir_name=$(basename "$dir") + + # 检查目录名称是否为合法日期格式，并且超过一个月 + if [[ $dir_name =~ ^[0-9]{4}-[0-9]{2}$ && "$dir_name" < "$current_date" ]]; then + # 遍历日志文件 + for log_file in "$dir"/linkis-audit-*.log + do + # 检查日志文件名称是否符合指定格式 + if [[ $(basename "$log_file") =~ ^linkis-audit-[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]+\.log$ ]]; then + # 提取年月信息 + year_month=$(basename "$log_file" | grep -oE '[0-9]{4}-[0-9]{2}') + + # 构建HDFS目标路径 + hdfs_target_dir="$hdfs_dir$year_month/" + + # 检查HDFS目录是否存在，不存在则创建 + hdfs dfs -test -d "$hdfs_target_dir" || hdfs dfs -mkdir -p "$hdfs_target_dir" + + # 将日志文件上传至HDFS目录 + hdfs dfs -put "$log_file" "$hdfs_target_dir" + + echo "Uploaded $log_file to $hdfs_target_dir" + + # 上传完成后删除本地文件 + rm "$log_file" + fi + done + # 删除上传完成后的本地目录 + rmdir "$dir" + fi + fi +done \ No newline at end of file diff --git a/linkis-dist/package/conf/application-engineconn.properties b/linkis-dist/package/conf/application-engineconn.properties new file mode 100644 index 00000000000..06b92b1f6d0 --- /dev/null +++ b/linkis-dist/package/conf/application-engineconn.properties @@ -0,0 +1,34 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +spring.application.name=linkis-cg-engineconn +spring.servlet.multipart.max-file-size=500MB +spring.servlet.multipart.max-request-size=500MB +spring.servlet.multipart.file-size-threshold=50MB +spring.main.allow-bean-definition-overriding=true + +eureka.instance.lease-renewal-interval-in-seconds=4 +eureka.instance.lease-expiration-duration-in-seconds=12 +eureka.instance.metadata-map.prometheus.path=${prometheus.path:/api/rest_j/v1/actuator/prometheus} +eureka.instance.metadata-map.linkis.app.version=${linkis.app.version} + +eureka.client.serviceUrl.defaultZone=http://127.0.0.1:20303/eureka/ +eureka.client.registry-fetch-interval-seconds=8 +eureka.client.initial-instance-info-replication-interval-seconds=10 + +management.endpoints.web.exposure.include=refresh,info,health,metrics + +logging.config=classpath:log4j2.xml + diff --git a/linkis-dist/package/conf/application-engineconn.yml b/linkis-dist/package/conf/application-engineconn.yml index 174ddd23057..4f9610513c6 100644 --- a/linkis-dist/package/conf/application-engineconn.yml +++ b/linkis-dist/package/conf/application-engineconn.yml @@ -13,17 +13,6 @@ # limitations under the License. # spring: - cloud: - loadbalancer: - cache: - enabled: false - main: - allow-circular-references: true - mvc: - pathmatch: - matching-strategy: ant_path_matcher - servlet: - path: /api/rest_j/v1 application: name: linkis-cg-engineconn servlet: @@ -52,4 +41,4 @@ management: include: refresh,info,health,metrics logging: - config: classpath:log4j2.xml \ No newline at end of file + config: classpath:log4j2.xml diff --git a/linkis-dist/package/conf/application-eureka.properties b/linkis-dist/package/conf/application-eureka.properties new file mode 100644 index 00000000000..b82409ddca9 --- /dev/null +++ b/linkis-dist/package/conf/application-eureka.properties @@ -0,0 +1,32 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +spring.application.name=linkis-mg-eureka +spring.profiles=eureka + +eureka.instance.hostname=${spring.cloud.client.ip-address} +eureka.instance.metadata-map.prometheus.path=${prometheus.path:/actuator/prometheus} +eureka.instance.metadata-map.linkis.app.version=${linkis.app.version} + +eureka.client.serviceUrl.defaultZone=http://127.0.0.1:20303/eureka/ + +eureka.server.enable-self-preservation=false +eureka.server.eviction-interval-timer-in-ms=5000 +eureka.server.response-cache-update-interval-ms=3000 + +management.endpoints.web.exposure.include=refresh,info,health,metrics + +server.port=20303 + diff --git a/linkis-dist/package/conf/application-linkis.properties b/linkis-dist/package/conf/application-linkis.properties new file mode 100644 index 00000000000..811a58029b9 --- /dev/null +++ b/linkis-dist/package/conf/application-linkis.properties @@ -0,0 +1,39 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +spring.servlet.multipart.max-file-size=500MB +spring.servlet.multipart.max-request-size=500MB +spring.servlet.multipart.file-size-threshold=50MB + +eureka.instance.lease-renewal-interval-in-seconds=4 +eureka.instance.lease-expiration-duration-in-seconds=12 +eureka.instance.metadata-map.prometheus.path=${prometheus.path:${prometheus.endpoint}} +eureka.instance.metadata-map.linkis.app.version=${linkis.app.version} + +eureka.client.serviceUrl.defaultZone=http://127.0.0.1:20303/eureka/ +eureka.client.registry-fetch-interval-seconds=8 + +knife4j.enable=true +knife4j.production=true + +logging.config=classpath:log4j2.xml + +management.endpoints.web.exposure.include=refresh,info,health,metrics + +pagehelper.helper-dialect=mysql +pagehelper.reasonable=true +pagehelper.support-methods-arguments=true +pagehelper.params=countSql + diff --git a/linkis-dist/package/conf/linkis-et-monitor-ims.properties b/linkis-dist/package/conf/linkis-et-monitor-ims.properties new file mode 100644 index 00000000000..22a45841d88 --- /dev/null +++ b/linkis-dist/package/conf/linkis-et-monitor-ims.properties @@ -0,0 +1,143 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# http://www.apache.org/licenses/LICENSE-2.0 +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +jobhistory.errorcode.01002={"alert_title":"Linkis Service load is too high, please contact Linkis owner","alert_info":"Linkis Linkis Service load is too high, please contact Linkis owner","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.errorcode.01003={"alert_title":"Linkis Service load is too high, please contact Linkis owner","alert_info":"Linkis Service load is too high, please contact Linkis owner","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.errorcode.01101={"alert_title":"Linkis Service load is too high, please contact Linkis owner","alert_info":" Service load is too high, please contact Linkis owner","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.errorcode.01102={"alert_title":"Linkis ECM memory Service load is too high, please contact Linkis owner","alert_info":" Service load is too high, please contact Linkis owner","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.errorcode.01103={"alert_title":"Linkis ECM CPU Service load is too high, please contact Linkis owner","alert_info":" Service load is too high, please contact Linkis owner","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.errorcode.01104={"alert_title":"Linkis ECM instances Service load is too high, please contact Linkis owner","alert_info":" Service load is too high, please contact Linkis owner","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.errorcode.01105={"alert_title":"Linkis Memory Service load is too high, please contact Linkis owner","alert_info":"The machine has insufficient memory. Please contact the administrator to expand the memory.","alert_way":"1,2,3","alert_reciver":"hadoop","alert_level":"2","alert_obj":"linkis-alert-obj","can_recover":"0"} +jobhistory.unfinished.time.exceed.sec.43200={"alert_title":"Linkis task execute timeout","alert_info":"Linkis task execute timeout 12h","alert_way":"1,2,3","alert_reciver":"","alert_level":"4","alert_obj":"Linkis","can_recover":"0"} + +ecm.resource.monitor.im.12003=\ + {"alert_title":"Linkis Monitor Resource Alert",\ + "alert_info":" $instance ECM Memory/cpu insufficient resource < 0.2 please contact Linkis owner: $name",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"3",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +ecm.resource.monitor.im.12004=\ + {"alert_title":"Linkis Monitor Resource Alert",\ + "alert_info":" $instance ECM Memory/cpu insufficient resource < 0.2 please contact Linkis owner: $name",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"2",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +ecm.resource.monitor.im.12005=\ + {"alert_title":"Linkis entrance user running task monitor",\ + "alert_info":"User $username runninng task at linkis ( $url ) > $runningtask ,please attention",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"3",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +ecm.resource.monitor.im.12006=\ + {"alert_title":"Linkis entrance user queued task monitor",\ + "alert_info":"User $username queued task at linkis ( $url ) > $queuedtask ,please attention",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"3",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +ecm.resource.monitor.im.12007=\ + {"alert_title":"Linkis entrance user total task monitor",\ + "alert_info":"User $username queued task at linkis ( $url ) > $tasktotal ,please attention",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"3",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +ecm.resource.monitor.im.12008=\ + {"alert_title":"Linkis entrance all running task monitor",\ + "alert_info":"linkis ( $url ) running task > $taskminor,please attention",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"3",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +ecm.resource.monitor.im.12009=\ + {"alert_title":"Linkis entrance all running task monitor",\ + "alert_info":"linkis ( $url ) all task > $taskmajor,please attention",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"2",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} + + + +user.mode.monitor.im.12011=\ + {"alert_title":"User mode execution timeout alarm",\ + "alert_info":"User mode execution timeout alarm Linkis url: $url engineType:$engineType Task ID: $jobId please attention $name",\ + "alert_way":"1,2,3",\ + "alert_reciver":"alexyang,hadoop",\ + "alert_level":"2",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +user.mode.monitor.im.12012=\ + {"alert_title":"User mode execution failure alarm",\ + "alert_info":"User mode execution failure alarm Linkis url: $url Engine: $engineType TaskID: $jobId ER=rrorCode?$errorCode errorMsg: $errorMsg please attention $name",\ + "alert_way":"1,2,3",\ + "alert_reciver":"alexyang,hadoop",\ + "alert_level":"2",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} + +user.mode.monitor.im.12017=\ + {"alert_title":"@alert_title",\ + "alert_info":"task execute failed, reason $msg",\ + "alert_way":"1,2,3",\ + "alert_reciver":"@alert_reciver",\ + "alert_level":"@alert_level",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +user.mode.monitor.im.12018=\ + {"alert_title":"@alert_title",\ + "alert_info":"task execute time out $timeout",\ + "alert_way":"1,2,3",\ + "alert_reciver":"@alert_reciver",\ + "alert_level":"@alert_level",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} + +thread.monitor.timeout.im.12014=\ + {"alert_title":" Linkis Shell Timeout Alert ",\ + "alert_info":"Monitor Shell execute time out $shellName",\ + "alert_way":"1,2,3",\ + "alert_reciver":"hadoop",\ + "alert_level":"3",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +jobhistory.result.monitor.im.12015=\ + {"alert_title":"Linkis \u4EFB\u52A1\u72B6\u6001\u901A\u77E5",\ + "alert_info":"Task ID:$id final status: $status",\ + "alert_way":"1,2,3",\ + "alert_reciver":"",\ + "alert_level":"4",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} +jobhistory.result.monitor.im.12016=\ + {"alert_title":"@alert_title",\ + "alert_info":"[Alarm time]$date\n[Subsystem]$sysid\n[Alarm IP]$ip\n[Alarm object]$object\n[Alarm information]$detail",\ + "alert_way":"1,2,3",\ + "alert_reciver":"@alert_reciver",\ + "alert_level":"@alert_level",\ + "alert_obj":"linkis_alert",\ + "can_recover":"0"} + + diff --git a/linkis-dist/package/conf/linkis-mg-gateway.properties b/linkis-dist/package/conf/linkis-mg-gateway.properties index 0e4275677cc..6e9f88289f6 100644 --- a/linkis-dist/package/conf/linkis-mg-gateway.properties +++ b/linkis-dist/package/conf/linkis-mg-gateway.properties @@ -27,7 +27,7 @@ wds.linkis.login_encrypt.enable=false ##LDAP wds.linkis.ldap.proxy.url= wds.linkis.ldap.proxy.baseDN= -wds.linkis.ldap.proxy.userNameFormat= +#wds.linkis.ldap.proxy.userNameFormat=cn=%s@xxx.com,OU=xxx,DC=xxx,DC=com wds.linkis.admin.user=hadoop #wds.linkis.admin.password= ##OAuth diff --git a/linkis-web-next/.husky/pre-commit b/linkis-dist/package/conf/version.properties similarity index 93% rename from linkis-web-next/.husky/pre-commit rename to linkis-dist/package/conf/version.properties index f61e74e3c8a..0da37e5c037 100644 --- a/linkis-web-next/.husky/pre-commit +++ b/linkis-dist/package/conf/version.properties @@ -1,4 +1,3 @@ -#!/bin/sh # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with @@ -14,6 +13,4 @@ # limitations under the License. # -. "$(dirname "$0")/_/husky.sh" - -npx lint-staged +cli.version=${project.version} \ No newline at end of file diff --git a/linkis-dist/package/db/index/linkis_indexes_mysql.sql b/linkis-dist/package/db/index/linkis_indexes_mysql.sql new file mode 100644 index 00000000000..ec82227e5e8 --- /dev/null +++ b/linkis-dist/package/db/index/linkis_indexes_mysql.sql @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +-- ============================================ +-- Linkis Database Index Definitions (MySQL) +-- ============================================ +-- This file contains all index definitions extracted from linkis_ddl.sql +-- Index naming conventions: +-- - Non-unique indexes: idx_fieldname[_fieldname] +-- - Unique indexes: uniq_fieldname[_fieldname] +-- ============================================ + +-- Configuration Module Indexes +-- linkis_ps_configuration_config_key +ALTER TABLE `linkis_ps_configuration_config_key` ADD UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`); + +-- linkis_ps_configuration_key_engine_relation +ALTER TABLE `linkis_ps_configuration_key_engine_relation` ADD UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `engine_type_label_id`); + +-- linkis_ps_configuration_config_value +ALTER TABLE `linkis_ps_configuration_config_value` ADD UNIQUE INDEX `uniq_kid_lid` (`config_key_id`, `config_label_id`); + +-- linkis_ps_configuration_category +ALTER TABLE `linkis_ps_configuration_category` ADD UNIQUE INDEX `uniq_label_id` (`label_id`); + +-- linkis_ps_configuration_template_config_key +ALTER TABLE `linkis_ps_configuration_template_config_key` ADD UNIQUE INDEX `uniq_tid_kid` (`template_uuid`, `key_id`); +ALTER TABLE `linkis_ps_configuration_template_config_key` ADD UNIQUE INDEX `uniq_tname_kid` (`template_uuid`, `key_id`); + +-- linkis_ps_configuration_key_limit_for_user +ALTER TABLE `linkis_ps_configuration_key_limit_for_user` ADD UNIQUE INDEX `uniq_com_label_kid` (`combined_label_value`, `key_id`); + +-- linkis_ps_configutation_lm_across_cluster_rule +ALTER TABLE `linkis_ps_configutation_lm_across_cluster_rule` ADD UNIQUE KEY `idx_creator_username` (`creator`, `username`); + +-- linkis_ps_configuration_across_cluster_rule +ALTER TABLE `linkis_ps_configuration_across_cluster_rule` ADD UNIQUE KEY `idx_creator_username` (`creator`, `username`); + +-- Job History Module Indexes +-- linkis_ps_job_history_group_history +ALTER TABLE `linkis_ps_job_history_group_history` ADD KEY `idx_created_time` (`created_time`); +ALTER TABLE `linkis_ps_job_history_group_history` ADD KEY `idx_submit_user` (`submit_user`); + +-- linkis_ps_job_history_diagnosis +ALTER TABLE `linkis_ps_job_history_diagnosis` ADD UNIQUE KEY `job_history_id` (`job_history_id`); + +-- Common Lock Module Indexes +-- linkis_ps_common_lock +ALTER TABLE `linkis_ps_common_lock` ADD UNIQUE KEY `uniq_lock_object` (`lock_object`); + +-- UDF Module Indexes +-- linkis_ps_udf_tree +ALTER TABLE `linkis_ps_udf_tree` ADD UNIQUE KEY `uniq_parent_name_uname_category` (`parent`,`name`,`user_name`,`category`); + +-- linkis_ps_udf_user_load +ALTER TABLE `linkis_ps_udf_user_load` ADD UNIQUE KEY `uniq_uid_uname` (`udf_id`, `user_name`); + +-- Variable Module Indexes +-- linkis_ps_variable_key_user +ALTER TABLE `linkis_ps_variable_key_user` ADD UNIQUE KEY `uniq_aid_kid_uname` (`application_id`,`key_id`,`user_name`); +ALTER TABLE `linkis_ps_variable_key_user` ADD KEY `idx_key_id` (`key_id`); +ALTER TABLE `linkis_ps_variable_key_user` ADD KEY `idx_aid` (`application_id`); + +-- linkis_ps_variable_key +ALTER TABLE `linkis_ps_variable_key` ADD KEY `idx_aid` (`application_id`); + +-- DataSource Module Indexes +-- linkis_ps_datasource_table +ALTER TABLE `linkis_ps_datasource_table` ADD UNIQUE KEY `uniq_db_name` (`database`,`name`); + +-- Context Service Module Indexes +-- linkis_ps_cs_context_map +ALTER TABLE `linkis_ps_cs_context_map` ADD UNIQUE KEY `uniq_key_cid_ctype` (`key`,`context_id`,`context_type`); +ALTER TABLE `linkis_ps_cs_context_map` ADD KEY `idx_keywords` (`keywords`(191)); + +-- linkis_ps_cs_context_history +ALTER TABLE `linkis_ps_cs_context_history` ADD KEY `idx_keyword` (`keyword`(191)); + +-- linkis_ps_cs_context_id +ALTER TABLE `linkis_ps_cs_context_id` ADD KEY `idx_instance` (`instance`); +ALTER TABLE `linkis_ps_cs_context_id` ADD KEY `idx_backup_instance` (`backup_instance`); +ALTER TABLE `linkis_ps_cs_context_id` ADD KEY `idx_instance_bin` (`instance`,`backup_instance`); + +-- BML Module Indexes +-- linkis_ps_bml_resources +ALTER TABLE `linkis_ps_bml_resources` ADD UNIQUE KEY `uniq_rid_eflag` (`resource_id`, `enable_flag`); + +-- linkis_ps_bml_resources_version +ALTER TABLE `linkis_ps_bml_resources_version` ADD UNIQUE KEY `uniq_rid_version` (`resource_id`, `version`); + +-- linkis_ps_bml_resources_task +ALTER TABLE `linkis_ps_bml_resources_task` ADD UNIQUE KEY `uniq_rid_version` (`resource_id`, `version`); + +-- linkis_ps_bml_project +ALTER TABLE `linkis_ps_bml_project` ADD UNIQUE KEY `uniq_name` (`name`); + +-- linkis_ps_bml_project_user +ALTER TABLE `linkis_ps_bml_project_user` ADD UNIQUE KEY `uniq_name_pid` (`username`, `project_id`); + +-- linkis_ps_bml_project_resource +ALTER TABLE `linkis_ps_bml_project_resource` ADD INDEX `idx_resource_id` (`resource_id`); + +-- linkis_ps_bml_cleaned_resources_version +ALTER TABLE `linkis_ps_bml_cleaned_resources_version` ADD UNIQUE KEY `resource_id_version` (`resource_id`,`version`); + +-- Instance Label Module Indexes +-- linkis_ps_instance_label +ALTER TABLE `linkis_ps_instance_label` ADD UNIQUE KEY `uniq_lk_lv` (`label_key`,`label_value`); + +-- linkis_ps_instance_label_value_relation +ALTER TABLE `linkis_ps_instance_label_value_relation` ADD UNIQUE KEY `uniq_lvk_lid` (`label_value_key`,`label_id`); + +-- linkis_ps_instance_label_relation +ALTER TABLE `linkis_ps_instance_label_relation` ADD UNIQUE KEY `uniq_lid_instance` (`label_id`,`service_instance`); + +-- linkis_ps_instance_info +ALTER TABLE `linkis_ps_instance_info` ADD UNIQUE KEY `uniq_instance` (`instance`); + +-- Error Code Module Indexes +-- linkis_ps_error_code +ALTER TABLE `linkis_ps_error_code` ADD UNIQUE INDEX `idx_error_regex` (`error_regex`(191)); + +-- Computation Governance Manager Module Indexes +-- linkis_cg_manager_service_instance +ALTER TABLE `linkis_cg_manager_service_instance` ADD UNIQUE KEY `uniq_instance` (`instance`); +ALTER TABLE `linkis_cg_manager_service_instance` ADD INDEX `idx_instance_name` (`instance`, `name`); + +-- linkis_cg_manager_label +ALTER TABLE `linkis_cg_manager_label` ADD UNIQUE KEY `uniq_lk_lv` (`label_key`,`label_value`); + +-- linkis_cg_manager_label_value_relation +ALTER TABLE `linkis_cg_manager_label_value_relation` ADD UNIQUE KEY `uniq_lvk_lid` (`label_value_key`,`label_id`); +ALTER TABLE `linkis_cg_manager_label_value_relation` ADD UNIQUE KEY `unlid_lvk_lvc` (`label_id`,`label_value_key`,`label_value_content`); + +-- linkis_cg_manager_label_resource +ALTER TABLE `linkis_cg_manager_label_resource` ADD UNIQUE KEY `uniq_label_id` (`label_id`); + +-- linkis_cg_ec_resource_info_record +ALTER TABLE `linkis_cg_ec_resource_info_record` ADD KEY `idx_ticket_id` (`ticket_id`); +ALTER TABLE `linkis_cg_ec_resource_info_record` ADD UNIQUE KEY `uniq_tid_lv` (`ticket_id`,`label_value`); +ALTER TABLE `linkis_cg_ec_resource_info_record` ADD UNIQUE KEY `uniq_sinstance_status_cuser_ctime` (`service_instance`, `status`, `create_user`, `create_time`); + +-- linkis_cg_manager_label_service_instance +ALTER TABLE `linkis_cg_manager_label_service_instance` ADD KEY `idx_lid_instance` (`label_id`,`service_instance`); + +-- linkis_cg_tenant_label_config +ALTER TABLE `linkis_cg_tenant_label_config` ADD UNIQUE KEY `uniq_user_creator` (`user`,`creator`); + +-- linkis_cg_user_ip_config +ALTER TABLE `linkis_cg_user_ip_config` ADD UNIQUE KEY `uniq_user_creator` (`user`,`creator`); + +-- linkis_cg_tenant_department_config +ALTER TABLE `linkis_cg_tenant_department_config` ADD UNIQUE KEY `uniq_creator_department` (`creator`,`department`); + +-- DataSource Manager Module Indexes +-- linkis_ps_dm_datasource +ALTER TABLE `linkis_ps_dm_datasource` ADD UNIQUE INDEX `uniq_datasource_name` (`datasource_name`); + +-- linkis_ps_dm_datasource_env +ALTER TABLE `linkis_ps_dm_datasource_env` ADD UNIQUE KEY `uniq_env_name` (`env_name`); +ALTER TABLE `linkis_ps_dm_datasource_env` ADD UNIQUE INDEX `uniq_name_dtid` (`env_name`, `datasource_type_id`); + +-- linkis_ps_dm_datasource_type +ALTER TABLE `linkis_ps_dm_datasource_type` ADD UNIQUE INDEX `uniq_name` (`name`); + +-- linkis_ps_dm_datasource_type_key +ALTER TABLE `linkis_ps_dm_datasource_type_key` ADD UNIQUE KEY `uniq_dstid_key` (`data_source_type_id`, `key`); + +-- Gateway Module Indexes +-- linkis_mg_gateway_auth_token +ALTER TABLE `linkis_mg_gateway_auth_token` ADD UNIQUE KEY `uniq_token_name` (`token_name`); + +-- linkis_mg_gateway_whitelist_config +ALTER TABLE `linkis_mg_gateway_whitelist_config` ADD UNIQUE KEY `address_uniq` (`allowed_user`, `client_address`); +ALTER TABLE `linkis_mg_gateway_whitelist_config` ADD KEY `linkis_mg_gateway_whitelist_config_allowed_user` (`allowed_user`); + +-- linkis_mg_gateway_whitelist_sensitive_user +ALTER TABLE `linkis_mg_gateway_whitelist_sensitive_user` ADD UNIQUE KEY `sensitive_username` (`sensitive_username`); + +-- linkis_mg_gateway_ecc_userinfo +ALTER TABLE `linkis_mg_gateway_ecc_userinfo` ADD UNIQUE INDEX `apply_itsm_id` (`apply_itsm_id`,`user_id`); diff --git a/linkis-dist/package/db/index/linkis_indexes_pg.sql b/linkis-dist/package/db/index/linkis_indexes_pg.sql new file mode 100644 index 00000000000..5782abab038 --- /dev/null +++ b/linkis-dist/package/db/index/linkis_indexes_pg.sql @@ -0,0 +1,195 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +-- ============================================ +-- Linkis Database Index Definitions (PostgreSQL) +-- ============================================ +-- This file contains all index definitions extracted from linkis_ddl_pg.sql +-- Index naming conventions: +-- - Non-unique indexes: idx_fieldname[_fieldname] +-- - Unique indexes: uniq_fieldname[_fieldname] +-- ============================================ + +-- Configuration Module Indexes +-- linkis_ps_configuration_config_key +CREATE UNIQUE INDEX IF NOT EXISTS uniq_key_ectype ON linkis_ps_configuration_config_key USING btree ("key", engine_conn_type); + +-- linkis_ps_configuration_key_engine_relation +CREATE UNIQUE INDEX IF NOT EXISTS uniq_kid_lid ON linkis_ps_configuration_key_engine_relation USING btree (config_key_id, engine_type_label_id); + +-- linkis_ps_configuration_config_value +CREATE UNIQUE INDEX IF NOT EXISTS uniq_kid_lid_config_value ON linkis_ps_configuration_config_value USING btree (config_key_id, config_label_id); + +-- linkis_ps_configuration_category +CREATE UNIQUE INDEX IF NOT EXISTS uniq_label_id_category ON linkis_ps_configuration_category USING btree (label_id); + +-- linkis_ps_configuration_template_config_key +CREATE UNIQUE INDEX IF NOT EXISTS uniq_tid_kid_template ON linkis_ps_configuration_template_config_key USING btree (template_uuid, key_id); +CREATE UNIQUE INDEX IF NOT EXISTS uniq_tname_kid_template ON linkis_ps_configuration_template_config_key USING btree (template_name, key_id); + +-- linkis_ps_configuration_key_limit_for_user +CREATE UNIQUE INDEX IF NOT EXISTS uniq_com_label_kid_limit ON linkis_ps_configuration_key_limit_for_user USING btree (combined_label_value, key_id); + +-- linkis_ps_configutation_lm_across_cluster_rule +CREATE UNIQUE INDEX IF NOT EXISTS idx_creator_username_lm ON linkis_ps_configutation_lm_across_cluster_rule USING btree (creator, username); + +-- linkis_ps_configuration_across_cluster_rule +CREATE UNIQUE INDEX IF NOT EXISTS idx_creator_username_config ON linkis_ps_configuration_across_cluster_rule USING btree (creator, username); + +-- Job History Module Indexes +-- linkis_ps_job_history_group_history +CREATE INDEX IF NOT EXISTS idx_created_time ON linkis_ps_job_history_group_history USING btree (created_time); +CREATE INDEX IF NOT EXISTS idx_submit_user ON linkis_ps_job_history_group_history USING btree (submit_user); + +-- linkis_ps_job_history_diagnosis +CREATE UNIQUE INDEX IF NOT EXISTS job_history_id ON linkis_ps_job_history_diagnosis USING btree (job_history_id); + +-- Common Lock Module Indexes +-- linkis_ps_common_lock +CREATE UNIQUE INDEX IF NOT EXISTS uniq_lock_object ON linkis_ps_common_lock USING btree (lock_object); + +-- UDF Module Indexes +-- linkis_ps_udf_tree +CREATE UNIQUE INDEX IF NOT EXISTS uniq_parent_name_uname_category ON linkis_ps_udf_tree USING btree (parent, "name", user_name, category); + +-- linkis_ps_udf_user_load +CREATE UNIQUE INDEX IF NOT EXISTS uniq_uid_uname ON linkis_ps_udf_user_load USING btree (udf_id, user_name); + +-- Variable Module Indexes +-- linkis_ps_variable_key_user +CREATE UNIQUE INDEX IF NOT EXISTS uniq_aid_kid_uname ON linkis_ps_variable_key_user USING btree (application_id, key_id, user_name); +CREATE INDEX IF NOT EXISTS idx_key_id_variable_user ON linkis_ps_variable_key_user USING btree (key_id); +CREATE INDEX IF NOT EXISTS idx_aid_variable_user ON linkis_ps_variable_key_user USING btree (application_id); + +-- linkis_ps_variable_key +CREATE INDEX IF NOT EXISTS idx_aid_variable_key ON linkis_ps_variable_key USING btree (application_id); + +-- DataSource Module Indexes +-- linkis_ps_datasource_table +CREATE UNIQUE INDEX IF NOT EXISTS uniq_db_name ON linkis_ps_datasource_table USING btree (database, "name"); + +-- Context Service Module Indexes +-- linkis_ps_cs_context_map +CREATE UNIQUE INDEX IF NOT EXISTS uniq_key_cid_ctype ON linkis_ps_cs_context_map USING btree ("key", context_id, context_type); +CREATE INDEX IF NOT EXISTS idx_keywords ON linkis_ps_cs_context_map USING btree (substring(keywords, 1, 191)); + +-- linkis_ps_cs_context_history +CREATE INDEX IF NOT EXISTS idx_keyword ON linkis_ps_cs_context_history USING btree (substring(keyword, 1, 191)); + +-- linkis_ps_cs_context_id +CREATE INDEX IF NOT EXISTS idx_instance ON linkis_ps_cs_context_id USING btree (instance); +CREATE INDEX IF NOT EXISTS idx_backup_instance ON linkis_ps_cs_context_id USING btree (backup_instance); +CREATE INDEX IF NOT EXISTS idx_instance_bin ON linkis_ps_cs_context_id USING btree (instance, backup_instance); + +-- BML Module Indexes +-- linkis_ps_bml_resources +CREATE UNIQUE INDEX IF NOT EXISTS uniq_rid_eflag ON linkis_ps_bml_resources USING btree (resource_id, enable_flag); + +-- linkis_ps_bml_resources_version +CREATE UNIQUE INDEX IF NOT EXISTS uniq_rid_version_bml ON linkis_ps_bml_resources_version USING btree (resource_id, version); + +-- linkis_ps_bml_resources_task +CREATE UNIQUE INDEX IF NOT EXISTS uniq_rid_version_task ON linkis_ps_bml_resources_task USING btree (resource_id, version); + +-- linkis_ps_bml_project +CREATE UNIQUE INDEX IF NOT EXISTS uniq_name_project ON linkis_ps_bml_project USING btree ("name"); + +-- linkis_ps_bml_project_user +CREATE UNIQUE INDEX IF NOT EXISTS uniq_name_pid ON linkis_ps_bml_project_user USING btree (username, project_id); + +-- linkis_ps_bml_project_resource +CREATE INDEX IF NOT EXISTS idx_resource_id ON linkis_ps_bml_project_resource USING btree (resource_id); + +-- linkis_ps_bml_cleaned_resources_version +CREATE UNIQUE INDEX IF NOT EXISTS resource_id_version ON linkis_ps_bml_cleaned_resources_version USING btree (resource_id, version); + +-- Instance Label Module Indexes +-- linkis_ps_instance_label +CREATE UNIQUE INDEX IF NOT EXISTS uniq_lk_lv_instance ON linkis_ps_instance_label USING btree (label_key, label_value); + +-- linkis_ps_instance_label_value_relation +CREATE UNIQUE INDEX IF NOT EXISTS uniq_lvk_lid_instance ON linkis_ps_instance_label_value_relation USING btree (label_value_key, label_id); + +-- linkis_ps_instance_label_relation +CREATE UNIQUE INDEX IF NOT EXISTS uniq_lid_instance ON linkis_ps_instance_label_relation USING btree (label_id, service_instance); + +-- linkis_ps_instance_info +CREATE UNIQUE INDEX IF NOT EXISTS uniq_instance_info ON linkis_ps_instance_info USING btree (instance); + +-- Error Code Module Indexes +-- linkis_ps_error_code +CREATE UNIQUE INDEX IF NOT EXISTS idx_error_regex ON linkis_ps_error_code USING btree (substring(error_regex, 1, 191)); + +-- Computation Governance Manager Module Indexes +-- linkis_cg_manager_service_instance +CREATE UNIQUE INDEX IF NOT EXISTS uniq_instance_manager ON linkis_cg_manager_service_instance USING btree (instance); +CREATE INDEX IF NOT EXISTS idx_instance_name ON linkis_cg_manager_service_instance USING btree (instance, name); + +-- linkis_cg_manager_label +CREATE UNIQUE INDEX IF NOT EXISTS uniq_lk_lv_manager ON linkis_cg_manager_label USING btree (label_key, label_value); + +-- linkis_cg_manager_label_value_relation +CREATE UNIQUE INDEX IF NOT EXISTS uniq_lvk_lid_manager ON linkis_cg_manager_label_value_relation USING btree (label_value_key, label_id); +CREATE UNIQUE INDEX IF NOT EXISTS unlid_lvk_lvc ON linkis_cg_manager_label_value_relation USING btree (label_id, label_value_key, label_value_content); + +-- linkis_cg_manager_label_resource +CREATE UNIQUE INDEX IF NOT EXISTS uniq_label_id_resource ON linkis_cg_manager_label_resource USING btree (label_id); + +-- linkis_cg_ec_resource_info_record +CREATE INDEX IF NOT EXISTS idx_ticket_id ON linkis_cg_ec_resource_info_record USING btree (ticket_id); +CREATE UNIQUE INDEX IF NOT EXISTS uniq_tid_lv ON linkis_cg_ec_resource_info_record USING btree (ticket_id, label_value); +CREATE UNIQUE INDEX IF NOT EXISTS uniq_sinstance_status_cuser_ctime ON linkis_cg_ec_resource_info_record USING btree (service_instance, status, create_user, create_time); + +-- linkis_cg_manager_label_service_instance +CREATE INDEX IF NOT EXISTS idx_lid_instance ON linkis_cg_manager_label_service_instance USING btree (label_id, service_instance); + +-- linkis_cg_tenant_label_config +CREATE UNIQUE INDEX IF NOT EXISTS uniq_user_creator_tenant ON linkis_cg_tenant_label_config USING btree ("user", creator); + +-- linkis_cg_user_ip_config +CREATE UNIQUE INDEX IF NOT EXISTS uniq_user_creator_ip ON linkis_cg_user_ip_config USING btree ("user", creator); + +-- linkis_cg_tenant_department_config +CREATE UNIQUE INDEX IF NOT EXISTS uniq_creator_department ON linkis_cg_tenant_department_config USING btree (creator, department); + +-- DataSource Manager Module Indexes +-- linkis_ps_dm_datasource +CREATE UNIQUE INDEX IF NOT EXISTS uniq_datasource_name ON linkis_ps_dm_datasource USING btree (datasource_name); + +-- linkis_ps_dm_datasource_env +CREATE UNIQUE INDEX IF NOT EXISTS uniq_env_name ON linkis_ps_dm_datasource_env USING btree (env_name); +CREATE UNIQUE INDEX IF NOT EXISTS uniq_name_dtid ON linkis_ps_dm_datasource_env USING btree (env_name, datasource_type_id); + +-- linkis_ps_dm_datasource_type +CREATE UNIQUE INDEX IF NOT EXISTS uniq_name_datasource_type ON linkis_ps_dm_datasource_type USING btree ("name"); + +-- linkis_ps_dm_datasource_type_key +CREATE UNIQUE INDEX IF NOT EXISTS uniq_dstid_key ON linkis_ps_dm_datasource_type_key USING btree (data_source_type_id, "key"); + +-- Gateway Module Indexes +-- linkis_mg_gateway_auth_token +CREATE UNIQUE INDEX IF NOT EXISTS uniq_token_name ON linkis_mg_gateway_auth_token USING btree (token_name); + +-- linkis_mg_gateway_whitelist_config +CREATE UNIQUE INDEX IF NOT EXISTS address_uniq ON linkis_mg_gateway_whitelist_config USING btree (allowed_user, client_address); +CREATE INDEX IF NOT EXISTS linkis_mg_gateway_whitelist_config_allowed_user ON linkis_mg_gateway_whitelist_config USING btree (allowed_user); + +-- linkis_mg_gateway_whitelist_sensitive_user +CREATE UNIQUE INDEX IF NOT EXISTS sensitive_username ON linkis_mg_gateway_whitelist_sensitive_user USING btree (sensitive_username); + +-- linkis_mg_gateway_ecc_userinfo +CREATE UNIQUE INDEX IF NOT EXISTS apply_itsm_id ON linkis_mg_gateway_ecc_userinfo USING btree (apply_itsm_id, user_id); diff --git a/linkis-dist/package/db/linkis_ddl.sql b/linkis-dist/package/db/linkis_ddl.sql index 8356b41dd8f..4b82b914a22 100644 --- a/linkis-dist/package/db/linkis_ddl.sql +++ b/linkis-dist/package/db/linkis_ddl.sql @@ -26,29 +26,30 @@ -- 组合索引建议包含所有字段名,过长的字段名可以采用缩写形式。例如idx_age_name_add -- 索引名尽量不超过50个字符，命名应该使用小写 + SET FOREIGN_KEY_CHECKS=0; DROP TABLE IF EXISTS `linkis_ps_configuration_config_key`; CREATE TABLE `linkis_ps_configuration_config_key`( - `id` bigint(20) NOT NULL AUTO_INCREMENT, - `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', - `description` varchar(200) DEFAULT NULL, - `name` varchar(50) DEFAULT NULL, - `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', - `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', - `validate_range` varchar(150) DEFAULT NULL COMMENT 'Validate range', - `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', - `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', - `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', - `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', - `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', - `boundary_type` TINYINT(2) NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', - `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', - `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', - `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', - `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', - UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), - PRIMARY KEY (`id`) + `id` bigint(20) NOT NULL AUTO_INCREMENT, + `key` varchar(50) DEFAULT NULL COMMENT 'Set key, e.g. spark.executor.instances', + `description` varchar(200) DEFAULT NULL, + `name` varchar(50) DEFAULT NULL, + `default_value` varchar(200) DEFAULT NULL COMMENT 'Adopted when user does not set key', + `validate_type` varchar(50) DEFAULT NULL COMMENT 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules', + `validate_range` varchar(150) COLLATE utf8_bin DEFAULT NULL COMMENT 'Validate range', + `engine_conn_type` varchar(50) DEFAULT '' COMMENT 'engine type,such as spark,hive etc', + `is_hidden` tinyint(1) DEFAULT NULL COMMENT 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end', + `is_advanced` tinyint(1) DEFAULT NULL COMMENT 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so', + `level` tinyint(1) DEFAULT NULL COMMENT 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets', + `treeName` varchar(20) DEFAULT NULL COMMENT 'Reserved field, representing the subdirectory of engineType', + `boundary_type` tinyint(2) NOT NULL DEFAULT '0' COMMENT '0 none/ 1 with mix /2 with max / 3 min and max both', + `en_description` varchar(200) DEFAULT NULL COMMENT 'english description', + `en_name` varchar(100) DEFAULT NULL COMMENT 'english name', + `en_treeName` varchar(100) DEFAULT NULL COMMENT 'english treeName', + `template_required` tinyint(1) DEFAULT 0 COMMENT 'template required 0 none / 1 must', + UNIQUE INDEX `uniq_key_ectype` (`key`,`engine_conn_type`), + PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; @@ -85,7 +86,62 @@ CREATE TABLE `linkis_ps_configuration_category` ( `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), UNIQUE INDEX `uniq_label_id` (`label_id`) -) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; + +DROP TABLE IF EXISTS `linkis_ps_configuration_template_config_key`; +CREATE TABLE IF NOT EXISTS `linkis_ps_configuration_template_config_key` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `template_name` VARCHAR(200) NOT NULL COMMENT 'Configuration template name redundant storage', + `template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid template id recorded by the third party', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT 'configuration value', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT 'upper limit value', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT 'Lower limit value (reserved)', + `validate_range` VARCHAR(50) NULL DEFAULT NULL COMMENT 'Verification regularity (reserved)', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT 'Is it valid? Reserved Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT 'Creator', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT 'Update by', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_tid_kid` (`template_uuid`, `key_id`), + UNIQUE INDEX `uniq_tname_kid` (`template_uuid`, `key_id`) + )ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; + +DROP TABLE IF EXISTS `linkis_ps_configuration_key_limit_for_user`; +CREATE TABLE IF NOT EXISTS `linkis_ps_configuration_key_limit_for_user` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT, + `user_name` VARCHAR(50) NOT NULL COMMENT 'username', + `combined_label_value` VARCHAR(128) NOT NULL COMMENT 'Combined label combined_userCreator_engineType such as hadoop-IDE,spark-2.4.3', + `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', + `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT 'configuration value', + `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT 'upper limit value', + `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT 'Lower limit value (reserved)', + `latest_update_template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid template id recorded by the third party', + `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT 'Is it valid? Reserved Y/N', + `create_by` VARCHAR(50) NOT NULL COMMENT 'Creator', + `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', + `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT 'Update by', + `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', + PRIMARY KEY (`id`), + UNIQUE INDEX `uniq_com_label_kid` (`combined_label_value`, `key_id`) +)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; + +DROP TABLE IF EXISTS `linkis_ps_configutation_lm_across_cluster_rule`; +CREATE TABLE IF NOT EXISTS linkis_ps_configutation_lm_across_cluster_rule ( + id INT AUTO_INCREMENT COMMENT 'Rule ID, auto-increment primary key', + cluster_name char(32) NOT NULL COMMENT 'Cluster name, cannot be empty', + creator char(32) NOT NULL COMMENT 'Creator, cannot be empty', + username char(32) NOT NULL COMMENT 'User, cannot be empty', + create_time datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time, cannot be empty', + create_by char(32) NOT NULL COMMENT 'Creator, cannot be empty', + update_time datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Modification time, cannot be empty', + update_by char(32) NOT NULL COMMENT 'Updater, cannot be empty', + rules varchar(256) NOT NULL COMMENT 'Rule content, cannot be empty', + is_valid VARCHAR(2) DEFAULT 'N' COMMENT 'Is it valid Y/N', + PRIMARY KEY (id), + UNIQUE KEY idx_creator_username (creator, username) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; -- -- New linkis job @@ -616,7 +672,8 @@ create table if not exists linkis_ps_bml_project_resource( `id` int(10) NOT NULL AUTO_INCREMENT, `project_id` int(10) NOT NULL, `resource_id` varchar(128) DEFAULT NULL, -PRIMARY KEY (`id`) + PRIMARY KEY (`id`), + KEY `idx_resource_id` (`resource_id`) )ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPACT; @@ -697,7 +754,8 @@ CREATE TABLE `linkis_cg_manager_service_instance` ( `creator` varchar(32) COLLATE utf8_bin DEFAULT NULL, `params` text COLLATE utf8_bin DEFAULT NULL, PRIMARY KEY (`id`), - UNIQUE KEY `uniq_instance` (`instance`) + UNIQUE KEY `uniq_instance` (`instance`), + KEY `idx_instance_name` (`instance`, `name`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; DROP TABLE IF EXISTS `linkis_cg_manager_linkis_resources`; @@ -991,6 +1049,7 @@ DROP TABLE IF EXISTS `linkis_mg_gateway_auth_token`; CREATE TABLE `linkis_mg_gateway_auth_token` ( `id` int(11) NOT NULL AUTO_INCREMENT, `token_name` varchar(128) NOT NULL, + `token_sign` TEXT DEFAULT NULL COMMENT '存储token密文', `legal_users` text, `legal_hosts` text, `business_owner` varchar(32), @@ -1144,51 +1203,6 @@ CREATE TABLE `linkis_ps_configuration_across_cluster_rule` ( UNIQUE KEY idx_creator_username (creator, username) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; --- ---------------------------- --- Table structure for linkis_ps_configuration_template_config_key --- ---------------------------- -DROP TABLE IF EXISTS `linkis_ps_configuration_template_config_key`; -CREATE TABLE `linkis_ps_configuration_template_config_key` ( - `id` BIGINT(20) NOT NULL AUTO_INCREMENT, - `template_name` VARCHAR(200) NOT NULL COMMENT '配置模板名称 冗余存储', - `template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', - `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', - `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', - `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', - `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值（预留）', - `validate_range` VARCHAR(50) NULL DEFAULT NULL COMMENT '校验正则(预留) ', - `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', - `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', - `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', - `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', - `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', - PRIMARY KEY (`id`), - UNIQUE INDEX `uniq_tid_kid` (`template_uuid`, `key_id`), - UNIQUE INDEX `uniq_tname_kid` (`template_uuid`, `key_id`) -)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; - --- ---------------------------- --- Table structure for linkis_ps_configuration_key_limit_for_user --- ---------------------------- -DROP TABLE IF EXISTS `linkis_ps_configuration_key_limit_for_user`; -CREATE TABLE `linkis_ps_configuration_key_limit_for_user` ( - `id` BIGINT(20) NOT NULL AUTO_INCREMENT, - `user_name` VARCHAR(50) NOT NULL COMMENT '用户名', - `combined_label_value` VARCHAR(128) NOT NULL COMMENT '组合标签 combined_userCreator_engineType 如 hadoop-IDE,spark-2.4.3', - `key_id` BIGINT(20) NOT NULL COMMENT 'id of linkis_ps_configuration_config_key', - `config_value` VARCHAR(200) NULL DEFAULT NULL COMMENT '配置值', - `max_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '上限值', - `min_value` VARCHAR(50) NULL DEFAULT NULL COMMENT '下限值（预留）', - `latest_update_template_uuid` VARCHAR(36) NOT NULL COMMENT 'uuid 第三方侧记录的模板id', - `is_valid` VARCHAR(2) DEFAULT 'Y' COMMENT '是否有效 预留 Y/N', - `create_by` VARCHAR(50) NOT NULL COMMENT '创建人', - `create_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', - `update_by` VARCHAR(50) NULL DEFAULT NULL COMMENT '更新人', - `update_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT 'update time', - PRIMARY KEY (`id`), - UNIQUE INDEX `uniq_com_label_kid` (`combined_label_value`, `key_id`) -)ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; - -- ---------------------------- @@ -1270,26 +1284,60 @@ CREATE TABLE `linkis_ps_python_module_info` ( `update_user` varchar(50) NOT NULL COMMENT '修改用户', `is_load` tinyint(1) NOT NULL DEFAULT '0' COMMENT '是否加载，0-未加载，1-已加载', `is_expire` tinyint(1) DEFAULT NULL COMMENT '是否过期，0-未过期，1-已过期）', + `python_module` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '依赖python模块', `create_time` datetime NOT NULL COMMENT '创建时间', `update_time` datetime NOT NULL COMMENT '修改时间', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='Python模块包信息表'; --- ---------------------------- --- Table structure for linkis_org_user_sync --- ---------------------------- -DROP TABLE IF EXISTS `linkis_org_user_sync`; -CREATE TABLE `linkis_org_user_sync` ( - `cluster_code` varchar(16) COMMENT '集群', - `user_type` varchar(64) COMMENT '用户类型', - `user_name` varchar(128) COMMENT '授权用户', - `org_id` varchar(16) COMMENT '部门ID', - `org_name` varchar(64) COMMENT '部门名字', - `queue_name` varchar(64) COMMENT '默认资源队列', - `db_name` varchar(64) COMMENT '默认操作数据库', - `interface_user` varchar(64) COMMENT '接口人', - `is_union_analyse` varchar(64) COMMENT '是否联合分析人', - `create_time` varchar(64) COMMENT '用户创建时间', - `user_itsm_no` varchar(64) COMMENT '用户创建单号', - PRIMARY KEY (`user_name`) -) ENGINE = InnoDB DEFAULT CHARSET = utf8mb4 COLLATE=utf8mb4_bin COMMENT ='用户部门统计INC表'; \ No newline at end of file + +DROP TABLE IF EXISTS `linkis_ps_job_history_diagnosis`; +CREATE TABLE `linkis_ps_job_history_diagnosis` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `diagnosis_content` text COLLATE utf8mb4_bin COMMENT 'Diagnosis failed task information', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `only_read` varchar(5) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '1 just read,can not update', + PRIMARY KEY (`id`), + UNIQUE KEY `job_history_id` (`job_history_id`) +) ENGINE=InnoDB AUTO_INCREMENT=17 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT ='任务诊断分析表'; + + +CREATE TABLE `linkis_mg_gateway_ecc_userinfo` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID，自增', + `om_tool` VARCHAR(255) NOT NULL COMMENT '工具系统', + `user_id` VARCHAR(255) NOT NULL COMMENT '申请授权用户', + `op_user_id` VARCHAR(255) NOT NULL COMMENT '协助运维账号', + `roles` VARCHAR(255) NOT NULL COMMENT '角色列表，多个逗号,分隔', + `auth_system_id` VARCHAR(500) NOT NULL COMMENT '授权子系统名称ID，多个逗号,分隔', + `apply_itsm_id` VARCHAR(255) NOT NULL COMMENT 'ITSM申请单号，唯一，重复推送时根据这个字段做更新', + `effective_datetime` DATETIME NOT NULL COMMENT '生效时间，允许登录的最早时间', + `expire_datetime` DATETIME NOT NULL COMMENT '失效时间，根据这个时间计算cookie的有效期', + `created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间，默认当前时间', + `updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间，默认当前时间，更新时修改', + PRIMARY KEY (`id`), + UNIQUE INDEX `apply_itsm_id` (`apply_itsm_id`,`user_id`) +) +ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='转协查用户授权表'; + + +-- AI 作业历史记录表 +CREATE TABLE IF NOT EXISTS `linkis_ps_ai_job_history` ( + `id` BIGINT(20) PRIMARY KEY AUTO_INCREMENT COMMENT '主键ID', + `job_history_id` VARCHAR(64) NOT NULL COMMENT '作业历史ID', + `submit_user` VARCHAR(50) NOT NULL COMMENT '提交用户', + `execute_user` VARCHAR(50) NOT NULL COMMENT '执行用户', + `submit_code` TEXT COMMENT '用户提交代码', + `execution_code` TEXT COMMENT '执行代码', + `metrics` text COMMENT 'metrics 信息', + `params` text COMMENT '任务参数', + `labels` text COMMENT '任务标签', + `error_code` int DEFAULT NULL COMMENT '错误码', + `error_desc` TEXT COMMENT '错误信息', + `engine_instances` VARCHAR(250) COMMENT '引擎实例', + `engine_type` VARCHAR(50) COMMENT '引擎类型', + `change_time` DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '切换时间', + `created_time` DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', + `updated_time` DATETIME DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间' +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='AI作业历史记录表'; diff --git a/linkis-dist/package/db/linkis_ddl_pg.sql b/linkis-dist/package/db/linkis_ddl_pg.sql index 5ce8fce3190..258340b4148 100644 --- a/linkis-dist/package/db/linkis_ddl_pg.sql +++ b/linkis-dist/package/db/linkis_ddl_pg.sql @@ -1,1188 +1,1363 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - --- Non-unique indexes are named according to "idx_fieldname[_fieldname]". For example idx_age_name --- The unique index is named according to "uniq_field name[_field name]". For example uniq_age_name --- It is recommended to include all field names for composite indexes, and the long field names can be abbreviated. For example idx_age_name_add --- The index name should not exceed 50 characters, and the name should be lowercase --- --- 非唯一索引按照“idx_字段名称[_字段名称]”进用行命名。例如idx_age_name --- 唯一索引按照“uniq_字段名称[_字段名称]”进用行命名。例如uniq_age_name --- 组合索引建议包含所有字段名,过长的字段名可以采用缩写形式。例如idx_age_name_add --- 索引名尽量不超过50个字符，命名应该使用小写 - - -DROP TABLE IF EXISTS "linkis_ps_configuration_config_key"; -CREATE TABLE linkis_ps_configuration_config_key ( - id bigserial NOT NULL, - "key" varchar(50) NULL, - description varchar(200) NULL, - "name" varchar(50) NULL, - default_value varchar(200) NULL, - validate_type varchar(50) NULL, - validate_range varchar(50) NULL, - engine_conn_type varchar(50) NULL, - is_hidden bool NULL, - is_advanced bool NULL, - "level" int2 NULL, - "treeName" varchar(20) NULL, - boundary_type int2 NOT NULL DEFAULT 0, - en_description varchar(200) NULL, - en_name varchar(100) NULL, - "en_treeName" varchar(100) NULL, - template_required bool NULL, - CONSTRAINT linkis_configuration_config_key_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."key" IS 'Set key, e.g. spark.executor.instances'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."engine_conn_type" IS 'engine type,such as spark,hive etc'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."default_value" IS 'Adopted when user does not set key'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."validate_type" IS 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."validate_range" IS 'Validate range'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."is_hidden" IS 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."is_advanced" IS 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."level" IS 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."treeName" IS 'Reserved field, representing the subdirectory of engineType'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."treeName" IS 'english description'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."treeName" IS 'english name'; -COMMENT ON COLUMN "linkis_ps_configuration_config_key"."treeName" IS 'english treeName'; -CREATE UNIQUE INDEX uniq_key_ectype ON linkis_ps_configuration_config_key USING btree ("key","engine_conn_type"); - -DROP TABLE IF EXISTS "linkis_ps_configuration_key_engine_relation"; -CREATE TABLE linkis_ps_configuration_key_engine_relation ( - id bigserial NOT NULL, - config_key_id int4 NOT NULL, - engine_type_label_id int4 NOT NULL, - CONSTRAINT linkis_ps_configuration_key_engine_relation_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_ckid_etlid ON linkis_ps_configuration_key_engine_relation USING btree (config_key_id, engine_type_label_id); -COMMENT ON COLUMN "linkis_ps_configuration_key_engine_relation"."config_key_id" IS 'config key id'; -COMMENT ON COLUMN "linkis_ps_configuration_key_engine_relation"."engine_type_label_id" IS 'engine label id'; - - -DROP TABLE IF EXISTS "linkis_ps_configuration_config_value"; -CREATE TABLE linkis_ps_configuration_config_value ( - id bigserial NOT NULL, - config_key_id int4 NULL, - config_value varchar(500) NULL, - config_label_id int8 NULL, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_configuration_config_value_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_ckid_clid ON linkis_ps_configuration_config_value USING btree (config_key_id, config_label_id); - - -DROP TABLE IF EXISTS "linkis_ps_configuration_category"; -CREATE TABLE linkis_ps_configuration_category ( - id bigserial NOT NULL, - label_id int4 NOT NULL, - "level" int4 NOT NULL, - description varchar(200) NULL, - tag varchar(200) NULL, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_configuration_category_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_label_id_cc ON linkis_ps_configuration_category USING btree (label_id); - -DROP TABLE IF EXISTS linkis_ps_configuration_template_config_key; -CREATE TABLE linkis_ps_configuration_template_config_key ( - id BIGINT PRIMARY KEY NOT NULL, - template_name VARCHAR(200) NOT NULL, - template_uuid VARCHAR(36) NOT NULL, - key_id BIGINT NOT NULL, - config_value VARCHAR(200) NULL DEFAULT NULL, - max_value VARCHAR(50) NULL DEFAULT NULL, - min_value VARCHAR(50) NULL DEFAULT NULL, - validate_range VARCHAR(50) NULL DEFAULT NULL, - is_valid VARCHAR(2) DEFAULT 'Y', - create_by VARCHAR(50) NOT NULL, - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW(), - update_by VARCHAR(50) NULL DEFAULT NULL, - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW() -); - -DROP TABLE IF EXISTS linkis_ps_configuration_key_limit_for_user; -CREATE TABLE IF NOT EXISTS linkis_ps_configuration_key_limit_for_user ( - id bigint NOT NULL GENERATED ALWAYS AS IDENTITY (START WITH 1), - user_name varchar(50) NOT NULL, - combined_label_value varchar(128) NOT NULL, - key_id bigint NOT NULL, - config_value varchar(200) NULL DEFAULT NULL, - max_value varchar(50) NULL DEFAULT NULL, - min_value varchar(50) NULL DEFAULT NULL, - latest_update_template_uuid varchar(36) NOT NULL, - is_valid varchar(2) DEFAULT 'Y', - create_by varchar(50) NOT NULL, - create_time timestamp without time zone DEFAULT now(), - update_by varchar(50) NULL DEFAULT NULL, - update_time timestamp without time zone DEFAULT now() - ); - -DROP TABLE IF EXISTS linkis_ps_configutation_lm_across_cluster_rule; -CREATE TABLE IF NOT EXISTS linkis_ps_configutation_lm_across_cluster_rule ( - rule_id bigserial NOT NULL, - cluster_name varchar(32) NOT NULL, - creator varchar(32) NOT NULL, - username varchar(32) NOT NULL, - create_time TIMESTAMP NOT NULL DEFAULT NOW(), - create_by varchar(32) NOT NULL, - update_time TIMESTAMP NOT NULL DEFAULT NOW(), - update_by varchar(32), - rules TEXT NOT NULL, - is_valid varchar(1) DEFAULT'N' -); - -DROP TABLE IF EXISTS "linkis_ps_job_history_group_history"; -CREATE TABLE linkis_ps_job_history_group_history ( - id bigserial NOT NULL, - job_req_id varchar(64) NULL, - submit_user varchar(50) NULL, - execute_user varchar(50) NULL, - "source" text NULL, - labels text NULL, - params text NULL, - progress varchar(32) NULL, - status varchar(50) NULL, - log_path varchar(200) NULL, - error_code int4 NULL, - error_desc varchar(1000) NULL, - created_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - updated_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - instances varchar(250) NULL, - metrics text NULL, - engine_type varchar(32) NULL, - execution_code text NULL, - result_location varchar(500) NULL, - observe_info varchar(500) NULL, - CONSTRAINT linkis_ps_job_history_group_history_pkey PRIMARY KEY (id) -); -CREATE INDEX idx_created_time ON linkis_ps_job_history_group_history USING btree (created_time); -CREATE INDEX idx_submit_user ON linkis_ps_job_history_group_history USING btree (submit_user); -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."id" IS 'Primary Key, auto increment'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."job_req_id" IS 'job execId'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."submit_user" IS 'who submitted this Job'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."execute_user" IS 'who actually executed this Job'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."source" IS 'job source'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."labels" IS 'job labels'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."params" IS 'job labels'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."progress" IS 'Job execution progress'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."status" IS 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."log_path" IS 'File path of the job log'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."error_code" IS 'Error code. Generated when the execution of the script fails'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."error_desc" IS 'Execution description. Generated when the execution of script fails'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."created_time" IS 'Creation time'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."updated_time" IS 'Update time'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."instances" IS 'Entrance instances'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."metrics" IS 'Job Metrics'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."engine_type" IS 'Engine type'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."execution_code" IS 'Job origin code or code path'; -COMMENT ON COLUMN "linkis_ps_job_history_group_history"."observe_info" IS 'The notification information configuration of this job'; - - -DROP TABLE IF EXISTS "linkis_ps_job_history_detail"; -CREATE TABLE linkis_ps_job_history_detail ( - id bigserial NOT NULL, - job_history_id int8 NOT NULL, - result_location varchar(500) NULL, - execution_content text NULL, - result_array_size int4 NULL DEFAULT 0, - job_group_info text NULL, - created_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - updated_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - status varchar(32) NULL, - priority int4 NULL DEFAULT 0, - CONSTRAINT linkis_ps_job_history_detail_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_job_history_detail"."id" IS 'Primary Key, auto increment'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."job_history_id" IS 'ID of JobHistory'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."result_location" IS 'File path of the resultsets'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."execution_content" IS 'The script code or other execution content executed by this Job'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."result_array_size" IS 'size of result array'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."job_group_info" IS 'Job group info/path'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."created_time" IS 'Creation time'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."updated_time" IS 'Update time'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."status" IS 'status'; -COMMENT ON COLUMN "linkis_ps_job_history_detail"."priority" IS 'order of subjob'; - - -DROP TABLE IF EXISTS "linkis_ps_common_lock"; -CREATE TABLE linkis_ps_common_lock ( - id bigserial NOT NULL, - lock_object varchar(255) NULL, - locker varchar(255) NOT NULL, - time_out text NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_ps_common_lock_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_lock_object ON linkis_ps_common_lock USING btree (lock_object); - - -DROP TABLE IF EXISTS "linkis_ps_udf_manager"; -CREATE TABLE linkis_ps_udf_manager ( - id bigserial NOT NULL, - user_name varchar(20) NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_udf_manager_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_udf_shared_group"; -CREATE TABLE linkis_ps_udf_shared_group ( - id bigserial NOT NULL, - udf_id int8 NOT NULL, - shared_group varchar(50) NOT NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_udf_shared_group_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_udf_shared_info"; -CREATE TABLE linkis_ps_udf_shared_info ( - id bigserial NOT NULL, - udf_id int8 NOT NULL, - user_name varchar(50) NOT NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_ps_udf_shared_info_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_udf_tree"; -CREATE TABLE linkis_ps_udf_tree ( - id bigserial NOT NULL, - parent int8 NOT NULL, - "name" varchar(100) NULL, - user_name varchar(50) NOT NULL, - description varchar(255) NULL, - create_time timestamp(6) NOT NULL, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - category varchar(50) NULL, - CONSTRAINT linkis_udf_tree_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_udf_tree"."name" IS 'Category name of the function. It would be displayed in the front-end'; -COMMENT ON COLUMN "linkis_ps_udf_tree"."category" IS 'Used to distinguish between udf and function'; -CREATE UNIQUE INDEX uniq_parent_name_uname_category ON linkis_ps_udf_tree USING btree (parent,name,user_name,category); - -DROP TABLE IF EXISTS "linkis_ps_udf_user_load"; -CREATE TABLE linkis_ps_udf_user_load ( - id bigserial NOT NULL, - udf_id int4 NOT NULL, - user_name varchar(50) NOT NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_ps_udf_user_load_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_uid_uname ON linkis_ps_udf_user_load USING btree (udf_id, user_name); - -DROP TABLE IF EXISTS "linkis_ps_udf_baseinfo"; -CREATE TABLE linkis_ps_udf_baseinfo ( - id bigserial NOT NULL, - create_user varchar(50) NOT NULL, - udf_name varchar(255) NOT NULL, - udf_type int4 NULL DEFAULT 0, - tree_id int8 NOT NULL, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - update_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "sys" varchar(255) NOT NULL DEFAULT 'ide', - cluster_name varchar(255) NOT NULL, - is_expire bool NULL, - is_shared bool NULL, - CONSTRAINT linkis_ps_udf_baseinfo_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_udf_version"; -CREATE TABLE linkis_ps_udf_version ( - id bigserial NOT NULL, - udf_id int8 NOT NULL, - "path" varchar(255) NOT NULL, - bml_resource_id varchar(50) NOT NULL, - bml_resource_version varchar(20) NOT NULL, - is_published bool NULL, - register_format varchar(255) NULL, - use_format varchar(255) NULL, - description varchar(255) NOT NULL, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - md5 varchar(100) NULL, - CONSTRAINT linkis_ps_udf_version_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_variable_key_user"; -CREATE TABLE linkis_ps_variable_key_user ( - id bigserial NOT NULL, - application_id int8 NULL, - key_id int8 NULL, - user_name varchar(50) NULL, - value varchar(200) NULL, - CONSTRAINT linkis_var_key_user_pkey PRIMARY KEY (id) -); -CREATE INDEX idx_aid_vku ON linkis_ps_variable_key_user USING btree (application_id); -CREATE UNIQUE INDEX uniq_aid_kid_uname ON linkis_ps_variable_key_user USING btree (application_id, key_id, user_name); -CREATE INDEX idx_key_id ON linkis_ps_variable_key_user USING btree (key_id); -COMMENT ON COLUMN "linkis_ps_variable_key_user"."application_id" IS 'Reserved word'; -COMMENT ON COLUMN "linkis_ps_variable_key_user"."value" IS 'Value of the global variable'; - - -DROP TABLE IF EXISTS "linkis_ps_variable_key"; -CREATE TABLE linkis_ps_variable_key ( - id bigserial NOT NULL, - "key" varchar(50) NULL, - description varchar(200) NULL, - "name" varchar(50) NULL, - application_id int8 NULL, - default_value varchar(200) NULL, - value_type varchar(50) NULL, - value_regex varchar(100) NULL, - CONSTRAINT linkis_var_key_pkey PRIMARY KEY (id) -); -CREATE INDEX idx_aid_vk ON linkis_ps_variable_key USING btree (application_id); -COMMENT ON COLUMN "linkis_ps_variable_key"."key" IS 'Key of the global variable'; -COMMENT ON COLUMN "linkis_ps_variable_key"."description" IS 'Reserved word'; -COMMENT ON COLUMN "linkis_ps_variable_key"."name" IS 'Reserved word'; -COMMENT ON COLUMN "linkis_ps_variable_key"."application_id" IS 'Reserved word'; -COMMENT ON COLUMN "linkis_ps_variable_key"."default_value" IS 'Reserved word'; -COMMENT ON COLUMN "linkis_ps_variable_key"."value_type" IS 'Reserved word'; -COMMENT ON COLUMN "linkis_ps_variable_key"."value_regex" IS 'Reserved word'; - - -DROP TABLE IF EXISTS "linkis_ps_datasource_access"; -CREATE TABLE linkis_ps_datasource_access ( - id bigserial NOT NULL, - table_id int8 NOT NULL, - visitor varchar(16) NOT NULL, - fields varchar(255) NULL, - application_id int4 NOT NULL, - access_time timestamp(6) NOT NULL, - CONSTRAINT linkis_mdq_access_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_datasource_field"; -CREATE TABLE linkis_ps_datasource_field ( - id bigserial NOT NULL, - table_id int8 NOT NULL, - "name" varchar(64) NOT NULL, - alias varchar(64) NULL, - "type" varchar(64) NOT NULL, - "comment" varchar(255) NULL, - express varchar(255) NULL, - "rule" varchar(128) NULL, - is_partition_field bool NOT NULL, - is_primary bool NOT NULL, - length int4 NULL, - mode_info varchar(128) NULL, - CONSTRAINT linkis_mdq_field_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_datasource_import"; -CREATE TABLE linkis_ps_datasource_import ( - id bigserial NOT NULL, - table_id int8 NOT NULL, - import_type int4 NOT NULL, - args varchar(255) NOT NULL, - CONSTRAINT linkis_mdq_import_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_datasource_lineage"; -CREATE TABLE linkis_ps_datasource_lineage ( - id bigserial NOT NULL, - table_id int8 NULL, - source_table varchar(64) NULL, - update_time timestamp(6) NULL, - CONSTRAINT linkis_mdq_lineage_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_datasource_table"; -CREATE TABLE linkis_ps_datasource_table ( - id bigserial NOT NULL, - "database" varchar(64) NOT NULL, - "name" varchar(64) NOT NULL, - alias varchar(64) NULL, - creator varchar(16) NOT NULL, - "comment" varchar(255) NULL, - create_time timestamp(6) NOT NULL, - product_name varchar(64) NULL, - project_name varchar(255) NULL, - "usage" varchar(128) NULL, - lifecycle int4 NOT NULL, - use_way int4 NOT NULL, - is_import bool NOT NULL, - model_level int4 NOT NULL, - is_external_use bool NOT NULL, - is_partition_table bool NOT NULL, - is_available bool NOT NULL, - CONSTRAINT linkis_mdq_table_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_db_name ON linkis_ps_datasource_table USING btree (database, name); - - -DROP TABLE IF EXISTS "linkis_ps_datasource_table_info"; -CREATE TABLE linkis_ps_datasource_table_info ( - id bigserial NOT NULL, - table_id int8 NOT NULL, - table_last_update_time timestamp(6) NOT NULL, - row_num int8 NOT NULL, - file_num int4 NOT NULL, - table_size varchar(32) NOT NULL, - partitions_num int4 NOT NULL, - update_time timestamp(6) NOT NULL, - field_num int4 NOT NULL, - CONSTRAINT linkis_mdq_table_info_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_cs_context_map"; -CREATE TABLE linkis_ps_cs_context_map ( - id serial4 NOT NULL, - "key" varchar(128) NULL, - context_scope varchar(32) NULL, - context_type varchar(32) NULL, - props text NULL, - value text NULL, - context_id int4 NULL, - keywords varchar(255) NULL, - update_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - access_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cs_context_map_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_key_cid_ctype ON linkis_ps_cs_context_map USING btree (key, context_id, context_type); -CREATE INDEX idx_keywords ON linkis_ps_cs_context_map USING btree (keywords); - - -DROP TABLE IF EXISTS "linkis_ps_cs_context_map_listener"; -CREATE TABLE linkis_ps_cs_context_map_listener ( - id serial4 NOT NULL, - listener_source varchar(255) NULL, - key_id int4 NULL, - update_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - access_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cs_context_map_listener_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_cs_context_history"; -CREATE TABLE linkis_ps_cs_context_history ( - id serial4 NOT NULL, - context_id int4 NULL, - "source" text NULL, - context_type varchar(32) NULL, - history_json text NULL, - keyword varchar(255) NULL, - update_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - access_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cs_context_history_pkey PRIMARY KEY (id) -); -CREATE INDEX idx_keyword ON linkis_ps_cs_context_history USING btree (keyword); - - -DROP TABLE IF EXISTS "linkis_ps_cs_context_id"; -CREATE TABLE linkis_ps_cs_context_id ( - id serial4 NOT NULL, - "user" varchar(32) NULL, - application varchar(32) NULL, - "source" varchar(255) NULL, - expire_type varchar(32) NULL, - expire_time timestamp(6) NULL, - "instance" varchar(128) NULL, - backup_instance varchar(255) NULL, - update_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - access_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cs_context_id_pkey PRIMARY KEY (id) -); -CREATE INDEX idx_backup_instance ON linkis_ps_cs_context_id USING btree (backup_instance); -CREATE INDEX idx_instance ON linkis_ps_cs_context_id USING btree (instance); -CREATE INDEX idx_instance_bin ON linkis_ps_cs_context_id USING btree (instance, backup_instance); - - -DROP TABLE IF EXISTS "linkis_ps_cs_context_listener"; -CREATE TABLE linkis_ps_cs_context_listener ( - id serial4 NOT NULL, - listener_source varchar(255) NULL, - context_id int4 NULL, - update_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - access_time timestamptz(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cs_context_listener_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_bml_resources"; -CREATE TABLE linkis_ps_bml_resources ( - id bigserial NOT NULL, - resource_id varchar(50) NOT NULL, - is_private bool NULL DEFAULT false, - resource_header int2 NULL DEFAULT 0, - downloaded_file_name varchar(200) NULL, - sys varchar(100) NOT NULL, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "owner" varchar(200) NOT NULL, - is_expire bool NULL DEFAULT false, - expire_type varchar(50) NULL, - expire_time varchar(50) NULL, - max_version int4 NULL DEFAULT 10, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - updator varchar(50) NULL, - enable_flag bool NOT NULL DEFAULT true, - CONSTRAINT linkis_resources_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_bml_resources"."id" IS '主键'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."resource_id" IS '资源id，资源的uuid'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."is_private" IS '资源是否私有，0表示私有，1表示公开'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."resource_header" IS '分类，0表示未分类，1表示已分类'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."downloaded_file_name" IS '下载时的文件名'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."sys" IS '所属系统'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."create_time" IS '创建时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."owner" IS '资源所属者'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."is_expire" IS '是否过期，0表示不过期，1表示过期'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."expire_type" IS '过期类型，date指到指定日期过期，TIME指时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."expire_time" IS '过期时间，默认一天'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."max_version" IS '默认为10，指保留最新的10个版本'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."update_time" IS '更新时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."updator" IS '更新者'; -COMMENT ON COLUMN "linkis_ps_bml_resources"."enable_flag" IS '状态，1：正常，0：冻结'; -CREATE UNIQUE INDEX uniq_rid_eflag ON linkis_ps_bml_resources USING btree (resource_id, enable_flag); - -DROP TABLE IF EXISTS "linkis_ps_bml_resources_version"; -CREATE TABLE linkis_ps_bml_resources_version ( - id bigserial NOT NULL, - resource_id varchar(50) NOT NULL, - file_md5 varchar(32) NOT NULL, - "version" varchar(20) NOT NULL, - "size" int4 NOT NULL, - start_byte numeric(20) NOT NULL DEFAULT 0, - end_byte numeric(20) NOT NULL DEFAULT 0, - resource varchar(2000) NOT NULL, - description varchar(2000) NULL, - start_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - end_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - client_ip varchar(200) NOT NULL, - updator varchar(50) NULL, - enable_flag bool NOT NULL DEFAULT true, - CONSTRAINT linkis_resources_version_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_rid_version ON linkis_ps_bml_resources_version USING btree (resource_id, version); -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."id" IS '主键'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."resource_id" IS '资源id，资源的uuid'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."file_md5" IS '文件的md5摘要'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."version" IS '资源版本（v 加上 五位数字）'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."size" IS '文件大小'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."resource" IS '资源内容（文件信息 包括 路径和文件名）'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."description" IS '描述'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."start_time" IS '开始时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."end_time" IS '结束时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."client_ip" IS '客户端ip'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."updator" IS '修改者'; -COMMENT ON COLUMN "linkis_ps_bml_resources_version"."enable_flag" IS '状态，1：正常，0：冻结'; - - -DROP TABLE IF EXISTS "linkis_ps_bml_resources_permission"; -CREATE TABLE linkis_ps_bml_resources_permission ( - id bigserial NOT NULL, - resource_id varchar(50) NOT NULL, - "permission" varchar(10) NOT NULL, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "system" varchar(50) DEFAULT 'dss', - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - updator varchar(50) NOT NULL, - CONSTRAINT linkis_resources_permission_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."id" IS '主键'; -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."resource_id" IS '资源id，资源的uuid'; -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."permission" IS '权限代码'; -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."create_time" IS '创建时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."system" IS '创建者'; -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."update_time" IS '更新时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources_permission"."updator" IS '更新者'; - - -DROP TABLE IF EXISTS "linkis_ps_resources_download_history"; -CREATE TABLE linkis_ps_resources_download_history ( - id bigserial NOT NULL, - start_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - end_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - client_ip varchar(200) NOT NULL, - state int2 NOT NULL, - resource_id varchar(50) NOT NULL, - "version" varchar(20) NOT NULL, - downloader varchar(50) NOT NULL, - CONSTRAINT linkis_resources_download_history_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_resources_download_history"."id" IS '主键'; -COMMENT ON COLUMN "linkis_ps_resources_download_history"."start_time" IS '开始时间'; -COMMENT ON COLUMN "linkis_ps_resources_download_history"."end_time" IS '结束时间'; -COMMENT ON COLUMN "linkis_ps_resources_download_history"."client_ip" IS '客户端ip'; -COMMENT ON COLUMN "linkis_ps_resources_download_history"."state" IS '下载状态，0下载成功，1下载失败'; -COMMENT ON COLUMN "linkis_ps_resources_download_history"."downloader" IS '下载者'; - - -DROP TABLE IF EXISTS "linkis_ps_bml_resources_task"; -CREATE TABLE linkis_ps_bml_resources_task ( - id bigserial NOT NULL, - resource_id varchar(50) NULL, - "version" varchar(20) NULL, - operation varchar(20) NOT NULL, - state varchar(20) NOT NULL DEFAULT 'Schduled'::character varying, - submit_user varchar(20) NOT NULL, - "system" varchar(20) NULL DEFAULT 'dss'::character varying, - "instance" varchar(128) NOT NULL, - client_ip varchar(50) NULL, - extra_params text NULL, - err_msg varchar(2000) NULL, - start_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - end_time timestamp(6) NULL, - last_update_time timestamp(6) NOT NULL, - CONSTRAINT linkis_resources_task_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."resource_id" IS '资源id，资源的uuid'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."version" IS '当前操作的资源版本号'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."operation" IS '操作类型.upload = 0, update = 1'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."state" IS '任务当前状态:Schduled, Running, Succeed, Failed,Cancelled'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."submit_user" IS '任务提交用户名'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."system" IS '子系统名 wtss'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."instance" IS '物料库实例'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."client_ip" IS '请求IP'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."extra_params" IS '额外关键信息.如批量删除的资源IDs及versions,删除资源下的所有versions'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."err_msg" IS '任务失败信息.e.getMessage'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."start_time" IS '开始时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."end_time" IS '结束时间'; -COMMENT ON COLUMN "linkis_ps_bml_resources_task"."last_update_time" IS '最后更新时间'; - - -DROP TABLE IF EXISTS "linkis_ps_bml_project"; -CREATE TABLE linkis_ps_bml_project ( - id bigserial NOT NULL, - "name" varchar(128) NULL, - "system" varchar(64) NOT NULL DEFAULT 'dss'::character varying, - "source" varchar(1024) NULL, - description varchar(1024) NULL, - creator varchar(128) NOT NULL, - enabled int2 NULL DEFAULT 1, - create_time timestamp(6) NULL DEFAULT now(), - CONSTRAINT linkis_bml_project_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_name_bp ON linkis_ps_bml_project USING btree (name); - - -DROP TABLE IF EXISTS "linkis_ps_bml_project_user"; -CREATE TABLE linkis_ps_bml_project_user ( - id bigserial NOT NULL, - project_id int4 NOT NULL, - username varchar(64) NULL, - priv int4 NOT NULL, - creator varchar(128) NOT NULL, - create_time timestamp(6) NULL DEFAULT now(), - expire_time timestamp(6) NULL, - CONSTRAINT linkis_bml_project_user_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_name_pid ON linkis_ps_bml_project_user USING btree (username, project_id); - - -DROP TABLE IF EXISTS "linkis_ps_bml_project_resource"; -CREATE TABLE linkis_ps_bml_project_resource ( - id serial4 NOT NULL, - project_id int4 NOT NULL, - resource_id varchar(128) NULL, - CONSTRAINT linkis_bml_project_resource_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_ps_instance_label"; -CREATE TABLE linkis_ps_instance_label ( - id bigserial NOT NULL, - label_key varchar(32) NOT NULL, - label_value varchar(255) NOT NULL, - label_feature varchar(16) NOT NULL, - label_value_size int4 NOT NULL, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_instance_label_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_lk_lv_il ON linkis_ps_instance_label USING btree (label_key, label_value); -COMMENT ON COLUMN "linkis_ps_instance_label"."label_key" IS 'string key'; -COMMENT ON COLUMN "linkis_ps_instance_label"."label_value" IS 'string value'; -COMMENT ON COLUMN "linkis_ps_instance_label"."label_feature" IS 'store the feature of label, but it may be redundant'; -COMMENT ON COLUMN "linkis_ps_instance_label"."label_value_size" IS 'size of key -> value map'; -COMMENT ON COLUMN "linkis_ps_instance_label"."update_time" IS 'update unix timestamp'; -COMMENT ON COLUMN "linkis_ps_instance_label"."create_time" IS 'update unix timestamp'; - - -DROP TABLE IF EXISTS "linkis_ps_instance_label_value_relation"; -CREATE TABLE linkis_ps_instance_label_value_relation ( - id bigserial NOT NULL, - label_value_key varchar(255) NOT NULL, - label_value_content varchar(255) NULL, - label_id int4 NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_ps_instance_label_value_relation_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_lvk_lid_ilvr ON linkis_ps_instance_label_value_relation USING btree (label_value_key, label_id); -COMMENT ON COLUMN "linkis_ps_instance_label_value_relation"."label_value_key" IS 'value key'; -COMMENT ON COLUMN "linkis_ps_instance_label_value_relation"."label_value_content" IS 'value content'; -COMMENT ON COLUMN "linkis_ps_instance_label_value_relation"."label_id" IS 'id reference linkis_ps_instance_label -> id'; -COMMENT ON COLUMN "linkis_ps_instance_label_value_relation"."update_time" IS 'update unix timestamp'; -COMMENT ON COLUMN "linkis_ps_instance_label_value_relation"."create_time" IS 'create unix timestamp'; - - -DROP TABLE IF EXISTS "linkis_ps_instance_label_relation"; -CREATE TABLE linkis_ps_instance_label_relation ( - id bigserial NOT NULL, - label_id int4 NULL, - service_instance varchar(128) NOT NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_instance_label_relation_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_lid_instance ON linkis_ps_instance_label_relation USING btree (label_id, service_instance); -COMMENT ON COLUMN "linkis_ps_instance_label_relation"."label_id" IS 'id reference linkis_ps_instance_label -> id'; -COMMENT ON COLUMN "linkis_ps_instance_label_relation"."service_instance" IS 'structure like ${host|machine}:${port}'; -COMMENT ON COLUMN "linkis_ps_instance_label_relation"."update_time" IS 'update unix timestamp'; -COMMENT ON COLUMN "linkis_ps_instance_label_relation"."create_time" IS 'create unix timestamp'; - - -DROP TABLE IF EXISTS "linkis_ps_instance_info"; -CREATE TABLE linkis_ps_instance_info ( - id bigserial NOT NULL, - "instance" varchar(128) NULL, - "name" varchar(128) NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_instance_info_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_instance_ii ON linkis_ps_instance_info USING btree (instance); -COMMENT ON COLUMN "linkis_ps_instance_info"."instance" IS 'structure like ${host|machine}:${port}'; -COMMENT ON COLUMN "linkis_ps_instance_info"."name" IS 'equal application name in registry'; -COMMENT ON COLUMN "linkis_ps_instance_info"."update_time" IS 'update unix timestamp'; -COMMENT ON COLUMN "linkis_ps_instance_info"."create_time" IS 'create unix timestamp'; - - -DROP TABLE IF EXISTS "linkis_ps_error_code"; -CREATE TABLE linkis_ps_error_code ( - id bigserial NOT NULL, - error_code varchar(50) NOT NULL, - error_desc varchar(1024) NOT NULL, - error_regex varchar(1024) NULL, - error_type int4 NULL DEFAULT 0, - CONSTRAINT linkis_ps_error_code_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX idx_error_regex ON linkis_ps_error_code USING btree (error_regex); - -DROP TABLE IF EXISTS "linkis_cg_manager_service_instance"; -CREATE TABLE linkis_cg_manager_service_instance ( - id serial4 NOT NULL, - "instance" varchar(128) NULL, - "name" varchar(32) NULL, - "owner" varchar(32) NULL, - mark varchar(32) NULL, - identifier varchar(32) NULL, - ticketId varchar(255) NULL DEFAULT NULL, - mapping_host varchar(128) NULL DEFAULT NULL, - mapping_ports varchar(128) NULL DEFAULT NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - updator varchar(32) NULL, - creator varchar(32) NULL, - CONSTRAINT linkis_manager_service_instance_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_instance_msi ON linkis_cg_manager_service_instance USING btree (instance); - - -DROP TABLE IF EXISTS "linkis_cg_manager_linkis_resources"; -CREATE TABLE linkis_cg_manager_linkis_resources ( - id serial4 NOT NULL, - max_resource varchar(1020) NULL, - min_resource varchar(1020) NULL, - used_resource varchar(1020) NULL, - left_resource varchar(1020) NULL, - expected_resource varchar(1020) NULL, - locked_resource varchar(1020) NULL, - "resourceType" varchar(255) NULL, - "ticketId" varchar(255) NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - updator varchar(255) NULL, - creator varchar(255) NULL, - CONSTRAINT linkis_manager_linkis_resources_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_cg_manager_lock"; -CREATE TABLE linkis_cg_manager_lock ( - id serial4 NOT NULL, - lock_object varchar(255) NULL, - time_out text NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_manager_lock_pkey PRIMARY KEY (id) -); - - - -DROP TABLE IF EXISTS "linkis_cg_rm_external_resource_provider"; -CREATE TABLE linkis_cg_rm_external_resource_provider ( - id serial4 NOT NULL, - resource_type varchar(32) NOT NULL, - "name" varchar(32) NOT NULL, - labels varchar(32) NULL, - config text NOT NULL, - CONSTRAINT linkis_external_resource_provider_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_cg_manager_engine_em"; -CREATE TABLE linkis_cg_manager_engine_em ( - id serial4 NOT NULL, - engine_instance varchar(128) NULL, - em_instance varchar(128) NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_manager_engine_em_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_cg_manager_label"; -CREATE TABLE linkis_cg_manager_label ( - id serial4 NOT NULL, - label_key varchar(50) NOT NULL, - label_value varchar(255) NOT NULL, - label_feature varchar(16) NOT NULL, - label_value_size int4 NOT NULL, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_manager_label_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_lk_lv_ml ON linkis_cg_manager_label USING btree (label_key, label_value); - - -DROP TABLE IF EXISTS "linkis_cg_manager_label_value_relation"; -CREATE TABLE linkis_cg_manager_label_value_relation ( - id serial4 NOT NULL, - label_value_key varchar(255) NOT NULL, - label_value_content varchar(255) NULL, - label_id int4 NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cg_manager_label_value_relation_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_lvk_lid_mlvr ON linkis_cg_manager_label_value_relation USING btree (label_value_key, label_id); - - -DROP TABLE IF EXISTS "linkis_cg_manager_label_resource"; -CREATE TABLE linkis_cg_manager_label_resource ( - id serial4 NOT NULL, - label_id int4 NULL, - resource_id int4 NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_manager_label_resource_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_label_id_mlr ON linkis_cg_manager_label_resource USING btree (label_id); - - -DROP TABLE IF EXISTS "linkis_cg_ec_resource_info_record"; -CREATE TABLE linkis_cg_ec_resource_info_record ( - id serial4 NOT NULL, - label_value varchar(255) NOT NULL, - create_user varchar(128) NOT NULL, - service_instance varchar(128) NULL, - ecm_instance varchar(128) NULL, - ticket_id varchar(100) NOT NULL, - status varchar(50) DEFAULT NULL, - log_dir_suffix varchar(128) NULL, - request_times int4 NULL, - request_resource varchar(1020) NULL, - used_times int4 NULL, - used_resource varchar(1020) NULL, - metrics TEXT NULL, - release_times int4 NULL, - released_resource varchar(1020) NULL, - release_time timestamp(6) NULL, - used_time timestamp(6) NULL, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_cg_ec_resource_info_record_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_tid_lv ON linkis_cg_ec_resource_info_record USING btree (ticket_id, label_value); -CREATE INDEX idx_ticket_id ON linkis_cg_ec_resource_info_record USING btree (ticket_id); -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."label_value" IS 'ec labels stringValue'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."create_user" IS 'ec create user'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."service_instance" IS 'ec instance info'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."ecm_instance" IS 'ecm instance info'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."ticket_id" IS 'ec ticket id'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."status" IS 'EC status: Starting,Unlock,Locked,Idle,Busy,Running,ShuttingDown,Failed,Success'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."log_dir_suffix" IS 'log path'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."request_times" IS 'resource request times'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."request_resource" IS 'request resource'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."used_times" IS 'resource used time'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."used_resource" IS 'used resource'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."metrics" IS 'ec metrics'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."release_times" IS 'resource request times'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."released_resource" IS 'request resource'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."release_time" IS 'resource used time'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."used_time" IS 'used time'; -COMMENT ON COLUMN "linkis_cg_ec_resource_info_record"."create_time" IS 'create time'; - - -DROP TABLE IF EXISTS "linkis_cg_manager_label_service_instance"; -CREATE TABLE linkis_cg_manager_label_service_instance ( - id serial4 NOT NULL, - label_id int4 NULL, - service_instance varchar(128) NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_manager_label_service_instance_pkey PRIMARY KEY (id) -); -CREATE INDEX idx_lid_instance ON linkis_cg_manager_label_service_instance USING btree (label_id,service_instance); - - -DROP TABLE IF EXISTS "linkis_cg_manager_label_user"; -CREATE TABLE linkis_cg_manager_label_user ( - id serial4 NOT NULL, - username varchar(255) NULL, - label_id int4 NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_manager_label_user_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_cg_manager_metrics_history"; -CREATE TABLE linkis_cg_manager_metrics_history ( - id serial4 NOT NULL, - instance_status int4 NULL, - overload varchar(255) NULL, - heartbeat_msg varchar(255) NULL, - healthy_status int4 NULL, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - creator varchar(255) NULL, - "ticketID" varchar(255) NULL, - "serviceName" varchar(255) NULL, - "instance" varchar(255) NULL, - CONSTRAINT linkis_cg_manager_metrics_history_pkey PRIMARY KEY (id) -); - - -DROP TABLE IF EXISTS "linkis_cg_manager_service_instance_metrics"; -CREATE TABLE linkis_cg_manager_service_instance_metrics ( - "instance" varchar(128) NOT NULL, - instance_status int4 NULL, - overload varchar(255) NULL, - heartbeat_msg text NULL, - healthy_status varchar(255) NULL, - update_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - description varchar(256) NULL, - CONSTRAINT linkis_manager_service_instance_metrics_pkey PRIMARY KEY (instance) -); - - -DROP TABLE IF EXISTS "linkis_cg_engine_conn_plugin_bml_resources"; -CREATE TABLE linkis_cg_engine_conn_plugin_bml_resources ( - id bigserial NOT NULL, - engine_conn_type varchar(100) NOT NULL, - "version" varchar(100) NULL, - file_name varchar(255) NULL, - file_size int8 NOT NULL DEFAULT 0, - last_modified int8 NULL, - bml_resource_id varchar(100) NOT NULL, - bml_resource_version varchar(200) NOT NULL, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - last_update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_engine_conn_plugin_bml_resources_pkey PRIMARY KEY (id) -); -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."id" IS '主键'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."engine_conn_type" IS 'Engine type'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."version" IS 'version'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."file_name" IS 'file name'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."file_size" IS 'file size'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."last_modified" IS 'File update time'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."bml_resource_id" IS 'Owning system'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."bml_resource_version" IS 'Resource owner'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."create_time" IS 'created time'; -COMMENT ON COLUMN "linkis_cg_engine_conn_plugin_bml_resources"."last_update_time" IS 'update time'; - - - -DROP TABLE IF EXISTS "linkis_ps_dm_datasource"; -CREATE TABLE linkis_ps_dm_datasource ( - id bigserial NOT NULL, - datasource_name varchar(255) NOT NULL, - "datasource_desc" varchar(255) NULL, - datasource_type_id int8 NOT NULL, - create_identify varchar(255) NULL, - create_system varchar(255) NULL, - "parameter" varchar(1024) NULL, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - modify_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_user varchar(255) NULL, - modify_user varchar(255) NULL, - labels varchar(255) NULL, - version_id int8 NULL, - expire boolean DEFAULT '0', - published_version_id int8 NULL, - CONSTRAINT linkis_ps_dm_datasource_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_datasource_name ON linkis_ps_dm_datasource USING btree (datasource_name); - - -DROP TABLE IF EXISTS "linkis_ps_dm_datasource_env"; -CREATE TABLE linkis_ps_dm_datasource_env ( - id bigserial NOT NULL, - env_name varchar(32) NOT NULL, - "env_desc" varchar(255) NULL, - datasource_type_id int8 NOT NULL, - "parameter" varchar(1024) NULL, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_user varchar(255) NULL, - modify_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - modify_user varchar(255) NULL, - CONSTRAINT linkis_ps_dm_datasource_env_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_name_dtid ON linkis_ps_dm_datasource_env USING btree (env_name,datasource_type_id); -CREATE UNIQUE INDEX uniq_env_name ON linkis_ps_dm_datasource_env USING btree (env_name); - - -DROP TABLE IF EXISTS "linkis_ps_dm_datasource_type"; -CREATE TABLE linkis_ps_dm_datasource_type ( - id bigserial NOT NULL, - "name" varchar(32) NOT NULL, - description varchar(255) NULL, - "option" varchar(32) NULL, - "classifier" varchar(32) NOT NULL, - "icon" varchar(255) NULL, - layers int4 NOT NULL, - description_en varchar(255) NULL, - option_en varchar(32) NULL, - classifier_en varchar(32) NULL, - CONSTRAINT linkis_ps_dm_datasource_type_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_name_ddt ON linkis_ps_dm_datasource_type USING btree (name); -COMMENT ON COLUMN "linkis_ps_dm_datasource_type"."description_en" IS 'english description'; -COMMENT ON COLUMN "linkis_ps_dm_datasource_type"."option_en" IS 'english option'; -COMMENT ON COLUMN "linkis_ps_dm_datasource_type"."classifier_en" IS 'english classifier'; - - -DROP TABLE IF EXISTS "linkis_ps_dm_datasource_type_key"; -CREATE TABLE linkis_ps_dm_datasource_type_key ( - id bigserial NOT NULL, - data_source_type_id int8 NOT NULL, - "key" varchar(32) NOT NULL, - "name" varchar(32) NOT NULL, - "name_en" varchar(32) NOT NULL, - default_value varchar(50) NULL, - value_type varchar(50) NOT NULL, - "scope" varchar(50) NULL, - "require" boolean DEFAULT '0', - description varchar(200) NULL, - description_en varchar(200) NULL, - value_regex varchar(200) NULL, - ref_id int8 NULL, - "ref_value" varchar(50) NULL, - "data_source" varchar(200) NULL, - update_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT linkis_ps_dm_datasource_type_key_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_dstid_key ON linkis_ps_dm_datasource_type_key USING btree (data_source_type_id,"key"); - - -DROP TABLE IF EXISTS "linkis_ps_dm_datasource_version"; -CREATE TABLE linkis_ps_dm_datasource_version ( - version_id bigserial NOT NULL, - datasource_id int8 NOT NULL, - "parameter" varchar(2048) NULL, - "comment" varchar(255) NULL, - create_time timestamp(6) NULL DEFAULT CURRENT_TIMESTAMP, - create_user varchar(255) NULL, - CONSTRAINT linkis_ps_dm_datasource_version_pkey PRIMARY KEY (version_id,datasource_id) -); -CREATE UNIQUE INDEX uniq_vid_did ON linkis_ps_dm_datasource_version USING btree (version_id,"datasource_id"); - - -DROP TABLE IF EXISTS "linkis_mg_gateway_auth_token"; -CREATE TABLE linkis_mg_gateway_auth_token ( - id bigserial NOT NULL, - "token_name" varchar(128) NOT NULL, - legal_users text NULL, - legal_hosts text NULL, - "business_owner" varchar(32) NULL, - create_time timestamp(6) NULL, - update_time timestamp(6) NULL, - elapse_day int8 NULL, - update_by varchar(32) NULL, - CONSTRAINT linkis_mg_gateway_auth_token_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_token_name ON linkis_mg_gateway_auth_token USING btree (token_name); - - -DROP TABLE IF EXISTS "linkis_cg_tenant_label_config"; -CREATE TABLE "linkis_cg_tenant_label_config" ( - "id" bigserial NOT NULL, - "user" varchar(50) NOT NULL, - "creator" varchar(50) NOT NULL, - "tenant_value" varchar(128) NOT NULL, - "create_time" timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "update_time" timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "desc" varchar(100) NOT NULL, - "bussiness_user" varchar(50) NOT NULL, - CONSTRAINT linkis_cg_tenant_label_config_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_user_creator_tlc ON linkis_cg_tenant_label_config USING btree ("user",creator); - - -DROP TABLE IF EXISTS "linkis_cg_user_ip_config"; -CREATE TABLE "linkis_cg_user_ip_config" ( - "id" bigserial NOT NULL, - "user" varchar(50) NOT NULL, - "creator" varchar(50) NOT NULL, - "ip_list" text NOT NULL, - "create_time" timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "update_time" timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP, - "desc" varchar(100) NOT NULL, - "bussiness_user" varchar(50) NOT NULL, - CONSTRAINT linkis_cg_user_ip_config_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_user_creator_uic ON linkis_cg_user_ip_config USING btree ("user",creator); - -DROP TABLE IF EXISTS linkis_org_user; - -CREATE TABLE linkis_org_user ( - cluster_code varchar(16) COMMENT 'cluster code', - user_type varchar(64) COMMENT 'user type', - user_name varchar(128) PRIMARY KEY COMMENT 'username', - org_id varchar(16) COMMENT 'org id', - org_name varchar(64) COMMENT 'org name', - queue_name varchar(64) COMMENT 'yarn queue name', - db_name varchar(64) COMMENT 'default db name', - interface_user varchar(64) COMMENT 'interface user', - is_union_analyse varchar(64) COMMENT 'is union analyse', - create_time varchar(64) COMMENT 'create time', - user_itsm_no varchar(64) COMMENT 'user itsm no' -) COMMENT ON TABLE linkis_org_user IS 'user org info'; +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +-- Non-unique indexes are named according to "idx_fieldname[_fieldname]". For example idx_age_name +-- The unique index is named according to "uniq_field name[_field name]". For example uniq_age_name +-- It is recommended to include all field names for composite indexes, and the long field names can be abbreviated. For example idx_age_name_add +-- The index name should not exceed 50 characters, and the name should be lowercase +-- +-- 非唯一索引按照“idx_字段名称[_字段名称]”进用行命名。例如idx_age_name +-- 唯一索引按照“uniq_字段名称[_字段名称]”进用行命名。例如uniq_age_name +-- 组合索引建议包含所有字段名,过长的字段名可以采用缩写形式。例如idx_age_name_add +-- 索引名尽量不超过50个字符，命名应该使用小写 + + +-- 注意事项 +-- 1. TDSQL层面做了硬性规定，对于varchar索引，字段总长度不能超过768个字节，建议组合索引的列的长度根据实际列数值的长度定义，比如身份证号定义长度为varchar(20)，不要定位为varchar(100)， +-- 同时，由于TDSQL默认采用UTF8字符集，一个字符3个字节，因此，实际索引所包含的列的长度要小于768/3=256字符长度。 +-- 2. AOMP 执行sql 语句 create table 可以带反撇号，alter 语句不能带反撇号 +-- 3. 使用 alter 添加、修改字段时请带要字符集和排序规则 CHARSET utf8mb4 COLLATE utf8mb4_bin + +DROP TABLE IF EXISTS "linkis_ps_configuration_config_key"; +CREATE TABLE linkis_ps_configuration_config_key ( + id bigserial NOT NULL, + "key" varchar(50) DEFAULT NULL, + description varchar(200) DEFAULT NULL, + "name" varchar(50) DEFAULT NULL, + default_value varchar(200) DEFAULT NULL, + validate_type varchar(50) DEFAULT NULL, + validate_range varchar(150) DEFAULT NULL, + engine_conn_type varchar(50) DEFAULT '', + is_hidden boolean DEFAULT NULL, + is_advanced boolean DEFAULT NULL, + "level" smallint DEFAULT NULL, + treeName varchar(20) DEFAULT NULL, + boundary_type smallint NOT NULL DEFAULT '0', + en_description varchar(200) DEFAULT NULL, + en_name varchar(100) DEFAULT NULL, + en_treeName varchar(100) DEFAULT NULL, + template_required boolean DEFAULT false, + CONSTRAINT linkis_ps_configuration_config_key_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_key_ectype ON linkis_ps_configuration_config_key USING btree ("key", engine_conn_type); +COMMENT ON COLUMN linkis_ps_configuration_config_key."key" IS 'Set key, e.g. spark.executor.instances'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.default_value IS 'Adopted when user does not set key'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.validate_type IS 'Validate type, one of the following: None, NumInterval, FloatInterval, Include, Regex, OPF, Custom Rules'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.validate_range IS 'Validate range'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.engine_conn_type IS 'engine type,such as spark,hive etc'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.is_hidden IS 'Whether it is hidden from user. If set to 1(true), then user cannot modify, however, it could still be used in back-end'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.is_advanced IS 'Whether it is an advanced parameter. If set to 1(true), parameters would be displayed only when user choose to do so'; +COMMENT ON COLUMN linkis_ps_configuration_config_key."level" IS 'Basis for displaying sorting in the front-end. Higher the level is, higher the rank the parameter gets'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.treeName IS 'Reserved field, representing the subdirectory of engineType'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.boundary_type IS '0 none/ 1 with mix /2 with max / 3 min and max both'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.en_description IS 'english description'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.en_name IS 'english name'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.en_treeName IS 'english treeName'; +COMMENT ON COLUMN linkis_ps_configuration_config_key.template_required IS 'template required 0 none / 1 must'; +DROP TABLE IF EXISTS "linkis_ps_configuration_key_engine_relation"; +CREATE TABLE linkis_ps_configuration_key_engine_relation ( + id bigserial NOT NULL, + config_key_id bigint NOT NULL, + engine_type_label_id bigint NOT NULL, + CONSTRAINT linkis_ps_configuration_key_engine_relation_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_kid_lid ON linkis_ps_configuration_key_engine_relation USING btree (config_key_id, engine_type_label_id); +COMMENT ON COLUMN linkis_ps_configuration_key_engine_relation.config_key_id IS 'config key id'; +COMMENT ON COLUMN linkis_ps_configuration_key_engine_relation.engine_type_label_id IS 'engine label id'; +DROP TABLE IF EXISTS "linkis_ps_configuration_config_value"; +CREATE TABLE linkis_ps_configuration_config_value ( + id bigserial NOT NULL, + config_key_id bigint, + config_value varchar(500), + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_configuration_config_value_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_kid_lid ON linkis_ps_configuration_config_value USING btree (config_key_id, config_label_id); +DROP TABLE IF EXISTS "linkis_ps_configuration_category"; +CREATE TABLE linkis_ps_configuration_category ( + id serial NOT NULL, + label_id integer NOT NULL, + "level" integer NOT NULL, + description varchar(200), + tag varchar(200), + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_configuration_category_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_label_id ON linkis_ps_configuration_category USING btree (label_id); +DROP TABLE IF EXISTS "linkis_ps_configuration_template_config_key"; +CREATE TABLE linkis_ps_configuration_template_config_key ( + id bigserial NOT NULL, + template_name VARCHAR(200) NOT NULL, + template_uuid VARCHAR(36) NOT NULL, + key_id bigint NOT NULL, + config_value VARCHAR(200) NULL DEFAULT NULL, + max_value VARCHAR(50) NULL DEFAULT NULL, + min_value VARCHAR(50) NULL DEFAULT NULL, + validate_range VARCHAR(50) NULL DEFAULT NULL, + is_valid VARCHAR(2) DEFAULT 'Y', + create_by VARCHAR(50) NOT NULL, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + update_by VARCHAR(50) NULL DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_configuration_template_config_key_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_tid_kid ON linkis_ps_configuration_template_config_key USING btree (template_uuid, key_id); +CREATE UNIQUE INDEX uniq_tname_kid ON linkis_ps_configuration_template_config_key USING btree (template_uuid, key_id); +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.template_name IS 'Configuration template name redundant storage'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.template_uuid IS 'uuid template id recorded by the third party'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.key_id IS 'id of linkis_ps_configuration_config_key'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.config_value IS 'configuration value'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.max_value IS 'upper limit value'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.min_value IS 'Lower limit value (reserved)'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.validate_range IS 'Verification regularity (reserved)'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.is_valid IS 'Is it valid? Reserved Y/N'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.create_by IS 'Creator'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.update_by IS 'Update by'; +COMMENT ON COLUMN linkis_ps_configuration_template_config_key.update_time IS 'update time'; +DROP TABLE IF EXISTS "linkis_ps_configuration_key_limit_for_user"; +CREATE TABLE linkis_ps_configuration_key_limit_for_user ( + id bigserial NOT NULL, + user_name VARCHAR(50) NOT NULL, + combined_label_value VARCHAR(128) NOT NULL, + key_id bigint NOT NULL, + config_value VARCHAR(200) NULL DEFAULT NULL, + max_value VARCHAR(50) NULL DEFAULT NULL, + min_value VARCHAR(50) NULL DEFAULT NULL, + latest_update_template_uuid VARCHAR(36) NOT NULL, + is_valid VARCHAR(2) DEFAULT 'Y', + create_by VARCHAR(50) NOT NULL, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + update_by VARCHAR(50) NULL DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_configuration_key_limit_for_user_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_com_label_kid ON linkis_ps_configuration_key_limit_for_user USING btree (combined_label_value, key_id); +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.user_name IS 'username'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.combined_label_value IS 'Combined label combined_userCreator_engineType such as hadoop-IDE,spark-2.4.3'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.key_id IS 'id of linkis_ps_configuration_config_key'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.config_value IS 'configuration value'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.max_value IS 'upper limit value'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.min_value IS 'Lower limit value (reserved)'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.latest_update_template_uuid IS 'uuid template id recorded by the third party'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.is_valid IS 'Is it valid? Reserved Y/N'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.create_by IS 'Creator'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.update_by IS 'Update by'; +COMMENT ON COLUMN linkis_ps_configuration_key_limit_for_user.update_time IS 'update time'; +DROP TABLE IF EXISTS "linkis_ps_configutation_lm_across_cluster_rule"; +CREATE TABLE linkis_ps_configutation_lm_across_cluster_rule ( + id INT AUTO_INCREMENT, + cluster_name char(32) NOT NULL, + creator char(32) NOT NULL, + username char(32) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_by char(32) NOT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_by char(32) NOT NULL, + rules varchar(256) NOT NULL, + is_valid VARCHAR(2) DEFAULT 'N', + CONSTRAINT linkis_ps_configutation_lm_across_cluster_rule_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX idx_creator_username ON linkis_ps_configutation_lm_across_cluster_rule USING btree (creator, username); +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.id IS 'Rule ID, auto-increment primary key'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.cluster_name IS 'Cluster name, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.creator IS 'Creator, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.username IS 'User, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.create_time IS 'Creation time, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.create_by IS 'Creator, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.update_time IS 'Modification time, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.update_by IS 'Updater, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.rules IS 'Rule content, cannot be empty'; +COMMENT ON COLUMN linkis_ps_configutation_lm_across_cluster_rule.is_valid IS 'Is it valid Y/N'; +DROP TABLE IF EXISTS "linkis_ps_job_history_group_history"; +CREATE TABLE linkis_ps_job_history_group_history ( + id bigserial NOT NULL, + job_req_id varchar(64) DEFAULT NULL, + submit_user varchar(50) DEFAULT NULL, + execute_user varchar(50) DEFAULT NULL, + "source" text DEFAULT NULL, + labels text DEFAULT NULL, + params text DEFAULT NULL, + progress varchar(32) DEFAULT NULL, + status varchar(50) DEFAULT NULL, + log_path varchar(200) DEFAULT NULL, + error_code int DEFAULT NULL, + error_desc varchar(1000) DEFAULT NULL, + created_time timestamp(3) DEFAULT CURRENT_TIMESTAMP, + updated_time timestamp(3) DEFAULT CURRENT_TIMESTAMP, + instances varchar(250) DEFAULT NULL, + metrics text DEFAULT NULL, + engine_type varchar(32) DEFAULT NULL, + execution_code text DEFAULT NULL, + result_location varchar(500) DEFAULT NULL, + observe_info varchar(500) DEFAULT NULL, + CONSTRAINT linkis_ps_job_history_group_history_pkey PRIMARY KEY (id) +); +CREATE INDEX idx_created_time ON linkis_ps_job_history_group_history USING btree (created_time); +CREATE INDEX idx_submit_user ON linkis_ps_job_history_group_history USING btree (submit_user); +COMMENT ON COLUMN linkis_ps_job_history_group_history.id IS 'Primary Key, auto increment'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.job_req_id IS 'job execId'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.submit_user IS 'who submitted this Job'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.execute_user IS 'who actually executed this Job'; +COMMENT ON COLUMN linkis_ps_job_history_group_history."source" IS 'job source'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.labels IS 'job labels'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.params IS 'job params'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.progress IS 'Job execution progress'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.status IS 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.log_path IS 'File path of the job log'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.error_code IS 'Error code. Generated when the execution of the script fails'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.error_desc IS 'Execution description. Generated when the execution of script fails'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.created_time IS 'Creation time'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.updated_time IS 'Update time'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.instances IS 'Entrance instances'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.metrics IS 'Job Metrics'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.engine_type IS 'Engine type'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.execution_code IS 'Job origin code or code path'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.result_location IS 'File path of the resultsets'; +COMMENT ON COLUMN linkis_ps_job_history_group_history.observe_info IS 'The notification information configuration of this job'; +DROP TABLE IF EXISTS "linkis_ps_job_history_detail"; +CREATE TABLE linkis_ps_job_history_detail ( + id bigserial NOT NULL, + job_history_id bigint NOT NULL, + result_location varchar(500) DEFAULT NULL, + execution_content text DEFAULT NULL, + result_array_size smallint DEFAULT 0, + job_group_info text DEFAULT NULL, + created_time timestamp(3) DEFAULT CURRENT_TIMESTAMP, + updated_time timestamp(3) DEFAULT CURRENT_TIMESTAMP, + status varchar(32) DEFAULT NULL, + priority smallint DEFAULT 0, + CONSTRAINT linkis_ps_job_history_detail_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_job_history_detail.id IS 'Primary Key, auto increment'; +COMMENT ON COLUMN linkis_ps_job_history_detail.job_history_id IS 'ID of JobHistory'; +COMMENT ON COLUMN linkis_ps_job_history_detail.result_location IS 'File path of the resultsets'; +COMMENT ON COLUMN linkis_ps_job_history_detail.execution_content IS 'The script code or other execution content executed by this Job'; +COMMENT ON COLUMN linkis_ps_job_history_detail.result_array_size IS 'size of result array'; +COMMENT ON COLUMN linkis_ps_job_history_detail.job_group_info IS 'Job group info/path'; +COMMENT ON COLUMN linkis_ps_job_history_detail.created_time IS 'Creation time'; +COMMENT ON COLUMN linkis_ps_job_history_detail.updated_time IS 'Update time'; +COMMENT ON COLUMN linkis_ps_job_history_detail.status IS 'status'; +COMMENT ON COLUMN linkis_ps_job_history_detail.priority IS 'order of subjob'; +DROP TABLE IF EXISTS "linkis_ps_common_lock"; +CREATE TABLE linkis_ps_common_lock ( + id serial NOT NULL, + lock_object varchar(255) DEFAULT NULL, + locker VARCHAR(255) DEFAULT NULL, + time_out text, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_common_lock_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_lock_object ON linkis_ps_common_lock USING btree (lock_object); +COMMENT ON COLUMN linkis_ps_common_lock.locker IS 'locker'; +DROP TABLE IF EXISTS "linkis_ps_udf_manager"; +CREATE TABLE linkis_ps_udf_manager ( + id bigserial NOT NULL, + user_name varchar(20) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_udf_manager_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_udf_shared_group"; +CREATE TABLE linkis_ps_udf_shared_group ( + id bigserial NOT NULL, + udf_id bigint NOT NULL, + shared_group varchar(50) NOT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_udf_shared_group_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_udf_shared_info"; +CREATE TABLE linkis_ps_udf_shared_info ( + id bigint PRIMARY KEY NOT NULL AUTO_INCREMENT, + udf_id bigint NOT NULL, + user_name varchar(50) NOT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP +); +DROP TABLE IF EXISTS "linkis_ps_udf_tree"; +CREATE TABLE linkis_ps_udf_tree ( + id bigserial NOT NULL, + parent bigint NOT NULL, + "name" varchar(50) DEFAULT NULL, + user_name varchar(50) NOT NULL, + description varchar(255) DEFAULT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP , + category varchar(50) DEFAULT NULL, + CONSTRAINT linkis_ps_udf_tree_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_parent_name_uname_category ON linkis_ps_udf_tree USING btree (parent, "name", user_name, category); +COMMENT ON COLUMN linkis_ps_udf_tree."name" IS 'Category name of the function. It would be displayed in the front-end'; +COMMENT ON COLUMN linkis_ps_udf_tree.category IS 'Used to distinguish between udf and function'; +DROP TABLE IF EXISTS "linkis_ps_udf_user_load"; +CREATE TABLE linkis_ps_udf_user_load ( + id bigserial NOT NULL, + udf_id bigint NOT NULL, + user_name varchar(50) NOT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_udf_user_load_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_uid_uname ON linkis_ps_udf_user_load USING btree (udf_id, user_name); +DROP TABLE IF EXISTS "linkis_ps_udf_baseinfo"; +CREATE TABLE linkis_ps_udf_baseinfo ( + id bigserial NOT NULL, + create_user varchar(50) NOT NULL, + udf_name varchar(255) NOT NULL, + udf_type integer DEFAULT '0', + tree_id bigint NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP , + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + sys varchar(255) NOT NULL DEFAULT 'ide', + cluster_name varchar(255) NOT NULL, + is_expire boolean DEFAULT NULL, + is_shared boolean DEFAULT NULL, + CONSTRAINT linkis_ps_udf_baseinfo_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_udf_baseinfo.sys IS 'source system'; +DROP TABLE IF EXISTS "linkis_ps_udf_version"; +CREATE TABLE linkis_ps_udf_version ( + id bigserial NOT NULL, + udf_id bigint NOT NULL, + path varchar(255) NOT NULL, + bml_resource_id varchar(50) NOT NULL, + bml_resource_version varchar(20) NOT NULL, + is_published boolean DEFAULT NULL, + register_format varchar(255) DEFAULT NULL, + use_format varchar(255) DEFAULT NULL, + description varchar(255) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP , + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP , + md5 varchar(100) DEFAULT NULL, + CONSTRAINT linkis_ps_udf_version_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_udf_version.path IS 'Source path for uploading files'; +COMMENT ON COLUMN linkis_ps_udf_version.is_published IS 'is published'; +COMMENT ON COLUMN linkis_ps_udf_version.description IS 'version desc'; +DROP TABLE IF EXISTS "linkis_ps_variable_key_user"; +CREATE TABLE linkis_ps_variable_key_user ( + id bigserial NOT NULL, + application_id bigint DEFAULT NULL, + key_id bigint DEFAULT NULL, + user_name varchar(50) DEFAULT NULL, + value varchar(200) DEFAULT NULL, + CONSTRAINT linkis_ps_variable_key_user_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_aid_kid_uname ON linkis_ps_variable_key_user USING btree (application_id, key_id, user_name); +CREATE INDEX idx_key_id ON linkis_ps_variable_key_user USING btree (key_id); +CREATE INDEX idx_aid ON linkis_ps_variable_key_user USING btree (application_id); +COMMENT ON COLUMN linkis_ps_variable_key_user.application_id IS 'Reserved word'; +COMMENT ON COLUMN linkis_ps_variable_key_user.value IS 'Value of the global variable'; +DROP TABLE IF EXISTS "linkis_ps_variable_key"; +CREATE TABLE linkis_ps_variable_key ( + id bigserial NOT NULL, + "key" varchar(50) DEFAULT NULL, + description varchar(200) DEFAULT NULL, + "name" varchar(50) DEFAULT NULL, + application_id bigint DEFAULT NULL, + default_value varchar(200) DEFAULT NULL, + value_type varchar(50) DEFAULT NULL, + value_regex varchar(100) DEFAULT NULL, + CONSTRAINT linkis_ps_variable_key_pkey PRIMARY KEY (id) +); +CREATE INDEX idx_aid ON linkis_ps_variable_key USING btree (application_id); +COMMENT ON COLUMN linkis_ps_variable_key."key" IS 'Key of the global variable'; +COMMENT ON COLUMN linkis_ps_variable_key.description IS 'Reserved word'; +COMMENT ON COLUMN linkis_ps_variable_key."name" IS 'Reserved word'; +COMMENT ON COLUMN linkis_ps_variable_key.application_id IS 'Reserved word'; +COMMENT ON COLUMN linkis_ps_variable_key.default_value IS 'Reserved word'; +COMMENT ON COLUMN linkis_ps_variable_key.value_type IS 'Reserved word'; +COMMENT ON COLUMN linkis_ps_variable_key.value_regex IS 'Reserved word'; +DROP TABLE IF EXISTS "linkis_ps_datasource_access"; +CREATE TABLE linkis_ps_datasource_access ( + id bigserial NOT NULL, + table_id bigint NOT NULL, + visitor varchar(16) NOT NULL, + fields varchar(255) DEFAULT NULL, + application_id smallint NOT NULL, + access_time timestamp NOT NULL, + CONSTRAINT linkis_ps_datasource_access_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_datasource_field"; +CREATE TABLE linkis_ps_datasource_field ( + id bigserial NOT NULL, + table_id bigint NOT NULL, + "name" varchar(64) NOT NULL, + alias varchar(64) DEFAULT NULL, + "type" varchar(64) NOT NULL, + comment varchar(255) DEFAULT NULL, + express varchar(255) DEFAULT NULL, + rule varchar(128) DEFAULT NULL, + is_partition_field boolean NOT NULL, + is_primary boolean NOT NULL, + length integer DEFAULT NULL, + mode_info varchar(128) DEFAULT NULL, + CONSTRAINT linkis_ps_datasource_field_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_datasource_import"; +CREATE TABLE linkis_ps_datasource_import ( + id bigserial NOT NULL, + table_id bigint NOT NULL, + import_type smallint NOT NULL, + args varchar(255) NOT NULL, + CONSTRAINT linkis_ps_datasource_import_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_datasource_lineage"; +CREATE TABLE linkis_ps_datasource_lineage ( + id bigserial NOT NULL, + table_id bigint DEFAULT NULL, + source_table varchar(64) DEFAULT NULL, + update_time timestamp DEFAULT NULL, + CONSTRAINT linkis_ps_datasource_lineage_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_datasource_table"; +CREATE TABLE linkis_ps_datasource_table ( + id bigserial NOT NULL, + database varchar(64) NOT NULL, + "name" varchar(64) NOT NULL, + alias varchar(64) DEFAULT NULL, + creator varchar(16) NOT NULL, + comment varchar(255) DEFAULT NULL, + create_time timestamp NOT NULL, + product_name varchar(64) DEFAULT NULL, + project_name varchar(255) DEFAULT NULL, + usage varchar(128) DEFAULT NULL, + lifecycle smallint NOT NULL, + use_way smallint NOT NULL, + is_import boolean NOT NULL, + model_level smallint NOT NULL, + is_external_use boolean NOT NULL, + is_partition_table boolean NOT NULL, + is_available boolean NOT NULL, + CONSTRAINT linkis_ps_datasource_table_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_db_name ON linkis_ps_datasource_table USING btree (database, "name"); +DROP TABLE IF EXISTS "linkis_ps_datasource_table_info"; +CREATE TABLE linkis_ps_datasource_table_info ( + id bigserial NOT NULL, + table_id bigint NOT NULL, + table_last_update_time timestamp NOT NULL, + row_num bigint NOT NULL, + file_num integer NOT NULL, + table_size varchar(32) NOT NULL, + partitions_num integer NOT NULL, + update_time timestamp NOT NULL, + field_num integer NOT NULL, + CONSTRAINT linkis_ps_datasource_table_info_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_ps_cs_context_map"; +CREATE TABLE linkis_ps_cs_context_map ( + id serial NOT NULL, + "key" varchar(128) DEFAULT NULL, + context_scope varchar(32) DEFAULT NULL, + context_type varchar(32) DEFAULT NULL, + props text, + value text, + context_id integer DEFAULT NULL, + keywords varchar(255) DEFAULT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + access_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_cs_context_map_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_key_cid_ctype ON linkis_ps_cs_context_map USING btree ("key", context_id, context_type); +CREATE INDEX idx_keywords ON linkis_ps_cs_context_map USING btree (keywords(191); +COMMENT ON COLUMN linkis_ps_cs_context_map.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_cs_context_map.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_cs_context_map.access_time IS 'last access time'; +DROP TABLE IF EXISTS "linkis_ps_cs_context_map_listener"; +CREATE TABLE linkis_ps_cs_context_map_listener ( + id serial NOT NULL, + listener_source varchar(255) DEFAULT NULL, + key_id integer DEFAULT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + access_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_cs_context_map_listener_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_cs_context_map_listener.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_cs_context_map_listener.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_cs_context_map_listener.access_time IS 'last access time'; +DROP TABLE IF EXISTS "linkis_ps_cs_context_history"; +CREATE TABLE linkis_ps_cs_context_history ( + id serial NOT NULL, + context_id integer DEFAULT NULL, + "source" text, + context_type varchar(32) DEFAULT NULL, + history_json text, + keyword varchar(255) DEFAULT NULL, + CONSTRAINT linkis_ps_cs_context_history_pkey PRIMARY KEY (id), + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + access_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX idx_keyword ON linkis_ps_cs_context_history USING btree (keyword(191); +COMMENT ON COLUMN linkis_ps_cs_context_history.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_cs_context_history.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_cs_context_history.access_time IS 'last access time'; +DROP TABLE IF EXISTS "linkis_ps_cs_context_id"; +CREATE TABLE linkis_ps_cs_context_id ( + id serial NOT NULL, + "user" varchar(32) DEFAULT NULL, + application varchar(32) DEFAULT NULL, + "source" varchar(255) DEFAULT NULL, + expire_type varchar(32) DEFAULT NULL, + expire_time timestamp DEFAULT NULL, + instance varchar(64) DEFAULT NULL, + backup_instance varchar(64) DEFAULT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + access_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_cs_context_id_pkey PRIMARY KEY (id) +); +CREATE INDEX idx_instance ON linkis_ps_cs_context_id USING btree (instance); +CREATE INDEX idx_backup_instance ON linkis_ps_cs_context_id USING btree (backup_instance); +CREATE INDEX idx_instance_bin ON linkis_ps_cs_context_id USING btree (instance, backup_instance); +COMMENT ON COLUMN linkis_ps_cs_context_id.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_cs_context_id.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_cs_context_id.access_time IS 'last access time'; +DROP TABLE IF EXISTS "linkis_ps_cs_context_listener"; +CREATE TABLE linkis_ps_cs_context_listener ( + id serial NOT NULL, + listener_source varchar(255) DEFAULT NULL, + context_id integer DEFAULT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + access_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_cs_context_listener_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_cs_context_listener.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_cs_context_listener.create_time IS 'create time'; +COMMENT ON COLUMN linkis_ps_cs_context_listener.access_time IS 'last access time'; +DROP TABLE IF EXISTS "linkis_ps_bml_resources"; +CREATE TABLE linkis_ps_bml_resources ( + id bigserial NOT NULL, + resource_id varchar(50) NOT NULL, + is_private boolean DEFAULT false, + resource_header smallint DEFAULT 0, + downloaded_file_name varchar(200) DEFAULT NULL, + sys varchar(100) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + owner varchar(200) NOT NULL, + is_expire boolean DEFAULT false, + expire_type varchar(50) DEFAULT null, + expire_time varchar(50) DEFAULT null, + max_version integer DEFAULT 10, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + updator varchar(50) DEFAULT NULL, + enable_flag boolean NOT NULL DEFAULT true, + CONSTRAINT linkis_ps_bml_resources_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_rid_eflag ON linkis_ps_bml_resources USING btree (resource_id, enable_flag); +COMMENT ON COLUMN linkis_ps_bml_resources.id IS 'Primary key'; +COMMENT ON COLUMN linkis_ps_bml_resources.resource_id IS 'resource uuid'; +COMMENT ON COLUMN linkis_ps_bml_resources.is_private IS 'Whether the resource is private, 0 means private, 1 means public'; +COMMENT ON COLUMN linkis_ps_bml_resources.resource_header IS 'Classification, 0 means unclassified, 1 means classified'; +COMMENT ON COLUMN linkis_ps_bml_resources.downloaded_file_name IS 'File name when downloading'; +COMMENT ON COLUMN linkis_ps_bml_resources.sys IS 'Owning system'; +COMMENT ON COLUMN linkis_ps_bml_resources.create_time IS 'Created time'; +COMMENT ON COLUMN linkis_ps_bml_resources.owner IS 'Resource owner'; +COMMENT ON COLUMN linkis_ps_bml_resources.is_expire IS 'Whether expired, 0 means not expired, 1 means expired'; +COMMENT ON COLUMN linkis_ps_bml_resources.expire_type IS 'Expiration type, date refers to the expiration on the specified date, TIME refers to the time'; +COMMENT ON COLUMN linkis_ps_bml_resources.expire_time IS 'Expiration time, one day by default'; +COMMENT ON COLUMN linkis_ps_bml_resources.max_version IS 'The default is 10, which means to keep the latest 10 versions'; +COMMENT ON COLUMN linkis_ps_bml_resources.update_time IS 'Updated time'; +COMMENT ON COLUMN linkis_ps_bml_resources.updator IS 'updator'; +COMMENT ON COLUMN linkis_ps_bml_resources.enable_flag IS 'Status, 1: normal, 0: frozen'; +DROP TABLE IF EXISTS "linkis_ps_bml_resources_version"; +CREATE TABLE linkis_ps_bml_resources_version ( + id bigserial NOT NULL, + resource_id varchar(50) NOT NULL, + file_md5 varchar(32) NOT NULL, + version varchar(20) NOT NULL, + size integer NOT NULL, + start_byte bigint UNSIGNED NOT NULL DEFAULT 0, + end_byte bigint UNSIGNED NOT NULL DEFAULT 0, + resource varchar(2000) NOT NULL, + description varchar(2000) DEFAULT NULL, + start_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + end_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + client_ip varchar(200) NOT NULL, + updator varchar(50) DEFAULT NULL, + enable_flag boolean NOT NULL DEFAULT true, + CONSTRAINT linkis_ps_bml_resources_version_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_rid_version ON linkis_ps_bml_resources_version USING btree (resource_id, version); +COMMENT ON COLUMN linkis_ps_bml_resources_version.id IS 'Primary key'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.resource_id IS 'Resource uuid'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.file_md5 IS 'Md5 summary of the file'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.version IS 'Resource version (v plus five digits)'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.size IS 'File size'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.resource IS 'Resource content (file information including path and file name)'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.description IS 'description'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.start_time IS 'Started time'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.end_time IS 'Stoped time'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.client_ip IS 'Client ip'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.updator IS 'updator'; +COMMENT ON COLUMN linkis_ps_bml_resources_version.enable_flag IS 'Status, 1: normal, 0: frozen'; +DROP TABLE IF EXISTS "linkis_ps_bml_resources_permission"; +CREATE TABLE linkis_ps_bml_resources_permission ( + id bigserial NOT NULL, + resource_id varchar(50) NOT NULL, + permission varchar(10) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + "system" varchar(50) default "dss", + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + updator varchar(50) NOT NULL, + CONSTRAINT linkis_ps_bml_resources_permission_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_bml_resources_permission.id IS 'Primary key'; +COMMENT ON COLUMN linkis_ps_bml_resources_permission.resource_id IS 'Resource uuid'; +COMMENT ON COLUMN linkis_ps_bml_resources_permission.permission IS 'permission'; +COMMENT ON COLUMN linkis_ps_bml_resources_permission.create_time IS 'created time'; +COMMENT ON COLUMN linkis_ps_bml_resources_permission."system" IS 'creator'; +COMMENT ON COLUMN linkis_ps_bml_resources_permission.update_time IS 'updated time'; +COMMENT ON COLUMN linkis_ps_bml_resources_permission.updator IS 'updator'; +DROP TABLE IF EXISTS "linkis_ps_resources_download_history"; +CREATE TABLE linkis_ps_resources_download_history ( + id bigserial NOT NULL, + start_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + end_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + client_ip varchar(200) NOT NULL, + state smallint NOT NULL, + resource_id varchar(50) not null, + version varchar(20) not null, + downloader varchar(50) NOT NULL, + CONSTRAINT linkis_ps_resources_download_history_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_resources_download_history.id IS 'primary key'; +COMMENT ON COLUMN linkis_ps_resources_download_history.start_time IS 'start time'; +COMMENT ON COLUMN linkis_ps_resources_download_history.end_time IS 'stop time'; +COMMENT ON COLUMN linkis_ps_resources_download_history.client_ip IS 'client ip'; +COMMENT ON COLUMN linkis_ps_resources_download_history.state IS 'Download status, 0 download successful, 1 download failed'; +COMMENT ON COLUMN linkis_ps_resources_download_history.downloader IS 'Downloader'; +DROP TABLE IF EXISTS "linkis_ps_bml_resources_task"; +CREATE TABLE linkis_ps_bml_resources_task ( + id bigserial NOT NULL, + resource_id varchar(50) DEFAULT NULL, + version varchar(20) DEFAULT NULL, + operation varchar(20) NOT NULL, + state varchar(20) NOT NULL DEFAULT 'Schduled', + submit_user varchar(20) NOT NULL DEFAULT '', + "system" varchar(20) DEFAULT 'dss', + instance varchar(128) NOT NULL, + client_ip varchar(50) DEFAULT NULL, + extra_params text, + err_msg varchar(2000) DEFAULT NULL, + start_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + end_time timestamp DEFAULT NULL, + last_update_time timestamp NOT NULL, + CONSTRAINT linkis_ps_bml_resources_task_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_rid_version ON linkis_ps_bml_resources_task USING btree (resource_id, version); +COMMENT ON COLUMN linkis_ps_bml_resources_task.resource_id IS 'resource uuid'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.version IS 'Resource version number of the current operation'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.operation IS 'Operation type. upload = 0, update = 1'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.state IS 'Current status of the task:Schduled, Running, Succeed, Failed,Cancelled'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.submit_user IS 'Job submission user name'; +COMMENT ON COLUMN linkis_ps_bml_resources_task."system" IS 'Subsystem name: wtss'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.instance IS 'Material library example'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.client_ip IS 'Request IP'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.extra_params IS 'Additional key information. Such as the resource IDs and versions that are deleted in batches, and all versions under the resource are deleted'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.err_msg IS 'Task failure information.e.getMessage'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.start_time IS 'Starting time'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.end_time IS 'End Time'; +COMMENT ON COLUMN linkis_ps_bml_resources_task.last_update_time IS 'Last update time'; +DROP TABLE IF EXISTS "linkis_ps_bml_project"; +CREATE TABLE linkis_ps_bml_project ( + id serial NOT NULL, + "name" varchar(128) DEFAULT NULL, + "system" varchar(64) not null default "dss", + "source" varchar(1024) default null, + description varchar(1024) default null, + creator varchar(128) not null, + enabled smallint default 1, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_bml_project_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_name ON linkis_ps_bml_project USING btree ("name"); +DROP TABLE IF EXISTS "linkis_ps_bml_project_user"; +CREATE TABLE linkis_ps_bml_project_user ( + id serial NOT NULL, + project_id integer NOT NULL, + username varchar(64) DEFAULT NULL, + priv integer not null default 7, -- rwx 421 The permission value is 7. 8 is the administrator, which can authorize other users, + creator varchar(128) not null, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + expire_time timestamp default null, + CONSTRAINT linkis_ps_bml_project_user_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_name_pid ON linkis_ps_bml_project_user USING btree (username, project_id); +DROP TABLE IF EXISTS "linkis_ps_bml_project_resource"; +CREATE TABLE linkis_ps_bml_project_resource ( + id serial NOT NULL, + project_id integer NOT NULL, + resource_id varchar(128) DEFAULT NULL, + CONSTRAINT linkis_ps_bml_project_resource_pkey PRIMARY KEY (id) +); +CREATE INDEX idx_resource_id ON linkis_ps_bml_project_resource USING btree (resource_id); +DROP TABLE IF EXISTS "linkis_ps_instance_label"; +CREATE TABLE linkis_ps_instance_label ( + id serial NOT NULL, + label_key varchar(32) NOT NULL, + label_value varchar(128) NOT NULL, + label_feature varchar(16) NOT NULL, + label_value_size integer NOT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_instance_label_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_lk_lv ON linkis_ps_instance_label USING btree (label_key, label_value); +COMMENT ON COLUMN linkis_ps_instance_label.label_key IS 'string key'; +COMMENT ON COLUMN linkis_ps_instance_label.label_value IS 'string value'; +COMMENT ON COLUMN linkis_ps_instance_label.label_feature IS 'store the feature of label, but it may be redundant'; +COMMENT ON COLUMN linkis_ps_instance_label.label_value_size IS 'size of key -> value map'; +COMMENT ON COLUMN linkis_ps_instance_label.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_instance_label.create_time IS 'update unix timestamp'; +DROP TABLE IF EXISTS "linkis_ps_instance_label_value_relation"; +CREATE TABLE linkis_ps_instance_label_value_relation ( + id serial NOT NULL, + label_value_key varchar(128) NOT NULL, + label_value_content varchar(255) DEFAULT NULL, + label_id integer DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_instance_label_value_relation_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_lvk_lid ON linkis_ps_instance_label_value_relation USING btree (label_value_key, label_id); +COMMENT ON COLUMN linkis_ps_instance_label_value_relation.label_value_key IS 'value key'; +COMMENT ON COLUMN linkis_ps_instance_label_value_relation.label_value_content IS 'value content'; +COMMENT ON COLUMN linkis_ps_instance_label_value_relation.label_id IS 'id reference linkis_ps_instance_label -> id'; +COMMENT ON COLUMN linkis_ps_instance_label_value_relation.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_instance_label_value_relation.create_time IS 'create unix timestamp'; +DROP TABLE IF EXISTS "linkis_ps_instance_label_relation"; +CREATE TABLE linkis_ps_instance_label_relation ( + id serial NOT NULL, + label_id integer DEFAULT NULL, + service_instance varchar(128) NOT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_instance_label_relation_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_lid_instance ON linkis_ps_instance_label_relation USING btree (label_id, service_instance); +COMMENT ON COLUMN linkis_ps_instance_label_relation.label_id IS 'id reference linkis_ps_instance_label -> id'; +COMMENT ON COLUMN linkis_ps_instance_label_relation.service_instance IS 'structure like ${host|machine}:${port}'; +COMMENT ON COLUMN linkis_ps_instance_label_relation.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_instance_label_relation.create_time IS 'create unix timestamp'; +DROP TABLE IF EXISTS "linkis_ps_instance_info"; +CREATE TABLE linkis_ps_instance_info ( + id serial NOT NULL, + instance varchar(128) DEFAULT NULL, + "name" varchar(128) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_instance_info_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_instance ON linkis_ps_instance_info USING btree (instance); +COMMENT ON COLUMN linkis_ps_instance_info.instance IS 'structure like ${host|machine}:${port}'; +COMMENT ON COLUMN linkis_ps_instance_info."name" IS 'equal application name in registry'; +COMMENT ON COLUMN linkis_ps_instance_info.update_time IS 'update unix timestamp'; +COMMENT ON COLUMN linkis_ps_instance_info.create_time IS 'create unix timestamp'; +DROP TABLE IF EXISTS "linkis_ps_error_code"; +CREATE TABLE linkis_ps_error_code ( + id bigserial NOT NULL, + error_code varchar(50) NOT NULL, + error_desc varchar(1024) NOT NULL, + error_regex varchar(1024) DEFAULT NULL, + error_type int(3) DEFAULT 0, + CONSTRAINT linkis_ps_error_code_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX idx_error_regex ON linkis_ps_error_code USING btree (error_regex(191); +DROP TABLE IF EXISTS "linkis_cg_manager_service_instance"; +CREATE TABLE linkis_cg_manager_service_instance ( + id serial NOT NULL, + instance varchar(128) DEFAULT NULL, + "name" varchar(32) DEFAULT NULL, + owner varchar(32) DEFAULT NULL, + mark varchar(32) DEFAULT NULL, + identifier varchar(32) DEFAULT NULL, + ticketId varchar(255) DEFAULT NULL, + mapping_host varchar(128) DEFAULT NULL, + mapping_ports varchar(128) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + updator varchar(32) DEFAULT NULL, + creator varchar(32) DEFAULT NULL, + params text DEFAULT NULL, + CONSTRAINT linkis_cg_manager_service_instance_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_instance ON linkis_cg_manager_service_instance USING btree (instance); +CREATE INDEX idx_instance_name ON linkis_cg_manager_service_instance USING btree (instance, "name"); +DROP TABLE IF EXISTS "linkis_cg_manager_linkis_resources"; +CREATE TABLE linkis_cg_manager_linkis_resources ( + id serial NOT NULL, + max_resource varchar(1020) DEFAULT NULL, + min_resource varchar(1020) DEFAULT NULL, + used_resource varchar(1020) DEFAULT NULL, + left_resource varchar(1020) DEFAULT NULL, + expected_resource varchar(1020) DEFAULT NULL, + locked_resource varchar(1020) DEFAULT NULL, + resourceType varchar(255) DEFAULT NULL, + ticketId varchar(255) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + updator varchar(255) DEFAULT NULL, + creator varchar(255) DEFAULT NULL, + CONSTRAINT linkis_cg_manager_linkis_resources_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_cg_manager_lock"; +CREATE TABLE linkis_cg_manager_lock ( + id serial NOT NULL, + lock_object varchar(255) DEFAULT NULL, + time_out text, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_lock_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_cg_rm_external_resource_provider"; +CREATE TABLE linkis_cg_rm_external_resource_provider ( + id serial NOT NULL, + resource_type varchar(32) NOT NULL, + "name" varchar(32) NOT NULL, + labels varchar(32) DEFAULT NULL, + config text NOT NULL, + CONSTRAINT linkis_cg_rm_external_resource_provider_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_cg_manager_engine_em"; +CREATE TABLE linkis_cg_manager_engine_em ( + id serial NOT NULL, + engine_instance varchar(128) DEFAULT NULL, + em_instance varchar(128) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_engine_em_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_cg_manager_label"; +CREATE TABLE linkis_cg_manager_label ( + id serial NOT NULL, + label_key varchar(32) NOT NULL, + label_value varchar(128) NOT NULL, + label_feature varchar(16) NOT NULL, + label_value_size integer NOT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_label_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_lk_lv ON linkis_cg_manager_label USING btree (label_key, label_value); +DROP TABLE IF EXISTS "linkis_cg_manager_label_value_relation"; +CREATE TABLE linkis_cg_manager_label_value_relation ( + id serial NOT NULL, + label_value_key varchar(128) NOT NULL, + label_value_content varchar(255) DEFAULT NULL, + label_id integer DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_label_value_relation_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_lvk_lid ON linkis_cg_manager_label_value_relation USING btree (label_value_key, label_id); +CREATE UNIQUE INDEX unlid_lvk_lvc ON linkis_cg_manager_label_value_relation USING btree (label_id, label_value_key, label_value_content); +DROP TABLE IF EXISTS "linkis_cg_manager_label_resource"; +CREATE TABLE linkis_cg_manager_label_resource ( + id serial NOT NULL, + label_id integer DEFAULT NULL, + resource_id integer DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_label_resource_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_label_id ON linkis_cg_manager_label_resource USING btree (label_id); +DROP TABLE IF EXISTS "linkis_cg_ec_resource_info_record"; +CREATE TABLE linkis_cg_ec_resource_info_record ( + id serial NOT NULL, + label_value VARCHAR(128) NOT NULL, + create_user VARCHAR(128) NOT NULL, + service_instance varchar(128) DEFAULT NULL, + ecm_instance varchar(128) DEFAULT NULL, + ticket_id VARCHAR(36) NOT NULL, + status varchar(50) DEFAULT NULL, + log_dir_suffix varchar(128) DEFAULT NULL, + request_times INT(8), + request_resource VARCHAR(1020), + used_times INT(8), + used_resource VARCHAR(1020), + metrics TEXT DEFAULT NULL, + release_times INT(8), + released_resource VARCHAR(1020), + release_time timestamp DEFAULT NULL, + used_time timestamp DEFAULT NULL, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_ec_resource_info_record_pkey PRIMARY KEY (id) +); +CREATE INDEX idx_ticket_id ON linkis_cg_ec_resource_info_record USING btree (ticket_id); +CREATE UNIQUE INDEX uniq_tid_lv ON linkis_cg_ec_resource_info_record USING btree (ticket_id, label_value); +CREATE UNIQUE INDEX uniq_sinstance_status_cuser_ctime ON linkis_cg_ec_resource_info_record USING btree (service_instance, status, create_user, create_time); +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.label_value IS 'ec labels stringValue'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.create_user IS 'ec create user'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.service_instance IS 'ec instance info'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.ecm_instance IS 'ecm instance info '; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.ticket_id IS 'ec ticket id'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.status IS 'EC status: Starting,Unlock,Locked,Idle,Busy,Running,ShuttingDown,Failed,Success'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.log_dir_suffix IS 'log path'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.request_times IS 'resource request times'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.request_resource IS 'request resource'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.used_times IS 'resource used times'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.used_resource IS 'used resource'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.metrics IS 'ec metrics'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.release_times IS 'resource released times'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.released_resource IS 'released resource'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.release_time IS 'released time'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.used_time IS 'used time'; +COMMENT ON COLUMN linkis_cg_ec_resource_info_record.create_time IS 'create time'; +DROP TABLE IF EXISTS "linkis_cg_manager_label_service_instance"; +CREATE TABLE linkis_cg_manager_label_service_instance ( + id serial NOT NULL, + label_id integer DEFAULT NULL, + service_instance varchar(128) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_label_service_instance_pkey PRIMARY KEY (id) +); +CREATE INDEX idx_lid_instance ON linkis_cg_manager_label_service_instance USING btree (label_id, service_instance); +DROP TABLE IF EXISTS "linkis_cg_manager_label_user"; +CREATE TABLE linkis_cg_manager_label_user ( + id serial NOT NULL, + username varchar(255) DEFAULT NULL, + label_id integer DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_manager_label_user_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_cg_manager_metrics_history"; +CREATE TABLE linkis_cg_manager_metrics_history ( + id serial NOT NULL, + instance_status integer DEFAULT NULL, + overload varchar(255) DEFAULT NULL, + heartbeat_msg varchar(255) DEFAULT NULL, + healthy_status integer DEFAULT NULL, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + creator varchar(255) DEFAULT NULL, + ticketID varchar(255) DEFAULT NULL, + serviceName varchar(255) DEFAULT NULL, + instance varchar(255) DEFAULT NULL, + CONSTRAINT linkis_cg_manager_metrics_history_pkey PRIMARY KEY (id) +); +DROP TABLE IF EXISTS "linkis_cg_manager_service_instance_metrics"; +CREATE TABLE linkis_cg_manager_service_instance_metrics ( + instance varchar(128) NOT NULL, + instance_status integer DEFAULT NULL, + overload varchar(255) DEFAULT NULL, + heartbeat_msg text DEFAULT NULL, + healthy_status varchar(255) DEFAULT NULL, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + description varchar(256) NOT NULL DEFAULT '', + CONSTRAINT linkis_cg_manager_service_instance_metrics_pkey PRIMARY KEY (instance) +); +DROP TABLE IF EXISTS "linkis_cg_engine_conn_plugin_bml_resources"; +CREATE TABLE linkis_cg_engine_conn_plugin_bml_resources ( + id bigserial NOT NULL, + engine_conn_type varchar(100) NOT NULL, + version varchar(100), + file_name varchar(255), + file_size bigint DEFAULT 0 NOT NULL, + last_modified bigint, + bml_resource_id varchar(100) NOT NULL, + bml_resource_version varchar(200) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + last_update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_cg_engine_conn_plugin_bml_resources_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.id IS 'Primary key'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.engine_conn_type IS 'Engine type'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.version IS 'version'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.file_name IS 'file name'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.file_size IS 'file size'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.last_modified IS 'File update time'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.bml_resource_id IS 'Owning system'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.bml_resource_version IS 'Resource owner'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.create_time IS 'created time'; +COMMENT ON COLUMN linkis_cg_engine_conn_plugin_bml_resources.last_update_time IS 'updated time'; +DROP TABLE IF EXISTS "linkis_ps_dm_datasource"; +CREATE TABLE linkis_ps_dm_datasource ( + id serial NOT NULL, + datasource_name varchar(255) NOT NULL, + datasource_desc varchar(255) DEFAULT NULL, + datasource_type_id integer NOT NULL, + create_identify varchar(255) DEFAULT NULL, + create_system varchar(255) DEFAULT NULL, + parameter varchar(2048) NULL DEFAULT NULL, + create_time timestamp NULL DEFAULT CURRENT_TIMESTAMP, + modify_time timestamp NULL DEFAULT CURRENT_TIMESTAMP, + create_user varchar(255) DEFAULT NULL, + modify_user varchar(255) DEFAULT NULL, + labels varchar(255) DEFAULT NULL, + version_id integer DEFAULT NULL, + expire boolean DEFAULT false, + published_version_id integer DEFAULT NULL, + CONSTRAINT linkis_ps_dm_datasource_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_datasource_name ON linkis_ps_dm_datasource USING btree (datasource_name); +COMMENT ON COLUMN linkis_ps_dm_datasource.version_id IS 'current version id'; +DROP TABLE IF EXISTS "linkis_ps_dm_datasource_env"; +CREATE TABLE linkis_ps_dm_datasource_env ( + id serial NOT NULL, + env_name varchar(32) NOT NULL, + env_desc varchar(255) DEFAULT NULL, + datasource_type_id integer NOT NULL, + parameter varchar(2048) DEFAULT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_user varchar(255) NULL DEFAULT NULL, + modify_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + modify_user varchar(255) NULL DEFAULT NULL, + CONSTRAINT linkis_ps_dm_datasource_env_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_env_name ON linkis_ps_dm_datasource_env USING btree (env_name); +CREATE UNIQUE INDEX uniq_name_dtid ON linkis_ps_dm_datasource_env USING btree (env_name, datasource_type_id); +DROP TABLE IF EXISTS "linkis_ps_dm_datasource_type"; +CREATE TABLE linkis_ps_dm_datasource_type ( + id serial NOT NULL, + "name" varchar(32) NOT NULL, + description varchar(255) DEFAULT NULL, + "option" varchar(32) DEFAULT NULL, + classifier varchar(32) NOT NULL, + icon varchar(255) DEFAULT NULL, + layers int(3) NOT NULL, + description_en varchar(255) DEFAULT NULL, + option_en varchar(32) DEFAULT NULL, + classifier_en varchar(32) DEFAULT NULL, + CONSTRAINT linkis_ps_dm_datasource_type_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_name ON linkis_ps_dm_datasource_type USING btree ("name"); +COMMENT ON COLUMN linkis_ps_dm_datasource_type.description_en IS 'english description'; +COMMENT ON COLUMN linkis_ps_dm_datasource_type.option_en IS 'english option'; +COMMENT ON COLUMN linkis_ps_dm_datasource_type.classifier_en IS 'english classifier'; +DROP TABLE IF EXISTS "linkis_ps_dm_datasource_type_key"; +CREATE TABLE linkis_ps_dm_datasource_type_key ( + id serial NOT NULL, + data_source_type_id integer NOT NULL, + "key" varchar(32) NOT NULL, + "name" varchar(32) NOT NULL, + name_en varchar(32) NULL DEFAULT NULL, + default_value varchar(50) NULL DEFAULT NULL, + value_type varchar(50) NOT NULL, + scope varchar(50) NULL DEFAULT NULL, + require boolean NULL DEFAULT false, + description varchar(200) NULL DEFAULT NULL, + description_en varchar(200) NULL DEFAULT NULL, + value_regex varchar(200) NULL DEFAULT NULL, + ref_id bigint NULL DEFAULT NULL, + ref_value varchar(50) NULL DEFAULT NULL, + data_source varchar(200) NULL DEFAULT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_ps_dm_datasource_type_key_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_dstid_key ON linkis_ps_dm_datasource_type_key USING btree (data_source_type_id, "key"); +DROP TABLE IF EXISTS "linkis_ps_dm_datasource_version"; +CREATE TABLE linkis_ps_dm_datasource_version ( + version_id serial NOT NULL, + datasource_id integer NOT NULL, + parameter varchar(2048) NULL DEFAULT NULL, + comment varchar(255) NULL DEFAULT NULL, + create_time timestamp NULL DEFAULT CURRENT_TIMESTAMP, + create_user varchar(255) NULL DEFAULT NULL +); +DROP TABLE IF EXISTS "linkis_mg_gateway_auth_token"; +CREATE TABLE linkis_mg_gateway_auth_token ( + id serial NOT NULL, + token_name varchar(128) NOT NULL, + token_sign TEXT DEFAULT NULL, + legal_users text, + legal_hosts text, + business_owner varchar(32), + create_time DATE DEFAULT NULL, + update_time DATE DEFAULT NULL, + elapse_day BIGINT DEFAULT NULL, + update_by varchar(32), + CONSTRAINT linkis_mg_gateway_auth_token_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_token_name ON linkis_mg_gateway_auth_token USING btree (token_name); +DROP TABLE IF EXISTS "linkis_cg_tenant_label_config"; +CREATE TABLE linkis_cg_tenant_label_config ( + id serial NOT NULL, + "user" varchar(50) NOT NULL, + creator varchar(50) NOT NULL, + tenant_value varchar(128) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + "desc" varchar(100) NOT NULL, + bussiness_user varchar(50) NOT NULL, + is_valid varchar(1) NOT NULL DEFAULT 'Y', + CONSTRAINT linkis_cg_tenant_label_config_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_user_creator ON linkis_cg_tenant_label_config USING btree ("user", creator); +COMMENT ON COLUMN linkis_cg_tenant_label_config.is_valid IS 'is valid'; +DROP TABLE IF EXISTS "linkis_cg_user_ip_config"; +CREATE TABLE linkis_cg_user_ip_config ( + id serial NOT NULL, + "user" varchar(50) NOT NULL, + creator varchar(50) NOT NULL, + ip_list text NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + "desc" varchar(100) NOT NULL, + bussiness_user varchar(50) NOT NULL, + CONSTRAINT linkis_cg_user_ip_config_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_user_creator ON linkis_cg_user_ip_config USING btree ("user", creator); +DROP TABLE IF EXISTS "linkis_org_user"; +CREATE TABLE linkis_org_user ( + cluster_code varchar(16), + user_type varchar(64), + user_name varchar(128), + org_id varchar(16), + org_name varchar(64), + queue_name varchar(64), + db_name varchar(64), + interface_user varchar(64), + is_union_analyse varchar(64), + create_time varchar(64), + user_itsm_no varchar(64), + CONSTRAINT linkis_org_user_pkey PRIMARY KEY (user_name) +); +COMMENT ON COLUMN linkis_org_user.cluster_code IS 'cluster code'; +COMMENT ON COLUMN linkis_org_user.user_type IS 'user type'; +COMMENT ON COLUMN linkis_org_user.user_name IS 'username'; +COMMENT ON COLUMN linkis_org_user.org_id IS 'org id'; +COMMENT ON COLUMN linkis_org_user.org_name IS 'org name'; +COMMENT ON COLUMN linkis_org_user.queue_name IS 'yarn queue name'; +COMMENT ON COLUMN linkis_org_user.db_name IS 'default db name'; +COMMENT ON COLUMN linkis_org_user.interface_user IS 'interface user'; +COMMENT ON COLUMN linkis_org_user.is_union_analyse IS 'is union analyse'; +COMMENT ON COLUMN linkis_org_user.create_time IS 'create time'; +COMMENT ON COLUMN linkis_org_user.user_itsm_no IS 'user itsm no'; +DROP TABLE IF EXISTS "linkis_cg_synckey"; +CREATE TABLE linkis_cg_synckey ( + username char(32) NOT NULL, + synckey char(32) NOT NULL, + instance varchar(32) NOT NULL, + create_time timestamp(3) NOT NULL +); +DROP TABLE IF EXISTS "linkis_et_validator_checkinfo"; +CREATE TABLE linkis_et_validator_checkinfo ( + id bigserial NOT NULL, + execute_user varchar(64) NOT NULL, + db_name varchar(64) DEFAULT NULL, + params text, + code_type varchar(32) NOT NULL, + operation_type varchar(32) NOT NULL, + status smallint(4) DEFAULT NULL, + code text, + msg text, + risk_level varchar(32) DEFAULT NULL, + hit_rules text, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT linkis_et_validator_checkinfo_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_et_validator_checkinfo.create_time IS 'create time'; +DROP TABLE IF EXISTS "linkis_ps_bml_cleaned_resources_version"; +CREATE TABLE linkis_ps_bml_cleaned_resources_version ( + id bigserial NOT NULL, + resource_id varchar(50) NOT NULL, + file_md5 varchar(32) NOT NULL, + version varchar(20) NOT NULL, + size integer NOT NULL, + start_byte bigint unsigned NOT NULL DEFAULT '0', + end_byte bigint unsigned NOT NULL DEFAULT '0', + resource varchar(2000) NOT NULL, + description varchar(2000) DEFAULT NULL, + start_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + end_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + client_ip varchar(200) NOT NULL, + updator varchar(50) DEFAULT NULL, + enable_flag boolean NOT NULL DEFAULT true, + old_resource varchar(2000) NOT NULL, + CONSTRAINT linkis_ps_bml_cleaned_resources_version_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX resource_id_version ON linkis_ps_bml_cleaned_resources_version USING btree (resource_id, version); +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.id IS '主键'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.resource_id IS '资源id，资源的uuid'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.file_md5 IS '文件的md5摘要'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.version IS '资源版本（v 加上 五位数字）'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.size IS '文件大小'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.resource IS '资源内容（文件信息 包括 路径和文件名）'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.description IS '描述'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.start_time IS '开始时间'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.end_time IS '结束时间'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.client_ip IS '客户端ip'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.updator IS '修改者'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.enable_flag IS '状态，1：正常，0：冻结'; +COMMENT ON COLUMN linkis_ps_bml_cleaned_resources_version.old_resource IS '旧的路径'; +DROP TABLE IF EXISTS "linkis_ps_configuration_across_cluster_rule"; +CREATE TABLE linkis_ps_configuration_across_cluster_rule ( + id INT AUTO_INCREMENT, + cluster_name char(32) NOT NULL, + creator char(32) NOT NULL, + username char(32) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_by char(32) NOT NULL, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_by char(32) NOT NULL, + rules varchar(512) NOT NULL, + is_valid VARCHAR(2) DEFAULT 'N', + CONSTRAINT linkis_ps_configuration_across_cluster_rule_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX idx_creator_username ON linkis_ps_configuration_across_cluster_rule USING btree (creator, username); +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.id IS '规则ID，自增主键'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.cluster_name IS '集群名称，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.creator IS '创建者，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.username IS '用户，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.create_time IS '创建时间，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.create_by IS '创建者，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.update_time IS '修改时间，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.update_by IS '更新者，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.rules IS '规则内容，不能为空'; +COMMENT ON COLUMN linkis_ps_configuration_across_cluster_rule.is_valid IS '是否有效 Y/N'; +DROP TABLE IF EXISTS "linkis_org_user_sync"; +CREATE TABLE linkis_org_user_sync ( + cluster_code varchar(16), + user_type varchar(64), + user_name varchar(128), + org_id varchar(16), + org_name varchar(64), + queue_name varchar(64), + db_name varchar(64), + interface_user varchar(64), + is_union_analyse varchar(64), + create_time varchar(64), + user_itsm_no varchar(64), + CONSTRAINT linkis_org_user_sync_pkey PRIMARY KEY (user_name) +); +COMMENT ON COLUMN linkis_org_user_sync.cluster_code IS '集群'; +COMMENT ON COLUMN linkis_org_user_sync.user_type IS '用户类型'; +COMMENT ON COLUMN linkis_org_user_sync.user_name IS '授权用户'; +COMMENT ON COLUMN linkis_org_user_sync.org_id IS '部门ID'; +COMMENT ON COLUMN linkis_org_user_sync.org_name IS '部门名字'; +COMMENT ON COLUMN linkis_org_user_sync.queue_name IS '默认资源队列'; +COMMENT ON COLUMN linkis_org_user_sync.db_name IS '默认操作数据库'; +COMMENT ON COLUMN linkis_org_user_sync.interface_user IS '接口人'; +COMMENT ON COLUMN linkis_org_user_sync.is_union_analyse IS '是否联合分析人'; +COMMENT ON COLUMN linkis_org_user_sync.create_time IS '用户创建时间'; +COMMENT ON COLUMN linkis_org_user_sync.user_itsm_no IS '用户创建单号'; +DROP TABLE IF EXISTS "linkis_cg_tenant_department_config"; +CREATE TABLE linkis_cg_tenant_department_config ( + id serial NOT NULL, + creator varchar(50) NOT NULL, + department varchar(64) NOT NULL, + department_id varchar(16) NOT NULL, + tenant_value varchar(128) NOT NULL, + create_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_by varchar(50) NOT NULL, + is_valid varchar(1) NOT NULL DEFAULT 'Y', + CONSTRAINT linkis_cg_tenant_department_config_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_creator_department ON linkis_cg_tenant_department_config USING btree (creator, department); +COMMENT ON COLUMN linkis_cg_tenant_department_config.id IS 'ID'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.creator IS '应用'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.department IS '部门名称'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.department_id IS '部门ID'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.tenant_value IS '部门租户标签'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.create_time IS '创建时间'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.update_time IS '更新时间'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.create_by IS '创建用户'; +COMMENT ON COLUMN linkis_cg_tenant_department_config.is_valid IS '是否有效'; +DROP TABLE IF EXISTS "linkis_mg_gateway_whitelist_config"; +CREATE TABLE linkis_mg_gateway_whitelist_config ( + id serial NOT NULL, + allowed_user varchar(128) NOT NULL, + client_address varchar(128) NOT NULL, + create_time timestamp DEFAULT NULL, + update_time timestamp DEFAULT NULL, + CONSTRAINT linkis_mg_gateway_whitelist_config_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX address_uniq ON linkis_mg_gateway_whitelist_config USING btree (allowed_user, client_address); +CREATE INDEX linkis_mg_gateway_whitelist_config_allowed_user ON linkis_mg_gateway_whitelist_config USING btree (allowed_user); +DROP TABLE IF EXISTS "linkis_mg_gateway_whitelist_sensitive_user"; +CREATE TABLE linkis_mg_gateway_whitelist_sensitive_user ( + id serial NOT NULL, + sensitive_username varchar(128) NOT NULL, + create_time timestamp DEFAULT NULL, + update_time timestamp DEFAULT NULL, + CONSTRAINT linkis_mg_gateway_whitelist_sensitive_user_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX sensitive_username ON linkis_mg_gateway_whitelist_sensitive_user USING btree (sensitive_username); +DROP TABLE IF EXISTS "linkis_ps_python_module_info"; +CREATE TABLE linkis_ps_python_module_info ( + id bigserial NOT NULL, + "name" varchar(255) NOT NULL, + description text, + path varchar(255) NOT NULL, + engine_type varchar(50) NOT NULL, + create_user varchar(50) NOT NULL, + update_user varchar(50) NOT NULL, + is_load boolean NOT NULL DEFAULT false, + is_expire boolean DEFAULT NULL, + python_module varchar(200) DEFAULT NULL, + create_time timestamp NOT NULL, + update_time timestamp NOT NULL, + CONSTRAINT linkis_ps_python_module_info_pkey PRIMARY KEY (id) +); +COMMENT ON COLUMN linkis_ps_python_module_info.id IS '自增id'; +COMMENT ON COLUMN linkis_ps_python_module_info."name" IS 'python模块名称'; +COMMENT ON COLUMN linkis_ps_python_module_info.description IS 'python模块描述'; +COMMENT ON COLUMN linkis_ps_python_module_info.path IS 'hdfs路径'; +COMMENT ON COLUMN linkis_ps_python_module_info.engine_type IS '引擎类型，python/spark/all'; +COMMENT ON COLUMN linkis_ps_python_module_info.create_user IS '创建用户'; +COMMENT ON COLUMN linkis_ps_python_module_info.update_user IS '修改用户'; +COMMENT ON COLUMN linkis_ps_python_module_info.is_load IS '是否加载，0-未加载，1-已加载'; +COMMENT ON COLUMN linkis_ps_python_module_info.is_expire IS '是否过期，0-未过期，1-已过期）'; +COMMENT ON COLUMN linkis_ps_python_module_info.python_module IS '依赖python模块'; +COMMENT ON COLUMN linkis_ps_python_module_info.create_time IS '创建时间'; +COMMENT ON COLUMN linkis_ps_python_module_info.update_time IS '修改时间'; +DROP TABLE IF EXISTS "linkis_ps_job_history_diagnosis"; +CREATE TABLE linkis_ps_job_history_diagnosis ( + id bigserial NOT NULL, + job_history_id bigint NOT NULL, + diagnosis_content text, + created_time timestamp(3) DEFAULT CURRENT_TIMESTAMP, + updated_time timestamp(3) DEFAULT CURRENT_TIMESTAMP, + only_read varchar(5) DEFAULT NULL, + CONSTRAINT linkis_ps_job_history_diagnosis_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX job_history_id ON linkis_ps_job_history_diagnosis USING btree (job_history_id); +COMMENT ON COLUMN linkis_ps_job_history_diagnosis.id IS 'Primary Key, auto increment'; +COMMENT ON COLUMN linkis_ps_job_history_diagnosis.job_history_id IS 'ID of JobHistory'; +COMMENT ON COLUMN linkis_ps_job_history_diagnosis.diagnosis_content IS 'Diagnosis failed task information'; +COMMENT ON COLUMN linkis_ps_job_history_diagnosis.created_time IS 'Creation time'; +COMMENT ON COLUMN linkis_ps_job_history_diagnosis.updated_time IS 'Update time'; +COMMENT ON COLUMN linkis_ps_job_history_diagnosis.only_read IS '1 just read,can not update'; +DROP TABLE IF EXISTS "linkis_mg_gateway_ecc_userinfo"; +CREATE TABLE linkis_mg_gateway_ecc_userinfo ( +CREATE UNIQUE INDEX apply_itsm_id ON linkis_mg_gateway_ecc_userinfo USING btree (apply_itsm_id, user_id); +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.id IS '主键ID，自增'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.om_tool IS '工具系统'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.user_id IS '申请授权用户'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.op_user_id IS '协助运维账号'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.roles IS '角色列表，多个逗号,分隔'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.auth_system_id IS '授权子系统名称ID，多个逗号,分隔'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.apply_itsm_id IS 'ITSM申请单号，唯一，重复推送时根据这个字段做更新'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.effective_datetime IS '生效时间，允许登录的最早时间'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.expire_datetime IS '失效时间，根据这个时间计算cookie的有效期'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.created_at IS '创建时间，默认当前时间'; +COMMENT ON COLUMN linkis_mg_gateway_ecc_userinfo.updated_at IS '更新时间，默认当前时间，更新时修改'; diff --git a/linkis-dist/package/db/linkis_dml.sql b/linkis-dist/package/db/linkis_dml.sql index 8e9ebdc4d1f..6dd4ab0c860 100644 --- a/linkis-dist/package/db/linkis_dml.sql +++ b/linkis-dist/package/db/linkis_dml.sql @@ -388,15 +388,25 @@ INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = labe -- nebula default configuration insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) -(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM `linkis_ps_configuration_key_engine_relation` relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @NEBULA_ALL); +(select relation.config_key_id AS config_key_id, + CASE + WHEN config.`key` = 'linkis.nebula.host' THEN '127.0.0.1' + WHEN config.`key` = 'linkis.nebula.port' THEN '9669' + WHEN config.`key` = 'linkis.nebula.username' THEN 'nebula' + WHEN config.`key` = 'linkis.nebula.password' THEN 'nebula' + ELSE '' + END AS config_value, + relation.engine_type_label_id AS config_label_id +FROM `linkis_ps_configuration_key_engine_relation` relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @NEBULA_ALL +INNER JOIN linkis_ps_configuration_config_key config ON relation.config_key_id = config.id); insert into `linkis_cg_rm_external_resource_provider`(`id`,`resource_type`,`name`,`labels`,`config`) values -(1,'Yarn','default',NULL,'{"rmWebAddress":"@YARN_RESTFUL_URL","hadoopVersion":"@HADOOP_VERSION","authorEnable":@YARN_AUTH_ENABLE,"user":"@YARN_AUTH_USER","pwd":"@YARN_AUTH_PWD","kerberosEnable":@YARN_KERBEROS_ENABLE,"principalName":"@YARN_PRINCIPAL_NAME","keytabPath":"@YARN_KEYTAB_PATH","krb5Path":"@YARN_KRB5_PATH"}'); +(1,'Yarn','default',NULL,'{"rmWebAddress":"http://127.0.0.1:8088","hadoopVersion":"3.3.4","authorEnable":false,"user":"@YARN_AUTH_USER","pwd":"@YARN_AUTH_PWD","kerberosEnable":false,"principalName":"@YARN_PRINCIPAL_NAME","keytabPath":"@YARN_KEYTAB_PATH","krb5Path":"@YARN_KRB5_PATH"}'); -- errorcode -- 01 linkis server -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01001','您的任务没有路由到后台ECM，请联系管理员','The em of labels',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01001','您的任务没有路由到后台ECM，请联系管理员','The ecm of labels',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01002','任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容','Unexpected end of file from server',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01003','任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容','failed to ask linkis Manager Can be retried SocketTimeoutException',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01004','引擎在启动时被Kill，请联系管理员',' [0-9]+ Killed',0); @@ -430,10 +440,10 @@ INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13001','Java进程内存溢出，建议优化脚本内容','OutOfMemoryError',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13002','任务运行内存超过设置内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','Container killed by YARN for exceeding memory limits',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13003','任务运行内存超过设置内存限制，请在管理台增加executor内存或调优sql后执行','read record exception',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13004','任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','failed because the engine quitted unexpectedly',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13005','任务运行内存超过设置内存限制，导致Spark app应用退出，请在管理台增加driver内存或在提交任务时通过spark.driver.memory调整内存','Spark application has already stopped',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13006','任务运行内存超过设置内存限制，导致Spark context应用退出，请在管理台增加driver内存或在提交任务时通过spark.driver.memory调整内存','Spark application sc has already stopped',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13007','任务运行内存超过设置内存限制，导致Pyspark子进程退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','Pyspark process has stopped',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13004','任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台调整内存后使用','failed because the (hive|python|shell|jdbc|io_file|io_hdfs|fps|pipeline|presto|nebula|flink|appconn|sqoop|datax|openlookeng|trino|elasticsearch|seatunnel|hbase|jobserver) engine quitted unexpectedly',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13005','任务运行内存超过设置内存限制，导致Spark app应用退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60','Spark application has already stopped',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13006','任务运行内存超过设置内存限制，导致Spark context应用退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60','Spark application sc has already stopped',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13007','任务运行内存超过设置内存限制，导致Pyspark子进程退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60','Pyspark process has stopped',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13008','任务产生的序列化结果总大小超过了配置的spark.driver.maxResultSize限制。请检查您的任务，看看是否有可能减小任务产生的结果大小，或则可以考虑压缩或合并结果，以减少传输的数据量','is bigger than spark.driver.maxResultSize',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13009','您的任务因为引擎退出（退出可能是引擎进程OOM或者主动kill引擎）导致失败','ERROR EC exits unexpectedly and actively kills the task',0); INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13010','任务运行内存超过设置内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','Container exited with a non-zero exit code',0); @@ -583,13 +593,13 @@ INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) -- ---------------------------- -- Default Tokens -- ---------------------------- -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('LINKIS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('WS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('DSS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('QUALITIS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('VALIDATOR-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('LINKISCLI-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('DSM-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('LINKIS-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('WS-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('DSS-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('QUALITIS-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('VALIDATOR-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('LINKISCLI-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('DSM-AUTH','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); INSERT INTO `linkis_ps_dm_datasource_type` (`name`, `description`, `option`, `classifier`, `icon`, `layers`, `description_en`, `option_en`, `classifier_en`) VALUES ('kafka', 'kafka', 'kafka', '消息队列', '', 2, 'Kafka', 'Kafka', 'Message Queue'); INSERT INTO `linkis_ps_dm_datasource_type` (`name`, `description`, `option`, `classifier`, `icon`, `layers`, `description_en`, `option_en`, `classifier_en`) VALUES ('hive', 'hive数据库', 'hive', '大数据存储', '', 3, 'Hive Database', 'Hive', 'Big Data storage'); @@ -830,3 +840,128 @@ VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL select @data_source_type_id := id from `linkis_ps_dm_datasource_type` where `name` = 'doris'; UPDATE linkis_ps_dm_datasource_type_key SET `require` = 0 WHERE `key` ="password" and `data_source_type_id` = @data_source_type_id; + +-- ============================================ +-- Additional Configuration (Integrated from upgrade scripts) +-- ============================================ + +-- Spark additional config key +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`, `en_description`, `en_name`, `en_treeName`) VALUES ('spark.external.default.jars', '取值范围：file:///xxx.jar 多个路径时 逗号分隔', 'spark 支持额外的jar包列表', NULL, 'Regex', '^file:\/\/\/[\u4e00-\u9fa5_a-zA-Z0-9-.\/]*\.jar(?:,\s*file:\/\/\/[\u4e00-\u9fa5_a-zA-Z0-9-.\/]*\.jar)*?$', '0', '1', '1', 'spark资源设置', 'spark','Value Range: file:///xxx.jar', 'Spark External Default Jars', 'Spark Resource Settings'); + +-- JDBC additional config keys +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.version', '取值范围：jdbc3,jdbc4', 'jdbc版本', 'jdbc4', 'OFT', '["jdbc3","jdbc4"]', 'jdbc', 0, 0, 1, '数据源配置', 0, 'DataSource Configuration', 'Value range: jdbc3, jdbc4', 'jdbc version', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.connect.max', '范围：1-20，单位：个', 'jdbc引擎最大连接数', '10', 'NumInterval', '[1,20]', 'jdbc', 0, 0, 1, '数据源配置', 3, 'DataSource Configuration', 'Range: 1-20, unit: piece', 'Maximum connections of jdbc engine', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.username', 'username', '数据库连接用户名', '', 'None', '', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'username', 'Database connection user name', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.password', 'password', '数据库连接密码', '', 'None', '', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'password', 'Database connection password', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.driver', '例如:com.mysql.jdbc.Driver', 'jdbc连接驱动', 'com.mysql.jdbc.Driver', 'None', NULL, 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'For Example: com.mysql.jdbc.Driver', 'JDBC Connection Driver', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'jdbc引擎初始化内存大小', '1g', 'Regex', '^([1-9]|10)(G|g)$', 'jdbc', 0, 0, 1, '用户配置', 0, 'Value range: 1-10, Unit: G', 'JDBC Engine Initialization Memory Size', 'User Configuration', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.time', '单位：分钟', 'jdbc任务超时告警时间', '', 'Regex', '^[1-9]\\d*$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Unit: Minutes', 'JDBC Task Timeout Alarm Time', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.user', '多人用英文逗号分隔', 'jdbc任务超时告警人', '', 'Regex', '^[a-zA-Z0-9,_-]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Multiple People Separated By Commas In English', 'JDBC Task Timeout Alert Person', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.level', '超时告警级别:1 critical,2 major,3 minor,4 warning,5 info', 'jdbc任务超时告警级别', '3', 'NumInterval', '[1,5]', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Timeout Alarm Levels: 1 Critical, 2 Major, 3 Minor, 4 Warning, 5 Info', 'JDBC Task Timeout Alarm Level', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.datasource.type', '多个数据源用英文逗号分隔', '超时告警支持数据源类型', 'starrocks', 'Regex', '^[a-zA-Z0-9,]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Separate multiple data sources with commas in English', 'Timeout alarm supports data source types', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkisJDBCPoolAbandonedTimeout', '范围：1-21600，单位：秒', '数据源链接超时自动关闭时间', '300', 'NumInterval', '[1,21600]', 'jdbc', 0, 0, 1, '数据源配置', 0, 'Data Source Configuration', 'Range: 1-21600, Unit: seconds', 'Data Source Auto Close Link Time', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间', '15m', 'OFT', '["1h","2h","30m","15m","3m"]', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'Value range: 3m, 15m, 30m, 1h, 2h', 'Engine unlock exit time', 0); + +-- Shell config key +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,`level`,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES ('wds.linkis.engine.running.job.max', '引擎运行最大任务数', '引擎运行最大任务数', '60', 'None', NULL, 'shell', 0, 0, 1, 'shell引擎设置', 0, 'shell Engine Settings', 'Maximum Number Of Tasks The Engine Can Run', 'Maximum Number For Engine', 0); + +-- Spark-3.4.4 labels +INSERT IGNORE INTO `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType','*-*,spark-3.4.4', 'OPTIONAL', 2, now(), now()); +INSERT IGNORE INTO `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType','*-IDE,spark-3.4.4', 'OPTIONAL', 2, now(), now()); +INSERT IGNORE INTO `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType','*-Visualis,spark-3.4.4', 'OPTIONAL', 2, now(), now()); +INSERT IGNORE INTO `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType','*-nodeexecution,spark-3.4.4', 'OPTIONAL', 2, now(), now()); + +-- Nebula IDE label +INSERT IGNORE INTO `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType','*-IDE,nebula-3.0.0', 'OPTIONAL', 2, now(), now()); + +-- Spark additional key-engine relation +INSERT IGNORE INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config + INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'spark' AND config.`key`='spark.external.default.jars' AND label.label_value = '*-*,spark-2.4.3'); + +-- JDBC key-engine relations +INSERT IGNORE INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config + INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' AND label.label_value = '*-*,jdbc-4'); + +-- Shell key-engine relation +INSERT IGNORE INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config + INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'shell' AND config.`key` = 'wds.linkis.engine.running.job.max' AND label.label_value = '*-*,shell-1'); + +-- Spark-3.4.4 key-engine relations +INSERT IGNORE INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config + INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'spark' AND label.label_value = '*-*,spark-3.4.4'); + +-- Nebula key-engine relations +INSERT IGNORE INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config + INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' AND label.label_value = '*-*,nebula-3.0.0'); + +-- Spark-3.4.4 configuration categories +INSERT IGNORE INTO linkis_ps_configuration_category (`label_id`, `level`) VALUES ((SELECT id FROM linkis_cg_manager_label WHERE `label_value` = '*-IDE,spark-3.4.4'), 2); +INSERT IGNORE INTO linkis_ps_configuration_category (`label_id`, `level`) VALUES ((SELECT id FROM linkis_cg_manager_label WHERE `label_value` = '*-Visualis,spark-3.4.4'), 2); +INSERT IGNORE INTO linkis_ps_configuration_category (`label_id`, `level`) VALUES ((SELECT id FROM linkis_cg_manager_label WHERE `label_value` = '*-nodeexecution,spark-3.4.4'), 2); + +-- Nebula IDE configuration category +INSERT IGNORE INTO linkis_ps_configuration_category (`label_id`, `level`) VALUES ((SELECT id FROM linkis_cg_manager_label WHERE `label_value` = '*-IDE,nebula-3.0.0'), 2); + +-- Spark additional config value +INSERT IGNORE INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(SELECT `relation`.`config_key_id` AS `config_key_id`, NULL AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN linkis_ps_configuration_config_key config ON relation.config_key_id=config.id AND config.`key`='spark.external.default.jars' AND config.engine_conn_type='spark' + INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id=label.id AND label.label_value='*-*,spark-2.4.3'); + +-- JDBC config values +INSERT IGNORE INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(SELECT `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = '*-*,jdbc-4'); + +-- Shell config value +INSERT IGNORE INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(SELECT `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id + INNER JOIN linkis_ps_configuration_config_key config ON relation.config_key_id = config.id AND config.`key` = 'wds.linkis.engine.running.job.max' + AND label.label_value = '*-*,shell-1'); + +-- Spark-3.4.4 config values +INSERT IGNORE INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(SELECT `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = '*-*,spark-3.4.4'); + +-- Nebula config values +INSERT IGNORE INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(SELECT `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = '*-*,nebula-3.0.0'); + +-- Additional error codes +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13010','任务实际运行内存超过了设置的内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory增加内存','Container exited with a non-zero exit code',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13011','广播表过大导致driver内存溢出，请在执行sql前增加参数后重试：set spark.sql.autoBroadcastJoinThreshold=-1;','dataFrame to local exception',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13012','driver内存不足，请增加driver内存后重试','Failed to allocate a page (\\S+.*\\)), try again.',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13013','使用spark默认变量sc导致后续代码执行失败','sc.setJobGroup(\\S+.*\\))',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13014','任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','failed because the spark engine quitted unexpectedly',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21004','Hive Metastore存在问题，生产请联系生产服务助手进行处理，测试请联系Hive开发','Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Cannot resolve function `(\\S+)',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43016','模块 %s 没有属性 %s ，请确认代码引用是否正常','AttributeError: \'(\\S+)\' object has no attribute \'(\\S+)\'',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43019','执行表在元数据库中存在meta缓存，meta信息与缓存不一致导致，请增加参数(--conf spark.sql.hive.convertMetastoreOrc=false)后重试','Unable to alter table.*Table is not allowed to be altered',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43042','插入数据表动态分区数超过配置值 %s ，请优化sql或调整配置hive.exec.max.dynamic.partitions后重试','Maximum was set to (\\S+) partitions per node',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43043','执行任务消耗内存超过限制，hive任务请修改map或reduce的内存，spark任务请修改executor端内存','Error：java heap space',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43044','表 %s 分区数超过阈值 %s，需要分批删除分区，再删除表','the partitions of table (\\S+) exceeds threshold (\\S+)',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43045','查询/操作的表 %s 分区数为 %s ，超过阈值 %s ，需要限制查询/操作的分区数量','Number of partitions scanned \\(=(\\d+)\\) on table (\\S+) exceeds limit \\(=(\\d+)\\)',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43046','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Number of dynamic partitions created is (\\S+), which is more than (\\S+)',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43047','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Maximum was set to (\\S+) partitions per node, number of dynamic partitions on this node: (\\S+)',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43048','参数引用错误，请检查参数 %s 是否正常引用','UnboundLocalError.*local variable (\\S+) referenced before assignment',0); +INSERT IGNORE INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43050','特殊UDF不支持在非sql脚本中使用','Not support spacial udf in non-SQL script',0); + +-- Additional Token +INSERT IGNORE INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('DOCTOR-AUTH-LEstzFKwKkrALsDOuGg', '*', '*', 'BDP', DATE_FORMAT(NOW(), '%Y-%m-%d'), DATE_FORMAT(NOW(), '%Y-%m-%d'), -1, 'LINKIS'); + +-- Additional datasource type keys +INSERT IGNORE INTO linkis_ps_dm_datasource_type_key (data_source_type_id, `key`, name, default_value, value_type, `scope`, `require`, description, value_regex, ref_id, ref_value, data_source, update_time, create_time, name_en, description_en) VALUES(5, 'userClientIp', 'userClientIp', NULL, 'TEXT', 'ENV', 0, 'userClientIp', NULL, NULL, NULL, NULL, now(),now(), 'user client ip', 'user client ip'); +INSERT IGNORE INTO `linkis_ps_dm_datasource_type_key` (`data_source_type_id`, `key`, `name`, `default_value`, `value_type`, `scope`, `require`, `description`, `value_regex`, `ref_id`, `ref_value`, `data_source`, `update_time`, `create_time`, `name_en`, `description_en`) VALUES ((SELECT id FROM `linkis_ps_dm_datasource_type` WHERE `name` = 'starrocks'), 'kill_task_time', '超时kill任务时间', NULL, 'TEXT', NULL, 0, '配置任务超时时间，满足配置执行kill，单位：分钟', '^[1-9]\\d*$', NULL, NULL, NULL, now(), now(), 'Timeout Kill Task Time', 'Configure Task Timeout To Meet The Requirement Of Executing The Kill Action'); diff --git a/linkis-dist/package/db/linkis_dml_pg.sql b/linkis-dist/package/db/linkis_dml_pg.sql index 18cbcb20b97..3ba865151d7 100644 --- a/linkis-dist/package/db/linkis_dml_pg.sql +++ b/linkis-dist/package/db/linkis_dml_pg.sql @@ -1,711 +1,1120 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - - --- 变量： -SET VAR.SPARK_LABEL="spark-3.2.1"; -SET VAR.HIVE_LABEL="hive-3.1.3"; -SET VAR.PYTHON_LABEL="python-python2"; -SET VAR.PIPELINE_LABEL="pipeline-1"; -SET VAR.JDBC_LABEL="jdbc-4"; -SET VAR.PRESTO_LABEL="presto-0.234"; -SET VAR.TRINO_LABEL="trino-371"; -SET VAR.IO_FILE_LABEL="io_file-1.0"; -SET VAR.OPENLOOKENG_LABEL="openlookeng-1.5.0"; -SET VAR.ELASTICSEARCH_LABEL="elasticsearch-7.6.2"; - -delete from linkis_ps_configuration_config_key; -alter sequence linkis_ps_configuration_config_key_id_seq restart with 1; --- Global Settings -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.yarnqueue', 'yarn队列名', 'yarn队列名', 'default', 'None', NULL, '0', '0', '1', '队列资源'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.yarnqueue.instance.max', '取值范围：1-128，单位：个', '队列实例最大个数', '128', 'Regex', '^(?:[1-9]\d?|[1234]\d{2}|128)$', '0', '0', '1', '队列资源'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.yarnqueue.cores.max', '取值范围：1-500，单位：个', '队列CPU使用上限', '500', 'Regex', '^(?:[1-9]\d?|[1234]\d{2}|500)$', '0', '0', '1', '队列资源'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.yarnqueue.memory.max', '取值范围：1-1000，单位：G', '队列内存使用上限', '1000G', 'Regex', '^([1-9]\d{0,2}|1000)(G|g)$', '0', '0', '1', '队列资源'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.client.memory.max', '取值范围：1-100，单位：G', '全局各个引擎内存使用上限', '100G', 'Regex', '^([1-9]\d{0,1}|100)(G|g)$', '0', '0', '1', '队列资源'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.client.core.max', '取值范围：1-128，单位：个', '全局各个引擎核心个数上限', '128', 'Regex', '^(?:[1-9]\d?|[1][0-2][0-8])$', '0', '0', '1', '队列资源'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', '全局各个引擎最大并发数', '20', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源'); --- spark -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'spark引擎最大并发数', '20', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.executor.instances', '取值范围：1-40，单位：个', 'spark执行器实例最大并发数', '1', 'NumInterval', '[1,40]', '0', '0', '2', 'spark资源设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.executor.cores', '取值范围：1-8，单位：个', 'spark执行器核心个数', '1', 'NumInterval', '[1,8]', '0', '0', '1','spark资源设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.executor.memory', '取值范围：1-15，单位：G', 'spark执行器内存大小', '1g', 'Regex', '^([1-9]|1[0-5])(G|g)$', '0', '0', '3', 'spark资源设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.driver.cores', '取值范围：只能取1，单位：个', 'spark驱动器核心个数', '1', 'NumInterval', '[1,1]', '0', '1', '1', 'spark资源设置','spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.driver.memory', '取值范围：1-15，单位：G', 'spark驱动器内存大小','1g', 'Regex', '^([1-9]|1[0-5])(G|g)$', '0', '0', '1', 'spark资源设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间','1h', 'OFT', '["1h","2h","30m","15m","3m"]', '0', '0', '1', 'spark引擎设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.tispark.pd.addresses', NULL, NULL, 'pd0:2379', 'None', NULL, '0', '0', '1', 'tidb设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.tispark.tidb.addr', NULL, NULL, 'tidb', 'None', NULL, '0', '0', '1', 'tidb设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.tispark.tidb.password', NULL, NULL, NULL, 'None', NULL, '0', '0', '1', 'tidb设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.tispark.tidb.port', NULL, NULL, '4000', 'None', NULL, '0', '0', '1', 'tidb设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.tispark.tidb.user', NULL, NULL, 'root', 'None', NULL, '0', '0', '1', 'tidb设置', 'spark'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('spark.python.version', '取值范围：python2,python3', 'python版本','python3', 'OFT', '["python3","python2"]', '0', '0', '1', 'spark引擎设置', 'spark'); --- hive -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'hive引擎最大并发数', '20', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源', 'hive'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'hive引擎初始化内存大小','1g', 'Regex', '^([1-9]|10)(G|g)$', '0', '0', '1', 'hive引擎设置', 'hive'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('hive.client.java.opts', 'hive客户端进程参数', 'hive引擎启动时jvm参数','', 'None', NULL, '1', '1', '1', 'hive引擎设置', 'hive'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('mapred.reduce.tasks', '范围：-1-10000，单位：个', 'reduce数', '-1', 'NumInterval', '[-1,10000]', '0', '1', '1', 'hive资源设置', 'hive'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间','1h', 'OFT', '["1h","2h","30m","15m","3m"]', '0', '0', '1', 'hive引擎设置', 'hive'); - --- python -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.client.memory.max', '取值范围：1-100，单位：G', 'python驱动器内存使用上限', '20G', 'Regex', '^([1-9]\d{0,1}|100)(G|g)$', '0', '0', '1', '队列资源', 'python'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.client.core.max', '取值范围：1-128，单位：个', 'python驱动器核心个数上限', '10', 'Regex', '^(?:[1-9]\d?|[1234]\d{2}|128)$', '0', '0', '1', '队列资源', 'python'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'python引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源', 'python'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.engineconn.java.driver.memory', '取值范围：1-2，单位：G', 'python引擎初始化内存大小', '1g', 'Regex', '^([1-2])(G|g)$', '0', '0', '1', 'python引擎设置', 'python'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('python.version', '取值范围：python2,python3', 'python版本','python2', 'OFT', '["python3","python2"]', '0', '0', '1', 'python引擎设置', 'python'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间','1h', 'OFT', '["1h","2h","30m","15m","3m"]', '0', '0', '1', 'python引擎设置', 'python'); - --- pipeline -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('pipeline.output.mold', '取值范围：csv或excel', '结果集导出类型','csv', 'OFT', '["csv","excel"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('pipeline.field.split', '取值范围：，或\t', 'csv分隔符',',', 'OFT', '[",","\\t"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('pipeline.output.charset', '取值范围：utf-8或gbk', '结果集导出字符集','gbk', 'OFT', '["utf-8","gbk"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('pipeline.output.isoverwrite', '取值范围：true或false', '是否覆写','true', 'OFT', '["true","false"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.instance', '范围：1-3，单位：个', 'pipeline引擎最大并发数','3', 'NumInterval', '[1,3]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'pipeline引擎初始化内存大小','2g', 'Regex', '^([1-9]|10)(G|g)$', '0', '0', '1', 'pipeline资源设置', 'pipeline'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('pipeline.output.shuffle.null.type', '取值范围：NULL或者BLANK', '空值替换','NULL', 'OFT', '["NULL","BLANK"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); --- jdbc -insert into "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.jdbc.connect.url', '例如:jdbc:mysql://127.0.0.1:10000', 'jdbc连接地址', 'jdbc:mysql://127.0.0.1:10000', 'Regex', '^\s*jdbc:\w+://([^:]+)(:\d+)(/[^\?]+)?(\?\S*)?$', '0', '0', '1', '数据源配置', 'jdbc'); -insert into "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.jdbc.driver', '例如:com.mysql.jdbc.Driver', 'jdbc连接驱动', '', 'None', '', '0', '0', '1', '用户配置', 'jdbc'); -insert into "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.jdbc.version', '取值范围：jdbc3,jdbc4', 'jdbc版本','jdbc4', 'OFT', '["jdbc3","jdbc4"]', '0', '0', '1', '用户配置', 'jdbc'); -insert into "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.jdbc.username', 'username', '数据库连接用户名', '', 'None', '', '0', '0', '1', '用户配置', 'jdbc'); -insert into "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.jdbc.password', 'password', '数据库连接密码', '', 'None', '', '0', '0', '1', '用户配置', 'jdbc'); -insert into "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.jdbc.connect.max', '范围：1-20，单位：个', 'jdbc引擎最大连接数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '数据源配置', 'jdbc'); - --- io_file -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'io_file引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', 'io_file引擎资源上限', 'io_file'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.client.memory.max', '取值范围：1-50，单位：G', 'io_file引擎最大内存', '20G', 'Regex', '^([1-9]\d{0,1}|100)(G|g)$', '0', '0', '1', 'io_file引擎资源上限', 'io_file'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "is_hidden", "is_advanced", "level", "treeName", "engine_conn_type") VALUES ('wds.linkis.rm.client.core.max', '取值范围：1-100，单位：个', 'io_file引擎最大核心数', '40', 'Regex', '^(?:[1-9]\d?|[1234]\d{2}|128)$', '0', '0', '1', 'io_file引擎资源上限', 'io_file'); - --- openlookeng -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.openlookeng.url', '例如:http://127.0.0.1:8080', '连接地址', 'http://127.0.0.1:8080', 'Regex', '^\s*http://([^:]+)(:\d+)(/[^\?]+)?(\?\S*)?$', 'openlookeng', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.openlookeng.catalog', 'catalog', 'catalog', 'system', 'None', '', 'openlookeng', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.openlookeng.source', 'source', 'source', 'global', 'None', '', 'openlookeng', '0', '0', 1, '数据源配置'); - --- elasticsearch -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.cluster', '例如:http://127.0.0.1:9200', '连接地址', 'http://127.0.0.1:9200', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.datasource', '连接别名', '连接别名', 'hadoop', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.username', 'username', 'ES集群用户名', '无', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.password', 'password', 'ES集群密码', '无', 'None', '','elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.auth.cache', '客户端是否缓存认证', '客户端是否缓存认证', 'false', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.sniffer.enable', '客户端是否开启 sniffer', '客户端是否开启 sniffer', 'false', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.http.method', '调用方式', 'HTTP请求方式', 'GET', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.http.endpoint', '/_search', 'JSON 脚本调用的 Endpoint', '/_search', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.sql.endpoint', '/_sql', 'SQL 脚本调用的 Endpoint', '/_sql', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.sql.format', 'SQL 脚本调用的模板，%s 替换成 SQL 作为请求体请求Es 集群', '请求体', '{"query":"%s"}', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.es.headers.*', '客户端 Headers 配置', '客户端 Headers 配置', '无', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.engineconn.concurrent.limit', '引擎最大并发', '引擎最大并发', '100', 'None', '', 'elasticsearch', '0', '0', 1, '数据源配置'); - --- presto -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.presto.url', 'Presto 集群连接', 'presto连接地址', 'http://127.0.0.1:8080', 'None', NULL, 'presto', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.presto.catalog', '查询的 Catalog ', 'presto连接的catalog', 'hive', 'None', NULL, 'presto', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.presto.schema', '查询的 Schema ', '数据库连接schema', '', 'None', NULL, 'presto', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('wds.linkis.presto.source', '查询使用的 source ', '数据库连接source', '', 'None', NULL, 'presto', '0', '0', 1, '数据源配置'); - --- trino -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.default.limit', '查询的结果集返回条数限制', '结果集条数限制', '5000', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.http.connectTimeout', '连接Trino服务器的超时时间', '连接超时时间（秒）', '60', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.http.readTimeout', '等待Trino服务器返回数据的超时时间', '传输超时时间（秒）', '60', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.resultSet.cache.max', 'Trino结果集缓冲区大小', '结果集缓冲区', '512k', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.url', 'Trino服务器URL', 'Trino服务器URL', 'http://127.0.0.1:9401', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.user', '用于连接Trino查询服务的用户名', '用户名', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.password', '用于连接Trino查询服务的密码', '密码', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.passwordCmd', '用于连接Trino查询服务的密码回调命令', '密码回调命令', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.catalog', '连接Trino查询时使用的catalog', 'Catalog', 'system', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.schema', '连接Trino查询服务的默认schema', 'Schema', '', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.insecured', '是否忽略服务器的SSL证书', '验证SSL证书', 'false', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.engineconn.concurrent.limit', '引擎最大并发', '引擎最大并发', '100', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.keystore', 'Trino服务器SSL keystore路径', 'keystore路径', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.keystore.type', 'Trino服务器SSL keystore类型', 'keystore类型', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.keystore.password', 'Trino服务器SSL keystore密码', 'keystore密码', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.truststore', 'Trino服务器SSL truststore路径', 'truststore路径', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.truststore.type', 'Trino服务器SSL truststore类型', 'truststore类型', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); -INSERT INTO "linkis_ps_configuration_config_key" ("key", "description", "name", "default_value", "validate_type", "validate_range", "engine_conn_type", "is_hidden", "is_advanced", "level", "treeName") VALUES ('linkis.trino.ssl.truststore.password', 'Trino服务器SSL truststore密码', 'truststore密码', 'null', 'None', '', 'trino', '0', '0', 1, '数据源配置'); - - -delete from linkis_cg_manager_label; -alter sequence linkis_cg_manager_label_id_seq restart with 1; --- Configuration first level directory -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType','*-全局设置,*-*', 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType','*-IDE,*-*', 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType','*-Visualis,*-*', 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType','*-nodeexecution,*-*', 'OPTIONAL', 2, now(), now()); - - --- Engine level default configuration -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType','*-*,*-*', 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-*,' || current_setting('VAR.SPARK_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-*,' || current_setting('VAR.HIVE_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-*,' || current_setting('VAR.PYTHON_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-*,' || current_setting('VAR.PIPELINE_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-*,' || current_setting('VAR.JDBC_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-*,' || current_setting('VAR.OPENLOOKENG_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType', (select '*-*,' || current_setting('VAR.ELASTICSEARCH_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType', (select '*-*,' || current_setting('VAR.PRESTO_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType', (select '*-*,' || current_setting('VAR.TRINO_LABEL')), 'OPTIONAL', 2, now(), now()); - -delete from linkis_ps_configuration_key_engine_relation; -alter sequence linkis_ps_configuration_key_engine_relation_id_seq restart with 1; --- Custom correlation engine (e.g. spark) and configKey value --- Global Settings -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type is null and label.label_value = '*-*,*-*'); - --- spark(Here choose to associate all spark type Key values with spark) -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'spark' and label.label_value = (select '*-*,' || current_setting('VAR.SPARK_LABEL'))); - --- hive -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'hive' and label_value = (select '*-*,' || current_setting('VAR.HIVE_LABEL'))); - --- python-python2 -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'python' and label_value = (select '*-*,' || current_setting('VAR.PYTHON_LABEL'))); - --- pipeline-* -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'pipeline' and label_value = (select '*-*,' || current_setting('VAR.PIPELINE_LABEL'))); - --- jdbc-4 -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and label_value = (select '*-*,' || current_setting('VAR.JDBC_LABEL'))); - --- io_file-1.0 -INSERT INTO "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(SELECT config.id AS "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'io_file' and label_value = (select '*-*,' || current_setting('VAR.IO_FILE_LABEL'))); - --- openlookeng-* -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'openlookeng' and label_value = (select '*-*,' || current_setting('VAR.OPENLOOKENG_LABEL'))); - --- elasticsearch-7.6.2 -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'elasticsearch' and label_value = (select '*-*,' || current_setting('VAR.ELASTICSEARCH_LABEL'))); - --- presto-0.234 -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as "config_key_id", label.id AS "engine_type_label_id" FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'presto' and label_value = (select '*-*,' || current_setting('VAR.PRESTO_LABEL'))); - - --- trino-371 -insert into "linkis_ps_configuration_key_engine_relation" ("config_key_id", "engine_type_label_id") -(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config -INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'trino' and label_value = (select '*-*,' || current_setting('VAR.TRINO_LABEL'))); - --- If you need to customize the parameters of the new engine, the following configuration does not need to write SQL initialization --- Just write the SQL above, and then add applications and engines to the management console to automatically initialize the configuration - - --- Configuration secondary directory (creator level default configuration) --- IDE -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-IDE,' || current_setting('VAR.SPARK_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-IDE,' || current_setting('VAR.HIVE_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-IDE,' || current_setting('VAR.PYTHON_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-IDE,' || current_setting('VAR.PIPELINE_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-IDE,' || current_setting('VAR.JDBC_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-IDE,' || current_setting('VAR.OPENLOOKENG_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType', (select '*-IDE,' || current_setting('VAR.ELASTICSEARCH_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType', (select '*-IDE,' || current_setting('VAR.PRESTO_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType', (select '*-IDE,' || current_setting('VAR.TRINO_LABEL')), 'OPTIONAL', 2, now(), now()); - --- Visualis -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-Visualis,' || current_setting('VAR.SPARK_LABEL')), 'OPTIONAL', 2, now(), now()); --- nodeexecution -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-nodeexecution,' || current_setting('VAR.SPARK_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-nodeexecution,' || current_setting('VAR.HIVE_LABEL')), 'OPTIONAL', 2, now(), now()); -insert into "linkis_cg_manager_label" ("label_key", "label_value", "label_feature", "label_value_size", "update_time", "create_time") VALUES ('combined_userCreator_engineType',(select '*-nodeexecution,' || current_setting('VAR.PYTHON_LABEL')), 'OPTIONAL', 2, now(), now()); - - -delete from linkis_ps_configuration_category; -alter sequence linkis_ps_configuration_category_id_seq restart with 1; --- Associate first-level and second-level directories -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = '*-全局设置,*-*'), 1); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = '*-IDE,*-*'), 1); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = '*-Visualis,*-*'), 1); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = '*-nodeexecution,*-*'), 1); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-IDE,' || current_setting('VAR.SPARK_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-IDE,' || current_setting('VAR.HIVE_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-IDE,' || current_setting('VAR.PYTHON_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-IDE,' || current_setting('VAR.PIPELINE_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-IDE,' || current_setting('VAR.JDBC_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-IDE,' || current_setting('VAR.OPENLOOKENG_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-Visualis,' || current_setting('VAR.SPARK_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-nodeexecution,' || current_setting('VAR.SPARK_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-nodeexecution,' || current_setting('VAR.HIVE_LABEL'))), 2); -insert into linkis_ps_configuration_category ("label_id", "level") VALUES ((select id from linkis_cg_manager_label where "label_value" = (select '*-nodeexecution,' || current_setting('VAR.PYTHON_LABEL'))), 2); - - -delete from linkis_ps_configuration_config_value; -alter sequence linkis_ps_configuration_config_value_id_seq restart with 1; --- Associate label and default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = '*-*,*-*'); - --- spark default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.SPARK_LABEL'))); - --- hive default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.HIVE_LABEL'))); - --- python default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.PYTHON_LABEL'))); - --- pipeline default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.PIPELINE_LABEL'))); - --- jdbc default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.JDBC_LABEL'))); - --- openlookeng default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.OPENLOOKENG_LABEL'))); - --- elasticsearch default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.ELASTICSEARCH_LABEL'))); - --- presto default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select "relation"."config_key_id" AS "config_key_id", '' AS "config_value", "relation"."engine_type_label_id" AS "config_label_id" FROM linkis_ps_configuration_key_engine_relation relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.PRESTO_LABEL'))); - --- trino default configuration -insert into "linkis_ps_configuration_config_value" ("config_key_id", "config_value", "config_label_id") -(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM "linkis_ps_configuration_key_engine_relation" relation -INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = (select '*-*,' || current_setting('VAR.TRINO_LABEL'))); - - -delete from linkis_cg_rm_external_resource_provider; -alter sequence linkis_cg_rm_external_resource_provider_id_seq restart with 1; -insert into "linkis_cg_rm_external_resource_provider"("resource_type","name","labels","config") values -('Yarn','default',NULL,'{"rmWebAddress":"@YARN_RESTFUL_URL","hadoopVersion":"@HADOOP_VERSION","authorEnable":@YARN_AUTH_ENABLE,"user":"@YARN_AUTH_USER","pwd":"@YARN_AUTH_PWD","kerberosEnable":@YARN_KERBEROS_ENABLE,"principalName":"@YARN_PRINCIPAL_NAME","keytabPath":"@YARN_KEYTAB_PATH","krb5Path":"@YARN_KRB5_PATH"}'); - - -delete from linkis_ps_error_code; -alter sequence linkis_ps_error_code_id_seq restart with 1; --- errorcode --- 01 linkis server -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01001','您的任务没有路由到后台ECM，请联系管理员','The em of labels',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01002','Linkis服务负载过高，请联系管理员扩容','Unexpected end of file from server',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01003','Linkis服务负载过高，请联系管理员扩容','failed to ask linkis Manager Can be retried SocketTimeoutException',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01004','引擎在启动时被Kill，请联系管理员',' [0-9]+ Killed',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01005','请求Yarn获取队列信息重试2次仍失败，请联系管理员','Failed to request external resourceClassCastException',0); - - --- 11 linkis resource 12 user resource 13 user task resouce -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01101','ECM资源不足，请联系管理员扩容','ECM resources are insufficient',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01102','ECM 内存资源不足，请联系管理员扩容','ECM memory resources are insufficient',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01103','ECM CPU资源不足，请联系管理员扩容','ECM CPU resources are insufficient',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01004','ECM 实例资源不足，请联系管理员扩容','ECM Insufficient number of instances',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01005','机器内存不足，请联系管理员扩容','Cannot allocate memory',0); - -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12001','队列CPU资源不足，可以调整Spark执行器个数','Queue CPU resources are insufficient',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12002','队列内存资源不足，可以调整Spark执行器个数','Insufficient queue memory',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12003','队列实例数超过限制','Insufficient number of queue instances',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12004','全局驱动器内存使用上限，可以设置更低的驱动内存','Drive memory resources are insufficient',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12005','超出全局驱动器CPU个数上限，可以清理空闲引擎','Drive core resources are insufficient',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12006','超出引擎最大并发数上限，可以清理空闲引擎','Insufficient number of instances',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12008','获取Yarn队列信息异常,可能是您设置的yarn队列不存在','获取Yarn队列信息异常',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12009','会话创建失败，%s队列不存在，请检查队列设置是否正确','queue (\S+) does not exist in YARN',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12010','集群队列内存资源不足，可以联系组内人员释放资源','Insufficient cluster queue memory',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12011','集群队列CPU资源不足，可以联系组内人员释放资源','Insufficient cluster queue cpu',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12012','集群队列实例数超过限制','Insufficient cluster queue instance',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12013','资源不足导致启动引擎超时，您可以进行任务重试','wait for DefaultEngineConn',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12014','请求引擎超时，可能是因为队列资源不足导致，请重试','wait for engineConn initial timeout',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12015','您设置的执行器内存已经超过了集群的限定值%s，请减少到限定值以下','is above the max threshold (\S+.+\))',0); - - -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13001','Java进程内存溢出，建议优化脚本内容','OutOfMemoryError',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13002','使用资源过大，请调优sql或者加大资源','Container killed by YARN for exceeding memory limits',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13003','使用资源过大，请调优sql或者加大资源','read record exception',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13004','引擎意外退出，可能是使用资源过大导致','failed because the engine quitted unexpectedly',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13005','Spark app应用退出，可能是复杂任务导致','Spark application has already stopped',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13006','Spark context退出，可能是复杂任务导致','Spark application sc has already stopped',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13007','Pyspark子进程意外退出，可能是复杂任务导致','Pyspark process has stopped',0); --- 21 cluster Authority 22 db Authority -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21001','会话创建失败，用户%s不能提交应用到队列：%s，请联系提供队列给您的人员','User (\S+) cannot submit applications to queue (\S+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21002','创建Python解释器失败，请联系管理员','initialize python executor failed',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21003','创建单机Python解释器失败，请联系管理员','PythonSession process cannot be initialized',0); - -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22001','%s无权限访问，请申请开通数据表权限，请联系您的数据管理人员','Permission denied:\s*user=[a-zA-Z0-9_]+,\s*access=[A-Z]+\s*,\s*inode="([a-zA-Z0-9/_\.]+)"',0); --- INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22002','您可能没有相关权限','Permission denied',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22003','所查库表无权限','Authorization failed:No privilege',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22004','用户%s在机器不存在，请确认是否申请了相关权限','user (\S+) does not exist',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22005','用户在机器不存在，请确认是否申请了相关权限','engineConnExec.sh: Permission denied',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22006','用户在机器不存在，请确认是否申请了相关权限','at com.sun.security.auth.UnixPrincipal',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22007','用户在机器不存在，请确认是否申请了相关权限','LoginException: java.lang.NullPointerException: invalid null input: name',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22008','用户在机器不存在，请确认是否申请了相关权限','User not known to the underlying authentication module',0); - --- 30 Space exceeded 31 user operation -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('30001','库超过限制','is exceeded',0); - -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('31001','用户主动kill任务','is killed by user',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('31002','您提交的EngineTypeLabel没有对应的引擎版本','EngineConnPluginNotFoundException',0); - --- 41 not exist 44 sql 43 python 44 shell 45 scala 46 importExport -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41001','数据库%s不存在，请检查引用的数据库是否有误','Database ''([a-zA-Z_0-9]+)'' not found',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41001','数据库%s不存在，请检查引用的数据库是否有误','Database does not exist: ([a-zA-Z_0-9]+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41002','表%s不存在，请检查引用的表是否有误','Table or view not found: (["\.a-zA-Z_0-9]+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41002','表%s不存在，请检查引用的表是否有误','Table not found ''([a-zA-Z_0-9]+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41002','表%s不存在，请检查引用的表是否有误','Table ([a-zA-Z_0-9]+) not found',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41003','字段%s不存在，请检查引用的字段是否有误','cannot resolve ''"(.+)"'' given input columns',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41003','字段%s不存在，请检查引用的字段是否有误',' Invalid table alias or column reference ''(.+)'':',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41003','字段%s不存在，请检查引用的字段是否有误','Column ''(.+)'' cannot be resolved',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','分区字段%s不存在，请检查引用的表%s是否为分区表或分区字段有误','([a-zA-Z_0-9]+) is not a valid partition column in table (["\.a-zA-Z_0-9]+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','分区字段%s不存在，请检查引用的表是否为分区表或分区字段有误','Partition spec \{(\S+)\} contains non-partition columns',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','分区字段%s不存在，请检查引用的表是否为分区表或分区字段有误','table is not partitioned but partition spec exists:\{(.+)\}',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','表对应的路径不存在，请联系您的数据管理人员','Path does not exist: viewfs',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41005','文件%s不存在','Caused by:\s*java.io.FileNotFoundException',0); - --- 42 sql -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42001','括号不匹配，请检查代码中括号是否前后匹配','extraneous input ''\)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42002','非聚合函数%s必须写在group by中，请检查代码的group by语法','expression ''(\S+)'' is neither present in the group by',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42002','非聚合函数%s必须写在group by中，请检查代码的group by语法','grouping expressions sequence is empty,\s?and ''(\S+)'' is not an aggregate function',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42002','非聚合函数%s必须写在group by中，请检查代码的group by语法','Expression not in GROUP BY key ''(\S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Undefined function: ''(\S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Invalid function ''(\S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42004','字段%s存在名字冲突，请检查子查询内是否有同名字段','Reference ''(\S+)'' is ambiguous',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42004','字段%s存在名字冲突，请检查子查询内是否有同名字段','Ambiguous column Reference ''(\S+)'' in subquery',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42005','字段%s必须指定表或者子查询别名，请检查该字段来源','Column ''(\S+)'' Found in more than One Tables/Subqueries',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库%s中已经存在，请删除相应表后重试','Table or view ''(\S+)'' already exists in database ''(\S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库中已经存在，请删除相应表后重试','Table (\S+) already exists',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库中已经存在，请删除相应表后重试','Table already exists',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库中已经存在，请删除相应表后重试','AnalysisException: (S+) already exists',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42007','插入目标表字段数量不匹配,请检查代码！','requires that the data to be inserted have the same number of columns as the target table',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42008','数据类型不匹配，请检查代码！','due to data type mismatch: differing types in',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42009','字段%s引用有误，请检查字段是否存在！','Invalid column reference (S+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42010','字段%s提取数据失败','Can''t extract value from (S+): need',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42011','括号或者关键字不匹配，请检查代码！','mismatched input ''(\S+)'' expecting',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42012','group by 位置2不在select列表中，请检查代码！','GROUP BY position (S+) is not in select list',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42013','字段提取数据失败请检查字段类型','Can''t extract value from (S+): need struct type but got string',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42014','插入数据未指定目标表字段%s，请检查代码！','Cannot insert into target table because column number/types are different ''(S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42015','表别名%s错误，请检查代码！','Invalid table alias ''(\S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42016','UDF函数未指定参数，请检查代码！','UDFArgumentException Argument expected',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42017','聚合函数%s不能写在group by 中，请检查代码！','aggregate functions are not allowed in GROUP BY',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42018','您的代码有语法错误，请您修改代码之后执行','SemanticException Error in parsing',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42019','表不存在，请检查引用的表是否有误','table not found',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42020','函数使用错误，请检查您使用的函数方式','No matching method',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42021','您的sql代码可能有语法错误，请检查sql代码','FAILED: ParseException',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42022','您的sql代码可能有语法错误，请检查sql代码','org.apache.spark.sql.catalyst.parser.ParseException',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42022','您的sql代码可能有语法错误，请检查sql代码','org.apache.hadoop.hive.ql.parse.ParseException',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42023','聚合函数不能嵌套','aggregate function in the argument of another aggregate function',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42024','聚合函数不能嵌套','aggregate function parameters overlap with the aggregation',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42025','union 的左右查询字段不一致','Union can only be performed on tables',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42025','hql报错，union 的左右查询字段不一致','both sides of union should match',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42025','union左表和右表类型不一致','on first table and type',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42026','您的建表sql不能推断出列信息','Unable to infer the schema',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42027','动态分区的严格模式需要指定列，您可用通过设置set hive.exec.dynamic.partition.mode=nostrict','requires at least one static partition',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42028','函数输入参数有误','Invalid number of arguments for function',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42029','sql语法报错，select * 与group by无法一起使用','not allowed in select list when GROUP BY ordinal',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42030','where/having子句之外不支持引用外部查询的表达式','the outer query are not supported outside of WHERE',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42031','sql语法报错，group by 后面不能跟一个表','show up in the GROUP BY list',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42032','hql报错，窗口函数中的字段重复','check for circular dependencies',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42033','sql中出现了相同的字段','Found duplicate column',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42034','sql语法不支持','not supported in current context',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42035','hql语法报错，嵌套子查询语法问题','Unsupported SubQuery Expression',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42036','hql报错，子查询中in 用法有误','in definition of SubQuery',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43037','表字段类型修改导致的转型失败，请联系修改人员','cannot be cast to',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43038','select 的表可能有误','Invalid call to toAttribute on unresolved object',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43039','语法问题，请检查脚本','Distinct window functions are not supported',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43040','查询一定要指定数据源和库信息','Schema must be specified when session schema is not set',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43041','用户UDF函数 %s 加载失败，请检查后再执行','Invalid function (\S+)',0); --- 43 python -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43001','代码中存在NoneType空类型变量，请检查代码','''NoneType'' object',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43002','数组越界','IndexError:List index out of range',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43003','您的代码有语法错误，请您修改代码之后执行','SyntaxError',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43004','python代码变量%s未定义','name ''(S+)'' is not defined',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43005','python udf %s 未定义','Undefined function:s+''(S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43006','python执行不能将%s和%s两种类型进行连接','cannot concatenate ''(S+)'' and ''(S+)''',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43007','pyspark执行失败，可能是语法错误或stage失败','Py4JJavaError: An error occurred',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43009','python代码缩进有误','unexpected indent',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43010','python代码反斜杠后面必须换行','unexpected character after line',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43011','导出Excel表超过最大限制1048575','Invalid row number',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43012','python save as table未指定格式，默认用parquet保存，hive查询报错','parquet.io.ParquetDecodingException',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43013','索引使用错误','IndexError',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43014','sql语法有问题','raise ParseException',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43015','当前节点需要的CS表解析失败，请检查当前CSID对应的CS表是否存在','Cannot parse cs table for node',0); - --- 46 importExport -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46001','找不到导入文件地址：%s','java.io.FileNotFoundException: (\S+) \(No such file or directory\)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46002','导出为excel时临时文件目录权限异常','java.io.IOException: Permission denied(.+)at org.apache.poi.xssf.streaming.SXSSFWorkbook.createAndRegisterSXSSFSheet',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46003','导出文件时无法创建目录：%s','java.io.IOException: Mkdirs failed to create (\S+) (.+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46004','导入模块错误，系统没有%s模块，请联系运维人员安装','ImportError: No module named (S+)',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46005','导出语句错误，请检查路径或命名','Illegal out script',0); - --- 91 wtss -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91001','找不到变量值，请确认您是否设置相关变量','not find variable substitution for',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91002','不存在的代理用户，请检查你是否申请过平台层（bdp或者bdap）用户','failed to change current working directory ownership',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91003','请检查提交用户在WTSS内是否有该代理用户的权限，代理用户中是否存在特殊字符，是否用错了代理用户，OS层面是否有该用户，系统设置里面是否设置了该用户为代理用户','没有权限执行当前任务',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91004','平台层不存在您的执行用户，请在ITSM申请平台层（bdp或者bdap）用户','使用chown命令修改',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91005','未配置代理用户，请在ITSM走WTSS用户变更单，为你的用户授权改代理用户','请联系系统管理员为您的用户添加该代理用户',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91006','您的用户初始化有问题，请联系管理员','java: No such file or directory',0); -INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91007','JobServer中不存在您的脚本文件，请将你的脚本文件放入对应的JobServer路径中', 'Could not open input file for reading%does not exist',0); - -delete from linkis_mg_gateway_auth_token; -alter sequence linkis_mg_gateway_auth_token_id_seq restart with 1; --- ---------------------------- --- Default Tokens --- ---------------------------- -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('LINKIS-UNAVAILABLE-TOKEN','*','*','BDP',now(),now(),-1,'LINKIS'); -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('WS-UNAVAILABLE-TOKEN','*','*','BDP',now(),now(),-1,'LINKIS'); -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('DSS-UNAVAILABLE-TOKEN','*','*','BDP',now(),now(),-1,'LINKIS'); -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('QUALITIS-UNAVAILABLE-TOKEN','*','*','BDP',now(),now(),-1,'LINKIS'); -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('VALIDATOR-UNAVAILABLE-TOKEN','*','*','BDP',now(),now(),-1,'LINKIS'); -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('DSM-UNAVAILABLE-TOKEN','*','*','BDP',now(),now(),-1,'LINKIS'); -delete from linkis_ps_dm_datasource_type; -alter sequence linkis_ps_dm_datasource_type_id_seq restart with 1; -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('kafka', 'kafka', 'kafka', '消息队列', '', 2, 'Kafka', 'Kafka', 'Message Queue'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('hive', 'hive数据库', 'hive', '大数据存储', '', 3, 'Hive Database', 'Hive', 'Big Data storage'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('elasticsearch', 'elasticsearch数据源', 'es无结构化存储', '分布式全文索引', '', 3, 'Elasticsearch Datasource', 'Es No Structured Storage', 'Distributed Full-Text Indexing'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('mongodb', 'mongodb', 'NoSQL文档存储', 'NoSQL', null, 3, 'mongodb', 'NoSQL Document Storage', 'NOSQL'); - --- jdbc -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('mysql', 'mysql数据库', 'mysql数据库', '关系型数据库', '', 3, 'Mysql Database', 'Mysql Database', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('oracle', 'oracle数据库', 'oracle', '关系型数据库', '', 3, 'Oracle Database', 'Oracle Relational Database', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('dm', '达梦数据库', 'dm', '关系型数据库', '', 3, 'Dameng Database', 'Dm', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('kingbase', '人大金仓数据库', 'kingbase', '关系型数据库', '', 3, 'Renmin Jincang Database', 'Kingbase', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('postgresql', 'postgresql数据库', 'postgresql', '关系型数据库', '', 3, 'Postgresql Database', 'Postgresql', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('sqlserver', 'sqlserver数据库', 'sqlserver', '关系型数据库', '', 3, 'Sqlserver Database', 'Sqlserver', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('db2', 'db2数据库', 'db2', '关系型数据库', '', 3, 'Db2 Database', 'Db2', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('greenplum', 'greenplum数据库', 'greenplum', '关系型数据库', '', 3, 'Greenplum Database', 'Greenplum', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('doris', 'doris数据库', 'doris', 'olap', '', 4, 'Doris Database', 'Doris', 'Olap'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('clickhouse', 'clickhouse数据库', 'clickhouse', 'olap', '', 4, 'Clickhouse Database', 'Clickhouse', 'Olap'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('tidb', 'tidb数据库', 'tidb', '关系型数据库', '', 3, 'TiDB Database', 'TiDB', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('starrocks', 'starrocks数据库', 'starrocks', 'olap', '', 4, 'StarRocks Database', 'StarRocks', 'Olap'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('gaussdb', 'gaussdb数据库', 'gaussdb', '关系型数据库', '', 3, 'GaussDB Database', 'GaussDB', 'Relational Database'); -INSERT INTO "linkis_ps_dm_datasource_type" ("name", "description", "option", "classifier", "icon", "layers", "description_en", "option_en", "classifier_en") VALUES ('oceanbase', 'oceanbase数据库', 'oceanbase', 'olap', '', 4, 'oceanbase Database', 'oceanbase', 'Olap'); - - -delete from linkis_ps_dm_datasource_type_key; -alter sequence linkis_ps_dm_datasource_type_key_id_seq restart with 1; -INSERT INTO "linkis_ps_dm_datasource_type_key" ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mongodb'), 'username', '用户名', 'Username', NULL, 'TEXT', NULL, '1', '用户名', 'Username', '^[0-9A-Za-z_-]+$', NULL, '', NULL, now(), now()); -INSERT INTO "linkis_ps_dm_datasource_type_key" ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mongodb'), 'password', '密码', 'Password', NULL, 'PASSWORD', NULL, '1', '密码', 'Password', '', NULL, '', NULL, now(), now()); -INSERT INTO "linkis_ps_dm_datasource_type_key" ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mongodb'), 'database', '默认库', 'Database', NULL, 'TEXT', NULL, '1', '默认库', 'Database', '^[0-9A-Za-z_-]+$', NULL, '', NULL, now(), now()); -INSERT INTO "linkis_ps_dm_datasource_type_key" ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mongodb'), 'host', 'Host', 'Host', NULL, 'TEXT', NULL, '1', 'mongodb Host', 'Host', NULL, NULL, NULL, NULL, now(), now()); -INSERT INTO "linkis_ps_dm_datasource_type_key" ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mongodb'), 'port', '端口', 'Port', NULL, 'TEXT', NULL, '1', '端口', 'Port', NULL, NULL, NULL, NULL, now(), now()); -INSERT INTO "linkis_ps_dm_datasource_type_key" ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mongodb'), 'params', '连接参数', 'Params', NULL, 'TEXT', NULL, '0', '输入JSON格式: {"param":"value"}', 'Input JSON Format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()); - - - -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'hive'), 'envId', '集群环境(Cluster env)', 'Cluster env', NULL, 'SELECT', NULL, '1', '集群环境(Cluster env)', 'Cluster env', NULL, NULL, NULL, (select '/data-source-manager/env-list/all/type/' || (select id from "linkis_ps_dm_datasource_type" where "name" = 'hive')), now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kafka'), 'envId', '集群环境(Cluster env)', 'Cluster env', NULL, 'SELECT', NULL, '1', '集群环境(Cluster env)', 'Cluster env', NULL, NULL, NULL, (select '/data-source-manager/env-list/all/type/' || (select id from "linkis_ps_dm_datasource_type" where "name" = 'kafka')), now(), now()); - - -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'elasticsearch'), 'username', '用户名(Username)' , 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, '', NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'elasticsearch'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, '', NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'elasticsearch'), 'elasticUrls', 'ES连接URL(Elastic Url)', 'Elastic Url', NULL, 'TEXT', NULL, '1', 'ES连接URL(Elastic Url)', 'Elastic Url', '', NULL, '', NULL, now(), now()); - --- https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url-format.html -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '0', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'port', '端口号(Port)','Port', NULL, 'TEXT', NULL, '0', '端口号(Port)','Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, '0', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '0', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '0', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'mysql'), 'databaseName', '数据库名(Database name)', 'Database name', NULL, 'TEXT', NULL, '0', '数据库名(Database name)', 'Database name', NULL, NULL, NULL, NULL, now(), now()); - --- https://docs.oracle.com/en/database/oracle/oracle-database/21/jajdb/oracle/jdbc/OracleDriver.html -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'oracle.jdbc.driver.OracleDriver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'sid', 'SID', 'SID', NULL, 'TEXT', NULL, '0', 'SID', 'SID', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'serviceName', 'service_name', 'service_name', NULL, 'TEXT', NULL, '0', 'service_name', 'service_name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'server', 'server', 'server', NULL, 'TEXT', NULL, '0', 'server', 'server', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oracle'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '0', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'dm.jdbc.driver.DmDriver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'dm'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.kingbase8.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'kingbase'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'instance', NULL, NULL, NULL, NULL, now(), now()); - --- https://jdbc.postgresql.org/documentation/use/ -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'org.postgresql.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'postgresql'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - --- https://learn.microsoft.com/zh-cn/sql/connect/jdbc/building-the-connection-url?redirectedfrom=MSDN&view=sql-server-ver16 -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.microsoft.sqlserver.jdbc.SQLServerDriver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'sqlserver'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - --- https://www.ibm.com/docs/en/db2/11.5?topic=cdsudidsdjs-url-format-data-server-driver-jdbc-sqlj-type-4-connectivity -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.ibm.db2.jcc.DB2Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'db2'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - --- https://greenplum.docs.pivotal.io/6-1/datadirect/datadirect_jdbc.html#topic_ylk_pbx_2bb -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.pivotal.jdbc.GreenplumDriver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'greenplum'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '0', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'doris'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - --- https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-jdbc -INSERT INTO "linkis_ps_dm_datasource_type_key" - ("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'ru.yandex.clickhouse.ClickHouseDriver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'clickhouse'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - - -delete from linkis_ps_dm_datasource_env; -alter sequence linkis_ps_dm_datasource_env_id_seq restart with 1; -INSERT INTO "linkis_ps_dm_datasource_env" ("env_name", "env_desc", "datasource_type_id", "parameter", "create_time", "create_user", "modify_time", "modify_user") VALUES ('测试环境SIT', '测试环境SIT', (select id from "linkis_ps_dm_datasource_type" where "name" = 'hive'), '{"uris":"thrift://localhost:9083", "hadoopConf":{"hive.metastore.execute.setugi":"true"}}', now(), NULL, now(), NULL); -INSERT INTO "linkis_ps_dm_datasource_env" ("env_name", "env_desc", "datasource_type_id", "parameter", "create_time", "create_user", "modify_time", "modify_user") VALUES ('测试环境UAT', '测试环境UAT', (select id from "linkis_ps_dm_datasource_type" where "name" = 'hive'), '{"uris":"thrift://localhost:9083", "hadoopConf":{"hive.metastore.execute.setugi":"true"}}', now(), NULL, now(), NULL); -INSERT INTO "linkis_ps_dm_datasource_env" ("env_name", "env_desc", "datasource_type_id", "parameter", "create_time", "create_user", "modify_time", "modify_user") VALUES ('kafka测试环境SIT', '开源测试环境SIT', (select id from "linkis_ps_dm_datasource_type" where "name" = 'kafka'), '{"uris":"thrift://localhost:9092"}', now(), NULL, now(), NULL); - - -INSERT INTO "linkis_ps_dm_datasource_type_key" -("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '0', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'tidb'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" -("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '0', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'starrocks'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" -("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'org.postgresql.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'gaussdb'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); - -INSERT INTO "linkis_ps_dm_datasource_type_key" -("data_source_type_id", "key", "name", "name_en", "default_value", "value_type", "scope", "require", "description", "description_en", "value_regex", "ref_id", "ref_value", "data_source", "update_time", "create_time") -VALUES ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'address', '地址', 'Address', NULL, 'TEXT', NULL, '0', '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, '1', '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, '1', '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, '1', '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, '0', '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, '1', '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, '1', '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), - ((select id from "linkis_ps_dm_datasource_type" where "name" = 'oceanbase'), 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, '1', '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + + +-- 变量： +SET VAR.SPARK_LABEL="spark-3.2.1"; +SET VAR.HIVE_LABEL="hive-3.1.3"; +SET VAR.PYTHON_LABEL="python-python2"; +SET VAR.PIPELINE_LABEL="pipeline-1"; +SET VAR.JDBC_LABEL="jdbc-4"; +SET VAR.PRESTO_LABEL="presto-0.234"; +SET VAR.TRINO_LABEL="trino-371"; +SET VAR.IO_FILE_LABEL="io_file-1.0"; +SET VAR.OPENLOOKENG_LABEL="openlookeng-1.5.0"; +SET VAR.ELASTICSEARCH_LABEL="elasticsearch-7.6.2"; + +-- SET @ELASTICSEARCH_ALL=CONCAT('*-*,',@ELASTICSEARCH_LABEL); -- MySQL user variable, not supported in PostgreSQL +-- SET @ELASTICSEARCH_IDE=CONCAT('*-IDE,',@ELASTICSEARCH_LABEL); -- MySQL user variable, not supported in PostgreSQL + +-- SET @NEBULA_ALL=CONCAT('*-*,',@NEBULA_LABEL); -- MySQL user variable, not supported in PostgreSQL +-- SET @NEBULA_IDE=CONCAT('*-IDE,',@NEBULA_LABEL); -- MySQL user variable, not supported in PostgreSQL + +-- Global Settings +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.yarnqueue', 'yarn队列名', 'yarn队列名', 'default', 'None', NULL, '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.yarnqueue.instance.max', '取值范围：1-128，单位：个', '队列实例最大个数', '30', 'Regex', '^(?:[1-9]\\d?|[1234]\\d{2}|128)$', '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.yarnqueue.cores.max', '取值范围：1-4000，单位：个', '队列CPU使用上限', '150', 'Regex', '^(?:[1-9]\\d{0,2}|[1-3]\\d{3}|4000)$', '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.yarnqueue.memory.max', '取值范围：1-10000，单位：G', '队列内存使用上限', '300G', 'Regex', '^(?:[1-9]\\d{0,3}|10000)(G|g)$', '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.client.memory.max', '取值范围：1-100，单位：G', '全局各个引擎内存使用上限', '20G', 'Regex', '^([1-9]\\d{0,1}|100)(G|g)$', '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.client.core.max', '取值范围：1-128，单位：个', '全局各个引擎核心个数上限', '10', 'Regex', '^(?:[1-9]\\d?|[1][0-2][0-8])$', '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', '全局各个引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_description, en_name, en_treeName, template_required) VALUES ('linkis.entrance.creator.job.concurrency.limit', 'Creator级别限制，范围：1-10000，单位：个', 'Creator最大并发数', '10000', 'NumInterval', '[1,10000]', '', 0, 1, 1, '队列资源', 3, 'creator maximum task limit', 'creator maximum task limit', 'QueueResources', '1'); +-- spark +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, template_required) VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'spark引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源', 'spark', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, template_required) VALUES ('spark.executor.instances', '取值范围：1-40，单位：个', 'spark执行器实例最大并发数', '1', 'NumInterval', '[1,40]', '0', '0', '2', 'spark资源设置', 'spark', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('spark.executor.cores', '取值范围：1-8，单位：个', 'spark执行器核心个数', '1', 'NumInterval', '[1,8]', '0', '0', '1','spark资源设置', 'spark'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, template_required) VALUES ('spark.executor.memory', '取值范围：1-28，单位：G', 'spark执行器内存大小', '1g', 'Regex', '^([1-9]|1[0-9]|2[0-8])(G|g)$', '0', '0', '3', 'spark资源设置', 'spark', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('spark.driver.cores', '取值范围：只能取1，单位：个', 'spark驱动器核心个数', '1', 'NumInterval', '[1,1]', '0', '1', '1', 'spark资源设置','spark'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, template_required) VALUES ('spark.driver.memory', '取值范围：1-15，单位：G', 'spark驱动器内存大小','1g', 'Regex', '^([1-9]|1[0-5])(G|g)$', '0', '0', '1', 'spark资源设置', 'spark', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h,6h,12h', '引擎空闲退出时间','1h', 'OFT', '[\"1h\",\"2h\",\"6h\",\"12h\",\"30m\",\"15m\",\"3m\"]', '0', '0', '1', 'spark引擎设置', 'spark'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('spark.python.version', '取值范围：python2,python3', 'python版本','python2', 'OFT', '[\"python3\",\"python2\"]', '0', '0', '1', 'spark引擎设置', 'spark'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name) VALUES ('spark.conf', '多个参数使用分号[;]分隔 例如spark.shuffle.compress=true;', 'spark自定义配置参数',null, 'None', NULL, 'spark',0, 1, 1,'spark资源设置', 0, 'Spark Resource Settings','Multiple parameters are separated by semicolons [;] For example, spark.shuffle.compress=true;', 'Spark Custom Configuration Parameters'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name) VALUES ('spark.locality.wait', '范围：0-3，单位：秒', '任务调度本地等待时间', '3s', 'OFT', '[\"0s\",\"1s\",\"2s\",\"3s\"]', 'spark', 0, 1, 1, 'spark资源设置', 0, 'Spark Resource Settings', 'Range: 0-3, Unit: second', 'Task Scheduling Local Waiting Time'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name) VALUES ('spark.memory.fraction', '范围：0.4,0.5,0.6，单位：百分比', '执行内存和存储内存的百分比', '0.6', 'OFT', '[\"0.4\",\"0.5\",\"0.6\"]', 'spark', 0, 1, 1, 'spark资源设置', 0, 'Spark Resource Settings', 'Range: 0.4, 0.5, 0.6, in percentage', 'Percentage Of Execution Memory And Storage Memory'); + +-- hive +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, template_required) VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'hive引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源', 'hive', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, template_required) VALUES ('wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'hive引擎初始化内存大小','1g', 'Regex', '^([1-9]|10)(G|g)$', '0', '0', '1', 'hive引擎设置', 'hive', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('hive.client.java.opts', 'hive客户端进程参数', 'hive引擎启动时jvm参数','', 'None', NULL, '1', '1', '1', 'hive引擎设置', 'hive'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间','1h', 'OFT', '[\"1h\",\"2h\",\"30m\",\"15m\",\"3m\"]', '0', '0', '1', 'hive引擎设置', 'hive'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type,en_description, en_name, en_treeName, template_required) VALUES ("mapreduce.job.running.reduce.limit", '范围：10-999，单位：个', 'hive引擎reduce限制', '999', 'NumInterval', '[10,999]', '0', '1', '1', 'MapReduce设置', 'hive','Value Range: 10-999, Unit: Piece', 'Number Limit Of MapReduce Job Running Reduce', 'MapReduce Settings', '1'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type,en_description, en_name, en_treeName, template_required) VALUES ('mapreduce.job.reduce.slowstart.completedmaps', '取值范围：0-1', 'Map任务数与总Map任务数之间的比例','0.05', 'Regex', '^(0(\\.\\d{1,2})?|1(\\.0{1,2})?)$', '0', '0', '1', 'hive引擎设置', 'hive', 'Value Range: 0-1', 'The Ratio Between The Number Of Map Tasks And The Total Number Of Map Tasks', 'Hive Engine Settings', '1'); +-- python +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.client.memory.max', '取值范围：1-100，单位：G', 'python驱动器内存使用上限', '20G', 'Regex', '^([1-9]\\d{0,1}|100)(G|g)$', '0', '0', '1', '队列资源', 'python'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.client.core.max', '取值范围：1-128，单位：个', 'python驱动器核心个数上限', '10', 'Regex', '^(?:[1-9]\\d?|[1234]\\d{2}|128)$', '0', '0', '1', '队列资源', 'python'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'python引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '队列资源', 'python'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.engineconn.java.driver.memory', '取值范围：1-2，单位：G', 'python引擎初始化内存大小', '1g', 'Regex', '^([1-2])(G|g)$', '0', '0', '1', 'python引擎设置', 'python'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('python.version', '取值范围：python2,python3', 'python版本','python2', 'OFT', '[\"python3\",\"python2\"]', '0', '0', '1', 'python引擎设置', 'python'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间','1h', 'OFT', '[\"1h\",\"2h\",\"30m\",\"15m\",\"3m\"]', '0', '0', '1', 'python引擎设置', 'python'); + +-- pipeline +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('pipeline.output.mold', '取值范围：csv或excel', '结果集导出类型','csv', 'OFT', '[\"csv\",\"excel\"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('pipeline.field.split', '取值范围：,或\\t或;或|', 'csv分隔符',',', 'OFT', '[\",\",\"\\\\t\",\"\\\\;\",\"\\\\|\"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('pipeline.output.charset', '取值范围：utf-8或gbk', '结果集导出字符集','gbk', 'OFT', '[\"utf-8\",\"gbk\"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('pipeline.output.isoverwrite', '取值范围：true或false', '是否覆写','true', 'OFT', '[\"true\",\"false\"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.instance', '范围：1-3，单位：个', 'pipeline引擎最大并发数','3', 'NumInterval', '[1,3]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'pipeline引擎初始化内存大小','2g', 'Regex', '^([1-9]|10)(G|g)$', '0', '0', '1', 'pipeline资源设置', 'pipeline'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('pipeline.output.shuffle.null.type', '取值范围：NULL或者BLANK', '空值替换','NULL', 'OFT', '[\"NULL\",\"BLANK\"]', '0', '0', '1', 'pipeline引擎设置', 'pipeline'); +-- jdbc +insert into linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.jdbc.connect.url', '例如:jdbc:mysql://127.0.0.1:10000', 'jdbc连接地址', 'jdbc:mysql://127.0.0.1:10000', 'Regex', '^\\s*jdbc:\\w+://([^:]+)(:\\d+)(/[^\\?]+)?(\\?\\S*)?$', '0', '0', '1', '数据源配置', 'jdbc'); +insert into linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.jdbc.driver', '例如:com.mysql.jdbc.Driver', 'jdbc连接驱动', '', 'None', '', '0', '0', '1', '用户配置', 'jdbc'); +insert into linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.jdbc.version', '取值范围：jdbc3,jdbc4', 'jdbc版本','jdbc4', 'OFT', '[\"jdbc3\",\"jdbc4\"]', '0', '0', '1', '用户配置', 'jdbc'); +insert into linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.jdbc.username', 'username', '数据库连接用户名', '', 'None', '', '0', '0', '1', '用户配置', 'jdbc'); +insert into linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.jdbc.password', 'password', '数据库连接密码', '', 'None', '', '0', '0', '1', '用户配置', 'jdbc'); +insert into linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.jdbc.connect.max', '范围：1-20，单位：个', 'jdbc引擎最大连接数', '10', 'NumInterval', '[1,20]', '0', '0', '1', '数据源配置', 'jdbc'); + +-- io_file +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.instance', '范围：1-20，单位：个', 'io_file引擎最大并发数', '10', 'NumInterval', '[1,20]', '0', '0', '1', 'io_file引擎资源上限', 'io_file'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.client.memory.max', '取值范围：1-50，单位：G', 'io_file引擎最大内存', '20G', 'Regex', '^([1-9]\\d{0,1}|100)(G|g)$', '0', '0', '1', 'io_file引擎资源上限', 'io_file'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type) VALUES ('wds.linkis.rm.client.core.max', '取值范围：1-100，单位：个', 'io_file引擎最大核心数', '40', 'Regex', '^(?:[1-9]\\d?|[1234]\\d{2}|128)$', '0', '0', '1', 'io_file引擎资源上限', 'io_file'); + +-- openlookeng +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.openlookeng.url', '例如:http://127.0.0.1:8080', '连接地址', 'http://127.0.0.1:8080', 'Regex', '^\\s*http://([^:]+)(:\\d+)(/[^\\?]+)?(\\?\\S*)?$', 'openlookeng', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.openlookeng.catalog', 'catalog', 'catalog', 'system', 'None', '', 'openlookeng', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.openlookeng.source', 'source', 'source', 'global', 'None', '', 'openlookeng', 0, 0, 1, '数据源配置'); + +-- elasticsearch +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.cluster', '例如:http://127.0.0.1:9200', '连接地址', 'http://127.0.0.1:9200', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.datasource', '连接别名', '连接别名', 'hadoop', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.username', 'username', 'ES集群用户名', '无', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.password', 'password', 'ES集群密码', '无', 'None', '','elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.auth.cache', '客户端是否缓存认证', '客户端是否缓存认证', 'false', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.sniffer.enable', '客户端是否开启 sniffer', '客户端是否开启 sniffer', 'false', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.http.method', '调用方式', 'HTTP请求方式', 'GET', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.http.endpoint', '/_search', 'JSON 脚本调用的 Endpoint', '/_search', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.sql.endpoint', '/_sql', 'SQL 脚本调用的 Endpoint', '/_sql', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.sql.format', 'SQL 脚本调用的模板，%s 替换成 SQL 作为请求体请求Es 集群', '请求体', '{"query":"%s"}', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.es.headers.*', '客户端 Headers 配置', '客户端 Headers 配置', '无', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.engineconn.concurrent.limit', '引擎最大并发', '引擎最大并发', '100', 'None', '', 'elasticsearch', 0, 0, 1, '数据源配置'); + +-- presto +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.presto.url', 'Presto 集群连接', 'presto连接地址', 'http://127.0.0.1:8080', 'None', NULL, 'presto', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.presto.catalog', '查询的 Catalog ', 'presto连接的catalog', 'hive', 'None', NULL, 'presto', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.presto.schema', '查询的 Schema ', '数据库连接schema', '', 'None', NULL, 'presto', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('wds.linkis.presto.source', '查询使用的 source ', '数据库连接source', '', 'None', NULL, 'presto', 0, 0, 1, '数据源配置'); + +-- trino +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.default.limit', '查询的结果集返回条数限制', '结果集条数限制', '5000', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.http.connectTimeout', '连接Trino服务器的超时时间', '连接超时时间（秒）', '60', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.http.readTimeout', '等待Trino服务器返回数据的超时时间', '传输超时时间（秒）', '60', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.resultSet.cache.max', 'Trino结果集缓冲区大小', '结果集缓冲区', '512k', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.url', 'Trino服务器URL', 'Trino服务器URL', 'http://127.0.0.1:9401', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.user', '用于连接Trino查询服务的用户名', '用户名', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.password', '用于连接Trino查询服务的密码', '密码', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.passwordCmd', '用于连接Trino查询服务的密码回调命令', '密码回调命令', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.catalog', '连接Trino查询时使用的catalog', 'Catalog', 'system', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.schema', '连接Trino查询服务的默认schema', 'Schema', '', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.insecured', '是否忽略服务器的SSL证书', '验证SSL证书', 'false', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.engineconn.concurrent.limit', '引擎最大并发', '引擎最大并发', '100', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.keystore', 'Trino服务器SSL keystore路径', 'keystore路径', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.keystore.type', 'Trino服务器SSL keystore类型', 'keystore类型', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.keystore.password', 'Trino服务器SSL keystore密码', 'keystore密码', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.truststore', 'Trino服务器SSL truststore路径', 'truststore路径', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.truststore.type', 'Trino服务器SSL truststore类型', 'truststore类型', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName) VALUES ('linkis.trino.ssl.truststore.password', 'Trino服务器SSL truststore密码', 'truststore密码', 'null', 'None', '', 'trino', 0, 0, 1, '数据源配置'); + +-- nebula +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.host','Nebula 连接地址','Nebula 连接地址',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Host','Nebula Host',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.port','Nebula 连接端口','Nebula 连接端口',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Port','Nebula Port',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.username','Nebula 连接用户名','Nebula 连接用户名',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Username','Nebula Username',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.password','Nebula 连接密码','Nebula 连接密码',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Password','Nebula Password',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.space', 'Nebula 图空间', 'Nebula 图空间', NULL, 'None', NULL, 'nebula', 0, 0, 1, 'Necula引擎设置', 0, 'Nebula Engine Settings', 'Nebula Space', 'Nebula Space', 0); + +-- Configuration first level directory +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType','*-全局设置,*-*', 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType','*-IDE,*-*', 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType','*-Visualis,*-*', 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType','*-nodeexecution,*-*', 'OPTIONAL', 2, now(), now()); + + +-- Engine level default configuration +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType','*-*,*-*', 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@SPARK_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@HIVE_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@PYTHON_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@PIPELINE_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@JDBC_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@OPENLOOKENG_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @ELASTICSEARCH_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @PRESTO_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @TRINO_ALL, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @NEBULA_IDE,'OPTIONAL',2,now(),now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @NEBULA_ALL,'OPTIONAL',2,now(),now()); +-- Custom correlation engine (e.g. spark) and configKey value +-- Global Settings +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = '' and label.label_value = "*-*,*-*"); + +-- spark(Here choose to associate all spark type Key values with spark) +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'spark' and label.label_value = @SPARK_ALL); + +-- hive +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'hive' and label_value = @HIVE_ALL); + +-- python-python2 +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'python' and label_value = @PYTHON_ALL); + +-- pipeline-* +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'pipeline' and label_value = @PIPELINE_ALL); + +-- jdbc-4 +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and label_value = @JDBC_ALL); + +-- io_file-1.0 +INSERT INTO linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(SELECT config.id AS config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'io_file' and label_value = @IO_FILE_ALL); + +-- openlookeng-* +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'openlookeng' and label_value = @OPENLOOKENG_ALL); + +-- elasticsearch-7.6.2 +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'elasticsearch' and label_value = @ELASTICSEARCH_ALL); + +-- presto-0.234 +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'presto' and label_value = @PRESTO_ALL); + + +-- trino-371 +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'trino' and label_value = @TRINO_ALL); + +-- nebula-3.0.0 +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and label_value = @NEBULA_ALL); + +-- If you need to customize the parameters of the new engine, the following configuration does not need to write SQL initialization +-- Just write the SQL above, and then add applications and engines to the management console to automatically initialize the configuration + + +-- Configuration secondary directory (creator level default configuration) +-- IDE +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@SPARK_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@HIVE_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@PYTHON_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@PIPELINE_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@JDBC_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@OPENLOOKENG_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @ELASTICSEARCH_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @PRESTO_IDE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType', @TRINO_IDE, 'OPTIONAL', 2, now(), now()); + +-- Visualis +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@SPARK_VISUALIS, 'OPTIONAL', 2, now(), now()); +-- nodeexecution +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@SPARK_NODE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@HIVE_NODE, 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',@PYTHON_NODE, 'OPTIONAL', 2, now(), now()); + + +-- Associate first-level and second-level directories +select @label_id := id from linkis_cg_manager_label where label_value = '*-全局设置,*-*'; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 1); + +select @label_id := id from linkis_cg_manager_label where label_value = '*-IDE,*-*'; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 1); + +select @label_id := id from linkis_cg_manager_label where label_value = '*-Visualis,*-*'; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 1); + +select @label_id := id from linkis_cg_manager_label where label_value = '*-nodeexecution,*-*'; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 1); + +select @label_id := id from linkis_cg_manager_label where label_value = @SPARK_IDE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @HIVE_IDE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @PYTHON_IDE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @PIPELINE_IDE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @JDBC_IDE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @OPENLOOKENG_IDE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @SPARK_VISUALIS; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @SPARK_NODE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @HIVE_NODE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +select @label_id := id from linkis_cg_manager_label where label_value = @PYTHON_NODE; +insert into linkis_ps_configuration_category (label_id, level) VALUES (@label_id, 2); + +-- Associate label and default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = '*-*,*-*'); + +-- spark default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @SPARK_ALL); + +-- hive default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @HIVE_ALL); + +-- python default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @PYTHON_ALL); + +-- pipeline default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @PIPELINE_ALL); + +-- jdbc default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @JDBC_ALL); + +-- openlookeng default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @OPENLOOKENG_ALL); + +-- elasticsearch default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @ELASTICSEARCH_ALL); + +-- presto default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @PRESTO_ALL); + +-- trino default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @TRINO_ALL); + + +-- nebula default configuration +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = @NEBULA_ALL); + +insert into linkis_cg_rm_external_resource_provider(id,resource_type,name,labels,config) values +(1,'Yarn','default',NULL,'{"rmWebAddress":"http://127.0.0.1:8088","hadoopVersion":"3.3.4","authorEnable":false,"user":"@YARN_AUTH_USER","pwd":"@YARN_AUTH_PWD","kerberosEnable":false,"principalName":"@YARN_PRINCIPAL_NAME","keytabPath":"@YARN_KEYTAB_PATH","krb5Path":"@YARN_KRB5_PATH"}'); + +-- errorcode +-- 01 linkis server +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01001','您的任务没有路由到后台ECM，请联系管理员','The em of labels',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01002','任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容','Unexpected end of file from server',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01003','任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容','failed to ask linkis Manager Can be retried SocketTimeoutException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01004','引擎在启动时被Kill，请联系管理员',' [0-9]+ Killed',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01005','请求Yarn获取队列信息重试2次仍失败，请联系管理员','Failed to request external resourceClassCastException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01006','没有健康可用的ecm节点，可能任务量大,导致节点资源处于不健康状态，尝试kill空闲引擎释放资源','There are corresponding ECM tenant labels',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01007','文件编码格式异常,请联系管理人员处理','UnicodeEncodeError.*characters',0); + +-- 11 linkis resource 12 user resource 13 user task resouce +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01101','ECM资源不足，请联系管理员扩容','ECM resources are insufficient',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01102','ECM 内存资源不足，可以设置更低的驱动内存','ECM memory resources are insufficient',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01103','ECM CPU资源不足，请联系管理员扩容','ECM CPU resources are insufficient',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01104','ECM 实例资源不足，请联系管理员扩容','ECM Insufficient number of instances',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01105','机器内存不足，请联系管理员扩容','Cannot allocate memory',0); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12001','队列CPU资源不足，可以调整Spark执行器个数','Queue CPU resources are insufficient',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12002','队列内存资源不足，可以调整Spark执行器个数','Insufficient queue memory',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12003','队列实例数超过限制','Insufficient number of queue instances',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12004','全局驱动器内存使用上限，可以设置更低的驱动内存','Drive memory resources are insufficient',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12005','超出全局驱动器CPU个数上限，可以清理空闲引擎','Drive core resources are insufficient',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12006','超出引擎最大并发数上限，可以清理空闲引擎','Insufficient number of instances',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12008','获取Yarn队列信息异常,可能是您设置的yarn队列不存在','获取Yarn队列信息异常',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12009','会话创建失败，%s队列不存在，请检查队列设置是否正确','queue (\\S+) does not exist in YARN',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12010','集群队列内存资源不足，可以联系组内人员释放资源','Insufficient cluster queue memory',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12011','集群队列CPU资源不足，可以联系组内人员释放资源','Insufficient cluster queue cpu',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12012','集群队列实例数超过限制','Insufficient cluster queue instance',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12013','资源不足导致启动引擎超时，您可以进行任务重试','wait for DefaultEngineConn',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12014','请求引擎超时，可能是因为队列资源不足导致，请重试','wait for engineConn initial timeout',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('12015','您设置的执行器内存已经超过了集群的限定值%s，请减少到限定值以下','is above the max threshold (\\S+.+\\))',0); + + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13001','Java进程内存溢出，建议优化脚本内容','OutOfMemoryError',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13002','任务运行内存超过设置内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','Container killed by YARN for exceeding memory limits',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13003','任务运行内存超过设置内存限制，请在管理台增加executor内存或调优sql后执行','read record exception',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13004','任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','failed because the engine quitted unexpectedly',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13005','任务运行内存超过设置内存限制，导致Spark app应用退出，请在管理台增加driver内存或在提交任务时通过spark.driver.memory调整内存','Spark application has already stopped',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13006','任务运行内存超过设置内存限制，导致Spark context应用退出，请在管理台增加driver内存或在提交任务时通过spark.driver.memory调整内存','Spark application sc has already stopped',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13007','任务运行内存超过设置内存限制，导致Pyspark子进程退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','Pyspark process has stopped',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13008','任务产生的序列化结果总大小超过了配置的spark.driver.maxResultSize限制。请检查您的任务，看看是否有可能减小任务产生的结果大小，或则可以考虑压缩或合并结果，以减少传输的数据量','is bigger than spark.driver.maxResultSize',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13009','您的任务因为引擎退出（退出可能是引擎进程OOM或者主动kill引擎）导致失败','ERROR EC exits unexpectedly and actively kills the task',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13010','任务运行内存超过设置内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','Container exited with a non-zero exit code',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13011','广播表过大导致driver内存溢出，请在执行sql前增加参数后重试：set spark.sql.autoBroadcastJoinThreshold=-1;','dataFrame to local exception',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13012','driver内存不足，请增加driver内存后重试','Failed to allocate a page (\\S+.*\\)), try again.',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13013','使用spark默认变量sc导致后续代码执行失败','sc.setJobGroup(\\S+.*\\))',0); +-- 21 cluster Authority 22 db Authority +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21001','会话创建失败，用户%s不能提交应用到队列：%s，请联系提供队列给您的人员','User (\\S+) cannot submit applications to queue ([A-Za-z._0-9]+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21002','创建Python解释器失败，请联系管理员','initialize python executor failed',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21003','创建单机Python解释器失败，请联系管理员','PythonSession process cannot be initialized',0); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22001','%s无权限访问，请申请开通数据表权限，请联系您的数据管理人员','Permission denied:\\s*user=[a-zA-Z0-9_]+[,，]\\s*access=[a-zA-Z0-9_]+\\s*[,，]\\s*inode="([a-zA-Z0-9/_\\.]+)"',0); +-- INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22002','您可能没有相关权限','Permission denied',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22003','所查库表无权限','Authorization failed:No privilege',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22004','用户%s在机器不存在，请确认是否申请了相关权限','user (\\S+) does not exist',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22005','用户在机器不存在，请确认是否申请了相关权限','engineConnExec.sh: Permission denied',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22006','用户在机器不存在，请确认是否申请了相关权限','at com.sun.security.auth.UnixPrincipal',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22007','用户在机器不存在，请确认是否申请了相关权限','LoginException: java.lang.NullPointerException: invalid null input: name',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22008','用户在机器不存在，请确认是否申请了相关权限','User not known to the underlying authentication module',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22009','用户组不存在','FileNotFoundException: /tmp/?',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('22010','用户组不存在','error looking up the name of group',0); + +-- 30 Space exceeded 31 user operation +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('30001','库超过限制','is exceeded',0); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('31001','用户主动kill任务','is killed by user',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('31002','您提交的EngineTypeLabel没有对应的引擎版本','EngineConnPluginNotFoundException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('30003','用户Token下发失败，请确认用户初始化是否成功。可联系BDP Hive运维处理','Auth failed for User',0); + +-- 41 not exist 44 sql 43 python 44 shell 45 scala 46 importExport +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41001','数据库%s不存在，请检查引用的数据库是否有误','Database ''([a-zA-Z_0-9]+)'' not found',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41001','数据库%s不存在，请检查引用的数据库是否有误','Database does not exist: ([a-zA-Z_0-9]+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41002','表%s不存在，请检查引用的表是否有误','Table or view not found: ([\\.a-zA-Z_0-9]+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41002','表%s不存在，请检查引用的表是否有误','Table not found ''([a-zA-Z_0-9]+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41002','表%s不存在，请检查引用的表是否有误','Table ([a-zA-Z_0-9]+) not found',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41003','字段%s不存在，请检查引用的字段是否有误','cannot resolve ''(.+)'' given input columns',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41003','字段%s不存在，请检查引用的字段是否有误',' Invalid table alias or column reference ''(.+)'':',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41003','字段%s不存在，请检查引用的字段是否有误','Column ''(.+)'' cannot be resolved',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','分区字段%s不存在，请检查引用的表%s是否为分区表或分区字段有误','([a-zA-Z_0-9]+) is not a valid partition column in table ([\\.a-zA-Z_0-9]+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','分区字段%s不存在，请检查引用的表是否为分区表或分区字段有误','Partition spec \\{(\\S+)\\} contains non-partition columns',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','分区字段%s不存在，请检查引用的表是否为分区表或分区字段有误','table is not partitioned but partition spec exists:\\{(.+)\\}',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41004','表对应的路径不存在，请联系您的数据管理人员','Path does not exist: viewfs',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('41005','文件%s不存在','Caused by:\\s*java.io.FileNotFoundException',0); + +-- 42 sql +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42001','括号不匹配，请检查代码中括号是否前后匹配','extraneous input ''\\)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42002','非聚合函数%s必须写在group by中，请检查代码的group by语法','expression ''(\\S+)'' is neither present in the group by',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42002','非聚合函数%s必须写在group by中，请检查代码的group by语法','grouping expressions sequence is empty,\\s?and ''(\\S+)'' is not an aggregate function',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42002','非聚合函数%s必须写在group by中，请检查代码的group by语法','Expression not in GROUP BY key ''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Undefined function: ''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Invalid function ''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42004','字段%s存在名字冲突，请检查子查询内是否有同名字段','Reference ''(\\S+)'' is ambiguous',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42004','字段%s存在名字冲突，请检查子查询内是否有同名字段','Ambiguous column Reference ''(\\S+)'' in subquery',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42005','字段%s必须指定表或者子查询别名，请检查该字段来源','Column ''(\\S+)'' Found in more than One Tables/Subqueries',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库%s中已经存在，请删除相应表后重试','Table or view ''(\\S+)'' already exists in database ''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库中已经存在，请删除相应表后重试','Table (\\S+) already exists',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库中已经存在，请删除相应表后重试','Table already exists',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42006','表%s在数据库中已经存在，请删除相应表后重试','AnalysisException: (\\S+) already exists',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42007','插入目标表字段数量不匹配,请检查代码！','requires that the data to be inserted have the same number of columns as the target table',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42008','数据类型不匹配，请检查代码！','due to data type mismatch: differing types in',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42009','字段%s引用有误，请检查字段是否存在！','Invalid column reference (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42010','字段%s提取数据失败','Can''t extract value from (\\S+): need',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42011','括号或者关键字不匹配，请检查代码！','mismatched input ''(\\S+)'' expecting',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42012','group by 位置2不在select列表中，请检查代码！','GROUP BY position (\\S+) is not in select list',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42013','字段提取数据失败请检查字段类型','Can''t extract value from (\\S+): need struct type but got string',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42014','插入数据未指定目标表字段%s，请检查代码！','Cannot insert into target table because column number/types are different ''(S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42015','表别名%s错误，请检查代码！','Invalid table alias ''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42016','UDF函数未指定参数，请检查代码！','UDFArgumentException Argument expected',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42017','聚合函数%s不能写在group by 中，请检查代码！','aggregate functions are not allowed in GROUP BY',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42018','您的代码有语法错误，请您修改代码之后执行','SemanticException Error in parsing',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42019','表不存在，请检查引用的表是否有误','table not found',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42020','函数使用错误，请检查您使用的函数方式','No matching method',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42021','您的sql代码可能有语法错误，请检查sql代码','FAILED: ParseException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42022','您的sql代码可能有语法错误，请检查sql代码','org.apache.spark.sql.catalyst.parser.ParseException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42022','您的sql代码可能有语法错误，请检查sql代码','org.apache.hadoop.hive.ql.parse.ParseException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42023','聚合函数不能嵌套','aggregate function in the argument of another aggregate function',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42024','聚合函数不能嵌套','aggregate function parameters overlap with the aggregation',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42025','union 的左右查询字段不一致','Union can only be performed on tables',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42025','hql报错，union 的左右查询字段不一致','both sides of union should match',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42025','union左表和右表类型不一致','on first table and type',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42026','您的建表sql不能推断出列信息','Unable to infer the schema',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42027','动态分区的严格模式需要指定列，您可用通过设置set hive.exec.dynamic.partition.mode=nostrict','requires at least one static partition',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42028','函数输入参数有误','Invalid number of arguments for function',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42029','sql语法报错，select * 与group by无法一起使用','not allowed in select list when GROUP BY ordinal',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42030','where/having子句之外不支持引用外部查询的表达式','the outer query are not supported outside of WHERE',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42031','sql语法报错，group by 后面不能跟一个表','show up in the GROUP BY list',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42032','hql报错，窗口函数中的字段重复','check for circular dependencies',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42033','sql中出现了相同的字段','Found duplicate column',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42034','sql语法不支持','not supported in current context',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42035','hql语法报错，嵌套子查询语法问题','Unsupported SubQuery Expression',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42036','hql报错，子查询中in 用法有误','in definition of SubQuery',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43037','表字段类型修改导致的转型失败，请联系修改人员','cannot be cast to',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43038','select 的表可能有误','Invalid call to toAttribute on unresolved object',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43039','语法问题，请检查脚本','Distinct window functions are not supported',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43040','查询一定要指定数据源和库信息','Schema must be specified when session schema is not set',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43041','用户UDF函数 %s 加载失败，请检查后再执行','Invalid function (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43042','插入数据表动态分区数超过配置值 %s ，请优化sql或调整配置hive.exec.max.dynamic.partitions后重试','Maximum was set to (\\S+) partitions per node',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43043','执行任务消耗内存超过限制，hive任务请修改map或reduce的内存，spark任务请修改executor端内存','Error：java heap space',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43044','表 %s 分区数超过阈值 %s，需要分批删除分区，再删除表','the partitions of table (\\S+) exceeds threshold (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43045','查询/操作的表 %s 分区数为 %s ，超过阈值 %s ，需要限制查询/操作的分区数量','Number of partitions scanned \\(=(\\d+)\\) on table (\\S+) exceeds limit \\(=(\\d+)\\)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43046','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Number of dynamic partitions created is (\\S+), which is more than (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43047','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Maximum was set to (\\S+) partitions per node, number of dynamic partitions on this node: (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43048','参数引用错误，请检查参数 %s 是否正常引用','UnboundLocalError.*local variable (\\S+) referenced before assignment',0); +-- 43 python +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43001','代码中存在NoneType空类型变量，请检查代码','''NoneType'' object',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43002','数组越界','IndexError:List index out of range',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43003','您的代码有语法错误，请您修改代码之后执行','SyntaxError',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43004','python代码变量%s未定义','name ''(\\S+)'' is not defined',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43005','python udf %s 未定义','Undefined function:s+''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43006','python执行不能将%s和%s两种类型进行连接','cannot concatenate ''(\\S+)'' and ''(\\S+)''',0); +-- INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43007','pyspark执行失败，可能是语法错误或stage失败','Py4JJavaError: An error occurred',0); +-- INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43008','python代码缩进对齐有误','unexpected indent',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43009','python代码缩进有误','unexpected indent',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43010','python代码反斜杠后面必须换行','unexpected character after line',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43011','导出Excel表超过最大限制1048575','Invalid row number',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43012','python save as table未指定格式，默认用parquet保存，hive查询报错','parquet.io.ParquetDecodingException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43013','索引使用错误','IndexError',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43014','sql语法有问题','raise ParseException',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43015','python代码变量%s未定义','ImportError: ''(\\S+)''',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43015','当前节点需要的CS表解析失败，请检查当前CSID对应的CS表是否存在','Cannot parse cs table for node',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43016','模块 %s 没有属性 %s ，请确认代码引用是否正常','AttributeError: \'(\\S+)\' object has no attribute \'(\\S+)\'',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43017','存在参数无效或拼写错误，请确认 %s 参数正确性','KeyError: (.*)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43018','文件未找到，请确认该路径( %s )是否存在','FileNotFoundError.*No such file or directory\\:\\s\'(\\S+)\'',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43019','执行表在元数据库中存在meta缓存，meta信息与缓存不一致导致，请增加参数(--conf spark.sql.hive.convertMetastoreOrc=false)后重试','Unable to alter table.*Table is not allowed to be altered',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43020','Python 进程已停止，查询失败！','python process has stopped',0); + +-- 46 importExport +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46001','找不到导入文件地址：%s','java.io.FileNotFoundException: (\\S+) \\(No such file or directory\\)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46002','导出为excel时临时文件目录权限异常','java.io.IOException: Permission denied(.+)at org.apache.poi.xssf.streaming.SXSSFWorkbook.createAndRegisterSXSSFSheet',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46003','导出文件时无法创建目录：%s','java.io.IOException: Mkdirs failed to create (\\S+) (.+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46004','导入模块错误，系统没有%s模块，请联系运维人员安装','ImportError: No module named (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46005','导出语句错误，请检查路径或命名','Illegal out script',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('46006','可能是并发访问同一个HDFS文件，导致Filesystem closed问题，尝试重试','java.io.IOException: Filesystem closed\\n\\s+(at org.apache.hadoop.hdfs.DFSClient.checkOpen)',0); +-- 47 tuning +-- INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('47001','诊断任务异常：%s，详细异常: %s','Tuning-Code: (\\S+), Tuning-Desc: (.+)',0); + + +-- 91 wtss +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91001','找不到变量值，请确认您是否设置相关变量','not find variable substitution for',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91002','不存在的代理用户，请检查你是否申请过平台层（bdp或者bdap）用户','failed to change current working directory ownership',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91003','请检查提交用户在WTSS内是否有该代理用户的权限，代理用户中是否存在特殊字符，是否用错了代理用户，OS层面是否有该用户，系统设置里面是否设置了该用户为代理用户','没有权限执行当前任务',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91004','平台层不存在您的执行用户，请在ITSM申请平台层（bdp或者bdap）用户','使用chown命令修改',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91005','未配置代理用户，请在ITSM走WTSS用户变更单，为你的用户授权改代理用户','请联系系统管理员为您的用户添加该代理用户',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91006','您的用户初始化有问题，请联系管理员','java: No such file or directory',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('91007','JobServer中不存在您的脚本文件，请将你的脚本文件放入对应的JobServer路径中', 'Could not open input file for reading%does not exist',0); + +-- ---------------------------- +-- Default Tokens +-- ---------------------------- +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('LINKIS-7e3fb30c1c4b436cbfb9a245924d665f','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('WS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('DSS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('QUALITIS-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('VALIDATOR-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('LINKISCLI-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('DSM-UNAVAILABLE-TOKEN','*','*','BDP',curdate(),curdate(),-1,'LINKIS'); + +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('kafka', 'kafka', 'kafka', '消息队列', '', 2, 'Kafka', 'Kafka', 'Message Queue'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('hive', 'hive数据库', 'hive', '大数据存储', '', 3, 'Hive Database', 'Hive', 'Big Data storage'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('elasticsearch', 'elasticsearch数据源', 'es无结构化存储', '分布式全文索引', '', 3, 'Elasticsearch Datasource', 'Es No Structured Storage', 'Distributed Full-Text Indexing'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('mongodb', 'mongodb', 'NoSQL文档存储', 'NoSQL', null, 3, 'mongodb', 'NoSQL Document Storage', 'NOSQL'); + +-- jdbc +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('mysql', 'mysql数据库', 'mysql数据库', '关系型数据库', '', 3, 'Mysql Database', 'Mysql Database', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('oracle', 'oracle数据库', 'oracle', '关系型数据库', '', 3, 'Oracle Database', 'Oracle Relational Database', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('dm', '达梦数据库', 'dm', '关系型数据库', '', 3, 'Dameng Database', 'Dm', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('kingbase', '人大金仓数据库', 'kingbase', '关系型数据库', '', 3, 'Renmin Jincang Database', 'Kingbase', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('postgresql', 'postgresql数据库', 'postgresql', '关系型数据库', '', 3, 'Postgresql Database', 'Postgresql', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('sqlserver', 'sqlserver数据库', 'sqlserver', '关系型数据库', '', 3, 'Sqlserver Database', 'Sqlserver', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('db2', 'db2数据库', 'db2', '关系型数据库', '', 3, 'Db2 Database', 'Db2', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('greenplum', 'greenplum数据库', 'greenplum', '关系型数据库', '', 3, 'Greenplum Database', 'Greenplum', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('doris', 'doris数据库', 'doris', 'olap', '', 4, 'Doris Database', 'Doris', 'Olap'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('clickhouse', 'clickhouse数据库', 'clickhouse', 'olap', '', 4, 'Clickhouse Database', 'Clickhouse', 'Olap'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('tidb', 'tidb数据库', 'tidb', '关系型数据库', '', 3, 'TiDB Database', 'TiDB', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('starrocks', 'starrocks数据库', 'starrocks', 'olap', '', 4, 'StarRocks Database', 'StarRocks', 'Olap'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('gaussdb', 'gaussdb数据库', 'gaussdb', '关系型数据库', '', 3, 'GaussDB Database', 'GaussDB', 'Relational Database'); +INSERT INTO linkis_ps_dm_datasource_type (name, description, option, classifier, icon, layers, description_en, option_en, classifier_en) VALUES ('oceanbase', 'oceanbase数据库', 'oceanbase', 'olap', '', 4, 'oceanbase Database', 'oceanbase', 'Olap'); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'hive'; +-- SET @data_source=CONCAT('/data-source-manager/env-list/all/type/',@data_source_type_id); -- MySQL user variable, not supported in PostgreSQL +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'envId', '集群环境(Cluster env)', 'Cluster env', NULL, 'SELECT', NULL, 1, '集群环境(Cluster env)', 'Cluster env', NULL, NULL, NULL, @data_source, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'kafka'; +-- SET @data_source=CONCAT('/data-source-manager/env-list/all/type/',@data_source_type_id); -- MySQL user variable, not supported in PostgreSQL +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'envId', '集群环境(Cluster env)', 'Cluster env', NULL, 'SELECT', NULL, 1, '集群环境(Cluster env)', 'Cluster env', NULL, NULL, NULL, @data_source, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'mongodb'; +-- SET @data_source=CONCAT('/data-source-manager/env-list/all/type/',@data_source_type_id); -- MySQL user variable, not supported in PostgreSQL +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'username', '用户名', 'Username', NULL, 'TEXT', NULL, 1, '用户名', 'Username', '^[0-9A-Za-z_-]+$', NULL, '', NULL, now(), now()), + (@data_source_type_id, 'password', '密码', 'Password', NULL, 'PASSWORD', NULL, 1, '密码', 'Password', '', NULL, '', NULL, now(), now()), + (@data_source_type_id, 'database', '默认库', 'Database', NULL, 'TEXT', NULL, 1, '默认库', 'Database', '^[0-9A-Za-z_-]+$', NULL, '', NULL, now(), now()), + (@data_source_type_id, 'host', 'Host', 'Host', NULL, 'TEXT', NULL, 1, 'mongodb Host', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口', 'Port', NULL, 'TEXT', NULL, 1, '端口', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数', 'Params', NULL, 'TEXT', NULL, 0, '输入JSON格式: {"param":"value"}', 'Input JSON Format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'elasticsearch'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'username', '用户名(Username)' , 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, '', NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, '', NULL, now(), now()), + (@data_source_type_id, 'elasticUrls', 'ES连接URL(Elastic Url)', 'Elastic Url', NULL, 'TEXT', NULL, 1, 'ES连接URL(Elastic Url)', 'Elastic Url', '', NULL, '', NULL, now(), now()); + +-- https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url-format.html +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'mysql'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 0, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)','Port', NULL, 'TEXT', NULL, 0, '端口号(Port)','Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, 0, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 0, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 0, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'databaseName', '数据库名(Database name)', 'Database name', NULL, 'TEXT', NULL, 0, '数据库名(Database name)', 'Database name', NULL, NULL, NULL, NULL, now(), now()); + +-- https://docs.oracle.com/en/database/oracle/oracle-database/21/jajdb/oracle/jdbc/OracleDriver.html +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'oracle'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'oracle.jdbc.driver.OracleDriver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'sid', 'SID', 'SID', NULL, 'TEXT', NULL, 0, 'SID', 'SID', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'serviceName', 'service_name', 'service_name', NULL, 'TEXT', NULL, 0, 'service_name', 'service_name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'server', 'server', 'server', NULL, 'TEXT', NULL, 0, 'server', 'server', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 0, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'dm'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'dm.jdbc.driver.DmDriver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'kingbase'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.kingbase8.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'instance', NULL, NULL, NULL, NULL, now(), now()); + +-- https://jdbc.postgresql.org/documentation/use/ +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'postgresql'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'org.postgresql.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +-- https://learn.microsoft.com/zh-cn/sql/connect/jdbc/building-the-connection-url?redirectedfrom=MSDN&view=sql-server-ver16 +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'sqlserver'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.microsoft.sqlserver.jdbc.SQLServerDriver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +-- https://www.ibm.com/docs/en/db2/11.5?topic=cdsudidsdjs-url-format-data-server-driver-jdbc-sqlj-type-4-connectivity +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'db2'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.ibm.db2.jcc.DB2Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +-- https://greenplum.docs.pivotal.io/6-1/datadirect/datadirect_jdbc.html#topic_ylk_pbx_2bb +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'greenplum'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.pivotal.jdbc.GreenplumDriver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'doris'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +-- https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-jdbc +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'clickhouse'; +INSERT INTO linkis_ps_dm_datasource_type_key + (data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'ru.yandex.clickhouse.ClickHouseDriver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'hive'; +INSERT INTO linkis_ps_dm_datasource_env (env_name, env_desc, datasource_type_id, parameter, create_time, create_user, modify_time, modify_user) VALUES ('测试环境SIT', '测试环境SIT', @data_source_type_id, '{"uris":"thrift://localhost:9083", "hadoopConf":{"hive.metastore.execute.setugi":"true"}}', now(), NULL, now(), NULL); +INSERT INTO linkis_ps_dm_datasource_env (env_name, env_desc, datasource_type_id, parameter, create_time, create_user, modify_time, modify_user) VALUES ('测试环境UAT', '测试环境UAT', @data_source_type_id, '{"uris":"thrift://localhost:9083", "hadoopConf":{"hive.metastore.execute.setugi":"true"}}', now(), NULL, now(), NULL); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'kafka'; +INSERT INTO linkis_ps_dm_datasource_env (env_name, env_desc, datasource_type_id, parameter, create_time, create_user, modify_time, modify_user) VALUES ('kafka测试环境SIT', '开源测试环境SIT', @data_source_type_id, '{"uris":"thrift://localhost:9092"}', now(), NULL, now(), NULL); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'tidb'; +INSERT INTO linkis_ps_dm_datasource_type_key +(data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 0, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'starrocks'; +INSERT INTO linkis_ps_dm_datasource_type_key +(data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 0, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'gaussdb'; +INSERT INTO linkis_ps_dm_datasource_type_key +(data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'org.postgresql.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'oceanbase'; +INSERT INTO linkis_ps_dm_datasource_type_key +(data_source_type_id, key, name, name_en, default_value, value_type, scope, require, description, description_en, value_regex, ref_id, ref_value, data_source, update_time, create_time) +VALUES (@data_source_type_id, 'address', '地址', 'Address', NULL, 'TEXT', NULL, 0, '地址(host1:port1,host2:port2...)', 'Address(host1:port1,host2:port2...)', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'host', '主机名(Host)', 'Host', NULL, 'TEXT', NULL, 1, '主机名(Host)', 'Host', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'port', '端口号(Port)', 'Port', NULL, 'TEXT', NULL, 1, '端口号(Port)', 'Port', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'driverClassName', '驱动类名(Driver class name)', 'Driver class name', 'com.mysql.jdbc.Driver', 'TEXT', NULL, 1, '驱动类名(Driver class name)', 'Driver class name', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'params', '连接参数(Connection params)', 'Connection params', NULL, 'TEXT', NULL, 0, '输入JSON格式(Input JSON format): {"param":"value"}', 'Input JSON format: {"param":"value"}', NULL, NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'username', '用户名(Username)', 'Username', NULL, 'TEXT', NULL, 1, '用户名(Username)', 'Username', '^[0-9A-Za-z_-]+$', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'password', '密码(Password)', 'Password', NULL, 'PASSWORD', NULL, 1, '密码(Password)', 'Password', '', NULL, NULL, NULL, now(), now()), + (@data_source_type_id, 'instance', '实例名(instance)', 'Instance', NULL, 'TEXT', NULL, 1, '实例名(instance)', 'Instance', NULL, NULL, NULL, NULL, now(), now()); + +select @data_source_type_id := id from linkis_ps_dm_datasource_type where name = 'doris'; +UPDATE linkis_ps_dm_datasource_type_key SET require = 0 WHERE key ="password" and data_source_type_id = @data_source_type_id; +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +-- ============================================ +-- Linkis Database Upgrade DML Script +-- From: Base Version (db/linkis_dml.sql) +-- To: 1.15.0 +-- Generated: $(date '+%Y-%m-%d %H:%M:%S') +-- ============================================ + + +-- ============================================ +-- Version: 1.1.7 (dev-1.1.7) +-- ============================================ + + + + +update linkis_ps_configuration_config_key set key = 'linkis.trino.url' where key = 'wds.linkis.trino.url' ; +update linkis_ps_configuration_config_key set key = 'linkis.trino.catalog' where key = 'wds.linkis.trino.catalog' ; +update linkis_ps_configuration_config_key set key = 'linkis.trino.schema' where key = 'wds.linkis.trino.schema' ; +update linkis_ps_configuration_config_key set key = 'linkis.trino.source' where key = 'wds.linkis.trino.source' ; + + +update linkis_ps_configuration_config_key set validate_range = '[\"1h\",\"2h\",\"6h\",\"12h\",\"30m\",\"15m\",\"3m\"]' where key = 'wds.linkis.engineconn.max.free.time' and engine_conn_type = 'spark'; +update linkis_ps_configuration_config_key set description = '取值范围：3m,15m,30m,1h,2h,6h,12h' where key = 'wds.linkis.engineconn.max.free.time' and engine_conn_type = 'spark' ; + + +-- ============================================ +-- Version: 1.1.10 (dev-1.1.10) +-- ============================================ + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, is_hidden, is_advanced, level, treeName, engine_conn_type, en_description, en_name, en_treeName) VALUES ("spark.external.default.jars", '取值范围：file:///xxx.jar 多个路径时 逗号分隔', 'spark 支持额外的jar包列表', NULL, 'Regex', '^file:\/\/\/[\u4e00-\u9fa5_a-zA-Z0-9-.\/]*\.jar(?:,\s*file:\/\/\/[\u4e00-\u9fa5_a-zA-Z0-9-.\/]*\.jar)*?$', '0', '1', '1', 'spark资源设置', 'spark','Value Range: file:///xxx.jar', 'Spark External Default Jars', 'Spark Resource Settings'); + +INSERT INTO linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +( + SELECT config.id AS config_key_id, label.id AS engine_type_label_id + FROM ( + select * from linkis_ps_configuration_config_key + where key="spark.external.default.jars" + and engine_conn_type="spark") config + INNER JOIN linkis_cg_manager_label label ON label.label_value ="*-*,spark-2.4.3" +); + +INSERT INTO linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +( + SELECT relation.config_key_id AS config_key_id, NULL AS config_value, relation.engine_type_label_id AS config_label_id + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN ( select * from linkis_ps_configuration_config_key where key="spark.external.default.jars" and engine_conn_type="spark") config on relation.config_key_id=config.id + INNER JOIN ( select * from linkis_cg_manager_label where label_value ="*-*,spark-2.4.3") label on label.id=relation.engine_type_label_id +); + + +-- ============================================ +-- Version: 1.1.12 (dev-1.1.12) +-- ============================================ + + +-- ============================================ +-- Version: 1.3.0 (dev-1.3.0) +-- ============================================ + +UPDATE linkis_ps_configuration_config_key SET validate_range ='[\",\",\"\\\\t\",\"\\\\;\",\"\\\\|\"]',description ="取值范围:,或\\t或;或|" WHERE key= "pipeline.field.split"; + +DELETE FROM linkis_ps_error_code WHERE error_code = "43007"; + +UPDATE linkis_ps_error_code SET error_regex='Permission denied:\\s*user=[a-zA-Z0-9_]+[,，]\\s*access=[a-zA-Z0-9_]+\\s*[,，]\\s*inode="([a-zA-Z0-9/_\\.]+)"' WHERE error_code = "22001"; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13010','任务实际运行内存超过了设置的内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory增加内存','Container exited with a non-zero exit code',0); + +UPDATE linkis_ps_configuration_config_key SET key="pipeline.output.isoverwrite" where key = "pipeline.output.isoverwtite"; + + +-- ============================================ +-- Version: 1.4.0 (dev-1.4.0) +-- ============================================ + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43042','插入数据表动态分区数超过配置值 %s ，请优化sql或调整配置hive.exec.max.dynamic.partitions后重试','Maximum was set to (\\S+) partitions per node',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43043','执行任务消耗内存超过限制，hive任务请修改map或reduce的内存，spark任务请修改executor端内存','Error：java heap space',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43044','表 %s 分区数超过阈值 %s，需要分批删除分区，再删除表','the partitions of table (\\S+) exceeds threshold (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43045','查询/操作的表 %s 分区数为 %s ，超过阈值 %s ，需要限制查询/操作的分区数量','Number of partitions scanned \\(=(\\d+)\\) on table (\\S+) exceeds limit \\(=(\\d+)\\)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43046','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Number of dynamic partitions created is (\\S+), which is more than (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43047','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Maximum was set to (\\S+) partitions per node, number of dynamic partitions on this node: (\\S+)',0); + + +-- ============================================ +-- Version: 1.6.0 (dev-1.6.0) +-- ============================================ + + +INSERT INTO linkis_ps_dm_datasource_type_key (data_source_type_id, key, name, default_value, value_type, scope, require, description, value_regex, ref_id, ref_value, data_source, update_time, create_time, name_en, description_en) VALUES(5, 'userClientIp', 'userClientIp', NULL, 'TEXT', 'ENV', 0, 'userClientIp', NULL, NULL, NULL, NULL, now(),now(), 'user client ip', 'user client ip'); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43019','执行表在元数据库中存在meta缓存，meta信息与缓存不一致导致，请增加参数(--conf spark.sql.hive.convertMetastoreOrc=false)后重试','Unable to alter table.*Table is not allowed to be altered',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13011','广播表过大导致driver内存溢出，请在执行sql前增加参数后重试：set spark.sql.autoBroadcastJoinThreshold=-1;','dataFrame to local exception',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43048','参数引用错误，请检查参数 %s 是否正常引用','UnboundLocalError.*local variable (\\S+) referenced before assignment',0); + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容" WHERE error_code = "01002"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容" WHERE error_code = "01003"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Spark app应用退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13005"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Spark context应用退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13006"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Pyspark子进程退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13007"; +UPDATE linkis_ps_error_code SET error_desc = "您的任务因为引擎退出（退出可能是引擎进程OOM或者主动kill引擎）导致失败" WHERE error_code = "13009"; + +-- 注意: 原1.6.0版本的DML中包含CREATE TABLE linkis_org_user_sync语句 +-- 该表已在基础DDL中定义,DML文件不应包含DDL语句,因此已移除 +-- 参考: db/linkis_ddl.sql 第1260行 + + +-- ============================================ +-- Version: 1.7.0 (dev-1.7.0) +-- ============================================ + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13012','driver内存不足，请增加driver内存后重试','Failed to allocate a page (\\S+.*\\)), try again.',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13013','使用spark默认变量sc导致后续代码执行失败','sc.setJobGroup(\\S+.*\\))',0); +DELETE FROM linkis_ps_error_code WHERE error_code = "43016"; +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43016','模块 %s 没有属性 %s ，请确认代码引用是否正常','AttributeError: \'(\\S+)\' object has no attribute \'(\\S+)\'',0); +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台调整内存参数。" WHERE error_code = "13004"; +INSERT INTO linkis_cg_manager_label (label_key,label_value,label_feature,label_value_size,update_time,create_time) VALUES ('combined_userCreator_engineType','*-IDE,nebula-3.0.0','OPTIONAL',2,now(),now()); +INSERT INTO linkis_cg_manager_label (label_key,label_value,label_feature,label_value_size,update_time,create_time) VALUES ('combined_userCreator_engineType','*-*,nebula-3.0.0','OPTIONAL',2,now(),now()); + +insert into linkis_ps_configuration_category (label_id, level) VALUES ((select id from linkis_cg_manager_label where label_value = '*-IDE,nebula-3.0.0'), 2); + +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.host','Nebula 连接地址','Nebula 连接地址',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Host','Nebula Host',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.port','Nebula 连接端口','Nebula 连接端口',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Port','Nebula Port',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.username','Nebula 连接用户名','Nebula 连接用户名',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Username','Nebula Username',0); +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.password','Nebula 连接密码','Nebula 连接密码',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Password','Nebula Password',0); + +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.key = 'linkis.nebula.host' and label_value = '*-*,nebula-3.0.0'); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.key = 'linkis.nebula.port' and label_value = '*-*,nebula-3.0.0'); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.key = 'linkis.nebula.username' and label_value = '*-*,nebula-3.0.0'); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.key = 'linkis.nebula.password' and label_value = '*-*,nebula-3.0.0'); + +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '127.0.0.1' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.nebula.host') AND label.label_value = '*-*,nebula-3.0.0'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '9669' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.nebula.port') AND label.label_value = '*-*,nebula-3.0.0'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, 'nebula' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.nebula.username') AND label.label_value = '*-*,nebula-3.0.0'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, 'nebula' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.nebula.password') AND label.label_value = '*-*,nebula-3.0.0'); + +INSERT INTO linkis_mg_gateway_auth_token(token_name,legal_users,legal_hosts,business_owner,create_time,update_time,elapse_day,update_by) VALUES ('DOCTOR-AUTH-LEstzFKwKkrALsDOuGg', '*', '*', 'BDP', DATE_FORMAT(NOW(), '%Y-%m-%d'), DATE_FORMAT(NOW(), '%Y-%m-%d'), -1, 'LINKIS'); + + +-- ============================================ +-- Version: 1.9.0 (dev-1.9.0) +-- ============================================ + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'wds.linkis.jdbc.driver', '例如:com.mysql.jdbc.Driver', 'jdbc连接驱动', '', 'None', '', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'For Example: com.mysql.jdbc.Driver', 'JDBC Connection Driver', 0); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.key = 'wds.linkis.jdbc.driver' and label_value = '*-*,jdbc-4'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'wds.linkis.jdbc.driver') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.time', '单位：分钟', 'jdbc任务任务超时告警时间', '', 'Regex', '^[1-9]\\d*$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Unit: Minutes', 'JDBC Task Timeout Alert Time', 0); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.key = 'linkis.jdbc.task.timeout.alert.time' and label_value = '*-*,jdbc-4'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.jdbc.task.timeout.alert.time') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.user', '多人用英文逗号分隔', 'jdbc任务任务超时告警人', '', 'Regex', '^[a-zA-Z0-9,_-]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Multiple People Separated By Commas In English', 'JDBC Task Timeout Alert Person', 0); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.key = 'linkis.jdbc.task.timeout.alert.user' and label_value = '*-*,jdbc-4'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.jdbc.task.timeout.alert.user') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.level', '超时告警级别:1 critical,2 major,3 minor,4 warning,5 info', 'jdbc任务任务超时告警级别', '3', 'NumInterval', '[1,5]', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Timeout Alert Levels: 1 Critical, 2 Major, 3 Minor, 4 Warning, 5 Info', 'JDBC Task Timeout Alert Level', 0); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.key = 'linkis.jdbc.task.timeout.alert.level' and label_value = '*-*,jdbc-4'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.jdbc.task.timeout.alert.level') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.datasource.type', '多个数据源用英文逗号分隔', '超时告警支持数据源类型', 'starrocks', 'Regex', '^[a-zA-Z0-9,]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Separate Multiple Data Sources With Commas In English', 'Timeout Alert Supports Data Source Types', 0); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.key = 'linkis.jdbc.task.timeout.alert.datasource.type' and label_value = '*-*,jdbc-4'); +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'linkis.jdbc.task.timeout.alert.datasource.type') AND label.label_value = '*-*,jdbc-4'); + + +INSERT INTO linkis_ps_dm_datasource_type_key (data_source_type_id, key, name, default_value, value_type, scope, require, description, value_regex, ref_id, ref_value, data_source, update_time, create_time, name_en, description_en) VALUES ((select id from linkis_ps_dm_datasource_type where name = 'starrocks'), 'kill_task_time', '超时kill任务时间', NULL, 'TEXT', NULL, 0, '配置任务超时时间，满足配置执行kill，单位：分钟', '^[1-9]\\d*$', NULL, NULL, NULL, now(), now(), 'Timeout Kill Task Time', 'Configure Task Timeout To Meet The Requirement Of Executing The Kill Action'); + +UPDATE linkis_ps_error_code set error_desc ="任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台调整内存后使用",error_regex="failed because the (hive|python|shell|jdbc|io_file|io_hdfs|fps|pipeline|presto|nebula|flink|appconn|sqoop|datax|openlookeng|trino|elasticsearch|seatunnel|hbase|jobserver) engine quitted unexpectedly" WHERE error_code = "13004"; +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13014','任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','failed because the spark engine quitted unexpectedly',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43050','特殊UDF不支持在非sql脚本中使用','Not support spacial udf in non-SQL script',0); + +INSERT INTO linkis_ps_configuration_config_key (key, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, level, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'jdbc引擎初始化内存大小', '1g', 'Regex', '^([1-9]|10)(G|g)$', 'jdbc', 0, 0, 1, '用户配置', 0, 'Value range: 1-10, Unit: G', 'JDBC Engine Initialization Memory Size', 'User Configuration', 0); +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) (select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.key = 'wds.linkis.engineconn.java.driver.memory' and label_value = '*-*,jdbc-4'); +INSERT INTO linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = (select id FROM linkis_ps_configuration_config_key where key = 'wds.linkis.engineconn.java.driver.memory'and engine_conn_type = 'jdbc') AND label.label_value = '*-*,jdbc-4'); + + +-- ============================================ +-- Version: 1.10.0 (dev-1.10.0) +-- ============================================ + +update linkis_ps_configuration_config_key set description ="取值范围：1-8000，单位：个",validate_range ="^(?:[1-9]\\d{0,2}|[1-7]\\d{3}|8000)$",en_description ="Value Range: 1-8000, Unit: Piece" where key = 'wds.linkis.rm.yarnqueue.cores.max'; + +update linkis_ps_configuration_config_key set description ="取值范围：1-20000，单位：G",validate_range ="^(?:[1-9]\\d{0,3}|[1]\\d{4}|20000)(G|g)$",en_description ="Value Range: 1-20000, Unit: G" where key = 'wds.linkis.rm.yarnqueue.memory.max'; + +update linkis_ps_configuration_config_key set description ="范围：1-100，单位：个",validate_range ="[1,100]",en_description ="Range: 1-100, unit: piece" where key = 'wds.linkis.rm.instance' and engine_conn_type ="spark"; + +update linkis_ps_configuration_key_limit_for_user set max_value ='8000' where key_id = (SELECT id FROM linkis_ps_configuration_config_key where key = 'wds.linkis.rm.yarnqueue.cores.max'); + +update linkis_ps_configuration_key_limit_for_user set max_value ='20000g' where key_id = (SELECT id FROM linkis_ps_configuration_config_key where key = 'wds.linkis.rm.yarnqueue.memory.max'); + +update linkis_ps_configuration_key_limit_for_user set max_value ='100' where key_id = (SELECT id FROM linkis_ps_configuration_config_key where key = 'wds.linkis.rm.instance' and engine_conn_type ='spark'); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21004','Hive Metastore存在问题，生产请联系生产服务助手进行处理，测试请联系Hive开发','Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient',0); + + +-- ============================================ +-- Version: 1.11.0 (dev-1.11.0) +-- ============================================ + +update linkis_ps_configuration_config_key set validate_range ="^(?!root\\.).*",validate_type ="Regex" where key="wds.linkis.rm.yarnqueue"; + + +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',CONCAT('*-*,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); + +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) +(select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'spark' and label.label_value = CONCAT('*-*,',"spark-3.4.4")); + + +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',CONCAT('*-IDE,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',CONCAT('*-Visualis,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); +insert into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) VALUES ('combined_userCreator_engineType',CONCAT('*-nodeexecution,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); + + +insert into linkis_ps_configuration_category (label_id, level) VALUES ((select id from linkis_cg_manager_label where label_value = CONCAT('*-IDE,',"spark-3.4.4")), 2); +insert into linkis_ps_configuration_category (label_id, level) VALUES ((select id from linkis_cg_manager_label where label_value = CONCAT('*-Visualis,',"spark-3.4.4")), 2); +insert into linkis_ps_configuration_category (label_id, level) VALUES ((select id from linkis_cg_manager_label where label_value = CONCAT('*-nodeexecution,',"spark-3.4.4")), 2); + + +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) +(select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = CONCAT('*-*,',"spark-3.4.4")); + + +INSERT IGNORE INTO linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) +SELECT label_key, REPLACE(label_value, 'spark-2.4.3', 'spark-3.4.4') AS label_value, label_feature, label_value_size, NOW() , NOW() +FROM linkis_cg_manager_label +WHERE label_key = 'combined_userCreator_engineType' + AND label_value LIKE '%-IDE,spark-2.4.3' + AND label_value != "*-IDE,spark-2.4.3"; + +INSERT IGNORE INTO linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id, update_time, create_time) +SELECT config_key.id as config_key_id,"python3" as config_value ,label.id as config_label_id, NOW() , NOW() FROM linkis_cg_manager_label label,linkis_ps_configuration_config_key config_key +WHERE config_key.key ="spark.python.version" +AND label.label_key = 'combined_userCreator_engineType' +AND label.label_value LIKE '%-IDE,spark-3.4.4' + + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Cannot resolve function (\\S+)',0); + + +-- ============================================ +-- Version: 1.13.0 (dev-1.13.0) +-- ============================================ + +update linkis_ps_configuration_config_key set default_value = "com.mysql.jdbc.Driver" where engine_conn_type = "jdbc" and key="wds.linkis.jdbc.driver"; + +insert IGNORE into linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) +select 'combined_userCreator_engineType',REPLACE(label_value, '2.4.3', '3.4.4'), 'OPTIONAL', 2, now(), now() +from linkis_cg_manager_label +where label_value like "%spark-2.4.3%"; + +insert IGNORE into linkis_ps_configuration_category (label_id, level) +select id as label_id, 2 as level +from linkis_cg_manager_label +where label_value in (select REPLACE(label_value, '2.4.3', '3.4.4') + from linkis_cg_manager_label + where id in ( + select cate.label_id + from linkis_ps_configuration_category cate, linkis_cg_manager_label label + where cate.label_id = label .id and label.label_value like "%spark-2.4.3%")); + +INSERT INTO linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id, update_time, create_time) +SELECT cofig.config_key_id AS config_key_id, cofig.config_value AS config_value, res.sp3 AS config_label_id, NOW() AS update_time, NOW() AS create_time +FROM + (SELECT * FROM linkis_ps_configuration_config_value + WHERE config_label_id IN + (SELECT id FROM linkis_cg_manager_label WHERE label_value LIKE "%spark-2.4.3%") + AND config_key_id IN + (SELECT id FROM linkis_ps_configuration_config_key WHERE engine_conn_type = "spark")) cofig, + (SELECT * FROM + (SELECT id AS sp3, label_value AS lp3 FROM linkis_cg_manager_label WHERE label_value LIKE "%spark-3.4.4%") a, + (SELECT id AS sp2, label_value AS lp2 FROM linkis_cg_manager_label WHERE label_value LIKE "%spark-2.4.3%") b + WHERE SUBSTRING_INDEX(a.lp3, ',', 1) = SUBSTRING_INDEX(b.lp2, ',', 1)) res +WHERE cofig.config_label_id = res.sp2 +ON DUPLICATE KEY UPDATE + config_value = VALUES(config_value), + update_time = NOW(); + + +UPDATE linkis_ps_configuration_config_value +SET config_value = "python3" +WHERE config_label_id IN ( + SELECT id + FROM linkis_cg_manager_label + WHERE label_value LIKE "%spark-3.4.4%" + ) +AND config_key_id = (select id from linkis_ps_configuration_config_key where key= "spark.python.version"); + + + + + + +-- ============================================ +-- Version: 1.14.0 (dev-1.14.0) +-- ============================================ + +INSERT INTO linkis_ps_configuration_config_key (key,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,level,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('wds.linkis.engine.running.job.max', '引擎运行最大任务数', '引擎运行最大任务数', '30', 'None', NULL, 'shell', 0, 0, 1, 'shell引擎设置', 0, 'Maximum Number Of Tasks The Engine Can Run', 'Maximum Number For Engine', 'shell Engine Settings', 0); + +insert into linkis_ps_configuration_key_engine_relation (config_key_id, engine_type_label_id) ( select config.id as config_key_id, label.id AS engine_type_label_id FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'shell' and config.key = 'wds.linkis.engine.running.job.max' and label_value = '*-*,shell-1'); + + +insert into linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id) ( select relation.config_key_id AS config_key_id, '' AS config_value, relation.engine_type_label_id AS config_label_id FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where key = 'wds.linkis.engine.running.job.max') AND label.label_value = '*-*,shell-1'); + diff --git a/linkis-dist/package/db/upgrade/1.4.1_schema/mysql/linkis_dml.sql b/linkis-dist/package/db/upgrade/1.4.1_schema/mysql/linkis_dml.sql new file mode 100644 index 00000000000..ff186fd3fb5 --- /dev/null +++ b/linkis-dist/package/db/upgrade/1.4.1_schema/mysql/linkis_dml.sql @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +update linkis_ps_configuration_config_key set engine_conn_type = "" where engine_conn_type is NULL; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13008','任务产生的序列化结果总大小超过了配置的spark.driver.maxResultSize限制。请检查您的任务，看看是否有可能减小任务产生的结果大小，或则可以考虑压缩或合并结果，以减少传输的数据量','is bigger than spark.driver.maxResultSize',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13009','您的任务因为引擎退出（退出可能是引擎进程OOM或者主动kill引擎）导致失败','ERROR EC exits unexpectedly and actively kills the task',0); + +update linkis_ps_configuration_config_key set template_required = 1 where `key` in ( +"spark.executor.instances", +"spark.executor.memory", +"spark.driver.memory", +"wds.linkis.engineconn.java.driver.memory", +"mapreduce.job.running.map.limit", +"mapreduce.job.running.reduce.limit", +) +update linkis_ps_configuration_config_key set template_required = 1 where `key` = "wds.linkis.rm.instance" and engine_conn_type in ("spark","hive"); +-- spark.conf +INSERT INTO linkis_ps_configuration_config_key +(`key`, description, name, +default_value, validate_type, validate_range, engine_conn_type, +is_hidden, is_advanced, `level`, +treeName, boundary_type, en_treeName, +en_description, en_name) +VALUES( +'spark.conf', '多个参数使用分号[;]分隔 例如spark.shuffle.spill=true;', 'spark自定义配置参数', +null, 'None', NULL, 'spark', +0, 1, 1, +'spark资源设置', 0, 'Spark Resource Settings', +'Multiple parameters are separated by semicolons [;] For example, spark.shuffle.compress=ture;', 'Spark Custom Configuration Parameters'); + +INSERT INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +( + SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` + FROM ( + select * from linkis_ps_configuration_config_key + where `key`="spark.conf" + and `engine_conn_type`="spark") config + INNER JOIN linkis_cg_manager_label label ON label.label_value ="*-*,spark-2.4.3" +); + + + +INSERT INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +( + SELECT `relation`.`config_key_id` AS `config_key_id`, NULL AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN ( select * from linkis_ps_configuration_config_key where `key`="spark.conf" and `engine_conn_type`="spark") config on relation.config_key_id=config.id + INNER JOIN ( select * from linkis_cg_manager_label where label_value ="*-*,spark-2.4.3") label on label.id=relation.engine_type_label_id +); + + +-- spark.locality.wait + +INSERT INTO `linkis_ps_configuration_config_key` +(`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`) +VALUES +('spark.locality.wait', '范围：0-3，单位：秒', '任务调度本地等待时间', '3s', 'OFT', '[\"0s\",\"1s\",\"2s\",\"3s\"]', 'spark', 0, 1, 1, 'spark资源设置', 0, 'Spark Resource Settings', 'Range: 0-3, Unit: second', 'Task Scheduling Local Waiting Time'); + + +-- all 默认 +INSERT INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +( + SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` + FROM ( + select * from linkis_ps_configuration_config_key + where `key`="spark.locality.wait" + and `engine_conn_type`="spark") config + INNER JOIN linkis_cg_manager_label label ON label.label_value ="*-*,spark-2.4.3" +); + + + +INSERT INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +( + SELECT `relation`.`config_key_id` AS `config_key_id`, NULL AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN ( select * from linkis_ps_configuration_config_key where `key`="spark.locality.wait" and `engine_conn_type`="spark") config on relation.config_key_id=config.id + INNER JOIN ( select * from linkis_cg_manager_label where label_value ="*-*,spark-2.4.3") label on label.id=relation.engine_type_label_id +); + + +-- spark.memory.fraction +INSERT INTO `linkis_ps_configuration_config_key` +(`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`) +VALUES +('spark.memory.fraction', '范围：0.4,0.5,0.6，单位：百分比', '执行内存和存储内存的百分比', '0.6', 'OFT', '[\"0.4\",\"0.5\",\"0.6\"]', 'spark', 0, 1, 1, 'spark资源设置', 0, 'Spark Resource Settings', 'Range: 0.4, 0.5, 0.6, in percentage', 'Percentage Of Execution Memory And Storage Memory'); + + +-- all 默认 +INSERT INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +( + SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` + FROM ( + select * from linkis_ps_configuration_config_key + where `key`="spark.memory.fraction" + and `engine_conn_type`="spark") config + INNER JOIN linkis_cg_manager_label label ON label.label_value ="*-*,spark-2.4.3" +); + +INSERT INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +( + SELECT `relation`.`config_key_id` AS `config_key_id`, NULL AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN ( select * from linkis_ps_configuration_config_key where `key`="spark.memory.fraction" and `engine_conn_type`="spark") config on relation.config_key_id=config.id + INNER JOIN ( select * from linkis_cg_manager_label where label_value ="*-*,spark-2.4.3") label on label.id=relation.engine_type_label_id +); + + +UPDATE linkis_ps_error_code SET error_regex = "User (\\S+) cannot submit applications to queue ([A-Za-z._0-9]+)" WHERE error_code = "21001"; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43016','模块 %s 没有属性 %s ，请确认代码引用是否正常','AttributeError: \'(\\S+)\' object has no attribute \'(\\S+)\'',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43017','存在参数无效或拼写错误，请确认 %s 参数正确性','KeyError: (\\(.+\\))',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43018','文件未找到，请确认该路径( %s )是否存在','FileNotFoundError.*No such file or directory\\:\\s\'(\\S+)\'',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01006','没有健康可用的ecm节点，可能任务量大,导致节点资源处于不健康状态，尝试kill空闲引擎释放资源','There are corresponding ECM tenant labels',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('01007','文件编码格式异常,请联系管理人员处理','UnicodeEncodeError.*characters',0); +UPDATE linkis_ps_error_code SET error_regex = "KeyError: (.*)" WHERE error_code = "43017"; + +UPDATE linkis_ps_error_code SET error_desc = "任务实际运行内存超过了设置的内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory增加内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13002"; +update linkis_ps_configuration_config_key set validate_range ='[\",\",\"\\\\t\",\"\\\\;\",\"\\\\|\"]',description ="取值范围：,或\\t或;或|" WHERE `key`= "pipeline.field.split"; + +DELETE FROM linkis_ps_error_code WHERE error_code = "43007"; + +UPDATE linkis_ps_error_code SET error_regex='Permission denied:\\s*user=[a-zA-Z0-9_]+[,，]\\s*access=[a-zA-Z0-9_]+\\s*[,，]\\s*inode="([a-zA-Z0-9/_\\.]+)"' WHERE error_code = "22001"; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13010','任务实际运行内存超过了设置的内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory增加内存','Container exited with a non-zero exit code',0); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43042','插入数据表动态分区数超过配置值 %s ，请优化sql或调整配置hive.exec.max.dynamic.partitions后重试','Maximum was set to (\\S+) partitions per node',0); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43043','执行任务消耗内存超过限制，hive任务请修改map或reduce的内存，spark任务请修改executor端内存','Error：java heap space',0); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43044','表 %s 分区数超过阈值 %s，需要分批删除分区，再删除表','the partitions of table (\\S+) exceeds threshold (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43045','查询/操作的表 %s 分区数为 %s ，超过阈值 %s ，需要限制查询/操作的分区数量','Number of partitions scanned \\(=(\\d+)\\) on table (\\S+) exceeds limit \\(=(\\d+)\\)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43046','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Number of dynamic partitions created is (\\S+), which is more than (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43047','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Maximum was set to (\\S+) partitions per node, number of dynamic partitions on this node: (\\S+)',0); + +INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`,`en_description`, `en_name`, `en_treeName`, `template_required`) VALUES ('mapreduce.job.reduce.slowstart.completedmaps', '取值范围：0-1', 'Map任务数与总Map任务数之间的比例','0.05', 'Regex', '^(0(\\.\\d{1,2})?|1(\\.0{1,2})?)$', '0', '0', '1', 'hive引擎设置', 'hive', 'Value Range: 0-1', 'The Ratio Between The Number Of Map Tasks And The Total Number Of Map Tasks', 'Hive Engine Settings', '1'); + +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'hive' and config.`key` = "mapreduce.job.reduce.slowstart.completedmaps" and label_value = "*-*,hive-2.3.3"); + + +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = (select id FROM linkis_ps_configuration_config_key where `key`="mapreduce.job.reduce.slowstart.completedmaps")AND label.label_value = '*-*,hive-2.3.3'); + + +update linkis_ps_dm_datasource_type_key set value_regex='^[0-9A-Za-z_-\\s]+$' where data_source_type_id=18 and `key`='username'; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13002"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13010"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，请在管理台增加executor内存或调优sql后执行" WHERE error_code = "13003"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13004"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Spark app应用退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13005"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Spark context应用退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13006"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Pyspark子进程退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13007"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整executor内存或联系管理员扩容" WHERE error_code = "01002"; + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整executor内存或联系管理员扩容" WHERE error_code = "01003"; + + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43020','Python 进程已停止，查询失败！','python process has stopped',0); \ No newline at end of file diff --git a/linkis-dist/package/db/upgrade/1.9.0_schema/mysql/linkis_ddl.sql b/linkis-dist/package/db/upgrade/1.9.0_schema/mysql/linkis_ddl.sql new file mode 100644 index 00000000000..ad6abe601a5 --- /dev/null +++ b/linkis-dist/package/db/upgrade/1.9.0_schema/mysql/linkis_ddl.sql @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +ALTER TABLE `linkis_ps_configuration_config_key` + CHANGE COLUMN `validate_range` `validate_range` VARCHAR(150) NULL DEFAULT NULL COMMENT 'Validate range' COLLATE 'utf8_bin' AFTER `validate_type`; + +ALTER TABLE linkis_cg_ec_resource_info_record MODIFY COLUMN metrics text CHARACTER SET utf8 COLLATE utf8_bin NULL COMMENT 'ec metrics'; + + +DROP TABLE IF EXISTS `linkis_mg_gateway_whitelist_config`; +CREATE TABLE `linkis_mg_gateway_whitelist_config` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `allowed_user` varchar(128) COLLATE utf8_bin NOT NULL, + `client_address` varchar(128) COLLATE utf8_bin NOT NULL, + `create_time` datetime DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `address_uniq` (`allowed_user`, `client_address`), + KEY `linkis_mg_gateway_whitelist_config_allowed_user` (`allowed_user`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; + +DROP TABLE IF EXISTS `linkis_mg_gateway_whitelist_sensitive_user`; +CREATE TABLE `linkis_mg_gateway_whitelist_sensitive_user` ( + `id` int(11) NOT NULL AUTO_INCREMENT, + `sensitive_username` varchar(128) COLLATE utf8_bin NOT NULL, + `create_time` datetime DEFAULT NULL, + `update_time` datetime DEFAULT NULL, + PRIMARY KEY (`id`), + UNIQUE KEY `sensitive_username` (`sensitive_username`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; + +ALTER TABLE linkis_ps_python_module_info ADD COLUMN python_module varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '依赖python模块'; +ALTER TABLE linkis_ps_bml_project_resource ADD INDEX idx_resource_id (resource_id); + +DROP TABLE IF EXISTS `linkis_ps_job_history_diagnosis`; +CREATE TABLE `linkis_ps_job_history_diagnosis` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment', + `job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory', + `diagnosis_content` text COLLATE utf8mb4_bin COMMENT 'Diagnosis failed task information', + `created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time', + `updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time', + `only_read` varchar(5) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '1 just read,can not update', + PRIMARY KEY (`id`), + UNIQUE KEY `job_history_id` (`job_history_id`) +) ENGINE=InnoDB AUTO_INCREMENT=17 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT ='任务诊断分析表'; + +CREATE TABLE `linkis_mg_gateway_ecc_userinfo` ( + `id` BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID，自增', + `om_tool` VARCHAR(255) NOT NULL COMMENT '工具系统', + `user_id` VARCHAR(255) NOT NULL COMMENT '申请授权用户', + `op_user_id` VARCHAR(255) NOT NULL COMMENT '协助运维账号', + `roles` VARCHAR(255) NOT NULL COMMENT '角色列表，多个逗号,分隔', + `auth_system_id` VARCHAR(500) NOT NULL COMMENT '授权子系统名称ID，多个逗号,分隔', + `apply_itsm_id` VARCHAR(255) NOT NULL COMMENT 'ITSM申请单号，唯一，重复推送时根据这个字段做更新', + `effective_datetime` DATETIME NOT NULL COMMENT '生效时间，允许登录的最早时间', + `expire_datetime` DATETIME NOT NULL COMMENT '失效时间，根据这个时间计算cookie的有效期', + `created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间，默认当前时间', + `updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间，默认当前时间，更新时修改', + PRIMARY KEY (`id`), + UNIQUE INDEX `apply_itsm_id` (`apply_itsm_id`,`user_id`) +) +ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='转协查用户授权表'; + +-- AI 作业历史记录表 +CREATE TABLE IF NOT EXISTS `linkis_ps_ai_job_history` ( + `id` BIGINT(20) PRIMARY KEY AUTO_INCREMENT COMMENT '主键ID', + `job_history_id` VARCHAR(64) NOT NULL COMMENT '作业历史ID', + `submit_user` VARCHAR(50) NOT NULL COMMENT '提交用户', + `execute_user` VARCHAR(50) NOT NULL COMMENT '执行用户', + `submit_code` TEXT COMMENT '用户提交代码', + `execution_code` TEXT COMMENT '执行代码', + `metrics` text COMMENT 'metrics 信息', + `params` text COMMENT '任务参数', + `labels` text COMMENT '任务标签', + `error_code` int DEFAULT NULL COMMENT '错误码', + `error_desc` TEXT COMMENT '错误信息', + `engine_instances` VARCHAR(250) COMMENT '引擎实例', + `engine_type` VARCHAR(50) COMMENT '引擎类型', + `change_time` DATETIME(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT '切换时间', + `created_time` DATETIME(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT '创建时间', + `updated_time` DATETIME(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT '更新时间' +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin COMMENT='AI作业历史记录表'; + + +ALTER TABLE linkis_cg_manager_service_instance +ADD INDEX idx_instance_name (instance, name); + +-- 添加 token 密文存储字段 +ALTER TABLE `linkis_mg_gateway_auth_token` ADD COLUMN `token_sign` TEXT DEFAULT NULL COMMENT '存储token密文' AFTER `token_name`; diff --git a/linkis-dist/package/db/upgrade/1.9.0_schema/mysql/linkis_dml.sql b/linkis-dist/package/db/upgrade/1.9.0_schema/mysql/linkis_dml.sql new file mode 100644 index 00000000000..e5625ca1958 --- /dev/null +++ b/linkis-dist/package/db/upgrade/1.9.0_schema/mysql/linkis_dml.sql @@ -0,0 +1,287 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +update linkis_ps_configuration_config_key set `key` = 'linkis.trino.url' where `key` = 'wds.linkis.trino.url' ; +update linkis_ps_configuration_config_key set `key` = 'linkis.trino.catalog' where `key` = 'wds.linkis.trino.catalog' ; +update linkis_ps_configuration_config_key set `key` = 'linkis.trino.schema' where `key` = 'wds.linkis.trino.schema' ; +update linkis_ps_configuration_config_key set `key` = 'linkis.trino.source' where `key` = 'wds.linkis.trino.source' ; + + +update `linkis_ps_configuration_config_key` set `validate_range` = '[\"1h\",\"2h\",\"6h\",\"12h\",\"30m\",\"15m\",\"3m\"]' where `key` = 'wds.linkis.engineconn.max.free.time' and `engine_conn_type` = 'spark'; +update `linkis_ps_configuration_config_key` set `description` = '取值范围：3m,15m,30m,1h,2h,6h,12h' where `key` = 'wds.linkis.engineconn.max.free.time' and `engine_conn_type` = 'spark' ; + + +INSERT INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `is_hidden`, `is_advanced`, `level`, `treeName`, `engine_conn_type`, `en_description`, `en_name`, `en_treeName`) VALUES ("spark.external.default.jars", '取值范围：file:///xxx.jar 多个路径时 逗号分隔', 'spark 支持额外的jar包列表', NULL, 'Regex', '^file:\/\/\/[\u4e00-\u9fa5_a-zA-Z0-9-.\/]*\.jar(?:,\s*file:\/\/\/[\u4e00-\u9fa5_a-zA-Z0-9-.\/]*\.jar)*?$', '0', '1', '1', 'spark资源设置', 'spark','Value Range: file:///xxx.jar', 'Spark External Default Jars', 'Spark Resource Settings'); + +INSERT INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +( + SELECT config.id AS `config_key_id`, label.id AS `engine_type_label_id` + FROM ( + select * from linkis_ps_configuration_config_key + where `key`="spark.external.default.jars" + and `engine_conn_type`="spark") config + INNER JOIN linkis_cg_manager_label label ON label.label_value ="*-*,spark-2.4.3" +); + +INSERT INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +( + SELECT `relation`.`config_key_id` AS `config_key_id`, NULL AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` + FROM linkis_ps_configuration_key_engine_relation relation + INNER JOIN ( select * from linkis_ps_configuration_config_key where `key`="spark.external.default.jars" and `engine_conn_type`="spark") config on relation.config_key_id=config.id + INNER JOIN ( select * from linkis_cg_manager_label where label_value ="*-*,spark-2.4.3") label on label.id=relation.engine_type_label_id +); + + +UPDATE linkis_ps_configuration_config_key SET validate_range ='[\",\",\"\\\\t\",\"\\\\;\",\"\\\\|\"]',description ="取值范围:,或\\t或;或|" WHERE `key`= "pipeline.field.split"; + +DELETE FROM linkis_ps_error_code WHERE error_code = "43007"; + +UPDATE linkis_ps_error_code SET error_regex='Permission denied:\\s*user=[a-zA-Z0-9_]+[,，]\\s*access=[a-zA-Z0-9_]+\\s*[,，]\\s*inode="([a-zA-Z0-9/_\\.]+)"' WHERE error_code = "22001"; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13010','任务实际运行内存超过了设置的内存限制，请在管理台增加executor内存或在提交任务时通过spark.executor.memory增加内存','Container exited with a non-zero exit code',0); + +UPDATE linkis_ps_configuration_config_key SET `key`="pipeline.output.isoverwrite" where `key` = "pipeline.output.isoverwtite"; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43042','插入数据表动态分区数超过配置值 %s ，请优化sql或调整配置hive.exec.max.dynamic.partitions后重试','Maximum was set to (\\S+) partitions per node',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43043','执行任务消耗内存超过限制，hive任务请修改map或reduce的内存，spark任务请修改executor端内存','Error：java heap space',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43044','表 %s 分区数超过阈值 %s，需要分批删除分区，再删除表','the partitions of table (\\S+) exceeds threshold (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43045','查询/操作的表 %s 分区数为 %s ，超过阈值 %s ，需要限制查询/操作的分区数量','Number of partitions scanned \\(=(\\d+)\\) on table (\\S+) exceeds limit \\(=(\\d+)\\)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43046','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Number of dynamic partitions created is (\\S+), which is more than (\\S+)',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43047','动态分区一次性写入分区数 %s ，超过阈值 %s,请减少一次性写入的分区数','Maximum was set to (\\S+) partitions per node, number of dynamic partitions on this node: (\\S+)',0); + +INSERT INTO linkis_ps_dm_datasource_type_key (data_source_type_id, `key`, name, default_value, value_type, `scope`, `require`, description, value_regex, ref_id, ref_value, data_source, update_time, create_time, name_en, description_en) VALUES(5, 'userClientIp', 'userClientIp', NULL, 'TEXT', 'ENV', 0, 'userClientIp', NULL, NULL, NULL, NULL, now(),now(), 'user client ip', 'user client ip'); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43019','执行表在元数据库中存在meta缓存，meta信息与缓存不一致导致，请增加参数(--conf spark.sql.hive.convertMetastoreOrc=false)后重试','Unable to alter table.*Table is not allowed to be altered',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13011','广播表过大导致driver内存溢出，请在执行sql前增加参数后重试：set spark.sql.autoBroadcastJoinThreshold=-1;','dataFrame to local exception',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43048','参数引用错误，请检查参数 %s 是否正常引用','UnboundLocalError.*local variable (\\S+) referenced before assignment',0); + +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容" WHERE error_code = "01002"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Linkis服务负载过高，请在管理台调整Driver内存或联系管理员扩容" WHERE error_code = "01003"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Spark app应用退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13005"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Spark context应用退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13006"; +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致Pyspark子进程退出，请在管理台增加Driver内存或在提交任务时通过spark.driver.memory调整内存。更多细节请参考Linkis常见问题Q60" WHERE error_code = "13007"; +UPDATE linkis_ps_error_code SET error_desc = "您的任务因为引擎退出（退出可能是引擎进程OOM或者主动kill引擎）导致失败" WHERE error_code = "13009"; + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13012','driver内存不足，请增加driver内存后重试','Failed to allocate a page (\\S+.*\\)), try again.',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13013','使用spark默认变量sc导致后续代码执行失败','sc.setJobGroup(\\S+.*\\))',0); +DELETE FROM linkis_ps_error_code WHERE error_code = "43016"; +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43016','模块 %s 没有属性 %s ，请确认代码引用是否正常','AttributeError: \'(\\S+)\' object has no attribute \'(\\S+)\'',0); +UPDATE linkis_ps_error_code SET error_desc = "任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台调整内存参数。" WHERE error_code = "13004"; +INSERT INTO linkis_cg_manager_label (label_key,label_value,label_feature,label_value_size,update_time,create_time) VALUES ('combined_userCreator_engineType','*-IDE,nebula-3.0.0','OPTIONAL',2,now(),now()); +INSERT INTO linkis_cg_manager_label (label_key,label_value,label_feature,label_value_size,update_time,create_time) VALUES ('combined_userCreator_engineType','*-*,nebula-3.0.0','OPTIONAL',2,now(),now()); + +insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES ((select id from linkis_cg_manager_label where `label_value` = '*-IDE,nebula-3.0.0'), 2); + +INSERT INTO linkis_ps_configuration_config_key (`key`,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,`level`,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.host','Nebula 连接地址','Nebula 连接地址',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Host','Nebula Host',0); +INSERT INTO linkis_ps_configuration_config_key (`key`,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,`level`,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.port','Nebula 连接端口','Nebula 连接端口',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Port','Nebula Port',0); +INSERT INTO linkis_ps_configuration_config_key (`key`,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,`level`,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.username','Nebula 连接用户名','Nebula 连接用户名',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Username','Nebula Username',0); +INSERT INTO linkis_ps_configuration_config_key (`key`,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,`level`,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('linkis.nebula.password','Nebula 连接密码','Nebula 连接密码',NULL,'None',NULL,'nebula',0,0,1,'Necula引擎设置',0,'Nebula Engine Settings','Nebula Password','Nebula Password',0); + +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.`key` = 'linkis.nebula.host' and label_value = '*-*,nebula-3.0.0'); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.`key` = 'linkis.nebula.port' and label_value = '*-*,nebula-3.0.0'); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.`key` = 'linkis.nebula.username' and label_value = '*-*,nebula-3.0.0'); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'nebula' and config.`key` = 'linkis.nebula.password' and label_value = '*-*,nebula-3.0.0'); + +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '127.0.0.1' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.nebula.host') AND label.label_value = '*-*,nebula-3.0.0'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '9669' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.nebula.port') AND label.label_value = '*-*,nebula-3.0.0'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, 'nebula' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.nebula.username') AND label.label_value = '*-*,nebula-3.0.0'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, 'nebula' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.nebula.password') AND label.label_value = '*-*,nebula-3.0.0'); + +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('DOCTOR-AUTH-LEstzFKwKkrALsDOuGg', '*', '*', 'BDP', DATE_FORMAT(NOW(), '%Y-%m-%d'), DATE_FORMAT(NOW(), '%Y-%m-%d'), -1, 'LINKIS'); + + +INSERT INTO linkis_ps_configuration_config_key (`key`, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, `level`, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'wds.linkis.jdbc.driver', '例如:com.mysql.jdbc.Driver', 'jdbc连接驱动', '', 'None', '', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'For Example: com.mysql.jdbc.Driver', 'JDBC Connection Driver', 0); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.`key` = 'wds.linkis.jdbc.driver' and label_value = '*-*,jdbc-4'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'wds.linkis.jdbc.driver') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (`key`, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, `level`, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.time', '单位：分钟', 'jdbc任务任务超时告警时间', '', 'Regex', '^[1-9]\\d*$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Unit: Minutes', 'JDBC Task Timeout Alert Time', 0); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.`key` = 'linkis.jdbc.task.timeout.alert.time' and label_value = '*-*,jdbc-4'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.jdbc.task.timeout.alert.time') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (`key`, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, `level`, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.user', '多人用英文逗号分隔', 'jdbc任务任务超时告警人', '', 'Regex', '^[a-zA-Z0-9,_-]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Multiple People Separated By Commas In English', 'JDBC Task Timeout Alert Person', 0); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.`key` = 'linkis.jdbc.task.timeout.alert.user' and label_value = '*-*,jdbc-4'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.jdbc.task.timeout.alert.user') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (`key`, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, `level`, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.level', '超时告警级别:1 critical,2 major,3 minor,4 warning,5 info', 'jdbc任务任务超时告警级别', '3', 'NumInterval', '[1,5]', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Timeout Alert Levels: 1 Critical, 2 Major, 3 Minor, 4 Warning, 5 Info', 'JDBC Task Timeout Alert Level', 0); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.`key` = 'linkis.jdbc.task.timeout.alert.level' and label_value = '*-*,jdbc-4'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.jdbc.task.timeout.alert.level') AND label.label_value = '*-*,jdbc-4'); + +INSERT INTO linkis_ps_configuration_config_key (`key`, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, `level`, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'linkis.jdbc.task.timeout.alert.datasource.type', '多个数据源用英文逗号分隔', '超时告警支持数据源类型', 'starrocks', 'Regex', '^[a-zA-Z0-9,]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alert Configuration', 'Separate Multiple Data Sources With Commas In English', 'Timeout Alert Supports Data Source Types', 0); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.`key` = 'linkis.jdbc.task.timeout.alert.datasource.type' and label_value = '*-*,jdbc-4'); +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'linkis.jdbc.task.timeout.alert.datasource.type') AND label.label_value = '*-*,jdbc-4'); + + +INSERT INTO `linkis_ps_dm_datasource_type_key` (`data_source_type_id`, `key`, `name`, `default_value`, `value_type`, `scope`, `require`, `description`, `value_regex`, `ref_id`, `ref_value`, `data_source`, `update_time`, `create_time`, `name_en`, `description_en`) VALUES ((select id from `linkis_ps_dm_datasource_type` where `name` = 'starrocks'), 'kill_task_time', '超时kill任务时间', NULL, 'TEXT', NULL, 0, '配置任务超时时间，满足配置执行kill，单位：分钟', '^[1-9]\\d*$', NULL, NULL, NULL, now(), now(), 'Timeout Kill Task Time', 'Configure Task Timeout To Meet The Requirement Of Executing The Kill Action'); + +UPDATE linkis_ps_error_code set error_desc ="任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台调整内存后使用",error_regex="failed because the (hive|python|shell|jdbc|io_file|io_hdfs|fps|pipeline|presto|nebula|flink|appconn|sqoop|datax|openlookeng|trino|elasticsearch|seatunnel|hbase|jobserver) engine quitted unexpectedly" WHERE error_code = "13004"; +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('13014','任务运行内存超过设置内存限制，导致引擎意外退出，请在管理台增加executor内存或在提交任务时通过spark.executor.memory或spark.executor.memoryOverhead调整内存','failed because the spark engine quitted unexpectedly',0); +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('43050','特殊UDF不支持在非sql脚本中使用','Not support spacial udf in non-SQL script',0); + +INSERT INTO linkis_ps_configuration_config_key (`key`, description, name, default_value, validate_type, validate_range, engine_conn_type, is_hidden, is_advanced, `level`, treeName, boundary_type, en_treeName, en_description, en_name, template_required) VALUES( 'wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'jdbc引擎初始化内存大小', '1g', 'Regex', '^([1-9]|10)(G|g)$', 'jdbc', 0, 0, 1, '用户配置', 0, 'Value range: 1-10, Unit: G', 'JDBC Engine Initialization Memory Size', 'User Configuration', 0); +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) (select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and config.`key` = 'wds.linkis.engineconn.java.driver.memory' and label_value = '*-*,jdbc-4'); +INSERT INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = (select id FROM linkis_ps_configuration_config_key where `key` = 'wds.linkis.engineconn.java.driver.memory'and engine_conn_type = 'jdbc') AND label.label_value = '*-*,jdbc-4'); + + +update linkis_ps_configuration_config_key set description ="取值范围：1-8000，单位：个",validate_range ="^(?:[1-9]\\d{0,2}|[1-7]\\d{3}|8000)$",en_description ="Value Range: 1-8000, Unit: Piece" where `key` = 'wds.linkis.rm.yarnqueue.cores.max'; + +update linkis_ps_configuration_config_key set description ="取值范围：1-20000，单位：G",validate_range ="^(?:[1-9]\\d{0,3}|[1]\\d{4}|20000)(G|g)$",en_description ="Value Range: 1-20000, Unit: G" where `key` = 'wds.linkis.rm.yarnqueue.memory.max'; + +update linkis_ps_configuration_config_key set description ="范围：1-100，单位：个",validate_range ="[1,100]",en_description ="Range: 1-100, unit: piece" where `key` = 'wds.linkis.rm.instance' and engine_conn_type ="spark"; + +update linkis_ps_configuration_key_limit_for_user set max_value ='8000' where key_id = (SELECT id FROM linkis_ps_configuration_config_key where `key` = 'wds.linkis.rm.yarnqueue.cores.max'); + +update linkis_ps_configuration_key_limit_for_user set max_value ='20000g' where key_id = (SELECT id FROM linkis_ps_configuration_config_key where `key` = 'wds.linkis.rm.yarnqueue.memory.max'); + +update linkis_ps_configuration_key_limit_for_user set max_value ='100' where key_id = (SELECT id FROM linkis_ps_configuration_config_key where `key` = 'wds.linkis.rm.instance' and engine_conn_type ='spark'); + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('21004','Hive Metastore存在问题，生产请联系生产服务助手进行处理，测试请联系Hive开发','Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient',0); + + +update linkis_ps_configuration_config_key set validate_range ="^(?!root\\.).*",validate_type ="Regex" where `key`="wds.linkis.rm.yarnqueue"; + + +insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',CONCAT('*-*,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); + +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'spark' and label.label_value = CONCAT('*-*,',"spark-3.4.4")); + + +insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',CONCAT('*-IDE,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); +insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',CONCAT('*-Visualis,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); +insert into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',CONCAT('*-nodeexecution,',"spark-3.4.4"), 'OPTIONAL', 2, now(), now()); + + +insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES ((select id from linkis_cg_manager_label where `label_value` = CONCAT('*-IDE,',"spark-3.4.4")), 2); +insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES ((select id from linkis_cg_manager_label where `label_value` = CONCAT('*-Visualis,',"spark-3.4.4")), 2); +insert into linkis_ps_configuration_category (`label_id`, `level`) VALUES ((select id from linkis_cg_manager_label where `label_value` = CONCAT('*-nodeexecution,',"spark-3.4.4")), 2); + + +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = CONCAT('*-*,',"spark-3.4.4")); + + +INSERT IGNORE INTO linkis_cg_manager_label (label_key, label_value, label_feature, label_value_size, update_time, create_time) +SELECT label_key, REPLACE(label_value, 'spark-2.4.3', 'spark-3.4.4') AS label_value, label_feature, label_value_size, NOW() , NOW() +FROM linkis_cg_manager_label +WHERE label_key = 'combined_userCreator_engineType' + AND label_value LIKE '%-IDE,spark-2.4.3' + AND label_value != "*-IDE,spark-2.4.3"; + +INSERT IGNORE INTO linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id, update_time, create_time) +SELECT config_key.id as config_key_id,"python3" as config_value ,label.id as config_label_id, NOW() , NOW() FROM linkis_cg_manager_label label,linkis_ps_configuration_config_key config_key +WHERE config_key.`key` ="spark.python.version" +AND label.label_key = 'combined_userCreator_engineType' +AND label.label_value LIKE '%-IDE,spark-3.4.4' + + +INSERT INTO linkis_ps_error_code (error_code,error_desc,error_regex,error_type) VALUES ('42003','未知函数%s，请检查代码中引用的函数是否有误','Cannot resolve function `(\\S+)',0); + + +update linkis_ps_configuration_config_key set default_value = "com.mysql.jdbc.Driver" where engine_conn_type = "jdbc" and `key`="wds.linkis.jdbc.driver"; + +insert IGNORE into `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) +select 'combined_userCreator_engineType',REPLACE(label_value, '2.4.3', '3.4.4'), 'OPTIONAL', 2, now(), now() +from linkis_cg_manager_label +where label_value like "%spark-2.4.3%"; + +insert IGNORE into linkis_ps_configuration_category (`label_id`, `level`) +select id as `label_id`, 2 as `level` +from linkis_cg_manager_label +where label_value in (select REPLACE(label_value, '2.4.3', '3.4.4') + from linkis_cg_manager_label + where id in ( + select cate.label_id + from linkis_ps_configuration_category cate, linkis_cg_manager_label label + where cate.label_id = label .id and label.label_value like "%spark-2.4.3%")); + +INSERT INTO linkis_ps_configuration_config_value (config_key_id, config_value, config_label_id, update_time, create_time) +SELECT cofig.config_key_id AS config_key_id, cofig.config_value AS config_value, res.sp3 AS config_label_id, NOW() AS update_time, NOW() AS create_time +FROM + (SELECT * FROM linkis_ps_configuration_config_value + WHERE config_label_id IN + (SELECT id FROM linkis_cg_manager_label WHERE label_value LIKE "%spark-2.4.3%") + AND config_key_id IN + (SELECT id FROM linkis_ps_configuration_config_key WHERE engine_conn_type = "spark")) cofig, + (SELECT * FROM + (SELECT id AS sp3, label_value AS lp3 FROM linkis_cg_manager_label WHERE label_value LIKE "%spark-3.4.4%") a, + (SELECT id AS sp2, label_value AS lp2 FROM linkis_cg_manager_label WHERE label_value LIKE "%spark-2.4.3%") b + WHERE SUBSTRING_INDEX(a.lp3, ',', 1) = SUBSTRING_INDEX(b.lp2, ',', 1)) res +WHERE cofig.config_label_id = res.sp2 +ON DUPLICATE KEY UPDATE + config_value = VALUES(config_value), + update_time = NOW(); + + +UPDATE linkis_ps_configuration_config_value +SET config_value = "python3" +WHERE config_label_id IN ( + SELECT id + FROM linkis_cg_manager_label + WHERE label_value LIKE "%spark-3.4.4%" + ) +AND config_key_id = (select id from linkis_ps_configuration_config_key where `key`= "spark.python.version"); + + +-- JDBC 配置项 +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.version', '取值范围：jdbc3,jdbc4', 'jdbc版本', 'jdbc4', 'OFT', '["jdbc3","jdbc4"]', 'jdbc', 0, 0, 1, '数据源配置', 0, 'DataSource Configuration', 'Value range: jdbc3, jdbc4', 'jdbc version', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.connect.max', '范围：1-20，单位：个', 'jdbc引擎最大连接数', '10', 'NumInterval', '[1,20]', 'jdbc', 0, 0, 1, '数据源配置', 3, 'DataSource Configuration', 'Range: 1-20, unit: piece', 'Maximum connections of jdbc engine', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.username', 'username', '数据库连接用户名', '', 'None', '', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'username', 'Database connection user name', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.password', 'password', '数据库连接密码', '', 'None', '', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'password', 'Database connection password', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.jdbc.driver', '例如:com.mysql.jdbc.Driver', 'jdbc连接驱动', NULL, 'None', NULL, 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'For Example: com.mysql.jdbc.Driver', 'JDBC Connection Driver', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.engineconn.java.driver.memory', '取值范围：1-10，单位：G', 'jdbc引擎初始化内存大小', '1g', 'Regex', '^([1-9]|10)(G|g)$', 'jdbc', 0, 0, 1, '用户配置', 0, 'Value range: 1-10, Unit: G', 'JDBC Engine Initialization Memory Size', 'User Configuration', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.time', '单位：分钟', 'jdbc任务超时告警时间', '', 'Regex', '^[1-9]\\d*$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Unit: Minutes', 'JDBC Task Timeout Alarm Time', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.user', '多人用英文逗号分隔', 'jdbc任务超时告警人', '', 'Regex', '^[a-zA-Z0-9,_-]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Multiple People Separated By Commas In English', 'JDBC Task Timeout Alert Person', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.level', '超时告警级别:1 critical,2 major,3 minor,4 warning,5 info', 'jdbc任务超时告警级别', '3', 'NumInterval', '[1,5]', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Timeout Alarm Levels: 1 Critical, 2 Major, 3 Minor, 4 Warning, 5 Info', 'JDBC Task Timeout Alarm Level', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkis.jdbc.task.timeout.alert.datasource.type', '多个数据源用英文逗号分隔', '超时告警支持数据源类型', 'starrocks', 'Regex', '^[a-zA-Z0-9,]+$', 'jdbc', 0, 0, 1, '超时告警配置', 0, 'Timeout Alarm Configuration', 'Separate multiple data sources with commas in English', 'Timeout alarm supports data source types', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('linkisJDBCPoolAbandonedTimeout', '范围：1-21600，单位：秒', '数据源链接超时自动关闭时间', '300', 'NumInterval', '[1,21600]', 'jdbc', 0, 0, 1, '数据源配置', 0, 'Data Source Configuration', 'Range: 1-21600, Unit: seconds', 'Data Source Auto Close Link Time', 0); +INSERT IGNORE INTO `linkis_ps_configuration_config_key` (`key`, `description`, `name`, `default_value`, `validate_type`, `validate_range`, `engine_conn_type`, `is_hidden`, `is_advanced`, `level`, `treeName`, `boundary_type`, `en_treeName`, `en_description`, `en_name`, `template_required`) VALUES('wds.linkis.engineconn.max.free.time', '取值范围：3m,15m,30m,1h,2h', '引擎空闲退出时间', '15m', 'OFT', '["1h","2h","30m","15m","3m"]', 'jdbc', 0, 0, 1, '用户配置', 0, 'User Configuration', 'Value range: 3m, 15m, 30m, 1h, 2h', 'Engine unlock exit time', 0); + +-- JDBC 标签和关系配置 +INSERT IGNORE INTO `linkis_cg_manager_label` (`label_key`, `label_value`, `label_feature`, `label_value_size`, `update_time`, `create_time`) VALUES ('combined_userCreator_engineType',CONCAT('*-*,',"jdbc-4"), 'OPTIONAL', 2, now(), now()); + +INSERT IGNORE INTO `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) +(select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config +INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'jdbc' and label_value = CONCAT('*-*,',"jdbc-4")); + +INSERT IGNORE INTO linkis_ps_configuration_category (`label_id`, `level`) VALUES ((select id from linkis_cg_manager_label where `label_value` = CONCAT('*-IDE,',"jdbc-4")), 2); + +-- jdbc default configuration +INSERT IGNORE INTO `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) +(select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation +INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id AND label.label_value = CONCAT('*-*,',"jdbc-4")); + +INSERT INTO `linkis_ps_configuration_config_key` (`key`,description,name,default_value,validate_type,validate_range,engine_conn_type,is_hidden,is_advanced,`level`,treeName,boundary_type,en_treeName,en_description,en_name,template_required) VALUES +('wds.linkis.engine.running.job.max', '引擎运行最大任务数', '引擎运行最大任务数', '30', 'None', NULL, 'shell', 0, 0, 1, 'shell引擎设置', 0, 'Maximum Number Of Tasks The Engine Can Run', 'Maximum Number For Engine', 'shell Engine Settings', 0); + +insert into `linkis_ps_configuration_key_engine_relation` (`config_key_id`, `engine_type_label_id`) ( select config.id as `config_key_id`, label.id AS `engine_type_label_id` FROM linkis_ps_configuration_config_key config INNER JOIN linkis_cg_manager_label label ON config.engine_conn_type = 'shell' and config.`key` = 'wds.linkis.engine.running.job.max' and label_value = '*-*,shell-1'); + + +insert into `linkis_ps_configuration_config_value` (`config_key_id`, `config_value`, `config_label_id`) ( select `relation`.`config_key_id` AS `config_key_id`, '' AS `config_value`, `relation`.`engine_type_label_id` AS `config_label_id` FROM linkis_ps_configuration_key_engine_relation relation INNER JOIN linkis_cg_manager_label label ON relation.engine_type_label_id = label.id and relation.config_key_id = ( select id FROM linkis_ps_configuration_config_key where `key` = 'wds.linkis.engine.running.job.max') AND label.label_value = '*-*,shell-1'); + +-- 更新 Shell 引擎运行最大任务数默认值从 30 改为 60 +UPDATE `linkis_ps_configuration_config_key` SET default_value = '60' WHERE `key` = 'wds.linkis.engine.running.job.max' AND engine_conn_type = 'shell'; + +-- 更新错误码正则表达式 +UPDATE linkis_ps_error_code SET error_regex = "The ecm of labels" WHERE error_code = "01001"; + diff --git a/linkis-dist/package/sbin/kill-process-by-pid.sh b/linkis-dist/package/sbin/kill-process-by-pid.sh index 0ab28032b0e..6cf413f8d33 100644 --- a/linkis-dist/package/sbin/kill-process-by-pid.sh +++ b/linkis-dist/package/sbin/kill-process-by-pid.sh @@ -1,4 +1,4 @@ -#!/bin/bash +#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with @@ -22,7 +22,7 @@ if [ "$LINKIS_HOME" = "" ]; then export LINKIS_HOME=$INSTALL_HOME fi if [ "$LINKIS_LOG_DIR" = "" ]; then - export LINKIS_LOG_DIR="$LINKIS_HOME/logs" + export LINKIS_LOG_DIR="/data/logs/bdpe-ujes" fi ecmPid=`cat $LINKIS_HOME/pid/linkis_cg-engineconnmanager.pid` month=`date '+%Y-%m'` @@ -46,7 +46,8 @@ function kill_proc_by_pid() { cnt=0 MAX_RETRY=6 if [ "${killProcessPID}" != "${ecmPid}" ]; then - pkill -TERM -P ${killProcessPID} + #pkill -TERM -P ${killProcessPID} + kill -- -$(ps -o pgid "$killProcessPID" | grep [0-9] | tr -d ' ') while ps -p ${killProcessPID} > /dev/null && [ $cnt -lt $MAX_RETRY ] do kill -15 ${killProcessPID} diff --git a/linkis-dist/pom.xml b/linkis-dist/pom.xml index a897f1cd570..dbc115f829c 100644 --- a/linkis-dist/pom.xml +++ b/linkis-dist/pom.xml @@ -57,6 +57,10 @@ com.google.code.findbugs jsr305 + + org.codehaus.jackson + jackson-core-asl + diff --git a/linkis-dist/src/main/assembly/distribution.xml b/linkis-dist/src/main/assembly/distribution.xml index a506c99b180..c1f876896a0 100644 --- a/linkis-dist/src/main/assembly/distribution.xml +++ b/linkis-dist/src/main/assembly/distribution.xml @@ -260,6 +260,20 @@ **/* + + + + + + ../linkis-extensions/linkis-et-monitor/target/out/lib + + + linkis-package/lib/linkis-extensions/linkis-et-monitor + + + **/* + + diff --git a/linkis-engineconn-plugins/doris/pom.xml b/linkis-engineconn-plugins/doris/pom.xml index c108c9c3ea1..c1cab4210be 100644 --- a/linkis-engineconn-plugins/doris/pom.xml +++ b/linkis-engineconn-plugins/doris/pom.xml @@ -55,6 +55,7 @@ org.apache.linkis + linkis-common ${project.version} provided diff --git a/linkis-engineconn-plugins/flink/flink-core/pom.xml b/linkis-engineconn-plugins/flink/flink-core/pom.xml index 3776825ea86..1eb3322db56 100644 --- a/linkis-engineconn-plugins/flink/flink-core/pom.xml +++ b/linkis-engineconn-plugins/flink/flink-core/pom.xml @@ -210,9 +210,14 @@ hadoop-common + org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs-client + + + org.apache.hadoop + hadoop-hdfs @@ -272,6 +277,11 @@ 2.0.0 + + org.yaml + snakeyaml + + diff --git a/linkis-engineconn-plugins/hbase/hbase-core/pom.xml b/linkis-engineconn-plugins/hbase/hbase-core/pom.xml index 28e79f0a796..555050b8357 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/pom.xml +++ b/linkis-engineconn-plugins/hbase/hbase-core/pom.xml @@ -225,6 +225,26 @@ + + + org.apache.hbase + hbase-server + ${hbase.version} + provided + + + org.apache.hbase + hbase-client + ${hbase.version} + provided + + + org.apache.hbase + hbase-common + ${hbase.version} + provided + + org.apache.linkis linkis-engineconn-plugin-core diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java b/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java index fb537479801..2d86b8b44df 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/HBaseConnectionManager.java @@ -17,12 +17,15 @@ package org.apache.linkis.manager.engineplugin.hbase; +import org.apache.linkis.common.utils.Utils; +import org.apache.linkis.hadoop.common.conf.HadoopConf; import org.apache.linkis.manager.engineplugin.hbase.errorcode.HBaseErrorCodeSummary; import org.apache.linkis.manager.engineplugin.hbase.exception.HBaseParamsIllegalException; import org.apache.linkis.manager.engineplugin.hbase.exception.JobExecutorException; import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.HConstants; import org.apache.hadoop.hbase.client.Connection; @@ -73,16 +76,16 @@ public class HBaseConnectionManager { private static final AtomicBoolean kerberosEnvInit = new AtomicBoolean(false); private static final int KERBEROS_RE_LOGIN_MAX_RETRY = 5; private static final long KERBEROS_RE_LOGIN_INTERVAL = 30 * 60 * 1000L; - private static volatile HBaseConnectionManager instance = null; + private static volatile HBaseConnectionManager instance = null; // NOSONAR private HBaseConnectionManager() { connectionMap = new ConcurrentHashMap<>(); } public static HBaseConnectionManager getInstance() { - if (instance == null) { - synchronized (HBaseConnectionManager.class) { - if (instance == null) { + if (instance == null) { // NOSONAR + synchronized (HBaseConnectionManager.class) { // NOSONAR + if (instance == null) { // NOSONAR instance = new HBaseConnectionManager(); } } @@ -178,6 +181,7 @@ private void doKerberosLogin(Configuration configuration) { doKerberosReLogin(); } catch (IOException e) { kerberosEnvInit.set(false); + LOG.info("Failed to doKerberosReLogin, principal {}, keytab {}", principal, keytab, e); throw new JobExecutorException( HBaseErrorCodeSummary.KERBEROS_AUTH_FAILED.getErrorCode(), HBaseErrorCodeSummary.KERBEROS_AUTH_FAILED.getErrorDesc()); @@ -256,11 +260,14 @@ private Configuration buildConfiguration(Map prop) { String dfsRootDir = HBasePropertiesParser.getString(prop, HBASE_DFS_ROOT_DIR, DEFAULT_HBASE_DFS_ROOT_DIR); configuration.set(HConstants.HBASE_DIR, dfsRootDir); - String securityAuth = HBasePropertiesParser.getString(prop, HBASE_SECURITY_AUTH, SIMPLE); + String securityAuth = HBasePropertiesParser.getString(prop, HBASE_SECURITY_AUTH, KERBEROS); configuration.set(HBASE_AUTH, securityAuth); if (isKerberosAuthType(configuration)) { configuration.set(HADOOP_SECURITY_AUTH, KERBEROS); - String kerberosPrincipal = HBasePropertiesParser.getString(prop, KERBEROS_PRINCIPAL, ""); + configuration.setBoolean(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, true); + String userName = Utils.getJvmUser(); + String kerberosPrincipal = + HBasePropertiesParser.getString(prop, KERBEROS_PRINCIPAL, userName); if (StringUtils.isBlank(kerberosPrincipal)) { throw new HBaseParamsIllegalException( HBaseErrorCodeSummary.KERBEROS_PRINCIPAL_NOT_NULL.getErrorCode(), @@ -269,15 +276,19 @@ private Configuration buildConfiguration(Map prop) { configuration.set(KERBEROS_PRINCIPAL, kerberosPrincipal); String keytabFile = HBasePropertiesParser.getString(prop, KERBEROS_KEYTAB_FILE, ""); if (StringUtils.isBlank(keytabFile)) { - throw new HBaseParamsIllegalException( - HBaseErrorCodeSummary.KERBEROS_KEYTAB_NOT_NULL.getErrorCode(), - HBaseErrorCodeSummary.KERBEROS_KEYTAB_NOT_NULL.getErrorDesc()); + keytabFile = HadoopConf.KEYTAB_FILE() + userName + ".keytab"; } configuration.set(KERBEROS_KEYTAB_FILE, keytabFile); String proxyUser = HBasePropertiesParser.getString(prop, KERBEROS_PROXY_USER, ""); - configuration.set(KERBEROS_PROXY_USER, proxyUser); + if (StringUtils.isNotBlank(proxyUser)) { + configuration.set(KERBEROS_PROXY_USER, proxyUser); + } String regionServerPrincipal = - HBasePropertiesParser.getString(prop, HBASE_REGION_SERVER_KERBEROS_PRINCIPAL, ""); + HBasePropertiesParser.getString( + prop, + HBASE_REGION_SERVER_KERBEROS_PRINCIPAL, + org.apache.linkis.manager.engineplugin.hbase.conf.HBaseConfiguration + .HBASE_DEFAULT_KERBEOS_PRINCIPAL()); if (StringUtils.isBlank(regionServerPrincipal)) { throw new HBaseParamsIllegalException( HBaseErrorCodeSummary.REGION_SERVER_KERBEROS_PRINCIPAL_NOT_NULL.getErrorCode(), @@ -285,7 +296,11 @@ private Configuration buildConfiguration(Map prop) { } configuration.set(REGION_SERVER_KERBEROS_PRINCIPAL, regionServerPrincipal); String masterPrincipal = - HBasePropertiesParser.getString(prop, HBASE_MASTER_KERBEROS_PRINCIPAL, ""); + HBasePropertiesParser.getString( + prop, + HBASE_MASTER_KERBEROS_PRINCIPAL, + org.apache.linkis.manager.engineplugin.hbase.conf.HBaseConfiguration + .HBASE_DEFAULT_KERBEOS_PRINCIPAL()); if (StringUtils.isBlank(masterPrincipal)) { throw new HBaseParamsIllegalException( HBaseErrorCodeSummary.MASTER_KERBEROS_PRINCIPAL_NOT_NULL.getErrorCode(), @@ -295,6 +310,14 @@ private Configuration buildConfiguration(Map prop) { String krb5Conf = HBasePropertiesParser.getString(prop, KRB5_CONF_PATH, DEFAULT_KRB5_CONF_PATH); System.setProperty(KRB5_CONF_PATH, krb5Conf); + LOG.info( + "Kerberos info zk: {}, zNodeParent: {}, KERBEROS_PRINCIPAL {}, KERBEROS_KEYTAB_FILE {}, HBASE_REGION_SERVER_KERBEROS_PRINCIPAL {}, HBASE_MASTER_KERBEROS_PRINCIPAL {}", + zkQuorum, + zNodeParent, + kerberosPrincipal, + keytabFile, + regionServerPrincipal, + masterPrincipal); } return configuration; } diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellCommands.java b/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellCommands.java index 6b095d05db9..5c278ba7523 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellCommands.java +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellCommands.java @@ -35,14 +35,14 @@ public class HBaseShellCommands { private static final Logger LOG = LoggerFactory.getLogger(HBaseShellCommands.class); private static final String COMMANDS_PATH = "hbase-ruby/shell/commands/"; - private static volatile Set commandsSet; + private static volatile Set commandsSet; // NOSONAR private HBaseShellCommands() {} public static Set getAllCommands() throws IOException { - if (commandsSet == null) { - synchronized (HBaseShellCommands.class) { - if (commandsSet == null) { + if (commandsSet == null) { // NOSONAR + synchronized (HBaseShellCommands.class) { // NOSONAR + if (commandsSet == null) { // NOSONAR Set sortedSet = new TreeSet<>(); URL commandFilesUrl = HBaseShellCommands.class.getClassLoader().getResource(COMMANDS_PATH); diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellSessionManager.java b/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellSessionManager.java index a599fe42a25..1406707f36c 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellSessionManager.java +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/shell/HBaseShellSessionManager.java @@ -29,7 +29,7 @@ public class HBaseShellSessionManager { private static final Logger LOG = LoggerFactory.getLogger(HBaseShellSessionManager.class); private final ConcurrentHashMap shellSessionMap; - private static volatile HBaseShellSessionManager instance = null; + private static volatile HBaseShellSessionManager instance = null; // NOSONAR private final ReentrantLock lock = new ReentrantLock(); @@ -38,9 +38,9 @@ private HBaseShellSessionManager() { } public static HBaseShellSessionManager getInstance() { - if (instance == null) { - synchronized (HBaseShellSessionManager.class) { - if (instance == null) { + if (instance == null) { // NOSONAR + synchronized (HBaseShellSessionManager.class) { // NOSONAR + if (instance == null) { // NOSONAR instance = new HBaseShellSessionManager(); } } diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/resources/log4j2.xml b/linkis-engineconn-plugins/hbase/hbase-core/src/main/resources/log4j2.xml index 2e36367c4a7..9416ae8608f 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/resources/log4j2.xml +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/resources/log4j2.xml @@ -87,7 +87,9 @@ - + + + diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/conf/HBaseConfiguration.scala b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/conf/HBaseConfiguration.scala index 5569b981f09..03799055e4c 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/conf/HBaseConfiguration.scala +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/conf/HBaseConfiguration.scala @@ -21,4 +21,8 @@ import org.apache.linkis.common.conf.CommonVars object HBaseConfiguration { val HBASE_CONCURRENT_LIMIT = CommonVars[Int]("linkis.engineconn.hbase.concurrent.limit", 100) + + val HBASE_DEFAULT_KERBEOS_PRINCIPAL = + CommonVars[String]("linkis.engineconn.hbase.default.kerberos.principal", "").getValue + } diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/exception/ExecutorInitException.java b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/ExecutorInitException.java similarity index 77% rename from linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/exception/ExecutorInitException.java rename to linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/ExecutorInitException.java index d0e739d9c48..5a805c848cf 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/java/org/apache/linkis/manager/engineplugin/hbase/exception/ExecutorInitException.java +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/ExecutorInitException.java @@ -17,10 +17,16 @@ package org.apache.linkis.manager.engineplugin.hbase.exception; -import org.apache.linkis.common.exception.ErrorException; +import org.apache.linkis.common.exception.ExceptionLevel; +import org.apache.linkis.common.exception.LinkisRuntimeException; -public class ExecutorInitException extends ErrorException { +public class ExecutorInitException extends LinkisRuntimeException { public ExecutorInitException(int errCode, String desc) { super(errCode, desc); } + + @Override + public ExceptionLevel getLevel() { + return ExceptionLevel.ERROR; + } } diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/HBaseParamsIllegalException.scala b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/HBaseParamsIllegalException.scala index 63f7e14ecc1..38b8a91dc01 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/HBaseParamsIllegalException.scala +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/HBaseParamsIllegalException.scala @@ -17,7 +17,9 @@ package org.apache.linkis.manager.engineplugin.hbase.exception -import org.apache.linkis.common.exception.ErrorException +import org.apache.linkis.common.exception.{ExceptionLevel, LinkisRuntimeException} case class HBaseParamsIllegalException(errorCode: Int, errorMsg: String) - extends ErrorException(errorCode, errorMsg) + extends LinkisRuntimeException(errorCode, errorMsg) { + override def getLevel: ExceptionLevel = ExceptionLevel.ERROR +} diff --git a/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/JobExecutorException.scala b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/JobExecutorException.scala index 882dd415a29..f3c695a21a1 100644 --- a/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/JobExecutorException.scala +++ b/linkis-engineconn-plugins/hbase/hbase-core/src/main/scala/org/apache/linkis/manager/engineplugin/hbase/exception/JobExecutorException.scala @@ -17,7 +17,9 @@ package org.apache.linkis.manager.engineplugin.hbase.exception -import org.apache.linkis.common.exception.ErrorException +import org.apache.linkis.common.exception.{ExceptionLevel, LinkisRuntimeException} case class JobExecutorException(errorCode: Int, errorMsg: String) - extends ErrorException(errorCode, errorMsg) + extends LinkisRuntimeException(errorCode, errorMsg) { + override def getLevel: ExceptionLevel = ExceptionLevel.ERROR +} diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/pom.xml b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/pom.xml new file mode 100644 index 00000000000..bb70fceb4b7 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/pom.xml @@ -0,0 +1,99 @@ + + + + 4.0.0 + + org.apache.linkis + linkis + ${revision} + ../../../pom.xml + + + hbase-shims-1.2.1 + + + 1.2.1 + + + + + org.apache.linkis + linkis-common + ${project.version} + provided + + + + org.apache.hbase + hbase-server + ${hbase.version} + + + org.apache.hadoop + hadoop-common + + + org.apache.hadoop + hadoop-auth + + + org.apache.hadoop + hadoop-mapreduce-client-core + + + log4j + log4j + + + org.mortbay.jetty + jetty + + + org.jruby.jcodings + jcodings + + + org.jruby.joni + joni + + + + + + org.apache.hbase + hbase-testing-util + ${hbase.version} + test + + + org.slf4j + slf4j-api + + + + + + + + + net.alchim31.maven + scala-maven-plugin + + + + diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase.rb new file mode 100644 index 00000000000..dcf77318189 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase.rb @@ -0,0 +1,107 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# HBase ruby classes. +# Has wrapper classes for org.apache.hadoop.hbase.client.Admin +# and for org.apache.hadoop.hbase.client.Table. Classes take +# Formatters on construction and outputs any results using +# Formatter methods. These classes are only really for use by +# the hirb.rb HBase Shell script; they don't make much sense elsewhere. +# For example, the exists method on Admin class prints to the formatter +# whether the table exists and returns nil regardless. +include Java + +java_import('java.lang.Integer') {|package,name| "J#{name}" } +java_import('java.lang.Long') {|package,name| "J#{name}" } +java_import('java.lang.Boolean') {|package,name| "J#{name}" } + +module HBaseConstants + COLUMN = "COLUMN" + COLUMNS = "COLUMNS" + TIMESTAMP = "TIMESTAMP" + TIMERANGE = "TIMERANGE" + NAME = org.apache.hadoop.hbase.HConstants::NAME + VERSIONS = org.apache.hadoop.hbase.HConstants::VERSIONS + IN_MEMORY = org.apache.hadoop.hbase.HConstants::IN_MEMORY + METADATA = org.apache.hadoop.hbase.HConstants::METADATA + STOPROW = "STOPROW" + STARTROW = "STARTROW" + ROWPREFIXFILTER = "ROWPREFIXFILTER" + ENDROW = STOPROW + RAW = "RAW" + LIMIT = "LIMIT" + METHOD = "METHOD" + MAXLENGTH = "MAXLENGTH" + CACHE_BLOCKS = "CACHE_BLOCKS" + ALL_METRICS = "ALL_METRICS" + METRICS = "METRICS" + REVERSED = "REVERSED" + REPLICATION_SCOPE = "REPLICATION_SCOPE" + INTERVAL = 'INTERVAL' + CACHE = 'CACHE' + FILTER = 'FILTER' + SPLITS = 'SPLITS' + SPLITS_FILE = 'SPLITS_FILE' + SPLITALGO = 'SPLITALGO' + NUMREGIONS = 'NUMREGIONS' + REGION_REPLICATION = 'REGION_REPLICATION' + REGION_REPLICA_ID = 'REGION_REPLICA_ID' + CONFIGURATION = org.apache.hadoop.hbase.HConstants::CONFIGURATION + ATTRIBUTES="ATTRIBUTES" + VISIBILITY="VISIBILITY" + AUTHORIZATIONS = "AUTHORIZATIONS" + SKIP_FLUSH = 'SKIP_FLUSH' + CONSISTENCY = "CONSISTENCY" + ENDPOINT_CLASSNAME = 'ENDPOINT_CLASSNAME' + CLUSTER_KEY = 'CLUSTER_KEY' + TABLE_CFS = 'TABLE_CFS' + CONFIG = 'CONFIG' + DATA = 'DATA' + USER = 'USER' + TABLE = 'TABLE' + NAMESPACE = 'NAMESPACE' + TYPE = 'TYPE' + NONE = 'NONE' + VALUE = 'VALUE' + + # Load constants from hbase java API + def self.promote_constants(constants) + # The constants to import are all in uppercase + constants.each do |c| + next if c =~ /DEFAULT_.*/ || c != c.upcase + next if eval("defined?(#{c})") + eval("#{c} = '#{c}'") + end + end + + promote_constants(org.apache.hadoop.hbase.HColumnDescriptor.constants) + promote_constants(org.apache.hadoop.hbase.HTableDescriptor.constants) +end + +# Include classes definition +require 'hbase/hbase' +require 'hbase/admin' +require 'hbase/table' +require 'hbase/quotas' +require 'hbase/replication_admin' +require 'hbase/security' +require 'hbase/visibility_labels' + + +include HBaseQuotasConstants \ No newline at end of file diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/admin.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/admin.rb new file mode 100644 index 00000000000..3f31eebd4f6 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/admin.rb @@ -0,0 +1,1069 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java +java_import java.util.Arrays +java_import org.apache.hadoop.hbase.util.Pair +java_import org.apache.hadoop.hbase.util.RegionSplitter +java_import org.apache.hadoop.hbase.util.Bytes +java_import org.apache.hadoop.hbase.ServerName +java_import org.apache.hadoop.hbase.TableName +java_import org.apache.hadoop.hbase.protobuf.generated.HBaseProtos::SnapshotDescription + +# Wrapper for org.apache.hadoop.hbase.client.HBaseAdmin + +module Hbase + class Admin + include HBaseConstants + + def initialize(admin, formatter) + @admin = admin + @connection = @admin.getConnection() + @formatter = formatter + end + + def close + @admin.close + end + + #---------------------------------------------------------------------------------------------- + # Returns a list of tables in hbase + def list(regex = ".*") + @admin.listTableNames(regex).map { |t| t.getNameAsString } + end + + #---------------------------------------------------------------------------------------------- + # Requests a table or region flush + def flush(table_or_region_name) + @admin.flush(table_or_region_name) + end + + #---------------------------------------------------------------------------------------------- + # Requests a table or region or column family compaction + def compact(table_or_region_name, family = nil) + if family == nil + @admin.compact(table_or_region_name) + else + # We are compacting a column family within a region. + @admin.compact(table_or_region_name, family) + end + end + + # Requests to compact all regions on the regionserver + def compact_regionserver(servername, major = false) + @admin.compactRegionServer(ServerName.valueOf(servername), major) + end + + #---------------------------------------------------------------------------------------------- + # Requests a table or region or column family major compaction + def major_compact(table_or_region_name, family = nil) + if family == nil + @admin.majorCompact(table_or_region_name) + else + # We are major compacting a column family within a region or table. + @admin.majorCompact(table_or_region_name, family) + end + end + + #---------------------------------------------------------------------------------------------- + # Requests a regionserver's WAL roll + def wal_roll(server_name) + @admin.rollWALWriter(ServerName.valueOf(server_name)) + end + # TODO remove older hlog_roll version + alias :hlog_roll :wal_roll + + #---------------------------------------------------------------------------------------------- + # Requests a table or region split + def split(table_or_region_name, split_point) + if split_point == nil + @admin.split(table_or_region_name) + else + @admin.split(table_or_region_name, split_point) + end + end + + def locate_region(table_name, row_key) + locator = @connection.getRegionLocator(TableName.valueOf(table_name)) + begin + return locator.getRegionLocation(Bytes.toBytesBinary(row_key)) + ensure + locator.close() + end + end + + #---------------------------------------------------------------------------------------------- + # Requests a cluster balance + # Returns true if balancer ran + def balancer() + @admin.balancer() + end + + #---------------------------------------------------------------------------------------------- + # Enable/disable balancer + # Returns previous balancer switch setting. + def balance_switch(enableDisable) + @admin.setBalancerRunning( + java.lang.Boolean::valueOf(enableDisable), java.lang.Boolean::valueOf(false)) + end + + #---------------------------------------------------------------------------------------------- + # Query the current state of the LoadBalancer. + # Returns the balancer's state (true is enabled). + def balancer_enabled?() + @admin.isBalancerEnabled() + end + + #---------------------------------------------------------------------------------------------- + # Requests region normalization for all configured tables in the cluster + # Returns true if normalizer ran successfully + def normalize() + @admin.normalize() + end + + #---------------------------------------------------------------------------------------------- + # Enable/disable region normalizer + # Returns previous normalizer switch setting. + def normalizer_switch(enableDisable) + @admin.setNormalizerRunning(java.lang.Boolean::valueOf(enableDisable)) + end + + #---------------------------------------------------------------------------------------------- + # Query the current state of region normalizer. + # Returns the state of region normalizer (true is enabled). + def normalizer_enabled?() + @admin.isNormalizerEnabled() + end + + #---------------------------------------------------------------------------------------------- + # Request a scan of the catalog table (for garbage collection) + # Returns an int signifying the number of entries cleaned + def catalogjanitor_run() + @admin.runCatalogScan() + end + + #---------------------------------------------------------------------------------------------- + # Enable/disable the catalog janitor + # Returns previous catalog janitor switch setting. + def catalogjanitor_switch(enableDisable) + @admin.enableCatalogJanitor(java.lang.Boolean::valueOf(enableDisable)) + end + + #---------------------------------------------------------------------------------------------- + # Query on the catalog janitor state (enabled/disabled?) + # Returns catalog janitor state (true signifies enabled). + def catalogjanitor_enabled() + @admin.isCatalogJanitorEnabled() + end + + #---------------------------------------------------------------------------------------------- + # Enables a table + def enable(table_name) + tableExists(table_name) + return if enabled?(table_name) + @admin.enableTable(table_name) + end + + #---------------------------------------------------------------------------------------------- + # Enables all tables matching the given regex + def enable_all(regex) + regex = regex.to_s + @admin.enableTables(regex) + end + + #---------------------------------------------------------------------------------------------- + # Disables a table + def disable(table_name) + tableExists(table_name) + return if disabled?(table_name) + @admin.disableTable(table_name) + end + + #---------------------------------------------------------------------------------------------- + # Disables all tables matching the given regex + def disable_all(regex) + regex = regex.to_s + @admin.disableTables(regex).map { |t| t.getTableName().getNameAsString } + end + + #--------------------------------------------------------------------------------------------- + # Throw exception if table doesn't exist + def tableExists(table_name) + raise ArgumentError, "Table #{table_name} does not exist." unless exists?(table_name) + end + + #---------------------------------------------------------------------------------------------- + # Is table disabled? + def disabled?(table_name) + @admin.isTableDisabled(table_name) + end + + #---------------------------------------------------------------------------------------------- + # Drops a table + def drop(table_name) + tableExists(table_name) + raise ArgumentError, "Table #{table_name} is enabled. Disable it first." if enabled?(table_name) + + @admin.deleteTable(org.apache.hadoop.hbase.TableName.valueOf(table_name)) + end + + #---------------------------------------------------------------------------------------------- + # Drops a table + def drop_all(regex) + regex = regex.to_s + failed = @admin.deleteTables(regex).map { |t| t.getTableName().getNameAsString } + return failed + end + + #---------------------------------------------------------------------------------------------- + # Returns ZooKeeper status dump + def zk_dump + @zk_wrapper = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new( + @admin.getConfiguration(), + "admin", + nil) + zk = @zk_wrapper.getRecoverableZooKeeper().getZooKeeper() + @zk_main = org.apache.zookeeper.ZooKeeperMain.new(zk) + org.apache.hadoop.hbase.zookeeper.ZKUtil::dump(@zk_wrapper) + end + + #---------------------------------------------------------------------------------------------- + # Parse arguments and update HTableDescriptor accordingly + def parse_htd_args(htd, arg) + htd.setNormalizationEnabled(JBoolean.valueOf(arg.delete(NORMALIZATION_ENABLED))) if arg[NORMALIZATION_ENABLED] + end + + #---------------------------------------------------------------------------------------------- + # Creates a table + def create(table_name, *args) + # Fail if table name is not a string + raise(ArgumentError, "Table name must be of type String") unless table_name.kind_of?(String) + + # Flatten params array + args = args.flatten.compact + has_columns = false + + # Start defining the table + htd = org.apache.hadoop.hbase.HTableDescriptor.new(org.apache.hadoop.hbase.TableName.valueOf(table_name)) + splits = nil + # Args are either columns or splits, add them to the table definition + # TODO: add table options support + args.each do |arg| + unless arg.kind_of?(String) || arg.kind_of?(Hash) + raise(ArgumentError, "#{arg.class} of #{arg.inspect} is not of Hash or String type") + end + + # First, handle all the cases where arg is a column family. + if arg.kind_of?(String) or arg.has_key?(NAME) + # If the arg is a string, default action is to add a column to the table. + # If arg has a name, it must also be a column descriptor. + descriptor = hcd(arg, htd); + # Warn if duplicate columns are added + if htd.hasFamily(descriptor.getName) + puts "Family '" + descriptor.getNameAsString() + "' already exists, the old one will be replaced" + htd.modifyFamily(descriptor) + else + htd.addFamily(descriptor) + end + has_columns = true + next + end + if arg.has_key?(REGION_REPLICATION) + region_replication = JInteger.valueOf(arg.delete(REGION_REPLICATION)) + htd.setRegionReplication(region_replication) + end + + # Get rid of the "METHOD", which is deprecated for create. + # We'll do whatever it used to do below if it's table_att. + if (method = arg.delete(METHOD)) + raise(ArgumentError, "table_att is currently the only supported method") unless method == 'table_att' + end + + # The hash is not a column family. Figure out what's in it. + # First, handle splits. + if arg.has_key?(SPLITS_FILE) + splits_file = arg.delete(SPLITS_FILE) + unless File.exist?(splits_file) + raise(ArgumentError, "Splits file #{splits_file} doesn't exist") + end + arg[SPLITS] = [] + File.foreach(splits_file) do |line| + arg[SPLITS].push(line.chomp) + end + htd.setValue(SPLITS_FILE, arg[SPLITS_FILE]) + end + + if arg.has_key?(SPLITS) + splits = Java::byte[][arg[SPLITS].size].new + idx = 0 + arg.delete(SPLITS).each do |split| + splits[idx] = org.apache.hadoop.hbase.util.Bytes.toBytesBinary(split) + idx = idx + 1 + end + elsif arg.has_key?(NUMREGIONS) or arg.has_key?(SPLITALGO) + # deprecated region pre-split API; if one of the above is specified, will be ignored. + raise(ArgumentError, "Number of regions must be specified") unless arg.has_key?(NUMREGIONS) + raise(ArgumentError, "Split algorithm must be specified") unless arg.has_key?(SPLITALGO) + raise(ArgumentError, "Number of regions must be greater than 1") unless arg[NUMREGIONS] > 1 + num_regions = arg.delete(NUMREGIONS) + split_algo = RegionSplitter.newSplitAlgoInstance(@conf, arg.delete(SPLITALGO)) + splits = split_algo.split(JInteger.valueOf(num_regions)) + end + + # Done with splits; apply formerly-table_att parameters. + htd.setOwnerString(arg.delete(OWNER)) if arg[OWNER] + htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if arg[MAX_FILESIZE] + htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] + htd.setCompactionEnabled(JBoolean.valueOf(arg.delete(COMPACTION_ENABLED))) if arg[COMPACTION_ENABLED] + htd.setMemStoreFlushSize(JLong.valueOf(arg.delete(MEMSTORE_FLUSHSIZE))) if arg[MEMSTORE_FLUSHSIZE] + # DEFERRED_LOG_FLUSH is deprecated and was replaced by DURABILITY. To keep backward compatible, it still exists. + # However, it has to be set before DURABILITY so that DURABILITY could overwrite if both args are set + if arg.include?(DEFERRED_LOG_FLUSH) + if arg.delete(DEFERRED_LOG_FLUSH).to_s.upcase == "TRUE" + htd.setDurability(org.apache.hadoop.hbase.client.Durability.valueOf("ASYNC_WAL")) + else + htd.setDurability(org.apache.hadoop.hbase.client.Durability.valueOf("SYNC_WAL")) + end + end + htd.setDurability(org.apache.hadoop.hbase.client.Durability.valueOf(arg.delete(DURABILITY))) if arg[DURABILITY] + htd.setFlushPolicyClassName(arg.delete(FLUSH_POLICY)) if arg[FLUSH_POLICY] + htd.setRegionSplitPolicyClassName(arg.delete(SPLIT_POLICY)) if arg[SPLIT_POLICY] + htd.setRegionMemstoreReplication(JBoolean.valueOf(arg.delete(REGION_MEMSTORE_REPLICATION))) if arg[REGION_MEMSTORE_REPLICATION] + htd.setRegionReplication(JInteger.valueOf(arg.delete(REGION_REPLICATION))) if arg[REGION_REPLICATION] + parse_htd_args(htd, arg) + set_user_metadata(htd, arg.delete(METADATA)) if arg[METADATA] + set_descriptor_config(htd, arg.delete(CONFIGURATION)) if arg[CONFIGURATION] + + arg.each_key do |ignored_key| + puts("An argument ignored (unknown or overridden): %s" % [ ignored_key ]) + end + end + + # Fail if no column families defined + raise(ArgumentError, "Table must have at least one column family") if !has_columns + + if splits.nil? + # Perform the create table call + @admin.createTable(htd) + else + # Perform the create table call + @admin.createTable(htd, splits) + end + end + + #---------------------------------------------------------------------------------------------- + # Closes a region. + # If server name is nil, we presume region_name is full region name (HRegionInfo.getRegionName). + # If server name is not nil, we presume it is the region's encoded name (HRegionInfo.getEncodedName) + def close_region(region_name, server) + if (server == nil || !closeEncodedRegion?(region_name, server)) + @admin.closeRegion(region_name, server) + end + end + + #---------------------------------------------------------------------------------------------- + #---------------------------------------------------------------------------------------------- + # Assign a region + def assign(region_name) + @admin.assign(region_name.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Unassign a region + def unassign(region_name, force) + @admin.unassign(region_name.to_java_bytes, java.lang.Boolean::valueOf(force)) + end + + #---------------------------------------------------------------------------------------------- + # Move a region + def move(encoded_region_name, server = nil) + @admin.move(encoded_region_name.to_java_bytes, server ? server.to_java_bytes: nil) + end + + #---------------------------------------------------------------------------------------------- + # Merge two regions + def merge_region(encoded_region_a_name, encoded_region_b_name, force) + @admin.mergeRegions(encoded_region_a_name.to_java_bytes, encoded_region_b_name.to_java_bytes, java.lang.Boolean::valueOf(force)) + end + + #---------------------------------------------------------------------------------------------- + # Returns table's structure description + def describe(table_name) + @admin.getTableDescriptor(TableName.valueOf(table_name)).to_s + end + + def get_column_families(table_name) + @admin.getTableDescriptor(TableName.valueOf(table_name)).getColumnFamilies() + end + + def get_table_attributes(table_name) + @admin.getTableDescriptor(TableName.valueOf(table_name)).toStringTableAttributes + end + + #---------------------------------------------------------------------------------------------- + # Truncates table (deletes all records by recreating the table) + def truncate(table_name, conf = @conf) + table_description = @admin.getTableDescriptor(TableName.valueOf(table_name)) + raise ArgumentError, "Table #{table_name} is not enabled. Enable it first." unless enabled?(table_name) + yield 'Disabling table...' if block_given? + @admin.disableTable(table_name) + + begin + yield 'Truncating table...' if block_given? + @admin.truncateTable(org.apache.hadoop.hbase.TableName.valueOf(table_name), false) + rescue => e + # Handle the compatibility case, where the truncate method doesn't exists on the Master + raise e unless e.respond_to?(:cause) && e.cause != nil + rootCause = e.cause + if rootCause.kind_of?(org.apache.hadoop.hbase.DoNotRetryIOException) then + # Handle the compatibility case, where the truncate method doesn't exists on the Master + yield 'Dropping table...' if block_given? + @admin.deleteTable(org.apache.hadoop.hbase.TableName.valueOf(table_name)) + + yield 'Creating table...' if block_given? + @admin.createTable(table_description) + else + raise e + end + end + end + + #---------------------------------------------------------------------------------------------- + # Truncates table while maintaing region boundaries (deletes all records by recreating the table) + def truncate_preserve(table_name, conf = @conf) + h_table = @connection.getTable(TableName.valueOf(table_name)) + locator = @connection.getRegionLocator(TableName.valueOf(table_name)) + begin + splits = locator.getAllRegionLocations(). + map{|i| Bytes.toString(i.getRegionInfo().getStartKey)}. + delete_if{|k| k == ""}.to_java :String + ensure + locator.close() + end + + table_description = @admin.getTableDescriptor(TableName.valueOf(table_name)) + yield 'Disabling table...' if block_given? + disable(table_name) + + begin + yield 'Truncating table...' if block_given? + @admin.truncateTable(org.apache.hadoop.hbase.TableName.valueOf(table_name), true) + rescue => e + # Handle the compatibility case, where the truncate method doesn't exists on the Master + raise e unless e.respond_to?(:cause) && e.cause != nil + rootCause = e.cause + if rootCause.kind_of?(org.apache.hadoop.hbase.DoNotRetryIOException) then + # Handle the compatibility case, where the truncate method doesn't exists on the Master + yield 'Dropping table...' if block_given? + @admin.deleteTable(org.apache.hadoop.hbase.TableName.valueOf(table_name)) + + yield 'Creating table with region boundaries...' if block_given? + @admin.createTable(table_description, splits) + else + raise e + end + end + end + + #---------------------------------------------------------------------------------------------- + # Check the status of alter command (number of regions reopened) + def alter_status(table_name) + # Table name should be a string + raise(ArgumentError, "Table name must be of type String") unless table_name.kind_of?(String) + + # Table should exist + raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) + + status = Pair.new() + begin + status = @admin.getAlterStatus(org.apache.hadoop.hbase.TableName.valueOf(table_name)) + if status.getSecond() != 0 + puts "#{status.getSecond() - status.getFirst()}/#{status.getSecond()} regions updated." + else + puts "All regions updated." + end + sleep 1 + end while status != nil && status.getFirst() != 0 + puts "Done." + end + + #---------------------------------------------------------------------------------------------- + # Change table structure or table options + def alter(table_name, wait = true, *args) + # Table name should be a string + raise(ArgumentError, "Table name must be of type String") unless table_name.kind_of?(String) + + # Table should exist + raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) + + # There should be at least one argument + raise(ArgumentError, "There should be at least one argument but the table name") if args.empty? + + # Get table descriptor + htd = @admin.getTableDescriptor(TableName.valueOf(table_name)) + + # Process all args + args.each do |arg| + + + # Normalize args to support column name only alter specs + arg = { NAME => arg } if arg.kind_of?(String) + + # Normalize args to support shortcut delete syntax + arg = { METHOD => 'delete', NAME => arg['delete'] } if arg['delete'] + + # There are 3 possible options. + # 1) Column family spec. Distinguished by having a NAME and no METHOD. + method = arg.delete(METHOD) + if method == nil and arg.has_key?(NAME) + descriptor = hcd(arg, htd) + column_name = descriptor.getNameAsString + + # If column already exist, then try to alter it. Create otherwise. + if htd.hasFamily(column_name.to_java_bytes) + @admin.modifyColumn(table_name, descriptor) + else + @admin.addColumn(table_name, descriptor) + end + + if wait == true + puts "Updating all regions with the new schema..." + alter_status(table_name) + end + + # We bypass descriptor when adding column families; refresh it to apply other args correctly. + htd = @admin.getTableDescriptor(TableName.valueOf(table_name)) + next + end + + # 2) Method other than table_att, with some args. + name = arg.delete(NAME) + if method != nil and method != "table_att" + # Delete column family + if method == "delete" + raise(ArgumentError, "NAME parameter missing for delete method") unless name + @admin.deleteColumn(table_name, name) + # Unset table attributes + elsif method == "table_att_unset" + raise(ArgumentError, "NAME parameter missing for table_att_unset method") unless name + if name.kind_of?(Array) + name.each do |key| + if (htd.getValue(key) == nil) + raise ArgumentError, "Could not find attribute: #{key}" + end + htd.remove(key) + end + else + if (htd.getValue(name) == nil) + raise ArgumentError, "Could not find attribute: #{name}" + end + htd.remove(name) + end + @admin.modifyTable(table_name.to_java_bytes, htd) + # Unknown method + else + raise ArgumentError, "Unknown method: #{method}" + end + + arg.each_key do |unknown_key| + puts("Unknown argument ignored: %s" % [unknown_key]) + end + + if wait == true + puts "Updating all regions with the new schema..." + alter_status(table_name) + end + + if method == "delete" + # We bypass descriptor when deleting column families; refresh it to apply other args correctly. + htd = @admin.getTableDescriptor(TableName.valueOf(table_name)) + end + next + end + + # 3) Some args for the table, optionally with METHOD => table_att (deprecated) + raise(ArgumentError, "NAME argument in an unexpected place") if name + htd.setOwnerString(arg.delete(OWNER)) if arg[OWNER] + htd.setMaxFileSize(JLong.valueOf(arg.delete(MAX_FILESIZE))) if arg[MAX_FILESIZE] + htd.setReadOnly(JBoolean.valueOf(arg.delete(READONLY))) if arg[READONLY] + htd.setCompactionEnabled(JBoolean.valueOf(arg.delete(COMPACTION_ENABLED))) if arg[COMPACTION_ENABLED] + parse_htd_args(htd, arg) + htd.setMemStoreFlushSize(JLong.valueOf(arg.delete(MEMSTORE_FLUSHSIZE))) if arg[MEMSTORE_FLUSHSIZE] + # DEFERRED_LOG_FLUSH is deprecated and was replaced by DURABILITY. To keep backward compatible, it still exists. + # However, it has to be set before DURABILITY so that DURABILITY could overwrite if both args are set + if arg.include?(DEFERRED_LOG_FLUSH) + if arg.delete(DEFERRED_LOG_FLUSH).to_s.upcase == "TRUE" + htd.setDurability(org.apache.hadoop.hbase.client.Durability.valueOf("ASYNC_WAL")) + else + htd.setDurability(org.apache.hadoop.hbase.client.Durability.valueOf("SYNC_WAL")) + end + end + htd.setDurability(org.apache.hadoop.hbase.client.Durability.valueOf(arg.delete(DURABILITY))) if arg[DURABILITY] + htd.setFlushPolicyClassName(arg.delete(FLUSH_POLICY)) if arg[FLUSH_POLICY] + htd.setRegionSplitPolicyClassName(arg.delete(SPLIT_POLICY)) if arg[SPLIT_POLICY] + htd.setRegionMemstoreReplication(JBoolean.valueOf(arg.delete(REGION_MEMSTORE_REPLICATION))) if arg[REGION_MEMSTORE_REPLICATION] + htd.setRegionReplication(JInteger.valueOf(arg.delete(REGION_REPLICATION))) if arg[REGION_REPLICATION] + set_user_metadata(htd, arg.delete(METADATA)) if arg[METADATA] + set_descriptor_config(htd, arg.delete(CONFIGURATION)) if arg[CONFIGURATION] + + # set a coprocessor attribute + valid_coproc_keys = [] + if arg.kind_of?(Hash) + arg.each do |key, value| + k = String.new(key) # prepare to strip + k.strip! + + if (k =~ /coprocessor/i) + v = String.new(value) + v.strip! + htd.addCoprocessorWithSpec(v) + valid_coproc_keys << key + end + end + + valid_coproc_keys.each do |key| + arg.delete(key) + end + + @admin.modifyTable(table_name.to_java_bytes, htd) + + arg.each_key do |unknown_key| + puts("Unknown argument ignored: %s" % [unknown_key]) + end + + if wait == true + puts "Updating all regions with the new schema..." + alter_status(table_name) + end + next + end + end + end + + def status(format, type) + status = @admin.getClusterStatus() + if format == "detailed" + puts("version %s" % [ status.getHBaseVersion() ]) + # Put regions in transition first because usually empty + puts("%d regionsInTransition" % status.getRegionsInTransition().size()) + for k, v in status.getRegionsInTransition() + puts(" %s" % [v]) + end + master = status.getMaster() + puts("active master: %s:%d %d" % [master.getHostname(), master.getPort(), master.getStartcode()]) + puts("%d backup masters" % [ status.getBackupMastersSize() ]) + for server in status.getBackupMasters() + puts(" %s:%d %d" % \ + [ server.getHostname(), server.getPort(), server.getStartcode() ]) + end + + master_coprocs = java.util.Arrays.toString(@admin.getMasterCoprocessors()) + if master_coprocs != nil + puts("master coprocessors: %s" % master_coprocs) + end + puts("%d live servers" % [ status.getServersSize() ]) + for server in status.getServers() + puts(" %s:%d %d" % \ + [ server.getHostname(), server.getPort(), server.getStartcode() ]) + puts(" %s" % [ status.getLoad(server).toString() ]) + for name, region in status.getLoad(server).getRegionsLoad() + puts(" %s" % [ region.getNameAsString().dump ]) + puts(" %s" % [ region.toString() ]) + end + end + puts("%d dead servers" % [ status.getDeadServers() ]) + for server in status.getDeadServerNames() + puts(" %s" % [ server ]) + end + elsif format == "replication" + #check whether replication is enabled or not + if (!@admin.getConfiguration().getBoolean(org.apache.hadoop.hbase.HConstants::REPLICATION_ENABLE_KEY, + org.apache.hadoop.hbase.HConstants::REPLICATION_ENABLE_DEFAULT)) + puts("Please enable replication first.") + else + puts("version %s" % [ status.getHBaseVersion() ]) + puts("%d live servers" % [ status.getServersSize() ]) + for server in status.getServers() + sl = status.getLoad(server) + rSinkString = " SINK :" + rSourceString = " SOURCE:" + rLoadSink = sl.getReplicationLoadSink() + rSinkString << " AgeOfLastAppliedOp=" + rLoadSink.getAgeOfLastAppliedOp().to_s + rSinkString << ", TimeStampsOfLastAppliedOp=" + + (java.util.Date.new(rLoadSink.getTimeStampsOfLastAppliedOp())).toString() + rLoadSourceList = sl.getReplicationLoadSourceList() + index = 0 + while index < rLoadSourceList.size() + rLoadSource = rLoadSourceList.get(index) + rSourceString << " PeerID=" + rLoadSource.getPeerID() + rSourceString << ", AgeOfLastShippedOp=" + rLoadSource.getAgeOfLastShippedOp().to_s + rSourceString << ", SizeOfLogQueue=" + rLoadSource.getSizeOfLogQueue().to_s + rSourceString << ", TimeStampsOfLastShippedOp=" + + (java.util.Date.new(rLoadSource.getTimeStampOfLastShippedOp())).toString() + rSourceString << ", Replication Lag=" + rLoadSource.getReplicationLag().to_s + index = index + 1 + end + puts(" %s:" % + [ server.getHostname() ]) + if type.casecmp("SOURCE") == 0 + puts("%s" % rSourceString) + elsif type.casecmp("SINK") == 0 + puts("%s" % rSinkString) + else + puts("%s" % rSourceString) + puts("%s" % rSinkString) + end + end + end + elsif format == "simple" + load = 0 + regions = 0 + master = status.getMaster() + puts("active master: %s:%d %d" % [master.getHostname(), master.getPort(), master.getStartcode()]) + puts("%d backup masters" % [ status.getBackupMastersSize() ]) + for server in status.getBackupMasters() + puts(" %s:%d %d" % \ + [ server.getHostname(), server.getPort(), server.getStartcode() ]) + end + puts("%d live servers" % [ status.getServersSize() ]) + for server in status.getServers() + puts(" %s:%d %d" % \ + [ server.getHostname(), server.getPort(), server.getStartcode() ]) + puts(" %s" % [ status.getLoad(server).toString() ]) + load += status.getLoad(server).getNumberOfRequests() + regions += status.getLoad(server).getNumberOfRegions() + end + puts("%d dead servers" % [ status.getDeadServers() ]) + for server in status.getDeadServerNames() + puts(" %s" % [ server ]) + end + puts("Aggregate load: %d, regions: %d" % [ load , regions ] ) + else + puts "1 active master, #{status.getBackupMastersSize} backup masters, #{status.getServersSize} servers, #{status.getDeadServers} dead, #{'%.4f' % status.getAverageLoad} average load" + end + end + + #---------------------------------------------------------------------------------------------- + # + # Helper methods + # + + # Does table exist? + def exists?(table_name) + @admin.tableExists(table_name) + end + + #---------------------------------------------------------------------------------------------- + # Is table enabled + def enabled?(table_name) + @admin.isTableEnabled(table_name) + end + + #---------------------------------------------------------------------------------------------- + #Is supplied region name is encoded region name + def closeEncodedRegion?(region_name, server) + @admin.closeRegionWithEncodedRegionName(region_name, server) + end + + #---------------------------------------------------------------------------------------------- + # Return a new HColumnDescriptor made of passed args + def hcd(arg, htd) + # String arg, single parameter constructor + return org.apache.hadoop.hbase.HColumnDescriptor.new(arg) if arg.kind_of?(String) + + raise(ArgumentError, "Column family #{arg} must have a name") unless name = arg.delete(NAME) + + family = htd.getFamily(name.to_java_bytes) + # create it if it's a new family + family ||= org.apache.hadoop.hbase.HColumnDescriptor.new(name.to_java_bytes) + + family.setBlockCacheEnabled(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKCACHE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKCACHE) + family.setScope(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::REPLICATION_SCOPE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::REPLICATION_SCOPE) + family.setCacheDataOnWrite(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_DATA_ON_WRITE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_DATA_ON_WRITE) + family.setCacheIndexesOnWrite(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_INDEX_ON_WRITE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_INDEX_ON_WRITE) + family.setCacheBloomsOnWrite(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_BLOOMS_ON_WRITE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_BLOOMS_ON_WRITE) + family.setEvictBlocksOnClose(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::EVICT_BLOCKS_ON_CLOSE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::EVICT_BLOCKS_ON_CLOSE) + family.setCacheDataInL1(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_DATA_IN_L1))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::CACHE_DATA_IN_L1) + family.setInMemory(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::IN_MEMORY) + family.setTimeToLive(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::TTL))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::TTL) + family.setDataBlockEncoding(org.apache.hadoop.hbase.io.encoding.DataBlockEncoding.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::DATA_BLOCK_ENCODING))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::DATA_BLOCK_ENCODING) + family.setBlocksize(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKSIZE))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::BLOCKSIZE) + family.setMaxVersions(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::VERSIONS))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::VERSIONS) + family.setMinVersions(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::MIN_VERSIONS))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::MIN_VERSIONS) + family.setKeepDeletedCells(org.apache.hadoop.hbase.KeepDeletedCells.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::KEEP_DELETED_CELLS).to_s.upcase)) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::KEEP_DELETED_CELLS) + family.setCompressTags(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESS_TAGS))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESS_TAGS) + family.setPrefetchBlocksOnOpen(JBoolean.valueOf(arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN))) if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::PREFETCH_BLOCKS_ON_OPEN) + family.setValue(COMPRESSION_COMPACT, arg.delete(COMPRESSION_COMPACT)) if arg.include?(COMPRESSION_COMPACT) + if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::BLOOMFILTER) + bloomtype = arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::BLOOMFILTER).upcase + unless org.apache.hadoop.hbase.regionserver.BloomType.constants.include?(bloomtype) + raise(ArgumentError, "BloomFilter type #{bloomtype} is not supported. Use one of " + org.apache.hadoop.hbase.regionserver.StoreFile::BloomType.constants.join(" ")) + else + family.setBloomFilterType(org.apache.hadoop.hbase.regionserver.BloomType.valueOf(bloomtype)) + end + end + if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESSION) + compression = arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::COMPRESSION).upcase + unless org.apache.hadoop.hbase.io.compress.Compression::Algorithm.constants.include?(compression) + raise(ArgumentError, "Compression #{compression} is not supported. Use one of " + org.apache.hadoop.hbase.io.compress.Compression::Algorithm.constants.join(" ")) + else + family.setCompressionType(org.apache.hadoop.hbase.io.compress.Compression::Algorithm.valueOf(compression)) + end + end + if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::ENCRYPTION) + algorithm = arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::ENCRYPTION).upcase + family.setEncryptionType(algorithm) + if arg.include?(org.apache.hadoop.hbase.HColumnDescriptor::ENCRYPTION_KEY) + key = org.apache.hadoop.hbase.io.crypto.Encryption.pbkdf128( + arg.delete(org.apache.hadoop.hbase.HColumnDescriptor::ENCRYPTION_KEY)) + family.setEncryptionKey(org.apache.hadoop.hbase.security.EncryptionUtil.wrapKey(@conf, key, + algorithm)) + end + end + + set_user_metadata(family, arg.delete(METADATA)) if arg[METADATA] + set_descriptor_config(family, arg.delete(CONFIGURATION)) if arg[CONFIGURATION] + family.setDFSReplication(JInteger.valueOf(arg.delete(org.apache.hadoop.hbase. + HColumnDescriptor::DFS_REPLICATION))) if arg.include?(org.apache.hadoop.hbase. + HColumnDescriptor::DFS_REPLICATION) + + arg.each_key do |unknown_key| + puts("Unknown argument ignored for column family %s: %s" % [name, unknown_key]) + end + + return family + end + + #---------------------------------------------------------------------------------------------- + # Enables/disables a region by name + def online(region_name, on_off) + # Open meta table + meta = @connection.getTable(org.apache.hadoop.hbase.TableName::META_TABLE_NAME) + + # Read region info + # FIXME: fail gracefully if can't find the region + region_bytes = region_name.to_java_bytes + g = org.apache.hadoop.hbase.client.Get.new(region_bytes) + g.addColumn(org.apache.hadoop.hbase.HConstants::CATALOG_FAMILY, org.apache.hadoop.hbase.HConstants::REGIONINFO_QUALIFIER) + hri_bytes = meta.get(g).value + + # Change region status + hri = org.apache.hadoop.hbase.util.Writables.getWritable(hri_bytes, org.apache.hadoop.hbase.HRegionInfo.new) + hri.setOffline(on_off) + + # Write it back + put = org.apache.hadoop.hbase.client.Put.new(region_bytes) + put.add(org.apache.hadoop.hbase.HConstants::CATALOG_FAMILY, org.apache.hadoop.hbase.HConstants::REGIONINFO_QUALIFIER, org.apache.hadoop.hbase.util.Writables.getBytes(hri)) + meta.put(put) + end + # Apply user metadata to table/column descriptor + def set_user_metadata(descriptor, metadata) + raise(ArgumentError, "#{METADATA} must be a Hash type") unless metadata.kind_of?(Hash) + for k,v in metadata + v = v.to_s unless v.nil? + descriptor.setValue(k, v) + end + end + + #---------------------------------------------------------------------------------------------- + # Take a snapshot of specified table + def snapshot(table, snapshot_name, *args) + if args.empty? + @admin.snapshot(snapshot_name.to_java_bytes, table.to_java_bytes) + else + args.each do |arg| + if arg[SKIP_FLUSH] == true + @admin.snapshot(snapshot_name.to_java_bytes, table.to_java_bytes, SnapshotDescription::Type::SKIPFLUSH) + else + @admin.snapshot(snapshot_name.to_java_bytes, table.to_java_bytes) + end + end + end + end + + #---------------------------------------------------------------------------------------------- + # Restore specified snapshot + def restore_snapshot(snapshot_name) + @admin.restoreSnapshot(snapshot_name.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Create a new table by cloning the snapshot content + def clone_snapshot(snapshot_name, table) + @admin.cloneSnapshot(snapshot_name.to_java_bytes, table.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Delete specified snapshot + def delete_snapshot(snapshot_name) + @admin.deleteSnapshot(snapshot_name.to_java_bytes) + end + + #---------------------------------------------------------------------------------------------- + # Deletes the snapshots matching the given regex + def delete_all_snapshot(regex) + @admin.deleteSnapshots(regex).to_a + end + + #---------------------------------------------------------------------------------------------- + # Returns a list of snapshots + def list_snapshot(regex = ".*") + @admin.listSnapshots(regex).to_a + end + + # Apply config specific to a table/column to its descriptor + def set_descriptor_config(descriptor, config) + raise(ArgumentError, "#{CONFIGURATION} must be a Hash type") unless config.kind_of?(Hash) + for k,v in config + v = v.to_s unless v.nil? + descriptor.setConfiguration(k, v) + end + end + + #---------------------------------------------------------------------------------------------- + # Updates the configuration of one regionserver. + def update_config(serverName) + @admin.updateConfiguration(ServerName.valueOf(serverName)); + end + + #---------------------------------------------------------------------------------------------- + # Updates the configuration of all the regionservers. + def update_all_config() + @admin.updateConfiguration(); + end + + #---------------------------------------------------------------------------------------------- + # Returns namespace's structure description + def describe_namespace(namespace_name) + namespace = @admin.getNamespaceDescriptor(namespace_name) + + unless namespace.nil? + return namespace.to_s + end + + raise(ArgumentError, "Failed to find namespace named #{namespace_name}") + end + + #---------------------------------------------------------------------------------------------- + # Returns a list of namespaces in hbase + def list_namespace(regex = ".*") + pattern = java.util.regex.Pattern.compile(regex) + list = @admin.listNamespaceDescriptors.map { |ns| ns.getName } + list.select {|s| pattern.match(s) } + end + + #---------------------------------------------------------------------------------------------- + # Returns a list of tables in namespace + def list_namespace_tables(namespace_name) + unless namespace_name.nil? + return @admin.listTableNamesByNamespace(namespace_name).map { |t| t.getQualifierAsString() } + end + + raise(ArgumentError, "Failed to find namespace named #{namespace_name}") + end + + #---------------------------------------------------------------------------------------------- + # Creates a namespace + def create_namespace(namespace_name, *args) + # Fail if table name is not a string + raise(ArgumentError, "Namespace name must be of type String") unless namespace_name.kind_of?(String) + + # Flatten params array + args = args.flatten.compact + + # Start defining the table + nsb = org.apache.hadoop.hbase.NamespaceDescriptor::create(namespace_name) + args.each do |arg| + unless arg.kind_of?(Hash) + raise(ArgumentError, "#{arg.class} of #{arg.inspect} is not of Hash or String type") + end + for k,v in arg + v = v.to_s unless v.nil? + nsb.addConfiguration(k, v) + end + end + @admin.createNamespace(nsb.build()); + end + + #---------------------------------------------------------------------------------------------- + # modify a namespace + def alter_namespace(namespace_name, *args) + # Fail if table name is not a string + raise(ArgumentError, "Namespace name must be of type String") unless namespace_name.kind_of?(String) + + nsd = @admin.getNamespaceDescriptor(namespace_name) + + unless nsd + raise(ArgumentError, "Namespace does not exist") + end + nsb = org.apache.hadoop.hbase.NamespaceDescriptor::create(nsd) + + # Flatten params array + args = args.flatten.compact + + # Start defining the table + args.each do |arg| + unless arg.kind_of?(Hash) + raise(ArgumentError, "#{arg.class} of #{arg.inspect} is not of Hash type") + end + method = arg[METHOD] + if method == "unset" + nsb.removeConfiguration(arg[NAME]) + elsif method == "set" + arg.delete(METHOD) + for k,v in arg + v = v.to_s unless v.nil? + + nsb.addConfiguration(k, v) + end + else + raise(ArgumentError, "Unknown method #{method}") + end + end + @admin.modifyNamespace(nsb.build()); + end + + + #---------------------------------------------------------------------------------------------- + # Drops a table + def drop_namespace(namespace_name) + @admin.deleteNamespace(namespace_name) + end + + #---------------------------------------------------------------------------------------------- + # Get security capabilities + def get_security_capabilities + @admin.getSecurityCapabilities + end + + # Abort a procedure + def abort_procedure?(proc_id, may_interrupt_if_running=nil) + if may_interrupt_if_running.nil? + @admin.abortProcedure(proc_id, true) + else + @admin.abortProcedure(proc_id, may_interrupt_if_running) + end + end + + # List all procedures + def list_procedures() + @admin.listProcedures() + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/hbase.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/hbase.rb new file mode 100644 index 00000000000..874b2ce8a9c --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/hbase.rb @@ -0,0 +1,79 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java +java_import org.apache.hadoop.hbase.client.ConnectionFactory +java_import org.apache.hadoop.hbase.HBaseConfiguration +java_import org.apache.linkis.manager.engineplugin.hbase.HBaseConnectionManager + +require 'hbase/admin' +require 'hbase/table' +require 'hbase/quotas' +require 'hbase/security' +require 'hbase/visibility_labels' + +module Hbase + class Hbase + attr_accessor :configuration + + def initialize(properties = nil) + # Create configuration + @connection = nil + if properties + self.configuration = HBaseConnectionManager.getInstance().getConfiguration(properties) + @connection = HBaseConnectionManager.getInstance().getConnection(self.configuration) + else + self.configuration = org.apache.hadoop.hbase.HBaseConfiguration.create + # Turn off retries in hbase and ipc. Human doesn't want to wait on N retries. + configuration.setInt("hbase.client.retries.number", 7) + configuration.setInt("hbase.ipc.client.connect.max.retries", 3) + @connection = ConnectionFactory.createConnection(self.configuration) + end + end + + def admin(formatter) + ::Hbase::Admin.new(@connection.getAdmin, formatter) + end + + # Create new one each time + def table(table, shell) + ::Hbase::Table.new(@connection.getTable(table), shell) + end + + def replication_admin(formatter) + ::Hbase::RepAdmin.new(configuration, formatter) + end + + def security_admin(formatter) + ::Hbase::SecurityAdmin.new(@connection.getAdmin, formatter) + end + + def visibility_labels_admin(formatter) + ::Hbase::VisibilityLabelsAdmin.new(@connection.getAdmin, formatter) + end + + def quotas_admin(formatter) + ::Hbase::QuotasAdmin.new(@connection.getAdmin, formatter) + end + + def shutdown + @connection.close + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/quotas.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/quotas.rb new file mode 100644 index 00000000000..0be428de8d7 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/quotas.rb @@ -0,0 +1,219 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java +java_import java.util.concurrent.TimeUnit +java_import org.apache.hadoop.hbase.TableName +java_import org.apache.hadoop.hbase.quotas.ThrottleType +java_import org.apache.hadoop.hbase.quotas.QuotaFilter +java_import org.apache.hadoop.hbase.quotas.QuotaRetriever +java_import org.apache.hadoop.hbase.quotas.QuotaSettingsFactory + +module HBaseQuotasConstants + GLOBAL_BYPASS = 'GLOBAL_BYPASS' + THROTTLE_TYPE = 'THROTTLE_TYPE' + THROTTLE = 'THROTTLE' + REQUEST = 'REQUEST' + WRITE = 'WRITE' + READ = 'READ' +end + +module Hbase + class QuotasAdmin + def initialize(admin, formatter) + @admin = admin + @formatter = formatter + end + + def close + @admin.close + end + + def throttle(args) + raise(ArgumentError, "Arguments should be a Hash") unless args.kind_of?(Hash) + type = args.fetch(THROTTLE_TYPE, REQUEST) + args.delete(THROTTLE_TYPE) + type, limit, time_unit = _parse_limit(args.delete(LIMIT), ThrottleType, type) + if args.has_key?(USER) + user = args.delete(USER) + if args.has_key?(TABLE) + table = TableName.valueOf(args.delete(TABLE)) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.throttleUser(user, table, type, limit, time_unit) + elsif args.has_key?(NAMESPACE) + namespace = args.delete(NAMESPACE) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.throttleUser(user, namespace, type, limit, time_unit) + else + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.throttleUser(user, type, limit, time_unit) + end + elsif args.has_key?(TABLE) + table = TableName.valueOf(args.delete(TABLE)) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.throttleTable(table, type, limit, time_unit) + elsif args.has_key?(NAMESPACE) + namespace = args.delete(NAMESPACE) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.throttleNamespace(namespace, type, limit, time_unit) + else + raise "One of USER, TABLE or NAMESPACE must be specified" + end + @admin.setQuota(settings) + end + + def unthrottle(args) + raise(ArgumentError, "Arguments should be a Hash") unless args.kind_of?(Hash) + if args.has_key?(USER) + user = args.delete(USER) + if args.has_key?(TABLE) + table = TableName.valueOf(args.delete(TABLE)) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.unthrottleUser(user, table) + elsif args.has_key?(NAMESPACE) + namespace = args.delete(NAMESPACE) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.unthrottleUser(user, namespace) + else + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.unthrottleUser(user) + end + elsif args.has_key?(TABLE) + table = TableName.valueOf(args.delete(TABLE)) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.unthrottleTable(table) + elsif args.has_key?(NAMESPACE) + namespace = args.delete(NAMESPACE) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.unthrottleNamespace(namespace) + else + raise "One of USER, TABLE or NAMESPACE must be specified" + end + @admin.setQuota(settings) + end + + def set_global_bypass(bypass, args) + raise(ArgumentError, "Arguments should be a Hash") unless args.kind_of?(Hash) + + if args.has_key?(USER) + user = args.delete(USER) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + settings = QuotaSettingsFactory.bypassGlobals(user, bypass) + else + raise "Expected USER" + end + @admin.setQuota(settings) + end + + def list_quotas(args = {}) + raise(ArgumentError, "Arguments should be a Hash") unless args.kind_of?(Hash) + + limit = args.delete("LIMIT") || -1 + count = 0 + + filter = QuotaFilter.new() + filter.setUserFilter(args.delete(USER)) if args.has_key?(USER) + filter.setTableFilter(args.delete(TABLE)) if args.has_key?(TABLE) + filter.setNamespaceFilter(args.delete(NAMESPACE)) if args.has_key?(NAMESPACE) + raise(ArgumentError, "Unexpected arguments: " + args.inspect) unless args.empty? + + # Start the scanner + scanner = @admin.getQuotaRetriever(filter) + begin + iter = scanner.iterator + + # Iterate results + while iter.hasNext + if limit > 0 && count >= limit + break + end + + settings = iter.next + owner = { + USER => settings.getUserName(), + TABLE => settings.getTableName(), + NAMESPACE => settings.getNamespace(), + }.delete_if { |k, v| v.nil? }.map {|k, v| k.to_s + " => " + v.to_s} * ', ' + + yield owner, settings.to_s + + count += 1 + end + ensure + scanner.close() + end + + return count + end + + def _parse_size(str_limit) + str_limit = str_limit.downcase + match = /(\d+)([bkmgtp%]*)/.match(str_limit) + if match + if match[2] == '%' + return match[1].to_i + else + return _size_from_str(match[1].to_i, match[2]) + end + else + raise "Invalid size limit syntax" + end + end + + def _parse_limit(str_limit, type_cls, type) + str_limit = str_limit.downcase + match = /(\d+)(req|[bkmgtp])\/(sec|min|hour|day)/.match(str_limit) + if match + if match[2] == 'req' + limit = match[1].to_i + type = type_cls.valueOf(type + "_NUMBER") + else + limit = _size_from_str(match[1].to_i, match[2]) + type = type_cls.valueOf(type + "_SIZE") + end + + if limit <= 0 + raise "Invalid throttle limit, must be greater then 0" + end + + case match[3] + when 'sec' then time_unit = TimeUnit::SECONDS + when 'min' then time_unit = TimeUnit::MINUTES + when 'hour' then time_unit = TimeUnit::HOURS + when 'day' then time_unit = TimeUnit::DAYS + end + + return type, limit, time_unit + else + raise "Invalid throttle limit syntax" + end + end + + def _size_from_str(value, suffix) + case suffix + when 'k' then value <<= 10 + when 'm' then value <<= 20 + when 'g' then value <<= 30 + when 't' then value <<= 40 + when 'p' then value <<= 50 + end + return value + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/replication_admin.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/replication_admin.rb new file mode 100644 index 00000000000..2a248298204 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/replication_admin.rb @@ -0,0 +1,174 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java + +java_import org.apache.hadoop.hbase.client.replication.ReplicationAdmin +java_import org.apache.hadoop.hbase.replication.ReplicationPeerConfig +java_import org.apache.hadoop.hbase.util.Bytes +java_import org.apache.hadoop.hbase.zookeeper.ZKConfig +java_import org.apache.hadoop.hbase.TableName + +# Wrapper for org.apache.hadoop.hbase.client.replication.ReplicationAdmin + +module Hbase + class RepAdmin + include HBaseConstants + + def initialize(configuration, formatter) + @replication_admin = ReplicationAdmin.new(configuration) + @configuration = configuration + @formatter = formatter + end + + #---------------------------------------------------------------------------------------------- + # Add a new peer cluster to replicate to + def add_peer(id, args = {}, peer_tableCFs = nil) + # make add_peer backwards compatible to take in string for clusterKey and peer_tableCFs + if args.is_a?(String) + cluster_key = args + @replication_admin.addPeer(id, cluster_key, peer_tableCFs) + elsif args.is_a?(Hash) + unless peer_tableCFs.nil? + raise(ArgumentError, "peer_tableCFs should be specified as TABLE_CFS in args") + end + + endpoint_classname = args.fetch(ENDPOINT_CLASSNAME, nil) + cluster_key = args.fetch(CLUSTER_KEY, nil) + + # Handle cases where custom replication endpoint and cluster key are either both provided + # or neither are provided + if endpoint_classname.nil? and cluster_key.nil? + raise(ArgumentError, "Either ENDPOINT_CLASSNAME or CLUSTER_KEY must be specified.") + elsif !endpoint_classname.nil? and !cluster_key.nil? + raise(ArgumentError, "ENDPOINT_CLASSNAME and CLUSTER_KEY cannot both be specified.") + end + + # Cluster Key is required for ReplicationPeerConfig for a custom replication endpoint + if !endpoint_classname.nil? and cluster_key.nil? + cluster_key = ZKConfig.getZooKeeperClusterKey(@configuration) + end + + # Optional parameters + config = args.fetch(CONFIG, nil) + data = args.fetch(DATA, nil) + table_cfs = args.fetch(TABLE_CFS, nil) + + # Create and populate a ReplicationPeerConfig + replication_peer_config = ReplicationPeerConfig.new + replication_peer_config.set_cluster_key(cluster_key) + + unless endpoint_classname.nil? + replication_peer_config.set_replication_endpoint_impl(endpoint_classname) + end + + unless config.nil? + replication_peer_config.get_configuration.put_all(config) + end + + unless data.nil? + # Convert Strings to Bytes for peer_data + peer_data = replication_peer_config.get_peer_data + data.each{|key, val| + peer_data.put(Bytes.to_bytes(key), Bytes.to_bytes(val)) + } + end + + @replication_admin.add_peer(id, replication_peer_config, table_cfs) + else + raise(ArgumentError, "args must be either a String or Hash") + end + end + + #---------------------------------------------------------------------------------------------- + # Remove a peer cluster, stops the replication + def remove_peer(id) + @replication_admin.removePeer(id) + end + + + #--------------------------------------------------------------------------------------------- + # Show replcated tables/column families, and their ReplicationType + def list_replicated_tables(regex = ".*") + pattern = java.util.regex.Pattern.compile(regex) + list = @replication_admin.listReplicated() + list.select {|s| pattern.match(s.get(ReplicationAdmin::TNAME))} + end + + #---------------------------------------------------------------------------------------------- + # List all peer clusters + def list_peers + @replication_admin.listPeers + end + + #---------------------------------------------------------------------------------------------- + # Get peer cluster state + def get_peer_state(id) + @replication_admin.getPeerState(id) ? "ENABLED" : "DISABLED" + end + + #---------------------------------------------------------------------------------------------- + # Restart the replication stream to the specified peer + def enable_peer(id) + @replication_admin.enablePeer(id) + end + + #---------------------------------------------------------------------------------------------- + # Stop the replication stream to the specified peer + def disable_peer(id) + @replication_admin.disablePeer(id) + end + + #---------------------------------------------------------------------------------------------- + # Show the current tableCFs config for the specified peer + def show_peer_tableCFs(id) + @replication_admin.getPeerTableCFs(id) + end + + #---------------------------------------------------------------------------------------------- + # Set new tableCFs config for the specified peer + def set_peer_tableCFs(id, tableCFs) + @replication_admin.setPeerTableCFs(id, tableCFs) + end + + #---------------------------------------------------------------------------------------------- + # Append a tableCFs config for the specified peer + def append_peer_tableCFs(id, tableCFs) + @replication_admin.appendPeerTableCFs(id, tableCFs) + end + + #---------------------------------------------------------------------------------------------- + # Remove some tableCFs from the tableCFs config of the specified peer + def remove_peer_tableCFs(id, tableCFs) + @replication_admin.removePeerTableCFs(id, tableCFs) + end + #---------------------------------------------------------------------------------------------- + # Enables a table's replication switch + def enable_tablerep(table_name) + tableName = TableName.valueOf(table_name) + @replication_admin.enableTableRep(tableName) + end + #---------------------------------------------------------------------------------------------- + # Disables a table's replication switch + def disable_tablerep(table_name) + tableName = TableName.valueOf(table_name) + @replication_admin.disableTableRep(tableName) + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/security.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/security.rb new file mode 100644 index 00000000000..d4dfa1b921e --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/security.rb @@ -0,0 +1,204 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java + +# Wrapper for org.apache.hadoop.hbase.client.HBaseAdmin + +module Hbase + class SecurityAdmin + include HBaseConstants + + def initialize(admin, formatter) + @admin = admin + @connection = @admin.getConnection() + @formatter = formatter + end + + def close + @admin.close + end + + #---------------------------------------------------------------------------------------------- + def grant(user, permissions, table_name=nil, family=nil, qualifier=nil) + security_available? + + # TODO: need to validate user name + + begin + # Verify that the specified permission is valid + if (permissions == nil || permissions.length == 0) + raise(ArgumentError, "Invalid permission: no actions associated with user") + end + + perm = org.apache.hadoop.hbase.security.access.Permission.new( + permissions.to_java_bytes) + + if (table_name != nil) + tablebytes=table_name.to_java_bytes + #check if the tablename passed is actually a namespace + if (isNamespace?(table_name)) + # Namespace should exist first. + namespace_name = table_name[1...table_name.length] + raise(ArgumentError, "Can't find a namespace: #{namespace_name}") unless + namespace_exists?(namespace_name) + + org.apache.hadoop.hbase.security.access.AccessControlClient.grant( + @connection, namespace_name, user, perm.getActions()) + else + # Table should exist + raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) + + tableName = org.apache.hadoop.hbase.TableName.valueOf(table_name.to_java_bytes) + htd = @admin.getTableDescriptor(tableName) + + if (family != nil) + raise(ArgumentError, "Can't find a family: #{family}") unless htd.hasFamily(family.to_java_bytes) + end + + fambytes = family.to_java_bytes if (family != nil) + qualbytes = qualifier.to_java_bytes if (qualifier != nil) + + org.apache.hadoop.hbase.security.access.AccessControlClient.grant( + @connection, tableName, user, fambytes, qualbytes, perm.getActions()) + end + else + # invoke cp endpoint to perform access controls + org.apache.hadoop.hbase.security.access.AccessControlClient.grant( + @connection, user, perm.getActions()) + end + end + end + + #---------------------------------------------------------------------------------------------- + def revoke(user, table_name=nil, family=nil, qualifier=nil) + security_available? + + # TODO: need to validate user name + + begin + if (table_name != nil) + #check if the tablename passed is actually a namespace + if (isNamespace?(table_name)) + # Namespace should exist first. + namespace_name = table_name[1...table_name.length] + raise(ArgumentError, "Can't find a namespace: #{namespace_name}") unless namespace_exists?(namespace_name) + + tablebytes=table_name.to_java_bytes + org.apache.hadoop.hbase.security.access.AccessControlClient.revoke( + @connection, namespace_name, user) + else + # Table should exist + raise(ArgumentError, "Can't find a table: #{table_name}") unless exists?(table_name) + + tableName = org.apache.hadoop.hbase.TableName.valueOf(table_name.to_java_bytes) + htd = @admin.getTableDescriptor(tableName) + + if (family != nil) + raise(ArgumentError, "Can't find a family: #{family}") unless htd.hasFamily(family.to_java_bytes) + end + + fambytes = family.to_java_bytes if (family != nil) + qualbytes = qualifier.to_java_bytes if (qualifier != nil) + + org.apache.hadoop.hbase.security.access.AccessControlClient.revoke( + @connection, tableName, user, fambytes, qualbytes) + end + else + perm = org.apache.hadoop.hbase.security.access.Permission.new(''.to_java_bytes) + org.apache.hadoop.hbase.security.access.AccessControlClient.revoke( + @connection, user, perm.getActions()) + end + end + end + + #---------------------------------------------------------------------------------------------- + def user_permission(table_regex=nil) + security_available? + all_perms = org.apache.hadoop.hbase.security.access.AccessControlClient.getUserPermissions( + @connection,table_regex) + res = {} + count = 0 + all_perms.each do |value| + user_name = String.from_java_bytes(value.getUser) + if (table_regex != nil && isNamespace?(table_regex)) + namespace = value.getNamespace() + else + namespace = (value.getTableName != nil) ? value.getTableName.getNamespaceAsString() : value.getNamespace() + end + table = (value.getTableName != nil) ? value.getTableName.getNameAsString() : '' + family = (value.getFamily != nil) ? + org.apache.hadoop.hbase.util.Bytes::toStringBinary(value.getFamily) : + '' + qualifier = (value.getQualifier != nil) ? + org.apache.hadoop.hbase.util.Bytes::toStringBinary(value.getQualifier) : + '' + + action = org.apache.hadoop.hbase.security.access.Permission.new value.getActions + + if block_given? + yield(user_name, "#{namespace},#{table},#{family},#{qualifier}: #{action.to_s}") + else + res[user_name] ||= {} + res[user_name]["#{family}:#{qualifier}"] = action + end + count += 1 + end + + return ((block_given?) ? count : res) + end + + # Does table exist? + def exists?(table_name) + @admin.tableExists(table_name) + end + + def isNamespace?(table_name) + table_name.start_with?('@') + end + + # Does Namespace exist + def namespace_exists?(namespace_name) + namespaceDesc = @admin.getNamespaceDescriptor(namespace_name) + if(namespaceDesc == nil) + return false + else + return true + end + end + + # Make sure that security features are available + def security_available?() + caps = [] + begin + # Try the getSecurityCapabilities API where supported. + # We only need to look at AUTHORIZATION, the AccessController doesn't support + # CELL_AUTHORIZATION without AUTHORIZATION also available. + caps = @admin.getSecurityCapabilities + rescue + # If we are unable to use getSecurityCapabilities, fall back with a check for + # deployment of the ACL table + raise(ArgumentError, "DISABLED: Security features are not available") unless \ + exists?(org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME) + return + end + raise(ArgumentError, "DISABLED: Security features are not available") unless \ + caps.include? org.apache.hadoop.hbase.client.security.SecurityCapability::AUTHORIZATION + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/table.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/table.rb new file mode 100644 index 00000000000..c86cab918b1 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/table.rb @@ -0,0 +1,716 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java + +# Wrapper for org.apache.hadoop.hbase.client.Table + +module Hbase + class Table + include HBaseConstants + + @@thread_pool = nil + + # Add the command 'name' to table s.t. the shell command also called via 'name' + # and has an internal method also called 'name'. + # + # e.g. name = scan, adds table.scan which calls Scan.scan + def self.add_shell_command(name) + self.add_command(name, name, name) + end + + # add a named command to the table instance + # + # name - name of the command that should added to the table + # (eg. sending 'scan' here would allow you to do table.scan) + # shell_command - name of the command in the shell + # internal_method_name - name of the method in the shell command to forward the call + def self.add_command(name, shell_command, internal_method_name) + method = name.to_sym + self.class_eval do + define_method method do |*args| + @shell.internal_command(shell_command, internal_method_name, self, *args) + end + end + end + + # General help for the table + # class level so we can call it from anywhere + def self.help + return <<-EOF +Help for table-reference commands. + +You can either create a table via 'create' and then manipulate the table via commands like 'put', 'get', etc. +See the standard help information for how to use each of these commands. + +However, as of 0.96, you can also get a reference to a table, on which you can invoke commands. +For instance, you can get create a table and keep around a reference to it via: + + hbase> t = create 't', 'cf' + +Or, if you have already created the table, you can get a reference to it: + + hbase> t = get_table 't' + +You can do things like call 'put' on the table: + + hbase> t.put 'r', 'cf:q', 'v' + +which puts a row 'r' with column family 'cf', qualifier 'q' and value 'v' into table t. + +To read the data out, you can scan the table: + + hbase> t.scan + +which will read all the rows in table 't'. + +Essentially, any command that takes a table name can also be done via table reference. +Other commands include things like: get, delete, deleteall, +get_all_columns, get_counter, count, incr. These functions, along with +the standard JRuby object methods are also available via tab completion. + +For more information on how to use each of these commands, you can also just type: + + hbase> t.help 'scan' + +which will output more information on how to use that command. + +You can also do general admin actions directly on a table; things like enable, disable, +flush and drop just by typing: + + hbase> t.enable + hbase> t.flush + hbase> t.disable + hbase> t.drop + +Note that after dropping a table, your reference to it becomes useless and further usage +is undefined (and not recommended). +EOF + end + + #--------------------------------------------------------------------------------------------- + + # let external objects read the underlying table object + attr_reader :table + # let external objects read the table name + attr_reader :name + + def initialize(table, shell) + @table = table + @name = @table.getName().getNameAsString() + @shell = shell + @converters = Hash.new() + end + + def close() + @table.close() + end + + # Note the below methods are prefixed with '_' to hide them from the average user, as + # they will be much less likely to tab complete to the 'dangerous' internal method + #---------------------------------------------------------------------------------------------- + + # Put a cell 'value' at specified table/row/column + def _put_internal(row, column, value, timestamp = nil, args = {}) + p = org.apache.hadoop.hbase.client.Put.new(row.to_s.to_java_bytes) + family, qualifier = parse_column_name(column) + if args.any? + attributes = args[ATTRIBUTES] + set_attributes(p, attributes) if attributes + visibility = args[VISIBILITY] + set_cell_visibility(p, visibility) if visibility + ttl = args[TTL] + set_op_ttl(p, ttl) if ttl + end + #Case where attributes are specified without timestamp + if timestamp.kind_of?(Hash) + timestamp.each do |k, v| + if k == 'ATTRIBUTES' + set_attributes(p, v) + elsif k == 'VISIBILITY' + set_cell_visibility(p, v) + elsif k == "TTL" + set_op_ttl(p, v) + end + end + timestamp = nil + end + if timestamp + p.add(family, qualifier, timestamp, value.to_s.to_java_bytes) + else + p.add(family, qualifier, value.to_s.to_java_bytes) + end + @table.put(p) + end + + #---------------------------------------------------------------------------------------------- + # Delete a cell + def _delete_internal(row, column, + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP, + args = {}, all_version = false) + _deleteall_internal(row, column, timestamp, args, all_version) + end + + #---------------------------------------------------------------------------------------------- + # Delete a row + def _deleteall_internal(row, column = nil, + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP, + args = {}, all_version = true) + # delete operation doesn't need read permission. Retaining the read check for + # meta table as a part of HBASE-5837. + if is_meta_table? + raise ArgumentError, "Row Not Found" if _get_internal(row).nil? + end + temptimestamp = timestamp + if temptimestamp.kind_of?(Hash) + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP + end + d = org.apache.hadoop.hbase.client.Delete.new(row.to_s.to_java_bytes, timestamp) + if temptimestamp.kind_of?(Hash) + temptimestamp.each do |k, v| + if v.kind_of?(String) + set_cell_visibility(d, v) if v + end + end + end + if args.any? + visibility = args[VISIBILITY] + set_cell_visibility(d, visibility) if visibility + end + if column && all_version + family, qualifier = parse_column_name(column) + d.deleteColumns(family, qualifier, timestamp) + elsif column && !all_version + family, qualifier = parse_column_name(column) + d.deleteColumn(family, qualifier, timestamp) + end + @table.delete(d) + end + + #---------------------------------------------------------------------------------------------- + # Increment a counter atomically + def _incr_internal(row, column, value = nil, args={}) + if value.kind_of?(Hash) + value = 1 + end + value ||= 1 + incr = org.apache.hadoop.hbase.client.Increment.new(row.to_s.to_java_bytes) + family, qualifier = parse_column_name(column) + if qualifier.nil? + raise ArgumentError, "Failed to provide both column family and column qualifier for incr" + end + if args.any? + attributes = args[ATTRIBUTES] + visibility = args[VISIBILITY] + set_attributes(incr, attributes) if attributes + set_cell_visibility(incr, visibility) if visibility + ttl = args[TTL] + set_op_ttl(incr, ttl) if ttl + end + incr.addColumn(family, qualifier, value) + result = @table.increment(incr) + return nil if result.isEmpty + + # Fetch cell value + cell = result.listCells[0] + org.apache.hadoop.hbase.util.Bytes::toLong(cell.getValue) + end + + #---------------------------------------------------------------------------------------------- + # appends the value atomically + def _append_internal(row, column, value, args={}) + append = org.apache.hadoop.hbase.client.Append.new(row.to_s.to_java_bytes) + family, qualifier = parse_column_name(column) + if qualifier.nil? + raise ArgumentError, "Failed to provide both column family and column qualifier for append" + end + if args.any? + attributes = args[ATTRIBUTES] + visibility = args[VISIBILITY] + set_attributes(append, attributes) if attributes + set_cell_visibility(append, visibility) if visibility + ttl = args[TTL] + set_op_ttl(append, ttl) if ttl + end + append.add(family, qualifier, value.to_s.to_java_bytes) + @table.append(append) + end + + #---------------------------------------------------------------------------------------------- + # Count rows in a table + def _count_internal(interval = 1000, caching_rows = 10) + # We can safely set scanner caching with the first key only filter + scan = org.apache.hadoop.hbase.client.Scan.new + scan.setCacheBlocks(false) + scan.setCaching(caching_rows) + scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new) + + # Run the scanner + scanner = @table.getScanner(scan) + count = 0 + iter = scanner.iterator + + # Iterate results + while iter.hasNext + row = iter.next + count += 1 + next unless (block_given? && count % interval == 0) + # Allow command modules to visualize counting process + yield(count, + org.apache.hadoop.hbase.util.Bytes::toStringBinary(row.getRow)) + end + + scanner.close() + # Return the counter + return count + end + + #---------------------------------------------------------------------------------------------- + # Get from table + def _get_internal(row, *args) + get = org.apache.hadoop.hbase.client.Get.new(row.to_s.to_java_bytes) + maxlength = -1 + @converters.clear() + + # Normalize args + args = args.first if args.first.kind_of?(Hash) + if args.kind_of?(String) || args.kind_of?(Array) + columns = [ args ].flatten.compact + args = { COLUMNS => columns } + end + + # + # Parse arguments + # + unless args.kind_of?(Hash) + raise ArgumentError, "Failed parse of of #{args.inspect}, #{args.class}" + end + + # Get maxlength parameter if passed + maxlength = args.delete(MAXLENGTH) if args[MAXLENGTH] + filter = args.delete(FILTER) if args[FILTER] + attributes = args[ATTRIBUTES] + authorizations = args[AUTHORIZATIONS] + consistency = args.delete(CONSISTENCY) if args[CONSISTENCY] + replicaId = args.delete(REGION_REPLICA_ID) if args[REGION_REPLICA_ID] + unless args.empty? + columns = args[COLUMN] || args[COLUMNS] + if args[VERSIONS] + vers = args[VERSIONS] + else + vers = 1 + end + if columns + # Normalize types, convert string to an array of strings + columns = [ columns ] if columns.is_a?(String) + + # At this point it is either an array or some unsupported stuff + unless columns.kind_of?(Array) + raise ArgumentError, "Failed parse column argument type #{args.inspect}, #{args.class}" + end + + # Get each column name and add it to the filter + columns.each do |column| + family, qualifier = parse_column_name(column.to_s) + if qualifier + get.addColumn(family, qualifier) + else + get.addFamily(family) + end + end + + # Additional params + get.setMaxVersions(vers) + get.setTimeStamp(args[TIMESTAMP]) if args[TIMESTAMP] + get.setTimeRange(args[TIMERANGE][0], args[TIMERANGE][1]) if args[TIMERANGE] + else + if attributes + set_attributes(get, attributes) + elsif authorizations + set_authorizations(get, authorizations) + else + # May have passed TIMESTAMP and row only; wants all columns from ts. + unless ts = args[TIMESTAMP] || tr = args[TIMERANGE] + raise ArgumentError, "Failed parse of #{args.inspect}, #{args.class}" + end + end + + get.setMaxVersions(vers) + # Set the timestamp/timerange + get.setTimeStamp(ts.to_i) if args[TIMESTAMP] + get.setTimeRange(args[TIMERANGE][0], args[TIMERANGE][1]) if args[TIMERANGE] + end + set_attributes(get, attributes) if attributes + set_authorizations(get, authorizations) if authorizations + end + + unless filter.class == String + get.setFilter(filter) + else + get.setFilter( + org.apache.hadoop.hbase.filter.ParseFilter.new.parseFilterString(filter.to_java_bytes)) + end + + get.setConsistency(org.apache.hadoop.hbase.client.Consistency.valueOf(consistency)) if consistency + get.setReplicaId(replicaId) if replicaId + + # Call hbase for the results + result = @table.get(get) + return nil if result.isEmpty + + # Print out results. Result can be Cell or RowResult. + res = {} + result.list.each do |kv| + family = String.from_java_bytes(kv.getFamily) + qualifier = org.apache.hadoop.hbase.util.Bytes::toStringBinary(kv.getQualifier) + + column = "#{family}:#{qualifier}" + value = to_string(column, kv, maxlength) + + if block_given? + yield(column, value) + else + res[column] = value + end + end + + # If block given, we've yielded all the results, otherwise just return them + return ((block_given?) ? nil : res) + end + + #---------------------------------------------------------------------------------------------- + # Fetches and decodes a counter value from hbase + def _get_counter_internal(row, column) + family, qualifier = parse_column_name(column.to_s) + # Format get request + get = org.apache.hadoop.hbase.client.Get.new(row.to_s.to_java_bytes) + get.addColumn(family, qualifier) + get.setMaxVersions(1) + + # Call hbase + result = @table.get(get) + return nil if result.isEmpty + + # Fetch cell value + cell = result.list[0] + org.apache.hadoop.hbase.util.Bytes::toLong(cell.getValue) + end + + def _hash_to_scan(args) + if args.any? + enablemetrics = args["ALL_METRICS"].nil? ? false : args["ALL_METRICS"] + enablemetrics = enablemetrics || !args["METRICS"].nil? + filter = args["FILTER"] + startrow = args["STARTROW"] || '' + stoprow = args["STOPROW"] + rowprefixfilter = args["ROWPREFIXFILTER"] + timestamp = args["TIMESTAMP"] + columns = args["COLUMNS"] || args["COLUMN"] || [] + # If CACHE_BLOCKS not set, then default 'true'. + cache_blocks = args["CACHE_BLOCKS"].nil? ? true: args["CACHE_BLOCKS"] + cache = args["CACHE"] || 0 + reversed = args["REVERSED"] || false + versions = args["VERSIONS"] || 1 + timerange = args[TIMERANGE] + raw = args["RAW"] || false + attributes = args[ATTRIBUTES] + authorizations = args[AUTHORIZATIONS] + consistency = args[CONSISTENCY] + # Normalize column names + columns = [columns] if columns.class == String + limit = args["LIMIT"] || -1 + unless columns.kind_of?(Array) + raise ArgumentError.new("COLUMNS must be specified as a String or an Array") + end + + scan = if stoprow + org.apache.hadoop.hbase.client.Scan.new(startrow.to_java_bytes, stoprow.to_java_bytes) + else + org.apache.hadoop.hbase.client.Scan.new(startrow.to_java_bytes) + end + + # This will overwrite any startrow/stoprow settings + scan.setRowPrefixFilter(rowprefixfilter.to_java_bytes) if rowprefixfilter + + # Clear converters from last scan. + @converters.clear() + + columns.each do |c| + family, qualifier = parse_column_name(c.to_s) + if qualifier + scan.addColumn(family, qualifier) + else + scan.addFamily(family) + end + end + + unless filter.class == String + scan.setFilter(filter) + else + scan.setFilter( + org.apache.hadoop.hbase.filter.ParseFilter.new.parseFilterString(filter.to_java_bytes)) + end + + scan.setScanMetricsEnabled(enablemetrics) if enablemetrics + scan.setTimeStamp(timestamp) if timestamp + scan.setCacheBlocks(cache_blocks) + scan.setReversed(reversed) + scan.setCaching(cache) if cache > 0 + scan.setMaxVersions(versions) if versions > 1 + scan.setTimeRange(timerange[0], timerange[1]) if timerange + scan.setRaw(raw) + scan.setCaching(limit) if limit > 0 + set_attributes(scan, attributes) if attributes + set_authorizations(scan, authorizations) if authorizations + scan.setConsistency(org.apache.hadoop.hbase.client.Consistency.valueOf(consistency)) if consistency + else + scan = org.apache.hadoop.hbase.client.Scan.new + end + + scan + end + + def _get_scanner(args) + @table.getScanner(_hash_to_scan(args)) + end + + #---------------------------------------------------------------------------------------------- + # Scans whole table or a range of keys and returns rows matching specific criteria + def _scan_internal(args = {}, scan = nil) + raise(ArgumentError, "Args should be a Hash") unless args.kind_of?(Hash) + raise(ArgumentError, "Scan argument should be org.apache.hadoop.hbase.client.Scan") \ + unless scan == nil || scan.kind_of?(org.apache.hadoop.hbase.client.Scan) + + limit = args["LIMIT"] || -1 + maxlength = args.delete("MAXLENGTH") || -1 + count = 0 + res = {} + + # Start the scanner + scan = scan == nil ? _hash_to_scan(args) : scan + scanner = @table.getScanner(scan) + iter = scanner.iterator + + # Iterate results + while iter.hasNext + row = iter.next + key = org.apache.hadoop.hbase.util.Bytes::toStringBinary(row.getRow) + + row.list.each do |kv| + family = String.from_java_bytes(kv.getFamily) + qualifier = org.apache.hadoop.hbase.util.Bytes::toStringBinary(kv.getQualifier) + + column = "#{family}:#{qualifier}" + cell = to_string(column, kv, maxlength) + + if block_given? + yield(key, "column=#{column}, #{cell}") + else + res[key] ||= {} + res[key][column] = cell + end + end + + # One more row processed + count += 1 + if limit > 0 && count >= limit + # If we reached the limit, exit before the next call to hasNext + break + end + end + + scanner.close() + return ((block_given?) ? count : res) + end + + # Apply OperationAttributes to puts/scans/gets + def set_attributes(oprattr, attributes) + raise(ArgumentError, "Attributes must be a Hash type") unless attributes.kind_of?(Hash) + for k,v in attributes + v = v.to_s unless v.nil? + oprattr.setAttribute(k.to_s, v.to_java_bytes) + end + end + + def set_cell_permissions(op, permissions) + raise(ArgumentError, "Permissions must be a Hash type") unless permissions.kind_of?(Hash) + map = java.util.HashMap.new + permissions.each do |user,perms| + map.put(user.to_s, org.apache.hadoop.hbase.security.access.Permission.new( + perms.to_java_bytes)) + end + op.setACL(map) + end + + def set_cell_visibility(oprattr, visibility) + oprattr.setCellVisibility( + org.apache.hadoop.hbase.security.visibility.CellVisibility.new( + visibility.to_s)) + end + + def set_authorizations(oprattr, authorizations) + raise(ArgumentError, "Authorizations must be a Array type") unless authorizations.kind_of?(Array) + auths = [ authorizations ].flatten.compact + oprattr.setAuthorizations( + org.apache.hadoop.hbase.security.visibility.Authorizations.new( + auths.to_java(:string))) + end + + def set_op_ttl(op, ttl) + op.setTTL(ttl.to_java(:long)) + end + + #---------------------------- + # Add general administration utilities to the shell + # each of the names below adds this method name to the table + # by callling the corresponding method in the shell + # Add single method utilities to the current class + # Generally used for admin functions which just have one name and take the table name + def self.add_admin_utils(*args) + args.each do |method| + define_method method do |*method_args| + @shell.command(method, @name, *method_args) + end + end + end + + #Add the following admin utilities to the table + add_admin_utils :enable, :disable, :flush, :drop, :describe, :snapshot + + #---------------------------- + #give the general help for the table + # or the named command + def help (command = nil) + #if there is a command, get the per-command help from the shell + if command + begin + return @shell.help_command(command) + rescue NoMethodError + puts "Command \'#{command}\' does not exist. Please see general table help." + return nil + end + end + return @shell.help('table_help') + end + + # Table to string + def to_s + cl = self.class() + return "#{cl} - #{@name}" + end + + # Standard ruby call to get the return value for an object + # overriden here so we get sane semantics for printing a table on return + def inspect + to_s + end + + #---------------------------------------------------------------------------------------- + # Helper methods + + # Returns a list of column names in the table + def get_all_columns + @table.table_descriptor.getFamilies.map do |family| + "#{family.getNameAsString}:" + end + end + + # Checks if current table is one of the 'meta' tables + def is_meta_table? + org.apache.hadoop.hbase.TableName::META_TABLE_NAME.equals(@table.getName()) + end + + # Returns family and (when has it) qualifier for a column name + def parse_column_name(column) + split = org.apache.hadoop.hbase.KeyValue.parseColumn(column.to_java_bytes) + set_converter(split) if split.length > 1 + return split[0], (split.length > 1) ? split[1] : nil + end + + # Make a String of the passed kv + # Intercept cells whose format we know such as the info:regioninfo in hbase:meta + def to_string(column, kv, maxlength = -1) + if is_meta_table? + if column == 'info:regioninfo' or column == 'info:splitA' or column == 'info:splitB' + hri = org.apache.hadoop.hbase.HRegionInfo.parseFromOrNull(kv.getValue) + return "timestamp=%d, value=%s" % [kv.getTimestamp, hri.toString] + end + if column == 'info:serverstartcode' + if kv.getValue.length > 0 + str_val = org.apache.hadoop.hbase.util.Bytes.toLong(kv.getValue) + else + str_val = org.apache.hadoop.hbase.util.Bytes.toStringBinary(kv.getValue) + end + return "timestamp=%d, value=%s" % [kv.getTimestamp, str_val] + end + end + + if kv.isDelete + val = "timestamp=#{kv.getTimestamp}, type=#{org.apache.hadoop.hbase.KeyValue::Type::codeToType(kv.getType)}" + else + val = "timestamp=#{kv.getTimestamp}, value=#{convert(column, kv)}" + end + (maxlength != -1) ? val[0, maxlength] : val + end + + def convert(column, kv) + #use org.apache.hadoop.hbase.util.Bytes as the default class + klazz_name = 'org.apache.hadoop.hbase.util.Bytes' + #use org.apache.hadoop.hbase.util.Bytes::toStringBinary as the default convertor + converter = 'toStringBinary' + if @converters.has_key?(column) + # lookup the CONVERTER for certain column - "cf:qualifier" + matches = /c\((.+)\)\.(.+)/.match(@converters[column]) + if matches.nil? + # cannot match the pattern of 'c(className).functionname' + # use the default klazz_name + converter = @converters[column] + else + klazz_name = matches[1] + converter = matches[2] + end + end + method = eval(klazz_name).method(converter) + return method.call(kv.getValue) # apply the converter + end + + # if the column spec contains CONVERTER information, to get rid of :CONVERTER info from column pair. + # 1. return back normal column pair as usual, i.e., "cf:qualifier[:CONVERTER]" to "cf" and "qualifier" only + # 2. register the CONVERTER information based on column spec - "cf:qualifier" + def set_converter(column) + family = String.from_java_bytes(column[0]) + parts = org.apache.hadoop.hbase.KeyValue.parseColumn(column[1]) + if parts.length > 1 + @converters["#{family}:#{String.from_java_bytes(parts[0])}"] = String.from_java_bytes(parts[1]) + column[1] = parts[0] + end + end + + #---------------------------------------------------------------------------------------------- + # Get the split points for the table + def _get_splits_internal() + locator = @table.getRegionLocator() + splits = locator.getAllRegionLocations(). + map{|i| Bytes.toStringBinary(i.getRegionInfo().getStartKey)}.delete_if{|k| k == ""} + locator.close() + puts("Total number of splits = %s" % [splits.size + 1]) + return splits + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/visibility_labels.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/visibility_labels.rb new file mode 100644 index 00000000000..676903f719e --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hbase/visibility_labels.rb @@ -0,0 +1,159 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +include Java +java_import org.apache.hadoop.hbase.security.visibility.VisibilityClient +java_import org.apache.hadoop.hbase.security.visibility.VisibilityConstants +java_import org.apache.hadoop.hbase.util.Bytes + +module Hbase + class VisibilityLabelsAdmin + + def initialize(admin, formatter) + @admin = admin + @config = @admin.getConfiguration() + @formatter = formatter + end + + def close + @admin.close + end + + def add_labels(*args) + visibility_feature_available? + # Normalize args + if args.kind_of?(Array) + labels = [ args ].flatten.compact + end + if labels.size() == 0 + raise(ArgumentError, "Arguments cannot be null") + end + + begin + response = VisibilityClient.addLabels(@config, labels.to_java(:string)) + if response.nil? + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") + end + labelsWithException = "" + list = response.getResultList() + list.each do |result| + if result.hasException() + labelsWithException += Bytes.toString(result.getException().getValue().toByteArray()) + end + end + if labelsWithException.length > 0 + raise(ArgumentError, labelsWithException) + end + end + end + + def set_auths(user, *args) + visibility_feature_available? + # Normalize args + if args.kind_of?(Array) + auths = [ args ].flatten.compact + end + + begin + response = VisibilityClient.setAuths(@config, auths.to_java(:string), user) + if response.nil? + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") + end + labelsWithException = "" + list = response.getResultList() + list.each do |result| + if result.hasException() + labelsWithException += Bytes.toString(result.getException().getValue().toByteArray()) + end + end + if labelsWithException.length > 0 + raise(ArgumentError, labelsWithException) + end + end + end + + def get_auths(user) + visibility_feature_available? + begin + response = VisibilityClient.getAuths(@config, user) + if response.nil? + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") + end + return response.getAuthList + end + end + + def list_labels(regex = ".*") + visibility_feature_available? + begin + response = VisibilityClient.listLabels(@config, regex) + if response.nil? + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") + end + return response.getLabelList + end + end + + def clear_auths(user, *args) + visibility_feature_available? + # Normalize args + if args.kind_of?(Array) + auths = [ args ].flatten.compact + end + + begin + response = VisibilityClient.clearAuths(@config, auths.to_java(:string), user) + if response.nil? + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") + end + labelsWithException = "" + list = response.getResultList() + list.each do |result| + if result.hasException() + labelsWithException += Bytes.toString(result.getException().getValue().toByteArray()) + end + end + if labelsWithException.length > 0 + raise(ArgumentError, labelsWithException) + end + end + end + + # Make sure that lables table is available + def visibility_feature_available?() + caps = [] + begin + # Try the getSecurityCapabilities API where supported. + caps = @admin.getSecurityCapabilities + rescue + # If we are unable to use getSecurityCapabilities, fall back with a check for + # deployment of the labels table + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") unless \ + exists?(VisibilityConstants::LABELS_TABLE_NAME) + return + end + raise(ArgumentError, "DISABLED: Visibility labels feature is not available") unless \ + caps.include? org.apache.hadoop.hbase.client.security.SecurityCapability::CELL_VISIBILITY + end + + # Does table exist? + def exists?(table_name) + @admin.tableExists(table_name) + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hirb.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hirb.rb new file mode 100644 index 00000000000..1c4f8aec892 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/hirb.rb @@ -0,0 +1,267 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# File passed to org.jruby.Main by bin/hbase. Pollutes jirb with hbase imports +# and hbase commands and then loads jirb. Outputs a banner that tells user +# where to find help, shell version, and loads up a custom hirb. +# +# In noninteractive mode, runs commands from stdin until completion or an error. +# On success will exit with status 0, on any problem will exit non-zero. Callers +# should only rely on "not equal to 0", because the current error exit code of 1 +# will likely be updated to diffentiate e.g. invalid commands, incorrect args, +# permissions, etc. + +# TODO: Interrupt a table creation or a connection to a bad master. Currently +# has to time out. Below we've set down the retries for rpc and hbase but +# still can be annoying (And there seem to be times when we'll retry for +# ever regardless) +# TODO: Add support for listing and manipulating catalog tables, etc. +# TODO: Encoding; need to know how to go from ruby String to UTF-8 bytes + +# Run the java magic include and import basic HBase types that will help ease +# hbase hacking. +include Java + +# Some goodies for hirb. Should these be left up to the user's discretion? +require 'irb/completion' +require 'pathname' + +# Add the directory names in hbase.jruby.sources commandline option +# to the ruby load path so I can load up my HBase ruby modules +# sources = java.lang.System.getProperty('hbase.ruby.sources') +# $LOAD_PATH.unshift Pathname.new(sources) + +$LOAD_PATH.unshift 'uri:classloader:/hbase-ruby' + +args_input_by_env = java.lang.System.getProperty('hbase.ruby.args') +if args_input_by_env.nil? + args = Array.new +else + args = args_input_by_env.split("$") +end + +# +# FIXME: Switch args processing to getopt +# +# See if there are args for this shell. If any, read and then strip from ARGV +# so they don't go through to irb. Output shell 'usage' if user types '--help' +cmdline_help = < exception + message = exception.to_s + # exception unwrapping in shell means we'll have to handle Java exceptions + # as a special case in order to format them properly. + if exception.kind_of? java.lang.Exception + $stderr.puts "java exception" + message = exception.get_message + end + # Include the 'ERROR' string to try to make transition easier for scripts that + # may have already been relying on grepping output. + puts "ERROR #{exception.class}: #{message}" + if $fullBacktrace + # re-raising the will include a backtrace and exit. + raise exception + else + exit 1 + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/irb/hirb.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/irb/hirb.rb new file mode 100644 index 00000000000..4d6d2771728 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/irb/hirb.rb @@ -0,0 +1,59 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +require 'rbconfig' + +module IRB + WINDOZE = RbConfig::CONFIG['host_os'] =~ /mswin|mingw/ + + # Subclass of IRB so can intercept methods + class HIRB < Irb + def initialize + # This is ugly. Our 'help' method above provokes the following message + # on irb construction: 'irb: warn: can't alias help from irb_help.' + # Below, we reset the output so its pointed at /dev/null during irb + # construction just so this message does not come out after we emit + # the banner. Other attempts at playing with the hash of methods + # down in IRB didn't seem to work. I think the worst thing that can + # happen is the shell exiting because of failed IRB construction with + # no error (though we're not blanking STDERR) + begin + # Map the '/dev/null' according to the runing platform + # Under Windows platform the 'dev/null' is not fully compliant with unix, + # and the 'NUL' object need to be use instead. + devnull = "/dev/null" + devnull = "NUL" if WINDOZE + f = File.open(devnull, "w") + $stdout = f + super + ensure + f.close() + $stdout = STDOUT + end + end + + def output_value + # Suppress output if last_value is 'nil' + # Otherwise, when user types help, get ugly 'nil' + # after all output. + if @context.last_value != nil + super + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell.rb new file mode 100644 index 00000000000..dcbe6e62d38 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell.rb @@ -0,0 +1,427 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Shell commands module +module Shell + @@commands = {} + def self.commands + @@commands + end + + @@command_groups = {} + def self.command_groups + @@command_groups + end + + def self.load_command(name, group, aliases=[]) + return if commands[name] + + # Register command in the group + raise ArgumentError, "Unknown group: #{group}" unless command_groups[group] + command_groups[group][:commands] << name + + # Load command + begin + require "shell/commands/#{name}" + klass_name = name.to_s.gsub(/(?:^|_)(.)/) { $1.upcase } # camelize + commands[name] = eval("Commands::#{klass_name}") + aliases.each do |an_alias| + commands[an_alias] = commands[name] + end + rescue => e + raise "Can't load hbase shell command: #{name}. Error: #{e}\n#{e.backtrace.join("\n")}" + end + end + + def self.load_command_group(group, opts) + raise ArgumentError, "No :commands for group #{group}" unless opts[:commands] + + command_groups[group] = { + :commands => [], + :command_names => opts[:commands], + :full_name => opts[:full_name] || group, + :comment => opts[:comment] + } + + all_aliases = opts[:aliases] || {} + + opts[:commands].each do |command| + aliases = all_aliases[command] || [] + load_command(command, group, aliases) + end + end + + #---------------------------------------------------------------------- + class Shell + attr_accessor :hbase + attr_accessor :formatter + attr_accessor :interactive + alias interactive? interactive + + @debug = false + attr_accessor :debug + + def initialize(hbase, formatter, interactive=true) + self.hbase = hbase + self.formatter = formatter + self.interactive = interactive + end + + def hbase_admin + @hbase_admin ||= hbase.admin(formatter) + end + + def hbase_table(name) + hbase.table(name, self) + end + + def hbase_replication_admin + @hbase_replication_admin ||= hbase.replication_admin(formatter) + end + + def hbase_security_admin + @hbase_security_admin ||= hbase.security_admin(formatter) + end + + def hbase_visibility_labels_admin + @hbase_visibility_labels_admin ||= hbase.visibility_labels_admin(formatter) + end + + def hbase_quotas_admin + @hbase_quotas_admin ||= hbase.quotas_admin(formatter) + end + + def export_commands(where) + ::Shell.commands.keys.each do |cmd| + # here where is the IRB namespace + # this method just adds the call to the specified command + # which just references back to 'this' shell object + # a decently extensible way to add commands + where.send :instance_eval, <<-EOF + def #{cmd}(*args) + ret = @shell.command('#{cmd}', *args) + puts + return ret + end + EOF + end + end + + def command_instance(command) + ::Shell.commands[command.to_s].new(self) + end + + #call the method 'command' on the specified command + def command(command, *args) + internal_command(command, :command, *args) + end + + #call a specific internal method in the command instance + # command - name of the command to call + # method_name - name of the method on the command to call. Defaults to just 'command' + # args - to be passed to the named method + def internal_command(command, method_name= :command, *args) + command_instance(command).command_safe(self.debug,method_name, *args) + end + + def print_banner + puts "HBase Shell; enter 'help' for list of supported commands." + puts 'Type "exit" to leave the HBase Shell' + print 'Version ' + command('version') + puts + end + + def help_multi_command(command) + puts "Command: #{command}" + puts command_instance(command).help + puts + return nil + end + + def help_command(command) + puts command_instance(command).help + return nil + end + + def help_group(group_name) + group = ::Shell.command_groups[group_name.to_s] + group[:commands].sort.each { |cmd| help_multi_command(cmd) } + if group[:comment] + puts '-' * 80 + puts + puts group[:comment] + puts + end + return nil + end + + def help(command = nil) + if command + return help_command(command) if ::Shell.commands[command.to_s] + return help_group(command) if ::Shell.command_groups[command.to_s] + puts "ERROR: Invalid command or command group name: #{command}" + puts + end + + puts help_header + puts + puts 'COMMAND GROUPS:' + ::Shell.command_groups.each do |name, group| + puts " Group name: " + name + puts " Commands: " + group[:command_names].sort.join(', ') + puts + end + unless command + puts 'SHELL USAGE:' + help_footer + end + return nil + end + + def help_header + return "HBase Shell, version #{org.apache.hadoop.hbase.util.VersionInfo.getVersion()}, " + + "r#{org.apache.hadoop.hbase.util.VersionInfo.getRevision()}, " + + "#{org.apache.hadoop.hbase.util.VersionInfo.getDate()}" + "\n" + + "Type 'help \"COMMAND\"', (e.g. 'help \"get\"' -- the quotes are necessary) for help on a specific command.\n" + + "Commands are grouped. Type 'help \"COMMAND_GROUP\"', (e.g. 'help \"general\"') for help on a command group." + end + + def help_footer + puts <<-HERE +Quote all names in HBase Shell such as table and column names. Commas delimit +command parameters. Type after entering a command to run it. +Dictionaries of configuration used in the creation and alteration of tables are +Ruby Hashes. They look like this: + + {'key1' => 'value1', 'key2' => 'value2', ...} + +and are opened and closed with curley-braces. Key/values are delimited by the +'=>' character combination. Usually keys are predefined constants such as +NAME, VERSIONS, COMPRESSION, etc. Constants do not need to be quoted. Type +'Object.constants' to see a (messy) list of all constants in the environment. + +If you are using binary keys or values and need to enter them in the shell, use +double-quote'd hexadecimal representation. For example: + + hbase> get 't1', "key\\x03\\x3f\\xcd" + hbase> get 't1', "key\\003\\023\\011" + hbase> put 't1', "test\\xef\\xff", 'f1:', "\\x01\\x33\\x40" + +The HBase shell is the (J)Ruby IRB with the above HBase-specific commands added. +For more on the HBase Shell, see http://hbase.apache.org/book.html + HERE + end + end +end + +# Load commands base class +require 'shell/commands' + +# Load all commands +Shell.load_command_group( + 'general', + :full_name => 'GENERAL HBASE SHELL COMMANDS', + :commands => %w[ + status + version + table_help + whoami + ] +) + +Shell.load_command_group( + 'ddl', + :full_name => 'TABLES MANAGEMENT COMMANDS', + :commands => %w[ + alter + create + describe + disable + disable_all + is_disabled + drop + drop_all + enable + enable_all + is_enabled + exists + list + show_filters + alter_status + alter_async + get_table + locate_region + ], + :aliases => { + 'describe' => ['desc'] + } +) + +Shell.load_command_group( + 'namespace', + :full_name => 'NAMESPACE MANAGEMENT COMMANDS', + :commands => %w[ + create_namespace + drop_namespace + alter_namespace + describe_namespace + list_namespace + list_namespace_tables + ] +) + +Shell.load_command_group( + 'dml', + :full_name => 'DATA MANIPULATION COMMANDS', + :commands => %w[ + count + delete + deleteall + get + get_counter + incr + put + scan + truncate + truncate_preserve + append + get_splits + ] +) + +Shell.load_command_group( + 'tools', + :full_name => 'HBASE SURGERY TOOLS', + :comment => "WARNING: Above commands are for 'experts'-only as misuse can damage an install", + :commands => %w[ + assign + balancer + balance_switch + balancer_enabled + normalize + normalizer_switch + normalizer_enabled + close_region + compact + flush + major_compact + move + split + merge_region + unassign + zk_dump + wal_roll + catalogjanitor_run + catalogjanitor_switch + catalogjanitor_enabled + compact_rs + trace + ], + # TODO remove older hlog_roll command + :aliases => { + 'wal_roll' => ['hlog_roll'] + } +) + +Shell.load_command_group( + 'replication', + :full_name => 'CLUSTER REPLICATION TOOLS', + :comment => "In order to use these tools, hbase.replication must be true.", + :commands => %w[ + add_peer + remove_peer + list_peers + enable_peer + disable_peer + show_peer_tableCFs + set_peer_tableCFs + list_replicated_tables + append_peer_tableCFs + remove_peer_tableCFs + enable_table_replication + disable_table_replication + ] +) + +Shell.load_command_group( + 'snapshots', + :full_name => 'CLUSTER SNAPSHOT TOOLS', + :commands => %w[ + snapshot + clone_snapshot + restore_snapshot + delete_snapshot + delete_all_snapshot + list_snapshots + ] +) + +Shell.load_command_group( + 'configuration', + :full_name => 'ONLINE CONFIGURATION TOOLS', + :commands => %w[ + update_config + update_all_config + ] +) + +Shell.load_command_group( + 'quotas', + :full_name => 'CLUSTER QUOTAS TOOLS', + :commands => %w[ + set_quota + list_quotas + ] +) + +Shell.load_command_group( + 'security', + :full_name => 'SECURITY TOOLS', + :comment => "NOTE: Above commands are only applicable if running with the AccessController coprocessor", + :commands => %w[ + list_security_capabilities + grant + revoke + user_permission + ] +) + +Shell.load_command_group( + 'procedures', + :full_name => 'PROCEDURES MANAGEMENT', + :commands => %w[ + abort_procedure + list_procedures + ] +) + +Shell.load_command_group( + 'visibility labels', + :full_name => 'VISIBILITY LABEL TOOLS', + :comment => "NOTE: Above commands are only applicable if running with the VisibilityController coprocessor", + :commands => %w[ + add_labels + list_labels + set_auths + get_auths + clear_auths + set_visibility + ] +) diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands.rb new file mode 100644 index 00000000000..b18063e66ab --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands.rb @@ -0,0 +1,152 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Command + + def initialize(shell) + @shell = shell + end + + #wrap an execution of cmd to catch hbase exceptions + # cmd - command name to execture + # args - arguments to pass to the command + def command_safe(debug, cmd = :command, *args) + # send is internal ruby method to call 'cmd' with *args + #(everything is a message, so this is just the formal semantics to support that idiom) + translate_hbase_exceptions(*args) { send(cmd,*args) } + rescue => e + rootCause = e + while rootCause != nil && rootCause.respond_to?(:cause) && rootCause.cause != nil + rootCause = rootCause.cause + end + if @shell.interactive? + puts + puts "ERROR: #{rootCause}" + puts "Backtrace: #{rootCause.backtrace.join("\n ")}" if debug + puts + puts "Here is some help for this command:" + puts help + puts + else + raise rootCause + end + end + + def admin + @shell.hbase_admin + end + + def table(name) + @shell.hbase_table(name) + end + + def replication_admin + @shell.hbase_replication_admin + end + + def security_admin + @shell.hbase_security_admin + end + + def visibility_labels_admin + @shell.hbase_visibility_labels_admin + end + + def quotas_admin + @shell.hbase_quotas_admin + end + + #---------------------------------------------------------------------- + + def formatter + @shell.formatter + end + + def format_simple_command + now = Time.now + yield + formatter.header + formatter.footer(now) + end + + def format_and_return_simple_command + now = Time.now + ret = yield + formatter.header + formatter.footer(now) + return ret + end + + def translate_hbase_exceptions(*args) + yield + rescue => e + # Since exceptions will be thrown from the java code, 'e' will always be NativeException. + # Check for the original java exception and use it if present. + raise e unless e.respond_to?(:cause) && e.cause != nil + cause = e.cause + + # let individual command handle exceptions first + if self.respond_to?(:handle_exceptions) + self.handle_exceptions(cause, *args) + end + # Global HBase exception handling below if not handled by respective command above + if cause.kind_of?(org.apache.hadoop.hbase.TableNotFoundException) then + raise "Unknown table #{args.first}!" + end + if cause.kind_of?(org.apache.hadoop.hbase.UnknownRegionException) then + raise "Unknown region #{args.first}!" + end + if cause.kind_of?(org.apache.hadoop.hbase.NamespaceNotFoundException) then + raise "Unknown namespace #{args.first}!" + end + if cause.kind_of?(org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException) then + raise "Unknown snapshot #{args.first}!" + end + if cause.kind_of?(org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException) then + exceptions = cause.getCauses + exceptions.each do |exception| + if exception.kind_of?(org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException) then + valid_cols = table(args.first).get_all_columns.map { |c| c + '*' } + raise "Unknown column family! Valid column names: #{valid_cols.join(", ")}" + end + end + end + if cause.kind_of?(org.apache.hadoop.hbase.TableExistsException) then + raise "Table already exists: #{args.first}!" + end + # To be safe, here only AccessDeniedException is considered. In future + # we might support more in more generic approach when possible. + if cause.kind_of?(org.apache.hadoop.hbase.security.AccessDeniedException) then + str = java.lang.String.new("#{cause}") + # Error message is merged with stack trace, reference StringUtils.stringifyException + # This is to parse and get the error message from the whole. + strs = str.split("\n") + if strs.size > 0 then + raise "#{strs[0]}" + end + end + + # Throw the other exception which hasn't been handled above + raise e + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/abort_procedure.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/abort_procedure.rb new file mode 100644 index 00000000000..6f77ab7059c --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/abort_procedure.rb @@ -0,0 +1,51 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AbortProcedure < Command + def help + return <<-EOF +Given a procedure Id (and optional boolean may_interrupt_if_running parameter, +default is true), abort a procedure in hbase. Use with caution. Some procedures +might not be abortable. For experts only. + +If this command is accepted and the procedure is in the process of aborting, +it will return true; if the procedure could not be aborted (eg. procedure +does not exist, or procedure already completed or abort will cause corruption), +this command will return false. + +Examples: + + hbase> abort_procedure proc_id + hbase> abort_procedure proc_id, true + hbase> abort_procedure proc_id, false +EOF + end + + def command(proc_id, may_interrupt_if_running=nil) + format_simple_command do + formatter.row([ + admin.abort_procedure?(proc_id, may_interrupt_if_running).to_s + ]) + end + end + end + end +end \ No newline at end of file diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/add_labels.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/add_labels.rb new file mode 100644 index 00000000000..65a1140f240 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/add_labels.rb @@ -0,0 +1,40 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AddLabels < Command + def help + return <<-EOF +Add a set of visibility labels. +Syntax : add_labels [label1, label2] + +For example: + + hbase> add_labels ['SECRET','PRIVATE'] +EOF + end + + def command(*args) + format_simple_command do + visibility_labels_admin.add_labels(args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/add_peer.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/add_peer.rb new file mode 100644 index 00000000000..be010416445 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/add_peer.rb @@ -0,0 +1,70 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AddPeer< Command + def help + return <<-EOF +A peer can either be another HBase cluster or a custom replication endpoint. In either case an id +must be specified to identify the peer. + +For a HBase cluster peer, a cluster key must be provided and is composed like this: +hbase.zookeeper.quorum:hbase.zookeeper.property.clientPort:zookeeper.znode.parent +This gives a full path for HBase to connect to another HBase cluster. An optional parameter for +table column families identifies which column families will be replicated to the peer cluster. +Examples: + + hbase> add_peer '1', "server1.cie.com:2181:/hbase" + hbase> add_peer '2', "zk1,zk2,zk3:2182:/hbase-prod" + hbase> add_peer '3', "zk4,zk5,zk6:11000:/hbase-test", "table1; table2:cf1; table3:cf1,cf2" + hbase> add_peer '4', CLUSTER_KEY => "server1.cie.com:2181:/hbase" + hbase> add_peer '5', CLUSTER_KEY => "server1.cie.com:2181:/hbase", + TABLE_CFS => { "table1" => [], "table2" => ["cf1"], "table3" => ["cf1", "cf2"] } + +For a custom replication endpoint, the ENDPOINT_CLASSNAME can be provided. Two optional arguments +are DATA and CONFIG which can be specified to set different either the peer_data or configuration +for the custom replication endpoint. Table column families is optional and can be specified with +the key TABLE_CFS. + + hbase> add_peer '6', ENDPOINT_CLASSNAME => 'org.apache.hadoop.hbase.MyReplicationEndpoint' + hbase> add_peer '7', ENDPOINT_CLASSNAME => 'org.apache.hadoop.hbase.MyReplicationEndpoint', + DATA => { "key1" => 1 } + hbase> add_peer '8', ENDPOINT_CLASSNAME => 'org.apache.hadoop.hbase.MyReplicationEndpoint', + CONFIG => { "config1" => "value1", "config2" => "value2" } + hbase> add_peer '9', ENDPOINT_CLASSNAME => 'org.apache.hadoop.hbase.MyReplicationEndpoint', + DATA => { "key1" => 1 }, CONFIG => { "config1" => "value1", "config2" => "value2" }, + hbase> add_peer '10', ENDPOINT_CLASSNAME => 'org.apache.hadoop.hbase.MyReplicationEndpoint', + TABLE_CFS => { "table1" => [], "table2" => ["cf1"], "table3" => ["cf1", "cf2"] } + hbase> add_peer '11', ENDPOINT_CLASSNAME => 'org.apache.hadoop.hbase.MyReplicationEndpoint', + DATA => { "key1" => 1 }, CONFIG => { "config1" => "value1", "config2" => "value2" }, + TABLE_CFS => { "table1" => [], "table2" => ["cf1"], "table3" => ["cf1", "cf2"] } + +Note: Either CLUSTER_KEY or ENDPOINT_CLASSNAME must be specified but not both. +EOF + end + + def command(id, args = {}, peer_tableCFs = nil) + format_simple_command do + replication_admin.add_peer(id, args, peer_tableCFs) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter.rb new file mode 100644 index 00000000000..2c3aa6f872b --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter.rb @@ -0,0 +1,101 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Alter < Command + def help + return <<-EOF +Alter a table. If the "hbase.online.schema.update.enable" property is set to +false, then the table must be disabled (see help 'disable'). If the +"hbase.online.schema.update.enable" property is set to true, tables can be +altered without disabling them first. Altering enabled tables has caused problems +in the past, so use caution and test it before using in production. + +You can use the alter command to add, +modify or delete column families or change table configuration options. +Column families work in a similar way as the 'create' command. The column family +specification can either be a name string, or a dictionary with the NAME attribute. +Dictionaries are described in the output of the 'help' command, with no arguments. + +For example, to change or add the 'f1' column family in table 't1' from +current value to keep a maximum of 5 cell VERSIONS, do: + + hbase> alter 't1', NAME => 'f1', VERSIONS => 5 + +You can operate on several column families: + + hbase> alter 't1', 'f1', {NAME => 'f2', IN_MEMORY => true}, {NAME => 'f3', VERSIONS => 5} + +To delete the 'f1' column family in table 'ns1:t1', use one of: + + hbase> alter 'ns1:t1', NAME => 'f1', METHOD => 'delete' + hbase> alter 'ns1:t1', 'delete' => 'f1' + +You can also change table-scope attributes like MAX_FILESIZE, READONLY, +MEMSTORE_FLUSHSIZE, DURABILITY, etc. These can be put at the end; +for example, to change the max size of a region to 128MB, do: + + hbase> alter 't1', MAX_FILESIZE => '134217728' + +You can add a table coprocessor by setting a table coprocessor attribute: + + hbase> alter 't1', + 'coprocessor'=>'hdfs:///foo.jar|com.foo.FooRegionObserver|1001|arg1=1,arg2=2' + +Since you can have multiple coprocessors configured for a table, a +sequence number will be automatically appended to the attribute name +to uniquely identify it. + +The coprocessor attribute must match the pattern below in order for +the framework to understand how to load the coprocessor classes: + + [coprocessor jar file location] | class name | [priority] | [arguments] + +You can also set configuration settings specific to this table or column family: + + hbase> alter 't1', CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'} + hbase> alter 't1', {NAME => 'f2', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}} + +You can also remove a table-scope attribute: + + hbase> alter 't1', METHOD => 'table_att_unset', NAME => 'MAX_FILESIZE' + + hbase> alter 't1', METHOD => 'table_att_unset', NAME => 'coprocessor$1' + +You can also set REGION_REPLICATION: + + hbase> alter 't1', {REGION_REPLICATION => 2} + +There could be more than one alteration in one command: + + hbase> alter 't1', { NAME => 'f1', VERSIONS => 3 }, + { MAX_FILESIZE => '134217728' }, { METHOD => 'delete', NAME => 'f2' }, + OWNER => 'johndoe', METADATA => { 'mykey' => 'myvalue' } +EOF + end + + def command(table, *args) + format_simple_command do + admin.alter(table, true, *args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_async.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_async.rb new file mode 100644 index 00000000000..bddff010a28 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_async.rb @@ -0,0 +1,65 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AlterAsync < Command + def help + return <<-EOF +Alter column family schema, does not wait for all regions to receive the +schema changes. Pass table name and a dictionary specifying new column +family schema. Dictionaries are described on the main help command output. +Dictionary must include name of column family to alter. For example, + +To change or add the 'f1' column family in table 't1' from defaults +to instead keep a maximum of 5 cell VERSIONS, do: + + hbase> alter_async 't1', NAME => 'f1', VERSIONS => 5 + +To delete the 'f1' column family in table 'ns1:t1', do: + + hbase> alter_async 'ns1:t1', NAME => 'f1', METHOD => 'delete' + +or a shorter version: + + hbase> alter_async 'ns1:t1', 'delete' => 'f1' + +You can also change table-scope attributes like MAX_FILESIZE +MEMSTORE_FLUSHSIZE, READONLY, and DEFERRED_LOG_FLUSH. + +For example, to change the max size of a family to 128MB, do: + + hbase> alter 't1', METHOD => 'table_att', MAX_FILESIZE => '134217728' + +There could be more than one alteration in one command: + + hbase> alter 't1', {NAME => 'f1'}, {NAME => 'f2', METHOD => 'delete'} + +To check if all the regions have been updated, use alter_status +EOF + end + + def command(table, *args) + format_simple_command do + admin.alter(table, false, *args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_namespace.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_namespace.rb new file mode 100644 index 00000000000..a16e10d8ffa --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_namespace.rb @@ -0,0 +1,44 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AlterNamespace < Command + def help + return <<-EOF +Alter namespace properties. + +To add/modify a property: + + hbase> alter_namespace 'ns1', {METHOD => 'set', 'PROPERTY_NAME' => 'PROPERTY_VALUE'} + +To delete a property: + + hbase> alter_namespace 'ns1', {METHOD => 'unset', NAME=>'PROPERTY_NAME'} +EOF + end + + def command(namespace, *args) + format_simple_command do + admin.alter_namespace(namespace, *args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_status.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_status.rb new file mode 100644 index 00000000000..72f044dbbaa --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/alter_status.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AlterStatus < Command + def help + return <<-EOF +Get the status of the alter command. Indicates the number of regions of the +table that have received the updated schema +Pass table name. + +hbase> alter_status 't1' +hbase> alter_status 'ns1:t1' +EOF + end + def command(table) + admin.alter_status(table) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/append.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/append.rb new file mode 100644 index 00000000000..a0ef36d28e0 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/append.rb @@ -0,0 +1,52 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Append < Command + def help + return <<-EOF +Appends a cell 'value' at specified table/row/column coordinates. + + hbase> append 't1', 'r1', 'c1', 'value', ATTRIBUTES=>{'mykey'=>'myvalue'} + hbase> append 't1', 'r1', 'c1', 'value', {VISIBILITY=>'PRIVATE|SECRET'} + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.append 'r1', 'c1', 'value', ATTRIBUTES=>{'mykey'=>'myvalue'} + hbase> t.append 'r1', 'c1', 'value', {VISIBILITY=>'PRIVATE|SECRET'} +EOF + end + + def command(table, row, column, value, args={}) + append(table(table), row, column, value, args) + end + + def append(table, row, column, value, args={}) + format_simple_command do + table._append_internal(row, column, value, args) + end + end + end + end +end + +#add incr comamnd to Table +::Hbase::Table.add_shell_command("append") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/append_peer_tableCFs.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/append_peer_tableCFs.rb new file mode 100644 index 00000000000..3919b203170 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/append_peer_tableCFs.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class AppendPeerTableCFs< Command + def help + return <<-EOF +Append a replicable table-cf config for the specified peer +Examples: + + # append a table / table-cf to be replicable for a peer + hbase> append_peer_tableCFs '2', "table4:cfA,cfB" + +EOF + end + + def command(id, table_cfs) + format_simple_command do + replication_admin.append_peer_tableCFs(id, table_cfs) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/assign.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/assign.rb new file mode 100644 index 00000000000..448a5460b4e --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/assign.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Assign < Command + def help + return <<-EOF +Assign a region. Use with caution. If region already assigned, +this command will do a force reassign. For experts only. +Examples: + + hbase> assign 'REGIONNAME' + hbase> assign 'ENCODED_REGIONNAME' +EOF + end + + def command(region_name) + format_simple_command do + admin.assign(region_name) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balance_switch.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balance_switch.rb new file mode 100644 index 00000000000..4d7778daa41 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balance_switch.rb @@ -0,0 +1,42 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class BalanceSwitch < Command + def help + return <<-EOF +Enable/Disable balancer. Returns previous balancer state. +Examples: + + hbase> balance_switch true + hbase> balance_switch false +EOF + end + + def command(enableDisable) + format_simple_command do + formatter.row([ + admin.balance_switch(enableDisable)? "true" : "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balancer.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balancer.rb new file mode 100644 index 00000000000..c329eced4b9 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balancer.rb @@ -0,0 +1,40 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Balancer < Command + def help + return <<-EOF +Trigger the cluster balancer. Returns true if balancer ran and was able to +tell the region servers to unassign all the regions to balance (the re-assignment itself is async). +Otherwise false (Will not run if regions in transition). +EOF + end + + def command() + format_simple_command do + formatter.row([ + admin.balancer()? "true": "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balancer_enabled.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balancer_enabled.rb new file mode 100644 index 00000000000..3b2f5c64fe5 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/balancer_enabled.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env hbase-jruby +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Prints the current balancer status + +module Shell + module Commands + class BalancerEnabled < Command + def help + return <<-EOF +Query the balancer's state. +Examples: + + hbase> balancer_enabled +EOF + end + + def command() + format_simple_command do + formatter.row([ + admin.balancer_enabled?.to_s + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_enabled.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_enabled.rb new file mode 100644 index 00000000000..b310c3a7ab0 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_enabled.rb @@ -0,0 +1,40 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CatalogjanitorEnabled < Command + def help + return <<-EOF +Query for the CatalogJanitor state (enabled/disabled?) +Examples: + + hbase> catalogjanitor_enabled +EOF + end + + def command() + format_simple_command do + formatter.row([ + admin.catalogjanitor_enabled()? "true" : "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_run.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_run.rb new file mode 100644 index 00000000000..03426cb4f14 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_run.rb @@ -0,0 +1,37 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CatalogjanitorRun < Command + def help + return <<-EOF +Catalog janitor command to run the (garbage collection) scan from command line. + + hbase> catalogjanitor_run + +EOF + end + def command() + format_simple_command do + admin.catalogjanitor_run() + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_switch.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_switch.rb new file mode 100644 index 00000000000..fce1925eecb --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/catalogjanitor_switch.rb @@ -0,0 +1,41 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CatalogjanitorSwitch < Command + def help + return <<-EOF +Enable/Disable CatalogJanitor. Returns previous CatalogJanitor state. +Examples: + + hbase> catalogjanitor_switch true + hbase> catalogjanitor_switch false +EOF + end + + def command(enableDisable) + format_simple_command do + formatter.row([ + admin.catalogjanitor_switch(enableDisable)? "true" : "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/clear_auths.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/clear_auths.rb new file mode 100644 index 00000000000..8553fa64713 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/clear_auths.rb @@ -0,0 +1,40 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ClearAuths < Command + def help + return <<-EOF +Clear visibility labels from a user or group +Syntax : clear_auths 'user',[label1, label2] + +For example: + + hbase> clear_auths 'user1', ['SECRET','PRIVATE'] + hbase> clear_auths '@group1', ['SECRET','PRIVATE'] +EOF + end + + def command(user, *args) + format_simple_command do + visibility_labels_admin.clear_auths(user, args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/clone_snapshot.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/clone_snapshot.rb new file mode 100644 index 00000000000..beedad197e8 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/clone_snapshot.rb @@ -0,0 +1,52 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CloneSnapshot < Command + def help + return <<-EOF +Create a new table by cloning the snapshot content. +There're no copies of data involved. +And writing on the newly created table will not influence the snapshot data. + +Examples: + hbase> clone_snapshot 'snapshotName', 'tableName' + hbase> clone_snapshot 'snapshotName', 'namespace:tableName' +EOF + end + + def command(snapshot_name, table) + format_simple_command do + admin.clone_snapshot(snapshot_name, table) + end + end + + def handle_exceptions(cause, *args) + if cause.kind_of?(org.apache.hadoop.hbase.TableExistsException) then + tableName = args[1] + raise "Table already exists: #{tableName}!" + end + if cause.is_a?(org.apache.hadoop.hbase.NamespaceNotFoundException) + namespace_name = args[1].split(':')[0] + raise "Unknown namespace: #{namespace_name}!" + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/close_region.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/close_region.rb new file mode 100644 index 00000000000..9e2900c31be --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/close_region.rb @@ -0,0 +1,61 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CloseRegion < Command + def help + return <<-EOF +Close a single region. Ask the master to close a region out on the cluster +or if 'SERVER_NAME' is supplied, ask the designated hosting regionserver to +close the region directly. Closing a region, the master expects 'REGIONNAME' +to be a fully qualified region name. When asking the hosting regionserver to +directly close a region, you pass the regions' encoded name only. A region +name looks like this: + + TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. +or + Namespace:TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. + +The trailing period is part of the regionserver name. A region's encoded name +is the hash at the end of a region name; e.g. 527db22f95c8a9e0116f0cc13c680396 +(without the period). A 'SERVER_NAME' is its host, port plus startcode. For +example: host187.example.com,60020,1289493121758 (find servername in master ui +or when you do detailed status in shell). This command will end up running +close on the region hosting regionserver. The close is done without the +master's involvement (It will not know of the close). Once closed, region will +stay closed. Use assign to reopen/reassign. Use unassign or move to assign +the region elsewhere on cluster. Use with caution. For experts only. +Examples: + + hbase> close_region 'REGIONNAME' + hbase> close_region 'REGIONNAME', 'SERVER_NAME' + hbase> close_region 'ENCODED_REGIONNAME' + hbase> close_region 'ENCODED_REGIONNAME', 'SERVER_NAME' +EOF + end + + def command(region_name, server = nil) + format_simple_command do + admin.close_region(region_name, server) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/compact.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/compact.rb new file mode 100644 index 00000000000..adeffa2d034 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/compact.rb @@ -0,0 +1,48 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Compact < Command + def help + return <<-EOF + Compact all regions in passed table or pass a region row + to compact an individual region. You can also compact a single column + family within a region. + Examples: + Compact all regions in a table: + hbase> compact 'ns1:t1' + hbase> compact 't1' + Compact an entire region: + hbase> compact 'r1' + Compact only a column family within a region: + hbase> compact 'r1', 'c1' + Compact a column family within a table: + hbase> compact 't1', 'c1' + EOF + end + + def command(table_or_region_name, family = nil) + format_simple_command do + admin.compact(table_or_region_name, family) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/compact_rs.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/compact_rs.rb new file mode 100644 index 00000000000..0ecdd21113f --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/compact_rs.rb @@ -0,0 +1,43 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CompactRs < Command + def help + return <<-EOF + Compact all regions on passed regionserver. + Examples: + Compact all regions on a regionserver: + hbase> compact_rs 'host187.example.com,60020' + or + hbase> compact_rs 'host187.example.com,60020,1289493121758' + Major compact all regions on a regionserver: + hbase> compact_rs 'host187.example.com,60020,1289493121758', true + EOF + end + + def command(regionserver, major = false) + format_simple_command do + admin.compact_regionserver(regionserver, major) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/count.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/count.rb new file mode 100644 index 00000000000..225005ee36e --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/count.rb @@ -0,0 +1,77 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Count < Command + def help + return <<-EOF +Count the number of rows in a table. Return value is the number of rows. +This operation may take a LONG time (Run '$HADOOP_HOME/bin/hadoop jar +hbase.jar rowcount' to run a counting mapreduce job). Current count is shown +every 1000 rows by default. Count interval may be optionally specified. Scan +caching is enabled on count scans by default. Default cache size is 10 rows. +If your rows are small in size, you may want to increase this +parameter. Examples: + + hbase> count 'ns1:t1' + hbase> count 't1' + hbase> count 't1', INTERVAL => 100000 + hbase> count 't1', CACHE => 1000 + hbase> count 't1', INTERVAL => 10, CACHE => 1000 + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding commands would be: + + hbase> t.count + hbase> t.count INTERVAL => 100000 + hbase> t.count CACHE => 1000 + hbase> t.count INTERVAL => 10, CACHE => 1000 +EOF + end + + def command(table, params = {}) + count(table(table), params) + end + + def count(table, params = {}) + # If the second parameter is an integer, then it is the old command syntax + params = { 'INTERVAL' => params } if params.kind_of?(Fixnum) + + # Merge params with defaults + params = { + 'INTERVAL' => 1000, + 'CACHE' => 10 + }.merge(params) + + # Call the counter method + now = Time.now + formatter.header + count = table._count_internal(params['INTERVAL'].to_i, params['CACHE'].to_i) do |cnt, row| + formatter.row([ "Current count: #{cnt}, row: #{row}" ]) + end + formatter.footer(now, count) + return count + end + end + end +end + +#Add the method table.count that calls count.count +::Hbase::Table.add_shell_command("count") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/create.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/create.rb new file mode 100644 index 00000000000..a5a125e15b1 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/create.rb @@ -0,0 +1,73 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Create < Command + def help + return <<-EOF +Creates a table. Pass a table name, and a set of column family +specifications (at least one), and, optionally, table configuration. +Column specification can be a simple string (name), or a dictionary +(dictionaries are described below in main help output), necessarily +including NAME attribute. +Examples: + +Create a table with namespace=ns1 and table qualifier=t1 + hbase> create 'ns1:t1', {NAME => 'f1', VERSIONS => 5} + +Create a table with namespace=default and table qualifier=t1 + hbase> create 't1', {NAME => 'f1'}, {NAME => 'f2'}, {NAME => 'f3'} + hbase> # The above in shorthand would be the following: + hbase> create 't1', 'f1', 'f2', 'f3' + hbase> create 't1', {NAME => 'f1', VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true} + hbase> create 't1', {NAME => 'f1', CONFIGURATION => {'hbase.hstore.blockingStoreFiles' => '10'}} + +Table configuration options can be put at the end. +Examples: + + hbase> create 'ns1:t1', 'f1', SPLITS => ['10', '20', '30', '40'] + hbase> create 't1', 'f1', SPLITS => ['10', '20', '30', '40'] + hbase> create 't1', 'f1', SPLITS_FILE => 'splits.txt', OWNER => 'johndoe' + hbase> create 't1', {NAME => 'f1', VERSIONS => 5}, METADATA => { 'mykey' => 'myvalue' } + hbase> # Optionally pre-split the table into NUMREGIONS, using + hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname) + hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit'} + hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO => 'HexStringSplit', REGION_REPLICATION => 2, CONFIGURATION => {'hbase.hregion.scan.loadColumnFamiliesOnDemand' => 'true'}} + hbase> create 't1', {NAME => 'f1', DFS_REPLICATION => 1} + +You can also keep around a reference to the created table: + + hbase> t1 = create 't1', 'f1' + +Which gives you a reference to the table named 't1', on which you can then +call methods. +EOF + end + + def command(table, *args) + format_simple_command do + ret = admin.create(table, *args) + end + #and then return the table you just created + table(table) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/create_namespace.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/create_namespace.rb new file mode 100644 index 00000000000..adb6897b719 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/create_namespace.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class CreateNamespace < Command + def help + return <<-EOF +Create namespace; pass namespace name, +and optionally a dictionary of namespace configuration. +Examples: + + hbase> create_namespace 'ns1' + hbase> create_namespace 'ns1', {'PROPERTY_NAME'=>'PROPERTY_VALUE'} +EOF + end + + def command(namespace, *args) + format_simple_command do + admin.create_namespace(namespace, *args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete.rb new file mode 100644 index 00000000000..dc5f9f9da0b --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete.rb @@ -0,0 +1,59 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Delete < Command + def help + return <<-EOF +Put a delete cell value at specified table/row/column and optionally +timestamp coordinates. Deletes must match the deleted cell's +coordinates exactly. When scanning, a delete cell suppresses older +versions. To delete a cell from 't1' at row 'r1' under column 'c1' +marked with the time 'ts1', do: + + hbase> delete 'ns1:t1', 'r1', 'c1', ts1 + hbase> delete 't1', 'r1', 'c1', ts1 + hbase> delete 't1', 'r1', 'c1', ts1, {VISIBILITY=>'PRIVATE|SECRET'} + +The same command can also be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.delete 'r1', 'c1', ts1 + hbase> t.delete 'r1', 'c1', ts1, {VISIBILITY=>'PRIVATE|SECRET'} +EOF + end + + def command(table, row, column, + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP, args = {}) + delete(table(table), row, column, timestamp, args) + end + + def delete(table, row, column, + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP, args = {}) + format_simple_command do + table._delete_internal(row, column, timestamp, args, false) + end + end + end + end +end + +#Add the method table.delete that calls delete.delete +::Hbase::Table.add_shell_command("delete") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete_all_snapshot.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete_all_snapshot.rb new file mode 100644 index 00000000000..bc072597bb2 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete_all_snapshot.rb @@ -0,0 +1,61 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DeleteAllSnapshot < Command + def help + return <<-EOF +Delete all of the snapshots matching the given regex. Examples: + + hbase> delete_all_snapshot 's.*' + +EOF + end + + def command(regex) + formatter.header([ "SNAPSHOT", "TABLE + CREATION TIME"]) + list = admin.list_snapshot(regex) + count = list.size + list.each do |snapshot| + creation_time = Time.at(snapshot.getCreationTime() / 1000).to_s + formatter.row([ snapshot.getName, snapshot.getTable + " (" + creation_time + ")" ]) + end + puts "\nDelete the above #{count} snapshots (y/n)?" unless count == 0 + answer = 'n' + answer = gets.chomp unless count == 0 + puts "No snapshots matched the regex #{regex.to_s}" if count == 0 + return unless answer =~ /y.*/i + format_simple_command do + admin.delete_all_snapshot(regex) + end + list = admin.list_snapshot(regex) + leftOverSnapshotCount = list.size + successfullyDeleted = count - leftOverSnapshotCount + puts "#{successfullyDeleted} snapshots successfully deleted." unless successfullyDeleted == 0 + return if leftOverSnapshotCount == 0 + puts "\nFailed to delete the below #{leftOverSnapshotCount} snapshots." + formatter.header([ "SNAPSHOT", "TABLE + CREATION TIME"]) + list.each do |snapshot| + creation_time = Time.at(snapshot.getCreationTime() / 1000).to_s + formatter.row([ snapshot.getName, snapshot.getTable + " (" + creation_time + ")" ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete_snapshot.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete_snapshot.rb new file mode 100644 index 00000000000..b8c3791a540 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/delete_snapshot.rb @@ -0,0 +1,37 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DeleteSnapshot < Command + def help + return <<-EOF +Delete a specified snapshot. Examples: + + hbase> delete_snapshot 'snapshotName', +EOF + end + + def command(snapshot_name) + format_simple_command do + admin.delete_snapshot(snapshot_name) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/deleteall.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/deleteall.rb new file mode 100644 index 00000000000..46f3bfa17a3 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/deleteall.rb @@ -0,0 +1,60 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Deleteall < Command + def help + return <<-EOF +Delete all cells in a given row; pass a table name, row, and optionally +a column and timestamp. Examples: + + hbase> deleteall 'ns1:t1', 'r1' + hbase> deleteall 't1', 'r1' + hbase> deleteall 't1', 'r1', 'c1' + hbase> deleteall 't1', 'r1', 'c1', ts1 + hbase> deleteall 't1', 'r1', 'c1', ts1, {VISIBILITY=>'PRIVATE|SECRET'} + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.deleteall 'r1' + hbase> t.deleteall 'r1', 'c1' + hbase> t.deleteall 'r1', 'c1', ts1 + hbase> t.deleteall 'r1', 'c1', ts1, {VISIBILITY=>'PRIVATE|SECRET'} +EOF + end + + def command(table, row, column = nil, + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP, args = {}) + deleteall(table(table), row, column, timestamp, args) + end + + def deleteall(table, row, column = nil, + timestamp = org.apache.hadoop.hbase.HConstants::LATEST_TIMESTAMP, args = {}) + format_simple_command do + table._deleteall_internal(row, column, timestamp, args, true) + end + end + end + end +end + +#Add the method table.deleteall that calls deleteall.deleteall +::Hbase::Table.add_shell_command("deleteall") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/describe.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/describe.rb new file mode 100644 index 00000000000..bfa16cdc2da --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/describe.rb @@ -0,0 +1,50 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Describe < Command + def help + return <<-EOF +Describe the named table. For example: + hbase> describe 't1' + hbase> describe 'ns1:t1' + +Alternatively, you can use the abbreviated 'desc' for the same thing. + hbase> desc 't1' + hbase> desc 'ns1:t1' +EOF + end + + def command(table) + now = Time.now + + column_families = admin.get_column_families(table) + + formatter.header(["Table " + table.to_s + " is " + if admin.enabled?(table) then "ENABLED" else "DISABLED" end]) + formatter.row([table.to_s + admin.get_table_attributes(table)], true) + formatter.header(["COLUMN FAMILIES DESCRIPTION"]) + column_families.each do |column_family| + formatter.row([ column_family.to_s ], true) + end + formatter.footer(now) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/describe_namespace.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/describe_namespace.rb new file mode 100644 index 00000000000..cf135da5584 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/describe_namespace.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DescribeNamespace < Command + def help + return <<-EOF +Describe the named namespace. For example: + hbase> describe_namespace 'ns1' +EOF + end + + def command(namespace) + now = Time.now + + desc = admin.describe_namespace(namespace) + + formatter.header([ "DESCRIPTION" ], [ 64 ]) + formatter.row([ desc ], true, [ 64 ]) + formatter.footer(now) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable.rb new file mode 100644 index 00000000000..79bcd864f7f --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Disable < Command + def help + return <<-EOF +Start disable of named table: + hbase> disable 't1' + hbase> disable 'ns1:t1' +EOF + end + + def command(table) + format_simple_command do + admin.disable(table) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_all.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_all.rb new file mode 100644 index 00000000000..212db249a8a --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_all.rb @@ -0,0 +1,50 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DisableAll < Command + def help + return <<-EOF +Disable all of tables matching the given regex: + +hbase> disable_all 't.*' +hbase> disable_all 'ns:t.*' +hbase> disable_all 'ns:.*' +EOF + end + + def command(regex) + list = admin.list(regex) + count = list.size + list.each do |table| + formatter.row([ table ]) + end + puts "\nDisable the above #{count} tables (y/n)?" unless count == 0 + answer = 'n' + answer = gets.chomp unless count == 0 + puts "No tables matched the regex #{regex.to_s}" if count == 0 + return unless answer =~ /y.*/i + failed = admin.disable_all(regex) + puts "#{count - failed.size} tables successfully disabled" + puts "#{failed.size} tables not disabled due to an exception: #{failed.join ','}" unless failed.size == 0 + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_peer.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_peer.rb new file mode 100644 index 00000000000..416545b21fb --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_peer.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DisablePeer< Command + def help + return <<-EOF +Stops the replication stream to the specified cluster, but still +keeps track of new edits to replicate. + +Examples: + + hbase> disable_peer '1' +EOF + end + + def command(id) + format_simple_command do + replication_admin.disable_peer(id) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_table_replication.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_table_replication.rb new file mode 100644 index 00000000000..5bf96679493 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/disable_table_replication.rb @@ -0,0 +1,40 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DisableTableReplication< Command + def help + return <<-EOF +Disable a table's replication switch. + +Examples: + + hbase> disable_table_replication 'table_name' +EOF + end + + def command(table_name) + format_simple_command do + replication_admin.disable_tablerep(table_name) + end + puts "The replication swith of table '#{table_name}' successfully disabled" + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop.rb new file mode 100644 index 00000000000..fc7b1344686 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Drop < Command + def help + return <<-EOF +Drop the named table. Table must first be disabled: + hbase> drop 't1' + hbase> drop 'ns1:t1' +EOF + end + + def command(table) + format_simple_command do + admin.drop(table) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop_all.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop_all.rb new file mode 100644 index 00000000000..73398fbf6e5 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop_all.rb @@ -0,0 +1,50 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DropAll < Command + def help + return <<-EOF +Drop all of the tables matching the given regex: + +hbase> drop_all 't.*' +hbase> drop_all 'ns:t.*' +hbase> drop_all 'ns:.*' +EOF + end + + def command(regex) + list = admin.list(regex) + count = list.size + list.each do |table| + formatter.row([ table ]) + end + puts "\nDrop the above #{count} tables (y/n)?" unless count == 0 + answer = 'n' + answer = gets.chomp unless count == 0 + puts "No tables matched the regex #{regex.to_s}" if count == 0 + return unless answer =~ /y.*/i + failed = admin.drop_all(regex) + puts "#{count - failed.size} tables successfully dropped" + puts "#{failed.size} tables not dropped due to an exception: #{failed.join ','}" unless failed.size == 0 + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop_namespace.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop_namespace.rb new file mode 100644 index 00000000000..b030d279c77 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/drop_namespace.rb @@ -0,0 +1,36 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class DropNamespace < Command + def help + return <<-EOF +Drop the named namespace. The namespace must be empty. +EOF + end + + def command(namespace) + format_simple_command do + admin.drop_namespace(namespace) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable.rb new file mode 100644 index 00000000000..deeb70cce25 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Enable < Command + def help + return <<-EOF +Start enable of named table: + hbase> enable 't1' + hbase> enable 'ns1:t1' +EOF + end + + def command(table) + format_simple_command do + admin.enable(table) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_all.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_all.rb new file mode 100644 index 00000000000..2b899f2ac07 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_all.rb @@ -0,0 +1,50 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class EnableAll < Command + def help + return <<-EOF +Enable all of the tables matching the given regex: + +hbase> enable_all 't.*' +hbase> enable_all 'ns:t.*' +hbase> enable_all 'ns:.*' +EOF + end + + def command(regex) + list = admin.list(regex) + count = list.size + list.each do |table| + formatter.row([ table ]) + end + puts "\nEnable the above #{count} tables (y/n)?" unless count == 0 + answer = 'n' + answer = gets.chomp unless count == 0 + puts "No tables matched the regex #{regex.to_s}" if count == 0 + return unless answer =~ /y.*/i + failed = admin.enable_all(regex) + puts "#{count - failed.size} tables successfully enabled" + puts "#{failed.size} tables not enabled due to an exception: #{failed.join ','}" unless failed.size == 0 + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_peer.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_peer.rb new file mode 100644 index 00000000000..55136ffb7fb --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_peer.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class EnablePeer< Command + def help + return <<-EOF +Restarts the replication to the specified peer cluster, +continuing from where it was disabled. + +Examples: + + hbase> enable_peer '1' +EOF + end + + def command(id) + format_simple_command do + replication_admin.enable_peer(id) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_table_replication.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_table_replication.rb new file mode 100644 index 00000000000..15e3133a895 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/enable_table_replication.rb @@ -0,0 +1,40 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class EnableTableReplication< Command + def help + return <<-EOF +Enable a table's replication switch. + +Examples: + + hbase> enable_table_replication 'table_name' +EOF + end + + def command(table_name) + format_simple_command do + replication_admin.enable_tablerep(table_name) + end + puts "The replication swith of table '#{table_name}' successfully enabled" + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/exists.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/exists.rb new file mode 100644 index 00000000000..bacf6c94c44 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/exists.rb @@ -0,0 +1,40 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Exists < Command + def help + return <<-EOF +Does the named table exist? + hbase> exists 't1' + hbase> exists 'ns1:t1' +EOF + end + + def command(table) + format_simple_command do + formatter.row([ + "Table #{table} " + (admin.exists?(table.to_s) ? "does exist" : "does not exist") + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/flush.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/flush.rb new file mode 100644 index 00000000000..2aefec52e9d --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/flush.rb @@ -0,0 +1,41 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Flush < Command + def help + return <<-EOF +Flush all regions in passed table or pass a region row to +flush an individual region. For example: + + hbase> flush 'TABLENAME' + hbase> flush 'REGIONNAME' + hbase> flush 'ENCODED_REGIONNAME' +EOF + end + + def command(table_or_region_name) + format_simple_command do + admin.flush(table_or_region_name) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get.rb new file mode 100644 index 00000000000..1ab13cb9cac --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get.rb @@ -0,0 +1,98 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Get < Command + def help + return <<-EOF +Get row or cell contents; pass table name, row, and optionally +a dictionary of column(s), timestamp, timerange and versions. Examples: + + hbase> get 'ns1:t1', 'r1' + hbase> get 't1', 'r1' + hbase> get 't1', 'r1', {TIMERANGE => [ts1, ts2]} + hbase> get 't1', 'r1', {COLUMN => 'c1'} + hbase> get 't1', 'r1', {COLUMN => ['c1', 'c2', 'c3']} + hbase> get 't1', 'r1', {COLUMN => 'c1', TIMESTAMP => ts1} + hbase> get 't1', 'r1', {COLUMN => 'c1', TIMERANGE => [ts1, ts2], VERSIONS => 4} + hbase> get 't1', 'r1', {COLUMN => 'c1', TIMESTAMP => ts1, VERSIONS => 4} + hbase> get 't1', 'r1', {FILTER => "ValueFilter(=, 'binary:abc')"} + hbase> get 't1', 'r1', 'c1' + hbase> get 't1', 'r1', 'c1', 'c2' + hbase> get 't1', 'r1', ['c1', 'c2'] + hbase> get 't1', 'r1', {COLUMN => 'c1', ATTRIBUTES => {'mykey'=>'myvalue'}} + hbase> get 't1', 'r1', {COLUMN => 'c1', AUTHORIZATIONS => ['PRIVATE','SECRET']} + hbase> get 't1', 'r1', {CONSISTENCY => 'TIMELINE'} + hbase> get 't1', 'r1', {CONSISTENCY => 'TIMELINE', REGION_REPLICA_ID => 1} + +Besides the default 'toStringBinary' format, 'get' also supports custom formatting by +column. A user can define a FORMATTER by adding it to the column name in the get +specification. The FORMATTER can be stipulated: + + 1. either as a org.apache.hadoop.hbase.util.Bytes method name (e.g, toInt, toString) + 2. or as a custom class followed by method name: e.g. 'c(MyFormatterClass).format'. + +Example formatting cf:qualifier1 and cf:qualifier2 both as Integers: + hbase> get 't1', 'r1' {COLUMN => ['cf:qualifier1:toInt', + 'cf:qualifier2:c(org.apache.hadoop.hbase.util.Bytes).toInt'] } + +Note that you can specify a FORMATTER by column only (cf:qualifier). You cannot specify +a FORMATTER for all columns of a column family. + +The same commands also can be run on a reference to a table (obtained via get_table or +create_table). Suppose you had a reference t to table 't1', the corresponding commands +would be: + + hbase> t.get 'r1' + hbase> t.get 'r1', {TIMERANGE => [ts1, ts2]} + hbase> t.get 'r1', {COLUMN => 'c1'} + hbase> t.get 'r1', {COLUMN => ['c1', 'c2', 'c3']} + hbase> t.get 'r1', {COLUMN => 'c1', TIMESTAMP => ts1} + hbase> t.get 'r1', {COLUMN => 'c1', TIMERANGE => [ts1, ts2], VERSIONS => 4} + hbase> t.get 'r1', {COLUMN => 'c1', TIMESTAMP => ts1, VERSIONS => 4} + hbase> t.get 'r1', {FILTER => "ValueFilter(=, 'binary:abc')"} + hbase> t.get 'r1', 'c1' + hbase> t.get 'r1', 'c1', 'c2' + hbase> t.get 'r1', ['c1', 'c2'] + hbase> t.get 'r1', {CONSISTENCY => 'TIMELINE'} + hbase> t.get 'r1', {CONSISTENCY => 'TIMELINE', REGION_REPLICA_ID => 1} +EOF + end + + def command(table, row, *args) + get(table(table), row, *args) + end + + def get(table, row, *args) + now = Time.now + formatter.header(["COLUMN", "CELL"]) + + table._get_internal(row, *args) do |column, value| + formatter.row([ column, value ]) + end + + formatter.footer(now) + end + end + end +end + +#add get command to table +::Hbase::Table.add_shell_command('get') diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_auths.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_auths.rb new file mode 100644 index 00000000000..1b758ef9495 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_auths.rb @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class GetAuths < Command + def help + return <<-EOF +Get the visibility labels set for a particular user or group +Syntax : get_auths 'user' + +For example: + + hbase> get_auths 'user1' + hbase> get_auths '@group1' +EOF + end + + def command(user) + format_simple_command do + list = visibility_labels_admin.get_auths(user) + list.each do |auths| + formatter.row([org.apache.hadoop.hbase.util.Bytes::toStringBinary(auths.toByteArray)]) + end + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_counter.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_counter.rb new file mode 100644 index 00000000000..6708c6a5d31 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_counter.rb @@ -0,0 +1,54 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class GetCounter < Command + def help + return <<-EOF +Return a counter cell value at specified table/row/column coordinates. +A counter cell should be managed with atomic increment functions on HBase +and the data should be binary encoded (as long value). Example: + + hbase> get_counter 'ns1:t1', 'r1', 'c1' + hbase> get_counter 't1', 'r1', 'c1' + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.get_counter 'r1', 'c1' +EOF + end + + def command(table, row, column) + get_counter(table(table), row, column) + end + + def get_counter(table, row, column) + if cnt = table._get_counter_internal(row, column) + puts "COUNTER VALUE = #{cnt}" + else + puts "No counter found at specified coordinates" + end + end + end + end +end + +::Hbase::Table.add_shell_command('get_counter') diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_splits.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_splits.rb new file mode 100644 index 00000000000..8b6ae825834 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_splits.rb @@ -0,0 +1,46 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class GetSplits < Command + def help + return <<-EOF +Get the splits of the named table: + hbase> get_splits 't1' + hbase> get_splits 'ns1:t1' + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.get_splits +EOF + end + + def command(table) + get_splits(table(table)) + end + + def get_splits(table) + table._get_splits_internal() + end + end + end +end + +::Hbase::Table.add_shell_command("get_splits") \ No newline at end of file diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_table.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_table.rb new file mode 100644 index 00000000000..43e7c1af708 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/get_table.rb @@ -0,0 +1,47 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class GetTable < Command + def help + return <<-EOF +Get the given table name and return it as an actual object to +be manipulated by the user. See table.help for more information +on how to use the table. +Eg. + + hbase> t1 = get_table 't1' + hbase> t1 = get_table 'ns1:t1' + +returns the table named 't1' as a table object. You can then do + + hbase> t1.help + +which will then print the help for that table. +EOF + end + + def command(table, *args) + format_and_return_simple_command do + table(table) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/grant.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/grant.rb new file mode 100644 index 00000000000..4a22004d4bf --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/grant.rb @@ -0,0 +1,119 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Grant < Command + def help + return <<-EOF +Grant users specific rights. +Syntax: grant , [, [, [, ]]] +Syntax: grant , , <@namespace> + +permissions is either zero or more letters from the set "RWXCA". +READ('R'), WRITE('W'), EXEC('X'), CREATE('C'), ADMIN('A') + +Note: Groups and users are granted access in the same way, but groups are prefixed with an '@' + character. Tables and namespaces are specified the same way, but namespaces are + prefixed with an '@' character. + +For example: + + hbase> grant 'bobsmith', 'RWXCA' + hbase> grant '@admins', 'RWXCA' + hbase> grant 'bobsmith', 'RWXCA', '@ns1' + hbase> grant 'bobsmith', 'RW', 't1', 'f1', 'col1' + hbase> grant 'bobsmith', 'RW', 'ns1:t1', 'f1', 'col1' +EOF + end + + def command(*args) + + # command form is ambiguous at first argument + table_name = user = args[0] + raise(ArgumentError, "First argument should be a String") unless user.kind_of?(String) + + if args[1].kind_of?(String) + + # Original form of the command + # user in args[0] + # permissions in args[1] + # table_name in args[2] + # family in args[3] or nil + # qualifier in args[4] or nil + + permissions = args[1] + raise(ArgumentError, "Permissions are not of String type") unless permissions.kind_of?( + String) + table_name = family = qualifier = nil + table_name = args[2] # will be nil if unset + if not table_name.nil? + raise(ArgumentError, "Table name is not of String type") unless table_name.kind_of?( + String) + family = args[3] # will be nil if unset + if not family.nil? + raise(ArgumentError, "Family is not of String type") unless family.kind_of?(String) + qualifier = args[4] # will be nil if unset + if not qualifier.nil? + raise(ArgumentError, "Qualifier is not of String type") unless qualifier.kind_of?( + String) + end + end + end + format_simple_command do + security_admin.grant(user, permissions, table_name, family, qualifier) + end + + elsif args[1].kind_of?(Hash) + + # New form of the command, a cell ACL update + # table_name in args[0], a string + # a Hash mapping users (or groups) to permisisons in args[1] + # a Hash argument suitable for passing to Table#_get_scanner in args[2] + # Useful for feature testing and debugging. + + permissions = args[1] + raise(ArgumentError, "Permissions are not of Hash type") unless permissions.kind_of?(Hash) + scan = args[2] + raise(ArgumentError, "Scanner specification is not a Hash") unless scan.kind_of?(Hash) + + t = table(table_name) + now = Time.now + scanner = t._get_scanner(scan) + count = 0 + iter = scanner.iterator + while iter.hasNext + row = iter.next + row.list.each do |cell| + put = org.apache.hadoop.hbase.client.Put.new(row.getRow) + put.add(cell) + t.set_cell_permissions(put, permissions) + t.table.put(put) + end + count += 1 + end + formatter.footer(now, count) + + else + raise(ArgumentError, "Second argument should be a String or Hash") + end + + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/incr.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/incr.rb new file mode 100644 index 00000000000..d223a45aafe --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/incr.rb @@ -0,0 +1,65 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Incr < Command + def help + return <<-EOF +Increments a cell 'value' at specified table/row/column coordinates. +To increment a cell value in table 'ns1:t1' or 't1' at row 'r1' under column +'c1' by 1 (can be omitted) or 10 do: + + hbase> incr 'ns1:t1', 'r1', 'c1' + hbase> incr 't1', 'r1', 'c1' + hbase> incr 't1', 'r1', 'c1', 1 + hbase> incr 't1', 'r1', 'c1', 10 + hbase> incr 't1', 'r1', 'c1', 10, {ATTRIBUTES=>{'mykey'=>'myvalue'}} + hbase> incr 't1', 'r1', 'c1', {ATTRIBUTES=>{'mykey'=>'myvalue'}} + hbase> incr 't1', 'r1', 'c1', 10, {VISIBILITY=>'PRIVATE|SECRET'} + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.incr 'r1', 'c1' + hbase> t.incr 'r1', 'c1', 1 + hbase> t.incr 'r1', 'c1', 10, {ATTRIBUTES=>{'mykey'=>'myvalue'}} + hbase> t.incr 'r1', 'c1', 10, {VISIBILITY=>'PRIVATE|SECRET'} +EOF + end + + def command(table, row, column, value = nil, args = {}) + incr(table(table), row, column, value, args) + end + + def incr(table, row, column, value = nil, args={}) + format_simple_command do + if cnt = table._incr_internal(row, column, value, args) + puts "COUNTER VALUE = #{cnt}" + else + puts "No counter found at specified coordinates" + end + end + end + end + end +end + +#add incr comamnd to Table +::Hbase::Table.add_shell_command("incr") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/is_disabled.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/is_disabled.rb new file mode 100644 index 00000000000..6da7046ba79 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/is_disabled.rb @@ -0,0 +1,40 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class IsDisabled < Command + def help + return <<-EOF +Is named table disabled? For example: + hbase> is_disabled 't1' + hbase> is_disabled 'ns1:t1' +EOF + end + + def command(table) + format_simple_command do + formatter.row([ + admin.disabled?(table)? "true" : "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/is_enabled.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/is_enabled.rb new file mode 100644 index 00000000000..960ade75724 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/is_enabled.rb @@ -0,0 +1,40 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class IsEnabled < Command + def help + return <<-EOF +Is named table enabled? For example: + hbase> is_enabled 't1' + hbase> is_enabled 'ns1:t1' +EOF + end + + def command(table) + format_simple_command do + formatter.row([ + admin.enabled?(table)? "true" : "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list.rb new file mode 100644 index 00000000000..dce0ae2acdc --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list.rb @@ -0,0 +1,49 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class List < Command + def help + return <<-EOF +List all tables in hbase. Optional regular expression parameter could +be used to filter the output. Examples: + + hbase> list + hbase> list 'abc.*' + hbase> list 'ns:abc.*' + hbase> list 'ns:.*' +EOF + end + + def command(regex = ".*") + now = Time.now + formatter.header([ "TABLE" ]) + + list = admin.list(regex) + list.each do |table| + formatter.row([ table ]) + end + + formatter.footer(now, list.size) + return list + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_labels.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_labels.rb new file mode 100644 index 00000000000..6c7f99137b7 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_labels.rb @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListLabels < Command + def help + return <<-EOF +List the visibility labels defined in the system. +Optional regular expression parameter could be used to filter the labels being returned. +Syntax : list_labels + +For example: + + hbase> list_labels 'secret.*' + hbase> list_labels +EOF + end + + def command(regex = ".*") + format_simple_command do + list = visibility_labels_admin.list_labels(regex) + list.each do |label| + formatter.row([org.apache.hadoop.hbase.util.Bytes::toStringBinary(label.toByteArray)]) + end + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_namespace.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_namespace.rb new file mode 100644 index 00000000000..5d25604afa7 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_namespace.rb @@ -0,0 +1,46 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListNamespace < Command + def help + return <<-EOF +List all namespaces in hbase. Optional regular expression parameter could +be used to filter the output. Examples: + + hbase> list_namespace + hbase> list_namespace 'abc.*' +EOF + end + + def command(regex = ".*") + now = Time.now + formatter.header([ "NAMESPACE" ]) + + list = admin.list_namespace(regex) + list.each do |table| + formatter.row([ table ]) + end + + formatter.footer(now, list.size) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_namespace_tables.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_namespace_tables.rb new file mode 100644 index 00000000000..29e18123e6b --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_namespace_tables.rb @@ -0,0 +1,45 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListNamespaceTables < Command + def help + return <<-EOF +List all tables that are members of the namespace. +Examples: + + hbase> list_namespace_tables 'ns1' +EOF + end + + def command(namespace) + now = Time.now + formatter.header([ "TABLE" ]) + + list = admin.list_namespace_tables(namespace) + list.each do |table| + formatter.row([ table ]) + end + + formatter.footer(now, list.size) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_peers.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_peers.rb new file mode 100644 index 00000000000..cc1be044169 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_peers.rb @@ -0,0 +1,48 @@ +# +# Copyright The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListPeers< Command + def help + return <<-EOF +List all replication peer clusters. + + hbase> list_peers +EOF + end + + def command() + now = Time.now + peers = replication_admin.list_peers + + formatter.header(["PEER_ID", "CLUSTER_KEY", "STATE", "TABLE_CFS"]) + + peers.entrySet().each do |e| + state = replication_admin.get_peer_state(e.key) + tableCFs = replication_admin.show_peer_tableCFs(e.key) + formatter.row([ e.key, e.value, state, tableCFs ]) + end + + formatter.footer(now) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_procedures.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_procedures.rb new file mode 100644 index 00000000000..f407547f34c --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_procedures.rb @@ -0,0 +1,46 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListProcedures < Command + def help + return <<-EOF +List all procedures in hbase. Examples: + + hbase> list_procedures +EOF + end + + def command() + now = Time.now + formatter.header([ "Id", "Name", "State", "Start_Time", "Last_Update" ]) + + list = admin.list_procedures() + list.each do |proc| + start_time = Time.at(proc.getStartTime / 1000).to_s + last_update = Time.at(proc.getLastUpdate / 1000).to_s + formatter.row([ proc.getProcId, proc.getProcName, proc.getProcState, start_time, last_update ]) + end + + formatter.footer(now, list.size) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_quotas.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_quotas.rb new file mode 100644 index 00000000000..682bb7197c2 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_quotas.rb @@ -0,0 +1,52 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListQuotas < Command + def help + return <<-EOF +List the quota settings added to the system. +You can filter the result based on USER, TABLE, or NAMESPACE. + +For example: + + hbase> list_quotas + hbase> list_quotas USER => 'bob.*' + hbase> list_quotas USER => 'bob.*', TABLE => 't1' + hbase> list_quotas USER => 'bob.*', NAMESPACE => 'ns.*' + hbase> list_quotas TABLE => 'myTable' + hbase> list_quotas NAMESPACE => 'ns.*' +EOF + end + + def command(args = {}) + now = Time.now + formatter.header(["OWNER", "QUOTAS"]) + + #actually do the scanning + count = quotas_admin.list_quotas(args) do |row, cells| + formatter.row([ row, cells ]) + end + + formatter.footer(now, count) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_replicated_tables.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_replicated_tables.rb new file mode 100644 index 00000000000..0db1d83aeb9 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_replicated_tables.rb @@ -0,0 +1,50 @@ +# +# Copyright The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListReplicatedTables< Command + def help + return <<-EOF +List all the tables and column families replicated from this cluster + + hbase> list_replicated_tables + hbase> list_replicated_tables 'abc.*' +EOF + end + + def command(regex = ".*") + now = Time.now + + formatter.header([ "TABLE:COLUMNFAMILY", "ReplicationType" ], [ 32 ]) + list = replication_admin.list_replicated_tables(regex) + list.each do |e| + if e.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::REPLICATIONTYPE) == org.apache.hadoop.hbase.client.replication.ReplicationAdmin::REPLICATIONGLOBAL + replicateType = "GLOBAL" + else + replicateType = "unknown" + end + formatter.row([e.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::TNAME) + ":" + e.get(org.apache.hadoop.hbase.client.replication.ReplicationAdmin::CFNAME), replicateType], true, [32]) + end + formatter.footer(now) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_security_capabilities.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_security_capabilities.rb new file mode 100644 index 00000000000..922ad11a36b --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_security_capabilities.rb @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ListSecurityCapabilities < Command + def help + return <<-EOF +List supported security capabilities + +Example: + hbase> list_security_capabilities +EOF + end + + def command() + begin + list = admin.get_security_capabilities + list.each do |s| + puts s.getName + end + return list.map { |s| s.getName() } + rescue Exception => e + if e.to_s.include? "UnsupportedOperationException" + puts "ERROR: Master does not support getSecurityCapabilities" + return [] + end + raise e + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_snapshots.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_snapshots.rb new file mode 100644 index 00000000000..4e68802f8dd --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/list_snapshots.rb @@ -0,0 +1,51 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +require 'time' + +module Shell + module Commands + class ListSnapshots < Command + def help + return <<-EOF +List all snapshots taken (by printing the names and relative information). +Optional regular expression parameter could be used to filter the output +by snapshot name. + +Examples: + hbase> list_snapshots + hbase> list_snapshots 'abc.*' +EOF + end + + def command(regex = ".*") + now = Time.now + formatter.header([ "SNAPSHOT", "TABLE + CREATION TIME"]) + + list = admin.list_snapshot(regex) + list.each do |snapshot| + creation_time = Time.at(snapshot.getCreationTime() / 1000).to_s + formatter.row([ snapshot.getName, snapshot.getTable + " (" + creation_time + ")" ]) + end + + formatter.footer(now, list.size) + return list.map { |s| s.getName() } + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/locate_region.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/locate_region.rb new file mode 100644 index 00000000000..b1e8c7bf485 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/locate_region.rb @@ -0,0 +1,44 @@ +# +# Copyright The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class LocateRegion < Command + def help + return <<-EOF +Locate the region given a table name and a row-key + + hbase> locate_region 'tableName', 'key0' +EOF + end + + def command(table, row_key) + now = Time.now + + region_location = admin.locate_region(table, row_key) + hri = region_location.getRegionInfo() + + formatter.header([ "HOST", "REGION" ]) + formatter.row([region_location.getHostnamePort(), hri.toString()]) + formatter.footer(now, 1) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/major_compact.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/major_compact.rb new file mode 100644 index 00000000000..825748336cf --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/major_compact.rb @@ -0,0 +1,49 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class MajorCompact < Command + def help + return <<-EOF + Run major compaction on passed table or pass a region row + to major compact an individual region. To compact a single + column family within a region specify the region name + followed by the column family name. + Examples: + Compact all regions in a table: + hbase> major_compact 't1' + hbase> major_compact 'ns1:t1' + Compact an entire region: + hbase> major_compact 'r1' + Compact a single column family within a region: + hbase> major_compact 'r1', 'c1' + Compact a single column family within a table: + hbase> major_compact 't1', 'c1' + EOF + end + + def command(table_or_region_name, family = nil) + format_simple_command do + admin.major_compact(table_or_region_name, family) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/merge_region.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/merge_region.rb new file mode 100644 index 00000000000..6afa2e5b503 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/merge_region.rb @@ -0,0 +1,49 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class MergeRegion < Command + def help + return <<-EOF +Merge two regions. Passing 'true' as the optional third parameter will force +a merge ('force' merges regardless else merge will fail unless passed +adjacent regions. 'force' is for expert use only). + +NOTE: You must pass the encoded region name, not the full region name so +this command is a little different from other region operations. The encoded +region name is the hash suffix on region names: e.g. if the region name were +TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. then +the encoded region name portion is 527db22f95c8a9e0116f0cc13c680396 + +Examples: + + hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME' + hbase> merge_region 'ENCODED_REGIONNAME', 'ENCODED_REGIONNAME', true +EOF + end + + def command(encoded_region_a_name, encoded_region_b_name, force = 'false') + format_simple_command do + admin.merge_region(encoded_region_a_name, encoded_region_b_name, force) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/move.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/move.rb new file mode 100644 index 00000000000..e6b28288e3f --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/move.rb @@ -0,0 +1,47 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Move < Command + def help + return <<-EOF +Move a region. Optionally specify target regionserver else we choose one +at random. NOTE: You pass the encoded region name, not the region name so +this command is a little different to the others. The encoded region name +is the hash suffix on region names: e.g. if the region name were +TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. then +the encoded region name portion is 527db22f95c8a9e0116f0cc13c680396 +A server name is its host, port plus startcode. For example: +host187.example.com,60020,1289493121758 +Examples: + + hbase> move 'ENCODED_REGIONNAME' + hbase> move 'ENCODED_REGIONNAME', 'SERVER_NAME' +EOF + end + + def command(encoded_region_name, server_name = nil) + format_simple_command do + admin.move(encoded_region_name, server_name) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalize.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalize.rb new file mode 100644 index 00000000000..7e6302c8514 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalize.rb @@ -0,0 +1,45 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Normalize < Command + def help + return <<-EOF +Trigger region normalizer for all tables which have NORMALIZATION_ENABLED flag set. Returns true + if normalizer ran successfully, false otherwise. Note that this command has no effect + if region normalizer is disabled (make sure it's turned on using 'normalizer_switch' command). + + Examples: + + hbase> normalize +EOF + end + + def command() + format_simple_command do + formatter.row([ + admin.normalize()? "true": "false" + ]) + end + end + end + end +end + diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalizer_enabled.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalizer_enabled.rb new file mode 100644 index 00000000000..1121b25eb05 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalizer_enabled.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env hbase-jruby +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with this +# work for additional information regarding copyright ownership. The ASF +# licenses this file to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Prints current region normalizer status + +module Shell + module Commands + class NormalizerEnabled < Command + def help + return <<-EOF +Query the state of region normalizer. +Examples: + + hbase> normalizer_enabled +EOF + end + + def command() + format_simple_command do + formatter.row([ + admin.normalizer_enabled?.to_s + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalizer_switch.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalizer_switch.rb new file mode 100644 index 00000000000..6d959c4977a --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/normalizer_switch.rb @@ -0,0 +1,43 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class NormalizerSwitch < Command + def help + return <<-EOF +Enable/Disable region normalizer. Returns previous normalizer state. +When normalizer is enabled, it handles all tables with 'NORMALIZATION_ENABLED' => true. +Examples: + + hbase> normalizer_switch true + hbase> normalizer_switch false +EOF + end + + def command(enableDisable) + format_simple_command do + formatter.row([ + admin.normalizer_switch(enableDisable)? "true" : "false" + ]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/put.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/put.rb new file mode 100644 index 00000000000..2b47a4d5c6e --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/put.rb @@ -0,0 +1,57 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Put < Command + def help + return <<-EOF +Put a cell 'value' at specified table/row/column and optionally +timestamp coordinates. To put a cell value into table 'ns1:t1' or 't1' +at row 'r1' under column 'c1' marked with the time 'ts1', do: + + hbase> put 'ns1:t1', 'r1', 'c1', 'value' + hbase> put 't1', 'r1', 'c1', 'value' + hbase> put 't1', 'r1', 'c1', 'value', ts1 + hbase> put 't1', 'r1', 'c1', 'value', {ATTRIBUTES=>{'mykey'=>'myvalue'}} + hbase> put 't1', 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}} + hbase> put 't1', 'r1', 'c1', 'value', ts1, {VISIBILITY=>'PRIVATE|SECRET'} + +The same commands also can be run on a table reference. Suppose you had a reference +t to table 't1', the corresponding command would be: + + hbase> t.put 'r1', 'c1', 'value', ts1, {ATTRIBUTES=>{'mykey'=>'myvalue'}} +EOF + end + + def command(table, row, column, value, timestamp=nil, args = {}) + put table(table), row, column, value, timestamp, args + end + + def put(table, row, column, value, timestamp = nil, args = {}) + format_simple_command do + table._put_internal(row, column, value, timestamp, args) + end + end + end + end +end + +#Add the method table.put that calls Put.put +::Hbase::Table.add_shell_command("put") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/remove_peer.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/remove_peer.rb new file mode 100644 index 00000000000..5ae5786fc4a --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/remove_peer.rb @@ -0,0 +1,39 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class RemovePeer< Command + def help + return <<-EOF +Stops the specified replication stream and deletes all the meta +information kept about it. Examples: + + hbase> remove_peer '1' +EOF + end + + def command(id) + format_simple_command do + replication_admin.remove_peer(id) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/remove_peer_tableCFs.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/remove_peer_tableCFs.rb new file mode 100644 index 00000000000..5b15b529651 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/remove_peer_tableCFs.rb @@ -0,0 +1,42 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class RemovePeerTableCFs < Command + def help + return <<-EOF +Remove a table / table-cf from the table-cfs config for the specified peer +Examples: + + # Remove a table / table-cf from the replicable table-cfs for a peer + hbase> remove_peer_tableCFs '2', "table1" + hbase> remove_peer_tableCFs '2', "table1:cf1" + +EOF + end + + def command(id, table_cfs) + format_simple_command do + replication_admin.remove_peer_tableCFs(id, table_cfs) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/restore_snapshot.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/restore_snapshot.rb new file mode 100644 index 00000000000..4d531711bca --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/restore_snapshot.rb @@ -0,0 +1,41 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class RestoreSnapshot < Command + def help + return <<-EOF +Restore a specified snapshot. +The restore will replace the content of the original table, +bringing back the content to the snapshot state. +The table must be disabled. + +Examples: + hbase> restore_snapshot 'snapshotName' +EOF + end + + def command(snapshot_name) + format_simple_command do + admin.restore_snapshot(snapshot_name) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/revoke.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/revoke.rb new file mode 100644 index 00000000000..fbb99ebcaac --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/revoke.rb @@ -0,0 +1,49 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Revoke < Command + def help + return <<-EOF +Revoke a user's access rights. +Syntax: revoke [,
[, [, ]]] +Syntax: revoke , <@namespace> + +Note: Groups and users access are revoked in the same way, but groups are prefixed with an '@' + character. Tables and namespaces are specified the same way, but namespaces are + prefixed with an '@' character. + +For example: + + hbase> revoke 'bobsmith' + hbase> revoke '@admins' + hbase> revoke 'bobsmith', '@ns1' + hbase> revoke 'bobsmith', 't1', 'f1', 'col1' + hbase> revoke 'bobsmith', 'ns1:t1', 'f1', 'col1' +EOF + end + + def command(user, table_name=nil, family=nil, qualifier=nil) + format_simple_command do + security_admin.revoke(user, table_name, family, qualifier) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/scan.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/scan.rb new file mode 100644 index 00000000000..72785787a85 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/scan.rb @@ -0,0 +1,128 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Scan < Command + def help + return <<-EOF +Scan a table; pass table name and optionally a dictionary of scanner +specifications. Scanner specifications may include one or more of: +TIMERANGE, FILTER, LIMIT, STARTROW, STOPROW, ROWPREFIXFILTER, TIMESTAMP, +MAXLENGTH or COLUMNS, CACHE or RAW, VERSIONS, ALL_METRICS or METRICS + +If no columns are specified, all columns will be scanned. +To scan all members of a column family, leave the qualifier empty as in +'col_family'. + +The filter can be specified in two ways: +1. Using a filterString - more information on this is available in the +Filter Language document attached to the HBASE-4176 JIRA +2. Using the entire package name of the filter. + +If you wish to see metrics regarding the execution of the scan, the +ALL_METRICS boolean should be set to true. Alternatively, if you would +prefer to see only a subset of the metrics, the METRICS array can be +defined to include the names of only the metrics you care about. + +Some examples: + + hbase> scan 'hbase:meta' + hbase> scan 'hbase:meta', {COLUMNS => 'info:regioninfo'} + hbase> scan 'ns1:t1', {COLUMNS => ['c1', 'c2'], LIMIT => 10, STARTROW => 'xyz'} + hbase> scan 't1', {COLUMNS => ['c1', 'c2'], LIMIT => 10, STARTROW => 'xyz'} + hbase> scan 't1', {COLUMNS => 'c1', TIMERANGE => [1303668804, 1303668904]} + hbase> scan 't1', {REVERSED => true} + hbase> scan 't1', {ALL_METRICS => true} + hbase> scan 't1', {METRICS => ['RPC_RETRIES', 'ROWS_FILTERED']} + hbase> scan 't1', {ROWPREFIXFILTER => 'row2', FILTER => " + (QualifierFilter (>=, 'binary:xyz')) AND (TimestampsFilter ( 123, 456))"} + hbase> scan 't1', {FILTER => + org.apache.hadoop.hbase.filter.ColumnPaginationFilter.new(1, 0)} + hbase> scan 't1', {CONSISTENCY => 'TIMELINE'} +For setting the Operation Attributes + hbase> scan 't1', { COLUMNS => ['c1', 'c2'], ATTRIBUTES => {'mykey' => 'myvalue'}} + hbase> scan 't1', { COLUMNS => ['c1', 'c2'], AUTHORIZATIONS => ['PRIVATE','SECRET']} +For experts, there is an additional option -- CACHE_BLOCKS -- which +switches block caching for the scanner on (true) or off (false). By +default it is enabled. Examples: + + hbase> scan 't1', {COLUMNS => ['c1', 'c2'], CACHE_BLOCKS => false} + +Also for experts, there is an advanced option -- RAW -- which instructs the +scanner to return all cells (including delete markers and uncollected deleted +cells). This option cannot be combined with requesting specific COLUMNS. +Disabled by default. Example: + + hbase> scan 't1', {RAW => true, VERSIONS => 10} + +Besides the default 'toStringBinary' format, 'scan' supports custom formatting +by column. A user can define a FORMATTER by adding it to the column name in +the scan specification. The FORMATTER can be stipulated: + + 1. either as a org.apache.hadoop.hbase.util.Bytes method name (e.g, toInt, toString) + 2. or as a custom class followed by method name: e.g. 'c(MyFormatterClass).format'. + +Example formatting cf:qualifier1 and cf:qualifier2 both as Integers: + hbase> scan 't1', {COLUMNS => ['cf:qualifier1:toInt', + 'cf:qualifier2:c(org.apache.hadoop.hbase.util.Bytes).toInt'] } + +Note that you can specify a FORMATTER by column only (cf:qualifier). You cannot +specify a FORMATTER for all columns of a column family. + +Scan can also be used directly from a table, by first getting a reference to a +table, like such: + + hbase> t = get_table 't' + hbase> t.scan + +Note in the above situation, you can still provide all the filtering, columns, +options, etc as described above. + +EOF + end + + def command(table, args = {}) + scan(table(table), args) + end + + #internal command that actually does the scanning + def scan(table, args = {}) + now = Time.now + formatter.header(["ROW", "COLUMN+CELL"]) + + scan = table._hash_to_scan(args) + #actually do the scanning + count = table._scan_internal(args, scan) do |row, cells| + formatter.row([ row, cells ]) + end + + formatter.footer(now, count) + + # if scan metrics were enabled, print them after the results + if (scan != nil && scan.isScanMetricsEnabled()) + formatter.scan_metrics(scan.getScanMetrics(), args["METRICS"]) + end + end + end + end +end + +#Add the method table.scan that calls Scan.scan +::Hbase::Table.add_shell_command("scan") diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_auths.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_auths.rb new file mode 100644 index 00000000000..4a52eb0ef7d --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_auths.rb @@ -0,0 +1,40 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class SetAuths < Command + def help + return <<-EOF +Add a set of visibility labels for a user or group +Syntax : set_auths 'user',[label1, label2] + +For example: + + hbase> set_auths 'user1', ['SECRET','PRIVATE'] + hbase> set_auths '@group1', ['SECRET','PRIVATE'] +EOF + end + + def command(user, *args) + format_simple_command do + visibility_labels_admin.set_auths(user, args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_peer_tableCFs.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_peer_tableCFs.rb new file mode 100644 index 00000000000..3a88dbb7412 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_peer_tableCFs.rb @@ -0,0 +1,47 @@ +# +# Copyright The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class SetPeerTableCFs< Command + def help + return <<-EOF + Set the replicable table-cf config for the specified peer + Examples: + + # set all tables to be replicable for a peer + hbase> set_peer_tableCFs '1', "" + hbase> set_peer_tableCFs '1' + # set table / table-cf to be replicable for a peer, for a table without + # an explicit column-family list, all replicable column-families (with + # replication_scope == 1) will be replicated + hbase> set_peer_tableCFs '2', "table1; table2:cf1,cf2; table3:cfA,cfB" + + EOF + end + + def command(id, peer_table_cfs = nil) + format_simple_command do + replication_admin.set_peer_tableCFs(id, peer_table_cfs) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_quota.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_quota.rb new file mode 100644 index 00000000000..a638b939702 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_quota.rb @@ -0,0 +1,80 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class SetQuota < Command + def help + return <<-EOF +Set a quota for a user, table, or namespace. +Syntax : set_quota TYPE => , + +TYPE => THROTTLE +User can either set quota on read, write or on both the requests together(i.e., read+write) +The read, write, or read+write(default throttle type) request limit can be expressed using +the form 100req/sec, 100req/min and the read, write, read+write(default throttle type) limit +can be expressed using the form 100k/sec, 100M/min with (B, K, M, G, T, P) as valid size unit +and (sec, min, hour, day) as valid time unit. +Currently the throttle limit is per machine - a limit of 100req/min +means that each machine can execute 100req/min. + +For example: + + hbase> set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => '10req/sec' + hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => READ, USER => 'u1', LIMIT => '10req/sec' + + hbase> set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => '10M/sec' + hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, USER => 'u1', LIMIT => '10M/sec' + + hbase> set_quota TYPE => THROTTLE, USER => 'u1', TABLE => 't2', LIMIT => '5K/min' + hbase> set_quota TYPE => THROTTLE, USER => 'u1', NAMESPACE => 'ns2', LIMIT => NONE + + hbase> set_quota TYPE => THROTTLE, NAMESPACE => 'ns1', LIMIT => '10req/sec' + hbase> set_quota TYPE => THROTTLE, TABLE => 't1', LIMIT => '10M/sec' + hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, TABLE => 't1', LIMIT => '10M/sec' + hbase> set_quota TYPE => THROTTLE, USER => 'u1', LIMIT => NONE + hbase> set_quota TYPE => THROTTLE, THROTTLE_TYPE => WRITE, USER => 'u1', LIMIT => NONE + + hbase> set_quota USER => 'u1', GLOBAL_BYPASS => true +EOF + end + + def command(args = {}) + if args.has_key?(TYPE) + qtype = args.delete(TYPE) + case qtype + when THROTTLE + if args[LIMIT].eql? NONE + args.delete(LIMIT) + quotas_admin.unthrottle(args) + else + quotas_admin.throttle(args) + end + else + raise "Invalid TYPE argument. got " + qtype + end + elsif args.has_key?(GLOBAL_BYPASS) + quotas_admin.set_global_bypass(args.delete(GLOBAL_BYPASS), args) + else + raise "Expected TYPE argument" + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_visibility.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_visibility.rb new file mode 100644 index 00000000000..59779fb138a --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/set_visibility.rb @@ -0,0 +1,73 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class SetVisibility < Command + def help + return <<-EOF +Set the visibility expression on one or more existing cells. + +Pass table name, visibility expression, and a dictionary containing +scanner specifications. Scanner specifications may include one or more +of: TIMERANGE, FILTER, STARTROW, STOPROW, ROWPREFIXFILTER, TIMESTAMP, or COLUMNS + +If no columns are specified, all columns will be included. +To include all members of a column family, leave the qualifier empty as in +'col_family:'. + +The filter can be specified in two ways: +1. Using a filterString - more information on this is available in the +Filter Language document attached to the HBASE-4176 JIRA +2. Using the entire package name of the filter. + +Examples: + + hbase> set_visibility 't1', 'A|B', {COLUMNS => ['c1', 'c2']} + hbase> set_visibility 't1', '(A&B)|C', {COLUMNS => 'c1', + TIMERANGE => [1303668804, 1303668904]} + hbase> set_visibility 't1', 'A&B&C', {ROWPREFIXFILTER => 'row2', + FILTER => "(QualifierFilter (>=, 'binary:xyz')) AND + (TimestampsFilter ( 123, 456))"} + +This command will only affect existing cells and is expected to be mainly +useful for feature testing and functional verification. +EOF + end + + def command(table, visibility, scan) + t = table(table) + now = Time.now + scanner = t._get_scanner(scan) + count = 0 + iter = scanner.iterator + while iter.hasNext + row = iter.next + row.list.each do |cell| + put = org.apache.hadoop.hbase.client.Put.new(row.getRow) + put.add(cell) + t.set_cell_visibility(put, visibility) + t.table.put(put) + end + count += 1 + end + formatter.footer(now, count) + end + + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/show_filters.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/show_filters.rb new file mode 100644 index 00000000000..cdbd9ed9b59 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/show_filters.rb @@ -0,0 +1,49 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +java_import org.apache.hadoop.hbase.filter.ParseFilter + +module Shell + module Commands + class ShowFilters < Command + def help + return <<-EOF +Show all the filters in hbase. Example: + hbase> show_filters + + ColumnPrefixFilter + TimestampsFilter + PageFilter + ..... + KeyOnlyFilter +EOF + end + + def command( ) + now = Time.now + parseFilter = ParseFilter.new + supportedFilters = parseFilter.getSupportedFilters + + supportedFilters.each do |filter| + formatter.row([filter]) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/show_peer_tableCFs.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/show_peer_tableCFs.rb new file mode 100644 index 00000000000..037630f5949 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/show_peer_tableCFs.rb @@ -0,0 +1,37 @@ +# +# Copyright The Apache Software Foundation +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ShowPeerTableCFs< Command + def help + return <<-EOF + Show replicable table-cf config for the specified peer. + + hbase> show_peer_tableCFs '2' + EOF + end + + def command(id) + puts replication_admin.show_peer_tableCFs(id) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/snapshot.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/snapshot.rb new file mode 100644 index 00000000000..15bf298e043 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/snapshot.rb @@ -0,0 +1,38 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Snapshot < Command + def help + return <<-EOF +Take a snapshot of specified table. Examples: + + hbase> snapshot 'sourceTable', 'snapshotName' + hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH => true} +EOF + end + + def command(table, snapshot_name, *args) + format_simple_command do + admin.snapshot(table, snapshot_name, *args) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/split.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/split.rb new file mode 100644 index 00000000000..9dc424ff687 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/split.rb @@ -0,0 +1,43 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Split < Command + def help + return <<-EOF +Split entire table or pass a region to split individual region. With the +second parameter, you can specify an explicit split key for the region. +Examples: + split 'tableName' + split 'namespace:tableName' + split 'regionName' # format: 'tableName,startKey,id' + split 'tableName', 'splitKey' + split 'regionName', 'splitKey' +EOF + end + + def command(table_or_region_name, split_point = nil) + format_simple_command do + admin.split(table_or_region_name, split_point) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/status.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/status.rb new file mode 100644 index 00000000000..b22b2723987 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/status.rb @@ -0,0 +1,43 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Status < Command + def help + return <<-EOF +Show cluster status. Can be 'summary', 'simple', 'detailed', or 'replication'. The +default is 'summary'. Examples: + + hbase> status + hbase> status 'simple' + hbase> status 'summary' + hbase> status 'detailed' + hbase> status 'replication' + hbase> status 'replication', 'source' + hbase> status 'replication', 'sink' +EOF + end + + def command(format = 'summary',type = 'both') + admin.status(format, type) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/table_help.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/table_help.rb new file mode 100644 index 00000000000..e5a1858a92c --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/table_help.rb @@ -0,0 +1,33 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class TableHelp < Command + def help + return Hbase::Table.help + end + + #just print the help + def command + # call the shell to get the nice formatting there + @shell.help_command 'table_help' + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/trace.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/trace.rb new file mode 100644 index 00000000000..5e0093092f8 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/trace.rb @@ -0,0 +1,76 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +HTrace = org.apache.htrace.Trace +java_import org.apache.htrace.Sampler +java_import org.apache.hadoop.hbase.trace.SpanReceiverHost + +module Shell + module Commands + class Trace < Command + def help + return <<-EOF +Start or Stop tracing using HTrace. +Always returns true if tracing is running, otherwise false. +If the first argument is 'start', new span is started. +If the first argument is 'stop', current running span is stopped. +('stop' returns false on success.) +If the first argument is 'status', just returns if or not tracing is running. +On 'start'-ing, you can optionally pass the name of span as the second argument. +The default name of span is 'HBaseShell'. +Repeating 'start' does not start nested span. + +Examples: + + hbase> trace 'start' + hbase> trace 'status' + hbase> trace 'stop' + + hbase> trace 'start', 'MySpanName' + hbase> trace 'stop' + +EOF + end + + def command(startstop="status", spanname="HBaseShell") + format_and_return_simple_command do + trace(startstop, spanname) + end + end + + def trace(startstop, spanname) + @@receiver ||= SpanReceiverHost.getInstance(@shell.hbase.configuration) + if startstop == "start" + if not tracing? + @@tracescope = HTrace.startSpan(spanname, Sampler.ALWAYS) + end + elsif startstop == "stop" + if tracing? + @@tracescope.close() + end + end + tracing? + end + + def tracing?() + HTrace.isTracing() + end + + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/truncate.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/truncate.rb new file mode 100644 index 00000000000..b7812fbe3dd --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/truncate.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Truncate < Command + def help + return <<-EOF + Disables, drops and recreates the specified table. +EOF + end + + def command(table) + format_simple_command do + puts "Truncating '#{table}' table (it may take a while):" + admin.truncate(table) { |log| puts " - #{log}" } + end + end + + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/truncate_preserve.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/truncate_preserve.rb new file mode 100644 index 00000000000..918b23289a9 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/truncate_preserve.rb @@ -0,0 +1,38 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class TruncatePreserve < Command + def help + return <<-EOF + Disables, drops and recreates the specified table while still maintaing the previous region boundaries. +EOF + end + + def command(table) + format_simple_command do + puts "Truncating '#{table}' table (it may take a while):" + admin.truncate_preserve(table) { |log| puts " - #{log}" } + end + end + + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/unassign.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/unassign.rb new file mode 100644 index 00000000000..5eea71f6e08 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/unassign.rb @@ -0,0 +1,45 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Unassign < Command + def help + return <<-EOF +Unassign a region. Unassign will close region in current location and then +reopen it again. Pass 'true' to force the unassignment ('force' will clear +all in-memory state in master before the reassign. If results in +double assignment use hbck -fix to resolve. To be used by experts). +Use with caution. For expert use only. Examples: + + hbase> unassign 'REGIONNAME' + hbase> unassign 'REGIONNAME', true + hbase> unassign 'ENCODED_REGIONNAME' + hbase> unassign 'ENCODED_REGIONNAME', true +EOF + end + + def command(region_name, force = 'false') + format_simple_command do + admin.unassign(region_name, force) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/update_all_config.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/update_all_config.rb new file mode 100644 index 00000000000..16d3728bf3b --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/update_all_config.rb @@ -0,0 +1,39 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class UpdateAllConfig < Command + def help + return <<-EOF +Reload a subset of configuration on all servers in the cluster. See +http://hbase.apache.org/book.html#dyn_config for more details. Here is how +you would run the command in the hbase shell: + hbase> update_all_config +EOF + end + + def command() + format_simple_command do + admin.update_all_config() + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/update_config.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/update_config.rb new file mode 100644 index 00000000000..05db24ff725 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/update_config.rb @@ -0,0 +1,40 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class UpdateConfig < Command + def help + return <<-EOF +Reload a subset of configuration on server 'servername' where servername is +host, port plus startcode. For example: host187.example.com,60020,1289493121758 +See http://hbase.apache.org/book.html#dyn_config for more details. Here is how +you would run the command in the hbase shell: + hbase> update_config 'servername' +EOF + end + + def command(serverName) + format_simple_command do + admin.update_config(serverName) + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/user_permission.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/user_permission.rb new file mode 100644 index 00000000000..71b98f370a8 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/user_permission.rb @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class UserPermission < Command + def help + return <<-EOF +Show all permissions for the particular user. +Syntax : user_permission
+ +Note: A namespace must always precede with '@' character. + +For example: + + hbase> user_permission + hbase> user_permission '@ns1' + hbase> user_permission '@.*' + hbase> user_permission '@^[a-c].*' + hbase> user_permission 'table1' + hbase> user_permission 'namespace1:table1' + hbase> user_permission '.*' + hbase> user_permission '^[A-C].*' +EOF + end + + def command(table_regex=nil) + #format_simple_command do + #admin.user_permission(table_regex) + now = Time.now + formatter.header(["User", "Namespace,Table,Family,Qualifier:Permission"]) + + count = security_admin.user_permission(table_regex) do |user, permission| + formatter.row([ user, permission]) + end + + formatter.footer(now, count) + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/version.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/version.rb new file mode 100644 index 00000000000..63e9712a52d --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/version.rb @@ -0,0 +1,37 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Version < Command + def help + return <<-EOF +Output this HBase version +EOF + end + + def command + # Output version. + puts "#{org.apache.hadoop.hbase.util.VersionInfo.getVersion()}, " + + "r#{org.apache.hadoop.hbase.util.VersionInfo.getRevision()}, " + + "#{org.apache.hadoop.hbase.util.VersionInfo.getDate()}" + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/wal_roll.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/wal_roll.rb new file mode 100644 index 00000000000..0fe1870af23 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/wal_roll.rb @@ -0,0 +1,42 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +module Shell + module Commands + class WalRoll < Command + def help + return <<-EOF +Roll the log writer. That is, start writing log messages to a new file. +The name of the regionserver should be given as the parameter. A +'server_name' is the host, port plus startcode of a regionserver. For +example: host187.example.com,60020,1289493121758 (find servername in +master ui or when you do detailed status in shell) +EOF + end + + def command(server_name) + format_simple_command do + admin.wal_roll(server_name) + end + end + end + #TODO remove old HLog version + class HlogRoll < WalRoll + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/whoami.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/whoami.rb new file mode 100644 index 00000000000..3b6b32d2576 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/whoami.rb @@ -0,0 +1,42 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class Whoami < Command + def help + return <<-EOF +Show the current hbase user. +Syntax : whoami +For example: + + hbase> whoami +EOF + end + + def command() + user = org.apache.hadoop.hbase.security.User.getCurrent() + puts "#{user.toString()}" + groups = user.getGroupNames().to_a + if not groups.nil? and groups.length > 0 + puts " groups: #{groups.join(', ')}" + end + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/zk_dump.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/zk_dump.rb new file mode 100644 index 00000000000..c0b509a3c6e --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/commands/zk_dump.rb @@ -0,0 +1,34 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +module Shell + module Commands + class ZkDump < Command + def help + return <<-EOF +Dump status of HBase cluster as seen by ZooKeeper. +EOF + end + + def command + puts admin.zk_dump + end + end + end +end diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/formatter.rb b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/formatter.rb new file mode 100644 index 00000000000..7f43efa3386 --- /dev/null +++ b/linkis-engineconn-plugins/hbase/hbase-shims-1.2.1/src/main/resources/hbase-ruby/shell/formatter.rb @@ -0,0 +1,201 @@ +# +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Results formatter +module Shell + module Formatter + # Base abstract class for results formatting. + class Base + attr_reader :row_count + + def is_valid_io?(obj) + obj.instance_of?(IO) || obj == Kernel + end + + def refresh_width() + if $stdout.tty? + @max_width = Java.jline.TerminalFactory.get.getWidth + else + @max_width = 0 + end + end + + # Takes an output stream and a print width. + def initialize(opts = {}) + options = { + :output_stream => Kernel, + }.merge(opts) + + @out = options[:output_stream] + refresh_width + @row_count = 0 + + # raise an error if the stream is not valid + raise(TypeError, "Type #{@out.class} of parameter #{@out} is not IO") unless is_valid_io?(@out) + end + + def header(args = [], widths = []) + refresh_width + row(args, false, widths) if args.length > 0 + @row_count = 0 + end + + # Output a row. + # Inset is whether or not to offset row by a space. + def row(args = [], inset = true, widths = []) + # Print out nothing + return if !args || args.empty? + + # Print a string + if args.is_a?(String) + output(args) + @out.puts + return + end + + # TODO: Look at the type. Is it RowResult? + if args.length == 1 + splits = split(@max_width, dump(args[0])) + for l in splits + output(@max_width, l) + @out.puts + end + elsif args.length == 2 + if @max_width == 0 + col1width = col2width = 0 + else + col1width = (not widths or widths.length == 0) ? @max_width / 4 : @max_width * widths[0] / 100 + col2width = (not widths or widths.length < 2) ? @max_width - col1width - 2 : @max_width * widths[1] / 100 - 2 + end + splits1 = split(col1width, dump(args[0])) + splits2 = split(col2width, dump(args[1])) + biggest = (splits2.length > splits1.length)? splits2.length: splits1.length + index = 0 + while index < biggest + # Inset by one space if inset is set. + @out.print(" ") if inset + output(col1width, splits1[index]) + # Add extra space so second column lines up w/ second column output + @out.print(" ") unless inset + @out.print(" ") + output(col2width, splits2[index]) + index += 1 + @out.puts + end + else + # Print a space to set off multi-column rows + print ' ' + first = true + for e in args + @out.print " " unless first + first = false + @out.print e + end + puts + end + @row_count += 1 + end + + # Output the scan metrics. Can be filtered to output only those metrics whose keys exists + # in the metric_filter + def scan_metrics(scan_metrics = nil, metric_filter = []) + return if scan_metrics == nil + raise(ArgumentError, \ + "Argument should be org.apache.hadoop.hbase.client.metrics.ScanMetrics") \ + unless scan_metrics.kind_of?(org.apache.hadoop.hbase.client.metrics.ScanMetrics) + # prefix output with empty line + @out.puts + # save row count to restore after printing metrics + # (metrics should not count towards row count) + saved_row_count = @row_count + iter = scan_metrics.getMetricsMap().entrySet().iterator() + metric_hash = Hash.new() + # put keys in hash so they can be sorted easily + while iter.hasNext + metric = iter.next + metric_hash[metric.getKey.to_s] = metric.getValue.to_s + end + # print in alphabetical order + row(["METRIC", "VALUE"], false) + metric_hash.sort.map do |key, value| + if (not metric_filter or metric_filter.length == 0 or metric_filter.include?(key)) + row([key, value]) + end + end + + @row_count = saved_row_count + return + end + + def split(width, str) + if width == 0 + return [str] + end + result = [] + index = 0 + while index < str.length do + result << str.slice(index, width) + index += width + end + result + end + + def dump(str) + return if str.instance_of?(Integer) + # Remove double-quotes added by 'dump'. + return str + end + + def output(str) + output(@max_width, str) + end + + def output(width, str) + if str == nil + str = '' + end + if not width or width == str.length + @out.print(str) + else + @out.printf('%-*s', width, str) + end + end + + def footer(start_time = nil, row_count = nil) + return unless start_time + row_count ||= @row_count + # Only output elapsed time and row count if startTime passed + @out.puts("%d row(s) in %.4f seconds" % [row_count, Time.now - start_time]) + end + end + + + class Console < Base + end + + class XHTMLFormatter < Base + # http://www.germane-software.com/software/rexml/doc/classes/REXML/Document.html + # http://www.crummy.com/writing/RubyCookbook/test_results/75942.html + end + + class JSON < Base + end + end +end + diff --git a/linkis-engineconn-plugins/hbase/hbase-shims-2.5.3/pom.xml b/linkis-engineconn-plugins/hbase/hbase-shims-2.5.3/pom.xml index 48c83560840..9938a354947 100644 --- a/linkis-engineconn-plugins/hbase/hbase-shims-2.5.3/pom.xml +++ b/linkis-engineconn-plugins/hbase/hbase-shims-2.5.3/pom.xml @@ -98,6 +98,11 @@ org.slf4jslf4j-api + + + org.apache.hadoop + hadoop-hdfs-client + diff --git a/linkis-engineconn-plugins/hbase/pom.xml b/linkis-engineconn-plugins/hbase/pom.xml index 0e0d22baef1..5cf46a82b6d 100644 --- a/linkis-engineconn-plugins/hbase/pom.xml +++ b/linkis-engineconn-plugins/hbase/pom.xml @@ -28,7 +28,7 @@ pom hbase-core - hbase-shims-1.2.0 + hbase-shims-1.2.1 hbase-shims-1.4.3 hbase-shims-2.2.6 hbase-shims-2.5.3 diff --git a/linkis-engineconn-plugins/hive/pom.xml b/linkis-engineconn-plugins/hive/pom.xml index 28ae9dd1ce9..30e5f7b4111 100644 --- a/linkis-engineconn-plugins/hive/pom.xml +++ b/linkis-engineconn-plugins/hive/pom.xml @@ -198,6 +198,14 @@ com.google.inject guice + + org.codehaus.jettison + jettison + + + org.apache.velocity + velocity + @@ -274,9 +282,14 @@ org.apache.hadoop hadoop-common + + + org.apache.hadoop + hadoop-hdfs-client + org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs diff --git a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/creation/HiveEngineConnFactory.scala b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/creation/HiveEngineConnFactory.scala index a9b217074dc..43d845e6f3c 100644 --- a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/creation/HiveEngineConnFactory.scala +++ b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/creation/HiveEngineConnFactory.scala @@ -59,10 +59,6 @@ import scala.collection.JavaConverters._ class HiveEngineConnFactory extends ComputationSingleExecutorEngineConnFactory with Logging { - private val HIVE_QUEUE_NAME: String = "mapreduce.job.queuename" - private val BDP_QUEUE_NAME: String = "wds.linkis.rm.yarnqueue" - private val HIVE_TEZ_QUEUE_NAME: String = "tez.queue.name" - override protected def newExecutor( id: Int, engineCreationContext: EngineCreationContext, @@ -188,10 +184,10 @@ class HiveEngineConnFactory extends ComputationSingleExecutorEngineConnFactory w } .foreach { case (k, v) => logger.info(s"key is $k, value is $v") - if (BDP_QUEUE_NAME.equals(k)) { - hiveConf.set(HIVE_QUEUE_NAME, v) + if (HiveEngineConfiguration.BDP_QUEUE_NAME.equals(k)) { + hiveConf.set(HiveEngineConfiguration.HIVE_QUEUE_NAME, v) if ("tez".equals(HiveEngineConfiguration.HIVE_ENGINE_TYPE)) { - hiveConf.set(HIVE_TEZ_QUEUE_NAME, v) + hiveConf.set(HiveEngineConfiguration.HIVE_TEZ_QUEUE_NAME, v) } } else hiveConf.set(k, v) } diff --git a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConcurrentConnExecutor.scala b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConcurrentConnExecutor.scala index 9fe37eb3d98..3f2510ab678 100644 --- a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConcurrentConnExecutor.scala +++ b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConcurrentConnExecutor.scala @@ -236,6 +236,10 @@ class HiveEngineConcurrentConnExecutor( engineExecutorContext.appendStdout( s"Your hive taskId: $taskId has $numberOfJobs MR jobs to do" ) + val queueName = hiveConf.get(HiveEngineConfiguration.HIVE_QUEUE_NAME) + engineExecutorContext.appendStdout( + s"Your task will be submitted to the $queueName queue" + ) } logger.info(s"there are ${numberOfJobs} jobs.") diff --git a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala index 793a65b1eb8..b2d0196c1e9 100644 --- a/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala +++ b/linkis-engineconn-plugins/hive/src/main/scala/org/apache/linkis/engineplugin/hive/executor/HiveEngineConnExecutor.scala @@ -277,6 +277,10 @@ class HiveEngineConnExecutor( } if (numberOfMRJobs > 0) { engineExecutorContext.appendStdout(s"Your hive sql has $numberOfMRJobs MR jobs to do") + val queueName = hiveConf.get(HiveEngineConfiguration.HIVE_QUEUE_NAME) + engineExecutorContext.appendStdout( + s"Your task will be submitted to the $queueName queue" + ) } if (thread.isInterrupted) { logger.error( @@ -696,7 +700,7 @@ class HiveEngineConnExecutor( currentProps.keys.foreach { key => if (!hiveTmpConf.contains(key)) { logger.info(s"Clearing extra configuration key: $key") - sessionConf.set(key, "") + sessionConf.unset(key) } } } else { diff --git a/linkis-engineconn-plugins/impala/src/main/java/org/apache/linkis/engineplugin/impala/password/CommandPasswordCallback.java b/linkis-engineconn-plugins/impala/src/main/java/org/apache/linkis/engineplugin/impala/password/CommandPasswordCallback.java new file mode 100644 index 00000000000..dbe22743641 --- /dev/null +++ b/linkis-engineconn-plugins/impala/src/main/java/org/apache/linkis/engineplugin/impala/password/CommandPasswordCallback.java @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.engineplugin.impala.password; + +import org.apache.commons.io.IOUtils; + +import javax.security.auth.callback.PasswordCallback; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; + +public class CommandPasswordCallback extends PasswordCallback { + + private static final Charset CHARSET = Charset.defaultCharset(); + + public CommandPasswordCallback(String prompt) { + this(prompt, false); + } + + public CommandPasswordCallback(String prompt, boolean echoOn) { + super(prompt, echoOn); + } + + private static String string(InputStream inputStream) throws IOException { + try { + return IOUtils.toString(inputStream, CHARSET); + } finally { + inputStream.close(); + } + } + + private static char[] array(InputStream inputStream) throws IOException { + try { + return IOUtils.toCharArray(inputStream, CHARSET); + } finally { + inputStream.close(); + } + } + + @Override + public char[] getPassword() { + Process process = null; + String prompt = getPrompt(); + try { + process = new ProcessBuilder().command(prompt).start(); + int ret = process.waitFor(); + if (ret != 0) throw new RuntimeException(string(process.getErrorStream())); + return array(process.getInputStream()); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new RuntimeException("Failed to authenticate with command: " + prompt, e); + } finally { + if (process != null) { + process.destroy(); + } + } + } +} diff --git a/linkis-engineconn-plugins/impala/src/main/java/org/apache/linkis/engineplugin/impala/password/StaticPasswordCallback.java b/linkis-engineconn-plugins/impala/src/main/java/org/apache/linkis/engineplugin/impala/password/StaticPasswordCallback.java new file mode 100644 index 00000000000..8f69c080578 --- /dev/null +++ b/linkis-engineconn-plugins/impala/src/main/java/org/apache/linkis/engineplugin/impala/password/StaticPasswordCallback.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.engineplugin.impala.password; + +import javax.security.auth.callback.PasswordCallback; + +public class StaticPasswordCallback extends PasswordCallback { + + private final char[] password; + + public StaticPasswordCallback(String prompt, boolean echoOn) { + super(prompt, echoOn); + this.password = prompt.toCharArray(); + } + + public StaticPasswordCallback(String prompt) { + super(prompt, false); + this.password = prompt.toCharArray(); + } + + @Override + public char[] getPassword() { + return password; + } +} diff --git a/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/conf/ImpalaConfiguration.scala b/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/conf/ImpalaConfiguration.scala index 211e88c118b..d7f82a947f1 100644 --- a/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/conf/ImpalaConfiguration.scala +++ b/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/conf/ImpalaConfiguration.scala @@ -31,8 +31,7 @@ object ImpalaConfiguration { val IMPALA_USER_ISOLATION_MODE = CommonVars[Boolean]("linkis.impala.user.isolation.mode", false) - val IMPALA_ENGINE_USER = - CommonVars("linkis.impala.engine.user", StorageConfiguration.HDFS_ROOT_USER().getValue) + val IMPALA_ENGINE_USER = CommonVars[String]("linkis.impala.engine.user", "hadoop") val IMPALA_SERVERS = CommonVars[String]("linkis.impala.servers", "127.0.0.1:21050") val IMPALA_MAX_CONNECTIONS = CommonVars[Int]("linkis.impala.maxConnections", 10) @@ -57,7 +56,9 @@ object ImpalaConfiguration { val IMPALA_SASL_AUTHORIZATION_ID = CommonVars[String]("linkis.impala.sasl.authorizationId", "") val IMPALA_SASL_PROTOCOL = CommonVars[String]("linkis.impala.sasl.protocol", "LDAP") val IMPALA_SASL_PROPERTIES = CommonVars[String]("linkis.impala.sasl.properties", "") - val IMPALA_SASL_USERNAME = CommonVars("linkis.impala.sasl.username", IMPALA_ENGINE_USER.getValue) + + val IMPALA_SASL_USERNAME = CommonVars[String]("linkis.impala.sasl.username", "") + val IMPALA_SASL_PASSWORD = CommonVars[String]("linkis.impala.sasl.password", "") val IMPALA_SASL_PASSWORD_CMD = CommonVars[String]("linkis.impala.sasl.password.cmd", "") diff --git a/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/executor/ImpalaEngineConnExecutor.scala b/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/executor/ImpalaEngineConnExecutor.scala index 97613f3f94d..c96fc77297c 100644 --- a/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/executor/ImpalaEngineConnExecutor.scala +++ b/linkis-engineconn-plugins/impala/src/main/scala/org/apache/linkis/engineplugin/impala/executor/ImpalaEngineConnExecutor.scala @@ -19,10 +19,6 @@ package org.apache.linkis.engineplugin.impala.executor import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{OverloadUtils, Utils} -import org.apache.linkis.engineconn.common.password.{ - CommandPasswordCallback, - StaticPasswordCallback -} import org.apache.linkis.engineconn.computation.executor.execute.{ ConcurrentComputationExecutor, EngineExecutionContext @@ -45,6 +41,10 @@ import org.apache.linkis.engineplugin.impala.client.thrift.{ } import org.apache.linkis.engineplugin.impala.conf.ImpalaConfiguration._ import org.apache.linkis.engineplugin.impala.conf.ImpalaEngineConfig +import org.apache.linkis.engineplugin.impala.password.{ + CommandPasswordCallback, + StaticPasswordCallback +} import org.apache.linkis.governance.common.paser.SQLCodeParser import org.apache.linkis.manager.common.entity.resource.{ CommonNodeResource, @@ -204,7 +204,7 @@ class ImpalaEngineConnExecutor(override val outputPrintLimit: Int, val id: Int) var row: Row = resultSet.next() while (row != null) { - val record = new TableRecord(row.getValues) + val record = new TableRecord(row.getValues.asInstanceOf[Array[Any]]) resultSetWriter.addRecord(record) rows += 1 row = resultSet.next() diff --git a/linkis-engineconn-plugins/jdbc/pom.xml b/linkis-engineconn-plugins/jdbc/pom.xml index 8136e290802..ebe9f4faf96 100644 --- a/linkis-engineconn-plugins/jdbc/pom.xml +++ b/linkis-engineconn-plugins/jdbc/pom.xml @@ -128,6 +128,14 @@ io.netty netty + + org.apache.avro + avro + + + org.xerial.snappy + snappy-java + diff --git a/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManager.java b/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManager.java index b3a9867d387..db1c2ce9c6f 100644 --- a/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManager.java +++ b/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManager.java @@ -57,7 +57,7 @@ public class ConnectionManager { private final Map dataSourceFactories; private final JDBCDataSourceConfigurations jdbcDataSourceConfigurations; - private static volatile ConnectionManager connectionManager; + private static volatile ConnectionManager connectionManager; // NOSONAR private ScheduledExecutorService scheduledExecutorService; private Integer kinitFailCount = 0; @@ -67,9 +67,9 @@ private ConnectionManager() { } public static ConnectionManager getInstance() { - if (connectionManager == null) { - synchronized (ConnectionManager.class) { - if (connectionManager == null) { + if (connectionManager == null) { // NOSONAR + synchronized (ConnectionManager.class) { // NOSONAR + if (connectionManager == null) { // NOSONAR connectionManager = new ConnectionManager(); } } @@ -185,6 +185,9 @@ protected DataSource buildDataSource(String dbUrl, Map propertie boolean removeAbandoned = JDBCPropertiesParser.getBool( properties, JDBCEngineConnConstant.JDBC_POOL_REMOVE_ABANDONED_ENABLED, true); + boolean logAbandoned = + JDBCPropertiesParser.getBool( + properties, JDBCEngineConnConstant.JDBC_POOL_REMOVE_ABANDONED_LOG_ENABLED, true); int removeAbandonedTimeout = JDBCPropertiesParser.getInt( properties, JDBCEngineConnConstant.JDBC_POOL_REMOVE_ABANDONED_TIMEOUT, 300); @@ -200,6 +203,9 @@ protected DataSource buildDataSource(String dbUrl, Map propertie DruidDataSource datasource = new DruidDataSource(); LOG.info("Database connection address information(数据库连接地址信息)=" + dbUrl); datasource.setUrl(dbUrl); + if (dbUrl.toLowerCase().contains("oracle")) { + datasource.setValidationQuery("SELECT 1 FROM DUAL"); + } datasource.setUsername(username); if (AESUtils.LINKIS_DATASOURCE_AES_SWITCH.getValue()) { // decrypt @@ -221,6 +227,7 @@ protected DataSource buildDataSource(String dbUrl, Map propertie datasource.setPoolPreparedStatements(poolPreparedStatements); datasource.setRemoveAbandoned(removeAbandoned); datasource.setRemoveAbandonedTimeout(removeAbandonedTimeout); + datasource.setLogAbandoned(logAbandoned); if (queryTimeout > 0) { datasource.setQueryTimeout(queryTimeout); } @@ -239,6 +246,9 @@ private Connection getConnectionFromDataSource( } } } + if (url.contains("oracle")) { + ((DruidDataSource) dataSource).setValidationQuery("SELECT 1 FROM DUAL"); + } return dataSource.getConnection(); } diff --git a/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/monitor/impl/TrinoProgressMonitor.java b/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/monitor/impl/TrinoProgressMonitor.java index 3adcc673bc2..c0aaf0dc2e8 100644 --- a/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/monitor/impl/TrinoProgressMonitor.java +++ b/linkis-engineconn-plugins/jdbc/src/main/java/org/apache/linkis/manager/engineplugin/jdbc/monitor/impl/TrinoProgressMonitor.java @@ -26,7 +26,7 @@ import io.trino.jdbc.TrinoStatement; public class TrinoProgressMonitor extends ProgressMonitor { - private volatile Runnable callback; + private volatile Runnable callback; // NOSONAR private volatile double sqlProgress = 0.0; private volatile int completedSplits = 0; private volatile int totalSplits = 0; @@ -58,7 +58,7 @@ public void callback(Runnable callback) { @Override public float getSqlProgress() { - return Double.valueOf(sqlProgress).floatValue(); + return Double.valueOf(sqlProgress).floatValue(); // NOSONAR } @Override diff --git a/linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/executor/JDBCEngineConnExecutor.scala b/linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/executor/JDBCEngineConnExecutor.scala index fba3a02d276..9d05100f3dd 100644 --- a/linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/executor/JDBCEngineConnExecutor.scala +++ b/linkis-engineconn-plugins/jdbc/src/main/scala/org/apache/linkis/manager/engineplugin/jdbc/executor/JDBCEngineConnExecutor.scala @@ -18,6 +18,7 @@ package org.apache.linkis.manager.engineplugin.jdbc.executor import org.apache.linkis.common.conf.Configuration +import org.apache.linkis.common.log.LogUtils import org.apache.linkis.common.utils.{OverloadUtils, Utils} import org.apache.linkis.engineconn.computation.executor.entity.EngineConnTask import org.apache.linkis.engineconn.computation.executor.execute.{ @@ -68,6 +69,7 @@ import org.apache.linkis.storage.resultset.table.{TableMetaData, TableRecord} import org.apache.commons.collections.MapUtils import org.apache.commons.io.IOUtils import org.apache.commons.lang3.StringUtils +import org.apache.commons.lang3.exception.ExceptionUtils import org.springframework.util.CollectionUtils @@ -235,6 +237,11 @@ class JDBCEngineConnExecutor(override val outputPrintLimit: Int, val id: Int) } catch { case e: Throwable => logger.error(s"Cannot run $code", e) + // 推送堆栈信息到前端 + val errorStr = ExceptionUtils.getStackTrace(e) + engineExecutorContext.appendStdout( + LogUtils.generateERROR(s"execute code failed!: $errorStr") + ) return ErrorExecuteResponse(e.getMessage, e) } finally { connectionManager.removeStatement(taskId) @@ -374,6 +381,9 @@ class JDBCEngineConnExecutor(override val outputPrintLimit: Int, val id: Int) val data = resultSet.getObject(i + 1) match { case value: Array[Byte] => new String(resultSet.getObject(i + 1).asInstanceOf[Array[Byte]]) + case value: java.sql.Timestamp => value.toString + case value: java.sql.Time => value.toString + case value: java.sql.Date => value.toString case value: Any => resultSet.getString(i + 1) case _ => null } diff --git a/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManagerTest.java b/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManagerTest.java index 97a65c57609..9e3adeeefc6 100644 --- a/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManagerTest.java +++ b/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ConnectionManagerTest.java @@ -51,7 +51,7 @@ public void testCreateJdbcConnAndExecSql() properties.put(JDBCEngineConnConstant.JDBC_SCRIPTS_EXEC_USER, "leo_jie"); ConnectionManager connectionManager = ConnectionManager.getInstance(); Connection conn = connectionManager.getConnection("jdbc-1", properties); - Statement statement = conn.createStatement(); + Statement statement = conn.createStatement(); // NOSONAR ResultSet rs = statement.executeQuery("show databases;"); while (rs.next()) { System.out.println(rs.getObject(1)); diff --git a/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ProgressMonitorTest.java b/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ProgressMonitorTest.java index 1c7f3a2b76f..c5b35e79699 100644 --- a/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ProgressMonitorTest.java +++ b/linkis-engineconn-plugins/jdbc/src/test/java/org/apache/linkis/manager/engineplugin/jdbc/ProgressMonitorTest.java @@ -41,7 +41,7 @@ public void testProgressMonitor() throws SQLException { String url = "jdbc:trino://127.0.0.1:8080/hive/test"; Properties properties = new Properties(); properties.setProperty("user", "test"); - Connection connection = DriverManager.getConnection(url, properties); + Connection connection = DriverManager.getConnection(url, properties); // NOSONAR monitor = ProgressMonitor.attachMonitor(connection.createStatement()); Assertions.assertNotNull(monitor); diff --git a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/constant/PipeLineConstant.scala b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/constant/PipeLineConstant.scala index 77c2a3a8a13..477addcfff5 100644 --- a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/constant/PipeLineConstant.scala +++ b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/constant/PipeLineConstant.scala @@ -30,4 +30,7 @@ object PipeLineConstant { "wds.linkis.engine.pipeline.field.quote.retoch.enable" val BLANK = "BLANK" + + val PIPELINE_MASKED_CONF = "pipeline.masked.field.names" + } diff --git a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/CSVExecutor.scala b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/CSVExecutor.scala index 68b5010f177..9f4a7833fa5 100644 --- a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/CSVExecutor.scala +++ b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/CSVExecutor.scala @@ -18,6 +18,7 @@ package org.apache.linkis.manager.engineplugin.pipeline.executor import org.apache.linkis.common.io.FsPath +import org.apache.linkis.common.utils.Logging import org.apache.linkis.engineconn.computation.executor.execute.EngineExecutionContext import org.apache.linkis.manager.engineplugin.pipeline.conf.PipelineEngineConfiguration.{ PIPELINE_FIELD_QUOTE_RETOUCH_ENABLE, @@ -25,6 +26,7 @@ import org.apache.linkis.manager.engineplugin.pipeline.conf.PipelineEngineConfig PIPELINE_OUTPUT_CHARSET_STR, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH } +import org.apache.linkis.manager.engineplugin.pipeline.constant.PipeLineConstant import org.apache.linkis.manager.engineplugin.pipeline.constant.PipeLineConstant._ import org.apache.linkis.manager.engineplugin.pipeline.errorcode.PopelineErrorCodeSummary._ import org.apache.linkis.manager.engineplugin.pipeline.exception.PipeLineErrorException @@ -32,13 +34,15 @@ import org.apache.linkis.scheduler.executer.ExecuteResponse import org.apache.linkis.storage.FSFactory import org.apache.linkis.storage.csv.CSVFsWriter import org.apache.linkis.storage.source.FileSource +import org.apache.linkis.storage.utils.ResultUtils import org.apache.linkis.storage.utils.StorageConfiguration.STORAGE_RS_FILE_SUFFIX import org.apache.commons.io.IOUtils +import org.apache.commons.lang3.StringUtils import java.io.OutputStream -class CSVExecutor extends PipeLineExecutor { +class CSVExecutor extends PipeLineExecutor with Logging { override def execute( sourcePath: String, @@ -57,35 +61,58 @@ class CSVExecutor extends PipeLineExecutor { NOT_A_RESULT_SET_FILE.getErrorDesc ) } + + // Extract masked field names from options + val maskedFieldNames = options.getOrDefault(PipeLineConstant.PIPELINE_MASKED_CONF, "") + val sourceFsPath = new FsPath(sourcePath) val destFsPath = new FsPath(destPath) val sourceFs = FSFactory.getFs(sourceFsPath) sourceFs.init(null) val destFs = FSFactory.getFs(destFsPath) destFs.init(null) - val fileSource = FileSource.create(sourceFsPath, sourceFs) - if (!FileSource.isTableResultSet(fileSource)) { - throw new PipeLineErrorException( - ONLY_RESULT_CONVERTED_TO_CSV.getErrorCode, - ONLY_RESULT_CONVERTED_TO_CSV.getErrorDesc + + try { + val fileSource = FileSource.create(sourceFsPath, sourceFs) + if (!FileSource.isTableResultSet(fileSource)) { + throw new PipeLineErrorException( + ONLY_RESULT_CONVERTED_TO_CSV.getErrorCode, + ONLY_RESULT_CONVERTED_TO_CSV.getErrorDesc + ) + } + + var nullValue = options.getOrDefault(PIPELINE_OUTPUT_SHUFFLE_NULL_TYPE, "NULL") + if (BLANK.equalsIgnoreCase(nullValue)) nullValue = "" + + val outputStream: OutputStream = + destFs.write(destFsPath, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH.getValue(options)) + OutputStreamCache.osCache.put(engineExecutionContext.getJobId.get, outputStream) + + val cSVFsWriter = CSVFsWriter.getCSVFSWriter( + PIPELINE_OUTPUT_CHARSET_STR.getValue(options), + PIPELINE_FIELD_SPLIT_STR.getValue(options), + PIPELINE_FIELD_QUOTE_RETOUCH_ENABLE.getValue(options), + outputStream ) + + try { + if (StringUtils.isNotBlank(maskedFieldNames)) { + // Apply field masking if maskedFieldNames is provided + ResultUtils.dealMaskedField(maskedFieldNames, cSVFsWriter, fileSource) + } else { + // Original stream write logic + logger.info("No field masking, using stream write for CSV export") + fileSource.addParams("nullValue", nullValue).write(cSVFsWriter) + } + } finally { + IOUtils.closeQuietly(cSVFsWriter) + IOUtils.closeQuietly(fileSource) + } + } finally { + IOUtils.closeQuietly(sourceFs) + IOUtils.closeQuietly(destFs) } - var nullValue = options.getOrDefault(PIPELINE_OUTPUT_SHUFFLE_NULL_TYPE, "NULL") - if (BLANK.equalsIgnoreCase(nullValue)) nullValue = "" - val outputStream: OutputStream = - destFs.write(destFsPath, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH.getValue(options)) - OutputStreamCache.osCache.put(engineExecutionContext.getJobId.get, outputStream) - val cSVFsWriter = CSVFsWriter.getCSVFSWriter( - PIPELINE_OUTPUT_CHARSET_STR.getValue(options), - PIPELINE_FIELD_SPLIT_STR.getValue(options), - PIPELINE_FIELD_QUOTE_RETOUCH_ENABLE.getValue(options), - outputStream - ) - fileSource.addParams("nullValue", nullValue).write(cSVFsWriter) - IOUtils.closeQuietly(cSVFsWriter) - IOUtils.closeQuietly(fileSource) - IOUtils.closeQuietly(sourceFs) - IOUtils.closeQuietly(destFs) + super.execute(sourcePath, destPath, engineExecutionContext) } diff --git a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/ExcelExecutor.scala b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/ExcelExecutor.scala index 42c0e27cd29..3d6d924c4e4 100644 --- a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/ExcelExecutor.scala +++ b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/ExcelExecutor.scala @@ -18,33 +18,41 @@ package org.apache.linkis.manager.engineplugin.pipeline.executor import org.apache.linkis.common.io.FsPath -import org.apache.linkis.common.utils.ResultSetUtils +import org.apache.linkis.common.utils.{Logging, ResultSetUtils} import org.apache.linkis.engineconn.computation.executor.execute.EngineExecutionContext import org.apache.linkis.manager.engineplugin.pipeline.conf.PipelineEngineConfiguration import org.apache.linkis.manager.engineplugin.pipeline.conf.PipelineEngineConfiguration.PIPELINE_OUTPUT_ISOVERWRITE_SWITCH +import org.apache.linkis.manager.engineplugin.pipeline.constant.PipeLineConstant import org.apache.linkis.manager.engineplugin.pipeline.constant.PipeLineConstant._ import org.apache.linkis.manager.engineplugin.pipeline.errorcode.PopelineErrorCodeSummary._ import org.apache.linkis.manager.engineplugin.pipeline.exception.PipeLineErrorException import org.apache.linkis.scheduler.executer.ExecuteResponse import org.apache.linkis.storage.FSFactory -import org.apache.linkis.storage.excel.{ExcelFsWriter, StorageMultiExcelWriter} +import org.apache.linkis.storage.conf.LinkisStorageConf +import org.apache.linkis.storage.conf.LinkisStorageConf.FIELD_TRUNCATION_ENABLED +import org.apache.linkis.storage.excel.{ExcelFsWriter, StorageExcelWriter, StorageMultiExcelWriter} import org.apache.linkis.storage.fs.FileSystem import org.apache.linkis.storage.source.FileSource +import org.apache.linkis.storage.utils.ResultUtils import org.apache.commons.io.IOUtils +import org.apache.commons.lang3.StringUtils import java.io.OutputStream import java.util -import scala.collection.JavaConverters.mapAsScalaMapConverter +import scala.collection.JavaConverters._ -class ExcelExecutor extends PipeLineExecutor { +class ExcelExecutor extends PipeLineExecutor with Logging { override def execute( sourcePath: String, destPath: String, engineExecutorContext: EngineExecutionContext ): ExecuteResponse = { + // Extract masked field names from options + val maskedFieldNames = options.getOrDefault(PipeLineConstant.PIPELINE_MASKED_CONF, "") + var fileSource: FileSource = null var excelFsWriter: ExcelFsWriter = null val sourceFsPath = new FsPath(sourcePath) @@ -53,56 +61,93 @@ class ExcelExecutor extends PipeLineExecutor { sourceFs.init(null) val destFs = FSFactory.getFs(destFsPath) destFs.init(null) - val outputStream: OutputStream = - destFs.write(destFsPath, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH.getValue(options)) - val paramsMap = new util.HashMap[String, String]() - engineExecutorContext.getProperties.asScala - .filter(_._2 != null) - .map(kv => (kv._1, kv._2.toString)) - .foreach(kv => paramsMap.put(kv._1, kv._2)) - val excelAutoFormat = PipelineEngineConfiguration.EXPORT_EXCEL_AUTO_FORMAT.getValue(paramsMap) - if (sourcePath.contains(".")) { - // sourcePaht 是文件形式 - // TODO: fs 加目录判断 - if (!FileSource.isResultSet(sourcePath)) { + + try { + val outputStream: OutputStream = + destFs.write(destFsPath, PIPELINE_OUTPUT_ISOVERWRITE_SWITCH.getValue(options)) + val paramsMap = new util.HashMap[String, String]() + engineExecutorContext.getProperties.asScala + .filter(_._2 != null) + .map(kv => (kv._1, kv._2.toString)) + .foreach(kv => paramsMap.put(kv._1, kv._2)) + val excelAutoFormat = + PipelineEngineConfiguration.EXPORT_EXCEL_AUTO_FORMAT.getValue(paramsMap) + if (sourcePath.contains(".")) { + // sourcePaht 是文件形式 + // TODO: fs 加目录判断 + if (!FileSource.isResultSet(sourcePath)) { + throw new PipeLineErrorException( + NOT_A_RESULT_SET_FILE.getErrorCode, + NOT_A_RESULT_SET_FILE.getErrorDesc + ) + } + fileSource = FileSource.create(sourceFsPath, sourceFs) + excelFsWriter = ExcelFsWriter.getExcelFsWriter( + DEFAULTC_HARSET, + DEFAULT_SHEETNAME, + DEFAULT_DATEFORMATE, + outputStream, + excelAutoFormat + ) + } else { + // 目录形式 + excelFsWriter = new StorageMultiExcelWriter(outputStream, excelAutoFormat) + val fsPathListWithError = + sourceFs.asInstanceOf[FileSystem].listPathWithError(sourceFsPath) + if (fsPathListWithError == null) { + throw new PipeLineErrorException(EMPTY_DIR.getErrorCode, EMPTY_DIR.getErrorDesc) + } + val fsPathList = fsPathListWithError.getFsPaths + ResultSetUtils.sortByNameNum(fsPathList) + fileSource = FileSource.create(fsPathList.toArray(Array[FsPath]()), sourceFs) + } + if (!FileSource.isTableResultSet(fileSource)) { throw new PipeLineErrorException( - NOT_A_RESULT_SET_FILE.getErrorCode, - NOT_A_RESULT_SET_FILE.getErrorDesc + ONLY_RESULT_CONVERTED_TO_EXCEL.getErrorCode, + ONLY_RESULT_CONVERTED_TO_EXCEL.getErrorDesc ) } - fileSource = FileSource.create(sourceFsPath, sourceFs) - excelFsWriter = ExcelFsWriter.getExcelFsWriter( - DEFAULTC_HARSET, - DEFAULT_SHEETNAME, - DEFAULT_DATEFORMATE, - outputStream, - excelAutoFormat - ) - } else { - // 目录形式 - excelFsWriter = new StorageMultiExcelWriter(outputStream, excelAutoFormat) - val fsPathListWithError = sourceFs.asInstanceOf[FileSystem].listPathWithError(sourceFsPath) - if (fsPathListWithError == null) { - throw new PipeLineErrorException(EMPTY_DIR.getErrorCode, EMPTY_DIR.getErrorDesc) + + var nullValue = options.getOrDefault(PIPELINE_OUTPUT_SHUFFLE_NULL_TYPE, "NULL") + if (BLANK.equalsIgnoreCase(nullValue)) nullValue = "" + OutputStreamCache.osCache.put(engineExecutorContext.getJobId.get, outputStream) + + try { + // Apply field masking if maskedFieldNames is provided + fileSource.addParams("nullValue", nullValue) + // 如果同时提供了字段屏蔽和字段截取参数，则先执行字段屏蔽，再执行字段截取 + if (StringUtils.isNotBlank(maskedFieldNames) && FIELD_TRUNCATION_ENABLED) { + // 同时执行字段屏蔽和字段截取 + ResultUtils.applyFieldMaskingAndTruncation( + maskedFieldNames, + excelFsWriter.asInstanceOf[StorageExcelWriter], + fileSource, + LinkisStorageConf.FIELD_EXPORT_MAX_LENGTH + ) + } else if (StringUtils.isNotBlank(maskedFieldNames)) { + // 只执行字段屏蔽 + ResultUtils.dealMaskedField(maskedFieldNames, excelFsWriter, fileSource) + } else if (FIELD_TRUNCATION_ENABLED) { + // 只执行字段截取 + ResultUtils.detectAndHandle( + excelFsWriter.asInstanceOf[StorageExcelWriter], + fileSource, + LinkisStorageConf.FIELD_EXPORT_MAX_LENGTH + ) + } else { + // Original stream write logic + logger.info("No field masking, using stream write for Excel export") + fileSource.write(excelFsWriter) + } + } finally { + IOUtils.closeQuietly(excelFsWriter) + IOUtils.closeQuietly(fileSource) } - val fsPathList = fsPathListWithError.getFsPaths - ResultSetUtils.sortByNameNum(fsPathList) - fileSource = FileSource.create(fsPathList.toArray(Array[FsPath]()), sourceFs) + } finally { + IOUtils.closeQuietly(sourceFs) + IOUtils.closeQuietly(destFs) } - if (!FileSource.isTableResultSet(fileSource)) { - throw new PipeLineErrorException( - ONLY_RESULT_CONVERTED_TO_EXCEL.getErrorCode, - ONLY_RESULT_CONVERTED_TO_EXCEL.getErrorDesc - ) - } - var nullValue = options.getOrDefault(PIPELINE_OUTPUT_SHUFFLE_NULL_TYPE, "NULL") - if (BLANK.equalsIgnoreCase(nullValue)) nullValue = "" - OutputStreamCache.osCache.put(engineExecutorContext.getJobId.get, outputStream) - fileSource.addParams("nullValue", nullValue).write(excelFsWriter) - IOUtils.closeQuietly(excelFsWriter) - IOUtils.closeQuietly(fileSource) - IOUtils.closeQuietly(sourceFs) - IOUtils.closeQuietly(destFs) + super.execute(sourcePath, destPath, engineExecutorContext) } diff --git a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/PipelineEngineConnExecutor.scala b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/PipelineEngineConnExecutor.scala index da1bd0e6e13..f22e2ce6bab 100644 --- a/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/PipelineEngineConnExecutor.scala +++ b/linkis-engineconn-plugins/pipeline/src/main/scala/org/apache/linkis/manager/engineplugin/pipeline/executor/PipelineEngineConnExecutor.scala @@ -30,12 +30,14 @@ import org.apache.linkis.manager.common.entity.resource.{ } import org.apache.linkis.manager.engineplugin.common.conf.EngineConnPluginConf import org.apache.linkis.manager.engineplugin.common.util.NodeResourceUtils +import org.apache.linkis.manager.engineplugin.pipeline.constant.PipeLineConstant import org.apache.linkis.manager.engineplugin.pipeline.errorcode.PopelineErrorCodeSummary._ import org.apache.linkis.manager.engineplugin.pipeline.exception.PipeLineErrorException import org.apache.linkis.manager.label.entity.Label import org.apache.linkis.protocol.engine.JobProgressInfo import org.apache.linkis.rpc.Sender import org.apache.linkis.scheduler.executer.ExecuteResponse +import org.apache.linkis.storage.conf.LinkisStorageConf import java.util @@ -66,11 +68,25 @@ class PipelineEngineConnExecutor(val id: Int) extends ComputationExecutor with L newOptions.put(keyAndValue._1, keyAndValue._2.toString) } newOptions.asScala.foreach({ case (k, v) => logger.info(s"key is $k, value is $v") }) - val regex = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s?".r + + // Regex patterns for Pipeline syntax + val regexWithMask = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s+without\\s+\"([^\"]+)\"\\s*".r + val regexNormal = "(?i)\\s*from\\s+(\\S+)\\s+to\\s+(\\S+)\\s*".r + try { thread = Thread.currentThread() code match { - case regex(sourcePath, destPath) => + case regexWithMask(sourcePath, destPath, maskedFields) => + logger.info(s"Pipeline execution with field masking: $maskedFields") + val enhancedOptions = new util.HashMap[String, String](newOptions) + if (LinkisStorageConf.FIELD_MASKED_ENABLED) { + enhancedOptions.put(PipeLineConstant.PIPELINE_MASKED_CONF, maskedFields) + } + PipelineExecutorSelector + .select(sourcePath, destPath, enhancedOptions.asInstanceOf[util.Map[String, String]]) + .execute(sourcePath, destPath, engineExecutorContext) + case regexNormal(sourcePath, destPath) => + logger.info("Pipeline execution without field masking") PipelineExecutorSelector .select(sourcePath, destPath, newOptions.asInstanceOf[util.Map[String, String]]) .execute(sourcePath, destPath, engineExecutorContext) diff --git a/linkis-engineconn-plugins/pom.xml b/linkis-engineconn-plugins/pom.xml index 054d300ed08..807a586e444 100644 --- a/linkis-engineconn-plugins/pom.xml +++ b/linkis-engineconn-plugins/pom.xml @@ -45,14 +45,7 @@ nebula doris repl + impala - - - compile-impala - - impala - - - diff --git a/linkis-engineconn-plugins/presto/pom.xml b/linkis-engineconn-plugins/presto/pom.xml index 2b2d234d740..67333788f5f 100644 --- a/linkis-engineconn-plugins/presto/pom.xml +++ b/linkis-engineconn-plugins/presto/pom.xml @@ -65,6 +65,17 @@ com.facebook.prestopresto-client${presto.version} + + + com.squareup.okio + okio + + + + + com.squareup.okio + okio + ${okio.version} com.facebook.presto diff --git a/linkis-engineconn-plugins/presto/src/main/resources/log4j2.xml b/linkis-engineconn-plugins/presto/src/main/resources/log4j2.xml index 2cd3e264c35..645a69a4280 100644 --- a/linkis-engineconn-plugins/presto/src/main/resources/log4j2.xml +++ b/linkis-engineconn-plugins/presto/src/main/resources/log4j2.xml @@ -87,5 +87,8 @@ + + + diff --git a/linkis-engineconn-plugins/python/src/main/scala/org/apache/linkis/manager/engineplugin/python/executor/PythonEngineConnExecutor.scala b/linkis-engineconn-plugins/python/src/main/scala/org/apache/linkis/manager/engineplugin/python/executor/PythonEngineConnExecutor.scala index 57943ca3299..1388c702c3e 100644 --- a/linkis-engineconn-plugins/python/src/main/scala/org/apache/linkis/manager/engineplugin/python/executor/PythonEngineConnExecutor.scala +++ b/linkis-engineconn-plugins/python/src/main/scala/org/apache/linkis/manager/engineplugin/python/executor/PythonEngineConnExecutor.scala @@ -32,7 +32,10 @@ import org.apache.linkis.manager.common.entity.resource.{ NodeResource } import org.apache.linkis.manager.engineplugin.common.conf.EngineConnPluginConf -import org.apache.linkis.manager.engineplugin.common.conf.EngineConnPluginConf.PYTHON_VERSION_KEY +import org.apache.linkis.manager.engineplugin.common.conf.EngineConnPluginConf.{ + PYTHON_VERSION_KEY, + SPARK_PYTHON_VERSION_KEY +} import org.apache.linkis.manager.engineplugin.common.util.NodeResourceUtils import org.apache.linkis.manager.engineplugin.python.conf.PythonEngineConfiguration import org.apache.linkis.manager.label.entity.Label diff --git a/linkis-engineconn-plugins/shell/src/main/scala/org/apache/linkis/manager/engineplugin/shell/configuration/AccessibleExecutorConfiguration.scala b/linkis-engineconn-plugins/shell/src/main/scala/org/apache/linkis/manager/engineplugin/shell/configuration/AccessibleExecutorConfiguration.scala index dd861f2c8d5..7693bc2a748 100644 --- a/linkis-engineconn-plugins/shell/src/main/scala/org/apache/linkis/manager/engineplugin/shell/configuration/AccessibleExecutorConfiguration.scala +++ b/linkis-engineconn-plugins/shell/src/main/scala/org/apache/linkis/manager/engineplugin/shell/configuration/AccessibleExecutorConfiguration.scala @@ -36,7 +36,6 @@ import org.springframework.context.annotation.{Bean, Configuration} class AccessibleExecutorConfiguration extends Logging { @Bean(Array("lockService")) - @ConditionalOnMissingBean def createLockManager(): LockService = { val lockService = new EngineConnConcurrentLockService logger.info("use shell engine conn configuration to create concurrent lockService") diff --git a/linkis-engineconn-plugins/spark/pom.xml b/linkis-engineconn-plugins/spark/pom.xml index ff5b2c93303..035b2fdb33b 100644 --- a/linkis-engineconn-plugins/spark/pom.xml +++ b/linkis-engineconn-plugins/spark/pom.xml @@ -194,7 +194,11 @@ org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs-client + + + org.apache.hadoop + hadoop-hdfs com.fasterxml.jackson.core @@ -243,7 +247,11 @@ org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs-client + + + org.apache.hadoop + hadoop-hdfs com.fasterxml.jackson.core @@ -285,7 +293,11 @@ org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs-client + + + org.apache.hadoop + hadoop-hdfs com.fasterxml.jackson.core @@ -330,7 +342,11 @@ org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs-client + + + org.apache.hadoop + hadoop-hdfs org.apache.commons @@ -464,8 +480,8 @@ ch.cern.sparkmeasure - spark-measure_2.12 - 0.24 + spark-measure_${scala.binary.version} + ${sparkmeasure.version} org.xerial.snappy @@ -545,11 +561,11 @@ spark-3 - 1.11.0-wds-spark3 3.7.0-M11 3.4.4 2.12.17 2.12 + 0.24 2.14.2 diff --git a/linkis-engineconn-plugins/spark/scala-2.12/pom.xml b/linkis-engineconn-plugins/spark/scala-2.12/pom.xml index 4dd5c56e4b0..7691c25473e 100644 --- a/linkis-engineconn-plugins/spark/scala-2.12/pom.xml +++ b/linkis-engineconn-plugins/spark/scala-2.12/pom.xml @@ -55,9 +55,14 @@ org.apache.hadoop hadoop-common + org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs-client + + + org.apache.hadoop + hadoop-hdfs com.fasterxml.jackson.core diff --git a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkConfiguration.scala b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkConfiguration.scala index ad3c5ef232b..b07d5013322 100644 --- a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkConfiguration.scala +++ b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/config/SparkConfiguration.scala @@ -229,6 +229,9 @@ object SparkConfiguration extends Logging { val SPARK_DRIVER_BLOCK_MANAGER_PORT = CommonVars[String]("spark.driver.blockManager.port", "") + val SPARK_PROHIBITS_DYNAMIC_RESOURCES_SWITCH = + CommonVars[Boolean]("linkis.spark.dynamic.resource.switch", false).getValue + private def getMainJarName(): String = { val somePath = ClassUtils.jarOfClass(classOf[SparkEngineConnFactory]) if (somePath.isDefined) { diff --git a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkSqlExecutor.scala b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkSqlExecutor.scala index b32a0744058..baeaf6e0408 100644 --- a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkSqlExecutor.scala +++ b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/executor/SparkSqlExecutor.scala @@ -121,18 +121,15 @@ class SparkSqlExecutor( ) ) - if (engineExecutionContext.isEnableDirectPush) { - submitResultSetIterator(lastTask.getTaskId, df) - } else { - SQLSession.showDF( - sparkEngineSession.sparkContext, - jobGroup, - df, - null, - SparkConfiguration.SHOW_DF_MAX_RES.getValue, - engineExecutionContext - ) - } + // Direct push feature removed, use standard showDF + SQLSession.showDF( + sparkEngineSession.sparkContext, + jobGroup, + df, + null, + SparkConfiguration.SHOW_DF_MAX_RES.getValue, + engineExecutionContext + ) // Stop capturing Spark metrics and output the records to the specified file. sparkMeasure.foreach { measure => diff --git a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnFactory.scala b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnFactory.scala index 9e71c765313..db585c5ad6e 100644 --- a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnFactory.scala +++ b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/factory/SparkEngineConnFactory.scala @@ -20,6 +20,7 @@ package org.apache.linkis.engineplugin.spark.factory import org.apache.linkis.common.conf.CommonVars import org.apache.linkis.common.utils.{JsonUtils, Logging, Utils} import org.apache.linkis.engineconn.common.creation.EngineCreationContext +import org.apache.linkis.engineconn.computation.executor.conf.ComputationExecutorConf import org.apache.linkis.engineconn.launch.EngineConnServer import org.apache.linkis.engineplugin.spark.client.context.{ExecutionContext, SparkConfig} import org.apache.linkis.engineplugin.spark.config.SparkConfiguration @@ -32,6 +33,8 @@ import org.apache.linkis.engineplugin.spark.exception.{ SparkCreateFileException, SparkSessionNullException } +import org.apache.linkis.engineplugin.spark.extension.SparkUDFCheckRule +import org.apache.linkis.engineplugin.spark.utils.EngineUtils import org.apache.linkis.manager.engineplugin.common.conf.EnvConfiguration import org.apache.linkis.manager.engineplugin.common.creation.{ ExecutorFactory, @@ -39,6 +42,7 @@ import org.apache.linkis.manager.engineplugin.common.creation.{ } import org.apache.linkis.manager.engineplugin.common.launch.process.Environment import org.apache.linkis.manager.engineplugin.common.launch.process.Environment.variable +import org.apache.linkis.manager.label.conf.LabelCommonConfig import org.apache.linkis.manager.label.entity.engine.EngineType import org.apache.linkis.manager.label.entity.engine.EngineType.EngineType import org.apache.linkis.manager.label.utils.LabelUtil @@ -215,6 +219,22 @@ class SparkEngineConnFactory extends MultiExecutorEngineConnFactory with Logging logger.info( "print current thread name " + Thread.currentThread().getContextClassLoader.toString ) + // 在所有配置加载完成后检查Spark版本 + // 如果不是3.4.4版本则关闭动态分配功能（这是最晚的配置设置点） + val sparkVersion = Utils.tryQuietly(EngineUtils.sparkSubmitVersion()) + if ( + SparkConfiguration.SPARK_PROHIBITS_DYNAMIC_RESOURCES_SWITCH && (!LabelCommonConfig.SPARK3_ENGINE_VERSION.getValue + .equals(sparkVersion)) + ) { + logger.info( + s"Spark version is $sparkVersion, not 3.4.4, disabling spark.dynamicAllocation.enabled" + ) + sparkConf.set("spark.dynamicAllocation.enabled", "false") + } else { + logger.info( + s"Spark version is $sparkVersion, keeping spark.dynamicAllocation.enabled as configured" + ) + } val sparkSession = createSparkSession(outputDir, sparkConf) if (sparkSession == null) { throw new SparkSessionNullException(CAN_NOT_NULL.getErrorCode, CAN_NOT_NULL.getErrorDesc) @@ -292,6 +312,13 @@ class SparkEngineConnFactory extends MultiExecutorEngineConnFactory with Logging } val builder = SparkSession.builder.config(conf) + if (ComputationExecutorConf.SPECIAL_UDF_CHECK_ENABLED.getValue) { + logger.info("inject sql check rule into spark extension.") + builder.withExtensions(extension => { + extension.injectOptimizerRule(SparkUDFCheckRule) + }) + } + builder.enableHiveSupport().getOrCreate() } @@ -325,7 +352,7 @@ class SparkEngineConnFactory extends MultiExecutorEngineConnFactory with Logging output }(t => { logger.warn("create spark repl classdir failed", t) - throw new SparkCreateFileException( + throw new SparkCreateFileException( // NOSONAR SPARK_CREATE_EXCEPTION.getErrorCode, SPARK_CREATE_EXCEPTION.getErrorDesc, t diff --git a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/sparkmeasure/SparkSqlMeasure.scala b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/sparkmeasure/SparkSqlMeasure.scala index 9b613f05561..d92d27cefd9 100644 --- a/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/sparkmeasure/SparkSqlMeasure.scala +++ b/linkis-engineconn-plugins/spark/src/main/scala/org/apache/linkis/engineplugin/spark/sparkmeasure/SparkSqlMeasure.scala @@ -121,11 +121,41 @@ class SparkSqlMeasure( private def collectMetrics( metrics: Either[StageMetrics, TaskMetrics] ): java.util.Map[String, Long] = { + import scala.collection.JavaConverters._ + + def toJavaMap(obj: Any): java.util.Map[String, Long] = { + try { + // Try newer API (0.24+) with JavaMap method + val javaMapMethod = obj.getClass.getMethod("aggregateStageMetricsJavaMap") + javaMapMethod.invoke(obj).asInstanceOf[java.util.Map[String, Long]] + } catch { + case _: NoSuchMethodException => + try { + // Try older API (0.17) that returns Scala Map + val aggregateMethod = obj.getClass.getMethod( + if (obj.isInstanceOf[StageMetrics]) "aggregateStageMetrics" + else "aggregateTaskMetrics" + ) + val result = aggregateMethod.invoke(obj) + // Convert Scala Map to Java Map + result + .asInstanceOf[Map[String, Any]] + .map { + case (k, v: Number) => k -> v.longValue() + case (k, v) => k -> v.toString.toLong + } + .asJava + } catch { + case e: Exception => + logger.warn(s"Failed to collect metrics: ${e.getMessage}") + new util.HashMap[String, Long]() + } + } + } + metrics match { - case Left(stageMetrics) => - stageMetrics.aggregateStageMetricsJavaMap() - case Right(taskMetrics) => - taskMetrics.aggregateTaskMetricsJavaMap() + case Left(stageMetrics) => toJavaMap(stageMetrics) + case Right(taskMetrics) => toJavaMap(taskMetrics) case _ => new util.HashMap[String, Long]() } } diff --git a/linkis-engineconn-plugins/sqoop/pom.xml b/linkis-engineconn-plugins/sqoop/pom.xml index 87fdae7c86a..ad456eb5a8a 100644 --- a/linkis-engineconn-plugins/sqoop/pom.xml +++ b/linkis-engineconn-plugins/sqoop/pom.xml @@ -72,8 +72,12 @@ servlet-api - com.google.inject - guice + org.apache.avro + avro + + + org.codehaus.jackson + jackson-core-asl @@ -162,9 +166,14 @@ org.slf4j slf4j-log4j12 + + + org.apache.hadoop + hadoop-hdfs-client + org.apache.hadoop - ${hadoop-hdfs-client.artifact} + hadoop-hdfs diff --git a/linkis-engineconn-plugins/trino/pom.xml b/linkis-engineconn-plugins/trino/pom.xml index d9ea2d6868f..767be9328a2 100644 --- a/linkis-engineconn-plugins/trino/pom.xml +++ b/linkis-engineconn-plugins/trino/pom.xml @@ -65,6 +65,17 @@ io.trinotrino-client${trino.version} + + + com.squareup.okio + okio + + + + + com.squareup.okio + okio + ${okio.version} diff --git a/linkis-engineconn-plugins/trino/src/main/java/org/apache/linkis/engineplugin/trino/socket/SocketChannelSocketFactory.java b/linkis-engineconn-plugins/trino/src/main/java/org/apache/linkis/engineplugin/trino/socket/SocketChannelSocketFactory.java index 9bd519f0a74..8cf9e6bb798 100644 --- a/linkis-engineconn-plugins/trino/src/main/java/org/apache/linkis/engineplugin/trino/socket/SocketChannelSocketFactory.java +++ b/linkis-engineconn-plugins/trino/src/main/java/org/apache/linkis/engineplugin/trino/socket/SocketChannelSocketFactory.java @@ -30,12 +30,12 @@ public class SocketChannelSocketFactory extends SocketFactory { @Override public Socket createSocket() throws IOException { - return SocketChannel.open().socket(); + return SocketChannel.open().socket(); // NOSONAR } @Override public Socket createSocket(String host, int port) throws IOException { - return SocketChannel.open(new InetSocketAddress(host, port)).socket(); + return SocketChannel.open(new InetSocketAddress(host, port)).socket(); // NOSONAR } @Override @@ -46,7 +46,7 @@ public Socket createSocket(String host, int port, InetAddress localAddress, int @Override public Socket createSocket(InetAddress address, int port) throws IOException { - return SocketChannel.open(new InetSocketAddress(address, port)).socket(); + return SocketChannel.open(new InetSocketAddress(address, port)).socket(); // NOSONAR } @Override diff --git a/linkis-extensions/linkis-et-monitor/pom.xml b/linkis-extensions/linkis-et-monitor/pom.xml index 565979119d6..d0f22e3d081 100644 --- a/linkis-extensions/linkis-et-monitor/pom.xml +++ b/linkis-extensions/linkis-et-monitor/pom.xml @@ -82,7 +82,36 @@ de.codecentric spring-boot-admin-starter-server - 2.7.16 + ${spring.boot.admin.version} + + + io.projectreactor.netty + reactor-netty-http + + + org.thymeleaf + thymeleaf + + + org.thymeleaf + thymeleaf-spring5 + + + + + org.thymeleaf + thymeleaf + 3.1.2.RELEASE + + + org.thymeleaf + thymeleaf-spring5 + 3.1.2.RELEASE + + + io.projectreactor.netty + reactor-netty-http + ${reactor-netty-http.version} diff --git a/linkis-extensions/linkis-et-monitor/src/main/assembly/distribution.xml b/linkis-extensions/linkis-et-monitor/src/main/assembly/distribution.xml index 0280a0857c3..462d7be9f85 100644 --- a/linkis-extensions/linkis-et-monitor/src/main/assembly/distribution.xml +++ b/linkis-extensions/linkis-et-monitor/src/main/assembly/distribution.xml @@ -164,8 +164,8 @@ org.apache.logging.log4j:log4j-slf4j-impl:jar org.apache.zookeeper:zookeeper:jar org.aspectj:aspectjweaver:jar - org.bouncycastle:bcpkix-jdk18on:jar - org.bouncycastle:bcprov-jdk18on:jar + org.bouncycastle:bcpkix-jdk15on:jar + org.bouncycastle:bcprov-jdk15on:jar org.codehaus.jackson:jackson-jaxrs:jar org.codehaus.jackson:jackson-xc:jar org.codehaus.jettison:jettison:jar diff --git a/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/config/MonitorConfig.java b/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/config/MonitorConfig.java index 8d1f94e79c6..65bf1ce9f4f 100644 --- a/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/config/MonitorConfig.java +++ b/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/config/MonitorConfig.java @@ -71,5 +71,5 @@ public class MonitorConfig { + "请关注是否任务正常，如果不正常您可以到Linkis/DSS管理台进行任务的kill，集群信息为BDAP({2})。详细解决方案见Q47：{3} "); public static final CommonVars JOBHISTORY_CLEAR_DAY = - CommonVars.apply("linkis.monitor.jobhistory.clear.day", "90"); + CommonVars.apply("linkis.monitor.jobhistory.clear.day", "60"); } diff --git a/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/scheduled/UserDepartmentInfoSync.java b/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/scheduled/UserDepartmentInfoSync.java index bdc4720b72e..338d220489b 100644 --- a/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/scheduled/UserDepartmentInfoSync.java +++ b/linkis-extensions/linkis-et-monitor/src/main/java/org/apache/linkis/monitor/scheduled/UserDepartmentInfoSync.java @@ -53,7 +53,7 @@ public class UserDepartmentInfoSync { private static final UserDepartmentInfoMapper userDepartmentInfoMapper = MapperFactory.getUserDepartmentInfoMapper(); - @Scheduled(cron = "${linkis.monitor.org.user.sync.cron:0 0 0 1/7 * ?}") + @Scheduled(cron = "${linkis.monitor.org.user.sync.cron:0 0 11 1/7 * ?}") public static void DepartmentInfoSync() { // 获取linkis_org_user_sync信息 // 收集异常用户 diff --git a/linkis-extensions/linkis-et-monitor/src/main/resources/mapper/common/UserDepartmentInfoMapper.xml b/linkis-extensions/linkis-et-monitor/src/main/resources/mapper/common/UserDepartmentInfoMapper.xml index 176e3b67030..d77f13ef9e0 100644 --- a/linkis-extensions/linkis-et-monitor/src/main/resources/mapper/common/UserDepartmentInfoMapper.xml +++ b/linkis-extensions/linkis-et-monitor/src/main/resources/mapper/common/UserDepartmentInfoMapper.xml @@ -15,7 +15,6 @@ ~ See the License for the specific language governing permissions and ~ limitations under the License. --> - diff --git a/linkis-extensions/linkis-et-monitor/src/main/scala/org/apache/linkis/monitor/jobhistory/jobtime/StarrocksTimeExceedRule.scala b/linkis-extensions/linkis-et-monitor/src/main/scala/org/apache/linkis/monitor/jobhistory/jobtime/StarrocksTimeExceedRule.scala index b616c5c02cb..93404aa1bc8 100644 --- a/linkis-extensions/linkis-et-monitor/src/main/scala/org/apache/linkis/monitor/jobhistory/jobtime/StarrocksTimeExceedRule.scala +++ b/linkis-extensions/linkis-et-monitor/src/main/scala/org/apache/linkis/monitor/jobhistory/jobtime/StarrocksTimeExceedRule.scala @@ -29,7 +29,7 @@ import org.apache.commons.collections.MapUtils import org.apache.commons.lang3.StringUtils import java.util -import java.util.{ArrayList, HashMap, List, Locale, Map} +import java.util.{ArrayList, HashMap, Locale, Map} import scala.collection.JavaConverters._ @@ -54,7 +54,7 @@ class StarrocksTimeExceedRule(hitObserver: Observer) val alertData: util.List[JobHistory] = new util.ArrayList[JobHistory]() for (scannedData <- data.asScala) { if (scannedData != null && scannedData.getData() != null) { - var taskMinID = 0L; + var taskMinID = 0L for (jobHistory <- scannedData.getData().asScala) { jobHistory match { case job: JobHistory => @@ -66,7 +66,7 @@ class StarrocksTimeExceedRule(hitObserver: Observer) Constants.JDBC_ENGINE.toUpperCase(Locale.getDefault) ) ) { - // 获取job所使用的数据源类型 + // 获取 job 所使用的数据源类型 val datasourceConfMap = getDatasourceConf(job) logger.info("starock datasourceConfMap: {}", datasourceConfMap) // 计算任务执行时间 @@ -83,6 +83,26 @@ class StarrocksTimeExceedRule(hitObserver: Observer) alertData.add(job) } } + // 获取超时 kill 配置信息 + if (StringUtils.isNotBlank(job.getParams)) { + val connectParamsMap = MapUtils.getMap( + datasourceConfMap, + "connectParams", + new util.HashMap[AnyRef, AnyRef] + ) + val killTime = MapUtils.getString(connectParamsMap, "kill_task_time", "") + logger.info("starock killTime: {}", killTime) + if (StringUtils.isNotBlank(killTime) && elapse > killTime.toLong * 60 * 1000) { + if (StringUtils.isNotBlank(timeValue)) { + val timeoutInSeconds = timeValue.toDouble + val timeoutInMillis = (timeoutInSeconds * 60 * 1000).toLong + if (elapse > timeoutInMillis) { + // 触发 kill 任务 + HttpsUntils.killJob(job) + } + } + } + } } if (taskMinID == 0L || taskMinID > job.getId) { taskMinID = job.getId @@ -109,7 +129,7 @@ class StarrocksTimeExceedRule(hitObserver: Observer) } private def getDatasourceConf(job: JobHistory): util.Map[_, _] = { - // 获取任务参数中datasourcename + // 获取任务参数中 datasourceName val parmMap = BDPJettyServerHelper.gson.fromJson(job.getParams, classOf[java.util.Map[String, String]]) val configurationMap = @@ -117,7 +137,7 @@ class StarrocksTimeExceedRule(hitObserver: Observer) val runtimeMap = MapUtils.getMap(configurationMap, "runtime", new util.HashMap[String, String]()) val datasourceName = MapUtils.getString(runtimeMap, Constants.JOB_DATASOURCE_CONF, "") - // 获取datasource信息 + // 获取 datasource 信息 if (StringUtils.isNotBlank(datasourceName)) { HttpsUntils.getDatasourceConf(job.getSubmitUser, datasourceName) } else { diff --git a/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/monitor/EngineConnMonitor.scala b/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/monitor/EngineConnMonitor.scala index e89193418ad..1479bc54769 100644 --- a/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/monitor/EngineConnMonitor.scala +++ b/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/monitor/EngineConnMonitor.scala @@ -30,10 +30,14 @@ import org.apache.linkis.governance.common.utils.GovernanceConstant import org.apache.linkis.manager.common.entity.enumeration.NodeStatus import org.apache.linkis.manager.common.protocol.node.{RequestNodeStatus, ResponseNodeStatus} import org.apache.linkis.manager.label.entity.Label +import org.apache.linkis.manager.label.entity.engine.EngineTypeLabel import org.apache.linkis.manager.label.utils.LabelUtil import org.apache.linkis.orchestrator.computation.conf.ComputationOrchestratorConf import org.apache.linkis.orchestrator.computation.execute.{CodeExecTaskExecutor, EngineConnTaskInfo} +import org.apache.linkis.orchestrator.ecm.entity.{Mark, MarkReq} +import org.apache.linkis.orchestrator.ecm.service.impl.ComputationEngineConnExecutor import org.apache.linkis.orchestrator.listener.task.{ + EngineQuitedUnexpectedlyEvent, TaskErrorResponseEvent, TaskLogEvent, TaskStatusEvent @@ -41,6 +45,8 @@ import org.apache.linkis.orchestrator.listener.task.{ import org.apache.linkis.rpc.Sender import org.apache.linkis.server.{toJavaMap, BDPJettyServerHelper} +import org.apache.commons.lang3.StringUtils + import java.util import java.util.concurrent.TimeUnit @@ -53,6 +59,8 @@ object EngineConnMonitor extends Logging { private val ENGINECONN_LASTUPDATE_TIMEOUT = ComputationOrchestratorConf.ENGINECONN_LASTUPDATE_TIMEOUT.getValue.toLong + private val engineTypeKey = "engineType" + private[linkis] def addEngineExecutorStatusMonitor( engineConnExecutorCache: util.Map[EngineConnTaskInfo, CodeExecTaskExecutor] ): Unit = { @@ -202,29 +210,27 @@ object EngineConnMonitor extends Logging { executors.foreach { executor => val execTask = executor.getExecTask Utils.tryAndError { - val labels: Array[Label[_]] = executor.getEngineConnExecutor.getLabels() - val engineType: String = LabelUtil.getEngineTypeLabel(labels.toList.asJava).getEngineType logger.warn( s"Will kill task ${execTask.getIDInfo()} because the engine ${executor.getEngineConnExecutor.getServiceInstance.toString} quited unexpectedly." ) - val errLog = LogUtils.generateERROR( - s"Your job : ${execTask.getIDInfo()} was failed because the ${engineType} engine quitted unexpectedly(任务${execTask - .getIDInfo()}失败，" + - s"原因是引擎意外退出,可能是复杂任务导致引擎退出，如OOM)." - ) - val logEvent = TaskLogEvent(execTask, errLog) - execTask.getPhysicalContext.pushLog(logEvent) - val errorResponseEvent = TaskErrorResponseEvent( + val event = EngineQuitedUnexpectedlyEvent( execTask, - "task failed，Engine quitted unexpectedly(任务运行失败原因是引擎意外退出,可能是复杂任务导致引擎退出，如OOM)." + executor.getEngineConnExecutor.getServiceInstance.toString ) - execTask.getPhysicalContext.broadcastSyncEvent(errorResponseEvent) - val statusEvent = TaskStatusEvent(execTask, ExecutionNodeStatus.Failed) - execTask.getPhysicalContext.broadcastSyncEvent(statusEvent) + execTask.getPhysicalContext.broadcastSyncEvent(event) } } } + private def getEngineType(labels: Array[Label[_]]): String = { + val labelArray: Array[Label[_]] = labels.filter(_.getLabelKey.equals(engineTypeKey)) + var engineType = "" + if (labelArray != null && labelArray.size > 0) { + engineType = labelArray(0).asInstanceOf[EngineTypeLabel].getEngineType + } + engineType + } + private def updateExecutorActivityTime( serviceInstance: ServiceInstance, engineConnExecutorCache: mutable.HashMap[ServiceInstance, ArrayBuffer[CodeExecTaskExecutor]] diff --git a/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/service/ComputationTaskExecutionReceiver.scala b/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/service/ComputationTaskExecutionReceiver.scala index 21451dbde88..d1497ae768c 100644 --- a/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/service/ComputationTaskExecutionReceiver.scala +++ b/linkis-orchestrator/linkis-computation-orchestrator/src/main/scala/org/apache/linkis/orchestrator/computation/service/ComputationTaskExecutionReceiver.scala @@ -17,6 +17,7 @@ package org.apache.linkis.orchestrator.computation.service +import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.utils.Logging import org.apache.linkis.governance.common.entity.ExecutionNodeStatus import org.apache.linkis.governance.common.protocol.task._ @@ -97,7 +98,11 @@ class ComputationTaskExecutionReceiver extends TaskExecutionReceiver with Loggin taskStatus match { case rte: ResponseTaskStatusWithExecuteCodeIndex => logger.info(s"execute error with index: ${rte.errorIndex}") - task.updateParams("execute.error.code.index", rte.errorIndex.toString) + task.updateParams(Configuration.EXECUTE_ERROR_CODE_INDEX.key, rte.errorIndex.toString) + task.updateParams( + Configuration.EXECUTE_RESULTSET_ALIAS_NUM.key, + rte.aliasNum.toString + ) case _ => } // 标识当前方法执行过，该方法是异步的，处理失败任务需要该方法执行完 diff --git a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/conf/OrchestratorConfiguration.scala b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/conf/OrchestratorConfiguration.scala index b66c2819f3e..04dcac6792a 100644 --- a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/conf/OrchestratorConfiguration.scala +++ b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/conf/OrchestratorConfiguration.scala @@ -67,7 +67,7 @@ object OrchestratorConfiguration { CommonVars("wds.linkis.orchestrator.task.scheduler.retry.wait.time", 100000) val TASK_SCHEDULER_THREAD_POOL = - CommonVars("wds.linkis.orchestrator.task.scheduler.thread.pool", 200) + CommonVars("wds.linkis.orchestrator.task.scheduler.thread.pool", 400) val ORCHESTRATOR_EXECUTION_FACTORY = CommonVars( "wds.linkis.orchestrator.execution.factory.class", diff --git a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/execution/impl/DefaultFailedTaskResponse.scala b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/execution/impl/DefaultFailedTaskResponse.scala index 2ded3662978..46bfc32d135 100644 --- a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/execution/impl/DefaultFailedTaskResponse.scala +++ b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/execution/impl/DefaultFailedTaskResponse.scala @@ -25,11 +25,16 @@ class DefaultFailedTaskResponse(errorMsg: String, errorCode: Int, throwable: Thr extends FailedTaskResponse { private var _errorIndex: Int = -1 + private var _aliasNum: Int = 0 def errorIndex: Int = _errorIndex def errorIndex_=(value: Int): Unit = _errorIndex = value + def aliasNum: Int = _aliasNum + + def aliasNum_=(value: Int): Unit = _aliasNum = value + override def getCause: Throwable = throwable override def getErrorMsg: String = errorMsg diff --git a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/plans/physical/PhysicalContextImpl.scala b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/plans/physical/PhysicalContextImpl.scala index d7909259af5..36b95e2f4e6 100644 --- a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/plans/physical/PhysicalContextImpl.scala +++ b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/plans/physical/PhysicalContextImpl.scala @@ -17,6 +17,7 @@ package org.apache.linkis.orchestrator.plans.physical +import org.apache.linkis.common.conf.Configuration import org.apache.linkis.common.listener.Event import org.apache.linkis.common.utils.Logging import org.apache.linkis.governance.common.entity.ExecutionNodeStatus @@ -82,7 +83,8 @@ class PhysicalContextImpl(private var rootTask: ExecTask, private var leafTasks: case job: AbstractJob => val labels: util.List[Label[_]] = job.getLabels val codeType: String = LabelUtil.getCodeType(labels) - if ("aisql".equals(codeType)) { + // 支持 aisql 和 jdbc 类型的断点续跑 + if ("aisql".equals(codeType) || "jdbc".equals(codeType)) { val params: Map[String, String] = this.rootTask.params var flag: Boolean = params.getOrElse("task.error.receiver.flag", "false").toBoolean val startTime: Long = System.currentTimeMillis() @@ -95,7 +97,10 @@ class PhysicalContextImpl(private var rootTask: ExecTask, private var leafTasks: flag = params.getOrElse("task.error.receiver.flag", "false").toBoolean } logger.info("task error receiver end.") - failedResponse.errorIndex = params.getOrElse("execute.error.code.index", "-1").toInt + failedResponse.errorIndex = + params.getOrElse(Configuration.EXECUTE_ERROR_CODE_INDEX.key, "-1").toInt + failedResponse.aliasNum = + params.getOrElse(Configuration.EXECUTE_RESULTSET_ALIAS_NUM.key, "0").toInt } case _ => } diff --git a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/strategy/async/AsyncTaskManager.scala b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/strategy/async/AsyncTaskManager.scala index 7392b3784b4..f309005eb59 100644 --- a/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/strategy/async/AsyncTaskManager.scala +++ b/linkis-orchestrator/linkis-orchestrator-core/src/main/scala/org/apache/linkis/orchestrator/strategy/async/AsyncTaskManager.scala @@ -19,6 +19,7 @@ package org.apache.linkis.orchestrator.strategy.async import org.apache.linkis.common.log.LogUtils import org.apache.linkis.governance.common.entity.ExecutionNodeStatus +import org.apache.linkis.manager.label.utils.LabelUtil import org.apache.linkis.orchestrator.execution.ExecTaskRunner import org.apache.linkis.orchestrator.execution.impl.DefaultTaskManager import org.apache.linkis.orchestrator.listener.OrchestratorSyncEvent @@ -122,8 +123,12 @@ class AsyncTaskManager ) } else { val execTask = event.execTask + val labels = execTask.getTaskDesc.getOrigin.getASTOrchestration.getASTContext.getLabels + val engineType: String = LabelUtil.getEngineTypeLabel(labels).getEngineType val errLog = LogUtils.generateERROR( - s"Your job : ${execTask.getIDInfo()} was failed because the engine quited unexpectedly(任务${execTask.getIDInfo()}失败，原因是引擎意外退出,可能是复杂任务导致引擎退出，如OOM)." + s"Your job : ${execTask.getIDInfo()} was failed because the ${engineType} engine quitted unexpectedly(任务${execTask + .getIDInfo()}失败，" + + s"原因是引擎意外退出,可能是复杂任务导致引擎退出，如OOM)." ) val logEvent = TaskLogEvent(execTask, errLog) execTask.getPhysicalContext.pushLog(logEvent) diff --git a/linkis-public-enhancements/distribution.xml b/linkis-public-enhancements/distribution.xml index b4cc88582e1..58b75bf1d74 100644 --- a/linkis-public-enhancements/distribution.xml +++ b/linkis-public-enhancements/distribution.xml @@ -243,9 +243,6 @@ org.slf4j:jul-to-slf4j:jar org.slf4j:slf4j-api:jar org.springframework.boot:spring-boot:jar - org.springframework.boot:spring-boot-actuator:jar - org.springframework.boot:spring-boot-actuator-autoconfigure:jar - org.springframework.boot:spring-boot-autoconfigure:jar org.springframework.boot:spring-boot-starter:jar org.springframework.boot:spring-boot-starter-actuator:jar org.springframework.boot:spring-boot-starter-aop:jar diff --git a/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java b/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java index 900dfca21a2..ed9ab45cf65 100644 --- a/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java +++ b/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlProjectRestful.java @@ -260,7 +260,7 @@ public Message updateShareResource( Map properties = new HashMap<>(); properties.put("clientIp", clientIp); ResourceTask resourceTask = null; - synchronized (resourceId.intern()) { + synchronized (resourceId.intern()) { // NOSONAR resourceTask = taskService.createUpdateTask(resourceId, DEFAULT_PROXY_USER, file, properties); } diff --git a/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java b/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java index 8125d706dbc..363833bf87b 100644 --- a/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java +++ b/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/restful/BmlRestfulApi.java @@ -748,7 +748,7 @@ public Message updateVersion( Map properties = new HashMap<>(); properties.put("clientIp", clientIp); ResourceTask resourceTask = null; - synchronized (resourceId.intern()) { + synchronized (resourceId.intern()) { // NOSONAR resourceTask = taskService.createUpdateTask(resourceId, user, file, properties); } message = Message.ok("The update resource task was submitted successfully(提交更新资源任务成功)"); @@ -891,7 +891,7 @@ public Message copyResourceToAnotherUser( properties.put("maxVersion", 0); properties.put("system", "dss"); ResourceTask resourceTask = null; - synchronized (resourceId.intern()) { + synchronized (resourceId.intern()) { // NOSONAR resourceTask = taskService.createCopyResourceTask(resourceId, anotherUser, properties); } message = Message.ok(); @@ -930,7 +930,7 @@ public Message rollbackVersion(HttpServletRequest request, @RequestBody JsonNode Map properties = new HashMap<>(); properties.put("clientIp", clientIp); ResourceTask resourceTask = null; - synchronized (resourceId.intern()) { + synchronized (resourceId.intern()) { // NOSONAR resourceTask = taskService.createRollbackVersionTask( resourceId, rollbackVersion, username, properties); diff --git a/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/service/impl/VersionServiceImpl.java b/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/service/impl/VersionServiceImpl.java index b2d9479c5f1..f6f4daa8edf 100644 --- a/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/service/impl/VersionServiceImpl.java +++ b/linkis-public-enhancements/linkis-bml-server/src/main/java/org/apache/linkis/bml/service/impl/VersionServiceImpl.java @@ -147,7 +147,7 @@ public boolean downloadResource( Fs fileSystem = FSFactory.getFsByProxyUser(new FsPath(path), user); fileSystem.init(new HashMap()); InputStream inputStream = fileSystem.read(new FsPath(path)); - inputStream.skip(startByte - 1); + inputStream.skip(startByte - 1); // NOSONAR logger.info( "{} downLoad source {} inputStream skipped {} bytes", user, resourceId, (startByte - 1)); byte[] buffer = new byte[1024]; diff --git a/linkis-public-enhancements/linkis-bml-server/src/test/java/org/apache/linkis/bml/dao/BmlProjectDaoTest.java b/linkis-public-enhancements/linkis-bml-server/src/test/java/org/apache/linkis/bml/dao/BmlProjectDaoTest.java index 8ccfc31ce58..3c78aa5fbf8 100644 --- a/linkis-public-enhancements/linkis-bml-server/src/test/java/org/apache/linkis/bml/dao/BmlProjectDaoTest.java +++ b/linkis-public-enhancements/linkis-bml-server/src/test/java/org/apache/linkis/bml/dao/BmlProjectDaoTest.java @@ -26,15 +26,11 @@ import java.util.List; import org.junit.jupiter.api.Test; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import static org.junit.jupiter.api.Assertions.assertTrue; class BmlProjectDaoTest extends BaseDaoTest { - private static final Logger logger = LoggerFactory.getLogger(BmlProjectDaoTest.class); - @Autowired BmlProjectDao bmlProjectDao; void insertNewProject() { @@ -82,7 +78,6 @@ void testSetProjectPriv() { @Test void testGetPrivInProject() { Integer privInt = bmlProjectDao.getPrivInProject("testName", "creCreatorUser"); - logger.info("privInt:" + privInt); assertTrue(privInt == 2); } @@ -94,7 +89,6 @@ void testAddProjectResource() { @Test void testGetProjectNameByResourceId() { String projectName = bmlProjectDao.getProjectNameByResourceId("123"); - logger.info("projectName:" + projectName); assertTrue(projectName.equals("testName")); } diff --git a/linkis-public-enhancements/linkis-bml-server/src/test/resources/application.properties b/linkis-public-enhancements/linkis-bml-server/src/test/resources/application.properties index 3659afe32b3..7d192e44228 100644 --- a/linkis-public-enhancements/linkis-bml-server/src/test/resources/application.properties +++ b/linkis-public-enhancements/linkis-bml-server/src/test/resources/application.properties @@ -28,7 +28,7 @@ springfox.documentation.auto-startup=false springfox.documentation.swagger-ui.enabled=false mybatis-plus.mapper-locations=classpath*:mapper/common/*.xml,classpath*:mapper/mysql/*.xml -mybatis-plus.type-aliases-package=org.apache.linkis.bml.ntity +mybatis-plus.type-aliases-package=org.apache.linkis.bml.entity mybatis-plus.configuration.log-impl=org.apache.ibatis.logging.stdout.StdOutImpl #disable eureka discovery client diff --git a/linkis-public-enhancements/linkis-configuration/src/test/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApiTest.java b/linkis-public-enhancements/linkis-configuration/src/test/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApiTest.java index 85b5037c881..b01e205115d 100644 --- a/linkis-public-enhancements/linkis-configuration/src/test/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApiTest.java +++ b/linkis-public-enhancements/linkis-configuration/src/test/java/org/apache/linkis/configuration/restful/api/ConfigurationRestfulApiTest.java @@ -161,7 +161,7 @@ public void TestCheckAdmin() throws Exception { public void sendUrl(String url, MultiValueMap paramsMap, String type, String msg) throws Exception { MvcUtils mvcUtils = new MvcUtils(mockMvc); - Message mvcResult = null; + Message mvcResult = null; // NOSONAR if (type.equals("get")) { if (paramsMap != null) { mvcResult = mvcUtils.getMessage(mvcUtils.buildMvcResultGet(url, paramsMap)); @@ -176,7 +176,7 @@ public void sendUrl(String url, MultiValueMap paramsMap, String mvcResult = mvcUtils.getMessage(mvcUtils.buildMvcResultPost(url)); } } - assertEquals(MessageStatus.SUCCESS(), mvcResult.getStatus()); + assertEquals(MessageStatus.SUCCESS(), mvcResult.getStatus()); // NOSONAR logger.info(String.valueOf(mvcResult)); } } diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/BinaryLogicCondition.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/BinaryLogicCondition.java index 4e13d04e65b..ce221e73619 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/BinaryLogicCondition.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/BinaryLogicCondition.java @@ -46,6 +46,6 @@ public Condition getRight() { } public void setRight(Condition right) { - right = right; + this.right = right; } } diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/construction/ConditionParser.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/construction/ConditionParser.java index 901ef0d5b7b..c002a563bd8 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/construction/ConditionParser.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/condition/construction/ConditionParser.java @@ -22,30 +22,32 @@ import java.util.HashMap; import java.util.List; import java.util.Map; - -import com.google.common.collect.Lists; +import java.util.stream.Collectors; +import java.util.stream.Stream; public interface ConditionParser { - public static Map parserMap = - new HashMap() { - { - List conditionParsers = - Lists.newArrayList( - new RegexConditionParser(), - new ContainsConditionParser(), - new ContextTypeConditionParser(), - new ContextScopeConditionParser(), - new AndConditionParser(), - new OrConditionParser(), - new NotConditionParser(), - new NearestConditionParser(), - new ContextValueTypeConditionParser()); - for (ConditionParser conditionParser : conditionParsers) { - put(conditionParser.getName(), conditionParser); - } - } - }; + Map parserMap = initializeParserMap(); + + static Map initializeParserMap() { + List conditionParsers = + Stream.of( + new RegexConditionParser(), + new ContainsConditionParser(), + new ContextTypeConditionParser(), + new ContextScopeConditionParser(), + new AndConditionParser(), + new OrConditionParser(), + new NotConditionParser(), + new NearestConditionParser(), + new ContextValueTypeConditionParser()) + .collect(Collectors.toList()); + Map map = new HashMap<>(); + for (ConditionParser conditionParser : conditionParsers) { + map.put(conditionParser.getName(), conditionParser); + } + return map; + } Condition parse(Map conditionMap); diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/DefaultContextCacheService.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/DefaultContextCacheService.java index 1c7ed0cfd92..58dc805a5c9 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/DefaultContextCacheService.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/DefaultContextCacheService.java @@ -166,7 +166,7 @@ public List getAllByScope( } catch (Exception e) { logger.error( "Failed to getAllByScope contextID({}) of ContextScope({}) of csType({})", - contextID.getContextId(), + contextID == null ? "NULL" : contextID.getContextId(), scope, csType, e); diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/cache/cskey/impl/ContextValueMapSetImpl.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/cache/cskey/impl/ContextValueMapSetImpl.java index 7ddb75dcaf3..3d30225bd85 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/cache/cskey/impl/ContextValueMapSetImpl.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/cache/cskey/impl/ContextValueMapSetImpl.java @@ -45,7 +45,7 @@ public Map getContextValueMap(ContextType contextType) } String csType = contextType.name(); if (!contextValueMapSet.containsKey(csType)) { - synchronized (csType.intern()) { + synchronized (csType.intern()) { // NOSONAR if (!contextValueMapSet.containsKey(csType)) { logger.info("For ContextType({}) init ContextValueMap", csType); contextValueMapSet.put(csType, new HashMap(16)); diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/index/ContextInvertedIndexSetImpl.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/index/ContextInvertedIndexSetImpl.java index ebe30bad7d5..f812d921ee5 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/index/ContextInvertedIndexSetImpl.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/contextcache/index/ContextInvertedIndexSetImpl.java @@ -36,7 +36,7 @@ public class ContextInvertedIndexSetImpl implements ContextInvertedIndexSet { public ContextInvertedIndex getContextInvertedIndex(ContextType contextType) { String csType = contextType.name(); if (!invertedIndexMap.containsKey(csType)) { - synchronized (csType.intern()) { + synchronized (csType.intern()) { // NOSONAR if (!invertedIndexMap.containsKey(csType)) { logger.info("For ContextType({}) init invertedIndex", csType); invertedIndexMap.put(csType, new DefaultContextInvertedIndex()); diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/BackupInstanceGeneratorImpl.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/BackupInstanceGeneratorImpl.java index f7f93c655a7..772a80525f5 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/BackupInstanceGeneratorImpl.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/BackupInstanceGeneratorImpl.java @@ -44,6 +44,8 @@ public class BackupInstanceGeneratorImpl implements BackupInstanceGenerator { @Autowired private ContextHAChecker contextHAChecker; + private Random random = new Random(); + @Override public String getBackupInstance(String haIDKey) throws CSErrorException { @@ -88,6 +90,6 @@ public String chooseBackupInstance(String mainInstanceAlias) throws CSErrorExcep private int getBackupInstanceIndex(List instanceList) { // todo refactor according to load-balance - return new Random().nextInt(instanceList.size()); + return random.nextInt(instanceList.size()); } } diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/ContextHACheckerImpl.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/ContextHACheckerImpl.java index fbaa878d3f9..da702a03d70 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/ContextHACheckerImpl.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/highavailable/ha/impl/ContextHACheckerImpl.java @@ -100,11 +100,11 @@ public String convertHAIDToHAKey(HAContextID haContextID) throws CSErrorExceptio throw new CSErrorException( CSErrorCode.INVALID_HAID, "Incomplete HAID Object cannot be encoded. mainInstance : " - + haContextID.getInstance() + + (haContextID == null ? "NULL" : haContextID.getInstance()) + ", backupInstance : " - + haContextID.getBackupInstance() + + (haContextID == null ? "NULL" : haContextID.getBackupInstance()) + ", contextID : " - + haContextID.getContextId()); + + (haContextID == null ? "NULL" : haContextID.getContextId())); } if (StringUtils.isNumeric(haContextID.getContextId())) { return encode(haContextID); diff --git a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/protocol/RestResponseProtocol.java b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/protocol/RestResponseProtocol.java index 304db9a55f8..eb7f697fad1 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/protocol/RestResponseProtocol.java +++ b/linkis-public-enhancements/linkis-cs-server/src/main/java/org/apache/linkis/cs/server/protocol/RestResponseProtocol.java @@ -55,7 +55,7 @@ public void waitTimeEnd(long mills) throws InterruptedException { public void notifyJob() { logger.info("notify the job"); synchronized (lock) { - lock.notify(); + lock.notify(); // NOSONAR } } diff --git a/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/parser/ApiJsonTest.java b/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/parser/ApiJsonTest.java index 29d5a7ceda4..7247ce97b3c 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/parser/ApiJsonTest.java +++ b/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/parser/ApiJsonTest.java @@ -74,7 +74,7 @@ public void temp() { String test = "{\"cols\":[{\"name\":\"birthday\",\"visualType\":\"string\",\"type\":\"category\",\"config\":true,\"field\":{\"alias\":\"\",\"desc\":\"\",\"useExpression\":false},\"format\":{\"formatType\":\"default\"},\"from\":\"cols\"},{\"name\":\"name\",\"visualType\":\"string\",\"type\":\"category\",\"config\":true,\"field\":{\"alias\":\"\",\"desc\":\"\",\"useExpression\":false},\"format\":{\"formatType\":\"default\"},\"from\":\"cols\"}],\"rows\":[],\"metrics\":[{\"name\":\"score@Visualis@6F01974E\",\"visualType\":\"number\",\"type\":\"value\",\"agg\":\"sum\",\"config\":true,\"chart\":{\"id\":1,\"name\":\"table\",\"title\":\"表格\",\"icon\":\"icon-table\",\"coordinate\":\"other\",\"rules\":[{\"dimension\":[0,9999],\"metric\":[0,9999]}],\"data\":{\"cols\":{\"title\":\"列\",\"type\":\"category\"},\"rows\":{\"title\":\"行\",\"type\":\"category\"},\"metrics\":{\"title\":\"指标\",\"type\":\"value\"},\"filters\":{\"title\":\"筛选\",\"type\":\"all\"}},\"style\":{\"table\":{\"fontFamily\":\"PingFang SC\",\"fontSize\":\"12\",\"color\":\"#666\",\"lineStyle\":\"solid\",\"lineColor\":\"#D9D9D9\",\"headerBackgroundColor\":\"#f7f7f7\",\"headerConfig\":[],\"columnsConfig\":[],\"leftFixedColumns\":[],\"rightFixedColumns\":[],\"headerFixed\":true,\"autoMergeCell\":false,\"bordered\":true,\"size\":\"default\",\"withPaging\":true,\"pageSize\":\"20\",\"withNoAggregators\":false},\"spec\":{}}},\"field\":{\"alias\":\"\",\"desc\":\"\",\"useExpression\":false},\"format\":{\"formatType\":\"default\"},\"from\":\"metrics\"}],\"filters\":[],\"color\":{\"title\":\"颜色\",\"type\":\"category\",\"value\":{\"all\":\"#509af2\"},\"items\":[]},\"chartStyles\":{\"richText\":{\"content\":\"

〖@dv_name_dv@〗

〖@dv_birthday_dv@〗

\"},\"spec\":{}},\"selectedChart\":15,\"data\":[],\"pagination\":{\"pageNo\":0,\"pageSize\":0,\"withPaging\":false,\"totalCount\":0},\"dimetionAxis\":\"col\",\"renderType\":\"rerender\",\"orders\":[],\"mode\":\"chart\",\"model\":{\"birthday\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"score\":{\"sqlType\":\"DOUBLE\",\"visualType\":\"number\",\"modelType\":\"value\"},\"teacher\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"city\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"sex\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"fee\":{\"sqlType\":\"DOUBLE\",\"visualType\":\"number\",\"modelType\":\"value\"},\"name\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"lesson\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"id\":{\"sqlType\":\"INT\",\"visualType\":\"number\",\"modelType\":\"value\"},\"class\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"exam_date\":{\"sqlType\":\"STRING\",\"visualType\":\"string\",\"modelType\":\"category\"},\"age\":{\"sqlType\":\"INT\",\"visualType\":\"number\",\"modelType\":\"value\"}},\"controls\":[],\"computed\":[],\"cache\":false,\"expired\":300,\"autoLoadData\":true,\"query\":{\"groups\":[\"birthday\",\"name\"],\"aggregators\":[{\"column\":\"score\",\"func\":\"sum\"}],\"filters\":[],\"orders\":[],\"pageNo\":0,\"pageSize\":0,\"nativeQuery\":false,\"cache\":false,\"expired\":0,\"flush\":false}}"; Set columns = getWidgetUsedColumns(test); - columns.size(); + columns.size(); // NOSONAR } private Set getWidgetUsedColumns(String config) { diff --git a/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextHistoryTest.java b/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextHistoryTest.java index 0c771ca9c1b..3f1049f24af 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextHistoryTest.java +++ b/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextHistoryTest.java @@ -46,10 +46,10 @@ public void before() { public void testcreateContextHistory() throws CSErrorException, JsonProcessingException { AContextHistory aContextHistory = new AContextHistory(); PersistenceContextID persistenceContextID = new PersistenceContextID(); - persistenceContextID.setContextId(String.valueOf(new Random().nextInt(100000))); + persistenceContextID.setContextId(String.valueOf(new Random().nextInt(100000))); // NOSONAR aContextHistory.setHistoryJson("json"); aContextHistory.setContextType(ContextType.DATA); - aContextHistory.setId(new Random().nextInt(100000)); + aContextHistory.setId(new Random().nextInt(100000)); // NOSONAR aContextHistory.setKeyword("keywords"); aContextHistory.setSource("source"); contextHistoryPersistence.createHistory(persistenceContextID, aContextHistory); diff --git a/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextMapTest.java b/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextMapTest.java index 7664c8889d4..f715215f49c 100644 --- a/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextMapTest.java +++ b/linkis-public-enhancements/linkis-cs-server/src/test/java/org/apache/linkis/cs/persistence/ContextMapTest.java @@ -45,7 +45,7 @@ public void before() { public void testcreateContextMap() throws CSErrorException, JsonProcessingException { AContextKeyValue aContextKeyValue = new AContextKeyValue(); PersistenceContextID persistenceContextID = new PersistenceContextID(); - persistenceContextID.setContextId(String.valueOf(new Random().nextInt(100000))); + persistenceContextID.setContextId(String.valueOf(new Random().nextInt(100000))); // NOSONAR AContextValue aContextValue = new AContextValue(); CSTable csTable = new CSTable(); csTable.setCreator("hadoop"); diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/pom.xml b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/pom.xml index baaf6d86286..99da167830c 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/pom.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/pom.xml @@ -51,7 +51,7 @@ - org.hibernate + org.hibernate.validator hibernate-validator 6.2.0.Final @@ -91,6 +91,7 @@ ${project.version}provided + diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/assembly/distribution.xml b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/assembly/distribution.xml index e124c84fc4d..e6149a8e3b0 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/assembly/distribution.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/assembly/distribution.xml @@ -140,9 +140,6 @@ org.scala-lang:*:jar org.slf4j:*:jar org.springframework.boot:spring-boot:jar - org.springframework.boot:spring-boot-actuator:jar - org.springframework.boot:spring-boot-actuator-autoconfigure:jar - org.springframework.boot:spring-boot-autoconfigure:jar org.springframework.boot:spring-boot-starter:jar org.springframework.boot:spring-boot-starter-actuator:jar org.springframework.boot:spring-boot-starter-aop:jar diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java index cf49d316871..8788b4b1c29 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceCoreRestfulApi.java @@ -1121,6 +1121,42 @@ public Message getDataSourceListByTypes( }, "Fail to get all types of data source[获取数据源列表失败]"); } + + @ApiOperation( + value = "getPublishedDataSourceByType", + notes = "get published data source by type and proxy user", + response = Message.class) + @ApiImplicitParams({ + @ApiImplicitParam(name = "dataSourceType", required = true, dataType = "String"), + @ApiImplicitParam(name = "proxyUser", required = true, dataType = "String") + }) + @RequestMapping(value = "/published/type", method = RequestMethod.GET) + public Message getPublishedDataSourceByType( + @RequestParam("dataSourceType") String dataSourceType, + @RequestParam("proxyUser") String proxyUser, + HttpServletRequest request) { + return RestfulApiHelper.doAndResponse( + () -> { + String userName = + ModuleUserUtils.getOperationUser(request, "getPublishedDataSourceByType"); + DataSource dataSource = + dataSourceInfoService.getPublishedDataSourceByType(dataSourceType, proxyUser); + if (dataSource == null) { + return Message.error( + "No published data source found for type: " + + dataSourceType + + " and proxy user: " + + proxyUser); + } + if (!AuthContext.hasPermission(dataSource, userName)) { + return Message.error("Don't have query permission for data source [没有数据源的查询权限]"); + } + dataSource.setConnectParams(null); + dataSource.setParameter(null); + return Message.ok().data("info", dataSource); + }, + "Fail to get published data source[获取已发布数据源信息失败]"); + } /** * Inner method to insert data source * diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceOperateRestfulApi.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceOperateRestfulApi.java index 4937f56b3d5..f9c26866794 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceOperateRestfulApi.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/DataSourceOperateRestfulApi.java @@ -126,14 +126,6 @@ protected void doConnect(String operator, DataSource dataSource) throws ErrorExc connectParams.replace("password", encrypt); dataSource.setConnectParams(connectParams); } - if (AESUtils.LINKIS_DATASOURCE_AES_SWITCH.getValue() - && !dataSource.getConnectParams().containsKey("isEncrypt")) { - String password = dataSource.getConnectParams().get("password").toString(); - String encrypt = AESUtils.encrypt(password, AESUtils.LINKIS_DATASOURCE_AES_KEY.getValue()); - Map connectParams = dataSource.getConnectParams(); - connectParams.replace("password", encrypt); - dataSource.setConnectParams(connectParams); - } List keyDefinitionList = dataSourceRelateService.getKeyDefinitionsByType(dataSource.getDataSourceTypeId()); dataSource.setKeyDefinitions(keyDefinitionList); diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/RestfulApiHelper.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/RestfulApiHelper.java index b42e2abe3b9..f03afe3d6e2 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/RestfulApiHelper.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/restful/RestfulApiHelper.java @@ -130,6 +130,7 @@ public static Message doAndResponse(TryOperation tryOperation, String failMessag } catch (WarnException e) { return Message.warn(e.getMessage()); } catch (Exception e) { + e.printStackTrace(); return Message.error(failMessage, e); } } diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoService.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoService.java index e037d45fe9a..a7387f5413b 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoService.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoService.java @@ -290,4 +290,13 @@ DataSource getDataSourceInfoForConnect(String dataSourceName, String envId) * @return */ boolean existDataSourceEnv(String dataSourceEnvName); + + /** + * Get published data source by type and proxy user + * + * @param dataSourceType data source type + * @param proxyUser proxy user + * @return data source entity + */ + DataSource getPublishedDataSourceByType(String dataSourceType, String proxyUser); } diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java index 1e5aa95e723..99ddbc98520 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java @@ -739,4 +739,29 @@ private void mergeVersionParams(DataSource dataSource, Long version) { } } } + + @Override + public DataSource getPublishedDataSourceByType(String dataSourceType, String proxyUser) { + try { + // 1. 查询数据源列表 + List dataSourceList = + dataSourceDao.selectDatasourcesByType(dataSourceType, proxyUser); + if (CollectionUtils.isEmpty(dataSourceList)) { + LOG.debug("No datasource found for type:{} and owner:{}", dataSourceType, proxyUser); + return null; + } + // 2. 筛选符合条件的已发布数据源 + return dataSourceList.stream() + .filter( + dataSource -> + (dataSource.getPublishedVersionId() != null) && (!dataSource.isExpire())) + .sorted(Comparator.comparing(DataSource::getCreateTime, Comparator.reverseOrder())) + .findFirst() + .orElse(null); + } catch (Exception e) { + LOG.error( + "Get published datasource failed, type:{}, proxyUser:{}", dataSourceType, proxyUser, e); + return null; + } + } } diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/metadata/query/server/utils/MetadataUtils.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/metadata/query/server/utils/MetadataUtils.java index 0107347e410..3b9337f4d3a 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/metadata/query/server/utils/MetadataUtils.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/java/org/apache/linkis/metadata/query/server/utils/MetadataUtils.java @@ -160,8 +160,7 @@ private static List searchMetaServiceClassFormURI( classNameList.add(className); } } else if (url.endsWith(JAR_SUF_NAME)) { - try { - JarFile jarFile = new JarFile(new File(url)); + try (JarFile jarFile = new JarFile(new File(url))) { Enumeration en = jarFile.entries(); while (en.hasMoreElements()) { String name = en.nextElement().getName(); diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml index d9cf5ff06d4..949c7d90160 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/main/resources/mapper/mysql/DataSouceMapper.xml @@ -256,6 +256,15 @@ t.`name` = #{datasourceTypeName} and d.`create_user` = #{datasourceUser} + UPDATE linkis_ps_dm_datasource diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/test/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoServiceTest.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/test/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoServiceTest.java index c9bd12f7994..0bddd61d42e 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/test/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoServiceTest.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/server/src/test/java/org/apache/linkis/datasourcemanager/core/service/DataSourceInfoServiceTest.java @@ -360,7 +360,7 @@ void testInsertDataSourceParameter() throws ErrorException { Long res = dataSourceInfoService.insertDataSourceParameter( keyDefinitionList, datasourceId, connectParams, username, comment); - assertTrue(expectedVersion == res); + assertTrue(expectedVersion.equals(res)); } @Test diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/hive/pom.xml b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/hive/pom.xml index 7b920dca824..1a069305299 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/hive/pom.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/hive/pom.xml @@ -75,6 +75,12 @@ + + org.apache.avro + avro + 1.11.3 + provided + org.apache.hadoop hadoop-mapreduce-client-core @@ -87,6 +93,10 @@ io.netty netty + + org.apache.avro + avro + @@ -134,8 +144,17 @@ com.github.joshelser dropwizard-metrics-hadoop-metrics2-reporter + + io.airlift + aircompressor + + + io.airlift + aircompressor + ${aircompressor.version} + org.apache.hive hive-exec @@ -245,6 +264,10 @@ org.apache.httpcomponents httpcore + + org.apache.velocity + velocity + diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/pom.xml b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/pom.xml index 135026cccca..44fbe141c47 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/pom.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/pom.xml @@ -27,7 +27,6 @@ linkis-metadata-query-service-jdbc - 42.3.8 0.4.6 diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/StarrocksMetaService.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/StarrocksMetaService.java new file mode 100644 index 00000000000..96b71ddd682 --- /dev/null +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/jdbc/src/main/java/org/apache/linkis/metadata/query/service/StarrocksMetaService.java @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.metadata.query.service; + +import org.apache.linkis.datasourcemanager.common.util.json.Json; +import org.apache.linkis.metadata.query.common.domain.MetaColumnInfo; +import org.apache.linkis.metadata.query.common.service.AbstractDbMetaService; +import org.apache.linkis.metadata.query.common.service.MetadataConnection; +import org.apache.linkis.metadata.query.service.conf.SqlParamsMapper; +import org.apache.linkis.metadata.query.service.starrocks.SqlConnection; + +import java.sql.SQLException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +public class StarrocksMetaService extends AbstractDbMetaService { + + @Override + public MetadataConnection getConnection( + String operator, Map params) throws Exception { + String host = + String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_HOST.getValue(), "")); + Integer port = + (Double.valueOf( + String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_PORT.getValue(), 0)))) + .intValue(); + String username = + String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_USERNAME.getValue(), "")); + String password = + String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_PASSWORD.getValue(), "")); + + String database = + String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_DATABASE.getValue(), "")); + Map extraParams = new HashMap<>(); + Object sqlParamObj = params.get(SqlParamsMapper.PARAM_SQL_EXTRA_PARAMS.getValue()); + if (null != sqlParamObj) { + if (!(sqlParamObj instanceof Map)) { + extraParams = + Json.fromJson(String.valueOf(sqlParamObj), Map.class, String.class, Object.class); + } else { + extraParams = (Map) sqlParamObj; + } + } + assert extraParams != null; + return new MetadataConnection<>( + new SqlConnection(host, port, username, password, database, extraParams)); + } + + @Override + public List queryDatabases(SqlConnection connection) { + try { + return connection.getAllDatabases(); + } catch (SQLException e) { + throw new RuntimeException("Fail to get Sql databases(获取数据库列表失败)", e); + } + } + + @Override + public List queryTables(SqlConnection connection, String database) { + try { + return connection.getAllTables(database); + } catch (SQLException e) { + throw new RuntimeException("Fail to get Sql tables(获取表列表失败)", e); + } + } + + @Override + public List queryColumns( + SqlConnection connection, String database, String table) { + try { + return connection.getColumns(database, table); + } catch (SQLException | ClassNotFoundException e) { + throw new RuntimeException("Fail to get Sql columns(获取字段列表失败)", e); + } + } +} diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/kafka/pom.xml b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/kafka/pom.xml index ed8723c536c..059333e9b19 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/kafka/pom.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/kafka/pom.xml @@ -45,16 +45,26 @@ ${project.version} provided - + + org.slf4j + slf4j-api + ${slf4j.version} + org.apache.kafka kafka-clients - 3.9.1 + ${kafka-client.version} + + + org.xerial.snappy + snappy-java + + - - org.slf4j - slf4j-api + org.xerial.snappy + snappy-java + ${snappy-java.version} diff --git a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/mongodb/src/main/java/org/apache/linkis/metadata/query/service/MongoDbConnection.java b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/mongodb/src/main/java/org/apache/linkis/metadata/query/service/MongoDbConnection.java index 3586982bb52..16de742e454 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/mongodb/src/main/java/org/apache/linkis/metadata/query/service/MongoDbConnection.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-datasource-manager/service/mongodb/src/main/java/org/apache/linkis/metadata/query/service/MongoDbConnection.java @@ -18,6 +18,7 @@ package org.apache.linkis.metadata.query.service; import org.apache.linkis.common.conf.CommonVars; +import org.apache.linkis.common.utils.AESUtils; import org.apache.linkis.metadata.query.common.domain.MetaColumnInfo; import java.io.Closeable; @@ -73,7 +74,9 @@ public MongoDbConnection( String database, Map extraParams) throws ClassNotFoundException, Exception { - connectMessage = new ConnectMessage(host, port, username, password, database, extraParams); + connectMessage = + new ConnectMessage( + host, port, username, AESUtils.isDecryptByConf(password), database, extraParams); conn = getDBConnection(connectMessage, database); } @@ -245,8 +248,9 @@ public ConnectMessage( this.database = database; if (extraParams != null) { this.extraParams = extraParams; + } else { + this.extraParams = new HashMap<>(); } - this.extraParams = extraParams; this.extraParams.put("connectTimeout", CONNECT_TIMEOUT.getValue()); this.extraParams.put("socketTimeout", SOCKET_TIMEOUT.getValue()); } diff --git a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/hive/config/LinkisMybatisConfig.java b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/hive/config/LinkisMybatisConfig.java index bbc7b22579a..7394ea1844d 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/hive/config/LinkisMybatisConfig.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/java/org/apache/linkis/metadata/hive/config/LinkisMybatisConfig.java @@ -34,6 +34,7 @@ import java.util.HashMap; +import com.alibaba.druid.pool.DruidDataSource; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -71,6 +72,17 @@ private DataSource mysqlDataSource() { return DataSourceUtils.buildDataSource(null, null, null); } + private DataSource rangerDataSource() { + DruidDataSource rangerDataSource = + (DruidDataSource) + DataSourceUtils.buildDataSource( + DWSConfig.RANGER_DB_URL.getValue(), + DWSConfig.RANGER_DB_USER.getValue(), + DWSConfig.RANGER_DB_PASSWORD.getValue()); + rangerDataSource.setMaxWait(DWSConfig.RANGER_DB_MAX_WAIT.getValue()); + return rangerDataSource; + } + @Bean(name = "dataSource") @Conditional(DataSourceCondition.class) public DynamicDataSource mutiDataSource() { @@ -80,6 +92,10 @@ public DynamicDataSource mutiDataSource() { HashMap hashMap = new HashMap<>(); hashMap.put(DSEnum.FIRST_DATA_SOURCE, hiveDataSource); hashMap.put(DSEnum.SECONDE_DATA_SOURCE, mysqlDataSource); + if (DWSConfig.RANGER_DB_ENABLE.getValue()) { + DataSource rangerDataSource = rangerDataSource(); + hashMap.put(DSEnum.THIRD_DATA_SOURCE, rangerDataSource); + } dynamicDataSource.setTargetDataSources(hashMap); dynamicDataSource.setDefaultTargetDataSource(mysqlDataSource); return dynamicDataSource; diff --git a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/HiveMetaDao.xml b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/HiveMetaDao.xml index e4e0ed0fefe..15ccfc0d3d8 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/HiveMetaDao.xml +++ b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/HiveMetaDao.xml @@ -40,6 +40,13 @@ where TBLS.TBL_NAME = #{tableName,jdbcType=VARCHAR} and `DBS`.NAME = #{dbName,jdbcType=VARCHAR}; + + + + + + + + + + + + + + + + + + + @@ -154,7 +269,17 @@ and `PARTITIONS`.PART_NAME = #{partitionName,jdbcType=VARCHAR} and TBLS.TBL_NAME = #{tableName,jdbcType=VARCHAR} and `DBS`.NAME = #{dbName,jdbcType=VARCHAR}; + + + + - + /*slave*/ SELECT c.COMMENT, c.COLUMN_NAME, c.TYPE_NAME + FROM COLUMNS_V2 c + INNER JOIN SDS s ON c.CD_ID = s.CD_ID + INNER JOIN TBLS t ON s.SD_ID = t.SD_ID + INNER JOIN DBS d ON t.DB_ID = d.DB_ID + WHERE t.TBL_NAME = #{tableName,jdbcType=VARCHAR} + AND d.NAME = #{dbName,jdbcType=VARCHAR} + ORDER BY c.INTEGER_IDX ASC; + + + + + + + + + + + + + + + + + + + + diff --git a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/RangerDao.xml b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/RangerDao.xml new file mode 100644 index 00000000000..fedfa1fb100 --- /dev/null +++ b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/main/resources/mapper/common/RangerDao.xml @@ -0,0 +1,51 @@ + + + + + + + + + + + + + + + + + + + + + diff --git a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/test/java/org/apache/linkis/metadata/hive/dao/HiveMetaDaoTest.java b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/test/java/org/apache/linkis/metadata/hive/dao/HiveMetaDaoTest.java index a2049a07cc3..92ddc471fc1 100644 --- a/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/test/java/org/apache/linkis/metadata/hive/dao/HiveMetaDaoTest.java +++ b/linkis-public-enhancements/linkis-datasource/linkis-metadata/src/test/java/org/apache/linkis/metadata/hive/dao/HiveMetaDaoTest.java @@ -114,7 +114,7 @@ public void getPartitionsTest() { queryParam.setTableName("employee"); List partitions = hiveMetaDao.getPartitions(queryParam); - Assertions.assertTrue(partitions.size() >= 0); + Assertions.assertTrue(partitions.size() >= 0); // NOSONAR } @Test @@ -126,7 +126,19 @@ public void getColumnsTest() { queryParam.setTableName("employee"); List> columns = hiveMetaDao.getColumns(queryParam); - Assertions.assertTrue(columns.size() >= 0); + Assertions.assertTrue(columns.size() >= 0); // NOSONAR + } + + @Test + @DisplayName("getStorageDescriptionIDByDbTableNameAndUserTest") + public void getStorageDescriptionIDByDbTableNameAndUserTest() { + MetadataQueryParam queryParam = new MetadataQueryParam(); + queryParam.setDbName("default"); + queryParam.setTableName("employee"); + queryParam.setUserName("admin"); + Map list = + hiveMetaDao.getStorageDescriptionIDByDbTableNameAndUserFromDB(queryParam); + Assertions.assertNull(list); } @Test @@ -136,7 +148,7 @@ public void getColumnsByStorageDescriptionIDTest() { MetadataQueryParam queryParam = new MetadataQueryParam(); queryParam.setSdId("1"); List> columns = hiveMetaDao.getColumnsByStorageDescriptionID(queryParam); - Assertions.assertTrue(columns.size() >= 0); + Assertions.assertTrue(columns.size() >= 0); // NOSONAR } @Test diff --git a/linkis-public-enhancements/linkis-datasource/pom.xml b/linkis-public-enhancements/linkis-datasource/pom.xml index b0eae4131a2..307d61cd9bd 100644 --- a/linkis-public-enhancements/linkis-datasource/pom.xml +++ b/linkis-public-enhancements/linkis-datasource/pom.xml @@ -34,6 +34,5 @@ linkis-datasource-manager/service/jdbc linkis-datasource-manager/service/hdfs linkis-datasource-manager/service/mongodb - linkis-metadata diff --git a/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/async/GenericAsyncConsumerQueue.java b/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/async/GenericAsyncConsumerQueue.java index dbb32e3d0d8..deecb6324ad 100644 --- a/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/async/GenericAsyncConsumerQueue.java +++ b/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/async/GenericAsyncConsumerQueue.java @@ -92,7 +92,9 @@ public synchronized void consumer( LOG.trace("wait until the next time: " + triggerTime); consumeLock.lock(); try { - consumeCondition.await(triggerTime - nowMillsTime, TimeUnit.MILLISECONDS); + if (!consumeCondition.await(triggerTime - nowMillsTime, TimeUnit.MILLISECONDS)) { + LOG.error("Interrupt in awaiting action"); + } } catch (InterruptedException e) { LOG.error("Interrupt in awaiting action, message: [" + e.getMessage() + "]", e); continue; diff --git a/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/entity/InsPersistenceLabel.java b/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/entity/InsPersistenceLabel.java index fedc036bba0..869b5744bf8 100644 --- a/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/entity/InsPersistenceLabel.java +++ b/linkis-public-enhancements/linkis-instance-label-server/src/main/java/org/apache/linkis/instance/label/entity/InsPersistenceLabel.java @@ -20,6 +20,7 @@ import org.apache.linkis.manager.label.entity.GenericLabel; import java.util.Date; +import java.util.Objects; /** like: PersistenceLabel in label-manager-common */ public class InsPersistenceLabel extends GenericLabel { @@ -93,4 +94,9 @@ public boolean equals(Object other) { } return false; } + + @Override + public int hashCode() { + return Objects.hash(this.getLabelKey(), this.getStringValue()); + } } diff --git a/linkis-public-enhancements/linkis-instance-label-server/src/test/java/org/apache/linkis/instance/label/dao/InstanceInfoDaoTest.java b/linkis-public-enhancements/linkis-instance-label-server/src/test/java/org/apache/linkis/instance/label/dao/InstanceInfoDaoTest.java new file mode 100644 index 00000000000..13136d5d983 --- /dev/null +++ b/linkis-public-enhancements/linkis-instance-label-server/src/test/java/org/apache/linkis/instance/label/dao/InstanceInfoDaoTest.java @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.instance.label.dao; + +import org.apache.linkis.common.ServiceInstance; +import org.apache.linkis.instance.label.entity.InstanceInfo; + +import org.springframework.beans.factory.annotation.Autowired; + +import java.util.Date; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertTrue; + +public class InstanceInfoDaoTest extends BaseDaoTest { + + @Autowired InstanceInfoDao instanceInfoDao; + + void insert() { + InstanceInfo instanceInfo = new InstanceInfo(); + instanceInfo.setId(1); + instanceInfo.setInstance("testInstance"); + instanceInfo.setApplicationName("testApplicationName"); + instanceInfoDao.insertOne(instanceInfo); + } + + @Test + public void testInsertOne() { + insert(); + ServiceInstance serviceInstance = new ServiceInstance(); + serviceInstance.setInstance("testInstance"); + serviceInstance.setApplicationName("testApplicationName"); + InstanceInfo instanceInfo = instanceInfoDao.getInstanceInfoByServiceInstance(serviceInstance); + assertTrue(instanceInfo.getInstance().equals("testInstance")); + } + + @Test + public void testRemoveInstance() { + insert(); + ServiceInstance serviceInstance = new ServiceInstance(); + serviceInstance.setInstance("testInstance"); + instanceInfoDao.removeInstance(serviceInstance); + ServiceInstance serviceInstances = new ServiceInstance(); + serviceInstances.setInstance("testInstance"); + serviceInstances.setApplicationName("testApplicationName"); + InstanceInfo instanceInfo = instanceInfoDao.getInstanceInfoByServiceInstance(serviceInstances); + assertTrue(instanceInfo == null); + } + + @Test + public void testUpdateInstance() { + insert(); + InstanceInfo instanceInfo = new InstanceInfo(); + instanceInfo.setId(1); + instanceInfo.setInstance("testInstance1"); + instanceInfo.setApplicationName("testApplicationName1"); + instanceInfo.setUpdateTime(new Date()); + instanceInfoDao.updateInstance(instanceInfo); + } + + @Test + public void testGetInstanceInfoByServiceInstance() { + insert(); + ServiceInstance serviceInstance = new ServiceInstance(); + serviceInstance.setInstance("testInstance"); + serviceInstance.setApplicationName("testApplicationName"); + InstanceInfo instanceInfo = instanceInfoDao.getInstanceInfoByServiceInstance(serviceInstance); + assertTrue(instanceInfo.getInstance().equals("testInstance")); + } +} diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/dao/JobDiagnosisMapper.java b/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/dao/JobDiagnosisMapper.java index 24efb1a183c..3d17b217a9b 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/dao/JobDiagnosisMapper.java +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/dao/JobDiagnosisMapper.java @@ -19,6 +19,8 @@ import org.apache.linkis.jobhistory.entity.JobDiagnosis; +import org.apache.ibatis.annotations.Param; + public interface JobDiagnosisMapper { void insert(JobDiagnosis jobDiagnosis); @@ -26,5 +28,6 @@ public interface JobDiagnosisMapper { void update(JobDiagnosis jobDiagnosis); - JobDiagnosis selectById(Long jobHistoryId); + JobDiagnosis selectByJobIdAndSource( + @Param("id") Long jobHistoryId, @Param("diagnosisSource") String diagnosisSource); } diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/entity/JobDiagnosis.java b/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/entity/JobDiagnosis.java index 9a232d1c6bb..5e8093e9510 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/entity/JobDiagnosis.java +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/entity/JobDiagnosis.java @@ -32,6 +32,8 @@ public class JobDiagnosis { private Date updatedTime; private String onlyRead; + private String diagnosisSource; + // Getters and Setters public Long getId() { @@ -82,6 +84,14 @@ public void setOnlyRead(String onlyRead) { this.onlyRead = onlyRead; } + public String getDiagnosisSource() { + return diagnosisSource; + } + + public void setDiagnosisSource(String diagnosisSource) { + this.diagnosisSource = diagnosisSource; + } + @Override public String toString() { return "JobDiagnosis{" @@ -91,11 +101,17 @@ public String toString() { + jobHistoryId + ", diagnosisContent='" + diagnosisContent - + '\'' + + "'" + ", createdTime=" + createdTime + ", updatedTime=" + updatedTime + + ", onlyRead='" + + onlyRead + + "'" + + ", diagnosisSource='" + + diagnosisSource + + "'" + '}'; } } diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java b/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java index 215c2071d76..8bbfb3f6f20 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/java/org/apache/linkis/jobhistory/restful/api/QueryRestfulApi.java @@ -787,9 +787,15 @@ private List getJobhistoryList( value = "diagnosis-query", notes = "query failed task diagnosis msg", response = Message.class) + @ApiImplicitParams({ + @ApiImplicitParam(name = "taskID", dataType = "String"), + @ApiImplicitParam(name = "diagnosisSource", dataType = "String", example = "doctoris"), + }) @RequestMapping(path = "/diagnosis-query", method = RequestMethod.GET) public Message queryFailedTaskDiagnosis( - HttpServletRequest req, @RequestParam(value = "taskID", required = false) String taskID) { + HttpServletRequest req, + @RequestParam(value = "taskID", required = false) String taskID, + @RequestParam(value = "diagnosisSource", required = false) String diagnosisSource) { String username = ModuleUserUtils.getOperationUser(req, "diagnosis-query"); if (StringUtils.isBlank(taskID)) { return Message.error("Invalid jobId cannot be empty"); @@ -797,6 +803,9 @@ public Message queryFailedTaskDiagnosis( if (!QueryUtils.checkNumberValid(taskID)) { throw new LinkisCommonErrorException(21304, "Invalid taskID : " + taskID); } + if (StringUtils.isBlank(diagnosisSource)) { + diagnosisSource = "linkis"; + } JobHistory jobHistory = null; boolean isAdmin = Configuration.isJobHistoryAdmin(username) || Configuration.isAdmin(username); boolean isDepartmentAdmin = Configuration.isDepartmentAdmin(username); @@ -829,14 +838,19 @@ public Message queryFailedTaskDiagnosis( String diagnosisMsg = ""; if (jobHistory != null) { String jobStatus = jobHistory.getStatus(); - JobDiagnosis jobDiagnosis = jobHistoryDiagnosisService.selectByJobId(Long.valueOf(taskID)); + JobDiagnosis jobDiagnosis = + jobHistoryDiagnosisService.selectByJobId(Long.valueOf(taskID), diagnosisSource); if (null == jobDiagnosis) { + if (diagnosisSource.equals("doctoris")) { + return Message.ok().data("diagnosisMsg", diagnosisMsg); + } diagnosisMsg = JobhistoryUtils.getDiagnosisMsg(taskID); jobDiagnosis = new JobDiagnosis(); jobDiagnosis.setJobHistoryId(Long.valueOf(taskID)); jobDiagnosis.setDiagnosisContent(diagnosisMsg); jobDiagnosis.setCreatedTime(new Date()); jobDiagnosis.setUpdatedDate(new Date()); + jobDiagnosis.setDiagnosisSource("linkis"); if (TaskStatus.isComplete(TaskStatus.valueOf(jobStatus))) { jobDiagnosis.setOnlyRead("1"); } diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml b/linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml index a14328a69d3..3241a21c1fb 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/resources/mapper/common/JobDiagnosisMapper.xml @@ -20,8 +20,8 @@ - INSERT INTO linkis_ps_job_history_diagnosis (job_history_id, diagnosis_content, created_time, updated_time, only_read) - VALUES (#{jobHistoryId}, #{diagnosisContent}, #{createdTime}, #{updatedTime}, #{onlyRead}) + INSERT INTO linkis_ps_job_history_diagnosis (job_history_id, diagnosis_content, created_time, updated_time, only_read, diagnosis_source) + VALUES (#{jobHistoryId}, #{diagnosisContent}, #{createdTime}, #{updatedTime}, #{onlyRead}, #{diagnosisSource}) @@ -36,10 +36,15 @@ updated_time = #{updatedTime}, only_read = #{onlyRead} - WHERE job_history_id = #{jobHistoryId} + WHERE job_history_id = #{jobHistoryId} and diagnosis_source = #{diagnosisSource} - + SELECT id, job_history_id, diagnosis_content, created_time, updated_time, only_read, diagnosis_source + FROM linkis_ps_job_history_diagnosis + WHERE job_history_id = #{id} + + AND diagnosis_source = #{diagnosisSource} + diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/JobHistoryDiagnosisService.java b/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/JobHistoryDiagnosisService.java index 3a2ba2326d5..3f335533496 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/JobHistoryDiagnosisService.java +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/JobHistoryDiagnosisService.java @@ -24,13 +24,11 @@ public interface JobHistoryDiagnosisService { - void insert(JobDiagnosis jobDiagnosis); + void insert(JobDiagnosis jobDiagnosis); - void deleteById(Long id); - - void update(JobDiagnosis jobDiagnosis); - - JobDiagnosis selectByJobId(Long id); + void deleteById(Long id); + void update(JobDiagnosis jobDiagnosis); + JobDiagnosis selectByJobId(Long jobId, String diagnosisSource); } diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryDiagnosisServicelmpl.scala b/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryDiagnosisServicelmpl.scala index d3eeff2588c..40c759af004 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryDiagnosisServicelmpl.scala +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryDiagnosisServicelmpl.scala @@ -45,8 +45,8 @@ class JobHistoryDiagnosisServicelmpl extends JobHistoryDiagnosisService with Log jobDiagnosisMapper.update(jobDiagnosis) } - override def selectByJobId(id: lang.Long): JobDiagnosis = { - jobDiagnosisMapper.selectById(id) + override def selectByJobId(jobId: lang.Long, diagnosisSource: String): JobDiagnosis = { + jobDiagnosisMapper.selectByJobIdAndSource(jobId, diagnosisSource) } } diff --git a/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryQueryServiceImpl.scala b/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryQueryServiceImpl.scala index 3e780b29fd6..675c6df49fe 100644 --- a/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryQueryServiceImpl.scala +++ b/linkis-public-enhancements/linkis-jobhistory/src/main/scala/org/apache/linkis/jobhistory/service/impl/JobHistoryQueryServiceImpl.scala @@ -22,6 +22,7 @@ import org.apache.linkis.common.utils.{Logging, Utils} import org.apache.linkis.governance.common.conf.GovernanceCommonConf import org.apache.linkis.governance.common.constant.job.JobRequestConstants import org.apache.linkis.governance.common.entity.job.{ + JobDiagnosisRequest, JobRequest, JobRequestWithDetail, QueryException, @@ -33,7 +34,9 @@ import org.apache.linkis.jobhistory.conf.JobhistoryConfiguration import org.apache.linkis.jobhistory.conversions.TaskConversions._ import org.apache.linkis.jobhistory.dao.JobHistoryMapper import org.apache.linkis.jobhistory.entity.{JobHistory, QueryJobHistory} +import org.apache.linkis.jobhistory.entity.JobDiagnosis import org.apache.linkis.jobhistory.errorcode.JobhistoryErrorCodeSummary +import org.apache.linkis.jobhistory.service.JobHistoryDiagnosisService import org.apache.linkis.jobhistory.service.JobHistoryQueryService import org.apache.linkis.jobhistory.transitional.TaskStatus import org.apache.linkis.jobhistory.util.QueryUtils @@ -68,6 +71,9 @@ class JobHistoryQueryServiceImpl extends JobHistoryQueryService with Logging { @Autowired private var jobHistoryMapper: JobHistoryMapper = _ + @Autowired + private var jobHistoryDiagnosisService: JobHistoryDiagnosisService = _ + private val unDoneTaskCache: Cache[String, Integer] = CacheBuilder .newBuilder() .concurrencyLevel(5) @@ -77,6 +83,49 @@ class JobHistoryQueryServiceImpl extends JobHistoryQueryService with Logging { .recordStats() .build() + @Receiver + def JobDiagnosisReqInsert(jobDiagnosisRequest: JobDiagnosisReqInsert): JobRespProtocol = { + logger.info(s"insert job diagnosis: ${jobDiagnosisRequest.toString}") + val jobResp = new JobRespProtocol + Utils.tryCatch { + // 先查询是否已存在该任务的诊断记录 + val jobid = jobDiagnosisRequest.jobReq.getJobHistoryId + val content = jobDiagnosisRequest.jobReq.getDiagnosisContent + val diagnosisSource = jobDiagnosisRequest.jobReq.getDiagnosisSource + var jobDiagnosis = jobHistoryDiagnosisService.selectByJobId(jobid, diagnosisSource) + + if (jobDiagnosis == null) { + // 创建新的诊断记录 + jobDiagnosis = new JobDiagnosis + jobDiagnosis.setJobHistoryId(jobid) + jobDiagnosis.setCreatedTime(new Date) + } + // 更新诊断内容和来源 + jobDiagnosis.setDiagnosisContent(content) + jobDiagnosis.setDiagnosisSource(diagnosisSource) + jobDiagnosis.setOnlyRead("1") + jobDiagnosis.setUpdatedDate(new Date) + + // 保存诊断记录 + if (jobDiagnosis.getId == null) { + jobHistoryDiagnosisService.insert(jobDiagnosis) + } else { + jobHistoryDiagnosisService.update(jobDiagnosis) + } + + jobResp.setStatus(0) + jobResp.setMsg("insert diagnosis success") + } { case exception: Exception => + logger.error( + s"Failed to insert job diagnosis ${jobDiagnosisRequest.toString}, should be retry", + exception + ) + jobResp.setStatus(2) + jobResp.setMsg(ExceptionUtils.getRootCauseMessage(exception)) + } + jobResp + } + @Receiver override def add(jobReqInsert: JobReqInsert): JobRespProtocol = { logger.info("Insert data into the database(往数据库中插入数据)：" + jobReqInsert.toString) @@ -123,8 +172,14 @@ class JobHistoryQueryServiceImpl extends JobHistoryQueryService with Logging { val oldStatus: String = jobHistoryMapper.selectJobHistoryStatusForUpdate(jobReq.getId) val startUpMap: util.Map[String, AnyRef] = TaskUtils.getStartupMap(jobReqUpdate.jobReq.getParams) - val aiSqlEnable: AnyRef = startUpMap.getOrDefault("linkis.ai.sql.enable", "false") - if (oldStatus != null && !shouldUpdate(oldStatus, jobReq.getStatus, aiSqlEnable.toString)) { + val taskRetrySwitch: AnyRef = startUpMap.getOrDefault("linkis.task.retry.switch", "false") + if ( + oldStatus != null && !shouldUpdate( + oldStatus, + jobReq.getStatus, + taskRetrySwitch.toString + ) + ) { throw new QueryException( 120001, s"jobId:${jobReq.getId}，oldStatus(在数据库中的task状态为)：${oldStatus}," + @@ -631,7 +686,7 @@ class JobHistoryQueryServiceImpl extends JobHistoryQueryService with Logging { jobReq.setMetrics(mergedMetrics) logger.info(s"""Merged metrics for job ${jobReq.getId}: - |added ${requestMetrics.size()} new entries to ${oldMetricsMap + |added ${requestMetrics.size()} new entries to ${oldMetricsMap .size()} existing entries""".stripMargin) } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/http/HttpContextClient.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/http/HttpContextClient.java index fffae20f389..06388e615a6 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/http/HttpContextClient.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/http/HttpContextClient.java @@ -91,9 +91,9 @@ private HttpContextClient(ContextClientConfig contextClientConfig) { } public static HttpContextClient getInstance(ContextClientConfig contextClientConfig) { - if (httpContextClient == null) { - synchronized (HttpContextClient.class) { - if (httpContextClient == null) { + if (httpContextClient == null) { // NOSONAR + synchronized (HttpContextClient.class) { // NOSONAR + if (httpContextClient == null) { // NOSONAR httpContextClient = new HttpContextClient(contextClientConfig); } } @@ -823,11 +823,7 @@ public int batchClearContextByHAID(List idList) throws ErrorException { } if (result instanceof ContextClearByIDResult) { ContextClearByIDResult contextClearByIDResult = (ContextClearByIDResult) result; - if (null != contextClearByIDResult) { - return contextClearByIDResult.num(); - } else { - return 0; - } + return contextClearByIDResult.num(); } else if (null == result) { throw new CSErrorException(80017, "Invalid null result "); } else { @@ -873,11 +869,7 @@ public int batchClearContextByTime( } if (result instanceof ContextClearByTimeResult) { ContextClearByTimeResult contextClearByTimeResult = (ContextClearByTimeResult) result; - if (null != contextClearByTimeResult) { - return contextClearByTimeResult.num(); - } else { - return 0; - } + return contextClearByTimeResult.num(); } else if (null == result) { throw new CSErrorException(80017, "Invalid null result "); } else { diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/listener/ContextClientListenerManager.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/listener/ContextClientListenerManager.java index 3694b38326e..9fc96376779 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/listener/ContextClientListenerManager.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/listener/ContextClientListenerManager.java @@ -26,9 +26,9 @@ public class ContextClientListenerManager { public static ContextClientListenerBus getContextClientListenerBus() { - if (contextClientListenerBus == null) { - synchronized (ContextClientListenerManager.class) { - if (contextClientListenerBus == null) { + if (contextClientListenerBus == null) { // NOSONAR + synchronized (ContextClientListenerManager.class) { // NOSONAR + if (contextClientListenerBus == null) { // NOSONAR contextClientListenerBus = new ContextClientListenerBus(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSMetaDataService.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSMetaDataService.java index 19e55909ae3..703cf3a12d5 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSMetaDataService.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSMetaDataService.java @@ -41,9 +41,9 @@ public class CSMetaDataService implements MetaDataService { private CSMetaDataService() {} public static CSMetaDataService getInstance() { - if (null == csMetaDataService) { - synchronized (CSMetaDataService.class) { - if (null == csMetaDataService) { + if (null == csMetaDataService) { // NOSONAR + synchronized (CSMetaDataService.class) { // NOSONAR + if (null == csMetaDataService) { // NOSONAR csMetaDataService = new CSMetaDataService(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSNodeServiceImpl.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSNodeServiceImpl.java index 3f79011e0e0..4c84bae6040 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSNodeServiceImpl.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSNodeServiceImpl.java @@ -38,9 +38,9 @@ public class CSNodeServiceImpl implements CSNodeService { private CSNodeServiceImpl() {} public static CSNodeService getInstance() { - if (null == csNodeService) { - synchronized (CSNodeServiceImpl.class) { - if (null == csNodeService) { + if (null == csNodeService) { // NOSONAR + synchronized (CSNodeServiceImpl.class) { // NOSONAR + if (null == csNodeService) { // NOSONAR csNodeService = new CSNodeServiceImpl(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResourceService.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResourceService.java index d08b11548e0..7e23c043ce1 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResourceService.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResourceService.java @@ -43,9 +43,9 @@ public class CSResourceService implements ResourceService { private CSResourceService() {} public static CSResourceService getInstance() { - if (null == csResourceService) { - synchronized (CSResourceService.class) { - if (null == csResourceService) { + if (null == csResourceService) { // NOSONAR + synchronized (CSResourceService.class) { // NOSONAR + if (null == csResourceService) { // NOSONAR csResourceService = new CSResourceService(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResultDataServiceImpl.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResultDataServiceImpl.java index bc682863c2f..a2d280aad3d 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResultDataServiceImpl.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSResultDataServiceImpl.java @@ -48,9 +48,9 @@ public class CSResultDataServiceImpl implements CSResultDataService { private CSResultDataServiceImpl() {} public static CSResultDataService getInstance() { - if (null == csResultDataService) { - synchronized (CSResultDataServiceImpl.class) { - if (null == csResultDataService) { + if (null == csResultDataService) { // NOSONAR + synchronized (CSResultDataServiceImpl.class) { // NOSONAR + if (null == csResultDataService) { // NOSONAR csResultDataService = new CSResultDataServiceImpl(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSTableService.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSTableService.java index 0c89f939dec..130c273e408 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSTableService.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSTableService.java @@ -54,9 +54,9 @@ public class CSTableService implements TableService { private CSTableService() {} public static CSTableService getInstance() { - if (null == csTableService) { - synchronized (CSTableService.class) { - if (null == csTableService) { + if (null == csTableService) { // NOSONAR + synchronized (CSTableService.class) { // NOSONAR + if (null == csTableService) { // NOSONAR csTableService = new CSTableService(); } } @@ -104,7 +104,7 @@ public List getUpstreamTables(String contextIDStr, String nodeName) if (null != rsList) logger.info( "contextID: {} and nodeName: {} succeed to get tables size {}", - contextID.getContextId(), + contextID == null ? "NULL" : contextID.getContextId(), nodeName, rsList.size()); return rsList; diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSVariableService.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSVariableService.java index bd45a990c59..425bd9798c4 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSVariableService.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSVariableService.java @@ -99,9 +99,9 @@ public void putVariable(String contextIDStr, String contextKeyStr, LinkisVariabl } public static CSVariableService getInstance() { - if (null == csVariableService) { - synchronized (CSVariableService.class) { - if (null == csVariableService) { + if (null == csVariableService) { // NOSONAR + synchronized (CSVariableService.class) { // NOSONAR + if (null == csVariableService) { // NOSONAR csVariableService = new CSVariableService(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSWorkServiceImpl.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSWorkServiceImpl.java index 5059e68faa5..fc3720175b9 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSWorkServiceImpl.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/CSWorkServiceImpl.java @@ -41,9 +41,9 @@ private CSWorkServiceImpl() {} private static CSWorkService csWorkService = null; public static CSWorkService getInstance() { - if (null == csWorkService) { - synchronized (CSWorkServiceImpl.class) { - if (null == csWorkService) { + if (null == csWorkService) { // NOSONAR + synchronized (CSWorkServiceImpl.class) { // NOSONAR + if (null == csWorkService) { // NOSONAR csWorkService = new CSWorkServiceImpl(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/ContextHistoryClientServiceImpl.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/ContextHistoryClientServiceImpl.java index c0ff85f96a8..63100fb7d1b 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/ContextHistoryClientServiceImpl.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/ContextHistoryClientServiceImpl.java @@ -44,9 +44,9 @@ public class ContextHistoryClientServiceImpl implements ContextHistoryClientServ private ContextHistoryClientServiceImpl() {} public static ContextHistoryClientService getInstance() { - if (null == contextHistoryClientService) { - synchronized (ContextHistoryClientServiceImpl.class) { - if (null == contextHistoryClientService) { + if (null == contextHistoryClientService) { // NOSONAR + synchronized (ContextHistoryClientServiceImpl.class) { // NOSONAR + if (null == contextHistoryClientService) { // NOSONAR contextHistoryClientService = new ContextHistoryClientServiceImpl(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/DefaultSearchService.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/DefaultSearchService.java index 6b60a50dba4..b04bc59d9bd 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/DefaultSearchService.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/DefaultSearchService.java @@ -186,9 +186,9 @@ public List searchUpstreamKeyValue( } public static SearchService getInstance() { - if (null == searchService) { - synchronized (DefaultSearchService.class) { - if (null == searchService) { + if (null == searchService) { // SONAR + synchronized (DefaultSearchService.class) { // NOSONAR + if (null == searchService) { // NOSONAR searchService = new DefaultSearchService(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/LinkisJobDataServiceImpl.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/LinkisJobDataServiceImpl.java index 9be63b07d61..83cb1dde253 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/LinkisJobDataServiceImpl.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/cs/client/service/LinkisJobDataServiceImpl.java @@ -45,9 +45,9 @@ public class LinkisJobDataServiceImpl implements LinkisJobDataService { private LinkisJobDataServiceImpl() {} public static LinkisJobDataService getInstance() { - if (null == linkisJobDataService) { - synchronized (LinkisJobDataServiceImpl.class) { - if (null == linkisJobDataService) { + if (null == linkisJobDataService) { // NOSONAR + synchronized (LinkisJobDataServiceImpl.class) { // NOSONAR + if (null == linkisJobDataService) { // NOSONAR linkisJobDataService = new LinkisJobDataServiceImpl(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/datasource/client/errorcode/DatasourceClientErrorCodeSummary.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/datasource/client/errorcode/DatasourceClientErrorCodeSummary.java index 4debe0ab308..ac195e85939 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/datasource/client/errorcode/DatasourceClientErrorCodeSummary.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/datasource/client/errorcode/DatasourceClientErrorCodeSummary.java @@ -31,6 +31,8 @@ public enum DatasourceClientErrorCodeSummary implements LinkisErrorCode { IP_NEEDED(31000, "ip is needed(ip为空)!"), PORT_NEEDED(31000, "port is needed(port为空)!"), OWNER_NEEDED(31000, "owner is needed(owner为空)!"), + DATASOURCE_TYPE_NEEDED(31000, "dataSourceType is needed(dataSourceType为空)!"), + PROXY_USER_NEEDED(31000, "proxyUser is needed(proxyUser为空)!"), DATASOURCE_NEEDED(31000, "datasourceTypeName is needed(datasourceTypeName为空)!"), CANNOT_SOURCE( 31000, "Cannot encode the name of data source:{0} for request(无法对请求的数据源名称进行编码：{0})"), diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/handler/LinkisErrorCodeHandler.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/handler/LinkisErrorCodeHandler.java index aa444ac55d9..8054485442f 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/handler/LinkisErrorCodeHandler.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/handler/LinkisErrorCodeHandler.java @@ -63,9 +63,9 @@ public class LinkisErrorCodeHandler new ThreadPoolExecutor.AbortPolicy()); public static LinkisErrorCodeHandler getInstance() { - if (null == linkisErrorCodeHandler) { - synchronized (LinkisErrorCodeHandler.class) { - if (null == linkisErrorCodeHandler) { + if (null == linkisErrorCodeHandler) { // NOSONAR + synchronized (LinkisErrorCodeHandler.class) { // NOSONAR + if (null == linkisErrorCodeHandler) { // NOSONAR linkisErrorCodeHandler = new LinkisErrorCodeHandler(); } } @@ -121,8 +121,9 @@ public void handle(String logFilePath, int type) { LOGGER.info("begin to handle logFilePath {}", logFilePath); // At the end of the file, write "error code information is being generated for you". - try { - writeToFile(logFilePath, ERROR_CODE_PRE); + try (BufferedWriter bufferedWriter = + new BufferedWriter(new OutputStreamWriter(new FileOutputStream(logFilePath, true)))) { + writeToFile(bufferedWriter, ERROR_CODE_PRE); } catch (Exception e) { // If there is a write exception, skip this question directly. LOGGER.error("Failed to append error code to log file {}", logFilePath, e); @@ -146,13 +147,14 @@ public void handle(String logFilePath, int type) { LOGGER.error("failed to handle log file {} ", logFilePath, e); return; } - try { + try (BufferedWriter bufferedWriter = + new BufferedWriter(new OutputStreamWriter(new FileOutputStream(logFilePath, true)))) { if (errorCodeSet.size() == 0) { - writeToFile(logFilePath, ERROR_CODE_FAILED); + writeToFile(bufferedWriter, ERROR_CODE_FAILED); } else { - writeToFile(logFilePath, ERROR_CODE_OK); + writeToFile(bufferedWriter, ERROR_CODE_OK); List retErrorCodes = new ArrayList<>(errorCodeSet); - writeToFile(logFilePath, retErrorCodes.toString()); + writeToFile(bufferedWriter, retErrorCodes.toString()); } } catch (Exception e) { LOGGER.error("failed to write to errorcodes to {} ", logFilePath, e); @@ -162,12 +164,9 @@ public void handle(String logFilePath, int type) { LOGGER.info("put handle into threadPool"); } - private void writeToFile(String filePath, String content) throws Exception { - BufferedWriter bufferedWriter = - new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filePath, true))); + private void writeToFile(BufferedWriter bufferedWriter, String content) throws Exception { bufferedWriter.write(content); bufferedWriter.write(NEW_LINE); - bufferedWriter.close(); } /** diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/manager/LinkisErrorCodeManager.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/manager/LinkisErrorCodeManager.java index a401beda72d..ae587eb4c34 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/manager/LinkisErrorCodeManager.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/manager/LinkisErrorCodeManager.java @@ -32,9 +32,9 @@ public class LinkisErrorCodeManager { private LinkisErrorCodeManager() {} public static LinkisErrorCodeManager getInstance() { - if (linkisErrorCodeManager == null) { - synchronized (LinkisErrorCodeManager.class) { - if (linkisErrorCodeManager == null) { + if (linkisErrorCodeManager == null) { // NOSONAR + synchronized (LinkisErrorCodeManager.class) { // NOSONAR + if (linkisErrorCodeManager == null) { // NOSONAR linkisErrorCodeManager = new LinkisErrorCodeManager(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/synchronizer/LinkisErrorCodeSynchronizer.java b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/synchronizer/LinkisErrorCodeSynchronizer.java index 2b3a3f9fed0..025483a75a1 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/synchronizer/LinkisErrorCodeSynchronizer.java +++ b/linkis-public-enhancements/linkis-pes-client/src/main/java/org/apache/linkis/errorcode/client/synchronizer/LinkisErrorCodeSynchronizer.java @@ -83,9 +83,9 @@ private void init() { } public static LinkisErrorCodeSynchronizer getInstance() { - if (linkisErrorCodeSynchronizer == null) { - synchronized (LinkisErrorCodeSynchronizer.class) { - if (linkisErrorCodeSynchronizer == null) { + if (linkisErrorCodeSynchronizer == null) { // NOSONAR + synchronized (LinkisErrorCodeSynchronizer.class) { // NOSONAR + if (linkisErrorCodeSynchronizer == null) { // NOSONAR linkisErrorCodeSynchronizer = new LinkisErrorCodeSynchronizer(); } } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/DataSourceRemoteClient.scala b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/DataSourceRemoteClient.scala index 3d541f95636..ba66f67062a 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/DataSourceRemoteClient.scala +++ b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/DataSourceRemoteClient.scala @@ -67,4 +67,9 @@ trait DataSourceRemoteClient extends RemoteClient { ): UpdateDataSourceParameterResult def getKeyDefinitionsByType(action: GetKeyTypeDatasourceAction): GetKeyTypeDatasourceResult + + def getPublishedDataSourceByType( + action: GetPublishedDataSourceByTypeAction + ): GetPublishedDataSourceByTypeResult + } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/impl/LinkisDataSourceRemoteClient.scala b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/impl/LinkisDataSourceRemoteClient.scala index 068e6235fda..dedf8f6eb9f 100644 --- a/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/impl/LinkisDataSourceRemoteClient.scala +++ b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/impl/LinkisDataSourceRemoteClient.scala @@ -151,4 +151,9 @@ class LinkisDataSourceRemoteClient(clientConfig: DWSClientConfig, clientName: St action: GetKeyTypeDatasourceAction ): GetKeyTypeDatasourceResult = execute(action).asInstanceOf[GetKeyTypeDatasourceResult] + override def getPublishedDataSourceByType( + action: GetPublishedDataSourceByTypeAction + ): GetPublishedDataSourceByTypeResult = + execute(action).asInstanceOf[GetPublishedDataSourceByTypeResult] + } diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/request/GetPublishedDataSourceByTypeAction.scala b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/request/GetPublishedDataSourceByTypeAction.scala new file mode 100644 index 00000000000..5ab82a8cde2 --- /dev/null +++ b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/request/GetPublishedDataSourceByTypeAction.scala @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.datasource.client.request + +import org.apache.linkis.datasource.client.config.DatasourceClientConfig.DATA_SOURCE_SERVICE_MODULE +import org.apache.linkis.datasource.client.errorcode.DatasourceClientErrorCodeSummary._ +import org.apache.linkis.datasource.client.exception.DataSourceClientBuilderException +import org.apache.linkis.httpclient.request.GetAction + +class GetPublishedDataSourceByTypeAction extends GetAction with DataSourceAction { + private var dataSourceType: String = _ + private var proxyUser: String = _ + + override def suffixURLs: Array[String] = + Array(DATA_SOURCE_SERVICE_MODULE.getValue, "published", "type") + + override def setUser(user: String): Unit = this.user = user + + override def getUser: String = this.user + + private var user: String = _ + +} + +object GetPublishedDataSourceByTypeAction { + def builder(): Builder = new Builder + + class Builder private[GetPublishedDataSourceByTypeAction] () { + private var dataSourceType: String = _ + private var proxyUser: String = _ + private var system: String = _ + private var user: String = _ + + def setUser(user: String): Builder = { + this.user = user + this + } + + def setDataSourceType(dataSourceType: String): Builder = { + this.dataSourceType = dataSourceType + this + } + + def setProxyUser(proxyUser: String): Builder = { + this.proxyUser = proxyUser + this + } + + def setSystem(system: String): Builder = { + this.system = system + this + } + + def build(): GetPublishedDataSourceByTypeAction = { + if (dataSourceType == null) { + throw new DataSourceClientBuilderException(DATASOURCE_TYPE_NEEDED.getErrorDesc) + } + if (proxyUser == null) { + throw new DataSourceClientBuilderException(PROXY_USER_NEEDED.getErrorDesc) + } + if (system == null) throw new DataSourceClientBuilderException(SYSTEM_NEEDED.getErrorDesc) + if (user == null) throw new DataSourceClientBuilderException(USER_NEEDED.getErrorDesc) + + val action = new GetPublishedDataSourceByTypeAction + action.dataSourceType = this.dataSourceType + action.proxyUser = this.proxyUser + action.setParameter("dataSourceType", dataSourceType) + action.setParameter("proxyUser", proxyUser) + action.setParameter("system", system) + action.setUser(user) + action + } + + } + +} diff --git a/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/response/GetPublishedDataSourceByTypeResult.scala b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/response/GetPublishedDataSourceByTypeResult.scala new file mode 100644 index 00000000000..5c8bd2997db --- /dev/null +++ b/linkis-public-enhancements/linkis-pes-client/src/main/scala/org/apache/linkis/datasource/client/response/GetPublishedDataSourceByTypeResult.scala @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.datasource.client.response + +import org.apache.linkis.datasourcemanager.common.domain.DataSource +import org.apache.linkis.httpclient.dws.DWSHttpClient +import org.apache.linkis.httpclient.dws.annotation.DWSHttpMessageResult +import org.apache.linkis.httpclient.dws.response.DWSResult + +import scala.beans.BeanProperty + +@DWSHttpMessageResult("/api/rest_j/v\\d+/data-source-manager/published/type") +class GetPublishedDataSourceByTypeResult extends DWSResult { + @BeanProperty var info: java.util.Map[String, Any] = _ + + def getDataSource: DataSource = { + val str = DWSHttpClient.jacksonJson.writeValueAsString(info) + DWSHttpClient.jacksonJson.readValue(str, classOf[DataSource]) + } + +} diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/helper/ContextSerializationHelper.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/helper/ContextSerializationHelper.java index 6c22c075c0a..253c7cf1712 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/helper/ContextSerializationHelper.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/helper/ContextSerializationHelper.java @@ -71,9 +71,9 @@ private void init() throws CSErrorException { private static ContextSerializationHelper contextSerializationHelper = null; public static ContextSerializationHelper getInstance() { - if (contextSerializationHelper == null) { - synchronized (ContextSerializationHelper.class) { - if (contextSerializationHelper == null) { + if (contextSerializationHelper == null) { // NOSONAR + synchronized (ContextSerializationHelper.class) { // NOSONAR + if (contextSerializationHelper == null) { // NOSONAR contextSerializationHelper = new ContextSerializationHelper(); try { contextSerializationHelper.init(); diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/context/LinkisHAWorkFlowContextIDSerializer.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/context/LinkisHAWorkFlowContextIDSerializer.java index 7a967af9f9c..2e752df1008 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/context/LinkisHAWorkFlowContextIDSerializer.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/context/LinkisHAWorkFlowContextIDSerializer.java @@ -37,7 +37,7 @@ public LinkisHAWorkFlowContextID fromJson(String json) throws CSErrorException { @Override public boolean accepts(Object obj) { - if (null != obj && obj.getClass().getName().equals(LinkisHAWorkFlowContextID.class.getName())) { + if (null != obj && obj.getClass().isAssignableFrom(LinkisHAWorkFlowContextID.class)) { return true; } return false; diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableLineageSerializer.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableLineageSerializer.java index 7b83f7cf5a4..5fda484c922 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableLineageSerializer.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableLineageSerializer.java @@ -64,6 +64,6 @@ public String getType() { @Override public boolean accepts(Object obj) { - return null != obj && obj.getClass().getName().equals(CSTableLineageHistory.class.getName()); + return null != obj && obj.getClass().isAssignableFrom(CSTableLineageHistory.class); } } diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableMetadataHistorySerializer.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableMetadataHistorySerializer.java index d6aafb06348..6c97a2840f4 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableMetadataHistorySerializer.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/common/serialize/impl/history/metadata/CSTableMetadataHistorySerializer.java @@ -57,7 +57,6 @@ public String getType() { @Override public boolean accepts(Object obj) { - return null != obj - && obj.getClass().getName().equals(CSTableMetadataContextHistory.class.getName()); + return null != obj && obj.getClass().isAssignableFrom(CSTableMetadataContextHistory.class); } } diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/ListenerBus/ContextAsyncListenerBus.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/ListenerBus/ContextAsyncListenerBus.java index c6e4116c1dc..7ba05ad96fb 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/ListenerBus/ContextAsyncListenerBus.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/ListenerBus/ContextAsyncListenerBus.java @@ -43,9 +43,9 @@ public void doPostEvent(L listener, E event) { private static ContextAsyncListenerBus contextAsyncListenerBus = null; public static ContextAsyncListenerBus getInstance() { - if (contextAsyncListenerBus == null) { - synchronized (ContextAsyncListenerBus.class) { - if (contextAsyncListenerBus == null) { + if (contextAsyncListenerBus == null) { // NOSONAR + synchronized (ContextAsyncListenerBus.class) { // NOSONAR + if (contextAsyncListenerBus == null) { // NOSONAR contextAsyncListenerBus = new ContextAsyncListenerBus(); contextAsyncListenerBus.start(); } diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextIDCallbackEngine.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextIDCallbackEngine.java index 3910deef775..e0ade199a3d 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextIDCallbackEngine.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextIDCallbackEngine.java @@ -147,9 +147,9 @@ public void onEventError(Event event, Throwable t) {} private DefaultContextIDCallbackEngine() {} public static DefaultContextIDCallbackEngine getInstance() { - if (singleDefaultContextIDCallbackEngine == null) { - synchronized (DefaultContextIDCallbackEngine.class) { - if (singleDefaultContextIDCallbackEngine == null) { + if (singleDefaultContextIDCallbackEngine == null) { // NOSONAR + synchronized (DefaultContextIDCallbackEngine.class) { // NOSONAR + if (singleDefaultContextIDCallbackEngine == null) { // NOSONAR singleDefaultContextIDCallbackEngine = new DefaultContextIDCallbackEngine(); DefaultContextListenerManager instanceContextListenerManager = DefaultContextListenerManager.getInstance(); diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextKeyCallbackEngine.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextKeyCallbackEngine.java index c78fdc58d2b..bfa6ff5159c 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextKeyCallbackEngine.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/callback/imp/DefaultContextKeyCallbackEngine.java @@ -61,7 +61,7 @@ public void registerClient(ListenerDomain listenerDomain) { } } // 针对cskey生成一个bean，cskey对应的value值目前为空 - if (contextKey != null) { + if (contextKey != null && contextID != null) { ContextKeyValueBean contextKeyValueBean = new ContextKeyValueBean(); contextKeyValueBean.setCsKey(contextKey); contextKeyValueBean.setCsID(contextID); @@ -160,9 +160,9 @@ public void onEventError(Event event, Throwable t) {} private DefaultContextKeyCallbackEngine() {} public static DefaultContextKeyCallbackEngine getInstance() { - if (singleDefaultContextKeyCallbackEngine == null) { - synchronized (DefaultContextKeyCallbackEngine.class) { - if (singleDefaultContextKeyCallbackEngine == null) { + if (singleDefaultContextKeyCallbackEngine == null) { // NOSONAR + synchronized (DefaultContextKeyCallbackEngine.class) { // NOSONAR + if (singleDefaultContextKeyCallbackEngine == null) { // NOSONAR singleDefaultContextKeyCallbackEngine = new DefaultContextKeyCallbackEngine(); DefaultContextListenerManager instanceContextListenerManager = DefaultContextListenerManager.getInstance(); diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/manager/imp/DefaultContextListenerManager.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/manager/imp/DefaultContextListenerManager.java index 51b9e78d082..6dddb464b86 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/manager/imp/DefaultContextListenerManager.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/cs/listener/manager/imp/DefaultContextListenerManager.java @@ -48,9 +48,9 @@ public DefaultContextKeyCallbackEngine getContextKeyCallbackEngine() { private DefaultContextListenerManager() {} public static DefaultContextListenerManager getInstance() { - if (singleDefaultContextListenerManager == null) { - synchronized (DefaultContextListenerManager.class) { - if (singleDefaultContextListenerManager == null) { + if (singleDefaultContextListenerManager == null) { // NOSONAR + synchronized (DefaultContextListenerManager.class) { // NOSONAR + if (singleDefaultContextListenerManager == null) { // NOSONAR singleDefaultContextListenerManager = new DefaultContextListenerManager(); } } diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/metadata/query/common/cache/ConnCacheManager.java b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/metadata/query/common/cache/ConnCacheManager.java index 65fbcdf3734..914f9f03760 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/metadata/query/common/cache/ConnCacheManager.java +++ b/linkis-public-enhancements/linkis-pes-common/src/main/java/org/apache/linkis/metadata/query/common/cache/ConnCacheManager.java @@ -29,9 +29,9 @@ public class ConnCacheManager implements CacheManager { private ConnCacheManager() {} public static CacheManager custom() { - if (null == manager) { - synchronized (ConnCacheManager.class) { - if (null == manager) { + if (null == manager) { // NOSONAR + synchronized (ConnCacheManager.class) { // NOSONAR + if (null == manager) { // NOSONAR manager = new ConnCacheManager(); } } diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/RequestPythonModuleProtocol.scala b/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/RequestPythonModuleProtocol.scala index 27cd071fb7c..d1ed9432578 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/RequestPythonModuleProtocol.scala +++ b/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/RequestPythonModuleProtocol.scala @@ -26,3 +26,8 @@ case class RequestPythonModuleProtocol(userName: String, engineType: String) with CacheableProtocol with PythonModuleProtocol with UdfProtocol + +case class RequestPythonInfo(pythonModule: String, username: String) + extends RetryableProtocol + with CacheableProtocol + with UdfProtocol diff --git a/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/ResponsePythonModuleProtocol.scala b/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/ResponsePythonModuleProtocol.scala index 4ff5c0f8db1..84a1ca27a88 100644 --- a/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/ResponsePythonModuleProtocol.scala +++ b/linkis-public-enhancements/linkis-pes-common/src/main/scala/org/apache/linkis/udf/api/rpc/ResponsePythonModuleProtocol.scala @@ -31,3 +31,5 @@ class ResponsePythonModuleProtocol(val pythonModules: java.util.List[PythonModul } } + +case class ResponsePythonInfo(pythonModuleInfoVO: PythonModuleInfoVO) extends PythonModuleProtocol diff --git a/linkis-public-enhancements/linkis-pes-publicservice/pom.xml b/linkis-public-enhancements/linkis-pes-publicservice/pom.xml index 86e45f43f86..756127ef4b2 100644 --- a/linkis-public-enhancements/linkis-pes-publicservice/pom.xml +++ b/linkis-public-enhancements/linkis-pes-publicservice/pom.xml @@ -23,20 +23,18 @@ ${revision} ../../pom.xml - linkis-pes-publicservice jar - org.apache.linkis - linkis-module + linkis-pes-client ${project.version} - org.ow2.asm - asm + org.apache.linkis + linkis-storage @@ -47,23 +45,6 @@ ${project.version} - - org.apache.linkis - linkis-pes-common - ${project.version} - - - - org.apache.linkis - linkis-pes-client - ${project.version} - - - org.apache.linkis - linkis-storage - - - org.apache.linkis linkis-storage @@ -88,10 +69,43 @@ provided + + org.apache.hbase + hbase-server + ${hbase.version} + + + org.apache.hadoop + hadoop-common + + + org.apache.hadoop + hadoop-auth + + + org.apache.hadoop + hadoop-mapreduce-client-core + + + org.jruby.jcodings + jcodings + + + org.jruby.joni + joni + + + org.mortbay.jetty + jetty + + + + org.apache.linkis - linkis-io_file-client + linkis-computation-governance-common ${project.version} + provided @@ -107,37 +121,6 @@ net.alchim31.maven scala-maven-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - false - - false - out - false - false - - src/main/assembly/distribution.xml - - - - - make-assembly - - single - - package - - - src/main/assembly/distribution.xml - - - - - - - diff --git a/linkis-public-enhancements/linkis-pes-publicservice/src/main/assembly/distribution.xml b/linkis-public-enhancements/linkis-pes-publicservice/src/main/assembly/distribution.xml index 174686fb773..8044e4c1c7a 100644 --- a/linkis-public-enhancements/linkis-pes-publicservice/src/main/assembly/distribution.xml +++ b/linkis-public-enhancements/linkis-pes-publicservice/src/main/assembly/distribution.xml @@ -6,9 +6,9 @@ ~ The ASF licenses this file to You under the Apache License, Version 2.0 ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at - ~ + ~ ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ + ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. @@ -18,12 +18,12 @@ - linkis-pes-publicservice + linkis-et-monitor dir false - linkis-pes-publicservice + linkis-et-monitor @@ -35,23 +35,49 @@ false false true - + + antlr:antlr:jar aopalliance:aopalliance:jar + asm:asm:jar + cglib:cglib:jar + com.amazonaws:aws-java-sdk-autoscaling:jar + com.amazonaws:aws-java-sdk-core:jar + com.amazonaws:aws-java-sdk-ec2:jar + com.amazonaws:aws-java-sdk-route53:jar + com.amazonaws:aws-java-sdk-sts:jar + com.amazonaws:jmespath-java:jar com.fasterxml.jackson.core:jackson-annotations:jar com.fasterxml.jackson.core:jackson-core:jar com.fasterxml.jackson.core:jackson-databind:jar + com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar + com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar + com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar + com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar + com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar + com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar + com.fasterxml.jackson.module:jackson-module-parameter-names:jar com.fasterxml.jackson.module:jackson-module-paranamer:jar com.fasterxml.jackson.module:jackson-module-scala_2.11:jar + com.github.andrewoma.dexx:dexx-collections:jar + com.github.vlsi.compactmap:compactmap:jar + com.google.code.findbugs:annotations:jar + com.google.code.findbugs:jsr305:jar com.google.code.gson:gson:jar com.google.guava:guava:jar + com.google.inject:guice:jar com.google.protobuf:protobuf-java:jar com.netflix.archaius:archaius-core:jar + com.netflix.eureka:eureka-client:jar + com.netflix.eureka:eureka-core:jar com.netflix.hystrix:hystrix-core:jar com.netflix.netflix-commons:netflix-commons-util:jar + com.netflix.netflix-commons:netflix-eventbus:jar + com.netflix.netflix-commons:netflix-infix:jar com.netflix.netflix-commons:netflix-statistics:jar com.netflix.ribbon:ribbon:jar com.netflix.ribbon:ribbon-core:jar + com.netflix.ribbon:ribbon-eureka:jar com.netflix.ribbon:ribbon-httpclient:jar com.netflix.ribbon:ribbon-loadbalancer:jar com.netflix.ribbon:ribbon-transport:jar @@ -60,21 +86,31 @@ com.sun.jersey.contribs:jersey-apache-client4:jar com.sun.jersey:jersey-client:jar com.sun.jersey:jersey-core:jar + com.sun.jersey:jersey-json:jar com.sun.jersey:jersey-server:jar + com.sun.jersey:jersey-servlet:jar com.sun.xml.bind:jaxb-impl:jar + com.thoughtworks.paranamer:paranamer:jar + com.thoughtworks.xstream:xstream:jar org.apache.linkis:linkis-common:jar + org.apache.linkis:linkis-module:jar commons-beanutils:commons-beanutils:jar commons-beanutils:commons-beanutils-core:jar commons-cli:commons-cli:jar + commons-codec:commons-codec:jar commons-collections:commons-collections:jar commons-configuration:commons-configuration:jar commons-daemon:commons-daemon:jar commons-dbcp:commons-dbcp:jar commons-digester:commons-digester:jar + commons-httpclient:commons-httpclient:jar commons-io:commons-io:jar + commons-jxpath:commons-jxpath:jar commons-lang:commons-lang:jar + commons-logging:commons-logging:jar commons-net:commons-net:jar commons-pool:commons-pool:jar + io.micrometer:micrometer-core:jar io.netty:netty:jar io.netty:netty-all:jar io.netty:netty-buffer:jar @@ -93,14 +129,22 @@ javax.inject:javax.inject:jar javax.servlet:javax.servlet-api:jar javax.servlet.jsp:jsp-api:jar + javax.validation:validation-api:jar + javax.websocket:javax.websocket-api:jar + javax.ws.rs:javax.ws.rs-api:jar javax.xml.bind:jaxb-api:jar javax.xml.stream:stax-api:jar + joda-time:joda-time:jar + log4j:log4j:jar + mysql:mysql-connector-java:jar net.databinder.dispatch:dispatch-core_2.11:jar net.databinder.dispatch:dispatch-json4s-jackson_2.11:jar org.antlr:antlr-runtime:jar org.antlr:stringtemplate:jar org.apache.commons:commons-compress:jar + org.apache.commons:commons-math:jar org.apache.commons:commons-math3:jar + org.apache.curator:curator-client:jar org.apache.curator:curator-framework:jar org.apache.curator:curator-recipes:jar org.apache.directory.api:api-asn1-api:jar @@ -112,12 +156,22 @@ org.apache.hadoop:hadoop-common:jar org.apache.hadoop:hadoop-hdfs:jar org.apache.htrace:htrace-core:jar + org.apache.httpcomponents:httpclient:jar + org.apache.httpcomponents:httpcore:jar org.apache.logging.log4j:log4j-api:jar + org.apache.logging.log4j:log4j-core:jar + org.apache.logging.log4j:log4j-jul:jar + org.apache.logging.log4j:log4j-slf4j-impl:jar org.apache.zookeeper:zookeeper:jar org.aspectj:aspectjweaver:jar - org.bouncycastle:bcpkix-jdk18on:jar - org.bouncycastle:bcprov-jdk18on:jar - org.codehaus.jettison:jettison:jar + org.bouncycastle:bcpkix-jdk15on:jar + org.bouncycastle:bcprov-jdk15on:jar + org.codehaus.jackson:jackson-jaxrs:jar + org.codehaus.jackson:jackson-xc:jar + org.codehaus.woodstox:stax2-api:jar + org.codehaus.woodstox:woodstox-core-asl:jar + org.eclipse.jetty:jetty-annotations:jar + org.eclipse.jetty:jetty-client:jar org.eclipse.jetty:jetty-continuation:jar org.eclipse.jetty:jetty-http:jar org.eclipse.jetty:jetty-io:jar @@ -126,17 +180,61 @@ org.eclipse.jetty:jetty-security:jar org.eclipse.jetty:jetty-server:jar org.eclipse.jetty:jetty-servlet:jar + org.eclipse.jetty:jetty-servlets:jar org.eclipse.jetty:jetty-util:jar org.eclipse.jetty:jetty-webapp:jar org.eclipse.jetty:jetty-xml:jar + org.eclipse.jetty.websocket:javax-websocket-client-impl:jar + org.eclipse.jetty.websocket:javax-websocket-server-impl:jar + org.eclipse.jetty.websocket:websocket-api:jar + org.eclipse.jetty.websocket:websocket-client:jar + org.eclipse.jetty.websocket:websocket-common:jar + org.eclipse.jetty.websocket:websocket-server:jar + org.eclipse.jetty.websocket:websocket-servlet:jar org.fusesource.leveldbjni:leveldbjni-all:jar + org.glassfish.hk2:class-model:jar + org.glassfish.hk2:config-types:jar + org.glassfish.hk2.external:aopalliance-repackaged:jar + org.glassfish.hk2.external:asm-all-repackaged:jar + org.glassfish.hk2.external:bean-validator:jar + org.glassfish.hk2.external:javax.inject:jar + org.glassfish.hk2:hk2:jar + org.glassfish.hk2:hk2-api:jar + org.glassfish.hk2:hk2-config:jar + org.glassfish.hk2:hk2-core:jar + org.glassfish.hk2:hk2-locator:jar + org.glassfish.hk2:hk2-runlevel:jar + org.glassfish.hk2:hk2-utils:jar + org.glassfish.hk2:osgi-resource-locator:jar + org.glassfish.hk2:spring-bridge:jar + org.glassfish.jersey.bundles:jaxrs-ri:jar + org.glassfish.jersey.bundles.repackaged:jersey-guava:jar + org.glassfish.jersey.containers:jersey-container-servlet:jar + org.glassfish.jersey.containers:jersey-container-servlet-core:jar + org.glassfish.jersey.core:jersey-client:jar + org.glassfish.jersey.core:jersey-common:jar + org.glassfish.jersey.core:jersey-server:jar + org.glassfish.jersey.ext:jersey-entity-filtering:jar + org.glassfish.jersey.ext:jersey-spring3:jar + org.glassfish.jersey.media:jersey-media-jaxb:jar + org.glassfish.jersey.media:jersey-media-json-jackson:jar + org.glassfish.jersey.media:jersey-media-multipart:jar org.hdrhistogram:HdrHistogram:jar + org.javassist:javassist:jar org.json4s:json4s-ast_2.11:jar org.json4s:json4s-core_2.11:jar org.json4s:json4s-jackson_2.11:jar org.jsoup:jsoup:jar + org.jvnet.mimepull:mimepull:jar + org.jvnet:tiger-types:jar + org.latencyutils:LatencyUtils:jar + org.mortbay.jasper:apache-el:jar org.mortbay.jetty:jetty:jar org.mortbay.jetty:jetty-util:jar + org.ow2.asm:asm-analysis:jar + org.ow2.asm:asm-commons:jar + org.ow2.asm:asm-tree:jar + org.reflections:reflections:jar org.scala-lang.modules:scala-parser-combinators_2.11:jar org.scala-lang.modules:scala-xml_2.11:jar org.scala-lang:scala-compiler:jar @@ -146,15 +244,25 @@ org.slf4j:jul-to-slf4j:jar org.slf4j:slf4j-api:jar org.springframework.boot:spring-boot:jar - org.springframework.boot:spring-boot-autoconfigure:jar org.springframework.boot:spring-boot-starter:jar + org.springframework.boot:spring-boot-starter-actuator:jar org.springframework.boot:spring-boot-starter-aop:jar + org.springframework.boot:spring-boot-starter-jetty:jar + org.springframework.boot:spring-boot-starter-json:jar + org.springframework.boot:spring-boot-starter-log4j2:jar + org.springframework.boot:spring-boot-starter-web:jar org.springframework.cloud:spring-cloud-commons:jar + org.springframework.cloud:spring-cloud-config-client:jar org.springframework.cloud:spring-cloud-context:jar org.springframework.cloud:spring-cloud-netflix-archaius:jar + org.springframework.cloud:spring-cloud-netflix-core:jar + org.springframework.cloud:spring-cloud-netflix-eureka-client:jar org.springframework.cloud:spring-cloud-netflix-ribbon:jar org.springframework.cloud:spring-cloud-starter:jar + org.springframework.cloud:spring-cloud-starter-config:jar + org.springframework.cloud:spring-cloud-starter-eureka:jar org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar + org.springframework.cloud:spring-cloud-starter-netflix-eureka-client:jar org.springframework.cloud:spring-cloud-starter-netflix-ribbon:jar org.springframework.security:spring-security-crypto:jar org.springframework.security:spring-security-rsa:jar @@ -163,16 +271,22 @@ org.springframework:spring-context:jar org.springframework:spring-core:jar org.springframework:spring-expression:jar - + org.springframework:spring-jcl:jar org.springframework:spring-web:jar + org.springframework:spring-webmvc:jar org.tukaani:xz:jar org.yaml:snakeyaml:jar + software.amazon.ion:ion-java:jar xerces:xercesImpl:jar - xml-apis:xml-apis:jar xmlenc:xmlenc:jar + xmlpull:xmlpull:jar + xpp3:xpp3_min:jar + + + diff --git a/linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/FsRestfulApi.java b/linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/FsRestfulApi.java index 0cf8d8334f0..60143cafafe 100644 --- a/linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/FsRestfulApi.java +++ b/linkis-public-enhancements/linkis-pes-publicservice/src/main/java/org/apache/linkis/filesystem/restful/api/FsRestfulApi.java @@ -40,12 +40,14 @@ import org.apache.linkis.storage.conf.LinkisStorageConf; import org.apache.linkis.storage.csv.CSVFsWriter; import org.apache.linkis.storage.domain.FsPathListWithError; +import org.apache.linkis.storage.entity.FieldTruncationResult; import org.apache.linkis.storage.excel.*; import org.apache.linkis.storage.exception.ColLengthExceedException; import org.apache.linkis.storage.fs.FileSystem; import org.apache.linkis.storage.script.*; import org.apache.linkis.storage.source.FileSource; import org.apache.linkis.storage.source.FileSource$; +import org.apache.linkis.storage.utils.ResultUtils; import org.apache.linkis.storage.utils.StorageUtils; import org.apache.commons.collections.CollectionUtils; @@ -71,9 +73,7 @@ import java.text.MessageFormat; import java.util.*; import java.util.concurrent.CompletableFuture; -import java.util.function.Predicate; import java.util.stream.Collectors; -import java.util.stream.IntStream; import com.fasterxml.jackson.databind.JsonNode; import com.github.xiaoymin.knife4j.annotations.ApiOperationSupport; @@ -635,7 +635,8 @@ public Message openFile( @RequestParam(value = "columnPage", required = false, defaultValue = "1") Integer columnPage, @RequestParam(value = "columnPageSize", required = false, defaultValue = "500") Integer columnPageSize, - @RequestParam(value = "maskedFieldNames", required = false) String maskedFieldNames) + @RequestParam(value = "maskedFieldNames", required = false) String maskedFieldNames, + @RequestParam(value = "truncateColumn", required = false) String truncateColumn) throws IOException, WorkSpaceException { Message message = Message.ok(); @@ -647,6 +648,9 @@ public Message openFile( throw WorkspaceExceptionManager.createException(80036, path); } + // 检查是否为管理台请求（enableLimit=true） + boolean enableLimitResult = Boolean.parseBoolean(enableLimit); + String userName = ModuleUserUtils.getOperationUser(req, "openFile " + path); LoggerUtils.setJobIdMDC("openFileThread_" + userName); LOGGER.info("userName {} start to open File {}", userName, path); @@ -663,6 +667,23 @@ public Message openFile( int[] columnIndices = null; FileSource fileSource = null; + String zh_msg = + MessageFormat.format( + "结果集存在字段值字符数超过{0}，如需查看全部数据请导出文件或使用字符串截取函数（substring、substr）截取相关字符即可前端展示数据内容", + LinkisStorageConf.LINKIS_RESULT_COL_LENGTH()); + String en_msg = + MessageFormat.format( + "There is a field value exceed {0} characters or col size exceed {1} in the result set. If you want to view it, please use the result set export function.", + LinkisStorageConf.LINKIS_RESULT_COL_LENGTH(), + LinkisStorageConf.LINKIS_RESULT_COLUMN_SIZE()); + String truncateColumn_msg = + MessageFormat.format( + "结果集存在字段值字符数超过{0}，如需查看全部数据请导出文件或确认截取展示数据内容", + LinkisStorageConf.LINKIS_RESULT_COL_LENGTH()); + String truncateColumn_en_msg = + MessageFormat.format( + "The result set contains field values exceeding {0} characters. To view the full data, please export the file or confirm the displayed content is truncated", + LinkisStorageConf.LINKIS_RESULT_COL_LENGTH()); try { fileSource = FileSource$.MODULE$.create(fsPath, fileSystem); if (nullValue != null && BLANK.equalsIgnoreCase(nullValue)) { @@ -677,7 +698,7 @@ public Message openFile( 80034, FILESYSTEM_RESULTSET_ROW_LIMIT.getValue()); } - if (StringUtils.isNotBlank(enableLimit) && "true".equals(enableLimit)) { + if (enableLimitResult) { LOGGER.info("set enable limit for thread: {}", Thread.currentThread().getName()); LinkisStorageConf.enableLimitThreadLocal().set(enableLimit); // 组装列索引 @@ -737,16 +758,68 @@ public Message openFile( } // 增加字段屏蔽 Object resultmap = newMap == null ? metaMap : newMap; - if (FileSource$.MODULE$.isResultSet(fsPath.getPath()) - && StringUtils.isNotBlank(maskedFieldNames)) { - // 如果结果集并且屏蔽字段不为空，则执行屏蔽逻辑，反之则保持原逻辑 - Set maskedFields = - new HashSet<>(Arrays.asList(maskedFieldNames.toLowerCase().split(","))); - Map[] metadata = filterMaskedFieldsFromMetadata(resultmap, maskedFields); - List fileContent = - removeFieldsFromContent(resultmap, result.getSecond(), maskedFields); - message.data("metadata", metadata).data("fileContent", fileContent); + if (FileSource$.MODULE$.isResultSet(fsPath.getPath()) && (resultmap instanceof Map[])) { + // 2. 类型转换（已通过校验，可安全强转） + Map[] filteredMetadata = (Map[]) resultmap; + List filteredContent = result.getSecond(); + // 优先过滤屏蔽字段 + if (LinkisStorageConf.FIELD_MASKED_ENABLED() + && StringUtils.isNotBlank(maskedFieldNames)) { + Set maskedFields = + new HashSet<>(Arrays.asList(maskedFieldNames.toLowerCase().split(","))); + filteredMetadata = ResultUtils.filterMaskedFieldsFromMetadata(resultmap, maskedFields); + filteredContent = + ResultUtils.removeFieldsFromContent(resultmap, filteredContent, maskedFields); + } + // 优先截取大字段 + if (LinkisStorageConf.FIELD_TRUNCATION_ENABLED() && enableLimitResult) { + // 管理台请求(enableLimit=true)不进行字段长度拦截，兼容旧逻辑 + FieldTruncationResult fieldTruncationResult = + ResultUtils.detectAndHandle( + filteredMetadata, + filteredContent, + LinkisStorageConf.FIELD_VIEW_MAX_LENGTH(), + false); + if (fieldTruncationResult.isHasOversizedFields()) { + // 检测到超长字段 + if (null == truncateColumn) { + message.data("type", fileSource.getFileSplits()[0].type()); + message.data("oversizedFields", fieldTruncationResult.getOversizedFields()); + message.data("display_prohibited", true); + message.data("zh_msg", truncateColumn_msg); + message.data("en_msg", truncateColumn_en_msg); + return message; + } + boolean truncateColumnSwitch = Boolean.parseBoolean(truncateColumn); + if (truncateColumnSwitch) { + // 用户选择截取 + FieldTruncationResult truncationResult = + ResultUtils.detectAndHandle( + filteredMetadata, + filteredContent, + LinkisStorageConf.FIELD_VIEW_MAX_LENGTH(), + truncateColumnSwitch); + filteredContent = truncationResult.getData(); + + } else { + // 用户未选择截取，提示用户 + message.data("type", fileSource.getFileSplits()[0].type()); + message.data("display_prohibited", true); + message.data("zh_msg", zh_msg); + message.data("en_msg", en_msg); + return message; + } + } + } + if (StringUtils.isNotBlank(maskedFieldNames) + || (LinkisStorageConf.FIELD_TRUNCATION_ENABLED() && enableLimitResult)) { + message.data("metadata", filteredMetadata).data("fileContent", filteredContent); + } else { + // 不执行字段屏蔽也不执行字段截取 + message.data("metadata", resultmap).data("fileContent", result.getSecond()); + } } else { + // 不执行字段屏蔽也不执行字段截取 message.data("metadata", resultmap).data("fileContent", result.getSecond()); } message.data("type", fileSource.getFileSplits()[0].type()); @@ -756,17 +829,8 @@ public Message openFile( LOGGER.info("Failed to open file {}", path, e); message.data("type", fileSource.getFileSplits()[0].type()); message.data("display_prohibited", true); - message.data( - "zh_msg", - MessageFormat.format( - "结果集存在字段值字符数超过{0}，如需查看全部数据请导出文件或使用字符串截取函数（substring、substr）截取相关字符即可前端展示数据内容", - LinkisStorageConf.LINKIS_RESULT_COL_LENGTH())); - message.data( - "en_msg", - MessageFormat.format( - "There is a field value exceed {0} characters or col size exceed {1} in the result set. If you want to view it, please use the result set export function.", - LinkisStorageConf.LINKIS_RESULT_COL_LENGTH(), - LinkisStorageConf.LINKIS_RESULT_COLUMN_SIZE())); + message.data("zh_msg", zh_msg); + message.data("en_msg", en_msg); return message; } } finally { @@ -778,87 +842,6 @@ public Message openFile( IOUtils.closeQuietly(fileSource); } } - /** - * 删除指定字段的内容 - * - * @param metadata 元数据数组，包含字段信息 - * @param contentList 需要处理的二维字符串数组 - * @param fieldsToRemove 需要删除的字段集合 - * @return 处理后的字符串数组，若输入无效返回空集合而非null - */ - @SuppressWarnings("unchecked") - private List removeFieldsFromContent( - Object metadata, List contentList, Set fieldsToRemove) { - // 1. 参数校验 - if (metadata == null - || fieldsToRemove == null - || fieldsToRemove.isEmpty() - || contentList == null - || !(metadata instanceof Map[])) { - return contentList; - } - - // 2. 安全类型转换 - Map[] fieldMetadata = (Map[]) metadata; - - // 3. 收集需要删除的列索引（去重并排序） - List columnsToRemove = - IntStream.range(0, fieldMetadata.length) - .filter( - i -> { - Map meta = fieldMetadata[i]; - Object columnName = meta.get("columnName"); - return columnName != null - && fieldsToRemove.contains(columnName.toString().toLowerCase()); - }) - .distinct() - .boxed() - .sorted((a, b) -> Integer.compare(b, a)) - .collect(Collectors.toList()); - - // 如果没有需要删除的列，直接返回副本 - if (columnsToRemove.isEmpty()) { - return new ArrayList<>(contentList); - } - // 4. 对每行数据进行处理（删除指定列） - return contentList.stream() - .map( - row -> { - if (row == null || row.length == 0) { - return row; - } - // 创建可变列表以便删除元素 - List rowList = new ArrayList<>(Arrays.asList(row)); - // 从后向前删除列，避免索引变化问题 - for (int columnIndex : columnsToRemove) { - if (columnIndex < rowList.size()) { - rowList.remove(columnIndex); - } - } - return rowList.toArray(new String[0]); - }) - .collect(Collectors.toList()); - } - - @SuppressWarnings("unchecked") - private Map[] filterMaskedFieldsFromMetadata(Object metadata, Set maskedFields) { - // 1. 参数校验 - if (metadata == null || maskedFields == null || !(metadata instanceof Map[])) { - return new Map[0]; - } - - // 2. 类型转换（已通过校验，可安全强转） - Map[] originalMaps = (Map[]) metadata; - - // 3. 过滤逻辑（提取谓词增强可读性） - Predicate> isNotMaskedField = - map -> !maskedFields.contains(map.get("columnName").toString().toLowerCase()); - - // 4. 流处理 + 结果转换 - return Arrays.stream(originalMaps) - .filter(isNotMaskedField) - .toArray(Map[]::new); // 等价于 toArray(new Map[0]) - } /** * 组装获取列索引 @@ -970,7 +953,12 @@ public Message saveScript(HttpServletRequest req, @RequestBody Map ByteTimeUtils.byteStringAsBytes(FILESYSTEM_FILE_CHECK_SIZE.getValue())) { throw WorkspaceExceptionManager.createException(80033, path); } + // 解析日志级别，支持多级别组合 + Set targetLevels = parseLogLevels(logLevel); try (FileSource fileSource = FileSource$.MODULE$.create(fsPath, fileSystem).addParams("ifMerge", "false")) { Pair> collect = fileSource.collect()[0]; @@ -1431,7 +1482,21 @@ public Message openLog( snd.stream() .map(f -> f[0]) .forEach( - s -> WorkspaceUtil.logMatch(s, start).forEach(i -> log[i].append(s).append("\n"))); + s -> { + List matchedIndices = WorkspaceUtil.logMatch(s, start); + if (targetLevels.contains(LogLevel.Type.ALL)) { + // 返回所有日志 + matchedIndices.forEach(i -> log[i].append(s).append("\n")); + } else { + // 多级别组合：只返回目标级别的日志 + for (LogLevel.Type level : targetLevels) { + int targetIndex = level.ordinal(); + if (matchedIndices.contains(targetIndex)) { + log[targetIndex].append(s).append("\n"); + } + } + } + }); LOGGER.info("userName {} Finished to openLog File {}", userName, path); LoggerUtils.removeJobIdMDC(); return Message.ok() @@ -1454,6 +1519,55 @@ private static void deleteAllFiles(FileSystem fileSystem, FsPath fsPath) throws fileSystem.delete(fsPath); } + /** 解析日志级别参数，支持多级别组合 */ + private Set parseLogLevels(String logLevel) { + Set levels = new HashSet<>(); + + if (StringUtils.isEmpty(logLevel)) { + levels.add(LogLevel.Type.ALL); + return levels; + } + + // 去除空格并转为大写 + String cleanedLevel = logLevel.replaceAll("\\s+", "").toUpperCase(); + + // 检查是否为 ALL + if ("ALL".equals(cleanedLevel)) { + levels.add(LogLevel.Type.ALL); + return levels; + } + + // 检查是否包含逗号（多级别组合） + if (cleanedLevel.contains(",")) { + String[] levelArray = cleanedLevel.split(","); + for (String levelStr : levelArray) { + try { + LogLevel.Type level = LogLevel.Type.valueOf(levelStr); + levels.add(level); + } catch (IllegalArgumentException e) { + LOGGER.warn("Invalid log level: {}, skipping", levelStr); + } + } + + // 如果没有有效的级别，使用默认的 ALL + if (levels.isEmpty()) { + LOGGER.warn("No valid log levels found in: {}, using ALL", logLevel); + levels.add(LogLevel.Type.ALL); + } + } else { + // 单级别情况 + try { + LogLevel.Type level = LogLevel.Type.valueOf(cleanedLevel); + levels.add(level); + } catch (IllegalArgumentException e) { + LOGGER.warn("Invalid logLevel: {}, use default level: ALL", logLevel); + levels.add(LogLevel.Type.ALL); + } + } + + return levels; + } + @ApiOperation(value = "chmod", notes = "file permission chmod", response = Message.class) @ApiImplicitParams({ @ApiImplicitParam(name = "filepath", required = true, dataType = "String", value = "filepath"), diff --git a/linkis-public-enhancements/linkis-udf-service/pom.xml b/linkis-public-enhancements/linkis-udf-service/pom.xml index 30a8652394c..4c3a87f7bd9 100644 --- a/linkis-public-enhancements/linkis-udf-service/pom.xml +++ b/linkis-public-enhancements/linkis-udf-service/pom.xml @@ -33,11 +33,6 @@ - - org.apache.linkis - linkis-pes-common - ${project.version} - org.apache.linkis linkis-pes-client diff --git a/linkis-public-enhancements/linkis-udf-service/src/main/java/org/apache/linkis/udf/service/impl/UDFServiceImpl.java b/linkis-public-enhancements/linkis-udf-service/src/main/java/org/apache/linkis/udf/service/impl/UDFServiceImpl.java index e5155c8ddbd..dc7bdfe5afd 100644 --- a/linkis-public-enhancements/linkis-udf-service/src/main/java/org/apache/linkis/udf/service/impl/UDFServiceImpl.java +++ b/linkis-public-enhancements/linkis-udf-service/src/main/java/org/apache/linkis/udf/service/impl/UDFServiceImpl.java @@ -407,28 +407,20 @@ public UDFInfo createSharedUdfInfo(UDFInfo udfInfo, Long shareParentId, FsPath s private UDFTree getOrCreateTree(String userName, String category, String treeName) throws UDFException { + Map selfTreemap = new HashMap<>(); + selfTreemap.put("parent", -1); + selfTreemap.put("userName", userName); + selfTreemap.put("category", category); // 个人函数目录 - List selfTree = - udfTreeDao.getTreesByParentId( - new HashMap() { - { - put("parent", -1); - put("userName", userName); - put("category", category); - } - }); + List selfTree = udfTreeDao.getTreesByParentId(selfTreemap); if (selfTree == null || selfTree.size() == 0) { throw new UDFException("该用户没有个人函数目录!"); } - List selfTreeChildren = - udfTreeDao.getTreesByParentId( - new HashMap() { - { - put("parent", selfTree.get(0).getId()); - put("userName", userName); - put("category", category); - } - }); + Map selfTreeChildrenMap = new HashMap<>(); + selfTreeChildrenMap.put("parent", selfTree.get(0).getId()); + selfTreeChildrenMap.put("userName", userName); + selfTreeChildrenMap.put("category", category); + List selfTreeChildren = udfTreeDao.getTreesByParentId(selfTreeChildrenMap); for (UDFTree tree : selfTreeChildren) { if (tree.getName().equals(treeName)) { return tree; @@ -612,17 +604,15 @@ public PageInfo getManagerPages( } } boolean finalCanExpire = canExpire; - l.setOperationStatus( - new HashMap() { - { - put("canUpdate", true); - put("canDelete", !finalCanExpire); - put("canExpire", finalCanExpire); - put("canShare", ismanager); - put("canPublish", ismanager && Boolean.TRUE.equals(l.getShared())); - put("canHandover", true); - } - }); + Map operationStatusMap = new HashMap<>(); + operationStatusMap.put("canUpdate", true); + operationStatusMap.put("canDelete", !finalCanExpire); + operationStatusMap.put("canExpire", finalCanExpire); + operationStatusMap.put("canShare", ismanager); + operationStatusMap.put( + "canPublish", ismanager && Boolean.TRUE.equals(l.getShared())); + operationStatusMap.put("canHandover", true); + l.setOperationStatus(operationStatusMap); }); } logger.info("end to get managerPages."); @@ -768,8 +758,7 @@ public Boolean addLoadInfo(Long id, String userName) throws UDFException { long loadCount = udfDao.getSameLoadCount(userName, udfInfo.getUdfName()); if (loadCount > 0) { throw new UDFException( - "There is a Jar package function with the same name(存在同名的Jar包函数)： " - + udfInfo.getUdfName()); + String.format("用户 %s 已加载过 %s 函数或用户操作过于频繁请刷新后重试", userName, udfInfo.getUdfName())); } // 校验jar包名字 // String path = udfInfo.getPath(); @@ -1079,17 +1068,15 @@ public List getUdfByNameList(List udfNameList, String creator) } } boolean finalCanExpire = canExpire; - udfInfo.setOperationStatus( - new HashMap() { - { - put("canUpdate", true); - put("canDelete", !finalCanExpire); - put("canExpire", finalCanExpire); - put("canShare", ismanager); - put("canPublish", ismanager && Boolean.TRUE.equals(udfInfo.getShared())); - put("canHandover", true); - } - }); + Map operationStatusMap = new HashMap<>(); + operationStatusMap.put("canUpdate", true); + operationStatusMap.put("canDelete", !finalCanExpire); + operationStatusMap.put("canExpire", finalCanExpire); + operationStatusMap.put("canShare", ismanager); + operationStatusMap.put( + "canPublish", ismanager && Boolean.TRUE.equals(udfInfo.getShared())); + operationStatusMap.put("canHandover", true); + udfInfo.setOperationStatus(operationStatusMap); }); return retList; } diff --git a/linkis-public-enhancements/linkis-udf-service/src/main/scala/org/apache/linkis/udf/api/rpc/UdfReceiver.scala b/linkis-public-enhancements/linkis-udf-service/src/main/scala/org/apache/linkis/udf/api/rpc/UdfReceiver.scala index 4527f8d40ad..06815d5cd25 100644 --- a/linkis-public-enhancements/linkis-udf-service/src/main/scala/org/apache/linkis/udf/api/rpc/UdfReceiver.scala +++ b/linkis-public-enhancements/linkis-udf-service/src/main/scala/org/apache/linkis/udf/api/rpc/UdfReceiver.scala @@ -30,7 +30,6 @@ import java.{lang, util} import scala.collection.JavaConverters.asScalaBufferConverter import scala.concurrent.duration.Duration -import scala.tools.nsc.interactive.Logger class UdfReceiver extends Receiver with Logging { diff --git a/linkis-public-enhancements/linkis-udf-service/src/test/java/org/apache/linkis/udf/api/PythonModuleRestfulApiTest.java b/linkis-public-enhancements/linkis-udf-service/src/test/java/org/apache/linkis/udf/api/PythonModuleRestfulApiTest.java new file mode 100644 index 00000000000..c97b0840426 --- /dev/null +++ b/linkis-public-enhancements/linkis-udf-service/src/test/java/org/apache/linkis/udf/api/PythonModuleRestfulApiTest.java @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.linkis.udf.api; + +import org.apache.linkis.udf.entity.PythonModuleInfo; +import org.apache.linkis.udf.service.PythonModuleInfoService; + +import java.util.ArrayList; +import java.util.List; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mock; +import org.mockito.MockitoAnnotations; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.when; + +/** PythonModuleRestfulApiTest 类用于测试 Python 模块服务相关功能。 */ +public class PythonModuleRestfulApiTest { + + @Mock private PythonModuleInfoService pythonModuleInfoService; + + @BeforeEach + public void setUp() { + MockitoAnnotations.openMocks(this); + } + + /** 测试Python模块列表功能 */ + @Test + public void testPythonList() throws Exception { + List mockList = new ArrayList<>(); + PythonModuleInfo info = new PythonModuleInfo(); + info.setId(1L); + info.setName("testModule"); + mockList.add(info); + when(pythonModuleInfoService.getByConditions(any())).thenReturn(mockList); + + List result = pythonModuleInfoService.getByConditions(new PythonModuleInfo()); + assertNotNull(result); + assertEquals(1, result.size()); + } + + /** 测试删除Python模块功能 */ + @Test + public void testPythonDelete() throws Exception { + when(pythonModuleInfoService.updatePythonModuleInfo(any())).thenReturn(1); + + int result = pythonModuleInfoService.updatePythonModuleInfo(new PythonModuleInfo()); + assertEquals(1, result); + } + + /** 测试保存Python模块功能 */ + @Test + public void testPythonSave() throws Exception { + when(pythonModuleInfoService.insertPythonModuleInfo(any())).thenReturn(1L); + when(pythonModuleInfoService.getByUserAndNameAndId(any())).thenReturn(null); + + Long result = pythonModuleInfoService.insertPythonModuleInfo(new PythonModuleInfo()); + assertEquals(1L, result.longValue()); + } + + /** 测试检查Python模块文件是否存在功能 */ + @Test + public void testPythonFileExist() throws Exception { + PythonModuleInfo mockInfo = new PythonModuleInfo(); + mockInfo.setId(1L); + when(pythonModuleInfoService.getByUserAndNameAndId(any())).thenReturn(mockInfo); + + PythonModuleInfo result = pythonModuleInfoService.getByUserAndNameAndId(new PythonModuleInfo()); + assertNotNull(result); + } +} diff --git a/linkis-public-enhancements/pom.xml b/linkis-public-enhancements/pom.xml index 72f8067a3f5..7ced9e1d753 100644 --- a/linkis-public-enhancements/pom.xml +++ b/linkis-public-enhancements/pom.xml @@ -34,6 +34,7 @@ linkis-ps-common-lock linkis-bml-server linkis-cs-server + linkis-datasource/linkis-metadata linkis-datasource linkis-udf-service linkis-jobhistory @@ -50,12 +51,12 @@ org.apache.linkis - linkis-metadata + linkis-datasource-manager-server ${project.version} org.apache.linkis - linkis-datasource-manager-server + linkis-metadata ${project.version} diff --git a/linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka/pom.xml b/linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka/pom.xml index 37805414680..4bfa76e589f 100644 --- a/linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka/pom.xml +++ b/linkis-spring-cloud-services/linkis-service-discovery/linkis-eureka/pom.xml @@ -26,54 +26,68 @@ linkis-eureka jar - - true - true - - - org.apache.linkis - linkis-module - provided + org.springframework.cloud + spring-cloud-starter - com.github.xiaoymin - knife4j-spring-boot-starter + org.springframework.boot + spring-boot-starter + + + + + org.springframework.cloud + spring-cloud-starter-netflix-eureka-server + - io.springfox - springfox-schema + org.springframework.boot + spring-boot-starter-logging - io.springfox - springfox-spring-web + com.fasterxml.jackson.core + jackson-databind - io.springfox - springfox-spring-webflux + javax.servlet + servlet-api - io.springfox - springfox-spring-webmvc + io.github.x-stream + mxparser + + org.springframework.boot + spring-boot-starter-log4j2 + + + com.fasterxml.jackson.core + jackson-databind + - org.springframework.cloud - spring-cloud-starter-netflix-eureka-server + org.springframework.boot + spring-boot-starter-actuator + ${spring.boot.version} - org.springframework.cloud - spring-cloud-starter + org.springframework.boot + spring-boot-starter-logging + + + org.yaml + snakeyaml - - com.sun.jersey.contribs - jersey-apache-client4 + io.micrometer + micrometer-registry-prometheus + compile diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/main/scala/org/apache/linkis/gateway/authentication/service/CachedTokenService.scala b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/main/scala/org/apache/linkis/gateway/authentication/service/CachedTokenService.scala index 4b3fcd85739..150bc9ad90d 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/main/scala/org/apache/linkis/gateway/authentication/service/CachedTokenService.scala +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/main/scala/org/apache/linkis/gateway/authentication/service/CachedTokenService.scala @@ -52,7 +52,7 @@ class CachedTokenService extends TokenService { private val tokenCache: LoadingCache[String, Token] = CacheBuilder.newBuilder .maximumSize(TokenConfiguration.TOKEN_CACHE_MAX_SIZE) - .refreshAfterWrite(TokenConfiguration.TOKEN_CACHE_EXPIRE_MINUTES, TimeUnit.MINUTES) + .expireAfterWrite(TokenConfiguration.TOKEN_CACHE_EXPIRE_MINUTES, TimeUnit.MINUTES) .build(new CacheLoader[String, Token]() { override def load(tokenName: String): Token = { diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/java/org/apache/linkis/gateway/authentication/service/CachedTokenServiceTest.java b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/java/org/apache/linkis/gateway/authentication/service/CachedTokenServiceTest.java index 05286487ed8..312ae7ef019 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/java/org/apache/linkis/gateway/authentication/service/CachedTokenServiceTest.java +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/java/org/apache/linkis/gateway/authentication/service/CachedTokenServiceTest.java @@ -91,5 +91,16 @@ void testDoAuth() { assertThrows( TokenAuthException.class, () -> tokenService.doAuth(TokenName, "test", "10.10.10.10")); logger.info("assertThrows：{}", exception.getMessage()); + + exception = + assertThrows( + TokenAuthException.class, () -> tokenService.doAuth("NOT-EXIST", "test", "127.0.0.1")); + logger.info("assertThrows：{}", exception.getMessage()); + + exception = + assertThrows( + TokenAuthException.class, + () -> tokenService.doAuth("LINKISCLI-AUTH", "test", "127.0.0.1")); + logger.info("assertThrows：{}", exception.getMessage()); } } diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create.sql b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create.sql index 8fc39b1e566..67e389f8062 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create.sql +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create.sql @@ -37,4 +37,4 @@ DELETE FROM linkis_mg_gateway_auth_token; -- ---------------------------- -- Default Tokens -- ---------------------------- -INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('LINKIS-UNAVAILABLE-TOKEN','test','127.0.0.1','BDP',curdate(),curdate(),-1,'LINKIS'); +INSERT INTO `linkis_mg_gateway_auth_token`(`token_name`,`legal_users`,`legal_hosts`,`business_owner`,`create_time`,`update_time`,`elapse_day`,`update_by`) VALUES ('LINKIS-UNAVAILABLE-TOKEN','test','127.0.0.1','BDP',curdate(),curdate(),-1,'LINKIS'); \ No newline at end of file diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create_pg.sql b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create_pg.sql index 725ed37bf14..85eea687da5 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create_pg.sql +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/create_pg.sql @@ -1,37 +1,37 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. -*/ - -DROP TABLE IF EXISTS "linkis_mg_gateway_auth_token"; -CREATE TABLE linkis_mg_gateway_auth_token ( - id serial NOT NULL, - "token_name" varchar(128) NOT NULL, - legal_users text NULL, - legal_hosts text NULL, - "business_owner" varchar(32) NULL, - create_time timestamp(6) NULL, - update_time timestamp(6) NULL, - elapse_day int8 NULL, - update_by varchar(32) NULL, - CONSTRAINT linkis_mg_gateway_auth_token_pkey PRIMARY KEY (id) -); -CREATE UNIQUE INDEX uniq_token_name ON linkis_mg_gateway_auth_token (token_name); - -delete from linkis_mg_gateway_auth_token; --- ---------------------------- --- Default Tokens --- ---------------------------- -INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('LINKIS-UNAVAILABLE-TOKEN','test','127.0.0.1','BDP',now(),now(),-1,'LINKIS'); +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +DROP TABLE IF EXISTS "linkis_mg_gateway_auth_token"; +CREATE TABLE linkis_mg_gateway_auth_token ( + id serial NOT NULL, + "token_name" varchar(128) NOT NULL, + legal_users text NULL, + legal_hosts text NULL, + "business_owner" varchar(32) NULL, + create_time timestamp(6) NULL, + update_time timestamp(6) NULL, + elapse_day int8 NULL, + update_by varchar(32) NULL, + CONSTRAINT linkis_mg_gateway_auth_token_pkey PRIMARY KEY (id) +); +CREATE UNIQUE INDEX uniq_token_name ON linkis_mg_gateway_auth_token (token_name); + +delete from linkis_mg_gateway_auth_token; +-- ---------------------------- +-- Default Tokens +-- ---------------------------- +INSERT INTO "linkis_mg_gateway_auth_token"("token_name","legal_users","legal_hosts","business_owner","create_time","update_time","elapse_day","update_by") VALUES ('LINKIS-UNAVAILABLE-TOKEN','test','127.0.0.1','BDP',now(),now(),-1,'LINKIS'); \ No newline at end of file diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/data.sql b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/data.sql new file mode 100644 index 00000000000..201701a10f0 --- /dev/null +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-authentication/src/test/resources/data.sql @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +-- This file is intentionally left with no data operations. +-- Initial data is loaded via create.sql +SELECT 1; diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/pom.xml b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/pom.xml index 322c2c61b7e..e7444893388 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/pom.xml +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/pom.xml @@ -74,6 +74,17 @@ io.springfox springfox-spring-web ${springfox.version} + + + io.github.classgraph + classgraph + + + + + io.github.classgraph + classgraph + ${classgraph.version} io.springfox diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/config/GatewayConfiguration.scala b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/config/GatewayConfiguration.scala index b9ef27105af..52103046188 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/config/GatewayConfiguration.scala +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/config/GatewayConfiguration.scala @@ -138,6 +138,7 @@ object GatewayConfiguration { val PROHIBIT_LOGIN_SWITCH = CommonVars("linkis.system.user.prohibit.login.switch", false) val PROHIBIT_LOGIN_PREFIX = - CommonVars("linkis.system.user.prohibit.login.prefix", "hduser,shduser").getValue.toLowerCase() + CommonVars("linkis.system.user.prohibit.login.prefix", "hduser,shduser,hadoop").getValue + .toLowerCase() } diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/SecurityFilter.scala b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/SecurityFilter.scala index 9f170e9dd2f..5881b6001ea 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/SecurityFilter.scala +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/SecurityFilter.scala @@ -134,7 +134,9 @@ object SecurityFilter extends Logging { val userName = Utils.tryCatch(GatewaySSOUtils.getLoginUser(gatewayContext)) { case n @ (_: NonLoginException | _: LoginExpireException) => if (Configuration.IS_TEST_MODE.getValue) None - else { + else if (GatewayConfiguration.ENABLE_SSO_LOGIN.getValue) { + None + } else { filterResponse( gatewayContext, Message.noLogin(n.getMessage) << gatewayContext.getRequest.getRequestURI diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/UserRestful.scala b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/UserRestful.scala index 7f8e1ffefa0..be9fbe3a07a 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/UserRestful.scala +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-core/src/main/scala/org/apache/linkis/gateway/security/UserRestful.scala @@ -307,10 +307,13 @@ abstract class UserPwdAbstractUserRestful extends AbstractUserRestful with Loggi return Message.error("Password can not be blank(密码不能为空)！") } - if ( - GatewayConfiguration.PROHIBIT_LOGIN_SWITCH.getValue && (!getRequestSource(gatewayContext) - .equals("client")) - ) { + // 从header获取webLogin标识，默认为false + val headers = gatewayContext.getRequest.getHeaders + val webLoginHeaders = headers.getOrDefault("webLogin", Array("false")) + val webLogin = java.lang.Boolean.parseBoolean(webLoginHeaders.head) + // 如果是web登录，检查是否为系统用户（包括hadoop用户） + if (GatewayConfiguration.PROHIBIT_LOGIN_SWITCH.getValue && webLogin) { + // 检查是否为系统用户（包括hadoop用户） PROHIBIT_LOGIN_PREFIX.split(",").foreach { prefix => if (userName.toLowerCase().startsWith(prefix)) { return Message.error("System users are prohibited from logging in（系统用户禁止登录）！") diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-httpclient-support/src/main/scala/org/apache/linkis/httpclient/dws/authentication/StaticAuthenticationStrategy.scala b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-httpclient-support/src/main/scala/org/apache/linkis/httpclient/dws/authentication/StaticAuthenticationStrategy.scala index c002e8bc306..a684152ad9d 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-httpclient-support/src/main/scala/org/apache/linkis/httpclient/dws/authentication/StaticAuthenticationStrategy.scala +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-httpclient-support/src/main/scala/org/apache/linkis/httpclient/dws/authentication/StaticAuthenticationStrategy.scala @@ -72,18 +72,15 @@ class StaticAuthenticationStrategy(override protected val sessionMaxAliveTime: L case userPwd: UserPwdAction => action.addRequestPayload("userName", userPwd.getUser) action.addRequestPayload("password", userPwd.getPassword.getOrElse(pwd)) - action.addRequestPayload("source", "client") case userAction: UserAction => action.addRequestPayload("userName", userAction.getUser) action.addRequestPayload("password", pwd) - action.addRequestPayload("source", "client") case _ => if (StringUtils.isBlank(getClientConfig.getAuthTokenKey)) { throw new AuthenticationFailedException(AUTHTOKENVALUE_BE_EXISTS.getErrorDesc) } action.addRequestPayload("userName", getClientConfig.getAuthTokenKey) action.addRequestPayload("password", pwd) - action.addRequestPayload("source", "client") } action } diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-server-support/pom.xml b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-server-support/pom.xml index 344fce259c3..2fd40c4c965 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-server-support/pom.xml +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-gateway-server-support/pom.xml @@ -81,17 +81,6 @@ ${project.version} - - - org.apache.linkis - linkis-jobhistory - ${project.version} - - - - com.fasterxml.jackson.core - jackson-databind - diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/pom.xml b/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/pom.xml index eead24e624f..e1a025a11da 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/pom.xml +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/pom.xml @@ -48,6 +48,10 @@ org.springframework.cloud spring-cloud-starter-gateway + + org.codehaus.jackson + jackson-core-asl + @@ -97,8 +101,17 @@ io.projectreactor.netty reactor-netty + + io.projectreactor.netty + reactor-netty-http + + + io.projectreactor.netty + reactor-netty-http + ${reactor-netty-http.version} + org.springframework.boot spring-boot-starter diff --git a/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/src/main/java/org/apache/linkis/gateway/springcloud/http/GatewayAuthorizationFilter.java b/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/src/main/java/org/apache/linkis/gateway/springcloud/http/GatewayAuthorizationFilter.java index edeee8f170b..cef85b843e9 100644 --- a/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/src/main/java/org/apache/linkis/gateway/springcloud/http/GatewayAuthorizationFilter.java +++ b/linkis-spring-cloud-services/linkis-service-gateway/linkis-spring-cloud-gateway/src/main/java/org/apache/linkis/gateway/springcloud/http/GatewayAuthorizationFilter.java @@ -210,7 +210,7 @@ private Mono gatewayDeal( realRd = rd; } } - String uri = realRd.getUri().toString(); + String uri = realRd.getUri().toString(); // NOSONAR if (uri != null) { uri = uri diff --git a/linkis-web-next/.husky/commit-msg b/linkis-web-next/.husky/commit-msg index 59600a66d34..7ef87280c71 100644 --- a/linkis-web-next/.husky/commit-msg +++ b/linkis-web-next/.husky/commit-msg @@ -16,4 +16,7 @@ . "$(dirname "$0")/_/husky.sh" -npx --no-install commitlint --edit $1 \ No newline at end of file +npx --no-install commitlint --edit $1 + +wds.linkis.entrance.log.push.interval.time=3000 + diff --git a/linkis-web-next/src/config/db.ts b/linkis-web-next/src/config/db.ts index 4e858e2961c..0fadff4b53b 100644 --- a/linkis-web-next/src/config/db.ts +++ b/linkis-web-next/src/config/db.ts @@ -29,4 +29,4 @@ export const config = { node: '&nodeId', tree: '&treeId', }, -}; +}; \ No newline at end of file diff --git a/linkis-web-next/src/dss/view/500.vue b/linkis-web-next/src/dss/view/500.vue index 5396a706191..a5421cc2c9e 100644 --- a/linkis-web-next/src/dss/view/500.vue +++ b/linkis-web-next/src/dss/view/500.vue @@ -5,9 +5,9 @@ ~ The ASF licenses this file to You under the Apache License, Version 2.0 ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at - ~ + ~ ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ + ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. diff --git a/linkis-web/.env b/linkis-web/.env index 5f7be0422c5..a778ea91fa3 100644 --- a/linkis-web/.env +++ b/linkis-web/.env @@ -1,7 +1,9 @@ VUE_APP_HOST= +BACKEND_URL=http://127.0.0.1:9001 VUE_APP_MN_CONFIG_PREFIX= VUE_APP_MN_CONFIG_SOCKET=/ws/api/entrance/connect VUE_APP_VERSION=1.8.0 # Enable or disable Python Module feature (true/false) -# Set to false if you cannot access @webank internal npm packages VUE_APP_ENABLE_PYTHON_MODULE=false +VUE_APP_DISALLOW_LOGIN=true +VUE_APP_DISALLOW_LOGIN_PREFIX=hduser,shduser,hadoop diff --git a/linkis-web/src/apps/PythonModule/.env b/linkis-web/src/apps/PythonModule/.env index 1b5d140b8e6..ff9b59ca8a8 100644 --- a/linkis-web/src/apps/PythonModule/.env +++ b/linkis-web/src/apps/PythonModule/.env @@ -5,12 +5,14 @@ * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at - * + * * http://www.apache.org/licenses/LICENSE-2.0 - * + * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ + +/// diff --git a/linkis-web/src/apps/PythonModule/src/letgo/global.css b/linkis-web/src/apps/PythonModule/src/letgo/global.css index d2232996530..6620c456fc7 100644 --- a/linkis-web/src/apps/PythonModule/src/letgo/global.css +++ b/linkis-web/src/apps/PythonModule/src/letgo/global.css @@ -5,15 +5,16 @@ * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at - * + * * http://www.apache.org/licenses/LICENSE-2.0 - * + * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ + .buttons .fes-form-item-content { display: flex; justify-content: space-between; diff --git a/linkis-web/src/apps/linkis/module/eurekaService/index.js b/linkis-web/src/apps/linkis/module/eurekaService/index.js index 889a16be0ca..c4128f9c0e5 100644 --- a/linkis-web/src/apps/linkis/module/eurekaService/index.js +++ b/linkis-web/src/apps/linkis/module/eurekaService/index.js @@ -5,17 +5,18 @@ * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at - * + * * http://www.apache.org/licenses/LICENSE-2.0 - * + * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ - + export default { - name: 'eurekaService', + name: 'pythonModule', + events: [], component: () => import('./index.vue'), -} \ No newline at end of file +}; diff --git a/linkis-web/src/apps/linkis/router.js b/linkis-web/src/apps/linkis/router.js index 4db08510fe0..0f63d27828a 100644 --- a/linkis-web/src/apps/linkis/router.js +++ b/linkis-web/src/apps/linkis/router.js @@ -281,6 +281,16 @@ export default [ publicPage: true, }, }] : []), + { + name: 'pythonModule', + path: 'pythonModule', + component: () => + import('./module/pythonModule/index.vue'), + meta: { + title: 'pythonModule', + publicPage: true, + }, + }, { name: 'datasourceAccess', path: 'datasourceAccess', diff --git a/linkis-web/src/apps/linkis/view/linkis/index.vue b/linkis-web/src/apps/linkis/view/linkis/index.vue index 816fa7eda90..e96bf1d0f86 100644 --- a/linkis-web/src/apps/linkis/view/linkis/index.vue +++ b/linkis-web/src/apps/linkis/view/linkis/index.vue @@ -30,7 +30,7 @@
@@ -116,7 +116,6 @@ diff --git a/pom.xml b/pom.xml index 61d506d1c28..ca06cf588f8 100644 --- a/pom.xml +++ b/pom.xml @@ -103,116 +103,128 @@ 1.8.0 - - 3.9.4 - 4.2.0 - 3.3.4 - hadoop-hdfs-client - 2.5.3 + 2.9.2 + 2.4.3 + 2.7.2 + 1.2.1 9.3.4.0 1.0.56 2.1.42 - 3.1.3 - 3.2.1 - 2.0.0 - 3.4.0 + hadoop-hdfs 2.7.2 - org.apache.hadoop - hadoop-common - hadoop-hdfs - ${hadoop.version} - provided + 3.8.4 + + 2.7.1 + 33.2.1-jre + 4.2.7.Final + 3.4.0 + 1.1.10.4 + 1.11.4 + 0.27 + 1.0.39 + + 5.4.1 + + 2.8.9 + 2.15.0 + 3.0.0 + 1.19.4 + 2.23.1 + 2.14.6 + 9.4.57.v20241219 + 4.5.13 + ${httpclient.version} + 2.0.9 + 2.10.5 + 1.4.21 + 0.9.12 + 3.5.7 + 8.0.28 + 1.2.4 + 3.27.0-GA + 3.2.2 + 2.6 + 3.12.0 + 1.2 + 1.10.0 + 3.6.1 + 2.11.0 + 1.11.0 + 1.21 + 1.10 + 1.3 + 1.6.0 + 3.25.5 + 3.3.0 + 9.3 + 1.7.30 + 2.17.2 + 5.2.3 + 2.0 + 1.5.4 + 6.4.0 + 4.8.112 + 3.9.1 + + + 2.9.1 + 1.2.30 + + + 1.3.5 + + + 1.10.0 + 1.5.8 provided provided + + 2.3.3 1.16.2 + 371 + 3.4.0.7.2.15.0-147 + 0.234 + 1.4.6 + 7.6.2 + 1.2.6 + 1.5.0 + 2.1.2 + + 0.9.3 1.3.0 1.0 4 - 371 - 3.4.0.7.2.15.0-147 - 1.5.0 1 - 0.234 - 3.0.0 - 1.2.6 1 python2 - 2.1.2 1 - 1.4.6 - 7.6.2 - - - 5.4.1 - - 2.8.9 - 2.13.4.20221013 + 2.9.0 1.5.4 - 1.4.21 6.4.0 - 2.0 - - 3.25.5 - - 1.10.0 - 1.5.8 - - 1.19.4 - 2.23.1 - 4.1.86.Final - 9.4.56.v20240826 - 4.5.13 - ${httpclient.version} - - 3.0.3 - 3.0.0 - 5.3.0 - 0.9.12 + 0.17 - - 3.9.0 + 3.4.3.4 - 8.0.28 test 42.3.9 - 1.2.23 3.12.8 - - 32.0.0-jre - 3.2.2 - 3.18.0 - 1.2 - 1.10.0 - 3.6.1 - 2.17.0 - 1.11.0 - 1.26.1 - 1.10 - 1.3 - 1.6.0 + 3.9.0 1.3.1 - 3.9.0 - 1.79 - - - 3.27.0-GA - 9.3 - 3.3.0 - - - 1.7.30 - 2.17.2 - - - 1.3.5 + 4.3.1 + provided + hadoop-common + hadoop-hdfs + org.apache.hadoop + ${hadoop.version} + 2.0.0 5.7.2 @@ -221,14 +233,14 @@ 2.2.220 - 5.3.27 - 5.7.8 - 2.7.11 + 5.3.34 + 5.7.11 + 2.7.12 3.1.7 2021.0.8 2021.0.6.0 3.1.7 - 1.2.30 + 2.7.16 UTF-8 @@ -237,8 +249,9 @@ 1.8 3.5.0 - 2.12.17 - 2.12 + 3.7.0-M11 + 2.11.12 + 2.11 1.10.12 @@ -312,6 +325,11 @@ linkis-mybatis ${project.version} + + com.thoughtworks.xstream + xstream + ${xstream.core.version} + @@ -334,7 +352,16 @@ scalap ${scala.version} - + + com.google.guava + guava + ${guava.version} + + + xml-apis + xml-apis + 1.4.01 + org.apache.logging.log4j log4j-bom @@ -347,7 +374,11 @@ slf4j-api ${slf4j.version} - + + org.javassist + javassist + ${javassist.version} + org.apache.ant ant @@ -365,23 +396,6 @@ ${reflections.version} - - redis.clients - jedis - ${jedis.version} - - - mysql - mysql-connector-java - ${mysql.connector.version} - ${mysql.connector.scope} - - - - com.google.guava - guava - ${guava.version} - commons-collections commons-collections @@ -390,7 +404,7 @@ commons-lang commons-lang - 2.6 + ${commons-lang.version} org.apache.commons @@ -432,38 +446,13 @@ commons-exec ${commons-exec.version} - - commons-fileupload - commons-fileupload - ${commons-fileupload.version} - - - commons-net - commons-net - ${commons-net.version} - - - commons-logging - commons-logging - ${commons-logging.version} - - - commons-cli - commons-cli - ${commons-cli.version} - - org.jline - jline - ${jline.version} - - - jline - jline - 2.14.6 + mysql + mysql-connector-java + ${mysql.connector.version} + test - org.junit junit-bom @@ -480,11 +469,6 @@ - - com.jayway.jsonpath - json-path - ${jsonpath.version} - com.google.code.gson gson @@ -497,49 +481,6 @@ pom import - - org.codehaus.jackson - jackson-core-asl - 1.9.13 - - - org.codehaus.jackson - jackson-mapper-asl - 1.9.13 - - - - com.thoughtworks.xstream - xstream - ${xstream.core.version} - - - xml-apis - xml-apis - 1.4.01 - - - com.fasterxml.woodstox - woodstox-core - ${woodstox.version} - - - org.codehaus.jettison - jettison - ${jettison.version} - - - - org.yaml - snakeyaml - ${snakeyaml.version} - - - - com.google.protobuf - protobuf-java - ${protobuf.version} - com.sun.jersey @@ -558,206 +499,62 @@ com.sun.jersey - jersey-core - ${jersey.version} - - - com.sun.jersey - jersey-json - ${jersey.version} - - - com.sun.jersey.contribs - jersey-guice - ${jersey.version} - - - com.sun.jersey.contribs - jersey-apache-client4 - ${jersey.version} - - - org.glassfish.jersey - jersey-bom - ${jersey.servlet.version} - pom - import - - - - org.eclipse.jetty - jetty-bom - ${jetty.version} - pom - import - - - - org.apache.curator - curator-recipes - ${curator.version} - - - org.apache.curator - curator-client - ${curator.version} - - - org.apache.curator - curator-framework - ${curator.version} - - - - org.apache.hbase - hbase-common - ${hbase.version} - - - org.apache.hbase - hbase-hadoop2-compat - ${hbase.version} - - - org.apache.hbase - hbase-hadoop-compat - ${hbase.version} - - - org.apache.hbase - hbase-server - ${hbase.version} - - - org.apache.hadoop - hadoop-hdfs - - - - - org.apache.hbase - hbase-mapreduce - ${hbase.version} - - - org.apache.hadoop - hadoop-hdfs - - - - - org.apache.hbase - hbase-protocol - ${hbase.version} - - - org.apache.hbase - hbase-client - ${hbase.version} - - - - org.apache.hive - hive-common - ${hive.version} - - - jline - jline - - - - - org.apache.hive - hive-service - ${hive.version} - - - org.apache.curator - apache-curator - - - org.apache.hbase - * - - - - - org.apache.hive - hive-jdbc - ${hive.version} - - - org.apache.hive - hive-shims - - - org.apache.hadoop - hadoop-hdfs - - - org.apache.hbase - hbase-client - - - org.eclipse.jetty - * - - - org.eclipse.jetty.aggregate - * - - - org.eclipse.jetty.orbit - * - - - org.mortbay.jetty - * - - - tomcat - * - - - junit - junit - - - log4j - log4j - - - log4j - apache-log4j-extras - - - org.slf4j - slf4j-log4j12 - - - ch.qos.logback - logback-classic - - - javax.servlet - servlet-api - - - io.netty - netty - - - jdk.tools - jdk.tools - - + jersey-core + ${jersey.version} - - org.apache.hadoop - hadoop-distcp - ${hadoop.version} + com.sun.jersey + jersey-json + ${jersey.version} + + + com.sun.jersey.contribs + jersey-guice + ${jersey.version} + + + com.google.protobuf + protobuf-java + ${protobuf.version} + + + com.sun.jersey.contribs + jersey-apache-client4 + ${jersey.version} + + + org.eclipse.jetty + jetty-bom + ${jetty.version} + pom + import + + + org.glassfish.jersey + jersey-bom + ${jersey.servlet.version} + pom + import + + + jline + jline + ${jline.version} + + + org.apache.curator + curator-recipes + ${curator.version} + + + org.apache.curator + curator-client + ${curator.version} + + + org.apache.curator + curator-framework + ${curator.version} org.apache.hadoop @@ -772,15 +569,6 @@ org.slf4j slf4j-log4j12 - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - javax.ws.rs jsr311-api @@ -857,15 +645,6 @@ org.slf4j slf4j-log4j12 - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - org.fusesource.leveldbjni leveldbjni-all @@ -942,7 +721,11 @@ - + + commons-logging + commons-logging + ${commons-logging.version} + org.apache.hadoop ${hadoop-hdfs-client.artifact} @@ -956,15 +739,6 @@ org.slf4j slf4j-log4j12 - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - netty io.netty @@ -1008,15 +782,6 @@ org.slf4j slf4j-log4j12 - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - org.apache.httpcomponents * @@ -1040,18 +805,8 @@ org.slf4j slf4j-log4j12 - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - - org.apache.hadoop hadoop-yarn-api @@ -1095,20 +850,11 @@ - - org.apache.hadoop - hadoop-yarn-registry - ${hadoop.version} - org.apache.hadoop hadoop-yarn-common ${hadoop.version} - - ch.qos.reload4j - reload4j - asm asm @@ -1156,15 +902,6 @@ log4j log4j - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - javax.servlet servlet-api @@ -1184,6 +921,13 @@ + + + org.apache.hadoop + hadoop-aliyun + ${hadoop-aliyun.version} + + org.apache.zookeeper zookeeper @@ -1193,27 +937,10 @@ log4j log4j - - ch.qos.logback - logback-core - - - ch.qos.logback - logback-classic - org.slf4j slf4j-log4j12 - - - ch.qos.reload4j - reload4j - - - org.slf4j - slf4j-reload4j - jline jline @@ -1233,23 +960,54 @@ - - org.bouncycastle - bcprov-jdk18on - ${bouncycastle.version} + org.apache.hbase + hbase-common + ${hbase.version} - org.bouncycastle - bcpkix-jdk18on - ${bouncycastle.version} + org.apache.hbase + hbase-hadoop2-compat + ${hbase.version} - - org.javassist - javassist - ${javassist.version} + org.apache.hbase + hbase-hadoop-compat + ${hbase.version} + + + org.apache.hbase + hbase-server + ${hbase.version} + + + org.apache.hadoop + hadoop-hdfs + + + + + org.apache.hbase + hbase-mapreduce + ${hbase.version} + + + org.apache.hadoop + hadoop-hdfs + + + + + org.apache.hbase + hbase-protocol + ${hbase.version} + + + org.apache.hbase + hbase-client + ${hbase.version} + org.ow2.asm asm-bom @@ -1263,6 +1021,24 @@ ${cglib.version} + + redis.clients + jedis + ${jedis.version} + + + + com.baomidou + mybatis-plus-boot-starter + ${mybatis-plus.boot.starter.version} + + + com.zaxxer + HikariCP + + + + jakarta.annotation @@ -1270,28 +1046,29 @@ ${annotation-api.version} - + - org.mockito - mockito-bom - ${mockito-bom.version} + com.azure + azure-sdk-bom + ${azure.blob.bom} pom import - - + - org.instancio - instancio-junit - 2.16.1 - test + com.azure + azure-storage-blob + 12.29.0 - - org.assertj - assertj-core - ${assertj.version} - test + com.azure + azure-storage-common + 12.29.0 + + + com.azure + azure-identity + 1.13.0 @@ -1365,27 +1142,9 @@ import - com.baomidou - mybatis-plus-boot-starter - ${mybatis-plus.version} - - - com.zaxxer - HikariCP - - - - - com.alibaba.cloud - spring-cloud-starter-alibaba-nacos-discovery - ${spring-cloud-alibaba.version} - - - com.azure - azure-sdk-bom - ${azure.blob.bom} - pom - import + org.springframework.boot + spring-boot-actuator + ${spring.boot.version} @@ -1403,12 +1162,39 @@ test + + + org.instancio + instancio-junit + 2.16.1 + test + + + + org.mockito + mockito-core + ${mockito-bom.version} + test + org.mockito mockito-inline + ${mockito-bom.version} + test + + + org.mockito + mockito-junit-jupiter + ${mockito-bom.version} test + + org.assertj + assertj-core + ${assertj.version} + test + org.springframework.boot spring-boot-starter-test @@ -1423,6 +1209,10 @@ org.springframework.boot spring-boot-starter-logging + + org.yaml + snakeyaml + @@ -1432,8 +1222,8 @@ test - com.zaxxer - HikariCP + org.springframework.boot + spring-boot-starter-jdbc test @@ -1441,12 +1231,6 @@ spring-test test - - org.postgresql - postgresql - ${postgresql.connector.version} - test - @@ -1612,17 +1396,16 @@ **/target/** **/out/** **/*.log - **/logs/** CONTRIBUTING.md CONTRIBUTING_CN.md README.md - .gitmodules **/META-INF/** .github/** compiler/** **/generated/** **/scritpis-test.sql - **/**.dolphin + .gitmodules + **/*.dolphin @@ -1792,35 +1575,150 @@ + + + hadoop-3.3 + + 3.3.1 + 4.2.0 + hadoop-hdfs-client + + hadoop-2.7 2.7.2 2.7.1 - hadoop-hdfs - - spark-2.4 + spark-3 + + 1.8.0-spark3 + 3.7.0-M11 + 3.4.4 + 2.12.17 + 2.12 + 2.14.2 + + + + + + + + + apache - org.apache.linkis - linkis-hadoop-hdfs-client-shade - linkis-hadoop-hdfs-client-shade - ${project.version} - compile - 2.4.3 - 2.11.12 - 2.11 + + 1.8.0 + + + 3.3.4 + hadoop-hdfs-client + 2.5.3 + 3.9.2 + 4.2.0 + 3.1.3 + + + 3.2.1 + 2.12.17 + 2.12 + + + provided + hadoop-common + hadoop-hdfs + org.apache.hadoop + ${hadoop.version} + 2.0.0 + provided + provided + 1.10.0 + 1.5.8 + + + 371 + 3.4.0.7.2.15.0-147 + 1.5.0 + 0.234 + 1.2.6 + 2.1.2 + 1.4.6 + 7.6.2 + + + 0.9.3 + 1.3.0 + 1.0 + 4 + 1 + 1 + python2 + 1 + + + 2.13.4.20221013 + 2.9.0 + 1.5.4 + + + 4.1.86.Final + 9.4.56.v20240826 + + + 5.3.27 + 5.7.8 + 2.7.11 + + + 3.0.3 + 3.0.0 + + + 3.9.0 + 3.4.3.4 + test + 42.3.9 + 1.2.23 + 3.12.8 + + + 32.0.0-jre + 2.17.0 + 1.26.1 + 1.9.4 + 1.5 + 3.9.0 + 1.3.1 + + + 1.78.1 + + + 4.3.1 + + + 5.3.0 + 3.9.0 + 6.4.0 + 0.24 + 1.16.2 + + + 3.3.4 + 1.2.30 + hbase-2.5 - !hbase.profile + hbase.profile @@ -1833,13 +1731,13 @@ hbase-1.2 - hbase.profile + !hbase.profile 1.2 hbase-1.2 - 1.2.0 + 1.2.1 @@ -1870,6 +1768,7 @@ 2.2.6 + jacoco @@ -1898,10 +1797,9 @@ org.apache.maven.plugins maven-surefire-plugin - 3 - true - -XX:MaxMetaspaceSize=2g ${extraJavaTestArgs} - -Dio.netty.tryReflectionSetAccessible=true + + + @@ -1966,19 +1864,5 @@ - - - storage-parquet - - compile - - - - - storage-orc - - compile - - diff --git a/quick-build.cmd b/quick-build.cmd new file mode 100644 index 00000000000..b788756fd4c --- /dev/null +++ b/quick-build.cmd @@ -0,0 +1,190 @@ +@echo off +chcp 65001 >nul 2>&1 +setlocal EnableDelayedExpansion + +REM ============================================================ +REM Linkis 混合编译脚本 (Windows 版本) +REM +REM 解决方案: 先并行编译所有模块，再串行打包 linkis-dist +REM 这样既能获得并行编译的性能提升，又能保证产物完整性 +REM +REM 预期效果: 性能提升 40-50%%，产物与串行编译完全一致 +REM +REM 用法: quick-build.cmd [选项] +REM --v2 编译 2.x 版本 (Hadoop 2 + Spark 2 + Hive 2) +REM 默认编译 3.x 版本 +REM ============================================================ + +set "THREADS=1C" +set "V2_MODE=false" +set "V2_PROFILE=" + +REM 解析命令行参数 +:parse_args +if "%~1"=="" goto :done_args +if "%~1"=="--v2" ( + set "V2_MODE=true" + set "V2_PROFILE=-Phadoop-2,spark-2,hive-2 -Dhadoop.profile=2" + shift + goto :parse_args +) +if "%~1"=="-t" ( + set "THREADS=%~2" + shift + shift + goto :parse_args +) +shift +goto :parse_args +:done_args + +echo. +echo ╔════════════════════════════════════════════════════════╗ +echo ║ Linkis 混合编译模式 (Hybrid Build) ║ +echo ╚════════════════════════════════════════════════════════╝ +echo. +echo 编译策略: +echo [1/2] 并行编译所有模块 (跳过 linkis-dist) - 使用 -T %THREADS% +echo [2/2] 串行打包 linkis-dist - 确保产物完整 +echo. +if "%V2_MODE%"=="true" ( + echo 版本: 2.x ^(Hadoop 2.7.2 + Spark 2.4.3 + Hive 2.3.3^) +) else ( + echo 版本: 3.x ^(Hadoop 3.3.4 + Spark 3.2.1 + Hive 3.1.3^) [默认] +) +echo. +echo 开始时间: %date% %time% +echo. + +REM 记录开始时间 +set "START_TIME=%time%" +call :GetSeconds "%START_TIME%" START_SECONDS + +REM ============================================================ +REM Step 1: 并行编译所有模块（跳过 linkis-dist） +REM ============================================================ +echo [1/2] 并行编译所有模块... +echo 执行: mvn clean install -T %THREADS% -DskipTests %V2_PROFILE% -pl "!:linkis-dist" +echo. + +set "STEP1_START=%time%" +call :GetSeconds "%STEP1_START%" STEP1_START_SEC + +call mvn clean install -T %THREADS% -DskipTests %V2_PROFILE% -pl "!:linkis-dist" +if %ERRORLEVEL% neq 0 ( + echo. + echo [错误] 步骤 1 编译失败! + exit /b 1 +) + +set "STEP1_END=%time%" +call :GetSeconds "%STEP1_END%" STEP1_END_SEC +set /a "STEP1_TIME=STEP1_END_SEC-STEP1_START_SEC" +if !STEP1_TIME! lss 0 set /a "STEP1_TIME+=86400" +set /a "STEP1_MIN=STEP1_TIME/60" +set /a "STEP1_SEC=STEP1_TIME%%60" + +echo. +echo [OK] 步骤 1 完成! 耗时: !STEP1_TIME! 秒 (!STEP1_MIN!分!STEP1_SEC!秒) +echo. + +REM ============================================================ +REM Step 2: 串行编译 linkis-dist +REM ============================================================ +echo [2/2] 串行打包 linkis-dist... +echo 执行: mvn install -pl :linkis-dist -DskipTests %V2_PROFILE% +echo. + +set "STEP2_START=%time%" +call :GetSeconds "%STEP2_START%" STEP2_START_SEC + +call mvn install -pl :linkis-dist -DskipTests %V2_PROFILE% +if %ERRORLEVEL% neq 0 ( + echo. + echo [错误] 步骤 2 编译失败! + exit /b 1 +) + +set "STEP2_END=%time%" +call :GetSeconds "%STEP2_END%" STEP2_END_SEC +set /a "STEP2_TIME=STEP2_END_SEC-STEP2_START_SEC" +if !STEP2_TIME! lss 0 set /a "STEP2_TIME+=86400" +set /a "STEP2_MIN=STEP2_TIME/60" +set /a "STEP2_SEC=STEP2_TIME%%60" + +echo. +echo [OK] 步骤 2 完成! 耗时: !STEP2_TIME! 秒 (!STEP2_MIN!分!STEP2_SEC!秒) +echo. + +REM ============================================================ +REM 计算总时间并显示结果 +REM ============================================================ +set "END_TIME=%time%" +call :GetSeconds "%END_TIME%" END_SECONDS +set /a "TOTAL_TIME=END_SECONDS-START_SECONDS" +if !TOTAL_TIME! lss 0 set /a "TOTAL_TIME+=86400" +set /a "TOTAL_MIN=TOTAL_TIME/60" +set /a "TOTAL_SEC=TOTAL_TIME%%60" + +echo ╔════════════════════════════════════════════════════════╗ +echo ║ 编译完成! ║ +echo ╚════════════════════════════════════════════════════════╝ +echo. +echo 耗时统计: +echo 步骤 1 (并行编译模块): !STEP1_TIME! 秒 (!STEP1_MIN!分!STEP1_SEC!秒) +echo 步骤 2 (串行打包): !STEP2_TIME! 秒 (!STEP2_MIN!分!STEP2_SEC!秒) +echo ──────────────────────────── +echo 总耗时: !TOTAL_TIME! 秒 (!TOTAL_MIN!分!TOTAL_SEC!秒) +echo. + +REM 检查产物 +set "DIST_DIR=linkis-dist\target\apache-linkis-1.8.0-bin" +if exist "%DIST_DIR%" ( + echo 产物信息: + echo 目录: %DIST_DIR% + + REM 统计文件数 + set "FILE_COUNT=0" + for /r "%DIST_DIR%" %%f in (*) do set /a "FILE_COUNT+=1" + echo 文件数: !FILE_COUNT! + echo. + + REM 检查关键目录 + echo 关键模块检查: + if exist "%DIST_DIR%\linkis-package\lib\linkis-computation-governance\linkis-cg-engineconnmanager" ( + echo [OK] linkis-cg-engineconnmanager + ) else ( + echo [X] linkis-cg-engineconnmanager (缺失!) + ) + if exist "%DIST_DIR%\linkis-package\lib\linkis-computation-governance\linkis-cg-entrance" ( + echo [OK] linkis-cg-entrance + ) else ( + echo [X] linkis-cg-entrance (缺失!) + ) + if exist "%DIST_DIR%\linkis-package\lib\linkis-computation-governance\linkis-cg-linkismanager" ( + echo [OK] linkis-cg-linkismanager + ) else ( + echo [X] linkis-cg-linkismanager (缺失!) + ) + echo. +) + +echo 混合编译完成! +echo 结束时间: %date% %time% +echo. + +exit /b 0 + +REM ============================================================ +REM 函数: 将时间转换为秒数 +REM ============================================================ +:GetSeconds +set "TIME_STR=%~1" +REM 处理时间格式 HH:MM:SS.CC 或 H:MM:SS.CC +for /f "tokens=1-4 delims=:,." %%a in ("%TIME_STR%") do ( + set /a "HOURS=%%a" + set /a "MINS=%%b" + set /a "SECS=%%c" +) +set /a "%~2=HOURS*3600+MINS*60+SECS" +exit /b diff --git a/quick-build.ps1 b/quick-build.ps1 new file mode 100644 index 00000000000..3a89dc26ea4 --- /dev/null +++ b/quick-build.ps1 @@ -0,0 +1,220 @@ +#Requires -Version 5.1 +<# +.SYNOPSIS + Linkis 混合编译脚本 (PowerShell 版本) + +.DESCRIPTION + 解决方案: 先并行编译所有模块，再串行打包 linkis-dist + 这样既能获得并行编译的性能提升，又能保证产物完整性 + 预期效果: 性能提升 40-50%，产物与串行编译完全一致 + +.PARAMETER Threads + 并行编译线程数，默认为 "1C" (使用 CPU 核心数) + +.PARAMETER SkipTests + 是否跳过测试，默认为 $true + +.PARAMETER V2 + 编译 2.x 版本 (Hadoop 2 + Spark 2 + Hive 2)，默认编译 3.x 版本 + +.EXAMPLE + .\quick-build.ps1 + 使用默认设置编译 3.x 版本 + +.EXAMPLE + .\quick-build.ps1 -V2 + 编译 2.x 版本 + +.EXAMPLE + .\quick-build.ps1 -Threads 4 + 使用 4 线程编译 + +.EXAMPLE + .\quick-build.ps1 -V2 -Threads 4 + 编译 2.x 版本，使用 4 线程 +#> + +param( + [string]$Threads = "1C", + [switch]$SkipTests = $true, + [switch]$V2 = $false +) + +$ErrorActionPreference = "Stop" +$OutputEncoding = [System.Text.Encoding]::UTF8 + +# 颜色定义 +function Write-ColorOutput { + param( + [string]$Message, + [string]$Color = "White" + ) + Write-Host $Message -ForegroundColor $Color +} + +function Write-Banner { + param([string]$Text) + Write-Host "" + Write-ColorOutput "╔════════════════════════════════════════════════════════╗" "Cyan" + Write-ColorOutput "║$($Text.PadLeft(29 + $Text.Length/2).PadRight(58))║" "Cyan" + Write-ColorOutput "╚════════════════════════════════════════════════════════╝" "Cyan" + Write-Host "" +} + +function Format-Duration { + param([TimeSpan]$Duration) + if ($Duration.TotalMinutes -ge 1) { + return "{0}分{1}秒" -f [int]$Duration.TotalMinutes, $Duration.Seconds + } + return "{0}秒" -f [int]$Duration.TotalSeconds +} + +# ============================================================ +# 主程序开始 +# ============================================================ + +Write-Banner "Linkis 混合编译模式 (Hybrid Build)" + +Write-ColorOutput "📋 编译策略:" "Yellow" +Write-Host " [1/2] 并行编译所有模块 (跳过 linkis-dist) - 使用 -T $Threads" +Write-Host " [2/2] 串行打包 linkis-dist - 确保产物完整" +Write-Host "" +if ($V2) { + Write-ColorOutput "🔧 版本: 2.x (Hadoop 2.7.2 + Spark 2.4.3 + Hive 2.3.3)" "Yellow" +} else { + Write-ColorOutput "🔧 版本: 3.x (Hadoop 3.3.4 + Spark 3.2.1 + Hive 3.1.3) [默认]" "Yellow" +} +Write-Host "" +Write-ColorOutput ("⏱️ 开始时间: " + (Get-Date -Format "yyyy-MM-dd HH:mm:ss")) "Yellow" +Write-Host "" + +$TotalStartTime = Get-Date +$SkipTestsArg = if ($SkipTests) { "-DskipTests" } else { "" } +$V2ProfileArg = if ($V2) { "-Phadoop-2,spark-2,hive-2 -Dhadoop.profile=2" } else { "" } + +# ============================================================ +# Step 1: 并行编译所有模块（跳过 linkis-dist） +# ============================================================ +Write-ColorOutput "[1/2] 🚀 并行编译所有模块..." "Green" +$cmd = "mvn clean install -T $Threads $SkipTestsArg $V2ProfileArg -pl `"!:linkis-dist`"" +Write-Host "执行: $cmd" +Write-Host "" + +$Step1Start = Get-Date + +try { + $mvnArgs = @("clean", "install", "-T", $Threads) + if ($SkipTestsArg) { $mvnArgs += $SkipTestsArg } + if ($V2) { + $mvnArgs += "-Phadoop-2,spark-2,hive-2" + $mvnArgs += "-Dhadoop.profile=2" + } + $mvnArgs += @("-pl", "!:linkis-dist") + + & mvn $mvnArgs + if ($LASTEXITCODE -ne 0) { + throw "Maven 编译失败，退出码: $LASTEXITCODE" + } +} catch { + Write-ColorOutput "❌ 步骤 1 编译失败: $_" "Red" + exit 1 +} + +$Step1End = Get-Date +$Step1Duration = $Step1End - $Step1Start + +Write-Host "" +Write-ColorOutput ("✅ 步骤 1 完成! 耗时: " + (Format-Duration $Step1Duration)) "Green" +Write-Host "" + +# ============================================================ +# Step 2: 串行编译 linkis-dist +# ============================================================ +Write-ColorOutput "[2/2] 📦 串行打包 linkis-dist..." "Green" +$cmd = "mvn install -pl :linkis-dist $SkipTestsArg $V2ProfileArg" +Write-Host "执行: $cmd" +Write-Host "" + +$Step2Start = Get-Date + +try { + $mvnArgs = @("install", "-pl", ":linkis-dist") + if ($SkipTestsArg) { $mvnArgs += $SkipTestsArg } + if ($V2) { + $mvnArgs += "-Phadoop-2,spark-2,hive-2" + $mvnArgs += "-Dhadoop.profile=2" + } + + & mvn $mvnArgs + if ($LASTEXITCODE -ne 0) { + throw "Maven 打包失败，退出码: $LASTEXITCODE" + } +} catch { + Write-ColorOutput "❌ 步骤 2 编译失败: $_" "Red" + exit 1 +} + +$Step2End = Get-Date +$Step2Duration = $Step2End - $Step2Start + +Write-Host "" +Write-ColorOutput ("✅ 步骤 2 完成! 耗时: " + (Format-Duration $Step2Duration)) "Green" +Write-Host "" + +# ============================================================ +# 显示结果 +# ============================================================ +$TotalEndTime = Get-Date +$TotalDuration = $TotalEndTime - $TotalStartTime + +Write-Banner "编译完成!" + +Write-ColorOutput "📊 耗时统计:" "Yellow" +Write-Host (" 步骤 1 (并行编译模块): " + (Format-Duration $Step1Duration)) +Write-Host (" 步骤 2 (串行打包): " + (Format-Duration $Step2Duration)) +Write-Host " ────────────────────────────" +Write-ColorOutput (" 总耗时: " + (Format-Duration $TotalDuration)) "Green" +Write-Host "" + +# 检查产物 +$DistDir = Join-Path $PSScriptRoot "linkis-dist\target\apache-linkis-1.8.0-bin" +if (Test-Path $DistDir) { + $Files = Get-ChildItem -Path $DistDir -Recurse -File + $FileCount = $Files.Count + $TotalSize = ($Files | Measure-Object -Property Length -Sum).Sum + $SizeFormatted = if ($TotalSize -ge 1GB) { + "{0:N2} GB" -f ($TotalSize / 1GB) + } elseif ($TotalSize -ge 1MB) { + "{0:N0} MB" -f ($TotalSize / 1MB) + } else { + "{0:N0} KB" -f ($TotalSize / 1KB) + } + + Write-ColorOutput "📦 产物信息:" "Yellow" + Write-Host " 目录: $DistDir" + Write-Host " 文件数: $FileCount" + Write-Host " 总大小: $SizeFormatted" + Write-Host "" + + # 检查关键目录 + Write-ColorOutput "🔍 关键模块检查:" "Yellow" + $Modules = @( + "linkis-cg-engineconnmanager", + "linkis-cg-entrance", + "linkis-cg-linkismanager" + ) + + foreach ($Module in $Modules) { + $ModulePath = Join-Path $DistDir "linkis-package\lib\linkis-computation-governance\$Module" + if (Test-Path $ModulePath) { + Write-ColorOutput " ✅ $Module" "Green" + } else { + Write-ColorOutput " ❌ $Module (缺失!)" "Red" + } + } + Write-Host "" +} + +Write-ColorOutput "🎉 混合编译完成!" "Green" +Write-Host (" 结束时间: " + (Get-Date -Format "yyyy-MM-dd HH:mm:ss")) +Write-Host "" diff --git a/quick-build.sh b/quick-build.sh new file mode 100755 index 00000000000..ec564845087 --- /dev/null +++ b/quick-build.sh @@ -0,0 +1,386 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# Linkis 混合编译脚本 (Linux/macOS) +# +# 解决方案：分阶段编译 +# Stage 1: 并行编译基础模块 (跳过 hbase 和 linkis-dist) +# Stage 2: 串行编译 hbase 模块 (依赖基础模块) +# Stage 3: 并行编译剩余模块和 linkis-dist +# +# 预期效果：性能提升 40-50%，产物与串行编译完全一致 +# +# 用法：./quick-build.sh [选项] +# -t, --threads 并行线程数，默认为 1C (CPU 核心数) +# -s, --skip-tests 跳过测试 (默认) +# -r, --run-tests 运行测试 +# --v2 编译默认版本 (Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11 + Hive 2.3.3), 同默认 +# --v3 编译 Apache 发布版本 (Hadoop 3.3.4 + Spark 3.2.1 + Scala 2.12 + Hive 3.1.3), 同 -Papache +# --hadoop 指定 Hadoop 完整版本号 (如 3.3.4, 2.7.2) +# --spark 指定 Spark 完整版本号 (如 3.4.4, 3.2.1, 2.4.3) +# --hive 指定 Hive 完整版本号 (如 3.1.3, 2.3.3) +# --scala 指定 Scala 完整版本号 (如 2.12.17, 2.11.12) +# --revision 指定 revision 版本号 (默认：1.8.0) +# -h, --help 显示帮助 +# +# 版本说明: +# 默认版本：Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11.12 + Hive 2.3.3 +# -Pspark-3: Spark 3.4.4 + Scala 2.12.17 (Hadoop/Hive 保持默认) +# -Papache / --v3: Hadoop 3.3.4 + Spark 3.2.1 + Scala 2.12.17 + Hive 3.1.3 +# +# 示例: +# ./quick-build.sh 使用默认版本编译 (Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11 + Hive 2.3.3) +# ./quick-build.sh --v2 同默认版本 (仅用于明确指定 v2) +# ./quick-build.sh --v3 编译 Apache 发布版本 (同 -Papache) +# ./quick-build.sh -Pspark-3 只升级 Spark 到 3.4.4 + Scala 2.12.17 +# ./quick-build.sh --revision 1.8.0-spark2 指定 revision 编译 (用于区分不同版本) +# ./quick-build.sh --spark 3.4.4 指定 Spark 3.4.4 编译 +# ./quick-build.sh --hadoop 3.3.4 --spark 3.4.4 --hive 3.1.3 指定完整版本组合 +# ./quick-build.sh --spark 3.4.4 --scala 2.12.17 --hadoop 3.3.1 自定义版本组合 +# ./quick-build.sh -t 4 使用 4 线程编译 +# ./quick-build.sh -r 运行测试 +# +# 注意：--hadoop/--spark/--hive/--scala 参数优先级高于 --v2/--v3，可覆盖 Profile 中的版本 +# + +set -e + +# 默认参数 +THREADS="1C" +SKIP_TESTS=true +V2_MODE=false +V3_MODE=false +CUSTOM_VERSION_MODE=false +HADOOP_VERSION="" +SPARK_VERSION="" +HIVE_VERSION="" +SCALA_VERSION="" +REVISION="" + +# 颜色定义 +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' + +# 获取脚本所在目录 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_DIR="$SCRIPT_DIR" + +# 帮助信息 +show_help() { + echo "用法：$0 [选项]" + echo "" + echo "选项:" + echo " -t, --threads 并行线程数，默认为 1C (CPU 核心数)" + echo " -s, --skip-tests 跳过测试 (默认)" + echo " -r, --run-tests 运行测试" + echo " --v2 编译默认版本 (Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11 + Hive 2.3.3), 同默认" + echo " --v3 编译 Apache 发布版本 (Hadoop 3.3.4 + Spark 3.2.1 + Scala 2.12 + Hive 3.1.3), 同 -Papache" + echo " --hadoop 指定 Hadoop 完整版本号 (如 3.3.4, 2.7.2)" + echo " --spark 指定 Spark 完整版本号 (如 3.4.4, 3.2.1, 2.4.3)" + echo " --hive 指定 Hive 完整版本号 (如 3.1.3, 2.3.3)" + echo " --scala 指定 Scala 完整版本号 (如 2.12.17, 2.11.12)" + echo " --revision 指定 revision 版本号 (默认：1.8.0)" + echo " -h, --help 显示帮助" + echo "" + echo "版本参数说明:" + echo " --hadoop/--spark/--hive/--scala 可单独使用，也可与 --v2/--v3 组合使用" + echo " 当与 --v2/--v3 同时使用时，指定的版本会覆盖对应预设版本的默认值" + echo "" + echo "编译方式说明:" + echo " 默认/-Pspark-2: Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11.12 + Hive 2.3.3" + echo " -Pspark-3: Spark 3.4.4 + Scala 2.12.17 (Hadoop/Hive 保持默认)" + echo " -Papache / --v3: Hadoop 3.3.4 + Spark 3.2.1 + Scala 2.12.17 + Hive 3.1.3" + echo "" + echo "示例:" + echo " ./quick-build.sh 默认版本 (Hadoop 2.7.2 + Spark 2.4.3)" + echo " ./quick-build.sh --v2 同默认版本" + echo " ./quick-build.sh --v3 Apache 发布版本 (同 -Papache)" + echo " ./quick-build.sh --spark 3.4.4 只升级 Spark 到 3.4.4" + echo " ./quick-build.sh --v3 --spark 3.4.4 --hadoop 3.3.1 Apache 基础上修改 Spark 和 Hadoop" + echo " ./quick-build.sh --hadoop 3.3.4 --spark 3.2.1 --hive 3.1.3 指定完整版本组合" + echo " ./quick-build.sh --revision 1.8.0-spark2 指定 revision 编译" + echo " ./quick-build.sh -t 4 使用 4 线程编译" + echo " ./quick-build.sh -r 运行测试编译" + echo "" + echo "双版本编译 (分别执行两次，避免 Maven 仓库覆盖):" + echo " ./quick-build.sh # 编译默认版本 (Spark 2.4.3)" + echo " ./quick-build.sh --v3 --revision 1.8.0-apache # 编译 Apache 版本 (Spark 3.2.1)" +} + +# 解析参数 +while [[ $# -gt 0 ]]; do + case $1 in + -t|--threads) + THREADS="$2" + shift 2 + ;; + -s|--skip-tests) + SKIP_TESTS=true + shift + ;; + -r|--run-tests) + SKIP_TESTS=false + shift + ;; + --v2) + V2_MODE=true + shift + ;; + --v3) + V3_MODE=true + shift + ;; + --hadoop) + CUSTOM_VERSION_MODE=true + HADOOP_VERSION="$2" + shift 2 + ;; + --spark) + CUSTOM_VERSION_MODE=true + SPARK_VERSION="$2" + shift 2 + ;; + --hive) + CUSTOM_VERSION_MODE=true + HIVE_VERSION="$2" + shift 2 + ;; + --scala) + CUSTOM_VERSION_MODE=true + SCALA_VERSION="$2" + shift 2 + ;; + --revision) + REVISION="$2" + shift 2 + ;; + -h|--help) + show_help + exit 0 + ;; + *) + echo "未知选项：$1" + show_help + exit 1 + ;; + esac +done + +# 构建 Maven 参数 +SKIP_TESTS_ARG="" +if [ "$SKIP_TESTS" = true ]; then + SKIP_TESTS_ARG="-DskipTests" +fi + +# Profile 参数 +PROFILE_ARG="" +if [ "$V3_MODE" = true ]; then + # --v3 模式：使用 -Papache profile (Hadoop 3.3.4 + Spark 3.2.1 + Scala 2.12 + Hive 3.1.3) + PROFILE_ARG="-Papache" +else + # 默认模式 (包括 --v2): 使用默认配置 (Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11 + Hive 2.3.3) + # 不需要额外指定 profile + PROFILE_ARG="" +fi + +# 自定义版本参数（优先级高于 --v2/--v3） +CUSTOM_VERSION_ARGS="" +if [ -n "$HADOOP_VERSION" ]; then + CUSTOM_VERSION_ARGS="$CUSTOM_VERSION_ARGS -Dhadoop.version=$HADOOP_VERSION" +fi +if [ -n "$SPARK_VERSION" ]; then + CUSTOM_VERSION_ARGS="$CUSTOM_VERSION_ARGS -Dspark.version=$SPARK_VERSION" +fi +if [ -n "$HIVE_VERSION" ]; then + CUSTOM_VERSION_ARGS="$CUSTOM_VERSION_ARGS -Dhive.version=$HIVE_VERSION" +fi +if [ -n "$SCALA_VERSION" ]; then + CUSTOM_VERSION_ARGS="$CUSTOM_VERSION_ARGS -Dscala.version=$SCALA_VERSION" +fi + +# Revision 参数 +REVISION_ARG="" +if [ -n "$REVISION" ]; then + REVISION_ARG="-Drevision=$REVISION" +fi + +# 格式化时间 +format_duration() { + local seconds=$1 + local minutes=$((seconds / 60)) + local secs=$((seconds % 60)) + if [ $minutes -gt 0 ]; then + echo "${minutes}分${secs}秒" + else + echo "${secs}秒" + fi +} + +# 确定显示的版本信息 +if [ "$CUSTOM_VERSION_MODE" = true ]; then + # 自定义版本模式 + VERSION_DISPLAY="自定义版本 (" + [ -n "$HADOOP_VERSION" ] && VERSION_DISPLAY="$VERSION_DISPLAY Hadoop $HADOOP_VERSION" + [ -n "$SPARK_VERSION" ] && VERSION_DISPLAY="$VERSION_DISPLAY Spark $SPARK_VERSION" + [ -n "$HIVE_VERSION" ] && VERSION_DISPLAY="$VERSION_DISPLAY Hive $HIVE_VERSION" + [ -n "$SCALA_VERSION" ] && VERSION_DISPLAY="$VERSION_DISPLAY Scala $SCALA_VERSION" + VERSION_DISPLAY="$VERSION_DISPLAY )" + DEFAULT_REVISION="1.8.0" +elif [ "$V3_MODE" = true ]; then + # --v3 模式：Apache 发布版本 + VERSION_DISPLAY="Apache 发布版本 (Hadoop 3.3.4 + Spark 3.2.1 + Scala 2.12.17 + Hive 3.1.3)" + DEFAULT_REVISION="1.8.0-apache" +else + # 默认模式 (包括 --v2) + VERSION_DISPLAY="默认版本 (Hadoop 2.7.2 + Spark 2.4.3 + Scala 2.11.12 + Hive 2.3.3)" + DEFAULT_REVISION="1.8.0" +fi + +# 如果未指定 revision，使用默认值 +if [ -z "$REVISION" ]; then + REVISION_ARG="-Drevision=$DEFAULT_REVISION" +fi + +echo "" +echo -e "${BLUE}╔════════════════════════════════════════════════════════╗${NC}" +echo -e "${BLUE}║ Linkis 混合编译模式 (Hybrid Build) ║${NC}" +echo -e "${BLUE}╚════════════════════════════════════════════════════════╝${NC}" +echo "" +echo -e "${YELLOW}📋 编译策略:${NC}" +echo " [1/3] 并行编译基础模块 (跳过 hbase 和 linkis-dist) - 使用 -T $THREADS" +echo " [2/3] 串行编译 hbase 模块 (依赖基础模块)" +echo " [3/3] 并行编译剩余模块和 linkis-dist" +echo "" +echo -e "${YELLOW}🔧 版本：${VERSION_DISPLAY}${NC}" +echo -e "${YELLOW}📦 Revision: ${REVISION_ARG#-Drevision=}${NC}" +echo "" +echo -e "${YELLOW}⏱️ 开始时间：$(date '+%Y-%m-%d %H:%M:%S')${NC}" +echo "" + +# 记录开始时间 +START_TIME=$(date +%s) + +# ============================================================ +# Step 1: 并行编译基础模块（跳过 hbase 和 linkis-dist） +# ============================================================ +echo -e "${GREEN}[1/3] 🚀 并行编译基础模块...${NC}" +echo "执行：mvn clean install -T $THREADS $SKIP_TESTS_ARG $PROFILE_ARG $CUSTOM_VERSION_ARGS $REVISION_ARG -pl '!org.apache.linkis:linkis-engineconn-plugin-hbase','!org.apache.linkis:linkis-engineconn-plugin-hbase-module','!org.apache.linkis:hbase-shims-1.2.1','!org.apache.linkis:hbase-shims-1.4.3','!org.apache.linkis:hbase-shims-2.2.6','!org.apache.linkis:hbase-shims-2.5.3','!org.apache.linkis:linkis-dist'" +echo "" + +cd "$PROJECT_DIR" +STEP1_START=$(date +%s) + +mvn clean install -T $THREADS $SKIP_TESTS_ARG $PROFILE_ARG $CUSTOM_VERSION_ARGS $REVISION_ARG -pl '!org.apache.linkis:linkis-engineconn-plugin-hbase','!org.apache.linkis:linkis-engineconn-plugin-hbase-module','!org.apache.linkis:hbase-shims-1.2.1','!org.apache.linkis:hbase-shims-1.4.3','!org.apache.linkis:hbase-shims-2.2.6','!org.apache.linkis:hbase-shims-2.5.3','!org.apache.linkis:linkis-dist' + +STEP1_END=$(date +%s) +STEP1_TIME=$((STEP1_END - STEP1_START)) + +echo "" +echo -e "${GREEN}✅ 步骤 1 完成！耗时：${STEP1_TIME} 秒 ($(format_duration $STEP1_TIME))${NC}" +echo "" + +# ============================================================ +# Step 2: 串行编译 hbase 模块（依赖基础模块） +# ============================================================ +echo -e "${GREEN}[2/3] 📦 串行编译 hbase 模块...${NC}" +echo "执行：mvn install -pl org.apache.linkis:linkis-engineconn-plugin-hbase-module -am $SKIP_TESTS_ARG $PROFILE_ARG $CUSTOM_VERSION_ARGS $REVISION_ARG" +echo "" + +STEP2_START=$(date +%s) + +# 编译 hbase 模块（使用 -am 确保依赖也被编译，但基础模块已编译会直接跳过） +mvn install -pl org.apache.linkis:linkis-engineconn-plugin-hbase-module -am $SKIP_TESTS_ARG $PROFILE_ARG $CUSTOM_VERSION_ARGS $REVISION_ARG + +STEP2_END=$(date +%s) +STEP2_TIME=$((STEP2_END - STEP2_START)) + +echo "" +echo -e "${GREEN}✅ 步骤 2 完成！耗时：${STEP2_TIME} 秒 ($(format_duration $STEP2_TIME))${NC}" +echo "" + +# ============================================================ +# Step 3: 并行编译剩余模块和 linkis-dist +# ============================================================ +echo -e "${GREEN}[3/3] 🚀 并行编译 linkis-dist...${NC}" +echo "执行：mvn install -T $THREADS -pl :linkis-dist $SKIP_TESTS_ARG $PROFILE_ARG $CUSTOM_VERSION_ARGS $REVISION_ARG" +echo "" + +STEP3_START=$(date +%s) + +mvn install -T $THREADS -pl :linkis-dist $SKIP_TESTS_ARG $PROFILE_ARG $CUSTOM_VERSION_ARGS $REVISION_ARG + +STEP3_END=$(date +%s) +STEP3_TIME=$((STEP3_END - STEP3_START)) + +echo "" +echo -e "${GREEN}✅ 步骤 3 完成！耗时：${STEP3_TIME} 秒 ($(format_duration $STEP3_TIME))${NC}" +echo "" + +# ============================================================ +# 显示结果 +# ============================================================ +END_TIME=$(date +%s) +TOTAL_TIME=$((END_TIME - START_TIME)) + +CURRENT_REVISION=${REVISION_ARG#-Drevision=} +DIST_DIR="$PROJECT_DIR/linkis-dist/target/apache-linkis-${CURRENT_REVISION}-bin" + +echo -e "${BLUE}╔════════════════════════════════════════════════════════╗${NC}" +echo -e "${BLUE}║ 编译完成！ ║${NC}" +echo -e "${BLUE}╚════════════════════════════════════════════════════════╝${NC}" +echo "" +echo -e "${YELLOW}📊 耗时统计:${NC}" +echo " 步骤 1 (并行编译基础模块): ${STEP1_TIME} 秒 ($(format_duration $STEP1_TIME))" +echo " 步骤 2 (串行编译 hbase): ${STEP2_TIME} 秒 ($(format_duration $STEP2_TIME))" +echo " 步骤 3 (并行编译剩余模块): ${STEP3_TIME} 秒 ($(format_duration $STEP3_TIME))" +echo " ──────────────────────────────────" +echo -e " ${GREEN}总耗时：${TOTAL_TIME} 秒 ($(format_duration $TOTAL_TIME))${NC}" +echo "" + +# 检查产物 +if [ -d "$DIST_DIR" ]; then + FILE_COUNT=$(find "$DIST_DIR" -type f 2>/dev/null | wc -l) + DIR_SIZE=$(du -sh "$DIST_DIR" 2>/dev/null | cut -f1) + + echo -e "${YELLOW}📦 产物信息:${NC}" + echo " 目录：$DIST_DIR" + echo " 文件数：$FILE_COUNT" + echo " 总大小：$DIR_SIZE" + echo "" + + # 检查关键目录 + echo -e "${YELLOW}🔍 关键模块检查:${NC}" + for module in "linkis-cg-engineconnmanager" "linkis-cg-entrance" "linkis-cg-linkismanager"; do + module_path="$DIST_DIR/linkis-package/lib/linkis-computation-governance/$module" + if [ -d "$module_path" ]; then + echo -e " ${GREEN}✅ $module${NC}" + else + echo -e " ${RED}❌ $module (缺失!)${NC}" + fi + done + echo "" +else + echo -e "${YELLOW}⚠️ 产物目录不存在：$DIST_DIR${NC}" + echo "" +fi + +echo -e "${GREEN}🎉 混合编译完成!${NC}" +echo " 结束时间：$(date '+%Y-%m-%d %H:%M:%S')" +echo "" diff --git a/tool/dependencies/known-dependencies.txt b/tool/dependencies/known-dependencies.txt index 3297af87ed9..3aec99db6b7 100644 --- a/tool/dependencies/known-dependencies.txt +++ b/tool/dependencies/known-dependencies.txt @@ -1,14 +1,17 @@ -HdrHistogram-2.1.12.jar -HdrHistogram-2.1.9.jar -LatencyUtils-2.0.3.jar -ST4-4.0.4.jar -SparseBitSet-1.3.jar +accessors-smart-2.4.11.jar accessors-smart-2.4.9.jar +activation-1.1.jar agrona-1.12.0.jar aircompressor-0.10.jar +aircompressor-0.27.jar +aircompressor-0.3.jar +akka-actor_2.11-2.5.21.jar akka-actor_2.12-2.5.21.jar +akka-protobuf_2.11-2.5.21.jar akka-protobuf_2.12-2.5.21.jar +akka-slf4j_2.11-2.5.21.jar akka-slf4j_2.12-2.5.21.jar +akka-stream_2.11-2.5.21.jar akka-stream_2.12-2.5.21.jar aliyun-java-sdk-core-4.5.10.jar aliyun-java-sdk-kms-2.11.0.jar @@ -17,13 +20,18 @@ aliyun-sdk-oss-3.16.0.jar annotations-13.0.jar annotations-17.0.0.jar ant-1.10.12.jar -ant-launcher-1.10.12.jar antisamy-1.4.3.jar +ant-launcher-1.10.12.jar antlr-2.7.7.jar antlr-runtime-3.4.jar antlr-runtime-3.5.2.jar aopalliance-1.0.jar aopalliance-repackaged-2.4.0-b34.jar +apache-curator-2.7.1.pom +apacheds-i18n-2.0.0-M15.jar +apacheds-kerberos-codec-2.0.0-M15.jar +api-asn1-api-1.0.0-M20.jar +api-util-1.0.0-M20.jar arrow-format-0.8.0.jar arrow-format-2.0.0.jar arrow-memory-0.8.0.jar @@ -36,23 +44,39 @@ asm-commons-9.3.jar asm-tree-9.3.jar aspectjweaver-1.9.7.jar attoparser-2.0.5.RELEASE.jar +attoparser-2.0.7.RELEASE.jar +audience-annotations-0.12.0.jar audience-annotations-0.13.0.jar audience-annotations-0.5.0.jar -audience-annotations-0.12.0.jar automaton-1.11-8.jar avatica-1.11.0.jar +avatica-1.8.0.jar +avatica-metrics-1.8.0.jar +avro-1.7.4.jar avro-1.7.7.jar avro-1.8.2.jar aws-java-sdk-core-1.12.261.jar aws-java-sdk-kms-1.12.261.jar aws-java-sdk-s3-1.12.261.jar +azure-core-1.54.1.jar +azure-core-http-netty-1.15.7.jar +azure-identity-1.13.0.jar +azure-json-1.3.0.jar +azure-storage-blob-12.29.0.jar +azure-storage-common-12.29.0.jar +azure-storage-internal-avro-12.14.0.jar +azure-xml-1.1.0.jar batik-css-1.7.jar batik-ext-1.7.jar batik-util-1.7.jar +bcpkix-jdk15on-1.69.jar bcpkix-jdk18on-1.78.1.jar bcpkix-jdk18on-1.79.jar +bcprov-jdk15on-1.69.jar bcprov-jdk18on-1.78.1.jar bcprov-jdk18on-1.79.jar +bcprov-jdk18on-1.80.jar +bcutil-jdk15on-1.69.jar bcutil-jdk18on-1.78.1.jar bcutil-jdk18on-1.79.jar bonecp-0.8.0.RELEASE.jar @@ -64,15 +88,22 @@ bval-jsr-2.0.0.jar byte-buddy-1.12.23.jar caffeine-2.8.4.jar caffeine-2.9.3.jar +calcite-core-1.10.0.jar calcite-core-1.16.0.jar +calcite-druid-1.10.0.jar calcite-druid-1.16.0.jar +calcite-linq4j-1.10.0.jar calcite-linq4j-1.16.0.jar cglib-nodep-3.2.5.jar +checker-qual-3.19.0.jar checker-qual-3.33.0.jar checker-qual-3.4.0.jar +checker-qual-3.42.0.jar checker-qual-3.5.0.jar -chill-java-0.7.6.jar +chill_2.11-0.7.6.jar chill_2.12-0.7.6.jar +chill-java-0.7.6.jar +classgraph-4.8.112.jar classgraph-4.8.83.jar classmate-1.5.1.jar clickhouse-jdbc-0.4.6.jar @@ -80,12 +111,15 @@ client-3.0.0.jar commons-beanutils-1.11.0.jar commons-beanutils-1.9.4.jar commons-beanutils-core-1.7.0.jar +commons-cli-1.2.jar commons-cli-1.3.1.jar +commons-cli-1.5.0.jar commons-codec-1.10.jar commons-codec-1.15.jar commons-collections-3.2.2.jar commons-collections4-4.4.jar commons-compiler-3.1.9.jar +commons-compress-1.21.jar commons-compress-1.26.1.jar commons-configuration-1.10.jar commons-configuration2-2.1.1.jar @@ -98,14 +132,15 @@ commons-exec-1.3.jar commons-fileupload-1.5.jar commons-fileupload-1.6.0.jar commons-httpclient-3.1.jar +commons-io-2.11.0.jar commons-io-2.17.0.jar commons-jxpath-1.3.jar commons-lang-2.6.jar commons-lang3-3.12.0.jar -commons-lang3-3.18.0.jar commons-logging-1.2.jar commons-math-2.2.jar commons-math3-3.6.1.jar +commons-net-3.1.jar commons-net-3.9.0.jar commons-pool-1.6.jar commons-pool2-2.11.1.jar @@ -113,9 +148,14 @@ commons-text-1.10.0.jar concurrent-0.191.jar config-1.3.3.jar configuration-0.191.jar +content-type-2.3.jar +curator-client-2.7.1.jar curator-client-4.2.0.jar +curator-framework-2.7.1.jar curator-framework-4.2.0.jar +curator-recipes-2.7.1.jar curator-recipes-4.2.0.jar +curvesapi-1.07.jar curvesapi-1.08.jar datanucleus-api-jdo-4.2.4.jar datanucleus-core-4.1.17.jar @@ -127,11 +167,14 @@ disruptor-3.4.2.jar dnsjava-2.1.7.jar dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar druid-1.2.23.jar +druid-1.2.4.jar +eigenbase-properties-1.1.5.jar elasticsearch-rest-client-6.8.15.jar elasticsearch-rest-client-7.6.2.jar elasticsearch-rest-client-sniffer-7.6.2.jar error_prone_annotations-2.10.0.jar error_prone_annotations-2.18.0.jar +error_prone_annotations-2.26.1.jar error_prone_annotations-2.3.4.jar esapi-2.1.0.jar esri-geometry-api-2.0.0.jar @@ -142,6 +185,7 @@ excel-streaming-reader-5.0.2.jar expiringmap-0.5.6.jar failsafe-2.4.0.jar failureaccess-1.0.1.jar +failureaccess-1.0.2.jar fastutil-6.5.6.jar feign-core-11.10.jar feign-form-3.8.0.jar @@ -154,14 +198,17 @@ flink-annotations-1.12.2.jar flink-annotations-1.16.2.jar flink-cep-1.16.2.jar flink-clients-1.16.2.jar +flink-clients_2.11-1.12.2.jar flink-clients_2.12-1.12.2.jar flink-connector-base-1.12.2.jar flink-connector-base-1.16.2.jar flink-connector-files-1.12.2.jar flink-connector-files-1.16.2.jar +flink-connector-hive_2.11-1.12.2.jar flink-connector-hive_2.12-1.12.2.jar flink-connector-hive_2.12-1.16.2.jar flink-connector-kafka-1.16.2.jar +flink-connector-kafka_2.11-1.12.2.jar flink-connector-kafka_2.12-1.12.2.jar flink-core-1.12.2.jar flink-core-1.16.2.jar @@ -174,17 +221,21 @@ flink-java-1.16.2.jar flink-json-1.12.2.jar flink-json-1.16.2.jar flink-kubernetes-1.16.2.jar +flink-kubernetes_2.11-1.12.2.jar flink-kubernetes_2.12-1.12.2.jar flink-metrics-core-1.12.2.jar flink-metrics-core-1.16.2.jar flink-optimizer-1.16.2.jar +flink-optimizer_2.11-1.12.2.jar flink-optimizer_2.12-1.12.2.jar flink-queryable-state-client-java-1.12.2.jar flink-queryable-state-client-java-1.16.2.jar flink-rpc-akka-loader-1.16.2.jar flink-rpc-core-1.16.2.jar flink-runtime-1.16.2.jar +flink-runtime_2.11-1.12.2.jar flink-runtime_2.12-1.12.2.jar +flink-scala_2.11-1.12.2.jar flink-scala_2.12-1.12.2.jar flink-scala_2.12-1.16.2.jar flink-shaded-asm-7-7.1-12.0.jar @@ -199,68 +250,98 @@ flink-shaded-netty-4.1.70.Final-15.0.jar flink-shaded-zookeeper-3-3.4.14-12.0.jar flink-shaded-zookeeper-3-3.5.9-15.0.jar flink-sql-client-1.16.2.jar +flink-sql-client_2.11-1.12.2.jar flink-sql-client_2.12-1.12.2.jar flink-sql-gateway-api-1.16.2.jar flink-streaming-java-1.16.2.jar +flink-streaming-java_2.11-1.12.2.jar flink-streaming-java_2.12-1.12.2.jar +flink-streaming-scala_2.11-1.12.2.jar flink-streaming-scala_2.12-1.12.2.jar flink-table-api-bridge-base-1.16.2.jar flink-table-api-java-1.12.2.jar flink-table-api-java-1.16.2.jar flink-table-api-java-bridge-1.16.2.jar +flink-table-api-java-bridge_2.11-1.12.2.jar flink-table-api-java-bridge_2.12-1.12.2.jar -flink-table-api-scala-bridge_2.12-1.12.2.jar +flink-table-api-scala_2.11-1.12.2.jar flink-table-api-scala_2.12-1.12.2.jar +flink-table-api-scala-bridge_2.11-1.12.2.jar +flink-table-api-scala-bridge_2.12-1.12.2.jar flink-table-common-1.12.2.jar flink-table-common-1.16.2.jar -flink-table-planner-blink_2.12-1.12.2.jar flink-table-planner_2.12-1.16.2.jar +flink-table-planner-blink_2.11-1.12.2.jar +flink-table-planner-blink_2.12-1.12.2.jar flink-table-runtime-1.16.2.jar +flink-table-runtime-blink_2.11-1.12.2.jar flink-table-runtime-blink_2.12-1.12.2.jar flink-yarn-1.16.2.jar +flink-yarn_2.11-1.12.2.jar flink-yarn_2.12-1.12.2.jar force-shading-1.12.2.jar freemarker-2.3.32.jar generex-1.0.2.jar +geronimo-annotation_1.0_spec-1.1.1.jar +geronimo-jaspic_1.0_spec-1.0.jar +geronimo-jta_1.1_spec-1.1.1.jar +grizzled-slf4j_2.11-1.3.2.jar grizzled-slf4j_2.12-1.3.2.jar gson-2.8.9.jar guava-32.0.0-jre.jar +guava-33.2.1-jre.jar guava-retrying-2.0.0.jar guice-3.0.jar guice-4.0.jar guice-4.1.0.jar guice-4.2.2.jar guice-assistedinject-3.0.jar +guice-servlet-3.0.jar guice-servlet-4.0.jar +hadoop-aliyun-2.9.1.jar hadoop-aliyun-3.3.4.jar hadoop-annotations-2.10.2.jar hadoop-annotations-2.7.2.jar hadoop-annotations-2.8.5.jar hadoop-annotations-3.3.4.jar +hadoop-auth-2.7.2.jar hadoop-auth-3.3.4.jar +hadoop-client-2.7.2.jar hadoop-client-3.3.4.jar hadoop-common-2.7.2.jar hadoop-common-3.3.4.jar +hadoop-distcp-2.10.2.jar +hadoop-distcp-2.8.5.jar hadoop-distcp-3.3.4.jar hadoop-hdfs-2.7.2.jar hadoop-hdfs-client-3.3.4.jar +hadoop-mapreduce-client-app-2.7.2.jar hadoop-mapreduce-client-common-2.5.1.jar +hadoop-mapreduce-client-common-2.7.2.jar hadoop-mapreduce-client-common-2.7.4.jar hadoop-mapreduce-client-common-3.3.4.jar +hadoop-mapreduce-client-core-2.7.2.jar hadoop-mapreduce-client-core-3.3.4.jar hadoop-mapreduce-client-jobclient-2.5.1.jar +hadoop-mapreduce-client-jobclient-2.7.2.jar hadoop-mapreduce-client-jobclient-2.7.4.jar hadoop-mapreduce-client-jobclient-3.3.4.jar hadoop-mapreduce-client-shuffle-2.5.1.jar +hadoop-mapreduce-client-shuffle-2.7.2.jar hadoop-mapreduce-client-shuffle-2.7.4.jar hadoop-registry-3.3.4.jar hadoop-shaded-guava-1.1.1.jar hadoop-shaded-protobuf_3_7-1.1.1.jar +hadoop-yarn-api-2.7.2.jar hadoop-yarn-api-3.3.4.jar +hadoop-yarn-client-2.7.2.jar hadoop-yarn-client-3.3.4.jar +hadoop-yarn-common-2.7.2.jar hadoop-yarn-common-3.3.4.jar +hadoop-yarn-registry-2.7.1.jar hadoop-yarn-registry-3.3.4.jar hadoop-yarn-server-common-2.5.1.jar +hadoop-yarn-server-common-2.7.2.jar hadoop-yarn-server-common-2.7.4.jar hadoop-yarn-server-nodemanager-2.5.1.jar hadoop-yarn-server-nodemanager-2.7.4.jar @@ -268,26 +349,32 @@ hamcrest-2.2.jar hamcrest-core-2.2.jar hazelcast-5.3.5.jar hbase-annotations-1.2.0.jar +hbase-annotations-1.2.1.jar hbase-annotations-1.4.3.jar hbase-asyncfs-2.5.3.jar hbase-client-1.2.0.jar +hbase-client-1.2.1.jar hbase-client-1.4.3.jar hbase-client-2.2.6.jar hbase-client-2.5.3.jar -hbase-common-1.2.0-tests.jar hbase-common-1.2.0.jar -hbase-common-1.4.3-tests.jar +hbase-common-1.2.0-tests.jar +hbase-common-1.2.1.jar +hbase-common-1.2.1-tests.jar hbase-common-1.4.3.jar +hbase-common-1.4.3-tests.jar hbase-common-2.2.6.jar hbase-common-2.5.3.jar -hbase-hadoop-compat-1.2.0.jar -hbase-hadoop-compat-1.4.3.jar -hbase-hadoop-compat-2.2.6.jar -hbase-hadoop-compat-2.5.3.jar hbase-hadoop2-compat-1.2.0.jar +hbase-hadoop2-compat-1.2.1.jar hbase-hadoop2-compat-1.4.3.jar hbase-hadoop2-compat-2.2.6.jar hbase-hadoop2-compat-2.5.3.jar +hbase-hadoop-compat-1.2.0.jar +hbase-hadoop-compat-1.2.1.jar +hbase-hadoop-compat-1.4.3.jar +hbase-hadoop-compat-2.2.6.jar +hbase-hadoop-compat-2.5.3.jar hbase-http-2.2.6.jar hbase-http-2.5.3.jar hbase-logging-2.5.3.jar @@ -298,12 +385,15 @@ hbase-metrics-api-1.4.3.jar hbase-metrics-api-2.2.6.jar hbase-metrics-api-2.5.3.jar hbase-prefix-tree-1.2.0.jar +hbase-prefix-tree-1.2.1.jar hbase-prefix-tree-1.4.3.jar hbase-procedure-1.2.0.jar +hbase-procedure-1.2.1.jar hbase-procedure-1.4.3.jar hbase-procedure-2.2.6.jar hbase-procedure-2.5.3.jar hbase-protocol-1.2.0.jar +hbase-protocol-1.2.1.jar hbase-protocol-1.4.3.jar hbase-protocol-2.2.6.jar hbase-protocol-2.5.3.jar @@ -312,6 +402,7 @@ hbase-protocol-shaded-2.5.3.jar hbase-replication-2.2.6.jar hbase-replication-2.5.3.jar hbase-server-1.2.0.jar +hbase-server-1.2.1.jar hbase-server-1.4.3.jar hbase-server-2.2.6.jar hbase-server-2.5.3.jar @@ -327,25 +418,42 @@ hbase-shaded-protobuf-4.1.4.jar hbase-unsafe-4.1.4.jar hbase-zookeeper-2.2.6.jar hbase-zookeeper-2.5.3.jar +HdrHistogram-2.1.12.jar +HdrHistogram-2.1.9.jar hibernate-validator-5.1.2.Final.jar hibernate-validator-6.2.0.Final.jar hibernate-validator-6.2.5.Final.jar +HikariCP-4.0.3.jar hive-classification-3.1.3.jar +hive-common-2.3.3.jar hive-common-3.1.3.jar +hive-exec-2.3.3.jar hive-exec-3.1.3.jar +hive-jdbc-2.3.3.jar hive-jdbc-3.1.3.jar +hive-llap-client-2.3.3.jar hive-llap-client-3.1.3.jar -hive-llap-common-3.1.3-tests.jar +hive-llap-common-2.3.3.jar +hive-llap-common-2.3.3-tests.jar hive-llap-common-3.1.3.jar +hive-llap-common-3.1.3-tests.jar +hive-llap-server-2.3.3.jar hive-llap-server-3.1.3.jar +hive-llap-tez-2.3.3.jar hive-llap-tez-3.1.3.jar +hive-metastore-2.3.3.jar hive-metastore-3.1.3.jar +hive-serde-2.3.3.jar hive-serde-3.1.3.jar +hive-service-2.3.3.jar hive-service-3.1.3.jar +hive-service-rpc-2.3.3.jar hive-service-rpc-3.1.3.jar hive-standalone-metastore-3.1.3.jar +hive-storage-api-2.4.0.jar hive-storage-api-2.7.0.jar hive-upgrade-acid-3.1.3.jar +hive-vector-code-gen-2.3.3.jar hive-vector-code-gen-3.1.3.jar hk2-api-2.4.0-b34.jar hk2-locator-2.4.0-b34.jar @@ -361,32 +469,51 @@ httpcore-nio-4.4.16.jar httpmime-4.5.13.jar httpmime-4.5.14.jar icu4j-67.1.jar +impala-frontend-3.4.0.7.2.15.0-147.jar +impala-minimal-hive-exec-3.4.0.7.2.15.0-147.jar ini4j-0.5.4.jar ion-java-1.0.2.jar istack-commons-runtime-3.0.12.jar ivy-2.4.0.jar j2objc-annotations-2.8.jar +j2objc-annotations-3.0.0.jar jackson-annotations-2.13.4.jar +jackson-annotations-2.15.0.jar jackson-core-2.13.4.jar +jackson-core-2.15.0.jar jackson-core-asl-1.9.13.jar jackson-databind-2.13.4.2.jar +jackson-databind-2.15.0.jar jackson-dataformat-cbor-2.13.4.jar +jackson-dataformat-cbor-2.15.0.jar jackson-dataformat-csv-2.13.4.jar +jackson-dataformat-csv-2.15.0.jar jackson-dataformat-properties-2.13.4.jar +jackson-dataformat-properties-2.15.0.jar jackson-dataformat-smile-2.13.4.jar +jackson-dataformat-smile-2.15.0.jar jackson-dataformat-xml-2.13.4.jar +jackson-dataformat-xml-2.15.0.jar jackson-dataformat-yaml-2.13.4.jar +jackson-dataformat-yaml-2.15.0.jar jackson-datatype-guava-2.13.4.jar +jackson-datatype-guava-2.15.0.jar jackson-datatype-jdk8-2.13.4.jar +jackson-datatype-jdk8-2.15.0.jar jackson-datatype-joda-2.13.4.jar +jackson-datatype-joda-2.15.0.jar jackson-datatype-jsr310-2.13.4.jar +jackson-datatype-jsr310-2.15.0.jar jackson-jaxrs-1.9.13.jar jackson-jaxrs-base-2.13.4.jar jackson-jaxrs-json-provider-2.13.4.jar jackson-mapper-asl-1.9.13.jar jackson-module-jaxb-annotations-2.13.4.jar jackson-module-parameter-names-2.13.4.jar +jackson-module-parameter-names-2.15.0.jar +jackson-module-scala_2.11-2.15.0.jar jackson-module-scala_2.12-2.13.4.jar +jackson-xc-1.9.13.jar jackson-xc-1.9.2.jar jakarta.activation-1.2.2.jar jakarta.activation-api-1.2.2.jar @@ -402,15 +529,18 @@ janino-3.1.9.jar jasper-compiler-5.5.23.jar jasper-runtime-5.5.23.jar javassist-3.27.0-GA.jar -javax-websocket-client-impl-9.4.56.v20240826.jar -javax-websocket-server-impl-9.4.56.v20240826.jar javax.activation-api-1.2.0.jar javax.annotation-api-1.3.2.jar javax.inject-1.jar javax.inject-2.4.0-b34.jar javax.jdo-3.2.0-m3.jar +javax.servlet-3.0.0.v201112011016.jar javax.servlet-api-4.0.1.jar javax.servlet.jsp-api-2.3.1.jar +javax-websocket-client-impl-9.4.56.v20240826.jar +javax-websocket-client-impl-9.4.57.v20241219.jar +javax-websocket-server-impl-9.4.56.v20240826.jar +javax-websocket-server-impl-9.4.57.v20241219.jar javax.ws.rs-api-2.0.1.jar javax.ws.rs-api-2.1.1.jar javolution-5.5.1.jar @@ -421,11 +551,15 @@ jboss-logging-3.4.3.Final.jar jcip-annotations-1.0-1.jar jcl-over-slf4j-1.7.36.jar jcodings-1.0.56.jar +jcodings-1.0.8.jar +jcommander-1.30.jar jcommander-1.81.jar +jctools-core-4.0.5.jar jdbi3-core-3.4.0.jar jdbi3-sqlobject-3.4.0.jar jdo-api-3.0.1.jar jdom2-2.0.6.1.jar +jedis-2.9.2.jar jedis-3.9.0.jar jersey-apache-client4-1.19.4.jar jersey-client-1.19.4.jar @@ -441,36 +575,57 @@ jersey-media-jaxb-2.23.1.jar jersey-server-1.19.4.jar jersey-server-2.23.1.jar jersey-servlet-1.19.4.jar +jettison-1.1.jar +jettison-1.4.0.jar +jettison-1.5.1.jar jettison-1.5.4.jar +jetty-all-7.6.0.v20120127.jar jetty-annotations-9.4.56.v20240826.jar +jetty-annotations-9.4.57.v20241219.jar jetty-client-9.4.56.v20240826.jar +jetty-client-9.4.57.v20241219.jar jetty-continuation-9.4.56.v20240826.jar +jetty-continuation-9.4.57.v20241219.jar jetty-http-9.4.56.v20240826.jar +jetty-http-9.4.57.v20241219.jar jetty-io-9.4.56.v20240826.jar +jetty-io-9.4.57.v20241219.jar jetty-plus-9.4.56.v20240826.jar +jetty-plus-9.4.57.v20241219.jar jetty-rewrite-9.4.56.v20240826.jar jetty-security-9.4.56.v20240826.jar +jetty-security-9.4.57.v20241219.jar jetty-server-9.4.56.v20240826.jar +jetty-server-9.4.57.v20241219.jar jetty-servlet-9.4.56.v20240826.jar +jetty-servlet-9.4.57.v20241219.jar jetty-servlets-9.4.56.v20240826.jar +jetty-servlets-9.4.57.v20241219.jar jetty-sslengine-6.1.26.jar jetty-util-6.1.26.jar jetty-util-9.4.56.v20240826.jar +jetty-util-9.4.57.v20241219.jar jetty-util-ajax-9.4.56.v20240826.jar +jetty-util-ajax-9.4.57.v20241219.jar jetty-webapp-9.4.56.v20240826.jar +jetty-webapp-9.4.57.v20241219.jar jetty-xml-9.4.56.v20240826.jar +jetty-xml-9.4.57.v20241219.jar jline-2.14.6.jar jline-3.9.0.jar jmespath-java-1.12.261.jar jmxutils-1.19.jar jna-5.12.1.jar jna-platform-5.12.1.jar +jna-platform-5.6.0.jar joda-time-2.10.10.jar joda-time-2.10.5.jar joda-time-2.3.jar joda-time-2.8.1.jar +joda-time-2.9.3.jar joda-time-2.9.9.jar jol-core-0.2.jar +joni-2.1.2.jar joni-2.1.42.jar jpam-1.1.jar jruby-complete-9.3.4.0.jar @@ -479,6 +634,7 @@ json-0.193.jar json-1.8.jar json-path-2.9.0.jar json-smart-2.4.10.jar +json-smart-2.4.11.jar jsp-2.1-6.1.14.jar jsp-api-2.1-6.1.14.jar jsp-api-2.1.jar @@ -493,7 +649,6 @@ jul-to-slf4j-1.7.36.jar junit-4.13.2.jar kafka-clients-2.7.2.jar kafka-clients-3.1.2.jar -kafka-clients-3.7.1.jar kafka-clients-3.9.1.jar kerb-admin-1.0.1.jar kerb-client-1.0.1.jar @@ -509,11 +664,17 @@ kerby-config-1.0.1.jar kerby-pkix-1.0.1.jar kerby-util-1.0.1.jar kerby-xdr-1.0.1.jar +knife4j-annotations-2.0.9.jar knife4j-annotations-3.0.3.jar +knife4j-core-2.0.9.jar knife4j-core-3.0.3.jar +knife4j-spring-2.0.9.jar knife4j-spring-3.0.3.jar +knife4j-spring-boot-autoconfigure-2.0.9.jar knife4j-spring-boot-autoconfigure-3.0.3.jar +knife4j-spring-boot-starter-2.0.9.jar knife4j-spring-boot-starter-3.0.3.jar +knife4j-spring-ui-2.0.9.jar knife4j-spring-ui-3.0.3.jar kotlin-stdlib-1.6.21.jar kotlin-stdlib-common-1.6.21.jar @@ -541,25 +702,31 @@ kubernetes-model-policy-5.10.2.jar kubernetes-model-rbac-5.10.2.jar kubernetes-model-scheduling-5.10.2.jar kubernetes-model-storageclass-5.10.2.jar +lang-tag-1.7.jar +LatencyUtils-2.0.3.jar leveldbjni-all-1.8.jar libfb303-0.9.3.jar +libthrift-0.15.0.jar libthrift-0.15.0.pom libthrift-0.9.3.jar listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar log-0.191.jar log-0.193.jar -log4j-1.2-api-2.17.2.jar log4j-1.2.17.jar +log4j-1.2-api-2.17.2.jar log4j-api-2.17.2.jar log4j-core-2.17.2.jar log4j-jcl-2.17.2.jar log4j-jul-2.17.2.jar log4j-slf4j-impl-2.17.2.jar log4j-web-2.17.2.jar +logback-classic-1.2.12.jar +logback-core-1.2.12.jar logging-interceptor-4.9.3.jar lz4-java-1.6.0.jar lz4-java-1.7.1.jar lz4-java-1.8.0.jar +mail-1.4.1.jar mapstruct-1.3.1.Final.jar mchange-commons-java-0.2.15.jar memory-0.9.0.jar @@ -568,69 +735,146 @@ metrics-core-4.2.18.jar metrics-json-4.2.18.jar metrics-jvm-4.2.18.jar micrometer-core-1.9.10.jar +micrometer-core-1.9.11.jar micrometer-registry-prometheus-1.9.10.jar +micrometer-registry-prometheus-1.9.11.jar minlog-1.3.0.jar mongo-java-driver-3.12.8.jar +msal4j-1.15.1.jar +msal4j-persistence-extension-1.3.0.jar +mybatis-3.5.16.jar mybatis-3.5.7.jar mybatis-plus-3.4.3.4.jar +mybatis-plus-3.5.7.jar mybatis-plus-annotation-3.4.3.4.jar +mybatis-plus-annotation-3.5.7.jar mybatis-plus-boot-starter-3.4.3.4.jar +mybatis-plus-boot-starter-3.5.7.jar mybatis-plus-core-3.4.3.4.jar +mybatis-plus-core-3.5.7.jar mybatis-plus-extension-3.4.3.4.jar +mybatis-plus-extension-3.5.7.jar +mybatis-plus-spring-boot-autoconfigure-3.5.7.jar mybatis-spring-2.0.6.jar +mybatis-spring-2.1.2.jar nekohtml-1.9.22.jar netflix-eventbus-0.3.0.jar netflix-infix-0.3.0.jar netty-3.10.6.Final.jar +netty-3.6.2.Final.jar netty-all-4.1.86.Final.jar +netty-all-4.2.7.Final.jar netty-buffer-4.1.86.Final.jar +netty-buffer-4.2.7.Final.jar netty-codec-4.1.86.Final.jar +netty-codec-4.2.7.Final.jar +netty-codec-base-4.2.7.Final.jar +netty-codec-classes-quic-4.2.7.Final.jar +netty-codec-compression-4.2.7.Final.jar netty-codec-dns-4.1.86.Final.jar +netty-codec-dns-4.2.7.Final.jar netty-codec-haproxy-4.1.86.Final.jar -netty-codec-http-4.1.86.Final.jar +netty-codec-haproxy-4.2.7.Final.jar netty-codec-http2-4.1.86.Final.jar +netty-codec-http2-4.2.7.Final.jar +netty-codec-http3-4.2.7.Final.jar +netty-codec-http-4.1.86.Final.jar +netty-codec-http-4.2.7.Final.jar +netty-codec-marshalling-4.2.7.Final.jar netty-codec-memcache-4.1.86.Final.jar +netty-codec-memcache-4.2.7.Final.jar netty-codec-mqtt-4.1.86.Final.jar +netty-codec-mqtt-4.2.7.Final.jar +netty-codec-native-quic-4.2.7.Final-linux-aarch_64.jar +netty-codec-native-quic-4.2.7.Final-linux-x86_64.jar +netty-codec-native-quic-4.2.7.Final-osx-aarch_64.jar +netty-codec-native-quic-4.2.7.Final-osx-x86_64.jar +netty-codec-native-quic-4.2.7.Final-windows-x86_64.jar +netty-codec-protobuf-4.2.7.Final.jar netty-codec-redis-4.1.86.Final.jar +netty-codec-redis-4.2.7.Final.jar netty-codec-smtp-4.1.86.Final.jar +netty-codec-smtp-4.2.7.Final.jar netty-codec-socks-4.1.86.Final.jar +netty-codec-socks-4.2.7.Final.jar netty-codec-stomp-4.1.86.Final.jar +netty-codec-stomp-4.2.7.Final.jar netty-codec-xml-4.1.86.Final.jar +netty-codec-xml-4.2.7.Final.jar netty-common-4.1.86.Final.jar +netty-common-4.2.7.Final.jar netty-handler-4.1.86.Final.jar +netty-handler-4.2.7.Final.jar netty-handler-proxy-4.1.86.Final.jar +netty-handler-proxy-4.2.7.Final.jar netty-handler-ssl-ocsp-4.1.86.Final.jar +netty-handler-ssl-ocsp-4.2.7.Final.jar netty-incubator-codec-classes-quic-0.0.39.Final.jar +netty-incubator-codec-classes-quic-0.0.40.Final.jar netty-incubator-codec-native-quic-0.0.39.Final-linux-x86_64.jar +netty-incubator-codec-native-quic-0.0.40.Final-linux-x86_64.jar netty-resolver-4.1.86.Final.jar +netty-resolver-4.2.7.Final.jar netty-resolver-dns-4.1.86.Final.jar +netty-resolver-dns-4.2.7.Final.jar netty-resolver-dns-classes-macos-4.1.86.Final.jar +netty-resolver-dns-classes-macos-4.2.7.Final.jar netty-resolver-dns-native-macos-4.1.86.Final-osx-aarch_64.jar netty-resolver-dns-native-macos-4.1.86.Final-osx-x86_64.jar +netty-resolver-dns-native-macos-4.2.7.Final-osx-aarch_64.jar +netty-resolver-dns-native-macos-4.2.7.Final-osx-x86_64.jar +netty-tcnative-boringssl-static-2.0.54.Final.jar netty-tcnative-boringssl-static-2.0.54.Final-linux-aarch_64.jar netty-tcnative-boringssl-static-2.0.54.Final-linux-x86_64.jar netty-tcnative-boringssl-static-2.0.54.Final-osx-aarch_64.jar netty-tcnative-boringssl-static-2.0.54.Final-osx-x86_64.jar netty-tcnative-boringssl-static-2.0.54.Final-windows-x86_64.jar -netty-tcnative-boringssl-static-2.0.54.Final.jar +netty-tcnative-boringssl-static-2.0.74.Final.jar +netty-tcnative-boringssl-static-2.0.74.Final-linux-aarch_64.jar +netty-tcnative-boringssl-static-2.0.74.Final-linux-x86_64.jar +netty-tcnative-boringssl-static-2.0.74.Final-osx-aarch_64.jar +netty-tcnative-boringssl-static-2.0.74.Final-osx-x86_64.jar +netty-tcnative-boringssl-static-2.0.74.Final-windows-x86_64.jar netty-tcnative-classes-2.0.54.Final.jar +netty-tcnative-classes-2.0.74.Final.jar netty-transport-4.1.86.Final.jar +netty-transport-4.2.7.Final.jar netty-transport-classes-epoll-4.1.86.Final.jar +netty-transport-classes-epoll-4.2.7.Final.jar +netty-transport-classes-io_uring-4.2.7.Final.jar netty-transport-classes-kqueue-4.1.86.Final.jar +netty-transport-classes-kqueue-4.2.7.Final.jar netty-transport-native-epoll-4.1.86.Final-linux-aarch_64.jar netty-transport-native-epoll-4.1.86.Final-linux-x86_64.jar +netty-transport-native-epoll-4.2.7.Final-linux-aarch_64.jar +netty-transport-native-epoll-4.2.7.Final-linux-riscv64.jar +netty-transport-native-epoll-4.2.7.Final-linux-x86_64.jar +netty-transport-native-io_uring-4.2.7.Final-linux-aarch_64.jar +netty-transport-native-io_uring-4.2.7.Final-linux-riscv64.jar +netty-transport-native-io_uring-4.2.7.Final-linux-x86_64.jar netty-transport-native-kqueue-4.1.86.Final-osx-aarch_64.jar netty-transport-native-kqueue-4.1.86.Final-osx-x86_64.jar +netty-transport-native-kqueue-4.2.7.Final-osx-aarch_64.jar +netty-transport-native-kqueue-4.2.7.Final-osx-x86_64.jar netty-transport-native-unix-common-4.1.86.Final.jar +netty-transport-native-unix-common-4.2.7.Final.jar netty-transport-rxtx-4.1.86.Final.jar +netty-transport-rxtx-4.2.7.Final.jar netty-transport-sctp-4.1.86.Final.jar +netty-transport-sctp-4.2.7.Final.jar netty-transport-udt-4.1.86.Final.jar +netty-transport-udt-4.2.7.Final.jar +nimbus-jose-jwt-9.37.3.jar nimbus-jose-jwt-9.8.1.jar +oauth2-oidc-sdk-11.9.1.jar objenesis-2.1.jar objenesis-3.2.jar +ognl-3.3.4.jar okhttp-4.9.3.jar okhttp-urlconnection-4.9.3.jar okio-2.8.0.jar +okio-3.4.0.jar +okio-jvm-3.4.0.jar opencsv-2.3.jar opentelemetry-api-1.15.0.jar opentelemetry-context-1.15.0.jar @@ -638,17 +882,23 @@ opentelemetry-semconv-1.15.0-alpha.jar opentracing-api-0.33.0.jar opentracing-noop-0.33.0.jar opentracing-util-0.33.0.jar +orc-core-1.3.3.jar orc-core-1.5.8.jar orc-shims-1.5.8.jar org.jacoco.agent-0.8.5-runtime.jar osgi-resource-locator-1.0.1.jar oshi-core-6.2.1.jar +oshi-core-6.4.0.jar pagehelper-5.3.1.jar paranamer-2.3.jar paranamer-2.8.jar parquet-hadoop-bundle-1.10.0.jar +parquet-hadoop-bundle-1.8.1.jar +poi-5.2.3.jar poi-5.3.0.jar +poi-ooxml-5.2.3.jar poi-ooxml-5.3.0.jar +poi-ooxml-lite-5.2.3.jar poi-ooxml-lite-5.3.0.jar presto-client-0.234.jar presto-client-1.5.0.jar @@ -667,19 +917,33 @@ reactor-core-3.4.29.jar reactor-extra-3.4.10.jar reactor-netty-0.9.7.RELEASE.jar reactor-netty-1.0.31.jar +reactor-netty-1.0.32.jar reactor-netty-core-1.0.31.jar +reactor-netty-core-1.0.32.jar reactor-netty-http-1.0.31.jar +reactor-netty-http-1.0.32.jar +reactor-netty-http-1.0.39.jar reactor-netty-http-brave-1.0.31.jar +reactor-netty-http-brave-1.0.32.jar reactor-netty-incubator-quic-0.0.20.jar +reactor-netty-incubator-quic-0.0.21.jar reflections-0.9.12.jar reload4j-1.2.19.jar +scala-compiler-2.11.12.jar scala-compiler-2.12.17.jar +scala-java8-compat_2.11-0.7.0.jar scala-java8-compat_2.12-0.8.0.jar +scala-library-2.11.12.jar scala-library-2.12.17.jar +scalap-2.11.12.jar +scalap-2.12.17.jar +scala-parser-combinators_2.11-1.0.4.jar scala-parser-combinators_2.12-1.1.1.jar +scala-reflect-2.11.12.jar scala-reflect-2.12.17.jar +scala-xml_2.11-1.0.5.jar scala-xml_2.12-2.1.0.jar -scalap-2.12.17.jar +scopt_2.11-3.5.0.jar scopt_2.12-3.5.0.jar seatunnel-api-2.3.1.jar seatunnel-common-2.3.1.jar @@ -707,38 +971,67 @@ sketches-core-0.9.0.jar slf4j-api-1.7.30.jar slf4j-reload4j-1.7.36.jar slice-0.38.jar +slider-core-0.90.2-incubating.jar +snakeyaml-1.30.jar snakeyaml-2.0.jar -snappy-java-1.1.10.5.jar +snappy-java-1.0.4.1.jar +snappy-java-1.0.5.jar +snappy-java-1.1.10.4.jar snappy-java-1.1.4.jar snappy-java-1.1.7.7.jar snappy-java-1.1.8.2.jar snappy-java-1.1.8.3.jar +spark-measure_2.11-0.17.jar +spark-measure_2.12-0.24.jar +spark-redis_2.11-2.6.0.jar spark-redis_2.12-2.6.0.jar +SparseBitSet-1.2.jar +SparseBitSet-1.3.jar spring-aop-5.3.27.jar +spring-aop-5.3.34.jar spring-beans-5.3.27.jar +spring-beans-5.3.34.jar spring-boot-2.7.11.jar +spring-boot-2.7.12.jar spring-boot-actuator-2.7.11.jar +spring-boot-actuator-2.7.12.jar spring-boot-actuator-autoconfigure-2.7.11.jar +spring-boot-actuator-autoconfigure-2.7.12.jar spring-boot-admin-server-2.7.16.jar spring-boot-admin-server-cloud-2.7.16.jar spring-boot-admin-server-ui-2.7.16.jar spring-boot-admin-starter-server-2.7.16.jar spring-boot-autoconfigure-2.7.11.jar +spring-boot-autoconfigure-2.7.12.jar spring-boot-starter-2.7.11.jar +spring-boot-starter-2.7.12.jar spring-boot-starter-actuator-2.7.11.jar +spring-boot-starter-actuator-2.7.12.jar spring-boot-starter-aop-2.7.11.jar +spring-boot-starter-aop-2.7.12.jar spring-boot-starter-cache-2.7.11.jar +spring-boot-starter-cache-2.7.12.jar spring-boot-starter-freemarker-2.7.11.jar +spring-boot-starter-freemarker-2.7.12.jar spring-boot-starter-jdbc-2.7.11.jar spring-boot-starter-jetty-2.7.11.jar +spring-boot-starter-jetty-2.7.12.jar spring-boot-starter-json-2.7.11.jar +spring-boot-starter-json-2.7.12.jar spring-boot-starter-log4j2-2.7.11.jar +spring-boot-starter-log4j2-2.7.12.jar spring-boot-starter-quartz-2.7.11.jar +spring-boot-starter-quartz-2.7.12.jar spring-boot-starter-reactor-netty-2.7.11.jar +spring-boot-starter-reactor-netty-2.7.12.jar spring-boot-starter-thymeleaf-2.7.11.jar +spring-boot-starter-thymeleaf-2.7.12.jar spring-boot-starter-validation-2.7.11.jar +spring-boot-starter-validation-2.7.12.jar spring-boot-starter-web-2.7.11.jar +spring-boot-starter-web-2.7.12.jar spring-boot-starter-webflux-2.7.11.jar +spring-boot-starter-webflux-2.7.12.jar spring-cloud-commons-3.1.7.jar spring-cloud-context-3.1.7.jar spring-cloud-gateway-server-3.1.8.jar @@ -753,35 +1046,57 @@ spring-cloud-starter-netflix-eureka-client-3.1.7.jar spring-cloud-starter-netflix-eureka-server-3.1.7.jar spring-cloud-starter-openfeign-3.1.8.jar spring-context-5.3.27.jar +spring-context-5.3.34.jar spring-context-support-5.3.27.jar +spring-context-support-5.3.34.jar spring-core-5.3.27.jar +spring-core-5.3.34.jar spring-expression-5.3.27.jar -spring-jcl-5.3.27.jar -spring-jdbc-5.3.27.jar -spring-plugin-core-2.0.0.RELEASE.jar -spring-plugin-metadata-2.0.0.RELEASE.jar -spring-security-crypto-5.7.8.jar -spring-security-rsa-1.0.11.RELEASE.jar -spring-tx-5.3.27.jar -spring-web-5.3.27.jar -spring-webflux-5.3.27.jar -spring-webmvc-5.3.27.jar -spring-retry-1.3.4.jar +spring-expression-5.3.34.jar +springfox-bean-validators-2.10.5.jar springfox-bean-validators-3.0.0.jar springfox-boot-starter-3.0.0.jar +springfox-core-2.10.5.jar springfox-core-3.0.0.jar springfox-data-rest-3.0.0.jar springfox-oas-3.0.0.jar +springfox-schema-2.10.5.jar springfox-schema-3.0.0.jar +springfox-spi-2.10.5.jar springfox-spi-3.0.0.jar +springfox-spring-web-2.10.5.jar springfox-spring-web-3.0.0.jar +springfox-spring-webflux-2.10.5.jar springfox-spring-webflux-3.0.0.jar +springfox-spring-webmvc-2.10.5.jar springfox-spring-webmvc-3.0.0.jar -springfox-swagger-common-3.0.0.jar +springfox-swagger2-2.10.5.jar springfox-swagger2-3.0.0.jar +springfox-swagger-common-2.10.5.jar +springfox-swagger-common-3.0.0.jar +spring-jcl-5.3.27.jar +spring-jcl-5.3.34.jar +spring-jdbc-5.3.27.jar +spring-jdbc-5.3.34.jar +spring-plugin-core-2.0.0.RELEASE.jar +spring-plugin-metadata-2.0.0.RELEASE.jar +spring-retry-1.3.4.jar +spring-security-crypto-5.7.11.jar +spring-security-crypto-5.7.8.jar +spring-security-rsa-1.0.11.RELEASE.jar +spring-tx-5.3.27.jar +spring-tx-5.3.34.jar +spring-web-5.3.27.jar +spring-web-5.3.34.jar +spring-webflux-5.3.27.jar +spring-webflux-5.3.34.jar +spring-webmvc-5.3.27.jar +spring-webmvc-5.3.34.jar sqlline-1.3.0.jar sqoop-1.4.6-hadoop200.jar +ssl-config-core_2.11-0.3.7.jar ssl-config-core_2.12-0.3.7.jar +ST4-4.0.4.jar stats-0.191.jar stax2-api-4.2.1.jar stringtemplate-3.2.1.jar @@ -794,10 +1109,13 @@ tephra-api-0.6.0.jar tephra-core-0.6.0.jar tephra-hbase-compat-1.0-0.6.0.jar thymeleaf-3.0.15.RELEASE.jar +thymeleaf-3.1.2.RELEASE.jar thymeleaf-extras-java8time-3.0.4.RELEASE.jar thymeleaf-spring5-3.0.15.RELEASE.jar +thymeleaf-spring5-3.1.2.RELEASE.jar token-provider-1.0.1.jar tomcat-embed-el-9.0.74.jar +tomcat-embed-el-9.0.75.jar transaction-api-1.1.jar trino-client-371.jar twill-api-0.6.0-incubating.jar @@ -812,15 +1130,22 @@ units-1.3.jar units-1.6.jar validation-api-2.0.1.Final.jar websocket-api-9.4.56.v20240826.jar +websocket-api-9.4.57.v20241219.jar websocket-client-9.4.56.v20240826.jar +websocket-client-9.4.57.v20241219.jar websocket-common-9.4.56.v20240826.jar +websocket-common-9.4.57.v20241219.jar websocket-server-9.4.56.v20240826.jar +websocket-server-9.4.57.v20241219.jar websocket-servlet-9.4.56.v20240826.jar +websocket-servlet-9.4.57.v20241219.jar +woodstox-core-6.2.1.jar woodstox-core-6.4.0.jar xalan-2.7.0.jar xercesImpl-2.9.1.jar xml-apis-1.4.01.jar xml-apis-ext-1.3.04.jar +xmlbeans-5.1.1.jar xmlbeans-5.2.1.jar xmlenc-0.52.jar xom-1.2.5.jar @@ -831,28 +1156,10 @@ zipkin-2.23.2.jar zipkin-reporter-2.16.3.jar zipkin-reporter-brave-2.16.3.jar zjsonpatch-0.3.0.jar +zookeeper-3.8.4.jar zookeeper-3.9.2.jar -zookeeper-3.9.4.jar +zookeeper-jute-3.8.4.jar zookeeper-jute-3.9.2.jar -zookeeper-jute-3.9.4.jar zstd-jni-1.4.5-6.jar zstd-jni-1.5.0-4.jar zstd-jni-1.5.6-4.jar -spark-measure_2.12-0.24.jar -azure-core-1.54.1.jar -azure-core-http-netty-1.15.7.jar -azure-identity-1.14.2.jar -azure-json-1.3.0.jar -azure-storage-blob-12.29.0.jar -azure-storage-common-12.28.0.jar -azure-storage-internal-avro-12.14.0.jar -azure-xml-1.1.0.jar -content-type-2.3.jar -jna-platform-5.13.0.jar -lang-tag-1.7.jar -msal4j-1.17.2.jar -msal4j-persistence-extension-1.3.0.jar -oauth2-oidc-sdk-11.18.jar -okio-3.4.0.jar -okio-jvm-3.4.0.jar -oshi-core-6.4.0.jar