Add Support for Custom pg_stat_statements View in Postgres Lineage Ingestion #25104
Conversation
…views Co-authored-by: SumanMaharana <59608519+SumanMaharana@users.noreply.github.com>
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
|
TypeScript types have been updated based on the JSON schema changes in the PR |
harshach
added
the
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
| SELECT | ||
| u.usename, | ||
| d.datname database_name, | ||
| s.query query_text, |
There was a problem hiding this comment.
🚨 Security: SQL injection via unvalidated queryStatementSource parameter
Details
The queryStatementSource configuration value is directly interpolated into SQL queries using Python string formatting ({query_statement_source}) without any validation or sanitization. A malicious user could provide a value like pg_stat_statements; DROP TABLE users; -- which would be directly inserted into the SQL query, potentially allowing arbitrary SQL execution.
Impact: An attacker with access to configuration could execute arbitrary SQL commands against the database, leading to data theft, corruption, or complete database compromise.
Suggested fix: Add input validation to ensure queryStatementSource only contains valid identifier characters (alphanumeric, underscores, dots for schema qualification). Consider using a regex pattern like ^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)*$ in the JSON schema validation:
"queryStatementSource": {
"title": "Query Statement Source",
"description": "...",
"type": "string",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*(\\.[a-zA-Z_][a-zA-Z0-9_]*)*$",
"default": "pg_stat_statements"
}Additionally, consider using proper identifier quoting in Python code as a defense-in-depth measure.
🛡️ TRIVY SCAN RESULT 🛡️ Target:
|
| Package | Vulnerability ID | Severity | Installed Version | Fixed Version |
|---|---|---|---|---|
com.fasterxml.jackson.core:jackson-core |
CVE-2025-52999 | 🚨 HIGH | 2.12.7 | 2.15.0 |
com.fasterxml.jackson.core:jackson-core |
CVE-2025-52999 | 🚨 HIGH | 2.13.4 | 2.15.0 |
com.fasterxml.jackson.core:jackson-databind |
CVE-2022-42003 | 🚨 HIGH | 2.12.7 | 2.12.7.1, 2.13.4.2 |
com.fasterxml.jackson.core:jackson-databind |
CVE-2022-42004 | 🚨 HIGH | 2.12.7 | 2.12.7.1, 2.13.4 |
com.google.code.gson:gson |
CVE-2022-25647 | 🚨 HIGH | 2.2.4 | 2.8.9 |
com.google.protobuf:protobuf-java |
CVE-2021-22569 | 🚨 HIGH | 3.3.0 | 3.16.1, 3.18.2, 3.19.2 |
com.google.protobuf:protobuf-java |
CVE-2022-3509 | 🚨 HIGH | 3.3.0 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2022-3510 | 🚨 HIGH | 3.3.0 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2024-7254 | 🚨 HIGH | 3.3.0 | 3.25.5, 4.27.5, 4.28.2 |
com.google.protobuf:protobuf-java |
CVE-2021-22569 | 🚨 HIGH | 3.7.1 | 3.16.1, 3.18.2, 3.19.2 |
com.google.protobuf:protobuf-java |
CVE-2022-3509 | 🚨 HIGH | 3.7.1 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2022-3510 | 🚨 HIGH | 3.7.1 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2024-7254 | 🚨 HIGH | 3.7.1 | 3.25.5, 4.27.5, 4.28.2 |
com.nimbusds:nimbus-jose-jwt |
CVE-2023-52428 | 🚨 HIGH | 9.8.1 | 9.37.2 |
com.squareup.okhttp3:okhttp |
CVE-2021-0341 | 🚨 HIGH | 3.12.12 | 4.9.2 |
commons-beanutils:commons-beanutils |
CVE-2025-48734 | 🚨 HIGH | 1.9.4 | 1.11.0 |
commons-io:commons-io |
CVE-2024-47554 | 🚨 HIGH | 2.8.0 | 2.14.0 |
dnsjava:dnsjava |
CVE-2024-25638 | 🚨 HIGH | 2.1.7 | 3.6.0 |
io.netty:netty-codec-http2 |
CVE-2025-55163 | 🚨 HIGH | 4.1.96.Final | 4.2.4.Final, 4.1.124.Final |
io.netty:netty-codec-http2 |
GHSA-xpw8-rcwv-8f8p | 🚨 HIGH | 4.1.96.Final | 4.1.100.Final |
io.netty:netty-handler |
CVE-2025-24970 | 🚨 HIGH | 4.1.96.Final | 4.1.118.Final |
net.minidev:json-smart |
CVE-2021-31684 | 🚨 HIGH | 1.3.2 | 1.3.3, 2.4.4 |
net.minidev:json-smart |
CVE-2023-1370 | 🚨 HIGH | 1.3.2 | 2.4.9 |
org.apache.avro:avro |
CVE-2024-47561 | 🔥 CRITICAL | 1.7.7 | 1.11.4 |
org.apache.avro:avro |
CVE-2023-39410 | 🚨 HIGH | 1.7.7 | 1.11.3 |
org.apache.derby:derby |
CVE-2022-46337 | 🔥 CRITICAL | 10.14.2.0 | 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0 |
org.apache.ivy:ivy |
CVE-2022-46751 | 🚨 HIGH | 2.5.1 | 2.5.2 |
org.apache.mesos:mesos |
CVE-2018-1330 | 🚨 HIGH | 1.4.3 | 1.6.0 |
org.apache.thrift:libthrift |
CVE-2019-0205 | 🚨 HIGH | 0.12.0 | 0.13.0 |
org.apache.thrift:libthrift |
CVE-2020-13949 | 🚨 HIGH | 0.12.0 | 0.14.0 |
org.apache.zookeeper:zookeeper |
CVE-2023-44981 | 🔥 CRITICAL | 3.6.3 | 3.7.2, 3.8.3, 3.9.1 |
org.eclipse.jetty:jetty-server |
CVE-2024-13009 | 🚨 HIGH | 9.4.56.v20240826 | 9.4.57.v20241219 |
org.lz4:lz4-java |
CVE-2025-12183 | 🚨 HIGH | 1.8.0 | 1.8.1 |
🛡️ TRIVY SCAN RESULT 🛡️
Target: Node.js
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: Python
Vulnerabilities (4)
| Package | Vulnerability ID | Severity | Installed Version | Fixed Version |
|---|---|---|---|---|
starlette |
CVE-2025-62727 | 🚨 HIGH | 0.48.0 | 0.49.1 |
urllib3 |
CVE-2025-66418 | 🚨 HIGH | 1.26.20 | 2.6.0 |
urllib3 |
CVE-2025-66471 | 🚨 HIGH | 1.26.20 | 2.6.0 |
urllib3 |
CVE-2026-21441 | 🚨 HIGH | 1.26.20 | 2.6.3 |
🛡️ TRIVY SCAN RESULT 🛡️
Target: /etc/ssl/private/ssl-cert-snakeoil.key
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/extended_sample_data.yaml
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/lineage.yaml
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/sample_data.json
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/sample_data.yaml
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/sample_data_aut.yaml
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/sample_usage.json
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/sample_usage.yaml
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /ingestion/pipelines/sample_usage_aut.yaml
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️ Target:
|
| Package | Vulnerability ID | Severity | Installed Version | Fixed Version |
|---|---|---|---|---|
com.fasterxml.jackson.core:jackson-core |
CVE-2025-52999 | 🚨 HIGH | 2.12.7 | 2.15.0 |
com.fasterxml.jackson.core:jackson-core |
CVE-2025-52999 | 🚨 HIGH | 2.13.4 | 2.15.0 |
com.fasterxml.jackson.core:jackson-databind |
CVE-2022-42003 | 🚨 HIGH | 2.12.7 | 2.12.7.1, 2.13.4.2 |
com.fasterxml.jackson.core:jackson-databind |
CVE-2022-42004 | 🚨 HIGH | 2.12.7 | 2.12.7.1, 2.13.4 |
com.google.code.gson:gson |
CVE-2022-25647 | 🚨 HIGH | 2.2.4 | 2.8.9 |
com.google.protobuf:protobuf-java |
CVE-2021-22569 | 🚨 HIGH | 3.3.0 | 3.16.1, 3.18.2, 3.19.2 |
com.google.protobuf:protobuf-java |
CVE-2022-3509 | 🚨 HIGH | 3.3.0 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2022-3510 | 🚨 HIGH | 3.3.0 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2024-7254 | 🚨 HIGH | 3.3.0 | 3.25.5, 4.27.5, 4.28.2 |
com.google.protobuf:protobuf-java |
CVE-2021-22569 | 🚨 HIGH | 3.7.1 | 3.16.1, 3.18.2, 3.19.2 |
com.google.protobuf:protobuf-java |
CVE-2022-3509 | 🚨 HIGH | 3.7.1 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2022-3510 | 🚨 HIGH | 3.7.1 | 3.16.3, 3.19.6, 3.20.3, 3.21.7 |
com.google.protobuf:protobuf-java |
CVE-2024-7254 | 🚨 HIGH | 3.7.1 | 3.25.5, 4.27.5, 4.28.2 |
com.nimbusds:nimbus-jose-jwt |
CVE-2023-52428 | 🚨 HIGH | 9.8.1 | 9.37.2 |
com.squareup.okhttp3:okhttp |
CVE-2021-0341 | 🚨 HIGH | 3.12.12 | 4.9.2 |
commons-beanutils:commons-beanutils |
CVE-2025-48734 | 🚨 HIGH | 1.9.4 | 1.11.0 |
commons-io:commons-io |
CVE-2024-47554 | 🚨 HIGH | 2.8.0 | 2.14.0 |
dnsjava:dnsjava |
CVE-2024-25638 | 🚨 HIGH | 2.1.7 | 3.6.0 |
io.netty:netty-codec-http2 |
CVE-2025-55163 | 🚨 HIGH | 4.1.96.Final | 4.2.4.Final, 4.1.124.Final |
io.netty:netty-codec-http2 |
GHSA-xpw8-rcwv-8f8p | 🚨 HIGH | 4.1.96.Final | 4.1.100.Final |
io.netty:netty-handler |
CVE-2025-24970 | 🚨 HIGH | 4.1.96.Final | 4.1.118.Final |
net.minidev:json-smart |
CVE-2021-31684 | 🚨 HIGH | 1.3.2 | 1.3.3, 2.4.4 |
net.minidev:json-smart |
CVE-2023-1370 | 🚨 HIGH | 1.3.2 | 2.4.9 |
org.apache.avro:avro |
CVE-2024-47561 | 🔥 CRITICAL | 1.7.7 | 1.11.4 |
org.apache.avro:avro |
CVE-2023-39410 | 🚨 HIGH | 1.7.7 | 1.11.3 |
org.apache.derby:derby |
CVE-2022-46337 | 🔥 CRITICAL | 10.14.2.0 | 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0 |
org.apache.ivy:ivy |
CVE-2022-46751 | 🚨 HIGH | 2.5.1 | 2.5.2 |
org.apache.mesos:mesos |
CVE-2018-1330 | 🚨 HIGH | 1.4.3 | 1.6.0 |
org.apache.thrift:libthrift |
CVE-2019-0205 | 🚨 HIGH | 0.12.0 | 0.13.0 |
org.apache.thrift:libthrift |
CVE-2020-13949 | 🚨 HIGH | 0.12.0 | 0.14.0 |
org.apache.zookeeper:zookeeper |
CVE-2023-44981 | 🔥 CRITICAL | 3.6.3 | 3.7.2, 3.8.3, 3.9.1 |
org.eclipse.jetty:jetty-server |
CVE-2024-13009 | 🚨 HIGH | 9.4.56.v20240826 | 9.4.57.v20241219 |
org.lz4:lz4-java |
CVE-2025-12183 | 🚨 HIGH | 1.8.0 | 1.8.1 |
🛡️ TRIVY SCAN RESULT 🛡️
Target: Node.js
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: Python
Vulnerabilities (9)
| Package | Vulnerability ID | Severity | Installed Version | Fixed Version |
|---|---|---|---|---|
Werkzeug |
CVE-2024-34069 | 🚨 HIGH | 2.2.3 | 3.0.3 |
aiohttp |
CVE-2025-69223 | 🚨 HIGH | 3.12.12 | 3.13.3 |
aiohttp |
CVE-2025-69223 | 🚨 HIGH | 3.13.2 | 3.13.3 |
deepdiff |
CVE-2025-58367 | 🔥 CRITICAL | 7.0.1 | 8.6.1 |
ray |
CVE-2025-62593 | 🔥 CRITICAL | 2.47.1 | 2.52.0 |
starlette |
CVE-2025-62727 | 🚨 HIGH | 0.48.0 | 0.49.1 |
urllib3 |
CVE-2025-66418 | 🚨 HIGH | 1.26.20 | 2.6.0 |
urllib3 |
CVE-2025-66471 | 🚨 HIGH | 1.26.20 | 2.6.0 |
urllib3 |
CVE-2026-21441 | 🚨 HIGH | 1.26.20 | 2.6.3 |
🛡️ TRIVY SCAN RESULT 🛡️
Target: /etc/ssl/private/ssl-cert-snakeoil.key
No Vulnerabilities Found
🛡️ TRIVY SCAN RESULT 🛡️
Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO
No Vulnerabilities Found
Jest test CoverageUI tests summary
|
| u.usename, | ||
| d.datname database_name, | ||
| s.query query_text, | ||
| s.{time_column_name} duration |
There was a problem hiding this comment.
🚨 Security: SQL injection via unvalidated queryStatementSource parameter
Details
The queryStatementSource configuration parameter is directly interpolated into SQL queries without any validation or sanitization. This creates a SQL injection vulnerability where a malicious user could execute arbitrary SQL commands.
Impact:
- A malicious actor with configuration access could inject arbitrary SQL like
pg_stat_statements; DROP TABLE users; --or use UNION-based attacks to extract sensitive data - This could lead to data exfiltration, data modification, or denial of service
Locations affected:
queries.pyline 22-24:{query_statement_source}in POSTGRES_SQL_STATEMENTqueries.pyline 140:{query_statement_source}in POSTGRES_TEST_GET_QUERIESconnection.pyline 9-10: Passing unvalidated value to query formatquery_parser.pyline 44-45: Passing unvalidated value to query format
Suggested fix:
- Add a validation pattern to the JSON schema to restrict input to valid SQL identifiers:
"queryStatementSource": {
"type": "string",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*(\\.[a-zA-Z_][a-zA-Z0-9_]*)?$",
"default": "pg_stat_statements"
}- Additionally, validate/sanitize the value in Python before use:
import re
def validate_query_source(source: str) -> str:
if not re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)?$', source):
raise ValueError(f"Invalid query statement source: {source}")
return sourceThis mirrors security patterns used elsewhere for similar configurable SQL identifiers.
🔍 CI failure analysis for fb07505: Integration tests fail to compile due to applyTag() signature change from main branchIssueBoth integration test jobs ( Root CauseCompilation Error in Integration Tests (unrelated to PR changes) File: Problem: The Error at lines 82, 131, and 194: The method now requires 8 parameters but the test code is passing only 7 parameters (with the 7th being null). DetailsThis is a breaking API change introduced in the main branch, most likely from one of these recent commits merged into this PR:
The "Tagging explanation" commit ( SolutionThe This is completely unrelated to the Postgres Code Review 🚫 BlockedFeature adds configurable query source for Postgres lineage but introduces a SQL injection vulnerability due to unvalidated user input being directly interpolated into SQL queries. 🚨 Security: SQL injection via unvalidated queryStatementSource parameter📄 ingestion/src/metadata/ingestion/source/database/postgres/queries.py:22 📄 ingestion/src/metadata/ingestion/source/database/postgres/queries.py:140 📄 openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/postgresConnection.json:109 🔗 CWE-89: SQL Injection The Attack vector: Impact:
Affected code:
Recommended fix:
"queryStatementSource": {
"type": "string",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*(\\.[a-zA-Z_][a-zA-Z0-9_]*)?$",
"default": "pg_stat_statements"
}
import re
if not re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)?$', query_statement_source):
raise ValueError("Invalid queryStatementSource format")This mirrors security patterns used in other database connectors that accept table/view names as configuration. 🚨 Security: SQL injection via unvalidated queryStatementSource parameter📄 ingestion/src/metadata/ingestion/source/database/postgres/queries.py:22 📄 ingestion/src/metadata/ingestion/source/database/postgres/queries.py:140 📄 openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/postgresConnection.json:109 🔗 CWE-89: SQL Injection The Impact:
Locations affected:
Suggested fix:
"queryStatementSource": {
"type": "string",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*(\\.[a-zA-Z_][a-zA-Z0-9_]*)?$",
"default": "pg_stat_statements"
}
import re
def validate_query_source(source: str) -> str:
if not re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)?$', source):
raise ValueError(f"Invalid query statement source: {source}")
return sourceThis mirrors security patterns used elsewhere for similar configurable SQL identifiers. 🚨 Security: SQL injection via unvalidated queryStatementSource parameter📄 ingestion/src/metadata/ingestion/source/database/postgres/queries.py:22 📄 ingestion/src/metadata/ingestion/source/database/postgres/queries.py:140 📄 openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/postgresConnection.json:109 🔗 CWE-89: SQL Injection The Impact: An attacker with access to configuration could execute arbitrary SQL commands against the database, leading to data theft, corruption, or complete database compromise. Suggested fix: Add input validation to ensure "queryStatementSource": {
"title": "Query Statement Source",
"description": "...",
"type": "string",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*(\\.[a-zA-Z_][a-zA-Z0-9_]*)*$",
"default": "pg_stat_statements"
}Additionally, consider using proper identifier quoting in Python code as a defense-in-depth measure. What Works WellThe implementation follows existing patterns in the codebase (similar to Snowflake connector's accountUsageSchema), includes comprehensive unit tests for both default and custom source scenarios, and provides thorough documentation for users. RecommendationsAdd input validation at both the schema level (JSON Schema pattern constraint) and application level (Python regex validation) to ensure the queryStatementSource only contains valid SQL identifier patterns (e.g., Tip Comment OptionsAuto-apply is off Gitar will not commit updates to this branch. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | This comment will update automatically (Docs) |
| u.usename, | ||
| d.datname database_name, | ||
| s.query query_text, | ||
| s.{time_column_name} duration |
There was a problem hiding this comment.
🚨 Security: SQL injection via unvalidated queryStatementSource parameter
Details
The queryStatementSource parameter is directly interpolated into SQL queries via Python's .format() method without any validation or sanitization. This creates a SQL injection vulnerability.
Attack vector:
A user could provide a malicious value like:
"queryStatementSource": "pg_stat_statements; DROP TABLE users; --"
Impact:
- Remote code execution via SQL injection
- Data exfiltration, modification, or deletion
- Complete database compromise
Affected code:
queries.pyline 22:{query_statement_source} sinPOSTGRES_SQL_STATEMENTqueries.pyline 140:{query_statement_source} sinPOSTGRES_TEST_GET_QUERIES- The JSON schema at
postgresConnection.jsonline 109 defines the field as a plain string with no pattern validation
Recommended fix:
- Add a strict regex pattern in the JSON schema to only allow valid SQL identifiers:
"queryStatementSource": {
"type": "string",
"pattern": "^[a-zA-Z_][a-zA-Z0-9_]*(\\.[a-zA-Z_][a-zA-Z0-9_]*)?$",
"default": "pg_stat_statements"
}- Additionally, add server-side validation in Python to double-check the format before interpolation:
import re
if not re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*(\.[a-zA-Z_][a-zA-Z0-9_]*)?$', query_statement_source):
raise ValueError("Invalid queryStatementSource format")This mirrors security patterns used in other database connectors that accept table/view names as configuration.
|
Describe your changes:
Adds
queryStatementSourceconfiguration property to Postgres and Timescale connectors, allowing users to specify a custom view/table for query logs instead of the defaultpg_stat_statements. This supports deployments that exposepg_stat_statementsthrough a custom view for security policy compliance.Changes:
queryStatementSourceproperty topostgresConnection.jsonandtimescaleConnection.jsonschemas (default:pg_stat_statements)POSTGRES_SQL_STATEMENTandPOSTGRES_TEST_GET_QUERIESto use{query_statement_source}placeholderPostgresQueryParserSource.get_sql_statement()and connection test methods to pass the configured sourcePostgres.mdandTimescale.mdExample configuration:
{ "type": "Postgres", "username": "user", "hostPort": "localhost:5432", "database": "postgres", "queryStatementSource": "my_schema.custom_pg_stat_statements" }This mirrors the existing pattern in the Snowflake connector (
accountUsageSchema).Type of change:
Checklist:
Fixes <issue-number>: <short explanation>or decision-making process is reflected in the issue.
Note on migrations: No migration script needed - this adds a new optional property with a default value that preserves existing behavior.
Warning
Firewall rules blocked me from connecting to one or more addresses (expand for details)
I tried to connect to the following addresses, but was blocked by firewall rules:
www.antlr.org/usr/bin/curl curl -O REDACTED(dns block)If you need me to access, download, or install something from one of these locations, you can either:
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.