Skip to content

feat(core): make elasticsearch compatible with remote_task_log #62121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Owen-CH-Leung wants to merge 3 commits into
base: main
Choose a base branch
from
+17 −22
Draft

feat(core): make elasticsearch compatible with remote_task_log #62121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
859624c
feat(core): make elasticsearch compatible with remote_task_log
Owen-CH-Leung Feb 18, 2026
84243cb
Merge branch 'main' into core/make-es-compatible-with-remote-log-io
Owen-CH-Leung Feb 20, 2026
1780ead
feat(core): add elasticsearch into min_version_override
Owen-CH-Leung Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 16 additions & 22 deletions airflow-core/src/airflow/config_templates/airflow_local_settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,35 +279,29 @@ def _default_conn_name_from(mod_path, hook_name):
)
remote_task_handler_kwargs = {}
elif ELASTICSEARCH_HOST:
ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get_mandatory_value("elasticsearch", "END_OF_LOG_MARK")
ELASTICSEARCH_FRONTEND: str = conf.get_mandatory_value("elasticsearch", "frontend")
from airflow.providers.elasticsearch.log.es_task_handler import ElasticsearchRemoteLogIO
Copy link
Contributor

@eladkal eladkal Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need also min elastic version in

airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Line 75 in 7c01e0b

MIN_VERSION_OVERRIDE: dict[str, Version] = {

Because we know this won't work with older version of elastic provider

Need to verify with @potiuk this is the right way to proceed

Copy link
Member

@potiuk potiuk Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it depends on whether there is min version that will work here.

Copy link
Contributor Author

@Owen-CH-Leung Owen-CH-Leung Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added "elasticsearch": parse_version("6.5.0") which should be the next es provider version release after the provider mr is merged.


ELASTICSEARCH_WRITE_STDOUT: bool = conf.getboolean("elasticsearch", "WRITE_STDOUT")
ELASTICSEARCH_WRITE_TO_ES: bool = conf.getboolean("elasticsearch", "WRITE_TO_ES")
ELASTICSEARCH_JSON_FORMAT: bool = conf.getboolean("elasticsearch", "JSON_FORMAT")
ELASTICSEARCH_JSON_FIELDS: str = conf.get_mandatory_value("elasticsearch", "JSON_FIELDS")
ELASTICSEARCH_TARGET_INDEX: str = conf.get_mandatory_value("elasticsearch", "TARGET_INDEX")
ELASTICSEARCH_HOST_FIELD: str = conf.get_mandatory_value("elasticsearch", "HOST_FIELD")
ELASTICSEARCH_OFFSET_FIELD: str = conf.get_mandatory_value("elasticsearch", "OFFSET_FIELD")
ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get_mandatory_value("elasticsearch", "LOG_ID_TEMPLATE")

REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(
host=ELASTICSEARCH_HOST,
target_index=ELASTICSEARCH_TARGET_INDEX,
write_stdout=ELASTICSEARCH_WRITE_STDOUT,
write_to_es=ELASTICSEARCH_WRITE_TO_ES,
offset_field=ELASTICSEARCH_OFFSET_FIELD,
host_field=ELASTICSEARCH_HOST_FIELD,
base_log_folder=BASE_LOG_FOLDER,
delete_local_copy=delete_local_copy,
json_format=ELASTICSEARCH_JSON_FORMAT,
log_id_template=ELASTICSEARCH_LOG_ID_TEMPLATE,
)

ELASTIC_REMOTE_HANDLERS: dict[str, dict[str, str | bool | None]] = {
"task": {
"class": "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler",
"formatter": "airflow",
"base_log_folder": BASE_LOG_FOLDER,
"end_of_log_mark": ELASTICSEARCH_END_OF_LOG_MARK,
"host": ELASTICSEARCH_HOST,
"frontend": ELASTICSEARCH_FRONTEND,
"write_stdout": ELASTICSEARCH_WRITE_STDOUT,
"write_to_es": ELASTICSEARCH_WRITE_TO_ES,
"target_index": ELASTICSEARCH_TARGET_INDEX,
"json_format": ELASTICSEARCH_JSON_FORMAT,
"json_fields": ELASTICSEARCH_JSON_FIELDS,
"host_field": ELASTICSEARCH_HOST_FIELD,
"offset_field": ELASTICSEARCH_OFFSET_FIELD,
},
}

DEFAULT_LOGGING_CONFIG["handlers"].update(ELASTIC_REMOTE_HANDLERS)
elif OPENSEARCH_HOST:
OPENSEARCH_END_OF_LOG_MARK: str = conf.get_mandatory_value("opensearch", "END_OF_LOG_MARK")
OPENSEARCH_PORT: str = conf.get_mandatory_value("opensearch", "PORT")
Expand Down
1 change: 1 addition & 0 deletions scripts/ci/prek/update_airflow_pyproject_toml.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@
"git": parse_version("0.0.2"),
"common.messaging": parse_version("2.0.0"),
"informatica": parse_version("0.1.0"),
"elasticsearch": parse_version("6.5.0"),
}


Expand Down
Loading