Skip to content

Request records contain request_response_data twice #151

Open

Description

@cvas22

Problem

Every request record contains the request/response data twice: once flattened into the record (request_received_at, response_time_ms, response_sent_at) and once as a nested request_response_data object:

{"request_response_data": {"request_received_at": "...", "response_time_ms": 0, "response_sent_at": "..."},
 "type": "request", "correlation_id": "-", "...": "...",
 "request_received_at": "...", "response_time_ms": 0, "response_sent_at": "..."}

With a custom request_response_dto_class that adds more fields, all of them are duplicated as well.

Cause

The integrations log with extra={'request_response_data': ...}. BaseJSONFormatter._get_extra_fields() copies every non built-in record attribute, and the DTO is a dict, so it is added as is. JSONRequestLogFormatter then also merges the DTO into the record with json_log_object.update(record.request_response_data).

Question

Removing the nested copy changes the output fields, so I'd like a maintainer decision before sending a fix. Options:

  1. Skip request_response_data in _get_extra_fields() (smaller records, but a change for anyone reading the nested object).
  2. Keep it and document it. Users who don't want it can already drop it with json_logging.EXCLUDED_FIELDS = ("request_response_data",).

Activity

  1. added a commit that references this issue on Nov 12, 2026
    5b58e35
  2. hanming5 commented on Nov 12, 2026

    @hanming5

    Following up on the review of #158, here is what removing the nested request_response_data object from request logs means for compatibility.

    What changes

    • Request logs no longer contain the nested request_response_data object.
    • The same values are still written once at the top level: request_received_at, response_time_ms, response_sent_at, and any field a custom request_response_dto_class adds. This matches the request log sample and field table in the README, which never listed the nested object.

    Who is affected

    • Only log consumers that read the nested path, for example request_response_data.response_time_ms in a dashboard query or alert. They need to switch to the top-level field (response_time_ms), which holds the same value.
    • Users who already drop it with EXCLUDED_FIELDS = ("request_response_data",) see no change, and keeping that setting is harmless.
    • Custom request formatters are not affected: record.request_response_data is still set on the log record.

    Escape hatch
    Anyone who still needs the nested object can add it back in a custom request formatter:

    class NestedDTORequestFormatter(json_logging.JSONRequestLogFormatter):
        def _format_log_object(self, record, request_util):
            json_log_object = super()._format_log_object(record, request_util)
            json_log_object["request_response_data"] = dict(record.request_response_data)
            return json_log_object

    Documentation
    #158 lists this under Changed in the Unreleased section of the CHANGELOG with a migration note, and adds a short README section with the snippet above.

    Strictly speaking this removes a field from the output, so it is a schema change for anyone reading the nested path. Since the nested copy was never documented, the top-level fields carry the same data, and 1.7.0 already documents another output change under Changed (#147), I'd suggest shipping it in 1.7.0 as a documented change rather than keeping the duplicate behind an option. I'm happy to add an opt-in flag instead if you prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions