You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Python: [Bug]: RedisHistoryProvider.get_messages aborts on a single malformed entry #9974
RedisHistoryProvider.get_messages deserializes every entry in the Redis list without per-entry error handling. A single empty string or corrupt JSON entry raises JSONDecodeError/ValueError and aborts the entire read, even when the remaining entries are valid.
Expected Behavior
Malformed entries should be skipped (with a warning) and the valid messages returned, matching how CosmosHistoryProvider.get_messages already handles this case.
Steps to Reproduce
Configure a RedisHistoryProvider.
Push a corrupt/empty entry into the underlying Redis list (e.g. ["", "not-json", <valid message>]).
Call get_messages.
Minimal Reproduction
# Redis list contains ["", "not-json", <valid entry>]# provider.get_messages() raises instead of returning the valid entry
🤖 Automated triage reproduction notes(agent-authored — trust but verify)
Agent analysis
python/packages/redis/agent_framework_redis/_history_provider.py::RedisHistoryProvider.get_messages at line 274 aborts when any Redis list entry cannot be decoded as a message. Minimal repro: mock lrange to return ["", "not-json", json.dumps(valid_message.to_dict())], call get_messages("s1"), and observe JSONDecodeError before the valid entry is returned.
/plan Resolve the remaining HOW questions from the maintainer review of #9909 before treating #9974 as implementation-ready. Compare explicit rejection with atomic recovery of complete tool exchanges; do not assume per-record skip-and-warn is safe or approved. Preserve call/result pairing, valid-history behavior, save-time replay/deduplication (coordinate with the existing #10127 plan), and payload-free diagnostics. Include corrupt JSON and malformed-message shapes, corrupted call/result pairs, and full-transcript replay tests. Estimate the smallest correct production-code change and any compatibility impact. This is planning only, not implementation authorization.
RedisHistoryProvider.get_messages reads the whole Redis list and decodes each entry with json.loads + Message.from_dict, with no error handling (_history_provider.py L261-289). One bad entry makes the whole read fail. The issue asks for skip-and-warn, as Cosmos does. Your /plan asks which behavior is actually safe, so I compared the options below. I did not run anything and did not look at the open fix PR.
What the code shows (confirmed from reading it)
save_messages also calls get_messages (L316) to work out which incoming messages are new. So today one bad entry blocks reads and saves for that session.
Dedup depends on an exact ordered match.filter_new_messages in _sessions.py L178 looks for the stored list as a contiguous run inside the incoming list (the "full transcript replay" case, e.g. AG-UI).
Cosmos skips per item (_history_provider.py L149-187). It catches only ValueError and logs the exception text. That is precedent, not an approved design, and it has the same hole described below.
Take stored history [user, assistant(call A), tool(result A), assistant] where the tool(result A) entry is corrupt.
Orphaned call. Skipping gives assistant(call A) with no result. Providers commonly reject this, so the agent fails later and further from the cause.
Cannot tell what was lost. A corrupt entry has no readable call_id, so you cannot know which exchange it belonged to. Dropping "the whole exchange" would be a guess based on unpaired neighbours.
Replay duplicates. The stored list now has a hole. When a client replays the full transcript, the contiguous-match check fails and the entire transcript is appended again. This is the superlinear growth that filter_new_messages exists to prevent.
Silent context change. The model sees a different conversation with only a log line as evidence.
Recommendation
Code change warranted, but explicit rejection, not skip-and-warn. Atomic recovery should wait for a maintainer decision.
Explicit rejection
Atomic recovery of whole exchanges
Pairing / replay
Preserved: nothing is returned or saved on partial data
Needs a new orphan-detection pass; corrupt entries have no call_id, so grouping is heuristic
Size
Small, Redis only
Larger, likely a shared helper in core plus a save-path change so dedup compares against the full stored list
User impact
Session is blocked until clear() or manual repair, but the failure is clear
Session keeps working with silently altered history
Compatibility
Error type changes from raw JSONDecodeError/ValueError/TypeError/KeyError
Return values change for corrupt data; new silent data loss
Recovery can be added later as an explicit opt-in if maintainers want it. Making it the default is the risky part.
Change get_messages in the Redis provider only. Wrap the per-entry decode and raise one dedicated error. It should subclass ValueError so existing except ValueError code keeps working. It should carry only the session key and entry index, never the entry text, JSON text, or the underlying exception message. Chain with from None, or log only the exception class name. save_messages then fails closed with no further code change.
Cover these cases in the existing TestRedisHistoryProviderGetMessages class (this is test work, not code I ran). Each failure should assert the error type, that the message names the index but not the payload, and that nothing was written:
corrupt JSON: "", "not-json", truncated JSON;
valid JSON with the wrong shape: a list, a string, null, a missing role, a bad contents;
a corrupted result in a call/result pair: a read error, and save_messages raising rather than re-appending;
a valid full history, unchanged;
full-transcript replay on a valid history: only the new tail is pushed (guards filter_new_messages behavior);
same replay with a corrupt entry present: no rpush.
Documentation: a short note in the package README or docstring, giving the error and recovery via clear(). Cosmos is left alone; aligning it is a separate compatibility decision.
Smallest production change: roughly 10-20 lines in one file, plus tests. No core changes. The compatibility impact is the exception type only, and valid data behaves exactly as before. I could not check which exceptions Message.from_dict raises for every bad shape, so step 3 should settle that before step 2 picks what to catch.
Done when: corrupt data raises the dedicated error with no payload in the message or logs, no partial history is returned, nothing is appended after a failed read, and the valid and replay tests pass unchanged.
Questions for maintainers
Is explicit rejection acceptable as the default, with recovery as a possible later opt-in?
If recovery is wanted, should it be a shared core helper (reused by Cosmos) rather than Redis-specific?
Should max_messages trimming (ltrim) also be checked so it never splits a tool exchange? That is existing behavior and out of scope here, so I left it out.
Generated by Issue Planning for #9974 · copilot · auto · ~41 AIC · ⌖ ~1 AIC · ⊞ 4.8K · ◷
Hi Eduard van Valkenburg (@eavanvalkenburg), question 3 is tracked in #9851: max_messages trimming can split a tool exchange and leave a tool result without its call, after which every later turn fails with a 400 from the model provider. On main 13956ad with gpt-6-luna, 10-turn chats with a tool break this way at limits 4, 6, 8 and 10, on Chat Completions and on Responses with store=False. PR #9890 changes that trimming to cut on complete groups.
That also shows point 1 in practice: an unpaired call or result in stored history breaks the whole session, not only the read.
Maintainer direction: malformed Redis history may be recovered only when recovery is atomic and replay-safe. Do not skip individual records in a way that can orphan a function call or result. Preserve complete call/result exchanges, and ensure that saving or replaying a full transcript after recovery cannot append duplicate history. If a malformed record cannot be associated with an exchange safely, fail closed rather than guess. Diagnostics must not include payload data. Please update the acceptance criteria and implementation tests for corrupt call/result pairs, unclassifiable malformed records, and full-transcript replay.
Activity
github-actions commentedon Nov 2, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
python/packages/redis/agent_framework_redis/_history_provider.py::RedisHistoryProvider.get_messagesat line 274 aborts when any Redis list entry cannot be decoded as a message. Minimal repro: mocklrangeto return["", "not-json", json.dumps(valid_message.to_dict())], callget_messages("s1"), and observeJSONDecodeErrorbefore the valid entry is returned.python/packages/redis/tests/test_providers.py::TestRedisHistoryProviderGetMessages::test_skips_malformed_entries1.0.0b261030eavanvalkenburg commentedon Nov 9, 2026
/plan Resolve the remaining HOW questions from the maintainer review of #9909 before treating #9974 as implementation-ready. Compare explicit rejection with atomic recovery of complete tool exchanges; do not assume per-record skip-and-warn is safe or approved. Preserve call/result pairing, valid-history behavior, save-time replay/deduplication (coordinate with the existing #10127 plan), and payload-free diagnostics. Include corrupt JSON and malformed-message shapes, corrupted call/result pairs, and full-transcript replay tests. Estimate the smallest correct production-code change and any compatibility impact. This is planning only, not implementation authorization.
github-actions commentedon Nov 9, 2026
Summary
RedisHistoryProvider.get_messagesreads the whole Redis list and decodes each entry withjson.loads+Message.from_dict, with no error handling (_history_provider.pyL261-289). One bad entry makes the whole read fail. The issue asks for skip-and-warn, as Cosmos does. Your/planasks which behavior is actually safe, so I compared the options below. I did not run anything and did not look at the open fix PR.What the code shows (confirmed from reading it)
save_messagesalso callsget_messages(L316) to work out which incoming messages are new. So today one bad entry blocks reads and saves for that session.filter_new_messagesin_sessions.pyL178 looks for the stored list as a contiguous run inside the incoming list (the "full transcript replay" case, e.g. AG-UI)._history_provider.pyL149-187). It catches onlyValueErrorand logs the exception text. That is precedent, not an approved design, and it has the same hole described below.RedisContextProvider(vector memory documents). It does not touch the history provider orfilter_new_messages. I did not re-read its plan comments. Coordination is only on the principle: replay must not create duplicates.Why per-record skip is not safe
Take stored history
[user, assistant(call A), tool(result A), assistant]where thetool(result A)entry is corrupt.assistant(call A)with no result. Providers commonly reject this, so the agent fails later and further from the cause.call_id, so you cannot know which exchange it belonged to. Dropping "the whole exchange" would be a guess based on unpaired neighbours.filter_new_messagesexists to prevent.Recommendation
Code change warranted, but explicit rejection, not skip-and-warn. Atomic recovery should wait for a maintainer decision.
call_id, so grouping is heuristicclear()or manual repair, but the failure is clearJSONDecodeError/ValueError/TypeError/KeyErrorRecovery can be added later as an explicit opt-in if maintainers want it. Making it the default is the risky part.
Proposed plan (nothing has been changed)
Retitle or re-scope Python: [Bug]: RedisHistoryProvider.get_messages aborts on a single malformed entry #9974 so the expected behavior is "fail clearly" rather than "skip and return valid entries". That is the open design decision for the maintainer, and the rest of the plan assumes it.
Change
get_messagesin the Redis provider only. Wrap the per-entry decode and raise one dedicated error. It should subclassValueErrorso existingexcept ValueErrorcode keeps working. It should carry only the session key and entry index, never the entry text, JSON text, or the underlying exception message. Chain withfrom None, or log only the exception class name.save_messagesthen fails closed with no further code change.Cover these cases in the existing
TestRedisHistoryProviderGetMessagesclass (this is test work, not code I ran). Each failure should assert the error type, that the message names the index but not the payload, and that nothing was written:"","not-json", truncated JSON;null, a missingrole, a badcontents;save_messagesraising rather than re-appending;filter_new_messagesbehavior);rpush.Documentation: a short note in the package README or docstring, giving the error and recovery via
clear(). Cosmos is left alone; aligning it is a separate compatibility decision.Smallest production change: roughly 10-20 lines in one file, plus tests. No core changes. The compatibility impact is the exception type only, and valid data behaves exactly as before. I could not check which exceptions
Message.from_dictraises for every bad shape, so step 3 should settle that before step 2 picks what to catch.Done when: corrupt data raises the dedicated error with no payload in the message or logs, no partial history is returned, nothing is appended after a failed read, and the valid and replay tests pass unchanged.
Questions for maintainers
max_messagestrimming (ltrim) also be checked so it never splits a tool exchange? That is existing behavior and out of scope here, so I left it out.LUCKYREDDY31 commentedon Nov 9, 2026
Hi Eduard van Valkenburg (@eavanvalkenburg), question 3 is tracked in #9851:
max_messagestrimming can split a tool exchange and leave a tool result without its call, after which every later turn fails with a 400 from the model provider. On main 13956ad withgpt-6-luna, 10-turn chats with a tool break this way at limits 4, 6, 8 and 10, on Chat Completions and on Responses withstore=False. PR #9890 changes that trimming to cut on complete groups.That also shows point 1 in practice: an unpaired call or result in stored history breaks the whole session, not only the read.
eavanvalkenburg commentedon Nov 12, 2026
Maintainer direction: malformed Redis history may be recovered only when recovery is atomic and replay-safe. Do not skip individual records in a way that can orphan a function call or result. Preserve complete call/result exchanges, and ensure that saving or replaying a full transcript after recovery cannot append duplicate history. If a malformed record cannot be associated with an exchange safely, fail closed rather than guess. Diagnostics must not include payload data. Please update the acceptance criteria and implementation tests for corrupt call/result pairs, unclassifiable malformed records, and full-transcript replay.