Random POST request failures after upgrade from 9.2.4 to 10.0.4 or 10.2.0

[mirakae]

Description

After upgrading Apache Traffic Server from 9.2.4 to 10.0.4 or even 10.2.0 we are experiencing random issues handling POST requests.

The problem appears intermittently: some POST requests do not receive any response from ATS.
The client connection remains open until timeout.

This issue did not occur on version 9.2.4.

Environment

• ATS Version: 10.0.4/10.2.0
• Previous Version: 9.2.4
• OS: Rocky 8
• Plugins enabled: @plugin=header_rewrite.so, regex_remap.so, cachekey.so, compress.so

Expected Behavior

POST requests should be forwarded upstream and a valid response should be returned to the client.

Actual Behavior

Some POST requests randomly hang.
No response is returned to the client.
The client eventually times out and fallback.
Read_timeout 30s --> this is reached for the fact that I don't get response

Frequency

Approximately 15% of POST requests (random pattern).

Steps to Reproduce

Unfortunately not deterministic, but with my config I'm able to reprodiuce it.

Debug Logs

Below is a relevant excerpt from debug logs (http|cache.|regex.):

Possible solution

Enable http2, the error rate drops close to 0. But why on ATS 9 I din't have this issue with http1.

post_issue_ats10.txt

[bryancall]

@brbzull0 do you mind taking a look at this? I'm concerned that we may have some post issues in the 10.x releases

[bryancall]

@mirakae Can you please post the configuration you are using?

[mirakae]

Let me know if this help @bryancall , @brbzull0 .

Thanks for your support.

config_vod_post.txt

Just let you know that enabling http2, was able to bypass the issue.