Patchwork: retry POST/PATCH (longer) with Retry + more log - #88
Merged
Conversation
By default, these methods are not retried, because they are not considered to be idempotent: multiple requests with the same parameters end with the same state. Here in this patchwork module, it looks like it is fine to retry multiple times in case of errors. It sounds better doing that with the lib than manual code here, with different timeout. Note that the default manual retry timeout was too low: recently, the nipa-upload-wireless service failed because it wasn't able to post a check, even with 3 retries. All attempts failed with a 504 error. With the previous code, it was retrying only during 3.5 minutes, compared to ~17 minutes with the lib. That leaves more time for server maintenance operations. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
s/post/patch/, to be able to distinct between post and patch. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
This can probably happen in case of errors, e.g. a 504 timeout error. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
Now that retries are handled by the lib, it looks interesting to find out how long the different requests took, similar to what is done with the GET methods. Use a different message for each method, to be able to monitor them individually if needed. Note that on the brancher service, the longest GET response time was at 86 seconds. So logging such info looks important to understand other errors. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
In case of issues, it looks interesting to have the timestamps in the logs, to find out a root cause, e.g. a server maintenance, or too many parallel requests. Signed-off-by: Matthieu Baerts <matttbe@kernel.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recently, the nipa-upload-wireless service failed because it wasn't able to POST a PW check, even after 3 retries. All attempts failed with a 504 error. With the previous code, it was retrying only during 3.5 minutes, compared to ~17 minutes with the lib. Switching to the lib: it should be better, also for maintenance purposes. That also leaves more time for PW server maintenance operations.
While at it, improve the log: typo, non-JSON data, response time, and timestamps.
That's not urgent.