Skip to content

fix registration submission deadlock - #12030

Open
bodintsov wants to merge 3 commits into
CenterForOpenScience:feature/pbs-26-22from
bodintsov:fix/ENG-12768
Open

bodintsov wants to merge 3 commits into
CenterForOpenScience:feature/pbs-26-22from
bodintsov:fix/ENG-12768

Conversation

@bodintsov

Copy link
Copy Markdown
Contributor

Ticket

https://openscience.atlassian.net/browse/ENG-12768

Purpose

Fix registration submission deadlock

Changes

Fix registration submission deadlock

Side Effects

TBD

QE Notes

TBD

CE Notes

TBD

Documentation

TBD

@sh-andriy sh-andriy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, tested it locally and the deadlock is fixed - on pbs-26-22 I got "deadlock detected" with the draft create holding the node and waiting on FOR UPDATE on the schema, with this branch it goes through. A few things I think should be changed:

  1. CollectionSubmissionActionListPermission (api/collections/permissions.py) - this was the only lock on the submission for POST /v2/collection_submission_actions/, the serializer loads the target with CollectionSubmission.load() without select_for_update. Checked locally with accept in flight + reject at the same time: on pbs-26-22 reject waits and gets 409, with this branch it returns 201 and overwrites accepted with rejected. I'd keep the lock there or lock the target in the serializer
  2. ModeratorMixin.get_provider - the provider lock was what serialized the "Cannot remove last admin" check in remove_from_group. Now two admins removing each other at the same time both go through and the provider ends up with no admins (on pbs-26-22 the second one waits and gets 403). Probably needs a lock just when removing admins
  3. Not from this PR but related - NodeMixin.get_node does a plain FOR UPDATE over the join with nodesettings and region, so every node write also locks the shared region row (us). Locally a PATCH on one project waited until another tx holding KEY SHARE on that row committed, which is what creating any node does. select_for_update(of=('self',)) in get_object_or_error should fix it, could be a follow-up ticket

@brianjgeiger

Copy link
Copy Markdown
Collaborator

Not from this PR but related - NodeMixin.get_node does a plain FOR UPDATE over the join with nodesettings and region, so every node write also locks the shared region row (us). Locally a PATCH on one project waited until another tx holding KEY SHARE on that row committed, which is what creating any node does. select_for_update(of=('self',)) in get_object_or_error should fix it, could be a follow-up ticket

I'd love to get that in here if it's easy enough rather than as a follow-up ticket.

@sh-andriy sh-andriy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, all 3 are fixed, checked locally - reject while accept is in flight gets 409 again, two admins removing each other leaves one admin, and a project PATCH no longer waits on the region row. One thing I think should be added:

  1. Test for RegistrationProviderSubmissionList.perform_create - the ticket asks for an API test on every changed line that's hit by a modifying request, all of them have one except this, nothing posts to /v2/providers/registrations/{provider_id}/submissions/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants