Skip to content

[ENG-12560] Fully implement the stress testing tool - #12027

Open
Ostap-Zherebetskyi wants to merge 6 commits into
CenterForOpenScience:feature/celery-improvementsfrom
Ostap-Zherebetskyi:feature/celery-improvements/stress_testing
Open

Ostap-Zherebetskyi wants to merge 6 commits into
CenterForOpenScience:feature/celery-improvementsfrom
Ostap-Zherebetskyi:feature/celery-improvements/stress_testing

Conversation

@Ostap-Zherebetskyi

@Ostap-Zherebetskyi Ostap-Zherebetskyi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Ticket

Purpose

Fully implement the stress testing tool

Changes

Side Effects

QE Notes

CE Notes

Documentation

@Ostap-Zherebetskyi
Ostap-Zherebetskyi marked this pull request as ready for review November 17, 2026 12:30

@antkryt antkryt left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design issues:
The form allows 1000 tasks/s for 3600s

  1. Every step locks the same suite row. Each publish, task start (simulated_workload) and task finish (finish_task) runs SELECT … FOR UPDATE on that one row, so instead of stress testing celery we're stress testing our db. I think we can leave db locks to write stats/finish/cancel/start suite etc., but stress tasks part shouldn't make any database calls unless they're specifically designed to do so
  2. That's also up to about 3.6M task rows per run (max), and nothing ever cleans them up

Also, I left a few comments in the PR

Comment thread osf/models/celery_test.py Resolved
Comment thread admin/base/celery_panel.py Resolved
Comment thread admin/base/celery_panel.py Resolved
Comment thread admin/base/forms.py Resolved
Comment thread admin/base/forms.py Resolved
Comment thread osf/celery_stress_test/scenarios.py Resolved
Comment thread admin/base/celery_panel.py Resolved
Comment thread osf_tests/test_celery_stress_control.py Resolved

@antkryt antkryt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
One thing to discuss: long-living coordinator (start_celery_stress_test)

@cslzchen cslzchen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great overall with all the options to stress test. Let's add docstring and comments and if necessary update the naming conventions.

Comment thread osf/models/celery_test.py
error = models.TextField(blank=True)

class Meta:
ordering = ['-created_at']

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: we don't expect there to be a lot of CeleryTestSuite and we won't have many operations that would benefit from index. So we decide not to create index to save time for insert/update/delete.

Comment thread osf/models/celery_test.py
return self.scenario.partition('.')[0]

class Meta:
indexes = [models.Index(fields=['suite', 'status'], name='celery_test_suite_status_idx')]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: there will be a lot of tasks, so use combined index suite with status works for our purpose. We will add other index if necessary (unlikely though).


return None
@shared_task(name='stress_testing.start_celery_stress_test')
def start_celery_stress_test(suite_id):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a short docstring, and add comments to describe steps/conditions

beat_schedule = {
'cleanup-celery-test-history': {
'task': 'stress_testing.cleanup_history',
'schedule': crontab(minute=30, hour=3),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Find the best time (UTC) and add comment with EDT

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants