Repository navigation
[ENG-12560] Fully implement the stress testing tool - #12027
Ostap-Zherebetskyi wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Design issues:
The form allows 1000 tasks/s for 3600s
- Every step locks the same suite row. Each publish, task start (
simulated_workload) and task finish (finish_task) runs SELECT … FOR UPDATE on that one row, so instead of stress testing celery we're stress testing our db. I think we can leave db locks to write stats/finish/cancel/start suite etc., but stress tasks part shouldn't make any database calls unless they're specifically designed to do so - That's also up to about 3.6M task rows per run (max), and nothing ever cleans them up
Also, I left a few comments in the PR
antkryt
left a comment
There was a problem hiding this comment.
LGTM
One thing to discuss: long-living coordinator (start_celery_stress_test)
cslzchen
left a comment
There was a problem hiding this comment.
Looks great overall with all the options to stress test. Let's add docstring and comments and if necessary update the naming conventions.
| error = models.TextField(blank=True) | ||
|
|
||
| class Meta: | ||
| ordering = ['-created_at'] |
There was a problem hiding this comment.
Note: we don't expect there to be a lot of CeleryTestSuite and we won't have many operations that would benefit from index. So we decide not to create index to save time for insert/update/delete.
| return self.scenario.partition('.')[0] | ||
|
|
||
| class Meta: | ||
| indexes = [models.Index(fields=['suite', 'status'], name='celery_test_suite_status_idx')] |
There was a problem hiding this comment.
Note: there will be a lot of tasks, so use combined index suite with status works for our purpose. We will add other index if necessary (unlikely though).
|
|
||
| return None | ||
| @shared_task(name='stress_testing.start_celery_stress_test') | ||
| def start_celery_stress_test(suite_id): |
There was a problem hiding this comment.
Add a short docstring, and add comments to describe steps/conditions
| beat_schedule = { | ||
| 'cleanup-celery-test-history': { | ||
| 'task': 'stress_testing.cleanup_history', | ||
| 'schedule': crontab(minute=30, hour=3), |
There was a problem hiding this comment.
- Find the best time (UTC) and add comment with EDT
Ticket
Purpose
Fully implement the stress testing tool
Changes
Side Effects
QE Notes
CE Notes
Documentation