-
-
Notifications
You must be signed in to change notification settings - Fork 65
Composite score is linear #1461
Copy link
Copy link
Open
Labels
platformEverything related to benchmark exection, including the PR workflow.Everything related to benchmark exection, including the PR workflow.testsAbout test design, issues with tests etc.About test design, issues with tests etc.
Description
Activity
Metadata
Metadata
Assignees
Labels
platformEverything related to benchmark exection, including the PR workflow.Everything related to benchmark exection, including the PR workflow.testsAbout test design, issues with tests etc.About test design, issues with tests etc.
Currently the composite score is linear in the sense that for each test (scored individually from 0 to 1000) the score is linear, if the top ranked is for example at 3 million rps, a framework at 1.5 million has 500 score.
This may not be the best approach, as throughput doesn't increase linearly and frameworks that score very bad in one or more tests should be heavily penalized. For example if a framework does ok in all tests but score very poorly in async test, this should be reflected at composite score in a meaningful way.
thoughts?