SDSTOR-21465: scrubber phase 1

[JacksonYao287]

No description provided.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 50.26212% with 759 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (stable/v4.x@e1c23e1). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/lib/homestore_backend/scrub_manager.cpp	56.28%	351 Missing and 87 partials ⚠️
src/lib/homestore_backend/hs_http_manager.cpp	0.42%	237 Missing ⚠️
src/lib/homestore_backend/hs_pg_manager.cpp	72.50%	21 Missing and 12 partials ⚠️
src/lib/homestore_backend/hs_http_manager.hpp	0.00%	20 Missing ⚠️
src/lib/homestore_backend/scrub_manager.hpp	72.97%	20 Missing ⚠️
...ib/homestore_backend/replication_state_machine.cpp	64.70%	5 Missing and 1 partial ⚠️
src/lib/homestore_backend/MPMCPriorityQueue.hpp	94.59%	1 Missing and 1 partial ⚠️
src/lib/homestore_backend/hs_homeobject.cpp	66.66%	0 Missing and 2 partials ⚠️
src/lib/homestore_backend/hs_shard_manager.cpp	88.88%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@              Coverage Diff               @@
##             stable/v4.x     #413   +/-   ##
==============================================
  Coverage               ?   53.14%           
==============================================
  Files                  ?       39           
  Lines                  ?     6902           
  Branches               ?      943           
==============================================
  Hits                   ?     3668           
  Misses                 ?     2823           
  Partials               ?      411           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xiaoxichen]

The NPE bug should be fixed before merging, other part LGTM.

The NPE can be triggered if some SM enabled scrubbing during config change/upgrade, then causing cluster wide crashing.

[xiaoxichen]

the ut failure is in scubber test, please take a look

[JacksonYao287]

I triggered this UT again and CI passed. Moreover, I run this UT locally for several times , but can not reproduce this failure.

I think it is a flaky case

[JacksonYao287]

I tried several times again, but still can not reproduce this case.

I suspect the root cause is the unexpected leader switch during UT, just like the flaky stuck case we can sometimes see in homestore_test_pg/shard/blob.

if unexpected leader switch happens during UT,
1 for scrubber test, the scrub task will be cancelled and leader will get an incorrect scrub report , which will result the UT failure.

2 for homestore_test_pg/shard/blob, follower will wait for something to happen, and leader think it is not leader any more( because of leader switch) and do not schedule some op, then all the member will sync and wait at some point, and thus the UT is stuck.

so, I think thing we need try to handle the leader switch case in raft test framework.

[xiaoxichen]

OK, lets wait a few days for other team members to review. If no further comment lets merge as is.