Skip to content

✨ S3UTILS-230 implement CRR multi-site V2 replication format#396

Merged
bert-e merged 16 commits into
development/1from
feature/S3UTILS-230/multi-crr
Jun 26, 2026
Merged

✨ S3UTILS-230 implement CRR multi-site V2 replication format#396
bert-e merged 16 commits into
development/1from
feature/S3UTILS-230/multi-crr

Conversation

@DarkIsDude

@DarkIsDude DarkIsDude commented May 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add V2 replication format support to `ReplicationStatusUpdater`, implementing the CRR to Multiple Sites design
  • V2 configs (rules with a `Filter` element) route through a new `_markObjectPendingV2` path that writes per-backend `destination`/`role` and removes legacy top-level `storageClass`/`storageType`/`destination`
  • V1 configs (no `Filter` on any rule) retain existing behavior unchanged
  • Arsenal static methods (`resolveBackends`, `resolveSourceRole`, `resolveDestinationRole`) are reused instead of maintaining parallel implementations; arsenal dependency pinned to `improvement/ARSN-571/crr-multi`

Changes

`CRR/ReplicationStatusUpdater.js`

  • `_isV1Format()` — V2 detection aligned with arsenal: a rule is V2 if it carries a `Filter` element
  • `_getMatchingRules()` — prefix-matches object key against enabled rules, deduplicates by site (highest priority wins); feeds the adapter for `resolveBackends`
  • `_computeTopLevelStatus()` — aggregates backend statuses: FAILED > PROCESSING > COMPLETED
  • `_removeV1Fields()` — strips legacy top-level `storageClass`/`storageType`/`destination` when migrating to V2
  • `_markObjectPendingV2()` — V2 metadata writer: adapts AWS SDK rule shape to arsenal's `ReplicationConfigurationMetadata`, calls `ReplicationConfiguration.resolveBackends` for prefix matching / priority dedup / per-backend destination+role stamping, then calls `resolveSourceRole` for the top-level role

`package.json` — arsenal pinned to `improvement/ARSN-571/crr-multi` for the new static methods

`tests/utils/crr.js` — V2 bucket replication config fixture

`tests/unit/CRR/ReplicationStatusUpdater.js` — 7 new V2 test cases (single match, dual-rule prefix overlap, no-match skip, SITE_NAME filter, all-up-to-date skip, PROCESSING aggregate status, forceUsingConfiguration)

Test plan

  • All 456 unit tests pass (`yarn test:unit`)
  • Lint clean (`yarn lint`)
  • V2 metadata written with per-backend `destination`/`role` and no legacy top-level fields
  • V1 configs produce identical output to before this change

Issue: S3UTILS-230

⚠️ The procedure at this page seems not up to date : https://scality.atlassian.net/wiki/spaces/TS/pages/2312536220/Ingestion+Replication+XDM#Step-one---Clear-the-replication-queue. The ENDPOINT should now be http://.... with the port :80. Should we ping CS or update the page directly ?

@DarkIsDude DarkIsDude self-assigned this May 28, 2026
@bert-e

This comment was marked as resolved.

@bert-e

This comment was marked as resolved.

Comment thread tests/unit/CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.08911% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.31%. Comparing base (eb9592d) to head (6ad3454).
⚠️ Report is 30 commits behind head on development/1.

Files with missing lines Patch % Lines
CRR/ReplicationStatusUpdater.js 91.08% 5 Missing and 4 partials ⚠️
Additional details and impacted files
@@                Coverage Diff                @@
##           development/1     #396      +/-   ##
=================================================
+ Coverage          44.90%   45.31%   +0.41%     
=================================================
  Files                 88       88              
  Lines               6456     6516      +60     
  Branches            1352     1369      +17     
=================================================
+ Hits                2899     2953      +54     
- Misses              3511     3513       +2     
- Partials              46       50       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from d5490b8 to f9fca0c Compare June 1, 2026 20:17
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@claude

This comment was marked as resolved.

@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude requested review from a team, delthas and maeldonn June 2, 2026 14:13
@bert-e

bert-e commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@maeldonn maeldonn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be simplified a lot by using arsenal logic

Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js Outdated
Comment thread CRR/ReplicationStatusUpdater.js
Comment thread CRR/ReplicationStatusUpdater.js Outdated
@bert-e

bert-e commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following reviewers are expecting changes from the author, or must review again:

Comment thread package.json
@claude

This comment was marked as resolved.

Comment thread package.json Outdated
@claude

This comment was marked as resolved.

@DarkIsDude DarkIsDude requested a review from maeldonn June 8, 2026 17:40
@DarkIsDude DarkIsDude requested a review from maeldonn June 17, 2026 13:14
@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from d1e9317 to dcbdaa0 Compare June 17, 2026 14:28
Comment thread CRR/ReplicationStatusUpdater.js Outdated

@maeldonn maeldonn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bert-e

bert-e commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@DarkIsDude DarkIsDude requested a review from delthas June 19, 2026 09:08
Comment thread package.json Outdated
@bert-e

bert-e commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

The following reviewers are expecting changes from the author, or must review again:

@DarkIsDude DarkIsDude requested a review from maeldonn June 23, 2026 14:56
@bert-e

bert-e commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@delthas

delthas commented Jun 26, 2026

Copy link
Copy Markdown

Sorry for the delay :) Good work

Comment thread package.json Outdated
@DarkIsDude DarkIsDude force-pushed the feature/S3UTILS-230/multi-crr branch from 006d95a to 6ad3454 Compare June 26, 2026 12:20
@DarkIsDude DarkIsDude removed the request for review from francoisferrand June 26, 2026 12:27
@DarkIsDude

Copy link
Copy Markdown
Contributor Author

/approve

@bert-e

bert-e commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/1

The following branches will NOT be impacted:

  • development/1.13
  • development/1.14
  • development/1.15
  • development/1.16
  • development/1.17
  • development/1.18
  • development/1.4
  • development/1.7

This pull request does not target the following hotfix branch(es) so they
will be left untouched:

  • hotfix/1.14.0

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

@bert-e

bert-e commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/1

The following branches have NOT changed:

  • development/1.13
  • development/1.14
  • development/1.15
  • development/1.16
  • development/1.17
  • development/1.18
  • development/1.4
  • development/1.7

Please check the status of the associated issue S3UTILS-230.

Goodbye darkisdude.

@bert-e bert-e merged commit 8555fe9 into development/1 Jun 26, 2026
18 checks passed
@bert-e bert-e deleted the feature/S3UTILS-230/multi-crr branch June 26, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants