Skip to content

CLDSRV-1083 add a replication compatibility mode - #6388

Open
nicolas2bert wants to merge 1 commit into
development/9.4from
improvement/CLDSRV-1083/replication-legacy-mode
Open

nicolas2bert wants to merge 1 commit into
development/9.4from
improvement/CLDSRV-1083/replication-legacy-mode

Conversation

@nicolas2bert

@nicolas2bert nicolas2bert commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Add a setting, replicationCascadeCompatMode, for the S3C upgrade to 10.1, the first S3C release with replication multi-destination and cascade.

The reason: an S3C upgrade goes node by node. For a while, old and new nodes work on the same buckets and objects, and the old nodes don't understand what the new cloudserver does for replication:

  • The old backbeat can't read the new object format, so replication fails (S3C-11677).
  • The old cloudserver crashes on a bucket whose replication configuration was written by a new node.
  • A site that is not upgraded yet sends changes that the new cloudserver refuses or drops (S3C-11679, S3C-11680).

So during the upgrade, the setting makes the new cloudserver behave like the old one: same format, no forwarding, no revision check. Once every node is upgraded, we turn it off and multi-destination and cascade start working.

For Artesca / Zenko nothing changes: the setting is off by default, and off is exactly today's behaviour.

During a node-by-node upgrade, the nodes that are not upgraded yet
read what an upgraded cloudserver writes, and they can't read the new
replication format: an old backbeat doesn't replicate the objects, and
an old cloudserver stops on the bucket replication configuration.

With replicationCascadeCompatMode set, cloudserver stores and does what a
cloudserver older than 9.4 does for replication:

- the replicationInfo of an object is written in the old format, and
  no microVersionId is set on a metadata change
- the bucket replication configuration is stored in the old shape. We
  refuse what an old cloudserver refuses: a single role for a CRR
  site, rules to different buckets, overlapping prefixes. Like it, we
  ignore the account of a rule
- a replica is stored as backbeat sent it, with the status REPLICA. We
  ignore x-scal-micro-version-id and the version check of putData, so
  we don't compare revisions and we don't replicate a replica further
- on an ACL change, we tell CRR from cloud destinations with the
  location type and not with the role of the backend: the old format
  has no role per backend. No change of behaviour outside the mode

When the setting is absent or false, nothing changes.
@bert-e

bert-e commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Hello nicolas2bert,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Branches have diverged

This pull request's source branch improvement/CLDSRV-1083/replication-legacy-mode has diverged from
development/9.6 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

  • Merge origin/development/9.6 into improvement/CLDSRV-1083/replication-legacy-mode
  • Rebase improvement/CLDSRV-1083/replication-legacy-mode onto origin/development/9.6

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

@nicolas2bert
nicolas2bert changed the base branch from development/9.6 to development/9.4 November 20, 2026 13:18
@bert-e

bert-e commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-1083 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.4.18

  • 9.5.8

  • 9.6.0

Please check the Fix Version/s of CLDSRV-1083, or the target
branch of this pull request.

config.replicationCascadeCompatMode = true;
bucketPutReplication(makeAuthInfo('accessKey1'), request, log, err => {
config.replicationCascadeCompatMode = false;
assert(err.is.InvalidRequest);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer comparing the error message over .is, so a failure shows the actual error name.

Suggested change
assert(err.is.InvalidRequest);
assert.strictEqual(err.message, 'InvalidRequest');

Comment on lines +42 to +51
(config, next) => {
if (!s3config.replicationCascadeCompatMode) {
return next(null, config);
}
const { refused, replicationConfig } = toLegacyConfiguration(config);
if (refused) {
return next(errorInstances[refused.code].customizeDescription(refused.description));
}
return next(null, replicationConfig);
},
@bert-e

bert-e commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.07843% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.82%. Comparing base (2b375a5) to head (3c10025).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
lib/Config.js 66.66% 1 Missing ⚠️
lib/api/bucketPutReplication.js 87.50% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
...api/apiUtils/bucket/getReplicationConfiguration.js 100.00% <100.00%> (ø)
lib/api/apiUtils/object/bumpMicroVersionId.js 100.00% <100.00%> (ø)
lib/api/apiUtils/object/getReplicationInfo.js 100.00% <100.00%> (ø)
lib/metadata/acl.js 98.78% <100.00%> (ø)
lib/routes/routeBackbeat.js 78.96% <100.00%> (+0.05%) ⬆️
lib/Config.js 82.15% <66.66%> (-0.06%) ⬇️
lib/api/bucketPutReplication.js 90.69% <87.50%> (-0.97%) ⬇️

... and 2 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.4    #6388      +/-   ##
===================================================
- Coverage            86.85%   86.82%   -0.03%     
===================================================
  Files                  214      214              
  Lines                14707    14751      +44     
===================================================
+ Hits                 12774    12808      +34     
- Misses                1933     1943      +10     
Flag Coverage Δ
checksums-disabled-tests 35.22% <9.80%> (-0.06%) ⬇️
file-ft-tests 69.64% <35.29%> (-0.19%) ⬇️
file-ft-tests-null-compat 70.13% <35.29%> (-0.14%) ⬇️
kmip-ft-tests 27.99% <9.80%> (-0.06%) ⬇️
mongo-v0-ft-tests 70.76% <35.29%> (-0.14%) ⬇️
mongo-v1-ft-tests 70.76% <35.29%> (-0.19%) ⬇️
multiple-backend 35.91% <17.64%> (-0.08%) ⬇️
s3c-ft-tests-v0 64.75% <35.29%> (-0.14%) ⬇️
s3c-ft-tests-v0-null-compat 64.80% <35.29%> (-0.12%) ⬇️
s3c-ft-tests-v1 64.73% <35.29%> (-0.12%) ⬇️
sur-tests 36.41% <9.80%> (-0.09%) ⬇️
sur-tests-inflights 39.19% <9.80%> (-0.12%) ⬇️
unit 75.22% <96.07%> (+0.05%) ⬆️
utapi-v2-tests 35.16% <9.80%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants