HDDS-15879. Ozone Versioning Metadata foundation

[symious]

What changes were proposed in this pull request?

This is the first basic task of Ozone versioning, including the following points

• bucket proto three-state enum
• set-property state machine
• OmKeyInfo proto extension
• versionedKeyTableColumnFamily
• layout feature gate

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15879

How was this patch tested?

unit test.

[chihsuan]

Thanks for the clean foundation work! @symious I verified that the five commits map nicely to the design doc. Left a few small comments below.

[chihsuan]

Nice work keeping the two fields in sync here! One spot that looks left behind:

OzoneManager#getBucketInfo copies only the legacy flag when resolving a link bucket:

ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java

Lines 3059 to 3072 in b981afb

	return bucketInfo.toBuilder()
	.setDefaultReplicationConfig(
	realBucket.getDefaultReplicationConfig())
	.setIsVersionEnabled(realBucket.getIsVersionEnabled())
	.setStorageType(realBucket.getStorageType())
	.setQuotaInBytes(realBucket.getQuotaInBytes())
	.setQuotaInNamespace(realBucket.getQuotaInNamespace())
	.setUsedBytes(realBucket.getUsedBytes())
	.setSnapshotUsedBytes(realBucket.getSnapshotUsedBytes())
	.setSnapshotUsedNamespace(realBucket.getSnapshotUsedNamespace())
	.setUsedNamespace(realBucket.getUsedNamespace())
	.addAllMetadata(realBucket.getMetadata())
	.setBucketLayout(realBucket.getBucketLayout())
	.build();

Since SUSPENDED maps to flag=false, a suspended bucket read through a link would come back as UNVERSIONED. Should we also add .setVersioningStatus(realBucket.getVersioningStatus()) there?

[symious]

Nice catch, Updated, PTAL.

[chihsuan]

Tiny one: toAuditMap() still only records IS_VERSION_ENABLED, so a status change via the new field would show up as isVersionEnabled=null in the audit log. Worth adding a versioningStatus entry while we're here?

ozone/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketArgs.java

Lines 199 to 206 in f74c331

	public Map<String, String> toAuditMap() {
	Map<String, String> auditMap = new LinkedHashMap<>();
	auditMap.put(OzoneConsts.VOLUME, this.volumeName);
	auditMap.put(OzoneConsts.BUCKET, this.bucketName);
	auditMap.put(OzoneConsts.GDPR_FLAG,
	getMetadata().get(OzoneConsts.GDPR_FLAG));
	auditMap.put(OzoneConsts.IS_VERSION_ENABLED,
	String.valueOf(this.isVersionEnabled));

[symious]

Updated, PTAL.

[chihsuan]

Could we also add these three fields to isKeyInfoSame() (and toString())? Right now two records differing only in versionId, or a delete marker vs. a regular key, compare as equal. Looks like the follow-up tasks will rely on OmKeyInfo equality in tests; having this in the foundation would prevent false-positive assertions later. OmBucketInfo already includes the new status in both, so this would keep the two helpers consistent.

ozone/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java

Lines 1004 to 1017 in f74c331

	public boolean isKeyInfoSame(OmKeyInfo omKeyInfo, boolean checkPath,
	boolean checkKeyLocationVersions,
	boolean checkModificationTime,
	boolean checkUpdateID,
	boolean checkOwnerName) {
	boolean isEqual = dataSize == omKeyInfo.dataSize &&
	creationTime == omKeyInfo.creationTime &&
	volumeName.equals(omKeyInfo.volumeName) &&
	bucketName.equals(omKeyInfo.bucketName) &&
	replicationConfig.equals(omKeyInfo.replicationConfig) &&
	Objects.equals(getMetadata(), omKeyInfo.getMetadata()) &&
	Objects.equals(acls, omKeyInfo.acls) &&
	Objects.equals(getTags(), omKeyInfo.getTags()) &&
	getObjectID() == omKeyInfo.getObjectID();

[symious]

Nice catch, Updated, PTAL.