KAFKA-20762 Add --zk-rollback-compatible flag to kafka-storage.sh format

[aiven-anton]

In order to support node replacements during KRaft migration, and for a cluster that has such replacements to remain possible to rollback, the storage formatting tool must be able to produce a storage directory that can be rolled back. Currently this is not the case, and the tool always produces a meta.properties file with version=1.

When rollback is triggered on such a cluster that has had node replacements, this results in a broker crash-loop with the below log.

Jul 02 14:56:32 kafka[25156]: [2026-07-02 14:56:32,647] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
                              java.lang.RuntimeException: Found unexpected version in /srv/kafka/meta.properties. ZK-based brokers that are not migrating only support version 0 (which is implicit when the `version` field is missing).
                                      at org.apache.kafka.metadata.properties.MetaPropertiesEnsemble.verify(MetaPropertiesEnsemble.java:489)
                                      at kafka.server.KafkaServer.startup(KafkaServer.scala:258)
                                      at kafka.Kafka$.main(Kafka.scala:112)
                                      at kafka.Kafka.main(Kafka.scala)
Jul 02 14:56:32 kafka[25156]: [2026-07-02 14:56:32,650] INFO shutting down (kafka.server.KafkaServer)

This commit remedies this by introducing a new flag to the formatter tool that makes it create a meta.properties with version=0 and compatible with rollback. This matches what the broker itself does during a normal from-ZK migration.

Documentation is updated both in the KRaft migration section and in the generic section for provisioning nodes.

Reviewers: Mickael Maison mickael.maison@gmail.com

[mimaison]

This adds a new command line flag so it will require a Kafka Improvement Proposal. See https://cwiki.apache.org/confluence/spaces/KAFKA/pages/50859233/Kafka+Improvement+Proposals for the process

[aiven-anton]

@mimaison Thanks for looking at this and reviewing!

Does that apply even if the CLI remains fully backwards compatible with how it works prior? Reading the linked doc, I get the impression the intention is mainly to care about not breaking any existing usages.

Public apis are next as they cause people to rebuild code and lead to compatibility issues in large multi-dependency projects (which end up requiring multiple incompatible versions). Configuration, monitoring, and command line tools can be faster and looser — changes here will break monitoring dashboards and require a bit of care during upgrades but aren't a huge burden.

The new flag introduced is optional, and when omitted the tool works exactly as before, so the change cannot affect any existing usages.

[mimaison]

Does that apply even if the CLI remains fully backwards compatible with how it works prior?

Yes