Skip to content

durable: an instance touched by two versions in a rolling deploy can't be recovered by either #253

Open

Description

@reka18

What happens

#235 raises durable's journal format to 2, and from then on Recover refuses a journal whose Records go below a format it has already seen. An older binary also refuses a Record whose format is lower than its checkpoint. So in a rolling deploy, an instance a new node starts (format-2 baseline) and an old node then fires on (format-1 Record) can't be recovered by either version:

  • the new version reports that an older version appended to a newer journal;
  • the old version reports that the Store didn't keep Record.Format.

Nothing documents a way to get the instance back, short of editing the journal by hand.

Why it's not fixed in #235

#235 documents "don't run mixed versions on one instance", and binaries already built can't be changed. The question is what recovery path to offer an operator who hits it anyway.

Options

  • a recovery tool, or a Recover option, that truncates the tail at the first Record from the other version, keeping the steps up to it;
  • deploy guidance: drain or pin instances to one version during a rollout;
  • a format-compatibility window, where version N also writes format N-1 until a flag flips.

Relates to: #235, #214

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions