S3 lifecycle policy and history files

[lgyurci]

This is more like a general question about S3 lifecycle policies, since there is some conflicting information about this topic. The docs (https://cloudnative-pg.io/plugin-barman-cloud/docs/object_stores/#s3-lifecycle-policy) explicitly recommend setting an S3 lifecycle policy on the current versions of the objects as well. However, this would also expire history files, which are not touched in the objectstore after transfering them once. Are these archived history files not needed for restoration? Then why archive them? If they are needed, then is there some mechanism that guarantees a timeline switch in each retention window?

[dosubot]

You've identified a real issue - timeline history files (.history) are essential for PostgreSQL recovery operations involving timeline switches (e.g., after failovers), and the S3 lifecycle policy recommendation does conflict with preserving them.

When these files are deleted by lifecycle policies, replicas cannot find the bridge to new timelines, causing errors like "requested timeline X is not a child of this server's history" and PITR failures.

There is no built-in mechanism that guarantees timeline switches happen within a retention window. However, the solution is to use the historyTags feature to preserve history files separately:

tags:
  backupRetentionPolicy: "expire"
historyTags:
  backupRetentionPolicy: "keep"

This allows other backup objects to expire normally while keeping history files indefinitely.

For S3 lifecycle policies specifically, you could configure tag-based rules to exclude .history files from deletion or use "Archive"/"Coldline" transitions instead of permanent deletion for those files.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}