Redesign Proposal: write snapshots to disk and implement sidecar design

[ognots]

Problem Statement:
Using the Lotus API, snapshot production is slow due to the constraints of Lotus's API. The average snapshot job takes 4h on average, post FVM upgrade.

Possible Solution:
Reports from other operators shows snapshot production being exponentially faster writing snapshots directly to disk. Many report <1h snapshot production.
Draft a design doc and prototype of a "sidecar" design, where filecoin-chain-archiver runs in a process with read-only access to a volume where Lotus can write snapshots directly to.

[ognots]

Design 1:

[ognots]

the first prototype of this is working successfully in theory, in #67
there are however a few relevant blockers:

• Chain ExportRangeInternal returns an EOF error when exporting stateroots lotus#10672 - by far the largest blocker, the ExportRangeInternal method curently returns an EOF error. this is reproduceable with lotus CLI and using filecoin-chain-archiver create
• feat:chain:export-range:configurable output dir and name lotus#10602 - needs reviewed and merged for this design to work. current implementation in lotus makes the filename indeterminate from automation, because the method names the file with a future timestamp