Skip to content

injectHeartbeat writing failed on teardown #1663

Description

@pasha132

Hi all!

Environment:
Azure Mysql Flexible Sever, 8.0.44-azure
gh-ost 1.1.8

We have an instance with a number of partitioned tables and a large schema metadata. With this setup, the dry run check fails with the following errors (from the first to the last):

Closed streamer connection. err=<nil>
Dropping table `<redacted>_ghc`
Table dropped
Error 1146 (42S02): Table '<redacted>_ghc' doesn't exist
...
Error 1146 (42S02): Table '<redacted>_ghc' doesn't exist
...
injectHeartbeat writing failed 61 times, last error: Error 1146 (42S02): Table '<redacted>_ghc' doesn't exist

It appears that on slow disks/thousands of tables, finalCleanup is executed too early and drops the table before teardown (where injectHeartbeat actually stops). And deleting a '_ghc' table takes longer than default-retries (60) * heartbeat-interval-millis (100 ms).

Currently, this can be fixed by increasing default-retries or heartbeat-interval-millis (or both), but this seems like a workaround.

Is it possible to fix the order of execution?

Thanks in advance!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions