Track scheduled function delay

[Shubham8287]

Summary

fixes: https://github.com/clockworklabs/SpacetimeDBPrivate/issues/3441.

It chooses an arbitrary threshold of 50ms, if a scheduled function starts more than that threshold after its expected time, we emit a warning and record a metric.

Note that, metric does not necessarily tells If something is wrong with Host as drift can also happen due to module's previous schedule function taking long time to finish.

[gefjon]

How much additional work would it be to make the threshold be, say, 25% of the requested delay, rather than a flat 50ms? I worry that 50ms is quite a long time, especially for games which want to run relatively frequent and relatively fast "tick" reducers.

[Shubham8287]

How much additional work would it be to make the threshold be, say, 25% of the requested delay, rather than a flat 50ms?

The scheduling function has precision of few milliseconds, so we would still need some minimum threshold. On top of that, as I mentioned in the description, long running schedule functions can itself contribute to the delay, potentially causing many false positives.

As you already suggested to record a metric for every run, we will be using this value only logging purpose, so it may be fine?

[gefjon]

How much additional work would it be to make the threshold be, say, 25% of the requested delay, rather than a flat 50ms?

The scheduling function has precision of few milliseconds, so we would still need some minimum threshold. On top of that, as I mentioned in the description, long running schedule functions can itself contribute to the delay, potentially causing many false positives.

As you already suggested to record a metric for every run, we will be using this value only logging purpose, so it may be fine?

Yeah, I don't plan to directly alert on any single schedule invocation violating the threshold, I want to detect patterns of functions deviating significantly.

[gefjon]

Looks good, thanks!