chore: log max run_queue and don't fail health checks based on it #883
Conversation
Whoops
force-pushed
the
wh-run-queue-rework
branch
from
ea8b4ac to
be41fd4
Compare
| This check always returns healthy as we don't want to kill tasks based on the run queue length. | ||
| Instead it logs the maximum run queue length across all schedulers for monitoring purposes. |
There was a problem hiding this comment.
suggestion (non-blocking): if this isn't going to be used as a "real" health check anymore, does it make sense to maybe split this out and make it its own GenServer living somewhere in the supervision tree? The only issue is that I'm not sure where exactly that "somewhere" would be, especially in the context of the API umbrella app setup.
There was a problem hiding this comment.
I both agree, but didn't feel like it was worth the overhead. I can easily be convinced otherwise. The rest of the metrics we collect come from a reporter built into ehmon. We don't currently have another way of surfacing metrics so it'd mean building a way of scheduling this to run on some interval, which just seemed like more code than it was worth for this one metric.
| defp pid_log(other) do | ||
| inspect(other) | ||
| defp max_queue_length do | ||
| Enum.max(:erlang.statistics(:run_queue_lengths)) |
There was a problem hiding this comment.
nitpick (non-blocking): this could be a good pipeline operator candidate
lemald
left a comment
lemald
left a comment
There was a problem hiding this comment.
A couple of non-blocking things but overall LGTM.
Summary of changes
Asana Ticket: 🍎💳 Resolve API Run Queue Too High Alerts