When requesting really really big data sets, it seems the worker can get OOMkilled by kubernetes
I don't understand this, because the http request is async and should stream down, yeilding to other processes, and should give plenty of time for the child process OOMKill to kill in and gracefully kill it.
I suppose it's possible that the request completes quite happily and keeps all the data in memory (at an in-memory cost within tolerance). But when serializing the dataclip out of the child process, the synchronous JSON.stringify call (to measure the serialized dataclip size) is chewing up too much memory before the child process can catch it.
Yeah. That'll be it.
That stuff is being looked at over in #1366 - but that's proving quite a problematic PR in its own right.
Data can be downloaded from a public URL without credentials, although I won't share it here. Maybe check api/Facility/All in et-mfr, or DM me for the URL. Or just find a bunch of json elsewhere :)
When requesting really really big data sets, it seems the worker can get OOMkilled by kubernetes
I don't understand this, because the http request is async and should stream down, yeilding to other processes, and should give plenty of time for the child process OOMKill to kill in and gracefully kill it.
I suppose it's possible that the request completes quite happily and keeps all the data in memory (at an in-memory cost within tolerance). But when serializing the dataclip out of the child process, the synchronous JSON.stringify call (to measure the serialized dataclip size) is chewing up too much memory before the child process can catch it.
Yeah. That'll be it.
That stuff is being looked at over in #1366 - but that's proving quite a problematic PR in its own right.
Data can be downloaded from a public URL without credentials, although I won't share it here. Maybe check
api/Facility/Allinet-mfr, or DM me for the URL. Or just find a bunch of json elsewhere :)