Refactor libevwrapper cleanup situation

[fruch]

libevwrapper.py is using atexit module to do it's cleanup of global loop

from time to time we see the following crashes during python shutdown:

• Artifact test crashes on python shutdown (libevwrapper) scylla-cluster-tests#11713
• Sporadic SIGSEGV (exit code -11) in Python when exiting the Alternator test scylladb#17564 (suspect, since we don't collect there the dumps, to see it)

for years we had a workaround of some sort in SCT that was clearing the atexit hooks, blindly.
we removed it since it breaking other part that SCT or other part are using atexit.

and now we are getting this crash from time to time.

there two things that I think needs to improve:

1. ability to stop the libev loop, so we won't get into case it wakes up during python shutdown
2. refactor code to no use global loop and do cleanup directly without atexit

complete vibe code example of part one, it got compiled.
but didn't have a clear path (nor time) to test it

diff --git a/cassandra/io/libevwrapper.c b/cassandra/io/libevwrapper.c
index f32504fa3..0689bf3b9 100644
--- a/cassandra/io/libevwrapper.c
+++ b/cassandra/io/libevwrapper.c
@@ -6,6 +6,7 @@
 typedef struct libevwrapper_Loop {
     PyObject_HEAD
     struct ev_loop *loop;
+    ev_async async_watcher;
 } libevwrapper_Loop;

 static void
@@ -30,12 +31,24 @@ Loop_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {
     return (PyObject *)self;
 };

+static void async_stop_cb(EV_P_ ev_async *w, int revents) {
+    ev_break(EV_A_ EVBREAK_ALL);
+}
+
+static PyObject *
+Loop_stop(libevwrapper_Loop *self, PyObject *args) {
+    ev_async_send(self->loop, &self->async_watcher);
+    Py_RETURN_NONE;
+}
+
 static int
 Loop_init(libevwrapper_Loop *self, PyObject *args, PyObject *kwds) {
     if (!PyArg_ParseTuple(args, "")) {
         PyErr_SetString(PyExc_TypeError, "Loop.__init__() takes no arguments");
         return -1;
     }
+    ev_async_init(&self->async_watcher, async_stop_cb);
+    ev_async_start(self->loop, &self->async_watcher);
     return 0;
 };

@@ -56,6 +69,7 @@ Loop_unref(libevwrapper_Loop *self, PyObject *args) {
 static PyMethodDef Loop_methods[] = {
     {"start", (PyCFunction)Loop_start, METH_NOARGS, "Start the event loop"},
     {"unref", (PyCFunction)Loop_unref, METH_NOARGS, "Unrefrence the event loop"},
+    {"stop", (PyCFunction)Loop_stop, METH_NOARGS, "Stop the event loop from any thread"},
     {NULL} /* Sentinel */
 };

[fruch]

NOTE:
we are using libev backend across the board in testing, it's the most recommended and tested one from POV.
this might be the cause of miss-using or leaving opening connection behind, but even if that happens we should prevent crashing python shutdown

[fruch]

@absurdfarce are you familiar with such issues during the python shutdown ?

I did a quick search, and found any references

[absurdfarce]

Hey @fruch , thanks for the ping!

The closest I can get to what you describe is PYTHON-1400. That ticket was created from a user report of random crashes when using the libev reactor; like your case it was intermittent and essentially impossible to reliably repro. We wound up tracking it down to a CPython bug (which was reported here) that highlighted a known issue with PyEval_RestoreThread() when daemon threads are in play. This issue is still present in current Pythons and could theoretically kick in for any event loop... however a quick perusal of your other tickets doesn't show any obvious references to the PyThreadState struct essential to the problem.

I'm certainly not saying it's the root cause of your problem... but it might be an avenue to investigate.

[fruch]

@absurdfarce thanks, all fit with my suspicion that we have in flight request (in our tests that it happens, the Cassandra/scylla server is still up, when we shutdown the interpreter), and since we don't hold reference to the libev and stop the loop, it's callback can be called after/during shutdown.

[swasik]

@fruch #708 mentions that it fixes this issue. So could we close it or is something still left to be done?