Registry: Neo4j out of memory

[Wenzel]

ERROR:neo4j:Failed to read from defunct connection IPv4Address(('localhost', 7687)) (ResolvedIPv4Address(('127.0.0.1', 7687)))
ERROR:root:Post Mortem: An unhandled exception occurred.
Traceback (most recent call last):
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_common.py", line 51, in _buffer_one_chunk
    receive_into_buffer(self._socket, self._buffer, 2)
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_common.py", line 306, in receive_into_buffer
    n = sock.recv_into(view[buffer.used:end], end - buffer.used)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_async_compat/network/_bolt_socket.py", line 493, in recv_into
    return self._wait_for_io(self._socket.recv_into, buffer, nbytes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_async_compat/network/_bolt_socket.py", line 468, in _wait_for_io
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/wenzel/Projets/grapheos-plugins/plugins/__main__.py", line 14, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/Projets/grapheos-plugins/plugins/__main__.py", line 67, in runner
    plugin(commit)
  File "/home/wenzel/Projets/grapheos-plugins/plugins/types.py", line 38, in __call__
    self.run(commit)
  File "/home/wenzel/Projets/grapheos-plugins/plugins/plugins/registry.py", line 140, in run
    node = self.dump_hive(hive_path, hive_local_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/Projets/grapheos-plugins/plugins/plugins/registry.py", line 165, in dump_hive
    self.insert_from_visited_node_cypher(node.return_value)
  File "/home/wenzel/Projets/grapheos-plugins/plugins/plugins/registry.py", line 172, in insert_from_visited_node_cypher
    self.insert_child_keys(node)
  File "/home/wenzel/Projets/grapheos-plugins/plugins/plugins/registry.py", line 221, in insert_child_keys
    self.neogit.db.cypher_query(query, {"parent_hash": node.hash, "unwind_param": unwind_param})
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neomodel/util.py", line 42, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neomodel/util.py", line 419, in cypher_query
    results, meta = self._run_cypher_query(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neomodel/util.py", line 455, in _run_cypher_query
    response = session.run(query, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/work/transaction.py", line 169, in run
    result._tx_ready_run(query, parameters)
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/work/result.py", line 131, in _tx_ready_run
    self._run(query, parameters, None, None, None, None, None, None)
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/work/result.py", line 181, in _run
    self._attach()
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/work/result.py", line 298, in _attach
    self._connection.fetch_message()
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_common.py", line 178, in inner
    func(*args, **kwargs)
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_bolt.py", line 846, in fetch_message
    tag, fields = self.inbox.pop(
                  ^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_common.py", line 72, in pop
    self._buffer_one_chunk()
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_common.py", line 68, in _buffer_one_chunk
    Util.callback(self.on_error, error)
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_async_compat/util.py", line 118, in callback
    return cb(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_bolt.py", line 872, in _set_defunct_read
    self._set_defunct(message, error=error, silent=silent)
  File "/home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_bolt.py", line 919, in _set_defunct
    raise ServiceUnavailable(message) from error
neo4j.exceptions.ServiceUnavailable: Failed to read from defunct connection IPv4Address(('localhost', 7687)) (ResolvedIPv4Address(('127.0.0.1', 7687)))
> /home/wenzel/.cache/pypoetry/virtualenvs/plugins-L2OWRlqL-py3.11/lib/python3.11/site-packages/neo4j/_sync/io/_bolt.py(919)_set_defunct()
    918             if error:
--> 919                 raise ServiceUnavailable(message) from error
    920             else:

ipdb> 

neo4j output

api-1    | Object types need a way to be resolved for field:  diffCommits
api-1    | 🚀 Server ready at http://localhost:4000/
neo4j-1  | Exception in thread "neo4j.Scheduler-1" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp168703427-289"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.Scheduler-1"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp168703427-287"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-33-thread-1"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JNA Cleaner"
neo4j-1  | Exception in thread "pool-35-thread-1" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | Exception in thread "qtp168703427-292" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | Exception in thread "Log4j2-TF-3-Scheduled-1" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.IndexSampling-9"
neo4j-1  | 2024-04-14 18:36:01.113+0000 ERROR Client triggered an unexpected error [Neo.DatabaseError.Statement.ExecutionFailed]: Java heap space, reference 6251f318-7da7-43e1-b86a-2c553a445892.

[Wenzel]

with neo4j 4.4.32:

poetry run runner a43419ee4637c981b31bdb8860baa6de932cb0b0 winreg 758,80s user 2100,43s system 98% cpu 48:12,29 total

[Wenzel]

post on neo4j community: https://community.neo4j.com/t/neo4j-5-18-100-on-all-cpus-followed-by-oom/67364/10

Hi !

I'm facing another issue after upgrading from 4.4.32 to 5.18.1.

One of the scripts i'm using to import and MERGE data ends up crashing, with an unresponsive data.

Basically I observe that at some point Neo4j takes 100% of the CPU (like almost 100 on every possible core).
At that point I event tried to execute some queries in the Neo4j browser, to understand what was going on, but the database wasn't responding to a simple SHOW INDEXES.

Furthermore my script is executing everything inside a transaction, and as far as I understand, transactions are still single executing on a core in Neo4j, so I don't get how it ends up using every possibles core at 100%.

api-1    | 🚀 Server ready at http://localhost:4000/
neo4j-1  | Exception in thread "neo4j.Scheduler-1" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp168703427-289"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.Scheduler-1"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp168703427-287"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-33-thread-1"
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JNA Cleaner"
neo4j-1  | Exception in thread "pool-35-thread-1" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | Exception in thread "qtp168703427-292" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | Exception in thread "Log4j2-TF-3-Scheduled-1" java.lang.OutOfMemoryError: Java heap space
neo4j-1  | 
neo4j-1  | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.IndexSampling-9"
neo4j-1  | 2024-04-14 18:36:01.113+0000 ERROR Client triggered an unexpected error [Neo.DatabaseError.Statement.ExecutionFailed]: Java heap space, reference 6251f318-7da7-43e1-b86a-2c553a445892.

Note: my script worked very well with Neo4j 4.4.32, so I don't understand what's happening here.

• Neo4j 5.18.1 (Docker)
• Python stack with neomodel and direct Cypher queries