Exception when inserting large text columns

[svr4]

Current Behavior

Hello, I'm having some intermittent issues using this package in AWS Fargate containers. Sometimes it works without a hitch and sometimes it doesn't work at all. I suspect certain row insertion payload sizes are causing this issue as I've faced text column insertion size issues locally. Here's the log output from AWS:

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 540, in _internal_socket_read
    data = sock.recv(1)
  File "/usr/local/lib/python3.13/ssl.py", line 1285, in recv
    return self.read(buflen)
           ~~~~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.13/ssl.py", line 1140, in read
    return self._sslobj.read(len)
           ~~~~~~~~~~~~~~~~~^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "//./main.py", line 304, in save_articles
    cursor.execute(add_story, _story)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/dbapi2.py", line 583, in execute
    result = self._driver.execute_statement(
        sql, parameters, self.connection.sqlitecloud_connection
    )
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 109, in execute_statement
    result = self._internal_run_command(connection, command)
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 487, in _internal_run_command
    return self._internal_socket_read(connection, main_socket)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 546, in _internal_socket_read
    raise SQLiteCloudException(
    ...<2 lines>...
    ) from exc
sqlitecloud.exceptions.SQLiteCloudException: An error occurred while reading command length from the socket.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 509, in _internal_socket_write
    sock.sendall(command)
    ~~~~~~~~~~~~^^^^^^^^^
  File "/usr/local/lib/python3.13/ssl.py", line 1263, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.13/ssl.py", line 1232, in send
    return self._sslobj.write(data)
           ~~~~~~~~~~~~~~~~~~^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "//./main.py", line 420, in <module>
    save_articles(conf)
    ~~~~~~~~~~~~~^^^^^^
  File "//./main.py", line 235, in save_articles
    with sqlite3.connect("database.db") if is_dev_env(conf) else sqlitecloud.connect(conf["SQLITE_URL"]) as cnx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/dbapi2.py", line 433, in __exit__
    self.rollback()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/dbapi2.py", line 377, in rollback
    self._driver.execute("ROLLBACK;", self.sqlitecloud_connection)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 92, in execute
    return self._internal_run_command(connection, command)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 486, in _internal_run_command
    self._internal_socket_write(connection, command, main_socket)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 511, in _internal_socket_write
    raise SQLiteCloudException(
    ...<2 lines>...
    ) from exc
sqlitecloud.exceptions.SQLiteCloudException: An error occurred while writing data.
An error occurred while writing data.

Preliminary Steps to Reproduce the Problem

1. Produce multiple insert statements with large text columns.
2. Try to insert.

Environment

• SQLiteCloud Version: 0.0.84
• Platform: AWS ARM/Linux
• Python Version: Python 3.13-slim
• Build Machine: MacBook Air M4, macOS Sequoia 15.6 (24G84)
• Docker Version: Docker Desktop 4.48.0 (207573); Engine 28.5.1; Compose v2.40.2-desktop.1

Any help is appreciated and if you need more information let me know.

[danielebriggi]

This error message reports an abruptly closure by the server and the reasons are many.
How large are the text columns you are sending?
When you say "multiple inserts statements" you mean multi inserts in a single execution (executemany(), executescript()? how many inserts do you send, if so?

[svr4]

Taking last month's data ingestion as a baseline we have:

Inserted records: 682
Average string length: 3509 characters
Average records inserted per code execution that produces data to insert: 5

Note: This is a task that runs hourly everyday and you could have a situation where as data is not being inserted it accumulates and you could have a batch to insert that's over 15 records, which locally runs fine.

When I say "multiple insert statements" I currently have a loop doing execute() but I used to have executemany() with the same issues.

Right now I'm seeing that between 5 and 6 AM AST I got more errors but then it started running fine. Here's the error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 540, in _internal_socket_read
    data = sock.recv(1)
  File "/usr/local/lib/python3.13/ssl.py", line 1285, in recv
    return self.read(buflen)
           ~~~~~~~~~^^^^^^^^
  File "/usr/local/lib/python3.13/ssl.py", line 1140, in read
    return self._sslobj.read(len)
           ~~~~~~~~~~~~~~~~~^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "//./main.py", line 304, in save_articles
    cursor.execute(add_story, _story)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/dbapi2.py", line 583, in execute
    result = self._driver.execute_statement(
        sql, parameters, self.connection.sqlitecloud_connection
    )
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 109, in execute_statement
    result = self._internal_run_command(connection, command)
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 487, in _internal_run_command
    return self._internal_socket_read(connection, main_socket)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 546, in _internal_socket_read
    raise SQLiteCloudException(
    ...<2 lines>...
    ) from exc
sqlitecloud.exceptions.SQLiteCloudException: An error occurred while reading command length from the socket.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 509, in _internal_socket_write
    sock.sendall(command)
    ~~~~~~~~~~~~^^^^^^^^^
  File "/usr/local/lib/python3.13/ssl.py", line 1263, in sendall
    v = self.send(byte_view[count:])
  File "/usr/local/lib/python3.13/ssl.py", line 1232, in send
    return self._sslobj.write(data)
           ~~~~~~~~~~~~~~~~~~^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "//./main.py", line 420, in <module>
    save_articles(conf)
    ~~~~~~~~~~~~~^^^^^^
  File "//./main.py", line 235, in save_articles
    with sqlite3.connect("database.db") if is_dev_env(conf) else sqlitecloud.connect(conf["SQLITE_URL"]) as cnx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/dbapi2.py", line 433, in __exit__
    self.rollback()
    ~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/dbapi2.py", line 377, in rollback
    self._driver.execute("ROLLBACK;", self.sqlitecloud_connection)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 92, in execute
    return self._internal_run_command(connection, command)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 486, in _internal_run_command
    self._internal_socket_write(connection, command, main_socket)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlitecloud/driver.py", line 511, in _internal_socket_write
    raise SQLiteCloudException(
    ...<2 lines>...
    ) from exc
sqlitecloud.exceptions.SQLiteCloudException: An error occurred while writing data.
An error occurred while writing data.

[danielebriggi]

Thank you for the details. Can you please send an email to support with the information of your project ID? We need to check if there are errors on your node.

.. it accumulates and you could have a batch to insert that's over 15 records, which locally runs fine.

you mean that the same amount of insertions run locally without any failure but on the remote Fairgate they sometimes fail?

[svr4]

That's exactly what I mean. I will email support and link to this thread. Thanks for the support.