Full DEV Reindex Fails Due to entity-api 500 Error (Unread result found)

[yuanzhou]

Full reindex DEV failed shortly after running the full reindex script via search-api.

[2026-08-25 01:21:32] INFO in fresh_indices: ############# Full index via script started at 01:21:31 #############
[2026-08-25 01:21:32] ERROR in hubmap_translator: [FULL] call_entity_api() failed to get entity of uuid 96a667104f92a38f5d4f97c38d94e738 via entity-api
NoneType: None
[2026-08-25 01:21:32] ERROR in hubmap_translator: [FULL] Exception during executing translate()
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 622, in enqueue_reindex
    entity = self.call_entity_api(entity_id=entity_id, endpoint_base='documents')
  File "/usr/src/app/src/hubmap_translator.py", line 1920, in call_entity_api
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/python3.13/lib/python3.13/site-packages/requests/models.py", line 1028, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://entity-api:8080/documents/96a667104f92a38f5d4f97c38d94e738
[2026-08-25 01:21:32] ERROR in hubmap_translator: [FULL] 500 Server Error: INTERNAL SERVER ERROR for url: http://entity-api:8080/documents/96a667104f92a38f5d4f97c38d94e738
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 272, in translate_full
    self.enqueue_reindex(uuid, reindex_queue, priority=1, index_override=index_override, job_type='FULL')
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/app/src/hubmap_translator.py", line 622, in enqueue_reindex
    entity = self.call_entity_api(entity_id=entity_id, endpoint_base='documents')
  File "/usr/src/app/src/hubmap_translator.py", line 1920, in call_entity_api
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/local/python3.13/lib/python3.13/site-packages/requests/models.py", line 1028, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://entity-api:8080/documents/96a667104f92a38f5d4f97c38d94e738
[2026-08-25 01:21:32] INFO in fresh_indices: 1 entity ids failed
[2026-08-25 01:21:32] DEBUG in fresh_indices: 96a667104f92a38f5d4f97c38d94e738
[2026-08-25 01:21:32] INFO in fresh_indices: ############# Full index job enqueuing complete at 01:21:32 #############
[2026-08-25 01:21:32] INFO in fresh_indices: ############# Full index job enqueuing took 00:00:01. #############

Both https://entity-api.dev.hubmapconsortium.org/entities/96a667104f92a38f5d4f97c38d94e738 and https://entity-api.dev.hubmapconsortium.org/documents/96a667104f92a38f5d4f97c38d94e738 returned

{
    "error": "500 Internal Server Error: Unexpected error: Unread result found"
}

The entity-api logs didn't reveal error details

[2026-08-25 01:27:45] INFO in schema_manager: Using HTTP response cache of GET http://uuid-api:8080/uuid/96a667104f92a38f5d4f97c38d94e738 at time 2026-08-25 01:27:45.449722
[pid: 295|app: 0|req: 20/23] 172.31.17.198 () {54 vars in 985 bytes} [Tue Aug 25 01:27:45 2026] GET /documents/96a667104f92a38f5d4f97c38d94e738 => generated 77 bytes in 1 msecs (HTTP/1.1 500) 2 headers in 90 bytes (1 switches on core 20)
[2026-08-25 01:27:50] INFO in schema_manager: Using HTTP response cache of GET http://uuid-api:8080/uuid/96a667104f92a38f5d4f97c38d94e738 at time 2026-08-25 01:27:50.733156
[pid: 295|app: 0|req: 21/24] 172.31.17.198 () {54 vars in 983 bytes} [Tue Aug 25 01:27:50 2026] GET /entities/96a667104f92a38f5d4f97c38d94e738 => generated 77 bytes in 1 msecs (HTTP/1.1 500) 2 headers in 90 bytes (1 switches on core 22)

[yuanzhou]

Had this same 500 error on https://entity-api.dev.hubmapconsortium.org/documents/dd258063b8389612a9596af734143906

[yuanzhou]

Finally found many of such errors in Cloudwatch insights:

SOURCE logGroups(namePrefix: [], class: "STANDARD") START=-604800s END=0s |
fields @timestamp, @message, @logGroupId, @log
| filter @message like "Unread result found"
| sort @timestamp desc
| limit 10000

One of the results shows the source is uuid-api:

2026-08-26T20:43:23.962Z  [2026-08-26 20:43:23] ERROR in app: Unread result found

2026-08-26T20:43:23.962Z  Traceback (most recent call last):

2026-08-26T20:43:23.962Z    File "/usr/src/app/src/app.py", line 169, in get_uuid

2026-08-26T20:43:23.962Z      info = worker.getIdInfo(uuid)

2026-08-26T20:43:23.962Z    File "/usr/src/app/src/uuid_worker.py", line 999, in getIdInfo

2026-08-26T20:43:23.962Z      with closing(dbConn.cursor(prepared=True)) as curs:

2026-08-26T20:43:23.962Z                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^

2026-08-26T20:43:23.962Z    File "/usr/local/lib/python3.13/site-packages/mysql/connector/connection_cext.py", line 771, in cursor

2026-08-26T20:43:23.962Z      self.handle_unread_result(prepared)

2026-08-26T20:43:23.962Z      ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^

2026-08-26T20:43:23.962Z    File "/usr/local/lib/python3.13/site-packages/mysql/connector/connection_cext.py", line 1047, in handle_unread_result

2026-08-26T20:43:23.962Z      raise InternalError("Unread result found")

2026-08-26T20:43:23.962Z  mysql.connector.errors.InternalError: Unread result found

2026-08-26T20:43:23.962Z  [pid: 49|app: 0|req: 37632/54913] 172.18.0.6 () {38 vars in 571 bytes} [Wed Aug 26 20:43:23 2026] GET /uuid/30398cd0f43dc9030e4066aa2d37681b => generated 37 bytes in 10 msecs (HTTP/1.1 500) 2 headers in 98 bytes (1 switches on core 4)