[Bug] Unexpected Error - Error occurred during internal service call

[olw-medcom]

Hey there!

So we have quite a few test sessions where we try to view the user-submitted text. It is simply normal MULTILINE_TEXT request elements.
For example:

Clicking on it results in:

It seems to happen pretty sporadically, it's not all request MULTILINE_TEXT request elements it happens for. I can't find anything in the UI or SRV logs unfortunately.
Furthermore - when I download the test data for the session the data from that request element aren't even included. It has started to happen pretty recently for files that we were able to view last patch.

Best regards!

[costas80]

Hi @olw-medcom!

Lets try to get to the bottom of this:

• In the screenshots you shared I see that the test session is marked as an error. Is this an expected test session failure, or not? If not, is it possible to view (and share here) the test session logs?
• Normally anything server-side that could result in an error like this is logged in gitb-ui or gitb-srv. If you don't see anything there, it could be an error that is purely on the frontend (maybe a communication issue or a UI bug). If you can reproduce this, could you then share the browser's console log trace and network call status (e.g. is it a 404 error)?
• Finally, now that this came up for this specific test session, is this something you can always reproduce for the session in question when trying to view the step report?

While waiting for your feedback on these points we'll be checking on our end to see what could be the cause.

[costas80]

... it's not all request MULTILINE_TEXT request elements it happens for ...

This is good news. If it was all such elements then this would have been spotted already by us in development. What would be great debugging-wise is that once this occurs for a given test session, it is reproduced whenever you click to see the step report.

[olw-medcom]

1. Yeah the error was expected for this test session. We are experiencing the same thing for succeeded test sessions with no test logs

First error: XHR GET https://test.itb.medcom.dk/api/repository/reports/teststep/72b77339-295f-43ee-83a2-06cd32c6094b/1__SQS__F__SQE__.1.xml [HTTP/2 404 58ms] Error caught: {"error":{"error_description":"Error occurred during internal service call"}} main.js:85:4163 ERROR Object { headers: {…}, status: 404, statusText: "OK", url: "https://test.itb.medcom.dk/api/repository/reports/teststep/72b77339-295f-43ee-83a2-06cd32c6094b/1__SQS__F__SQE__.1.xml", ok: false, type: undefined, redirected: undefined, responseType: undefined, name: "HttpErrorResponse", message: "Http failure response for https://test.itb.medcom.dk/api/repository/reports/teststep/72b77339-295f-43ee-83a2-06cd32c6094b/1__SQS__F__SQE__.1.xml: 404 OK", … } main.js:6:23231

Second error:
Error caught: {"error":{"error_description":"Error occurred during internal service call"}} main.js:85:4163

Third error:
ERROR
Object { headers: {…}, status: 404, statusText: "OK", url: "https://test.itb.medcom.dk/api/repository/reports/teststep/72b77339-295f-43ee-83a2-06cd32c6094b/1__SQS__F__SQE__.1.xml", ok: false, type: undefined, redirected: undefined, responseType: undefined, name: "HttpErrorResponse", message: "Http failure response for https://test.itb.medcom.dk/api/repository/reports/teststep/72b77339-295f-43ee-83a2-06cd32c6094b/1__SQS__F__SQE__.1.xml: 404 OK", … }
main.js:6:23231

3. No, when I run it myself I'm able to view it afterwards.
   Viewing the test case history it looks like this:

Where the red means we can't view the data they have provided, and green means we can. Whether that's something to do with the time period they have submitted or something else, I don't know.

[costas80]

Ok - what you shared matches my investigation up to now: This is a 404 error because the step's report cannot be found. In addition, this is the only case where the backend will not log anything regarding this, and the reason why in the test data archive you would not see data for the test step.

The next question now is why - in these cases - is the step report not being generated. All the examples you shared indicate that this interact was in a conditional if block, and more specifically the else part. I'll look into this and let you know as soon as I have an update.

If you have time on your end, and readily available test sessions where the step report works fine, could you check to see if the cases that work are (a) not in conditional if steps, and (b) in case they are whether they are within the then branch (as opposed to the else)?

[olw-medcom]

Hey costas,

So the test setup where it works and where it doesn't are all within the same flow. It is true that they are within an <else> branch of an if but that is the case for all of the runs.
However upon investigating this it is weird...

This is the code:

<steps>
      <if static="true">
         <cond>$isAPITestExecution</cond>
         <then>
            <log>"Running as API execution"</log>
            <assign to="APIExecutionTestsHeaders{Content-Type}">"application/json"</assign>
            <log>"Please provide your Shared Note document with all optional fields filled out at the POST URL below. Please note that this is a static URL except the sessionId. "</log>
            <assign to="receiveInputBody">'OK'</assign>
            <receive id="receiveInput1"
                     desc="Receive inputs through the API"
                     handler="HttpMessagingV2"
                     to="TestBed"
                     from="ITBUser">
               <input name="method">"POST"</input>
               <input name="body">$receiveInputBody</input>
               <input name="status">"200"</input>
               <input name="headers">$APIExecutionTestsHeaders</input>
               <input name="uriExtension" asTemplate="true">"/receiveInput/providedFHIRDocument/${SESSION{sessionId}}"</input>
            </receive>
            <send id="sendToBeSureItStopsTheTest"
                  hidden="true"
                  from="TestBed"
                  to="TestBed"
                  handler="HttpMessagingV2">
               <input name="uri" asTemplate="true">
                "https://test.itb.medcom.dk/itbsrv/api/http/${SYSTEM{apiKey}}/instructions/${SESSION{sessionId}}"
              </input>
            </send>
            <assign to="step1UserData{providedFHIRDocument}">$receiveInput1{request}{body}</assign>
            <log>"Received the input 'providedFHIRDocument':" || $step1UserData{providedFHIRDocument}</log>
         </then>
         <else>
            <log>"Running through the UI - Did you know that you can run this through the API? See instructions in the testcase documentation by clicking the 'i' button"</log>
            <interact id="step1UserData"
                      desc="Step 1 - Provide Shared Note document for MedCom review">
               <request name="providedFHIRDocument"
                        desc="Please provide your Shared Note document with all optional fields filled out"
                        inputType="MULTILINE_TEXT"
                        size="15"
                        required="true"/>
            </interact>
         </else>
      </if>

And as you can see there is a <log> that states "Running through the UI ..."
And all of the tests where it fails the log does not include this. In fact the log is entirely empty? Normally all tests will have the following LOG [2026-09-07 07:00:13] INFO - Starting session atleast?

But as you can also see the <receive> aren't there either, so it's not because they have been running it other than through the UI?

[costas80]

Thanks @olw-medcom, that helps!

When you view a test session's result through the dashboard, the information is sourced from two locations:

• The ITB's DB. The overall result, executed steps and status for each step are loaded from here. This appears to be fine in the cases you shared.
• The ITB's filesystem. The specific test step reports, test session data (that goes into the archive), and test session log are loaded from here. This seems to be where the problem is coming from.

The fact that you get a 404 when requesting a test session report clearly points to a file system issue. In addition, when you say...

And all of the tests where it fails the log does not include this. In fact the log is entirely empty?

... means that likely the test session's folder cannot be found. Checking the implementation I see that in this case you wouldn't get a 404 but rather an empty response, resulting in a log displayed as follows:

I assume this is what you see on the UI right?

Now, coming back to the list of test sessions you highlighted (also very helpful). All affected test sessions are from August and this is important. The ITB every month automatically archives the test session data from the previous month to optimize storage. New test sessions that you run have their data directly recorded on the file system which is why they are working fine, and why you cannot replicate this with new test sessions. This could also be the reason why this seems to have happened recently (in September archival was triggered for August - during August itself these test sessions were not archived). What is odd however is that test sessions from June/July seem to be working fine, suggesting that there is no general issue with the data archival process.

To double-check this, could you please:

• Ensure that test sessions from earlier on indeed seem to be working fine in terms of retrieving their log and showing step reports?
• In case you have one available, check on a dev/test instance whether you see the same behavior relating to August and previous months?

I'm thinking that there could have been an error when the archival process ran for August that resulted in the August data not getting archived, but the ITB thinking it should be archived. To check this on the specific instance in question, the best way would be to:

• Obtain a shell on the gitb-ui pod.
• Go to folder /gitb-repository/status-updates
• Go to folder 2026 and check the contents. This should list the current month as a folder 9 and all other months in ZIP archives as 1.zip, 2.zip etc. It could be that 8 still appears as a folder and not as 8.zip.
• The contents of the folder or ZIP archive includes then folders per specific day, and within each of these, folders per specific test session. It would be great to see within this if the specific test session folder is present and check its contents.

But hey! Coming back to the very initial comment you made, you mention a test session from 02/09 (i.e. this month) which has the same issue, but then the list of test sessions does not include this. Is this indeed part of the problematic test cases? If you have already connected to the filesystem as described above, you could cross-check using the same approach whether the folder for the specific test session is present and whether its contents are as expected.

[costas80]

Yup, I hacked one of my ITB instances and replicated 100% the behavior you are seeing. For the hack I went to the status-updates folder and renamed the folder for a given day. This results in the same errors when clicking to view step reports and empty log contents.

The question now is to see where the problem comes from:

• Was there an issue persisting test session data in the filesystem to begin with? If you check the test sessions in question and find no trace on the filesystem, this would be the root issue (concerning, as we don't know yet when/why this would happen - however you said that "It has started to happen pretty recently for files that we were able to view last patch." which suggests this is not the cause).
• Was there an issue with the automated archival process that failed? If for example you find a 8 folder and not a 8.zip this could well be the case (and one we can more easily explain and address). However, your first example you mentioned is from the current month which does not involve the archival at all (good to cross-check if you indeed get this for September test sessions).
• Is there an issue with the lookup process (with the underlying data all being present)? Highly unlikely as the observed behaviour matches exactly what you would get for a missing (or mis-archived) data folder.

[olw-medcom]

Roger, I unfortunately don't have a shell access to the gitb-ui pod, but I have requested a view of the file tree within the folder you've mentioned.
I neither have a test setup :/

This just to explain why it make take awhile before getting back to this issue ^^

But hey! Coming back to the very initial comment you made, you mention a test session from 02/09 (i.e. this month) which has the same issue, but then the list of test sessions does not include this. Is this indeed part of the problematic test cases?

Indeed the test case from september also have the problem :/ here however there is a bit of a log output:
[2026-09-02 06:40:36] INFO - Session finished with result [ERROR]

Weirdly enough it doesn't write a 'session started' log.

[costas80]

Weirdly enough it doesn't write a 'session started' log.

Hmm this is odd. Could it be that the test session in question started in August and completed in September (thus making it also related to whatever happened in the August-to-September transition)? In general, are the tests in question maybe starting in month A and completing in month B? It could be also that some environment/operational glitch happened in August that bonked filesystem persistence...

I'm also curious to see whether there were any errors going back in time. I'm thinking odd logged errors from August (if you go so far back), or from when you restarted the last time in September. When the archival process runs you get log entries such as Archived test session folder for year [2026] and month [8]. It would be interesting to see when this ran for your instance and if anything fishy was reported aroung that.

On my end, after reviewing the archival process I think there could well be an issue for test sessions that are still pending completion when the archival process runs. In case test sessions are pending for a long time, I can well imagine that the archival running in the meantime could result in odd consequences (step reports not getting written or getting written when the relevant month has already been archived, logs missing or being incomplete depending on when they were flushed etc.). I will need to replicate this in a test environment to determine exactly what could happen and when, but I'm certain that there is a correction/safeguard needed here to prevent issues. Also, I'm not surprised we haven't heard about this from other users because most projects run test sessions that are fairly short lived.

In the meantime lets see also what your operations team reports back. The ideal in fact would be (if possible) that you ask them to make a listing of the status-updates/2026 folder, and share with you an exported copy of the contents of status-updates/2026, specifically folders and archives named 8, 9, 8.zip and 9.zip (if the entire folder is too big). Like that we can check all sorts of odd scenarios without going back and forth to them.

[olw-medcom]

Hey @costas80 ,

I got some more info here:

root@gitb-ui-7bcb4bb976-dl2nk:/gitb-repository/status-updates/2026# ls -R
.:
1.zip  2.zip  3.zip  4.zip  5.zip  6.zip  7.zip  8  8.zip  9
./8:
21
./8/21:
72b77339-295f-43ee-83a2-06cd32c6094b
./8/21/72b77339-295f-43ee-83a2-06cd32c6094b:
./9:
1  2  3  4  7
./9/1:
13eea718-1ac7-4e88-8762-74f5da158e3b  204c6514-c693-47e4-a185-bf388a5416e0  4ad54ab5-1b8a-4b6d-a657-646665be9670  60d1845e-16e1-46a1-a165-98ca2e7a7c57  6c7d8070-4e81-4088-b8a4-ad11737cdc7d  8b36c636-bcf6-4358-9dd5-a74e0d6f6628  abfbeac8-1793-4682-b627-48a57a3ed68d  de275eb7-fcdf-45a7-bcbb-6e5bbb14607b  fd6bba44-a2a9-4978-ac25-f599b2dc8ada
[... more down here from each day in september]

It seems indeed the 8 is still there and the 8.zip is also there?

Could it be that the test session in question started in August and completed in September (thus making it also related to whatever happened in the August-to-September transition)?

Absolutely, it is most likely what happened. It was started in september (which I can't verify for some reason I can't find the start time in the session dashboard anymore?) and then we completed it here in august with our admin interact elements.

I unfortunately can't easily access the logs from august, because the pods are regularly updated where the log from my point of view is wiped :(
Lemme know if you need me to ask the my vendor to provide the logs from previous pods.

[olw-medcom]

And just to add: The session id for the session that started in august and finished in september is 786be33e-25d0-4fa7-8630-3af20356ed60

[costas80]

Absolutely, it is most likely what happened.

Ah I think it all makes sense now! Let me explain:

1. The test sessions from August completed just fine in August and everything was recorded as expected. Accessing reports and logs worked fine before the archival process kicked in.
2. The last 72b77339-295f-43ee-83a2-06cd32c6094b session started in August and also started writing its file system output to the /8 folder location. Critically, this test session has not completed yet.
3. In September the archival process ran, ignored the fact that there were pending sessions, and archived /8 as /8.zip.
4. You then completed the pending test session at which point it wrote its pending state. In doing so it defensively recreated the missing /8 folder and wrote the pending state into it.

This explains perfectly the odd behavior you see:

• Reviewing the 72b77339-295f-43ee-83a2-06cd32c6094b session you get a half-empty log file with only the final log output present. I'm guessing also that this also includes the step report for the admin interaction you did to complete it. The start of the log and the previous interaction report which fails are in the /8.zip archive.
• When you retrieve a step's report or log, the ITB looks first for a non-archived month folder and then for an archived one. As it finds /8 it doesn't think that it would be incomplete - it proceeds to read from it but the files its looking for are no longer there. In addition, as it found a non-archived folder it never attempts to look for an archive.

So basically the fix for this is to avoid running the archival process while there are active test sessions. To correct the current issues you could manually do the following:

• Tell your operations team to give you the content of /8 and /8.zip.
• You merge the content of /8 into /8.zip.
• You give the updated /8.zip back to the ops team telling them to (a) replace the /8.zip archive and (b) delete the /8 folder.

Doing the above will restore the problematic test sessions. Alternatively you could wait for a patch that will:

• Adapt the archival process to ensure it never runs while active sessions are running (and blocks new tests until finished).
• Do a sanity check pass verifying that there is no folder plus archive for a given month and if so, does the discussed merge and folder deletion automatically.

I can't find the start time in the session dashboard anymore?

Ah yes! We now show the end time by default in the session dashboard for completed test sessions to avoid showing superfluous columns and restricting space for more meaningful ones. When this update was done however we overlooked to include the start and end time in the More popup, and also overlooked including the Start time in the displayed column preferences.

I'll ping you when both these patches are available to pull (archival issues plus start time column).

P.S: While waiting for the update could you please ask your operations team to provide you the contents of /8 and /8.zip? Like that we can 100% confirm that what I described above was the cause (plus you can choose to fix this immediately as discussed).

[olw-medcom]

Hey @costas80,
Fantastic that we got to the cause of it! I will try and reach out to my operations team and then let's see what takes the longest, your patch or the manual fix 😅

ask your operations team to provide you the contents of /8 and /8.zip?

Absolutely! Will do and report back to you

When this update was done however we overlooked to include the start and end time in the More popup, and also overlooked including the Start time in the displayed column preferences

Perfect, I'm glad I wasn't blind 😆

I'm looking forward to the patches! Thanks as always ^^