Check tokens created within all tokens processing time window

[stefangutica]

Reasoning

• getAllTokensRaw, run by the cache warmer every 2 minutes, takes tens of seconds: it fetches the token list once at the start and then processes every token. Any token created after that initial fetch is missing from the result until the next refresh.
• This also races with the token creation handler in the transaction processor. If warming starts at t1, a token is created at t2 and appended to the cached list by the transaction processor, and warming finishes at t3 (t1 < t2 < t3), the warmer overwrites the cache with a list that does not contain the new token. The token appears and then disappears until the next refresh.

Proposed Changes

• After the full processing pass, re-fetch the fungible token properties and MetaESDT collections, and process only the tokens that were not in the initial list. They are merged in before sorting.
• This narrows the window for both issues from the whole processing time (tens of seconds) to the duration of the second, much smaller pass (a few seconds at most).
• If the second fetch fails, the error is logged and the already processed list is still returned, so a failure there cannot discard the full refresh.
• Extracted the list fetch into fetchAllTokensWithoutDetails so both passes share it.

How to test

• Run npx jest src/test/unit/services/tokens.spec.ts. The new cases check that only tokens created in the meantime are processed in the second pass and that they are included in the result, and that a failing second fetch keeps the already processed tokens.
• On devnet, issue a token or register a MetaESDT collection while the All Tokens invalidations cron is running. The token should still be in GET /tokens after the warmer finishes, and the logs should show Processing N tokens created while processing all tokens.
• When no tokens are created during a refresh, GET /tokens returns the same count and order as before.

[github-actions]

k6 load testing comparison.
Base Commit Hash: 4a67c69
Target Commit Hash: 27fbe52

Metric	Base				Target				Diff
	Avg	Max	90	95	Avg	Max	90	95	Avg	Max	90	95
Nodes	68.89	15789.12	52.20	55.05	63.33	17112.46	59.40	62.59	-8.06% ✅	+8.38% 🔴	+13.79% 🔴	+13.69% 🔴
Blocks	42.92	837.04	52.70	56.97	47.19	708.76	58.09	61.97	+9.96% 🔴	-15.33% ✅	+10.21% 🔴	+8.78% 🔴
Transactions	73.24	14363.78	53.40	57.66	67.74	16092.62	59.34	61.96	-7.52% ✅	+12.04% 🔴	+11.13% 🔴	+7.46% 🔴
Mex	32.47	186.02	42.50	51.77	38.90	429.55	54.78	59.59	+19.80% 🔴	+130.91% 🔴	+28.90% 🔴	+15.12% 🔴
Tokens	32.68	582.95	42.97	51.87	39.12	512.35	54.92	59.68	+19.70% 🔴	-12.11% ✅	+27.81% 🔴	+15.07% 🔴
Accounts	34.10	1523.18	45.53	53.03	40.03	578.87	56.44	60.55	+17.37% 🔴	-62.00% ✅	+23.95% 🔴	+14.20% 🔴
Pool	32.68	524.78	42.97	51.77	39.13	490.34	54.87	59.72	+19.74% 🔴	-6.56% ✅	+27.71% 🔴	+15.36% 🔴
Test Run Duration	60002.77				60002.66

Legend: Avg - Average Response Time, Max - Maximum Response Time, 90 - 90th Percentile, 95 - 95th Percentile
All times are in milliseconds.