regression

[carlosreyes210177]

Is it possible to revert the "Horse.Provider.Console.pas" unit to how it was in version 3.2?
It becomes unstable due to changes introduced in version 3.3.0.

[regyssilveira]

Obrigado pelo relato. Para conseguirmos localizar a regressão sem reverter correções importantes do provider Console, poderia complementar com:

• versão exata do Delphi ou FPC e sistema operacional;
• versão mais recente do Horse em que o comportamento ainda funcionava;
• o que significa “instável” no seu cenário: exceção, travamento, encerramento, perda de conexão ou comportamento incorreto;
• mensagem de erro, stack trace ou log;
• configuração usada (Listen, porta/host, SSL, keep-alive, múltiplas instâncias etc.);
• um projeto mínimo ou trecho de código que reproduza o problema;
• se possível, resultado do mesmo teste nas versões 3.2.0, 3.3.0 e 3.3.8.

Com esses dados conseguimos identificar a mudança específica e criar um teste de regressão antes de alterar Horse.Provider.Console.pas.

[carlosreyes210177]

Delphi 12.3
Windows Server 2022
here is a Windows service using the aforementioned drive.
To be honest, it is difficult to provide a specific example project that demonstrates the "instability," but here is what is happening:
We have over 50 clients connecting to Horse for more than 8 hours and generating 75,000 transactions; with version 3.2, we have no issues—it is stable and works perfectly.
When we switched to version 3.3.0, we started noticing problems: not all clients could connect, or connections would hang or get stuck in a waiting state. When transactions were sent to the server, they took too long or sometimes hung completely, or we would get a 500 error.
We tested versions 3.3.0 through 3.3.8.
When we reverted to version 3.2, we didn't experience any similar issues.
Upon reviewing and comparing the "Horse.Provider.Console.pas" unit from version 3.2 with the new versions,
we noticed that it had been modified.
We made some changes to the unit for testing purposes, then reverted those changes and compiled using version 3.3.8; we haven't encountered the issues mentioned since then, which is why I asked if we could revert that specific unit.

modified unit from version 3.3.8
unit Horse.Provider.Console;

{ ===========================================================================
PATCH-CONSOLE-1 — ListenWithConfig override
Without this, THorseProviderAbstract.ListenWithConfig just calls the no-arg
Listen, which enters InternalListen with FPort = 0 and falls back to
DEFAULT_PORT (9000), silently ignoring the caller-supplied APort.
Fix: override ListenWithConfig in THorseProvider (Console) so that it sets
Console's own FPort via SetPort before calling InternalListen.
AConfig is intentionally ignored — Console/Indy has no use for
THorseCrossSocketConfig; only CrossSocket overrides ListenWithConfig fully.
=========================================================================== }

interface

{$IF NOT DEFINED(FPC)}
uses
Horse.Provider.Abstract,
{$IFNDEF MSWINDOWS}
{ PATCH-CONSOLE-1: Horse.Provider.Config needed for THorseCrossSocketConfig }
Horse.Provider.Config,
{$ENDIF}
{ =========================================================================== }
Horse.Provider.IOHandleSSL.Contract,
IdHTTPWebBrokerBridge,
IdContext,
System.SyncObjs,
System.SysUtils,
Horse.Core;

type
THorseProvider = class(THorseProviderAbstract)
private
class var FPort: Integer;
class var FHost: string;
class var FRunning: Boolean;
class var FEvent: TEvent;
class var FMaxConnections: Integer;
class var FListenQueue: Integer;
class var FKeepConnectionAlive: Boolean;
class var FIdHTTPWebBrokerBridge: TIdHTTPWebBrokerBridge;
class var FHorseProviderIOHandleSSL: IHorseProviderIOHandleSSL;
class function GetDefaultHTTPWebBroker: TIdHTTPWebBrokerBridge;
class function GetDefaultHorseProviderIOHandleSSL: IHorseProviderIOHandleSSL;
class function GetDefaultEvent: TEvent;
class function HTTPWebBrokerIsNil: Boolean;
class procedure OnAuthentication(AContext: TIdContext; const AAuthType, AAuthData: String; var VUsername, VPassword: String; var VHandled: Boolean);
class procedure OnQuerySSLPort(APort: Word; var VUseSSL: Boolean);
{$IFNDEF MSWINDOWS}
class procedure OnConnect(AContext: TIdContext);
{$ENDIF}
class procedure SetListenQueue(const AValue: Integer); static;
class procedure SetMaxConnections(const AValue: Integer); static;
class procedure SetPort(const AValue: Integer); static;
class procedure SetIOHandleSSL(const AValue: IHorseProviderIOHandleSSL); static;
class procedure SetHost(const AValue: string); static;
class procedure SetKeepConnectionAlive(const AValue: Boolean); static;
class function GetListenQueue: Integer; static;
class function GetMaxConnections: Integer; static;
class function GetPort: Integer; static;
class function GetDefaultPort: Integer; static;
class function GetDefaultHost: string; static;
class function GetIOHandleSSL: IHorseProviderIOHandleSSL; static;
class function GetHost: string; static;
class function GetKeepConnectionAlive: Boolean; static;
class procedure InternalListen; virtual;
class procedure InternalStopListen; virtual;
class procedure InitServerIOHandlerSSLOpenSSL(const AIdHTTPWebBrokerBridge: TIdHTTPWebBrokerBridge; const AHorseProviderIOHandleSSL: IHorseProviderIOHandleSSL);
public
class property Host: string read GetHost write SetHost;
class property Port: Integer read GetPort write SetPort;
class property MaxConnections: Integer read GetMaxConnections write SetMaxConnections;
class property ListenQueue: Integer read GetListenQueue write SetListenQueue;
class property KeepConnectionAlive: Boolean read GetKeepConnectionAlive write SetKeepConnectionAlive;
class property IOHandleSSL: IHorseProviderIOHandleSSL read GetIOHandleSSL write SetIOHandleSSL;
{$IFNDEF MSWINDOWS}
class function GetActivePort: Integer; override;
class procedure TriggerBeforeListen; override;
class procedure TriggerAfterListen; override;
class procedure TriggerBeforeStop; override;
class procedure TriggerAfterStop; override;
class procedure StopListenGraceful(const ATimeoutMS: Integer = 5000); override;
{$ENDIF}
class procedure StopListen; override;
class procedure Listen; overload; override;
class procedure Listen(const APort: Integer; const AHost: string = '0.0.0.0'; const ACallbackListen: TProc = nil; const ACallbackStopListen: TProc = nil); reintroduce; overload; static;
class procedure Listen(const APort: Integer; const ACallbackListen: TProc; const ACallbackStopListen: TProc = nil); reintroduce; overload; static;
class procedure Listen(const AHost: string; const ACallbackListen: TProc = nil; const ACallbackStopListen: TProc = nil); reintroduce; overload; static;
class procedure Listen(const ACallbackListen: TProc; const ACallbackStopListen: TProc = nil); reintroduce; overload; static;
{$IFNDEF MSWINDOWS}
{ ===========================================================================
PATCH-CONSOLE-1 — ListenWithConfig declaration
=========================================================================== }
class procedure ListenWithConfig(const APort: Integer; const AConfig: THorseCrossSocketConfig); override;
{ =========================================================================== }
{$ENDIF}
class function IsRunning: Boolean;
class destructor UnInitialize;
end;
{$ENDIF}

implementation

{$IF NOT DEFINED(FPC)}
uses
Web.WebReq,
Horse.WebModule,
IdCustomTCPServer,
Horse.Constants,
Horse.Provider.IOHandleSSL,
IdSSLOpenSSL
{$IFNDEF MSWINDOWS}
,IdSchedulerOfThreadPool,
System.Classes,
IdSocketHandle,
Horse.Core.Base,
Horse.Instance
{$ENDIF}
;

class function THorseProvider.GetDefaultHTTPWebBroker: TIdHTTPWebBrokerBridge;
begin
if HTTPWebBrokerIsNil then
begin
FIdHTTPWebBrokerBridge := TIdHTTPWebBrokerBridge.Create(nil);
{$IFNDEF MSWINDOWS}
FIdHTTPWebBrokerBridge.Scheduler := TIdSchedulerOfThreadPool.Create(FIdHTTPWebBrokerBridge);
FIdHTTPWebBrokerBridge.OnConnect := OnConnect;
{$ENDIF}
FIdHTTPWebBrokerBridge.OnParseAuthentication := OnAuthentication;
FIdHTTPWebBrokerBridge.OnQuerySSLPort := OnQuerySSLPort;

end;
Result := FIdHTTPWebBrokerBridge;
end;

class function THorseProvider.HTTPWebBrokerIsNil: Boolean;
begin
Result := FIdHTTPWebBrokerBridge = nil;
end;

class procedure THorseProvider.OnQuerySSLPort(APort: Word; var VUseSSL: Boolean);
begin
VUseSSL := (FHorseProviderIOHandleSSL <> nil) and FHorseProviderIOHandleSSL.Active;
end;
{$IFNDEF MSWINDOWS}
{ Disable Nagle (TCP_NODELAY) on each accepted connection. Without it, on Linux
loopback the keep-alive request/response ping-pong collides with the ~40 ms
delayed-ACK timer -> a flat ~44 ms/request floor that cripples throughput.
Harmless (beneficial) on Windows. See bench-analysis-report.md §7.5. }
class procedure THorseProvider.OnConnect(AContext: TIdContext);
begin
AContext.Binding.UseNagle := False;
AContext.Binding.SetKeepAliveValues(True, 60000, 1000);
if THorseProviderAbstract.ReadTimeout > 0 then
AContext.Connection.Socket.ReadTimeout := THorseProviderAbstract.ReadTimeout;
end;
{$ENDIF}

class function THorseProvider.GetDefaultEvent: TEvent;
begin
if FEvent = nil then
FEvent := TEvent.Create;
Result := FEvent;
end;

class function THorseProvider.GetDefaultHorseProviderIOHandleSSL: IHorseProviderIOHandleSSL;
begin
if FHorseProviderIOHandleSSL = nil then
FHorseProviderIOHandleSSL := THorseProviderIOHandleSSL.New;
Result := FHorseProviderIOHandleSSL;
end;

class function THorseProvider.GetDefaultHost: string;
begin
Result := DEFAULT_HOST;
end;

class function THorseProvider.GetDefaultPort: Integer;
begin
Result := DEFAULT_PORT;
end;

class function THorseProvider.GetHost: string;
begin
Result := FHost;
end;

class function THorseProvider.GetKeepConnectionAlive: Boolean;
begin
Result := FKeepConnectionAlive;
end;

class function THorseProvider.IsRunning: Boolean;
begin
Result := FRunning;
end;

class function THorseProvider.GetIOHandleSSL: IHorseProviderIOHandleSSL;
begin
Result := GetDefaultHorseProviderIOHandleSSL;
end;

class function THorseProvider.GetListenQueue: Integer;
begin
Result := FListenQueue;
end;

class function THorseProvider.GetMaxConnections: Integer;
begin
Result := FMaxConnections;
end;

class function THorseProvider.GetPort: Integer;
begin
Result := FPort;
end;

{$IFNDEF MSWINDOWS}
class function THorseProvider.GetActivePort: Integer;
begin
Result := FPort;
end;

class procedure THorseProvider.TriggerBeforeListen;
begin
inherited;
end;

class procedure THorseProvider.TriggerAfterListen;
begin
inherited;
end;

class procedure THorseProvider.TriggerBeforeStop;
begin
inherited;
end;

class procedure THorseProvider.TriggerAfterStop;
begin
inherited;
end;
{$ENDIF}
class procedure THorseProvider.InitServerIOHandlerSSLOpenSSL(const AIdHTTPWebBrokerBridge: TIdHTTPWebBrokerBridge; const AHorseProviderIOHandleSSL: IHorseProviderIOHandleSSL);
var
LIOHandleSSL: TIdServerIOHandlerSSLOpenSSL;
begin
LIOHandleSSL := TIdServerIOHandlerSSLOpenSSL.Create(AIdHTTPWebBrokerBridge);
LIOHandleSSL.SSLOptions.CertFile := AHorseProviderIOHandleSSL.CertFile;
LIOHandleSSL.SSLOptions.RootCertFile := AHorseProviderIOHandleSSL.RootCertFile;
LIOHandleSSL.SSLOptions.KeyFile := AHorseProviderIOHandleSSL.KeyFile;
LIOHandleSSL.SSLOptions.Method := AHorseProviderIOHandleSSL.Method;
LIOHandleSSL.SSLOptions.SSLVersions := AHorseProviderIOHandleSSL.SSLVersions;
LIOHandleSSL.SSLOptions.CipherList := AHorseProviderIOHandleSSL.CipherList;
LIOHandleSSL.SSLOptions.DHParamsFile := AHorseProviderIOHandleSSL.DHParamsFile;
LIOHandleSSL.OnGetPassword := AHorseProviderIOHandleSSL.OnGetPassword;
AIdHTTPWebBrokerBridge.IOHandler := LIOHandleSSL;
end;

{$IFDEF MSWINDOWS}
class procedure THorseProvider.InternalListen;
var
LAttach: string;
LIdHTTPWebBrokerBridge: TIdHTTPWebBrokerBridge;
begin
inherited;
if FPort <= 0 then
FPort := GetDefaultPort;

if FHost.IsEmpty then
FHost := GetDefaultHost;

LIdHTTPWebBrokerBridge := GetDefaultHTTPWebBroker;
WebRequestHandler.WebModuleClass := WebModuleClass;
try
if FMaxConnections > 0 then
begin
WebRequestHandler.MaxConnections := FMaxConnections;
GetDefaultHTTPWebBroker.MaxConnections := FMaxConnections;
end;

if FListenQueue = 0 then
  FListenQueue := IdListenQueueDefault;

if FHorseProviderIOHandleSSL <> nil then
  InitServerIOHandlerSSLOpenSSL(LIdHTTPWebBrokerBridge, GetDefaultHorseProviderIOHandleSSL);
LIdHTTPWebBrokerBridge.ListenQueue := FListenQueue;

LIdHTTPWebBrokerBridge.Bindings.Clear;
if FHost <> GetDefaultHost then
begin
  LIdHTTPWebBrokerBridge.Bindings.Add;
  LIdHTTPWebBrokerBridge.Bindings.Items[0].IP := FHost;
  LIdHTTPWebBrokerBridge.Bindings.Items[0].Port := FPort;
end;

LIdHTTPWebBrokerBridge.KeepAlive := FKeepConnectionAlive;
LIdHTTPWebBrokerBridge.DefaultPort := FPort;
LIdHTTPWebBrokerBridge.Active := True;
LIdHTTPWebBrokerBridge.StartListening;
FRunning := True;
DoOnListen;

if IsConsole then
begin
  while FRunning do
    GetDefaultEvent.WaitFor();
end

except
on E: Exception do
begin
if IsConsole then
begin
Writeln(E.ClassName, ': ', E.Message);
Read(LAttach);
end
else
{$IF CompilerVersion >= 32.0}
raise AcquireExceptionObject;
{$ELSE}
raise;
{$ENDIF}
end;
end;
end;

{$ELSE}
class procedure THorseProvider.InternalListen;
var
LAttach: string;
LIdHTTPWebBrokerBridge: TIdHTTPWebBrokerBridge;
LPortKey: Integer;

LInstance: THorseInstance;
LBinding: TIdSocketHandle;
begin
if FPort <= 0 then
FPort := GetDefaultPort;

if FHost.IsEmpty then
FHost := GetDefaultHost;

LIdHTTPWebBrokerBridge := GetDefaultHTTPWebBroker;
WebRequestHandler.WebModuleClass := WebModuleClass;
try
if FMaxConnections > 0 then
begin
WebRequestHandler.MaxConnections := FMaxConnections;
GetDefaultHTTPWebBroker.MaxConnections := FMaxConnections;
end;

if FListenQueue = 0 then
  FListenQueue := IdListenQueueDefault;

if FHorseProviderIOHandleSSL <> nil then
  InitServerIOHandlerSSLOpenSSL(LIdHTTPWebBrokerBridge, GetDefaultHorseProviderIOHandleSSL);
LIdHTTPWebBrokerBridge.ListenQueue := FListenQueue;

LIdHTTPWebBrokerBridge.Bindings.Clear;
if (GInstances <> nil) and (GInstances.Count > 0) then
begin
  GInstancesLock.Enter;
  try
    for LPortKey in GInstances.Keys do
    begin
      LBinding := LIdHTTPWebBrokerBridge.Bindings.Add;
      LInstance := THorseInstance(GInstances.Items[LPortKey]);
      if LInstance.Host.IsEmpty then
        LBinding.IP := '127.0.0.1'
      else
        LBinding.IP := LInstance.Host;
      LBinding.Port := LPortKey;
    end;
  finally
    GInstancesLock.Leave;
  end;
end
else
begin
  LBinding := LIdHTTPWebBrokerBridge.Bindings.Add;
  if FHost.IsEmpty then
    LBinding.IP := '127.0.0.1'
  else
    LBinding.IP := FHost;
  LBinding.Port := FPort;
end;

TriggerBeforeListen;
LIdHTTPWebBrokerBridge.KeepAlive := FKeepConnectionAlive;
LIdHTTPWebBrokerBridge.DefaultPort := FPort;
if LIdHTTPWebBrokerBridge.Active then
  LIdHTTPWebBrokerBridge.Active := False;
LIdHTTPWebBrokerBridge.Active := True;
LIdHTTPWebBrokerBridge.StartListening;
FRunning := True;
DoOnListen;

if IsConsole then
begin
  while FRunning do
    GetDefaultEvent.WaitFor();
end;

except
on E: Exception do
begin
if IsConsole and (Trim(GetEnvironmentVariable('HORSE_TEST_SILENCE')) <> '1') and not FindCmdLineSwitch('silence', True) and not FindCmdLineSwitch('non-interactive', True) then
begin
Writeln(E.ClassName, ': ', E.Message);
Read(LAttach);
end;
raise;
end;
end;
end;
{$ENDIF}

{$IFDEF MSWINDOWS}
class procedure THorseProvider.InternalStopListen;
begin
if not HTTPWebBrokerIsNil then
begin
GetDefaultHTTPWebBroker.StopListening;
GetDefaultHTTPWebBroker.Active := False;
DoOnStopListen;
FRunning := False;
if FEvent <> nil then
GetDefaultEvent.SetEvent;
end
else
raise Exception.Create('Horse not listen');
end;
{$ELSE}
class procedure THorseProvider.InternalStopListen;
var
LContexts: TList;
I: Integer;
begin
TriggerBeforeStop;
if not HTTPWebBrokerIsNil then
begin
GetDefaultHTTPWebBroker.StopListening;
try
LContexts := GetDefaultHTTPWebBroker.Contexts.LockList;
try
for I := LContexts.Count - 1 downto 0 do
begin
try
TIdContext(LContexts[I]).Connection.Disconnect;
except
end;
end;
finally
GetDefaultHTTPWebBroker.Contexts.UnlockList;
end;
except
end;
GetDefaultHTTPWebBroker.Active := False;
DoOnStopListen;
FRunning := False;
if FEvent <> nil then
GetDefaultEvent.SetEvent;
end
else
raise Exception.Create('Horse not listen');
end;
{$ENDIF}
class procedure THorseProvider.StopListen;
begin
InternalStopListen;
end;
{$IFNDEF MSWINDOWS}
class procedure THorseProvider.StopListenGraceful(const ATimeoutMS: Integer);
var
LStart: Int64;
LContexts: TList;
LCount: Integer;
begin
TriggerBeforeStop;
if not HTTPWebBrokerIsNil then
begin
THorseCore.SetIsShuttingDown(True);
try
GetDefaultHTTPWebBroker.StopListening;

  LStart := TThread.GetTickCount;
  while (TThread.GetTickCount - LStart < ATimeoutMS) do
  begin
    LCount := 0;
    try
      LContexts := GetDefaultHTTPWebBroker.Contexts.LockList;
      try
        LCount := LContexts.Count;
      finally
        GetDefaultHTTPWebBroker.Contexts.UnlockList;
      end;
    except
    end;

    if (THorseCore.GetActiveRequests = 0) and (LCount = 0) then
      Break;

    TThread.Sleep(50);
  end;
  
  TThread.Sleep(100);
  
  GetDefaultHTTPWebBroker.Active := False;
  DoOnStopListen;
  FRunning := False;
  if FEvent <> nil then
    GetDefaultEvent.SetEvent;
finally
  THorseCore.SetIsShuttingDown(False);
end;

end
else
raise Exception.Create('Horse not listen');
end;
{$ENDIF}

class procedure THorseProvider.Listen;
begin
InternalListen;
end;

class procedure THorseProvider.Listen(const APort: Integer; const AHost: string; const ACallbackListen, ACallbackStopListen: TProc);
begin
SetPort(APort);
SetHost(AHost);
SetOnListen(ACallbackListen);
SetOnStopListen(ACallbackStopListen);
InternalListen;
end;

class procedure THorseProvider.Listen(const AHost: string; const ACallbackListen, ACallbackStopListen: TProc);
begin
Listen(FPort, AHost, ACallbackListen, ACallbackStopListen);
end;

class procedure THorseProvider.Listen(const ACallbackListen, ACallbackStopListen: TProc);
begin
Listen(FPort, FHost, ACallbackListen, ACallbackStopListen);
end;

class procedure THorseProvider.Listen(const APort: Integer; const ACallbackListen, ACallbackStopListen: TProc);
begin
Listen(APort, FHost, ACallbackListen, ACallbackStopListen);
end;
{$IFNDEF MSWINDOWS}
{ ===========================================================================
PATCH-CONSOLE-1 — ListenWithConfig implementation
Sets Console's own FPort before starting, so the port is honoured even when
the caller goes through the abstract ListenWithConfig entry point.
AConfig is intentionally ignored — Indy/Console has no use for CrossSocket
configuration. CrossSocket overrides ListenWithConfig completely.
=========================================================================== }
class procedure THorseProvider.ListenWithConfig(const APort: Integer; const AConfig: THorseCrossSocketConfig);
begin
SetPort(APort);
InternalListen;
end;
{ =========================================================================== }
{$ENDIF}
class procedure THorseProvider.OnAuthentication(AContext: TIdContext; const AAuthType, AAuthData: String; var VUsername, VPassword: String; var VHandled: Boolean);
begin
VHandled := True;
end;

class procedure THorseProvider.SetHost(const AValue: string);
begin
FHost := AValue.Trim;
end;

class procedure THorseProvider.SetKeepConnectionAlive(const AValue: Boolean);
begin
FKeepConnectionAlive := AValue;
end;

class procedure THorseProvider.SetIOHandleSSL(const AValue: IHorseProviderIOHandleSSL);
begin
FHorseProviderIOHandleSSL := AValue;
end;

class procedure THorseProvider.SetListenQueue(const AValue: Integer);
begin
FListenQueue := AValue;
end;

class procedure THorseProvider.SetMaxConnections(const AValue: Integer);
begin
FMaxConnections := AValue;
end;

class procedure THorseProvider.SetPort(const AValue: Integer);
begin
FPort := AValue;
end;

class destructor THorseProvider.UnInitialize;
begin
FreeAndNil(FIdHTTPWebBrokerBridge);
if FEvent <> nil then
FreeAndNil(FEvent);
end;

initialization
THorseProvider.SetKeepConnectionAlive(True);
{$ENDIF}

end.

To run the tests, we used
{$IFNDEF MSWINDOWS}
With the new changes,
we encountered no issues during compilation.

[regyssilveira]

Obrigado pelos detalhes e pelo código comparativo. Não vamos reverter integralmente Horse.Provider.Console.pas, pois isso removeria correções posteriores não relacionadas. Vamos isolar o comportamento no Windows e criar uma reprodução automatizada que cubra conexões concorrentes de longa duração, keep-alive, scheduler, latência e respostas 500. Os primeiros candidatos serão avaliados separadamente: TIdSchedulerOfThreadPool e SetKeepAliveValues, mantendo as demais correções. Publicaremos os resultados e qualquer ajuste somente após o teste identificar a causa.

[regyssilveira]

We created PR #585 with a dedicated stability test for the reported scenario, without reverting the provider or changing production code at this stage.

The test starts 50 concurrent HTTP/1.1 clients using persistent connections, performs 1,500 requests per client (75,000 total), and fails on timeouts, exceptions, non-200 responses, or invalid response bodies.

On our environment with Delphi 12/Win32 and the current master, the quick run completed successfully:

clients=50 requests=75000 delay_ms=0 errors=0 max_latency_ms=1313

We also noticed that 75,000 transactions from 50 clients over eight hours corresponds to approximately 19.2 seconds between requests from each client. Since that idle time may be relevant to persistent-connection behavior, PR #585 now allows the workload to be paced through environment variables.

Could you please run the following on Windows Server 2022 using Horse without the MSWINDOWS modification and share the output?

Quick run:

powershell -ExecutionPolicy Bypass -File tests/console-stability/run-console-stability-test.ps1

Eight-hour paced run matching the reported rate:

$env:HORSE_STABILITY_CLIENTS = '50'
$env:HORSE_STABILITY_REQUESTS_PER_CLIENT = '1500'
$env:HORSE_STABILITY_REQUEST_DELAY_MS = '19200'
powershell -ExecutionPolicy Bypass -File tests/console-stability/run-console-stability-test.ps1

If it fails in that environment, we will use the same reproduction to compare the Indy scheduler and socket keep-alive settings independently. This will let us correct the actual cause while preserving the other fixes made since version 3.2.

[carlosreyes210177]

I forgot to mention that we have a connection pool using FireDAC—200 for reads and 100 for writes; I don't know if that helps.

[regyssilveira]

Thank you for the additional information about the FireDAC connection pools. It may help explain the HTTP 500 responses, but it does not yet explain why some clients cannot connect or remain waiting.

To isolate the problem, could you please provide:

• the output of the quick and paced tests from PR test(console): add persistent connection stability stress #585 on Windows Server 2022;
• the exception message and stack trace corresponding to the HTTP 500 responses;
• the configured THorse.MaxConnections, ListenQueue, and ReadTimeout values;
• whether each request acquires and releases its FireDAC connection inside a try...finally block;
• the FireDAC pool acquisition timeout and the exact error produced when the pool is exhausted;
• the result of the same workload against a simple endpoint that does not access FireDAC;
• confirmation of whether the instability disappears when only the custom TIdSchedulerOfThreadPool assignment is removed, while keeping OnConnect and SetKeepAliveValues.

The scheduler is currently our main candidate because it was introduced in Horse 3.3.0 and its Indy PoolSize remains at the default value of zero. However, we need the A/B result before changing production code. This will allow us to distinguish an HTTP scheduler regression from a database-pool or application-level failure.