[SPARK-59613] Fix intermittent 404s on SparkSQL ExecutionPage

[samra-h]

Jira ticket

https://issues.apache.org/jira/browse/SPARK-59613

What changes were proposed in this pull request?

This PR fixes intermittent 404 responses for /static/sql/* requests in the Spark History Server (SHS), which cause the SparkSQL tab's SQL -> ExecutionPage -> Plan Visualization to fail to render its graphs.
The /static/sql Jetty static handler was previously registered only inside SQLTab's constructor:
parent.addStaticHandler(SQLTab.STATIC_RESOURCE_DIR, "/static/sql")
SQLTab is instantiated lazily, per-application, via SQLHistoryServerPlugin.setupUI — only when an application containing SQL/DataFrame executions is loaded into the SHS application cache. Its handler therefore shared the lifetime of a per-application SparkUI, which is cached and evicted by ApplicationCache.
This change decouples the /static/sql handler from per-application SparkUI creation/eviction by registering it as part of the core SHS UI at startup (alongside the main /static handler wired up by HistoryServer), so the handler exists unconditionally for the lifetime of the server.

Why are the changes needed?

The SparkSQL ExecutionPage intermittently rendered no execution graphs, with browser network logs showing GET /static/sql returning 404.
Because the /static/sql handler's lifetime was tied to whichever SparkUI happened to register it, evicting that application from ApplicationCache tore the handler down, and /static/sql/* requests began 404ing until another SQL-bearing application was loaded and re-registered it.
This is significantly worse in multi-pod SHS deployments: each pod maintains its own independent ApplicationCache and its own set of dynamically-registered handlers. The core /static handler exists on every pod (registered at startup), but /static/sql only exists on a pod that currently has a SQL application loaded. A single browser page load fans out across pods — the ExecutionPage HTML may be served by a pod that has the handler, while the follow-up /static/sql/* asset request is routed to a different pod that never instantiated SQLTab, producing the seemingly random 404s. This also explains why manually opening the SQL tab once was only an unreliable, temporary workaround (it registered the handler on a single pod for a single eviction window).
Registering the handler at startup ensures /static/sql is available on every pod for the server's entire lifetime, eliminating both the eviction-driven and routing-driven 404s.

Does this PR introduce any user-facing change?

Yes — a bug fix. Previously, the SparkSQL ExecutionPage in the History Server intermittently failed to render execution graphs because /static/sql/* resources returned 404 (dependent on application-cache eviction, and pronounced in multi-pod deployments). After this change, the /static/sql resources are consistently served, so the SparkSQL execution graphs render reliably. There is no change to APIs, configuration, or output format.

How was this patch tested?

Manually verified against a live multi-pod SHS deployment (6 History Server pods behind the Service/load balancer).

Reproduction of the bug (before the fix):

• Restarted the SHS pods.
• Opened the SparkSQL plan visualization in the SHS UI — the execution graph failed to render.

• Checked each of the 6 pods individually for the /static/sql static-resource registration and found the handler was registered on only one pod.

• Concluded the graph-rendering request was being routed by the load balancer to one of the "cold" pods that had never instantiated SQLTab and therefore had no /static/sql handler, resulting in the 404 and the missing graph.

Verification (after the fix):

• Restarted the SHS pods again.
• Checked all 6 pods for the /static/sql static-resource registration — the resources were present on all 6 pods immediately on startup, without needing any SQL-bearing application to be loaded first.

• Repeatedly opened the SparkSQL plan visualization in the SHS UI and was unable to reproduce the missing-graph bug — the execution graph rendered consistently regardless of which pod served the request.

Was this patch authored or co-authored using generative AI tooling?

No