Skip to content

feat(app): add a duration heatmap view to trace search #3184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alex-fedotyev wants to merge 7 commits into
base: alex/red-metrics-trace-search-v2
Choose a base branch
from
Open

feat(app): add a duration heatmap view to trace search #3184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a690c51
fix(app): count legacy STATUS_CODE_ERROR spans as errors
alex-fedotyev Sep 22, 2026
7de8c84
refactor(app): scope the error predicate to span status only
alex-fedotyev Sep 24, 2026
76e4dca
feat(app): show RED metrics on the trace search results view
alex-fedotyev Sep 22, 2026
3c58c91
refactor(app): follow the error predicate rename
alex-fedotyev Sep 24, 2026
2d62337
refactor(app): name the duration-heatmap number format
alex-fedotyev Sep 22, 2026
8bb3d02
feat(app): add a duration heatmap view to trace search
alex-fedotyev Sep 22, 2026
46b3d9e
fix(app): pin live tail off while the trace heatmap is visible
alex-fedotyev Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/error-span-predicate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@hyperdx/app': patch
---

Count spans recorded with the legacy `STATUS_CODE_ERROR` status as errors on the Services and LLM dashboards. The errored-span definition now lives in one place and matches both that value and the short `Error` form, so those dashboards no longer undercount errors from collectors that emit the long form.
5 changes: 5 additions & 0 deletions .changeset/red-metrics-trace-search.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@hyperdx/app': minor
---

Show RED metrics (Throughput, Errors, Duration) above the trace search results instead of the single count histogram. The three charts share a synced hover cursor, Errors toggles between rate and volume, and clicking a status series on the volume chart filters the results. Logs and other sources keep the existing histogram.
5 changes: 5 additions & 0 deletions .changeset/trace-search-duration-heatmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@hyperdx/app': minor
---

Add a RED/Heatmap switch to the trace search results view. Heatmap shows the duration distribution over time as the same tile the dashboard renders, and the choice is kept in the URL so a reload or a shared link reopens the same view.
31 changes: 25 additions & 6 deletions packages/app/src/ChartUtils.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {
BuilderSavedChartConfig,
ChartConfigWithDateRange,
ChartConfigWithOptDateRange,
ChartPaletteToken,
DisplayType,
Filter,
isSearchableSource,
Expand All @@ -49,7 +50,12 @@ import {
import { getMetricNameSql } from './otelSemanticConventions';
import { AggFn, TableChartSeries, TimeChartSeries } from './types';
import { NumberFormat } from './types';
import { getColorProps, getLogLevelColorOrder, logLevelColor } from './utils';
import {
getColorFromCSSToken,
getColorProps,
getLogLevelColorOrder,
logLevelColor,
} from './utils';

type SortOrder = 'asc' | 'desc';

Expand Down Expand Up @@ -585,6 +591,7 @@ interface LineDataWithOptionalColor extends Omit<LineData, 'color'> {

function setLineColors(
sortedLineData: LineDataWithOptionalColor[],
seriesColors?: Record<string, ChartPaletteToken>,
): LineData[] {
// Ensure that the current and previous period lines are the same color
const lineColorByCurrentPeriodKey = new Map<string, string>();
Expand All @@ -595,10 +602,15 @@ function setLineColors(
if (lineColorByCurrentPeriodKey.has(currentPeriodKey)) {
line.color = lineColorByCurrentPeriodKey.get(currentPeriodKey);
} else if (!line.color) {
line.color = getColorProps(
colorIndex++,
line.displayName ?? line.dataKey,
);
const seriesName = line.displayName ?? line.dataKey;
// A caller-pinned color (keyed by series name) wins over the positional
// palette, so a series keeps the same hue no matter how the data sorts
// it. A pinned series does not consume a positional slot, so unpinned
// series keep their natural palette order.
const pinnedToken = seriesColors?.[seriesName];
line.color = pinnedToken
? getColorFromCSSToken(pinnedToken)
: getColorProps(colorIndex++, seriesName);
lineColorByCurrentPeriodKey.set(currentPeriodKey, line.color);
} else {
lineColorByCurrentPeriodKey.set(currentPeriodKey, line.color);
Expand Down Expand Up @@ -749,6 +761,7 @@ export function formatResponseForTimeChart({
hiddenSeries = [],
previousPeriodOffsetSeconds = 0,
maxSeries = MAX_RENDERED_TIME_CHART_SERIES,
seriesColors,
}: {
dateRange: [Date, Date];
granularity?: SQLInterval;
Expand All @@ -764,6 +777,12 @@ export function formatResponseForTimeChart({
* every series (the "load all" escape hatch behind the hidden-series notice).
*/
maxSeries?: number;
/**
* Pin specific series (by display name) to a palette token instead of the
* positional palette, so their color is stable across sessions and can avoid
* hues reserved for other meanings (e.g. keeping latency off error red).
*/
seriesColors?: Record<string, ChartPaletteToken>;
}) {
const meta = currentPeriodResponse.meta;

Expand Down Expand Up @@ -998,7 +1017,7 @@ export function formatResponseForTimeChart({
(a, b) => a[timestampColumn.name] - b[timestampColumn.name],
);

const sortedLineDataWithColors = setLineColors(sortedLineData);
const sortedLineDataWithColors = setLineColors(sortedLineData, seriesColors);

// Count of LOGICAL groups actually rendered, in the same unit as
// hiddenSeriesCount — so a comparison chart (current + previous entries per
Expand Down
165 changes: 139 additions & 26 deletions packages/app/src/DBSearchPage.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ import {
Group,
Modal,
Paper,
SegmentedControl,
Select,
Stack,
Text,
Expand Down Expand Up @@ -140,6 +141,10 @@ import DBSqlRowTableWithSideBar from './components/DBSqlRowTableWithSidebar';
import PatternTable from './components/PatternTable';
import { DBSearchHeatmapChart } from './components/Search/DBSearchHeatmapChart';
import DirectTraceSidePanel from './components/Search/DirectTraceSidePanel';
import {
type TraceChartMode,
TraceRedMetricsChart,
} from './components/Search/TraceRedMetricsChart';
import SourceSchemaPreview, {
isSourceSchemaPreviewEnabled,
} from './components/SourceSchemaPreview';
Expand Down Expand Up @@ -206,6 +211,13 @@ const SearchConfigSchema = z.object({
type SearchConfigFromSchema = z.infer<typeof SearchConfigSchema>;

const QUERY_KEY_PREFIX = 'search';
// The RED tiles are auxiliary viz, not the results query, so they key their
// queries under a separate prefix to stay out of the `search executed`
// telemetry (which matches [QUERY_KEY_PREFIX] only). Otherwise the slow
// avg/p95/p99 duration query would become what the reported search latency
// measures. The live-tail pause, on the other hand, does wait on this prefix
// (see the pause signal below) so live ticks don't stack fresh quantile scans.
const RED_QUERY_KEY_PREFIX = `${QUERY_KEY_PREFIX}-red`;

// Clicks inside the results panel keep the row side panel open (so users can
// scroll the table or select a different row); clicks anywhere else on the page
Expand Down Expand Up @@ -1061,6 +1073,14 @@ export function DBSearchPage() {
]).withDefault('results'),
);

// RED metrics vs heatmap for the trace results chart area. URL state (like
// the other view toggles on this page) so a reload or shared link keeps the
// chosen view; the switch lives inline in the search stats row.
const [traceChartMode, setTraceChartMode] = useQueryState(
'traceChartMode',
parseAsStringEnum<TraceChartMode>(['red', 'heatmap']).withDefault('red'),
);

const [patternColumn, setPatternColumn] = useQueryState(
'patternColumn',
parseAsString,
Expand Down Expand Up @@ -1611,6 +1631,16 @@ export function DBSearchPage() {
queryKey: [QUERY_KEY_PREFIX],
}) > 0;

// Live tail also waits on the RED metric queries. They live under a separate
// prefix (kept out of the search-latency telemetry above), so without this
// each ~10s tick would fire a fresh avg/p95/p99 scan without waiting for the
// previous one, stacking slow quantile queries while live tail runs. Resolves
// to 0 when the RED tiles aren't mounted, so it's a no-op off the trace view.
const isRedQueryFetching =
useIsFetching({
queryKey: [RED_QUERY_KEY_PREFIX],
}) > 0;

const { searchElapsedMs } = useSearchTelemetry({
isAnyQueryFetching,
isLive: isLive ?? false,
Expand Down Expand Up @@ -1665,7 +1695,8 @@ export function DBSearchPage() {
interval,
refreshFrequency,
onTimeRangeSelect,
pause: isAnyQueryFetching || !queryReady || !isTabVisible,
pause:
isAnyQueryFetching || isRedQueryFetching || !queryReady || !isTabVisible,
});

// This ensures we only render this conditionally on the client
Expand Down Expand Up @@ -1864,6 +1895,53 @@ export function DBSearchPage() {

const aliasWith = useMemo(() => aliasMapToWithClauses(aliasMap), [aliasMap]);

// The trace results chart shows RED metrics (and a heatmap) only for a trace
// source that exposes a duration column; otherwise the single histogram
// stays. Derived once, as the narrowed source or null, so the stats-row
// toggle and the chart branch can't drift apart on which one renders.
const traceRedMetricsSource =
searchedSource != null &&
isTraceSource(searchedSource) &&
searchedSource.durationExpression
? searchedSource
: null;

// Reset to the default RED view when the source actually changes, so
// returning to a trace source after viewing another one doesn't silently
// reopen in Heatmap. The ref starts undefined and is only compared once a
// real id has been seen, so the async source resolution on first load
// (undefined -> id) doesn't count as a change and a shared link that pins
// ?traceChartMode=heatmap survives.
const prevSourceIdRef = useRef<string | undefined>(undefined);

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New mode-switch and reset logic has no tests

No test renders TraceRedMetricsChart in either mode. None checks that the heatmap queries carry queryKeyPrefix, which is what lets live tail wait on them, and none covers the source-change reset effect (a first-load undefined→id transition must keep heatmap; a change from one id to another must reset). Add component tests, for example next to components/Search/__tests__/traceRedMetrics.test.ts, that assert the rendered chart and query keys for each mode, plus a hook-level test for the reset effect.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

useEffect(() => {

@github-actions github-actions Bot Sep 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Reset effect discards the user's Heatmap choice even when the new source can render it

The effect fires on any searchedSource?.id change, so switching from trace source A to trace source B — both with a durationExpression, so both render the toggle via traceRedMetricsSource — silently flips the view back to RED. The sibling view toggle analysisMode (:1067) is deliberately not reset on source change even though delta needs a trace source too; that branch just renders nothing (:2637). Drop the effect (both the switch and the chart are already hidden when traceRedMetricsSource == null, so a lingering ?traceChartMode=heatmap is inert), or gate the reset on the new source not supporting the heatmap.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

if (
prevSourceIdRef.current !== undefined &&
prevSourceIdRef.current !== searchedSource?.id
) {
setTraceChartMode('red');

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Every source switch adds ?traceChartMode=red to the URL, even for log sources

The app uses nuqs 1.17.0, where clearOnDefault is off by default. setTraceChartMode('red') therefore writes the default value into the URL each time the source changes, including sources that have no RED view. Call setTraceChartMode(null) to clear the param instead.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

}
prevSourceIdRef.current = searchedSource?.id;
}, [searchedSource?.id, setTraceChartMode]);

// Pin live tail off while the heatmap is the visible chart. A live tick
// moves the absolute time range, and the heatmap keys its uPlot instance on
// the chart config, so every refresh would tear the chart down and rebuild
// it with a recomputed y-axis instead of updating in place. Delta and
// pattern modes already pin it off (see the analysisMode effect above), so
// the delta-view heatmap never hit this; results mode defaults to live, so
// the view has to opt out for itself. Only pinned while the heatmap is
// actually rendered, so a stale ?traceChartMode=heatmap on a log source
// can't disable live tail for a search that shows the histogram.
const isHeatmapChartVisible =
analysisMode === 'results' &&
traceRedMetricsSource != null &&

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Resume live tail button undoes the heatmap's live-tail pin

The effect only calls setIsLive(false) when isHeatmapChartVisible changes. Once it runs, shouldShowLiveModeHint becomes true and ResumeLiveTailButton shows up right next to the new switch (line ~2704). Clicking it, or picking a live range in the time picker (setIsLive(true) at ~2125), turns live back on while the heatmap is still visible, and the effect doesn't run again. Each tick then rebuilds the uPlot instance, which is the problem the comment says this prevents. Fix: hide the resume button and the live option while isHeatmapChartVisible, or leave isLive alone and gate useLiveUpdate's pause on isHeatmapChartVisible.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Switching back to RED leaves live tail off

The pin writes isLive=false into the URL, and nothing restores it. A user who looks at the heatmap for a moment and switches back to RED is left on a frozen time range with live tail off. Pausing live updates through useLiveUpdate's pause while the heatmap is visible, instead of overwriting isLive, would bring live tail back when the user switches to RED.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

traceChartMode === 'heatmap';

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Resume Live Tail gets around the heatmap live-tail pin

The pin effect only runs when isHeatmapChartVisible changes. Once live has been pinned off, shouldShowLiveModeHint renders ResumeLiveTailButton right next to the Heatmap switch (line ~2706). Clicking it, or picking a live interval (the setIsLive(true) path at ~2125), turns live back on and the effect never runs again. The heatmap then tears down and rebuilds on every tick, because it keys on JSON.stringify(config) in DBHeatmapChart.tsx:873, which this effect was meant to prevent. Fix: hide or disable ResumeLiveTailButton and the live interval option while isHeatmapChartVisible. Alternatively, have the live-enable handlers leave the heatmap first (for example setTraceChartMode('red')), or pass isLive && !isHeatmapChartVisible to useLiveUpdate.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

useEffect(() => {
if (isHeatmapChartVisible) {
setIsLive(false);
}
}, [isHeatmapChartVisible, setIsLive]);

const histogramTimeChartConfig = useMemo(() => {
if (chartConfig == null) {
return undefined;
Expand Down Expand Up @@ -2491,11 +2569,7 @@ export function DBSearchPage() {
setAnalysisMode={setAnalysisMode}
chartConfig={filtersChartConfig}
sourceId={inputSourceObj?.id}
showDelta={
!!(searchedSource?.kind === SourceKind.Trace
? searchedSource.durationExpression
: undefined)
}
showDelta={traceRedMetricsSource != null}
onColumnToggle={toggleColumn}
displayedColumns={displayedColumns}
onCollapse={() => setIsFilterSidebarCollapsed(true)}
Expand Down Expand Up @@ -2612,6 +2686,21 @@ export function DBSearchPage() {
enableParallelQueries
/>
<Group gap="sm" align="center">
{traceRedMetricsSource != null && (
<SegmentedControl
size="xs"
value={traceChartMode}
onChange={v =>
setTraceChartMode(
v === 'heatmap' ? 'heatmap' : 'red',
)
}
data={[
{ label: 'RED', value: 'red' },
{ label: 'Heatmap', value: 'heatmap' },
]}
/>
)}
{shouldShowLiveModeHint &&
denoiseResults != true && (
<ResumeLiveTailButton
Expand All @@ -2632,26 +2721,50 @@ export function DBSearchPage() {
</Group>
</Group>
</Box>
{!hasQueryError && (
<Box
className={searchPageStyles.timeChartContainer}
mih="0"
>
<DBTimeChart
sourceId={searchedConfig.source ?? undefined}
showLegend={false}
config={histogramTimeChartConfig}
enabled={isReady}
showDisplaySwitcher={false}
showMVOptimizationIndicator={false}
showDateRangeIndicator={false}
queryKeyPrefix={QUERY_KEY_PREFIX}
onTimeRangeSelect={handleTimeRangeSelect}
onFocusSeries={handleFocusSeries}
enableParallelQueries
/>
</Box>
)}
{!hasQueryError &&
(traceRedMetricsSource != null ? (
<Box
className={searchPageStyles.timeChartContainer}
mih="0"
h={240}
>
<TraceRedMetricsChart
mode={traceChartMode}
histogramTimeChartConfig={
histogramTimeChartConfig
}
heatmapChartConfig={{

@github-actions github-actions Bot Sep 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — heatmapChartConfig is a fresh object literal every render, so the tile-config memo never memoizes

The inline {...chartConfig, dateRange: searchedTimeRange, with: aliasWith} gets a new identity on every render, which is the single dep of useMemo in TraceRedMetricsChart.tsx:121-135 — so toHeatmapChartConfig and both buildHeatmap*Config calls rerun each render, exactly the failure the sibling durationFormat memo comment at TraceRedMetricsChart.tsx:86 warns about. The page already memoizes the near-identical shape as filtersChartConfig (:2035, which adds only orderBy: undefined) and histogramTimeChartConfig (:1926). Hoist one useMemo for {...chartConfig, dateRange: searchedTimeRange, with: aliasWith} and pass it to both this callsite and the DBSearchHeatmapChart one at :2641.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The inline heatmapChartConfig object makes the heatmap useMemo recompute on every render

heatmapChartConfig={{...chartConfig, dateRange, with: aliasWith}} creates a new object on each DBSearchPage render. TraceRedMetricsChart lists it as a dependency of the toHeatmapChartConfig memo, so the memo never reuses its result, even while in RED mode. Memoize the config in DBSearchPage with useMemo over chartConfig, searchedTimeRange and aliasWith, or build it inside TraceRedMetricsChart from its inputs.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions github-actions Bot Sep 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Inline heatmapChartConfig literal means the heatmap useMemo never reuses its result

heatmapChartConfig={{...chartConfig, dateRange: searchedTimeRange, with: aliasWith}} creates a new object on every DBSearchPage render. It is a dependency of the toHeatmapChartConfig useMemo in TraceRedMetricsChart.tsx:314, so that memo recomputes every render and does no work. Fix: build the config once with useMemo in DBSearchPage, keyed on chartConfig, searchedTimeRange and aliasWith, and pass the memoized value. Otherwise drop the useMemo.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

...chartConfig,
dateRange: searchedTimeRange,
with: aliasWith,
}}
source={traceRedMetricsSource}
isReady={isReady}
queryKeyPrefix={RED_QUERY_KEY_PREFIX}
onTimeRangeSelect={handleTimeRangeSelect}
onFocusSeries={handleFocusSeries}
/>
</Box>
) : (
<Box
className={searchPageStyles.timeChartContainer}
mih="0"
>
<DBTimeChart
sourceId={searchedConfig.source ?? undefined}
showLegend={false}
config={histogramTimeChartConfig}
enabled={isReady}
showDisplaySwitcher={false}
showMVOptimizationIndicator={false}
showDateRangeIndicator={false}
queryKeyPrefix={QUERY_KEY_PREFIX}
onTimeRangeSelect={handleTimeRangeSelect}
onFocusSeries={handleFocusSeries}
enableParallelQueries
/>
</Box>
))}
</>
)}
{hasQueryError && queryError ? (
Expand Down
Loading
Loading