Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/red-metrics-trace-search.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@hyperdx/app': minor
---

Show RED metrics (Throughput, Errors, Duration) above the trace search results instead of the single count histogram. The three charts share a synced hover cursor, Errors toggles between rate and volume, and a RED/Heatmap switch flips the area to the duration heatmap. Logs and other sources keep the existing histogram.

The shared "error span" definition now also matches the legacy `STATUS_CODE_ERROR` status value in addition to `Error`, so error counts on the Services and LLM dashboards include those spans too.
31 changes: 25 additions & 6 deletions packages/app/src/ChartUtils.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {
BuilderSavedChartConfig,
ChartConfigWithDateRange,
ChartConfigWithOptDateRange,
ChartPaletteToken,
DisplayType,
Filter,
isSearchableSource,
Expand All @@ -49,7 +50,12 @@ import {
import { getMetricNameSql } from './otelSemanticConventions';
import { AggFn, TableChartSeries, TimeChartSeries } from './types';
import { NumberFormat } from './types';
import { getColorProps, getLogLevelColorOrder, logLevelColor } from './utils';
import {
getColorFromCSSToken,
getColorProps,
getLogLevelColorOrder,
logLevelColor,
} from './utils';

type SortOrder = 'asc' | 'desc';

Expand Down Expand Up @@ -585,6 +591,7 @@ interface LineDataWithOptionalColor extends Omit<LineData, 'color'> {

function setLineColors(
sortedLineData: LineDataWithOptionalColor[],
seriesColors?: Record<string, ChartPaletteToken>,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — seriesColors adds a second per-series color mechanism alongside select[i].color

ChartConfigSchema already carries color: ChartPaletteTokenSchema.optional() on each select item (common-utils types.ts:416), consumed positionally against meta[i].name in DBTableChart.tsx:149-175, and types.ts:1502-1503 states line/bar/pie per-series colors are meant to ship through that same field. Keying a side map on displayName instead means the pin lives apart from the alias that defines it — the hazard traceRedMetrics.ts:146-151 documents rather than removes. Resolve pins from select[i].color in formatResponseForTimeChart (same positional meta mapping useChartNumberFormats already uses) and set color: 'chart-cyan' on the p95 select entry instead of adding the seriesColors prop.

): LineData[] {
// Ensure that the current and previous period lines are the same color
const lineColorByCurrentPeriodKey = new Map<string, string>();
Expand All @@ -595,10 +602,15 @@ function setLineColors(
if (lineColorByCurrentPeriodKey.has(currentPeriodKey)) {
line.color = lineColorByCurrentPeriodKey.get(currentPeriodKey);
} else if (!line.color) {
line.color = getColorProps(
colorIndex++,
line.displayName ?? line.dataKey,
);
const seriesName = line.displayName ?? line.dataKey;
// A caller-pinned color (keyed by series name) wins over the positional
// palette, so a series keeps the same hue no matter how the data sorts
// it. A pinned series does not consume a positional slot, so unpinned
// series keep their natural palette order.
const pinnedToken = seriesColors?.[seriesName];
line.color = pinnedToken
? getColorFromCSSToken(pinnedToken)
: getColorProps(colorIndex++, seriesName);
lineColorByCurrentPeriodKey.set(currentPeriodKey, line.color);
} else {
lineColorByCurrentPeriodKey.set(currentPeriodKey, line.color);
Expand Down Expand Up @@ -749,6 +761,7 @@ export function formatResponseForTimeChart({
hiddenSeries = [],
previousPeriodOffsetSeconds = 0,
maxSeries = MAX_RENDERED_TIME_CHART_SERIES,
seriesColors,
}: {
dateRange: [Date, Date];
granularity?: SQLInterval;
Expand All @@ -764,6 +777,12 @@ export function formatResponseForTimeChart({
* every series (the "load all" escape hatch behind the hidden-series notice).
*/
maxSeries?: number;
/**
* Pin specific series (by display name) to a palette token instead of the
* positional palette, so their color is stable across sessions and can avoid
* hues reserved for other meanings (e.g. keeping latency off error red).
*/
seriesColors?: Record<string, ChartPaletteToken>;
}) {
const meta = currentPeriodResponse.meta;

Expand Down Expand Up @@ -998,7 +1017,7 @@ export function formatResponseForTimeChart({
(a, b) => a[timestampColumn.name] - b[timestampColumn.name],
);

const sortedLineDataWithColors = setLineColors(sortedLineData);
const sortedLineDataWithColors = setLineColors(sortedLineData, seriesColors);

// Count of LOGICAL groups actually rendered, in the same unit as
// hiddenSeriesCount — so a comparison chart (current + previous entries per
Expand Down
146 changes: 120 additions & 26 deletions packages/app/src/DBSearchPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ import {
Group,
Modal,
Paper,
SegmentedControl,
Select,
Stack,
Text,
Expand Down Expand Up @@ -140,6 +141,10 @@ import DBSqlRowTableWithSideBar from './components/DBSqlRowTableWithSidebar';
import PatternTable from './components/PatternTable';
import { DBSearchHeatmapChart } from './components/Search/DBSearchHeatmapChart';
import DirectTraceSidePanel from './components/Search/DirectTraceSidePanel';
import {
type TraceChartMode,
TraceRedMetricsChart,
} from './components/Search/TraceRedMetricsChart';
import SourceSchemaPreview, {
isSourceSchemaPreviewEnabled,
} from './components/SourceSchemaPreview';
Expand Down Expand Up @@ -206,6 +211,13 @@ const SearchConfigSchema = z.object({
type SearchConfigFromSchema = z.infer<typeof SearchConfigSchema>;

const QUERY_KEY_PREFIX = 'search';
// The RED tiles are auxiliary viz, not the results query, so they key their
// queries under a separate prefix to stay out of the `search executed`
// telemetry (which matches [QUERY_KEY_PREFIX] only). Otherwise the slow
// avg/p95/p99 duration query would become what the reported search latency
// measures. The live-tail pause, on the other hand, does wait on this prefix
// (see the pause signal below) so live ticks don't stack fresh quantile scans.
const RED_QUERY_KEY_PREFIX = `${QUERY_KEY_PREFIX}-red`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — RED queries bypass the live-tail pause gate, so live ticks can stack them

useLiveUpdate pauses on isAnyQueryFetching, which is useIsFetching({queryKey: [QUERY_KEY_PREFIX]}) (line 1597) and therefore never matches search-red. With the default 10s refreshFrequency, every tick advances searchedTimeRange, changing the RED query keys and firing three fresh queries — including the avg/p95/p99 quantile scan the comment itself calls slow — without waiting for the previous ones, so on a large trace table those accumulate unbounded while live tail runs. If the goal is only to keep RED out of the reported search latency, split the two consumers: keep useSearchTelemetry on the search-only count and give useLiveUpdate a pause signal that also counts [RED_QUERY_KEY_PREFIX].


// Clicks inside the results panel keep the row side panel open (so users can
// scroll the table or select a different row); clicks anywhere else on the page
Expand Down Expand Up @@ -1061,6 +1073,14 @@ export function DBSearchPage() {
]).withDefault('results'),
);

// RED metrics vs heatmap for the trace results chart area. URL state (like
// the other view toggles on this page) so a reload or shared link keeps the
// chosen view; the switch lives inline in the search stats row.
const [traceChartMode, setTraceChartMode] = useQueryState(
'traceChartMode',
parseAsStringEnum<TraceChartMode>(['red', 'heatmap']).withDefault('red'),
);

const [patternColumn, setPatternColumn] = useQueryState(
'patternColumn',
parseAsString,
Expand Down Expand Up @@ -1611,6 +1631,16 @@ export function DBSearchPage() {
queryKey: [QUERY_KEY_PREFIX],
}) > 0;

// Live tail also waits on the RED metric queries. They live under a separate
// prefix (kept out of the search-latency telemetry above), so without this
// each ~10s tick would fire a fresh avg/p95/p99 scan without waiting for the
// previous one, stacking slow quantile queries while live tail runs. Resolves
// to 0 when the RED tiles aren't mounted, so it's a no-op off the trace view.
const isRedQueryFetching =
useIsFetching({
queryKey: [RED_QUERY_KEY_PREFIX],
}) > 0;

const { searchElapsedMs } = useSearchTelemetry({
isAnyQueryFetching,
isLive: isLive ?? false,
Expand Down Expand Up @@ -1665,7 +1695,8 @@ export function DBSearchPage() {
interval,
refreshFrequency,
onTimeRangeSelect,
pause: isAnyQueryFetching || !queryReady || !isTabVisible,
pause:
isAnyQueryFetching || isRedQueryFetching || !queryReady || !isTabVisible,
});

// This ensures we only render this conditionally on the client
Expand Down Expand Up @@ -1864,6 +1895,34 @@ export function DBSearchPage() {

const aliasWith = useMemo(() => aliasMapToWithClauses(aliasMap), [aliasMap]);

// The trace results chart shows RED metrics (and a heatmap) only for a trace
// source that exposes a duration column; otherwise the single histogram
// stays. Derived once, as the narrowed source or null, so the stats-row
// toggle and the chart branch can't drift apart on which one renders.
const traceRedMetricsSource =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — traceRedMetricsSource restates the predicate already computed for showDelta

showDelta at DBSearchPage.tsx:2471 computes the same "trace source with a durationExpression" test inline; the two must now be kept in sync by hand. Pass showDelta={traceRedMetricsSource != null} and drop the inline ternary.

searchedSource != null &&
isTraceSource(searchedSource) &&
searchedSource.durationExpression
? searchedSource
: null;

// Reset to the default RED view when the source actually changes, so
// returning to a trace source after viewing another one doesn't silently
// reopen in Heatmap. The ref starts undefined and is only compared once a
// real id has been seen, so the async source resolution on first load
// (undefined -> id) doesn't count as a change and a shared link that pins
// ?traceChartMode=heatmap survives.
const prevSourceIdRef = useRef<string | undefined>(undefined);
useEffect(() => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Source-change reset fires on the first source resolution, so ?traceChartMode=heatmap never survives a load

searchedSource comes from useResolvedSourceParam → useSources() (a react-query fetch), so on a fresh page load the first render has searchedSource === undefined and prevSourceIdRef is seeded with undefined. When the sources arrive the id changes from undefined → the real id and the effect runs setTraceChartMode('red'), discarding the heatmap value the shared link carried — the exact case the comment says it skips. It also writes traceChartMode=red into the URL of every search page (nuqs 1.17 does not clear-on-default), including log sources where the param is meaningless. Seed the ref with a sentinel and only reset when the previous id was defined, e.g. const prevSourceIdRef = useRef<string | null | undefined>(undefined) plus if (prevSourceIdRef.current !== undefined && prevSourceIdRef.current !== searchedSource?.id) setTraceChartMode('red'), always assigning the ref.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Source-change reset clobbers traceChartMode on back/forward navigation

The effect compares only searchedSource?.id, so a history pop back to a URL that pins ?traceChartMode=heatmap (e.g. saved search A on source S1 → saved search B on source S2 → Back) is seen as a source change and rewritten to red, discarding the mode the restored URL carried. The existing source-change effect at DBSearchPage.tsx:1410-1476 already distinguishes a user-initiated switch from the form catching up to the config; gate the reset on the same signal rather than on a bare id comparison.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

if (
prevSourceIdRef.current !== undefined &&
prevSourceIdRef.current !== searchedSource?.id
) {
setTraceChartMode('red');
}
prevSourceIdRef.current = searchedSource?.id;
}, [searchedSource?.id, setTraceChartMode]);

const histogramTimeChartConfig = useMemo(() => {
if (chartConfig == null) {
return undefined;
Expand Down Expand Up @@ -2491,11 +2550,7 @@ export function DBSearchPage() {
setAnalysisMode={setAnalysisMode}
chartConfig={filtersChartConfig}
sourceId={inputSourceObj?.id}
showDelta={
!!(searchedSource?.kind === SourceKind.Trace
? searchedSource.durationExpression
: undefined)
}
showDelta={traceRedMetricsSource != null}
onColumnToggle={toggleColumn}
displayedColumns={displayedColumns}
onCollapse={() => setIsFilterSidebarCollapsed(true)}
Expand Down Expand Up @@ -2612,6 +2667,21 @@ export function DBSearchPage() {
enableParallelQueries
/>
<Group gap="sm" align="center">
{traceRedMetricsSource != null && (
<SegmentedControl

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — RED/Heatmap toggle stays visible when the chart area is suppressed by a query error

The SegmentedControl is inside the stats row, which renders whenever chartConfig && histogramTimeChartConfig, while the chart itself is gated on !hasQueryError at line 2637. Type an invalid WHERE clause on a trace source and the toggle renders above the error panel with nothing to switch. Gate it on !hasQueryError as well.

size="xs"
value={traceChartMode}
onChange={v =>
setTraceChartMode(
v === 'heatmap' ? 'heatmap' : 'red',
)
}
data={[
{ label: 'RED', value: 'red' },
{ label: 'Heatmap', value: 'heatmap' },
]}
/>
)}
{shouldShowLiveModeHint &&
denoiseResults != true && (
<ResumeLiveTailButton
Expand All @@ -2632,26 +2702,50 @@ export function DBSearchPage() {
</Group>
</Group>
</Box>
{!hasQueryError && (
<Box
className={searchPageStyles.timeChartContainer}
mih="0"
>
<DBTimeChart
sourceId={searchedConfig.source ?? undefined}
showLegend={false}
config={histogramTimeChartConfig}
enabled={isReady}
showDisplaySwitcher={false}
showMVOptimizationIndicator={false}
showDateRangeIndicator={false}
queryKeyPrefix={QUERY_KEY_PREFIX}
onTimeRangeSelect={handleTimeRangeSelect}
onFocusSeries={handleFocusSeries}
enableParallelQueries
/>
</Box>
)}
{!hasQueryError &&
(traceRedMetricsSource != null ? (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Trace searches now run the status-grouped count histogram with nothing rendering it, plus a second ungrouped count for Throughput

SearchResultsCountGroup (DBSearchPage.tsx:2661) still derives "N Results" from histogramTimeChartConfig under QUERY_KEY_PREFIX, and SearchTotalCountChart deliberately mirrors DBTimeChart's config/queryKey so the two dedupe into one query (see the comment at components/SearchTotalCountChart.tsx:48 and the guard test __tests__/DBSearchPageQueryKey.test.tsx). For a trace source that chart is no longer rendered, so the grouped count() GROUP BY StatusCode, bucket scan now runs purely to produce the total, while throughputConfig runs a near-identical ungrouped count() over the same rows under search-red — two count scans per search and per ~10s live-tail tick where there used to be one. Pass the throughput config to SearchResultsCountGroup and render the Throughput tile under QUERY_KEY_PREFIX (only Errors/Duration need the separate prefix) so the count and the visible chart share a key again.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

<Box
className={searchPageStyles.timeChartContainer}
mih="0"
h={240}
>
<TraceRedMetricsChart
mode={traceChartMode}
histogramTimeChartConfig={
histogramTimeChartConfig
}
heatmapChartConfig={{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — heatmapChartConfig is a fresh object literal every render, so the heatmap memo in TraceRedMetricsChart never hits

heatmapChartConfig={{...chartConfig, dateRange: searchedTimeRange, with: aliasWith}} is rebuilt on every render of DBSearchPage, so useMemo(..., [heatmapChartConfig, durationMsExpression]) at TraceRedMetricsChart.tsx:121 re-runs toHeatmapChartConfig and hands DBHeatmapChart a new config object each time — the same identity trap the file already guards against for durationFormat (TraceRedMetricsChart.tsx:86-96). Memoize the literal in DBSearchPage the way filtersChartConfig (DBSearchPage.tsx:2035) already is, and pass the memoized value.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

...chartConfig,
dateRange: searchedTimeRange,
with: aliasWith,
}}
source={traceRedMetricsSource}
isReady={isReady}
queryKeyPrefix={RED_QUERY_KEY_PREFIX}
onTimeRangeSelect={handleTimeRangeSelect}
onFocusSeries={handleFocusSeries}
/>
</Box>
) : (
<Box
className={searchPageStyles.timeChartContainer}
mih="0"
>
<DBTimeChart
sourceId={searchedConfig.source ?? undefined}
showLegend={false}
config={histogramTimeChartConfig}
enabled={isReady}
showDisplaySwitcher={false}
showMVOptimizationIndicator={false}
showDateRangeIndicator={false}
queryKeyPrefix={QUERY_KEY_PREFIX}
onTimeRangeSelect={handleTimeRangeSelect}
onFocusSeries={handleFocusSeries}
enableParallelQueries
/>
</Box>
))}
</>
)}
{hasQueryError && queryError ? (
Expand Down
Loading
Loading