Skip to content

fix(core): [Callback Errors 2] Drop failed processor data #6142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adinauer wants to merge 1 commit into
base: fix/callback-error-handling-discard-reason
Choose a base branch
from
+231 −12
Open

fix(core): [Callback Errors 2] Drop failed processor data #6142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@

- Fix SDK callback error handling ([#6140](https://github.com/getsentry/sentry-java/pull/6140))
- Add `DiscardReason.CALLBACK_ERROR` and use it for telemetry dropped when a `beforeSend*` callback throws. `OnDiscardCallback` can now receive this value.
- Drop telemetry and record `callback_error` when an event processor throws instead of continuing with a potentially partially processed item.
- Disable URL caching when reading `META-INF/MANIFEST.MF` files during version detection so that the SDK no longer keeps jar file handles open for the life of the process ([#6124](https://github.com/getsentry/sentry-java/pull/6124)
- Keep the `EventListener` wrapped by `SentryOkHttpEventListener` per `Call` ([#6003](https://github.com/getsentry/sentry-java/pull/6003))

Expand Down
24 changes: 24 additions & 0 deletions sentry/src/main/java/io/sentry/SentryClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ private SentryEvent processEvent(
e,
"An exception occurred while processing event by processor: %s",
processor.getClass().getName());
options
.getClientReportRecorder()
.recordLostEvent(DiscardReason.CALLBACK_ERROR, DataCategory.Error);
return null;
}

if (event == null) {
Expand Down Expand Up @@ -557,6 +561,8 @@ private SentryLogEvent processLogEvent(
e,
"An exception occurred while processing log event by processor: %s",
processor.getClass().getName());
recordLostLogEvent(DiscardReason.CALLBACK_ERROR, eventBeforeProcessor);
return null;
}

if (event == null) {
Expand Down Expand Up @@ -590,6 +596,8 @@ private SentryMetricsEvent processMetricsEvent(
e,
"An exception occurred while processing metrics event by processor: %s",
processor.getClass().getName());
recordLostMetricsEvent(DiscardReason.CALLBACK_ERROR, eventBeforeProcessor);
return null;
}

if (event == null) {
Expand Down Expand Up @@ -622,6 +630,14 @@ private SentryMetricsEvent processMetricsEvent(
e,
"An exception occurred while processing transaction by processor: %s",
processor.getClass().getName());
options
.getClientReportRecorder()
.recordLostEvent(DiscardReason.CALLBACK_ERROR, DataCategory.Transaction);
options
.getClientReportRecorder()
.recordLostEvent(
DiscardReason.CALLBACK_ERROR, DataCategory.Span, spanCountBeforeProcessor + 1);
return null;
}
final int spanCountAfterProcessor = transaction == null ? 0 : transaction.getSpans().size();

Expand Down Expand Up @@ -675,6 +691,10 @@ private SentryReplayEvent processReplayEvent(
e,
"An exception occurred while processing replay event by processor: %s",
processor.getClass().getName());
options
.getClientReportRecorder()
.recordLostEvent(DiscardReason.CALLBACK_ERROR, DataCategory.Replay);
return null;
}

if (replayEvent == null) {
Comment on lines 691 to 700

@sentry sentry Bot Sep 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: If an EventProcessor mutates an event and then throws, the client report will incorrectly record the byte size of the mutated event, not the original.
Severity: MEDIUM

Suggested Fix

Calculate the byte size of the event before calling the processor, rather than after catching an exception. Alternatively, create a defensive copy of the event object before passing it to the processor if the object supports cloning.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: sentry/src/main/java/io/sentry/SentryClient.java#L691-L700

Potential issue: In `processLogEvent` and `processMetricsEvent`, a reference to the
event is stored in `eventBeforeProcessor` before it is passed to an `EventProcessor`. If
the processor mutates the event in-place and then throws an exception, the subsequent
call to `recordLostLogEvent` or `recordLostMetricsEvent` will calculate the byte size
based on the mutated event, not the original one. This leads to inaccurate byte counts
being recorded in client reports for dropped events, as the `EventProcessor` interface
explicitly allows for in-place mutation.

Also affects:

  • sentry/src/main/java/io/sentry/SentryClient.java:729~738

Did we get this right? 👍 / 👎 to inform future reviews.

Expand Down Expand Up @@ -709,6 +729,10 @@ private SentryEvent processFeedbackEvent(
e,
"An exception occurred while processing feedback event by processor: %s",
processor.getClass().getName());
options
.getClientReportRecorder()
.recordLostEvent(DiscardReason.CALLBACK_ERROR, DataCategory.Feedback);
return null;
}

if (feedbackEvent == null) {
Expand Down
218 changes: 206 additions & 12 deletions sentry/src/test/java/io/sentry/SentryClientTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package io.sentry

import com.google.common.truth.Truth.assertThat
import io.sentry.Scope.IWithPropagationContext
import io.sentry.SentryLevel.WARNING
import io.sentry.Session.State.Crashed
Expand Down Expand Up @@ -474,6 +475,48 @@ class SentryClientTest {
)
}

@Test
fun `throwing log processor drops log and stops callbacks`() {
val scope = createScope()
val logEvent = SentryLogEvent(SentryId(), SentryNanotimeDate(), "message", SentryLogLevel.WARN)
val logEventNumberOfBytes =
JsonSerializationUtils.byteSizeOf(
fixture.sentryOptions.serializer,
fixture.sentryOptions.logger,
logEvent,
)
val throwingProcessor = mock<EventProcessor>()
val nextProcessor = mock<EventProcessor>()
val beforeSend = mock<SentryOptions.Logs.BeforeSendLogCallback>()
val onDiscard = mock<SentryOptions.OnDiscardCallback>()
whenever(throwingProcessor.process(any<SentryLogEvent>()))
.thenThrow(IllegalStateException("test"))
scope.addEventProcessor(throwingProcessor)
scope.addEventProcessor(nextProcessor)
fixture.sentryOptions.logs.beforeSend = beforeSend
fixture.sentryOptions.onDiscard = onDiscard

fixture.getSut().captureLog(logEvent, scope)

verify(nextProcessor, never()).process(any<SentryLogEvent>())
verify(beforeSend, never()).execute(any())
verify(fixture.loggerBatchProcessor, never()).add(any())
assertClientReport(
fixture.sentryOptions.clientReportRecorder,
listOf(
DiscardedEvent(DiscardReason.CALLBACK_ERROR.reason, DataCategory.LogItem.category, 1),
DiscardedEvent(
DiscardReason.CALLBACK_ERROR.reason,
DataCategory.LogByte.category,
logEventNumberOfBytes,
),
),
)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.LogItem, 1)
verify(onDiscard)
.execute(DiscardReason.CALLBACK_ERROR, DataCategory.LogByte, logEventNumberOfBytes)
}

@Test
fun `when beforeSendLog is returns new instance, new instance is sent`() {
val scope = createScope()
Expand Down Expand Up @@ -595,6 +638,56 @@ class SentryClientTest {
)
}

@Test
fun `throwing metric processor drops metric and stops callbacks`() {
val scope = createScope()
val metricsEvent = SentryMetricsEvent(SentryId(), SentryNanotimeDate(), "name", "gauge", 123.0)
val metricsEventNumberOfBytes =
JsonSerializationUtils.byteSizeOf(
fixture.sentryOptions.serializer,
fixture.sentryOptions.logger,
metricsEvent,
)
val throwingProcessor = mock<EventProcessor>()
val nextProcessor = mock<EventProcessor>()
val beforeSend = mock<SentryOptions.Metrics.BeforeSendMetricCallback>()
val onDiscard = mock<SentryOptions.OnDiscardCallback>()
whenever(throwingProcessor.process(any<SentryMetricsEvent>(), anyOrNull()))
.thenThrow(IllegalStateException("test"))
scope.addEventProcessor(throwingProcessor)
scope.addEventProcessor(nextProcessor)
fixture.sentryOptions.metrics.beforeSend = beforeSend
fixture.sentryOptions.onDiscard = onDiscard

fixture.getSut().captureMetric(metricsEvent, scope, null)

verify(nextProcessor, never()).process(any<SentryMetricsEvent>(), anyOrNull())
verify(beforeSend, never()).execute(any(), anyOrNull())
verify(fixture.metricsBatchProcessor, never()).add(any())
assertClientReport(
fixture.sentryOptions.clientReportRecorder,
listOf(
DiscardedEvent(
DiscardReason.CALLBACK_ERROR.reason,
DataCategory.TraceMetric.category,
1,
),
DiscardedEvent(
DiscardReason.CALLBACK_ERROR.reason,
DataCategory.TraceMetricByte.category,
metricsEventNumberOfBytes,
),
),
)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.TraceMetric, 1)
verify(onDiscard)
.execute(
DiscardReason.CALLBACK_ERROR,
DataCategory.TraceMetricByte,
metricsEventNumberOfBytes,
)
}

@Test
fun `when beforeSendMetric is returns new instance, new instance is sent`() {
val scope = createScope()
Expand Down Expand Up @@ -1242,6 +1335,42 @@ class SentryClientTest {
)
}

@Test
fun `throwing transaction processor drops transaction and stops callbacks`() {
val throwingProcessor = mock<EventProcessor>()
val nextProcessor = mock<EventProcessor>()
val beforeSend = mock<SentryOptions.BeforeSendTransactionCallback>()
val onDiscard = mock<SentryOptions.OnDiscardCallback>()
whenever(throwingProcessor.process(any<SentryTransaction>(), anyOrNull()))
.thenThrow(IllegalStateException("test"))
fixture.sentryOptions.addEventProcessor(throwingProcessor)
fixture.sentryOptions.addEventProcessor(nextProcessor)
fixture.sentryOptions.beforeSendTransaction = beforeSend
fixture.sentryOptions.onDiscard = onDiscard

val id =
fixture
.getSut()
.captureTransaction(
SentryTransaction(fixture.sentryTracer),
fixture.sentryTracer.traceContext(),
)

assertThat(id).isEqualTo(SentryId.EMPTY_ID)
verify(nextProcessor, never()).process(any<SentryTransaction>(), anyOrNull())
verify(beforeSend, never()).execute(any(), anyOrNull())
verify(fixture.transport, never()).send(any(), anyOrNull())
assertClientReport(
fixture.sentryOptions.clientReportRecorder,
listOf(
DiscardedEvent(DiscardReason.CALLBACK_ERROR.reason, DataCategory.Transaction.category, 1),
DiscardedEvent(DiscardReason.CALLBACK_ERROR.reason, DataCategory.Span.category, 2),
),
)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.Transaction, 1)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.Span, 2)
}

@Test
fun `transaction dropped by ignoredTransactions is recorded`() {
fixture.sentryOptions.setIgnoredTransactions(listOf("a-transaction"))
Expand Down Expand Up @@ -1927,10 +2056,29 @@ class SentryClientTest {
}

@Test
fun `exception thrown by an event processor is handled gracefully`() {
fixture.sentryOptions.addEventProcessor(eventProcessorThrows())
val sut = fixture.getSut()
sut.captureEvent(SentryEvent())
fun `exception thrown by an event processor drops event and stops callbacks`() {
val throwingProcessor = mock<EventProcessor>()
val nextProcessor = mock<EventProcessor>()
val beforeSend = mock<SentryOptions.BeforeSendCallback>()
val onDiscard = mock<SentryOptions.OnDiscardCallback>()
whenever(throwingProcessor.process(any<SentryEvent>(), anyOrNull()))
.thenThrow(IllegalStateException("test"))
fixture.sentryOptions.addEventProcessor(throwingProcessor)
fixture.sentryOptions.addEventProcessor(nextProcessor)
fixture.sentryOptions.beforeSend = beforeSend
fixture.sentryOptions.onDiscard = onDiscard

val id = fixture.getSut().captureEvent(SentryEvent())

assertThat(id).isEqualTo(SentryId.EMPTY_ID)
verify(nextProcessor, never()).process(any<SentryEvent>(), anyOrNull())
verify(beforeSend, never()).execute(any(), anyOrNull())
verify(fixture.transport, never()).send(any(), anyOrNull())
assertClientReport(
fixture.sentryOptions.clientReportRecorder,
listOf(DiscardedEvent(DiscardReason.CALLBACK_ERROR.reason, DataCategory.Error.category, 1)),
)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.Error, 1)
}

@Test
Expand Down Expand Up @@ -3524,6 +3672,32 @@ class SentryClientTest {
verify(onDiscardMock, times(1)).execute(DiscardReason.EVENT_PROCESSOR, DataCategory.Replay, 1)
}

@Test
fun `throwing replay processor drops replay and stops callbacks`() {
val throwingProcessor = mock<EventProcessor>()
val nextProcessor = mock<EventProcessor>()
val beforeSend = mock<SentryOptions.BeforeSendReplayCallback>()
val onDiscard = mock<SentryOptions.OnDiscardCallback>()
whenever(throwingProcessor.process(any<SentryReplayEvent>(), anyOrNull()))
.thenThrow(IllegalStateException("test"))
fixture.sentryOptions.addEventProcessor(throwingProcessor)
fixture.sentryOptions.addEventProcessor(nextProcessor)
fixture.sentryOptions.beforeSendReplay = beforeSend
fixture.sentryOptions.onDiscard = onDiscard

val id = fixture.getSut().captureReplayEvent(createReplayEvent(), createScope(), null)

assertThat(id).isEqualTo(SentryId.EMPTY_ID)
verify(nextProcessor, never()).process(any<SentryReplayEvent>(), anyOrNull())
verify(beforeSend, never()).execute(any(), anyOrNull())
verify(fixture.transport, never()).send(any(), anyOrNull())
assertClientReport(
fixture.sentryOptions.clientReportRecorder,
listOf(DiscardedEvent(DiscardReason.CALLBACK_ERROR.reason, DataCategory.Replay.category, 1)),
)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.Replay, 1)
}

@Test
fun `calls captureReplay on replay controller for error events`() {
var called = false
Expand Down Expand Up @@ -4086,6 +4260,34 @@ class SentryClientTest {
verify(onDiscardMock, times(1)).execute(DiscardReason.EVENT_PROCESSOR, DataCategory.Feedback, 1)
}

@Test
fun `throwing feedback processor drops feedback and stops callbacks`() {
val throwingProcessor = mock<EventProcessor>()
val nextProcessor = mock<EventProcessor>()
val beforeSend = mock<SentryOptions.BeforeSendCallback>()
val onDiscard = mock<SentryOptions.OnDiscardCallback>()
whenever(throwingProcessor.process(any<SentryEvent>(), anyOrNull()))
.thenThrow(IllegalStateException("test"))
fixture.sentryOptions.addEventProcessor(throwingProcessor)
fixture.sentryOptions.addEventProcessor(nextProcessor)
fixture.sentryOptions.beforeSendFeedback = beforeSend
fixture.sentryOptions.onDiscard = onDiscard

val id = fixture.getSut().captureFeedback(Feedback("message"), null, createScope())

assertThat(id).isEqualTo(SentryId.EMPTY_ID)
verify(nextProcessor, never()).process(any<SentryEvent>(), anyOrNull())
verify(beforeSend, never()).execute(any(), anyOrNull())
verify(fixture.transport, never()).send(any(), anyOrNull())
assertClientReport(
fixture.sentryOptions.clientReportRecorder,
listOf(
DiscardedEvent(DiscardReason.CALLBACK_ERROR.reason, DataCategory.Feedback.category, 1)
),
)
verify(onDiscard).execute(DiscardReason.CALLBACK_ERROR, DataCategory.Feedback, 1)
}

// endregion

private fun givenScopeWithStartedSession(
Expand Down Expand Up @@ -4352,14 +4554,6 @@ class SentryClientTest {
override fun timestamp(): Long? = null
}

private fun eventProcessorThrows(): EventProcessor {
return object : EventProcessor {
override fun process(event: SentryEvent, hint: Hint): SentryEvent? {
throw Throwable()
}
}
}

private class BackfillableHint : Backfillable {
override fun shouldEnrich(): Boolean = false
}
Expand Down
Loading