Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
a59ac4f
Run bulk actions on everything matching a filter
ejsmith Nov 3, 2026
fe83fc0
Count selectable rows in the bulk delete e2e test
ejsmith Nov 3, 2026
74cbb57
Harden bulk operations: paging cost, active index, pinned stack filte…
ejsmith Nov 3, 2026
d8cd35e
Continue bulk operations in a new work item after temporary errors
ejsmith Nov 3, 2026
7080146
Renew the bulk operation lease after every batch
ejsmith Nov 3, 2026
dff385c
Reject unknown bulk selection kinds and never restart a finished id s…
ejsmith Nov 3, 2026
8928092
Fail a bulk operation that could not be queued
ejsmith Nov 3, 2026
11181b5
Pin relative dates inside a bulk operation's filter
ejsmith Nov 3, 2026
794c82e
Pin relative dates only on date fields and retire stale bulk operations
ejsmith Nov 3, 2026
629d02b
Pin relative dates in ranges that take their field from a group
ejsmith Nov 3, 2026
9914213
Merge remote-tracking branch 'origin/main' into feature/bulk-operations
ejsmith Nov 4, 2026
fd19ca0
Track bulk operations and project work as background tasks
ejsmith Nov 4, 2026
18d15d1
Show every organization task in the progress panel
ejsmith Nov 4, 2026
b6249f9
Add background task backend tests
ejsmith Nov 4, 2026
bd86727
Hide internal errors from task messages and let abandoned tasks be cl…
ejsmith Nov 4, 2026
a7e836d
Give the relative date freezer tests an Arrange section
ejsmith Nov 4, 2026
b81fe3a
Fix background task lifecycle and active task visibility
niemyjski Nov 10, 2026
9594bd4
Merge main into feature/bulk-operations
niemyjski Nov 13, 2026
d14373f
fix: fail safely when background task startup cannot complete
niemyjski Nov 13, 2026
b5593a5
fix: preserve confirmed deletions and asynchronous task updates
niemyjski Nov 13, 2026
a723561
fix: validate and pin bulk selection time ranges
niemyjski Nov 13, 2026
3d3cc99
test: separate background task endpoint journey phases
niemyjski Nov 13, 2026
94b990a
fix: retain newer task state and event selections
niemyjski Nov 13, 2026
affc791
fix: reject unsuccessful background task cache writes
niemyjski Nov 13, 2026
070ddde
fix: preserve bulk progress and close worker failure paths
niemyjski Nov 13, 2026
4e1b317
fix: preserve task reconciliation and selection intent
niemyjski Nov 13, 2026
dda123f
Merge remote-tracking branch 'origin/main' into feature/bulk-operations
niemyjski Nov 13, 2026
7231062
Pin bulk selections and harden task failure recovery
niemyjski Nov 13, 2026
aec942e
Refresh task lists when returning to an organization
niemyjski Nov 13, 2026
de51181
Preserve empty bulk selections through serialization
niemyjski Nov 13, 2026
fc804c0
Separate setup and actions in frontend tests
niemyjski Nov 13, 2026
27087d9
Harden task lifecycle and batch task discovery
niemyjski Nov 14, 2026
4566256
Load only required event fields for bulk deletion
niemyjski Nov 14, 2026
aa494c3
Keep task controls reachable and coalesce completion refreshes
niemyjski Nov 14, 2026
557b4fa
Document task contracts and reproducible performance measurements
niemyjski Nov 14, 2026
4e2826f
Keep background task architecture guide out of the public site
niemyjski Nov 14, 2026
01b02c3
Wait for dependent event metrics before browser remounts
niemyjski Nov 14, 2026
54f7cf2
Synchronize deletion assertions with search visibility
niemyjski Nov 14, 2026
826d4f4
Harden task state and bulk recovery boundaries
niemyjski Nov 14, 2026
aec2b97
Keep task discovery reliable during continuous progress
niemyjski Nov 14, 2026
ba4c3b3
Document task boundaries and expanded performance measurements
niemyjski Nov 14, 2026
2130ec1
Preserve worker outcomes through shutdown and accounting failures
niemyjski Nov 15, 2026
ec1125e
Reconcile task outcomes across initial discovery and pending reads
niemyjski Nov 15, 2026
b6b0779
Clarify task recovery and reconciliation guarantees
niemyjski Nov 15, 2026
e0320bb
Preserve task discovery through retries and history cleanup
niemyjski Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ Notes:
3. Use Aspire CLI 13.6.5 or newer (`aspire update --self` to update). The AppHost uses the CLI bundle for the dashboard and resource orchestrator; when no CLI is installed, the SDK downloads the matching version through DNX.
4. Try `dotnet build -mt` for multithreaded builds. Performance depends on the workload and machine.

See the [background task developer guide](docs/background-tasks.md) for task lifecycle, API contracts, cancellation, bulk execution, and extension points.

## UI Development

The Svelte 5 UI in `src/Exceptionless.Web/ClientApp` is the only application frontend and serves at `/`. Aspire starts Svelte together with the API. Primary hash-route bookmarks resolve to the current pages, and existing signed-in sessions are retained.
Expand Down
12 changes: 12 additions & 0 deletions benchmarks/BackgroundTasks/BackgroundTasks.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net11.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="../../src/Exceptionless.Core/Exceptionless.Core.csproj" />
</ItemGroup>
</Project>
176 changes: 176 additions & 0 deletions benchmarks/BackgroundTasks/Program.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
using System.Diagnostics;
using System.Text.Json;
using Exceptionless.Core.Models.BackgroundTasks;
using Exceptionless.Core.Models.BulkOperations;
using Exceptionless.Core.Services.BackgroundTasks;
using Foundatio.Caching;
using Foundatio.Jobs;
using Foundatio.Lock;
using Foundatio.Messaging;
using Foundatio.Queues;
using Foundatio.Repositories.Utility;
using Foundatio.Resilience;
using Microsoft.Extensions.Logging.Abstractions;

// A manual service microbenchmark, deliberately outside the correctness test suite.
// Compare the same command before and after a change. This does not measure Redis network latency.
string label = args.FirstOrDefault() ?? "current";
bool stress = args.Skip(1).Contains("--stress", StringComparer.Ordinal);
int[] sizes = args.Length > 1 && !stress ? args.Skip(1).Select(Int32.Parse).ToArray() : [100, 1_000, 10_000, 50_001];
Workload[] workloads = stress
? [new("large-selection-history", 1_000, 2, 1_000), new("many-active-tasks", 1_000, 200, 0), new("same-second-history", 1_000, 2, 1_000, SameSecond: true)]
: sizes.Select(size => new Workload("history", size, 2, 0)).ToArray();
foreach (var workload in workloads)
{
int historyCount = workload.HistoryCount;
using var cache = new InMemoryCacheClient(new InMemoryCacheClientOptions { CloneValues = true, MaxItems = null });
using var messageBus = new InMemoryMessageBus();
using var queue = new InMemoryQueue<WorkItemData>();
var measuredCache = new MeasuredCacheClient(cache);
var locks = new CacheLockProvider(cache, messageBus);
var tasks = new BackgroundTaskService(measuredCache, locks, messageBus, queue,
new ResiliencePolicyProvider(), new BackgroundTaskOptions(), TimeProvider.System, NullLogger<BackgroundTaskService>.Instance);

const string organizationId = "537650f3b77efe23a47914f4";
var createdUtc = DateTime.UtcNow.AddSeconds(-historyCount - workload.ActiveCount);
createdUtc = new DateTime(createdUtc.Ticks - createdUtc.Ticks % TimeSpan.TicksPerSecond, DateTimeKind.Utc);
var ids = new List<string>(historyCount + workload.ActiveCount);
var selectionIds = Enumerable.Range(0, workload.SelectionIds).Select(_ => ObjectId.GenerateNewId().ToString()).ToList();
for (int i = 0; i < historyCount + workload.ActiveCount; i++)
{
bool active = i < workload.ActiveCount;
var taskCreatedUtc = workload.SameSecond && !active
? createdUtc.AddSeconds(workload.ActiveCount).AddTicks(i - workload.ActiveCount)
: createdUtc.AddSeconds(i);
var task = new BackgroundTask
{
// Model servers whose ObjectId machine bytes disagree with precise creation order inside one second.
Id = workload.SameSecond && !active
? new ObjectId(taskCreatedUtc, machine: historyCount + workload.ActiveCount - i, pid: 1, increment: 0).ToString()
: ObjectId.GenerateNewId(taskCreatedUtc).ToString(),
OrganizationId = organizationId,
Type = "benchmark.work",
Description = "Benchmark task",
CreatedByUserId = "537650f3b77efe23a47914f5",
CreatedUtc = taskCreatedUtc,
UpdatedUtc = taskCreatedUtc,
Status = active ? BackgroundTaskStatus.Running : BackgroundTaskStatus.Completed,
Progress = active ? 10 : 100,
Bulk = selectionIds.Count == 0 ? null : new BulkOperationDetails
{
Selection = new BulkOperationSelection { Kind = BulkOperationSelectionKind.Ids, Ids = selectionIds }
}
};
ids.Add(task.Id);
await cache.SetAsync(BackgroundTaskService.GetTaskCacheKey(task.Id), task, BackgroundTaskService.TaskLifetime);
}

// Seed equivalent persisted state directly so index construction is outside the measurement.
await cache.ListAddAsync(BackgroundTaskService.GetOrganizationIndexCacheKey(organizationId), ids, BackgroundTaskService.TaskLifetime);
await cache.ListAddAsync(BackgroundTaskService.GetActiveIndexCacheKey(organizationId), ids.Take(workload.ActiveCount), BackgroundTaskService.TaskLifetime);
async Task<bool> RunAsync(string operation)
{
if (operation == "list")
{
var result = await tasks.GetByOrganizationAsync(organizationId);
return result.Count == Math.Min(historyCount, BackgroundTaskService.ListLimit) + workload.ActiveCount
&& result[0].Id == ids[^1]
&& result.Count(t => t.IsActive) == workload.ActiveCount;
}

var updated = await tasks.UpdateAsync(ids[0], task =>
{
task.Progress = operation == "progress" ? 20 : 99;
return operation == "progress";
});
return updated is { Progress: 20, IsActive: true };
}

foreach (string operation in new[] { "list", "progress", "no-op" })
{
for (int i = 0; i < 3; i++)
await RunAsync(operation);

int iterations = historyCount >= 10_000 ? 7 : 21;
var milliseconds = new List<double>(iterations);
var allocatedBytes = new List<long>(iterations);
bool correct = true;
for (int i = 0; i < iterations; i++)
{
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
measuredCache.Reset();
long beforeAllocation = GC.GetTotalAllocatedBytes(precise: true);
long started = Stopwatch.GetTimestamp();
correct &= await RunAsync(operation);
milliseconds.Add(Stopwatch.GetElapsedTime(started).TotalMilliseconds);
allocatedBytes.Add(GC.GetTotalAllocatedBytes(precise: true) - beforeAllocation);
}

milliseconds.Sort();
allocatedBytes.Sort();
Console.WriteLine(JsonSerializer.Serialize(new
{
label,
workload = workload.Name,
operation,
historyCount,
activeCount = workload.ActiveCount,
selectionIds = workload.SelectionIds,
iterations,
medianMilliseconds = milliseconds[iterations / 2],
medianAllocatedBytes = allocatedBytes[iterations / 2],
measuredCache.ListReads,
measuredCache.SingleReads,
measuredCache.BatchReads,
measuredCache.ListWrites,
measuredCache.ExpirationWrites,
correct
}));
}
}

sealed record Workload(string Name, int HistoryCount, int ActiveCount, int SelectionIds, bool SameSecond = false);

// Instrument calls while preserving the real cache implementation and its cloning behavior.
sealed class MeasuredCacheClient(ICacheClient inner) : ScopedCacheClient(inner, null), ICacheClient
{
public int ListReads { get; private set; }
public int SingleReads { get; private set; }
public int BatchReads { get; private set; }
public int ListWrites { get; private set; }
public int ExpirationWrites { get; private set; }

public void Reset() => ListReads = SingleReads = BatchReads = ListWrites = ExpirationWrites = 0;

Task<CacheValue<T>> ICacheClient.GetAsync<T>(string key)
{
SingleReads++;
return UnscopedCache.GetAsync<T>(key);
}

Task<IDictionary<string, CacheValue<T>>> ICacheClient.GetAllAsync<T>(IEnumerable<string> keys)
{
BatchReads++;
return UnscopedCache.GetAllAsync<T>(keys);
}

Task<CacheValue<ICollection<T>>> ICacheClient.GetListAsync<T>(string key, int? page, int pageSize)
{
ListReads++;
return UnscopedCache.GetListAsync<T>(key, page, pageSize);
}

Task<long> ICacheClient.ListAddAsync<T>(string key, IEnumerable<T> values, TimeSpan? expiresIn)
{
ListWrites++;
return UnscopedCache.ListAddAsync(key, values, expiresIn);
}

Task ICacheClient.SetExpirationAsync(string key, TimeSpan expiresIn)
{
ExpirationWrites++;
return UnscopedCache.SetExpirationAsync(key, expiresIn);
}
}
85 changes: 85 additions & 0 deletions benchmarks/BackgroundTasks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Background task service benchmark

This manual executable measures task listing, active progress updates, and skipped mutations using the real `BackgroundTaskService` and Foundatio in-memory cache. It is outside the normal test suite and solution. It adds no benchmark package or production dependency.

## Run

From the repository root with the .NET 11 SDK:

```sh
dotnet run --project benchmarks/BackgroundTasks/BackgroundTasks.csproj --configuration Release -- current
```

The first argument labels the results. Optional following arguments replace the finished-history sizes:

```sh
dotnet run --project benchmarks/BackgroundTasks/BackgroundTasks.csproj --configuration Release -- current 100 1000 10000 50001
```

The stress profile checks larger task payloads, a larger active index, and creation bursts across servers:

```sh
dotnet run --project benchmarks/BackgroundTasks/BackgroundTasks.csproj --configuration Release -- current --stress
```

| Stress workload | Finished tasks | Active tasks | Explicit selection IDs per task | Creation order |
| --- | ---: | ---: | ---: | --- |
| Large selection history | 1,000 | 2 | 1,000 | Separate seconds |
| Many active tasks | 1,000 | 200 | 0 | Separate seconds |
| Same-second history | 1,000 | 2 | 1,000 | All finished tasks in one second, with ID order opposite to precise creation order |

To compare revisions, create a separate worktree at the baseline, copy this folder's `Program.cs` and `.csproj` into the same location there, and run the same command in each checkout. Keep the SDK, configuration, dependencies, and machine identical. Run the measurements sequentially without simultaneous builds or test suites.

## What is measured

- Seed each history size plus the profile's older active tasks. Setup is outside the measurement. Cache values are cloned as in application use; capacity eviction is disabled so it does not change the selected workload.
- Warm each operation three times, then take 21 samples for smaller histories or seven for histories of 10,000 or more.
- Report median elapsed time and total managed allocation per call, including asynchronous work. Force collection outside each sample to make retained setup state comparable.
- Count service-level index reads, individual and batched record reads, index renewals, and expiry updates. The measuring decorator delegates directly to the underlying cache.
- Check that listing returns the newest finished task, the expected count, and every active task. Check that progress updates retain the running task and requested progress. A skipped mutation deliberately changes its local copy before returning false; the returned result must still describe the stored progress.

Allocation means bytes allocated during an operation, not retained process memory. This is a service microbenchmark, not an Elasticsearch throughput test or a Redis latency forecast. `GetAllAsync` enables batching, but provider behavior and Redis Cluster hash-slot placement determine actual network requests. The default profile uses two active entries; the stress profile measures a larger active index separately.

## Recorded comparison

Measured on Linux x64, .NET SDK 11.0.100, Release, Foundatio 13.0.5. The baseline is [`fc804c08`](https://github.com/exceptionless/Exceptionless/commit/fc804c084506f5f78fc80b11209b49e437423587). The candidate is the service implementation in [`54f7cf26`](https://github.com/exceptionless/Exceptionless/commit/54f7cf264bbeb3f6a79227edd5e1c3972d56bf55), before the same-second ordering and skipped-mutation corrections. [comparison.json](comparison.json) contains every recorded sample summary and the candidate service blob identifier.

| Finished history | Baseline listing | Candidate listing | Baseline allocation | Candidate allocation |
| ---: | ---: | ---: | ---: | ---: |
| 100 | 0.1340 ms | 0.1573 ms | 0.050 MB | 0.061 MB |
| 1,000 | 0.9845 ms | 0.6104 ms | 0.308 MB | 0.188 MB |
| 10,000 | 16.6519 ms | 5.2518 ms | 10.578 MB | 1.364 MB |
| 50,001 | 152.1701 ms | 17.0455 ms | 233.967 MB | 6.241 MB |

The 50,001 baseline result is **incorrect**: the old page cap omits the newest task. Its timing is included to expose the old work and failure, not as a comparison between two correct implementations. All candidate result checks passed.

At 10,000 finished tasks, both versions return the correct result; the candidate used about 87% fewer allocated bytes and 68% less elapsed time. The candidate performs two index reads and two batched record reads for these fixtures. The baseline performs 22 index reads and 52 individual record reads at 10,000 tasks, rising to 101 index reads at 50,001.

The small-history tradeoff is visible: at 100 tasks, dictionary batching adds about 12 KB of allocation and 0.023 ms of in-process work. It reduces sequential cache calls, which this local benchmark does not price as remote latency. Avoid adding provider-specific branches or caching layers to optimize that small absolute cost without a representative workload.

### Retention cost

Progress updates now renew the small active index and its key expiry so a task remains discoverable for the lifetime of its saved record. They do not rewrite the history index. With two active tasks, median allocation increased from 6,640 to 9,128 bytes at the first three history sizes: about 2.5 KB per update. At 1,000 history entries, the observed median was 0.0773 ms before and 0.1047 ms after. Submillisecond timings vary with runtime and host scheduling; the extra renewal and expiry calls are the stable change.

Terminal updates renew history before removing active membership. These measurements cover active progress, not terminal-history writes. Index retention is a correctness obligation; report its write cost alongside read improvements instead of claiming that every operation became faster.

## Remaining scaling boundary

The portable cache API has no ordered recent-history query. Listing still reads and sorts all live history IDs, and returning all active tasks requires materializing that active set. Correct ordering across servers also requires reading every candidate in the cutoff second, even when only a few finished records are requested. The service trims finished candidates between bounded read batches, so retained task payloads scale with active tasks, the requested limit, and one batch rather than the entire burst. A future store with ordered discovery can change that access pattern when real volume warrants it; the [task guide](../../docs/background-tasks.md) describes that extraction boundary.

## Correctness and stress comparison

[extended-comparison.json](extended-comparison.json) compares `54f7cf26` with the corrected service, using identical expanded benchmark source. It records both service blob identifiers, the benchmark blob, all 42 summaries, and the same environment and sampling rules. All 21 candidate checks passed. The baseline fails every skipped-mutation return check and the same-second listing check.

| Listing workload | Before | After | Allocation before | Allocation after | Record batches after |
| --- | ---: | ---: | ---: | ---: | ---: |
| 1,000 history, each with 1,000 selection IDs | 0.5614 ms | 0.5584 ms | 625,304 B | 623,520 B | 2 |
| 1,000 history, 200 active tasks | 1.0354 ms | 0.8396 ms | 371,792 B | 363,496 B | 5 |
| 10,000 history, 2 active tasks | 3.0749 ms | 3.0660 ms | 1,364,320 B | 1,362,536 B | 2 |
| 1,000 same-second tasks with large selections | 0.7746 ms **incorrect** | 5.0146 ms | 625,640 B **incorrect** | 9,282,976 B | 21 |

Correct same-second ordering costs more reads and decoding allocation because the old version silently discarded newer records. The corrected service retains only 50 finished candidates between batches, with at most another 50 candidates during trimming; it does not retain all 1,000 task payloads. This is an algorithmic bound, not a measured peak-memory claim.

The successful progress path keeps the same cache-call counts and allocation in these fixtures. At 200 active tasks it allocates 26,248 bytes per update, versus 9,208 with two active tasks; the cache's active-index work is a real scaling cost. A skipped mutation adds one read only on that path: 488 bytes for the small record and 8,896 bytes when its selection contains 1,000 IDs. This upholds the stored-result contract without cloning every successful update.

Small-history elapsed times varied in both directions: the 100-task list measured 0.1344 ms before and 0.1722 ms after, while its allocation fell by 1,784 bytes. These results do not support a claim that every operation became faster. The stable benefits are bounded retained history payloads, fewer temporary sorting allocations, unchanged successful-update cache calls, and correct results under creation bursts.
Loading
Loading