Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
5bf5dc8
GPU CMake: Mark GPUCA_DETERMINISTIC_MODE=1 invalid
davidrohr Nov 19, 2026
54f6f25
GPU TPC: Remove option to retry refit if cluster rejection breaks the…
davidrohr Sep 4, 2025
50ed2cd
GPU TPC: Add possibility to run last way of TPC fit in separate kerne…
davidrohr Sep 4, 2025
b198923
GPU TPC: Search hit closest to interpolated point in track rebuilding
davidrohr Sep 23, 2025
db6586f
GPU TPC: Fix index for hit interpolation in rows with multiple clusters
davidrohr Oct 5, 2025
6590cdb
GPU TPC: Implement ambiguity solving for rebuild tracks cluster assoc…
davidrohr Sep 29, 2025
33ee65c
GPU: Additional debug dumps
davidrohr Feb 19, 2026
778d176
GPU TPC: Split TPC Merger Fit() into multiple functions
davidrohr Oct 6, 2025
de29468
GPU: Add looperFollowMode option
davidrohr Oct 23, 2025
d3d6110
GPU: simplifaction, refactoring and some minor improvements
davidrohr Oct 23, 2025
7e007de
GPU TPC: Refactor cluster merging
davidrohr Oct 27, 2025
5cc435b
GPU: Change indexing of interpolated hit temporary data
davidrohr Oct 24, 2025
5318bc3
GPU: Move some functionality to CAMath
davidrohr Oct 27, 2025
6051641
GPU TPC: Store track position for interpolation also when not updatin…
davidrohr Oct 29, 2025
af80e5e
GPU TPC: Extract CE crossing handling in function, and apply before s…
davidrohr Oct 30, 2025
39e82a3
GPU TPC: If value from other side of interpolation not stored, use on…
davidrohr Oct 30, 2025
a427f2b
GPU TPC: Mark clusters of rebuilt track as high incli when interpolat…
davidrohr Oct 30, 2025
bdf7c31
GPU TPC: Pick up best hit from interpolation only, if current positio…
davidrohr Oct 30, 2025
65c1152
GPU TPC: Interpolate missing pad rows when rebuilding track
davidrohr Oct 31, 2025
0199544
Revert "GPU TPC: Remove option to retry refit if cluster rejection br…
davidrohr Oct 31, 2025
8186efe
GPU TPC: Relax chi2cut for single-pad clusters
davidrohr Nov 2, 2025
f6cc188
GPU TPC: Clean up and fix resetT0 code
davidrohr Nov 2, 2025
cbe59f3
GPU TPC: Don't break for high sinPhi if only the linearization model …
davidrohr Nov 2, 2025
cc02414
GPU: Add option to steer which attachment steps to run during rebuild
davidrohr Nov 18, 2025
57f4f59
GPU TPC: Extrapolate track inward and outward when rebuilding
davidrohr Nov 2, 2025
ba6593a
GPU: Add debug option to create temporary MC labels for collected mer…
davidrohr Dec 4, 2025
1166d05
GPU TPC: Correctly handle rowGap and fit Failures when interpolating …
davidrohr Nov 10, 2026
bc43f53
GPU TPC: Improve track extrapolation for better efficiency / less fak…
davidrohr Nov 10, 2026
7d4d7c6
GPU TPC: Let fallback clusters compete for ambiguity solving as all o…
davidrohr Nov 11, 2026
6158e28
GPU TPC: Add debug option to dump merged track ncl to text file
davidrohr Nov 11, 2026
73793ea
GPU TPC: Continuous and less strict (during refit) NCl cut for given Pt
davidrohr Nov 12, 2026
9a9226b
GPU GPUCA_CADEBUG: add possibility to have a condition for the debug …
davidrohr Nov 12, 2026
b6c1037
GPU TPC: Make extrapolation more robust by stopping early when there …
davidrohr Nov 12, 2026
006f5d4
GPU TPC: revert state after extrapolation in case of problems
davidrohr Nov 12, 2026
1136df4
GPU TPC: Fix some cases of fit failues after track rebuild (bad initi…
davidrohr Nov 12, 2026
c6c7031
GPU TPC: Fix deltaTimeFwd/deltaTimeBwd after track rebuild
davidrohr Nov 12, 2026
eed55d6
GPU TPC: Fix storing OuterParam after implementing track rebuild
davidrohr Nov 13, 2026
e3239be
GPU TPC: Option to run interpolation-rejection after track rebuild
davidrohr Nov 13, 2026
e7c5043
GPU TPC: Some fixes for NTolerated and nMissed, and some cleanup
davidrohr Nov 15, 2026
c0f3f46
GPU QA: Improve Efficiency Stat Printing
davidrohr Nov 16, 2026
23838ad
GPU TPC: Make AttachClusters work with track rebuilding
davidrohr Nov 16, 2026
b2e3ab9
GPU TPC: Mark hits with findAdjacentClusters done in previous iterati…
davidrohr Nov 17, 2026
859770d
GPU TPC: Fix cluster adjacent flags with Rebuild (and improve without)
davidrohr Nov 17, 2026
2d620b6
GPU QA: Add histograms for adjacent clusters without rejection, Fix s…
davidrohr Nov 17, 2026
8fe99e5
GPU TPC: Add tpcSortClusters option
davidrohr Nov 17, 2026
35768af
GPU Standalone / QA: Add testSyncAsyncMC option to pass cluster MC la…
davidrohr Nov 17, 2026
941fa0f
GPU Standalone: add testSyncAsyncNoSyncSettings option
davidrohr Nov 17, 2026
7438ba4
GPU QA: Add extra histograms for pt > 50 / 200 MeV
davidrohr Nov 17, 2026
18a8b41
GPU Math: Add support for half-precision min/max in CUDA
davidrohr Nov 18, 2026
ec7b77b
GPU CPUDevice: Add consistency check for some RTC parameters
davidrohr Nov 18, 2026
5b5d978
GPU TPC: synchronize threads in warp between fit iterations to reduce…
davidrohr Nov 19, 2026
68affbc
GPU: Provide separate fit kernels for rebuild/retry, remove mergerSor…
davidrohr Nov 18, 2026
d966c19
GPU TPC: Refactor rebuild data structures to remove GPU memory usage
davidrohr Nov 18, 2026
b414331
GPU TPC: Faster implementation for search adjacent clusters in missin…
davidrohr Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Detectors/TPC/reconstruction/test/testGPUCATracking.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ BOOST_AUTO_TEST_CASE(CATracking_test1)
config.configGRP.solenoidBzNominalGPU = solenoidBz;
config.configGRP.grpContinuousMaxTimeBin = continuous ? GPUSettings::TPC_MAX_TF_TIME_BIN : 0; // Number of timebins in timeframe if continuous, 0 otherwise

config.configReconstruction.tpc.nWays = 3; // Should always be 3!
config.configReconstruction.tpc.searchWindowDZDR = 2.5f; // Should always be 2.5 for looper-finding and/or continuous tracking
config.configReconstruction.tpc.trackReferenceX = refX;

Expand Down
14 changes: 14 additions & 0 deletions GPU/Common/GPUCommonMath.h
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ class GPUCommonMath
GPUd() constexpr static double Copysignd(double x, double y);
GPUd() constexpr static float TwoPi() { return 6.2831853f; }
GPUd() constexpr static float Pi() { return 3.1415927f; }
GPUd() constexpr static float Deg2Rad() { return Pi() / 180.f; }
GPUd() constexpr static float Round(float x);
GPUd() constexpr static float Floor(float x) { return GPUCA_CHOICE(floorf(x), floorf(x), floor(x)); }
GPUd() static uint32_t Float2UIntReint(const float& x);
Expand Down Expand Up @@ -235,6 +236,19 @@ class GPUCommonMath

typedef GPUCommonMath CAMath;

#ifdef __CUDACC__
template <>
GPUhd() half GPUCommonMath::Min(const half x, const half y) // BUG: CUDA __hmin / __hmax is not constexpr, fix once fixed in CUDA
{
return __hmin(x, y);
}
template <>
GPUhd() half GPUCommonMath::Max(const half x, const half y)
{
return __hmax(x, y);
}
#endif

template <typename... Args>
GPUhdi() constexpr float GPUCommonMath::Sum2(float w, Args... args)
{
Expand Down
1 change: 1 addition & 0 deletions GPU/GPUTracking/Base/GPUParam.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ void GPUParam::UpdateSettings(const GPUSettingsGRP* g, const GPUSettingsProcessi
if (d) {
rec.dyn = *d;
}
rec.tpc.nWays = 1 + 2 * ((rec.tpc.mergerInitial3wayFit > 0) + (rec.tpc.mergerInterpolateErrors > 0) + (rec.tpc.rebuildTrackInFit > 0));
}

void GPUParam::UpdateBzOnly(float newSolenoidBz, bool assumeConstantBz)
Expand Down
4 changes: 2 additions & 2 deletions GPU/GPUTracking/Base/GPUParam.h
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ struct GPUParam : public internal::GPUParam_t<GPUSettingsRec, GPUSettingsParam>
void UpdateRun3ClusterErrors(const float* yErrorParam, const float* zErrorParam);
#endif

GPUd() constexpr uint32_t tpcMinHitsB5(float qPtB5) const { return CAMath::Abs(qPtB5) > 10 ? 10 : (CAMath::Abs(qPtB5) > 5 ? 15 : 29); } // Minimum hits should depend on Pt, low Pt tracks can have few hits. 29 Hits default, 15 for < 200 mev, 10 for < 100 mev
GPUd() constexpr float Alpha(int32_t iSector) const // TODO: Check if this is faster, or the lookup from GPUTPCGeometry
GPUd() constexpr uint32_t tpcMinHitsB5Abs(float qPtB5) const { return qPtB5 > 10.f ? 10 : (qPtB5 > 5.f ? (uint32_t)(20 - qPtB5) : (qPtB5 > 3.6f ? (uint32_t)(65 - 10.f * qPtB5) : 29)); }
GPUd() constexpr float Alpha(int32_t iSector) const // TODO: Check if this is faster, or the lookup from GPUTPCGeometry
{
if (iSector >= (int32_t)o2::tpc::constants::MAXSECTOR / 2) {
iSector -= o2::tpc::constants::MAXSECTOR / 2;
Expand Down
5 changes: 1 addition & 4 deletions GPU/GPUTracking/Base/GPUReconstruction.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -295,15 +295,12 @@ int32_t GPUReconstruction::InitPhaseBeforeDevice()
if (GetProcessingSettings().deterministicGPUReconstruction && GetProcessingSettings().debugLevel >= 6) {
mProcessingSettings->nTPCClustererLanes = 1;
}
if (GetProcessingSettings().createO2Output > 1 && GetProcessingSettings().runQA && GetProcessingSettings().qcRunFraction == 100.f) {
if (GetProcessingSettings().createO2Output > 1 && ((GetProcessingSettings().runQA && GetProcessingSettings().qcRunFraction == 100.f) || GetProcessingSettings().debug.mergerDumpTrackNCl)) {
mProcessingSettings->createO2Output = 1;
}
if (!GetProcessingSettings().createO2Output || !IsGPU()) {
mProcessingSettings->clearO2OutputFromGPU = false;
}
if (!(mRecoSteps.stepsGPUMask & gpudatatypes::RecoStep::TPCMerging)) {
mProcessingSettings->mergerSortTracks = false;
}
if (GetProcessingSettings().debugLevel > 3 || !IsGPU() || GetProcessingSettings().deterministicGPUReconstruction) {
mProcessingSettings->delayedOutput = false;
}
Expand Down
4 changes: 2 additions & 2 deletions GPU/GPUTracking/Base/GPUReconstructionCPU.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -216,10 +216,10 @@ int32_t GPUReconstructionCPU::ExitDevice()
int32_t GPUReconstructionCPU::RunChains()
{
mMemoryScalers->temporaryFactor = 1.;
if (GetProcessingSettings().memoryScalingFuzz) {
if (GetProcessingSettings().debug.memoryScalingFuzz) {
static std::mt19937 rng;
static std::uniform_int_distribution<uint64_t> dist(0, 1000000);
uint64_t fuzzFactor = GetProcessingSettings().memoryScalingFuzz == 1 ? dist(rng) : GetProcessingSettings().memoryScalingFuzz;
uint64_t fuzzFactor = GetProcessingSettings().debug.memoryScalingFuzz == 1 ? dist(rng) : GetProcessingSettings().debug.memoryScalingFuzz;
GPUInfo("Fuzzing memory scaling factor with %lu", fuzzFactor);
mMemoryScalers->fuzzScalingFactor(fuzzFactor);
}
Expand Down
12 changes: 9 additions & 3 deletions GPU/GPUTracking/Base/cpudevice/GPUReconstructionCPUDevice.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,7 @@ GPUReconstructionCPUDevice::GPUReconstructionCPUDevice(const GPUSettingsDeviceBa
{
if (mMaster == nullptr) {
mInternals = new GPUReconstructionCPUDeviceInternals;
*mParDevice = *mParCPU; // We run the CPU kernels on the "device", so we use the CPU kernel parameters
mParDevice->par_ALTERNATE_BORDER_SORT = 1; // Required by the GPU code path without RTC
mParDevice->par_CF_SCAN_WORKGROUP_SIZE = 1; // No block-wide scan with 1 thread per block, the clusterizer uses the CPU stream compaction
*mParDevice = *mParCPU; // We run the host-compiled CPU kernels on the "device", so the device parameters must be the CPU kernel parameters. Other values are possible only with RTC (genRTC), checked in InitDevice_Runtime
for (auto& v : mParDevice->par_LB_maxThreads) {
v = 1;
}
Expand Down Expand Up @@ -72,6 +70,14 @@ int32_t GPUReconstructionCPUDevice::InitDevice_Runtime()
exit(0);
}

if (mProcessingSettings->alternateBorderSort == -1) {
mProcessingSettings->alternateBorderSort = 1; // Required by the GPU code path without RTC
}
if (!GetProcessingSettings().rtc.enable && memcmp(&mParDevice->par_LB_forceBlocks + 1, &mParCPU->par_LB_forceBlocks + 1, (const char*)(mParCPU + 1) - (const char*)(&mParCPU->par_LB_forceBlocks + 1))) { // All members behind the launch bounds
GPUError("CPUDevice kernel parameters differ from the parameters the host kernels were compiled with, this is possible only with RTC");
return 1;
}

if (mMaster == nullptr) {
mMultiprocessorCount = std::max<int32_t>(1, mMaxHostThreads); // One device block runs as one TBB task
mThreadCount = 1;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ int32_t GPUReconstructionCPUDevice::genRTC(std::string& filename, uint32_t& nCom
const std::string baseCommand = rtc::baseCommand(proc, std::string_view(_binary_GPUReconstructionCPUDevicertc_command_start, _binary_GPUReconstructionCPUDevicertc_command_len)) + " " + proc.rtctech.overrideArchitecture;
const std::string noFastMath(_binary_GPUReconstructionCPUDevicertc_command_no_fast_math_start, _binary_GPUReconstructionCPUDevicertc_command_no_fast_math_len);

mParDevice->par_CF_SCAN_WORKGROUP_SIZE = 1; // Must match the 1 thread per block for the scan kernels to compile, the clusterizer uses the CPU stream compaction
rtc::loadLaunchBounds(proc, *mParDevice);
const std::string launchBounds = rtc::launchBoundsCode(proc, *mParDevice, 0, mWarpSize);
if (proc.rtctech.printLaunchBounds >= 2) {
Expand Down
1 change: 1 addition & 0 deletions GPU/GPUTracking/DataCompression/GPUTPCCompression.h
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ class GPUTPCCompression : public GPUProcessor
void DumpCompressedClusters(std::ostream& out);
#endif
GPUd() bool rejectCluster(int32_t idx, const GPUParam& param, const GPUTrackingInOutPointers& ioPtrs) const;
GPUd() static bool isClusterRejected(int32_t idx, const GPUParam& param, const GPUTrackingInOutPointers& ioPtrs);

protected:
struct memory {
Expand Down
12 changes: 7 additions & 5 deletions GPU/GPUTracking/DataCompression/GPUTPCCompressionKernels.cxx
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ GPUdii() void GPUTPCCompressionKernels::Thread<GPUTPCCompressionKernels::step0at

int32_t hitId = hit.num;
int32_t attach = ioPtrs.mergedTrackHitAttachment[hitId];
if ((attach & gputpcgmmergertypes::attachTrackMask) != i) {
if (gputpcgmmergertypes::getAttachedTrackId(attach) != i) {
continue; // Main attachment to different track
}
bool rejectCluster = processors.param.rec.tpc.rejectionStrategy >= GPUSettings::RejectionStrategyA && !(attach & gputpcgmmergertypes::attachProtect) && (rejectTrk || GPUTPCClusterRejection::GetIsRejected(attach));
Expand Down Expand Up @@ -189,9 +189,11 @@ GPUd() bool GPUTPCCompressionKernels::GPUTPCCompressionKernels_Compare<4>::opera

GPUd() bool GPUTPCCompression::rejectCluster(int32_t idx, const GPUParam& GPUrestrict() param, const GPUTrackingInOutPointers& GPUrestrict() ioPtrs) const
{
if (mClusterStatus[idx]) {
return true;
}
return mClusterStatus[idx] || isClusterRejected(idx, param, ioPtrs);
}

GPUd() bool GPUTPCCompression::isClusterRejected(int32_t idx, const GPUParam& GPUrestrict() param, const GPUTrackingInOutPointers& GPUrestrict() ioPtrs)
{
int32_t attach = ioPtrs.mergedTrackHitAttachment[idx];
bool unattached = attach == 0;

Expand All @@ -206,7 +208,7 @@ GPUd() bool GPUTPCCompression::rejectCluster(int32_t idx, const GPUParam& GPUres
if (attach & gputpcgmmergertypes::attachProtect) {
return false;
}
int32_t id = attach & gputpcgmmergertypes::attachTrackMask;
int32_t id = gputpcgmmergertypes::getAttachedTrackId(attach);
auto& trk = ioPtrs.mergedTracks[id];
if (GPUTPCClusterRejection::IsTrackRejected(trk, param)) {
return true;
Expand Down
1 change: 1 addition & 0 deletions GPU/GPUTracking/DataTypes/GPUDataTypesIO.h
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ struct GPUTrackingInOutPointers {
uint32_t nOutputClusRefsTPCO2 = 0;
const o2::MCCompLabel* outputTracksTPCO2MC = nullptr;
const o2::tpc::CompressedClustersFlat* tpcCompressedClusters = nullptr;
const o2::dataformats::ConstMCTruthContainerView<o2::MCCompLabel>* tpcCompressedClustersMC = nullptr;
const o2::tpc::ClusterNativeAccess* clustersNativeReduced = nullptr;

// TPC links
Expand Down
6 changes: 6 additions & 0 deletions GPU/GPUTracking/DataTypes/GPUTPCGeometry.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@
#define GPUTPCGEOMETRY_H

#include "GPUCommonDef.h"
#include "GPUCommonMath.h"
#include "DataFormatsTPC/Constants.h"
#ifndef GPUCA_GPUCODE_DEVICE
#include <cmath>
#endif

namespace o2::gpu
{
Expand Down Expand Up @@ -174,6 +178,8 @@ class GPUTPCGeometry
const float v = (sector >= NSECTORS / 2) ? -z : z;
return (250.f - v) * FACTOR_Z2T; // Used in compression, must remain constant at 250cm
}

GPUd() static constexpr float kSectAngle() { return 2 * CAMath::Pi() / 18.f; }
};

} // namespace o2::gpu
Expand Down
22 changes: 18 additions & 4 deletions GPU/GPUTracking/Definitions/GPUDef.h
Original file line number Diff line number Diff line change
Expand Up @@ -70,16 +70,30 @@
#ifdef CADEBUG
#undef CADEBUG
#endif
#if GPUCA_CADEBUG == 1 && !defined(GPUCA_GPUCODE)
#define CADEBUG(...) __VA_ARGS__
#define CADEBUG2(cmd, ...) {__VA_ARGS__; cmd;}
#define GPUCA_CADEBUG_ENABLED
#ifdef CADEBUG2
#undef CADEBUG2
#endif
#ifdef CADEBUG3
#undef CADEBUG3
#endif
#if GPUCA_CADEBUG >= 1 && !defined(GPUCA_GPUCODE)
#if GPUCA_CADEBUG >= 2
#define CADEBUG(...) if (CADEBUG_CONDITION) {__VA_ARGS__;}
#define CADEBUG2(cmd, ...) if (CADEBUG_CONDITION) {__VA_ARGS__; cmd;}
#define GPUCA_CADEBUG_ENABLED 2
#else
#define CADEBUG(...) __VA_ARGS__
#define CADEBUG2(cmd, ...) {__VA_ARGS__; cmd;}
#define GPUCA_CADEBUG_ENABLED 1
#endif
#define CADEBUG3(...) __VA_ARGS__
#endif
#undef GPUCA_CADEBUG
#endif

#ifndef CADEBUG
#define CADEBUG(...)
#define CADEBUG2(cmd, ...) {cmd;}
#define CADEBUG3(...)
#endif
// clang-format on
2 changes: 1 addition & 1 deletion GPU/GPUTracking/Definitions/GPUGetConstexpr.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@

// This is a temporary workaround required for clang (with c++20), until we can go to C++23 with P2280R4, which allows getting constexpr static values from references

#if defined(__clang__) && __cplusplus >= 202002L && __cplusplus < 202302L
#if (defined(__clang__) && __cplusplus >= 202002L && __cplusplus < 202302L) || defined(__CPUDEVICE__) // BUG: The workaround is needed for GCC if used inside a templated function.

namespace o2::gpu::internal
{
Expand Down
Loading