diff --git a/.gitignore b/.gitignore index 99bca7d..48539b4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # Generated files build/ +build-*/ generated/ # Prerequisites diff --git a/docs/build.md b/docs/build.md index b329859..8d5b299 100644 --- a/docs/build.md +++ b/docs/build.md @@ -106,6 +106,10 @@ python3 scripts/run_performance_tests.py \ --output perf-current.json ``` +It runs every performance binary, tags each result with the commit, compiler, and +backend, and merges them into one file. Pass `--test ` (repeatable) to run +a subset — `--test perf_memory_pool` for the allocator work alone. + Compare two local runs with: ```bash @@ -118,6 +122,56 @@ Each result reports `unit`, `mean`, and `median` as separate fields. The default memory sweep runs up to 16 MiB; set `INFINI_RT_PERF_ENABLE_LARGE=1` to include the 256 MiB case. +`perf_memory_pool` is an A/B benchmark: it runs each workload twice, once +straight through the backend allocator and once through `ArenaMemoryPool`. Both +arms share the workload, the iteration count, and the unit, and emit one JSON row +each differing only in the `allocator` param (`direct` vs `arena`), so a consumer +can divide one by the other. A human-readable speedup table is also written to +stderr at the end of the run; stdout stays pure JSON. + +The paired workloads are `SingleBlock` (allocate one block, free it, repeat), +`WorkingSetChurn` (rotate a window of 8 live blocks, the shape an inference loop +produces), `MixedSizeClasses` (rotate 32 sizes, the arena's most favorable +shape), `FirstTouchGrowth` (2000 live blocks with nothing freed until the end), +`ThreadScaling` (up to 64 threads on one device, which exposes the cost of the +arena's single mutex), and `ConcurrentMixedSizes` (the same thread counts but +with 16 sizes per thread). Unpaired arena-only rows — `MissPath`, +`ConcurrentMissPath`, `AlignedHit`, `ReleaseCached`, `GetStats`, +`AllocateZeroBytes`, `DeallocateNullptr` — measure costs that exist only for the +arena. `ConcurrentMissPath` drops every cached block after each operation, so it +prices the upstream call under contention rather than the cache hit. + +Concurrent workloads calibrate their op count at run time to fill a ~40 ms +sample window. A fixed count would let a fast allocator finish a batch in a few +microseconds, where thread startup and scheduler placement dominate the +measurement; `ops_per_thread` is therefore an output in the JSON params rather +than a constant, and it differs between backends and between the two arms. + +`perf_allocator_matrix` covers what `perf_memory_pool` does not: gigabyte-scale +growth, trim cost split into bookkeeping and upstream frees, device-only effects, +and `cudaMallocAsync` as a third arm. `scripts/compare_allocators.py` configures +both a CPU and an NVIDIA build, runs it in each, and prints `direct` vs `arena` +and `cuda_async` vs `arena` per backend: + +```bash +python3 scripts/compare_allocators.py --jobs 32 +python3 scripts/compare_allocators.py --quick --backend cpu # shorter arms +``` + +The arena is instantiated over the `runtime::` dispatch API, so one binary +measures whichever backend the library was built with. To compare backends by +hand, configure one build dir per backend and run each: + +```bash +cmake -S . -B build-perf-cuda \ + -DCMAKE_BUILD_TYPE=Release \ + -DWITH_CPU=OFF -DWITH_NVIDIA=ON \ + -DINFINI_RT_BUILD_PERFORMANCE_TESTING=ON +cmake --build build-perf-cuda -j +python3 scripts/run_performance_tests.py \ + --build-dir build-perf-cuda --output perf-nvidia.json +``` + ## Documentation Enable the Doxygen documentation target with: diff --git a/include/infini/rt/arena_memory_pool.h b/include/infini/rt/arena_memory_pool.h new file mode 100644 index 0000000..b531f35 --- /dev/null +++ b/include/infini/rt/arena_memory_pool.h @@ -0,0 +1,1030 @@ +#ifndef INFINI_RT_ARENA_MEMORY_POOL_H_ +#define INFINI_RT_ARENA_MEMORY_POOL_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "infini/rt/detail/node_arena.h" +#include "infini/rt/detail/pointer_table.h" + +namespace infini::rt { + +struct DefaultArenaConfig { + static constexpr std::size_t kInitialCapacity = 64ull << 20; + + static constexpr std::size_t kMaxCapacity = 512ull << 20; + + static constexpr std::size_t kSmallThreshold = 1ull << 20; + + static constexpr std::size_t kMinSliceAlignment = 512; + + static constexpr std::size_t kMinSplitRemainder = 512; + + static constexpr std::size_t kShrinkThreshold = 16; + + static constexpr std::uint32_t kEmptyScansToDestroy = 2; + + static constexpr std::size_t kThreadCacheDepth = 1; + + static constexpr std::size_t kThreadCacheBytes = 8ull << 20; +}; + +namespace detail { + +template +struct ArenaCacheDepth + : std::integral_constant {}; + +template +struct ArenaCacheDepth> + : std::integral_constant {}; + +template +struct ArenaCacheBytes + : std::integral_constant {}; + +template +struct ArenaCacheBytes> + : std::integral_constant {}; + +} + +template +class ArenaMemoryPool { + public: + using Error = typename Upstream::Error; + + struct Stats { + std::size_t bytes_in_use = 0; + + std::size_t bytes_reserved = 0; + + std::size_t peak_bytes_in_use = 0; + + std::size_t peak_bytes_reserved = 0; + + std::size_t alloc_count = 0; + + std::size_t free_count = 0; + + std::size_t cache_hit_count = 0; + + std::size_t cache_miss_count = 0; + + std::size_t upstream_alloc_count = 0; + + std::size_t upstream_free_count = 0; + + std::size_t backing_count = 0; + + std::size_t bytes_free_in_backings = 0; + + std::size_t largest_free_chunk = 0; + + std::size_t bytes_internal_waste = 0; + + std::size_t bytes_unusable = 0; + + std::size_t shrink_count = 0; + }; + + ArenaMemoryPool() + : registry_(std::make_shared()), mutex_(registry_->mutex) { + registry_->pool = this; + } + + ArenaMemoryPool(const ArenaMemoryPool&) = delete; + ArenaMemoryPool& operator=(const ArenaMemoryPool&) = delete; + + ~ArenaMemoryPool() { + { + std::lock_guard lock(registry_->mutex); + for (ThreadCache* cache = registry_->head; cache != nullptr; + cache = cache->next) { + ReclaimCacheLocked(cache); + } + registry_->pool = nullptr; + } + + for (const auto& backing : backings_) { + DestroyChunks(backing.get()); + Upstream::Free(backing->base); + } + } + + Error Allocate(void** ptr, std::size_t size, std::size_t alignment = 0) { + if (ptr == nullptr) { + return InvalidValue(); + } + + *ptr = nullptr; + if (size == 0) { + return Upstream::kSuccess; + } + + const std::size_t align = SliceAlignment(alignment); + const std::size_t rounded = RoundSize(size); + + if (align == Config::kMinSliceAlignment) { + if (void* cached = TryCacheAllocate(rounded); cached != nullptr) { + *ptr = cached; + return Upstream::kSuccess; + } + } + + const std::size_t needed = rounded + align - Config::kMinSliceAlignment; + + std::unique_lock lock(mutex_); + + { + std::vector> doomed; + UpdateShrinkState(size, &doomed); + if (!doomed.empty()) { + lock.unlock(); + FreeBackings(doomed); + doomed.clear(); + lock.lock(); + } + } + + if (Chunk* chunk = FindFit(needed); chunk != nullptr) { + ++stats_.cache_hit_count; + *ptr = Serve(chunk, rounded, align); + return Upstream::kSuccess; + } + + std::size_t capacity = NextCapacity(needed); + lock.unlock(); + + void* base = nullptr; + Error status = Upstream::Malloc(&base, capacity); + if (status != Upstream::kSuccess) { + status = AllocateFallback(&base, &capacity, needed, status); + } + + lock.lock(); + + if (status != Upstream::kSuccess) { + if (Chunk* chunk = FindFit(needed); chunk != nullptr) { + ++stats_.cache_hit_count; + *ptr = Serve(chunk, rounded, align); + return Upstream::kSuccess; + } + return status; + } + + ++stats_.upstream_alloc_count; + Chunk* chunk = AdoptBacking(base, capacity); + ++stats_.cache_miss_count; + *ptr = Serve(chunk, rounded, align); + return Upstream::kSuccess; + } + + Error Deallocate(void* ptr) { + if (ptr == nullptr) { + return Upstream::kSuccess; + } + + std::lock_guard lock(mutex_); + + Chunk* chunk = nullptr; + if (!allocated_.Find(ptr, &chunk) || chunk->cached) { + return InvalidValue(); + } + + ++stats_.free_count; + + const std::size_t bin = BinIndex(chunk->size); + if (bin != kNoBin) { + if (ThreadCache* cache = LocalCacheIfPresent(); + cache != nullptr && TryParkCached(cache, bin, chunk)) { + return Upstream::kSuccess; + } + } + + allocated_.Take(ptr, &chunk); + ReleaseChunk(chunk); + return Upstream::kSuccess; + } + + void ReleaseCached() { + std::vector> doomed; + { + std::lock_guard lock(mutex_); + + ReclaimAllCaches(); + for (std::size_t i = backings_.size(); i-- > 0;) { + if (IsDrained(backings_[i].get())) { + doomed.push_back(Detach(i)); + } + } + if (backings_.empty()) { + next_capacity_ = Config::kInitialCapacity; + } + } + + FreeBackings(doomed); + } + + Stats GetStats() const { + std::lock_guard lock(mutex_); + Stats stats = stats_; + stats.backing_count = backings_.size(); + stats.largest_free_chunk = LargestFreeChunk(); + + std::size_t cached_bytes = 0; + SumCaches(&cached_bytes, &stats.alloc_count, &stats.cache_hit_count); + + stats.bytes_in_use -= cached_bytes; + stats.bytes_free_in_backings += cached_bytes; + return stats; + } + + private: + static_assert( + std::is_invocable_v, + "`Upstream::Malloc` must be callable with `(void**, size_t)`."); + static_assert(std::is_invocable_v, + "`Upstream::Free` must be callable with `(void*)`."); + static_assert( + std::is_same_v, Error>, + "`Upstream` must define `static constexpr Error kSuccess`."); + static_assert( + Config::kMinSliceAlignment != 0 && + (Config::kMinSliceAlignment & (Config::kMinSliceAlignment - 1)) == 0, + "`Config::kMinSliceAlignment` must be a power of two."); + static_assert(Config::kInitialCapacity <= Config::kMaxCapacity, + "`Config::kInitialCapacity` must not exceed " + "`Config::kMaxCapacity`."); + + struct BackingStore; + + enum class Location : std::uint8_t { kNone, kFastBin, kTree }; + + struct Chunk { + BackingStore* owner = nullptr; + Chunk* prev = nullptr; + Chunk* next = nullptr; + void* ptr = nullptr; + std::size_t size = 0; + + std::size_t requested = 0; + bool is_free = true; + + bool cached = false; + + Location location = Location::kNone; + + Chunk* fast_prev = nullptr; + Chunk* fast_next = nullptr; + }; + + struct BackingStore { + void* base = nullptr; + std::size_t capacity = 0; + Chunk* head = nullptr; + + std::size_t live_chunks = 0; + + bool resident = false; + + bool oversize = false; + std::uint32_t empty_scans = 0; + }; + + struct BySizeThenAddress { + bool operator()(const Chunk* lhs, const Chunk* rhs) const { + if (lhs->size != rhs->size) { + return lhs->size < rhs->size; + } + return reinterpret_cast(lhs->ptr) < + reinterpret_cast(rhs->ptr); + } + }; + + using FreeSet = + std::set>; + + static constexpr std::size_t kFastBinCount = 128; + static constexpr std::size_t kFastBinWords = (kFastBinCount + 63) / 64; + + static constexpr std::size_t kNoBin = static_cast(-1); + + static Error InvalidValue() { return static_cast(1); } + + static std::size_t CountTrailingZeros(std::uint64_t word) { +#if defined(__GNUC__) || defined(__clang__) + return static_cast(__builtin_ctzll(word)); +#else + std::size_t count = 0; + while ((word & 1ull) == 0) { + word >>= 1; + ++count; + } + return count; +#endif + } + + static std::size_t BinIndex(std::size_t size) { + if (size == 0 || + (size & (Config::kMinSliceAlignment - 1)) != 0) { + return kNoBin; + } + const std::size_t multiples = size / Config::kMinSliceAlignment; + return multiples <= kFastBinCount ? multiples - 1 : kNoBin; + } + + static std::size_t FirstEligibleBin(std::size_t needed) { + const std::size_t multiples = + (needed + Config::kMinSliceAlignment - 1) / Config::kMinSliceAlignment; + if (multiples == 0) { + return 0; + } + return multiples <= kFastBinCount ? multiples - 1 : kNoBin; + } + + static constexpr std::size_t kCacheDepth = + detail::ArenaCacheDepth::value; + static constexpr std::size_t kCacheBytes = + detail::ArenaCacheBytes::value; + + struct ThreadCache { + Chunk* lists[kFastBinCount] = {}; + std::size_t depths[kFastBinCount] = {}; + std::size_t bytes = 0; + + std::size_t alloc_count = 0; + std::size_t cache_hit_count = 0; + + std::atomic busy{false}; + + ThreadCache* next = nullptr; + }; + + struct Registry { + std::mutex mutex; + ArenaMemoryPool* pool = nullptr; + ThreadCache* head = nullptr; + }; + + class ThreadCacheHandle { + public: + explicit ThreadCacheHandle(std::shared_ptr registry) + : registry_(std::move(registry)), cache_(new ThreadCache) { + std::lock_guard lock(registry_->mutex); + cache_->next = registry_->head; + registry_->head = cache_; + } + + ~ThreadCacheHandle() { + { + std::lock_guard lock(registry_->mutex); + if (registry_->pool != nullptr) { + registry_->pool->RetireCacheLocked(cache_); + } + Unlink(cache_); + } + delete cache_; + } + + ThreadCacheHandle(const ThreadCacheHandle&) = delete; + ThreadCacheHandle& operator=(const ThreadCacheHandle&) = delete; + + ThreadCache* get() const { return cache_; } + + private: + void Unlink(ThreadCache* cache) { + ThreadCache** link = ®istry_->head; + while (*link != nullptr && *link != cache) { + link = &(*link)->next; + } + if (*link != nullptr) { + *link = cache->next; + } + } + + std::shared_ptr registry_; + ThreadCache* cache_; + }; + + class CacheGuard { + public: + explicit CacheGuard(ThreadCache* cache) : cache_(cache) { + while (cache_->busy.exchange(true, std::memory_order_acquire)) { + } + } + ~CacheGuard() { cache_->busy.store(false, std::memory_order_release); } + + CacheGuard(const CacheGuard&) = delete; + CacheGuard& operator=(const CacheGuard&) = delete; + + private: + ThreadCache* cache_; + }; + + struct CacheEntry { + const Registry* key; + std::unique_ptr handle; + }; + + static std::vector& CacheMap() { + static thread_local std::vector caches; + return caches; + } + + ThreadCache* LocalCache() { + if (ThreadCache* cache = LocalCacheIfPresent(); cache != nullptr) { + return cache; + } + + CacheMap().push_back( + CacheEntry{registry_.get(), + std::make_unique(registry_)}); + return CacheMap().back().handle->get(); + } + + ThreadCache* LocalCacheIfPresent() { + for (const CacheEntry& entry : CacheMap()) { + if (entry.key == registry_.get()) { + return entry.handle->get(); + } + } + return nullptr; + } + + static std::size_t RoundUp(std::size_t size, std::size_t granularity) { + return (size + granularity - 1) / granularity * granularity; + } + + static std::size_t RoundSize(std::size_t size) { + return RoundUp(size, Config::kMinSliceAlignment); + } + + static std::size_t SliceAlignment(std::size_t alignment) { + return alignment > Config::kMinSliceAlignment ? alignment + : Config::kMinSliceAlignment; + } + + static void* Offset(void* ptr, std::size_t bytes) { + return static_cast(ptr) + bytes; + } + + static std::size_t AlignPadding(void* ptr, std::size_t alignment) { + const auto address = reinterpret_cast(ptr); + const auto aligned = (address + alignment - 1) & + ~static_cast(alignment - 1); + return static_cast(aligned - address); + } + + static bool IsDrained(const BackingStore* backing) { + return backing->live_chunks == 0; + } + + Chunk* NewChunk() { + void* storage = chunk_arena_.Allocate(sizeof(Chunk), alignof(Chunk)); + return new (storage) Chunk(); + } + + void DeleteChunk(Chunk* chunk) { + chunk->~Chunk(); + chunk_arena_.Deallocate(chunk, sizeof(Chunk), alignof(Chunk)); + } + + void DestroyChunks(BackingStore* backing) { + Chunk* chunk = backing->head; + while (chunk != nullptr) { + Chunk* next = chunk->next; + DeleteChunk(chunk); + chunk = next; + } + backing->head = nullptr; + } + + void InsertFree(Chunk* chunk) { + const std::size_t bin = BinIndex(chunk->size); + if (bin == kNoBin) { + chunk->location = Location::kTree; + free_chunks_.insert(chunk); + return; + } + + chunk->location = Location::kFastBin; + chunk->fast_prev = nullptr; + chunk->fast_next = fast_bins_[bin]; + if (chunk->fast_next != nullptr) { + chunk->fast_next->fast_prev = chunk; + } + fast_bins_[bin] = chunk; + fast_bitmap_[bin / 64] |= 1ull << (bin % 64); + } + + void EraseFree(Chunk* chunk) { + if (chunk->location == Location::kNone) { + return; + } + if (chunk->location == Location::kTree) { + chunk->location = Location::kNone; + free_chunks_.erase(chunk); + return; + } + + const std::size_t bin = BinIndex(chunk->size); + if (chunk->fast_prev != nullptr) { + chunk->fast_prev->fast_next = chunk->fast_next; + } else { + fast_bins_[bin] = chunk->fast_next; + } + if (chunk->fast_next != nullptr) { + chunk->fast_next->fast_prev = chunk->fast_prev; + } + chunk->fast_prev = nullptr; + chunk->fast_next = nullptr; + chunk->location = Location::kNone; + + if (fast_bins_[bin] == nullptr) { + fast_bitmap_[bin / 64] &= ~(1ull << (bin % 64)); + } + } + + void* TryCacheAllocate(std::size_t rounded) { + const std::size_t bin = BinIndex(rounded); + if (bin == kNoBin) { + return nullptr; + } + + ThreadCache* cache = LocalCache(); + CacheGuard guard(cache); + Chunk* chunk = cache->lists[bin]; + if (chunk == nullptr) { + return nullptr; + } + + cache->lists[bin] = chunk->fast_next; + --cache->depths[bin]; + cache->bytes -= chunk->size; + ++cache->alloc_count; + ++cache->cache_hit_count; + + chunk->fast_next = nullptr; + chunk->cached = false; + + chunk->requested = rounded; + return chunk->ptr; + } + + bool TryParkCached(ThreadCache* cache, std::size_t bin, Chunk* chunk) { + CacheGuard guard(cache); + if (cache->depths[bin] >= kCacheDepth || + cache->bytes + chunk->size > kCacheBytes) { + return false; + } + + stats_.bytes_internal_waste -= chunk->size - chunk->requested; + chunk->requested = chunk->size; + chunk->cached = true; + + chunk->fast_next = cache->lists[bin]; + cache->lists[bin] = chunk; + ++cache->depths[bin]; + cache->bytes += chunk->size; + return true; + } + + void ReclaimCacheLocked(ThreadCache* cache) { + CacheGuard guard(cache); + if (cache->bytes == 0) { + return; + } + + for (std::size_t bin = 0; bin < kFastBinCount; ++bin) { + Chunk* chunk = cache->lists[bin]; + cache->lists[bin] = nullptr; + cache->depths[bin] = 0; + while (chunk != nullptr) { + Chunk* next = chunk->fast_next; + chunk->fast_next = nullptr; + chunk->cached = false; + ReleaseChunk(chunk); + chunk = next; + } + } + cache->bytes = 0; + } + + void RetireCacheLocked(ThreadCache* cache) { + ReclaimCacheLocked(cache); + stats_.alloc_count += cache->alloc_count; + stats_.cache_hit_count += cache->cache_hit_count; + cache->alloc_count = 0; + cache->cache_hit_count = 0; + } + + void ReclaimAllCaches() { + for (ThreadCache* cache = registry_->head; cache != nullptr; + cache = cache->next) { + ReclaimCacheLocked(cache); + } + } + + void SumCaches(std::size_t* bytes, std::size_t* allocs, + std::size_t* hits) const { + for (ThreadCache* cache = registry_->head; cache != nullptr; + cache = cache->next) { + CacheGuard guard(cache); + *bytes += cache->bytes; + *allocs += cache->alloc_count; + *hits += cache->cache_hit_count; + } + } + + void ReleaseChunk(Chunk* chunk) { + stats_.bytes_in_use -= chunk->size; + stats_.bytes_internal_waste -= chunk->size - chunk->requested; + + chunk->is_free = true; + chunk->requested = 0; + stats_.bytes_free_in_backings += chunk->size; + + BackingStore* owner = chunk->owner; + + coalesce_dirty_ = true; + InsertFree(chunk); + + if (--owner->live_chunks == 0 && !owner->resident) { + CoalesceBacking(owner); + ++drained_candidates_; + } + } + + Chunk* ScanBins(std::size_t from) const { + if (from >= kFastBinCount) { + return nullptr; + } + std::size_t word = from / 64; + std::uint64_t bits = fast_bitmap_[word] & (~0ull << (from % 64)); + while (bits == 0) { + if (++word >= kFastBinWords) { + return nullptr; + } + bits = fast_bitmap_[word]; + } + return fast_bins_[word * 64 + CountTrailingZeros(bits)]; + } + + Chunk* FindFit(std::size_t needed) { + if (Chunk* chunk = FindFitIndexed(needed); chunk != nullptr) { + return chunk; + } + if (coalesce_dirty_) { + CoalesceAll(); + if (Chunk* chunk = FindFitIndexed(needed); chunk != nullptr) { + return chunk; + } + } + + if (!AnyCached()) { + return nullptr; + } + ReclaimAllCaches(); + if (Chunk* chunk = FindFitIndexed(needed); chunk != nullptr) { + return chunk; + } + + if (!coalesce_dirty_) { + return nullptr; + } + CoalesceAll(); + return FindFitIndexed(needed); + } + + bool AnyCached() const { + for (ThreadCache* cache = registry_->head; cache != nullptr; + cache = cache->next) { + CacheGuard guard(cache); + if (cache->bytes != 0) { + return true; + } + } + return false; + } + + Chunk* FindFitIndexed(std::size_t needed) { + const std::size_t first = FirstEligibleBin(needed); + + if (first != kNoBin && fast_bins_[first] != nullptr) { + return fast_bins_[first]; + } + + Chunk* binned = first == kNoBin ? nullptr : ScanBins(first + 1); + + Chunk probe{}; + probe.size = needed; + probe.ptr = nullptr; + auto it = free_chunks_.lower_bound(&probe); + Chunk* treed = it == free_chunks_.end() ? nullptr : *it; + + if (binned == nullptr) { + return treed; + } + if (treed == nullptr) { + return binned; + } + return treed->size < binned->size ? treed : binned; + } + + std::size_t LargestFreeChunk() const { + std::size_t largest = 0; + for (const auto& backing : backings_) { + std::size_t run = 0; + for (const Chunk* chunk = backing->head; chunk != nullptr; + chunk = chunk->next) { + if (chunk->is_free || chunk->cached) { + run += chunk->size; + if (run > largest) { + largest = run; + } + } else { + run = 0; + } + } + } + return largest; + } + + void CoalesceBacking(BackingStore* backing) { + for (Chunk* chunk = backing->head; chunk != nullptr;) { + if (!chunk->is_free) { + chunk = chunk->next; + continue; + } + + Chunk* next = chunk->next; + if (next == nullptr || !next->is_free) { + chunk = next; + continue; + } + EraseFree(chunk); + while (next != nullptr && next->is_free) { + EraseFree(next); + chunk->size += next->size; + Chunk* after = next->next; + DeleteChunk(next); + next = after; + } + chunk->next = next; + if (next != nullptr) { + next->prev = chunk; + } + InsertFree(chunk); + chunk = next; + } + } + + void CoalesceAll() { + for (const auto& backing : backings_) { + CoalesceBacking(backing.get()); + } + coalesce_dirty_ = false; + } + + Chunk* SplitAt(Chunk* chunk, std::size_t offset) { + Chunk* tail = NewChunk(); + tail->owner = chunk->owner; + tail->ptr = Offset(chunk->ptr, offset); + tail->size = chunk->size - offset; + tail->is_free = chunk->is_free; + tail->prev = chunk; + tail->next = chunk->next; + if (chunk->next != nullptr) { + chunk->next->prev = tail; + } + chunk->next = tail; + chunk->size = offset; + return tail; + } + + void* Serve(Chunk* chunk, std::size_t rounded, std::size_t alignment) { + EraseFree(chunk); + BackingStore* owner = chunk->owner; + if (owner->live_chunks++ == 0) { + DropCandidate(owner); + } + stats_.bytes_free_in_backings -= chunk->size; + + chunk->is_free = false; + + const std::size_t padding = AlignPadding(chunk->ptr, alignment); + if (padding != 0) { + Chunk* body = SplitAt(chunk, padding); + chunk->is_free = true; + stats_.bytes_free_in_backings += chunk->size; + InsertFree(chunk); + chunk = body; + } + + if (chunk->size - rounded >= Config::kMinSplitRemainder) { + Chunk* tail = SplitAt(chunk, rounded); + tail->is_free = true; + stats_.bytes_free_in_backings += tail->size; + InsertFree(tail); + } + + chunk->requested = rounded; + allocated_.Insert(chunk->ptr, chunk); + + stats_.bytes_in_use += chunk->size; + stats_.bytes_internal_waste += chunk->size - chunk->requested; + if (stats_.bytes_in_use > stats_.peak_bytes_in_use) { + stats_.peak_bytes_in_use = stats_.bytes_in_use; + } + ++stats_.alloc_count; + return chunk->ptr; + } + + std::size_t NextCapacity(std::size_t needed) const { + const std::size_t floor = needed + Config::kMinSliceAlignment; + return next_capacity_ > floor ? next_capacity_ : floor; + } + + Chunk* AdoptBacking(void* base, std::size_t capacity) { + auto backing = std::make_unique(); + backing->base = base; + backing->capacity = capacity; + backing->oversize = capacity > Config::kMaxCapacity; + + backing->resident = !backing->oversize && !HasResident(); + + const std::size_t lead = AlignPadding(base, Config::kMinSliceAlignment); + + Chunk* chunk = NewChunk(); + chunk->owner = backing.get(); + chunk->ptr = Offset(base, lead); + chunk->size = capacity - lead; + chunk->is_free = true; + backing->head = chunk; + + stats_.bytes_reserved += capacity; + if (stats_.bytes_reserved > stats_.peak_bytes_reserved) { + stats_.peak_bytes_reserved = stats_.bytes_reserved; + } + stats_.bytes_free_in_backings += chunk->size; + stats_.bytes_unusable += lead; + InsertFree(chunk); + + if (!backing->resident) { + ++drained_candidates_; + } + + if (!backing->oversize) { + next_capacity_ = capacity >= Config::kMaxCapacity / 2 + ? Config::kMaxCapacity + : capacity * 2; + } + + backings_.push_back(std::move(backing)); + return chunk; + } + + void DropCandidate(const BackingStore* backing) { + if (!backing->resident) { + --drained_candidates_; + } + } + + bool HasResident() const { + for (const auto& backing : backings_) { + if (backing->resident) { + return true; + } + } + return false; + } + + std::unique_ptr Detach(std::size_t index) { + std::unique_ptr backing = std::move(backings_[index]); + backings_.erase(backings_.begin() + static_cast(index)); + + CoalesceBacking(backing.get()); + DropCandidate(backing.get()); + + EraseFree(backing->head); + stats_.bytes_free_in_backings -= backing->head->size; + stats_.bytes_unusable -= backing->capacity - backing->head->size; + stats_.bytes_reserved -= backing->capacity; + ++stats_.upstream_free_count; + DestroyChunks(backing.get()); + return backing; + } + + static void FreeBackings( + const std::vector>& doomed) { + for (const auto& backing : doomed) { + Upstream::Free(backing->base); + } + } + + void UpdateShrinkState(std::size_t size, + std::vector>* doomed) { + if (drained_candidates_ == 0) { + return; + } + + if (size > Config::kSmallThreshold) { + small_alloc_since_last_trim_ = 0; + return; + } + + if (++small_alloc_since_last_trim_ < Config::kShrinkThreshold) { + return; + } + + small_alloc_since_last_trim_ = 0; + + ReclaimAllCaches(); + + for (std::size_t i = backings_.size(); i-- > 0;) { + BackingStore* backing = backings_[i].get(); + if (backing->resident) { + continue; + } + if (!IsDrained(backing)) { + backing->empty_scans = 0; + continue; + } + ++backing->empty_scans; + + if (!backing->oversize && + backing->empty_scans < Config::kEmptyScansToDestroy) { + continue; + } + doomed->push_back(Detach(i)); + ++stats_.shrink_count; + } + } + + Error AllocateFallback(void** base, std::size_t* capacity, + std::size_t needed, Error failure) { + std::vector> doomed; + { + std::lock_guard lock(mutex_); + + ReclaimAllCaches(); + for (std::size_t i = backings_.size(); i-- > 0;) { + if (IsDrained(backings_[i].get())) { + doomed.push_back(Detach(i)); + } + } + } + + if (!doomed.empty()) { + FreeBackings(doomed); + doomed.clear(); + if (Upstream::Malloc(base, *capacity) == Upstream::kSuccess) { + return Upstream::kSuccess; + } + } + + const std::size_t minimum = needed + Config::kMinSliceAlignment; + if (*capacity > minimum && + Upstream::Malloc(base, minimum) == Upstream::kSuccess) { + *capacity = minimum; + return Upstream::kSuccess; + } + return failure; + } + + std::shared_ptr registry_; + + std::mutex& mutex_; + + detail::NodeArena chunk_arena_; + detail::NodeArena free_set_arena_; + + detail::PointerTable allocated_; + FreeSet free_chunks_{BySizeThenAddress{}, + detail::ArenaAllocator{&free_set_arena_}}; + + Chunk* fast_bins_[kFastBinCount] = {}; + std::uint64_t fast_bitmap_[kFastBinWords] = {}; + + std::vector> backings_; + std::size_t next_capacity_ = Config::kInitialCapacity; + std::size_t small_alloc_since_last_trim_ = 0; + + std::size_t drained_candidates_ = 0; + + bool coalesce_dirty_ = false; + Stats stats_; +}; + +} + +#endif diff --git a/include/infini/rt/detail/node_arena.h b/include/infini/rt/detail/node_arena.h new file mode 100644 index 0000000..98a6b61 --- /dev/null +++ b/include/infini/rt/detail/node_arena.h @@ -0,0 +1,128 @@ +#ifndef INFINI_RT_DETAIL_NODE_ARENA_H_ +#define INFINI_RT_DETAIL_NODE_ARENA_H_ + +#include +#include +#include +#include +#include + +namespace infini::rt::detail { + +class NodeArena { + public: + NodeArena() = default; + + NodeArena(const NodeArena&) = delete; + NodeArena& operator=(const NodeArena&) = delete; + + ~NodeArena() { + for (void* block : blocks_) { + ::operator delete(block, std::align_val_t{node_align_}); + } + } + + void* Allocate(std::size_t bytes, std::size_t alignment) { + if (node_size_ == 0) { + node_size_ = std::max(bytes, sizeof(void*)); + node_align_ = std::max(alignment, alignof(void*)); + } + + if (bytes > node_size_ || alignment > node_align_) { + return ::operator new(bytes, std::align_val_t{alignment}); + } + + if (free_ == nullptr) { + Grow(); + } + void* node = free_; + + std::memcpy(&free_, node, sizeof(void*)); + return node; + } + + void Deallocate(void* node, std::size_t bytes, std::size_t alignment) { + if (node == nullptr) { + return; + } + if (bytes > node_size_ || alignment > node_align_) { + ::operator delete(node, std::align_val_t{alignment}); + return; + } + std::memcpy(node, &free_, sizeof(void*)); + free_ = node; + } + + private: + static constexpr std::size_t kInitialNodes = 32; + static constexpr std::size_t kMaxNodesPerBlock = 4096; + + void Grow() { + const std::size_t count = next_count_; + next_count_ = std::min(next_count_ * 2, kMaxNodesPerBlock); + + const std::size_t stride = + (node_size_ + node_align_ - 1) / node_align_ * node_align_; + void* block = + ::operator new(stride * count, std::align_val_t{node_align_}); + blocks_.push_back(block); + + char* cursor = static_cast(block); + for (std::size_t i = 0; i < count; ++i) { + Deallocate(cursor + i * stride, node_size_, node_align_); + } + } + + std::vector blocks_; + void* free_ = nullptr; + std::size_t node_size_ = 0; + std::size_t node_align_ = alignof(std::max_align_t); + std::size_t next_count_ = kInitialNodes; +}; + +template +class ArenaAllocator { + public: + using value_type = T; + + explicit ArenaAllocator(NodeArena* arena) : arena_(arena) {} + + template + ArenaAllocator(const ArenaAllocator& other) + : arena_(other.arena()) {} + + T* allocate(std::size_t count) { + if (count != 1) { + return static_cast( + ::operator new(count * sizeof(T), std::align_val_t{alignof(T)})); + } + return static_cast(arena_->Allocate(sizeof(T), alignof(T))); + } + + void deallocate(T* ptr, std::size_t count) { + if (count != 1) { + ::operator delete(ptr, std::align_val_t{alignof(T)}); + return; + } + arena_->Deallocate(ptr, sizeof(T), alignof(T)); + } + + NodeArena* arena() const { return arena_; } + + template + bool operator==(const ArenaAllocator& other) const { + return arena_ == other.arena(); + } + + template + bool operator!=(const ArenaAllocator& other) const { + return arena_ != other.arena(); + } + + private: + NodeArena* arena_; +}; + +} + +#endif diff --git a/include/infini/rt/detail/pointer_table.h b/include/infini/rt/detail/pointer_table.h new file mode 100644 index 0000000..aa10b15 --- /dev/null +++ b/include/infini/rt/detail/pointer_table.h @@ -0,0 +1,149 @@ +#ifndef INFINI_RT_DETAIL_POINTER_TABLE_H_ +#define INFINI_RT_DETAIL_POINTER_TABLE_H_ + +#include +#include +#include +#include + +namespace infini::rt::detail { + +template +class PointerTable { + public: + void Insert(void* key, const Value& value) { + if ((occupied_ + 1) * 2 > slots_.size()) { + Rehash(); + } + + const std::size_t mask = slots_.size() - 1; + std::size_t index = Hash(key) & mask; + std::size_t tombstone = kNoSlot; + + for (;; index = (index + 1) & mask) { + Slot& slot = slots_[index]; + if (slot.state == State::kOccupied) { + if (slot.key == key) { + slot.value = value; + return; + } + continue; + } + if (slot.state == State::kTombstone) { + if (tombstone == kNoSlot) { + tombstone = index; + } + continue; + } + break; + } + + if (tombstone != kNoSlot) { + index = tombstone; + --tombstones_; + } else { + ++occupied_; + } + + slots_[index] = Slot{key, value, State::kOccupied}; + ++live_; + } + + bool Find(void* key, Value* out) const { + if (live_ == 0) { + return false; + } + + const std::size_t mask = slots_.size() - 1; + for (std::size_t index = Hash(key) & mask;; index = (index + 1) & mask) { + const Slot& slot = slots_[index]; + if (slot.state == State::kEmpty) { + return false; + } + if (slot.state == State::kOccupied && slot.key == key) { + *out = slot.value; + return true; + } + } + } + + bool Take(void* key, Value* out) { + if (live_ == 0) { + return false; + } + + const std::size_t mask = slots_.size() - 1; + for (std::size_t index = Hash(key) & mask;; index = (index + 1) & mask) { + Slot& slot = slots_[index]; + if (slot.state == State::kEmpty) { + return false; + } + if (slot.state == State::kOccupied && slot.key == key) { + *out = slot.value; + slot.state = State::kTombstone; + slot.key = nullptr; + ++tombstones_; + --live_; + return true; + } + } + } + + template + void ForEach(Visitor&& visitor) const { + for (const Slot& slot : slots_) { + if (slot.state == State::kOccupied) { + visitor(slot.value); + } + } + } + + std::size_t Size() const { return live_; } + + private: + enum class State : std::uint8_t { kEmpty, kOccupied, kTombstone }; + + struct Slot { + void* key = nullptr; + Value value{}; + State state = State::kEmpty; + }; + + static constexpr std::size_t kInitialSlots = 16; + static constexpr std::size_t kNoSlot = static_cast(-1); + + static std::size_t Hash(void* key) { + auto value = + static_cast(reinterpret_cast(key)); + value *= 0x9e3779b97f4a7c15ULL; + return static_cast(value >> 29); + } + + void Rehash() { + std::size_t capacity = kInitialSlots; + while (capacity <= (live_ + 1) * 2) { + capacity *= 2; + } + + std::vector old_slots(capacity); + old_slots.swap(slots_); + occupied_ = 0; + tombstones_ = 0; + live_ = 0; + + for (const Slot& slot : old_slots) { + if (slot.state == State::kOccupied) { + Insert(slot.key, slot.value); + } + } + } + + std::vector slots_; + std::size_t occupied_ = 0; + std::size_t tombstones_ = 0; + std::size_t live_ = 0; +}; + +} + +#endif diff --git a/scripts/compare_allocators.py b/scripts/compare_allocators.py new file mode 100644 index 0000000..b41c034 --- /dev/null +++ b/scripts/compare_allocators.py @@ -0,0 +1,438 @@ +"""Compare every allocation strategy across the CPU and device backends. + +The question this answers is three questions, and they need different builds to +answer honestly: + + direct vs arena Is the arena worth having at all on this backend? + pool vs arena Which pool design wins, and on which shapes? + arena vs cuda_async Does a hand-written arena beat the vendor's own pool? + +The third only exists on a device, and the first has a completely different +answer on each backend -- a host `malloc` costs tens of nanoseconds, so nothing +the arena saves can pay for its bookkeeping, while a `cudaMalloc` costs hundreds +of microseconds and the same arena wins by orders of magnitude. Reporting one +number for "the allocator" would average those into something true of neither. +So this configures and runs both builds and prints each comparison per backend. + +Usage: + + python scripts/compare_allocators.py # build, run, compare + python scripts/compare_allocators.py --quick # shorter arms + python scripts/compare_allocators.py --no-build # reuse what is built + python scripts/compare_allocators.py --backend cpu # one backend only + +A note on why the builds are serialized rather than parallel: `generated/` is +written into the *source* tree at configure time and its contents depend on which +backends are enabled, so two configured build trees cannot be compiled +concurrently -- the second would compile against the first's headers. Each +backend is therefore configured, built, and only then is anything run. +""" + +import argparse +import json +import pathlib +import platform +import re +import subprocess +import sys + +# The executables that emit per-allocator rows. Only `perf_allocator_matrix` +# does: it is the one binary that runs every arm over one set of workloads, which +# is what makes its rows pivotable into the tables below. `perf_memory_pool` +# covers the general shapes, but only over `direct`/`arena`, and it is driven by +# `scripts/run_performance_tests.py` instead. +_MATRIX_TESTS = ("perf_allocator_matrix",) + +# Each backend's build tree, the CMake options that select it, and whether a +# device is required. Ordered CPU-first so a machine without a GPU still gets a +# useful run before anything fails. +_BACKENDS = ( + { + "name": "cpu", + "build_dir": "build-perf-cpu", + "options": ["-DWITH_CPU=ON", "-DWITH_NVIDIA=OFF"], + }, + { + "name": "nvidia", + "build_dir": "build-perf-cuda", + "options": ["-DWITH_CPU=OFF", "-DWITH_NVIDIA=ON"], + }, +) + +# The two comparisons, as (baseline, candidate) arm pairs. `arena` is the +# candidate in every one -- including against `cuda_async`, where the natural +# phrasing would put the vendor's pool second. Keeping the arena in the candidate +# column means the win column always answers the same question, "should we adopt +# this thing", instead of flipping direction in the middle of the report. +_COMPARISONS = ( + ("direct", "arena", "the backend allocator vs the arena"), + ("cuda_async", "arena", "the vendor's stream-ordered pool vs the arena"), +) + +# Units where a larger number is the better outcome. Everything else here is a +# cost -- a duration, a byte count, a call count -- and lower wins. +_HIGHER_IS_BETTER = frozenset({"GiB/s", "seq_per_s"}) + +# Ratio rows are already a ratio; comparing two of them is meaningless. +# +# `x` covers both the retention-amplification rows and the fragmentation probe +# success rate. The latter is the one row here where a *ratio* is the primary +# result rather than a derived one, and it is deliberately not turned into a win +# column: 8/8 vs 8/8 is the expected outcome for both designs, and a 1.00x win +# column would read as "no difference measured" rather than "both succeeded". +_RATIO_UNITS = frozenset({"x"}) + +# Diagnostics rather than costs. `LedgerAccuracy` asks whether one allocator's +# self-reported retention matches what the driver says it took -- a question about +# that allocator's honesty, answered by the binary's own table. Racing two arms' +# answers would print a win column for a row where neither arm is competing. +_DIAGNOSTIC_BENCHMARKS = frozenset({"allocator_matrix.LedgerAccuracy"}) + + +def _repo_root(): + return pathlib.Path(__file__).resolve().parents[1] + + +def _run(command, **kwargs): + printable = " ".join(str(part) for part in command) + print(f"$ {printable}", flush=True) + return subprocess.run(command, check=True, **kwargs) + + +def _configure_and_build(backend, jobs): + """Configure and compile one backend's tree. + + Configuring rewrites `generated/` in the source tree, so this must complete + before another backend is configured. + """ + build_dir = _repo_root() / backend["build_dir"] + _run( + [ + "cmake", + "-S", + str(_repo_root()), + "-B", + str(build_dir), + "-DCMAKE_BUILD_TYPE=Release", + "-DINFINI_RT_BUILD_PERFORMANCE_TESTING=ON", + *backend["options"], + ], + stdout=subprocess.DEVNULL, + ) + for test in _MATRIX_TESTS: + _run( + [ + "cmake", + "--build", + str(build_dir), + "--target", + test, + "-j", + str(jobs), + ], + stdout=subprocess.DEVNULL, + ) + return build_dir + + +def _find_executable(build_dir, name): + for candidate in ( + build_dir / "tests" / "performance" / name, + build_dir / "tests" / "performance" / "Release" / name, + build_dir / name, + ): + if candidate.exists(): + return candidate + return None + + +def _run_matrix(build_dir, quick): + """Run one backend's benchmarks and return the parsed JSON result lines. + + stderr is forwarded rather than captured: it carries the binary's own + per-backend table and its skip messages, and a run that skipped an arm is + something the reader needs to see next to the numbers. + """ + results = [] + for test in _MATRIX_TESTS: + executable = _find_executable(build_dir, test) + if executable is None: + print(f" {test}: not built, skipping", file=sys.stderr) + continue + + command = [str(executable)] + if quick: + command.append("--quick") + completed = subprocess.run( + command, cwd=executable.parent, text=True, capture_output=True, + check=False, + ) + if completed.stderr: + sys.stderr.write(completed.stderr) + if completed.returncode != 0: + raise RuntimeError( + f"{executable} exited with status {completed.returncode}" + ) + for line in completed.stdout.splitlines(): + stripped = line.strip() + if stripped.startswith("{"): + results.append(json.loads(stripped)) + return results + + +def _git(args): + try: + return subprocess.check_output( + ["git", *args], cwd=_repo_root(), text=True, stderr=subprocess.DEVNULL + ).strip() + except (OSError, subprocess.CalledProcessError): + return "unknown" + + +def _row_key(result): + """Identity of a measurement with the allocator dimension removed. + + Two rows collapse to the same key exactly when they are the same workload at + the same parameters measured on different arms, which is what makes them + comparable. `arena_config` is deliberately part of the key: the reduced host + config and the production device config are not the same measurement, and + merging them would silently compare a 8 MB ramp against a 64 MB one. + """ + params = { + name: value + for name, value in result.get("params", {}).items() + if name != "allocator" + } + # `arena_config` stays in the key but out of the rendered label: it is + # constant within a backend, so printing it on every row is noise, while + # keeping it in the key is what stops a reduced-config host row from being + # merged with a production-config device row. + config = params.pop("arena_config", "") + rendered = ", ".join(f"{name}={value}" for name, value in sorted(params.items())) + return (result["benchmark"], rendered, config) + + +def _pivot(results): + """Group results into {(benchmark, params, config): {arm: result}}.""" + table = {} + for result in results: + arm = result.get("params", {}).get("allocator") + if arm is None: + continue + table.setdefault(_row_key(result), {})[arm] = result + return table + + +def _shorten(benchmark): + return re.sub(r"^allocator_matrix\.", "", benchmark) + + +def _format_value(value, unit): + if unit == "count": + return f"{value:,.0f}" + if unit == "bytes": + return f"{value / (1024 * 1024):,.1f} MiB" + # Sub-microsecond latencies and single-digit-percent probe rates both lose + # their meaning at two decimals, so small magnitudes get more. + if unit in ("us", "ms", "x") and 0.0 < abs(value) < 1.0: + return f"{value:,.4f}" + return f"{value:,.2f}" + + +def _speedup(baseline, candidate, unit): + """How many times better the candidate is. None where that has no meaning. + + A zero on either side is not reported as a ratio: for a call count zero is a + real and important outcome (the arena served a whole workload without going + upstream once), and dividing by it would turn the best result in the table + into a blank. + """ + if unit in _RATIO_UNITS: + return None + good, bad = (candidate, baseline) if unit in _HIGHER_IS_BETTER else ( + baseline, + candidate, + ) + if good <= 0.0 or bad <= 0.0: + return None + return good / bad + + +def _describe_zero(baseline, candidate, baseline_arm, candidate_arm): + if baseline == 0.0 and candidate == 0.0: + return "both zero" + if candidate == 0.0: + return f"{candidate_arm} zero" + if baseline == 0.0: + return f"{baseline_arm} zero" + return "-" + + +def _print_comparison(backend, table, baseline_arm, candidate_arm, caption): + rows = [] + configs = set() + for (benchmark, params, config), arms in table.items(): + if benchmark in _DIAGNOSTIC_BENCHMARKS: + continue + if baseline_arm not in arms or candidate_arm not in arms: + continue + configs.add(config) + rows.append( + (_shorten(benchmark), params, arms[baseline_arm], arms[candidate_arm]) + ) + + if not rows: + print( + f"\n[{backend}] {baseline_arm} vs {candidate_arm}: " + "no comparable rows (an arm did not run on this backend)." + ) + return + + rows.sort(key=lambda row: (row[0], row[1])) + + # Sized to the content rather than to a guess, so a long parameter list + # cannot push the numeric columns out of alignment. + name_width = max(len("workload"), *(len(row[0]) for row in rows)) + 2 + params_width = max(len("params"), *(len(row[1]) for row in rows)) + 2 + + config_note = f" (arena config: {', '.join(sorted(configs))})" if configs else "" + print(f"\n=== [{backend}] {caption}{config_note} ===") + win_label = f"{candidate_arm} win" + win_width = max(len(win_label), 12) + 2 + header = ( + f"{'workload':<{name_width}}{'params':<{params_width}}" + f"{baseline_arm:>16}{candidate_arm:>16}{'unit':>9}" + f"{win_label:>{win_width}}" + ) + print(header) + print("-" * len(header)) + + for workload, params, baseline, candidate in rows: + unit = baseline.get("unit", "") + base_value = baseline["median"] + cand_value = candidate["median"] + ratio = _speedup(base_value, cand_value, unit) + if ratio is None: + verdict = _describe_zero( + base_value, cand_value, baseline_arm, candidate_arm + ) + else: + verdict = f"{ratio:,.2f}x" + print( + f"{workload:<{name_width}}{params:<{params_width}}" + f"{_format_value(base_value, unit):>16}" + f"{_format_value(cand_value, unit):>16}" + f"{unit:>9}{verdict:>{win_width}}" + ) + + print( + f"\n`{candidate_arm} win` > 1 means {candidate_arm} is better on that row. " + "Rows in\nthe `x` unit are already ratios, so no win column is computed " + "for them." + ) + if candidate_arm == "cuda_async" or baseline_arm == "cuda_async": + print( + "cuda_async is stream-ordered: its release does not wait for pending\n" + "device work, so it offers a weaker guarantee than the other arms and\n" + "these ratios are not a drop-in speedup." + ) + + +def main(): + parser = argparse.ArgumentParser( + description="Compare allocation strategies across backends." + ) + parser.add_argument( + "--quick", + action="store_true", + help="Fewer iterations and thread counts. Shapes hold, noise is higher.", + ) + parser.add_argument( + "--no-build", + action="store_true", + help="Run whatever is already built instead of configuring first.", + ) + parser.add_argument( + "--backend", + action="append", + dest="backends", + default=None, + choices=[backend["name"] for backend in _BACKENDS], + help="Limit to one backend. Can be passed more than once.", + ) + parser.add_argument("--jobs", type=int, default=16) + parser.add_argument( + "--output", + type=pathlib.Path, + default=None, + help="Write the merged raw results here as JSON.", + ) + args = parser.parse_args() + + selected = [ + backend + for backend in _BACKENDS + if args.backends is None or backend["name"] in args.backends + ] + + metadata = { + "commit": _git(["rev-parse", "HEAD"]), + "system": platform.platform(), + "quick": args.quick, + } + + # Build every selected backend before running any of them. Not an + # optimization -- configuring rewrites the shared `generated/` tree, so a + # build interleaved with another backend's configure would compile against + # the wrong headers. + build_dirs = {} + for backend in selected: + if args.no_build: + build_dirs[backend["name"]] = _repo_root() / backend["build_dir"] + continue + try: + build_dirs[backend["name"]] = _configure_and_build(backend, args.jobs) + except subprocess.CalledProcessError: + print( + f"{backend['name']}: configure or build failed, skipping it.", + file=sys.stderr, + ) + + merged = [] + for backend in selected: + build_dir = build_dirs.get(backend["name"]) + if build_dir is None or not build_dir.exists(): + continue + + print(f"\n### running {backend['name']} ###", flush=True) + try: + results = _run_matrix(build_dir, args.quick) + except (RuntimeError, OSError) as exc: + print(f"{backend['name']}: {exc}", file=sys.stderr) + continue + + for result in results: + result.update(metadata) + merged.extend(results) + + table = _pivot(results) + for baseline_arm, candidate_arm, caption in _COMPARISONS: + _print_comparison( + backend["name"], table, baseline_arm, candidate_arm, caption + ) + + if args.output is not None: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text( + json.dumps(merged, indent=2) + "\n", encoding="utf-8" + ) + print(f"\nwrote {len(merged)} raw results to {args.output}") + + if not merged: + print("\nno results: nothing ran successfully.", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/run_performance_tests.py b/scripts/run_performance_tests.py index d0b3776..2831ff0 100644 --- a/scripts/run_performance_tests.py +++ b/scripts/run_performance_tests.py @@ -145,6 +145,7 @@ def main(): tests = args.tests or [ "perf_runtime_dispatch", "perf_memory", + "perf_memory_pool", "perf_tensor_view", "perf_tensor_view_footprint", ] diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index bb05ed2..4d49393 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -39,11 +39,30 @@ function(add_infini_rt_backend_graph_test backend device_type supports_graph_cap "INFINI_RT_TEST_SUPPORTS_GRAPH_CAPTURE=${supports_graph_capture}") endfunction() +function(add_infini_rt_backend_arena_memory_pool_test backend device_type + runtime_header) + string(TOLOWER "${backend}" backend_lower) + set(target "test_${backend_lower}_arena_memory_pool") + add_infini_rt_test(${target} test_arena_memory_pool_backend.cc) + target_compile_definitions(${target} + PRIVATE + "INFINI_RT_TEST_BACKEND_NAME=\"${backend}\"" + "INFINI_RT_TEST_DEVICE_TYPE=${device_type}" + "INFINI_RT_TEST_RUNTIME_HEADER=\"${runtime_header}\"") +endfunction() + add_infini_rt_test(test_smoke test_smoke.cc) add_infini_rt_test(test_core test_core.cc) add_infini_rt_test(test_small_vector test_small_vector.cc) add_infini_rt_test(test_metadata_view test_metadata_view.cc) add_infini_rt_test(test_shape_strides_storage test_shape_strides_storage.cc) + +# The arena pool's concurrency test drives the allocator from several threads, +# which is the only way to observe that upstream calls happen with the pool's +# lock released without deadlocking or handing the same slice to two callers. +add_infini_rt_test(test_arena_memory_pool test_arena_memory_pool.cc) +find_package(Threads REQUIRED) +target_link_libraries(test_arena_memory_pool PRIVATE Threads::Threads) if(CMAKE_SYSTEM_NAME STREQUAL "Linux") add_infini_rt_test(test_tensor_view_allocations test_tensor_view_allocations.cc) @@ -56,6 +75,8 @@ if(WITH_CPU) add_infini_rt_backend_runtime_test( CPU infini::rt::Device::Type::kCpu infini/rt/cpu/runtime_.h 0 1 0 1 0 1 1 1) + add_infini_rt_backend_arena_memory_pool_test( + CPU infini::rt::Device::Type::kCpu infini/rt/cpu/runtime_.h) endif() if(WITH_NVIDIA) @@ -66,6 +87,8 @@ if(WITH_NVIDIA) 1 1 1 1 1 1 1 1) add_infini_rt_backend_graph_test( NVIDIA infini::rt::Device::Type::kNvidia 1) + add_infini_rt_backend_arena_memory_pool_test( + NVIDIA infini::rt::Device::Type::kNvidia infini/rt/nvidia/runtime_.h) endif() if(WITH_ILUVATAR) diff --git a/tests/performance/CMakeLists.txt b/tests/performance/CMakeLists.txt index 7db86de..2b17ad0 100644 --- a/tests/performance/CMakeLists.txt +++ b/tests/performance/CMakeLists.txt @@ -29,6 +29,37 @@ endfunction() add_infini_rt_performance_test(perf_runtime_dispatch perf_runtime_dispatch.cc) add_infini_rt_performance_test(perf_memory perf_memory.cc) +add_infini_rt_performance_test(perf_memory_pool perf_memory_pool.cc) add_infini_rt_performance_test(perf_tensor_view perf_tensor_view.cc) add_infini_rt_performance_test(perf_tensor_view_footprint perf_tensor_view_footprint.cc) + +find_package(Threads REQUIRED) +target_link_libraries(perf_memory_pool PRIVATE Threads::Threads) + +# Every allocation strategy the current backend offers, measured on one set of +# workloads. Drives the real dispatch API, so the numbers are the backend's own +# -- which is the whole point on a device. +# +# `INFINI_RT_PERF_LARGE_ARENA_CONFIG` switches the arena to the production +# 64 MB -> 512 MB ramp and scales the footprints up to match. Only meaningful +# where memory is a device's: on a host build the same values would reserve +# gigabytes and measure the page allocator instead of the pool. +# +# Registered by hand rather than through the helper above, because the full sweep +# runs 64-thread and multi-stream arms across three allocators and the ctest +# target wants `--quick`: as a test the point is that every arm runs clean, not +# that the numbers are publication grade. Run the binary directly for those. +add_executable(perf_allocator_matrix perf_allocator_matrix.cc) +target_link_libraries(perf_allocator_matrix PRIVATE infinirt Threads::Threads) +target_include_directories(perf_allocator_matrix PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) +target_compile_definitions(perf_allocator_matrix + PRIVATE "INFINI_RT_PERF_BACKEND_NAME=\"${INFINI_RT_PERF_BACKEND_NAME}\"") +if(NOT INFINI_RT_PERF_BACKEND_NAME STREQUAL "cpu") + target_compile_definitions(perf_allocator_matrix + PRIVATE INFINI_RT_PERF_LARGE_ARENA_CONFIG) +endif() +add_test(NAME perf_allocator_matrix COMMAND perf_allocator_matrix --quick) +set_tests_properties(perf_allocator_matrix PROPERTIES + LABELS performance + TIMEOUT 900) diff --git a/tests/performance/perf_allocator_matrix.cc b/tests/performance/perf_allocator_matrix.cc new file mode 100644 index 0000000..c4367ed --- /dev/null +++ b/tests/performance/perf_allocator_matrix.cc @@ -0,0 +1,1471 @@ +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "perf_common.h" + +namespace { + +namespace perf = infini::rt::perf; +namespace runtime = infini::rt::runtime; + +bool Success(runtime::Error status) { return status == runtime::kSuccess; } + +#if defined(INFINI_RT_PERF_LARGE_ARENA_CONFIG) +struct MatrixArenaConfig { + static constexpr std::size_t kInitialCapacity = 64ull << 20; + static constexpr std::size_t kMaxCapacity = 512ull << 20; + static constexpr std::size_t kSmallThreshold = 1ull << 20; + static constexpr std::size_t kMinSliceAlignment = 512; + static constexpr std::size_t kMinSplitRemainder = 512; + static constexpr std::size_t kShrinkThreshold = 16; + static constexpr std::uint32_t kEmptyScansToDestroy = 2; +}; +constexpr const char* kConfigName = "production"; +#else +struct MatrixArenaConfig { + static constexpr std::size_t kInitialCapacity = 8ull << 20; + static constexpr std::size_t kMaxCapacity = 64ull << 20; + static constexpr std::size_t kSmallThreshold = 1ull << 20; + static constexpr std::size_t kMinSliceAlignment = 512; + static constexpr std::size_t kMinSplitRemainder = 512; + static constexpr std::size_t kShrinkThreshold = 16; + static constexpr std::uint32_t kEmptyScansToDestroy = 2; +}; +constexpr const char* kConfigName = "reduced"; +#endif + +#if defined(INFINI_RT_PERF_LARGE_ARENA_CONFIG) +constexpr std::size_t kFootprintDivisor = 1; +constexpr bool kDeviceBackend = true; +#else +constexpr std::size_t kFootprintDivisor = 16; +constexpr bool kDeviceBackend = false; +#endif + +struct DispatchUpstream { + using Error = runtime::Error; + static constexpr Error kSuccess = runtime::kSuccess; + + static Error Malloc(void** ptr, std::size_t size) { + return runtime::Malloc(ptr, size); + } + static Error Free(void* ptr) { return runtime::Free(ptr); } +}; + +class DirectArm { + public: + static constexpr const char* kName = "direct"; + static constexpr bool kStreamOrdered = false; + + static constexpr bool kHasCache = false; + + static bool Available() { return true; } + + runtime::Error Allocate(void** ptr, std::size_t size) { + ++upstream_allocs_; + return runtime::Malloc(ptr, size); + } + + runtime::Error Deallocate(void* ptr) { + if (ptr == nullptr) { + return runtime::kSuccess; + } + ++upstream_frees_; + return runtime::Free(ptr); + } + + void ReleaseCached() {} + void Sync() {} + + std::size_t UpstreamAllocs() const { return upstream_allocs_; } + std::size_t UpstreamFrees() const { return upstream_frees_; } + + bool TracksBytes() const { return false; } + std::size_t BytesReserved() const { return 0; } + + private: + std::size_t upstream_allocs_ = 0; + std::size_t upstream_frees_ = 0; +}; + +class ArenaArm { + public: + static constexpr const char* kName = "arena"; + static constexpr bool kStreamOrdered = false; + static constexpr bool kHasCache = true; + + static bool Available() { return true; } + + runtime::Error Allocate(void** ptr, std::size_t size) { + return pool_.Allocate(ptr, size); + } + runtime::Error Deallocate(void* ptr) { return pool_.Deallocate(ptr); } + void ReleaseCached() { pool_.ReleaseCached(); } + void Sync() {} + + std::size_t UpstreamAllocs() const { + return pool_.GetStats().upstream_alloc_count; + } + std::size_t UpstreamFrees() const { + return pool_.GetStats().upstream_free_count; + } + bool TracksBytes() const { return true; } + std::size_t BytesReserved() const { return pool_.GetStats().bytes_reserved; } + + private: + infini::rt::ArenaMemoryPool pool_; +}; + +class CudaAsyncArm { + public: + static constexpr const char* kName = "cuda_async"; + static constexpr bool kStreamOrdered = true; + + static constexpr bool kHasCache = false; + + static bool Available() { + runtime::Stream stream{}; + if (!Success(runtime::StreamCreate(&stream))) { + return false; + } + void* ptr = nullptr; + const bool ok = Success(runtime::MallocAsync(&ptr, 4096, stream)) && + Success(runtime::StreamSynchronize(stream)); + if (ok) { + runtime::FreeAsync(ptr, stream); + runtime::StreamSynchronize(stream); + } + runtime::StreamDestroy(stream); + return ok; + } + + CudaAsyncArm() { runtime::StreamCreate(&stream_); } + ~CudaAsyncArm() { + if (stream_ != runtime::Stream{}) { + runtime::StreamSynchronize(stream_); + runtime::StreamDestroy(stream_); + } + } + + CudaAsyncArm(const CudaAsyncArm&) = delete; + CudaAsyncArm& operator=(const CudaAsyncArm&) = delete; + + runtime::Error Allocate(void** ptr, std::size_t size) { + ++upstream_allocs_; + return runtime::MallocAsync(ptr, size, stream_); + } + + runtime::Error Deallocate(void* ptr) { + if (ptr == nullptr) { + return runtime::kSuccess; + } + ++upstream_frees_; + return runtime::FreeAsync(ptr, stream_); + } + + void ReleaseCached() {} + void Sync() { runtime::StreamSynchronize(stream_); } + + std::size_t UpstreamAllocs() const { return upstream_allocs_; } + std::size_t UpstreamFrees() const { return upstream_frees_; } + bool TracksBytes() const { return false; } + std::size_t BytesReserved() const { return 0; } + + private: + runtime::Stream stream_{}; + std::size_t upstream_allocs_ = 0; + std::size_t upstream_frees_ = 0; +}; + +struct Cell { + std::string workload; + std::string params; + std::string arm; + std::string unit; + double value = 0.0; + bool present = true; +}; + +std::vector g_cells; +std::vector g_arm_order; + +void RecordCell(const std::string& workload, const std::string& params, + const std::string& arm, const std::string& unit, double value, + bool present = true) { + g_cells.push_back(Cell{workload, params, arm, unit, value, present}); + if (std::find(g_arm_order.begin(), g_arm_order.end(), arm) == + g_arm_order.end()) { + g_arm_order.push_back(arm); + } +} + +std::vector WithArm(std::vector params, + const char* arm) { + params.push_back(perf::StringParam("allocator", arm)); + params.push_back(perf::StringParam("arena_config", kConfigName)); + return params; +} + +std::string DescribeSize(std::size_t size) { + if (size >= 1024 * 1024) { + return std::to_string(size / (1024 * 1024)) + " MiB"; + } + if (size >= 1024) { + return std::to_string(size / 1024) + " KiB"; + } + return std::to_string(size) + " B"; +} + +template +void BenchCrossThresholdRotation(std::size_t iterations) { + const std::size_t sizes[] = {1024, 2ull << 20, 4096}; + constexpr std::size_t kCount = 3; + + Arm arm; + std::size_t index = 0; + const auto measurement = perf::RunBenchmarkMeasured( + "allocator_matrix.CrossThresholdRotation", + WithArm({perf::NumberParam("size_count", kCount)}, Arm::kName), iterations, + "us", [&arm, &index, &sizes] { + const std::size_t size = sizes[index]; + index = (index + 1) % kCount; + void* ptr = nullptr; + auto status = arm.Allocate(&ptr, size); + perf::DoNotOptimize(status); + if (Success(status)) { + status = arm.Deallocate(ptr); + perf::DoNotOptimize(status); + } + }); + arm.Sync(); + + RecordCell("CrossThreshold", "1K/2M/4K", Arm::kName, "us", + measurement.median); + RecordCell("CrossThreshold/upstream", "1K/2M/4K", Arm::kName, "calls", + static_cast(arm.UpstreamAllocs())); + RecordCell("CrossThreshold/reserved", "1K/2M/4K", Arm::kName, "B", + static_cast(arm.BytesReserved()), arm.TracksBytes()); + + perf::PrintResult("allocator_matrix.CrossThresholdRotationUpstreamCalls", + WithArm({perf::NumberParam("size_count", kCount)}, + Arm::kName), + iterations, "count", + static_cast(arm.UpstreamAllocs()), + static_cast(arm.UpstreamAllocs())); +} + +template +void BenchLargeBlockRecycle(std::size_t iterations) { + const std::size_t sizes[] = {9ull << 20, 11ull << 20, 13ull << 20}; + constexpr std::size_t kCount = 3; + + { + std::vector probe; + bool ok = true; + for (std::size_t i = 0; i < kCount && ok; ++i) { + void* ptr = nullptr; + ok = Success(runtime::Malloc(&ptr, sizes[i])) && ptr != nullptr; + if (ok) { + probe.push_back(ptr); + } + } + for (void* ptr : probe) { + runtime::Free(ptr); + } + if (!ok) { + perf::SkipBenchmark("allocator_matrix.LargeBlockRecycle", + "device cannot hold the working set"); + return; + } + } + + Arm arm; + std::size_t index = 0; + const auto measurement = perf::RunBenchmarkMeasured( + "allocator_matrix.LargeBlockRecycle", + WithArm({perf::NumberParam("size_count", kCount)}, Arm::kName), iterations, + "us", [&arm, &index, &sizes] { + const std::size_t size = sizes[index]; + index = (index + 1) % kCount; + void* ptr = nullptr; + auto status = arm.Allocate(&ptr, size); + perf::DoNotOptimize(status); + if (Success(status)) { + status = arm.Deallocate(ptr); + perf::DoNotOptimize(status); + } + }); + arm.Sync(); + + const std::size_t demand = sizes[kCount - 1]; + RecordCell("LargeRecycle", "9/11/13 MiB", Arm::kName, "us", + measurement.median); + RecordCell("LargeRecycle/reserved", "9/11/13 MiB", Arm::kName, "B", + static_cast(arm.BytesReserved()), arm.TracksBytes()); + RecordCell("LargeRecycle/amplif", "9/11/13 MiB", Arm::kName, "x", + static_cast(arm.BytesReserved()) / + static_cast(demand), + arm.TracksBytes()); + RecordCell("LargeRecycle/upstream", "9/11/13 MiB", Arm::kName, "calls", + static_cast(arm.UpstreamAllocs())); +} + +template +void BenchHighWaterGrowth() { + const std::size_t target = (1ull << 30) / kFootprintDivisor; + constexpr std::size_t kBlock = 1ull << 20; + const std::size_t blocks = target / kBlock; + + Arm arm; + std::vector live; + live.reserve(blocks); + + const auto start = std::chrono::steady_clock::now(); + for (std::size_t i = 0; i < blocks; ++i) { + void* ptr = nullptr; + if (!Success(arm.Allocate(&ptr, kBlock))) { + break; + } + live.push_back(ptr); + } + arm.Sync(); + const auto end = std::chrono::steady_clock::now(); + + const auto elapsed_us = + std::chrono::duration(end - start).count(); + const auto reserved = arm.BytesReserved(); + const auto upstream = arm.UpstreamAllocs(); + const bool tracks = arm.TracksBytes(); + + for (void* ptr : live) { + arm.Deallocate(ptr); + } + arm.Sync(); + + const std::string params = DescribeSize(live.size() * kBlock) + " live"; + RecordCell("HighWater", params, Arm::kName, "us", elapsed_us); + RecordCell("HighWater/upstream", params, Arm::kName, "calls", + static_cast(upstream)); + RecordCell("HighWater/reserved", params, Arm::kName, "B", + static_cast(reserved), tracks); + + perf::PrintResult("allocator_matrix.HighWaterGrowth", + WithArm({perf::NumberParam( + "live_bytes", static_cast( + live.size() * kBlock))}, + Arm::kName), + live.size(), "us", elapsed_us, elapsed_us); + perf::PrintResult("allocator_matrix.HighWaterGrowthUpstreamCalls", + WithArm({perf::NumberParam( + "live_bytes", static_cast( + live.size() * kBlock))}, + Arm::kName), + live.size(), "count", static_cast(upstream), + static_cast(upstream)); +} + +template +void BenchTrimCost(std::size_t cached_blocks) { + constexpr std::size_t kSize = 64 * 1024; + constexpr std::size_t kIterations = 100; + + if (!Arm::kHasCache) { + perf::SkipBenchmark("allocator_matrix.TrimCost", + std::string(Arm::kName) + " has no releasable cache"); + return; + } + + Arm arm; + std::vector blocks(cached_blocks, nullptr); + + auto fill_and_trim = [&arm, &blocks] { + for (void*& block : blocks) { + arm.Allocate(&block, kSize); + } + for (void*& block : blocks) { + arm.Deallocate(block); + block = nullptr; + } + arm.ReleaseCached(); + }; + + const auto measurement = perf::RunBenchmarkMeasured( + "allocator_matrix.TrimCost", + WithArm({perf::NumberParam("cached_blocks", cached_blocks), + perf::NumberParam("size_bytes", kSize)}, + Arm::kName), + kIterations, "us", fill_and_trim); + arm.Sync(); + + const auto before_frees = arm.UpstreamFrees(); + fill_and_trim(); + arm.Sync(); + const auto frees_per_trim = + static_cast(arm.UpstreamFrees() - before_frees); + + const std::string params = std::to_string(cached_blocks) + " cached"; + RecordCell("TrimCost", params, Arm::kName, "us", measurement.median); + RecordCell("TrimCost/frees", params, Arm::kName, "calls", frees_per_trim); +} + +template +perf::Measurement RunThreaded(const std::string& benchmark, + const std::vector& params, + std::size_t threads, std::size_t ops_per_thread, + Op&& op) { + constexpr std::size_t kSamples = 7; + const auto total_ops = threads * ops_per_thread; + std::vector samples; + samples.reserve(kSamples); + + for (std::size_t sample = 0; sample < kSamples + 1; ++sample) { + std::atomic go{false}; + std::vector workers; + workers.reserve(threads); + + for (std::size_t t = 0; t < threads; ++t) { + workers.emplace_back([&go, &op, ops_per_thread, t] { + while (!go.load(std::memory_order_acquire)) { + } + for (std::size_t i = 0; i < ops_per_thread; ++i) { + op(t, i); + } + }); + } + + const auto start = std::chrono::steady_clock::now(); + go.store(true, std::memory_order_release); + for (auto& worker : workers) { + worker.join(); + } + const auto end = std::chrono::steady_clock::now(); + + if (sample == 0) { + continue; + } + samples.push_back( + std::chrono::duration(end - start).count() / + static_cast(total_ops)); + } + + const perf::Measurement measurement{perf::Mean(samples), + perf::Median(samples)}; + perf::PrintResult(benchmark, params, total_ops, "ns", measurement.mean, + measurement.median); + return measurement; +} + +template +void BenchThreadScaling(std::size_t threads, std::size_t ops_per_thread) { + constexpr std::size_t kClasses = 16; + constexpr std::size_t kStride = 512; + + Arm arm; + const auto measurement = RunThreaded( + "allocator_matrix.ThreadScaling", + WithArm({perf::NumberParam("threads", threads), + perf::NumberParam("size_classes", kClasses)}, + Arm::kName), + threads, ops_per_thread, [&arm](std::size_t thread, std::size_t op) { + const std::size_t size = ((thread + op) % kClasses + 1) * kStride; + void* ptr = nullptr; + auto status = arm.Allocate(&ptr, size); + if (Success(status)) { + status = arm.Deallocate(ptr); + } + perf::DoNotOptimize(status); + }); + arm.Sync(); + + RecordCell("ThreadScaling", std::to_string(threads) + "T", Arm::kName, "ns", + measurement.median); +} + +template +void BenchImplicitSyncCost() { + constexpr std::size_t kSize = 64 * 1024; + constexpr std::size_t kCachedBlocks = 64; + + constexpr std::size_t kBusyOps = 200; + const std::size_t busy_bytes = (32ull << 20) / kFootprintDivisor; + + if (!Arm::kHasCache) { + perf::SkipBenchmark("allocator_matrix.ImplicitSyncCost", + std::string(Arm::kName) + " has no releasable cache"); + return; + } + + runtime::Stream stream{}; + if (!Success(runtime::StreamCreate(&stream))) { + perf::SkipBenchmark("allocator_matrix.ImplicitSyncCost", + "stream creation failed"); + return; + } + + void* busy_buffer = nullptr; + if (!Success(runtime::Malloc(&busy_buffer, busy_bytes))) { + runtime::StreamDestroy(stream); + perf::SkipBenchmark("allocator_matrix.ImplicitSyncCost", + "could not reserve the interference buffer"); + return; + } + + Arm arm; + std::vector blocks(kCachedBlocks, nullptr); + + auto fill_cache = [&arm, &blocks] { + for (void*& block : blocks) { + arm.Allocate(&block, kSize); + } + for (void*& block : blocks) { + arm.Deallocate(block); + block = nullptr; + } + }; + + auto time_trim = [&](bool with_interference) { + constexpr std::size_t kSamples = 5; + std::vector samples; + samples.reserve(kSamples); + + for (std::size_t sample = 0; sample < kSamples + 1; ++sample) { + fill_cache(); + if (with_interference) { + for (std::size_t i = 0; i < kBusyOps; ++i) { + runtime::MemsetAsync(busy_buffer, static_cast(i & 0xff), + busy_bytes, stream); + } + } + + const auto start = std::chrono::steady_clock::now(); + arm.ReleaseCached(); + arm.Sync(); + const auto end = std::chrono::steady_clock::now(); + + runtime::StreamSynchronize(stream); + if (sample == 0) { + continue; + } + samples.push_back( + std::chrono::duration(end - start).count()); + } + return perf::Median(samples); + }; + + const double idle = time_trim(false); + const double loaded = time_trim(true); + + runtime::Free(busy_buffer); + runtime::StreamDestroy(stream); + + const auto params = WithArm( + {perf::NumberParam("cached_blocks", kCachedBlocks), + perf::NumberParam("queued_ops", kBusyOps)}, + Arm::kName); + perf::PrintResult("allocator_matrix.TrimIdleStream", params, kCachedBlocks, + "us", idle, idle); + perf::PrintResult("allocator_matrix.TrimBusyStream", params, kCachedBlocks, + "us", loaded, loaded); + + RecordCell("Trim/idle stream", "64 cached", Arm::kName, "us", idle); + RecordCell("Trim/busy stream", "64 cached", Arm::kName, "us", loaded); + + RecordCell("Trim/sync stall", "64 cached", Arm::kName, "us", + std::max(0.0, loaded - idle)); +} + +template +void MeasureLedgerAccuracy() { + constexpr std::size_t kBlock = 1ull << 20; + const std::size_t blocks = (256ull << 20) / kFootprintDivisor / kBlock; + + if (!kDeviceBackend) { + perf::SkipBenchmark("allocator_matrix.LedgerAccuracy", + "MemGetInfo is system-wide on the host backend"); + return; + } + + std::size_t free_before = 0; + std::size_t total = 0; + if (!Success(runtime::MemGetInfo(&free_before, &total)) || free_before == 0) { + perf::SkipBenchmark("allocator_matrix.LedgerAccuracy", + "the backend does not report device memory"); + return; + } + + Arm arm; + if (!arm.TracksBytes()) { + return; + } + + std::vector live; + live.reserve(blocks); + for (std::size_t i = 0; i < blocks; ++i) { + void* ptr = nullptr; + if (!Success(arm.Allocate(&ptr, kBlock))) { + break; + } + + runtime::Memset(ptr, 0, kBlock); + live.push_back(ptr); + } + arm.Sync(); + runtime::DeviceSynchronize(); + + std::size_t free_after = 0; + runtime::MemGetInfo(&free_after, &total); + const double device_consumed = + free_before > free_after ? static_cast(free_before - free_after) + : 0.0; + const double reported = static_cast(arm.BytesReserved()); + + for (void* ptr : live) { + arm.Deallocate(ptr); + } + arm.Sync(); + + const std::string params = DescribeSize(live.size() * kBlock) + " live"; + RecordCell("Ledger/reported", params, Arm::kName, "B", reported); + RecordCell("Ledger/device", params, Arm::kName, "B", device_consumed); + RecordCell("Ledger/ratio", params, Arm::kName, "x", + reported == 0.0 ? 0.0 : device_consumed / reported); + + perf::PrintResult( + "allocator_matrix.LedgerAccuracy", + WithArm({perf::NumberParam( + "live_bytes", + static_cast(live.size() * kBlock))}, + Arm::kName), + live.size(), "bytes", device_consumed, device_consumed); +} + +template +void BenchMultiStream(std::size_t streams, std::size_t ops_per_stream) { + constexpr std::size_t kSize = 256 * 1024; + + std::vector handles(streams, runtime::Stream{}); + for (auto& stream : handles) { + if (!Success(runtime::StreamCreate(&stream))) { + for (auto& created : handles) { + if (created != runtime::Stream{}) { + runtime::StreamDestroy(created); + } + } + perf::SkipBenchmark("allocator_matrix.MultiStream", + "stream creation failed"); + return; + } + } + + Arm arm; + const auto measurement = RunThreaded( + "allocator_matrix.MultiStream", + WithArm({perf::NumberParam("streams", streams), + perf::NumberParam("size_bytes", kSize)}, + Arm::kName), + streams, ops_per_stream, + [&arm, &handles](std::size_t index, std::size_t) { + void* ptr = nullptr; + if (!Success(arm.Allocate(&ptr, kSize))) { + return; + } + + runtime::MemsetAsync(ptr, 0, kSize, handles[index]); + runtime::StreamSynchronize(handles[index]); + const auto status = arm.Deallocate(ptr); + perf::DoNotOptimize(status); + }); + arm.Sync(); + + for (auto& stream : handles) { + runtime::StreamDestroy(stream); + } + + RecordCell("MultiStream", std::to_string(streams) + " streams", Arm::kName, + "ns", measurement.median); +} + +template +void BenchLayerwiseInference(std::size_t decode_steps) { + constexpr std::size_t kLayers = 32; + const std::size_t weight_bytes = (4ull << 20) / kFootprintDivisor; + const std::size_t activation_bytes = (2ull << 20) / kFootprintDivisor; + const std::size_t kv_step_bytes = (128ull << 10) / kFootprintDivisor; + const std::size_t decode_bytes = (64ull << 10) / kFootprintDivisor; + + Arm arm; + + std::vector weights; + weights.reserve(kLayers); + for (std::size_t layer = 0; layer < kLayers; ++layer) { + void* ptr = nullptr; + if (!Success(arm.Allocate(&ptr, weight_bytes))) { + for (void* held : weights) { + arm.Deallocate(held); + } + perf::SkipBenchmark("allocator_matrix.LayerwiseInference", + "device cannot hold the model weights"); + return; + } + weights.push_back(ptr); + } + arm.Sync(); + + std::vector kv_cache; + kv_cache.reserve(kLayers * (decode_steps + 1)); + + const auto before_upstream = arm.UpstreamAllocs(); + const auto start = std::chrono::steady_clock::now(); + + void* previous = nullptr; + for (std::size_t layer = 0; layer < kLayers; ++layer) { + void* activation = nullptr; + if (Success(arm.Allocate(&activation, activation_bytes))) { + if (previous != nullptr) { + arm.Deallocate(previous); + } + previous = activation; + } + void* kv = nullptr; + + if (Success(arm.Allocate(&kv, kv_step_bytes + layer * 512))) { + kv_cache.push_back(kv); + } + } + if (previous != nullptr) { + arm.Deallocate(previous); + previous = nullptr; + } + + for (std::size_t step = 0; step < decode_steps; ++step) { + for (std::size_t layer = 0; layer < kLayers; ++layer) { + void* activation = nullptr; + if (Success(arm.Allocate(&activation, decode_bytes))) { + arm.Deallocate(activation); + } + void* kv = nullptr; + if (Success(arm.Allocate(&kv, kv_step_bytes + step * 256))) { + kv_cache.push_back(kv); + } + } + } + arm.Sync(); + + const auto end = std::chrono::steady_clock::now(); + const auto elapsed_ms = + std::chrono::duration(end - start).count(); + const auto upstream = arm.UpstreamAllocs() - before_upstream; + const auto reserved = arm.BytesReserved(); + const bool tracks = arm.TracksBytes(); + + for (void* ptr : kv_cache) { + arm.Deallocate(ptr); + } + for (void* ptr : weights) { + arm.Deallocate(ptr); + } + arm.Sync(); + + const std::string params = std::to_string(decode_steps) + " steps"; + RecordCell("Inference", params, Arm::kName, "ms", elapsed_ms); + RecordCell("Inference/upstream", params, Arm::kName, "calls", + static_cast(upstream)); + RecordCell("Inference/reserved", params, Arm::kName, "B", + static_cast(reserved), tracks); + + const auto json_params = + WithArm({perf::NumberParam("layers", kLayers), + perf::NumberParam("decode_steps", decode_steps)}, + Arm::kName); + perf::PrintResult("allocator_matrix.LayerwiseInference", json_params, 1, "ms", + elapsed_ms, elapsed_ms); + perf::PrintResult("allocator_matrix.LayerwiseInferenceUpstreamCalls", + json_params, 1, "count", static_cast(upstream), + static_cast(upstream)); +} + +struct ConcurrentInferenceResult { + double p50 = 0.0; + double p99 = 0.0; + double max = 0.0; + double throughput = 0.0; + std::size_t upstream_allocs = 0; + std::size_t bytes_reserved = 0; + bool tracks_bytes = false; +}; + +template +ConcurrentInferenceResult BenchConcurrentInference(std::size_t threads, + std::size_t sequences, + std::size_t decode_steps) { + constexpr std::size_t kLayers = 8; + const std::size_t weight_bytes = (4ull << 20) / kFootprintDivisor; + const std::size_t activation_bytes = (2ull << 20) / kFootprintDivisor; + const std::size_t kv_step_bytes = (128ull << 10) / kFootprintDivisor; + const std::size_t decode_bytes = (64ull << 10) / kFootprintDivisor; + + std::vector handles(threads, runtime::Stream{}); + for (auto& stream : handles) { + if (!Success(runtime::StreamCreate(&stream))) { + for (auto& created : handles) { + if (created != runtime::Stream{}) { + runtime::StreamDestroy(created); + } + } + perf::SkipBenchmark("allocator_matrix.ConcurrentInference", + "stream creation failed"); + return {}; + } + } + + Arm arm; + + std::vector> weights(threads); + bool weights_ok = true; + for (std::size_t t = 0; t < threads && weights_ok; ++t) { + for (std::size_t layer = 0; layer < kLayers; ++layer) { + void* ptr = nullptr; + if (!Success(arm.Allocate(&ptr, weight_bytes))) { + weights_ok = false; + break; + } + weights[t].push_back(ptr); + } + } + if (!weights_ok) { + for (const auto& held : weights) { + for (void* ptr : held) { + arm.Deallocate(ptr); + } + } + for (auto& stream : handles) { + runtime::StreamDestroy(stream); + } + perf::SkipBenchmark("allocator_matrix.ConcurrentInference", + "device cannot hold one model per thread"); + return {}; + } + arm.Sync(); + + std::vector> samples(threads); + for (auto& per_thread : samples) { + per_thread.reserve(sequences); + } + + const auto before_upstream = arm.UpstreamAllocs(); + std::atomic go{false}; + std::vector workers; + workers.reserve(threads); + + for (std::size_t t = 0; t < threads; ++t) { + workers.emplace_back([&, t] { + const runtime::Stream stream = handles[t]; + std::vector kv_cache; + kv_cache.reserve(kLayers * (decode_steps + 1)); + + while (!go.load(std::memory_order_acquire)) { + } + + for (std::size_t sequence = 0; sequence < sequences; ++sequence) { + const auto start = std::chrono::steady_clock::now(); + + void* previous = nullptr; + for (std::size_t layer = 0; layer < kLayers; ++layer) { + void* activation = nullptr; + if (Success(arm.Allocate(&activation, activation_bytes))) { + runtime::MemsetAsync(activation, 0, activation_bytes, stream); + if (previous != nullptr) { + runtime::StreamSynchronize(stream); + arm.Deallocate(previous); + } + previous = activation; + } + void* kv = nullptr; + if (Success(arm.Allocate(&kv, kv_step_bytes + layer * 512))) { + kv_cache.push_back(kv); + } + } + if (previous != nullptr) { + runtime::StreamSynchronize(stream); + arm.Deallocate(previous); + } + + for (std::size_t step = 0; step < decode_steps; ++step) { + for (std::size_t layer = 0; layer < kLayers; ++layer) { + void* activation = nullptr; + if (Success(arm.Allocate(&activation, decode_bytes))) { + runtime::MemsetAsync(activation, 0, decode_bytes, stream); + runtime::StreamSynchronize(stream); + arm.Deallocate(activation); + } + void* kv = nullptr; + if (Success(arm.Allocate(&kv, kv_step_bytes + step * 256))) { + kv_cache.push_back(kv); + } + } + } + + runtime::StreamSynchronize(stream); + for (void* ptr : kv_cache) { + arm.Deallocate(ptr); + } + kv_cache.clear(); + + const auto end = std::chrono::steady_clock::now(); + samples[t].push_back( + std::chrono::duration(end - start).count()); + } + }); + } + + const auto wall_start = std::chrono::steady_clock::now(); + go.store(true, std::memory_order_release); + for (auto& worker : workers) { + worker.join(); + } + const auto wall_end = std::chrono::steady_clock::now(); + arm.Sync(); + + ConcurrentInferenceResult result; + result.upstream_allocs = arm.UpstreamAllocs() - before_upstream; + result.bytes_reserved = arm.BytesReserved(); + result.tracks_bytes = arm.TracksBytes(); + + const double wall_seconds = + std::chrono::duration(wall_end - wall_start).count(); + result.throughput = wall_seconds > 0.0 + ? static_cast(threads * sequences) / + wall_seconds + : 0.0; + + std::vector all; + all.reserve(threads * sequences); + for (const auto& per_thread : samples) { + all.insert(all.end(), per_thread.begin(), per_thread.end()); + } + if (!all.empty()) { + std::sort(all.begin(), all.end()); + result.p50 = all[all.size() / 2]; + const auto p99_index = + std::min(all.size() - 1, static_cast( + static_cast(all.size()) * 0.99)); + result.p99 = all[p99_index]; + result.max = all.back(); + } + + for (const auto& held : weights) { + for (void* ptr : held) { + arm.Deallocate(ptr); + } + } + arm.Sync(); + for (auto& stream : handles) { + runtime::StreamDestroy(stream); + } + + const auto params = + WithArm({perf::NumberParam("threads", threads), + perf::NumberParam("layers", kLayers), + perf::NumberParam("decode_steps", decode_steps)}, + Arm::kName); + const std::size_t total = threads * sequences; + perf::PrintResult("allocator_matrix.ConcurrentInferenceP50", params, total, + "ms", result.p50, result.p50); + perf::PrintResult("allocator_matrix.ConcurrentInferenceP99", params, total, + "ms", result.p99, result.p99); + perf::PrintResult("allocator_matrix.ConcurrentInferenceThroughput", params, + total, "seq_per_s", result.throughput, result.throughput); + perf::PrintResult("allocator_matrix.ConcurrentInferenceUpstreamCalls", params, + total, "count", + static_cast(result.upstream_allocs), + static_cast(result.upstream_allocs)); + + const std::string label = std::to_string(threads) + "T"; + RecordCell("ConcInfer/p50", label, Arm::kName, "ms", result.p50); + RecordCell("ConcInfer/p99", label, Arm::kName, "ms", result.p99); + RecordCell("ConcInfer/max", label, Arm::kName, "ms", result.max); + RecordCell("ConcInfer/upstream", label, Arm::kName, "calls", + static_cast(result.upstream_allocs)); + RecordCell("ConcInfer/reserved", label, Arm::kName, "B", + static_cast(result.bytes_reserved), result.tracks_bytes); + return result; +} + +template +void BenchFragmentation(std::size_t operations) { + constexpr std::size_t kSlots = 256; + + const std::size_t probe_bytes = (32ull << 20) / kFootprintDivisor; + constexpr std::size_t kProbes = 8; + + struct Slot { + void* ptr = nullptr; + std::size_t expires_at = 0; + }; + + Arm arm; + std::vector slots(kSlots); + + std::uint64_t state = 0x2545f4914f6cdd1dull; + auto next = [&state] { + state = state * 6364136223846793005ull + 1442695040888963407ull; + return static_cast(state >> 33); + }; + + auto random_size = [&next] { + const std::uint32_t decade = next() % 4; + const std::size_t base = 1024ull << (4 * decade); + return base + (next() % base); + }; + + std::size_t failures = 0; + for (std::size_t op = 0; op < operations; ++op) { + Slot& slot = slots[next() % kSlots]; + if (slot.ptr != nullptr) { + if (slot.expires_at > op) { + continue; + } + arm.Deallocate(slot.ptr); + slot.ptr = nullptr; + } + void* ptr = nullptr; + if (Success(arm.Allocate(&ptr, random_size()))) { + slot.ptr = ptr; + + slot.expires_at = op + 1 + (next() % 4096); + } else { + ++failures; + } + } + arm.Sync(); + + const auto churn_reserved = arm.BytesReserved(); + const auto churn_upstream = arm.UpstreamAllocs(); + + std::vector probes; + probes.reserve(kProbes); + for (std::size_t i = 0; i < kProbes; ++i) { + void* ptr = nullptr; + if (!Success(arm.Allocate(&ptr, probe_bytes)) || ptr == nullptr) { + break; + } + probes.push_back(ptr); + } + arm.Sync(); + const double probe_rate = + static_cast(probes.size()) / static_cast(kProbes); + const auto probe_upstream = arm.UpstreamAllocs() - churn_upstream; + + for (void* ptr : probes) { + arm.Deallocate(ptr); + } + for (Slot& slot : slots) { + if (slot.ptr != nullptr) { + arm.Deallocate(slot.ptr); + slot.ptr = nullptr; + } + } + arm.Sync(); + + const std::string label = std::to_string(operations / 1000) + "k ops"; + RecordCell("Fragment/reserved", label, Arm::kName, "B", + static_cast(churn_reserved), arm.TracksBytes()); + RecordCell("Fragment/upstream", label, Arm::kName, "calls", + static_cast(churn_upstream)); + RecordCell("Fragment/fail", label, Arm::kName, "calls", + static_cast(failures)); + RecordCell("Fragment/probe ok", label, Arm::kName, "x", probe_rate); + RecordCell("Fragment/probe up", label, Arm::kName, "calls", + static_cast(probe_upstream)); + + const auto params = + WithArm({perf::NumberParam("operations", operations), + perf::NumberParam("probe_bytes", probe_bytes)}, + Arm::kName); + perf::PrintResult("allocator_matrix.FragmentationReserved", params, + operations, "bytes", static_cast(churn_reserved), + static_cast(churn_reserved)); + perf::PrintResult("allocator_matrix.FragmentationProbeSuccess", params, + kProbes, "x", probe_rate, probe_rate); + perf::PrintResult("allocator_matrix.FragmentationProbeUpstreamCalls", params, + kProbes, "count", static_cast(probe_upstream), + static_cast(probe_upstream)); +} + +template +void BenchLatencyTail(std::size_t size, std::size_t operations) { + Arm arm; + std::vector samples; + samples.reserve(operations); + + for (std::size_t i = 0; i < 64; ++i) { + void* ptr = nullptr; + if (Success(arm.Allocate(&ptr, size))) { + arm.Deallocate(ptr); + } + } + arm.Sync(); + + for (std::size_t op = 0; op < operations; ++op) { + void* ptr = nullptr; + const auto start = std::chrono::steady_clock::now(); + const auto status = arm.Allocate(&ptr, size); + const auto end = std::chrono::steady_clock::now(); + if (Success(status)) { + arm.Deallocate(ptr); + } + samples.push_back( + std::chrono::duration(end - start).count()); + } + arm.Sync(); + + if (samples.empty()) { + return; + } + std::sort(samples.begin(), samples.end()); + auto quantile = [&samples](double q) { + const auto index = std::min( + samples.size() - 1, + static_cast(static_cast(samples.size()) * q)); + return samples[index]; + }; + + const std::string label = DescribeSize(size); + RecordCell("Latency/p50", label, Arm::kName, "us", quantile(0.50)); + RecordCell("Latency/p99", label, Arm::kName, "us", quantile(0.99)); + RecordCell("Latency/p999", label, Arm::kName, "us", quantile(0.999)); + RecordCell("Latency/max", label, Arm::kName, "us", samples.back()); + + const auto params = WithArm( + {perf::NumberParam("size_bytes", static_cast(size))}, + Arm::kName); + perf::PrintResult("allocator_matrix.LatencyP50", params, operations, "us", + quantile(0.50), quantile(0.50)); + perf::PrintResult("allocator_matrix.LatencyP99", params, operations, "us", + quantile(0.99), quantile(0.99)); + perf::PrintResult("allocator_matrix.LatencyP999", params, operations, "us", + quantile(0.999), quantile(0.999)); + perf::PrintResult("allocator_matrix.LatencyMax", params, operations, "us", + samples.back(), samples.back()); +} + +template +void BenchBandwidth() { + const std::size_t size = (64ull << 20) / kFootprintDivisor; + constexpr std::size_t kIterations = 32; + + runtime::Stream stream{}; + if (!Success(runtime::StreamCreate(&stream))) { + perf::SkipBenchmark("allocator_matrix.Bandwidth", "stream creation failed"); + return; + } + + Arm arm; + + void* leading = nullptr; + arm.Allocate(&leading, 4096); + + void* buffer = nullptr; + if (!Success(arm.Allocate(&buffer, size)) || buffer == nullptr) { + if (leading != nullptr) { + arm.Deallocate(leading); + } + runtime::StreamDestroy(stream); + perf::SkipBenchmark("allocator_matrix.Bandwidth", + "device cannot hold the bandwidth buffer"); + return; + } + + runtime::MemsetAsync(buffer, 0, size, stream); + runtime::StreamSynchronize(stream); + + std::vector samples; + samples.reserve(kIterations); + for (std::size_t i = 0; i < kIterations; ++i) { + const auto start = std::chrono::steady_clock::now(); + runtime::MemsetAsync(buffer, static_cast(i & 0xff), size, stream); + runtime::StreamSynchronize(stream); + const auto end = std::chrono::steady_clock::now(); + const double seconds = std::chrono::duration(end - start).count(); + if (seconds > 0.0) { + samples.push_back(static_cast(size) / seconds / + (1024.0 * 1024.0 * 1024.0)); + } + } + + arm.Deallocate(buffer); + if (leading != nullptr) { + arm.Deallocate(leading); + } + arm.Sync(); + runtime::StreamDestroy(stream); + + if (samples.empty()) { + return; + } + const double median = perf::Median(samples); + RecordCell("Bandwidth", DescribeSize(size), Arm::kName, "GiB/s", median); + perf::PrintResult( + "allocator_matrix.Bandwidth", + WithArm({perf::NumberParam("size_bytes", + static_cast(size))}, + Arm::kName), + kIterations, "GiB/s", perf::Mean(samples), median); +} + +void PrintMatrix() { + if (g_cells.empty()) { + return; + } + + std::vector> rows; + for (const Cell& cell : g_cells) { + const auto key = std::make_pair(cell.workload, cell.params); + if (std::find(rows.begin(), rows.end(), key) == rows.end()) { + rows.push_back(key); + } + } + + std::cerr << "\n=== " << INFINI_RT_PERF_BACKEND_NAME + << " allocator matrix (median; arena config: " << kConfigName + << ") ===\n"; + std::cerr << std::left << std::setw(24) << "workload" << std::setw(14) + << "params"; + for (const std::string& arm : g_arm_order) { + std::cerr << std::right << std::setw(16) << arm; + } + std::cerr << std::right << std::setw(12) << "unit" << "\n"; + + for (const auto& [workload, params] : rows) { + std::cerr << std::left << std::setw(24) << workload << std::setw(14) + << params; + std::string unit; + for (const std::string& arm : g_arm_order) { + const auto found = std::find_if( + g_cells.begin(), g_cells.end(), [&](const Cell& cell) { + return cell.workload == workload && cell.params == params && + cell.arm == arm; + }); + if (found == g_cells.end()) { + std::cerr << std::right << std::setw(16) << "-"; + } else if (!found->present) { + std::cerr << std::right << std::setw(16) << "n/a"; + unit = found->unit; + } else { + std::cerr << std::right << std::setw(16) << std::fixed + << std::setprecision(found->unit == "calls" ? 0 : 2) + << found->value; + unit = found->unit; + } + } + std::cerr << std::right << std::setw(12) << unit << "\n"; + } + + std::cerr + << "\nLower is better except `x` ratio rows. `calls` rows are exact\n" + "counts, not timings, and are the cause behind the timing above them.\n" + "`n/a` means the metric does not apply to that arm; `-` means the arm\n" + "did not run that workload.\n" + "cuda_async is stream-ordered: its `Deallocate` does not wait for\n" + "pending device work, so it offers a weaker guarantee than the other\n" + "three arms and its timings are not a drop-in speedup.\n"; + std::cerr << std::endl; +} + +bool PrepareRuntime() { + int device_count = 0; + if (!Success(runtime::GetDeviceCount(&device_count)) || device_count <= 0) { + std::cerr << "perf_allocator_matrix skipped: no available device." + << std::endl; + return false; + } + if (!Success(runtime::SetDevice(0))) { + std::cerr << "perf_allocator_matrix skipped: device 0 is not available." + << std::endl; + return false; + } + return true; +} + +template