Skip to content

feat: [DSM-142] Use CanisterStates in ReplicatedState#10287

Open
alin-at-dfinity wants to merge 3 commits into
alin/DSM-142-canister-states-cold-statsfrom
alin/DSM-142-canister-states-integration
Open

feat: [DSM-142] Use CanisterStates in ReplicatedState#10287
alin-at-dfinity wants to merge 3 commits into
alin/DSM-142-canister-states-cold-statsfrom
alin/DSM-142-canister-states-integration

Conversation

@alin-at-dfinity
Copy link
Copy Markdown
Contributor

Switches ReplicatedState::canister_states from a flat BTreeMap<CanisterId, Arc<CanisterState>> to CanisterStates, exposing the hot/cold partition to the rest of the system and migrating every caller.

ReplicatedState changes:

  • canister_states field is now CanisterStates.
  • Drop canisters_iter_mut(). Round-level callers move to hot_canisters_iter_mut() (skips the long tail of cold canisters); bulk callers move to canisters_for_each_mut / canisters_try_for_each_mut, which iterate every canister and re-establish the partition afterwards.
  • Add hot_canisters_iter() for read-only hot-only iteration.
  • Add repartition_canister_states(), called from StateManager::commit_and_certify after flush_checkpoint_ops_and_page_maps to drive canisters that went quiet during the round back into cold before checkpointing, so that replicas continuing through a checkpoint and replicas (re)starting from it agree on the partition.
  • take_canister_states / put_canister_states now exchange the CanisterStates directly instead of going through a flat BTreeMap round-trip.
  • Aggregator delegations: total_compute_allocation, memory_taken, total_canister_memory_usage, guaranteed_response_message_memory_taken, best_effort_message_memory_taken, callback_count now delegate to CanisterStates and run in O(|hot|).

state_manager:

  • commit_and_certify calls state.repartition_canister_states() after flush_checkpoint_ops_and_page_maps and before tip handover.
  • validate_eq_canister_states calls CanisterStates::validate_strict_split on the reference state to verify that the persisted partition matches what CanisterStates::new would produce on a fresh load.
  • flush_checkpoint_ops_and_page_maps and switch_to_checkpoint switch from canisters_iter_mut to canisters_for_each_mut / canisters_try_for_each_mut.
  • Bench: bench_traversal likewise.

execution_environment:

  • scheduler.rs: scheduler hot-only iteration where appropriate (add_heartbeat_and_global_timer_tasks, purge_expired_ingress_messages, the ongoing_long_install_code check); migrate charge_canisters_for_resource_allocation_and_usage and the log-memory-store migration loop to canisters_for_each_mut.
  • round_schedule.rs: partition_canisters_to_cores now takes / returns a CanisterStates; idle canisters are dropped before the main hot-canister iteration.
  • query_handler.rs, execution_environment.rs: callers updated.
  • canister_manager/tests.rs, scheduler tests (scheduling.rs, metrics.rs, dts.rs, ecdsa.rs, round_schedule/tests.rs, test_utilities.rs, tests.rs) updated.
  • benches/scheduler.rs: updated.

canonical_state:

  • lazy_tree_conversion.rs: new CanisterStatesFork<'_> that presents a CanisterStates as a LazyFork over the merged hot+cold pools in CanisterId order.

messaging:

  • stream_builder/tests.rs, state_machine/tests.rs, tests/common/mod.rs: caller updates.

replicated_state queues and system_state:

  • CanisterQueues / SystemState local_canisters parameter type flips from &BTreeMap<CanisterId, Arc<CanisterState>> to &CanisterStates (no behavioral change; queues only need contains_key).

metrics.rs:

  • check_dts walks hot_canisters_iter() (only hot canisters can have non-empty task queues).
  • check_subnet_memory_usage switches to CanisterStates::memory_taken() for O(|hot|) aggregation.

test_utilities and state_tool:

  • test_utilities/execution_environment, test_utilities/state, and state_tool/src/commands/canister_metrics.rs updated to use the new iteration APIs.

Switches `ReplicatedState::canister_states` from a flat
`BTreeMap<CanisterId, Arc<CanisterState>>` to `CanisterStates`,
exposing the hot/cold partition to the rest of the system and
migrating every caller.

`ReplicatedState` changes:

  * `canister_states` field is now `CanisterStates`.
  * Drop `canisters_iter_mut()`. Round-level callers move to
    `hot_canisters_iter_mut()` (skips the long tail of cold
    canisters); bulk callers move to `canisters_for_each_mut` /
    `canisters_try_for_each_mut`, which iterate every canister and
    re-establish the partition afterwards.
  * Add `hot_canisters_iter()` for read-only hot-only iteration.
  * Add `repartition_canister_states()`, called from
    `StateManager::commit_and_certify` after
    `flush_checkpoint_ops_and_page_maps` to drive canisters that went
    quiet during the round back into `cold` before checkpointing, so
    that replicas continuing through a checkpoint and replicas
    (re)starting from it agree on the partition.
  * `take_canister_states` / `put_canister_states` now exchange the
    `CanisterStates` directly instead of going through a flat
    `BTreeMap` round-trip.
  * Aggregator delegations: `total_compute_allocation`, `memory_taken`,
    `total_canister_memory_usage`,
    `guaranteed_response_message_memory_taken`,
    `best_effort_message_memory_taken`, `callback_count` now delegate
    to `CanisterStates` and run in `O(|hot|)`.

`state_manager`:

  * `commit_and_certify` calls `state.repartition_canister_states()`
    after `flush_checkpoint_ops_and_page_maps` and before tip
    handover.
  * `validate_eq_canister_states` calls
    `CanisterStates::validate_strict_split` on the reference state to
    verify that the persisted partition matches what
    `CanisterStates::new` would produce on a fresh load.
  * `flush_checkpoint_ops_and_page_maps` and
    `switch_to_checkpoint` switch from `canisters_iter_mut` to
    `canisters_for_each_mut` / `canisters_try_for_each_mut`.
  * Bench: `bench_traversal` likewise.

`execution_environment`:

  * `scheduler.rs`: scheduler hot-only iteration where appropriate
    (`add_heartbeat_and_global_timer_tasks`,
    `purge_expired_ingress_messages`, the
    `ongoing_long_install_code` check); migrate
    `charge_canisters_for_resource_allocation_and_usage` and the
    log-memory-store migration loop to `canisters_for_each_mut`.
  * `round_schedule.rs`: `partition_canisters_to_cores` now takes /
    returns a `CanisterStates`; idle canisters are dropped before the
    main hot-canister iteration.
  * `query_handler.rs`, `execution_environment.rs`: callers updated.
  * `canister_manager/tests.rs`, scheduler tests
    (`scheduling.rs`, `metrics.rs`, `dts.rs`, `ecdsa.rs`,
    `round_schedule/tests.rs`, `test_utilities.rs`, `tests.rs`)
    updated.
  * `benches/scheduler.rs`: updated.

`canonical_state`:

  * `lazy_tree_conversion.rs`: new `CanisterStatesFork<'_>` that
    presents a `CanisterStates` as a `LazyFork` over the merged
    hot+cold pools in `CanisterId` order.

`canister_sandbox`:

  * `sandboxed_execution_controller.rs`: switch
    `evict_sandbox_processes` to per-id `state.canister_state(id)` /
    `state.canister_priority(id)` lookups (also enables removing the
    bulk `canister_accumulated_priorities` method). This duplicates
    the standalone "perf: Look up sandbox scheduler priorities per
    canister" PR; whichever lands first, the other becomes a no-op.

`messaging`:

  * `stream_handler/tests.rs`: pre-heat `LOCAL_CANISTER` in the
    `out_of_memory` reject-signal test so that the expected and
    inducted states share the same hot/cold partition.
  * `stream_builder/tests.rs`, `state_machine/tests.rs`,
    `tests/common/mod.rs`: caller updates.

`replicated_state` queues and system_state:

  * `CanisterQueues` / `SystemState` `local_canisters` parameter type
    flips from `&BTreeMap<CanisterId, Arc<CanisterState>>` to
    `&CanisterStates` (no behavioural change; queues only need
    `contains_key`).
  * `replicated_state.rs` deletes the now-unused
    `canister_accumulated_priorities` method.

`metrics.rs`:

  * `check_dts` walks `hot_canisters_iter()` (only hot canisters can
    have non-empty task queues).
  * `check_subnet_memory_usage` switches to
    `CanisterStates::memory_taken()` for `O(|hot|)` aggregation.

`test_utilities` and `state_tool`:

  * `test_utilities/execution_environment`, `test_utilities/state`,
    and `state_tool/src/commands/canister_metrics.rs` updated to use
    the new iteration APIs.

Co-authored-by: Cursor <cursoragent@cursor.com>
@alin-at-dfinity alin-at-dfinity requested a review from a team as a code owner May 22, 2026 10:25
@github-actions github-actions Bot added the feat label May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant