kerf init: reconcile the live pool instead of writing the baseline once - #17
Merged
Conversation
The kernel now owns pool memory as a list of contiguous chunks, each on one NUMA node, and accepts per-node size requests instead of a single donated range, so MemoryAllocation can no longer be a base and a size. Model it as a list of PoolMemoryRegion (base, size, node) read back from the kernel plus a node-to-bytes request map, keep memory_pool_base, memory_pool_bytes and memory_pool_end as derived properties so the existing consumers keep working, and convert every constructor site in the tree and the tests. Signed-off-by: Cong Wang <cwang@multikernel.io>
The root device tree now reports pool memory as memory@<base> nodes with a reg pair and a numa-node-id, and a baseline request names per-node sizes instead of a single base/bytes pair. Parse both shapes, keep cpus- available as the free subset of the pool, reject the legacy memory-base and memory-bytes properties, and emit the request shape from the extractor so the baseline still round-trips through the parser. Signed-off-by: Cong Wang <cwang@multikernel.io>
…ples A NUMA topology node describes itself with memory-base and memory-size, so scanning the whole /resources body for the legacy pool properties rejected trees that were already migrated. Split the body into its own properties and its direct children, check only the former, read pool chunks only from depth-0 memory@ nodes, migrate the example baselines to memory@N requests, reject a /resources that describes no memory at all, and catch a lone memory-base on the DTB path. Signed-off-by: Cong Wang <cwang@multikernel.io>
The baseline written to the kernel is a request, not a description of memory the host already set aside, so the extractor emits one memory@N node per requested node carrying a size and, when the node is explicit, a numa-node-id, exactly the shape baseline.c parses. This commit adds the test that pins that format; the emitter itself landed with the parser change so the suite never saw a broken round-trip. Signed-off-by: Cong Wang <cwang@multikernel.io>
…seline Re-initializing the pool means moving resources between the host and the pool rather than writing a new baseline, so kerf needs to know what differs. Add compute_pool_diff(): set differences for CPUs and PCI devices, and a per-node memory comparison that grows a node by the missing amount or releases whole chunks when it holds too much, preferring chunks with no instance allocation and never splitting one. The request is the desired state, so memory on nodes it does not mention counts as surplus. Signed-off-by: Cong Wang <cwang@multikernel.io>
The most consequential behaviour of the pool diff, that a request naming only some nodes releases the memory on every other node, had no test, nor did the larger-first order in which idle chunks are returned. Add both cases and state the desired-state rule in the docstring so a future change cannot silently turn it into a merge. Signed-off-by: Cong Wang <cwang@multikernel.io>
The kernel dropped the mk,instance property and now addresses every overlay fragment by the standard target-path: /resources for the pool, /instances/<name> for an instance and /instances for create and remove. Generate pool transactions from a PoolDiff as a /resources fragment whose operations read from the pool's point of view, convert the instance update and create overlays to target-path, rename the memory items to memory@N and the NUMA property to numa-node-id, and carry numa- node-id on an instance's memory-add so a grow of a node-pinned instance stays on its node. Signed-off-by: Cong Wang <cwang@multikernel.io>
The write-and-check-the-transaction sequence lived in three copies, and the iomem helpers still assumed a single pool line although the kernel now registers one line per chunk. Collapse the copies into DeviceTreeManager.apply_dtbo(), report every chunk and note which ones still hold an allocation, and emit fragment unit-addresses in hex because the kernel parses them base 16. Signed-off-by: Cong Wang <cwang@multikernel.io>
The kernel refuses a baseline write once the pool holds resources, so re-running kerf init failed with EBUSY. Read the pool back, diff it against the request and apply the difference as a /resources overlay transaction, leaving the baseline write for an empty pool only. The --memory option now describes the request rather than a runtime allocation, so the lazy_cma client is gone, and --teardown hands every pool resource back to the host. Signed-off-by: Cong Wang <cwang@multikernel.io>
…list Before a pool exists the kernel still publishes a root device tree, and it describes the host: the online CPUs land in cpus with no cpus- available and no memory node, so reading it back raised a ParseError that aborted the very first init, and the CPU list alone made an empty pool look live. Treat an unreadable read-back as no pool yet and take liveness from cpus-available, the pool chunks and the pool devices, which only the pool branch of the kernel's tree emits; a teardown of an already empty pool no longer writes a baseline the kernel would reject, one that would strand instances names them instead of failing the transaction, and a read-back that fails after the transaction landed is a note rather than an error. Signed-off-by: Cong Wang <cwang@multikernel.io>
kerf show printed a single pool base and size, which cannot describe a pool made of several chunks on different nodes. List every chunk with its base, size and node, show pool membership and the free subset of CPUs separately now that the kernel reports both, hide the total and host-reserved lines when the read-back carries no such numbers, and stop the validation reporter from dividing by a zero total. Signed-off-by: Cong Wang <cwang@multikernel.io>
The host drops a CPU from /proc/cpuinfo once the pool owns it, so every re-init of a live pool rejected the very CPUs it was asked to keep. Read the pool before building the requested tree and treat its CPUs as valid APIC ids, and skip the post-teardown read-back that always failed because an emptied pool has no memory@N left to report. Signed-off-by: Cong Wang <cwang@multikernel.io>
The pool is a list of chunks, but validation still compared the read- back against the first chunk in /proc/iomem and bounded instance memory by base plus sum, so a two-chunk pool failed validation and kerf create aborted before reaching the kernel. Compare the whole chunk set, check that an instance region lies inside one chunk, stop picking a base in kerf create since the kernel places instance memory itself (a base given on the command line is still checked), and let an instance grow only into the chunk it already sits in because the memory-add overlay names an existing range. Signed-off-by: Cong Wang <cwang@multikernel.io>
A transaction can hand back every chunk and keep the pool CPUs, so a read-back with cpus-available but no memory@ node describes a real pool, not a broken one; rejecting it made kerf init fall back to writing a baseline, which the kernel refuses while it owns resources. Accept that shape as a pool with no memory while still requiring a request baseline to name its memory, since nothing else says how much pool to build. Signed-off-by: Cong Wang <cwang@multikernel.io>
Only whole idle chunks go back to the host, so a request smaller than every chunk leaves the pool as it is and the diff comes out empty, which looked like success with no explanation. Report the surplus on the paths that move nothing, the empty diff and --dry-run, the way the path that applies a transaction already does, and give sizes coming from --input the same page-alignment check as --memory since the kernel rejects the rest either way. Signed-off-by: Cong Wang <cwang@multikernel.io>
The root read-back publishes pool membership in cpus and the free members in cpus-available, and carries no instances section. Deriving free CPUs by subtracting the tree's instances from membership therefore offered CPUs already lent to a running instance, and kerf create only found out when the kernel refused the transaction; prefer cpus-available whenever the tree carries it. Signed-off-by: Cong Wang <cwang@multikernel.io>
Nothing here changes what the kernel is asked to do: name the pool transaction file's contents for what they are, the target path of the overlay's first fragment rather than an instance; make the device tree lock public so callers stop reaching into a private method with a pylint waiver; drop the DTB writers and the u64 reader that nothing calls, and a hardware check whose message contradicted its condition; spell the any-node NUMA id as pool_diff.ANY_NODE; say "No memory pool configured" after a teardown instead of printing nothing; and name the example's device NUMA property numa-node-id as everywhere else. Signed-off-by: Cong Wang <cwang@multikernel.io>
The diff preferred idle chunks but still fell back to a chunk holding a running instance when no idle chunk was small enough to cover the surplus; the kernel refuses that chunk and fails the whole transaction, so kerf init turned an unshrinkable pool into an error instead of the note it already prints for that case. Only whole idle chunks go back, so never propose a busy one. Signed-off-by: Cong Wang <cwang@multikernel.io>
Switch the per-node syntax of --memory from nodeK:SIZE to SIZE@N, matching the device-tree unit address convention and the kernel's own memory@N read-back instead of inventing a bespoke node prefix; a plain SIZE still means an unpinned request and the two forms cannot be mixed. Signed-off-by: Cong Wang <cwang@multikernel.io>
A --memory size without an @n used to reach the kernel with no numa- node-id, leaving the kernel to choose where the pool memory came from; placement is policy and policy belongs in kerf. Resolve the node here, from the NUMA node of the requested CPUs (the node they share, or the lowest APIC id's node when they straddle nodes), falling back to the node of the chunks the pool already holds when its CPUs are offline and the host no longer places them, and to node 0 when the host reports no topology at all; baselines read from --input get the same treatment, so the kernel is always handed an explicit node. Signed-off-by: Cong Wang <cwang@multikernel.io>
CI runs a pylint release that flags yielding elements one by one where yield from applies and comparing a result against an empty dict instead of testing its truthiness. Neither changes what the tests verify, so adopt the suggested spellings to keep the lint gate green. Signed-off-by: Cong Wang <cwang@multikernel.io>
Closed
Poetry builds sdists through "pip --isolated", which ignores PIP_CONSTRAINT along with every other pip setting, so the previous pin never reached the environment that compiles pylibfdt and its wrapper still came out with the PyInt_AsLong call swig 4.5 no longer maps. Install swig below 4.5 into the project venv, build pylibfdt there without build isolation so it uses that swig, and let poetry install find the package already satisfied. Signed-off-by: Cong Wang <cwang@multikernel.io>
congwang-mk
force-pushed
the
pool-reinit
branch
from
August 22, 2026 22:11
f5da5a4 to
51b4894
Compare
A kerf init request is the desired state of the pool, so an option that names no resource is already the way to say "none of this": --cpus=none, --memory=none (or --memory=0) and --devices=none each ask for none of that resource, and a request that asks for nothing at all returns the whole pool to the host. The --teardown flag only duplicated that one request behind a second spelling, along with a build_teardown_tree() that built the same tree the ordinary path builds once the available lists are empty, so both are gone and a full teardown is now written "kerf init --cpus=none --memory=none". Mixing none with real entries is rejected, --cpus and --memory stay required so a forgotten --memory cannot silently release the pool, and the instance guard, the empty-pool message, the shortfall note and the dry-run plan all keep working off the same "is the request empty" test as before. While here, drop the meaningless "Host Reserved: 0 cpus: []" line kerf show printed for a tree read back from the kernel, which describes the pool rather than the host. Signed-off-by: Cong Wang <cwang@multikernel.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kerf initcan now be run any number of times. The first run writes the baseline; every later run reads the live pool back from the kernel, diffs it against the request, and applies the difference as a/resourcesoverlay transaction. This pairs with the kernel series onmultikernel-latestthat makes the pool a set of kernel-owned NUMA-node chunks, addresses overlay fragments bytarget-path, and lets the pool grow, shrink and be torn down.User-visible changes
kerf init --cpus=... --memory=SIZEis idempotent: re-running with a different request grows or shrinks the pool in place (CPUs, memory, PCI devices).--memorytakesSIZEorSIZE@N,...(8GB@0,8GB@1), mirroring the device-treememory@Nunit address. A plainSIZEis resolved by kerf to the NUMA node of the requested CPUs, so the kernel is always handed an explicit node.kerf init --teardownreturns every pool resource to the host; it refuses while instances exist and names them.--dry-runprints the planned moves and why a pool cannot shrink further (only whole idle chunks go back).kerf showlists every pool chunk with its base, size and node, and separates pool CPUs from the free subset.Device-tree contract (no backward compatibility)
/resources { cpus; memory@N { size; numa-node-id }; devices };memory-base/memory-bytesare rejected.cpus(members),cpus-available(free),memory@<base> { device_type; reg; numa-node-id }per chunk.target-path(/resources,/instances/<name>,/instances);mk,instanceis gone; memory items arememory@N; the NUMA property isnuma-node-id.Internals
PoolMemoryRegion/ per-noderequestedin the model; parser for both baseline shapes;compute_pool_diff();OverlayGenerator.generate_pool_overlay(); oneDeviceTreeManager.apply_dtbo()write path; multi-chunk/proc/iomemhelpers; validation against the chunk set instead of a single range.Testing
python3 -m pytest tests -q), pylint clean on touched files.--teardownrefused while an instance exists, full teardown, re-init from scratch,SIZE@Nand unpinned-node resolution (including the offline-pool-CPU fallback).Follow-ups (not in this PR)
--teardown --dry-rundoes not warn about running instances.-EBUSY.kerf updaterange math after a multi-region grow needs the kernel to publish per-regionmemory@nodes under/instances/<name>.Close #8