Skip to content

[3/4] xtensa/esp32s3: Per-process address environments and POSIX fork() - #19797

Draft
casaroli wants to merge 13 commits into
apache:masterfrom
casaroli:xtensa-split-3-esp32s3
Draft

[3/4] xtensa/esp32s3: Per-process address environments and POSIX fork()#19797
casaroli wants to merge 13 commits into
apache:masterfrom
casaroli:xtensa-split-3-esp32s3

Conversation

@casaroli

@casaroli casaroli commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

depends-on: [/pull/19796]

Summary

Part 3 of 4, split from #19772. Needs #19795 (merged) for up_fork() and #19796 for BUILD_KERNEL on Xtensa.

The ESP32-S3 half: a per-process address environment, and fork() on top of it.

This does not yet contain a user process. It gives each process its own address space, so one cannot see another's memory. It does not program PMS for a kernel build: esp32s3_userspace.c is still built and compiled under CONFIG_BUILD_PROTECTED alone, so a kernel build never splits the worlds. A user process can still reach kernel memory and the registers that control the mapping. That is #19798, part 4 of 4.

The address environment is built on the SoC's cache MMU, not the Xtensa core, which offers only coarse 512 MB regions with no paging. That is why this is chip code: a non-Espressif LX7 would share none of it.

up_addrenv_fork() duplicates an address environment rather than sharing it, which is what separates fork() from vfork().

The page pool is deliberately not mapped in the kernel. It is carved out of the PSRAM the user processes run from, and external memory permissions are indexed by physical address, so a permanent kernel window on the pool would be a window on every process that no permission setting could close. The kernel reaches a page through a scratch mapping made for one operation and invalidated after.

Two review points are addressed. ARCH_HAVE_FORK is selected by the architecture rather than defaulted inside its own definition, and repeats the ARCH_ADDRENV dependency because a select bypasses depends on. And the chip stops duplicating ARCH_PGPOOL_PBASE and ARCH_PGPOOL_SIZE, which were reachable only under ARCH_PGPOOL_MAPPING although only a virtual base needs a mapping.

Impact

The ESP32-S3 gains CONFIG_BUILD_KERNEL and fork(). Flat and protected configurations are unaffected.

A kernel build here is not a security boundary. It is one after #19798.

Testing

ESP32-S3-DevKitC, two modules: WROOM-2 N32R8V (32 MB octal, 8 MB PSRAM) and WROOM-1 N8R2 (8 MB DIO, 2 MB PSRAM).

module configuration mode result
WROOM-2 ostest flat vfork() passes
WROOM-2 knsh protected vfork() passes
WROOM-2 kernel_oct kernel vfork() and fork() pass
N8R2 ostest flat vfork() passes
N8R2 kernel_n8r2 kernel vfork() and fork() pass 
vfork_test: Child 6 ran and exited before the parent resumed
fork_test: Child running independently (child)
fork_test: Parent and child had independent memory
ostest_main: Exiting with status 0

tools/checkpatch.sh -c -u -m -g passes.

@github-actions github-actions Bot added Area: Documentation Improvements or additions to documentation Arch: xtensa Issues related to the Xtensa architecture Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces. Board: xtensa labels Aug 11, 2026
@casaroli
casaroli marked this pull request as draft August 11, 2026 14:23
@casaroli casaroli changed the title xtensa/esp32s3: Per-process address environments and POSIX fork() [3/3] xtensa/esp32s3: Per-process address environments and POSIX fork() Aug 11, 2026
@github-actions

github-actions Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown

MemBrowse Memory Report

No memory changes detected for:

@casaroli
casaroli force-pushed the xtensa-split-3-esp32s3 branch from 43336c4 to c5c1dcc Compare August 11, 2026 15:42
@casaroli

casaroli commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Keeping this as draft until we merge both prerequisites

@casaroli casaroli mentioned this pull request Aug 11, 2026
Draft
@casaroli
casaroli force-pushed the xtensa-split-3-esp32s3 branch from c5c1dcc to 3be3fe2 Compare August 11, 2026 20:06
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown

❌ Cross-repo dependency could not be applied

The Build report says the declared dependency PR(s) could not be applied, so CI did not run against the combined code:

Reason: cherry-pick failed (if your PR has merge commits, rebase instead)

CI run: https://github.com/apache/nuttx/actions/runs/31531214704

@casaroli casaroli mentioned this pull request Aug 11, 2026
Draft
@casaroli

casaroli commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

This work is now in #19798, together with the ESP32-S3 chip code and the user mode isolation that a kernel build needs.

Please review #19798 instead of this one.

This pull request stays open for reference, and it will be closed when #19798 is merged.

casaroli added 10 commits August 28, 2026 19:48
@casaroli
mm/pgalloc: support 32 KB and 64 KB page sizes
6ea4e3d 
Add CONFIG_MM_PGSIZE == 32768 and 65536 to the page-size switch (and the
Kconfig help text). The 64 KB size matches the ESP32-S3 cache-MMU page
granularity, so an address-environment port there can use one mm_pgalloc()
page per cache-MMU page (naturally 64 KB-aligned by the granule allocator)
instead of coalescing several smaller pages. Inert for existing configs:
MM_PGSIZE is only used when CONFIG_MM_PGALLOC is enabled (BUILD_KERNEL).

Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa: Support BUILD_KERNEL.
2f8f2f9 
Add what a kernel build needs on Xtensa:  a crt0 for a user process, the
kernel stack allocation that a system call switches to, the syscall entry and
return path for an unprivileged caller, and the initial register state that
starts a user task at EL0 with its save area on the kernel stack.

On the ESP32-S3 the arch code that runs while the flash mapping is in flux
moves to IRAM, and the kernel heap is placed above the user .bss so that
up_allocate_kheap() and the user address environment do not overlap.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: add recoverable cache-attribute fault dispatcher (Uni…
a79b510 
…t B)

Route the precise cache-attribute permission faults -- Load/Store/InstrFetch
Prohibited (EXCCAUSE 28/29/20) -- from xtensa_user() to a new dispatcher,
esp32s3_pagefault_dispatch().  On a serviced fault the register frame is
returned so the exception vector's RFE re-executes the faulting instruction;
otherwise it declines to the existing panic path.  Gated by
CONFIG_ESP32S3_PAGEFAULT (default n, depends on BUILD_PROTECTED); the build
is unchanged when the option is off.

This is the recoverable-fault primitive the address-environment / demand-paging
work builds on.  Proven on the ESP32-S3-DevKitC WROOM-2:

- A precise LoadProhibited carries a tracking EXCVADDR (the exact faulting
  address), and RFE cleanly re-executes the faulted load on return -- verified
  with CONFIG_ESP32S3_PAGEFAULT_SELFTEST (the identical instruction restarts
  three times, then steps past, and the task resumes with the shell alive).
- ESP32-S3 PMS (World Controller) permission violations are NOT delivered as
  these precise causes; they raise the asynchronous DRAM0/IRAM0 PMS-monitor
  interrupt, so PMS is an isolation (kill) mechanism, not a restartable one.

No regression: esp32s3-devkit:knsh (WROOM-2) boots to nsh and ostest passes
with the option enabled.

Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: keep octal-flash init in IRAM for the protected build
21144c2 
The protected kernel linker (kernel-space.ld) placed the octal (OPI)
flash bring-up helpers -- esp_rom_spiflash / esp_rom_opiflash_*,
spi_flash_oct_flash_init, mmu_hal, mspi_timing_*, bootloader_flash*,
efuse_hal/efuse_utility, esp_mmu_map and esp32s3_spi_timing -- in mapped
flash.  During configure_cpu_caches() / spi_flash_init_chip_state() in
__start these run while the flash mapping is being reconfigured, which
faults (illegal instruction) on octal-flash modules such as the
ESP32-S3-WROOM-2.  Quad-flash parts never exercise the OPI path, so the
problem was latent.

Place those functions in .iram0.text (mirroring the flat sections
script) so they are safe to execute during flash reconfiguration.

Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Implement per-process address environments.
46f934a 
Give the ESP32-S3 the arch_addrenv_t machinery that BUILD_KERNEL needs:  a
per-process page directory built from the 64 KiB MMU pages of the chip, with
allocation, teardown, and the vaddr-to-paddr translation that the kernel uses
to reach a user buffer.

The MMU, PMS and WCL primitives are exposed as an arch API first, because the
address environment code and the protected user split both need them and
neither owns them.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
boards/esp32s3-devkit: add a kernel-build configuration
1cb2926 
kernel_oct, with the user-program layout and the boot ROMFS a kernel build
loads its programs from.  The ROMFS placeholder is rebuilt with the image,
the generated copy is ignored, and the programs are given stack sizes and
room for a fork() child.

Folds in:
  esp32s3-devkit: user-program layout and boot ROMFS for kernel builds
  boards/esp32s3-devkit: add a kernel-build configuration
  boards/esp32s3-devkit: give kernel_oct's programs their stacks back
  esp32s3-devkit: ignore the generated boot ROMFS
  boards/esp32s3-devkit: rebuild the ROMFS placeholder with the image
  boards/esp32s3-devkit: leave kernel_oct room for a fork() child

Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Reach a page pool page through a scratch mapping.
d5a5e6a 
The page pool is carved out of the PSRAM that user processes run from, and
the external memory permissions are indexed by physical address, so a
permanent kernel window onto the pool is a window onto every process, which
no permission setting can close.

Stop mapping the pool.  The kernel reaches a pool page through a small
scratch region instead, mapped for one operation and invalidated afterwards.
esp32s3_pgmap() takes a slot, esp32s3_pgunmap() releases it, and
ARCH_KMAP_VBASE and ARCH_KMAP_NPAGES describe the region.  Two slots are
enough, because the deepest user is up_addrenv_fork(), which holds a source
and a destination page at once.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
arch/xtensa: Provide POSIX fork() on the ESP32-S3.
6dd8adf 
up_addrenv_fork() duplicates an address environment into freshly allocated
pages mapped at the same virtual addresses.  The text, data and heap regions
of the source are walked one page at a time and copied into fresh pages hung
off the child's own directory, using the two kmap slots that
CONFIG_ARCH_KMAP_NPAGES reserves for exactly this.

xtensa_fork.c already took both paths:  a child that keeps the parent's stack
addresses needs no relocation, which is what a duplicated address environment
gives it.  Only the hook and the Kconfig default were missing.

fork() is offered on a kernel build, which is the only mode with per-process
address environments.

Verified on an ESP32-S3-WROOM-2 with esp32s3-devkit:kernel_oct.  ostest
reports "Parent and child had independent memory" and exits with status 0.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
boards/esp32s3-devkit: Add kernel_n8r2, a kernel build for 2 MB PSRAM.
3982852 
kernel_oct targets a WROOM-2 N32R8V:  octal flash, and 8 MB of PSRAM for the
page pool.  The defaults size the pool for that part, with 8 pages of 64 KiB
for each of the text, data and heap regions, so 1.5 MB per process.  fork()
duplicates the address environment, so a parent and a child need 3 MB at once
and a module with 2 MB of PSRAM cannot do it.

kernel_n8r2 sizes the same build for such a module.  Each region is 2 pages,
so a process takes 384 KiB and a fork() peaks at 768 KiB, inside a 1.5 MB pool
placed at 0x80000 to leave the start of the PSRAM alone.

The flash is quad and runs in DIO mode, so this configuration also exercises
the CONFIG_ESP32S3_FLASH_MODE_OCT guard in kernel-space.ld from the quad side,
which kernel_oct cannot.

This is tight by construction.  ostest has 115 KiB of text against a 128 KiB
text region.  A larger program needs a module with more PSRAM, not a larger
pool.

Verified on an ESP32-S3-DevKitC with an N8R2 module, 8 MB flash in DIO mode
and 2 MB of embedded quad PSRAM.  ostest reports "Parent and child had
independent memory" and exits with status 0.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
Documentation/esp32s3-devkit: Describe the kernel build configurations.
8e3af6c 
Describe kernel_oct and kernel_n8r2 next to the other configurations of this
board.

The entry for kernel_oct carries what a user needs and cannot guess:  a KERNEL
build is the only mode with fork() on this chip, the page pool is reached
through a scratch mapping rather than a permanent window, the ROMFS is linked
into the kernel image so a change to an application needs the whole
export-import-mkromfsimg-relink chain, how to confirm that the ROMFS is really
in the image, and that the shell needs the full path of a program.

The entry for kernel_n8r2 states its limit.  Each region of a process is 2
pages, ostest has 115 KiB of text against a 128 KiB text region, and a larger
program needs a module with more PSRAM.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
arch/xtensa: Address the review of the fork series.
d2077e2 
Two points from the review of apache#19772 that belong with the ESP32-S3 work.

ARCH_HAVE_FORK is now selected by the architecture rather than defaulted from
inside its own definition, so the condition sits where a reader of arch/Kconfig
will look for it.  It repeats the ARCH_ADDRENV dependency, because a select
bypasses depends on and without that an architecture could offer fork() with no
address environment to duplicate.

The page pool no longer carries chip-specific copies of settings the common
address environment already defines.  ARCH_PGPOOL_PBASE and ARCH_PGPOOL_SIZE
were only reachable under ARCH_PGPOOL_MAPPING, which does not apply here:  the
pool is deliberately left unmapped, because it is carved out of the PSRAM the
user processes run from and the external memory permissions are indexed by
physical address.  But a physical base and a size describe the pool whether or
not it is mapped -- only a virtual base needs the mapping -- so those two move
out of that block and the chip uses them.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Wire the chip into the kernel build.
bbd433e 
The common Xtensa BUILD_KERNEL support needs the chip to say what it can do
and where its memory goes.

The chip selects the address environment options it now implements, keeps the
kernel and user heaps apart, and the linker scripts separate kernel from user
text and data so the two worlds can be given different permissions.

Split out of the same change as the common code, so that arch/xtensa/src/common
can be reviewed without the chip in the way.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Fix the nxstyle errors CI reports.
977fdc3 
CI feeds nxstyle the diff hunks with three lines of context, so a style
error anywhere near a hunk fails the check job.  Five are this series' own,
in esp32s3_addrenv.c, esp32s3_pgalloc.c and esp32s3_userspace.c: a
declaration with no blank line after it, and a bare scope block indented one
level too deep.  Two in esp32s3_bringup.c are older than the series and are
in the lines the kernel configurations touch.

Whitespace only, no change in behaviour.

Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
casaroli force-pushed the xtensa-split-3-esp32s3 branch from 3be3fe2 to 977fdc3 Compare August 28, 2026 17:53
@casaroli casaroli changed the title [3/3] xtensa/esp32s3: Per-process address environments and POSIX fork() [3/4] xtensa/esp32s3: Per-process address environments and POSIX fork() Aug 28, 2026
@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown

🔗 Cross-repo PR dependencies

The read-only Build run reported the following dependent PR(s) and fetched head SHA(s):

CI run: https://github.com/apache/nuttx/actions/runs/33196800847

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Donny9 Donny9 Awaiting requested review from Donny9 Donny9 is a code owner
@jerpelea jerpelea Awaiting requested review from jerpelea jerpelea is a code owner
@fdcavalcanti fdcavalcanti Awaiting requested review from fdcavalcanti fdcavalcanti is a code owner
@tmedicci tmedicci Awaiting requested review from tmedicci tmedicci is a code owner
@eren-terzioglu eren-terzioglu Awaiting requested review from eren-terzioglu eren-terzioglu is a code owner
@acassis acassis Awaiting requested review from acassis acassis is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

Arch: xtensa Issues related to the Xtensa architecture Area: Documentation Improvements or additions to documentation Board: xtensa Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@casaroli