Skip to content

Commit 67d2296

Browse files
kartbenkartben
committed
doc: instrumentation: add documentation for the instrumentation subsystem
Add documentation for the instrumentation subsystem as this was missing from the initial contribution. Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
1 parent 91b1b84 commit 67d2296

File tree

4 files changed

+183
-0
lines changed

4 files changed

+183
-0
lines changed

‎doc/services/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ OS Services
1717
file_system/index.rst
1818
formatted_output.rst
1919
input/index.rst
20+
instrumentation/index.rst
2021
ipc/index.rst
2122
llext/index.rst
2223
logging/index.rst

‎doc/services/instrumentation/index.rst‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
.. _instrumentation:
2+
3+
Instrumentation
4+
###############
5+
6+
The instrumentation subsystem provides compiler-managed runtime system instrumentation capabilities
7+
for Zephyr applications. It enables developers to trace function calls, observe context switches,
8+
and profile application performance with minimal manual instrumentation effort.
9+
10+
Unlike the :ref:`tracing <tracing>` subsystem, which provides RTOS-aware tracing with structured
11+
event APIs, the instrumentation subsystem works at a lower level by leveraging compiler
12+
instrumentation hooks. This approach makes it possible to capture virtually any function entry and
13+
exit events without requiring manual tracing calls in the code.
14+
15+
.. admonition:: Tracing vs. Instrumentation
16+
:class: hint
17+
18+
**When to use Tracing**: Choose the tracing subsystem when you need RTOS-aware event tracing with
19+
third-party tool integration (SystemView, Tracealyzer) and want to minimize overhead in
20+
production builds.
21+
22+
**When to use Instrumentation**: Choose instrumentation when you need comprehensive
23+
function-level profiling during development, want automatic call graph reconstruction, or need to
24+
identify performance bottlenecks without manually adding trace points.
25+
26+
27+
The instrumentation subsystem relies on compiler support for automatic function instrumentation.
28+
When enabled, the compiler automatically inserts calls to special instrumentation handler functions
29+
at the entry and exit of every function in your application (excluding those explicitly marked with
30+
``__no_instrumentation__``). Currently, only GCC is supported with the ``-finstrument-functions``
31+
compiler flag.
32+
33+
The subsystem initializes automatically after RAM initialization and uses trigger/stopper functions
34+
to control when recording is active. The default trigger and stopper functions are both set to
35+
``main()`` (configurable via Kconfig), meaning instrumentation captures the entire execution from
36+
when ``main()`` starts until it returns.
37+
38+
The recorded data is stored in RAM and can be accessed from a host computer thanks to a UART backend
39+
that exposes a set of simple commands. :zephyr_file:`scripts/instrumentation/zaru.py` script allows
40+
to execute these commands through a high-level command-line interface and makes it easy to obtain
41+
data in a format suitable for further analysis (e.g. using `Perfetto`_).
42+
43+
Operational Modes
44+
*****************
45+
46+
The instrumentation subsystem supports two modes that can be enabled independently or together:
47+
48+
Callgraph Mode (Tracing)
49+
=========================
50+
51+
In callgraph mode (enabled with :kconfig:option:`CONFIG_INSTRUMENTATION_MODE_CALLGRAPH`), the
52+
subsystem records function entry and exit events along with timestamps and context information in a
53+
memory buffer. This enables:
54+
55+
- Reconstruction of the complete function call graph
56+
- Observation of thread context switches
57+
- Analysis of execution flow and timing relationships
58+
59+
The trace buffer can operate in ring buffer mode (default, overwrites old entries) or fixed buffer
60+
mode (stops when full). Buffer size is configurable via
61+
:kconfig:option:`CONFIG_INSTRUMENTATION_MODE_CALLGRAPH_TRACE_BUFFER_SIZE`.
62+
63+
Statistical Mode (Profiling)
64+
=============================
65+
66+
In statistical mode (enabled with :kconfig:option:`CONFIG_INSTRUMENTATION_MODE_STATISTICAL`), the
67+
subsystem accumulates timing statistics for each unique function executed between the trigger and
68+
stopper points. This provides total execution time per function and helps identify performance
69+
bottlenecks. The subsystem tracks up to
70+
:kconfig:option:`CONFIG_INSTRUMENTATION_MODE_STATISTICAL_MAX_NUM_FUNC` unique functions.
71+
72+
Configuration
73+
*************
74+
75+
Enable instrumentation with:
76+
77+
.. code-block:: kconfig
78+
79+
CONFIG_INSTRUMENTATION=y
80+
CONFIG_INSTRUMENTATION_MODE_CALLGRAPH=y # For tracing
81+
CONFIG_INSTRUMENTATION_MODE_STATISTICAL=y # For profiling
82+
83+
The instrumentation subsystem uses :ref:`retained memory <retention_api>` to persist trigger/stopper
84+
function addresses across reboots. This must be configured in the devicetree:
85+
86+
.. code-block:: devicetree
87+
88+
/ {
89+
sram@2003FC00 {
90+
compatible = "zephyr,memory-region", "mmio-sram";
91+
reg = <0x2003FC00 DT_SIZE_K(1)>;
92+
zephyr,memory-region = "RetainedMem";
93+
94+
retainedmem {
95+
compatible = "zephyr,retained-ram";
96+
status = "okay";
97+
98+
instrumentation_triggers: retention@0 {
99+
compatible = "zephyr,retention";
100+
status = "okay";
101+
reg = <0x0 0x10>;
102+
};
103+
};
104+
};
105+
};
106+
107+
/* Adjust main SRAM to exclude retained region */
108+
&sram0 {
109+
reg = <0x20000000 DT_SIZE_K(255)>;
110+
};
111+
112+
See the :zephyr:code-sample:`instrumentation` sample for complete configuration examples.
113+
Additional options include buffer sizes, trigger functions, and function/file exclusion lists (see
114+
Kconfig options starting with :kconfig:option-regex:`CONFIG_INSTRUMENTATION_*`).
115+
116+
``zaru.py`` Usage
117+
*****************
118+
119+
The ``zaru.py`` command-line tool (located in :zephyr_file:`scripts/instrumentation/zaru.py`)
120+
provides an interface for controlling instrumentation and extracting data from the target over UART.
121+
122+
The tool offers several commands:
123+
124+
- ``status``: Check if the target device supports callgraph (tracing) and statistical (profiling)
125+
modes.
126+
- ``trace``: Capture and display function call traces.
127+
- ``profile``: Capture and display function profiling data.
128+
- ``reboot``: Reboot the target device.
129+
130+
You can get help for each command by running ``zaru.py <command> --help``.
131+
132+
By default, ``zaru.py`` attempts to connect to the target device using ``/dev/ttyACM0``. You can
133+
specify a different serial port using the ``--serial`` option:
134+
135+
.. code-block:: console
136+
137+
$ ./scripts/instrumentation/zaru.py --serial /dev/ttyACM1 status
138+
139+
The ``--build-dir`` option can be used to specify the Zephyr build directory, which is needed to
140+
locate the ELF file for symbol resolution. If not provided, ``zaru.py`` will attempt to find it
141+
automatically.
142+
143+
See the :zephyr:code-sample:`instrumentation` sample documentation for detailed usage instructions.
144+
145+
Limitations and Considerations
146+
*******************************
147+
148+
- **Compiler support**: Currently requires GCC with ``-finstrument-functions`` support. Other
149+
compilers are not supported.
150+
151+
- **Stack size requirements**: Instrumentation adds overhead to every function call, which increases
152+
stack usage. You will likely need to increase thread stack sizes to accommodate the additional
153+
space required by instrumentation handlers and nested function calls.
154+
155+
- **Execution overhead**: All function calls incur instrumentation overhead. Code size will increase
156+
due to added instrumentation calls, and performance will be impacted.
157+
158+
- **Initialization constraints**: Code that runs before RAM initialization (e.g., early boot
159+
functions) is not captured.
160+
161+
To reduce overhead, use trigger/stopper functions to instrument only code regions of interest, and
162+
exclude performance-critical functions via
163+
:kconfig:option:`CONFIG_INSTRUMENTATION_EXCLUDE_FUNCTION_LIST` and
164+
:kconfig:option:`CONFIG_INSTRUMENTATION_EXCLUDE_FILE_LIST`.
165+
166+
API Reference
167+
*************
168+
169+
.. doxygengroup:: instrumentation_api
170+
171+
.. _Perfetto: https://perfetto.dev/

‎include/zephyr/instrumentation/instrumentation.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@
77
#ifndef ZEPHYR_INCLUDE_INSTRUMENTATION_INSTRUMENTATION_H_
88
#define ZEPHYR_INCLUDE_INSTRUMENTATION_INSTRUMENTATION_H_
99

10+
/**
11+
* @defgroup instrumentation_api Instrumentation
12+
* @ingroup os_services
13+
* @{
14+
*/
15+
1016
#include <zephyr/kernel.h>
1117

1218
#ifdef __cplusplus
@@ -212,4 +218,8 @@ void *instr_get_stop_func(void);
212218
}
213219
#endif
214220

221+
/**
222+
* @}
223+
*/
224+
215225
#endif /* ZEPHYR_INCLUDE_INSTRUMENTATION_INSTRUMENTATION_H_ */

‎samples/subsys/instrumentation/README.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
.. zephyr:code-sample:: instrumentation
22
:name: Instrumentation
3+
:relevant-api: instrumentation_api
34

45
Demonstrate the instrumentation subsystem tracing and profiling features.
56

0 commit comments

Comments
 (0)