Skip to content

Commit f6df1da

Browse files
kartbenkartben
committed
doc: instrumentation: add documentation for the instrumentation subsystem
Add documentation for the instrumentation subsystem as this was missing from the initial contribution. Signed-off-by: Benjamin Cabé <benjamin@zephyrproject.org>
1 parent 91b1b84 commit f6df1da

File tree

4 files changed

+181
-0
lines changed

4 files changed

+181
-0
lines changed

‎doc/services/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ OS Services
1717
file_system/index.rst
1818
formatted_output.rst
1919
input/index.rst
20+
instrumentation/index.rst
2021
ipc/index.rst
2122
llext/index.rst
2223
logging/index.rst

‎doc/services/instrumentation/index.rst‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
.. _instrumentation:
2+
3+
Instrumentation
4+
###############
5+
6+
The instrumentation subsystem provides compiler-managed runtime system instrumentation capabilities
7+
for Zephyr applications. It enables developers to trace function calls, observe context switches,
8+
and profile application performance with minimal manual instrumentation effort.
9+
10+
Unlike the :ref:`tracing <tracing>` subsystem, which provides RTOS-aware tracing with structured
11+
event APIs, the instrumentation subsystem works at a lower level by leveraging compiler
12+
instrumentation hooks. This approach makes it possible to capture virtually any function entry and
13+
exit events without requiring manual tracing calls in the code.
14+
15+
.. admonition:: Tracing vs. Instrumentation
16+
:class: hint
17+
18+
**When to use Tracing**: Choose the tracing subsystem when you need RTOS-aware event tracing
19+
(e.g. thread switches, semaphore operations, etc.) and want to minimize overhead.
20+
21+
**When to use Instrumentation**: Choose instrumentation when you need a detailed view of
22+
function-level execution to better understand code flow, or to identify performance bottlenecks
23+
without adding manual trace points.
24+
25+
The instrumentation subsystem relies on compiler support for automatic function instrumentation.
26+
When enabled, the compiler automatically inserts calls to special instrumentation handler functions
27+
at the entry and exit of every function in your application (excluding those explicitly marked with
28+
``__no_instrumentation__``). Currently, only GCC is supported with the ``-finstrument-functions``
29+
compiler flag.
30+
31+
The subsystem initializes automatically after RAM initialization and uses trigger/stopper functions
32+
to control when recording is active. The default trigger and stopper functions are both set to
33+
``main()`` (configurable via Kconfig), meaning instrumentation captures the entire execution from
34+
when ``main()`` starts until it returns.
35+
36+
The recorded data is stored in RAM and can be accessed from a host computer thanks to a UART backend
37+
that exposes a set of simple commands. :zephyr_file:`scripts/instrumentation/zaru.py` script allows
38+
to execute these commands through a high-level command-line interface and makes it easy to obtain
39+
data in a format suitable for further analysis (e.g. using `Perfetto`_).
40+
41+
Operational Modes
42+
*****************
43+
44+
The instrumentation subsystem supports two modes that can be enabled independently or together:
45+
46+
Callgraph Mode (Tracing)
47+
=========================
48+
49+
In callgraph mode (enabled with :kconfig:option:`CONFIG_INSTRUMENTATION_MODE_CALLGRAPH`), the
50+
subsystem records function entry and exit events along with timestamps and context information in a
51+
memory buffer. This enables:
52+
53+
- Reconstruction of the complete function call graph
54+
- Observation of thread context switches
55+
- Analysis of execution flow and timing relationships
56+
57+
The trace buffer can operate in ring buffer mode (default, overwrites old entries) or fixed buffer
58+
mode (stops when full). Buffer size is configurable via
59+
:kconfig:option:`CONFIG_INSTRUMENTATION_MODE_CALLGRAPH_TRACE_BUFFER_SIZE`.
60+
61+
Statistical Mode (Profiling)
62+
=============================
63+
64+
In statistical mode (enabled with :kconfig:option:`CONFIG_INSTRUMENTATION_MODE_STATISTICAL`), the
65+
subsystem accumulates timing statistics for each unique function executed between the trigger and
66+
stopper points. This provides total execution time per function and helps identify performance
67+
bottlenecks. The subsystem tracks up to
68+
:kconfig:option:`CONFIG_INSTRUMENTATION_MODE_STATISTICAL_MAX_NUM_FUNC` unique functions.
69+
70+
Configuration
71+
*************
72+
73+
Enable instrumentation with:
74+
75+
.. code-block:: kconfig
76+
77+
CONFIG_INSTRUMENTATION=y
78+
CONFIG_INSTRUMENTATION_MODE_CALLGRAPH=y # For tracing
79+
CONFIG_INSTRUMENTATION_MODE_STATISTICAL=y # For profiling
80+
81+
The instrumentation subsystem uses :ref:`retained memory <retention_api>` to persist trigger/stopper
82+
function addresses across reboots. This must be configured in the devicetree:
83+
84+
.. code-block:: devicetree
85+
86+
/ {
87+
sram@2003FC00 {
88+
compatible = "zephyr,memory-region", "mmio-sram";
89+
reg = <0x2003FC00 DT_SIZE_K(1)>;
90+
zephyr,memory-region = "RetainedMem";
91+
92+
retainedmem {
93+
compatible = "zephyr,retained-ram";
94+
status = "okay";
95+
96+
instrumentation_triggers: retention@0 {
97+
compatible = "zephyr,retention";
98+
status = "okay";
99+
reg = <0x0 0x10>;
100+
};
101+
};
102+
};
103+
};
104+
105+
/* Adjust main SRAM to exclude retained region */
106+
&sram0 {
107+
reg = <0x20000000 DT_SIZE_K(255)>;
108+
};
109+
110+
See the :zephyr:code-sample:`instrumentation` sample for complete configuration examples.
111+
Additional options include buffer sizes, trigger functions, and function/file exclusion lists (see
112+
Kconfig options starting with :kconfig:option-regex:`CONFIG_INSTRUMENTATION_*`).
113+
114+
``zaru.py`` Usage
115+
*****************
116+
117+
The ``zaru.py`` command-line tool (located in :zephyr_file:`scripts/instrumentation/zaru.py`)
118+
provides an interface for controlling instrumentation and extracting data from the target over UART.
119+
120+
The tool offers several commands:
121+
122+
- ``status``: Check if the target device supports callgraph (tracing) and statistical (profiling)
123+
modes.
124+
- ``trace``: Capture and display function call traces.
125+
- ``profile``: Capture and display function profiling data.
126+
- ``reboot``: Reboot the target device.
127+
128+
You can get help for each command by running ``zaru.py <command> --help``.
129+
130+
By default, ``zaru.py`` attempts to connect to the target device using ``/dev/ttyACM0``. You can
131+
specify a different serial port using the ``--serial`` option:
132+
133+
.. code-block:: console
134+
135+
$ ./scripts/instrumentation/zaru.py --serial /dev/ttyACM1 status
136+
137+
The ``--build-dir`` option can be used to specify the Zephyr build directory, which is needed to
138+
locate the ELF file for symbol resolution. If not provided, ``zaru.py`` will attempt to find it
139+
automatically.
140+
141+
See the :zephyr:code-sample:`instrumentation` sample documentation for detailed usage instructions.
142+
143+
Limitations and Considerations
144+
*******************************
145+
146+
- **Compiler support**: Currently requires GCC with ``-finstrument-functions`` support. Other
147+
compilers are not supported.
148+
149+
- **Stack size requirements**: Instrumentation adds overhead to every function call, which increases
150+
stack usage. You will likely need to increase thread stack sizes to accommodate the additional
151+
space required by instrumentation handlers and nested function calls.
152+
153+
- **Execution overhead**: All function calls incur instrumentation overhead. Code size will increase
154+
due to added instrumentation calls, and performance will be impacted.
155+
156+
- **Initialization constraints**: Code that runs before RAM initialization (e.g., early boot
157+
functions) is not captured.
158+
159+
To reduce overhead, use trigger/stopper functions to instrument only code regions of interest, and
160+
exclude performance-critical functions via
161+
:kconfig:option:`CONFIG_INSTRUMENTATION_EXCLUDE_FUNCTION_LIST` and
162+
:kconfig:option:`CONFIG_INSTRUMENTATION_EXCLUDE_FILE_LIST`.
163+
164+
API Reference
165+
*************
166+
167+
.. doxygengroup:: instrumentation_api
168+
169+
.. _Perfetto: https://perfetto.dev/

‎include/zephyr/instrumentation/instrumentation.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,12 @@
77
#ifndef ZEPHYR_INCLUDE_INSTRUMENTATION_INSTRUMENTATION_H_
88
#define ZEPHYR_INCLUDE_INSTRUMENTATION_INSTRUMENTATION_H_
99

10+
/**
11+
* @defgroup instrumentation_api Instrumentation
12+
* @ingroup os_services
13+
* @{
14+
*/
15+
1016
#include <zephyr/kernel.h>
1117

1218
#ifdef __cplusplus
@@ -212,4 +218,8 @@ void *instr_get_stop_func(void);
212218
}
213219
#endif
214220

221+
/**
222+
* @}
223+
*/
224+
215225
#endif /* ZEPHYR_INCLUDE_INSTRUMENTATION_INSTRUMENTATION_H_ */

‎samples/subsys/instrumentation/README.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
.. zephyr:code-sample:: instrumentation
22
:name: Instrumentation
3+
:relevant-api: instrumentation_api
34

45
Demonstrate the instrumentation subsystem tracing and profiling features.
56

0 commit comments

Comments
 (0)