ci(tests): Add support for multiple duts

[lucasssvaz]

Description of Change

This pull request adds robust support for multi-device tests in the CI build and run scripts. The changes introduce new functions for detecting, building, and running multi-device tests, and ensure that these tests are handled separately from regular single-device tests. The updates also improve argument handling, error reporting, and test selection logic for both build and run workflows.

Multi-device test support:

• Added is_multi_device_test functions to both .github/scripts/tests_build.sh and .github/scripts/tests_run.sh to detect tests configured for multiple devices via multi_device in ci.yml. [1] [2]
• Implemented build_multi_device_sketch and build_multi_device_test functions in tests_build.sh to build all required device sketches for multi-device tests, using custom build directories and handling errors per device.
• Added run_multi_device_test function in tests_run.sh to run multi-device tests with proper argument construction, port assignment, and result aggregation, including retries and platform checks.

Build and run workflow changes:

• Updated both scripts to search for and handle multi-device test directories when a sketch name is provided, falling back to regular sketch lookup if not found. [1] [2]
• Modified chunked and regular build/run logic to process all multi-device tests first, then proceed with regular single-device tests, ensuring proper error handling and reporting for each. [1] [2] [3]

General improvements and bug fixes:

• Improved argument parsing and error messages for missing targets and ports, including default port assignment for multi-device scenarios.
• Refactored test invocation to consistently use arrays for pytest command construction, improving quoting and argument handling.
• Updated count_sketches in sketch_utils.sh to skip sketches that are part of multi-device tests, preventing duplicate builds.

Test Scenarios

Tested locally

Related links

Requires espressif/pytest-embedded#393

[github-actions]

	Messages
📖	This PR seems to be quite large (total lines of code: 1582), you might consider splitting it into smaller PRs

👋 Hello lucasssvaz, we appreciate your contribution to this project!

---

📘 Please review the project's Contributions Guide for key guidelines on code, documentation, testing, and more.

---

🖊️ Please also make sure you have read and signed the Contributor License Agreement for this project.

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- Addressing info messages (📖) is strongly recommended; they're less critical but valuable.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests.

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
4. If the change is approved and passes the tests it is merged into the default branch.

Generated by 🚫 dangerJS against 0bea36f

[github-actions]

Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 0bea36f.

♻️ This comment has been updated with latest results.

[github-actions]

Memory usage test (comparing PR against master branch)

The table below shows the summary of memory usage change (decrease - increase) in bytes and percentage for each target.

Memory	FLASH [bytes]		FLASH [%]		RAM [bytes]		RAM [%]
Target	DEC	INC	DEC	INC	DEC	INC	DEC	INC
ESP32C5	0	⚠️ +76	0.00	⚠️ +0.01	0	0	0.00	0.00
ESP32P4	0	⚠️ +76	0.00	⚠️ +0.01	0	0	0.00	0.00
ESP32S3	0	⚠️ +64	0.00	⚠️ +0.01	0	0	0.00	0.00
ESP32S2	0	0	0.00	0.00	0	0	0.00	0.00
ESP32C3	0	⚠️ +76	0.00	⚠️ +0.01	0	0	0.00	0.00
ESP32C6	0	⚠️ +76	0.00	⚠️ +0.01	0	0	0.00	0.00
ESP32H2	0	⚠️ +76	0.00	⚠️ +0.01	0	0	0.00	0.00
ESP32	0	⚠️ +20	0.00	0.00	0	0	0.00	0.00

Click to expand the detailed deltas report [usage change in BYTES]

Target	ESP32C5		ESP32P4		ESP32S3		ESP32S2		ESP32C3		ESP32C6		ESP32H2		ESP32
Example	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM	FLASH	RAM
libraries/BLE/examples/Server	⚠️ +76	0	⚠️ +76	0	⚠️ +64	0	-	-	⚠️ +76	0	⚠️ +76	0	⚠️ +76	0	⚠️ +20	0
libraries/BLE/examples/Server_secure_authorization	⚠️ +76	0	-	-	⚠️ +64	0	-	-	⚠️ +76	0	⚠️ +76	0	⚠️ +76	0	-	-
libraries/BLE/examples/Server_secure_static_passkey	⚠️ +76	0	⚠️ +76	0	⚠️ +64	0	-	-	⚠️ +76	0	⚠️ +76	0	⚠️ +76	0	⚠️ +20	0
libraries/Insights/examples/MinimalDiagnostics	0	0	-	-	0	0	0	0	0	0	0	0	-	-	0	0
libraries/NetworkClientSecure/examples/WiFiClientSecure	0	0	0	0	0	0	0	0	0	0	0	0	-	-	0	0
libraries/ESP32/examples/Camera/CameraWebServer	-	-	-	-	0	0	0	0	-	-	-	-	-	-	0	0
ESP32/examples/Camera/CameraWebServer (2)	-	-	-	-	0	0	0	0	-	-	-	-	-	-	0	0
ESP32/examples/Camera/CameraWebServer (3)	-	-	-	-	0	0	-	-	-	-	-	-	-	-	-	-

[Copilot]

Pull request overview

This pull request adds comprehensive support for multi-device testing in the CI infrastructure, enabling tests that require communication between multiple ESP32 devices (e.g., WiFi AP/client, BLE server/client). The changes introduce detection, building, and running capabilities for multi-device tests while maintaining backward compatibility with single-device tests.

Key Changes

• Added multi-device test infrastructure with functions to detect, build, and run tests requiring multiple devices
• Implemented two example multi-device test suites: WiFi AP/client and BLE server/client communication tests
• Enhanced documentation with comprehensive guide for creating, building, and running multi-device tests

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
tests/validation/wifi_ap/test_wifi_ap.py	Python test orchestrating WiFi AP and client communication
tests/validation/wifi_ap/ap/ap.ino	Arduino sketch for WiFi Access Point device
tests/validation/wifi_ap/client/client.ino	Arduino sketch for WiFi client device
tests/validation/wifi_ap/ci.yml	Configuration defining multi-device setup and platform support
tests/validation/ble/test_ble.py	Python test coordinating BLE server/client pairing and secure communication
tests/validation/ble/server/server.ino	BLE server sketch with secure characteristic support
tests/validation/ble/client/client.ino	BLE client sketch with service discovery and authentication
tests/validation/ble/ci.yml	BLE test configuration with multi-device specification
tests/conftest.py	Added helper functions for IP validation and random string generation
.github/scripts/tests_run.sh	Enhanced with multi-device test detection, port management, and execution logic
.github/scripts/tests_build.sh	Updated with multi-device build functions and custom build directory handling
.github/scripts/sketch_utils.sh	Modified sketch counting to skip multi-device test sketches
docs/en/contributing.rst	Comprehensive documentation for creating and running multi-device tests

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.