Skip to content

Commit ad4afed

Browse files
bkhouribkhouri
committed
SwiftTesting: Enhance Event Stream JSON ABI
1 parent d2111b0 commit ad4afed

File tree

1 file changed

+222
-0
lines changed

1 file changed

+222
-0
lines changed

‎proposals/testing/NNNN-augment-event-json-abi.md‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
# Enhance Swift Testing JSON ABI with Rich Test Metadata
2+
3+
* Proposal: [ST-NNNN](NNNN-augment-event-json-abi.md)
4+
* Authors: [Sam Khouri](https://github.com/bkhouri),
5+
* Review Manager: TBD
6+
* Status: **Awaiting review**
7+
* Implementation: [swiftlang/swift-testing#1429](https://github.com/swiftlang/swift-testing/pull/1429)
8+
* Review: [pitch](https://forums.swift.org/t/adding-additional-information-to-the-abi-json/83426)
9+
10+
<!-- When filling out this template, you should delete or replace all of the text
11+
except for the section headers and the header fields above. For example, you
12+
should delete everything from this paragraph down to the Introduction section
13+
below.
14+
15+
As a proposal author, you should fill out all of the header fields except
16+
`Review Manager`. The review manager will set that field and change several
17+
others as part of initiating the review. Delete any header fields marked _if
18+
applicable_ that are not applicable to your proposal.
19+
20+
When sharing a link to the proposal while it is still a PR, be sure to share a
21+
live link to the proposal, not an exact commit, so that readers will always see
22+
the latest version when you make changes. On GitHub, you can find this link by
23+
browsing the PR branch: from the PR page, click the "username wants to merge ...
24+
from username:my-branch-name" link and find the proposal file in that branch.
25+
26+
`Status` should reflect the current implementation status while the proposal is
27+
still a PR. The proposal cannot be reviewed until an implementation is available,
28+
but early readers should see the correct status.
29+
30+
`Bug` should be used when this proposal is fixing a bug with significant
31+
discussion in the bug report. It is not necessary to link bugs that do not
32+
contain significant discussion or that merely duplicate discussion linked
33+
somewhere else. Do not link bugs from private bug trackers.
34+
35+
`Implementation` should link to the PR(s) implementing the feature. If the
36+
proposal has not been implemented yet, or if it simply codifies existing
37+
behavior, just say that. If the implementation has already been committed to the
38+
main branch (as an experimental feature or SPI), mention that. If the
39+
implementation is spread across multiple PRs, just link to the most important
40+
ones.
41+
42+
`Previous Proposal` should be used when there is a specific line of succession
43+
between this proposal and another proposal. For example, this proposal might
44+
have been removed from a previous proposal so that it can be reviewed separately,
45+
or this proposal might supersede a previous proposal in some way that was felt
46+
to exceed the scope of a "revision". Include text briefly explaining the
47+
relationship, such as "Supersedes ST-1234" or "Extracted from ST-01234". If
48+
possible, link to a post explaining the relationship, such as a review decision
49+
that asked for part of the proposal to be split off. Otherwise, you can just
50+
link to the previous proposal.
51+
52+
`Previous Revision` should be added after a major substantive revision of a
53+
proposal that has undergone review. It links to the previously reviewed revision.
54+
It is not necessary to add or update this field after minor editorial changes.
55+
-->
56+
## Introduction
57+
58+
This proposal enhances Swift Testing's event JSON ABI by exposing test
59+
metadata that is currently unavailable to external tools. By including test
60+
tags, bug associations, and time limits in the JSON output, this allows third-party
61+
tools to provide richer insights and more sophisticated test management capabilities.
62+
63+
## Motivation
64+
65+
Swift Testing's event JSON stream provides data for external tooling,
66+
enabling developers to build test analysis and reporting tools.
67+
However, the current implementation lacks access to some test metadata that
68+
developers may want to use to organize and manage their test suites.
69+
70+
Currently missing from the JSON output are:
71+
- **Test tags**: Used for categorization
72+
- **Bug associations**: Critical for tracking which tests verify specific bug fixes
73+
- **Time limits**: Essential for performance monitoring and timeout management
74+
75+
This missing metadata limits the capabilities of external tools. For example:
76+
- IDE extensions cannot provide tag-based test filtering
77+
- CI/CD systems cannot generate reports grouped by test categories
78+
- Performance monitoring tools cannot track tests with specific time constraints
79+
- Bug tracking integrations cannot correlate test failures with known issues
80+
81+
By exposing this information, we unlock new possibilities for Swift Testing
82+
tooling ecosystem.
83+
84+
## Proposed solution
85+
86+
We propose enriching the test payload in the event JSON stream by adding three
87+
metadata fields:
88+
89+
- **`tags`**: An array of string tags associated with the test, enabling categorization
90+
and filtering
91+
- **`bugs`**: An array of bug references, providing traceability between tests
92+
and issue tracking
93+
- **`timeLimit`**: The test's time limit in seconds, enabling performance monitoring
94+
and timeout analysis
95+
96+
These additions leverage existing internal data structures, ensuring minimal performance
97+
impact while maximizing the value delivered to external tools.
98+
99+
## Detailed design
100+
101+
This enhancement builds upon the existing test metadata infrastructure already used
102+
internally by Swift Testing. The implementation reuses established data structures,
103+
ensuring consistency and minimizing complexity.
104+
105+
### Implementation Strategy
106+
107+
The new fields are conditionally included in the JSON output based on test
108+
configuration:
109+
- Fields are only included when the test actually uses the corresponding traits
110+
- Empty or unused traits result in omitted fields, keeping the JSON clean and
111+
efficient
112+
- This approach maintains backward compatibility while providing rich information
113+
when available
114+
115+
### JSON Schema Changes
116+
117+
The **Modified Backus-Naur Form (BNF)** delta would be:
118+
119+
```diff
120+
diff --git a/Documentation/ABI/JSON.md b/Documentation/ABI/JSON.md
121+
index e4ff24a..1a82996 100644
122+
--- a/Documentation/ABI/JSON.md
123+
+++ b/Documentation/ABI/JSON.md
124+
@@ -157,10 +157,26 @@ additional `"testCases"` field describing the individual test cases.
125+
["displayName": <string>,] ; the user-supplied custom display name
126+
"sourceLocation": <source-location>, ; where the test is defined
127+
"id": <test-id>,
128+
- "isParameterized": <bool> ; is this a parameterized test function or not?
129+
+ "isParameterized": <bool>, ; is this a parameterized test function or not?
130+
+ ["tags": <array:tag>,] ; the tags associated with this test function
131+
+ ["bugs": <array:bug>,] ; the bugs associated with this test function
132+
+ ["timeLimit": <timeLimit>] ; the time limits associated with this test function
133+
+
134+
}
135+
136+
<test-id> ::= <string> ; an opaque string representing the test case
137+
+
138+
+<tag> ::= <string> ; a string representation of a tag
139+
+
140+
+<bug> ::= {
141+
+ ["url": <string>,] ; the bug url
142+
+ ["id": <string>,] ; the bug id
143+
+ "title": <string> ; the human readable bug title
144+
+} ;
145+
+
146+
+<timeLimit> ::= <number> ; the test time limit (in seconds)
147+
```
148+
149+
## Source compatibility
150+
151+
This proposal maintains full backward compatibility through careful design:
152+
153+
- **ABI Version Protection**: New fields are conditionally included based on ABI
154+
version checks, ensuring older tools continue to function without modification
155+
- **Experimental Feature Migration**: The existing experimental `_tag` field is
156+
replaced with the `tags` array. Since experimental features don't provide
157+
stability guarantees, this replacement doesn't constitute a breaking change
158+
- **Graceful Degradation**: Tools that don't expect the new fields will simply ignore
159+
them, while updated tools can leverage the enhanced metadata
160+
161+
No existing functionality is affected, making this a purely additive enhancement.
162+
163+
## Integration with supporting tools
164+
165+
The enhanced JSON ABI opens up exciting possibilities for the Swift Testing ecosystem:
166+
167+
### Immediate Benefits for Tool Developers
168+
- **IDE Extensions**: Can now provide tag-based test filtering and organization
169+
- **CI/CD Integrations**: Can generate more detailed reports with test categorization
170+
- **Performance Monitoring**: Can track and alert on time limit violations
171+
- **Bug Tracking Integration**: Can correlate test results with known issues
172+
173+
### Migration Path
174+
Existing tools will continue to work unchanged, as the new fields are purely additive.
175+
Tool developers can incrementally adopt the enhanced metadata at their own pace,
176+
choosing which fields provide the most value for their specific use cases.
177+
178+
### Enhanced Capabilities
179+
With access to this metadata, tools can now offer features like:
180+
- Filtering test runs by tags for faster feedback cycles
181+
- Generating reports that group results by test categories
182+
- Tracking performance regressions against defined time limits
183+
- Automatically linking test failures to relevant bug reports
184+
185+
## Future directions
186+
187+
This enhancement establishes future richer tooling experiences:
188+
189+
### Potential Extensions
190+
- **Additional Metadata**: Other test traits could be exposed as the ecosystem evolves
191+
- **Enhanced Bug Integration**: More sophisticated bug tracking integration with status updates
192+
- **Performance Analytics**: Historical time limit data for performance trend analysis
193+
194+
### Ecosystem Growth
195+
By providing this metadata, we anticipate growth in the Swift Testing tooling
196+
ecosystem, with new tools emerging to take advantage of the richer data available.
197+
This proposal positions Swift Testing as a platform that truly enables innovative
198+
testing tools and workflows.
199+
200+
## Alternatives considered
201+
202+
### Alternative Field Naming
203+
- **`timeLimitInSeconds` vs `timeLimit`**: We chose the shorter `timeLimit` name for
204+
consistency with Swift Testing's existing API, with the time unit documented in the
205+
schema specification
206+
207+
### Alternative Data Structures
208+
- **Flattened vs Structured Bug Information**: We chose a structured approach for bug
209+
metadata to accommodate various bug tracking systems while maintaining extensibility
210+
211+
### Implementation Approaches
212+
- **Always Include vs Conditional Fields**: We selected conditional inclusion to
213+
keep JSON output clean and avoid null values, improving the developer experience
214+
for tools consuming the data
215+
216+
## Acknowledgments
217+
218+
Thanks to [Jonathan Grynspan](https://github.com/grynspan) for suggesting to me
219+
I write this proposal and for providing feedback.
220+
221+
Thanks to [Paul LeMarqaund](https://github.com/plemarquand) for providing proposal
222+
feedback before it was posted.

0 commit comments

Comments
 (0)