Currently the test runner collect .pre and .ps1 test files from the tests/cases/passing/ and tests/cases/failing directories, invokes them and determine whether they passed based on the exit code. This results in extreme fragmentation, with one file per test.
Instead, store tests as JSON files matching this schema:
{
"type": "object",
"$defs": {
"testCase": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The name of this test case"
},
"language": {
"type": "string",
"enum": ["Prefix", "PowerShell"],
"description": "What language this test case is written in"
},
"source": {
"type": "string",
"description": "The test case source code"
}
},
"required": ["name", "language", "source"],
"additionalProperties": false
}
},
"properties": {
"passing": {
"type": "array",
"items": { "$ref": "#/$defs/testCase" }
},
"failing": {
"type": "array",
"items": { "$ref": "#/$defs/testCase" }
}
},
"anyOf": [
{ "required": ["passing"] },
{ "required": ["failing"] }
],
"additionalProperties": false
}
The test runner gathers all .json files from the tests/cases/ directory, parses them, and runs the tests.
Each testCase is a separate case regarding thread deployment, result validation, and CLI output. Failures should be reported as tests/cases/<file>.json: <testCase.name>: <UnexpectedFailure|UnexpectedSuccess>, which is more readable than the current output.
JSON files should be organized at a fine granularity: one file per operator, statement, literal type, or similar single-concern topic. Do not create a single large file for an entire category (e.g., one operators.json containing every operator). For example:
tests/cases/literals/string-literal.json
tests/cases/operators/arithmetic.json
tests/cases/statements/if.json
This grouping by topic improves test readability and browsing, while the JSON format enables a unified multi-language test system where both Prefix source and PowerShell integration tests can live in the same file.
Currently the test runner collect
.preand.ps1test files from thetests/cases/passing/andtests/cases/failingdirectories, invokes them and determine whether they passed based on the exit code. This results in extreme fragmentation, with one file per test.Instead, store tests as JSON files matching this schema:
{ "type": "object", "$defs": { "testCase": { "type": "object", "properties": { "name": { "type": "string", "description": "The name of this test case" }, "language": { "type": "string", "enum": ["Prefix", "PowerShell"], "description": "What language this test case is written in" }, "source": { "type": "string", "description": "The test case source code" } }, "required": ["name", "language", "source"], "additionalProperties": false } }, "properties": { "passing": { "type": "array", "items": { "$ref": "#/$defs/testCase" } }, "failing": { "type": "array", "items": { "$ref": "#/$defs/testCase" } } }, "anyOf": [ { "required": ["passing"] }, { "required": ["failing"] } ], "additionalProperties": false }The test runner gathers all
.jsonfiles from thetests/cases/directory, parses them, and runs the tests.Each
testCaseis a separate case regarding thread deployment, result validation, and CLI output. Failures should be reported astests/cases/<file>.json: <testCase.name>: <UnexpectedFailure|UnexpectedSuccess>, which is more readable than the current output.JSON files should be organized at a fine granularity: one file per operator, statement, literal type, or similar single-concern topic. Do not create a single large file for an entire category (e.g., one
operators.jsoncontaining every operator). For example:tests/cases/literals/string-literal.jsontests/cases/operators/arithmetic.jsontests/cases/statements/if.jsonThis grouping by topic improves test readability and browsing, while the JSON format enables a unified multi-language test system where both Prefix source and PowerShell integration tests can live in the same file.