Skip to content

Update test format #236

Description

@python-processing-unit

Currently the test runner collect .pre and .ps1 test files from the tests/cases/passing/ and tests/cases/failing directories, invokes them and determine whether they passed based on the exit code. This results in extreme fragmentation, with one file per test.

Instead, store tests as JSON files matching this schema:

{
  "type": "object",
  "$defs": {
    "testCase": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "The name of this test case"
        },
        "language": {
          "type": "string",
          "enum": ["Prefix", "PowerShell"],
          "description": "What language this test case is written in"
        },
        "source": {
          "type": "string",
          "description": "The test case source code"
        }
      },
      "required": ["name", "language", "source"],
      "additionalProperties": false
    }
  },
  "properties": {
    "passing": {
      "type": "array",
      "items": { "$ref": "#/$defs/testCase" }
    },
    "failing": {
      "type": "array",
      "items": { "$ref": "#/$defs/testCase" }
    }
  },
  "anyOf": [
    { "required": ["passing"] },
    { "required": ["failing"] }
  ],
  "additionalProperties": false
}

The test runner gathers all .json files from the tests/cases/ directory, parses them, and runs the tests.

Each testCase is a separate case regarding thread deployment, result validation, and CLI output. Failures should be reported as tests/cases/<file>.json: <testCase.name>: <UnexpectedFailure|UnexpectedSuccess>, which is more readable than the current output.

JSON files should be organized at a fine granularity: one file per operator, statement, literal type, or similar single-concern topic. Do not create a single large file for an entire category (e.g., one operators.json containing every operator). For example:

  • tests/cases/literals/string-literal.json
  • tests/cases/operators/arithmetic.json
  • tests/cases/statements/if.json

This grouping by topic improves test readability and browsing, while the JSON format enables a unified multi-language test system where both Prefix source and PowerShell integration tests can live in the same file.

Metadata

Metadata

Labels

enhancementNew feature or requestpatchRequires a patch version change.testSomething related to the test suite.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions