From 4fa22a3fc5d1d290c9b30bd255faf646c3a0e883 Mon Sep 17 00:00:00 2001 From: Huy Tieu Date: Wed, 9 Sep 2026 11:16:00 +0700 Subject: [PATCH 1/2] Add 18 Katalon True Platform testing skills Adds the katalon-* skill family covering the test lifecycle on Katalon True Platform: planning and estimation, test-case design, execution (manual, Run-with-AI, Playwright), result upload and reporting, failure triage, maintenance, and release readiness. Skills are agent-neutral and operate the platform through the Katalon MCP server (https://platform.katalon.io/mcp, streamable HTTP, OAuth 2.1). Source of truth is the public MIT-licensed repo katalon-labs/true-skills; katalon-platform-setup covers connecting the MCP before the other skills are used. Validated with npm run skill:validate (436 skills pass) and docs/README.skills.md regenerated via the build. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VjgFMQPShixvLoyCZfwrfL --- docs/README.skills.md | 18 + skills/katalon-analyze-failures/SKILL.md | 53 +++ .../references/failure-triage.md | 33 ++ skills/katalon-create-test-cases/SKILL.md | 124 +++++++ .../references/capability-boundaries.md | 32 ++ .../references/istqb-coverage.md | 64 ++++ .../references/manual-test-case-format.md | 45 +++ .../references/requirement-analysis.md | 33 ++ skills/katalon-execute-test/SKILL.md | 90 +++++ .../references/capability-boundaries.md | 32 ++ .../references/execution-workflow.md | 64 ++++ skills/katalon-exploratory-charter/SKILL.md | 177 ++++++++++ .../references/charter-and-session.md | 116 +++++++ .../references/oracles-and-tours.md | 75 ++++ skills/katalon-platform-setup/SKILL.md | 160 +++++++++ .../references/capability-boundaries.md | 32 ++ skills/katalon-playwright-execute/SKILL.md | 99 ++++++ .../playwright-katalon-reporting.md | 133 ++++++++ skills/katalon-release-analyze/SKILL.md | 124 +++++++ .../references/capability-boundaries.md | 32 ++ .../references/release-quality-gates.md | 72 ++++ .../SKILL.md | 320 ++++++++++++++++++ .../katalon-test-case-to-playwright/SKILL.md | 96 ++++++ .../references/playwright-typescript.md | 128 +++++++ skills/katalon-test-data/SKILL.md | 186 ++++++++++ .../references/data-design.md | 63 ++++ .../references/seed-and-teardown.md | 83 +++++ skills/katalon-test-estimation/SKILL.md | 228 +++++++++++++ .../references/estimation-model.md | 105 ++++++ .../references/mcp-evidence-map.md | 56 +++ skills/katalon-test-maintenance/SKILL.md | 54 +++ .../references/maintenance-loop.md | 41 +++ skills/katalon-test-management/SKILL.md | 50 +++ .../references/traceability.md | 35 ++ skills/katalon-test-plan/SKILL.md | 53 +++ .../references/planning-workflow.md | 44 +++ skills/katalon-test-reporting/SKILL.md | 202 +++++++++++ .../references/metric-selection.md | 46 +++ .../references/snapshot-and-trend.md | 78 +++++ skills/katalon-test-review/SKILL.md | 55 +++ .../references/review-rubric.md | 35 ++ skills/katalon-true-platform-testing/SKILL.md | 262 ++++++++++++++ .../references/combination-recipes.md | 45 +++ .../references/execution-workflow.md | 64 ++++ .../references/istqb-coverage.md | 64 ++++ .../references/lifecycle-map.md | 110 ++++++ .../references/manual-test-case-format.md | 45 +++ .../references/mcp-tool-index.md | 66 ++++ .../references/prompt-recipes.md | 39 +++ .../references/requirement-analysis.md | 33 ++ .../references/unavailable-capabilities.md | 32 ++ skills/katalon-upload-report/SKILL.md | 183 ++++++++++ .../references/capability-boundaries.md | 32 ++ .../references/cli-report-upload.md | 63 ++++ 54 files changed, 4604 insertions(+) create mode 100644 skills/katalon-analyze-failures/SKILL.md create mode 100644 skills/katalon-analyze-failures/references/failure-triage.md create mode 100644 skills/katalon-create-test-cases/SKILL.md create mode 100644 skills/katalon-create-test-cases/references/capability-boundaries.md create mode 100644 skills/katalon-create-test-cases/references/istqb-coverage.md create mode 100644 skills/katalon-create-test-cases/references/manual-test-case-format.md create mode 100644 skills/katalon-create-test-cases/references/requirement-analysis.md create mode 100644 skills/katalon-execute-test/SKILL.md create mode 100644 skills/katalon-execute-test/references/capability-boundaries.md create mode 100644 skills/katalon-execute-test/references/execution-workflow.md create mode 100644 skills/katalon-exploratory-charter/SKILL.md create mode 100644 skills/katalon-exploratory-charter/references/charter-and-session.md create mode 100644 skills/katalon-exploratory-charter/references/oracles-and-tours.md create mode 100644 skills/katalon-platform-setup/SKILL.md create mode 100644 skills/katalon-platform-setup/references/capability-boundaries.md create mode 100644 skills/katalon-playwright-execute/SKILL.md create mode 100644 skills/katalon-playwright-execute/references/playwright-katalon-reporting.md create mode 100644 skills/katalon-release-analyze/SKILL.md create mode 100644 skills/katalon-release-analyze/references/capability-boundaries.md create mode 100644 skills/katalon-release-analyze/references/release-quality-gates.md create mode 100644 skills/katalon-test-case-to-katalon-studio/SKILL.md create mode 100644 skills/katalon-test-case-to-playwright/SKILL.md create mode 100644 skills/katalon-test-case-to-playwright/references/playwright-typescript.md create mode 100644 skills/katalon-test-data/SKILL.md create mode 100644 skills/katalon-test-data/references/data-design.md create mode 100644 skills/katalon-test-data/references/seed-and-teardown.md create mode 100644 skills/katalon-test-estimation/SKILL.md create mode 100644 skills/katalon-test-estimation/references/estimation-model.md create mode 100644 skills/katalon-test-estimation/references/mcp-evidence-map.md create mode 100644 skills/katalon-test-maintenance/SKILL.md create mode 100644 skills/katalon-test-maintenance/references/maintenance-loop.md create mode 100644 skills/katalon-test-management/SKILL.md create mode 100644 skills/katalon-test-management/references/traceability.md create mode 100644 skills/katalon-test-plan/SKILL.md create mode 100644 skills/katalon-test-plan/references/planning-workflow.md create mode 100644 skills/katalon-test-reporting/SKILL.md create mode 100644 skills/katalon-test-reporting/references/metric-selection.md create mode 100644 skills/katalon-test-reporting/references/snapshot-and-trend.md create mode 100644 skills/katalon-test-review/SKILL.md create mode 100644 skills/katalon-test-review/references/review-rubric.md create mode 100644 skills/katalon-true-platform-testing/SKILL.md create mode 100644 skills/katalon-true-platform-testing/references/combination-recipes.md create mode 100644 skills/katalon-true-platform-testing/references/execution-workflow.md create mode 100644 skills/katalon-true-platform-testing/references/istqb-coverage.md create mode 100644 skills/katalon-true-platform-testing/references/lifecycle-map.md create mode 100644 skills/katalon-true-platform-testing/references/manual-test-case-format.md create mode 100644 skills/katalon-true-platform-testing/references/mcp-tool-index.md create mode 100644 skills/katalon-true-platform-testing/references/prompt-recipes.md create mode 100644 skills/katalon-true-platform-testing/references/requirement-analysis.md create mode 100644 skills/katalon-true-platform-testing/references/unavailable-capabilities.md create mode 100644 skills/katalon-upload-report/SKILL.md create mode 100644 skills/katalon-upload-report/references/capability-boundaries.md create mode 100644 skills/katalon-upload-report/references/cli-report-upload.md diff --git a/docs/README.skills.md b/docs/README.skills.md index 24473fa6f..dd9b436e1 100644 --- a/docs/README.skills.md +++ b/docs/README.skills.md @@ -251,6 +251,24 @@ See [CONTRIBUTING.md](../CONTRIBUTING.md#adding-skills) for guidelines on how to | [java-springboot](../skills/java-springboot/SKILL.md)
`gh skills install github/awesome-copilot java-springboot` | Get best practices for developing applications with Spring Boot. | None | | [javascript-typescript-jest](../skills/javascript-typescript-jest/SKILL.md)
`gh skills install github/awesome-copilot javascript-typescript-jest` | Best practices for writing JavaScript/TypeScript tests using Jest, including mocking strategies, test structure, and common patterns. | None | | [javax-to-jakarta-migration](../skills/javax-to-jakarta-migration/SKILL.md)
`gh skills install github/awesome-copilot javax-to-jakarta-migration` | Migrate Java code from javax.* to jakarta.* namespace. Use when upgrading to Tomcat 11, Jakarta EE 10, or when javax imports are detected in the codebase. | None | +| [katalon-analyze-failures](../skills/katalon-analyze-failures/SKILL.md)
`gh skills install github/awesome-copilot katalon-analyze-failures` | Triage Katalon True Platform/TestOps test failures and file defects. Use when you need to investigate failed test results, classify each failure as product defect vs automation defect vs environment/data issue, cluster failures by common signature, find likely root cause from execution data, and optionally create ALM-linked defects for real product bugs. This is failure diagnosis and defect filing; for the overall ship/no-ship release call use katalon-release-analyze, and for repairing the tests themselves use katalon-test-maintenance. Written for the manual tester and the automation tester looking at a red run and needing to know whether the application broke or the test did. | `references/failure-triage.md` | +| [katalon-create-test-cases](../skills/katalon-create-test-cases/SKILL.md)
`gh skills install github/awesome-copilot katalon-create-test-cases` | Create, update, organize, and link Katalon True Platform/TestOps manual test cases from a synced requirement key such as CEL-6, a read requirement, or free-text product behavior. Use when you need to analyze requirements, design manual test cases using ISTQB techniques as a reference, check existing Katalon coverage, avoid duplicate test cases, import only missing cases, update or link existing cases, or create/reuse a test suite for newly designed cases. For full requirement-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has a written requirement in hand and no cases for it yet. | `references/capability-boundaries.md`
`references/istqb-coverage.md`
`references/manual-test-case-format.md`
`references/requirement-analysis.md` | +| [katalon-execute-test](../skills/katalon-execute-test/SKILL.md)
`gh skills install github/awesome-copilot katalon-execute-test` | Execute Katalon True Platform/TestOps tests when the input is an existing test case, manual test case list, test suite, suite collection, execution request, or "run with AI" instruction. Use when you need to create a manual test run, start Run with AI, poll AI session results, schedule automated suites, read execution/test results, or summarize pass/fail/blocked outcomes. For full requirement-to-test-design-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has cases and needs a result, by hand or through Run with AI. A coded suite driven from a framework starts at katalon-playwright-execute or katalon-upload-report. | `references/capability-boundaries.md`
`references/execution-workflow.md` | +| [katalon-exploratory-charter](../skills/katalon-exploratory-charter/SKILL.md)
`gh skills install github/awesome-copilot katalon-exploratory-charter` | Write, run, and debrief exploratory testing charters against Katalon True Platform/TestOps when there is no script to follow. Use when you need to turn a vague area into a charter (mission, areas, oracles, timebox), run a timeboxed unscripted session, log what you find as session notes, judge which findings are real defects, and promote what was learned into atomic manual cases plus the folder or suite that holds them. This is unscripted discovery and the debrief after it; when a written requirement is already in hand and the job is designing cases from it, use katalon-create-test-cases instead. Written for the manual tester who has an hour, a feature, and no cases to run against it. | `references/charter-and-session.md`
`references/oracles-and-tours.md` | +| [katalon-platform-setup](../skills/katalon-platform-setup/SKILL.md)
`gh skills install github/awesome-copilot katalon-platform-setup` | Set up, verify, and troubleshoot Katalon True Platform/TestOps MCP connectivity for any AI coding agent. Use when you need to install or configure the Katalon MCP server, connect to a Katalon platform account, verify available Katalon MCP tools, diagnose missing authentication or repository access, confirm project/repository discovery, or explain what platform operations are and are not exposed through the MCP. One-time cross-cutting setup, run once by whoever wires the agent to the platform, before any lifecycle skill can work. | `references/capability-boundaries.md` | +| [katalon-playwright-execute](../skills/katalon-playwright-execute/SKILL.md)
`gh skills install github/awesome-copilot katalon-playwright-execute` | Run Playwright tests or suites and upload the resulting report to Katalon True Platform. Use when you need to execute Playwright scripts, package scripts, spec files, projects, or suites, configure or verify @katalon/playwright-reporter, upload Playwright reports with Katalon CLI/reporter commands, and verify uploaded Katalon Platform runs or reports. Written for the automation tester who wants the suite driven and the results landed in the platform in one step. For a report from a run that has already finished, or from any other framework, use katalon-upload-report. | `references/playwright-katalon-reporting.md` | +| [katalon-release-analyze](../skills/katalon-release-analyze/SKILL.md)
`gh skills install github/awesome-copilot katalon-release-analyze` | Analyze Katalon True Platform/TestOps release readiness from testing quality data. Use when you need to use Katalon MCP metrics and results to assess whether a release, sprint, iteration, version, test plan, suite, or repository is ready to ship; summarize requirement coverage, execution health, defect risk, test stability, configuration coverage, release blockers, quality gaps, and produce a Ready / Ready with risk / Not ready recommendation. Written for the test manager who owns the ship call and the test lead who has to defend it. | `references/capability-boundaries.md`
`references/release-quality-gates.md` | +| [katalon-test-case-to-katalon-studio](../skills/katalon-test-case-to-katalon-studio/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-case-to-katalon-studio` | Convert Katalon True Platform/TestOps manual test cases into Katalon Studio automation inside a local Studio Test Project checkout. Use when you need to author or extend a .tc test case file and its paired Groovy script under Scripts/, keep test case variable GUIDs consistent with the .ts test suite bindings that read them, model page elements as .rs test objects in the Object Repository instead of inline selectors, write WebUI, Mobile, or WS keyword steps in Groovy with Java interop, or wire data files, execution profiles, and GlobalVariable into a generated case. Also use to decide whether a headless katalonc run is possible at all, since Katalon Runtime Engine is a separately licensed paid runtime whose CLI requires -apiKey. Once a Studio run has finished and its report exists, hand off to katalon-upload-report. Written for the automation tester who has to land a manual case inside a Katalon Studio project rather than a code-first framework. | None | +| [katalon-test-case-to-playwright](../skills/katalon-test-case-to-playwright/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-case-to-playwright` | Convert Katalon True Platform/TestOps manual test cases, test suites, or requirement-linked cases into Playwright TypeScript automation. Use when you need to fetch/read Katalon Platform test cases and implement Playwright scripts, create or adapt a Playwright framework, apply Page Object Model and fixtures, or translate manual steps into meaningful automated test keywords. Written for the automation tester converting a manual case into code that fits an existing page-object layer. | `references/playwright-typescript.md` | +| [katalon-test-data](../skills/katalon-test-data/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-data` | Design, source, seed, and tear down the test data a Katalon True Platform test case or an automated suite runs on. Use when the steps are already settled and the blocker is the values, for example which data classes a case needs, which records must exist before a run, how to keep literals out of the step text and into the Test Data column or a fixture, and how to reset state afterwards so the next run starts clean. Covers choosing between static, generated, and cloned production data, keeping credentials out of test data, and the boundary that the Katalon MCP has no test data, fixture, seeding, or secrets tool of its own. If the cases do not exist yet, start at katalon-create-test-cases. Written for the manual tester filling in a case's Test Data column and precondition, and the automation tester wiring fixtures and teardown for a suite. | `references/data-design.md`
`references/seed-and-teardown.md` | +| [katalon-test-estimation](../skills/katalon-test-estimation/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-estimation` | Estimate testing effort, duration, and resourcing for a Katalon True Platform/TestOps cycle. Use when the question is how long testing will take, how many testers it needs, whether the scope fits the sprint window, or what a scope change costs in person-hours. Sizes design, manual execution, automated execution and triage, and rework separately, counts the countable part from platform data (case counts, automation split, historical pass and stability rates, configuration matrix), calibrates the rest against a rate the team supplies, and returns a three-point range with a confidence label instead of a single number. Splits resourcing across the manual and automated lanes and names the assumptions that would move the number most. For what to test and in what order, use katalon-test-plan; for a verdict on a cycle that has already run, use katalon-release-analyze. Written for the test lead sizing a cycle before it starts and the test manager who has to fund it. | `references/estimation-model.md`
`references/mcp-evidence-map.md` | +| [katalon-test-maintenance](../skills/katalon-test-maintenance/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-maintenance` | Maintain and evolve a Katalon True Platform/TestOps regression suite as the application changes. Use when you need to detect which tests broke or became flaky from stability and result history, diagnose whether a case needs repair vs regeneration, repair test assets (update, move, reorganize cases), refresh coverage after application or requirement changes, and feed the resulting gap list back into planning. Closes the lifecycle loop. For classifying a specific run's failures use katalon-analyze-failures; for authoring brand-new cases use katalon-create-test-cases. Written for the automation tester repairing a suite that has started lying, and the test lead who has to keep it trustworthy. | `references/maintenance-loop.md` | +| [katalon-test-management](../skills/katalon-test-management/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-management` | Organize, classify, and trace Katalon True Platform/TestOps test assets. Use when you need to structure test cases into folders and suites, move or reorganize cases, search and find existing assets at scale, link or unlink requirements to test cases, or produce a requirement-to-test traceability report (which requirements have coverage, which cases are orphaned, coverage percentage). Prefer this skill for inventory hygiene and traceability audits. For authoring new cases use katalon-create-test-cases; for coverage quality verdicts use katalon-test-review. Written for the test lead doing inventory hygiene on a repository nobody has curated in months. | `references/traceability.md` | +| [katalon-test-plan](../skills/katalon-test-plan/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-plan` | Plan Katalon True Platform/TestOps testing for a release, sprint, or feature. Use when you need to translate quality goals into scope, prioritize testing by requirement coverage and risk, decide what to test first, or build the executable plan structure (folders, suites, and sprint/release association) that stands in for a formal Test Plan. Reads project/repository/iteration context and requirement coverage, then proposes and materializes a prioritized plan. For designing the actual test cases, hand off to katalon-create-test-cases; for the ship/no-ship call, hand off to katalon-release-analyze. Written for the test lead who owns the cycle and has to decide what gets tested first. | `references/planning-workflow.md` | +| [katalon-test-reporting](../skills/katalon-test-reporting/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-reporting` | Report Katalon True Platform/TestOps quality metrics to people outside QA. Use when you need to answer a stakeholder question with testing data, choose the few metrics that actually answer it, trend coverage, execution health, defect risk and stability across several releases, sprints, or iterations rather than inside one, and write the summary a manager presents upward at a steering committee, an exec review, or a quality business review. Produces a headline answer with the named specifics behind it, never a metric dump. The MCP returns current state with no history tool, so trends are built from execution dates plus snapshots this skill stores per period. For the ship or no-ship call on a single release, use katalon-release-analyze; for a pre-pipeline suite verdict, use katalon-test-review. Written for the test manager who has to present quality upward and the test lead who assembles the numbers. | `references/metric-selection.md`
`references/snapshot-and-trend.md` | +| [katalon-test-review](../skills/katalon-test-review/SKILL.md)
`gh skills install github/awesome-copilot katalon-test-review` | Review Katalon True Platform/TestOps test quality and coverage before tests enter the delivery pipeline. Use when you need to check whether a suite is ready to run, review requirement and configuration coverage, assess test-case quality and flakiness/stability, spot weak or unreliable cases, and produce a review verdict (Approve / Approve with fixes / Reject for pipeline) with the specific cases to fix. This is a pre-execution quality gate, not a ship decision; for the release ship/no-ship call use katalon-release-analyze. Written for the test lead gating what is allowed into the pipeline, one suite at a time. | `references/review-rubric.md` | +| [katalon-true-platform-testing](../skills/katalon-true-platform-testing/SKILL.md)
`gh skills install github/awesome-copilot katalon-true-platform-testing` | End-to-end Katalon True Platform testing workflow and lifecycle router. Use when one request spans several stages and no single skill owns all of it, for example analyze a requirement, design and import the cases, build a suite, run it with AI, and report the outcome. Also use to route any testing request across the full 7-stage lifecycle (plan, design, manage, review, execute, analyze, maintain) to the right focused skill, and for deciding what is and is not available through the Katalon MCP tools. Start here when a request names a job rather than one task, such as drive the whole chain from requirement to ship call. Also start here when the asker wants orienting before acting, for example where do I start, which skill do I need, or I own quality here and do not know where to begin. Routes the asker to the skill that owns their next step, whether they say manual tester, QA analyst, test analyst, automation tester, SDET, automation engineer, QA engineer, test lead, QA lead, test manager, or QA manager. | `references/combination-recipes.md`
`references/execution-workflow.md`
`references/istqb-coverage.md`
`references/lifecycle-map.md`
`references/manual-test-case-format.md`
`references/mcp-tool-index.md`
`references/prompt-recipes.md`
`references/requirement-analysis.md`
`references/unavailable-capabilities.md` | +| [katalon-upload-report](../skills/katalon-upload-report/SKILL.md)
`gh skills install github/awesome-copilot katalon-upload-report` | Run automation and upload or verify Katalon Platform reports for Katalon Studio/KRE, JUnit XML, and Playwright reports. Use when you need to combine Katalon MCP project/result discovery with Katalon CLI execution, Katalon Report Uploader, or @katalon/playwright-reporter; configure report folders, report types, Platform/TestOps IDs, verify uploaded runs, handle failed auto-upload, or summarize execution results after automation completes. Written for the automation tester whose framework has already run and whose report is sitting on disk. If the Playwright run has not happened yet, start at katalon-playwright-execute. | `references/capability-boundaries.md`
`references/cli-report-upload.md` | | [kotlin-mcp-server-generator](../skills/kotlin-mcp-server-generator/SKILL.md)
`gh skills install github/awesome-copilot kotlin-mcp-server-generator` | Generate a complete Kotlin MCP server project with proper structure, dependencies, and implementation using the official io.modelcontextprotocol:kotlin-sdk library. | None | | [kotlin-springboot](../skills/kotlin-springboot/SKILL.md)
`gh skills install github/awesome-copilot kotlin-springboot` | Get best practices for developing applications with Spring Boot and Kotlin. | None | | [landing-page-conversion-audit](../skills/landing-page-conversion-audit/SKILL.md)
`gh skills install github/awesome-copilot landing-page-conversion-audit` | Audit a landing page, sales page or checkout page for conversion leaks and return a fix list ordered by expected revenue impact. Use when asked to review, critique or improve a landing page, sales page, opt-in page, product page or checkout flow, when conversion rate is low, when paid traffic is not converting, or when someone asks "why isn't this page converting" or wants a CRO / landing page review. | None | diff --git a/skills/katalon-analyze-failures/SKILL.md b/skills/katalon-analyze-failures/SKILL.md new file mode 100644 index 000000000..ec073eb05 --- /dev/null +++ b/skills/katalon-analyze-failures/SKILL.md @@ -0,0 +1,53 @@ +--- +name: katalon-analyze-failures +description: Triage Katalon True Platform/TestOps test failures and file defects. Use when you need to investigate failed test results, classify each failure as product defect vs automation defect vs environment/data issue, cluster failures by common signature, find likely root cause from execution data, and optionally create ALM-linked defects for real product bugs. This is failure diagnosis and defect filing; for the overall ship/no-ship release call use katalon-release-analyze, and for repairing the tests themselves use katalon-test-maintenance. Written for the manual tester and the automation tester looking at a red run and needing to know whether the application broke or the test did. +--- + +# Katalon Analyze Failures + +Use this skill for the **failure-analysis** part of the report/analysis stage: turn a set of failed results into a diagnosis and, when warranted, filed defects. The core value is **classification** — separating real product bugs from automation and environment noise. + +## Availability Boundary + +- **Available via MCP:** read results (`read_test_result`, `read_execution_test_results`, `find_test_results`, `read_execution`), defect context (`fetch_defect_data`), ALM discovery + filing (`find_alm_integration_projects`, `create_defect`). +- **Not directly available:** AI root-cause summarization and automation-error-pattern analytics are TestOps/Studio product features, not MCP calls — narrate their availability, do not claim to call them. `create_defect` requires a **known failed test result ID** and ALM integration details; there is no ID-less defect creation. + +## Triage Workflow + +```text ++---------------------+ +----------------------+ +----------------------+ +| Collect failures | --> | Classify each | --> | Cluster by signature | +| read results | | product/auto/env | | | ++---------------------+ +----------------------+ +----------------------+ + | + v + +----------------------+ + | File defects (asked) | + +----------------------+ +``` + +## Steps and tool rules + +1. **Collect the failures.** `find_test_results` (recent/specific) or `read_execution_test_results` for a run; `read_test_result` per failed case for detail. +2. **Classify each failure** into one bucket: + - **Product defect** — the application behaved wrong (assertion on real behavior failed, unexpected error/state). Candidate for a filed defect. + - **Automation defect** — the test is wrong (bad locator, timing, stale data, broken step). Route to `katalon-test-maintenance`. + - **Environment / data** — infra, account, network, fixture, or AUT-state issue. Route to re-run after fix. +3. **Cluster by signature.** Group failures with the same error message / step / object so one root cause is not filed as N defects. +4. **Check existing defects.** `fetch_defect_data` to avoid duplicate filings. +5. **File defects only when asked and only for product defects.** `find_alm_integration_projects` -> `create_defect` with the failed result ID. Ask before creating unless the user explicitly requested defect filing. +6. **Report.** Per cluster: classification, likely cause, affected cases, and action (file / repair / re-run). + +## Prompt recipes + +- `Triage the failures in execution 8842: which are product bugs vs flaky tests vs environment?` +- `Cluster today's failed results by root cause and tell me what to file.` +- `File defects for the confirmed product bugs in the checkout suite and link them to the failed results.` + +## Hand-offs + +- Automation defects / flaky -> `katalon-test-maintenance`. +- Ship decision from the failure picture -> `katalon-release-analyze`. +- Coverage gap exposed by a failure -> `katalon-test-plan`. + +Read `references/failure-triage.md` before classifying. Consult the orchestrator's `references/unavailable-capabilities.md` for defect-filing boundaries. diff --git a/skills/katalon-analyze-failures/references/failure-triage.md b/skills/katalon-analyze-failures/references/failure-triage.md new file mode 100644 index 000000000..fc018e450 --- /dev/null +++ b/skills/katalon-analyze-failures/references/failure-triage.md @@ -0,0 +1,33 @@ +# Failure triage reference + +## Classification signals + +| Bucket | Typical signals | Action | +|---|---|---| +| Product defect | assertion on real app behavior failed; unexpected app error/state; reproducible across runs and environments | file defect (with failed result ID) | +| Automation defect | element-not-found / locator error; timing / wait failure; stale test data; step no longer matches UI; passes on re-run without app change | route to `katalon-test-maintenance` | +| Environment / data | infra/network error; account/permission; missing fixture; wrong AUT state; TestCloud/agent issue | fix environment, then re-run | + +Reproducibility is the strongest signal: a failure that repeats across environments and runs leans product; one that vanishes on re-run leans automation/environment. + +## Clustering + +Group by the tuple `(failing step, error message, object/locator)`. One cluster = one candidate root cause = at most one defect. Report the cluster size so a single bug is not filed N times. + +## Defect filing rules + +- Only for **product defects**, only when the user asked (or explicitly pre-approved failure filing). +- Requires a **failed test result ID** — get it from `read_test_result` / `read_execution_test_results`. +- `find_alm_integration_projects` first if the ALM project/integration IDs are unknown. +- `fetch_defect_data` to check for an existing defect before creating a duplicate. +- Never invent a stack trace, defect ID, or ALM project. If the ID or integration is missing, report blocked with the exact missing input. + +## Report template + +```text +Cluster 1 — (N cases) + Classification: product | automation | environment + Likely cause: + Cases: + Action: file DEF-xxx | repair (maintenance) | re-run after +``` diff --git a/skills/katalon-create-test-cases/SKILL.md b/skills/katalon-create-test-cases/SKILL.md new file mode 100644 index 000000000..8512ff5f6 --- /dev/null +++ b/skills/katalon-create-test-cases/SKILL.md @@ -0,0 +1,124 @@ +--- +name: katalon-create-test-cases +description: Create, update, organize, and link Katalon True Platform/TestOps manual test cases from a synced requirement key such as CEL-6, a read requirement, or free-text product behavior. Use when you need to analyze requirements, design manual test cases using ISTQB techniques as a reference, check existing Katalon coverage, avoid duplicate test cases, import only missing cases, update or link existing cases, or create/reuse a test suite for newly designed cases. For full requirement-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has a written requirement in hand and no cases for it yet. +--- + +# Katalon Create Test Cases + +Use this skill for the test design and import portion of Katalon True Platform work. Keep the larger `katalon-true-platform-testing` skill available as the end-to-end orchestrator; this skill is only the focused create/update/link workflow. + +## Availability Boundary + +State the Katalon MCP boundary before promising writes: + +- Available: list projects/repositories, find/read requirements, create/read/update/move test cases, find/manage folders, find/read/manage test suites, and link requirements to test cases. +- Not directly available: create requirements, create a formal Test Plan entity, inspect the live AUT UI, or guarantee downstream AI execution. +- Workaround for a test plan: create or reuse a named test suite/folder as the executable planning structure. + +Read `references/capability-boundaries.md` when the user asks whether Katalon can do a specific operation. + +## Resolve Context First + +Before mutating Katalon data: + +1. Call `list_projects`. +2. Call `list_repositories`. +3. Resolve the repository/Test Project from the user's wording, requirement key, or unique available repository. + +Rules: + +- Treat repository and Test Project as the same resolution target. +- If exactly one repository exists, use it. +- If multiple equally plausible repositories remain, ask the user to choose. +- If the user says "Katalon Cloud" or "cloud repo", prefer a repository named `Katalon Cloud` when present. +- Do not scan every repository just to avoid asking. + +## Analyze Requirement + +If the user provides a requirement key, use `find_requirements` or `read_requirement`. If the user provides free text, analyze it locally and only link requirements when a real requirement ID is known. + +Output or internally track: + +- Requirement intent +- Personas +- Main flows +- Alternate and negative flows +- Data and environment assumptions +- Risk areas +- Coverage recommendations + +Read `references/requirement-analysis.md` before analyzing non-trivial requirements. + +## Design Coverage + +Design manual cases using ISTQB techniques as a reference for coverage (the techniques guide the design; the deliverable is plain platform test cases, not an ISTQB certification): + +- Equivalence partitioning for input classes, statuses, roles, filters, and product states. +- Boundary value analysis for ranges, quantities, prices, dates, pagination, and text lengths. +- Decision tables for business rules with multiple conditions. +- State transitions for lifecycle flows such as cart, checkout, status, and execution. +- Use-case scenarios for realistic end-to-end user journeys. +- Error guessing for ecommerce, account, permissions, environment, and data risks. + +### Make each case atomic + +Each test case targets **one validation condition (one acceptance-criteria line)** — but it must still be a **complete, independently runnable flow**, not a single bare assertion. + +- "Atomic" describes the *scope under test* (one rule per case), not the step count. Every case walks the real path to that condition: precondition/navigation -> enter the surrounding valid data -> perform the action under test -> verify the expected result. A case that is a lone step like "count the columns" is too thin; lead with the steps to reach and exercise that state so the case runs on its own (and so Run with AI can execute it). +- Split each input class, each required field, each boundary value, and each error message into its own case. Example: a password rule of "min 8 chars, has a letter, has a number, no spaces" becomes separate cases for too-short (7), exactly-8 valid, letters-only, numbers-only, and contains-space — each one a full fill-the-form-and-submit flow, differing only in the field under test. +- Cover both the happy-path flow and its edge cases. For every feature, include at least the main success flow plus the boundary and negative variants around it; do not stop at the positive path. +- Reserve genuinely combined multi-feature cases for true end-to-end scenarios (e.g. register -> log in -> land on dashboard), never as a container for unrelated checks. +- Why: a case mixing several *conditions* fails as a whole, so the result cannot tell you which rule broke and you lose 1 requirement-line -> 1 test-result traceability. Atomic-scope cases pinpoint the failing rule and map cleanly back to the requirement. +- Quote expected error and UI strings **verbatim** from the requirement, including any source typos. Flag suspected typos separately; never silently "correct" them in the expected result, or the test will assert behavior the app does not produce. + +Read `references/istqb-coverage.md` before creating cases from requirements, and `references/manual-test-case-format.md` before importing several cases. + +## Platform Constraints + +- Test case **names** accept only letters, numbers, spaces, and `( ) . , _ -`. Avoid other symbols (such as `@`, `:`, `/`) in titles; keep them in descriptions or steps instead. Folder paths may use `/`. +- Prefer `update_test_case` over delete-and-recreate when adjusting an existing set. Deletion can fail server-side, and updating in place keeps IDs, links, and history intact. + +## Check Existing Cases + +This step is mandatory before every create/import attempt, including retries after partial failure: + +1. If requirement IDs are known, call `find_test_cases_by_requirement`. +2. Search by requirement key, title keywords, feature area, and target folder with `find_test_cases`. +3. Read likely matches with `read_test_case` when the title alone is not enough to judge coverage. +4. Reuse, update, move, or link existing cases when they already cover the behavior. +5. Create new cases only for uncovered behavior, missing coverage classes, or clearly obsolete/incorrect cases. + +Never create a duplicate just because a previous create call failed. + +## Create Or Update Cases + +Use Katalon tools in this order: + +1. `create_test_case` only for uncovered manual cases. +2. `update_test_case` for revisions, passing all intended updates in one call. +3. `link_requirements_to_test_case` only after requirement IDs are known. +4. `read_test_case` after creation or update when verification matters. +5. `manage_test_folder` or `move_test_case` only when organization is requested or clearly needed. + +Use this manual case shape: + +- Title +- Description +- Pre-condition +- Steps +- Expected results +- Test data +- Priority +- Requirement links + +## Test Suite Handling + +When the user asks to create a test suite/test plan from the cases: + +1. Search existing suites with `find_test_suites` before creating anything. +2. Read likely matching suites with `read_test_suite`. +3. Reuse the matching suite and add missing cases instead of creating a duplicate. +4. Create a new suite with `manage_test_suite` only when no suitable suite exists. +5. Verify the final suite with `read_test_suite`. + +Report what was reused, updated, newly created, linked, and left uncovered. diff --git a/skills/katalon-create-test-cases/references/capability-boundaries.md b/skills/katalon-create-test-cases/references/capability-boundaries.md new file mode 100644 index 000000000..2cb5fcf95 --- /dev/null +++ b/skills/katalon-create-test-cases/references/capability-boundaries.md @@ -0,0 +1,32 @@ +# Katalon MCP Capability Boundaries + +## Available + +- Project discovery: `list_projects`. +- Repository/Test Project discovery: `list_repositories`. +- Requirement discovery: `find_requirements`, `read_requirement`. +- Requirement coverage: `fetch_requirement_data`. +- Test case operations: `create_test_case`, `read_test_case`, `update_test_case`, `duplicate_test_case`, `delete_test_case`, `move_test_case`, `find_test_cases`. +- Test folder operations: `find_test_folders`, `manage_test_folder`. +- Test suite operations: `find_test_suites`, `read_test_suite`, `manage_test_suite`. +- Requirement links: `link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`. +- Manual execution: `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`. +- Automated execution: `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, `build_schedule`, `schedule_test_run`. +- Execution results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Quality data: requirement, defect, test case, test stability, and configuration coverage fetch tools. +- ALM defects: `find_alm_integration_projects`, `create_defect`. + +## Not Directly Available + +- Create requirements in Katalon True Platform. Requirements are synced from Jira/Azure and can be found/read/linked. +- Create a formal Test Plan entity. Use test suites/folders/executions as the executable planning structure. +- Guarantee Run with AI completion. The platform may block, fail, or require AUT/account state. +- Inspect AUT pages through Katalon MCP. Use Browser/Playwright for website exploration. +- Create defects without a failed test result ID and ALM integration details. + +## Recommended Workarounds + +- Requirement creation: create in Jira/Azure first, then sync/find/link in Katalon. +- Test plan: create a named folder and/or test suite, link to sprint/release, and create execution from that suite. +- AUT exploration: use Browser/Playwright to understand the product, then import manual cases into Katalon. +- AI execution blocked: report blocked state with required fixture, AUT, account, or environment action. diff --git a/skills/katalon-create-test-cases/references/istqb-coverage.md b/skills/katalon-create-test-cases/references/istqb-coverage.md new file mode 100644 index 000000000..cfd05ce03 --- /dev/null +++ b/skills/katalon-create-test-cases/references/istqb-coverage.md @@ -0,0 +1,64 @@ +# Coverage Guide (ISTQB techniques as reference) + +Use this guide before writing or importing test cases. ISTQB is the **reference toolkit** for designing coverage here, not the deliverable: the output is plain platform test cases designed *using* these techniques, not "ISTQB cases" certified against the standard. The goal is enough risk-based coverage, not maximum case count. + +## Technique Selection + +- Equivalence Partitioning: use for valid/invalid classes such as product categories, brands, stock states, payment methods, account roles, and form input classes. +- Boundary Value Analysis: use for numeric or ordered values such as price range, quantity, pagination, character limits, dates, and timeout thresholds. +- Decision Table Testing: use when outcomes depend on combinations of conditions, such as selected variant + stock + quantity, checkout field validity, shipping eligibility, or payment availability. +- State Transition Testing: use when behavior depends on prior state, such as empty cart -> item added -> quantity updated -> removed, checkout step progression, or execution TODO -> IN_TESTING -> PASSED/FAILED. +- Use Case / Scenario Testing: use for end-to-end journeys that represent user goals, such as browse -> select variant -> add to cart -> checkout. +- Error Guessing / Checklist-Based Testing: use for likely failures based on domain knowledge, such as broken images, stale cart totals, invalid email, unavailable product, duplicate submission, or navigation loss. +- Pairwise / Combinatorial Testing: use when many variables interact, such as browser x device x category x filter x sort, while preserving explicitly high-risk combinations. + +## Minimum Coverage Expectations + +For each requirement, identify: + +- At least one happy-path scenario. +- At least one negative or validation scenario when user input or branching exists. +- Boundary cases for numeric/date/range fields. +- State transitions for multi-step workflows. +- Role/permission coverage when roles exist. +- Data setup and cleanup assumptions. +- Requirement-to-test traceability. + +## Coverage Output Format + +Before importing tests, prepare a short coverage note: + +```text +Coverage Techniques: +- Use case testing: ... +- Equivalence partitions: ... +- Boundary values: ... +- Decision table/state transition: ... +- Error guessing risks: ... + +Selected Test Cases: +- P0: ... +- P1: ... +- P2: ... + +Deferred / Not Covered: +- ... +Reason: +- ... +``` + +## Case Granularity + +- Keep each case **atomic in scope**: one validation condition / one acceptance-criteria line per case, so a failure pinpoints the exact rule and each requirement line maps 1:1 to a result. +- Atomic does not mean a single step. Every case is a **complete, runnable flow**: precondition/navigation -> enter surrounding valid data -> perform the action under test -> verify the result. Avoid lone-step cases like "count the columns"; include the steps to reach and exercise that state so the case executes on its own (including under Run with AI). +- Cover the happy-path flow **and** its edge cases for every feature: main success flow plus boundary and negative variants. Do not stop at the positive path. +- Quote expected error/UI strings verbatim from the requirement, including source typos; flag suspected typos separately rather than correcting them in the expected result. + +## Practical Rules + +- Do not create redundant tests that exercise the same partition and same expected behavior. +- Prefer fewer strong tests over many shallow tests. +- Mark P0 for revenue, checkout, account, data-loss, or broken-entry-point risks. +- Mark P1 for important catalog, filter, sort, and traceability behaviors. +- Mark P2 for cosmetic, footer, secondary navigation, or low-risk edge cases. +- If a test is primarily exploratory or visual, say so and include what evidence is needed. diff --git a/skills/katalon-create-test-cases/references/manual-test-case-format.md b/skills/katalon-create-test-cases/references/manual-test-case-format.md new file mode 100644 index 000000000..9a0600219 --- /dev/null +++ b/skills/katalon-create-test-cases/references/manual-test-case-format.md @@ -0,0 +1,45 @@ +# Manual Test Case Format + +Use this format when generating cases for Katalon True Platform. + +## Fields + +- Name: short verb-led title. +- Description: one or two sentences explaining the behavior under test. +- Pre-condition: environment, account, seed data, cart state, login state, AUT URL, and any browser/device assumptions. +- Test Steps: manual style actions. Start with navigation when page context matters. +- Expected Results: observable outcome for each step. +- Test Data: concrete values used by the step, or `N/A`. +- Priority: P0 critical path, P1 important functional path, P2 secondary/edge path. +- Requirement IDs: internal requirement IDs or source keys when linking is requested. + +## Step Quality Rules + +- Write one user action per step. +- Use visible labels and URLs, not implementation selectors. +- Put validations in Expected Results, not the action. +- Avoid "verify everything looks correct"; specify what must be visible or changed. +- Split long end-to-end flows when setup makes individual failures hard to diagnose. + +## Katalon Import Notes + +- Before creating test cases, search existing coverage with `find_test_cases_by_requirement` when requirement IDs are known and `find_test_cases` by title, requirement key, feature area, and folder. +- Reuse or update matching existing cases instead of creating duplicates. +- Create manual test cases with `create_test_case`. +- Link requirements after creation with `link_requirements_to_test_case`. +- Verify important created cases with `read_test_case`. +- Revise existing cases with one `update_test_case` call containing all updates. + +## Example Shape + +```text +Name: Add selected product variant to cart +Description: Verify that a shopper can select a product variant and add the selected quantity to cart. +Pre-condition: Storefront is available. Cart is empty. + +| Step | Test Step | Expected Result | Test Data | +| 1 | Navigate to product detail page | Product page is displayed with product heading. | https://... | +| 2 | Select color Ultramarine | Ultramarine option is selected. | Ultramarine | +| 3 | Select storage 128 GB | 128 GB option is selected and SKU/stock are displayed. | 128 GB | +| 4 | Click Buy | Product added confirmation is displayed and cart badge updates. | N/A | +``` diff --git a/skills/katalon-create-test-cases/references/requirement-analysis.md b/skills/katalon-create-test-cases/references/requirement-analysis.md new file mode 100644 index 000000000..66c3d1dea --- /dev/null +++ b/skills/katalon-create-test-cases/references/requirement-analysis.md @@ -0,0 +1,33 @@ +# Requirement Analysis Checklist + +Use this checklist before writing tests from requirements. + +## Inputs To Gather + +- Requirement source: Jira key, Azure item, Katalon requirement ID, pasted text, or URL. +- Target AUT/environment. +- Persona or role. +- Business objective. +- Acceptance criteria. +- Known constraints, data, permissions, and dependencies. +- Sprint/release scope if test assets should be organized by iteration. + +## Analysis Output + +Produce: + +- Requirement summary: what must be true for the feature to be accepted. +- Main flow: happy path from user intent to expected outcome. +- Alternate flows: optional paths, branching choices, pagination, sorting, filtering, retries. +- Negative flows: invalid data, missing data, permissions, empty states, out-of-stock or unavailable state. +- Data matrix: values needed for manual tests. +- Risk matrix: high-risk areas that deserve P0/P1 tests. +- Coverage map: requirement or story -> proposed test cases. + +## Katalon Requirement Handling + +- Use `find_requirements` to search synced Jira/Azure requirements. +- Use `read_requirement` for a specific requirement by source key or internal ID. +- Use `find_test_cases_by_requirement` to inspect existing coverage. +- Use `link_requirements_to_test_case` after test cases exist. +- Do not claim that the MCP can create requirements; current available tools only find/read/link synced requirements. diff --git a/skills/katalon-execute-test/SKILL.md b/skills/katalon-execute-test/SKILL.md new file mode 100644 index 000000000..579f16a72 --- /dev/null +++ b/skills/katalon-execute-test/SKILL.md @@ -0,0 +1,90 @@ +--- +name: katalon-execute-test +description: Execute Katalon True Platform/TestOps tests when the input is an existing test case, manual test case list, test suite, suite collection, execution request, or "run with AI" instruction. Use when you need to create a manual test run, start Run with AI, poll AI session results, schedule automated suites, read execution/test results, or summarize pass/fail/blocked outcomes. For full requirement-to-test-design-to-execution workflows, combine with or defer to katalon-true-platform-testing. Written for the manual tester who has cases and needs a result, by hand or through Run with AI. A coded suite driven from a framework starts at katalon-playwright-execute or katalon-upload-report. +--- + +# Katalon Execute Test + +Use this skill for running already-selected Katalon test cases or suites and reporting execution outcomes. Keep the larger `katalon-true-platform-testing` skill as the end-to-end orchestrator; this skill is the focused execution workflow. + +## Availability Boundary + +State the execution boundary before running: + +- Available: read AUT environments, create manual test runs, start Run with AI, poll AI sessions, schedule automated suite runs, and read execution/test results. +- Not directly available: guarantee AI completion, inspect the live AUT UI through Katalon MCP, or run manual test cases through the automated scheduler. +- Use Browser/Playwright only when AUT exploration or visual verification outside Katalon MCP is needed. + +Read `references/capability-boundaries.md` when capability scope is unclear. + +## Resolve Context First + +Before creating or scheduling any run: + +1. Call `list_projects`. +2. Call `list_repositories`. +3. Resolve the repository/Test Project from the user's wording or unique available repository. + +Ask only when multiple equally valid repositories remain, required credentials are missing, or the next action is destructive. + +## Choose Execution Type + +Use manual execution when: + +- The user provides manual test cases. +- The user asks for Run with AI. +- The test cases were just created in Katalon. +- The input is a manual suite. + +Use automated execution only when the input is an automated suite or suite collection. + +If the user only says "run tests" and the type cannot be inferred, ask whether they want manual or automated execution. + +## Manual Run With AI + +Always follow this order: + +1. Call `read_auts` immediately before `create_manual_test_run`. +2. Resolve the manual cases or suites. +3. If a suite is provided, call `read_test_suite` before creating the run. +4. If individual cases are provided, call `read_test_case` for unclear or risky inputs. +5. Create the manual run with `create_manual_test_run`. +6. Start Run with AI automatically with `create_manual_ai_session` unless the user explicitly says not to run AI. +7. Poll `read_manual_ai_session` until all items leave TODO/IN_TESTING, or the platform returns an external timeout/error. +8. Read available execution/test result details before responding. + +AUT rules: + +- Never reuse AUT environment selection from an earlier turn or earlier run. +- Choose the AUT/environment whose URL or name clearly matches the target AUT. +- If no AUT exists and a URL is known from the user, requirement, or test data, use it as `default_aut_environment_url`. +- Ask only when multiple AUTs are equally plausible or no target URL can be resolved. + +Read `references/execution-workflow.md` before creating manual executions. + +## Automated Execution + +Use automated execution only for automated suites: + +1. Find automated suites or suite collections with `find_test_suites`. +2. Find execution profiles with `find_execution_profiles`. +3. Find TestCloud environments with `list_test_cloud_environments`. +4. Build the run configuration with `build_run_configuration`. +5. Optionally build a schedule with `build_schedule`. +6. Run with `schedule_test_run`. +7. Read execution and test results with `read_execution`, `read_execution_test_results`, and `read_test_result` as needed. + +Do not call `create_manual_test_run` for automated suites, and do not call `schedule_test_run` for individual manual test cases. + +## Report Results + +After execution, report: + +- Run name and link when returned. +- Final status. +- Passed, failed, blocked/incomplete, and not-run counts. +- Failed cases with concise failure reasons. +- Defects created or recommended. +- Gaps, skipped cases, and manual follow-up required. + +Ask before creating ALM-linked defects unless the user explicitly requested defect creation for failures. diff --git a/skills/katalon-execute-test/references/capability-boundaries.md b/skills/katalon-execute-test/references/capability-boundaries.md new file mode 100644 index 000000000..2cb5fcf95 --- /dev/null +++ b/skills/katalon-execute-test/references/capability-boundaries.md @@ -0,0 +1,32 @@ +# Katalon MCP Capability Boundaries + +## Available + +- Project discovery: `list_projects`. +- Repository/Test Project discovery: `list_repositories`. +- Requirement discovery: `find_requirements`, `read_requirement`. +- Requirement coverage: `fetch_requirement_data`. +- Test case operations: `create_test_case`, `read_test_case`, `update_test_case`, `duplicate_test_case`, `delete_test_case`, `move_test_case`, `find_test_cases`. +- Test folder operations: `find_test_folders`, `manage_test_folder`. +- Test suite operations: `find_test_suites`, `read_test_suite`, `manage_test_suite`. +- Requirement links: `link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`. +- Manual execution: `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`. +- Automated execution: `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, `build_schedule`, `schedule_test_run`. +- Execution results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Quality data: requirement, defect, test case, test stability, and configuration coverage fetch tools. +- ALM defects: `find_alm_integration_projects`, `create_defect`. + +## Not Directly Available + +- Create requirements in Katalon True Platform. Requirements are synced from Jira/Azure and can be found/read/linked. +- Create a formal Test Plan entity. Use test suites/folders/executions as the executable planning structure. +- Guarantee Run with AI completion. The platform may block, fail, or require AUT/account state. +- Inspect AUT pages through Katalon MCP. Use Browser/Playwright for website exploration. +- Create defects without a failed test result ID and ALM integration details. + +## Recommended Workarounds + +- Requirement creation: create in Jira/Azure first, then sync/find/link in Katalon. +- Test plan: create a named folder and/or test suite, link to sprint/release, and create execution from that suite. +- AUT exploration: use Browser/Playwright to understand the product, then import manual cases into Katalon. +- AI execution blocked: report blocked state with required fixture, AUT, account, or environment action. diff --git a/skills/katalon-execute-test/references/execution-workflow.md b/skills/katalon-execute-test/references/execution-workflow.md new file mode 100644 index 000000000..65a26beb2 --- /dev/null +++ b/skills/katalon-execute-test/references/execution-workflow.md @@ -0,0 +1,64 @@ +# Execution Workflow + +## Manual Run With AI + +1. Resolve project and repository. +2. Search existing coverage first, then resolve existing test cases or create only missing cases. +3. Add cases to a manual test suite if grouping is needed. +4. Call `read_auts`. +5. Choose the matching AUT/environment automatically when one clearly matches the target AUT. If none exist and a URL is known, use it as `default_aut_environment_url` for AI execution. +6. Call `create_manual_test_run`. +7. Continue to AI automatically after creating any manual execution unless the user explicitly says not to run AI. +8. Call `create_manual_ai_session`. +9. Poll with `read_manual_ai_session` until every test case leaves TODO/IN_TESTING. +10. Summarize result in chat. + +## Manual Run Rules + +- Never call `create_manual_test_run` without a fresh `read_auts` first. +- Never reuse AUT environment choices from earlier turns. +- Start Run with AI without asking again after any manual run is created. +- Do not ask whether to continue with AI unless the user explicitly requests a manual run without AI. +- If the manual run contains newly created test cases, they are manual by default. +- Render returned execution paths as markdown links. +- Wait for AI completion before final response. If the platform stays pending/running for an unusually long time, keep polling at practical intervals and only report in-progress status when the user asks or the platform returns a timeout/error. +- Ask for user input only when required data is missing, multiple AUTs are equally plausible, the user explicitly disables AI, or the next action is destructive. + +## Automated Run + +1. Resolve repository. +2. Find automated test suites or suite collections. +3. Find execution profiles. +4. Select TestCloud environments. +5. Build run configuration. +6. Optionally build schedule. +7. Call `schedule_test_run`. +8. Read execution and results. + +## Automated Run Rules + +- Use `schedule_test_run` only for automated suites. +- Do not run manual test cases through automated scheduling. +- For mobile native, ensure app details are present. +- For mobile availability filters, clarify automation/manual vs live testing when needed. + +## Result Reporting Template + +```text +Run: +- Name: +- Link: +- Status: + +Summary: +- Passed: +- Failed: +- Blocked/Incomplete: +- Not run: + +Findings: +- ... + +Next actions: +- ... +``` diff --git a/skills/katalon-exploratory-charter/SKILL.md b/skills/katalon-exploratory-charter/SKILL.md new file mode 100644 index 000000000..b59f2233e --- /dev/null +++ b/skills/katalon-exploratory-charter/SKILL.md @@ -0,0 +1,177 @@ +--- +name: katalon-exploratory-charter +description: Write, run, and debrief exploratory testing charters against Katalon True Platform/TestOps when there is no script to follow. Use when you need to turn a vague area into a charter (mission, areas, oracles, timebox), run a timeboxed unscripted session, log what you find as session notes, judge which findings are real defects, and promote what was learned into atomic manual cases plus the folder or suite that holds them. This is unscripted discovery and the debrief after it; when a written requirement is already in hand and the job is designing cases from it, use katalon-create-test-cases instead. Written for the manual tester who has an hour, a feature, and no cases to run against it. +--- + +# Katalon Exploratory Charter + +Use this skill for the **unscripted** part of the design stage: a timeboxed session that learns something the written requirement does not say, then converts what it learned into scripted coverage. A charter is not a test case and a session is not a test run. The value is the *learning plus the promotion*, so a session that produces no notes and no promoted cases has failed even if nothing crashed. + +Treat the human as a tool. Ask when the area, the timebox, the environment, or the risk appetite cannot be resolved safely. Never invent an oracle. + +## Availability Boundary + +**There is no exploratory session object in the platform.** Say that before promising to "run a session in Katalon". + +- **Available via MCP:** resolve scope (`list_projects`, `list_repositories`), read the oracles you already have (`find_requirements`, `read_requirement`, `fetch_requirement_data`), check whether an area is already covered before spending the timebox on it (`find_test_cases`, `find_test_cases_by_requirement`), and land the outcome (`manage_test_folder`, `create_test_case`, `link_requirements_to_test_case`, `manage_test_suite`, `read_test_suite`). +- **Not available, and no workaround exists:** a charter / session / session-sheet entity, a session timer or duration field, attachment upload for screenshots or video or HAR files, a tool that sets a manual test result to failed, custom fields or tags to mark a case as exploratory in origin, and creating a requirement for behavior the charter discovered. Do not reach for `create_charter`, `upload_attachment`, `set_test_result_status`, or `manage_tags` - none of them exists. +- **The defect trap.** `create_defect` requires a known **failed test result ID**. An unscripted session produces no test result, so a charter finding cannot be filed through MCP as it stands. In order of preference: (1) file it in Jira or Azure DevOps directly, outside the MCP; (2) execute the reproduction case and mark it failed in the TestOps UI, then file from that run. A third path is MCP-legal but conditional - promote the finding to a case, `create_manual_test_run`, `create_manual_ai_session`, and if the AI run genuinely fails, `create_defect` with that result ID. It only works when Run with AI actually reproduces the bug, so never offer it as the default. +- **The Katalon MCP cannot see the application.** It reads and writes platform objects only. Every observation in a session comes from a browser you drive yourself, so use Browser or Playwright for the product and the MCP for the artifacts. +- **Where the charter lives.** Author the charter and the session sheet as a local artifact the user keeps. Optionally mirror the charter into the platform as one test case in the charter folder, mission in Description and session log in Steps. Say which you did. + +Read `references/charter-and-session.md` before writing a charter. Consult `katalon-true-platform-testing/references/unavailable-capabilities.md` when the user asks whether Katalon can run exploratory sessions for them. + +## Session Workflow + +```text ++------------------+ +------------------+ +------------------+ +| Frame the area | --> | Write charter | --> | Run the timebox | +| scope + coverage | | mission/oracles | | notes as you go | ++------------------+ +------------------+ +------------------+ + | + v ++------------------+ +------------------+ +------------------+ +| Promote to cases | <-- | Debrief | <-- | Classify findings| +| hand to authoring| | timebox account | | oracle broken | ++------------------+ +------------------+ +------------------+ +``` + +## Steps and tool rules + +1. **Resolve scope and existing coverage.** `list_projects` -> `list_repositories`. Then `find_test_cases` on the area keywords and `find_test_cases_by_requirement` when a requirement key is known. **Coverage is an input to the charter, not an afterthought**: an area that already has twenty cases is a poor use of the timebox, and the gaps are where the charter should point. +2. **Write the charter before touching the application.** Six fields, all required: mission, areas, oracles, timebox, out of scope, done when. Show it to the user and get one confirmation. A charter without oracles is a browsing session; refuse to start until at least one oracle is named. +3. **Choose the tours that fit the mission.** Pick two or three from `references/oracles-and-tours.md`. Name them in the charter so the session is reproducible by another tester. +4. **Run the timebox and take notes as you go, not afterwards.** Every note is timestamped and typed - `SETUP`, `OBS`, `Q`, `BUG`, `RISK`, `IDEA`. A `BUG` note is worthless without the steps that produced it, so write the repro before moving on. Use Browser or Playwright for the application itself; the Katalon MCP cannot see the AUT UI. +5. **Stop when the timebox ends, not when you run out of ideas.** Record the timebox accounting - minutes on charter, minutes on bug investigation and writeup, minutes on setup. A session that spent 40 of 60 minutes on setup is a finding about the environment; report it as one. +6. **Classify every finding against the oracle it broke.** Product defect, katalon-test-data or environment issue, question for the product owner, or design risk. A finding with no broken oracle is an observation, not a defect - say so plainly rather than inflating it. +7. **Debrief.** Report what the mission was, what was covered, what was not reached, the findings by class, the timebox accounting, and the charter you would run next. The next charter is part of the deliverable. +8. **Promote, then hand off.** Turn confirmed findings into a promotion list - title, priority, requirement link if one exists, and the one-line expected result. `manage_test_folder` to create or reuse the charter folder, then **hand authoring to `katalon-create-test-cases`**, which owns duplicate checking, the atomic-case rules, and import. Do not reimplement those rules here. Add the promoted cases to the relevant suite with `manage_test_suite` and confirm with `read_test_suite`. + +## Hard rules + +- **No oracle, no charter.** At least one named, checkable oracle before the timebox starts. *It feels wrong* is not an oracle. +- **The timebox ends the session.** Not the idea supply. An overrun charter is two charters that were never split. +- **A `BUG` note carries its repro or it is an `OBS`.** Write the reproduction before moving to the next observation, while the state is still on screen. +- **A finding with no broken oracle is not a defect.** Report it as a question or a risk. Inflating observations into defects is how a session loses its credibility. +- **Never explore production without explicit confirmation.** Confirm the environment from `read_auts` or from the user before the first action. +- **A session with no promoted cases and no notes has failed.** Nothing crashing is not a result. + +## Naming and platform constraints + +- Test case and folder **names** accept only letters, numbers, spaces, and `( ) . , _ -`. Charter titles love colons, and a colon will be rejected. Use `CH-01 Explore guest checkout with expired card`, never `CH-01: Explore ...`. Folder paths may use `/`. +- There is no tag or custom field via MCP, so the only durable marker of exploratory origin is the naming convention plus the folder. Keep both: folder `Exploratory / `, and prefix promoted case titles with the charter id. +- Prefer `update_test_case` over delete-and-recreate when a later session revises a promoted case. Deletion can fail server-side and updating in place keeps IDs, links, and history. + +## Worked example + +**User asks:** *We shipped guest checkout last week with no cases for it. Run an hour of exploratory testing on the payment step in Cellphone Shop and tell me what to write up.* + +**Step 1 - scope and coverage.** + +```text +list_projects -> Cellphone Shop (id 4021), Demo Bank (id 3980) +list_repositories -> Katalon Cloud (id 118) [exactly one, so use it] +find_requirements "CEL" -> CEL-6 Guest checkout (synced from Jira) +find_test_cases_by_requirement CEL-6 + -> 4 cases, all on the cart step, none on payment +``` + +Coverage gap confirmed, so the payment step is worth the timebox. + +**Step 2 - the charter, confirmed with the user before starting.** + +```text +CHARTER CH-01 +Mission Explore guest checkout payment with declined, expired, and + mistyped cards, to discover error-handling and cart-state defects +Areas Payment form, order confirmation, cart persistence, email receipt +Oracles CEL-6 acceptance lines 3 and 4 (an unsuccessful payment keeps + the cart intact and shows a retry message) + Consistency with the logged-in checkout flow, which is covered + Consistency with purpose (no order record means no charge) + The payment provider's published test-card behavior +Tours Error-message tour, then money tour +Timebox 60 minutes, one tester, staging +Out of scope Saved cards, subscriptions, refunds, mobile web +Done when Every declined-card path has a recorded outcome and every + finding is classified +``` + +**Step 4 - session sheet excerpt.** + +```text +00:00 SETUP Staging reachable, cart seeded with 1 phone, provider test keys present +00:07 OBS CVV field accepts 5 digits without complaint, submit still enabled +00:09 BUG-1 Expired card 01/24 -> spinner 41s -> blank page. Back button shows an + EMPTY cart. No order in Orders. No charge on the provider dashboard. + Repro 3/3. Oracle broken - CEL-6 line 3, cart must survive a decline +00:24 Q-1 Is a 5-digit CVV meant to be accepted, or is the provider silently + truncating? No acceptance line covers CVV length +00:31 BUG-2 Declined card shows the raw provider string "card_declined" instead + of a user message. Oracle broken - CEL-6 line 4 +00:38 RISK-1 Retrying after a decline re-submits the same idempotency key. Could + double-charge under a slow network. Not reproduced in the timebox +00:52 OBS Receipt email never arrives for guest orders. Out of scope, logged +01:00 STOP +``` + +**Step 5 - timebox accounting.** 44 minutes on charter, 11 on bug investigation and writeup, 5 on setup. + +**Step 6 - findings.** + +| ID | Class | Oracle broken | Severity | Promote | +|---|---|---|---|---| +| BUG-1 | product defect | CEL-6 line 3 | P0 | yes, 2 cases | +| BUG-2 | product defect | CEL-6 line 4 | P1 | yes, 1 case | +| Q-1 | question for the product owner | none, the rule does not exist | n/a | no, ask first | +| RISK-1 | design risk, unreproduced | none yet | n/a | no, next charter | + +**Step 8 - promotion list handed to `katalon-create-test-cases`.** + +```text +CH-01 Guest checkout keeps the cart after an expired card is declined P0 CEL-6 +CH-01 Guest checkout shows a retry message after a decline P0 CEL-6 +CH-01 Guest checkout shows a readable message not a provider code P1 CEL-6 +``` + +```text +find_test_cases "checkout expired card" -> 0 matches [dedupe before create] +manage_test_folder create "Exploratory / CH-01 Guest checkout payment" + -> folder id 9134 +-> hand to katalon-create-test-cases for create_test_case x3 + link_requirements_to_test_case +manage_test_suite add the 3 cases to "CEL-6 Regression" +read_test_suite -> 7 cases, 3 new [confirmed] +``` + +**Expected report back to the user.** Name every one of these; a report missing the boundary line is incomplete: + +```text +Charter CH-01, guest checkout payment, 60 minutes on staging. Timebox: 44 charter / 11 writeup / 5 setup. +Findings: 2 product defects (BUG-1 P0 cart lost on decline, BUG-2 P1 raw provider string), + 1 open question (CVV length, no acceptance line covers it), 1 unreproduced risk (duplicate idempotency key). +Promoted: 3 cases under folder 9134 "Exploratory / CH-01 Guest checkout payment", added to the CEL-6 + Regression suite. read_test_suite confirms 7 cases, 3 new. +Not reached: email receipt path, out of scope by the charter. +Next charter: CH-02, retry and idempotency under a slow network, 45 minutes. +Boundary: the two defects were NOT filed through MCP. create_defect needs a failed test result ID and this + session produced none. File them in Jira, or run the three promoted cases and file from the failed run. +``` + +## Prompt recipes + +- `Run a 60 minute exploratory charter on guest checkout payment in Cellphone Shop and give me the findings.` +- `Write a charter for the new search filters. I have no requirement, just the feature.` +- `Debrief my session notes below and tell me which findings should become test cases.` +- `We have an hour before the release. Which area is worth exploring given current coverage?` +- `Which of these findings are real defects and which are just observations?` + +## Hand-offs + +- Promoted cases to author, dedupe, and import -> `katalon-create-test-cases`. +- The data a promoted case needs before anyone can run it -> `katalon-test-data`. +- Running the promoted cases, by hand or with AI -> `katalon-execute-test`. +- **A red run, not a session.** If the user already has failed test results, this is not the skill - `katalon-analyze-failures` owns failures with a result ID. This skill owns findings that have no result ID at all. +- A charter programme across a sprint or release -> `katalon-test-plan`. +- Coverage gaps the session exposed across the repository -> `katalon-test-management`. + +Read `references/charter-and-session.md` before writing a charter, and `references/oracles-and-tours.md` before choosing how to explore. diff --git a/skills/katalon-exploratory-charter/references/charter-and-session.md b/skills/katalon-exploratory-charter/references/charter-and-session.md new file mode 100644 index 000000000..5249657ba --- /dev/null +++ b/skills/katalon-exploratory-charter/references/charter-and-session.md @@ -0,0 +1,116 @@ +# Charter and Session + +## The six-field charter template + +Copy this verbatim. All six fields are required. The hard rule first: **no oracle, no charter.** + +```text +CHARTER +Mission +Areas +Oracles +Tours +Timebox <45, 60, or 90 minutes, how many testers, which environment> +Out of scope +Done when +``` + +| Field | The one rule | +|---|---| +| Mission | A target plus an information goal. If it does not say what you want to *learn*, it is not a mission. | +| Areas | Named surfaces, not the whole feature. Four is a session, twelve is a release. | +| Oracles | Each one must be checkable by a second person. Point at an acceptance line, a covered flow, a published behavior, or a stated purpose. | +| Tours | Named in advance so another tester can reproduce the session, not just the bugs. | +| Timebox | Includes the environment. A charter that does not say where it runs will be run in the wrong place. | +| Out of scope | Written down so a tempting detour becomes a note instead of a lost 20 minutes. | +| Done when | An observable condition. "When I have looked at it enough" is not one. | + +## Good versus bad missions + +The difference is always the same: a target plus an information goal. + +| Bad | Good | What changed | +|---|---|---| +| Test checkout. | Explore guest checkout payment with declined, expired, and mistyped cards, to discover error-handling and cart-state defects. | Named the surface, named the inputs, named what you want to learn. | +| Have a look at the new search. | Explore search filters with empty, single, and conflicting selections, to discover result-count and reset defects. | Replaced "have a look" with the conditions that could break it. | +| Check the mobile layout works. | Explore the order history screen at 320px and on a slow network, to discover truncation, overflow, and loading-state defects. | Gave the vague quality bar concrete conditions and an information goal. | + +## The session sheet + +One line per note. Timestamped, typed, written as you go and never reconstructed afterwards. + +| Type | Means | Rule | +|---|---|---| +| `SETUP` | Getting the environment ready | Counts against the timebox. Log it honestly. | +| `OBS` | Something observed, no rule broken | The default. Most notes are this. | +| `Q` | A question no oracle answers | Names the missing rule, and who should answer. | +| `BUG` | An oracle was broken | **Carries its repro, or it is an `OBS`.** | +| `RISK` | A plausible failure not reproduced in the timebox | Says what would confirm it. | +| `IDEA` | A charter worth running later | Feeds the next-charter field of the debrief. | + +Rules that make the sheet worth keeping: + +- **Timestamp every line** as `MM:SS` from the session start. The timestamps are what produce the accounting below. +- **Write the repro before moving on.** The state is on screen now and gone in 90 seconds. Steps, the data used, and how many times out of how many it reproduced. +- **Name the broken oracle inside the `BUG` note**, not later. A `BUG` that cannot name its oracle is an `OBS` that got excited. +- **Log out-of-scope discoveries as `OBS` and keep going.** That is what the out-of-scope field is for. +- **Evidence capture is on the tester.** The MCP has no attachment upload, so screenshots and recordings go in the ALM ticket, not the platform. + +## Timebox accounting + +Three buckets, reported every time: + +| Bucket | What counts | +|---|---| +| On charter | Actually exploring the mission | +| Bug investigation and writeup | Reproducing, narrowing, and writing findings | +| Setup | Environment, accounts, data, tooling | + +Read the shape, do not just report the numbers: + +- **Setup over 25 percent is an environment finding.** Report it as one. The next tester will pay the same tax. +- **Bug writeup over 50 percent means the session found one thing** and should have been split into an investigation and a fresh charter. +- **On charter under 50 percent means the session did not really happen.** Say so rather than presenting the survivors as a full sweep. + +## The debrief checklist + +Six questions. Each has a weak answer to avoid. + +| Question | Weak answer looks like | +|---|---| +| What was the mission? | Restating the feature name instead of the information goal. | +| What was covered? | "Checkout" instead of the specific paths actually walked. | +| What was not reached? | Silence. Unreached areas are the most useful line in the debrief. | +| What was found, by class? | An undifferentiated bug list with no oracle named per item. | +| Where did the time go? | No accounting, or accounting with no reading of its shape. | +| What charter would you run next? | "More of the same." The next charter is part of the deliverable. | + +## The promotion rubric + +A finding becomes a scripted case when **all three** hold: + +1. It is **reproducible**, with steps that work from a stated starting state. +2. It maps to a **rule someone will defend**, an oracle a product owner would stand behind. +3. It **would not be caught by existing coverage**, checked with `find_test_cases` before proposing it. + +Otherwise it stays where it is: + +- **No oracle covers it** -> it stays a question. Ask the product owner; do not invent the rule and write a case that enforces your invention. +- **Not reproducible inside the timebox** -> it stays a risk. Propose it as the next charter with what would confirm it. +- **Already covered** -> it is a duplicate. Say so and move on. + +**The handoff contract to `katalon-create-test-cases`** is four fields per promoted finding and nothing else: + +```text + +Expected: +``` + +`katalon-create-test-cases` owns duplicate checking, the atomic-case rules, and import. Do not pre-write steps here and do not reimplement those rules. + +## Charter sizing + +- **45 minutes** for a narrow surface with a known oracle. +- **60 minutes** is the default and the one to reach for when unsure. +- **90 minutes** only with a second tester or a heavy setup cost, and only when the mission genuinely cannot be split. +- **A charter that needs more than 90 minutes is two charters.** Split it on the mission, not on the clock. diff --git a/skills/katalon-exploratory-charter/references/oracles-and-tours.md b/skills/katalon-exploratory-charter/references/oracles-and-tours.md new file mode 100644 index 000000000..15d983932 --- /dev/null +++ b/skills/katalon-exploratory-charter/references/oracles-and-tours.md @@ -0,0 +1,75 @@ +# Oracles and Tours + +An oracle is how you decide something is wrong. A tour is how you decide where to look. The charter needs both: oracles make findings defensible, tours make the timebox productive. + +## The oracle taxonomy + +Seven kinds. Name which one you are using in the charter, because that is what a finding gets judged against later. + +| Oracle | The question it asks | Katalon-flavoured example | +|---|---|---| +| **Consistency within the product** | Do two parts of the product disagree? | The logged-in checkout keeps the cart after a decline; the guest checkout empties it. | +| **Consistency with history** | Did this behave differently before? | Order search returned partial matches last release and now requires an exact ID. | +| **Consistency with a comparable product** | Does a peer product set the expectation? | Every storefront in the market keeps the cart on a payment failure. | +| **Consistency with claims** | Does a written statement say otherwise? | `read_requirement` on CEL-6 line 3 states the cart survives an unsuccessful payment. | +| **Consistency with user expectations** | Would a reasonable user be surprised? | A 41 second spinner with no progress indicator on the payment step. | +| **Consistency with purpose** | Does it defeat what the feature is for? | No order record exists but the provider dashboard shows a charge. | +| **Consistency with standards or statutes** | Is there a rule outside the product? | The payment provider's published test-card behavior, or a WCAG contrast minimum. | + +The strongest findings break **two or more** at once. BUG-1 in the skill's worked example breaks claims (CEL-6 line 3), within-product consistency (the logged-in flow), and purpose (cart state lost) together, which is why it is a P0 and not an argument. + +## Naming an oracle so it is checkable + +The test: **could a second person verify the oracle without you in the room?** + +| Not an oracle | An oracle | +|---|---| +| It feels wrong. | The logged-in flow keeps the cart after a decline and the guest flow does not. | +| This is bad UX. | The error shows the raw provider string `card_declined` rather than a user message, against CEL-6 line 4. | +| It seems slow. | The payment step takes 41 seconds with no progress indicator, against the 5 second budget in the requirement. | +| Users will hate this. | Every competitor keeps the cart on failure, and support ticket CS-882 says ours does not. | + +Point at one of four things, always: + +1. **An acceptance line** you can quote, from `read_requirement`. +2. **A covered flow** you can name, from `find_test_cases`. +3. **A published behavior** you can link, such as provider docs or a standard. +4. **A stated purpose** the feature would defeat. + +If you cannot point at any of the four, you have a `Q`, not a `BUG`. Log it and ask. + +## Tours + +A tour is a lens that constrains where you look. Pick two or three, never all of them. + +| Tour | Walk it by | It pays when | +|---|---|---| +| **Feature tour** | Visiting every capability once, shallowly | The feature is new and nobody has mapped it | +| **Money tour** | Following the paths that take or move money | Checkout, billing, subscriptions, refunds | +| **Landmark tour** | Hopping between the key screens in varied orders | Navigation and state are suspect | +| **Data tour** | Feeding empty, huge, unicode, and malformed values | Forms, imports, search | +| **Configuration tour** | Changing settings, then re-walking a flow | Feature flags, roles, tenant settings | +| **Error-message tour** | Deliberately provoking every failure path | Validation and error handling are the risk | +| **Back-button tour** | Browser back, forward, refresh, and deep links at every step | Multi-step flows and wizards | +| **Interruption tour** | Killing the network, the tab, or the session mid-flow | Long transactions, payments, uploads | +| **Anti-social tour** | Doing the least reasonable legal thing at each step | The happy path is already covered | + +## Choosing tours from the mission + +| Mission shape | Tours that pay | +|---|---| +| A new feature nobody has tested | Feature tour, then data tour | +| A rewrite of something that worked | Consistency-driven landmark tour, then back-button tour | +| A cluster of related bugs | Error-message tour, then interruption tour | +| A pre-release sweep with an hour | Money tour, then feature tour | +| A payment or checkout surface | Error-message tour, then money tour, then interruption tour | +| A form-heavy or search surface | Data tour, then anti-social tour | +| A permissions or multi-tenant change | Configuration tour, then landmark tour | + +Two or three. A charter naming six tours has not chosen; it has listed. + +## The boundary, restated + +- **The Katalon MCP cannot see the application.** Drive the product with Browser or Playwright. The MCP reads requirements and writes cases, folders, and suites, and that is all it does here. +- **Evidence capture is on the tester.** There is no attachment upload tool, so screenshots, recordings, and HAR files belong in the ALM ticket, not in the platform. Reference them from the finding by ticket ID. +- **There is no exploratory session object**, so the charter and the session sheet are local artifacts unless you deliberately mirror the charter into a test case. Say which you did. diff --git a/skills/katalon-platform-setup/SKILL.md b/skills/katalon-platform-setup/SKILL.md new file mode 100644 index 000000000..94f5a7300 --- /dev/null +++ b/skills/katalon-platform-setup/SKILL.md @@ -0,0 +1,160 @@ +--- +name: katalon-platform-setup +description: Set up, verify, and troubleshoot Katalon True Platform/TestOps MCP connectivity for any AI coding agent. Use when you need to install or configure the Katalon MCP server, connect to a Katalon platform account, verify available Katalon MCP tools, diagnose missing authentication or repository access, confirm project/repository discovery, or explain what platform operations are and are not exposed through the MCP. One-time cross-cutting setup, run once by whoever wires the agent to the platform, before any lifecycle skill can work. +--- + +# Katalon Platform Setup + +Use this skill for Katalon MCP installation, connection, and readiness checks. Do not use it for designing or running tests unless the setup check leads directly into `katalon-create-test-cases`, `katalon-execute-test`, or the larger `katalon-true-platform-testing` skill. + +This skill is agent-neutral. The Katalon MCP server is the same for every agent (Claude Code, Codex, Copilot, Cursor, Kiro, Windsurf, and others); only the config file location and format differ per agent. For the exact per-agent config path and snippet, read the repository `README.md` install section. + +## Setup Boundary + +Be explicit about what can be verified from the current environment: + +- Available through Katalon MCP once connected: project/repository discovery, requirement reads, test case/suite operations, manual runs, AI sessions, automated scheduling, results, metrics, and ALM-linked defects. +- Not directly available through Katalon MCP: create requirements, create a formal Test Plan entity, inspect live AUT pages, or guarantee AI execution completion. +- If no Katalon MCP tools are exposed in the current session, do not claim the platform is connected. + +Read `references/capability-boundaries.md` when explaining available operations. + +## Required Setup Inputs + +For setup requests, collect the minimum routing and authentication choices before writing or testing MCP config: + +1. Resolve the Katalon MCP endpoint: + - Default to the canonical endpoint `https://platform.katalon.io/mcp`. It serves every workspace and presents a picker after sign-in, so do not ask for a subdomain unless something below applies. + - If the installed config still contains the `` placeholder, replace it with the canonical endpoint rather than asking the user to fill it in. + - Only ask for a domain when the user names a dedicated Katalon domain, or when the local config mentions more than one Katalon domain - then list the candidates and ask which one they want. + - Normalize a bare subdomain to `https://.katalon.io/mcp`. +2. Ask how the user wants to authenticate when auth is required: + - Browser OAuth flow with an agent restart/reload after login. + - CLI/local `mcp-remote` login bootstrap. + - Login email/account label only for account selection. +3. Never ask for passwords, API tokens, cookies, JWTs, MFA codes, or OAuth callback URLs in chat. If the user offers "login information", accept only non-secret routing details such as login email, organization, domain, project name, or repository/Test Project name. +4. After authentication succeeds and Katalon tools are available, call `list_projects`. If more than one project is returned, list the projects and ask which project to use before calling repository-specific tools. +5. After project selection, call `list_repositories` with the selected `project_id`. If more than one repository/Test Project is returned and none is clearly implied by the user's request, list the repositories and ask which one to use. + +## Connection Check + +Start every setup/debugging request with a non-destructive check: + +1. Follow "Required Setup Inputs" first when the endpoint, domain, or auth route is not already resolved. +2. Search for Katalon/TestOps MCP tools using the agent's tool discovery mechanism. +3. If tools exist, call `list_projects`. +4. If multiple projects are returned, list them and ask which project to use. +5. Call `list_repositories` with the selected or only `project_id` after projects are available. +6. If multiple repositories/Test Projects are returned and no repository is clearly implied, list them and ask which repository/Test Project to use. +7. Confirm at least one project and repository/Test Project can be discovered. +8. If a requirement key or repo name is provided, verify access by searching or reading that target. + +If Katalon tools are not exposed in the active agent session, do not stop at "reload required" yet. First try the local proxy bootstrap in "MCP Remote OAuth Bootstrap" below. A successful bootstrap proves the account and endpoint are usable even if the current agent's MCP host has not reloaded the server list. + +Report the exact boundary: + +- Connected and usable. +- MCP tools exposed but auth/access failed. +- MCP works through local `mcp-remote` proxy, but tools are not exposed in the current agent session. +- MCP tools missing from the session and local proxy bootstrap failed or could not be attempted. +- Project exists but repository/Test Project cannot be resolved. + +## Install Or Configure MCP + +When MCP tools are missing: + +1. Inspect local agent/plugin context for an existing Katalon MCP install path or configuration. +2. If the config still contains a `` placeholder, replace it with `https://platform.katalon.io/mcp`. Never leave a placeholder in an active user config. +3. Prefer a native remote MCP entry. Agents that speak remote MCP directly (Claude Code, Copilot/VS Code, Copilot CLI, Cursor) need only the endpoint and run the browser OAuth flow themselves: + + ```json + { "mcpServers": { "katalon-prod-mcp": { "type": "http", "url": "https://platform.katalon.io/mcp" } } } + ``` + + See the bundled `.mcp.json` for the canonical shape. +4. Fall back to the `mcp-remote` wrapper only for agents without native remote-MCP support (Kiro, Continue, Codex): + + ```sh + npx -y mcp-remote https://platform.katalon.io/mcp --transport http-first + ``` + + - Codex declares the same command/args in TOML under `[mcp_servers.katalon-prod-mcp]`. + - For the exact file path per agent, read the repository `README.md` install section. +5. If the install command, package name, or server URL is not present in local context, use official Katalon-provided setup instructions or ask the user for the MCP package/source. Do not invent install commands. +6. Configure only the minimum required MCP server entry for the user's agent environment. +7. Store secrets using the environment's secret mechanism or environment variables. Never write access tokens, passwords, cookies, or raw auth callback URLs into repo files, skill files, logs, or final answers. +8. Re-run the connection check. Restart or reload the MCP host only if Katalon works through the local proxy but the current agent session still does not expose the tools. + +If the user explicitly asks to install but the environment does not permit plugin/MCP installation from the current session, give the exact missing prerequisite and the safest next step. + +## MCP Remote OAuth Bootstrap + +Use this when Katalon MCP tools are missing from the active session or direct remote MCP config returns `401 Invalid JWT token format`. + +1. Ask only for non-secret information needed to target the right server or verify access, such as: + - Katalon MCP URL, only if the user is on a dedicated domain rather than the canonical `https://platform.katalon.io/mcp`. + - Project name or project ID. + - Repository/Test Project name, if the user wants a specific target verified. + - Login email, only if it helps the user choose the right account in the browser. +2. Do not ask the user to paste passwords, API tokens, cookies, JWTs, or OAuth callback URLs into chat. If authentication is needed, trigger the browser-based OAuth flow and have the user complete login in the browser. +3. Run the proxy with the Katalon endpoint: + + ```sh + npx -y mcp-remote "https://platform.katalon.io/mcp" --transport http-first + ``` + + If this prints an authorization URL or opens the browser, wait for the user/browser callback to complete. `mcp-remote` stores OAuth state under its own auth cache, such as `~/.mcp-auth`, not in the workspace. + +4. Verify through a local MCP client, if Katalon tools are still not exposed through the agent's tool discovery: + - `listTools` should include `list_projects` and `list_repositories`. + - Call `list_projects`. + - Call `list_repositories` with the exact schema field `project_id`. +5. If local proxy verification succeeds but the agent's tool discovery still does not expose Katalon tools, report that MCP is authenticated and usable through `mcp-remote`, but the active agent host needs a reload/new session/new thread to expose the tools natively. + +Known working MCP server entries. Prefer the native form; use the `mcp-remote` form only where the agent cannot reach a remote server directly (Codex uses the same command/args in TOML): + +```json +{ + "mcpServers": { + "katalon-prod-mcp": { "type": "http", "url": "https://platform.katalon.io/mcp" } + } +} +``` + +```json +{ + "mcpServers": { + "katalon-prod-mcp": { + "command": "npx", + "args": ["-y", "mcp-remote", "https://platform.katalon.io/mcp", "--transport", "http-first"] + } + } +} +``` + +Known verification observations: + +- `mcp-remote` may say `Invalid JWT token format` before OAuth completes; treat this as an auth challenge, not proof the server is unavailable. +- The valid transport strategy is `http-first` or `http-only`; `http` can be rejected by `mcp-remote`. +- `list_repositories` expects `project_id`, not `projectId`. + +## Authentication And Access + +When tools exist but calls fail: + +- Treat 401/403-style failures as auth or account-scope problems. +- Ask the user to authenticate through the platform's secure browser flow. Never ask them to paste secrets into normal chat when a secure mechanism exists. +- You may ask for non-secret account routing details, such as subdomain, project ID/name, repository name, and login email. The user should enter passwords/SSO/MFA only in the browser or secure credential flow. +- If multiple authenticated Katalon domains are available, ask which domain to use before selecting projects. +- If multiple projects are available, list the project names/IDs and ask which project to use before listing repositories. +- If multiple repositories/Test Projects are available, list them and ask which repository/Test Project to use unless the user's request clearly names one. +- For empty project/repository results, verify account membership, organization, project role, and repository/Test Project access. +- For requirement lookup failures, verify that Jira/Azure sync exists and the requirement key belongs to the connected project. + +## Handoff + +After setup is verified: + +- Use `katalon-create-test-cases` for requirement or free-text test design/import. +- Use `katalon-execute-test` for running an existing case or suite. +- Use `katalon-true-platform-testing` for end-to-end flows that include requirement analysis, case creation, suite management, execution, AI polling, and result reporting. diff --git a/skills/katalon-platform-setup/references/capability-boundaries.md b/skills/katalon-platform-setup/references/capability-boundaries.md new file mode 100644 index 000000000..2cb5fcf95 --- /dev/null +++ b/skills/katalon-platform-setup/references/capability-boundaries.md @@ -0,0 +1,32 @@ +# Katalon MCP Capability Boundaries + +## Available + +- Project discovery: `list_projects`. +- Repository/Test Project discovery: `list_repositories`. +- Requirement discovery: `find_requirements`, `read_requirement`. +- Requirement coverage: `fetch_requirement_data`. +- Test case operations: `create_test_case`, `read_test_case`, `update_test_case`, `duplicate_test_case`, `delete_test_case`, `move_test_case`, `find_test_cases`. +- Test folder operations: `find_test_folders`, `manage_test_folder`. +- Test suite operations: `find_test_suites`, `read_test_suite`, `manage_test_suite`. +- Requirement links: `link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`. +- Manual execution: `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`. +- Automated execution: `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, `build_schedule`, `schedule_test_run`. +- Execution results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Quality data: requirement, defect, test case, test stability, and configuration coverage fetch tools. +- ALM defects: `find_alm_integration_projects`, `create_defect`. + +## Not Directly Available + +- Create requirements in Katalon True Platform. Requirements are synced from Jira/Azure and can be found/read/linked. +- Create a formal Test Plan entity. Use test suites/folders/executions as the executable planning structure. +- Guarantee Run with AI completion. The platform may block, fail, or require AUT/account state. +- Inspect AUT pages through Katalon MCP. Use Browser/Playwright for website exploration. +- Create defects without a failed test result ID and ALM integration details. + +## Recommended Workarounds + +- Requirement creation: create in Jira/Azure first, then sync/find/link in Katalon. +- Test plan: create a named folder and/or test suite, link to sprint/release, and create execution from that suite. +- AUT exploration: use Browser/Playwright to understand the product, then import manual cases into Katalon. +- AI execution blocked: report blocked state with required fixture, AUT, account, or environment action. diff --git a/skills/katalon-playwright-execute/SKILL.md b/skills/katalon-playwright-execute/SKILL.md new file mode 100644 index 000000000..e7dcd8f35 --- /dev/null +++ b/skills/katalon-playwright-execute/SKILL.md @@ -0,0 +1,99 @@ +--- +name: katalon-playwright-execute +description: Run Playwright tests or suites and upload the resulting report to Katalon True Platform. Use when you need to execute Playwright scripts, package scripts, spec files, projects, or suites, configure or verify @katalon/playwright-reporter, upload Playwright reports with Katalon CLI/reporter commands, and verify uploaded Katalon Platform runs or reports. Written for the automation tester who wants the suite driven and the results landed in the platform in one step. For a report from a run that has already finished, or from any other framework, use katalon-upload-report. +--- + +# Katalon Playwright Execute + +Use this skill when the user asks to execute a Playwright script/suite and send results to Katalon True Platform. The workflow is not complete until local execution finishes, report artifacts are collected, and upload is either verified or clearly blocked. + +## Execution Workflow + +```text ++------------------+ --> +------------------+ --> +------------------+ +| Resolve target | | Preflight repo | | Configure report | ++------------------+ +------------------+ +------------------+ + | + v ++------------------+ <-- +------------------+ <-- +------------------+ +| Verify Platform | | Upload report | | Run Playwright | ++------------------+ +------------------+ +------------------+ +``` + +## Required Context + +Resolve or ask for only missing required values: + +- Playwright target: spec file, grep, project, package script, suite command, or user-provided command. +- Repository/root directory that contains `package.json` and Playwright config. +- Katalon Platform project ID. Use Katalon MCP `list_projects` when available. +- Katalon API key from a secure source such as `KATALON_API_KEY`; check whether it is already available before asking. +- Platform base URL only when non-default or when the project uses QA/prod custom domains. +- Whether the agent may modify Playwright config/package files if the Katalon reporter is missing. + +Treat the human as a tool: ask concise questions when target, credentials, project, or permission cannot be resolved safely. Never ask the user to paste secrets into normal chat when a secure secret mechanism or environment variable can be used. + +## Preflight + +Before running: + +- Inspect the repository for `package.json`, `playwright.config.*`, `@playwright/test`, existing reporters, package manager lockfiles, and scripts. +- Verify Node.js is version 18 or newer when possible. +- Check for `KATALON_API_KEY` in the current process environment or the repo's approved secret mechanism before composing any upload command. +- If `KATALON_API_KEY` is missing, stop before execution/upload and ask the user to provide it through a secure environment variable or secret mechanism. +- Run dependency installation only when the repo convention is clear and the user has allowed dependency changes. +- Confirm `@katalon/playwright-reporter` is installed or can be installed. +- Verify a deterministic report output directory, preferably `./reports` or the repo's existing Katalon report directory. +- Build commands with secrets redacted in user-facing output. + +Read `references/playwright-katalon-reporting.md` before editing config or composing upload commands. + +## Configure Reporter + +Prefer the existing project convention. If the Katalon reporter is absent and the user allowed edits: + +- Add `@katalon/playwright-reporter` as a dev dependency. +- Add `getGlobalSetupPath()` from `@katalon/playwright-reporter` to Playwright `globalSetup` unless the existing config already has a global setup that must be preserved. +- Add the Katalon reporter with `outputDir`. +- Configure platform upload through environment variables, not hard-coded values. +- Preserve existing reporters such as `list`, `html`, `junit`, or custom reporters. + +Do not overwrite unrelated Playwright projects, retries, timeouts, use settings, web servers, or CI-specific behavior. + +## Run And Upload + +Use one of these modes: + +- Upload during execution: set `KATALON_API_KEY` and `KATALON_PROJECT_ID`, then run the selected Playwright command. +- Separate upload: run Playwright to generate report files, then run `playwright-reporter upload`. +- Existing report upload: skip execution only when the user explicitly provides an existing Playwright reporter output directory. + +Prefer upload during execution when the project already has the Katalon reporter configured. Prefer separate upload when the user asks to run first, upload later, or when troubleshooting import failures. + +If the Playwright command fails after generating report artifacts, still attempt upload when the reporter output exists unless the user asked to skip upload on failure. Test failures are valid report data. + +## Platform Verification + +After upload: + +- Use Katalon MCP tools to list/read recent executions, test results, or quality data when available. +- Match by project ID, run time, test names, and framework metadata. +- Resolve the Platform result URL from reporter output, upload output, MCP execution data, or the known project/execution path. +- Always provide the best available Platform result URL so the user can open the uploaded run/report. +- If MCP cannot verify, inspect CLI output for upload success and report that Platform verification is blocked. +- If upload is asynchronous/importing, poll when a read/list tool exists; otherwise report the importing state. + +## Response Contract + +Report: + +- Playwright command run and exit status, with secrets redacted. +- Katalon upload mode: during execution, separate upload, or existing report upload. +- Local report output directory and key artifacts. +- Platform upload status: verified, importing, failed, or not verifiable. +- Platform report/run URL for the user to open when returned or discoverable; if no exact URL is available, provide the closest project reports/executions URL and state why it is not exact. +- Passed, failed, skipped, and timed-out counts when available. +- Failed tests with concise reasons. +- Any required follow-up: missing API key, missing project ID, install blocked, config permission missing, upload failed, or MCP verification unavailable. + +Do not claim success if Playwright ran locally but the Katalon upload did not happen or cannot be verified. diff --git a/skills/katalon-playwright-execute/references/playwright-katalon-reporting.md b/skills/katalon-playwright-execute/references/playwright-katalon-reporting.md new file mode 100644 index 000000000..2aac41c35 --- /dev/null +++ b/skills/katalon-playwright-execute/references/playwright-katalon-reporting.md @@ -0,0 +1,133 @@ +# Playwright Katalon Reporting + +## Official References + +- Playwright Integration: https://docs.katalon.com/katalon-platform/integrations/testing-framework/playwright-integration +- Katalon API Key: https://docs.katalon.com/katalon-platform/administer/profile/katalon-api-key-in-katalon-testops +- Katalon CLI Integration: https://docs.katalon.com/katalon-platform/integrations/ci-cd/katalon-cli-integration + +## Dependencies + +Required: + +- Node.js 18+ +- `@playwright/test` 1.0+ +- `@katalon/playwright-reporter` +- `KATALON_API_KEY` for upload +- `KATALON_PROJECT_ID` for upload + +Install with the repo package manager: + +```bash +npm install --save-dev @katalon/playwright-reporter +pnpm add -D @katalon/playwright-reporter +yarn add -D @katalon/playwright-reporter +``` + +Use the package manager already used by the repo. Do not install dependencies globally unless the repo already does that. + +## API Key Check + +Before running or uploading, check whether the API key is already available: + +```bash +test -n "$KATALON_API_KEY" && echo "KATALON_API_KEY is set" || echo "KATALON_API_KEY is missing" +``` + +Do not print the key value. If missing, ask the user to provide it through a secure environment variable or secret mechanism, for example by setting `KATALON_API_KEY` in the shell/session. Stop before upload until the key is available. + +## Playwright Config Pattern + +Add the reporter without removing existing reporters: + +```ts +import { defineConfig } from '@playwright/test'; +import { getGlobalSetupPath } from '@katalon/playwright-reporter'; + +export default defineConfig({ + globalSetup: getGlobalSetupPath(), + reporter: [ + ['list'], + [ + '@katalon/playwright-reporter', + { + outputDir: './reports', + ...(process.env.KATALON_API_KEY && { + platform: { + apiKey: process.env.KATALON_API_KEY, + projectId: process.env.KATALON_PROJECT_ID, + baseUrl: process.env.KATALON_BASE_URL, + authUrl: process.env.KATALON_AUTH_URL, + }, + }), + }, + ], + ], +}); +``` + +When an existing `globalSetup` exists, preserve it. If combining setup is non-trivial, ask before editing and prefer separate upload after execution. + +## Run Commands + +Run the user's target command when provided. Otherwise use the narrowest Playwright command: + +```bash +npx playwright test +npx playwright test tests/example.spec.ts +npx playwright test --grep "TC-07" +npx playwright test --project chromium +``` + +Upload during execution: + +```bash +KATALON_API_KEY="$KATALON_API_KEY" \ +KATALON_PROJECT_ID="" \ +KATALON_BASE_URL="" \ +npx playwright test +``` + +Omit `KATALON_BASE_URL` when the default Katalon endpoint is correct. + +## Separate Upload + +Use this when the report already exists or when run and upload are separate: + +```bash +npx playwright test +playwright-reporter upload \ + --output ./reports \ + --api-key "$KATALON_API_KEY" \ + --project-id "" \ + --base-url "" +``` + +Omit `--base-url` when the default endpoint is correct. + +## Secret Handling + +- Never print `KATALON_API_KEY` or auth tokens. +- Do not write API keys to `playwright.config.*`, `package.json`, logs, reports, or skill files. +- Prefer environment variables, shell secret injection, or the platform's secure secret mechanism. +- Redact secret-like values before summarizing command output. + +## Result URL + +After upload, return a URL the user can open: + +- Prefer an exact URL returned by `@katalon/playwright-reporter` or `playwright-reporter upload`. +- If MCP can find the execution/report, use that execution/report path. +- If only `project_id` is known, provide the closest Platform project reports or executions URL and mark it as a fallback. +- For custom Platform domains, build links from `KATALON_BASE_URL`; otherwise use the default Katalon Platform base URL expected by the reporter. + +Do not finish with only "uploaded successfully"; include the result URL or explicitly state why no URL could be resolved. + +## Common Failure Handling + +- Missing dependency: install `@katalon/playwright-reporter` only with permission to modify dependencies. +- Missing API key: stop before upload and ask for a secure secret source. +- Missing project ID: use Katalon MCP `list_projects`; ask only if multiple possible projects remain. +- Playwright test failures: still upload reports if artifacts exist. +- Reporter output missing: inspect Playwright config and stdout; do not run separate upload without an output directory. +- Upload returns success but MCP cannot find a run: report upload output and state that Platform verification is not available from current tools. diff --git a/skills/katalon-release-analyze/SKILL.md b/skills/katalon-release-analyze/SKILL.md new file mode 100644 index 000000000..5a7caa3f1 --- /dev/null +++ b/skills/katalon-release-analyze/SKILL.md @@ -0,0 +1,124 @@ +--- +name: katalon-release-analyze +description: Analyze Katalon True Platform/TestOps release readiness from testing quality data. Use when you need to use Katalon MCP metrics and results to assess whether a release, sprint, iteration, version, test plan, suite, or repository is ready to ship; summarize requirement coverage, execution health, defect risk, test stability, configuration coverage, release blockers, quality gaps, and produce a Ready / Ready with risk / Not ready recommendation. Written for the test manager who owns the ship call and the test lead who has to defend it. +--- + +# Katalon Release Analyze + +Use this skill to assess release readiness from Katalon MCP data. The output must be a testing-quality decision, not just a metric dump. + +## Availability Boundary + +State the MCP boundary before promising an assessment: + +- Available through Katalon MCP: project/repository discovery, requirements, test cases, test suites, executions/results, quality summaries, defect data, stability data, and configuration coverage when tools are exposed. +- Not directly available: create formal Test Plan entities, inspect live AUT UI, guarantee AI execution completion, or decide product/business readiness beyond test evidence. +- If MCP tools are unavailable, provide only a framework/template and mark release confidence as Low until data is verified. + +Read `references/capability-boundaries.md` when capability scope is unclear. + +## Required Scope + +Resolve or ask for the minimum missing scope: + +- Katalon project and repository/Test Project. +- Release identifier, sprint, iteration, version, test suite, suite collection, execution ID, or date range. +- Target environment/configuration matrix when readiness depends on browsers, devices, OS, or deployment environments. +- Quality gate thresholds when the team has defined them. + +If the user gives only a release name or issue key, use MCP discovery first. Ask only when multiple equally plausible scopes remain. + +## MCP Data Collection + +Always resolve context before analysis: + +1. Call `list_projects`. +2. Call `list_repositories`. +3. Resolve the repository/Test Project. +4. Find the release scope using available requirement, suite, execution, iteration, or result tools. + +Collect evidence with the available Katalon MCP tools: + +- Requirement and traceability: `find_requirements`, `read_requirement`, `fetch_requirement_data`. +- Test case quality: `find_test_cases`, `find_test_cases_by_requirement`, `fetch_test_case_data`. +- Suites and executable scope: `find_test_suites`, `read_test_suite`. +- Executions and results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Defect risk: `fetch_defect_data`. +- Stability: `fetch_test_stability_data`. +- Configuration coverage: `fetch_test_configuration_data`. + +Use the freshest data available for the requested release. If comparing trend or stability, include the date range used. + +## Key Metrics + +Report the metrics that are available and relevant: + +- Requirement coverage: total, covered, uncovered, linked test cases, requirements without passing evidence. +- Test coverage: total tests, automated/manual split, unexecuted, blocked, stale, missing ownership or incomplete cases. +- Execution health: latest executions, pass/fail/blocked/skipped/not-run counts, pass rate, failed P0/P1 tests. +- Defect risk: open critical/high defects, release blockers, reopened defects, failed tests with linked defects, untriaged failures. +- Stability: flaky tests, repeated failures, unstable suites/configurations, recent trend. +- Configuration coverage: environments, browsers, devices, OS, execution profiles, and missing target combinations. +- Data confidence: missing MCP data, stale executions, ambiguous release scope, or incomplete links. + +Read `references/release-quality-gates.md` before making the final readiness decision. + +## Readiness Decision + +Return one of: + +- `Ready`: no testing blocker remains; critical requirements have coverage and recent passing evidence; target configurations are sufficiently tested. +- `Ready with risk`: no hard blocker remains, but moderate gaps or accepted risks exist. +- `Not ready`: release-blocking defects, failed/blocked/unrun critical tests, missing critical coverage, stale evidence, or untested required configurations remain. + +If thresholds are provided, apply them. If not, use risk-based judgment and state assumptions. + +Never mark `Ready` when: + +- Critical/high release-blocking defects remain open. +- P0/P1 tests are failed, blocked, or not run without accepted risk. +- Critical requirements have no tests or no passing evidence. +- Required target configurations are untested. +- MCP data is unavailable or too incomplete to support confidence. + +## Response Format + +Respond with: + +```text +Release Readiness: Ready | Ready with risk | Not ready +Confidence: High | Medium | Low + +Key Metrics: +- Requirement coverage: +- Test execution: +- Defect risk: +- Stability: +- Configuration coverage: + +Blocking Issues: +- ... + +Risks / Gaps: +- ... + +Recommendation: +- ... + +Evidence: +- ... +``` + +Evidence should include execution IDs, suite names, requirement keys, defect IDs, or links when returned by the platform. + +## Follow-Up Actions + +Recommend concrete next steps: + +- Create or link missing tests for uncovered critical requirements. +- Re-run failed, flaky, or stale suites. +- Resolve or accept specific defects. +- Expand configuration coverage for missing release targets. +- Use `katalon-execute-test` when execution is needed. +- Use `katalon-create-test-cases` when coverage gaps require new manual cases. +- Use `katalon-upload-report` when external automation results must be uploaded before assessment. diff --git a/skills/katalon-release-analyze/references/capability-boundaries.md b/skills/katalon-release-analyze/references/capability-boundaries.md new file mode 100644 index 000000000..2cb5fcf95 --- /dev/null +++ b/skills/katalon-release-analyze/references/capability-boundaries.md @@ -0,0 +1,32 @@ +# Katalon MCP Capability Boundaries + +## Available + +- Project discovery: `list_projects`. +- Repository/Test Project discovery: `list_repositories`. +- Requirement discovery: `find_requirements`, `read_requirement`. +- Requirement coverage: `fetch_requirement_data`. +- Test case operations: `create_test_case`, `read_test_case`, `update_test_case`, `duplicate_test_case`, `delete_test_case`, `move_test_case`, `find_test_cases`. +- Test folder operations: `find_test_folders`, `manage_test_folder`. +- Test suite operations: `find_test_suites`, `read_test_suite`, `manage_test_suite`. +- Requirement links: `link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`. +- Manual execution: `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`. +- Automated execution: `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, `build_schedule`, `schedule_test_run`. +- Execution results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Quality data: requirement, defect, test case, test stability, and configuration coverage fetch tools. +- ALM defects: `find_alm_integration_projects`, `create_defect`. + +## Not Directly Available + +- Create requirements in Katalon True Platform. Requirements are synced from Jira/Azure and can be found/read/linked. +- Create a formal Test Plan entity. Use test suites/folders/executions as the executable planning structure. +- Guarantee Run with AI completion. The platform may block, fail, or require AUT/account state. +- Inspect AUT pages through Katalon MCP. Use Browser/Playwright for website exploration. +- Create defects without a failed test result ID and ALM integration details. + +## Recommended Workarounds + +- Requirement creation: create in Jira/Azure first, then sync/find/link in Katalon. +- Test plan: create a named folder and/or test suite, link to sprint/release, and create execution from that suite. +- AUT exploration: use Browser/Playwright to understand the product, then import manual cases into Katalon. +- AI execution blocked: report blocked state with required fixture, AUT, account, or environment action. diff --git a/skills/katalon-release-analyze/references/release-quality-gates.md b/skills/katalon-release-analyze/references/release-quality-gates.md new file mode 100644 index 000000000..fb788a9f2 --- /dev/null +++ b/skills/katalon-release-analyze/references/release-quality-gates.md @@ -0,0 +1,72 @@ +# Release Quality Gates + +Use this reference to turn Katalon MCP quality data into a release readiness assessment. + +## Core Metrics + +- Requirement coverage: total in-scope requirements, linked requirements, unlinked requirements, requirements with no tests, and requirements with no recent passing result. +- Test coverage: total test cases, automated/manual split when available, unexecuted tests, stale tests, and blocked tests. +- Execution health: latest execution status, pass/fail/blocked/skipped/not-run counts, pass rate, failed critical tests, and rerun status. +- Defect risk: open defects, critical/high defects, defects linked to failed tests, reopened defects, and unresolved release-blocking bugs. +- Stability: flaky tests, repeated failures, recent pass/fail trend, and unstable configurations. +- Configuration coverage: tested browsers/devices/environments/OS combinations versus the release target matrix. +- Data quality: missing execution data, missing requirement links, missing test ownership, incomplete test steps, and ambiguous release scope. + +## Suggested Readiness Decision + +Use `Ready` only when: + +- No open critical/high release-blocking defects remain. +- Critical requirements have test coverage and recent passing execution evidence. +- Latest execution pass rate is acceptable for the release context. +- No P0/P1 tests are failed, blocked, or not run without accepted risk. +- Target configurations have enough coverage for the supported release matrix. + +Use `Ready with risk` when: + +- No known hard blocker remains, but moderate risks exist. +- Some lower-priority requirements, configurations, or tests are unexecuted. +- Flaky tests exist but are understood and not tied to critical paths. +- Risk owners or mitigations are clear. + +Use `Not ready` when: + +- Critical/high release-blocking defects are open. +- Critical requirements lack tests or passing execution evidence. +- P0/P1 tests are failed, blocked, or not run. +- Execution results are too stale or incomplete to support a release decision. +- Required environments/configurations are untested. + +## Output Template + +```text +Release Readiness: Ready | Ready with risk | Not ready +Confidence: High | Medium | Low + +Key Metrics: +- Requirement coverage: +- Test execution: +- Defect risk: +- Stability: +- Configuration coverage: + +Blocking Issues: +- ... + +Risks / Gaps: +- ... + +Recommendation: +- ... + +Evidence: +- ... +``` + +## Practical Rules + +- Prefer specific counts and links over generic statements. +- Distinguish actual risk from missing data. +- Do not mark a release Ready when MCP data is unavailable; mark confidence Low and say what must be verified. +- If thresholds are not provided, use risk-based judgment and state assumptions. +- If a release/sprint/version scope is unclear, ask for it before making a final readiness call. diff --git a/skills/katalon-test-case-to-katalon-studio/SKILL.md b/skills/katalon-test-case-to-katalon-studio/SKILL.md new file mode 100644 index 000000000..ea5144e95 --- /dev/null +++ b/skills/katalon-test-case-to-katalon-studio/SKILL.md @@ -0,0 +1,320 @@ +--- +name: katalon-test-case-to-katalon-studio +description: Convert Katalon True Platform/TestOps manual test cases into Katalon Studio automation inside a local Studio Test Project checkout. Use when you need to author or extend a .tc test case file and its paired Groovy script under Scripts/, keep test case variable GUIDs consistent with the .ts test suite bindings that read them, model page elements as .rs test objects in the Object Repository instead of inline selectors, write WebUI, Mobile, or WS keyword steps in Groovy with Java interop, or wire data files, execution profiles, and GlobalVariable into a generated case. Also use to decide whether a headless katalonc run is possible at all, since Katalon Runtime Engine is a separately licensed paid runtime whose CLI requires -apiKey. Once a Studio run has finished and its report exists, hand off to katalon-upload-report. Written for the automation tester who has to land a manual case inside a Katalon Studio project rather than a code-first framework. +--- + +# Katalon Test Case To Katalon Studio + +Use this skill to turn Katalon Platform/TestOps manual test cases into Katalon Studio automation: `.tc` metadata, the paired Groovy script, Object Repository test objects, and the test suite that binds data to them. Prefer the conventions the target project already uses. Treat the human as a tool: ask concise questions whenever a project path, credential, selector, or test data value cannot be discovered safely. + +## Availability Boundary + +State the boundary before promising automation. It has three parts and none of them is optional. + +**Available through Katalon MCP** + +- Resolve context with `list_projects` and `list_repositories`. +- Read the source case with `find_test_cases` and `read_test_case`; resolve a requirement key with `find_requirements`, `read_requirement`, and `find_test_cases_by_requirement`. +- Read platform-side suites with `find_test_suites` and `read_test_suite`. +- Write the generated Studio path back onto the platform case with `update_test_case`, for traceability. + +**Not available through Katalon MCP** + +- No MCP tool reads or writes a Studio project file. `.prj`, `.tc`, `.rs`, `.ts`, `.glbl`, `.dat`, and everything under `Scripts/` live on disk only. Work on a local checkout and ask the human for the path. +- No MCP tool captures web objects. Web Spy and Record are Studio desktop features; write `.rs` files by hand. +- No MCP tool inspects the live application. Use Browser/Playwright to discover selectors, then encode them as test objects. +- No MCP tool runs anything. `katalonc` is a local command. +- No MCP tool uploads a Studio report. That is `katalon-upload-report`. +- No MCP tool parses or validates a generated `.tc`, `.ts`, or `.rs`. Studio is the only parser, so this skill verifies XML well-formedness and GUID consistency and reports what stays unverified. + +**Command-line execution is a paid licence, and this is the part people get wrong** + +Katalon Studio's desktop IDE authors and runs a test case interactively. Running the same suite from a terminal or a pipeline is a different, separately licensed product: **Katalon Runtime Engine**. `katalonc -runMode=console` lists `-apiKey` among its required arguments and will not execute without an activated KRE licence, and a floating KRE licence allows one active session at a time. + +So everything this skill generates is usable by a reader on the free Studio tier, and none of it runs headlessly for them. Say that before promising a CI story. Never imply the generated files are pipeline-ready without the licence. + +Report flag, version-dependent: `-report.folderPath` is available from Katalon Studio 11.1.2 and is the current flag. `-reportFolder` is deprecated from 11.1.2, still accepted, and can produce reports with missing test-suite-collection and test-suite folder names. Emit `-report.folderPath` unless the project is pinned below 11.1.2, and say which you chose. + +## Workflow + +```text ++-------------------+ --> +-------------------+ --> +---------------------+ +| Resolve Katalon | | Read test cases | | Resolve project | ++-------------------+ +-------------------+ +---------------------+ + | + v ++-------------------+ <-- +-------------------+ <-- +---------------------+ +| Verify pairings | | Write .tc + .rs | | Map manual steps | +| | | + script + suite | | to keywords | ++-------------------+ +-------------------+ +---------------------+ +``` + +## Katalon Source + +Resolve the Katalon context before writing files: + +- Use Katalon MCP tools to list projects and repositories, find test suites, find test cases, and read each selected case. +- If the user gives a test suite, read the suite and every included case before generating anything. +- If the user gives requirement keys, find requirement-linked cases first. +- If MCP tools are unavailable or authentication fails, ask for exported cases, case URLs, case keys, or the case text. +- Preserve traceability in both directions: put the platform case key in the `.tc` `` and ``, and write the generated `Test Cases/...` path back onto the platform case with `update_test_case`. + +Extract for each case: title, priority, requirement links, folder, preconditions, test data, manual steps, expected results, AUT URL, roles and accounts, environment, browser or device assumptions, and cleanup. Flag ambiguous selectors and business data that need human input. + +## Project Resolution + +Inspect the workspace before creating anything: + +- Search for `*.prj` at a directory root, plus the sibling folders `Test Cases/`, `Scripts/`, `Object Repository/`, `Test Suites/`, `Keywords/`, `Profiles/`, `Data Files/`, `Include/`, `Libs/`, and `settings/`. A `.prj` with those siblings is a Studio Test Project. +- If a project exists, summarise the detected path, its folder conventions, its existing test objects, and the files you intend to add, then ask the user to confirm before writing. +- If several projects exist, ask which one. +- If none exists, ask whether the user has a repository or path to use. Creating a Studio project from scratch outside the Studio IDE is possible but brittle; prefer an existing project or one created once in Studio. +- Never write into `Libs/`. `Libs/internal/GlobalVariable.groovy` is generated by Studio from `Profiles/*.glbl` and carries a do-not-modify banner. Edit the `.glbl` instead. +- `Reports/` is build output and is git-ignored in Katalon's own sample projects. Do not commit it. + +Read `references/studio-project-anatomy.md` before writing any file. Read `references/groovy-authoring.md` before writing script or custom keyword code. + +## The Two Pairings + +A Katalon Studio test case is two files, and a data-driven suite adds a third relationship. All three are silent when broken: nothing raises an error, and the run continues on defaults. + +**1. `.tc` to its script, paired by folder path.** + +```text +Test Cases/Authentication/TC-1042 Login.tc +Scripts/Authentication/TC-1042 Login/Script1754870400000.groovy +``` + +`Scripts/` mirrors the `Test Cases/` relative path, then adds a folder named exactly after the case, holding exactly one `Script*.groovy`. The number is an epoch-millis stamp and nothing references it. Rename or move the `.tc` without moving that folder and Studio sees a case with no steps. + +**2. `.ts` to the `.tc`, paired by project-relative path.** The suite's `` is the path with no extension, for example `Test Cases/Authentication/TC-1042 Login`. + +**3. `.ts` variable bindings to `.tc` variables, paired by GUID.** Each `` in the `.tc` carries an ``; each `` in the suite carries a `` that must equal it, plus a `` that must equal the `` in the same file. + +**When a `variableId` does not match, nothing errors.** The variable keeps its `` and every iteration runs on that default. Depending on the assertion, that shows up as a late timeout that looks like an application bug, or as a green suite over meaningless data. Cross-check the GUIDs after every write; the check is in the Verification section. + +## Object Repository Is The Page-Object Layer + +Katalon Studio has no page-object classes. `Object Repository/` is the equivalent layer, and the discipline is the same one Page Object Model asks for. + +- One `.rs` file per element, named for what it is (`input_Username`, `button_SignIn`), grouped in a folder per page. +- A new Object Repository folder needs a `.meta` `` with `WEBELEMENT`. A new Test Cases folder needs one with `TESTCASE`. +- Reference objects from the script only as `findTestObject('Login Page/input_Username')`. Never inline a raw XPath or CSS string in a script step. +- Reuse an existing test object when one already matches. Glob `Object Repository/**/*.rs` and read `` plus the selected `webElementProperties` before creating a duplicate. +- Prefer stable properties: `id`, `name`, `aria-label`, visible text. Set `true` only on the properties you actually want matched. Fall back to XPath only when the application gives nothing better, and say so. +- Test objects hold locators. They do not hold assertions. + +## Authoring Rules + +- Translate one manual step into one keyword call, in step order, so a manual tester can still read the script. +- Use `WebUI` for web, `Mobile` for mobile, `WS` for API. Import them with the alias form the project already uses. +- Turn expected results into `WebUI.verify*` calls next to the action that produces them. +- Use test case variables for anything the manual case parameterises, with a safe ``. Use `GlobalVariable` for environment values that come from a profile. +- Use `setEncryptedText` only when the source value is Katalon-encrypted; use `setText` otherwise. Mark a credential variable `true`. +- Put shared logic in `Keywords/` as a Groovy class with `@Keyword` methods, and call it through `CustomKeywords`. Groovy is Java-compatible, so a Java library on the project classpath can be imported directly. +- Keep one Studio case aligned to one Katalon Platform case unless the project groups scenarios differently. +- Do not invent credentials, URLs, product IDs, account state, or selectors. Ask, or leave a narrow TODO with the exact missing value named. + +## Worked Example + +Input, read with `read_test_case`: + +```text +Key: TC-1042 +Name: Login with valid credentials +Folder: Authentication +Requirement: CEL-6 +Test data: username, password +Steps: 1 Open the login page 2 Enter 3 Enter 4 Click Sign in +Expected: The dashboard header "System dashboard" is visible +``` + +Output, four files plus a suite. `Test Cases/Authentication/TC-1042 Login with valid credentials.tc`: + +```xml + + + Generated from Katalon Platform test case TC-1042. Requirement CEL-6. + TC-1042 Login with valid credentials + TC-1042 + + 3f8b1c02-6d41-4a97-b0e5-9c2a7d15e830 + + '' + Login username, bound from the suite data file + a41d55e6-2b78-4c10-9f33-6e0b8c74d219 + false + username + + + '' + Login password, bound from the suite data file + c07e93a4-1f65-48db-8a52-77b1e9f0d3cc + true + password + + +``` + +`Scripts/Authentication/TC-1042 Login with valid credentials/Script1754870400000.groovy`: + +```groovy +import static com.kms.katalon.core.testobject.ObjectRepository.findTestObject +import com.kms.katalon.core.webui.keyword.WebUiBuiltInKeywords as WebUI +import internal.GlobalVariable as GlobalVariable + +WebUI.openBrowser(GlobalVariable.URL + '/login') + +WebUI.waitForElementVisible(findTestObject('Login Page/input_Username'), 30) + +WebUI.setText(findTestObject('Login Page/input_Username'), username) + +WebUI.setText(findTestObject('Login Page/input_Password'), password) + +WebUI.click(findTestObject('Login Page/button_SignIn')) + +WebUI.waitForElementVisible(findTestObject('Dashboard Page/header_Dashboard'), 30) + +WebUI.verifyTextPresent('System dashboard', false) + +WebUI.closeBrowser() +``` + +`username` and `password` are unqualified: Studio injects test case variables into the script binding by name, and the GUID never appears in Groovy. The password step uses `setText` because the data file holds plain values; `setEncryptedText` is only for a Katalon-encrypted value, and `true` on the variable is a separate decision from either. + +`Object Repository/Login Page/input_Username.rs`: + +```xml + + + + input_Username + + b8d0f472-5e19-4c86-a1d7-2f9c63b04e55 + + + BASIC + + + + BASIC + false + + false + equals + tag + Main + input + + + true + equals + id + Main + username + + +``` + +`useRalativeImagePath` is spelled that way in the schema. Do not "fix" it. + +`Test Suites/Regression/Login Regression.ts`, where the GUIDs have to line up: + +```xml + + + + Login Regression + + false + + 0 + 30 + true + false + 91a3c6f8-4d27-4b0e-9c85-3ad2f7e61b40 + + 5c2e8b31-9a04-4f7d-b6e2-18d95c0a7f63 + false + true + Test Cases/Authentication/TC-1042 Login with valid credentials + + ONE + 7e14a09c-3b52-4d68-8f01-c5b7e2493a1d + + ALL + + + Data Files/valid-accounts + + + 7e14a09c-3b52-4d68-8f01-c5b7e2493a1d + DATA_COLUMN + Username + a41d55e6-2b78-4c10-9f33-6e0b8c74d219 + + + 7e14a09c-3b52-4d68-8f01-c5b7e2493a1d + DATA_COLUMN + Password + c07e93a4-1f65-48db-8a52-77b1e9f0d3cc + + + +``` + +**The failure this example exists to show.** Change one character of the first `` to `...d21f`. There is no XML error, no Studio dialog, no CLI warning. `username` is unbound, keeps `''`, and every data row logs in with an empty username. Here it surfaces late as a `waitForElementVisible` timeout on the dashboard header, which reads like an application bug. In a search or filter case with a tolerant assertion it does not surface at all and the suite reports green over meaningless data. + +## Verification + +Studio is the only parser for these files, so verify what you can and name what you cannot. + +1. **XML well-formedness** on every file written: `xmllint --noout ` or any equivalent parser. +2. **Path pairing.** For each generated `.tc`, assert that `Scripts///` exists and holds exactly one `Script*.groovy`. +3. **Suite-to-case pairing.** Each `` in the suite resolves to an existing `.tc` at that path plus `.tc`. +4. **Variable GUID cross-check.** Every `` in the suite must appear as a `` in the linked case: + + ```bash + comm -23 \ + <(grep -ho '[^<]*' "Test Suites/Regression/Login Regression.ts" | sed 's///' | sort -u) \ + <(grep -ho '[^<]*' "Test Cases/Authentication/TC-1042 Login with valid credentials.tc" | sed 's///' | sort -u) + ``` + + Empty output means the bindings hold. Any printed line is a variable that will silently run on its default. +5. **Object references.** Every `findTestObject('')` in the script resolves to an existing `.rs` under `Object Repository/`. +6. **Report what stays unverified.** Groovy compilation, keyword arity, selector correctness against the live application, and whether Studio opens the project are all outside what you can check here. Name them; do not imply a green check covers them. + +If the user has a Katalon Runtime Engine licence and asks for a run, print the command and hand off: + +```text +katalonc -noSplash -runMode=console \ + -projectPath="" \ + -testSuitePath="Test Suites/" \ + -executionProfile="" \ + -browserType="" \ + -apiKey="$KATALON_API_KEY" \ + -report.folderPath="" +``` + +Never ask the user to paste an API key into chat; source it from a secure variable. + +## Report Back + +Report created and updated files with full paths, the Katalon Platform cases they map to, every test object created or reused, the GUID cross-check result, what remains unverified, and any value still needed from a human. If the user has no Katalon Runtime Engine licence, say plainly that the generated suite runs from the Studio IDE and not from a terminal or a pipeline. + +## Prompt recipes + +- `Turn TC-1042 into a Katalon Studio test case in ~/work/shop-tests.` +- `Convert every case linked to requirement CEL-6 into Studio cases and one data-driven suite.` +- `Add username and password variables to this Studio case and bind them to Data Files/valid-accounts.` +- `Check that my Login Regression suite's variable bindings actually resolve.` + +## Hand-offs + +- A Studio run has finished and a report exists -> `katalon-upload-report`. +- The target framework is Playwright, not Studio -> `katalon-test-case-to-playwright`. +- The manual case does not exist yet -> `katalon-create-test-cases`. +- The generated suite has started failing intermittently -> `katalon-test-maintenance`. +- A red run needs triage -> `katalon-analyze-failures`. + +Read `references/studio-project-anatomy.md` for the file formats and folder rules. Read `references/groovy-authoring.md` for keywords, custom keywords, profiles, and data files. Read the orchestrator `katalon-true-platform-testing/references/unavailable-capabilities.md` when the user asks what Katalon can do through MCP. diff --git a/skills/katalon-test-case-to-playwright/SKILL.md b/skills/katalon-test-case-to-playwright/SKILL.md new file mode 100644 index 000000000..24cb252a6 --- /dev/null +++ b/skills/katalon-test-case-to-playwright/SKILL.md @@ -0,0 +1,96 @@ +--- +name: katalon-test-case-to-playwright +description: Convert Katalon True Platform/TestOps manual test cases, test suites, or requirement-linked cases into Playwright TypeScript automation. Use when you need to fetch/read Katalon Platform test cases and implement Playwright scripts, create or adapt a Playwright framework, apply Page Object Model and fixtures, or translate manual steps into meaningful automated test keywords. Written for the automation tester converting a manual case into code that fits an existing page-object layer. +--- + +# Katalon Test Case To Playwright Script + +Use this skill to turn Katalon Platform/TestOps test cases into maintainable Playwright TypeScript automation. Prefer existing project patterns when a Playwright framework already exists. Treat the human as a tool: ask concise questions whenever a required target, credential, repository, AUT detail, or test data value cannot be discovered safely. + +## Workflow + +```text ++-------------------+ --> +-------------------+ --> +---------------------+ +| Resolve Katalon | | Read test cases | | Resolve framework | ++-------------------+ +-------------------+ +---------------------+ + | + v ++-------------------+ <-- +-------------------+ <-- +---------------------+ +| Verify scripts | | Write automation | | Map manual steps | ++-------------------+ +-------------------+ +---------------------+ +``` + +## Katalon Source + +Resolve the Katalon context before writing code: + +- Use Katalon MCP tools when available to list projects, list repositories/Test Projects, find test suites, find test cases, and read each selected test case. +- If the user gives a test suite, read the suite and every included test case before generating scripts. +- If the user gives requirement keys, find requirement-linked cases first. +- If MCP tools are unavailable or authentication fails, ask the user for exported test cases, case URLs, case IDs, or the test case text. +- Preserve traceability by keeping Katalon case IDs or titles in test annotations, comments, tags, or test names according to the target framework's style. + +Extract for each case: + +- Title, priority, requirement links, folder/suite, preconditions, test data, manual steps, and expected results. +- AUT URL, user roles/accounts, environment, browser/device assumptions, and cleanup requirements. +- Ambiguous selectors or business data that require human input. + +## Framework Resolution + +Inspect the workspace before creating anything: + +- Search with `rg --files` for `playwright.config.*`, `package.json`, `tests/`, `e2e/`, `fixtures/`, `pages/`, and existing `*.spec.ts` files. +- If a Playwright framework exists in the workspace, summarize the detected path, conventions, and intended files, then ask the user to confirm before modifying it. +- If multiple candidate frameworks exist, ask the user which one to use. +- If no Playwright framework exists, ask whether the user wants to provide a Git repository/path or wants the agent to create/copy a framework into a target directory. +- If the user does not provide a repository/path after that prompt and the current workspace is writable, initialize a Playwright TypeScript framework in the current workspace. + +When initializing a new framework, use TypeScript and include: + +- `playwright.config.ts` +- `tests/` for specs +- `pages/` for Page Object Model classes +- `fixtures/` for custom fixtures and shared test data +- meaningful helper methods that read like domain actions, not raw selector operations + +Read `references/playwright-typescript.md` before scaffolding a new framework or making broad changes to an existing one. + +## Automation Design + +Translate manual Katalon steps into Playwright scripts using these rules: + +- Use Page Object Model for page structure and domain actions. +- Use fixtures for authenticated users, test data, API setup, reusable pages, and environment URLs. +- Write spec names and test steps that remain understandable to a manual tester. +- Prefer stable user-facing locators: role, label, placeholder, text, test id. Avoid brittle CSS/XPath unless the app gives no better option. +- Convert manual expected results into assertions near the action that produces them. +- Keep one automated test aligned to one Katalon test case unless the existing framework groups scenarios differently. +- Do not silently invent credentials, URLs, product IDs, account state, or selectors. Ask the human or mark a small TODO only when the missing value cannot be discovered. + +Use meaningful keywords in page objects and fixtures, for example: + +```ts +await productCatalog.searchForPhone(testData.phoneName); +await productCatalog.expectPhoneVisible(testData.phoneName); +await cart.addVisibleProductToCart(testData.phoneName); +await checkout.expectOrderSummaryTotal(expectedTotal); +``` + +## Implementation Rules + +- Follow the existing repository's naming, linting, folder, fixture, and assertion conventions when present. +- Keep generated code idiomatic TypeScript with explicit domain names and minimal comments. +- Store test data in the existing data fixture pattern; if none exists, create typed fixture data rather than scattering literals across specs. +- Add tags or annotations for Katalon case IDs when the framework supports it. +- Avoid changing unrelated framework config unless required for the requested tests. +- If the AUT must be inspected to identify selectors, use Browser/Playwright exploration and keep selectors stable. +- If live AUT access is blocked, implement the structure and mark only selector/test data gaps that require human input. + +## Verification + +After writing scripts: + +- Run the narrowest available check: TypeScript compile, lint, Playwright list, or a targeted `npx playwright test`. +- If the test cannot run because credentials, AUT access, or dependencies are missing, report the exact blocker and what remains unverified. +- Report created/updated files, mapped Katalon cases, commands run, and any manual inputs still needed. diff --git a/skills/katalon-test-case-to-playwright/references/playwright-typescript.md b/skills/katalon-test-case-to-playwright/references/playwright-typescript.md new file mode 100644 index 000000000..dcc44903a --- /dev/null +++ b/skills/katalon-test-case-to-playwright/references/playwright-typescript.md @@ -0,0 +1,128 @@ +# Playwright TypeScript Reference + +## Framework Detection + +Use these signals to detect an existing framework: + +- `playwright.config.ts`, `playwright.config.js`, or `playwright.config.mts` +- `@playwright/test` in `package.json` +- spec files under `tests/`, `e2e/`, `specs/`, or feature folders +- existing `pages/`, `page-objects/`, `fixtures/`, `katalon-test-data/`, or `utils/` +- custom test exports such as `fixtures/base.ts`, `test.extend`, or `export const test` + +When a framework exists, inspect a few representative specs, page objects, and fixtures before editing. Match imports, file naming, tag style, fixture names, and assertion style. + +## New Framework Shape + +For a new TypeScript framework, use a small structure: + +```text +playwright.config.ts +tests/ + .spec.ts +pages/ + .page.ts +fixtures/ + test.ts + katalon-test-data.ts +``` + +Prefer `npm init playwright@latest` or the repository's package manager equivalent. Choose TypeScript, install browser dependencies only when needed, and avoid overwriting existing application files. + +## Page Object Model + +Page objects should expose business actions and assertions: + +```ts +import { expect, type Locator, type Page } from '@playwright/test'; + +export class ProductCatalogPage { + readonly page: Page; + readonly searchInput: Locator; + + constructor(page: Page) { + this.page = page; + this.searchInput = page.getByRole('searchbox', { name: /search/i }); + } + + async goto(baseURL: string) { + await this.page.goto(baseURL); + } + + async searchForPhone(phoneName: string) { + await this.searchInput.fill(phoneName); + await this.page.getByRole('button', { name: /search/i }).click(); + } + + async expectPhoneVisible(phoneName: string) { + await expect(this.page.getByText(phoneName, { exact: false })).toBeVisible(); + } +} +``` + +Avoid methods named only after low-level mechanics such as `clickButton` or `fillInput` unless they are private helpers. Use names that reflect the Katalon manual step's intent. + +## Fixtures + +Use fixtures to centralize reusable pages, users, and test data: + +```ts +import { test as base } from '@playwright/test'; +import { ProductCatalogPage } from '../pages/product-catalog.page'; +import { testData } from './katalon-test-data'; + +type Fixtures = { + productCatalog: ProductCatalogPage; + data: typeof testData; +}; + +export const test = base.extend({ + productCatalog: async ({ page }, use) => { + await use(new ProductCatalogPage(page)); + }, + data: async ({}, use) => { + await use(testData); + }, +}); + +export { expect } from '@playwright/test'; +``` + +## Spec Mapping + +Keep Katalon traceability visible: + +```ts +import { test, expect } from '../fixtures/test'; + +test.describe('Search and filter', () => { + test('TC-123 Verify phone price and stock search', async ({ productCatalog, data }, testInfo) => { + testInfo.annotations.push({ type: 'katalonCaseId', description: 'TC-123' }); + + await test.step('Open cellphone storefront', async () => { + await productCatalog.goto(data.baseURL); + }); + + await test.step('Search for an in-stock phone', async () => { + await productCatalog.searchForPhone(data.phones.inStock.name); + await productCatalog.expectPhoneVisible(data.phones.inStock.name); + }); + }); +}); +``` + +Each `test.step` should correspond to a meaningful manual action or assertion group, not every tiny Playwright call. + +## Human-As-Tool Questions + +Ask the human when any of these are missing and cannot be discovered: + +- target repository or directory +- permission to modify an existing framework +- AUT URL or environment +- credentials, roles, or setup data +- unique product/order/user records +- selector strategy when the UI cannot be inspected +- whether to scaffold a new framework or use a provided repository + +Keep questions short and actionable. Ask only for required values needed to continue. diff --git a/skills/katalon-test-data/SKILL.md b/skills/katalon-test-data/SKILL.md new file mode 100644 index 000000000..584ec98e7 --- /dev/null +++ b/skills/katalon-test-data/SKILL.md @@ -0,0 +1,186 @@ +--- +name: katalon-test-data +description: Design, source, seed, and tear down the test data a Katalon True Platform test case or an automated suite runs on. Use when the steps are already settled and the blocker is the values, for example which data classes a case needs, which records must exist before a run, how to keep literals out of the step text and into the Test Data column or a fixture, and how to reset state afterwards so the next run starts clean. Covers choosing between static, generated, and cloned production data, keeping credentials out of test data, and the boundary that the Katalon MCP has no test data, fixture, seeding, or secrets tool of its own. If the cases do not exist yet, start at katalon-create-test-cases. Written for the manual tester filling in a case's Test Data column and precondition, and the automation tester wiring fixtures and teardown for a suite. +--- + +# Katalon Test Data + +Use this skill for the **data** half of test design: which values a case runs on, which records must already exist when it starts, and what has to be true again when it finishes. The steps belong to `katalon-create-test-cases`; the code belongs to `katalon-test-case-to-playwright`; the values, the state, and the cleanup belong here. + +One artifact serves both lanes. Call it the **data contract**: what must exist, which values, and who resets it. A manual case carries the contract in its Pre-condition and per-step Test Data column. An automated suite carries the same contract in a fixture plus a teardown. Design it once, write it down twice. + +## Availability Boundary + +State this before promising anything, because three different things in the Katalon world are called "test data" and the MCP reaches only one of them. + +- **Available via MCP:** resolve scope (`list_projects`, `list_repositories`); read the requirement the data must satisfy (`find_requirements`, `read_requirement`); find the cases the data belongs to (`find_test_cases`, `find_test_cases_by_requirement`, `read_test_case`); write the Pre-condition and the per-step Test Data column into a case (`create_test_case`, `update_test_case`); read the target environment (`read_auts`); group data-dependent cases so they run together (`manage_test_folder`, `find_test_suites`, `read_test_suite`, `manage_test_suite`). +- **Not available via MCP, at all:** + - **No katalon-test-data entity, data file, or data binding.** The standalone Test Data object (CSV, Excel, internal, database) is a **Katalon Studio desktop** artifact. The MCP does not create, read, or bind it. + - **No secrets or variables.** Project `SECRET` and `VAR` entries live in the platform UI under Settings -> Configurations -> Secrets & Variables, are referenced in steps as `{KEY}`, apply to cloud-hosted execution only, and are scoped to the project. There is no tool to read or write them. + - **No seeding, no teardown, no database access, no API-call tool.** Nothing in the MCP can create or destroy application state. + - **No requirement creation.** Requirements sync from Jira or Azure; create them there, then find and link. +- **Do not invent tools.** `manage_test_data`, `create_test_data`, `read_secrets`, `seed_data`, `reset_environment`, and `query_database` **do not exist**. If a workflow seems to need one, use the workaround below and say the boundary out loud. +- **Run with AI reads the case body and nothing else.** Data that is not in the Pre-condition or the Test Data column does not reach the AI runner. This is why the placement rule below is not a style preference. +- **Workaround, the MCP-legal data contract:** + - Manual lane: the case's **Pre-condition + per-step Test Data column**, written through `update_test_case`, is the data contract of record. + - Automation lane: a **checked-in fixture or factory**, plus the **seed command named in the case Pre-condition** so the manual and coded lanes agree on the same starting state. + - Either way the contract is versioned, reviewable, and needs no tool that does not exist. + +## Workflow + +```text ++------------------+ --> +---------------------+ --> +----------------------+ +| Resolve scope | | Derive data classes | | Choose data strategy | ++------------------+ +---------------------+ +----------------------+ + | + v ++------------------+ <-- +---------------------+ <-- +----------------------+ +| Verify, run twice| | Plan seed + teardown| | Place the data | ++------------------+ +---------------------+ +----------------------+ +``` + +## Steps and tool rules + +1. **Resolve scope.** `list_projects` -> `list_repositories`. If exactly one matches the user's wording or context, use it. Call `read_auts` to learn which environment the data has to exist in; data designed for the wrong environment is worse than no data. +2. **Derive the data classes, do not guess values.** Read the requirement (`read_requirement`) and the cases (`find_test_cases_by_requirement`, then `read_test_case` on each). Turn each condition in the requirement into a data class using equivalence partitioning and boundary value analysis, exactly as `katalon-create-test-cases` does for coverage. One class per case, matching the atomic-case rule. Read `references/data-design.md` before deriving classes for a non-trivial requirement. +3. **Choose the strategy per class.** Static, generated, or cloned from production, decided per class and never by default. The trade-offs and the decision table are in `references/data-design.md`. Say which one you chose and why in the report. +4. **Place the data.** Apply the placement rule below. This is the step that stops data rotting inside step text. +5. **Plan seed and teardown together.** Never design a seed without its teardown in the same breath. Use the seeding ladder in `references/seed-and-teardown.md`: application API first, CLI or SQL second, UI last. Record the seed command in the case Pre-condition so both lanes start from the same state. +6. **Verify, then run it twice.** `read_test_case` to confirm the data actually landed in the case. Then the real check: the suite must pass **twice in a row from the state the first run left behind**, with no manual reset in between. A second-run failure is a data defect, not a flaky test. + +## The placement rule + +Concrete values do not belong in step text or in a spec file. They belong in a named place, one per lane. + +| Data | Manual lane | Automation lane | +|---|---|---| +| Values the step consumes | per-step **Test Data** column | typed fixture or factory module | +| State that must pre-exist | case **Pre-condition**, with the seed command named | `globalSetup` or a fixture that creates it | +| Environment or AUT URL | Pre-condition, or the AUT environment from `read_auts` | config or env var, never a literal in a spec | +| Credentials and tokens | `{KEY}` reference to Secrets & Variables | environment variable, never checked in | +| Cleanup | an explicit reset step in the Pre-condition or final step | teardown in the same fixture that created the state | + +Two rules that follow from it: + +- **A step reads `Enter the loyalty email`, not `Enter qa+cel9@example.com`.** The value goes in the Test Data column so the same step survives a data change. +- **A fixture owns creation and destruction of the same record.** If one function creates it and a different one deletes it, they will drift and the second run will fail. + +## Credentials are not test data + +This is a hard rule, not a preference. + +- **Never** write a password, token, API key, cookie, JWT, or MFA code into a step, a Test Data column, a fixture file, a Pre-condition, or the chat. +- Reference a platform secret as `{KEY}` and tell the human to create it in Settings -> Configurations -> Secrets & Variables. That page is UI-only; there is no MCP tool for it. +- In code, read from an environment variable and commit only the variable's name. +- If a user pastes a real credential, do not store it in a case or a file. Ask them to add it as a secret and use the `{KEY}` reference instead. +- Production clones carry real people's data. Mask or synthesize personally identifying fields before anything is written into a case, a fixture, or a report. Details in `references/data-design.md`. + +## Seed and teardown + +The short version; the full ladder, idempotency rules, and per-lane patterns are in `references/seed-and-teardown.md`. + +- **Prefer the application's own API** to create state. It is fast, it is the same code path the product uses, and it fails loudly. +- **Fall back to CLI or SQL** when no API exists, and say in the report that the seed bypasses application rules. +- **Seed through the UI only as a last resort**, and never inside the test that is measuring the behaviour, or a setup failure will be reported as a product failure. +- **Make every seed idempotent.** Running it twice must leave one record, not two. +- **Make every key unique per run.** A run stamp such as `qa+cel9-20260810-1432@example.com` prevents cross-run collisions and lets teardown find exactly what it created. +- **Tear down what you created, in the place that created it.** Not in a nightly cleanup job, not by hand. +- **Never seed or clean a production environment.** Confirm the target from `read_auts` before any destructive step. + +## Worked example + +**Input.** Project `Cellphone Shop`, requirement `CEL-9` synced from Jira: + +> A returning shopper whose loyalty tier is Silver or above receives 10% off when the cart subtotal reaches $500. The discount applies once per order and never applies to gift cards. + +`find_test_cases_by_requirement(requirement = "CEL-9")` returns three existing cases with empty Test Data columns: + +```text +TC-2101 Apply loyalty discount at the subtotal threshold +TC-2102 No loyalty discount below the subtotal threshold +TC-2103 Loyalty discount excludes gift cards +``` + +**Step 2 output, the data classes.** Four conditions in the requirement, so four classes, plus the two boundary values the threshold demands: + +| Class | Account | Cart | Expected | +|---|---|---|---| +| At threshold, eligible tier | Silver | subtotal exactly 500.00 | 10% applied, total 450.00 | +| Below threshold, eligible tier | Silver | subtotal 499.99 | no discount, total 499.99 | +| At threshold, ineligible tier | Bronze | subtotal 500.00 | no discount | +| Gift card in cart | Silver | 1 gift card 500.00 | no discount | + +**Step 3 output, the strategy.** Generated, not static and not cloned. Loyalty tier and order history are mutated by the test, so a shared static account goes stale after the first run, and cloned production accounts carry real customer PII for no benefit. + +**Step 4 output, the manual lane.** One `update_test_case` call per case, all edits in a single call. For `TC-2101`: + +```text +Pre-condition: + Storefront reachable at the Cellphone Shop AUT environment. + Seed a Silver-tier account with zero orders this month, using: + npm run seed -- --profile loyalty-silver --stamp + Cart is empty. Log in as the seeded account. + +| Step | Test Step | Expected Result | Test Data | +| 1 | Navigate to the storefront | Storefront home page is displayed | {AUT_URL} | +| 2 | Log in as the loyalty shopper | Account menu shows the Silver badge | qa+cel9-@example.com / {LOYALTY_PASSWORD} | +| 3 | Add the listed phone to the cart | Cart subtotal shows 500.00 | Model A-500, quantity 1, 500.00 | +| 4 | Open the cart | Loyalty discount line shows -50.00 and the order total shows 450.00 | N/A | + +Teardown: delete the seeded account with + npm run seed -- --teardown --stamp +``` + +**Expected output of the verification call.** `read_test_case(id = "TC-2101")` returns the case with a non-empty `testData` value on steps 1 to 3, the seed command inside `precondition`, `{LOYALTY_PASSWORD}` as a reference rather than a literal, and the requirement link to `CEL-9` intact. If the Test Data column comes back empty, the update did not land; re-issue the single `update_test_case` call rather than creating a second case. + +**Step 4 output, the automation lane.** The same contract as a typed factory, with creation and destruction in one place: + +```ts +// fixtures/loyalty-data.ts +export type LoyaltyShopper = { email: string; tier: 'Bronze' | 'Silver'; teardown: () => Promise }; + +export async function createLoyaltyShopper(tier: 'Bronze' | 'Silver'): Promise { + const stamp = `${Date.now()}`; + const email = `qa+cel9-${stamp}@example.com`; + await api.post('/test-support/accounts', { email, tier, password: process.env.LOYALTY_PASSWORD }); + return { email, tier, teardown: () => api.delete(`/test-support/accounts/${email}`) }; +} + +export const CART = { + atThreshold: { sku: 'A-500', quantity: 1, subtotal: 500.00, expectedTotal: 450.00 }, + belowThreshold:{ sku: 'A-499', quantity: 1, subtotal: 499.99, expectedTotal: 499.99 }, + giftCard: { sku: 'GIFT-500', quantity: 1, subtotal: 500.00, expectedTotal: 500.00 }, +}; +``` + +**Expected report to the user.** Name every one of these; a report missing the boundary line is incomplete: + +```text +Data contract for CEL-9, 4 classes across 3 cases. +Strategy: generated per run. Reason: the test mutates tier and order history. +Manual lane: TC-2101, TC-2102, TC-2103 updated. Pre-condition carries the seed command, Test Data column carries the values. +Automation lane: fixtures/loyalty-data.ts creates and tears down the shopper in one factory. +Credentials: LOYALTY_PASSWORD referenced as {LOYALTY_PASSWORD} and as an env var. Not stored in any case or file. +Second-run check: pending. Run the suite twice with no reset between runs before trusting these cases. +Boundary: the Katalon MCP has no katalon-test-data, seeding, or secrets tool. {LOYALTY_PASSWORD} must be created by hand + in Settings -> Configurations -> Secrets & Variables, which is UI-only and applies to cloud-hosted runs only. +``` + +## Prompt recipes + +- `Design the test data for requirement CEL-9 and write it into the existing cases.` +- `These cases have values hard-coded in the step text. Move them into the Test Data column and give me the seed command.` +- `Seed and tear down the order data for tonight's run, manual lane and Playwright lane using the same records.` +- `Our suite passes once and fails on the second run. Find the missing teardown.` +- `What data does this requirement need before anyone writes a case for it?` + +## Hand-offs + +- No cases exist yet -> `katalon-create-test-cases`, then come back for the data. +- Turning the case into code, where the fixture will live -> `katalon-test-case-to-playwright`. +- Running the cases once the data exists -> `katalon-execute-test`, or `katalon-playwright-execute` for the coded lane. +- A red run that may be a data problem rather than a product problem -> `katalon-analyze-failures`, which routes "environment / data" failures back here. +- Cases that keep breaking on stale data -> `katalon-test-maintenance`. +- Scoping the cycle before any of this -> `katalon-test-plan`. + +Read `references/data-design.md` before deriving data classes or choosing a strategy. Read `references/seed-and-teardown.md` before writing any seed or teardown. Read the orchestrator's `katalon-true-platform-testing/references/unavailable-capabilities.md` when the user asks "can Katalon manage test data?". diff --git a/skills/katalon-test-data/references/data-design.md b/skills/katalon-test-data/references/data-design.md new file mode 100644 index 000000000..ab1bc944f --- /dev/null +++ b/skills/katalon-test-data/references/data-design.md @@ -0,0 +1,63 @@ +# Test Data Design + +## From requirement to data classes + +The steps come from the requirement's actions. The data comes from its **conditions**. Read the requirement once for each and do not mix the passes. + +1. List every condition the requirement states: thresholds, statuses, roles, tiers, quantities, dates, formats, and exclusions. +2. Partition each condition into equivalence classes. One class is one case, matching the atomic-case rule in `katalon-create-test-cases`. +3. Add boundary values for every ordered condition: at the boundary, one below, one above. A threshold of "500 or more" produces 499.99, 500.00, and 500.01, not "a big number". +4. Add the exclusions the requirement names explicitly. "Never applies to gift cards" is a data class, not an edge case to remember later. +5. Stop when every condition has at least one class. Combinatorial explosion is handled by pairwise reduction, not by inventing more accounts. + +Write the classes as a table with the account state, the input values, and the expected outcome, and confirm it with the user before touching a single case. + +## Choosing a strategy, per class + +Decide per class. A suite that uses one strategy for everything has not made a decision. + +| Strategy | Use when | Costs | +|---|---|---| +| **Static** | The data is read-only and shared safely, such as a product catalogue or a country list | Goes stale silently; any test that mutates it poisons every other test | +| **Generated per run** | The test mutates the record: accounts, orders, carts, subscriptions, anything with a lifecycle | Needs a seed path and a teardown; needs unique keys | +| **Cloned from production** | The behaviour only reproduces at real-world scale or shape | Carries real PII, needs masking, needs a refresh policy, and is the slowest to provision | + +Default to **generated** whenever the test writes. Reach for **static** only for read-only reference data. Reach for **cloned** only when the other two provably cannot reproduce the behaviour, and say so in the report. + +## The placement rule, and why it is not cosmetic + +Concrete values inside step text are invisible to everyone who needs them: the next tester, the automation engineer converting the case, and the AI runner that reads the case body. + +- **Manual lane.** Values go in the per-step **Test Data** column. State that must pre-exist goes in the **Pre-condition**, with the seed command written out. Environment URLs come from the AUT environment or the Pre-condition. +- **Automation lane.** Values go in a typed fixture or factory. Nothing that varies by environment is a literal in a spec. +- **Both lanes reference the same records.** If the manual case says "a Silver-tier account seeded by `npm run seed -- --profile loyalty-silver`" and the fixture calls a different endpoint, the two lanes are testing two different applications. + +A step that reads `Enter the loyalty email` survives a data change. A step that reads `Enter qa+cel9@example.com` has to be rewritten every time, in every case that mentions it. + +## Personally identifying data + +- **Never** copy real names, emails, phone numbers, addresses, payment details, or government identifiers into a test case, a fixture, a screenshot, or a report. +- Mask or synthesize before the data leaves the source system, not after it has been written somewhere. +- Use reserved, non-routable values: `@example.com` for email, documented reserved ranges for phone numbers, and test card numbers published by the payment provider. +- Production clones are the highest-risk path in this whole skill. If a clone is unavoidable, name in the report what was masked, what was not, and who approved it. + +## Credentials + +Credentials are not test data and never live with it. + +- Platform secrets live in Settings -> Configurations -> Secrets & Variables and are referenced as `{KEY}`. The page is UI-only; the MCP has no tool for it, and secrets apply to cloud-hosted execution only. +- Code reads credentials from environment variables. Commit the variable name, never the value. +- Never paste a password, token, cookie, JWT, or MFA code into chat, a case, or a file, and never ask a user to. + +## Data quality checklist + +Run this before declaring a data design done. + +- [ ] Every requirement condition maps to at least one data class. +- [ ] Every ordered condition has its boundary values, not just a representative value. +- [ ] Every class names its expected outcome, not just its inputs. +- [ ] No value that the test mutates is shared between cases. +- [ ] Every key that must be unique carries a run stamp. +- [ ] No credential, token, or real personal detail appears anywhere in the contract. +- [ ] Every seed has a teardown written at the same time. +- [ ] The design says which environment it targets, and it is not production. diff --git a/skills/katalon-test-data/references/seed-and-teardown.md b/skills/katalon-test-data/references/seed-and-teardown.md new file mode 100644 index 000000000..1f996960b --- /dev/null +++ b/skills/katalon-test-data/references/seed-and-teardown.md @@ -0,0 +1,83 @@ +# Seed And Teardown + +The Katalon MCP cannot create or destroy application state. Everything in this file happens outside it, and the case Pre-condition is how the platform learns what happened. + +## The seeding ladder + +Take the highest rung that works. + +1. **The application's own API.** Fastest, uses the product's real validation, and fails loudly with a status code. Prefer a dedicated test-support endpoint when the product has one. +2. **A CLI or a seed script the product already ships.** Good, but say in the report that it may bypass some application rules. +3. **Direct SQL.** Only when no API and no CLI exists. It bypasses every application rule, so a row can be created that the product itself would refuse. Note that risk explicitly. +4. **The UI.** Last resort. Never seed through the UI inside the test that measures the behaviour, or a setup failure gets reported as a product failure. If the UI is the only path, seed in a separate step and let it fail separately. + +## Rules that make a seed trustworthy + +- **Idempotent.** Running it twice leaves one record, not two, and does not error on the second run. +- **Unique per run.** Stamp every key that must be unique: `qa+cel9-20260810-1432@example.com`, `ORDER-`. The stamp is what teardown searches for. +- **Self-describing.** A seeded record should be identifiable as test data by looking at it, so a human who finds one in a shared environment knows what it is. +- **Scoped to what the case needs.** Seeding a hundred orders to test one discount makes the failure harder to read and the teardown slower. +- **Environment-checked.** Confirm the target from `read_auts` before anything destructive. Never seed or clean production. +- **Recorded in the Pre-condition.** Write the exact command in the case Pre-condition so the manual lane and the coded lane start from the same state. + +## Teardown + +- **The thing that created the state destroys it**, in the same module or the same fixture. Split creation and destruction and they will drift. +- **Teardown runs even when the test fails.** Use the framework's guaranteed hook, not a line at the end of the test body. +- **Teardown is scoped to this run's stamp.** Never delete by a broad pattern in a shared environment. +- **Leave nothing behind that changes another test's outcome**, including cart state, session state, and feature-flag overrides. +- **A nightly cleanup job is not teardown.** It hides the defect until the day it does not run. + +## Manual lane pattern + +```text +Pre-condition: + Target environment: + Seed: npm run seed -- --profile --stamp + Login as: qa+-@example.com / {SECRET_KEY} + State: cart empty, no orders this month + +... steps, with values in the Test Data column ... + +Teardown: + npm run seed -- --teardown --stamp +``` + +Write it into the case with a single `update_test_case` call containing every edit, then confirm with `read_test_case`. If the Test Data column comes back empty, re-issue the update; never create a duplicate case because an update appeared to fail. + +## Automation lane pattern + +Creation and destruction in one factory, teardown attached to the fixture that owns it: + +```ts +// fixtures/data.ts +import { test as base } from '@playwright/test'; + +type Fixtures = { shopper: { email: string } }; + +export const test = base.extend({ + shopper: async ({}, use) => { + const stamp = `${Date.now()}`; + const email = `qa+cel9-${stamp}@example.com`; + await api.post('/test-support/accounts', { email, tier: 'Silver', password: process.env.LOYALTY_PASSWORD }); + await use({ email }); + await api.delete(`/test-support/accounts/${email}`); // runs even if the test fails + }, +}); +``` + +Use `globalSetup` only for state that is genuinely shared and read-only across the whole run. Anything a test mutates belongs to a per-test fixture, or the tests become order-dependent. + +## The acceptance test for any data design + +**Run the suite twice in a row, from the state the first run left behind, with no manual reset.** + +A second-run failure is a data defect and it has exactly three causes: + +| Symptom on the second run | Cause | Fix | +|---|---|---| +| "already exists" or a uniqueness error | The key is not run-stamped | Stamp the key | +| The assertion sees data from the first run | Teardown missing or not guaranteed | Move teardown into the fixture hook | +| Passes alone, fails in the suite | One test depends on another's state | Give each test its own seeded record | + +Report this check by name. If it has not been run, say so rather than implying the data design is proven. diff --git a/skills/katalon-test-estimation/SKILL.md b/skills/katalon-test-estimation/SKILL.md new file mode 100644 index 000000000..cdac1510a --- /dev/null +++ b/skills/katalon-test-estimation/SKILL.md @@ -0,0 +1,228 @@ +--- +name: katalon-test-estimation +description: Estimate testing effort, duration, and resourcing for a Katalon True Platform/TestOps cycle. Use when the question is how long testing will take, how many testers it needs, whether the scope fits the sprint window, or what a scope change costs in person-hours. Sizes design, manual execution, automated execution and triage, and rework separately, counts the countable part from platform data (case counts, automation split, historical pass and stability rates, configuration matrix), calibrates the rest against a rate the team supplies, and returns a three-point range with a confidence label instead of a single number. Splits resourcing across the manual and automated lanes and names the assumptions that would move the number most. For what to test and in what order, use katalon-test-plan; for a verdict on a cycle that has already run, use katalon-release-analyze. Written for the test lead sizing a cycle before it starts and the test manager who has to fund it. +--- + +# Katalon Test Estimation + +Use this skill for the **estimation** half of the planning stage: turn a scoped cycle into person-hours, a date, and a headcount. The output is a **range with a confidence label**, never a single number. `katalon-test-plan` decides what gets tested; this skill decides what that costs. + +## Availability Boundary + +State this before producing any number. + +**The Katalon MCP can count the work. It cannot measure the effort.** + +- **Available via MCP:** case and suite counts (`find_test_cases`, `read_test_suite`, `find_test_cases_by_requirement`), uncovered requirements (`fetch_requirement_data`), automated/manual split when exposed (`fetch_test_case_data`), failure and flake rates (`fetch_test_stability_data`, `find_test_results`), the configuration matrix (`fetch_test_configuration_data`), the cycle window (`find_iterations`), and a past execution's machine-recorded elapsed time (`read_execution`, `read_execution_test_results`). +- **Not available at all:** how long a person takes. No tool books effort against a case, a run, or a person. A manual run's open and close timestamps are calendar elapsed, not work. +- **Also not available:** team roster, capacity, PTO, or working calendar; any cost or rate model; a field to write the estimate back into; any cross-cycle aggregate such as average cycle duration. +- **Looks like MCP data but is not:** session concurrency. `list_test_cloud_environments` and `find_execution_profiles` list environments and profiles, not the number of parallel sessions the plan allows. Tag it team-supplied. +- **Field-level rule:** read one record with `read_execution` and confirm the elapsed-time field is present before modelling on it. If it is absent, ask for the figure and label it team-supplied. Never let a missing field become a zero. + +The consequence: **every per-unit rate is team-supplied and must be tagged as such in the output.** Read `references/mcp-evidence-map.md` for the full signal-to-tool map and the human-input checklist. + +## Estimation Workflow + +```text ++------------------+ +---------------------+ +----------------------+ +| Fix the scope | --> | Count from platform | --> | Get the rates (human)| ++------------------+ +---------------------+ +----------------------+ + | | | + v v v ++------------------+ +---------------------+ +----------------------+ +| Size five buckets| --> | Three-point + band | --> | Resource the lanes | ++------------------+ +---------------------+ +----------------------+ + | + v + +----------------------+ + | State the uncertainty| + +----------------------+ +``` + +## Steps and tool rules + +1. **Fix the scope, or refuse.** An estimate of an unfixed scope is theatre. Resolve project and repository (`list_projects`, `list_repositories`), bind the cycle window with `find_iterations`, and get the release, sprint, folder, or suite in writing. If scope is still moving, produce the model with the scope stated as an assumption and force confidence to Low. +2. **Count the countable part.** Never estimate what the platform can count. `fetch_requirement_data` for uncovered requirements, `find_test_cases` and `read_test_suite` for case counts, `fetch_test_case_data` for the automated/manual split when available, `fetch_test_configuration_data` for the matrix multiplier, `fetch_test_stability_data` for the flake burden. If a result set is paginated or truncated, say so rather than reporting a partial count as a total. +3. **Read one comparable execution.** Pick the closest finished cycle with the human, then `read_execution` and `read_execution_test_results` on it. Extract pass rate and, if the field is present, automated elapsed time. This is your only platform-side calibration, and it is n=1. Say so. +4. **Get the rates from the human.** Manual execution minutes per case, design minutes per new case, triage minutes per failing result, automation authoring hours per case, productive hours per tester per day, fixed-overhead percentage, session concurrency. Rank the sources: measured in this project on a recent comparable cycle beats team declaration beats an industry default. Tag each with its source and the date it was last measured. +5. **Size five buckets separately.** Design, manual execution, automated execution and triage, automation authoring, fixed overhead. A bucket deliberately out of scope is recorded as zero **with the reason**, never dropped. +6. **Three-point every bucket.** Optimistic, most likely, pessimistic. PERT mean is `(O + 4M + P) / 6`; standard deviation is `(P - O) / 6`. Combine independent buckets by summing the means and root-sum-squaring the deviations. Report the 80 percent band as `mean +/- 1.28 SD`, rounded outward. +7. **Resource both lanes, and keep the units apart.** Manual work is person-hours. Automated execution is **wall clock**, bounded by session concurrency, and its human cost is triage rather than running. Run-with-AI is wall clock plus review time. Never add wall clock to person-hours. +8. **Answer the question that was asked.** Fixed date gives headcount. Fixed team gives a date. Show both directions and name the serialized part that no extra person can shorten - usually triage and reporting, which cannot start before execution ends. +9. **State the uncertainty honestly.** Confidence label, the two or three assumptions that move the number most with their sensitivity in hours, what is excluded, and the one measurement that would raise confidence a level. Read `references/estimation-model.md` for the confidence rubric before choosing the label. + +## The five buckets + +Most people say "estimate the test cycle" and mean bucket B alone. Sizing only B is the most common way a test estimate comes in low. + +- **A. Design** new coverage - person-hours, driven by uncovered requirements x cases per requirement. +- **B. Manual execution** - person-hours, driven by manual case count x configurations in the manual matrix. +- **C. Automated execution and triage** - wall clock for the run, person-hours for the triage of what fails. +- **D. Automation authoring** - person-hours, driven by the cases selected for automation. +- **E. Fixed overhead** - a percentage of A+B+C+D, typically 10 to 20 percent. + +Three rules govern them: + +- A bucket that is out of scope is **recorded as zero with the reason**. Silence reads as forgetting. +- **Rework is not a sixth bucket.** Retest of failures lives inside B, triage of failures inside C. Modelling it separately double-counts. +- Exploratory or session-based testing, when the cycle includes it, is **timeboxed rather than estimated**. Add the timebox as a stated line, not as a derived figure. + +Read `references/estimation-model.md` for the arithmetic, the rework multiplier, and the resourcing corrections before sizing any bucket. + +## Output format + +```text +Estimate: +Confidence: High | Medium | Low Basis: + +Person-hours (PERT mean, with O / M / P): + A Design new coverage ( / /

) + B Manual execution ( / /

) + C Automated triage ( / /

) + D Automation authoring or 0 - + E Fixed overhead (%) + --------------------------------------------------- + Total 80% band - h + +Wall clock (not effort): + h for a full automated pass at parallel sessions x configurations + +Resourcing: + Fixed window working days -> testers at the mean, at P80 + Fixed team testers -> working days at the mean + Serialized + +Assumptions that move the number most: + 1. - a % miss moves the total by h + 2. ... + +Excluded from this estimate: + - - + +Evidence: + MCP ->

+ Team
-> +``` + +## Never do + +- Never emit a single number. Mean, band, and confidence label, always. +- Never quote more precision than the inputs carry. A rate of "about 20 minutes" produces "about 80 hours", not 81.1. +- Never turn person-hours into a date without stating productive hours per day and the serialized portion. +- Never treat automated elapsed time as person-effort, or a manual run's open duration as work. +- Never let missing platform data become zero. A bucket with no data is an unknown with a stated range. +- Never re-estimate to hit a date. If the number does not fit, cut scope or add people, re-run the model, and say which lever was pulled. +- Never invent production traffic, change frequency, defect rates, or team velocity the MCP does not return. If the human supplies them, use them and cite the source. + +## Worked example + +**Input, verbatim from the user** + +> How long will it take us to test release 3.2 of Cellphone Shop, and how many testers do I need? + +**Step 1, inputs with provenance.** Every line is tagged. Nothing untagged enters the model. + +| Input | Value | Source | +|---|---|---| +| Cycle window | 2026-08-17 to 2026-08-28, 10 working days | MCP `find_iterations` | +| In-scope requirements | 34, of which 26 have coverage and 8 have none | MCP `fetch_requirement_data` | +| Cases linked to covered requirements | 187 | MCP `find_test_cases_by_requirement` | +| Cases in the release folder | 214 (187 linked + 27 unlinked regression) | MCP `find_test_cases`, full page, not truncated | +| Automated / manual split | 96 / 118 | MCP `fetch_test_case_data` | +| Unstable cases, last 30 days | 11 | MCP `fetch_test_stability_data` | +| Automated pass rate, sprint 3.1 | 91% | MCP `read_execution_test_results` on the sprint 3.1 execution | +| Target configuration matrix | Chrome, Safari, Android Chrome. Automated covers all 3, manual covers Chrome only | MCP `fetch_test_configuration_data` | +| Sprint 3.1 automated elapsed | 41 min for 96 results | MCP `read_execution`, elapsed field confirmed present on the record | +| Session concurrency | 4 parallel TestCloud sessions | **Team-supplied.** TestCloud plan, not an MCP field | +| Manual execution rate | 18 min per case, median | **Team-supplied.** Measured on sprint 3.1 | +| Manual design rate | 25 min per new case | **Team-supplied.** Team declaration | +| New cases per uncovered requirement | 4 | **Team-supplied.** Team's own historical ratio | +| Triage rate | 20 min per failing result | **Team-supplied.** Team declaration | +| Productive hours per tester per day | 6 | **Team-supplied** | +| Fixed overhead | 15% | **Team-supplied** | +| Automation authoring this cycle | deferred | **Team decision** | + +**Step 2, the arithmetic, reproducible.** + +**A. Design new coverage.** 8 uncovered x 4 new cases = 32 cases. 32 x 25 min = 800 min = **13.3 h** most likely. O 10.0, P 22.0. +PERT = (10.0 + 4 x 13.3 + 22.0) / 6 = 85.2 / 6 = **14.2 h**. SD = (22.0 - 10.0) / 6 = **2.00 h**. + +**B. Manual execution, one pass, Chrome only.** 118 existing manual + 32 new = 150 cases. 150 x 18 min = 2700 min = **45.0 h** most likely. O 37.0, P 63.0. +PERT = (37.0 + 180.0 + 63.0) / 6 = 280.0 / 6 = **46.7 h**. SD = (63.0 - 37.0) / 6 = **4.33 h**. + +**C. Automated execution and triage.** 96 automated x 3 configurations = 288 results. At 91% pass, about 26 fail. 26 x 20 min = 520 min = **8.7 h** most likely. O 5.0, P 18.0 - the pessimistic branch is wide because the 11 unstable cases run on all 3 configurations and are the likeliest contributors. +PERT = (5.0 + 34.8 + 18.0) / 6 = 57.8 / 6 = **9.6 h**. SD = (18.0 - 5.0) / 6 = **2.17 h**. +Wall clock, tracked separately and never added to person-hours: 3 x 41 min = 123 min = **2.1 h** at 4 parallel sessions. + +**D. Automation authoring.** **0 h**, excluded by team decision this cycle. Recorded, not dropped. + +**E. Fixed overhead.** 15% of (14.2 + 46.7 + 9.6 = 70.5) = **10.6 h**. + +**Total.** 70.5 + 10.6 = **81.1 h**. +Combined SD = sqrt(2.00^2 + 4.33^2 + 2.17^2) = sqrt(27.46) = 5.24 h, scaled by the 15% overhead = **6.03 h**. +80 percent band = 81.1 +/- 1.28 x 6.03 = 81.1 +/- 7.7, rounded outward = **73 to 89 person-hours**. + +**Expected output** + +```text +Estimate: Cellphone Shop, release 3.2 test cycle +Confidence: Medium Basis: manual rate measured on sprint 3.1 in this project, n=1 + +Person-hours (PERT mean, with O / M / P): + A Design new coverage 14.2 h (10.0 / 13.3 / 22.0) + B Manual execution 46.7 h (37.0 / 45.0 / 63.0) + C Automated triage 9.6 h ( 5.0 / 8.7 / 18.0) + D Automation authoring 0.0 h excluded - team deferred authoring this cycle + E Fixed overhead (15%) 10.6 h + --------------------------------------------------- + Total 81.1 h 80% band 73 - 89 h + +Wall clock (not effort): + 2.1 h for a full automated pass at 4 parallel sessions x 3 configurations + +Resourcing: + Fixed window 10 working days -> 1.4 testers at the mean, 1.5 at P80 + Fixed team 2 manual testers -> 7 working days at the mean + Serialized triage cannot start before execution ends - reserve the last 2 days + +Assumptions that move the number most: + 1. Manual execution rate, 18 min per case - a 25% miss moves the total by 11.7 h, + which is wider than the entire design bucket's 80% band + 2. New cases per uncovered requirement, 4 - a 50% miss moves the total by 7.1 h + 3. Session concurrency, 4 - halving it doubles wall clock but changes person-hours by 0 + +Excluded from this estimate: + - Automation authoring - team deferred it this cycle + - Manual execution on Safari and Android Chrome - matrix currently Chrome only + - Any money figure - the platform holds no rate + +Evidence: + MCP fetch_requirement_data -> 34 in scope, 8 uncovered + MCP find_test_cases -> 214 cases in the release folder + MCP fetch_test_case_data -> 96 automated / 118 manual + MCP fetch_test_stability_data -> 11 unstable in 30 days + MCP read_execution_test_results -> 91% pass on sprint 3.1 + MCP fetch_test_configuration_data -> 3 target configurations + Team 18 min per manual case -> measured by the team on sprint 3.1 + Team 4 parallel sessions -> TestCloud plan, not an MCP field +``` + +**What the reader should take from it.** The listed scope needs **2 manual testers, not 3**. The third tester is not funding the listed scope; the exposure sits in the two exclusions. Sizing the first one on request - adding Safari to the manual lane at a team-supplied 0.7 repeat-pass factor is 150 x 18 min x 0.7 = 31.5 h, which with overhead takes the total from 81 h to about 117 h. That is 19.6 person-days instead of 13.5, so sustained headcount goes from 1.4 to 2.0 and the cycle costs **about 6 extra person-days**. Android Chrome and the P80 branch are what the third tester actually covers. The point is not the number, it is that the third seat is now a decision with a price on it rather than a habit. + +## Prompt recipes + +- `How long will it take us to test release 3.2, and how many people do I need?` +- `If we add the Safari and Android configurations, what does that cost us in tester days?` +- `We have 2 testers and the sprint ends on the 28th. Does the release 3.2 scope fit, and what would I have to cut?` +- `Size the testing effort for the 8 uncovered requirements in Cellphone Shop, and tell me which assumption you are least sure about.` + +## Hand-offs + +- Scope not yet fixed or ranked -> `katalon-test-plan`. Estimate only what that skill has already scoped. +- The plan of record for the estimated scope -> `katalon-test-plan`, whose folder + suite is where the estimate should be recorded, since the platform has no field for it. +- The uncovered requirements this estimate sized -> `katalon-create-test-cases`. +- Automation authoring effort, when it is in scope -> `katalon-test-case-to-playwright` for the per-case shape. +- Flake burden behind the triage bucket -> `katalon-test-maintenance`, and `katalon-analyze-failures` for a specific red run. +- The verdict once the estimated cycle has run -> `katalon-release-analyze`. + +Read `references/estimation-model.md` before sizing any bucket, and `references/mcp-evidence-map.md` before claiming any figure came from the platform. Read the orchestrator `katalon-true-platform-testing/references/unavailable-capabilities.md` when the user asks whether Katalon can hold the estimate itself. diff --git a/skills/katalon-test-estimation/references/estimation-model.md b/skills/katalon-test-estimation/references/estimation-model.md new file mode 100644 index 000000000..96e492162 --- /dev/null +++ b/skills/katalon-test-estimation/references/estimation-model.md @@ -0,0 +1,105 @@ +# Estimation model reference + +The method behind `katalon-test-estimation`. Five buckets, three points each, one combined band. + +## The five buckets + +Most people say "estimate the test cycle" and mean bucket B alone. Sizing only B is the most common way a test estimate comes in low. + +| # | Bucket | Unit | Countable input | Rate source | +|---|---|---|---|---| +| A | **Design** new coverage | person-hours | uncovered requirements x cases per requirement | team, minutes per new case | +| B | **Manual execution** | person-hours | manual case count x configurations in the manual matrix | team, minutes per case | +| C | **Automated execution and triage** | wall clock + person-hours | automated case count x configurations; failure rate | platform for elapsed, team for triage minutes per failing result | +| D | **Automation authoring** | person-hours | cases selected for automation | team, hours per case | +| E | **Fixed overhead** | percentage of A+B+C+D | none | team, typically 10 to 20 percent | + +Rules: + +- A bucket that is out of scope is **recorded as zero with the reason**. Silence reads as forgetting. +- Rework is not a sixth bucket. Retest of failures lives inside B, triage of failures inside C. Modelling it separately double-counts. +- Exploratory or session-based testing, when the cycle includes it, is timeboxed rather than estimated. Add the timebox as a stated line, not as a derived figure. + +## Three-point arithmetic + +For each bucket collect optimistic `O`, most likely `M`, pessimistic `P`. + +```text +PERT mean E = (O + 4M + P) / 6 +Deviation SD = (P - O) / 6 +``` + +Combine independent buckets: + +```text +Total mean = sum of the bucket means +Total SD = sqrt( SD_A^2 + SD_B^2 + SD_C^2 + SD_D^2 ) +``` + +Overhead E is a percentage of the rest, so scale the combined SD by the same factor rather than giving it its own three points. + +Report the **80 percent band** as `mean +/- 1.28 x SD`, rounded outward to whole hours. Use 90 percent (`1.64 x SD`) only when the ask is a commitment date rather than a plan figure, and say which multiplier you used. + +Setting the three points honestly: + +- `M` comes from count x rate. It is the only one with arithmetic behind it. +- `O` assumes nothing goes wrong and no rework. Usually 0.8 x M. +- `P` is the one that carries the information. Set it from a named risk, not a multiplier - "the 8 uncovered requirements turn out to need 7 cases each, not 4". If you cannot name the risk, `P` is 1.5 x M and confidence cannot be High. + +## Lane units, and why they must not be added + +| Lane | What consumes it | Unit | What shortens it | +|---|---|---|---| +| Manual | a person, per case, per configuration | person-hours | more people, until design and triage serialize | +| Automated | machines, bounded by session concurrency | wall clock | more parallel sessions - and this changes person-hours by zero | +| Automated, human side | triage of failing results | person-hours | a lower failure rate, or fewer flaky cases | +| Run with AI | platform wall clock, plus a human reviewing the AI session | wall clock + person-hours | nothing the estimator controls; treat the wall clock as fixed and estimate only the review | + +**Never add wall clock to person-hours.** They answer different questions - one is "will it fit in the window", the other is "how many people do I need". + +## Rework multiplier + +Failures generate two costs, and both are already inside a bucket: + +```text +failing results = total results x (1 - pass rate) +triage hours = failing results x triage minutes / 60 -> bucket C +manual retest = failing manual cases x execution minutes / 60 -> bucket B +``` + +Flaky cases inflate the failing count without indicating product defects. Use the unstable-case count as the driver of `P` rather than of `M`, since a flake may or may not fire on any given run. + +## Resourcing arithmetic + +```text +person-days = person-hours / productive hours per day +Fixed window testers = person-days / working days in the window +Fixed team days = person-days / testers +``` + +Two corrections that stop the naive division from lying: + +1. **Serialization.** Triage and reporting cannot start before execution ends. Subtract that tail from the window before dividing, and state it. +2. **Parallelism limits.** Manual execution parallelizes well across cases. Design parallelizes to roughly two people before coordination cost exceeds the gain. Triage of one suite rarely parallelizes at all. Adding people to a late cycle moves B and almost nothing else. + +Report both directions - headcount for a fixed date, and a date for a fixed team - because the asker usually has one of them fixed and has not said which. + +## Confidence rubric + +| Label | Conditions | Typical band width | +|---|---|---| +| **High** | The per-unit rate was measured in this project on a comparable cycle within the last 90 days, scope is frozen, and the platform returned a count for every bucket | under +/- 15% | +| **Medium** | The rate is team-declared, or measured on a different project or an older cycle, or one bucket's count is an estimate rather than a platform figure | +/- 15 to 35% | +| **Low** | No measured rate anywhere, or scope is still moving, or the platform returned nothing for a bucket that matters | wider than +/- 35% | + +At **Low**, deliver the model and the shape but refuse the headline number, and name the one measurement that would move it to Medium. That is usually "time 20 real cases". + +## Sensitivity + +Find the dominant term before publishing. For each team-supplied rate, recompute the total with the rate 25 percent higher and report the delta in hours. Rank the drivers and print the top two or three. + +The dominant term is almost always the largest bucket's rate. If a 25 percent miss on one input moves the total by more than the entire 80 percent band, that input is the estimate - say so, and ask for it to be measured before the number is used to commit to anything. + +## Re-estimation + +An estimate is a snapshot of a scope. When scope changes, re-run the model rather than adjusting the number, and publish the delta with the driver named: "+31.5 h, adding Safari to the manual lane". Never quietly revise a published estimate downward to fit a date; state which lever was pulled - scope, people, or window. diff --git a/skills/katalon-test-estimation/references/mcp-evidence-map.md b/skills/katalon-test-estimation/references/mcp-evidence-map.md new file mode 100644 index 000000000..f67e759ef --- /dev/null +++ b/skills/katalon-test-estimation/references/mcp-evidence-map.md @@ -0,0 +1,56 @@ +# MCP evidence map for estimation + +Every figure in an estimate is either a platform fact or a human input. This file draws the line. Tag every line of the output with which side it came from. + +## Platform side - what the MCP actually returns + +| Estimation input | Tool | What it gives you | Trap | +|---|---|---|---| +| Cycle window | `find_iterations` | sprint/iteration start and end | Calendar days, not working days. Subtract holidays yourself | +| Uncovered requirements | `fetch_requirement_data` | coverage status per requirement | "Covered" means linked, not passing. A linked-but-never-run requirement still costs execution time | +| Case counts | `find_test_cases`, `read_test_suite`, `find_test_cases_by_requirement` | the cases in a folder, suite, or requirement link | Check for pagination. A truncated page reported as a total silently shrinks the estimate | +| Automated / manual split | `fetch_test_case_data` | quality signals including the split when exposed | Not guaranteed present. If absent, ask, and tag it team-supplied | +| Failure and flake burden | `fetch_test_stability_data`, `find_test_results` | unstable cases, recent pass and fail history | A flaky case inflates the failing count without being a defect. Drive `P`, not `M`, with it | +| Configuration matrix | `fetch_test_configuration_data` | which environments, browsers, and devices are covered against the target set | The multiplier applies per lane. Automated may cover three configurations while manual covers one | +| Past automated elapsed time | `read_execution`, `read_execution_test_results` | a finished execution's records, including machine-recorded elapsed time | **Confirm the field is on the record before modelling on it.** Version and runner dependent. If it is absent, ask for the figure | +| Comparable cycle for calibration | `read_execution` on a named past execution | one sample | It is n=1. There is no aggregate-across-cycles tool. Say n=1 in the confidence basis | + +Discovery first, always: `list_projects` then `list_repositories`, exactly as `katalon-test-plan` does. + +## Human side - the checklist + +None of these exist in the platform. Ask for all of them that the chosen buckets need, and record who supplied each and when it was last measured. + +- Manual execution minutes per case (median, not mean - a few long cases skew the mean) +- Design minutes per new case +- New cases per uncovered requirement +- Triage minutes per failing result +- Automation authoring hours per case, if bucket D is in scope +- Productive hours per tester per day +- Number of testers available, and any part-time fractions +- Working days lost to holidays or PTO inside the window +- Fixed-overhead percentage +- Session concurrency for the automated lane +- Repeat-pass factor, if a configuration is being run a second time +- Any cost rate, if the answer must be in money + +## Rate source ranking + +1. **Measured in this project on a comparable cycle within 90 days.** The only source that supports High confidence. +2. **Team declaration.** Medium. Ask what it is based on; a number someone remembers is weaker than one someone tracked. +3. **A different project, or an older cycle.** Medium, and say which project. +4. **An industry default.** Low. Print it as a placeholder, name it as a placeholder, and ask for a real measurement. + +## Things the platform cannot hold + +- **No effort field.** Nothing books time against a case, run, or person. +- **No roster or capacity.** No team members, FTE fractions, PTO, or working calendar. +- **No cost model.** No rates, currency, license consumption, or TestCloud minute pricing. +- **No place to store the estimate.** No estimated-effort field on a case, folder, suite, or requirement, and per `unavailable-capabilities.md` no Test Plan, Release, or Build entity to attach one to. Record it in the plan suite's description or the linked Jira/Azure issue, and hand off to `katalon-test-plan`. +- **No cross-cycle aggregate.** Result and execution tools are record-level. Trending across releases belongs to a reporting skill that does not exist yet - state the boundary rather than assembling a trend from single reads and presenting it as one. + +## Things that look like MCP facts and are not + +- **Session concurrency.** `list_test_cloud_environments` lists environments; `find_execution_profiles` lists profiles. Neither returns how many sessions run at once. +- **A manual run's duration.** Open and close timestamps are calendar elapsed. A run left open over a weekend is not a weekend of work. +- **Production traffic, change frequency, and defect rates.** Not returned by any tool. If the human supplies them from TrueTest Test Gap Analysis, an analytics product, or Jira, use them and cite the source - the same rule `katalon-test-plan` already applies. diff --git a/skills/katalon-test-maintenance/SKILL.md b/skills/katalon-test-maintenance/SKILL.md new file mode 100644 index 000000000..803c3cd7d --- /dev/null +++ b/skills/katalon-test-maintenance/SKILL.md @@ -0,0 +1,54 @@ +--- +name: katalon-test-maintenance +description: Maintain and evolve a Katalon True Platform/TestOps regression suite as the application changes. Use when you need to detect which tests broke or became flaky from stability and result history, diagnose whether a case needs repair vs regeneration, repair test assets (update, move, reorganize cases), refresh coverage after application or requirement changes, and feed the resulting gap list back into planning. Closes the lifecycle loop. For classifying a specific run's failures use katalon-analyze-failures; for authoring brand-new cases use katalon-create-test-cases. Written for the automation tester repairing a suite that has started lying, and the test lead who has to keep it trustworthy. +--- + +# Katalon Test Maintenance + +Use this skill for the **maintenance** stage: keep the regression suite healthy over time and feed learning back to the plan. It consumes failure/stability signals and produces repairs plus a refreshed coverage gap list. + +## Availability Boundary + +- **Available via MCP:** change/impact signals (`fetch_test_stability_data`, `find_test_results`, `read_execution`), and case-level repair (`update_test_case`, `move_test_case`, `duplicate_test_case`). +- **Not directly available:** Self-healing tests, Time Capsule, Studio Tracer, object refactoring, and StudioAssist Agent-mode edits are **Studio-desktop** features. TrueTest regeneration from live journeys is a **TrueTest** surface. Narrate these; MCP repairs are case-level content edits, not object/script self-healing. + +## Maintenance Workflow + +```text ++---------------------+ +----------------------+ +----------------------+ +| Detect change impact| --> | Diagnose: repair vs | --> | Repair case assets | +| stability + history | | regenerate vs retire | | update/move/dup | ++---------------------+ +----------------------+ +----------------------+ + | + v + +----------------------+ + | Validate + feed back | + | to plan (gap list) | + +----------------------+ +``` + +## Steps and tool rules + +1. **Detect impact.** `fetch_test_stability_data` for flakiness trend; `find_test_results` + `read_execution` for what changed across recent runs. Build the list of broken / flaky / degraded cases. +2. **Diagnose each.** Decide per case: + - **Repair** — the case is salvageable: fix data, steps, or expected result via `update_test_case`; reorganize via `move_test_case`; branch a variant via `duplicate_test_case`. + - **Regenerate** — behavior changed enough that the case should be re-derived (TrueTest regeneration / re-author) -> hand to `katalon-create-test-cases`; narrate the TrueTest boundary. + - **Retire** — behavior removed: unlink/retire (prefer update/flag over unreliable `delete_test_case`); get approval. +3. **Repair (MCP-legal edits only).** Pass all intended updates in one `update_test_case` call. Do not claim to have self-healed locators/objects — that is Studio. +4. **Validate.** Re-run through `katalon-execute-test` and confirm via `find_test_results` / `read_execution`. +5. **Feed back.** Emit the refreshed coverage gap list to `katalon-test-plan` so maintenance closes the loop instead of dead-ending. + +## Prompt recipes + +- `Which regression cases became flaky this month, and which should we repair vs regenerate?` +- `Repair the login suite after the new auth flow: update the affected cases and tell me what still needs re-authoring.` +- `After this release's UI change, refresh coverage and give me the gap list for next sprint.` + +## Hand-offs + +- Classify a specific failed run first -> `katalon-analyze-failures`. +- Re-author changed behavior -> `katalon-create-test-cases`. +- Confirm repairs -> `katalon-execute-test` then `katalon-test-review`. +- Refreshed gaps -> `katalon-test-plan`. + +Read `references/maintenance-loop.md` before repairing. Consult the orchestrator's `references/unavailable-capabilities.md` for the Studio/TrueTest boundary. diff --git a/skills/katalon-test-maintenance/references/maintenance-loop.md b/skills/katalon-test-maintenance/references/maintenance-loop.md new file mode 100644 index 000000000..046c53195 --- /dev/null +++ b/skills/katalon-test-maintenance/references/maintenance-loop.md @@ -0,0 +1,41 @@ +# Maintenance loop reference + +## Detect -> diagnose -> repair -> validate -> feed back + +```text +stability/history -> per-case decision -> MCP-legal edit -> re-run -> gap list -> plan +``` + +## Per-case decision table + +| Signal | Decision | MCP action | +|---|---|---| +| passes on re-run, no app change | flaky -> stabilize | `update_test_case` (waits/data); flag for Studio object review | +| consistent fail, app behavior changed | regenerate | hand to `katalon-create-test-cases`; narrate TrueTest regeneration | +| step/element gone from app | repair or retire | `update_test_case` to match new flow; retire with approval | +| duplicate/near-duplicate drift | consolidate | `duplicate_test_case` base + `move_test_case`; keep one canonical | +| wrong folder/suite after refactor | reorganize | `move_test_case`, `manage_test_suite` | + +## Boundaries to narrate (never claim as MCP) + +- Self-healing locators, Time Capsule object repair, Studio Tracer, object-repository refactor = **Studio desktop**. +- TrueTest regeneration from live user journeys = **TrueTest** surface. +- MCP maintenance = **test-case content edits** (title, steps, expected results, data, links, location) + reorganization. + +## Closing the loop + +Maintenance is not done at "green again." Emit: + +```text +Repaired: +Regenerate: +Retired: +New gaps: -> katalon-test-plan +Stability: +``` + +## Safe-mutation rules + +- Prefer `update_test_case` / `move_test_case` over `delete_test_case` (delete unreliable). +- One `update_test_case` call carries all intended edits for a case. +- Approve bulk changes and any retire/delete before executing. diff --git a/skills/katalon-test-management/SKILL.md b/skills/katalon-test-management/SKILL.md new file mode 100644 index 000000000..b49952e74 --- /dev/null +++ b/skills/katalon-test-management/SKILL.md @@ -0,0 +1,50 @@ +--- +name: katalon-test-management +description: Organize, classify, and trace Katalon True Platform/TestOps test assets. Use when you need to structure test cases into folders and suites, move or reorganize cases, search and find existing assets at scale, link or unlink requirements to test cases, or produce a requirement-to-test traceability report (which requirements have coverage, which cases are orphaned, coverage percentage). Prefer this skill for inventory hygiene and traceability audits. For authoring new cases use katalon-create-test-cases; for coverage quality verdicts use katalon-test-review. Written for the test lead doing inventory hygiene on a repository nobody has curated in months. +--- + +# Katalon Test Management + +Use this skill for the **management** stage: keep the test inventory organized, findable, and fully traceable to requirements. Traceability is the platform's differentiator, so the headline output is a **requirement <-> test case <-> suite** map, not just folder tidying. + +## Availability Boundary + +- **Available via MCP:** folders (`find_test_folders`, `manage_test_folder`), suites (`find_test_suites`, `manage_test_suite`, `read_test_suite`), case organization (`move_test_case`, `duplicate_test_case`, `delete_test_case`), search (`find_test_cases`), and requirement links (`link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`, `fetch_requirement_data`). +- **Not directly available:** custom fields and tags authoring, dynamic test suites, Git repository configuration, script-repo upload, and project governance/notifications — these are TestOps-UI or Studio operations. Name them as boundaries; do not claim MCP writes for them. +- **Naming charset:** test case names allow only letters, numbers, spaces and `( ) . , _ -`. Folder paths allow `/`. Prefer `update`/`move` over delete-and-recreate (`delete_test_case` has been unreliable). + +## Traceability Workflow + +```text ++---------------------+ +-----------------------+ +----------------------+ +| Read requirements | --> | Map cases <-> reqs | --> | Find orphans + gaps | ++---------------------+ +-----------------------+ +----------------------+ + | | | + v v v ++---------------------+ +-----------------------+ +----------------------+ +| Link/unlink to fix | --> | Organize into suites | --> | Report traceability | ++---------------------+ +-----------------------+ +----------------------+ +``` + +## Steps and tool rules + +1. **Establish scope.** `list_projects` -> `list_repositories`. Resolve one target from context. +2. **Build the trace map.** For each requirement key, `find_test_cases_by_requirement`; use `fetch_requirement_data` for coverage status. Classify each requirement as: covered, partially covered, or orphan (no case). +3. **Find loose assets.** `find_test_cases` by feature/folder/keyword to surface cases with no requirement link (orphan cases) and duplicates. +4. **Fix links (only when known and requested).** `link_requirements_to_test_case` after requirement IDs are confirmed; `unlink_requirements_from_test_case` for wrong links. Never guess a link. +5. **Organize inventory.** `manage_test_folder` and `move_test_case` for structure; `manage_test_suite` to group runnable cases. Reuse before creating; summarize bulk moves and get approval before executing them. +6. **Report the traceability matrix.** requirement -> case(s) -> suite(s), coverage %, orphan requirements, orphan cases, and duplicate candidates. This is the deliverable. + +## Prompt recipes + +- `Audit requirement-to-test traceability for project X and list every requirement with no test case.` +- `Reorganize the smoke suite: move all P0 login cases into Test Cases/Auth/Smoke and report the new structure.` +- `Link test cases TC-1042 and TC-1043 to requirement CEL-6 and confirm the coverage.` +- `Find duplicate test cases in the Checkout folder and propose which to keep.` + +## Hand-offs + +- Coverage gaps found -> `katalon-create-test-cases` (author) or `katalon-test-plan` (schedule). +- Quality verdict on the organized suite -> `katalon-test-review`. + +Read `references/traceability.md` before running an audit. Consult the orchestrator's `references/unavailable-capabilities.md` for the full boundary list. diff --git a/skills/katalon-test-management/references/traceability.md b/skills/katalon-test-management/references/traceability.md new file mode 100644 index 000000000..da4360db0 --- /dev/null +++ b/skills/katalon-test-management/references/traceability.md @@ -0,0 +1,35 @@ +# Traceability reference + +## The trace map (the deliverable) + +```text +Requirement -> Test case(s) -> Suite(s) Coverage +CEL-6 TC-1042, TC-1043 Auth/Smoke covered +CEL-7 TC-1050 (none) covered, not in a suite +CEL-8 (none) (none) ORPHAN requirement +— TC-1099 (none) ORPHAN case (no requirement link) +``` + +Coverage % = requirements with >=1 linked case / total in-scope requirements. + +## Tool sequence for an audit + +1. `find_requirements` (or the supplied keys) — the in-scope requirement set. +2. `find_test_cases_by_requirement` per key — the linked cases. +3. `fetch_requirement_data` — platform coverage status to reconcile against the link map. +4. `find_test_cases` by folder/feature — surface cases that never appear in step 2 (orphan cases). +5. Diff: requirements with no case = orphan requirements; cases with no requirement = orphan cases. + +## Classification rules + +- **Covered:** requirement has >=1 linked, current test case. +- **Partially covered:** linked cases exist but leave known behavior untested (defer the judgment to `katalon-test-review`). +- **Orphan requirement:** no linked case — the top gap; route to planning/design. +- **Orphan case:** case with no requirement link — either link it (if a requirement is known) or flag for review; do not delete without approval. + +## Safe-mutation rules + +- Prefer `update_test_case` / `move_test_case` over `delete_test_case` (delete has been unreliable, HTTP 500 observed). +- Test case name charset: letters, numbers, spaces, `( ) . , _ -` only. No `@ : /` in titles. +- Summarize any bulk move/link/delete and get approval before executing. +- Never fabricate a requirement<->case link; link only confirmed IDs. diff --git a/skills/katalon-test-plan/SKILL.md b/skills/katalon-test-plan/SKILL.md new file mode 100644 index 000000000..e47429802 --- /dev/null +++ b/skills/katalon-test-plan/SKILL.md @@ -0,0 +1,53 @@ +--- +name: katalon-test-plan +description: Plan Katalon True Platform/TestOps testing for a release, sprint, or feature. Use when you need to translate quality goals into scope, prioritize testing by requirement coverage and risk, decide what to test first, or build the executable plan structure (folders, suites, and sprint/release association) that stands in for a formal Test Plan. Reads project/repository/iteration context and requirement coverage, then proposes and materializes a prioritized plan. For designing the actual test cases, hand off to katalon-create-test-cases; for the ship/no-ship call, hand off to katalon-release-analyze. Written for the test lead who owns the cycle and has to decide what gets tested first. +--- + +# Katalon Test Plan + +Use this skill for the **planning** stage of the Katalon lifecycle: turn quality goals into a scoped, risk-prioritized, executable plan. The output is a plan the team can act on plus the folder/suite/release structure that makes it runnable. Prefer Katalon MCP tools for platform data; use the human as a tool when scope, risk appetite, or release target cannot be resolved safely. + +## Availability Boundary + +State the boundary before promising a plan: + +- **Available via MCP:** resolve scope (`list_projects`, `list_repositories`, `find_iterations`), read requirement coverage and gaps (`fetch_requirement_data`, `find_requirements`, `find_test_cases_by_requirement`), and build the executable plan structure (`manage_test_folder`, `manage_test_suite`, add cases to suites). +- **Not directly available:** create a Release or Build entity, create a formal Test Plan entity, or author release gates through MCP. These live in the TestOps UI. +- **Workaround (the MCP-legal test plan):** a **named folder + test suite associated with a sprint/release** is the executable plan. Create it, populate it, and report it as the plan of record. + +## Planning Workflow + +```text ++---------------+ +----------------------+ +----------------------+ +| Resolve scope | --> | Read coverage + risk | --> | Prioritize what first| ++---------------+ +----------------------+ +----------------------+ + | | | + v v v ++---------------+ +----------------------+ +----------------------+ +| Propose plan | --> | Build folder + suite | --> | Associate sprint/rel | ++---------------+ +----------------------+ +----------------------+ +``` + +## Steps and tool rules + +1. **Resolve scope.** `list_projects` -> `list_repositories`. If exactly one matches the user's wording or context, use it. Use `find_iterations` to bind the plan to a sprint/release when named. +2. **Read coverage and gaps.** Use `fetch_requirement_data` for requirement coverage status and `find_test_cases_by_requirement` to see which requirements already have cases. `find_requirements` / `read_requirement` when requirement keys are supplied. +3. **Prioritize by risk and evidence.** Rank scope using: requirement coverage gaps first, then high-change/high-traffic areas, then areas with historically unstable tests (defer stability detail to `katalon-test-review`). State the risk basis; do not invent traffic numbers the MCP does not return. +4. **Propose the plan in chat.** Objectives, in-scope / out-of-scope, coverage matrix (requirement -> planned cases), risk ranking, target environments, and the suite/folder structure you will create. +5. **Materialize the executable structure.** `manage_test_folder` for the plan folder, `manage_test_suite` for the runnable suite, add the selected/known cases. Reuse an existing folder/suite when one already matches instead of creating a duplicate. +6. **Verify and report.** `read_test_suite` to confirm; report the plan of record with the suite as the executable artifact and the open gaps to fill next. + +## Prompt recipes + +- `Plan testing for sprint 3.2 in project Cellphone Shop: show coverage gaps and build the executable suite.` +- `What should we test first for release R-2026-Q3 based on requirement coverage? Propose a risk-ranked plan.` +- `Create a test plan structure (folder + suite) for requirement CEL-6 and associate it with the current sprint.` + +## Hand-offs + +- Gaps to fill -> `katalon-create-test-cases`. +- Plan ready to run -> `katalon-execute-test`. +- Ship decision on the planned scope -> `katalon-release-analyze`. +- Traceability audit of the plan -> `katalon-test-management`. + +Read `references/planning-workflow.md` before proposing scope. Read the orchestrator `katalon-true-platform-testing/references/unavailable-capabilities.md` when the user asks "can Katalon plan X?". diff --git a/skills/katalon-test-plan/references/planning-workflow.md b/skills/katalon-test-plan/references/planning-workflow.md new file mode 100644 index 000000000..e959e00eb --- /dev/null +++ b/skills/katalon-test-plan/references/planning-workflow.md @@ -0,0 +1,44 @@ +# Planning workflow reference + +## Scope resolution order + +1. Project (`list_projects`) — one match from context, else ask. +2. Repository / Test Project (`list_repositories`) — treat repository and Test Project as the same target; prefer `Katalon Cloud` when the user says "cloud repo". +3. Iteration / sprint / release (`find_iterations`) — bind the plan when the user names a sprint or release. + +## Risk-prioritization inputs (only what MCP returns) + +| Signal | Tool | Use | +|---|---|---| +| Requirement coverage status | `fetch_requirement_data` | Untested / partially-covered requirements rank first. | +| Requirement -> case links | `find_test_cases_by_requirement` | Orphan requirements (no case) are the top gap. | +| Configuration coverage | `fetch_test_configuration_data` | Under-covered browsers/platforms are planning risk. | +| Test stability | `fetch_test_stability_data` | Unstable areas need re-planning, not just re-run (defer detail to katalon-test-review). | + +Do not fabricate production traffic, change frequency, or defect rates the MCP does not expose. If the user supplies them (from TrueTest Test Gap Analysis, PostHog, or Jira), use them and cite the source. + +## The executable "test plan" structure + +Because MCP cannot create a Test Plan entity, the plan of record is: + +```text +Folder: Test Cases// +Suite: — (manage_test_suite) + + selected test cases added +Assoc: sprint/release via find_iterations context +``` + +Report the suite ID as the plan artifact. List the requirement->case coverage matrix and the open gaps as the plan's backlog. + +## Plan output template + +```text +Objective: +In scope: +Out of scope: +Coverage matrix: requirement -> planned/existing cases (mark gaps) +Risk ranking: 1..n with the signal that drove each rank +Environments: +Executable plan: folder + suite (IDs) associated to +Next: gaps to fill (-> katalon-create-test-cases) +``` diff --git a/skills/katalon-test-reporting/SKILL.md b/skills/katalon-test-reporting/SKILL.md new file mode 100644 index 000000000..660b3c0c6 --- /dev/null +++ b/skills/katalon-test-reporting/SKILL.md @@ -0,0 +1,202 @@ +--- +name: katalon-test-reporting +description: Report Katalon True Platform/TestOps quality metrics to people outside QA. Use when you need to answer a stakeholder question with testing data, choose the few metrics that actually answer it, trend coverage, execution health, defect risk and stability across several releases, sprints, or iterations rather than inside one, and write the summary a manager presents upward at a steering committee, an exec review, or a quality business review. Produces a headline answer with the named specifics behind it, never a metric dump. The MCP returns current state with no history tool, so trends are built from execution dates plus snapshots this skill stores per period. For the ship or no-ship call on a single release, use katalon-release-analyze; for a pre-pipeline suite verdict, use katalon-test-review. Written for the test manager who has to present quality upward and the test lead who assembles the numbers. +--- + +# Katalon Test Reporting + +Use this skill when someone outside QA asks a question about quality and the answer has to survive being asked where it came from. It reads the **analyze** stage and feeds the **plan** stage. The output is an **answer with the named specifics behind it**, never a metric dump. `katalon-release-analyze` judges one release; this skill reports the direction across several and writes what a manager says out loud. + +## Availability Boundary + +State the boundary before promising a report, because most of what "reporting" normally means is not an MCP call. + +- **Available via MCP.** The metric surface: `fetch_requirement_data` (coverage status), `fetch_test_case_data` (case quality signals), `fetch_test_stability_data` (flakiness), `fetch_test_configuration_data` (configuration coverage), `fetch_defect_data` (defect status and context), `find_test_results` / `read_execution` / `read_execution_test_results` (execution history, and the only data that carries its own dates), `find_iterations` (the period spine), and `list_projects` / `list_repositories` / `find_test_suites` / `read_test_suite` / `find_test_cases` to establish what each number is counted over. +- **Not directly available, and no tool exists to add.** + - **No trend or aggregation tool.** Nothing in the MCP returns a series, an average, or a delta. Every trend here is assembled by looping periods yourself. + - **No point-in-time read.** Every `fetch_*_data` tool returns *current* state. You cannot ask what requirement coverage was at release 3.0. Only execution records carry dates. + - **No dashboard, chart, or report entity.** Dashboards are a TestOps UI surface. This skill writes text and a snapshot file, nothing else. + - **No publishing surface.** The MCP cannot post to Slack, Confluence, email, or a deck. Hand the written brief back to the user and let them place it. + - **No Release or Build entity.** Bind periods to iterations via `find_iterations`, or to explicit date ranges. Never to a release object. + - **No escaped-defect flag.** `fetch_defect_data` returns status and context, not whether a defect reached production. Ask the user which ALM field classifies it. If there is none, report "defects opened after the release run" and label it as a proxy. + - **No effort, cost, headcount, or cycle-time data.** That is the `katalon-test-estimation` gap. Say so and stop. Never estimate effort from test counts. + - **No custom fields or tags.** Any team dimension (component, squad, product line) has to come from folder or suite structure, or from the user. +- **The workaround that makes trending real.** A **snapshot file** written on every run, plus the execution-dated series that needs no snapshot. See `references/snapshot-and-trend.md`. The first run establishes a baseline and says so instead of inferring a direction. + +## Reporting Workflow + +```text ++----------------------+ +----------------------+ +----------------------+ +| Get the question | --> | Choose <= 5 metrics | --> | Pull per period | +| and the audience | | that answer it | | iterations or dates | ++----------------------+ +----------------------+ +----------------------+ + | + v ++----------------------+ +----------------------+ +----------------------+ +| Write the brief | <-- | Name what moved it | <-- | Compare to snapshot | +| answer + specifics | | reqs, suites, defects| | then write a new one | ++----------------------+ +----------------------+ +----------------------+ +``` + +## Steps and tool rules + +1. **Get the question and the audience before calling a tool.** Who is asking, what decision are they making, and what will they do differently depending on the answer. If the user says only "a status report", ask which of the four questions in `references/metric-selection.md` they are actually being asked. A report written without a question is a metric dump by construction, and no amount of formatting fixes it later. +2. **Choose at most five metrics.** Pick from `references/metric-selection.md` against the question, and justify each in one clause. Everything the question does not need is cut, however cheap it is to fetch. Five is a hard cap, not a target. +3. **Fix the period spine.** `find_iterations` gives sprints and iterations with dates. Otherwise use explicit date ranges. Three to six periods; fewer than three cannot show a direction. State the spine in the report, because two different spines produce two different trends from the same data. +4. **Pull per period.** Resolve scope with `list_projects` then `list_repositories`. For execution-dated metrics, call `find_test_results` and `read_execution` once per period. For current-state metrics, call each `fetch_*_data` tool **once**, they have no period parameter, and calling them repeatedly returns the same answer. +5. **Compare against the snapshot, then write a new one.** Read the prior snapshot for this project and question. Current-state metrics can only be trended against it. If none exists, this run is the baseline. Write the new snapshot before writing the brief so a crash mid-report still leaves the next run better off. +6. **Name what moved the number.** Every trend line gets a named cause: a requirement key, a suite name, a defect id, a configuration. If you cannot name one, report the number as unexplained and say so. A trend with no named cause is not an answer to anything. +7. **Write the brief in the format below**, and hand it back as text. Do not claim to have published it anywhere. + +## Choosing the few metrics + +The question decides the metrics. Short map here; full definitions, exact derivation from each MCP return, and the anti-patterns are in `references/metric-selection.md`. + +| The stakeholder actually asked | Report these, nothing else | From | +|---|---|---| +| Is quality getting better or worse | pass-rate trend, open critical defect trend, flaky-case count trend | `find_test_results`, `read_execution`, `fetch_defect_data`, `fetch_test_stability_data` | +| Are we testing the right things | requirement coverage percent, uncovered critical requirements **by key**, configuration coverage against the target matrix | `fetch_requirement_data`, `find_test_cases_by_requirement`, `fetch_test_configuration_data` | +| Can we trust the results we are being shown | flaky-case count and trend, cases with erratic recent history, share of failures already triaged to a defect | `fetch_test_stability_data`, `find_test_results`, `fetch_defect_data` | +| Where should the next investment go | coverage gap by area, flakiness concentration by suite, defect concentration by area | `fetch_requirement_data`, `fetch_test_stability_data`, `fetch_defect_data`, `read_test_suite` | + +Three rules that keep this from becoming a dump: + +- **Five metrics, hard cap.** If a metric would not change the decision, it does not go in. Offer it as available on request instead. +- **Every number carries its denominator and its date.** "82 percent" is not a metric. "50 of 61 in-scope requirements, as of the 2026-08-08 run" is. +- **A metric with no decision attached is deleted, not demoted to an appendix.** An appendix is where a dump hides. + +## Trending without a history tool + +Two classes of data, and they are not equally trendable. Confusing them is how a report invents history it never had. + +| Data class | Metrics | Trendable how | Honesty rule | +|---|---|---|---| +| **Execution-dated** | pass rate, failure counts, run counts, results per suite | directly, from `find_test_results` and `read_execution` over per-period date ranges | trendable on the very first run | +| **Current-state** | requirement coverage, stability, configuration coverage, defect standing, case quality | only against snapshots written by earlier runs of this skill | first run reports **baseline established, no trend yet**. Never infer a direction from one reading | + +Rules: + +- **Never present a two-point difference as a trend.** Two points are a change. Three or more are a direction. Say which one you have. +- **Never backfill a snapshot you did not take.** If the user wants four sprints of coverage history and the first snapshot is from last week, report execution-dated metrics across all four and coverage from the snapshot date forward, and state the split. +- **A metric that moved less than its own noise did not move.** `references/snapshot-and-trend.md` gives the noise floor per metric. + +## Report format + +```text +Answer: +Direction: Improving | Flat | Degrading | Baseline only +Period spine: +Confidence: High | Medium | Low () + +What moved it: +- because + +Named specifics: +- -> + +Not in this report: +- + +The ask: +- +``` + +Rules on the format: + +- **`Answer` comes first and is one sentence.** If it needs two, the question was two questions; split the report or pick one. +- **`The ask` is mandatory.** A report to a steering committee with nothing to decide wastes the slot. If there is genuinely no ask, write "no decision needed, informational" and mean it. +- **`Not in this report` is where the boundary lands in the audience's language**, not in MCP tool names. "We do not track how long a fix takes" beats "no cycle-time tool". +- **Every named specific is a real key returned by the platform.** Never invent a requirement key, suite name, or defect id to make a sentence land. + +## Worked example + +**Input, verbatim from the user** + +> I present to the steering committee on Thursday. Project Cellphone Shop, last four sprints. Are we getting better or worse, and what do I tell them? + +**Step 1, question and audience.** Question is *is quality getting better or worse*. Audience is a steering committee, so the report needs a direction and an ask, not a table. + +**Step 2, metrics chosen.** Three, not five: pass-rate trend, open critical defect trend, flaky-case count. Requirement coverage is deliberately cut, it answers a different question and the user did not ask it. Configuration coverage is cut for the same reason. + +**Step 3, period spine.** + +```text +list_projects -> Cellphone Shop (id 41207) +list_repositories -> Cellphone Shop Web (id 8891) +find_iterations(41207) -> Sprint 12 2026-05-19 .. 2026-06-01 + Sprint 13 2026-06-02 .. 2026-06-15 + Sprint 14 2026-06-16 .. 2026-06-29 + Sprint 15 2026-06-30 .. 2026-07-13 +``` + +Four periods. Enough for a direction. + +**Step 4, pull per period.** + +```text +find_test_results(repo 8891, 2026-05-19..2026-06-01) -> 214 results, 176 passed, 38 failed +find_test_results(repo 8891, 2026-06-02..2026-06-15) -> 221 results, 189 passed, 32 failed +find_test_results(repo 8891, 2026-06-16..2026-06-29) -> 218 results, 194 passed, 24 failed +find_test_results(repo 8891, 2026-06-30..2026-07-13) -> 226 results, 208 passed, 18 failed +fetch_defect_data(repo 8891) -> open 14, of which critical 3, high 6; reopened 2 +fetch_test_stability_data(repo 8891) -> 9 cases below the stability threshold, + 6 of them in "Checkout Regression" +``` + +**Step 5, compare against the snapshot.** `.katalon-reporting/cellphone-shop.json` exists from the run of 2026-06-16 and holds `open_critical_defects: 7`, `flaky_cases: 11`. Pass rate needs no snapshot because the results carry their own dates. + +**Step 6, name the causes.** `read_test_suite` on the six flaky cases resolves them to **Checkout Regression**. `fetch_defect_data` names the three open criticals as **CEL-241**, **CEL-258**, **CEL-263**, all in checkout. + +**Expected output** + +```text +Answer: Quality is improving on every metric we track, but the entire remaining + risk is concentrated in checkout. +Direction: Improving +Period spine: Sprints 12-15, 2026-05-19 to 2026-07-13, four periods from find_iterations +Confidence: Medium (defect and flakiness history starts 2026-06-16, the date of our + first snapshot; pass rate covers all four sprints) + +What moved it: +- Pass rate 82.2% -> 92.0% across four sprints (176/214 -> 208/226) because the checkout + failures that dominated Sprint 12 were fixed in Sprint 14 +- Open critical defects 7 -> 3 since 2026-06-16, two points not four, so a change and not + yet a direction +- Flaky cases 11 -> 9 since 2026-06-16, which is inside the noise floor and should be read + as flat, not improving + +Named specifics: +- Checkout Regression suite -> holds 6 of the 9 flaky cases, so the suite most people + watch is the one they can trust least +- CEL-241, CEL-258, CEL-263 -> all three remaining open criticals, all in checkout +- Sprint 15 -> 18 failures, the lowest of the four, and none in the smoke core + +Not in this report: +- Whether any of these defects reached a customer. We do not have that classification + in the platform and nobody has told us which field carries it +- How much effort the improvement cost. We do not track effort + +The ask: +- One sprint of dedicated work on Checkout Regression. It is 6 of our 9 unreliable cases + and 3 of our 3 open critical defects, and it is the flow the committee cares about. +``` + +Note what the example does **not** do: it does not report requirement coverage (not asked), it does not call the flaky improvement real (inside the noise floor), it does not claim a direction on defects (two points), and it does not claim any defect escaped to a customer (no such field). Each of those is a place a metric dump would have said more and meant less. + +## Prompt recipes + +- `Cellphone Shop, last four sprints, is quality getting better or worse? I present to the steering committee Thursday.` +- `Trend requirement coverage and defect risk across the last three releases and tell me where the next tester should go.` +- `Write the quality section of the QBR for this project. One page, and I need an ask at the end.` +- `Can I trust the numbers in our last release report? Show me flakiness and how much of it is untriaged.` + +## Hand-offs + +- Ship or no-ship on one release -> `katalon-release-analyze`. +- Suite verdict before the pipeline -> `katalon-test-review`. +- Orphan requirements and traceability detail behind a coverage number -> `katalon-test-management`. +- Repairing the flaky cases this report named -> `katalon-test-maintenance`. +- Turning an ask into next cycle's scope -> `katalon-test-plan`. +- Effort, headcount, or duration questions -> not built. Say so and stop. + +Read `references/metric-selection.md` before choosing metrics and `references/snapshot-and-trend.md` before claiming any trend. Consult the orchestrator's `katalon-true-platform-testing/references/unavailable-capabilities.md` when the user asks whether the platform can report X. diff --git a/skills/katalon-test-reporting/references/metric-selection.md b/skills/katalon-test-reporting/references/metric-selection.md new file mode 100644 index 000000000..42fc47463 --- /dev/null +++ b/skills/katalon-test-reporting/references/metric-selection.md @@ -0,0 +1,46 @@ +# Metric selection reference + +The point of this file is subtraction. Every metric below is available; almost none of them belong in any single report. + +## The four questions + +Almost every request that reaches this skill is one of four questions wearing different clothes. Identify which one before choosing anything. + +| Question | What the asker will do with the answer | Metrics that answer it | Metrics that do not | +|---|---|---|---| +| **Is quality getting better or worse** | decide whether the current approach continues | pass-rate trend, open critical defect trend, flaky-case trend | requirement coverage (measures effort, not outcome), case counts | +| **Are we testing the right things** | decide where testers go next | requirement coverage percent, uncovered critical requirements by key, configuration coverage vs the target matrix | pass rate (a suite testing the wrong things passes beautifully) | +| **Can we trust the results** | decide whether to believe the last report | flaky-case count and trend, cases with erratic history, share of failures triaged to a defect | total test count, pass rate | +| **Where should the next investment go** | allocate people or budget | gap and flakiness and defect concentration, each by area or suite | anything aggregate; concentration is the whole point | + +If the request is genuinely two questions, write two reports or ask which one matters on Thursday. A report that answers two questions answers neither. + +## Metric definitions, and exactly what each can claim + +| Metric | Derived from | Claims | Does **not** claim | +|---|---|---|---| +| **Pass rate** | `find_test_results` over a date range; passed / total | how the suite behaved in that window | that the product is good. A narrow suite passes easily | +| **Requirement coverage** | `fetch_requirement_data`; covered / in-scope. Confirm with `find_test_cases_by_requirement` before naming a requirement uncovered | that a linked case exists | that the case is good, or that it passed. Link is not evidence | +| **Uncovered critical requirements** | the uncovered set from `fetch_requirement_data`, filtered by the requirement's own priority | the specific keys nobody is testing | completeness, if requirement priority is not maintained in the ALM | +| **Open critical defects** | `fetch_defect_data`, filtered to critical and high | current standing risk | escape to production. There is no such field | +| **Flaky cases** | `fetch_test_stability_data`, cases below the platform's stability threshold | that the platform's own signal flags them | a computed flake rate. Report the platform's signal; never recompute it | +| **Erratic history** | `find_test_results` per case across periods; alternating outcomes with no code change | a case whose result is not information | a cause. Hand to `katalon-test-maintenance` | +| **Configuration coverage** | `fetch_test_configuration_data` vs a target matrix the user supplies | which combinations were exercised | which combinations matter. The target matrix is an input, not a platform fact | +| **Triage share** | failures from `find_test_results` that have a linked defect in `fetch_defect_data` | how much of the red is understood | that the untriaged remainder is benign | +| **Concentration** | any metric above, grouped by suite via `read_test_suite` or by folder via `find_test_folders` | where the problem lives | a root cause | + +## Anti-patterns + +Each of these has appeared in a real QA status report and each one is a dump wearing a suit. + +- **Total test count.** Grows with copy-paste. Correlates with nothing a stakeholder cares about. If someone asks for it, give it and say what it does not mean. +- **Automation percentage without a denominator.** 60 percent of what, and did the manual 40 percent get run. +- **Pass rate with no scope.** A pass rate over "all results" mixes a smoke run with a full regression and means nothing. +- **A coverage percentage with no uncovered list.** The percentage is the least useful part; the names are the report. +- **Defect counts with no severity split.** Twenty cosmetic defects and three criticals are not comparable, and summing them hides the three. +- **Any metric presented without a date.** Current-state fetches are read at report time, not at period end. Say when. +- **A trend line with no named cause.** This is the specific failure `katalon-test-review` calls a metric dump. If the cause cannot be named, report the number as unexplained. + +## The subtraction test + +Before a metric goes in the report, answer: *if this number were removed, would the reader make a different decision?* If no, cut it. Applied honestly this usually leaves two or three metrics, which is the right size for a slide someone reads out loud. diff --git a/skills/katalon-test-reporting/references/snapshot-and-trend.md b/skills/katalon-test-reporting/references/snapshot-and-trend.md new file mode 100644 index 000000000..1ea3b627b --- /dev/null +++ b/skills/katalon-test-reporting/references/snapshot-and-trend.md @@ -0,0 +1,78 @@ +# Snapshot and trend reference + +The Katalon MCP has **no trend tool, no aggregation tool, and no point-in-time read**. Every `fetch_*_data` tool returns current state. This file is how a trend gets built anyway, honestly. + +## Two classes of data + +| Class | Metrics | Why | +|---|---|---| +| **Execution-dated** | pass rate, failure counts, run counts, results per suite | `find_test_results` and `read_execution` return records that carry their own timestamps, so history is already there. Trendable on the first run | +| **Current-state** | requirement coverage, stability, configuration coverage, defect standing, case-quality signals | the `fetch_*_data` tools take no date parameter and return only now. History exists only if this skill wrote it down | + +Never mix the two in one trend line without saying so. A four-sprint pass-rate trend beside a one-week coverage delta is two different claims sharing a bullet. + +## The snapshot file + +Written on every run, to `.katalon-reporting/.json` in the user's workspace. Plain JSON, appended to, never rewritten. Ask before creating the directory the first time. + +```json +{ + "project": "Cellphone Shop", + "project_id": 41207, + "repository_id": 8891, + "snapshots": [ + { + "taken_at": "2026-06-16", + "period_label": "Sprint 14 start", + "requirement_coverage": { "covered": 44, "in_scope": 61 }, + "uncovered_critical": ["CEL-88", "CEL-102"], + "defects": { "open": 22, "critical": 7, "high": 9, "reopened": 4 }, + "flaky_cases": 11, + "configuration_coverage": { "tested": 6, "target": 9 }, + "source_tools": ["fetch_requirement_data", "fetch_defect_data", + "fetch_test_stability_data", "fetch_test_configuration_data"] + } + ] +} +``` + +Rules: + +- **Write the snapshot before writing the brief.** A run that dies mid-report should still have improved the next one. +- **Record `taken_at` as the date the fetch happened**, never the period it is being attributed to. They are different, and pretending otherwise is how a report acquires history it never had. +- **Record `source_tools`.** A later reader has to be able to tell which numbers came from where. +- **Never edit a past snapshot.** If one was wrong, add a new entry with a note. Rewriting history in a file whose only job is history defeats the file. + +## The period loop + +1. `find_iterations(project_id)` for the spine. If the team does not use iterations, take explicit date ranges from the user. +2. For each period, call the execution-dated tools with that period's range. +3. Call each current-state tool **once**, not per period. They have no period parameter and repeated calls return the same answer. +4. Read the snapshot file and align its entries to the spine by `taken_at`. +5. Append the new snapshot. + +Three to six periods. Fewer than three cannot show a direction; more than six is a chart, and this skill writes prose. + +## Noise floors + +A metric that moved less than its noise floor did not move. These are working defaults; if the team has its own, use theirs and say so. + +| Metric | Treat as flat when | +|---|---| +| Pass rate | the change is under 2 percentage points, or the run count changed by more than 20 percent between periods | +| Flaky-case count | the change is 2 cases or fewer | +| Open defect count | the change is under 15 percent, or is fully explained by a triage sweep rather than by fixes | +| Requirement coverage | the change is under 2 percentage points, or the in-scope denominator changed | + +**The denominator rule outranks all of them.** If the denominator moved — requirements added, suite re-scoped, a repository split — the metric is not comparable across that boundary. Say the denominator changed and report the periods separately. This is the single most common way a quality trend lies. + +## Language for what you actually have + +| What you have | Say | +|---|---| +| one reading | "baseline established, no trend yet" | +| two readings | "changed from X to Y" — a change, not a direction | +| three or more, same denominator | "improving / degrading / flat across N periods" | +| three or more, denominator moved | "not comparable across ; here is each side" | +| a move inside the noise floor | "flat" | +| a move with no identifiable cause | "moved from X to Y, cause not identified in the platform data" | diff --git a/skills/katalon-test-review/SKILL.md b/skills/katalon-test-review/SKILL.md new file mode 100644 index 000000000..f98182ccc --- /dev/null +++ b/skills/katalon-test-review/SKILL.md @@ -0,0 +1,55 @@ +--- +name: katalon-test-review +description: Review Katalon True Platform/TestOps test quality and coverage before tests enter the delivery pipeline. Use when you need to check whether a suite is ready to run, review requirement and configuration coverage, assess test-case quality and flakiness/stability, spot weak or unreliable cases, and produce a review verdict (Approve / Approve with fixes / Reject for pipeline) with the specific cases to fix. This is a pre-execution quality gate, not a ship decision; for the release ship/no-ship call use katalon-release-analyze. Written for the test lead gating what is allowed into the pipeline, one suite at a time. +--- + +# Katalon Test Review + +Use this skill for the **review** stage: inspect coverage, quality, and reliability so weak tests do not enter the pipeline. The output is a **verdict with named weak cases**, never a metric dump. + +## Availability Boundary + +- **Available via MCP:** coverage review (`fetch_requirement_data`, `find_test_cases_by_requirement`, `fetch_test_configuration_data`), quality review (`fetch_test_case_data`), reliability review (`fetch_test_stability_data`, `find_test_results`), and environment readiness (`read_auts`). +- **Not directly available:** code/object review, local debug, StudioAssist Ask — these are Studio-desktop operations. For code-lane review, defer to `katalon-test-case-to-playwright` / `katalon-playwright-execute`. Use Browser/Playwright only for AUT sanity checks when asked. + +## Review Workflow + +```text ++---------------------+ +----------------------+ +----------------------+ +| Coverage review | --> | Quality review | --> | Reliability review | +| reqs + config | | case design signals | | flakiness/stability | ++---------------------+ +----------------------+ +----------------------+ + | + v + +----------------------+ + | Verdict + weak cases | + +----------------------+ +``` + +## Steps and tool rules + +1. **Coverage review.** `fetch_requirement_data` + `find_test_cases_by_requirement` for requirement coverage; `fetch_test_configuration_data` for browser/platform coverage. Flag orphan requirements and under-covered configurations. +2. **Quality review.** `fetch_test_case_data` for design signals; read representative cases with `read_test_case` when a signal is ambiguous. Flag non-atomic cases (many assertions), missing negative/boundary variants, and vague expected results. +3. **Reliability review.** `fetch_test_stability_data` for flakiness; `find_test_results` for recent pass/fail history. Flag probabilistically flaky cases that will erode pipeline trust. +4. **Environment readiness.** `read_auts` to confirm an executable AUT exists for the suite. +5. **Verdict.** One of **Approve** / **Approve with fixes** / **Reject for pipeline**, followed by the specific cases to fix and why. State the risk if approving with known gaps. + +## Verdict rubric + +- **Approve:** coverage meets the plan, no flaky cases in the critical path, expected results are observable. +- **Approve with fixes:** ship-able but list the exact cases needing a fix (flaky, non-atomic, weak expected result) and the owner action. +- **Reject for pipeline:** orphan critical requirements, or flaky cases in the smoke/regression core — fix before the suite runs. + +## Prompt recipes + +- `Review the regression suite for release 3.2: is it ready for the pipeline? Give a verdict and list weak cases.` +- `Check requirement and configuration coverage for project X and flag anything under-covered.` +- `Which cases in the smoke suite are flaky enough to reject before we wire them into CI?` + +## Hand-offs + +- Fixes needed -> `katalon-create-test-cases` (redesign) or `katalon-test-maintenance` (repair/flaky). +- Approved -> `katalon-execute-test`. +- Ship decision after execution -> `katalon-release-analyze`. + +Read `references/review-rubric.md` before issuing a verdict. Consult the orchestrator's `references/unavailable-capabilities.md` for boundaries. diff --git a/skills/katalon-test-review/references/review-rubric.md b/skills/katalon-test-review/references/review-rubric.md new file mode 100644 index 000000000..7158a1299 --- /dev/null +++ b/skills/katalon-test-review/references/review-rubric.md @@ -0,0 +1,35 @@ +# Review rubric reference + +## Three review lenses (all MCP-backed) + +| Lens | Tools | What to flag | +|---|---|---| +| Coverage | `fetch_requirement_data`, `find_test_cases_by_requirement`, `fetch_test_configuration_data` | orphan requirements; under-covered browsers/platforms/OS | +| Quality | `fetch_test_case_data`, `read_test_case` | non-atomic cases; missing negative/boundary variants; vague or non-observable expected results; duplicate coverage | +| Reliability | `fetch_test_stability_data`, `find_test_results` | probabilistically flaky cases; cases with erratic recent history; long-broken cases | + +## Verdict decision table + +| Coverage | Flaky in critical path | Verdict | +|---|---|---| +| meets plan | none | Approve | +| meets plan | some non-critical | Approve with fixes | +| minor gaps | none | Approve with fixes | +| critical orphan reqs | any | Reject for pipeline | +| any | flaky in smoke/regression core | Reject for pipeline | + +## Output template + +```text +Verdict: Approve | Approve with fixes | Reject for pipeline +Coverage: reqs covered X/Y; configs covered A/B; orphans: +Quality flags: +Reliability: flaky: +Fix list: action -> owner hint> +Risk if shipped as-is: +``` + +## Boundaries + +- Case code, locators, and object maintainability are Studio-side — say so; do not claim to have reviewed script internals via MCP. +- "Probabilistic flakiness" is a platform signal read through `fetch_test_stability_data`; report it, do not recompute it. diff --git a/skills/katalon-true-platform-testing/SKILL.md b/skills/katalon-true-platform-testing/SKILL.md new file mode 100644 index 000000000..dc7b180c8 --- /dev/null +++ b/skills/katalon-true-platform-testing/SKILL.md @@ -0,0 +1,262 @@ +--- +name: katalon-true-platform-testing +description: End-to-end Katalon True Platform testing workflow and lifecycle router. Use when one request spans several stages and no single skill owns all of it, for example analyze a requirement, design and import the cases, build a suite, run it with AI, and report the outcome. Also use to route any testing request across the full 7-stage lifecycle (plan, design, manage, review, execute, analyze, maintain) to the right focused skill, and for deciding what is and is not available through the Katalon MCP tools. Start here when a request names a job rather than one task, such as drive the whole chain from requirement to ship call. Also start here when the asker wants orienting before acting, for example where do I start, which skill do I need, or I own quality here and do not know where to begin. Routes the asker to the skill that owns their next step, whether they say manual tester, QA analyst, test analyst, automation tester, SDET, automation engineer, QA engineer, test lead, QA lead, test manager, or QA manager. +--- + +# Katalon True Platform Testing + +Use this skill for requirement-to-execution workflows in Katalon True Platform/TestOps, and as the **router across the full 7-stage testing lifecycle**. Prefer Katalon MCP tools for platform operations and Browser/Playwright only for external AUT exploration or visual verification. + +## Role Routing + +Ask who is in front of you before asking which stage they are in. A skill is listed here when a request phrased in role terms, naming no skill, should land there first. Everything else is reached by handoff. + +| Role | Comes here to | Starts at | Then | +|---|---|---|---| +| Manual tester | turn a written requirement into cases and run them | `katalon-create-test-cases` | `katalon-execute-test`, `katalon-analyze-failures` | +| Automation tester | turn cases into code, run it, ship the results | `katalon-test-case-to-playwright` | `katalon-playwright-execute`, `katalon-upload-report`, `katalon-test-maintenance` | +| Test lead | scope the cycle, judge readiness, keep the suite healthy | `katalon-test-plan` | `katalon-test-review`, `katalon-test-management`, `katalon-release-analyze` | +| Test manager | read coverage and risk, and call ship | `katalon-release-analyze` | `katalon-test-review`, `katalon-test-management` | + +`katalon-platform-setup` is role-neutral and comes first for everyone who has not connected the MCP yet. Synonyms (SDET, automation engineer, QA engineer, QA analyst, test analyst, QA lead, QA manager), the full intent-to-skill map, and what to fall back to for the parts not built yet are in `references/lifecycle-map.md`. + +## Lifecycle Routing + +The Katalon testing lifecycle has 7 stages. For a focused request, route to the matching skill; for a full-flow request, run the stages in order and stop when the user's goal is met. Full map + tool lists: `references/lifecycle-map.md`. + +| Stage | Route to skill | When | +|---|---|---| +| 1 Plan | `katalon-test-plan` | scope, risk-prioritized plan, executable folder+suite for a sprint/release | +| 2 Design | `katalon-create-test-cases` (+ `katalon-test-case-to-playwright`) | author/import cases from requirements | +| 3 Manage | `katalon-test-management` | organize inventory, classify, requirement traceability audit | +| 4 Review | `katalon-test-review` | pre-pipeline coverage/quality/flakiness verdict | +| 5 Execute | `katalon-execute-test` (+ `katalon-upload-report`, `katalon-playwright-execute`) | manual, Run with AI, automated, cloud | +| 6 Analyze | `katalon-analyze-failures` + `katalon-release-analyze` | failure triage/defects; ship/no-ship call | +| 7 Maintain | `katalon-test-maintenance` | repair flaky/broken cases, regenerate, feed gaps back to plan | +| pre / cross | `katalon-platform-setup` | connect and verify the MCP | + +For multi-skill plays (requirement-to-ship, coverage rescue, flaky cleanup, manual-to-automation, cross-lane trust check, traceability audit) read `references/combination-recipes.md`. For copy-paste prompts and cross-model/cross-agent execution notes read `references/prompt-recipes.md`. For the full MCP tool list read `references/mcp-tool-index.md`. + +This skill can run any stage inline itself (the workflows below cover requirement->execution->report); route to a focused skill when the user wants only that stage or a deeper treatment (traceability, review verdict, failure triage, maintenance). + +## Autonomy Policy + +When the user asks for an end-to-end Katalon flow, try to complete the full available workflow without pausing for optional decisions: + +```text ++--------------+ --> +-------------+ --> +--------------+ --> +-------------+ --> +-------------+ +| Analyze reqs | | Create/link | | Create suite | | Create run | | Run with AI | ++--------------+ +-------------+ +--------------+ +-------------+ +-------------+ + | + v + +---------------+ + | Report result | + +---------------+ +``` + +Default assumptions: + +- If exactly one Katalon project or repository matches the user's wording or current context, use it. +- If exactly one repository exists, use it. +- If no AUT environments exist and the user supplied or requirement contains a URL, use that URL as `default_aut_environment_url` for AI execution. +- If AUT environments exist, choose the environment whose URL/name best matches the target AUT. Ask only if no match is clear. +- After creating any manual test execution, start Run with AI automatically and wait for completion unless the user explicitly says not to run AI. +- If a matching test case already exists, reuse and update/link it instead of creating a duplicate. +- If a matching test suite already exists, reuse it and add missing cases instead of creating a duplicate. + +Ask the user only when a required value cannot be resolved safely, multiple equally valid choices remain, credentials/accounts are missing, the next action is destructive, or the user explicitly asks for approval gates. + +## Availability First + +Before promising a workflow, state the automation boundary: + +- Available through Katalon MCP: list projects/repositories, find/read requirements, create/read/update test cases, link requirements, find/manage test suites and folders, create manual test runs, start Run with AI, poll AI sessions, read execution/test results, fetch quality metrics, and create ALM-linked defects. +- Not directly available through Katalon MCP: create requirements, create a formal Test Plan entity, guarantee AI execution completion, or inspect the live AUT UI without Browser/Playwright. +- Workaround for test plans: use a named test suite or folder plus release/sprint association as the executable test plan structure. + +For details, read `references/unavailable-capabilities.md` when the user asks "can Katalon do X?" or when planning scope. + +## Required Context Workflow + +Always resolve context in this order before mutating Katalon data: + +```text ++---------------+ --> +---------------------+ --> +-----------------------+ +| list_projects | | list_repositories | | resolve repository | ++---------------+ +---------------------+ +-----------------------+ +``` + +Rules: + +- Treat repository and Test Project as the same resolution target. +- If the user says "Katalon Cloud" or "cloud repo", prefer a repository named `Katalon Cloud` when present. +- Do not scan all repositories to avoid choosing. Resolve from context or ask when ambiguous. +- For full-flow requests, perform non-destructive writes without asking again once project/repository/requirement scope is resolved. +- Before destructive writes or bulk changes outside the requested flow, summarize the intended changes and get approval. + +## Requirement Analysis + +Use `find_requirements` or `read_requirement` when requirements exist in Jira/Azure integration. If requirements are provided in chat, analyze them locally and only use Katalon to create/link test assets. + +Output requirement analysis as: + +- Requirement intent +- User roles/personas +- Main flows +- Alternate and negative flows +- Data and environment assumptions +- Risk areas +- Coverage recommendations + +Read `references/requirement-analysis.md` for the checklist. + +## Manual Test Case Design + +Write manual test cases in a platform-importable style: + +- Title: concise and action-oriented. +- Description: what behavior is verified. +- Pre-condition: environment, data, account, AUT state. +- Steps: manual tester phrasing, each starting with a concrete action. +- Expected results: observable UI/API/platform result per step. +- Test data: values, URLs, accounts, or `N/A`. +- Priority: P0/P1/P2 when useful. +- Requirement links: source keys or internal requirement IDs when available. + +Mirror the style of related existing test cases before drafting new or updated cases: + +- Read representative existing cases for the same requirement, feature area, folder, suite, product flow, or repository. +- Use their naming convention, field structure, step granularity, vocabulary, pre-condition style, test data style, and expected-result detail level. +- Keep new coverage consistent with the local suite unless the existing style is clearly incomplete or obsolete. +- If existing cases are weak, preserve platform compatibility while improving only what is needed for correctness and coverage. +- When no related cases exist, use the default manual test case format below. + +Design enough coverage using ISTQB test design techniques as a reference before importing cases (the techniques guide the design; the output is plain platform test cases, not an ISTQB certification): + +- Use equivalence partitioning for input classes, filters, statuses, user roles, and product states. +- Use boundary value analysis for numeric ranges, quantities, prices, dates, pagination, and length limits. +- Use decision table testing for business rules with combinations of conditions. +- Use state transition testing for workflows such as cart, checkout, execution status, and lifecycle changes. +- Use use-case/scenario testing for end-to-end user journeys. +- Use error guessing/checklist-based testing for common ecommerce/platform risks. +- Use pairwise or combinatorial reduction when variants explode, while preserving high-risk combinations. + +Keep each case **atomic in scope** (one validation condition per case) so a failure pinpoints the exact rule and each requirement line maps 1:1 to a result. Atomic is about scope, not step count: every case is still a complete, runnable flow (precondition/navigation -> enter surrounding valid data -> action under test -> verify), never a lone bare assertion. Cover the happy-path flow and its edge cases (boundary + negative variants), not just the positive path. Reserve combined cases for true end-to-end scenarios. Quote expected error/UI strings verbatim from the requirement, including source typos (flag them separately). If coverage is intentionally reduced, state the risk-based rationale. + +Read `references/istqb-coverage.md` before designing cases from requirements. Read `references/manual-test-case-format.md` before creating many cases or when the user asks for a specific format. + +## Existing Test Case Check + +Before creating or importing any test case, check whether suitable coverage already exists: + +- If requirement IDs are known, call `find_test_cases_by_requirement` first. +- Search by requirement key, title keywords, feature area, and target folder with `find_test_cases`. +- Read likely matches with `read_test_case` when title alone is not enough to judge coverage. +- Read enough related cases to infer the local writing style before drafting new cases, even when the related cases do not fully cover the requested behavior. +- Reuse, update, move, or link existing cases when they already cover the behavior. +- Create new cases only for uncovered behavior, missing coverage classes, or clearly obsolete/incorrect existing coverage. +- Report what was reused, what was updated, and what was newly created. + +This check is mandatory for write/import/full-flow requests, including retries after partial failure. Do not create duplicate cases simply because a previous create attempt failed. If no related cases can be found, say that the new cases follow the default skill format. + +## Import And Traceability Workflow + +Use this flow for "write test cases and import to platform": + +```text ++----------------------+ +-----------------------+ +----------------------+ +| Analyze requirements | --> | Draft coverage design | --> | Find existing cases | ++----------------------+ +-----------------------+ +----------------------+ + | | | + v v v ++----------------------+ +-----------------------+ +----------------------+ +| create/update needed | --> | link_requirements | --> | read_test_case verify| ++----------------------+ +-----------------------+ +----------------------+ +``` + +Tool rules: + +- Search for existing matching test cases before every create/import action. +- Use `create_test_case` only for manual test cases that do not already have suitable coverage. +- Use `link_requirements_to_test_case` only after requirement IDs are known. +- Use `read_test_case` after creation when verification matters. +- Use `update_test_case` for revisions; pass all intended updates in one call. +- Use `manage_test_folder` or `move_test_case` for organization when requested. + +## Test Suite And "Test Plan" Workflow + +When the user says "search and design test plan": + +1. Use `find_test_cases`, `find_requirements`, and `find_iterations` as needed to understand scope. +2. Propose a test plan structure in chat: objectives, in-scope/out-of-scope, coverage matrix, risks, environments, suites. +3. Implement the executable structure with `manage_test_suite` and optional folders. +4. Add selected test cases to suites. +5. Verify with `read_test_suite`. + +For formal Test Plan entity creation, state that the current MCP does not expose a direct create-katalon-test-plan tool. + +## Manual Execution And Run With AI + +Use this flow for manual execution: + +```text ++------------------+ +----------------------+ +-----------------------+ +| read_auts | --> | create_manual_run | --> | read_test_suite | ++------------------+ +----------------------+ +-----------------------+ + | | | + v v v ++------------------+ +----------------------+ +-----------------------+ +| create_ai_session| --> | read_ai_session poll | --> | report results | ++------------------+ +----------------------+ +-----------------------+ +``` + +Rules: + +- If the user asks for the full flow, test cases just created, or Run with AI, treat the execution as manual unless they explicitly ask for automated execution. +- If the user asks only to "run tests" and no test type can be inferred, ask: "Do you want to run manually or automated?" +- For manual execution, always call `read_auts` immediately before `create_manual_test_run`. +- Choose the best matching AUT/environment automatically when URL/name clearly matches the target AUT. Ask only when multiple AUTs are equally plausible. +- Never reuse AUT environment selection from an earlier turn or earlier run. +- After any manual test execution is created, start Run with AI automatically unless the user explicitly says not to. Do not ask whether to continue with AI. +- Before `create_manual_ai_session`, call `read_test_suite` for every suite from the manual run and pass non-empty test case lists. +- Poll `read_manual_ai_session` until all items are no longer TODO/IN_TESTING. If the session stays queued or running for a long time, keep polling at practical intervals and report an in-progress state only when the user asks for status or an external platform timeout/error is observed. + +Read `references/execution-workflow.md` before creating executions. + +## Automated Execution + +Use automated execution only for automated test suites: + +```text ++------------------+ --> +--------------------+ --> +----------------------+ +| find_test_suites | | build run config | | schedule_test_run | ++------------------+ +--------------------+ +----------------------+ +``` + +Rules: + +- Use `schedule_test_run`, never `create_manual_test_run`, for automated suites. +- Do not run individual manual test cases through `schedule_test_run`. +- Use `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, and optionally `build_schedule`. +- For mobile runs, ask whether the target is mobile browser or mobile app when app details are missing. + +## Result Review And Response + +After execution, read results before responding: + +- Manual AI run: `read_manual_ai_session`, then relevant execution/result tools if IDs are available. +- TestOps execution: `read_execution`, `read_execution_test_results`, `read_test_result`. +- Recent or specific results: `find_test_results`. +- Quality summaries: `fetch_requirement_data`, `fetch_test_case_data`, `fetch_defect_data`, `fetch_test_configuration_data`, `fetch_test_stability_data`. + +Report in chat with: + +- Execution/run link when returned by the platform. +- Pass/fail/blocked counts. +- Failed cases and concise failure reason. +- Defects created or recommended. +- Gaps, skipped items, and what needs manual follow-up. + +## Defects + +Use `create_defect` only when a failed test result ID is known. First call `find_alm_integration_projects` if ALM integration IDs are unknown. Ask the user before creating defects unless they explicitly requested defect creation for failures. diff --git a/skills/katalon-true-platform-testing/references/combination-recipes.md b/skills/katalon-true-platform-testing/references/combination-recipes.md new file mode 100644 index 000000000..07474304b --- /dev/null +++ b/skills/katalon-true-platform-testing/references/combination-recipes.md @@ -0,0 +1,45 @@ +# Combination recipes — multi-skill, multi-tool playbooks + +Each recipe chains several skills and MCP tools into one end-to-end play. Trigger phrase, skill order, key tools, stop condition, boundary. + +## R1 — Requirement to ship call (stages 1->6) +- **Trigger:** "test CEL-6 end to end and tell me if we can ship." +- **Skills:** `katalon-test-plan` -> `katalon-create-test-cases` -> `katalon-execute-test` -> `katalon-analyze-failures` -> `katalon-release-analyze`. +- **Tools:** `fetch_requirement_data`, `find_test_cases_by_requirement`, `create_test_case`, `link_requirements_to_test_case`, `manage_test_suite`, `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`, `read_test_result`, then the `fetch_*` quality tools. +- **Stop when:** release verdict issued (Ready / Ready with risk / Not ready). +- **Boundary:** no release-gate authoring; the ship call reads gates, it does not set them. + +## R2 — Coverage rescue (stages 3->4->2) +- **Trigger:** "we have gaps, fix our coverage for project X." +- **Skills:** `katalon-test-management` (find orphan requirements/cases) -> `katalon-test-review` (verdict on what exists) -> `katalon-create-test-cases` (fill). +- **Tools:** `find_test_cases_by_requirement`, `fetch_requirement_data`, `find_test_cases`, `fetch_test_case_data`, then `create_test_case` + `link_requirements_to_test_case`. +- **Stop when:** every in-scope requirement has >=1 linked case and the review verdict is Approve / Approve-with-fixes. + +## R3 — Flaky-suite cleanup (stages 6->7->4) +- **Trigger:** "our regression is flaky, clean it up." +- **Skills:** `katalon-analyze-failures` (classify automation defects) -> `katalon-test-maintenance` (repair/regenerate) -> `katalon-test-review` (re-approve). +- **Tools:** `find_test_results`, `read_test_result`, `fetch_test_stability_data`, `update_test_case`, `move_test_case`, then re-run + `fetch_test_stability_data` again. +- **Stop when:** flaky cases in the critical path are repaired or retired and the suite re-approves. + +## R4 — Manual to automation graduation (stages 2->5) +- **Trigger:** "turn these manual cases into automation and run them into the platform." +- **Skills:** `katalon-create-test-cases` -> `katalon-test-case-to-playwright` -> `katalon-playwright-execute` -> `katalon-upload-report`. +- **Tools:** `read_test_case`, `find_test_cases`, then Playwright codegen + `@katalon/playwright-reporter` upload, verified via `read_execution` / `find_test_results`. +- **Stop when:** the Playwright run is uploaded and the platform run is verified. + +## R5 — Cross-lane trust check (stages 5->6) +- **Trigger:** "don't trust the AI pass, verify it against real automation." +- **Skills:** `katalon-execute-test` (manual Run with AI) + `katalon-playwright-execute` (code lane) -> `katalon-analyze-failures`. +- **Tools:** `create_manual_ai_session`, `read_manual_ai_session`, Playwright run + `@katalon/playwright-reporter` upload, then `read_test_result` on both. +- **Stop when:** every critical case's manual AI verdict is confirmed against the code-lane ground truth; discrepancies recorded. +- **Boundary:** a self-reported AI PASS contradicted by the code lane is not a PASS. + +## R6 — Traceability audit (stage 3) +- **Trigger:** "show me requirement-to-test coverage." +- **Skills:** `katalon-test-management`. +- **Tools:** `find_requirements`, `find_test_cases_by_requirement`, `fetch_requirement_data`, `find_test_cases`. +- **Stop when:** the requirement <-> case <-> suite matrix + orphan lists + coverage % are reported. + +## Composition rule + +Always resolve project/repository once at the start, reuse that scope across the whole recipe, and state the boundary before promising any step. Ask only when a required value cannot be resolved safely. diff --git a/skills/katalon-true-platform-testing/references/execution-workflow.md b/skills/katalon-true-platform-testing/references/execution-workflow.md new file mode 100644 index 000000000..65a26beb2 --- /dev/null +++ b/skills/katalon-true-platform-testing/references/execution-workflow.md @@ -0,0 +1,64 @@ +# Execution Workflow + +## Manual Run With AI + +1. Resolve project and repository. +2. Search existing coverage first, then resolve existing test cases or create only missing cases. +3. Add cases to a manual test suite if grouping is needed. +4. Call `read_auts`. +5. Choose the matching AUT/environment automatically when one clearly matches the target AUT. If none exist and a URL is known, use it as `default_aut_environment_url` for AI execution. +6. Call `create_manual_test_run`. +7. Continue to AI automatically after creating any manual execution unless the user explicitly says not to run AI. +8. Call `create_manual_ai_session`. +9. Poll with `read_manual_ai_session` until every test case leaves TODO/IN_TESTING. +10. Summarize result in chat. + +## Manual Run Rules + +- Never call `create_manual_test_run` without a fresh `read_auts` first. +- Never reuse AUT environment choices from earlier turns. +- Start Run with AI without asking again after any manual run is created. +- Do not ask whether to continue with AI unless the user explicitly requests a manual run without AI. +- If the manual run contains newly created test cases, they are manual by default. +- Render returned execution paths as markdown links. +- Wait for AI completion before final response. If the platform stays pending/running for an unusually long time, keep polling at practical intervals and only report in-progress status when the user asks or the platform returns a timeout/error. +- Ask for user input only when required data is missing, multiple AUTs are equally plausible, the user explicitly disables AI, or the next action is destructive. + +## Automated Run + +1. Resolve repository. +2. Find automated test suites or suite collections. +3. Find execution profiles. +4. Select TestCloud environments. +5. Build run configuration. +6. Optionally build schedule. +7. Call `schedule_test_run`. +8. Read execution and results. + +## Automated Run Rules + +- Use `schedule_test_run` only for automated suites. +- Do not run manual test cases through automated scheduling. +- For mobile native, ensure app details are present. +- For mobile availability filters, clarify automation/manual vs live testing when needed. + +## Result Reporting Template + +```text +Run: +- Name: +- Link: +- Status: + +Summary: +- Passed: +- Failed: +- Blocked/Incomplete: +- Not run: + +Findings: +- ... + +Next actions: +- ... +``` diff --git a/skills/katalon-true-platform-testing/references/istqb-coverage.md b/skills/katalon-true-platform-testing/references/istqb-coverage.md new file mode 100644 index 000000000..cfd05ce03 --- /dev/null +++ b/skills/katalon-true-platform-testing/references/istqb-coverage.md @@ -0,0 +1,64 @@ +# Coverage Guide (ISTQB techniques as reference) + +Use this guide before writing or importing test cases. ISTQB is the **reference toolkit** for designing coverage here, not the deliverable: the output is plain platform test cases designed *using* these techniques, not "ISTQB cases" certified against the standard. The goal is enough risk-based coverage, not maximum case count. + +## Technique Selection + +- Equivalence Partitioning: use for valid/invalid classes such as product categories, brands, stock states, payment methods, account roles, and form input classes. +- Boundary Value Analysis: use for numeric or ordered values such as price range, quantity, pagination, character limits, dates, and timeout thresholds. +- Decision Table Testing: use when outcomes depend on combinations of conditions, such as selected variant + stock + quantity, checkout field validity, shipping eligibility, or payment availability. +- State Transition Testing: use when behavior depends on prior state, such as empty cart -> item added -> quantity updated -> removed, checkout step progression, or execution TODO -> IN_TESTING -> PASSED/FAILED. +- Use Case / Scenario Testing: use for end-to-end journeys that represent user goals, such as browse -> select variant -> add to cart -> checkout. +- Error Guessing / Checklist-Based Testing: use for likely failures based on domain knowledge, such as broken images, stale cart totals, invalid email, unavailable product, duplicate submission, or navigation loss. +- Pairwise / Combinatorial Testing: use when many variables interact, such as browser x device x category x filter x sort, while preserving explicitly high-risk combinations. + +## Minimum Coverage Expectations + +For each requirement, identify: + +- At least one happy-path scenario. +- At least one negative or validation scenario when user input or branching exists. +- Boundary cases for numeric/date/range fields. +- State transitions for multi-step workflows. +- Role/permission coverage when roles exist. +- Data setup and cleanup assumptions. +- Requirement-to-test traceability. + +## Coverage Output Format + +Before importing tests, prepare a short coverage note: + +```text +Coverage Techniques: +- Use case testing: ... +- Equivalence partitions: ... +- Boundary values: ... +- Decision table/state transition: ... +- Error guessing risks: ... + +Selected Test Cases: +- P0: ... +- P1: ... +- P2: ... + +Deferred / Not Covered: +- ... +Reason: +- ... +``` + +## Case Granularity + +- Keep each case **atomic in scope**: one validation condition / one acceptance-criteria line per case, so a failure pinpoints the exact rule and each requirement line maps 1:1 to a result. +- Atomic does not mean a single step. Every case is a **complete, runnable flow**: precondition/navigation -> enter surrounding valid data -> perform the action under test -> verify the result. Avoid lone-step cases like "count the columns"; include the steps to reach and exercise that state so the case executes on its own (including under Run with AI). +- Cover the happy-path flow **and** its edge cases for every feature: main success flow plus boundary and negative variants. Do not stop at the positive path. +- Quote expected error/UI strings verbatim from the requirement, including source typos; flag suspected typos separately rather than correcting them in the expected result. + +## Practical Rules + +- Do not create redundant tests that exercise the same partition and same expected behavior. +- Prefer fewer strong tests over many shallow tests. +- Mark P0 for revenue, checkout, account, data-loss, or broken-entry-point risks. +- Mark P1 for important catalog, filter, sort, and traceability behaviors. +- Mark P2 for cosmetic, footer, secondary navigation, or low-risk edge cases. +- If a test is primarily exploratory or visual, say so and include what evidence is needed. diff --git a/skills/katalon-true-platform-testing/references/lifecycle-map.md b/skills/katalon-true-platform-testing/references/lifecycle-map.md new file mode 100644 index 000000000..dd0e7256a --- /dev/null +++ b/skills/katalon-true-platform-testing/references/lifecycle-map.md @@ -0,0 +1,110 @@ +# Lifecycle map — 7 stages -> skills -> MCP tools + +Single source for the README diagram and the orchestrator's routing. The Katalon testing lifecycle has 7 stages; each maps to one or more skills and a set of MCP tools. + +```text +1 PLAN ......... katalon-test-plan + list_projects, list_repositories, find_iterations, + fetch_requirement_data, find_test_cases_by_requirement, + manage_test_folder, manage_test_suite + +2 DESIGN ....... katalon-create-test-cases (+ katalon-test-case-to-playwright) + find_requirements, read_requirement, create_test_case, + read_test_case, update_test_case, find_test_cases + +3 MANAGE ....... katalon-test-management + find_test_folders, manage_test_folder, find_test_suites, + manage_test_suite, move_test_case, duplicate_test_case, + link_requirements_to_test_case, unlink_requirements_from_test_case, + find_test_cases_by_requirement, fetch_requirement_data + +4 REVIEW ....... katalon-test-review + fetch_requirement_data, fetch_test_case_data, + fetch_test_stability_data, fetch_test_configuration_data, + find_test_cases_by_requirement, read_auts + +5 EXECUTE ...... katalon-execute-test (+ katalon-upload-report, katalon-playwright-execute) + read_auts, create_manual_test_run, create_manual_ai_session, + read_manual_ai_session, find_execution_profiles, + list_test_cloud_environments, build_run_configuration, + build_schedule, schedule_test_run, read_execution, + read_execution_test_results + +6 ANALYZE ...... katalon-analyze-failures + katalon-release-analyze + read_test_result, read_execution_test_results, find_test_results, + fetch_defect_data, fetch_test_case_data, fetch_test_stability_data, + fetch_test_configuration_data, fetch_requirement_data, + find_alm_integration_projects, create_defect + +7 MAINTAIN ..... katalon-test-maintenance + fetch_test_stability_data, find_test_results, read_execution, + update_test_case, move_test_case, duplicate_test_case + | + +--> feeds the gap list back to 1 PLAN (the loop closes) + +CROSS-CUTTING .. katalon-platform-setup (connect) · katalon-true-platform-testing (router) +``` + +## Stage boundaries (no MCP) + +- Object/action capture, data design, resilience design (stage 2): Studio desktop. +- Custom fields/tags, Git config, governance (stage 3): TestOps UI. +- Code/object review, local debug (stage 4): Studio desktop. +- Rerun / terminate / Live Monitor (stage 5): TestOps UI (MCP reads results only). +- AI root-cause, self-healing, Time Capsule, TrueTest regeneration (stages 6-7): product surfaces, not MCP. + +Use Browser/Playwright for AUT exploration; use Studio for object/script work; use the MCP for everything in the tool lists above. + +## Role map + +Four canonical roles. The synonyms are prose only, so someone who uses the industry's noun instead of Katalon's still finds their row. + +| Role | Also called | Comes here to | Starts at | Then | +|---|---|---|---|---| +| Manual tester | QA analyst, test analyst, QA engineer | turn a written requirement into cases and run them | `katalon-create-test-cases` | `katalon-execute-test`, `katalon-analyze-failures` | +| Automation tester | SDET, automation engineer, QA engineer | turn cases into code, run it, ship the results | `katalon-test-case-to-playwright` | `katalon-playwright-execute`, `katalon-upload-report`, `katalon-test-maintenance` | +| Test lead | QA lead, QE lead, test coordinator | scope the cycle, judge readiness, keep the suite healthy | `katalon-test-plan` | `katalon-test-review`, `katalon-test-management`, `katalon-release-analyze` | +| Test manager | QA manager, QE manager, head of quality | read coverage and risk, and call ship | `katalon-release-analyze` | `katalon-test-review`, `katalon-test-management` | + +A skill is named here when a request phrased in role terms, naming no skill, should land there first. Everything else is reached by handoff. `QA engineer` maps to two roles on purpose: resolve it by asking one question, never by guessing. + +`katalon-platform-setup` is role-neutral. It routes on the words connect, install, and MCP rather than on a role, and everyone runs it once. + +## Intent to skill + +**When a row's owning skill does not exist yet, route to the fallback and say the boundary out loud. Never invent a capability to fill a row.** + +| Intent, in the tester's words | Route to today | When the gap closes | +|---|---|---| +| Where do I start, I own quality for this and do not know the tooling | `katalon-true-platform-testing` | covered | +| Connect the platform, nothing works | `katalon-platform-setup` | covered | +| Analyze this requirement before I write anything | `katalon-create-test-cases` | covered | +| I just got requirement CEL-6 and need cases | `katalon-create-test-cases` | covered | +| What test data do these cases need | `katalon-create-test-cases`, per-case test data field only | `katalon-test-data` | +| Seed and tear down data for the run | **boundary only.** No skill owns this. State it | `katalon-test-data` | +| Run an exploratory session on checkout | `katalon-exploratory-charter` | covered | +| What should we test first this sprint | `katalon-test-plan` | covered | +| How many testers, how long, for this release | `katalon-test-plan`, scope only. State the boundary | `katalon-test-estimation` | +| Which product areas carry the most risk this quarter | `katalon-test-plan`, per-cycle risk ranking only. State the boundary | `risk-portfolio` | +| Organize our test cases, they are a mess | `katalon-test-management` | covered | +| Which requirements have no coverage | `katalon-test-management` | covered | +| Is this suite good enough for the pipeline | `katalon-test-review` | covered | +| Run these cases and tell me what broke | `katalon-execute-test` | covered | +| Run it with AI, I do not have time to click through | `katalon-execute-test` | covered | +| Schedule the automated suite on TestCloud | `katalon-execute-test` | covered | +| Turn TC-1042 into a Playwright spec | `katalon-test-case-to-playwright` | covered | +| Get my Playwright run into the platform | `katalon-playwright-execute` | covered | +| I have a JUnit or Katalon report on disk | `katalon-upload-report` | covered | +| Turn this manual case into a Cypress spec | `katalon-test-case-to-playwright` as the pattern, `katalon-upload-report` via a Mocha JUnit reporter. State the boundary | Cypress pair | +| Turn this manual case into a Selenium test | `katalon-upload-report`, JUnit XML path only. State the boundary | Selenium pair | +| Turn this manual case into a Katalon Studio test | `katalon-upload-report` runs `katalonc`, execution only, no authoring. State the boundary | Katalon Studio pair | +| Wire this into CI | `katalon-playwright-execute` + `katalon-upload-report`, both expose CI-invocable commands | `ci-setup` | +| This run failed, is it us or the app | `katalon-analyze-failures` | covered | +| File the bugs for these failures | `katalon-analyze-failures` | covered | +| Our Cypress suite is flaky | `katalon-test-maintenance` | Cypress pair, for the framework-specific rerun path | +| Which tests went flaky this month | `katalon-test-maintenance` | covered | +| Can we ship 3.2 | `katalon-release-analyze` | covered | +| I need a status deck for the steering committee | `katalon-release-analyze`, single-release verdict only. State the boundary | `katalon-test-reporting` | +| Trend escaped defects across the last four releases | `katalon-release-analyze`, single release only. State the boundary | `katalon-test-reporting` | +| What should we change about how we test | `katalon-test-maintenance`, asset-level gap list only. State the boundary | `test-retro` | +| Assert on an email, a PDF, a visual, a credential, or a database row inside a test | **boundary only.** No skill owns this. State it | not yet named, script distribution still blocks it | diff --git a/skills/katalon-true-platform-testing/references/manual-test-case-format.md b/skills/katalon-true-platform-testing/references/manual-test-case-format.md new file mode 100644 index 000000000..9a0600219 --- /dev/null +++ b/skills/katalon-true-platform-testing/references/manual-test-case-format.md @@ -0,0 +1,45 @@ +# Manual Test Case Format + +Use this format when generating cases for Katalon True Platform. + +## Fields + +- Name: short verb-led title. +- Description: one or two sentences explaining the behavior under test. +- Pre-condition: environment, account, seed data, cart state, login state, AUT URL, and any browser/device assumptions. +- Test Steps: manual style actions. Start with navigation when page context matters. +- Expected Results: observable outcome for each step. +- Test Data: concrete values used by the step, or `N/A`. +- Priority: P0 critical path, P1 important functional path, P2 secondary/edge path. +- Requirement IDs: internal requirement IDs or source keys when linking is requested. + +## Step Quality Rules + +- Write one user action per step. +- Use visible labels and URLs, not implementation selectors. +- Put validations in Expected Results, not the action. +- Avoid "verify everything looks correct"; specify what must be visible or changed. +- Split long end-to-end flows when setup makes individual failures hard to diagnose. + +## Katalon Import Notes + +- Before creating test cases, search existing coverage with `find_test_cases_by_requirement` when requirement IDs are known and `find_test_cases` by title, requirement key, feature area, and folder. +- Reuse or update matching existing cases instead of creating duplicates. +- Create manual test cases with `create_test_case`. +- Link requirements after creation with `link_requirements_to_test_case`. +- Verify important created cases with `read_test_case`. +- Revise existing cases with one `update_test_case` call containing all updates. + +## Example Shape + +```text +Name: Add selected product variant to cart +Description: Verify that a shopper can select a product variant and add the selected quantity to cart. +Pre-condition: Storefront is available. Cart is empty. + +| Step | Test Step | Expected Result | Test Data | +| 1 | Navigate to product detail page | Product page is displayed with product heading. | https://... | +| 2 | Select color Ultramarine | Ultramarine option is selected. | Ultramarine | +| 3 | Select storage 128 GB | 128 GB option is selected and SKU/stock are displayed. | 128 GB | +| 4 | Click Buy | Product added confirmation is displayed and cart badge updates. | N/A | +``` diff --git a/skills/katalon-true-platform-testing/references/mcp-tool-index.md b/skills/katalon-true-platform-testing/references/mcp-tool-index.md new file mode 100644 index 000000000..8823a4a28 --- /dev/null +++ b/skills/katalon-true-platform-testing/references/mcp-tool-index.md @@ -0,0 +1,66 @@ +# MCP tool index — every Katalon MCP tool, one line, stage-tagged + +Stages: 1 plan · 2 design · 3 manage · 4 review · 5 execute · 6 analyze · 7 maintain. + +## Discovery +- `list_projects` — list Katalon projects. [1,2,3,5] +- `list_repositories` — list repositories / Test Projects. [1,3] +- `find_iterations` — find sprints/iterations for a project. [1] + +## Requirements +- `find_requirements` — find synced requirements (Jira/Azure). [2] +- `read_requirement` — read one requirement's detail. [2] +- `fetch_requirement_data` — requirement coverage status. [1,4,6] + +## Test cases +- `create_test_case` — create a manual test case. [2] +- `read_test_case` — read a test case. [2,4] +- `update_test_case` — update a case (all edits in one call). [2,7] +- `duplicate_test_case` — copy a case. [7] +- `delete_test_case` — delete a case (unreliable; prefer update/move). [3] +- `move_test_case` — move a case between folders. [3,7] +- `find_test_cases` — search cases by key/title/feature/folder. [2,3] + +## Folders and suites +- `find_test_folders` — list folders. [3] +- `manage_test_folder` — create/organize folders. [1,3] +- `find_test_suites` — list suites. [3,5] +- `read_test_suite` — read a suite's cases. [3,5] +- `manage_test_suite` — create/edit a suite. [1,3] + +## Requirement links +- `link_requirements_to_test_case` — link req -> case. [3] +- `unlink_requirements_from_test_case` — remove a link. [3] +- `find_test_cases_by_requirement` — cases linked to a requirement. [1,3,4] + +## Manual execution +- `read_auts` — read applications-under-test / environments (call right before a manual run). [4,5] +- `create_manual_test_run` — start a manual run. [5] +- `create_manual_ai_session` — start Run with AI. [5] +- `read_manual_ai_session` — poll AI session until no item is TODO/IN_TESTING. [5] + +## Automated execution +- `find_execution_profiles` — execution profiles. [5] +- `list_test_cloud_environments` — TestCloud environments. [5] +- `build_run_configuration` — build a run config. [1,5] +- `build_schedule` — build a schedule. [5] +- `schedule_test_run` — schedule an automated run (never for manual cases). [1,5] + +## Results +- `read_execution` — read an execution. [5,6,7] +- `read_execution_test_results` — results within an execution. [5,6] +- `read_test_result` — one test result's detail (needed for defect filing). [6,7] +- `find_test_results` — find recent/specific results. [6,7] + +## Quality data +- `fetch_defect_data` — defect status/context. [6,7] +- `fetch_test_case_data` — test-case quality signals. [4,6] +- `fetch_test_stability_data` — flakiness/stability signal. [4,6,7] +- `fetch_test_configuration_data` — configuration coverage. [4,6] + +## ALM defects +- `find_alm_integration_projects` — ALM integration projects. [6] +- `create_defect` — create an ALM-linked defect (requires a failed result ID). [6] + +## Not available via MCP (state as boundary) +Create requirements · create Release/Build/Test-Plan entity · author release gates · guarantee Run-with-AI completion · inspect AUT UI · self-healing / Time Capsule / Tracer / object refactor · TrueTest regeneration · rerun/terminate/Live-Monitor · custom fields & tags · Git repo config · project governance. Use Jira/Azure, Studio, TestOps UI, or Browser/Playwright for these. diff --git a/skills/katalon-true-platform-testing/references/prompt-recipes.md b/skills/katalon-true-platform-testing/references/prompt-recipes.md new file mode 100644 index 000000000..c7bc70151 --- /dev/null +++ b/skills/katalon-true-platform-testing/references/prompt-recipes.md @@ -0,0 +1,39 @@ +# Prompt recipes + cross-model / cross-agent execution notes + +These skills are agent-neutral and must run on any coding agent (Claude, ChatGPT/Codex, Gemini, Copilot, Cursor, Kiro, Windsurf, Cline, Continue) and any model tier (Opus down to Haiku/small). Smaller models reason less, so the skills are written to be **followed, not interpreted**. + +## Cross-model design principles (why the skills read the way they do) + +1. **Deterministic tool order.** Steps are numbered and the tool sequence is fixed. A small model should never have to decide *which* tool — only execute the next numbered call. +2. **Explicit stop conditions.** Every loop states when to stop (e.g. "poll `read_manual_ai_session` until no item is TODO/IN_TESTING, then stop"). No open-ended "keep checking". +3. **One decision at a time.** Resolve project -> repository -> requirement in that fixed order. Ask only when a value cannot be resolved safely. +4. **Boundary stated up front.** Each skill names what the MCP cannot do first, so a weaker model does not hallucinate create-requirement / create-release / self-heal. +5. **Report template supplied.** Each workflow ends with an exact output shape so weaker models produce a structured answer instead of rambling. +6. **Platform tool names only.** Reference MCP tools by their platform name (`create_manual_test_run`), never an agent-specific wrapper, so the same instruction works in Copilot/Cursor/Kiro/Windsurf. + +## Small-model guardrails (Haiku / small Copilot / Gemini Flash) + +- Do one recipe at a time; do not chain R1..R6 in a single prompt. +- Confirm scope (project + repository) before any write. +- After each mutating call, read back (`read_test_case` / `read_test_suite` / `read_execution`) before moving on. +- If a tool errors, report the exact error and stop; do not retry blindly or invent a workaround. + +## Copy-paste prompts by intent + +Setup: `Set up the Katalon MCP and verify my projects.` +Plan: `Plan testing for sprint 3.2: show coverage gaps and build the executable suite.` +Design: `Design and import atomic manual cases from requirement CEL-6 and link them.` +Manage/trace: `Audit requirement-to-test traceability for project X and list orphan requirements.` +Review: `Review the regression suite for release 3.2 and give a verdict with weak cases.` +Execute: `Run this suite with AI and report pass/fail/blocked.` +Analyze: `Triage execution 8842: product bug vs flaky vs environment, and what to file.` +Release: `Is release 3.2 ready to ship based on the quality metrics?` +Maintain: `Which cases went flaky this month, and repair vs regenerate?` +Cross-verify: `Run this critical suite with AI and with Playwright, then reconcile any disagreement.` +Full chain: `Analyze CEL-6, design and import cases, build a suite, run with AI, and tell me if we can ship.` + +## Non-Claude agent notes + +- **Copilot / Cursor / Kiro / Windsurf / Cline / Continue:** invoke the skill by its name or `/skill-name`; the MCP tool calls are identical. The generated adapter for each agent carries the same skill body. +- **Codex / ChatGPT:** the plugin declares the MCP in `.mcp.json`; follow the numbered steps exactly, they are model-agnostic. +- **Any agent via AGENTS.md:** read `AGENTS.md`, match the request to a skill description, open that `SKILL.md`, follow it. diff --git a/skills/katalon-true-platform-testing/references/requirement-analysis.md b/skills/katalon-true-platform-testing/references/requirement-analysis.md new file mode 100644 index 000000000..66c3d1dea --- /dev/null +++ b/skills/katalon-true-platform-testing/references/requirement-analysis.md @@ -0,0 +1,33 @@ +# Requirement Analysis Checklist + +Use this checklist before writing tests from requirements. + +## Inputs To Gather + +- Requirement source: Jira key, Azure item, Katalon requirement ID, pasted text, or URL. +- Target AUT/environment. +- Persona or role. +- Business objective. +- Acceptance criteria. +- Known constraints, data, permissions, and dependencies. +- Sprint/release scope if test assets should be organized by iteration. + +## Analysis Output + +Produce: + +- Requirement summary: what must be true for the feature to be accepted. +- Main flow: happy path from user intent to expected outcome. +- Alternate flows: optional paths, branching choices, pagination, sorting, filtering, retries. +- Negative flows: invalid data, missing data, permissions, empty states, out-of-stock or unavailable state. +- Data matrix: values needed for manual tests. +- Risk matrix: high-risk areas that deserve P0/P1 tests. +- Coverage map: requirement or story -> proposed test cases. + +## Katalon Requirement Handling + +- Use `find_requirements` to search synced Jira/Azure requirements. +- Use `read_requirement` for a specific requirement by source key or internal ID. +- Use `find_test_cases_by_requirement` to inspect existing coverage. +- Use `link_requirements_to_test_case` after test cases exist. +- Do not claim that the MCP can create requirements; current available tools only find/read/link synced requirements. diff --git a/skills/katalon-true-platform-testing/references/unavailable-capabilities.md b/skills/katalon-true-platform-testing/references/unavailable-capabilities.md new file mode 100644 index 000000000..2cb5fcf95 --- /dev/null +++ b/skills/katalon-true-platform-testing/references/unavailable-capabilities.md @@ -0,0 +1,32 @@ +# Katalon MCP Capability Boundaries + +## Available + +- Project discovery: `list_projects`. +- Repository/Test Project discovery: `list_repositories`. +- Requirement discovery: `find_requirements`, `read_requirement`. +- Requirement coverage: `fetch_requirement_data`. +- Test case operations: `create_test_case`, `read_test_case`, `update_test_case`, `duplicate_test_case`, `delete_test_case`, `move_test_case`, `find_test_cases`. +- Test folder operations: `find_test_folders`, `manage_test_folder`. +- Test suite operations: `find_test_suites`, `read_test_suite`, `manage_test_suite`. +- Requirement links: `link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`. +- Manual execution: `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`. +- Automated execution: `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, `build_schedule`, `schedule_test_run`. +- Execution results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Quality data: requirement, defect, test case, test stability, and configuration coverage fetch tools. +- ALM defects: `find_alm_integration_projects`, `create_defect`. + +## Not Directly Available + +- Create requirements in Katalon True Platform. Requirements are synced from Jira/Azure and can be found/read/linked. +- Create a formal Test Plan entity. Use test suites/folders/executions as the executable planning structure. +- Guarantee Run with AI completion. The platform may block, fail, or require AUT/account state. +- Inspect AUT pages through Katalon MCP. Use Browser/Playwright for website exploration. +- Create defects without a failed test result ID and ALM integration details. + +## Recommended Workarounds + +- Requirement creation: create in Jira/Azure first, then sync/find/link in Katalon. +- Test plan: create a named folder and/or test suite, link to sprint/release, and create execution from that suite. +- AUT exploration: use Browser/Playwright to understand the product, then import manual cases into Katalon. +- AI execution blocked: report blocked state with required fixture, AUT, account, or environment action. diff --git a/skills/katalon-upload-report/SKILL.md b/skills/katalon-upload-report/SKILL.md new file mode 100644 index 000000000..6586f53ce --- /dev/null +++ b/skills/katalon-upload-report/SKILL.md @@ -0,0 +1,183 @@ +--- +name: katalon-upload-report +description: Run automation and upload or verify Katalon Platform reports for Katalon Studio/KRE, JUnit XML, and Playwright reports. Use when you need to combine Katalon MCP project/result discovery with Katalon CLI execution, Katalon Report Uploader, or @katalon/playwright-reporter; configure report folders, report types, Platform/TestOps IDs, verify uploaded runs, handle failed auto-upload, or summarize execution results after automation completes. Written for the automation tester whose framework has already run and whose report is sitting on disk. If the Playwright run has not happened yet, start at katalon-playwright-execute. +--- + +# Katalon Upload Report + +Use this skill after or during an automated execution when the user wants Katalon, JUnit, or Playwright reports collected, uploaded to Katalon Platform, and summarized. Use Katalon MCP for platform discovery and post-upload verification; use the report-specific CLI/uploader for local execution and upload. + +## Capability Boundary + +State the boundary before running or uploading: + +- Katalon CLI/KRE can run local automation and generate Katalon Studio reports. With Platform/TestOps arguments configured, current Katalon Studio/Runtime Engine executions are expected to upload results automatically to Katalon True Platform. +- Katalon Report Uploader can upload Katalon Studio and JUnit reports to Katalon True Platform with report `type` values such as `katalon` and `junit`. +- `@katalon/playwright-reporter` can generate and upload Playwright report data to Katalon True Platform during `npx playwright test` or through a separate `playwright-reporter upload` command. +- Katalon MCP can discover projects/repositories, read executions/results, find test results, and fetch quality data when those tools are exposed. +- Katalon MCP does not inspect the live AUT UI, create requirements, guarantee report import completion, or replace Katalon CLI execution. + +Read `references/capability-boundaries.md` when tool availability is unclear. Read `references/cli-report-upload.md` before composing CLI commands or choosing the Katalon, JUnit, or Playwright upload flow. + +## Required Inputs + +Resolve or ask for only the missing required values: + +- Report type: `katalon`, `junit`, or `playwright`. +- Execution mode: run tests now, upload an existing report, or verify a report that should already be uploaded. +- Report path: Katalon report folder, JUnit XML file/folder, or Playwright reporter output directory. +- Test target when running tests: Katalon test suite/collection path, Playwright project/test command, or explicit command provided by the user. +- Katalon API key through a secure mechanism or existing environment variable. +- Katalon Platform/TestOps project ID. +- Organization ID for Katalon Studio/KRE execution when required. +- Target Platform/TestOps server only when not default or when using a non-production environment. +- Browser/device/execution profile when required by the project. + +Never ask the user to paste secrets into normal chat when a secure secret mechanism exists. + +## MCP Context First + +When Katalon MCP tools are available: + +1. Call `list_projects`. +2. Call `list_repositories`. +3. Resolve the repository/Test Project. +4. If execution IDs, suite IDs, or project IDs are provided, read them before running. + +If Katalon MCP tools are missing, continue with CLI-only execution only when all CLI inputs are available, and clearly state that platform verification cannot be completed through MCP in this session. + +## Choose Report Flow + +Use this decision order: + +1. If the user says Playwright or the repo has `@playwright/test`, use the Playwright flow. +2. If the user provides `.xml` JUnit files or says JUnit, use the JUnit flow. +3. If the user provides a Katalon Studio/KRE report folder or `.prj` run target, use the Katalon flow. +4. If unclear, inspect local files for `.prj`, `playwright.config.*`, or JUnit XML before asking. + +Do not convert Playwright to JUnit unless the user already has only JUnit XML output or explicitly requests JUnit upload. + +## Preflight + +Before running: + +1. Verify the required CLI exists: Katalon CLI/KRE, Java/Docker for Report Uploader, or Node/npm/pnpm for Playwright. +2. Verify source paths exist: `.prj`, report folder, JUnit XML, Playwright config, or Playwright output directory. +3. Create or clean only a run-specific report folder. Do not delete broad `Reports/`, `test-results/`, or `playwright-report/` folders. +4. Build the command with redacted secrets for display. + +## Katalon Report Flow + +When running Katalon Studio/KRE automation, prefer this command shape for current True Platform uploads: + +```text +katalonc -noSplash -runMode=console \ + -projectPath="" \ + -testSuitePath="Test Suites/" \ + -executionProfile="" \ + -browserType="" \ + -apiKey="$KATALON_API_KEY" \ + -orgID="" \ + -testOpsProjectId="" \ + --config \ + -webui.autoUpdateDrivers=true \ + -reportFolder="" +``` + +Adapt for `katalon.exe`, suite collections, mobile, API-only suites, proxy settings, retries, and repository-specific conventions. + +When uploading an existing Katalon Studio report folder, use Report Uploader with `--type=katalon` if auto-upload did not happen or the user explicitly asks for uploader-based upload. + +## JUnit Report Flow + +Use this flow when the input is JUnit XML from any automation framework: + +1. Verify the XML file or report folder exists. +2. Verify the files are JUnit-style XML, not Playwright HTML or trace artifacts. +3. Use Katalon Report Uploader with `--type=junit`. +4. Prefer API key as the uploader password value, sourced from a secure variable. +5. After upload, use MCP result tools to find/read the new run when available. + +Command shape: + +```text +java -jar katalon-report-uploader-.jar \ + --projectId="" \ + --path="" \ + --password="$KATALON_API_KEY" \ + --type=junit \ + --server="" +``` + +Omit `--server` when the default target is correct. + +## Playwright Report Flow + +Use this flow when the input is Playwright: + +1. Verify Node.js, `@playwright/test`, and Playwright config. +2. Install or verify `@katalon/playwright-reporter`. +3. Configure `playwright.config.ts` or `playwright.config.js` with the Katalon reporter and stable `outputDir`, typically `./reports`. +4. Upload during execution by setting `KATALON_API_KEY` and `KATALON_PROJECT_ID`, or upload separately with `playwright-reporter upload`. +5. Preserve Playwright screenshots, videos, and traces as attachments when configured. +6. Verify uploaded runs through MCP when possible. + +Upload during execution: + +```text +KATALON_API_KEY="$KATALON_API_KEY" \ +KATALON_PROJECT_ID="" \ +KATALON_BASE_URL="" \ +npx playwright test +``` + +Separate upload: + +```text +playwright-reporter upload \ + --output "" \ + --api-key "$KATALON_API_KEY" \ + --project-id "" \ + --base-url "" +``` + +Omit `KATALON_BASE_URL` or `--base-url` when the default target is correct. + +## Run And Collect + +1. Run the Katalon CLI command. +2. For JUnit-only uploads, run the producing test command only if requested; otherwise upload the existing report. +3. For Playwright, run `npx playwright test` or the user's package script when execution is requested. +4. Preserve stdout/stderr logs, but redact secrets before summarizing. +5. Locate the generated report folder. +6. Check for common artifacts such as execution logs, JUnit XML, HTML reports, screenshots, videos, traces, and attachments. +7. Record local pass/fail/error signals from CLI exit code and report files. + +If the CLI run fails before report generation, report the failure and skip upload verification unless a partial report exists. + +## Upload Or Confirm Upload + +For current Katalon True Platform: + +1. Treat Katalon CLI execution with `-apiKey`, `-orgID`, and `-testOpsProjectId` as the Katalon Studio report upload path. +2. Treat Report Uploader with `--type=katalon` or `--type=junit` as the Katalon/JUnit report upload path when uploading existing reports. +3. Treat `@katalon/playwright-reporter` or `playwright-reporter upload` as the Playwright report upload path. +4. Use MCP result tools after the run to find/read the new execution or test results. +5. If the platform shows the run as importing, poll practical intervals when a read/poll tool exists. +6. If auto-upload fails but local report files exist, state the available manual upload path and do not claim the upload succeeded. + +## Report Back + +Always report: + +- Report type: `katalon`, `junit`, or `playwright`. +- CLI command status, with secrets redacted. +- Local report folder path. +- Upload status: verified uploaded, importing, failed, or not verifiable because MCP is unavailable. +- Platform execution/run link when returned. +- Passed, failed, skipped, blocked/incomplete, and error counts when available. +- Failed tests and concise failure reasons. +- Artifacts created: HTML/XML/screenshots/videos/logs. +- Follow-up needed for missing credentials, import failure, or manual upload. + +Do not mark the task complete merely because local automation finished; completion requires either verified Platform upload or a clear explanation that upload verification is blocked. diff --git a/skills/katalon-upload-report/references/capability-boundaries.md b/skills/katalon-upload-report/references/capability-boundaries.md new file mode 100644 index 000000000..2cb5fcf95 --- /dev/null +++ b/skills/katalon-upload-report/references/capability-boundaries.md @@ -0,0 +1,32 @@ +# Katalon MCP Capability Boundaries + +## Available + +- Project discovery: `list_projects`. +- Repository/Test Project discovery: `list_repositories`. +- Requirement discovery: `find_requirements`, `read_requirement`. +- Requirement coverage: `fetch_requirement_data`. +- Test case operations: `create_test_case`, `read_test_case`, `update_test_case`, `duplicate_test_case`, `delete_test_case`, `move_test_case`, `find_test_cases`. +- Test folder operations: `find_test_folders`, `manage_test_folder`. +- Test suite operations: `find_test_suites`, `read_test_suite`, `manage_test_suite`. +- Requirement links: `link_requirements_to_test_case`, `unlink_requirements_from_test_case`, `find_test_cases_by_requirement`. +- Manual execution: `read_auts`, `create_manual_test_run`, `create_manual_ai_session`, `read_manual_ai_session`. +- Automated execution: `find_execution_profiles`, `list_test_cloud_environments`, `build_run_configuration`, `build_schedule`, `schedule_test_run`. +- Execution results: `read_execution`, `read_execution_test_results`, `read_test_result`, `find_test_results`. +- Quality data: requirement, defect, test case, test stability, and configuration coverage fetch tools. +- ALM defects: `find_alm_integration_projects`, `create_defect`. + +## Not Directly Available + +- Create requirements in Katalon True Platform. Requirements are synced from Jira/Azure and can be found/read/linked. +- Create a formal Test Plan entity. Use test suites/folders/executions as the executable planning structure. +- Guarantee Run with AI completion. The platform may block, fail, or require AUT/account state. +- Inspect AUT pages through Katalon MCP. Use Browser/Playwright for website exploration. +- Create defects without a failed test result ID and ALM integration details. + +## Recommended Workarounds + +- Requirement creation: create in Jira/Azure first, then sync/find/link in Katalon. +- Test plan: create a named folder and/or test suite, link to sprint/release, and create execution from that suite. +- AUT exploration: use Browser/Playwright to understand the product, then import manual cases into Katalon. +- AI execution blocked: report blocked state with required fixture, AUT, account, or environment action. diff --git a/skills/katalon-upload-report/references/cli-report-upload.md b/skills/katalon-upload-report/references/cli-report-upload.md new file mode 100644 index 000000000..086295842 --- /dev/null +++ b/skills/katalon-upload-report/references/cli-report-upload.md @@ -0,0 +1,63 @@ +# Katalon CLI And Report Upload Reference + +Use this reference when a workflow needs to run automation locally, collect Katalon/JUnit/Playwright reports, upload or confirm upload to Katalon Platform, and summarize the result. + +## Official Docs Checked + +- Katalon Runtime Engine command-line syntax: https://docs.katalon.com/katalon-studio/execute-tests/katalon-runtime-engine/command-line-syntax-in-katalon-runtime-engine +- Upload test results from Katalon Studio to Katalon True Platform manually: https://docs.katalon.com/katalon-studio/test-reports/upload-test-results-from-katalon-studio-to-katalon-testops-manually +- View test reports in Katalon True Platform: https://docs.katalon.com/katalon-platform/analyze/view-test-results +- Katalon True Platform Report Uploader: https://docs.katalon.com/katalon-platform/integrations/ci-cd/report-uploader-integration +- Playwright integration: https://docs.katalon.com/katalon-platform/integrations/testing-framework/playwright-integration +- Katalon Report Uploader Legacy: https://docs.katalon.com/katalon-platform/analyze/reports/upload-test-reports/use-katalon-report-uploader + +## Katalon Reports + +- After Katalon Studio/Runtime Engine execution, test results are expected to upload automatically to Katalon True Platform when TestOps/Platform arguments are configured. +- Use CLI arguments such as `-apiKey`, `-orgID`, `-testOpsProjectId`, and `-reportFolder` with the normal execution command. +- Keep `-reportFolder` deterministic so the agent can locate generated artifacts after the run. +- Use Katalon MCP result tools after the run to confirm whether the platform imported the execution. + +## JUnit Reports + +Use Katalon Report Uploader when the input is existing JUnit XML or a report folder containing JUnit XML. + +Common inputs: + +- `PROJECT_ID`: Katalon Platform/TestOps project ID. +- `REPORT_PATH`: Local report folder path. +- `TYPE`: `junit`. +- `PASSWORD`: Prefer an API key, never a plaintext password. +- `SERVER`: Optional Platform/TestOps endpoint. + +## Report Uploader Types + +Current Katalon True Platform Report Uploader supports: + +- `katalon`: Katalon Studio report folder. +- `junit`: JUnit XML report file/folder. +- `katalon_recorder`: Katalon Recorder reports. + +Use `katalon` and `junit` for this skill unless the user explicitly asks for Katalon Recorder. + +## Playwright Reports + +Use `@katalon/playwright-reporter`, not the generic JUnit uploader, when the source project is Playwright and can be configured. + +Required inputs: + +- `KATALON_API_KEY`: API key for upload. +- `KATALON_PROJECT_ID`: target Katalon project ID. +- `KATALON_BASE_URL`: optional custom Platform URL. +- `KATALON_AUTH_URL`: optional custom auth URL. + +The reporter can upload during `npx playwright test` or upload a generated output directory later through `playwright-reporter upload --output --api-key --project-id `. + +If the user only has JUnit XML produced by Playwright and does not want to configure the Katalon reporter, use the JUnit flow and state that Playwright-specific attachments/metadata may be reduced. + +## Secret Handling + +- Never print API keys, passwords, cookies, or raw auth callback URLs. +- Prefer existing environment variables or secure secret stores. +- Redact command output before reporting it in chat. +- Do not write secrets into skill files, repo files, test reports, or logs. From 852fa94e54756a42510cd9fe7acd4d7038b32871 Mon Sep 17 00:00:00 2001 From: Huy Tieu Date: Wed, 9 Sep 2026 14:14:41 +0700 Subject: [PATCH 2/2] Address risk-scan findings: drop npx auto-yes from shell examples Both High findings were `npx -y` in copy-paste shell commands. Removed the auto-yes so the reader approves the install themselves, and noted why the spawned-server JSON/TOML config still needs `-y` (no TTY to answer the prompt on). Also unpinned-version: `npm init playwright@latest` -> `npm init playwright`. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01VjgFMQPShixvLoyCZfwrfL --- skills/katalon-platform-setup/SKILL.md | 6 ++++-- .../references/playwright-typescript.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/skills/katalon-platform-setup/SKILL.md b/skills/katalon-platform-setup/SKILL.md index 94f5a7300..0ad948179 100644 --- a/skills/katalon-platform-setup/SKILL.md +++ b/skills/katalon-platform-setup/SKILL.md @@ -75,9 +75,11 @@ When MCP tools are missing: 4. Fall back to the `mcp-remote` wrapper only for agents without native remote-MCP support (Kiro, Continue, Codex): ```sh - npx -y mcp-remote https://platform.katalon.io/mcp --transport http-first + npx mcp-remote https://platform.katalon.io/mcp --transport http-first ``` + On first run `npx` asks before installing `mcp-remote`; answer it yourself rather than pre-approving. The JSON/TOML config form below passes `-y` because a spawned MCP server has no terminal to answer on. + - Codex declares the same command/args in TOML under `[mcp_servers.katalon-prod-mcp]`. - For the exact file path per agent, read the repository `README.md` install section. 5. If the install command, package name, or server URL is not present in local context, use official Katalon-provided setup instructions or ask the user for the MCP package/source. Do not invent install commands. @@ -100,7 +102,7 @@ Use this when Katalon MCP tools are missing from the active session or direct re 3. Run the proxy with the Katalon endpoint: ```sh - npx -y mcp-remote "https://platform.katalon.io/mcp" --transport http-first + npx mcp-remote "https://platform.katalon.io/mcp" --transport http-first ``` If this prints an authorization URL or opens the browser, wait for the user/browser callback to complete. `mcp-remote` stores OAuth state under its own auth cache, such as `~/.mcp-auth`, not in the workspace. diff --git a/skills/katalon-test-case-to-playwright/references/playwright-typescript.md b/skills/katalon-test-case-to-playwright/references/playwright-typescript.md index dcc44903a..d677a4126 100644 --- a/skills/katalon-test-case-to-playwright/references/playwright-typescript.md +++ b/skills/katalon-test-case-to-playwright/references/playwright-typescript.md @@ -27,7 +27,7 @@ fixtures/ katalon-test-data.ts ``` -Prefer `npm init playwright@latest` or the repository's package manager equivalent. Choose TypeScript, install browser dependencies only when needed, and avoid overwriting existing application files. +Prefer `npm init playwright` or the repository's package manager equivalent. Choose TypeScript, install browser dependencies only when needed, and avoid overwriting existing application files. ## Page Object Model