[skill] Add mz-tests skill for running and adding tests in Materialize#35218
[skill] Add mz-tests skill for running and adding tests in Materialize#35218mtabebe wants to merge 1 commit intoMaterializeInc:mainfrom
Conversation
|
Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone. PR title guidelines
Pre-merge checklist
|
Add a Claude skill that runs or helps add tests: sqllogictest (.slt), testdrive (.td), pgtest (.pt), and Rust unit tests. Includes a quick reference table, environment setup (cargo env), and steps for each framework plus guidance for adding new tests. Defers to doc/developer/guide-testing.md for more detail.
mtabebe
force-pushed
the
ma/skill/mz-tests
branch
from
dbdc408 to
d1a2cf7
Compare
| allowed-tools: [Bash, Read, Edit, Grep, Glob, Task] | ||
| --- | ||
|
|
||
| Run or help add tests in the Materialize repo. Supports sqllogictest, testdrive, pgtest, and Rust unit tests. |
There was a problem hiding this comment.
This is missing the most interesting test frameworks unfortunately, the zoo mentioned in https://www.notion.so/materialize/You-ve-been-handed-a-Bug-by-QA-what-now-26c13f48d37b807c88deed9257ae97ac
| - Always show the full error output before attempting fixes. | ||
| - Ask before modifying test files. | ||
| - Do NOT commit test changes — I'll decide when to commit. |
There was a problem hiding this comment.
Those sound like your individual preferences? Not sure they belong in a skill
| - For Rust: use `#[mz_ore::test]` (or `#[mz_ore::test(tokio::test)]` for async). | ||
| - Do NOT modify files in `test/sqllogictest/sqlite` or `test/sqllogictest/cockroach` (upstream). | ||
|
|
||
| 4. Show full output. If tests fail, show failures clearly and offer to help fix. |
There was a problem hiding this comment.
In my experience when you tell an LLM that, it will sometimes just "fix" the test by changing the expected output to what the actual output is, instead of fixing the bug in Materialize ;)
|
|
||
| ### Running `slt` (sqllogictest) | ||
| ```bash | ||
| ./bin/sqllogictest [--release] -- test/sqllogictest/FILE.slt -v |
There was a problem hiding this comment.
Please mention --optimized. It's what you should actually use by default. It's about as fast to build as Debug, but nearly as fast as Release to execute.
| cd test/testdrive | ||
| ./mzcompose run default FILE.td |
There was a problem hiding this comment.
Instead of changing directory just do bin/mzcompose --find testdrive run default FILE.td.
|
I wish we could have the same guide for humans as for LLMs, but that's more of a general complaint. |
2 participants
Add a Claude skill that runs or helps add tests: sqllogictest (.slt), testdrive (.td), pgtest (.pt), and Rust unit tests.
Includes a quick reference table, environment setup (cargo env), and steps for each framework plus guidance for adding new tests.
Defers to doc/developer/guide-testing.md for more detail.