ENT-13699: Add shell test framework and migrate all tests

[larsewi]

Summary

• Added tests/shell/testlib.sh with lifecycle functions (test_init, test_finish, skip_unless_unsafe), assertion helpers (assert_file_exists, assert_file_contains, assert_failure, assert_equal, etc.), and capture utilities (run, run_expect_failure)
• Updated tests/shell/all.sh runner with per-test timing, pass/fail/skip reporting, and summary output
• Migrated all 47 shell tests to use the new framework, eliminating repeated boilerplate
• Consolidated common.sh git helpers into testlib.sh as assert_git_tracks and assert_git_no_diffs

🤖 Generated with Claude Code

Example output:

$ bash tests/shell/all.sh
Warning: These shell based tests use the cfbs you have installed
         If you haven't already, run: pip install .
--- PASS: 001_init (2s)
--- PASS: 002_add (2s)
--- PASS: 003_download (2s)
--- PASS: 004_build (12s)
--- PASS: 005_alias (3s)
--- PASS: 006_search (2s)
--- SKIP: 007_install (0s)
--- PASS: 008_remove (2s)
 ---snip---
--- PASS: 047_absolute_path_modules (4s)

==============================
Test Results: 46 passed, 0 failed, 1 skipped (total: 47, 156s)
==============================
All cfbs shell tests completed successfully!

If a test fails it will look like this:

$ bash tests/shell/all.sh
Warning: These shell based tests use the cfbs you have installed
         If you haven't already, run: pip install .
--- FAIL: 001_init (6s)
++ _test_name_from_path tests/shell/001_init.sh
++ basename tests/shell/001_init.sh .sh
+ _TEST_NAME=001_init
+ rm -rf ./tests/tmp/
+ mkdir -p ./tests/tmp/
+ cd ./tests/tmp/
++ date +%s
+ _TEST_START_TIME=1771345070
+ echo '--- START: 001_init ---'
--- START: 001_init ---
+ cfbs --non-interactive init
Initialized empty Git repository in /home/larsewi/ntech/cfbs/tests/tmp/.git/
[main (root-commit) bcfc025] Initialized a new CFEngine Build project
 1 file changed, 7 insertions(+)
 create mode 100644 cfbs.json
[main d75f629] Added module 'masterfiles'
 1 file changed, 16 insertions(+), 1 deletion(-)
Committing using git:


Initialized an empty project called 'Example project' in 'cfbs.json'
Added module: masterfiles
Committing using git:


+ assert_file_exists cfbs.json
+ '[' -e cfbs.json ']'
+ false
---
--- PASS: 002_add (2s)
--- PASS: 003_download (4s)
 ---snip---

[larsewi]

@cf-bottom Jenkins please :)

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/13396/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-13396/

[craigcomstock]

I can't say I love this PR. There is a lot of change and now the scripts can't be written free-hand in bash... you must refer to the testlib often to know what functions to use.

Also, I am not sure how much is improved other than maybe silencing "good" outputs to make analysis easier... which is a welcome change so +1 on that account.

I think all the migrated commits could be squashed into one for easier review... #next-time

[craigcomstock]

In order to debug fully it would be nice to save the complete output to an artifacts directory that is then slurped up and archived with the github workflow.

[craigcomstock]

is shellcheck running on this? It should complain about x-prefix comparisons that are not generally needed.

[craigcomstock]

assert_file_matches_regex() would be more descriptive... I was thinking "entire file match" as-is.

[craigcomstock]

maybe assert_string_occurrences() to be clear: not a regex (string instead) and check the number of matches.

[craigcomstock]

why this change? The result of the change would add a prefixed backslash for no reason right?

R: Hello from \foo

[craigcomstock]

probably don't need this prefixed backslash in the output.

[craigcomstock]

and at no time are we testing the output of this policy so pretty inconsequential either way.