Skip to content

DAOS-19248 test: Fix no tag match for Functional VM stages#18597

Open
phender wants to merge 20 commits into
masterfrom
hendersp/DAOS-19247-fix
Open

DAOS-19248 test: Fix no tag match for Functional VM stages#18597
phender wants to merge 20 commits into
masterfrom
hendersp/DAOS-19247-fix

Conversation

@phender

@phender phender commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Ensure Functional VM stages are skipped if no test tags match.

Test-tag: test_always_passes_hw
Skip-functional-on-el-8: false
Skip-functional-on-el-9: false
Skip-functional-on-leap-15: false
Skip-functional-on-sles-15: false

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

Ensure Functional VM stages are skipped if no test tags match.

Test-tag: test_always_passes_hw
Skip-functional-on-el-8: false
Skip-functional-on-el-9: false
Skip-functional-on-leap-15: false
Skip-functional-on-sles-15: false

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender phender requested a review from daltonbohning July 1, 2026 18:06
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

Ticket title is 'Update testing branches'
Status is 'Resolved'
https://daosio.atlassian.net/browse/DAOS-19248

Comment thread Jenkinsfile Outdated
Skip-test-hardware: true
Test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/5/display/redirect

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/7/display/redirect

phender added 2 commits July 2, 2026 01:24
skip-unit-tests: true
skip-test-hardware: true
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

phender added 2 commits July 2, 2026 17:56
Skip-cancel-previous-builds: true
Skip-functional-test: true
SKip-fault-injection-testing: true
Skip-unit-tests: true

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@phender

phender commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/15/execution/node/831/log

In Build 15 the Test RPMs on Leap 15.5 was corrected to install and run on 15.5 - previously (in build 13) it was installing and running on 15.6. Now the stage fails with:

[2026-07-02T23:03:56.240Z] Error: 
[2026-07-02T23:03:56.240Z]  Problem: package daos-server-tests-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires daos-server = 2.9.100-4.50.g33213bba.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - package daos-server-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires libpmemobj1 = 2.1.3-2.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - conflicting requests
[2026-07-02T23:03:56.240Z]   - nothing provides libc.so.6(GLIBC_2.38)(64bit) needed by libpmemobj1-2.1.3-2.suse.lp156.x86_64

According to https://daosio.atlassian.net/browse/DAOS-18925, we need pciutils-3.13.0-150300.13.12.1.x86_64 but that's only available in 15.6.

~> sudo dnf provides pciutils
Last metadata expiration check: 0:01:33 ago on Fri Jul  3 03:30:09 2026.
allow_vendor_change is disabled. This option is currently not supported for downgrade and distro-sync commands
pciutils-3.5.6-150300.13.6.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : opensuse-oss-proxy-artifactory
Matched from:
Provide    : pciutils = 3.5.6-150300.13.6.1

pciutils-3.13.0-150300.13.12.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : @System
Matched from:
Provide    : pciutils = 3.13.0-150300.13.12.1

phender added 2 commits July 2, 2026 22:48
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
sjip-cancel-previous-builds: true
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-9: true
skip-fault-injection-testing: true

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/17/display/redirect

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-9: true
skip-fault-injection-testing: true

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/18/display/redirect

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@phender

phender commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/20/execution/node/1087/log

This failure is expected until https://daosio.atlassian.net/browse/DAOS-15233 has been resolved.

@phender

phender commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Note the improved artifacts for the RPM Test RPM stages:

@phender phender requested review from a team as code owners July 7, 2026 15:01
phender added 2 commits July 7, 2026 11:18
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender

phender commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/15/execution/node/831/log

In Build 15 the Test RPMs on Leap 15.5 was corrected to install and run on 15.5 - previously (in build 13) it was installing and running on 15.6. Now the stage fails with:

[2026-07-02T23:03:56.240Z] Error: 
[2026-07-02T23:03:56.240Z]  Problem: package daos-server-tests-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires daos-server = 2.9.100-4.50.g33213bba.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - package daos-server-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires libpmemobj1 = 2.1.3-2.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - conflicting requests
[2026-07-02T23:03:56.240Z]   - nothing provides libc.so.6(GLIBC_2.38)(64bit) needed by libpmemobj1-2.1.3-2.suse.lp156.x86_64

According to https://daosio.atlassian.net/browse/DAOS-18925, we need pciutils-3.13.0-150300.13.12.1.x86_64 but that's only available in 15.6.

~> sudo dnf provides pciutils
Last metadata expiration check: 0:01:33 ago on Fri Jul  3 03:30:09 2026.
allow_vendor_change is disabled. This option is currently not supported for downgrade and distro-sync commands
pciutils-3.5.6-150300.13.6.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : opensuse-oss-proxy-artifactory
Matched from:
Provide    : pciutils = 3.5.6-150300.13.6.1

pciutils-3.13.0-150300.13.12.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : @System
Matched from:
Provide    : pciutils = 3.13.0-150300.13.12.1

The resolution for this is run the Test RPMs on Leap 15 stage with Leap 15.6.

Comment thread ci/rpm/test_daos_node.sh

set -uex

echo "${PRETTY_NAME:-Unknown OS}"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also considered just running cat /etc/os-release, but this file is sourced in line 3.

Comment thread ci/rpm/test_daos.sh
PYTHON_VERSION=\"${PYTHON_VERSION}\" \
$(cat "$mydir/test_daos_node.sh")"
"bash -lc 'NODE=${nodelist[0]} DAOS_PKG_VERSION=$DAOS_PKG_VERSION PYTHON_VERSION=$PYTHON_VERSION \
bash -s 2>&1 | tee /tmp/test_daos_rpms.log; exit \${PIPESTATUS[0]}'" \

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered just redirecting all output to the log, but I figured it would be good to have a backup incase the logs didn't get archived for some reason.

Comment thread Jenkinsfile
label: params.CI_UNIT_VM1_LABEL,
jobStatus: job_status_internal,
nextVersion: next_version(),
imageVersion: 'el9.6',

@phender phender Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now runs on EL 9.6 as intended (instead of EL 9.7)

Comment thread Jenkinsfile
label: params.CI_UNIT_VM1_LABEL,
jobStatus: job_status_internal,
nextVersion: next_version(),
imageVersion: 'leap15.6',

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Continues to run on Leap 15.6

Comment thread Jenkinsfile
imageVersion: 'el9.6',
instRpms: 'mercury-libfabric',
nodeList: env.NODELIST,
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on EL 9\'',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is weird now because the reference to ci/rpm/test_daos.sh is hardcoded in pipeline-lib, but ci/rpm/test_daos_post.sh is specified here. IMO they should both be passed here and not hardcoded in pipeline-lib

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, if that is a bigger fix I do not want to block on it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated defaults in daos-stack/pipeline-lib#526

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is

  • test_daos.sh
    • lives in daos-stack/daos
    • hardcoded reference in daos-stack/pipeline-lib
      • IMO this is bad design but is pre-existing
  • test_daos_post.sh
    • lives in daos-stack/daos
    • argument reference in daos-stack/daos

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. Yes I feel it would be a bit out of scope to address updating the defaults in testRpm.groovy as they would still affect other brnaches.

Comment thread ci/rpm/test_daos_post.sh Outdated
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender phender requested a review from daltonbohning July 9, 2026 03:31
@daosbuild3

Copy link
Copy Markdown
Collaborator

phender added 2 commits July 9, 2026 01:41
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender phender requested a review from grom72 July 9, 2026 16:33
Comment thread Jenkinsfile
imageVersion: 'el9.6',
instRpms: 'mercury-libfabric',
nodeList: env.NODELIST,
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on EL 9\'',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is

  • test_daos.sh
    • lives in daos-stack/daos
    • hardcoded reference in daos-stack/pipeline-lib
      • IMO this is bad design but is pre-existing
  • test_daos_post.sh
    • lives in daos-stack/daos
    • argument reference in daos-stack/daos

@grom72 grom72 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you adjust PR description - it is hard to follow the goal of this changes.

Comment thread Jenkinsfile
Comment on lines +1315 to +1316
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
}
}
} // parallel
} // script

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The team seems to go back and forth on these types of comments. I've maintained them in the past only to have someone else remove them. When viewing code in a modern IDE, I think we can leave them out.

Comment thread Jenkinsfile
archiveArtifactsArgs: [
artifacts: 'Test RPMs on EL 9/']
),
'Test RPMs on Leap 15': scriptedTestRpmStage(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something is wrong here:
We have to use the string 'Test RPMs on Leap 15' in 5 places

@phender phender Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of this is inevitable:

  • one is used for the build parameter name (trying to keep it in sync with the stage name for ease of use)
  • three are used for the stage name - one to define a unique parallel() entry, another to define the stage name in the method linked to said entry, and a third to reference the aforementioned matching build parameter name.

The other two are used as part of the runStage() map. I did try to reduce the number of times the stages are repeated here, but the only clean way to seed the runStage map w/ the stage names from the Jenkins build parameters keys resulted in an unordered list. Having an ordered list is very useful for displayRunStage(), so having one reference in the stageOrder list won out. The other references in updateRunStage() are needed for processing special build params / modes/ commit pragmas - for this stage there is only one when handling quickFunctional().

I did keep maintainability in mind, and at least a simple search and replace should cover most updates.

Comment thread Jenkinsfile
imageVersion: 'leap15.6',
instRpms: 'mercury-libfabric',
nodeList: env.NODELIST,
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on Leap 15\'',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's indicate when this script is used.

Suggested change
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on Leap 15\'',
alwaysPostScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on Leap 15\'',

@phender phender Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The always_script argument name comes from the unitTest() method. It would be out of scope of this PR to update the names for the arguments in unitTest.groovy and I think it makes the code clearer if the argument names are maintained - especially when passed through.

Comment thread ci/rpm/test_daos.sh
Comment on lines +14 to +16
"bash -lc 'NODE=${nodelist[0]} DAOS_PKG_VERSION=$DAOS_PKG_VERSION PYTHON_VERSION=$PYTHON_VERSION \
bash -s 2>&1 | tee /tmp/test_daos_rpms.log; exit \${PIPESTATUS[0]}'" \
< "$mydir/test_daos_node.sh"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Less bash call easier to follow.

Suggested change
"bash -lc 'NODE=${nodelist[0]} DAOS_PKG_VERSION=$DAOS_PKG_VERSION PYTHON_VERSION=$PYTHON_VERSION \
bash -s 2>&1 | tee /tmp/test_daos_rpms.log; exit \${PIPESTATUS[0]}'" \
< "$mydir/test_daos_node.sh"
"export NODE='${nodelist[0]}' DAOS_PKG_VERSION='$DAOS_PKG_VERSION' PYTHON_VERSION='$PYTHON_VERSION';
bash -s < /dev/stdin |& tee /tmp/test_daos_rpms.log;
exit \${PIPESTATUS[0]}" \
< "$mydir/test_daos_node.sh"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I find that to be clearer. It might just be a personal preference thing.

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants