SRE-3893 build: Consolidate build scripts by grom72 · Pull Request #19016 · daos-stack/daos

grom72 · 2026-09-04T11:11:37Z

Summary

Consolidate dependency installation, dependency builds, DAOS builds,
package generation, and RPM verification under utils/build. Use these
entry points consistently from Jenkins, Docker builds, and bare-host
workflows while retaining support for custom scons options.

By default, generate dependency and DAOS RPMs directly under
<repo_root>/rpms/deps and <repo_root>/rpms/daos, respectively, and
create repository metadata under <repo_root>/rpms/repodata. Allow
RPM_OUTPUT_DIR to override the repository root.

Add distro auto-detection, optional external dependency repositories,
and configurable package-verification severity. Remove the superseded
wrappers from ci/rpm and utils/rpms.

Update Jenkins to copy dependency RPMs produced in the Docker build
image into the workspace repository so that dependency and DAOS RPMs
are included in the artifact upload.

Changes

Add shared utils/build scripts for:
- installing pre-built dependencies;
- building missing dependencies;
- building DAOS;
- generating and verifying packages.
Remove the superseded ci/rpm and utils/rpms/build_packages.sh wrappers.
Add automatic EL and openSUSE Leap distro detection.
Support an optional temporary dependency repository through
DAOS_DEPS_EXT_REPO for the install_deps.sh script (to be used mainly as bare-host builds pre-step).
Allow additional options and variables to pass through to scons in build_daos.sh and build_deps.sh scripts.
Update the EL9 and Leap 15 Docker build flows to use the unified scripts.
Update Jenkins RPM generation and artifact collection.
Document the unified four-step build procedure.

Package verification

Package verification runs automatically after each nonempty RPM group.

verify=yes treats validation findings as errors.
verify=no reports validation findings as warnings.
Missing tooling and other critical setup errors remain fatal.

Requires:

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-09-04T11:11:57Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3893

daosbuild3 · 2026-09-04T11:33:42Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/2/execution/node/442/log

daosbuild3 · 2026-09-04T13:27:26Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/3/execution/node/493/log

daosbuild3 · 2026-09-07T09:27:31Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/10/execution/node/858/log

Consolidate dependency installation, dependency builds, DAOS builds, package generation, and RPM verification under utils/build. Use these entry points consistently from Jenkins, Docker builds, and bare-host workflows while retaining support for custom scons options. By default, generate dependency and DAOS RPMs directly under <repo_root>/rpms/deps and <repo_root>/rpms/daos, respectively, and create repository metadata under <repo_root>/rpms/repodata. Allow RPM_OUTPUT_DIR to override the repository root. Add distro auto-detection, optional external dependency repositories, and configurable package-verification severity. Remove the superseded wrappers from ci/rpm and utils/rpms. Update Jenkins to copy dependency RPMs produced in the Docker build image into the workspace repository so that dependency and DAOS RPMs are included in the artifact upload. skip-cancel-previous-builds: true skip-python-bandit-check: true skip-unit-tests: true skip-test: true skip-test-hardware: true skip-functional-on-leap-15: false Priority: 2 Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Document the shared scripts for installing dependencies, building DAOS, generating packages, and creating a complete RPM repository. Update Docker examples and packaging script paths to match the new layout. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> skip-cancel-previous-builds: true skip-python-bandit-check: true skip-unit-tests: true skip-test: true skip-test-hardware: true skip-functional-on-leap-15: false

Move distro detection from build_packages.sh to build_utils.sh so it can also be reused by install_deps.sh. Propagate detection failures explicitly from command substitutions and avoid importing variables from /etc/os-release into the calling shell. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2

janekmi · 2026-09-10T21:11:13Z

+
+## Unified DAOS Build Procedure
+
+The scripts under [`utils/build`](../../utils/build) provide a straightforward,


We shouldn't use relative ULRs in this document. This content is published on https://docs.daos.io/ where the reference to GitHub is completly lost and these links are broken there.

At the same time, I understand we should link to a specific branch e.g. release/2.8 so the content of the file matches the content of the document linking to it.

Good luck.

Ref: https://docs.daos.io/v2.8/dev/development/#updating-a-3rd-party-component

It is not a part of this PR to prepare official documentation for publishing.

janekmi · 2026-09-10T21:16:12Z

+   dependencies can also be built from source in the next step. Set
+   `DAOS_DEPS_EXT_REPO` to pull in a custom RPM set published by the DAOS
+   project (e.g. from [packages.daos.io](https://packages.daos.io/)) as
+   an extra dnf-format repo, registered only for the duration of the script.


I am not sure what dnf-format means in this context. And actually you adding a yum repository. So, I would go with the following.

Suggested change

an extra dnf-format repo, registered only for the duration of the script.

an extra repo, registered only for the duration of the script.

janekmi · 2026-09-10T21:18:40Z

+1. **`build_deps.sh`** builds any dependency not already satisfied by
+   `install_deps.sh` from source, via `scons install --build-deps=only`.
+1. **`build_daos.sh`** builds and installs DAOS itself with scons
+   (`USE_INSTALLED=all` to reuse the dependencies from the previous steps).


It is not clear from the text if it is something the user has to add themselve or is it an automatic bit?

janekmi · 2026-09-10T21:20:12Z

+   (`USE_INSTALLED=all` to reuse the dependencies from the previous steps).
+1. **`build_packages.sh [deps|daos|all] [yes|no]`** builds dependency
+   RPMs, DAOS RPMs, or both with `fpm`. The build type defaults to `all`,
+   and verification defaults to `yes`.


It is only implied the second argument is about the verification of the produced packages. You may want to introduce it properly here.

janekmi · 2026-09-10T21:23:26Z

+   `RPM_OUTPUT_DIR`, which defaults to `<repo_root>/rpms`. For supported RPM
+   distributions, the script also generates repository metadata under


What is a "RPM distribution"? I think without the "RPM" bit it is clear you mean a Linux distribution. Because you mean a Linux distribution, don't you?

Suggested change

`RPM_OUTPUT_DIR`, which defaults to `<repo_root>/rpms`. For supported RPM

distributions, the script also generates repository metadata under

`RPM_OUTPUT_DIR`, which defaults to `<repo_root>/rpms`. For supported

distributions, the script also generates repository metadata under

janekmi · 2026-09-10T23:13:02Z

+ARG DAOS_TARGET_TYPE=
+ARG DEPS_JOBS=
+
+ARG DAOS_BUILD_TYPE=$DAOS_TARGET_TYPE


Why these arguments are named differently by mean the same thing? 😕

Please ask scons ;)

janekmi · 2026-09-10T23:38:54Z

+        utils/build/build_deps.sh \
+            ${DEPS_JOBS:+--jobs "$DEPS_JOBS"} \
+            ${DAOS_BUILD_TYPE:+BUILD_TYPE="$DAOS_BUILD_TYPE"} \
+            ${DAOS_TARGET_TYPE:+TARGET_TYPE="$DAOS_TARGET_TYPE"} && \


Does truely the cleanup step depend on the build step hence you need && ?

It is not about cleanup it is about failing the whole Docker step.

janekmi · 2026-09-10T23:40:53Z

+        [ "$DAOS_KEEP_BUILD" != "no" ] || { \
        rm -rf .sconf_temp .sconsign.dblite config.log && \
-        ([ "$DAOS_KEEP_BUILD" != "no" ] || /bin/rm -rf build *.gz); \
+            /bin/rm -rf build *.gz; \


Indentation.

/bin/rm seriously?

You can probably combine both calls.

Suggested change

/bin/rm -rf build *.gz; \

rm -rf .sconf_temp .sconsign.dblite config.log build *.gz;

janekmi · 2026-09-10T23:49:23Z

 ARG DAOS_DEPS_BUILD
 ARG DAOS_BUILD=$DAOS_DEPS_BUILD
-ARG DAOS_JAVA_BUILD=$DAOS_BUILD
+ARG DAOS_JAVA_BUILD=no


Redundant?

Suggested change

ARG DAOS_JAVA_BUILD=no

ARG DAOS_JAVA_BUILD

janekmi · 2026-09-10T23:52:58Z

+        mkdir -p /home/daos/daos/src && \
+       cp -r /tmp/daos-src/. /home/daos/daos/src && \


Indentation.

Suggested change

mkdir -p /home/daos/daos/src && \

cp -r /tmp/daos-src/. /home/daos/daos/src && \

mkdir -p /home/daos/daos/src && \

cp -r /tmp/daos-src/. /home/daos/daos/src && \

daosbuild3 · 2026-09-11T07:30:08Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/36/execution/node/1869/log

grom72 force-pushed the grom72/SRE-3893-script2build branch from f31fa70 to db5ee07 Compare September 4, 2026 11:21

grom72 force-pushed the grom72/SRE-3893-script2build branch 4 times, most recently from a6ee981 to b423038 Compare September 4, 2026 20:37

grom72 force-pushed the grom72/SRE-3893-script2build branch 7 times, most recently from e0cbb87 to 32eb1f3 Compare September 9, 2026 15:03

janekmi requested changes Sep 9, 2026

View reviewed changes

grom72 force-pushed the grom72/SRE-3893-script2build branch 4 times, most recently from 6deddc7 to 93864f9 Compare September 10, 2026 09:23

grom72 added 3 commits September 10, 2026 18:20

grom72 force-pushed the grom72/SRE-3893-script2build branch from 93864f9 to 9d48806 Compare September 10, 2026 16:21

Final documentation polishing

282c33c

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2

grom72 requested a review from janekmi September 10, 2026 16:53

grom72 changed the title ~~SRE-3893 build: script2build~~ SRE-3893 build: Consolidate build scripts Sep 10, 2026

grom72 marked this pull request as ready for review September 10, 2026 16:54

grom72 requested a review from a team as a code owner September 10, 2026 16:54

grom72 requested review from daltonbohning, knard38 and phender and removed request for a team September 10, 2026 16:54

janekmi requested changes Sep 10, 2026

View reviewed changes


		## Unified DAOS Build Procedure

		The scripts under [`utils/build`](../../utils/build) provide a straightforward,

	an extra dnf-format repo, registered only for the duration of the script.
	an extra repo, registered only for the duration of the script.

		`RPM_OUTPUT_DIR`, which defaults to `<repo_root>/rpms`. For supported RPM
		distributions, the script also generates repository metadata under

	/bin/rm -rf build *.gz; \
	rm -rf .sconf_temp .sconsign.dblite config.log build *.gz;

		mkdir -p /home/daos/daos/src && \
		cp -r /tmp/daos-src/. /home/daos/daos/src && \

Uh oh!

Conversation

grom72 commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Package verification

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grom72 Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

grom72 commented Sep 4, 2026 •

edited

Loading

grom72 Sep 11, 2026 •

edited

Loading