Skip to content

server-builders.md

Latest commit

 

History

History
84 lines (57 loc) · 7.02 KB

server-builders.md

File metadata and controls

84 lines (57 loc) · 7.02 KB
Raw

Server builders

How a MariaDB Server change is built, where the server builders come from, and what to change to add a platform or a version.

From a push to packages

push or pull request on MariaDB/server
└─ tarball-docker                                   master-protected-branches
   │  source tarball saved at <ARTIFACTS_URL>/<tarbuildnum>/
   ├─ s_protected_branches: GITHUB_STATUS_BUILDERS in SUPPORTED_PLATFORMS[version]
   └─ s_upstream_all:       the rest of SUPPORTED_PLATFORMS[version], on any master
      └─ e.g. aarch64-debian-12                     autogen: RelWithDebInfo build, default MTR suites
         └─ aarch64-debian-12-deb-autobake          autogen: release packages, no tests
            ├─ ...-install                          master-libvirt
            ├─ ...-minor-upgrade-all, -major-upgrade, -distro-upgrade, ...
            └─ Docker Library tests                 master-nonlatent

tarbuildnum is the build number of tarball-docker. Every build that follows from it refers to it, and the packages are saved under <ARTIFACTS_URL>/<tarbuildnum>/<builder>/. The release workflows, in the release repository on git.mariadb.org, take the packages from there.

Packages are saved only for the branches in SAVED_PACKAGE_BRANCHES (the main branches, bb-*-release, preview-*, *pkgtest*), so the install and upgrade tests run only on those. Pull requests and bb-* branches get the test builders and the -autobake builds, without saving packages.

A trigger step uses the schedulers of the master it runs on, so the masters that run server builds all load the server's Triggerable schedulers (schedulers_definition.py); master-web, master-galera and master-migration replace them with their own. The build requests these schedulers create are taken by whichever master has the builder, so tarball-docker starts a server builder on any master, as long as the builder is in SUPPORTED_PLATFORMS for the version.

Versions and branches

Builders are chosen per server version. tarball-docker reads the version from the source (mariadb_version, e.g. mariadb-11.8.3) and keeps the major and minor (master_branch, 11.8), which selects SUPPORTED_PLATFORMS[master_branch]. Every version built needs:

  • an entry in BRANCHES_MAIN, so that pushes to its branch start a build and save packages;
  • an entry in SUPPORTED_PLATFORMS, usually a copy of the previous version's list. Without one, the trigger for that version fails.

Pushes to branches matching BB_TEST_BRANCHES (bb-*, st-*, prot-*, preview-1[0-9].*, jpsn-*) and pull requests are built too.

SUPPORTED_PLATFORMS and the other server lists in constants.py apply to server builders only. Galera, the Connectors and Foundry have their own entry points and settings.

Platforms: os_info.yaml

Each entry of os_info.yaml is a distribution release:

debian-12:
  image_tag: debian12        # build image: <CONTAINER_REGISTRY_URL><image_tag>
  tags:
    - release_packages       # shows its builders on the Server Release Status page
  arch: [amd64, aarch64, x86, ppc64le]
  type: deb                  # deb or rpm
  install_only: False        # True: only install/upgrade tests, see below

For every architecture of an entry, this generates:

Builder Master Does
<arch>-<os> autogen RelWithDebInfo build, default MTR suites (nm)
<arch>-<os>-<type>-autobake autogen Release packages, saved to ci.mariadb.org. No tests.
gal-<arch>-<os> master-galera Galera packages
...-autobake-install, -minor-upgrade-all, -major-upgrade, -distro-upgrade, -minor-upgrade-columnstore (amd64, aarch64) master-libvirt Install and upgrade tests, in a VM. None for s390x and x86, which have no VMs, nor for now for openSUSE 16.0.

An install_only entry (AlmaLinux, Rocky Linux) generates only the install and upgrade builders, which test the RHEL packages on that distribution. The derived lists are at the end of constants.py (BUILDERS_AUTOBAKE, BUILDERS_INSTALL, BUILDERS_UPGRADE, BUILDERS_GALERA).

The galera, s3 and ssl suites run on a <arch>-<os> builder when it is listed in BUILDERS_GALERA_MTR, BUILDERS_S3_MTR or BUILDERS_SSL_MTR.

A builder in GITHUB_STATUS_BUILDERS is not generated by the autogen masters, which still generate its -autobake builder. It is defined in master-protected-branches instead.

All the generated builders share the same factories (common_factories.py), so one of them can't easily do things differently. A builder with a non-standard configuration ends up defined on its own, on master-protected-branches or the master-docker-nonstandard masters. Define new ones on master-migration.

Autogen masters

define_masters.py creates autogen/<arch>-master-<n>/ from master.cfg, one or more masters per architecture: int(platforms / max_builds) + 1, with max_builds from master-private.cfg. Each master of an architecture gets all of its platforms, so the masters are duplicates and share the load: each takes build requests as it can. max_builds was tuned over time; it is an estimate, not an exact limit.

Check the number of masters when you change os_info.yaml. The script prints one line per master. The masters that run are the ones listed in MASTER_DIRECTORIES in generate-config.py, which gives them their ports in list order. A new master runs only once it is added there, and adding one changes the port of every master after it, while non-latent workers connect to a fixed port. A master that is no longer generated stays in the list, and its container fails to start. Plan either as a change of the infrastructure.

Adding a release platform

  1. Add its build image to the matrix of the image workflow of its family. Merging it to dev publishes the image for dev, merging it to main for production. See Build images. Until Galera packages exist for the platform, build the image with nogalera: true.
  2. Deploy its VMs for the install and upgrade tests, on the dev libvirt hosts first. See master-libvirt.
  3. Add it to os_info.yaml, and its <arch>-<os> builders to SUPPORTED_PLATFORMS for every version that supports it. A version is supported on a distribution when the distribution's own MariaDB package is the same major version or older.
  4. Check the number of autogen masters (see above).
  5. Once a Galera build has run for it on mariadb-4.x, rebuild its image with Galera.

Release builders on master-migration

The -autobake-migration builders on master-migration build the release packages, install them, and run MTR from the installed tree, as the old Buildbot (buildbot.mariadb.net) did. See master-migration.