[Bug]: trapperkeeper sends READY=1 too early to systemd

[bastelfreak]

Is this a critical security issue?

• This is not a security issue.

Describe the Bug

In #23 we added support for sd-notify. This is a systemd feature. It enables services to inform systemd when the service is up and running after a start/restart/reload. This is enabled by setting Type=notify-reload in the [Service] section (upstream docs).

Our implementation successfully sends READY=1 after a reload. But the service isn't fully started yet. This happens here: https://github.com/openvoxproject/trapperkeeper/blob/main/src/puppetlabs/trapperkeeper/internal.clj#L677-L684

(the two different services we manage via trapperkeeper are openvoxdb and openvox-server).

At this time trapperkeeper doesn't actually know if the service is fully up and running. The old startup implementation had a workaround for this. It parsed the restart-counter and looped until the value in the restart counter increased.

new systemd unit:

[Unit]
Description=puppetserver Service
After=syslog.target network.target nss-lookup.target

[Service]
Type=notify-reload
LogsDirectory=puppetlabs/puppetserver
RuntimeDirectory=puppetlabs/puppetserver
EnvironmentFile=/etc/sysconfig/puppetserver
User=puppet
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PrivateTmp=true
TasksMax=4915

#set default privileges to -rw-r-----
UMask=027

ExecStart=/usr/lib/jvm/jre-25/bin/java $JAVA_ARGS -Dlogappender=F1 \
  -XX:+CrashOnOutOfMemoryError \
  -XX:ErrorFile="${LOGS_DIRECTORY}/puppetserver_err_pid%p.log" \
  -cp "${INSTALL_DIR}/puppet-server-release.jar" \
  clojure.main \
  -m puppetlabs.trapperkeeper.main \
  --config "${CONFIG}" \
  --bootstrap-config "${BOOTSTRAP_CONFIG}" \
  $TK_ARGS

KillMode=process

ExecReload=kill -HUP $MAINPID
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

The old unit file:

[Unit]
Description=puppetserver Service
After=syslog.target network.target nss-lookup.target

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/puppetserver
User=puppet
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PIDFile=/run/puppetlabs/puppetserver/puppetserver.pid
PrivateTmp=true

# https://tickets.puppetlabs.com/browse/EZ-129
# Prior to systemd v228, TasksMax was unset by default, and unlimited. Starting in 228 a default of '512'
# was implemented. This is low enough to cause problems for certain applications. In systemd 231, the
# default was changed to be 15% of the default kernel limit. This explicitly sets TasksMax to 4915,
# which should match the default in systemd 231 and later.
# See https://github.com/systemd/systemd/issues/3211#issuecomment-233676333
TasksMax=4915

#set default privileges to -rw-r-----
UMask=027


ExecReload=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver reload
ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver start
ExecStop=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver stop

KillMode=process

SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

• /opt/puppetlabs/server/apps/puppetserver/bin/puppetserver: https://github.com/OpenVoxProject/ezbake/blob/2.7.8/resources/puppetlabs/lein-ezbake/staging-templates/cli-app.erb
• /opt/puppetlabs/server/apps/puppetserver/cli/apps/reload: https://github.com/OpenVoxProject/ezbake/blob/2.7.8/resources/puppetlabs/lein-ezbake/template/global/ext/cli/reload.erb
• /opt/puppetlabs/server/apps/puppetserver/ezbake-functions.sh: https://github.com/OpenVoxProject/ezbake/blob/2.7.8/resources/puppetlabs/lein-ezbake/template/global/ext/ezbake-functions.sh.erb

Expected Behavior

trapperkeeper sends READY=1\n to systemd when the service is actually started. openvox-server has a dedicated section for this: https://github.com/OpenVoxProject/openvox-server/blob/main/src/clj/puppetlabs/services/master/master_service.clj#L207-L210 . Same for OpenVoxDB: https://github.com/OpenVoxProject/openvoxdb/blob/main/src/puppetlabs/puppetdb/pdb_routing.clj#L131-L136

Steps to Reproduce

On EL10:

dnf update
dnf install epel-release
dnf install tig podman-docker bundler git make ruby-devel gcc
dnf install https://yum.voxpupuli.org/openvox8-release-el-10.noarch.rpm
git clone https://github.com/OpenVoxProject/openvox-server.git
cd openvox-server
docker build -t ezbake-builder .
bundle config set path .vendor
bundle install
rm -rf output/
EZBAKE_VERSION=4.0.0 bundle exec rake vox:build
dnf install output/el/10/openvox9/x86_64/openvox-server-*.el10.noarch.rpm
systemctl start puppetserver
puppet agent -t
systemctl reload puppetserver &
puppet agent -t # will now fail

Environment

openvoxdb / openvox-server built with ezbake 4.0.0

Additional Context

No response

Relevant log output