Skip to content

Fix race condition causing sshd start failure during provisioning #460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
s4heid wants to merge 1 commit into
base: ubuntu-jammy
Choose a base branch
from
Open

Fix race condition causing sshd start failure during provisioning #460

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions stemcell_builder/stages/base_ubuntu_firstboot/apply.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ set -e
base_dir=$(readlink -nf $(dirname $0)/../..)
source $base_dir/lib/prelude_apply.bash

cp $assets_dir/etc/rc.local $chroot/etc/rc.local
cp $assets_dir/root/firstboot.sh $chroot/root/firstboot.sh
chmod u+x "${chroot}/etc/rc.local"
chmod 0755 $chroot/root/firstboot.sh
install -D -m 0644 \
$assets_dir/etc/systemd/system/firstboot.service \
$chroot/etc/systemd/system/firstboot.service

run_in_chroot $chroot "systemctl enable firstboot.service"
9 changes: 0 additions & 9 deletions stemcell_builder/stages/base_ubuntu_firstboot/assets/etc/rc.local
View file
Open in desktop

This file was deleted.

15 changes: 15 additions & 0 deletions stemcell_builder/stages/base_ubuntu_firstboot/assets/etc/systemd/system/firstboot.service
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[Unit]
Description=Run first boot tasks
ConditionPathExists=!/root/firstboot_done
Before=ssh.service
Copy link

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding "Before=sshd-keygen.service" to ensure firstboot.service completes key regeneration before the system's default SSH key generation service (sshd-keygen.service) attempts to run. While both services likely check for existing keys, explicitly ordering them would prevent any potential race condition between the two key generation mechanisms.

Suggested change
Before=ssh.service
Before=sshd-keygen.service ssh.service

Copilot uses AI. Check for mistakes.

[Service]
Type=oneshot
ExecStartPre=/bin/sh -c '/bin/rm -f /etc/ssh/ssh_host*key*'
ExecStart=/usr/bin/ssh-keygen -A -v
ExecStartPost=/usr/sbin/dpkg-reconfigure -fnoninteractive sysstat
ExecStartPost=/usr/bin/touch /root/firstboot_done
RemainAfterExit=yes

Comment on lines +9 to +13
Copy link

Copilot AI Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the ssh-keygen command in ExecStart fails, the ExecStartPost commands (including the creation of the marker file) will still execute. This could leave the system in an inconsistent state where firstboot_done exists but SSH keys were never generated. Consider moving the marker file creation to a separate ExecStartPost command that only runs on success, or use a more robust approach to ensure the service fails if key generation fails.

Suggested change
ExecStart=/usr/bin/ssh-keygen -A -v
ExecStartPost=/usr/sbin/dpkg-reconfigure -fnoninteractive sysstat
ExecStartPost=/usr/bin/touch /root/firstboot_done
RemainAfterExit=yes
ExecStart=/bin/sh -c '/usr/bin/ssh-keygen -A -v && /usr/bin/touch /root/firstboot_done'
ExecStartPost=/usr/sbin/dpkg-reconfigure -fnoninteractive sysstat
RemainAfterExit=yes

Copilot uses AI. Check for mistakes.
[Install]
WantedBy=multi-user.target
6 changes: 0 additions & 6 deletions stemcell_builder/stages/base_ubuntu_firstboot/assets/root/firstboot.sh
View file
Open in desktop

This file was deleted.

6 changes: 5 additions & 1 deletion stemcell_builder/stages/system_azure_init/apply.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,16 @@ cat > $chroot/etc/logrotate.d/waagent <<EOS
}
EOS

#setup cloud-init
# Setup cloud-init
rm $chroot/etc/cloud/*.cfg
rm $chroot/etc/cloud/cloud.cfg.d/*.cfg
cp -f $dir/assets/etc/cloud-init/cloud.cfg $chroot/etc/cloud/cloud.cfg
cp -f $dir/assets/etc/cloud-init/*-*.cfg $chroot/etc/cloud/cloud.cfg.d/

# Ensures that cloud-init waits until host keys have been regenerated.
mkdir -p $chroot/etc/systemd/system/cloud-config.service.d
cp -f $dir/assets/etc/systemd/system/cloud-config.service.d/firstboot-blocker.conf \
$chroot/etc/systemd/system/cloud-config.service.d/firstboot-blocker.conf

# this will append the following two relevant lines (plus a few commented out lines)
# to the default-conf:
Expand Down
3 changes: 3 additions & 0 deletions ...system_azure_init/assets/etc/systemd/system/cloud-config.service.d/firstboot-blocker.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[Unit]
Wants=firstboot.service
After=firstboot.service
6 changes: 2 additions & 4 deletions stemcell_builder/stages/system_azure_init/assets/etc/waagent/walinuxagent.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,15 @@
[Unit]
Description=Azure Linux Agent

After=network-online.target cloud-init.service
# NON-DEFAULT: Must run after the firstboot.service, which regenerates the ssh keys
After=firstboot.service network-online.target cloud-init.service
Wants=network-online.target sshd.service sshd-keygen.service

ConditionFileIsExecutable=/usr/sbin/waagent
ConditionPathExists=/etc/waagent.conf

[Service]
Type=simple
# stemcells on Azure re-generate the SSH Hostkey upon first reboot
# waagent has to wait until the file was recreated
ExecStartPre=/bin/bash -c "while [ ! -f /root/firstboot_done ]; do sleep 1; done"
ExecStart=/usr/bin/python3 -u /usr/sbin/waagent -daemon
Restart=always
Slice=azure.slice
Expand Down
Loading