Don't save migrated startup-config, fail-secure hardening

board/common/rootfs/etc/finit.d/10-infix.conf

@@ -1,3 +1,7 @@
 task name:ixinit [S] <pid/syslogd> \
 	/usr/libexec/finit/runparts -bp /usr/libexec/infix/init.d \
 	-- Probing system
+
+task name:slot-check [2345] <service/rauc/running> \
+	/usr/libexec/infix/slot-check \
+	-- Checking software partitions

board/common/rootfs/etc/watchdogd.conf

@@ -62,6 +62,11 @@ device /dev/watchdog {
 #    priority = 98
 #    script = "/path/to/supervisor-script.sh"
 #}
+supervisor {
+    enabled  = true
+    priority = 98
+    script   = "/usr/libexec/infix/supervisor"
+}
 
 ### Reset reason #######################################################
 # The following section controls if/how the reset reason & reset counter

board/common/rootfs/usr/libexec/infix/slot-check

@@ -0,0 +1,14 @@
+#!/bin/sh
+# Remind the user to upgrade the other partition, issue #1637
+
+rauc status --detailed --output-format=json 2>/dev/null | jq -r '
+    [.slots | add | .[] | select(.class == "rootfs")] as $slots
+    | ($slots[] | select(.state == "booted") | .slot_status.bundle.version) as $booted
+    | $slots[] | select(.state != "booted")
+    | select(.slot_status.bundle.version != $booted)
+    | "\(.bootname) \(.slot_status.bundle.version // "unknown") \($booted)"' |
+while read -r name version booted; do
+    msg="NOTE: the $name partition has $version, this is $booted.  Use 'upgrade' to update it."
+    logger -t slot-check -p user.notice "$msg"
+    printf "\n%s\n" "$msg" | tee -a /etc/banner /etc/issue /etc/issue.net >/dev/null
+done

board/common/rootfs/usr/libexec/infix/supervisor

@@ -0,0 +1,16 @@
+#!/bin/sh
+# Called by watchdogd when a supervised process misses its deadline:
+#
+#    $0 supervisor CODE PID LABEL
+#
+# A hung confd bootstrap goes to fail-secure on next boot, issue #1637.
+# Exit non-zero so watchdogd saves the reset reason and resets.  A lost
+# kick reply is not a hang, exit 0 and watchdogd drops the supervision.
+
+if [ "$4" = "confd-bootstrap" ]; then
+    [ "$2" = 5 ] || exit 0	# WDOG_FAILED_TO_MEET_DEADLINE
+    touch /mnt/aux/startup-config.failed
+    sync
+fi
+
+exit 1

doc/ChangeLog.md

@@ -121,6 +121,19 @@ All notable changes to the project are documented in this file.
   which are written into the DHCP client service file
 - Restrict the allowed characters in DHCP server static-host match values
 - Restrict the allowed characters in a hardware component `name`
+- Fix #1637: a startup-config migrated on upgrade was saved to disk at
+  boot, so the image on the other partition could no longer read it.
+  The migrated configuration is now only applied to running-config,
+  use `copy running-config startup-config` to save it.  A failed
+  migration reverts to failure-config.  After a downgrade, a newer
+  startup-config is loaded as-is, unless it uses settings the older
+  version does not know.  A note at login, in `show software`, and on
+  the WebUI software page shows when the other partition has a
+  different version
+- A startup-config that fails to load now resets the unit, and the next
+  boot goes straight to failure-config from a clean state, issue #1637.
+  The same applies if loading startup-config hangs, the system watchdog
+  then resets the unit
 - Fix #1619: Raspberry Pi kernel panic when configure Wi-Fi
 - WebUI: "Save" in the interface editor and "OK" in Add Interface
   did nothing for Wi-Fi and WireGuard interfaces. The inline "+ New"

doc/boot.md

@@ -182,9 +182,18 @@ in the second case will cause the device to fail on the next boot.
 #### Broken startup-config
 
 If loading `startup-config` fails for some reason, e.g., invalid JSON
-syntax, failed validation against the system's YANG model, or a bug in
-the system's `confd` service, the *Fail Secure Mode* is triggered and
-`failure-config` is loaded (unless VPD Failure, see above).
+syntax, a failed migration or validation against the system's YANG
+model, or a bug in the system's `confd` service, the *Fail Secure Mode*
+is triggered and `failure-config` is loaded (unless VPD Failure, see
+above).  The system then marks the boot as failed and resets, so that
+the next boot starts from a clean state and goes straight to *Fail
+Secure Mode*.  The mark is cleared on that boot, so a reboot after it
+tries `startup-config` again.
+
+The same happens if loading `startup-config` hangs, e.g., a service that
+never responds.  The system watchdog then resets the unit, by default
+after 70 seconds, and the next boot goes straight to *Fail Secure
+Mode*.
 
 > [!TIP]
 > Please see the [Branding & Releases](branding.md) document for how to

doc/upgrade.md

@@ -109,9 +109,13 @@ booted from one partition, an `upgrade` will apply to the other
 
 > [!CAUTION]
 > During boot (step 5), the unit may [migrate](#configuration-migration)
-> the startup configuration for any syntax changes.  It is therefore
-> important that you make sure to upgrade the other partition as well
-> after reboot, of course after having verified your setup.
+> the startup configuration for any syntax changes.  The migrated
+> configuration is only applied to `running-config`, and the file on
+> disk is kept as-is until you save it.  Once saved, the old image on
+> the other partition may not be able to read it, so upgrade the other
+> partition as well after you have verified your setup.  Until then, a
+> note at login, in `show software`, and on the WebUI software page
+> shows that the other partition has a different version.
 
 The CLI example below shows steps 2-5.
 
@@ -414,14 +418,14 @@ The example above illustrated an upgrade from Infix v25.01.0 to
 v25.03.1. Inbetween these versions, YANG configuration definitions
 changed slightly (more details given below).
 
-During boot, Infix inspects the `version` meta information within the
-startup configuration file to determine if configuration migration is
-needed. In this specific case, the configuration file has version
+During boot, the system inspects the `version` meta information within
+the startup configuration file to determine if configuration migration
+is needed. In this specific case, the configuration file has version
 `1.4` while the booted software expects version `1.5` (the
 configuration version numbering differs from the Infix image version
 numbering).  The startup configuration is migrated to `1.5`
-definitions and stored, while a backup previous startup configuration
-is stored in directory `/cfg/backup/`.
+definitions and applied to `running-config`, while a backup of the
+original startup configuration is stored in directory `/cfg/backup/`.
 
 <pre class="cli"><code>admin@example:/> <b>dir /cfg/backup/</b>
 /cfg/backup/ directory
@@ -430,8 +434,26 @@ startup-config-1.4.cfg
 admin@example:/>
 </code></pre>
 
-The modifications made to the startup configuration can be viewed by
-comparing the files from the *shell*. An example is shown below.
+The file `/cfg/startup-config.cfg` itself is *not* changed.  If the new
+image fails, the unit can fall back to the old image on the other
+partition, and its startup configuration is intact.  The migration is
+repeated at every boot until the configuration is saved.  Until then, a
+note at login says the configuration is not saved, and the WebUI shows
+unsaved changes, since the `startup-config` datastore reports the old
+version.
+
+When you have verified the unit works as expected, save the migrated
+configuration:
+
+<pre class="cli"><code>admin@example:/> <b>copy running-config startup-config</b>
+admin@example:/>
+</code></pre>
+
+If the migration fails, the unit reverts to its [failure config][3].
+
+After saving, the modifications made to the startup configuration can
+be viewed by comparing the files from the *shell*. An example is shown
+below.
 
 <pre class="cli"><code>admin@example:/> <b>exit</b>
 admin@example:~$ <b>diff /cfg/backup/startup-config-1.4.cfg /cfg/startup-config.cfg</b>
@@ -457,6 +479,10 @@ Downgrading to an earlier version is possible, however, downgrading is
 up with the downgraded version, it may fail to apply the *startup
 config*, and instead apply its [failure config][3].
 
+A startup configuration of a newer version than the downgraded software
+supports is loaded as-is.  It only fails if it uses settings the older
+version does not know.
+
 We consider two cases: downgrading with and without applying a backup
 startup configuration before rebooting.
 

package/confd/Config.in

@@ -6,6 +6,7 @@ config BR2_PACKAGE_CONFD
 	select BR2_PACKAGE_SYSREPO
 	select BR2_PACKAGE_LIBSRX
 	select BR2_PACKAGE_SUPPORT
+	select BR2_PACKAGE_WATCHDOGD
 	help
 	  A plugin to sysrepo that provides the core YANG models used to
 	  manage an Infix based system.  Configuration can be done using

package/confd/confd.mk

@@ -10,7 +10,7 @@ CONFD_SITE = $(BR2_EXTERNAL_INFIX_PATH)/src/confd
 CONFD_LICENSE = BSD-3-Clause
 CONFD_LICENSE_FILES = LICENSE
 CONFD_REDISTRIBUTE = NO
-CONFD_DEPENDENCIES = host-sysrepo sysrepo rousette netopeer2 jansson libite sysrepo libsrx libglib2 libev sysklogd
+CONFD_DEPENDENCIES = host-sysrepo sysrepo rousette netopeer2 jansson libite sysrepo libsrx libglib2 libev sysklogd watchdogd
 CONFD_AUTORECONF = YES
 CONFD_CONF_OPTS += --disable-silent-rules --with-crypt=$(BR2_PACKAGE_CONFD_DEFAULT_CRYPT)
 CONFD_SYSREPO_SHM_PREFIX = sr_buildroot$(subst /,_,$(CONFIG_DIR))_confd

src/confd/configure.ac

@@ -127,6 +127,7 @@ PKG_CHECK_MODULES([libyang], [libyang >= 4.2.2])
 PKG_CHECK_MODULES([libsrx],  [libsrx >= 1.0.0])
 PKG_CHECK_MODULES([libsyslog], [libsyslog >= 2.7.0])
 PKG_CHECK_MODULES([libcrypto], [libcrypto])
+PKG_CHECK_MODULES([libwdog], [libwdog >= 4.0])
 
 AC_CHECK_HEADER([ev.h],
     [saved_LIBS="$LIBS"

src/confd/src/Makefile.am

@@ -6,8 +6,8 @@ plugin_LTLIBRARIES      = confd-plugin.la
 
 sbin_PROGRAMS           = confd
 
-confd_CFLAGS            = $(sysrepo_CFLAGS) $(libyang_CFLAGS) $(jansson_CFLAGS) $(libite_CFLAGS) $(libsrx_CFLAGS)
-confd_LDADD             = $(sysrepo_LIBS)   $(libyang_LIBS)   $(jansson_LIBS)   $(libite_LIBS)   $(libsrx_LIBS) $(EV_LIBS) -ldl
+confd_CFLAGS            = $(sysrepo_CFLAGS) $(libyang_CFLAGS) $(jansson_CFLAGS) $(libite_CFLAGS) $(libsrx_CFLAGS) $(libwdog_CFLAGS)
+confd_LDADD             = $(sysrepo_LIBS)   $(libyang_LIBS)   $(jansson_LIBS)   $(libite_LIBS)   $(libsrx_LIBS) $(libwdog_LIBS) $(EV_LIBS) -ldl
 confd_SOURCES           = main.c
 
 confd_plugin_la_LDFLAGS = -module -avoid-version -shared

src/confd/src/core.c

@@ -191,6 +191,19 @@ int finit_disablef(const char *fmt, ...)
 }
 
 
+/*
+ * After a migration the startup datastore has the old version of the file
+ * until saved, so stamp every write after bootstrap, issue #1637.
+ */
+static int startup_version(sr_session_ctx_t *session, uint32_t sub_id, const char *model,
+			   const char *xpath, sr_event_t event, unsigned request_id, void *priv)
+{
+	if (event != SR_EV_UPDATE || systemf("runlevel >/dev/null 2>&1"))
+		return SR_ERR_OK;
+
+	return meta_set_version(session);
+}
+
 static int startup_save(sr_session_ctx_t *session, uint32_t sub_id, const char *model,
 			const char *xpath, sr_event_t event, unsigned request_id, void *priv)
 {
@@ -921,6 +934,12 @@ int sr_plugin_init_cb(sr_session_ctx_t *session, void **priv)
 		ERROR("Failed to subscribe to infix-meta");
 		goto err;
 	}
+	rc = sr_module_change_subscribe(confd.startup, "infix-meta", "//.", startup_version, NULL,
+					CB_PRIO_PRIMARY, SR_SUBSCR_UPDATE | SR_SUBSCR_NO_THREAD, &confd.sub);
+	if (rc) {
+		ERROR("Failed to subscribe to infix-meta in startup");
+		goto err;
+	}
 	rc = subscribe_model("ieee1588-ptp-tt", &confd, 0);
 	if (rc) {
 		ERROR("Failed to subscribe to ieee1588-ptp-tt");

src/confd/src/core.h

@@ -271,6 +271,7 @@ int factory_rpc_init(struct confd *confd);
 int factory_default_rpc_init(struct confd *confd);
 
 /* meta.c */
+int meta_set_version(sr_session_ctx_t *session);
 int meta_change_cb(sr_session_ctx_t *session, struct lyd_node *config, struct lyd_node *diff, sr_event_t event, struct confd *confd);
 
 /* system-software.c */