cloudstack-usage fails after fresh 4.22.0.0 install on Ubuntu 24.04 because /etc/cloudstack/usage/key is missing

[Eddi1306]

I am running a fresh Ubuntu 24.04 LTS host and installed Apache CloudStack Management Server 4.22.0.0 following the official quick installation guide.

https://docs.cloudstack.apache.org/en/4.22.0.0/quickinstallationguide/qig.html

After installing the CloudStack packages, I noticed the following:

root@server:~# service cloudstack-usage status
● cloudstack-usage.service - CloudStack Usage Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Tue 2026-05-26 16:05:18 CEST; 5s ago
Docs: http://www.cloudstack.org/
Process: 2443 ExecStartPre=/usr/share/cloudstack-common/scripts/installer/pre-check.sh (code=exited, status=0/SUCCESS)
Process: 2469 ExecStart=/bin/sh -ec /usr/bin/java -Dpid=${JAVA_PID} $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS (code=exited, status=1/FAILURE)
Main PID: 2469 (code=exited, status=1/FAILURE)
CPU: 5.738s

Log file:

Caused by: com.cloud.utils.exception.CloudRuntimeException: key File containing secret key not found in the classpath:
at com.cloud.utils.crypt.EncryptionSecretKeyChecker.check(EncryptionSecretKeyChecker.java:80)
at com.cloud.utils.db.DbProperties.getDbProperties(DbProperties.java:80)
at com.cloud.utils.db.TransactionLegacy.(TransactionLegacy.java:1025)

The cloudstack-usage demon is not running properly due to missing decryption key for the database.

You can see here that the key has been created properly:

root@server:/etc/cloudstack/management# ll
total 88
drwxr-xr-x 2 root root 4096 May 26 16:04 ./
drwxr-xr-x 5 root root 4096 May 26 15:29 ../
-rw-r--r-- 1 root root 20976 Nov 7 2025 config.json
-rw-r----- 1 root cloud 4789 May 26 16:04 db.properties
-rw-r--r-- 1 root root 22806 Nov 7 2025 ehcache.xml
-rw-r--r-- 1 root root 981 Nov 7 2025 environment.properties
-rw-r--r-- 1 root root 931 Nov 7 2025 java.security.ciphers
-rw-r----- 1 root cloud 8 May 26 16:04 key
-rw-r--r-- 1 root root 5338 Nov 7 2025 log4j-cloud.xml
lrwxrwxrwx 1 root root 42 May 26 15:29 log4j2.xml -> /etc/cloudstack/management/log4j-cloud.xml
-rw-r--r-- 1 root root 2390 Nov 7 2025 server.properties

Here you can see that there is no symlink:

root@server:/etc/cloudstack/usage# ll
total 12
drwxr-xr-x 2 root root 4096 May 26 15:29 ./
drwxr-xr-x 5 root root 4096 May 26 15:29 ../
lrwxrwxrwx 1 root root 40 May 26 15:29 db.properties -> /etc/cloudstack/management/db.properties
-rw-r--r-- 1 root root 2396 Nov 7 2025 log4j-cloud.xml

I was able to fix this issue with creating a symlink for the key:

ln -s /etc/cloudstack/management/key /etc/cloudstack/usage/key

systemctl restart cloudstack-usage

root@server:/etc/cloudstack/usage# service cloudstack-usage status
● cloudstack-usage.service - CloudStack Usage Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; enabled; preset: enabled)
Active: active (running) since Tue 2026-05-26 16:09:25 CEST; 5s ago
Docs: http://www.cloudstack.org/
Process: 4860 ExecStartPre=/usr/share/cloudstack-common/scripts/installer/pre-check.sh (code=exited, status=0/SUCCESS)
Main PID: 4883 (sh)
Tasks: 29 (limit: 976)
Memory: 149.1M (peak: 155.1M)
CPU: 7.956s
CGroup: /system.slice/cloudstack-usage.service
├─4883 /bin/sh -ec "/usr/bin/java -Dpid=$$ $JAVA_OPTS $JAVA_DEBUG -cp $CLASSPATH $JAVA_CLASS"
└─4884 /usr/bin/java -Dpid=4883 -Xms256m -Xmx2048m --add-opens=java.base/java.lang=ALL-UNNAMED -cp "/usr/share/cloudstack-usage/:/usr/share/cloudstack-usage/lib/:/usr/share/c>

May 26 16:09:25 server.local systemd[1]: Starting cloudstack-usage.service - CloudStack Usage Server...
May 26 16:09:25 server.local systemd[1]: Started cloudstack-usage.service - CloudStack Usage Server.
lines 1-15/15 (END)

Is this a known issue/bug in cloudstack version 4.22.0.0 ? Did anyone encounter the same issue?

[kiranchavala]

Able to reproduce the issue

cc @weizhouapache @FabriceBrazier

[winterhazel]

@Eddi1306 @kiranchavala after investigating a bit, this seems like an installation order issue rather than an actual bug.

Could you confirm if you still encounter the problem when performing the installation in the following order?

1. Install the cloudstack-management package.
2. Prepare the database and run cloudstack-setup-databases to initialize it.
3. Install the cloudstack-usage package.

The Usage Server's symbolic link to the key is created only during its post-installation script if /etc/cloudstack/management/key exists, while /etc/cloudstack/management/key is created by cloudstack-setup-databases. Hence, installing the Usage Server before initializing the database results in the symbolic link not being automatically created.

If that is indeed what caused the issue, we can improve this behavior in the future by having cloudstack-setup-databases also create this symbolic link if the Usage Server is already installed on the same machine. However, for 4.23, I think this might just warrant a note in the official installation guide explaining this behavior.

[kiranchavala]

Thanks @winterhazel

Issue is not observed if the mentioned steps are followed, the key file present.

Issue occurs only if cloudstack-usage is installed before initializing the database

Install the cloudstack-management package.
Prepare the database and run cloudstack-setup-databases to initialize it.
Install the cloudstack-usage package.

root@test:/etc/cloudstack/usage# ls
db.properties  key  log4j-cloud.xml
root@test:/etc/cloudstack/usage# ls -la
total 12
drwxr-xr-x 2 root root 4096 Jul 21 05:08 .
drwxr-xr-x 5 root root 4096 Jul 21 05:08 ..
lrwxrwxrwx 1 root root   40 Jul 21 05:08 db.properties -> /etc/cloudstack/management/db.properties
lrwxrwxrwx 1 root root   30 Jul 21 05:08 key -> /etc/cloudstack/management/key

[Eddi1306]

@winterhazel Thank you very much for your answer! I followed your steps and can confirm that the key has been created successfully.

root@test-cm:/etc/cloudstack/usage# ll
total 12
drwxr-xr-x 2 root root 4096 Jul 28 10:22 ./
drwxr-xr-x 5 root root 4096 Jul 28 10:22 ../
lrwxrwxrwx 1 root root 40 Jul 28 10:22 db.properties -> /etc/cloudstack/management/db.properties
lrwxrwxrwx 1 root root 30 Jul 28 10:22 key -> /etc/cloudstack/management/key
-rw-r--r-- 1 root root 2396 Nov 7 2025 log4j-cloud.xml

[weizhouapache]

thanks @winterhazel @kiranchavala @Eddi1306