RootCAProvider error on management server with L3-network config

[mosys0815]

I am currently preparing a cluster and installed a new (the 1st) management server.
That server's network is a fully routed L3-network.
Starting the cloudstack-management service does not create the server certificate from root-ca with the following error:

2025-06-10 08:28:57,082 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RootCAProvider
2025-06-10 08:28:58,589 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [RootCAProvider] in [Ca Providers Registry]
2025-06-10 08:28:58,589 DEBUG [o.a.c.s.l.r.RegistryLifecycle] (main:[]) (logid:) Registered org.apache.cloudstack.ca.provider.RootCAProvider@611f82a8
2025-06-10 08:28:58,589 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Configuring CloudStack Components
2025-06-10 08:28:58,589 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider.
2025-06-10 08:28:59,037 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ].
2025-06-10 08:28:59,043 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [105870] for command [/bin/bash -c ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}' ].
2025-06-10 08:28:59,127 ERROR [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) Error on configuring bean RootCAProvider - Cannot invoke "java.net.NetworkInterface.getInterfaceAddresses()" because "nic" is null java.lang.NullPointerException: Cannot invoke "java.net.NetworkInterface.getInterfaceAddresses()" because "nic" is null

• these are the active network-interfaces, "hostip" is the bgp-announced ip on a virtual interface for networking, eth1* are the ethernet interfaces

~# ip -4 -br a | egrep '(eth1|hostip)'
eth1a            UP             10.72.44.198/30
eth1b            UP             10.72.45.198/30
hostip           UNKNOWN        10.72.44.3/32

• CS did retrieve the correct ip from hostip interface

~# ip route show default 0.0.0.0/0 | head -1 | awk '{print $5}'
10.72.44.3

• certificate check

~# openssl s_client -connect 10.72.44.3:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltNam
Could not read certificate from <stdin>
4087895B58700000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:../crypto/store/store_result.c:151:
Unable to load certificate

For testing i installed 2 virtual machines with cloudstack-management and a new mysql-database on one of these. Both instances are connected on a layer 2 network.
Here the management servers (one after another) came once with fully functional certificates, both servers see each other as peers in the cs-ui management server overview.

• certificates SAN on one of the test-instances:

~# openssl s_client -connect 10.65.254.48:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    IP Address:FE80:0:0:0:1C00:B1FF:FE00:164, IP Address:10.65.254.48, DNS:<redacted> DNS:cloudstack.internal

I then connected the former server with a fresh installation to the database of my test-setup.
I got the same error as mentioned above.

I suspect some issue with the L3-network and retrieving some networking-stuff by cloudstack-management-server.

Any idea how to proceed here?

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

what's the output of command ip route show default 0.0.0.0/0 ?

[mosys0815]

~# ip route show default 0.0.0.0/0
default proto bird src 10.72.44.3 metric 32
	nexthop via inet6 fe80::429e:a4ff:fe79:f50f dev eth1b weight 1
	nexthop via inet6 fe80::429e:a4ff:fe7b:8f0f dev eth1a weight 1
default via 10.72.44.197 dev eth1a proto static src 10.72.44.198 metric 1024 onlink
default via 10.72.45.197 dev eth1b proto static src 10.72.45.198 metric 1024 onlink

[weizhouapache]

~# ip route show default 0.0.0.0/0
default proto bird src 10.72.44.3 metric 32
	nexthop via inet6 fe80::429e:a4ff:fe79:f50f dev eth1b weight 1
	nexthop via inet6 fe80::429e:a4ff:fe7b:8f0f dev eth1a weight 1
default via 10.72.44.197 dev eth1a proto static src 10.72.44.198 metric 1024 onlink
default via 10.72.45.197 dev eth1b proto static src 10.72.45.198 metric 1024 onlink

maybe we should add a default via (or at least via) to make sure the command gets the default device, like

ip route show default 0.0.0.0/0 | grep ' via ' | head -1 | awk '{print $5}'

[mosys0815]

What exactly is the device used for?
In our setup we explicitly do not want to use a fixed interface, since these are supposed to "failover" each other.

[weizhouapache]

What exactly is the device used for? In our setup we explicitly do not want to use a fixed interface, since these are supposed to "failover" each other.

@mosys0815
the command gets the default device, and certificate is for all IPs of the default device

what IP would you like to use primarily ? 10.72.44.3 ?

btw: is the management server running , or stopped after Error on configuring bean RootCAProvider ?

[mosys0815]

AH, thx for clarification :)

Yes, i' like to expect the ip "10.72.44.3" added to the certificate, in fact with your current ip route command it already gets the correct ip.

The server is up and running, but https://10.72.44.3:9090/clusterservice is not available due to a missing certificate.
Therefore a lot of certificate error messages appear in the logs and the both test virtual management servers are not able to communicate with this one server.

[weizhouapache]

Yes, i' like to expect the ip "10.72.44.3" added to the certificate, in fact with your current ip route command it already gets the correct ip.

well, the command intends to get the default device name, but it returns the IP ... so it is unexpected

to support this special user case, some simple code changes are required I think

[mosys0815]

yes, I thought so, a one-liner using structured data could like this:

ip -j a | jq -r '.[] | .addr_info | map(select(.local == "'"$(ip -j r s default | jq -r '.[0] | .prefsrc')"'")) | .[].label'

output on my L2-network test virtual machine:

~# ip -j a | jq -r '.[] | .addr_info | map(select(.local == "'"$(ip -j r s default | jq -r '.[0] | .prefsrc')"'")) | .[].label'
ens3

output on L3-network server:

~# ip -j a | jq -r '.[] | .addr_info | map(select(.local == "'"$(ip -j r s default | jq -r '.[0] | .prefsrc')"'")) | .[].label'
hostip

Feel free to use and adapt :)

But for now, is there anything I can do to get the certificate, even manually?

[weizhouapache]

But for now, is there anything I can do to get the certificate, even manually?

I do not think so, unless the command returns results like

$ ip route show default 0.0.0.0/0
default via 10.72.44.3 dev hostip

[mosys0815]

ok, thx, I will have a look into the source and try to get it running :)

[mosys0815]

Sry for the delay, meanwhile other topics got higher priority.

I managed to build cloudstack from source (fork), even though setting up a build-server was quite a hassle, the documentation is outdated and not working for current distributions like ubuntu 24.04 (noble).

I made a slight change to the interface detection command.
It now works on our L3 network and on a L2-Network inside a virtual machine.
Unfortunately we do not have any hardware servers in common L2 networks anymore, so I am not able to test this with for example bond-setups.

[mosys0815]

here the output from management server start:

2025-07-14 07:32:55,999 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider.
2025-07-14 07:32:56,076 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Executing command [/bin/bash -c /bin/ip -j a | /bin/jq -r '.[] | .addr_info | map(select(.local == "'`/bin/ip -j r s default | /bin/jq -r '.[0] | .prefsrc'`'")) | .[].label' ].
2025-07-14 07:32:56,086 DEBUG [c.c.u.s.Script] (main:[]) (logid:) Successfully executed process [3384384] for command [/bin/bash -c /bin/ip -j a | /bin/jq -r '.[] | .addr_info | map(select(.local == "'`/bin/ip -j r s default | /bin/jq -r '.[0] | .prefsrc'`'")) | .[].label' ].
2025-07-14 07:32:56,087 DEBUG [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Trying to find management IP in CIDR range [10.65.252.0/22].
2025-07-14 07:32:56,521 INFO  [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Creating new management server certificate and keystore

and the certificate

openssl s_client -connect 10.72.44.3:9090 </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
X509v3 Subject Alternative Name:
    IP Address:2A13:B240:2010:1302:0:0:0:1, IP Address:FE80:0:0:0:4002:62FF:FEBB:8E8E, IP Address:10.72.44.3, DNS:<redacted>, DNS:cloudstack.internal

[DaanHoogland]

@mosys0815 does this mean your issue is fixed?

[mosys0815]

@DaanHoogland in my build it is fixed, any chance to get that into upstream?

[weizhouapache]

@mosys0815
you can create a pull request

[DaanHoogland]

can you refer your PR here and close the discussion @mosys0815 ?

[mosys0815]

PR: #11800