Cloudstack 4.22 SSVM fails to start

[dpthakar]

Hello,

Problem:-
SSVM Fails to start.

Versions:-
Cloudstack agent 4.22
Cloudstack Management 4.22
ubuntu 24.04

Error:-

==> /var/log/cloudstack/agent/agent.log <==
2025-11-11 16:07:12,873 INFO  [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:6786196e) Trying to fetch storage pool 3b4573e2-11b4-3b02-81b3-a7d8f8c97101 from libvirt
2025-11-11 16:07:12,942 WARN  [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-1:[]) (logid:6786196e) Execution of process [41853] for command [/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 18 -p bond1 -b brbond1-18 -o add ] failed.
2025-11-11 16:07:12,943 WARN  [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-1:[]) (logid:6786196e) Process [41853] for command [/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 18 -p bond1 -b brbond1-18 -o add ] encountered the error: [Error: 8021q: VLAN device already exists./usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh: line 37: /proc/sys/net/ipv6/conf/bond1.18/disable_ipv6: No such file or directoryCannot find device "bond1.18"Failed to create vlan 18 on pif: bond1.].
2025-11-11 16:07:12,943 WARN  [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-1:[]) (logid:6786196e) InternalErrorException com.cloud.exception.InternalErrorException: Failed to create vnet 18: Error: 8021q: VLAN device already exists./usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh: line 37: /proc/sys/net/ipv6/conf/bond1.18/disable_ipv6: No such file or directoryCannot find device "bond1.18"Failed to create vlan 18 on pif: bond1.
	at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.createVnet(BridgeVifDriver.java:338)
	at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.createVnetBr(BridgeVifDriver.java:320)
	at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.createStorageVnetBridgeIfNeeded(BridgeVifDriver.java:203)
	at com.cloud.hypervisor.kvm.resource.BridgeVifDriver.plug(BridgeVifDriver.java:272)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVif(LibvirtComputingResource.java:3861)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVifs(LibvirtComputingResource.java:3438)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:86)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:52)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:2280)
	at com.cloud.agent.Agent.processRequest(Agent.java:813)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1295)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Earlier SSVM was working fine on CS Agent 4.20, after upgrading CS Agent to 4.22 AND CS Mgmt to 4,22 we are getting above mentioned error.

Below is current network config.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: ens3f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:8d:f4 brd ff:ff:ff:ff:ff:ff
    altname enp99s0f0np0
3: ens3f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:8d:f5 brd ff:ff:ff:ff:ff:ff
    altname enp99s0f1np1
4: ens2f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9214 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff permaddr 7c:c2:55:42:8b:54
    altname enp68s0f0np0
5: ens2f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9214 qdisc mq state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:8b:55 brd ff:ff:ff:ff:ff:ff
    altname enp68s0f1np1
6: enp3s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:8e:34 brd ff:ff:ff:ff:ff:ff
7: enp3s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:8e:35 brd ff:ff:ff:ff:ff:ff
8: enp36s0f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9214 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff permaddr 7c:c2:55:42:85:44
9: enp36s0f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9214 qdisc mq master bond1 state UP group default qlen 1000
    link/ether ca:9f:8f:ec:8e:1f brd ff:ff:ff:ff:ff:ff permaddr 7c:c2:55:42:85:45
10: enp227s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:85:24 brd ff:ff:ff:ff:ff:ff
11: enp51s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 7c:c2:55:7b:58:d6 brd ff:ff:ff:ff:ff:ff
12: enp227s0f1np1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9214 qdisc mq master bond1 state UP group default qlen 1000
    link/ether ca:9f:8f:ec:8e:1f brd ff:ff:ff:ff:ff:ff permaddr 7c:c2:55:42:85:25
13: enp196s0f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9214 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff permaddr 7c:c2:55:42:84:84
14: enp196s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9214 qdisc mq state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:84:85 brd ff:ff:ff:ff:ff:ff
15: enp131s0f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:85:34 brd ff:ff:ff:ff:ff:ff
16: enp131s0f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:85:35 brd ff:ff:ff:ff:ff:ff
17: enp164s0f0np0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9214 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff permaddr 7c:c2:55:42:85:54
18: enp164s0f1np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9214 qdisc mq state DOWN group default qlen 1000
    link/ether 7c:c2:55:42:85:55 brd ff:ff:ff:ff:ff:ff
19: enp51s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master hostbr0 state UP group default qlen 1000
    link/ether 7c:c2:55:7b:58:d7 brd ff:ff:ff:ff:ff:ff
20: enxbe3af2b6059f: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether be:3a:f2:b6:05:9f brd ff:ff:ff:ff:ff:ff
21: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9214 qdisc noqueue master cloudbr0 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff
22: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP group default qlen 1000
    link/ether ca:9f:8f:ec:8e:1f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c89f:8fff:feec:8e1f/64 scope link 
       valid_lft forever preferred_lft forever
23: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP group default qlen 1000
    link/ether 76:55:ab:fd:45:64 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7455:abff:fefd:4564/64 scope link 
       valid_lft forever preferred_lft forever
24: csmgmtbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP group default qlen 1000
    link/ether da:30:cd:f0:20:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.131/23 metric 100 brd 192.168.11.255 scope global dynamic csmgmtbr0
       valid_lft 7773596sec preferred_lft 7773596sec
    inet6 fe80::d830:cdff:fef0:200a/64 scope link 
       valid_lft forever preferred_lft forever
25: hostbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:e0:4f:7c:cb:40 brd ff:ff:ff:ff:ff:ff
    inet 192.168.14.116/23 metric 100 brd 192.168.15.255 scope global dynamic hostbr0
       valid_lft 7773572sec preferred_lft 7773572sec
    inet6 fe80::fce0:4fff:fe7c:cb40/64 scope link 
       valid_lft forever preferred_lft forever
26: strbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP group default qlen 1000
    link/ether 7e:44:09:e3:06:76 brd ff:ff:ff:ff:ff:ff
    inet 192.168.18.105/24 brd 192.168.18.255 scope global strbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::7c44:9ff:fee3:676/64 scope link 
       valid_lft forever preferred_lft forever
27: vlan18@bond1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue master strbr0 state UP group default qlen 1000
    link/ether ca:9f:8f:ec:8e:1f brd ff:ff:ff:ff:ff:ff
28: vlan10@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue master csmgmtbr0 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff
29: cloud0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4a:3b:6e:d2:39:6d brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.1/16 scope global cloud0
       valid_lft forever preferred_lft forever
    inet6 fe80::483b:6eff:fed2:396d/64 scope link 
       valid_lft forever preferred_lft forever
42: bond0.4090@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue master brbond0-4090 state UP group default qlen 1000
    link/ether 0a:b7:ba:ab:5b:13 brd ff:ff:ff:ff:ff:ff
43: brbond0-4090: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue state UP group default qlen 1000
    link/ether c2:88:40:83:d0:27 brd ff:ff:ff:ff:ff:ff
44: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cloud0 state UNKNOWN group default qlen 1000
    link/ether fe:00:a9:fe:3c:68 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc00:a9ff:fefe:3c68/64 scope link 
       valid_lft forever preferred_lft forever
45: vnet7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue master csmgmtbr0 state UNKNOWN group default qlen 1000
    link/ether fe:00:48:00:00:0b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc00:48ff:fe00:b/64 scope link 
       valid_lft forever preferred_lft forever
46: vnet8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc noqueue master brbond0-4090 state UNKNOWN group default qlen 1000
    link/ether fe:00:9f:00:00:0c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc00:9fff:fe00:c/64 scope link 
       valid_lft forever preferred_lft forever

To fix it, we tried following but did not help.
Reboot all hosts & mgmgt
Downgrade CS Agent to 4.20
Erase CS Agent package & re-add host

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

@dpthakar
the CPVM is working, right ?

can you check database table dc_storage_network_ip_range if vlan is 18 ?
I think it should be NULL in your case.

[dpthakar]

@dpthakar the CPVM is working, right ?

can you check database table dc_storage_network_ip_range if vlan is 18 ? I think it should be NULL in your case.

Yes, CPVM works fine.
Below is table output.

mysql> select * from dc_storage_network_ip_range;
+----+--------------------------------------+---------------+---------------+--------------+------+---------------+----------------+--------+------------+
| id | uuid                                 | start_ip      | end_ip        | gateway      | vlan | netmask       | data_center_id | pod_id | network_id |
+----+--------------------------------------+---------------+---------------+--------------+------+---------------+----------------+--------+------------+
|  1 | 92d3991a-19ad-4c3a-af5b-e2593d637537 | 192.168.18.81 | 192.168.18.85 | 192.168.18.1 |   18 | 255.255.255.0 |              1 |      1 |        203 |
+----+--------------------------------------+---------------+---------------+--------------+------+---------------+----------------+--------+------------+
1 row in set (0.00 sec)

[dpthakar]

@weizhouapache

thanks for your help, after removing vlan 18 from zone config, it worked.
reason being we are already using vlan18 on hosts for rbd mount.