Network isolation creation error

[Hossein4185]

problem

I get an error when I try to create an isolated network.

2025-09-23 10:03:48,650 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Allocating the VR with id=5337 in datacenter Zone {"id": "3", "name": "PVCL.TEH.IR", "uuid": "d04f1ed7-09e9-4beb-853b-b4f4adbce9bd"} with the hypervisor type VMware
2025-09-23 10:03:48,679 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) VMware won't support system vm, skip it
2025-09-23 10:03:48,690 DEBUG [o.a.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Lock is released for network [id: 210] (Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115}) as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(3)-Pod(null)-Cluster(null)-Host(null)-Storage()]
2025-09-23 10:03:48,698 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Failed to implement network Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115} elements and resources as a part of network restart due to com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:3] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:250)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1753)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1678)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3719)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2955)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2977)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy200.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:97)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-09-23 10:03:48,699 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Network Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115} failed to restart.
2025-09-23 10:03:48,733 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Complete async job-52623, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to restart network"}
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Publish async job-52623 complete on message bus
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Wake up jobs related to job-52623
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Update db status for job-52623
2025-09-23 10:03:48,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Wake up jobs joined with job-52623 and disjoin all subjobs created from job- 52623
2025-09-23 10:03:48,751 DEBUG [c.c.a.ApiServer] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Retrieved cmdEventType from job info: NETWORK.RESTART
2025-09-23 10:03:48,775 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-52623
2025-09-23 10:03:48,775 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Remove job-52623 from job monitoring
2025-09-23 10:03:49,217 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 0 routers to update status.
2025-09-23 10:03:49,222 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 1 VPC's to update Redundant State.
2025-09-23 10:03:49,224 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 0 networks to update RvR status.
2025-09-23 10:03:49,459 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-deea6087]) (logid:40ac84bb) DbCollector is running...
2025-09-23 10:03:49,580 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-82351737]) (logid:070d357e) HA health check task is running...
2025-09-23 10:03:49,802 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e48fac]) (logid:c375191b) No inactive management server node found
2025-09-23 10:03:49,802 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e48fac]) (logid:c375191b) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms

versions

Cloudstack Management 4.21
Vmware 8.0.3

[kiranchavala]

@Hossein4185 Looks like cloudstack was not able to reach the virtual router on vmware

Could check if the virtual router is running on vcenter and is reachable from management server

[Hossein4185]

@kiranchavala
Running on host but unreachable on cloudstack.

2025-09-29 17:43:27,305 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-0415c95b, ctx-61551cf1]) (logid:de53d393) ===END=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:27,479 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca]) (logid:3a198344) ===START=== 172.17.0.247 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:27,480 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca]) (logid:3a198344) Two factor authentication is already verified for the user 12, so skipping
2025-09-29 17:43:27,497 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) CIDRs from which account 'Account [{"accountName":"ho.karami","id":9,"uuid":"aa8fee67-15f6-41c8-a5e6-4cf64bd0878e"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:27,503 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) Account for user id 13a153b4-c31b-4a3a-833f-d895c29928c3 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:27,504 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:27,504 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:27,516 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) ===END=== 172.17.0.247 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:27,775 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:27,775 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:27,775 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-09-29 17:43:28,158 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d2786022]) (logid:e4d9f092) HA health check task is running...
2025-09-29 17:43:29,276 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:29,276 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:29,276 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms
2025-09-29 17:43:29,420 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648]) (logid:e83af821) ===START=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,420 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648]) (logid:e83af821) Two factor authentication is already verified for the user 9, so skipping
2025-09-29 17:43:29,435 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) CIDRs from which account 'Account [{"accountName":"a.bahri","id":15,"uuid":"95be6584-640d-434d-8a1f-7df726a918ae"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:29,443 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) Account for user id e33a25d0-28c7-48df-b19f-131956f501a4 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:29,444 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:29,444 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:29,458 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) ===END=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,961 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73]) (logid:e197f43d) ===START=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,961 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73]) (logid:e197f43d) Two factor authentication is already verified for the user 9, so skipping
2025-09-29 17:43:29,977 DEBUG [c.c.a.ApiServer] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) CIDRs from which account 'Account [{"accountName":"a.bahri","id":15,"uuid":"95be6584-640d-434d-8a1f-7df726a918ae"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:29,985 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) Account for user id e33a25d0-28c7-48df-b19f-131956f501a4 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:29,985 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:29,985 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:29,997 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) ===END=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:30,376 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c]) (logid:76b7253c) ===START=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:30,376 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c]) (logid:76b7253c) Two factor authentication is already verified for the user 12, so skipping
2025-09-29 17:43:30,381 DEBUG [c.c.a.m.D.Task] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Response Received:
2025-09-29 17:43:30,381 DEBUG [c.c.a.t.Request] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"Stop VM r-5387-VM Succeed","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,382 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250305: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }
2025-09-29 17:43:30,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Details from executing class com.cloud.agent.api.StopCommand: Stop VM r-5387-VM Succeed
2025-09-29 17:43:30,382 ERROR [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Failed to start instance VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} com.cloud.utils.exception.ExecutionException: Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1601)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

2025-09-29 17:43:30,392 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) CIDRs from which account 'Account [{"accountName":"ho.karami","id":9,"uuid":"aa8fee67-15f6-41c8-a5e6-4cf64bd0878e"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:30,394 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Cleaning up resources for the vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,401 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) Account for user id 13a153b4-c31b-4a3a-833f-d895c29928c3 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:30,401 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:30,401 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:30,433 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) ===END=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:30,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Wait time setting on com.cloud.agent.api.StopCommand is 1800 seconds
2025-09-29 17:43:30,445 DEBUG [c.c.a.m.ClusteredDirectAgentAttache] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Routed from 345050399512
2025-09-29 17:43:30,445 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Sending { Cmd , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"172.17.26.167","forceStop":"false","vlanToPersistenceMap":{"1000":"true"},"expungeVM":"false","volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,445 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Executing: { Cmd , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"172.17.26.167","forceStop":"false","vlanToPersistenceMap":{"1000":"true"},"expungeVM":"false","volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,446 DEBUG [c.c.a.m.D.Task] (DirectAgent-59:[ctx-b1ed20e4]) (logid:6f34189c) Seq 16-4644900065679250306: Executing request
2025-09-29 17:43:30,446 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Executing resource command StopCommand: [{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"172.17.26.167","forceStop":false,"vlanToPersistenceMap":{"1000":true},"expungeVM":false,"volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":true,"wait":0,"bypassHostMaintenance":false}].
2025-09-29 17:43:30,467 DEBUG [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) find VM r-5387-VM on host
2025-09-29 17:43:30,467 INFO [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) VM r-5387-VM not found in host cache
2025-09-29 17:43:30,467 DEBUG [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) load VM cache on host
2025-09-29 17:43:30,565 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) VM r-5387-VM is already in stopped state
2025-09-29 17:43:30,566 DEBUG [c.c.a.m.D.Task] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Seq 16-4644900065679250306: Response Received:
2025-09-29 17:43:30,566 DEBUG [c.c.a.t.Request] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Seq 16-4644900065679250306: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"VM r-5387-VM is already in stopped state","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,566 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }
2025-09-29 17:43:30,704 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Released nic: NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} for vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"}
2025-09-29 17:43:30,704 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Released nic: NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} for vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"}
2025-09-29 17:43:30,704 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) The nic Nic {"broadcastUri":null,"deviceId":1,"iPv4Address":"172.17.26.101","id":55319,"instanceId":5387,"reservationId":"1772dbf2-0235-4112-99ac-e84ae568fb8d","uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b"} on NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} was released according to VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} by guru com.cloud.network.guru.ControlNetworkGuru@1909b900, now updating record.
2025-09-29 17:43:30,728 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully released network resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,749 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully released storage resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,750 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully cleaned up resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:30,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:30,771 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-09-29 17:43:30,787 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Stopped","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Hosts's actual total CPU: 301728 and CPU after applying overprovisioning: 301728
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Hosts's actual total RAM: (1023.66 GB) 1099147345920 and RAM after applying overprovisioning: (1023.66 GB) 1099147378688
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) release cpu from host: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}, old used: 6500, reserved: 0, actual total: 301728, total with overprovisioning: 301728; new used: 6000,reserved:0; movedfromreserved: false,moveToReservered: false
2025-09-29 17:43:30,810 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) release mem from host: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}, old used: (4.50 GB) 4831838208, reserved: (0 bytes) 0, total: (1023.66 GB) 1099147378688; new used: (4.00 GB) 4294967296, reserved: (0 bytes) 0; movedfromreserved: false, moveToReservered: false
2025-09-29 17:43:30,820 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Invocation exception, caused by: com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
2025-09-29 17:43:30,820 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Rethrow exception com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
2025-09-29 17:43:30,820 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 5387, job origin: 53846
2025-09-29 17:43:30,820 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Unable to complete AsyncJob {"accountId":9,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAJAAAAAAAAAAwAAAAAAAAVC3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZkbUZzZFdWNGNBRXhw","cmdVersion":0,"completeMsid":null,"created":"Mon Sep 29 17:40:32 IRST 2025","id":53849,"initMsid":345050399512,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":12,"uuid":"cd1036aa-8b98-4906-b35f-f6b3503f4ace"}, job origin: 53846 com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1637)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1601)
... 20 more

2025-09-29 17:43:30,827 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Complete async job-53849, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC1jb20uY2xvdWQuZXhjZXB0aW9uLkFnZW50VW5hdmFpbGFibGVFeGNlcHRpb25WTU9wAAAAFwIAAHhyADBjb20uY2xvdWQuZXhjZXB0aW9uLlJlc291cmNlVW5hdmFpbGFibGVFeGNlcHRpb25WTU9wAAAAHwIAAkoAA19pZEwABl9zY29wZXQAEUxqYXZhL2xhbmcvQ2xhc3M7eHIAImNvbS5jbG91ZC5leGNlcHRpb24uQ2xvdWRFeGNlcHRpb2556I4b8eRExwIAAkwAC2NzRXJyb3JDb2RldAATTGphdmEvbGFuZy9JbnRlZ2VyO0wABmlkTGlzdHQAFUxqYXZhL3V0aWwvQXJyYXlMaXN0O3hyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwc3IALGNvbS5jbG91ZC51dGlscy5leGNlcHRpb24uRXhlY3V0aW9uRXhjZXB0aW9uVk1PcAAAAA4CAAFJAAtjc0Vycm9yQ29kZXhxAH4ABnEAfgAOdABkVW5hYmxlIHRvIHN0YXJ0ICBWTTpjNWZjNTRlZC04MWY5LTQ3NDMtOGJkYi00OGRjNjM3OWY5ZjggZHVlIHRvIGVycm9yIGluIGZpbmFsaXplU3RhcnQsIG5vdCByZXRyeWluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAFXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ACUwADmRlY2xhcmluZ0NsYXNzcQB-AAlMAAhmaWxlTmFtZXEAfgAJTAAKbWV0aG9kTmFtZXEAfgAJTAAKbW9kdWxlTmFtZXEAfgAJTAANbW9kdWxlVmVyc2lvbnEAfgAJeHABAAAGQXQAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgASAQAAFsNxAH4AFHEAfgAVcQB-ABZxAH4AF3Bwc3EAfgASAv____5wdAAtamRrLmludGVybmFsLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2UwdAAJamF2YS5iYXNldAAHMTcuMC4xNnNxAH4AEgIAAABNcHEAfgAacQB-ABt0AAZpbnZva2VxAH4AHXEAfgAec3EAfgASAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAgcQB-AB1xAH4AHnNxAH4AEgIAAAI5cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACBxAH4AHXEAfgAec3EAfgASAQAAAGZxAH4AFHQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgASAQAAFz9xAH4AFHEAfgAVcQB-ABZxAH4AKnBwc3EAfgASAQAAAGNxAH4AFHQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-ABIBAAACsXEAfgAUdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-ABIBAAAAMXEAfgAUdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4AEgEAAAA4cQB-ABR0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4AEgEAAABncQB-ABR0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADp0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4AEgEAAAA1cQB-ABRxAH4APXEAfgA6dAAOcnVuV2l0aENvbnRleHRwcHNxAH4AEgEAAAAucQB-ABR0ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ANnEAfgA3cHBzcQB-ABIBAAACfXEAfgAUcQB-ADFxAH4AMnEAfgA3cHBzcQB-ABICAAACG3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AO3EAfgAdcQB-AB5zcQB-ABICAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ADdxAH4AHXEAfgAec3EAfgASAgAABHBwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AHXEAfgAec3EAfgASAgAAAntwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBMcQB-ADdxAH4AHXEAfgAec3EAfgASAgAAA0hwdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ADdxAH4AHXEAfgAec3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAAAB0ALBSZXNvdXJjZSBbSG9zdDoxNl0gaXMgdW5yZWFjaGFibGU6IEhvc3QgMTY6IFVuYWJsZSB0byBzdGFydCBpbnN0YW5jZSBkdWUgdG8gVW5hYmxlIHRvIHN0YXJ0ICBWTTpjNWZjNTRlZC04MWY5LTQ3NDMtOGJkYi00OGRjNjM3OWY5ZjggZHVlIHRvIGVycm9yIGluIGZpbmFsaXplU3RhcnQsIG5vdCByZXRyeWluZ3VxAH4AEAAAABVzcQB-ABIBAAAGZXEAfgAUcQB-ABVxAH4AFnEAfgAXcHBzcQB-ABIBAAAWw3EAfgAUcQB-ABVxAH4AFnEAfgAXcHBzcQB-ABIC_____nBxAH4AGnEAfgAbcQB-ABxxAH4AHXEAfgAec3EAfgASAgAAAE1wcQB-ABpxAH4AG3EAfgAgcQB-AB1xAH4AHnNxAH4AEgIAAAArcHEAfgAicQB-ACNxAH4AIHEAfgAdcQB-AB5zcQB-ABICAAACOXBxAH4AJXEAfgAmcQB-ACBxAH4AHXEAfgAec3EAfgASAQAAAGZxAH4AFHEAfgAocQB-AClxAH4AKnBwc3EAfgASAQAAFz9xAH4AFHEAfgAVcQB-ABZxAH4AKnBwc3EAfgASAQAAAGNxAH4AFHEAfgAtcQB-AC5xAH4AL3Bwc3EAfgASAQAAArFxAH4AFHEAfgAxcQB-ADJxAH4AM3Bwc3EAfgASAQAAADFxAH4AFHEAfgA1cQB-ADZxAH4AN3Bwc3EAfgASAQAAADhxAH4AFHEAfgA5cQB-ADpxAH4AO3Bwc3EAfgASAQAAAGdxAH4AFHEAfgA9cQB-ADpxAH4APnBwc3EAfgASAQAAADVxAH4AFHEAfgA9cQB-ADpxAH4AQHBwc3EAfgASAQAAAC5xAH4AFHEAfgBCcQB-ADZxAH4AN3Bwc3EAfgASAQAAAn1xAH4AFHEAfgAxcQB-ADJxAH4AN3Bwc3EAfgASAgAAAhtwcQB-AEVxAH4ARnEAfgA7cQB-AB1xAH4AHnNxAH4AEgIAAAEIcHEAfgBIcQB-AElxAH4AN3EAfgAdcQB-AB5zcQB-ABICAAAEcHBxAH4AS3EAfgBMcQB-AE1xAH4AHXEAfgAec3EAfgASAgAAAntwcQB-AE9xAH4ATHEAfgA3cQB-AB1xAH4AHnNxAH4AEgIAAANIcHEAfgBRcQB-AFJxAH4AN3EAfgAdcQB-AB5xAH4AVHhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQvXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeAAAAAAAAAAQdnIAE2NvbS5jbG91ZC5ob3N0Lkhvc3S4fD08IdIeQAIAAHhw
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Publish async job-53849 complete on message bus
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Wake up jobs related to job-53849
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Update db status for job-53849

[weizhouapache]

I suspect if the systemvm template is Ready
please also check if the global setting router.template.vmware is set correctly

[Hossein4185]

@weizhouapache
The global settings are as follows:

I can use the shared network as shown in the image below, but I am having problems with Isoleted Network.