OOBM Issue during KVM HA

[muthukrishnang1100]

Hi, I have an issue. I am trying to Test KVM HA test on my 3 host cloudstack. When I enable the OOBM on any one of the host 5 to minutes later the error comes. and after that node not able to login dashboard. Please anyone face this isuue if please what i do for this issue? and also after i disable and restart the Cloudstack management it will works fine no error.

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2025-12-30 12:51:49,231 ERROR [c.c.s.StatsCollector] (StatsCollector-1:ctx-2c2139dd) (logid:e41447a0) Error trying to monitor autoscaling
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)
at com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:960)
at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:939)
at com.cloud.cluster.dao.ManagementServerHostDaoImpl.findOneInUpState(ManagementServerHostDaoImpl.java:270)
at jdk.internal.reflect.GeneratedMethodAccessor284.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy161.findOneInUpState(Unknown Source)
at com.cloud.server.StatsCollector$AutoScaleMonitor.runInContext(StatsCollector.java:1766)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2025-12-30 12:51:49,240 ERROR [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8d4322d8) (logid:dcafa96b) Fail to complete the CheckRouterTask!
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)

[ingox]

@muthukrishnang1100 Your error message from the log does not directly belong to the KVM HA setup. The CloudStack management server does not get a connection to the database. Please check your database server and connection limits.

[muthukrishnang1100]

but, this error only occurs when i enable the OOBM on the host, so why it is comes only enable the OOBM? and also this error occurs when enable OOBM on host-1 error comes on host-1 or sometimes hots-2, 3.2025-12-31 12:45:37,062 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-2:null) (logid:7aa7e2e6) Process standard output for command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]: [Chassis Power is on
].
2025-12-31 12:45:37,062 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-2:null) (logid:7aa7e2e6) Process standard error output command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]: [Loading IANA PEN Registry...
Using best available cipher suite 3

Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20
Invalid completion code received: Invalid command
Discovered IPMB address 0x0
].
2025-12-31 12:45:37,062 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (pool-1-thread-2:null) (logid:7aa7e2e6) The command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p **** -U ADMIN -P **** chassis power status] was successful and got the result [Chassis Power is on].
2025-12-31 12:45:39,070 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-5:ctx-e9676f5b) (logid:a4534a37) HA health check task is running...
2025-12-31 12:45:41,710 DEBUG [c.c.a.ApiServlet] (qtp341748265-310:ctx-fcb0e415) (logid:5e23a71d) ===START=== 10.1.12.54 -- GET command=readyForShutdown&response=json&sessionkey=5cpt1hqcFjsg3EJGa1t8rtgrwnQ
2025-12-31 12:45:41,710 DEBUG [c.c.a.ApiServlet] (qtp341748265-310:ctx-fcb0e415) (logid:5e23a71d) Two factor authentication is already verified for the user 2, so skipping

after, this error comes we cant accessible the dashboard of that node

[ingox]

@muthukrishnang1100 you may check this:
while dashboard becomes inaccessible:

• check high CPU on management server
• are there many IPMI processes running? ps -ef | grep ipmitool | wc -l

under regular circumstances check:

• run several times: time ipmitool -I lanplus -R 1 -N 3 -H 192.168.10.117 -p 623 -U ADMIN -P '*****' chassis power status

What latency do you have between management and your hypervisor?