Agent stuck in "Connecting" state, 4.21

[bradh352]

I have an agent stuck in the "Connecting" state, it seems like its the fault of the management server in some way as the agent just keeps restarting

The interesting log on the management server says it can't acquire a lock on the host, I'm assuming this is something DB related?

2025-10-28 15:04:36,040 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-2:[ctx-abe4ba59]) (logid:c6a383ed) Failed to handle host connection: com.cloud.exception.ConnectionException: Unable to acquire lock on host Host {"id":31,"name":"hv102.internal.nexthop.ai","type":"Routing","uuid":"8a8a2f0b-be76-4179-a2b9-c9785a81d061"}, to process agent connection

[bradh352]

I restarted by DB nodes and the agent was then able to connect, so seems like some long lived SQL lock? The lock survived restarting of all the management nodes, which seems exceptionally strange. Locks should always die when the connection dies, no?

[weizhouapache]

I faced this issue some times in the past, but did not encounter recently.

[DaanHoogland]

this is an “interesting” problem. Could be in the connector or the DB itself. bit of an expensive experiment, but I wonder if the lock would get freed on a tcp timeout…?

[bhouse-nexthop]

I think ultimately everything is inter-related to the libvirt 10.0.0 issue on ubuntu 24. I haven't yet tried @weizhouapache 10.6.0 deb package, but its on my list. For now restarting libvirt and cloudstack-agent daily has helped a lot.

[weizhouapache]

I think ultimately everything is inter-related to the libvirt 10.0.0 issue on ubuntu 24. I haven't yet tried @weizhouapache 10.6.0 deb package, but its on my list. For now restarting libvirt and cloudstack-agent daily has helped a lot.

yes, could be related.
let's see if this will appear again when libvirt/agent are restarted daily.