domain user metrics - request failed

[PPisz]

Hello,

After upgrading CS to 4.22, all domain user across all domains lost the ability to view VM metrics. A "Request failed" message displays, with no visible error in the mgmt log. All domain admins and root admins can see metrics as usual.
Has anyone else encountered this error? Or what might be causing it?

Piotr

[DaanHoogland]

@PPisz , I hadn’t heard this before. from what version did you upgrade?

[PPisz]

From 4.21, and it definitely worked there.

[PPisz]

On management log we have:
`2025-12-11 10:55:41,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (StatsCollector-5:[ctx-77b16c86]) (logid:0809b460) Seq 4-4433512358169719428: Routed from 37951217891278
2025-12-11 10:55:42,442 DEBUG [c.c.a.ApiServlet] (qtp1201454821-19817:[ctx-c80e9b6d]) (logid:d4eefa21) ===START=== 45.93.161.24 -- GET id=2a7e988d-6c4f-4ece-9420-be21e4c3c520&startDate=2025-12-11T09:55:43%2B01:00&endDate=2025-12-11T10:55:43%2B01:00&command=listVirtualMachinesUsageHistory&response=json&&projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
2025-12-11 10:55:42,442 DEBUG [c.c.a.ApiServlet] (qtp1201454821-19817:[ctx-c80e9b6d]) (logid:d4eefa21) Two factor authentication is already verified for the user 307, so skipping
2025-12-11 10:55:42,450 DEBUG [c.c.a.ApiServer] (qtp1201454821-19817:[ctx-c80e9b6d, ctx-2f1b7979]) (logid:d4eefa21) CIDRs from which account 'Account [{"accountName":"usecrypt","id":104,"uuid":"d9591bbb-be72-4348-97ec-3dea3e76cb72"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-12-11 10:55:42,463 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1201454821-19817:[ctx-c80e9b6d, ctx-2f1b7979]) (logid:d4eefa21) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-12-11 10:55:42,476 DEBUG [c.c.a.ApiServlet] (qtp1201454821-19817:[ctx-c80e9b6d, ctx-2f1b7979]) (logid:d4eefa21) ===END=== 45.93.161.24 -- GET id=2a7e988d-6c4f-4ece-9420-be21e4c3c520&startDate=2025-12-11T09:55:43%2B01:00&endDate=2025-12-11T10:55:43%2B01:00&command=listVirtualMachinesUsageHistory&response=json&&projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
2025-12-11 10:55:42,762 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-3:[]) (logid:) Seq 5-7783909005956528746: MgmtId 165190512198206 : Resp: Routing to peer
2025-12-11 10:55:42,886 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-2:[]) (logid:) Seq 4-8870402416059633629: Routing from 165190512198206
2025-12-11 10:55:42,909 DEBUG [c.c.a.t.Request] (StatsCollector-5:[ctx-77b16c86]) (logid:0809b460) Seq 5-4433512358169719428: Received: { Ans: , MgmtId: 37951217891278, via: 5(sdc.dco.webdisk.io), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2025-12-11 10:55:43,010 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-11:[]) (logid:) Seq 9-8870402416059633629: MgmtId 165190512198206 : Resp: Routing to peer
2025-12-11 10:55:43,085 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a60498c]) (logid:8583a3c8) No inactive management server node found
2025-12-11 10:55:43,085 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-6a60498c]) (logid:8583a3c8) Peer scan is finished. profiler: Done. Duration: 3ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-12-11 10:55:43,121 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StatsCollector-5:[ctx-77b16c86]) (logid:0809b460) Wait time setting on com.cloud.agent.api.GetVmStatsCommand is 1800 seconds

If I use cmk as root admin:

(localcloud) 🐱 > list virtualmachinesusagehistory startdate="2025-12-11 09:55:43" enddate="2025-12-11 10:55:43" id=2a7e988d-6c4f-4ece-9420-be21e4c3c520 projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
{
"count": 1,
"virtualmachine": [
{
"displayname": "usecrypt-jump",
"id": "2a7e988d-6c4f-4ece-9420-be21e4c3c520",
"name": "i-107-2479-VM",
"stats": [
{
"cpuused": "0.54%",
"diskiopstotal": 56,
"diskioread": 0,
"diskiowrite": 56,
"diskkbsread": 0,
"diskkbswrite": 280,
"memoryintfreekbs": 444348,
"memorykbs": 2097152,
"memorytargetkbs": 2097152,
"networkkbsread": 12,
"networkkbswrite": 0,
"networkread": "0.01 MiB",
"networkwrite": "0.00 MiB",
"timestamp": "2025-12-11T09:56:13+0100"
},

If I use cmk as domain user:

(localcloud) 🐱 > list virtualmachinesusagehistory startdate="2025-12-11 09:55:43" enddate="2025-12-11 10:55:43" id=2a7e988d-6c4f-4ece-9420-be21e4c3c520 projectid=d4e998d6-c4c5-4715-94a5-dc94519687d4
(localcloud) 🐱 >

What else could I check?

[winterhazel]

@PPisz thanks for providing the API parameters and the logs. I was able to identify that there was a regression in commit c8d44d9 (on 4.22.0 and 4.20.2) that prevents non-admin accounts from listing the metrics of VMs and volumes that belong to a project. #12284 should restore the previous behavior.

For now, unfortunately, there isn't a workaround to allow users listing the metrics of these VMs.

[PPisz]

Thank you @winterhazel , for me this is important information, better a known CS bug than strange behavior on my environment :-)

[DaanHoogland]

@PPisz , can you test the PR @winterhazel created? (#12284)

[kiranchavala]

@DaanHoogland tested the PR, issue is resolved