failed VM migration cloudstack 4.20.1

[Chih80]

problem

migration error on some VMs to a different host. Some VMs are able to migrate while others failed at the start.

versions

cloudstack 4.20.1
managment and mysql on ubuntu 22.04
all host on ubuntu 24.04
All host are using local storage for VMs.

The steps to reproduce the bug

1. select 'migrate instance to another host' in UI on the selected VM.
2. select host and migrate withstorage
3. failed output:
   Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13- fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"38"}] and VM to host [{vm: "520", from: "57", to:"62"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/efcb9266-8ee1-44e7-a569-f1efb3d7f4aa' for incremental storage migration of disk 'vda' is not supported].].

What to do about it?

On the same host, able to migrate some VMs. The ones that can't have the error output stated above.

Log:

2025-08-22 23:05:47,373 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) STATS: Failed to alloc resource from host: Host {"id":61,"name":"reno-host-b03","type":"Routing","uuid":"2f275bc0-c8df-4c73-a497-855a6e134673"} reservedCpu: 0, requested cpu: 48000, reservedMem: (0 bytes) 0, requested mem: (64.00 GB) 68719476736
2025-08-22 23:05:47,505 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8c5c2b95]) (logid:d1497d39) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms
2025-08-22 23:05:48,551 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[ctx-c3e7bb0f]) (logid:) Seq 61-1951747488511695638: Processing: { Ans: , MgmtId: 98960401734289, via: 61, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Can't find volume:b79015df-7d75-421c-8930-e3892ef0052a","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-22 23:05:48,963 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[ctx-0ce0c857]) (logid:) Seq 57-5361253881407871722: Processing: { Ans: , MgmtId: 98960401734289, via: 57, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-22 23:05:48,973 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:[]) (logid:) Seq 61-1951747488511695640: Processing: { Ans: , MgmtId: 98960401734289, via: 61, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool ee9296f3-ed01-4bbd-856a-84beb37ed244 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'ee9296f3-ed01-4bbd-856a-84beb37ed244'
2025-08-22 23:05:48,973 DEBUG [o.a.c.s.m.KvmNonManagedStorageDataMotionStrategy] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) Failed to disconnect one or more (original) dest volumes com.cloud.exception.AgentUnavailableException: Resource [Host:61] is unreachable: Host 61: Unable to rollback prepare for migration due to the following: com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool ee9296f3-ed01-4bbd-856a-84beb37ed244 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid 'ee9296f3-ed01-4bbd-856a-84beb37ed244'
2025-08-22 23:05:49,005 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-321b6563]) (logid:e9f8869b) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms
2025-08-22 23:05:50,505 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-8a18f456]) (logid:fc5f1db6) Peer scan is finished. profiler: Done. Duration: 2ms , profilerQueryActiveList: Done. Duration: 0ms, , profilerSyncClusterInfo: Done. Duration: 0ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 0ms
2025-08-22 23:05:51,174 ERROR [o.a.c.s.m.KvmNonManagedStorageDataMotionStrategy] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported]. com.cloud.utils.exception.CloudRuntimeException: Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported
2025-08-22 23:05:51,184 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) Failed to copy volume com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].
2025-08-22 23:05:51,184 ERROR [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].].
2025-08-22 23:05:51,204 WARN [c.c.a.AlertManagerImpl] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) alertType=[17] dataCenterId=[1] podId=[1] clusterId=[null] message=[Unable to migrate vm i-5-520-VM from host reno-host-b01 in zone Reno-halfbat and pod Reno-halfbat].
2025-08-22 23:05:51,234 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Running","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"} state transited from [Migrating] to [Running] with event [OperationFailed]. VM's original host: Host {"id":57,"name":"reno-host-b01","type":"Routing","uuid":"d56c66e1-d646-4453-9879-80ad11719fc6"}, new host: Host {"id":57,"name":"reno-host-b01","type":"Routing","uuid":"d56c66e1-d646-4453-9879-80ad11719fc6"}, host before state transition: Host {"id":61,"name":"reno-host-b03","type":"Routing","uuid":"2f275bc0-c8df-4c73-a497-855a6e134673"}
2025-08-22 23:05:51,254 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].].
2025-08-22 23:05:51,254 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661, ctx-83f5b0a5]) (logid:2557fd83) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].].
2025-08-22 23:05:51,254 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661]) (logid:2557fd83) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkMigrateWithStorage","cmdInfo":"rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAIIdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAAA9AAAAAAAAADlzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAFNHNxAH4ACAAAAAAAAAAleA","cmdVersion":0,"completeMsid":null,"created":"Fri Aug 22 23:05:47 UTC 2025","id":18661,"initMsid":98960401734289,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"6f9bb3ab-0026-4b47-ab5b-bc04401d2e8f"}, job origin: 18660 com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].].
2025-08-22 23:05:51,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-118:[ctx-7da649c7, job-18660/job-18661]) (logid:2557fd83) Complete async job-18661, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AplGYWlsZWQgdG8gbWlncmF0ZSBWTSBbVk0gaW5zdGFuY2UgeyJpZCI6NTIwLCJpbnN0YW5jZU5hbWUiOiJpLTUtNTIwLVZNIiwic3RhdGUiOiJNaWdyYXRpbmciLCJ0eXBlIjoiVXNlciIsInV1aWQiOiI1YzI0MmY0ZC02NDg2LTQ2M2YtOGUxMy1mZWEwODlkYjJlOWIifV0gYWxvbmcgd2l0aCBpdHMgdm9sdW1lcyBkdWUgdG8gW2NvbS5jbG91ZC51dGlscy5leGNlcHRpb24uQ2xvdWRSdW50aW1lRXhjZXB0aW9uOiBDb3B5IHZvbHVtZShzKSB0byBzdG9yYWdlKHMpIFt7dm9sdW1lOiAiMTMzMiIsIGZyb206ICIzMyIsIHRvOiIzNyJ9XSBhbmQgVk0gdG8gaG9zdCBbe3ZtOiAiNTIwIiwgZnJvbTogIjU3IiwgdG86IjYxIn1dIGZhaWxlZCBpbiBTdG9yYWdlU3lzdGVtRGF0YU1vdGlvblN0cmF0ZWd5LmNvcHlBc3luYy4gRXJyb3IgbWVzc2FnZTogW0V4Y2VwdGlvbiBkdXJpbmcgbWlncmF0ZTogb3JnLmxpYnZpcnQuTGlidmlydEV4Y2VwdGlvbjogT3BlcmF0aW9uIG5vdCBzdXBwb3J0ZWQ6IHByZS1jcmVhdGlvbiBvZiBzdG9yYWdlIHRhcmdldCAnL3Zhci9saWIvbGlidmlydC9pbWFnZXMvZTc5MzQ3MDgtOTU1ZC00OTlkLThlZGQtZWVmYWQ3NzNjNDRlJyBmb3IgaW5jcmVtZW50YWwgc3RvcmFnZSBtaWdyYXRpb24gb2YgZGlzayAndmRhJyBpcyBub3Qgc3VwcG9ydGVkXS5dLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAH3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAFpHQAA2FwcHQAPW9yZy5hcGFjaGUuY2xvdWRzdGFjay5lbmdpbmUub3JjaGVzdHJhdGlvbi5Wb2x1bWVPcmNoZXN0cmF0b3J0ABdWb2x1bWVPcmNoZXN0cmF0b3IuamF2YXQADm1pZ3JhdGVWb2x1bWVzcHBzcQB-AAwA_____3B0ADBqZGsuaW50ZXJuYWwucmVmbGVjdC5HZW5lcmF0ZWRNZXRob2RBY2Nlc3NvcjIxNTZwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxNy4wLjE2c3EAfgAMAgAAAjlwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAABWHEAfgAOdAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25wcHNxAH4ADAEAAADGcQB-AA50ADxvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb250AB9SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbi5qYXZhdAAPaW52b2tlSm9pbnBvaW50cHBzcQB-AAwBAAAAo3EAfgAOcQB-ACJxAH4AI3QAB3Byb2NlZWRwcHNxAH4ADAEAAABhcQB-AA50AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAUcHBzcQB-AAwBAAAAunEAfgAOcQB-ACJxAH4AI3EAfgAmcHBzcQB-AAwBAAAA13EAfgAOdAA0b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLkpka0R5bmFtaWNBb3BQcm94eXQAF0pka0R5bmFtaWNBb3BQcm94eS5qYXZhcQB-ABRwcHNxAH4ADAD_____cHQAFGpkay5wcm94eTMuJFByb3h5MjY4cHEAfgARdAAKamRrLnByb3h5M3BzcQB-AAwBAAAM-HEAfgAOdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZU1pZ3JhdGVXaXRoU3RvcmFnZXBwc3EAfgAMAQAAFZ5xAH4ADnEAfgAycQB-ADNxAH4ANHBwc3EAfgAMAP____9wdAAwamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3IyMTU0cHEAfgAUcHBzcQB-AAwCAAAAK3BxAH4AFnEAfgAXcQB-ABRxAH4AGHEAfgAZc3EAfgAMAgAAAjlwcQB-ABtxAH4AHHEAfgAUcQB-ABhxAH4AGXNxAH4ADAEAAABmcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABXqcQB-AA5xAH4AMnEAfgAzcQB-AD1wcHNxAH4ADAEAAABjcQB-AA50ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgAMAQAAAoxxAH4ADnQAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgAMAQAAADFxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-AAwBAAAAOHEAfgAOdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-AAwBAAAAZ3EAfgAOdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgBNdAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-AAwBAAAANXEAfgAOcQB-AFBxAH4ATXQADnJ1bldpdGhDb250ZXh0cHBzcQB-AAwBAAAALnEAfgAOdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-AElxAH4ASnBwc3EAfgAMAQAAAlhxAH4ADnEAfgBEcQB-AEVxAH4ASnBwc3EAfgAMAgAAAhtwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-AE5xAH4AGHEAfgAZc3EAfgAMAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgBKcQB-ABhxAH4AGXNxAH4ADAIAAARwcHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-ABhxAH4AGXNxAH4ADAIAAAJ7cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4AX3EAfgBKcQB-ABhxAH4AGXNxAH4ADAIAAANIcHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgBKcQB-ABhxAH4AGXNyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHgAABCadwgAAAAAAAAAAHg
2025-08-22 23:05:51,279 WARN [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-105:[ctx-15ae3173, job-18660, ctx-81b6ceec]) (logid:2557fd83) Skipping delivering event Event {"description":"{"event":"VM.MIGRATE","status":"Completed"}","eventId":null,"eventType":"VM.MIGRATE","eventUuid":null,"resourceType":"VirtualMachine","resourceUUID":null} to any webhook as account ID is missing
2025-08-22 23:05:51,280 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-105:[ctx-15ae3173, job-18660, ctx-81b6ceec]) (logid:2557fd83) Failed to publish event [category: ActionEvent, type: VM.MIGRATE] on bus webhookEventBus
2025-08-22 23:05:51,290 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-105:[ctx-15ae3173, job-18660]) (logid:2557fd83) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].].
2025-08-22 23:05:51,290 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-105:[ctx-15ae3173, job-18660]) (logid:2557fd83) Complete async job-18660, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to migrate VM [VM instance {"id":520,"instanceName":"i-5-520-VM","state":"Migrating","type":"User","uuid":"5c242f4d-6486-463f-8e13-fea089db2e9b"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "1332", from: "33", to:"37"}] and VM to host [{vm: "520", from: "57", to:"61"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Operation not supported: pre-creation of storage target '/var/lib/libvirt/images/e7934708-955d-499d-8edd-eefad773c44e' for incremental storage migration of disk 'vda' is not supported].]."}

[kiranchavala]

@Chih80 If the vm are using local storage then live migration between 2 hosts on kvm is not supported

https://docs.cloudstack.apache.org/en/4.20.1.0/adminguide/virtual_machines.html#moving-instances-between-hosts-manual-live-migration

I believe this is a libvirt limiation

As a workaround please stop the vm and then try to migrate

https://docs.cloudstack.apache.org/en/4.20.1.0/adminguide/virtual_machines.html#moving-instance-s-volumes-between-storage-pools-offline-volume-migration

[Chih80]

@Chih80 If the vm are using local storage then live migration between 2 hosts on kvm is not supported

https://docs.cloudstack.apache.org/en/4.20.1.0/adminguide/virtual_machines.html#moving-instances-between-hosts-manual-live-migration

I believe this is a libvirt limiation

As a workaround please stop the vm and then try to migrate

https://docs.cloudstack.apache.org/en/4.20.1.0/adminguide/virtual_machines.html#moving-instance-s-volumes-between-storage-pools-offline-volume-migration

This is weird/confusing, I have live migrated some VMs without issue.

[DaanHoogland]

@Chih80 , do you only have local storage? And have you maybe migrated without storage?

[Chih80]

@Chih80 , do you only have local storage? And have you maybe migrated without storage?

I'm only using local storage and I have always migrated with storage since I manually select storage destination.

[DaanHoogland]

Can you check support for this in libvirt/qemu? Have you upgraded any of these recently? life migration on KVM is a bit of an issue from time to time and the underlying software versions are usually the culprit.

[Chih80]

Can you check support for this in libvirt/qemu? Have you upgraded any of these recently? life migration on KVM is a bit of an issue from time to time and the underlying software versions are usually the culprit.

I'm currently on version 10.0 / 8.2.2