Problem with live migration.

[HonorQ]

Hello.
I have problem with IO delay while live migration. Two nodes are Supermicro 2U servers connected with 10G link and proxmox verson 7.1-10.
Now 1st node is in test stage without VM's. It's 40 x Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz (2 Sockets) witch 187.58 GiB RAM.
Both nodes have same zfs storage configured. For test in destination machine I'm using SSD pool (3x 512GB Intel disks).

When I start migration VM (2 disks 100G) to empty node it takes 1:10:35 with average speed 514 MiB/s (secured):

While migration is in progress the destination node get very big load average (~50) and IO delay about 30%. Htop on this machine show same load but CPU/DISK usage is jumping - few seconds normal and jump for few seconds to ~80%.
With empty node there is't too much problem (sometimes few tasks on VM need to be restarted), but there is worst when I start to migrate second VM to this node (same pool).

IO delay and load average hit same amount. CPU/DISK usage jumping too from:

to:


The problem is with other VM's on destination node. Some tasks stoping working and syslog show kernel errors. Htop on VM not show any problem - no CPU/DISK usage and almost 0 load average. After some times and few more kernel error VM freez with log in console "Reboot after 5 secs".

I don't know where to search reason. In configuration I disabled secured migration - this gave me faster transfer.

Have you any clue what could be wrong? Thanks in advance.

 

[LnxBil]

Could you please share the VM configuration:

cat /etc/pve/qemu-server/101.conf

 

[HonorQ]

balloon: 0
boot: order=scsi1
cores: 4
ide2: none,media=cdrom
memory: 4096
name: testing
net0: virtio=7A:0C:56:0B:5B:48,bridge=vmbr1
net1: virtio=36:0B:39:C0:DB:C5,bridge=vmbr1,tag=126
ostype: l26
scsi0: ssd2:vm-101-disk-4,cache=writeback,discard=on,format=raw,size=100G,ssd=1
scsi1: ssd2:vm-101-disk-5,cache=writeback,discard=on,format=raw,size=100G,ssd=1
scsihw: virtio-scsi-pci
sockets: 2
startup: order=1,up=1

 

[HonorQ]

I made backup from other VM to this node and 240G system disk was backed up in 8 min.

109: 2022-02-23 12:45:11 INFO: backup is sparse: 119.36 GiB (49%) total zero data
109: 2022-02-23 12:45:11 INFO: transferred 240.00 GiB in 529 seconds (464.6 MiB/s)
109: 2022-02-23 12:45:12 INFO: archive file size: 45.30GB
109: 2022-02-23 12:45:12 INFO: prune older backups with retention: keep-last=17
109: 2022-02-23 12:45:12 INFO: removing backup 'proxmox1_nfs:backup/vzdump-qemu-109-2021_10_18-04_06_39.vma.lzo'
109: 2022-02-23 12:45:12 INFO: pruned 1 backup(s) not covered by keep-retention policy
109: 2022-02-23 12:45:12 INFO: Finished Backup of VM 109 (00:08:50)

 

[HonorQ]

At this moment I know what causing this problem.
It begin at 1st phase of live migration, while destination machine allocating disk space for migrated machine.
When source VM hard disk has enabled discard flag then allocating probably fill all space with zeroes (very big use of Disk Write without network).
This generating so big IO delay which killing other VM's on this node (kernel panic/freeze/reboot).
When i disabled discard flag on source disk and started migration then it start immediately with phase 2 (% progress with network usage) and do not generate IO delay.

So now I get the new problem. Changing options in VM require reboot of it. I need discard option because backing up without trim absorb a lot of space. I found this problem on bugzilla:
https://bugzilla.proxmox.com/show_bug.cgi?id=2631
where Alucard1 wrote about changing SCSI controller resolving IO problem, but in my case it's not helping.

Is there any option to use discard option without writing zeroes while allocating space?

 

[hevisko]

Have you tried ZFS as the storage yet?
I haven't seen those type of funs yet with ZFS, but then the method I "prepare" for the migrations, is to first use a replication task (ZFS specific I believe?) to pre-empt the biggest parts of the moves, then the rest is much quicker on the IO and delays

 

[HonorQ]

Both nodes have same zfs storage configured. Changing one option - discard - on source VM hdd make this problem. When on then IO killing other VM's on destination node before "data" start transfer (it's take about 10 mins). If I disable this option then "data" transfer start immediately and not cause so big IO delay.

 

[HonorQ]

Hello,
the problem still exist and I found same issue here:
https://forum.proxmox.com/threads/v...m-thin-with-discard-results-in-high-i-o.97647
Sadly I didn't find there fix for this problem.
Do anyone know is there and option to start live migration on destination node with ionice or something else which would stop killing other VMs?

 

[LnxBil]

Do anyone know is there and option to start live migration on destination node with ionice or something else which would stop killing other VMs?

If you migrate via ssh, one option could be traffic shaping and therefore restricting the amount of data transfered. Problem is however it'll take VERY long and on the source side could be more changes occuring than you transfer through your tunnel and you end up with nothing.

Both nodes have same zfs storage configured. Changing one option - discard - on source VM hdd make this problem. When on then IO killing other VM's on destination node before "data" start transfer (it's take about 10 mins). If I disable this option then "data" transfer start immediately and not cause so big IO delay.

If that is really the case, this is a bug. ZFS on the source and destination side should not need to zeroize the disk, zvols are created with zeros everywhere.

 

[HonorQ]

If you migrate via ssh, one option could be traffic shaping and therefore restricting the amount of data transfered. Problem is however it'll take VERY long and on the source side could be more changes occuring than you transfer through your tunnel and you end up with nothing.

I have tryed limiting bandwith even by hardware (configured other interface to run live migration with hardware 100 Mbps ports). This not helped because there was not too much network traffic while zeroing disk. While it zeroing htop show max usage of disk IO write which cause the problem.

If that is really the case, this is a bug. ZFS on the source and destination side should not need to zeroize the disk, zvols are created with zeros everywhere.

Without "discard=on" everything works great (network transfer start immediately).

 

[LnxBil]

Without "discard=on" everything works great (network transfer start immediately).

Yes, you wrote that. ZFS does not need zeroing stuff, so this should not be done on a newly created zvol (it's another story for existing ones).

 

[HonorQ]

So you suggest to test it on newly created zvol?

 

[HonorQ]

Newly created zvol do not resolve this problem. I think this is not problem with zfs.

 

[LnxBil]

So you suggest to test it on newly created zvol?

No, PVE should not zeroize a newly created zvol, because it is already full of zeros. This needs to be taken care of the source code.