Backups cause problems in VMs

[mgiammarco]

Hello,
I have a proxmox 6.4 cluster with ceph 14.2.20. If I do long backups (for example full backups) the day after people have micro pauses using windows remote desktop. If I migrate the windows vm to another server the VM start working perfectly.
With fast backups (for example incremental backup to a Proxmox Backup Server) this problem not happens.
What can I do?
Thanks,
Mario

 

[mgiammarco]

I have just discovered an important thing. Yesterday I have powered off some VMs (so Proxmox Backup Server loses modified data and does a full backup instead). Today after several full backups again I have the slowness and micro pause problem. I have tried the migration trick and it does nothing this time. So I restarted ceph.target and, bingo, now all VMs runs fast again. So it seems that on ceph 14.2.20 and previous versions it has a bug that after a lot of i/o you need to restart daemons.

 

[Matteo Calorio]

Hi, I confirm that problem is still present on ceph version 16.2.10 pacific (stable) and that tha solution worked for me, too, for one VM.

As soon as I get the OK to proceed with others I'll do.

Thanks for writing down your solution and to floh8 who pointed me to this thread!

Matteo

 

[Matteo Calorio]

Hi,

we still noticed a strange behavior, after restarting ceph.target: 3 and only 3 VMs out of 200 still suffer the problem that backups crash and crash the VM itself.

In particular, these are VMs with windows 2019 os and a disk (P) dedicated to the 15GB pagefile, a configuration different from all the others.

Moving all disks or even just disk P from ceph-ssd to ceph-hdd, our two CEPH clusters, with identical installations and versions, problems disappear.

This doesn't make sense to me, but it's the only solution that currently works.

This is the config of one of the machines with problems:

agent: 1
balloon: 512
boot: cdn
bootdisk: virtio0
cores: 4
ide2: none,media=cdrom
memory: 65536
name: ***
net0: virtio=9A:3E:76:C6:88:ED,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=e8972118-4803-424e-872d-4f09310d4507
sockets: 2
virtio0: ceph-ssd:vm-8059-disk-0,iothread=1,size=50G
virtio1: ceph-ssd:vm-8059-disk-1,iothread=1,size=50G
virtio2: ceph-ssd:vm-8059-disk-2,iothread=1,size=200G
virtio3: ceph-ssd:vm-8059-disk-3,iothread=1,size=15G
virtio4: ceph-ssd:vm-8059-disk-4,iothread=1,size=50G
virtio5: ceph-ssd:vm-8059-disk-5,iothread=1,size=50G
virtio6: ceph-hdd:vm-8059-disk-0,iothread=1,size=15G
vmgenid: d53010c1-9388-4c04-a315-c2cdd102de06

Matteo