After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs again

shartenauer

Active Member
Aug 15, 2013
27
0
41
Hello Proxmox Forum,

before 2 days I updated my Proxmox 3.2 Host with the actual proxmox updates.
Everthing works fine, but the backup makes problems again.
The backup is done via a mounted cifs volume.
The backup now is very slow and hangs at at the 1. VM by around 22%.
The VM is then locked and frozen.
I can only qm unlock and qm stop to recover from this situation.

The problem was the last 2 nights after the upgrade.
Today I switched back to the 2.6.32-27 kernel.
In this constellation the problem doesn't ocure.

In the CPU Monitoring I can see constant iowaits from 15% during the backup with the kernel 2.6.32-29.

configuration before update:
pveversion -v
proxmox-ve-2.6.32: 3.2-124 (running kernel: 2.6.32-27-pve)
pve-manager: 3.2-2 (running version: 3.2-2/82599a65)
pve-kernel-2.6.32-27-pve: 2.6.32-121
pve-kernel-2.6.32-28-pve: 2.6.32-124
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-15
pve-firmware: 1.1-2
libpve-common-perl: 3.0-14
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-6
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

configuration after update:
pveversion -v
proxmox-ve-2.6.32: 3.2-126 (running kernel: 2.6.32-29-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-2.6.32-27-pve: 2.6.32-121
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-29-pve: 2.6.32-126
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-16
pve-firmware: 1.1-3
libpve-common-perl: 3.0-18
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-8
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

The problem also occures with the Kernel 2.6.32-28.

Best regards Stephan
 
Re: After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs ag

Thanks for reporting this. I ran into the same problem on 2 different servers. I'm going to try downgrading to 2.6.32-27 to see what happens.
 
Re: After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs ag

Update: downgrading the kernel solved the problem for me too. There must be something in the new PVE kernels that causes this.
 
Re: After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs ag

vasek125, I think you posted the same thread link twice. The way newer vzdump backups work for kvm images is fundamentally different AFAIK. They use the kvm/qemu native snapshot ability when used with qcow2 images. This might not be as reliable as expected. I'm just shooting in the dark of course, but in the meantime it would be very useful to be able to switch snapshot modes for more reliable operation of the backup, if nothing else.

On the other hand, I also have problems with several OpenVZ containers. Their backup runs excruciatingly slow, a backup that usually runs for minutes, can run several hours on the new kernels. I haven't had the chance do downgrade the kernel where this problem occurs, and it might or might not be related to the problem with qcow2 snapshotted backups, but both problems started to appear on the newer kernels. So as far as I can guess, the problems could also be caused by a patch Proxmox introduced in their newer kernel. As a (hopefully) temporary solution to this latter problem was alleviated by using a CIFS share instead of NFS.
 
Re: After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs ag

They use the kvm/qemu native snapshot ability when used with qcow2 images.

Note: We do not use qcow2 snapshot ability for backup (we use new KVM snapshot backup block-jobs).
 
Re: After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs ag

I found two ways to make backup working: Use sshfs - always works for me (NFS nor CIFS dont) or make your custom backup script which uses LVM snapshot. LVM snapshot is a good feature so I dont think it was neccessary to reinvent the wheel with the vzdump custom backup method which is slower and not reliable.
 
Re: After update to kernel vmlinuz-2.6.32-28-pve or 2.6.32-29-pve the backup hangs ag

Yes, earlier when NFS was totally borked and unreliable in the 1.8 and early 2.x series (not necessarily in connection with PVE, could well have been an upstream kernel issue) I also used sshfs with success. But it's also a bit slower and has the unnecessary overhead of encryption. For now and for my setup CIFS is working fine. I'll do some tests with the 3.10 kernels later.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!