Fresh installation with ZFS crashes when restoring 1st VM from backup

May 28, 2018
68
11
13
38
I just installed the latest version of Proxmox (version 6.2-4), using ZFS RAID1.

Installed a few tools for ZFS, like auto-snapshot en ZFS-ZED,
setup a zfs-auto-snapshot schedule in /etc/cron.d/zfs-auto-snapshot
and enabled Let's Encrypt,
nothing else (so you see I did very very little to the new setup)

Copied a backupped VM over from another, slightly older Proxmox (version 5.3-8) server and ran:

qmrestore s2-test-vzdump-qemu-101-2020_05_11-12_08_20.vma.lzo 201 --storage local-zfs

The restore gets up to 86%, then the whole system freezes and needs a manual reboot by a datacenter tech.

Tried this twice now. 2nd time the restore got up to 90%, but froze again.

I'm shocked. Is Proxmox unstable / unusable?

The progress looks like this untill I lose the SSH connection:
restore vma archive: lzop -d -c /root/s1-production-vzdump-qemu-100-2020_05_26-03_00_01.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp32309.fifo - /var/tmp/vzdumptmp32309
CFG: size: 452 name: qemu-server.conf
DEV: dev_id=1 size: 1073741824000 devname: drive-scsi0
CTIME: Tue May 26 03:00:02 2020
new volume ID is 'local-zfs:vm-201-disk-0'
map 'drive-scsi0' to '/dev/zvol/rpool/data/vm-201-disk-0' (write zeros = 0)
progress 1% (read 10737418240 bytes, duration 13 sec)
progress 2% (read 21474836480 bytes, duration 24 sec)
progress 3% (read 32212254720 bytes, duration 28 sec)
...

Just tried a 3rd time and Proxmox froze completely on 91%.

The vzdump file appears to be intact.
 
Last edited:
Well, this is a brand new server. Fresh install, no VMs on it. Not doing anything.
Have been using Proxmox for many years, but never ran into this.
This contradicts your claim, doesn't it? ;)

I could not find anything in /var/log/syslog Just a big hole where the freeze/reboot was.
Besides whatever was logged and not written out. I would look at the performance of the system while the restore is running.
 
Yeah, you're right. Proxmox is stable. At least in my past experience. Should not have said so, sorry. I'm just really frustrated at this simple install, yet breaking big time. Wasted a whole day on this and no positive outlook :(

Maybe the latest version, ZFS or the hardware might be unstable, that I cannot say at this time.

Judging the performance is hard since there is nothing running on the machine, no?
The restore process runs fine up until around 90%, then complete freeze. This includes my SSH terminal, the web interface, everything really...

Waiting 30 minutes does not help.

I have ordered a full hardware scan and am waiting for results. In the past I have encountered bad hard drives that caused the backup process to break, but never a complete freeze like today.

Any other ideas?
This is very weird, no?
 
Does it restore on a different machine with the same PVE versions?
 
At the datacenter they did a full hardware scan. Turns out a fan was faulty and they replaced it. However... still same problem.
This time the server froze at 20% during the restore.

This time I also had KVM connected. At the time of freezing, the video signal just went dead.

The datacenter guys are now claiming there is a problem with the OS, but since this is a fresh install with nothing on it, I don't know anymore.
I'm trying to get another server, but I don't know if they will provide me with one :(

Simultaneously I'm trying to restore the VM on another Proxmox, but it's taking a long long time on that machine, so I'll know in a few hours...
 
This time I also had KVM connected. At the time of freezing, the video signal just went dead.
In my experience this sounds a lot like a hardware issue.

The datacenter guys are now claiming there is a problem with the OS, but since this is a fresh install with nothing on it, I don't know anymore.
I'm trying to get another server, but I don't know if they will provide me with one :(
That's easy for them to say. ;) You can try with an older pve-kernel (5.0, 5.3, 5.4) and see if it repeats.
 
The restore of the VM worked on another Proxmox, so looks indeed like a hardware issue.
I have ordered a new server (same type) and will restart everything...
Microcode and BIOS updates may help mitigate the issue.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!