Fresh installation with ZFS crashes when restoring 1st VM from backup

[Wouter Samaey]

I just installed the latest version of Proxmox (version 6.2-4), using ZFS RAID1.

Installed a few tools for ZFS, like auto-snapshot en ZFS-ZED,
setup a zfs-auto-snapshot schedule in /etc/cron.d/zfs-auto-snapshot
and enabled Let's Encrypt,
nothing else (so you see I did very very little to the new setup)

Copied a backupped VM over from another, slightly older Proxmox (version 5.3-8) server and ran:

qmrestore s2-test-vzdump-qemu-101-2020_05_11-12_08_20.vma.lzo 201 --storage local-zfs

The restore gets up to 86%, then the whole system freezes and needs a manual reboot by a datacenter tech.

Tried this twice now. 2nd time the restore got up to 90%, but froze again.

I'm shocked. Is Proxmox unstable / unusable?

The progress looks like this untill I lose the SSH connection:

restore vma archive: lzop -d -c /root/s1-production-vzdump-qemu-100-2020_05_26-03_00_01.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp32309.fifo - /var/tmp/vzdumptmp32309
CFG: size: 452 name: qemu-server.conf
DEV: dev_id=1 size: 1073741824000 devname: drive-scsi0
CTIME: Tue May 26 03:00:02 2020
new volume ID is 'local-zfs:vm-201-disk-0'
map 'drive-scsi0' to '/dev/zvol/rpool/data/vm-201-disk-0' (write zeros = 0)
progress 1% (read 10737418240 bytes, duration 13 sec)
progress 2% (read 21474836480 bytes, duration 24 sec)
progress 3% (read 32212254720 bytes, duration 28 sec)
...

Just tried a 3rd time and Proxmox froze completely on 91%.

The vzdump file appears to be intact.

 

[Alwin]

I'm shocked. Is Proxmox unstable / unusable?

How do you arrive at this conclusion?

Just tried a 3rd time and Proxmox froze completely on 91%.

What was the systems performance during the backup and when it "froze"? What do the logs say?

 

[Wouter Samaey]

How do you arrive at this conclusion?

Well, this is a brand new server. Fresh install, no VMs on it. Not doing anything.
Have been using Proxmox for many years, but never ran into this.

What do the logs say?

I could not find anything in /var/log/syslog Just a big hole where the freeze/reboot was.

What can I check?

 

[Alwin]

Well, this is a brand new server. Fresh install, no VMs on it. Not doing anything.
Have been using Proxmox for many years, but never ran into this.

This contradicts your claim, doesn't it?

I could not find anything in /var/log/syslog Just a big hole where the freeze/reboot was.

Besides whatever was logged and not written out. I would look at the performance of the system while the restore is running.

 

[Wouter Samaey]

Yeah, you're right. Proxmox is stable. At least in my past experience. Should not have said so, sorry. I'm just really frustrated at this simple install, yet breaking big time. Wasted a whole day on this and no positive outlook

Maybe the latest version, ZFS or the hardware might be unstable, that I cannot say at this time.

Judging the performance is hard since there is nothing running on the machine, no?
The restore process runs fine up until around 90%, then complete freeze. This includes my SSH terminal, the web interface, everything really...

Waiting 30 minutes does not help.

I have ordered a full hardware scan and am waiting for results. In the past I have encountered bad hard drives that caused the backup process to break, but never a complete freeze like today.

Any other ideas?
This is very weird, no?

 

[mira]

Does it restore on a different machine with the same PVE versions?

 

[Wouter Samaey]

At the datacenter they did a full hardware scan. Turns out a fan was faulty and they replaced it. However... still same problem.
This time the server froze at 20% during the restore.

This time I also had KVM connected. At the time of freezing, the video signal just went dead.

The datacenter guys are now claiming there is a problem with the OS, but since this is a fresh install with nothing on it, I don't know anymore.
I'm trying to get another server, but I don't know if they will provide me with one

Simultaneously I'm trying to restore the VM on another Proxmox, but it's taking a long long time on that machine, so I'll know in a few hours...

 

[Alwin]

This time I also had KVM connected. At the time of freezing, the video signal just went dead.

In my experience this sounds a lot like a hardware issue.

The datacenter guys are now claiming there is a problem with the OS, but since this is a fresh install with nothing on it, I don't know anymore.
I'm trying to get another server, but I don't know if they will provide me with one

That's easy for them to say. You can try with an older pve-kernel (5.0, 5.3, 5.4) and see if it repeats.

 

[Wouter Samaey]

The restore of the VM worked on another Proxmox, so looks indeed like a hardware issue.
I have ordered a new server (same type) and will restart everything...

Fingers crossed.

 

[Alwin]

The restore of the VM worked on another Proxmox, so looks indeed like a hardware issue.
I have ordered a new server (same type) and will restart everything...

Microcode and BIOS updates may help mitigate the issue.

 

[Wouter Samaey]

I got the new replacement server and everything works so far...

Thanks for your help!