missing all VMs' located on ZFS (local HDD) storage

[zolwie]

Good day,
I am a new Proxmox user.
I installed it for testing on an HP Z420 box.

configuration relevant to the problem:
2 x SD disk 120GB
2 x HDD disks 1TG

Proxmox installed on a created RAID of 120GB SSD disks - no problem with that

What is important is DATASTOR.

2 x 1TB in RAID and ZFS pool
It mapped nicely in the datacenter.
I created 3 VMs and everything worked. VMs installed on HDD-Storage (on this ZFS)
I turned off the virtual machines for the night.

And in the morning PROBLEM.

There is none of the created machines in the datacenter.
Datastore mapped correctly, but I do not see any VM.

What happened? And how can I recover them?

virtual machine 110 (test) was created after the problem was discovered

It looks like data are still stored on the ZFS pool, but VM are not visible in the proxmox




root@proxmox:~# lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1
├─sda2 vfat FAT32 0D88-8D9E
└─sda3 zfs_member 5000 rpool 9858725616785542742
sdb
├─sdb1
├─sdb2 vfat FAT32 0D87-C769
└─sdb3 zfs_member 5000 rpool 9858725616785542742
sdc
├─sdc1 zfs_member 5000 HDD-storage 12506070680966649297
└─sdc9
sdd
├─sdd1 zfs_member 5000 HDD-storage 12506070680966649297
└─sdd9
zd0
├─zd0p1
└─zd0p2 ext4 1.0 f5805798-e084-48e7-8fef-8d86d21f9a9c
zd16
├─zd16p1
└─zd16p2 ext4 1.0 2c9fde65-c76d-4c3a-9618-a494bd244f97
zd32
├─zd32p1 ntfs System Reserved C036AD1436AD0D0A
└─zd32p2 ntfs DCC2ADF1C2ADD056
zd48
├─zd48p1 xfs 37c850ff-21d7-4eb0-b725-ae2864ff48b1
└─zd48p2 LVM2_member LVM2 001 4THQKI-BzVg-0ykQ-4vhR-v0SX-3bfv-twLZf1

The directory has property size

root@proxmox:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
HDD-storage 154G 745G 96K /HDD-storage
HDD-storage/vm-100-disk-0 30.5G 773G 2.47G -
HDD-storage/vm-101-disk-0 30.5G 773G 2.80G -
HDD-storage/vm-102-disk-0 60.9G 795G 10.6G -
HDD-storage/vm-110-disk-0 32.5G 773G 4.58G -
rpool 11.1G 103G 104K /rpool
rpool/ROOT 1.40G 103G 96K /rpool/ROOT
rpool/ROOT/pve-1 1.40G 103G 1.40G /
rpool/data 96K 103G 96K /rpool/data
rpool/var-lib-vz 9.66G 103G 9.66G /var/lib/vz








root@proxmox:~# zpool status
pool: HDD-storage
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
HDD-storage ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-WDC_WD10EZEX-08WN4A0_WD-WCC6Y4YTU0EK ONLINE 0 0 0
ata-WDC_WD10EZEX-08WN4A0_WD-WCC6Y7JUYN8S ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-SAMSUNG_MZ7LN128HCHP-000H1_S1ZRNX0H757325-part3 ONLINE 0 0 0
ata-SAMSUNG_MZ7LN128HCHP-000H1_S1ZRNX0H758230-part3 ONLINE 0 0 0






Please help me solve the problem

Regards

 

[UdoB]

First look into this folder:

~# ls -al /etc/pve/local/qemu-server/

Do you see more than "110.conf"? If not then you've lost configuration files.

Then examine the commands logged in the Zpool history like this:

~# zpool history HDD-storage

Are there any "destroy" commands listed?

 

[zolwie]

root@proxmox:/etc/pve/local/qemu-server# ls -la
total 2
drwxr-xr-x 2 root www-data 0 Jan 27 11:36 .
drwxr-xr-x 2 root www-data 0 Jan 27 11:36 ..
-rw-r----- 1 root www-data 336 Jan 29 13:27 100.conf.gz
-rw-r----- 1 root www-data 383 Jan 29 13:07 101.conf.gz
-rw-r----- 1 root www-data 429 Jan 29 13:33 102.conf.gz
-rw-r----- 1 root www-data 416 Jan 30 11:04 110.conf


zpool history HDD-storage
History for 'HDD-storage':
2025-01-27.12:02:06 zpool create -o ashift=12 HDD-storage mirror /dev/disk/by-id/ata-WDC_WD10EZEX-08WN4A0_WD-WCC6Y4YTU0EK /dev/disk/by-id/ata-WDC_WD10EZEX-08WN4A0_WD-WCC6Y7JUYN8S
2025-01-27.12:02:07 zfs set compression=off HDD-storage
2025-01-28.14:20:22 zfs create -V 31457280k HDD-storage/vm-100-disk-0
2025-01-29.07:42:28 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-29.07:46:22 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-29.07:51:07 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-29.08:03:38 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-29.12:28:53 zfs create -V 31457280k HDD-storage/vm-101-disk-0
2025-01-29.13:01:33 zfs create -V 8900608k HDD-storage/vm-101-state-aa
2025-01-29.13:01:51 zfs snapshot HDD-storage/vm-101-disk-0@aa
2025-01-29.13:04:35 zfs rollback HDD-storage/vm-101-disk-0@aa
2025-01-29.13:06:28 zfs rollback HDD-storage/vm-101-disk-0@aa
2025-01-29.13:07:06 zfs destroy -r HDD-storage/vm-101-state-aa
2025-01-29.13:07:06 zfs destroy HDD-storage/vm-101-disk-0@aa
2025-01-29.13:32:50 zfs create -V 62914560k HDD-storage/vm-102-disk-0
2025-01-30.10:08:46 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-30.10:12:27 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-30.10:36:53 zfs create -V 33554432k HDD-storage/vm-110-disk-0
2025-01-30.11:08:46 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-30.11:56:45 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage
2025-01-30.12:07:01 zpool import -N -d /dev/disk/by-id -o cachefile=none HDD-storage

 

[zolwie]

Progress
Thank you UdoB so much.

I found archived config files.
I decompressed it and VKs' came back to live.

What happened files was archived to .gz ?

 

[UdoB]

I found archived config files.

Great

What happened files was archived to .gz ?

I have no idea. I am using PVE on several machines for several years now and I have never seen something like this happen.

Any chance you did install some third-party scripts on the host?