[SOLVED] I don't know if it's a bug, but my disk is gone

[fever_wits]

Hello.
I don't know if what I'm going to describe is a bug.
I have 2 hypervisors. Server1 and Server2.
On server 1, the disk array in which the images of the virtual servers are stored is shared via nfs with server 2.
Server 2 has its own local storage
Virtual machine numbers start at 100
Virtual machine numbers start at 1000 - they are entered manually. The reason for this is that sometimes virtual servers move from server 1 to server 2.
What happened to me is this:
On server 2, I mistakenly created a virtual server number 100 in the local storage. After realizing the error, I deleted the virtual server as well as the disk to it through the web administration. As a result, the disk and the folder on server 1 disappeared.
Fortunately, the virtual server number 100 on server 1 was still working and I was able to retrieve the information.

Proxmox version on server 1
# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.65-1-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Proxmox version on server2
pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

Best Regards,
Aleksandr Hristov

 

[mailinglists]

It is not a bug, but expected behavior. You told PM to delete VM with it's disk and it did just that.
All data was actually deleted only after the running VM was shut down, because it still had file pointer open to the disk file, allowing you to transfer data off.

 

[fever_wits]

Hello,
Maybe I didn't explain exactly.
Server 1 and server 2 are not in a cluster.
I am deleting a VM100 from server 2.
The VM100 disk is located in /mnt/SSD/images/.
The disk is deleted from /mnt/SSD/images/ but it is also deleted from the mount /mnt/pve/nfs-server1/images/
In mount /mnt/pve/nfs-server1/images/ there is a disk on VM100, which runs on server 1.
In fact, my problem is that 2 disks in two different mounts on one server have been deleted

I don't understand why it erases disk from mount /mnt/pve/nfs-server1/images/?

 

[apoc]

In fact, my problem is that 2 disks in two different mounts on one server have been deleted

It doesnt matter if it is mounted differently if it is the same source. It reads different but how is it underneath?
Any chance that it is pointing to the same storage?

I'd try to repro. Create a VM with a high ID 4444 for instance. The try repeating your steps. This should give an indication if and what is going on...

 

[fever_wits]

On server1 I created a VM with number 4444
VM 4444 is visible in stor1

On server2 I also see VM 4444 in nfs_stor (created on server 1) and mounted on nfs from server1 to server2
On server2 I created a VM with 4444 on stor2
I currently have 2 disks named vm-4444-disk-0.qcow2
One on stor1 and one on stor2
I am deleting VM 4444 from server2.
Disk vm-4444-disk-0.qcow2 on stor2 is missing, but disk vm-4444-disk-0.qcow2 on stor1 is also missing.
I deleted one VM, but deleted 2 disks on 2 arrays.

I hope I was able to explain correctly?
My English is not very good.


I have to attach a simple diagram of disk arrays and servers

 

[fiona]

Hi,
glad to hear that you were able to recover. When deleting a VM PVE currently scans for disks with the same ID on all storages and deletes them too (it's planned to change this for the next major release to avoid such surprises in the future). When you deleted the VM with ID 100 on Server 2, PVE also found the disk with ID 100 on nfs_stor1 and thought "it must belong to this guest, so I should delete it". It couldn't know that there is another guest on another cluster that this disk actually belonged to.

 

[fever_wits]

Hello,

Thank you for the information.

I'm glad this will be fixed.

Best Regards,
Aleksandr Hristov

 

[apoc]

When deleting a VM PVE currently scans for disks with the same ID on all storages and deletes them too

Uh. Thats dangerous...

it's planned to change this for the next major release to avoid such surprises in the future

Yes please.