VM disk missing after reboot or empty after moving from smb-share (btrfs issues?!)..

[tbox23]

Hello there,

I'm test driving PM for some days, I had 2 weird issues with VM disk image files being gone or suddenly empty. I have a very basic setup - there is only 1 SATA disk attached in the system, everything is running off that disk. I use BTRFS with zstd-compression, since it worked fine for me in the past, no fancy RAID mode/options or something (1 disk setup).

1) After powering down/up via power button, a fresh installed Win7 VM misses a disk/file. The VM disk was placed in a directory "/tmp" I created earlier, because I had to move the disk file from "local-btrfs" to "elsewere" to be able to convert from *.raw to *.qcow2 format (because format option when moving disk is not available when trying in-place convert for some reason). Was it a bad idea to choose "/tmp" for the disk file? Is this directory cleaned up automatically by linux or proxmox somehow?

2) I also tried to use a SMB-Share to move/convert disk image. It was moved and converted successfully onto the SMB-Share, but when moving it back, the process took maybe 5 seconds for 24GB and resulted in the disk file being 0 byte and result "OK" - the disk image was gone.

Can these issues be related to BTRFS? I actually don't expect it to be the problem, but maybe you know better.

Thank you! o)

 

[mira]

How did you use a disk on /tmp for a VM?

Please provide the output of pveversion -v, cat /etc/pve/storage.cfg and qm config <VMID>.

 

[tbox23]

How did you use a disk on /tmp for a VM?

I created "/tmp" directory in Datacenter -> Storage -> Add "Directory".. (as said, I needed another location to get the conversion done).
I am not aware about "/tmp" being a problematic directory, but I will choose another name next time for sure, just to be safe. o)




Please find the requested output below..

Thank you for taking the time! o)

root@pve:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

root@pve:~# cat /etc/pve/storage.cfg
dir: local
        disable
        path /var/lib/vz
        content vztmpl,backup,iso

btrfs: local-btrfs
        path /var/lib/pve/local-btrfs
        content rootdir,iso,vztmpl,images,backup

dir: tmp
        path /tmp
        content images
        prune-backups keep-all=1
        shared 0

cifs: smb10.22.0.1_e
        path /mnt/pve/smb10.22.0.1_e
        server 10.22.0.1
        share E$
        content vztmpl,images,iso,rootdir,snippets,backup
        preallocation off
        prune-backups keep-all=1
        username ******

cifs: smb10.22.0.1_os
        path /mnt/pve/smb10.22.0.1_os
        server 10.22.0.1
        share os
        content snippets,backup,images,vztmpl,iso,rootdir
        preallocation off
        prune-backups keep-all=1
        username *****

root@pve:~# qm config 103
agent: 1,fstrim_cloned_disks=1
balloon: 1536
boot: order=scsi0;ide2
cores: 2
cpu: kvm64,flags=+aes
ide2: local-btrfs:iso/virtio-win-0.1.173.iso,media=cdrom,size=385112K
machine: pc-q35-7.1
memory: 4096
meta: creation-qemu=7.1.0,ctime=1672140597
name: win7u
net0: virtio=2A:86:06:FE:08:15,bridge=vmbr0,firewall=1
numa: 0
ostype: win7
scsi0: tmp:103/vm-103-disk-0.qcow2,cache=writeback,iothread=1,size=24G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=594db9b1-5af5-45ab-b0d1-caa3bb6fd246
sockets: 1
tablet: 0
unused1: local-btrfs:103/vm-103-disk-0.raw
vga: virtio,memory=64
vmgenid: e418067e-47de-4579-b11d-4c824bd185d6

 

[mira]

Yes, /tmp is cleared on reboot. So it's not meant for persistent things.

How did you move/convert the image to the samba share and from it?
Are there any logs still available?

 

[tbox23]

Unfortunately no logs.. (you mean the output in the move-progress dialog I guess?) If not, I'm eager to search elsewhere if I can find more.
In the meantime I tried to reproduce the issue, but this time moving/converting worked as expected two times to and from a smb-share, despite..

The move/convert dialog always forcing conversion to *.raw format, when I want to move the disk image back to it's default "local-btrfs" location. For this target location, the format option is always greyed out and hard coded it seems. It is available when moving to smb or my weird /tmp directory though. Obviously this makes it impossible to get away with the PM GUI if I want the resulting *.qcow2 image to reside in "local-btrfs". Manual move on the console and manual editing of VM configs is required and I wonder why is it like that?

My /tmp directory still makes me wonder, I cannot see a mountpoint or something interfering with tmpfs? (I'm kind of new to linux, please apologize my noobness! o).
EDIT: After researching some more, I did not find what exact mechanic/script/daemon is responsible, but /tmp is cleared very likely on reboot in many distributions, so I will avoid putting disk images there in the future! o) The /tmp folder did NOT exist before, so I felt kind of safe to use it - I learned something new! o)

root@pve:~# df
Filesystem       1K-blocks        Used  Available Use% Mounted on
udev               3928768           0    3928768   0% /dev
tmpfs               792456         924     791532   1% /run
/dev/sda3         49250304    22270992   26578672  46% /
tmpfs              3962268       53040    3909228   2% /dev/shm
tmpfs                 5120           0       5120   0% /run/lock
/dev/fuse           131072          20     131052   1% /etc/pve
//10.22.0.1/os  3221225468  2944146112  277079356  92% /mnt/pve/smb10.22.0.1_os
//10.22.0.1/E$ 13672249340 10915710680 2756538660  80% /mnt/pve/smb10.22.0.1_e
tmpfs               792452           0     792452   0% /run/user/0

Thank you, really appreciate your help!

 

[mira]

Yes, it's not just `tmpfs`. I've updated my post yesterday to remove that part.

Regarding the format, that's interesting since we use BTRFS as a file level storage [0].
Can you specify the qcow2 format when using `qm disk move <VMID> <DISK> <STORAGE> --format qcow2`?
See `man qm` for the complete list of parameters.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_storage_types