Disks disappearing

[gardar]

Hello. I have had few occurrences where a vm's disk file (.qcow2) has disappeared without a trace.

Thankfully I do a backup each day so the vm's can easily be restored.



Today I got a vzdump backup status mail that includes this error:

volume 'local:101/vm-101-disk-1.qcow2' does not exist

The disk file has indeed disappered but the vm is still running

 /usr/bin/kvm -id 101 -chardev socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -name test -smp 8,sockets=2,cores=4,maxcpus=8 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga qxl -cpu kvm64,+lahf_lm,+x2apic,+sep -m 16384 -k is -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=DES-CBC3-SHA,seamless-migration=on -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:5b8b5fbeb32f -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/var/lib/vz/images/101/vm-101-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,aio=native,cache=none,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=52:4F:BD:91:45:9B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

I see no information on what has happened to the disk but it seems to be completely gone from the filesystem.

Any ideas why this is happeneng and how I can investigate this?

 

[dietmar]

I see no information on what has happened to the disk but it seems to be completely gone from the filesystem.

What kind of disk subsystem do you use (single disk, md raid, hw raid, zfs, ...)?

 

[gardar]

hw raid 10 (dell perc) and ext4.


The problem looks like it's on file level since it's just the .qcow2 file disappearing

 

[fortechitsolutions]

Do you have any other odd behaviour at filesystem level, things logged in dmesg / system logs ?

Certainly it smells to me more like weirdness at filesystem level and not a KVM-Proxmox issue per se. But the fact the problem is contained just to qcow file(s) is odd. (For that matter - is it always the same QCOW disappearing for the same VM, or is it "any VM Image file, for any VM, and happens more or less randomly across all my VMs on regular basis" ?

presumably disks and raid are healthy, and output from "smartctl" diagnostics show that life is good and disks are happy / not pending fail ?

Definitely a weird thing - I have never heard of a situation quite like this before.

Tim