losted Data on VM Disks

N

Neb

Well-Known Member
Apr 27, 2017
35
0
46
29
Hello !

I have a proxmox cluster with 3 machines on which are installed Ceph.
So i have configured one ceph pool with 3 osds, one on each nodes. After i have created a rbd storage over ceph pool. I have created about ten virtual machines which use this rbd storage for their disks.
All works fine until this morning.

At one point, all my virtual machines began to restart without apparent reasons. But since, data on their disks have been removed. Now, there is no bootable disks when I try to start my virtual machines. I don't why and I don't how this happened.

Please, there is a way to recover or prevent this problem ? Somebody has a similar problem ?

This is very uncomfortable because I planned to use this in production environment but this kind of thing is very annoying ...

Thank you !

Some outputs from command lines :

Code: 
root@be1proxmox1:~# cat /etc/pve/storage.cfg
rbd: ceph-vm
    monhost 10.53.100.211 10.53.100.212 10.53.100.213
    content images
    krbd 0
    pool ceph-vm

dir: local
    path /var/lib/vz
    content images,rootdir,vztmpl,iso
    maxfiles 0


Code: 
root@be1proxmox1:~# rbd ls ceph-vm
vm-100-disk-1
vm-101-disk-1
vm-102-disk-1
vm-103-disk-1
vm-104-disk-1
vm-105-disk-1
vm-106-disk-2

Code: 
root@be1proxmox1:~# ceph -s
  cluster:
    id:     4e1fa8c2-ed36-46a4-9830-04d5575cc97f
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum be1proxmox1,be1proxmox2,be1proxmox3
    mgr: be1proxmox1(active), standbys: be1proxmox3, be1proxmox2
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   1 pools, 128 pgs
    objects: 31480 objects, 122 GB
    usage:   368 GB used, 4389 GB / 4758 GB avail
    pgs:     128 active+clean
 
  io:
    client:   810 kB/s wr, 0 op/s rd, 102 op/s wr

This next is suspicious ?

Code: 
Sep 18 12:13:11 be1proxmox1 sh[3227]: mount: Mounting /dev/disk/by-partuuid/80cb587b-94c6-47c7-b22a-855859bec57d on /var/lib/ceph/tmp/mnt.sjeO6z with options noatime,inode64
Sep 18 12:13:11 be1proxmox1 sh[3227]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/disk/by-partuuid/80cb587b-94c6-47c7-b22a-855859bec57d /var/lib/ceph/tmp/mnt.sjeO6z
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/sbin/ceph-disk", line 11, in <module>
Sep 18 12:13:11 be1proxmox1 sh[3227]:     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5699, in run
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5650, in main
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5400, in <lambda>
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4141, in main_activate_space
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3494, in mount_activate
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 1446, in mount
Sep 18 12:13:11 be1proxmox1 sh[3227]: ceph_disk.main.MountError: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/disk/by-partuuid/80cb587b-94c6-47c7-b22a-855859bec57d', '/var/lib/ceph/tmp/mnt.sjeO6z']' returned non-zero exit status 32
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/sbin/ceph-disk", line 11, in <module>
Sep 18 12:13:11 be1proxmox1 sh[3227]:     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5699, in run
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5650, in main
Sep 18 12:13:11 be1proxmox1 sh[3227]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4853, in main_trigger
Sep 18 12:13:11 be1proxmox1 sh[3227]: ceph_disk.main.Error: Error: return code 1
Sep 18 12:13:11 be1proxmox1 systemd[1]: ceph-disk@dev-sdc2.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 12:13:11 be1proxmox1 systemd[1]: ceph-disk@dev-sdc2.service: Unit entered failed state.
Sep 18 12:13:11 be1proxmox1 systemd[1]: ceph-disk@dev-sdc2.service: Failed with result 'exit-code'.
Sep 18 12:13:11 be1proxmox1 sh[3221]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
Sep 18 12:13:11 be1proxmox1 sh[3221]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
Sep 18 12:13:11 be1proxmox1 sh[3221]: mount: Mounting /dev/sdc1 on /var/lib/ceph/tmp/mnt.f5aEho with options noatime,inode64
Sep 18 12:13:11 be1proxmox1 sh[3221]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sdc1 /var/lib/ceph/tmp/mnt.f5aEho
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/sbin/ceph-disk", line 11, in <module>
Sep 18 12:13:11 be1proxmox1 sh[3221]:     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5699, in run
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5650, in main
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3754, in main_activate
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3494, in mount_activate
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 1446, in mount
Sep 18 12:13:11 be1proxmox1 sh[3221]: ceph_disk.main.MountError: Mounting filesystem failed: Command '['/bin/mount', '-t', u'xfs', '-o', 'noatime,inode64', '--', '/dev/sdc1', '/var/lib/ceph/tmp/mnt.f5aEho']' returned non-zero exit status 32
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/sbin/ceph-disk", line 11, in <module>
Sep 18 12:13:11 be1proxmox1 sh[3221]:     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5699, in run
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5650, in main
Sep 18 12:13:11 be1proxmox1 sh[3221]:   File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4853, in main_trigger
Sep 18 12:13:11 be1proxmox1 sh[3221]: ceph_disk.main.Error: Error: return code 1
Sep 18 12:13:11 be1proxmox1 systemd[1]: ceph-disk@dev-sdc1.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 12:13:11 be1proxmox1 systemd[1]: ceph-disk@dev-sdc1.service: Unit entered failed state.
Sep 18 12:13:11 be1proxmox1 systemd[1]: ceph-disk@dev-sdc1.service: Failed with result 'exit-code'.

Versions :

Code: 
root@be1proxmox1:~# pveversion --verbose
proxmox-ve: 5.0-20 (running kernel: 4.10.17-2-pve)
pve-manager: 5.0-30 (running version: 5.0-30/5ab26bc)
pve-kernel-4.10.17-2-pve: 4.10.17-20
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-14
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.0-4
pve-container: 2.0-15
pve-firewall: 3.0-2
pve-ha-manager: 2.0-2
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
ceph: 12.1.2-pve1

Tell me if you need more details
 
Hi. I have disable pve-enterprise.list because I use my own local package repository. I can't proceed in an other way because I use proxmox inside a closed network. But, Proxmox seems to work fine before.
I have restarted a new install of proxmox but the issue appears a new time.
No alerts from cluster, and now all packages are updated from my local repository.

Code: 
root@be1proxmox2:~# pveversion --verbose
proxmox-ve: 5.0-20 (running kernel: 4.10.17-2-pve)
pve-manager: 5.0-31 (running version: 5.0-31/27769b1f)
pve-kernel-4.10.17-2-pve: 4.10.17-20
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-14
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.0-5
pve-container: 2.0-15
pve-firewall: 3.0-2
pve-ha-manager: 2.0-2
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
ceph: 12.1.2-pve1

Code: 
root@be1proxmox1:~# ceph osd dump
epoch 126
fsid 1cfcc937-020c-43b9-abc6-0c69f8bc0445
created 2017-09-19 12:09:55.001033
modified 2017-09-20 18:27:37.258377
flags sortbitwise,recovery_deletes
crush_version 7
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
pool 1 'ceph-vm' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 27 flags hashpspool stripe_width 0 application rbd
    removed_snaps [1~3]
max_osd 3
osd.0 up   in  weight 1 up_from 32 up_thru 124 down_at 28 last_clean_interval [5,27) 10.53.100.213:6800/3226 10.53.100.213:6801/3226 10.53.100.213:6802/3226 10.53.100.213:6803/3226 exists,up a6583c2e-e365-4357-b8d2-a63756abf905
osd.1 up   in  weight 1 up_from 115 up_thru 124 down_at 113 last_clean_interval [110,112) 10.53.100.212:6800/2884 10.53.100.212:6801/2884 10.53.100.212:6802/2884 10.53.100.212:6803/2884 exists,up 0b2344e5-0394-4102-a8dd-568558ccc9bb
osd.2 up   in  weight 1 up_from 124 up_thru 124 down_at 122 last_clean_interval [119,121) 10.53.100.211:6800/2703 10.53.100.211:6801/2703 10.53.100.211:6802/2703 10.53.100.211:6803/2703 exists,up 3297aeff-945f-49f3-85fe-3b81e9b2cc20
 
There needs to be something more in the logs, besides ceph not being able to mount drives. Some hardware issues in kernel.log maybe?
 
Thank you for help

In logs, i can see this :

Code: 
Sep 20 18:27:16 be1proxmox3 ceph-mgr[15594]: 2017-09-20 18:27:16.595234 7fce301c3700 -1 mgr handle_mgr_map I was active but no longer am
Sep 20 18:27:16 be1proxmox3 ceph-mgr[15594]: global_init: error reading config file.
Sep 20 18:27:16 be1proxmox3 systemd[1]: ceph-mgr@be1proxmox3.service: Main process exited, code=exited, status=1/FAILURE
Sep 20 18:27:16 be1proxmox3 systemd[1]: ceph-mgr@be1proxmox3.service: Unit entered failed state.
Sep 20 18:27:16 be1proxmox3 systemd[1]: ceph-mgr@be1proxmox3.service: Failed with result 'exit-code'.
Sep 20 18:27:16 be1proxmox3 systemd[1]: ceph-mgr@be1proxmox3.service: Service hold-off time over, scheduling restart.
Sep 21 00:17:01 be1proxmox3 ceph-mon[1699]: 2017-09-21 00:17:01.312134 7f13b6134700 -1 received  signal: Hangup from  PID: 19775 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Sep 21 00:17:01 be1proxmox3 ceph-osd[3226]: 2017-09-21 00:17:01.312629 7fcf7b0b2700 -1 received  signal: Hangup from  PID: 19775 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Sep 21 00:17:01 be1proxmox3 ceph-mgr[25243]: 2017-09-21 00:17:01.313727 7feab9519700 -1 received  signal: Hangup from  PID: 19775 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0

Yesterday at 6 pm, all works fine. And this morning (in Europe) VM does not detect bootable disk.
I try to understand this problem in order to avoid it, but I don't know why. I'm continuing to search ..
Ceph Luminous is not the current stable release too ?

Moreover, there is several time these lines from logs :

Code: 
Sep 21 10:43:00 be1proxmox3 systemd[1]: Starting Proxmox VE replication runner...
Sep 21 10:43:00 be1proxmox3 systemd[1]: Started Proxmox VE replication runner.
Sep 21 10:44:00 be1proxmox3 systemd[1]: Starting Proxmox VE replication runner...
Sep 21 10:44:00 be1proxmox3 systemd[1]: Started Proxmox VE replication runner.
Sep 21 10:45:00 be1proxmox3 systemd[1]: Starting Proxmox VE replication runner...
Sep 21 10:45:00 be1proxmox3 systemd[1]: Started Proxmox VE replication runner.
Sep 21 10:46:00 be1proxmox3 systemd[1]: Starting Proxmox VE replication runner...
Sep 21 10:46:00 be1proxmox3 systemd[1]: Started Proxmox VE replication runner.
Sep 21 10:47:00 be1proxmox3 systemd[1]: Starting Proxmox VE replication runner...
Sep 21 10:47:00 be1proxmox3 systemd[1]: Started Proxmox VE replication runner.

Code: 
root@be1proxmox1:~# rbd -p ceph-vm disk-usage
NAME          PROVISIONED   USED
vm-100-disk-1     102400M  1828M
vm-101-disk-1        200G      0
vm-101-disk-2        200G 10580M
vm-102-disk-1        200G   588M
vm-103-disk-1      32768M 10716M
<TOTAL>              732G 23712M
 
Last edited:
Ceph Luminous 12.1.x are RC. http://docs.ceph.com/docs/master/releases/ We have newer packages for ceph in the repository. Please try do update your PVE servers, so we can rule out any bugs that have been fixed already.

Does this happen on all nodes on the cluster?
How does your 'pvecm status' look like, when it happens?
 
It's ok, I have updated Ceph v12.1 to the LTS Ceph Release 12.2.0 now.

the output of pvecm status :

Code: 
root@be1proxmox1:~# pvecm status
Quorum information
------------------
Date:             Thu Sep 21 11:33:11 2017
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000001
Ring ID:          1/228
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.53.100.211 (local)
0x00000002          1 10.53.100.212
0x00000003          1 10.53.100.213

Is what I need to remove my ceph pool and my rbd storage over this and create a new ceph pool after to have updated Ceph ? Anyway, I have losted all data from vm disks (no bootable disk found when i start virtual machines).
 
You don't need to recreate the pool on update. There are 122GB on your ceph cluster, these are still there. I might just be that your VM disks have a corrupt boot sector/partition. Depends on how you setup your VM (eg. Bus/Device), VirtIO SCSI disk with a VirtIO SCSI controller is more robust, on none responsive storage issues, then using IDE and LSI controller.
 
  • Like
Reactions: Neb
I use SATA Bus/Device but maybe I have to change to VirtIO, (better performance ?). However, it is worrying to know that it is possible to corrupt VM disks after to have installed an OS as guest on host. I really need to know how this problem is appeared to prevent this in production environment..
 
Neb said:
I use SATA Bus/Device but maybe I have to change to VirtIO, (better performance ?).
Click to expand...
Yes, VirtIO SCSI has better performance.

Neb said:
However, it is worrying to know that it is possible to corrupt VM disks after to have installed an OS as guest on host.
Click to expand...
If on all your nodes, ceph stops working then no VM can access their disks. For the OS inside the VM it is like a disk failure.

Neb said:
I really need to know how this problem is appeared to prevent this in production environment..
Click to expand...
On "be1proxmox3" all ceph services have been killed. You need to find out why.
 
Yes at the first time, but this morning (for me - I don't know where you are ) when this problem appeared for the second time, Ceph worked without interruption. I migrated VMs and then I get the result "no bootable disk". I'm rummaging through log files
 
So there have been two problems? How does the config of one of your VMs look like (qm config <vmid>)?
 
  • Like
Reactions: Neb
This is the same problem. It have already act last Monday. Ceph was in error because there was conflicts with an other block device. Since I have resolved conflicts for sur. But this morning I can't see ceph error for the moment.

Code: 
root@be1proxmox1:~# qm config 103
bootdisk: sata0
cores: 4
ide2: none,media=cdrom
memory: 32000
name: be1mgmtdb1
net0: virtio=5E:18:CA:0B:7A:BC,bridge=vmbr0
numa: 0
ostype: l26
sata0: ceph-vm:vm-103-disk-1,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=08ad7968-b177-41e4-b4ee-ba357e39dd2e
sockets: 1
unused1: ceph-vm:vm-103-disk-2

The disks that I use for ceph is block devices on each proxmox machines which correspond at different LUN in a SAN.

I see in logs proxmox logs :
Code: 
Sep 20 17:07:01 be1proxmox2 kernel: [   44.191757] sd 2:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
Sep 21 10:29:59 be1proxmox2 kernel: [   35.971104] sd 2:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical

Can that be a relationship ?
 
Last edited:
  • Like
Reactions: Neb
It was flickering network. For the moment, I'm testing proxmox in our testing environment. I will reinstall ceph and proxmox on direct attached storage with one 300GB SAS disks in additional on each server for testing in this configuration. In production, we have planned to use direct attached storage in the futur anyway.

Actually, Proxmox is installed on RAID level 1 hardware. I need to break this RAID in order to have two disks (300GB each one) DAS. It's for this reason that I need to reinstall Proxmox.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back