Ceph rbd error - sysfs write failed

elurex

elurex

Active Member
Proxmox Subscriber
Oct 28, 2015
204
14
38
Taiwan
I am running PVE 6.0 and on this pve node, I have already started 2 vm with rbd disks to my ceph storage, However, sometime if I need to start third VM using rbd disk, PVE will error with following msg. It only can be solve if I reboot the pve node and then I can start 3 VM or more using rbd disk

Code: 
/dev/rbd6
/dev/rbd7
/dev/rbd8
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-104-disk-1: rbd: sysfs write failed
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-104-disk-2: rbd: sysfs write failed
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-104-disk-0: rbd: sysfs write failed
TASK ERROR: start failed: command '/usr/bin/kvm -id 104 -name winclient104 -chardev 'socket,id=qmp,path=/var/run/qemu-server/104.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/104.pid -daemonize -smbios 'type=1,uuid=b9595b11-142c-41b5-bba9-25f359770a91' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/rbd/rbd/vm-104-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' -m 16384 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=b2:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=b2:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=b2:00.2,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2' -device 'vfio-pci,host=b2:00.3,id=hostpci0.3,bus=ich9-pcie-port-1,addr=0x0.3' -chardev 'socket,path=/var/run/qemu-server/104.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cb85eb190f0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/rbd/rbd/vm-104-disk-1,if=none,id=drive-scsi0,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=/dev/rbd/rbd/vm-104-disk-2,if=none,id=drive-scsi1,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3A:C2:94:31:A4:C4,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-3.1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout

Code: 
rbd image 'vm-104-disk-1':
        size 60 GiB in 15360 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 433efbc6fb9b03
        block_name_prefix: rbd_data.433efbc6fb9b03
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Fri Sep  6 01:24:48 2019
        access_timestamp: Fri Sep  6 01:24:48 2019
        modify_timestamp: Fri Sep  6 01:24:48 2019

root@gpu01:~# rados -p rbd listwatchers rbd_data.433efbc6fb9b03
error listing watchers rbd/rbd_data.433efbc6fb9b03: (2) No such file or directory

root@gpu01:~# rbd status -p rbd vm-104-disk-1
Watchers: none

root@gpu01:~# rbd showmapped
id pool namespace image         snap device
0  rbd            vm-101-disk-1 -    /dev/rbd0
1  rbd            vm-101-disk-2 -    /dev/rbd1
2  rbd            vm-101-disk-0 -    /dev/rbd2
3  rbd            vm-102-disk-1 -    /dev/rbd3
4  rbd            vm-102-disk-2 -    /dev/rbd4
5  rbd            vm-102-disk-0 -    /dev/rbd5

root@gpu01:~#  ceph health detail
HEALTH_OK

There is no other rbd client open that image

# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-5.0.21-2-pve: 5.0.21-3
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 14.2.2-pve1
ceph-fuse: 14.2.2-pve1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-8
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2
Click to expand...
 
Last edited:
elurex said:
I am running PVE 6.0 and on this pve node,
Click to expand...
IS this a client or part of a Ceph cluster? If part of PVE + Ceph cluster, can you please post a ceph versions?

elurex said:
root@gpu01:~# rados -p rbd listwatchers rbd_data.433efbc6fb9b03 error listing watchers rbd/rbd_data.433efbc6fb9b03: (2) No such file or directory
Click to expand...

elurex said:
rbd: sysfs write failed can't unmap rbd device /dev/rbd/rbd/vm-104-disk-1: rbd: sysfs write failed
Click to expand...
Is there anything in the logs (journal/syslog, ceph)?
 
Code: 
root@gpu01:~# ceph versions
{
    "mon": {
        "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 3
    },
    "mgr": {
        "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 1
    },
    "osd": {
        "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 96
    },
    "mds": {},
    "overall": {
        "ceph version 14.2.2 (a887fe9a5d3d97fe349065d3c1c9dbd7b8870855) nautilus (stable)": 100
    }
}

syslog

Code: 
root@gpu01:/var/log# cat syslog.1|grep rbd|grep unmap
Oct  2 20:37:38 gpu01-la3 pvedaemon[3351852]: can't unmap rbd device /dev/rbd/rbd/vm-103-disk-1: rbd: sysfs write failed
Oct  2 20:37:38 gpu01-la3 pvedaemon[3351852]: can't unmap rbd device /dev/rbd/rbd/vm-103-disk-2: rbd: sysfs write failed
Oct  2 20:37:39 gpu01-la3 pvedaemon[3351852]: can't unmap rbd device /dev/rbd/rbd/vm-103-disk-0: rbd: sysfs write failed
Oct  2 20:37:39 gpu01-la3 pvedaemon[3351852]: start failed: command '/usr/bin/kvm -id 103 -name winclient103 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=e5190607-223f-4886-a9eb-1650123a9b66' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/rbd/rbd/vm-103-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' -m 16384 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=b2:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=b2:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=b2:00.2,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2' -device 'vfio-pci,host=b2:00.3,id=hostpci0.3,bus=ich9-pcie-port-1,addr=0x0.3' -chardev 'socket,path=/var/run/qemu-server/103.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cb85eb190f0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/rbd/rbd/vm-103-disk-1,if=none,id=drive-scsi0,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=/dev/rbd/rbd/vm-103-disk-2,if=none,id=drive-scsi1,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=3A:C2:94:31:A4:C4,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=sle
w,base=localtime' -machine 'type=pc-q35-3.1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout
Oct  2 20:37:39 gpu01-la3 pvedaemon[508130]: <mquery@pam> end task UPID:gpu01:0033252C:0467553A:5D956CE3:qmstart:103:mquery@pam: start failed: command '/usr/bin/kvm -id 103 -name winclient103 -chardev 'socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/103.pid -daemonize -smbios 'type=1,uuid=e5190607-223f-4886-a9eb-1650123a9b66' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/rbd/rbd/vm-103-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' -m 16384 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=b2:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=b2:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=b2:00.2,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2' -device 'vfio-pci,host=b2:00.3,id=hostpci0.3,bus=ich9-pcie-port-1,addr=0x0.3' -chardev 'socket,path=/var/run/qemu-server/103.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cb85eb190f0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/rbd/rbd/vm-103-disk-1,if=none,id=drive-scsi0,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=/dev/rbd/rbd/vm-103-disk-2,if=none,id=drive-scsi1,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pc
i,mac=3A:C2:94:31:A4:C4,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-3.1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout

Nothing on the ceph log
 
Code: 
rbd image 'vm-104-disk-1':
        size 60 GiB in 15360 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 433efbc6fb9b03
        block_name_prefix: rbd_data.433efbc6fb9b03
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Fri Sep  6 01:24:48 2019
        access_timestamp: Fri Sep  6 01:24:48 2019
        modify_timestamp: Fri Sep  6 01:24:48 2019

provided in first post


spirit said:
also, what is the result of

#rbd info rbd/vm-104-disk-1
Click to expand...
 
currently proxmox disable some features

/usr/share/perl5/PVE/Storage/RBDPlugin.pm

my $krbd_feature_blacklist = ['deep-flatten', 'fast-diff', 'object-map', 'exclusive-lock'];

This was for old kernel on proxmox5, but this has not been updated.

I'm thinking about exclusive-lock (since kernel 4.10), maybe it could help in your case.

#rbd feature enable rbd/vm-104-disk-1 exclusive-lock
 
spirit said:
.....

#rbd feature enable rbd/vm-104-disk-1 exclusive-lock
Click to expand...

Does not work
........
root@gpu05:~# rbd feature enable rbd/vm-104-disk-0 exclusive-lock
root@gpu05:~# rbd feature enable rbd/vm-104-disk-1 exclusive-lock
root@gpu05:~# rbd feature enable rbd/vm-104-disk-2 exclusive-lock

1570504539728.png

Code: 
/dev/rbd9
/dev/rbd10
/dev/rbd11
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-104-disk-1: rbd: sysfs write failed
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-104-disk-2: rbd: sysfs write failed
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/rbd/vm-104-disk-0: rbd: sysfs write failed
TASK ERROR: start failed: command '/usr/bin/kvm -id 104 -name winclient104 -chardev 'socket,id=qmp,path=/var/run/qemu-server/104.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/104.pid -daemonize -smbios 'type=1,uuid=b9595b11-142c-41b5-bba9-25f359770a91' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/rbd/rbd/vm-104-disk-0' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,kvm=off' -m 16384 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa 'node,nodeid=0,cpus=0-7,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=3e:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=3e:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'vfio-pci,host=3e:00.2,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2' -device 'vfio-pci,host=3e:00.3,id=hostpci0.3,bus=ich9-pcie-port-1,addr=0x0.3' -chardev 'socket,path=/var/run/qemu-server/104.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cb85eb190f0' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/rbd/rbd/vm-104-disk-1,if=none,id=drive-scsi0,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -drive 'file=/dev/rbd/rbd/vm-104-disk-2,if=none,id=drive-scsi1,cache=writeback,discard=on,throttling.bps-read=419430400,throttling.iops-read=8000,throttling.bps-write=419430400,throttling.iops-write=8000,format=raw,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1' -netdev 'type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=86:FE:6B:F7:96:F4,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=pc-q35-3.1' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeout
 
Code: 
root@gpu01:~# rbd showmapped
id pool namespace image         snap device
0  rbd            vm-101-disk-1 -    /dev/rbd0
1  rbd            vm-101-disk-2 -    /dev/rbd1
2  rbd            vm-101-disk-0 -    /dev/rbd2
3  rbd            vm-104-disk-1 -    /dev/rbd3
4  rbd            vm-104-disk-2 -    /dev/rbd4
5  rbd            vm-104-disk-0 -    /dev/rbd5
6  rbd            vm-102-disk-1 -    /dev/rbd6
7  rbd            vm-102-disk-2 -    /dev/rbd7
8  rbd            vm-102-disk-0 -    /dev/rbd8

root@gpu01:~# rbd map rbd/vm-103-disk-2
/dev/rbd9

root@gpu01:~# rbd unmap rbd/vm-103-disk-2

This is strange because doing via cli it all works, but when pve trying to start a vm with RBD disk, it will error
 
no, I don't....

Code: 
root@gpu01:/etc/pve/priv/ceph# ls -al
total 0
drwx------ 2 root www-data 0 Aug 12 05:10 .
drwx------ 2 root www-data 0 May  3 05:02 ..

A temporarily work around this issue is to reboot pve host,
then it will stop working again mounting additional rbd disk with sysfs write error after sometimes
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back