Linux VM on Ceph RDB Backed Disk won't boot

Hello

Could I see the output of lsblock, pvs, lvs and vgs please?

Regards
Philipp
 
Absolutely:

Code: 
root@pveclua:~# lsblk
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                                                                                                     8:0    0  16.4T  0 disk
└─ceph--2b87cc00--81f8--4c5d--b5f1--7214f244ff22-osd--block--b31291c0--33a2--4b74--b544--51d68aaad424 253:0    0  16.4T  0 lvm 
sdb                                                                                                     8:16   0  16.4T  0 disk
└─ceph--bb18bfc4--f008--4504--88dd--4956d61313bf-osd--block--6360eadb--4785--4a3d--8c22--c6b8ea319ccc 253:1    0  16.4T  0 lvm 
sdc                                                                                                     8:32   0   1.8T  0 disk
├─sdc1                                                                                                  8:33   0  1007K  0 part
├─sdc2                                                                                                  8:34   0   512M  0 part
└─sdc3                                                                                                  8:35   0   1.8T  0 part
sdd                                                                                                     8:48   0  16.4T  0 disk
└─ceph--18b7e9d2--0fca--42bd--9d22--2bd0f42ff268-osd--block--1c2aaa3f--6ba3--4a2d--b092--4835a32ee5bc 253:2    0  16.4T  0 lvm 
zd0                                                                                                   230:0    0   256G  0 disk
├─zd0p1                                                                                               230:1    0   255G  0 part
├─zd0p2                                                                                               230:2    0     1K  0 part
└─zd0p5                                                                                               230:5    0   975M  0 part
zd16                                                                                                  230:16   0     4M  0 disk
zd32                                                                                                  230:32   0   630G  0 disk
├─zd32p1                                                                                              230:33   0     1G  0 part
└─zd32p2                                                                                              230:34   0   629G  0 part
zd48                                                                                                  230:48   0   256G  0 disk
├─zd48p1                                                                                              230:49   0   100M  0 part
├─zd48p2                                                                                              230:50   0    16M  0 part
├─zd48p3                                                                                              230:51   0 255.4G  0 part
└─zd48p4                                                                                              230:52   0   524M  0 part
zd64                                                                                                  230:64   0   256G  0 disk
├─zd64p1                                                                                              230:65   0   255G  0 part
├─zd64p2                                                                                              230:66   0     1K  0 part
└─zd64p5                                                                                              230:69   0   975M  0 part
zd80                                                                                                  230:80   0     1M  0 disk
zd96                                                                                                  230:96   0     4M  0 disk
nvme0n1                                                                                               259:0    0   1.8T  0 disk
├─nvme0n1p1                                                                                           259:1    0  1007K  0 part
├─nvme0n1p2                                                                                           259:2    0   512M  0 part
└─nvme0n1p3                                                                                           259:3    0   1.8T  0 part

Code: 
root@pveclua:~# pvs
  PV         VG                                        Fmt  Attr PSize  PFree
  /dev/sda   ceph-2b87cc00-81f8-4c5d-b5f1-7214f244ff22 lvm2 a--  16.37t    0
  /dev/sdb   ceph-bb18bfc4-f008-4504-88dd-4956d61313bf lvm2 a--  16.37t    0
  /dev/sdd   ceph-18b7e9d2-0fca-42bd-9d22-2bd0f42ff268 lvm2 a--  16.37t    0
root@pveclua:~#

Code: 
root@pveclua:~# lvs
  LV                                             VG                                        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-block-1c2aaa3f-6ba3-4a2d-b092-4835a32ee5bc ceph-18b7e9d2-0fca-42bd-9d22-2bd0f42ff268 -wi-ao---- 16.37t                                                   
  osd-block-b31291c0-33a2-4b74-b544-51d68aaad424 ceph-2b87cc00-81f8-4c5d-b5f1-7214f244ff22 -wi-ao---- 16.37t                                                   
  osd-block-6360eadb-4785-4a3d-8c22-c6b8ea319ccc ceph-bb18bfc4-f008-4504-88dd-4956d61313bf -wi-ao---- 16.37t                                                   
root@pveclua:~#
Code: 
root@pveclua:~# vgs
  VG                                        #PV #LV #SN Attr   VSize  VFree
  ceph-18b7e9d2-0fca-42bd-9d22-2bd0f42ff268   1   1   0 wz--n- 16.37t    0
  ceph-2b87cc00-81f8-4c5d-b5f1-7214f244ff22   1   1   0 wz--n- 16.37t    0
  ceph-bb18bfc4-f008-4504-88dd-4956d61313bf   1   1   0 wz--n- 16.37t    0
root@pveclua:~#

https://ss.ecansol.com/uploads/2023/11/09/chrome_2023-11-09_14-16-19.png
https://ss.ecansol.com/uploads/2023/11/09/chrome_2023-11-09_14-16-59.png
 
Hello

Can I see your /etc/pve/storage.cfg? Also, the config file of the effected vm /etc/pve/qemu-server/<vmid>.conf?
 
@Phillip Hufnagl

Code: 
root@pveclua:~# cat /etc/pve/storage.cfg 
dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

cephfs: mainceph
        path /mnt/pve/mainceph
        content vztmpl,iso,backup
        fs-name mainceph

zfspool: SixTBnVME
        pool SixTBnVME
        content rootdir,images
        mountpoint /SixTBnVME
        nodes pveclud

rbd: RDBBlock
        content images,rootdir
        krbd 0
        pool RDBBlock

cephfs: cephfs
        path /mnt/pve/cephfs
        content backup,vztmpl,iso
        fs-name cephfs

rbd: RDBLargeMedia
        content rootdir,images
        krbd 0
        pool RDBLargeMedia

rbd: RDBS3
        content rootdir,images
        krbd 0
        pool RDBS3

rbd: RDBRedMail
        content rootdir,images
        krbd 0
        pool RDBRedMail

and

Code: 
root@pveclua:~# cat /etc/pve/qemu-server/108.conf 
agent: 1
balloon: 0
boot: order=scsi0
cores: 7
cpu: x86-64-v2-AES
hotplug: 0
memory: 65536
meta: creation-qemu=8.0.2,ctime=1699386271
name: RedMailA
net0: virtio=02:72:D1:89:11:52,bridge=vmbr1
numa: 0
onboot: 1
ostype: l26
scsi0: RDBRedMail:vm-108-disk-0,backup=0,discard=on,iothread=1,size=6512G
scsihw: virtio-scsi-single
smbios1: uuid=b5b9b53e-5535-47d2-9ab4-c2577f380b28
sockets: 1
vmgenid: f42ea4a9-40ea-4f94-97e9-0aab42e6f336
 
OHHHH I think this might be a generic Linux issue. For root file systems over a certain size, you have to use LVM instead of standard partition stuff yeah?
 
No, I don't think that is the issue. I do not understand why the disk is not found. Are you sure it's on the storage?


Can you show me rados ls -p RDBRedMail and pveceph status?
 
the first command is a wall of text that continues with more of:
Code: 
root@pveclua:~# rados ls -p RDBRedMail|more
rbd_data.b904aebd3b3049.000000000003ae00
rbd_data.b904aebd3b3049.0000000000000123
rbd_data.b904aebd3b3049.0000000000135e00
rbd_data.b904aebd3b3049.00000000000e5c00
rbd_data.b904aebd3b3049.000000000007d200
rbd_data.b904aebd3b3049.0000000000121e00
rbd_data.b904aebd3b3049.0000000000038000
rbd_data.b904aebd3b3049.00000000000a9e00
rbd_data.b904aebd3b3049.0000000000000fa0
rbd_data.b904aebd3b3049.0000000000107c00
rbd_data.b904aebd3b3049.00000000000ec200
rbd_data.b904aebd3b3049.00000000000f3400
rbd_data.b904aebd3b3049.0000000000110e00
rbd_data.b904aebd3b3049.0000000000009400
rbd_data.b904aebd3b3049.000000000015d200
rbd_data.b904aebd3b3049.0000000000001e60
rbd_data.b904aebd3b3049.000000000014d800
rbd_data.b904aebd3b3049.0000000000015000
rbd_data.b904aebd3b3049.0000000000114c00
rbd_data.b904aebd3b3049.0000000000024000
rbd_data.b904aebd3b3049.0000000000074400
rbd_data.b904aebd3b3049.00000000000de800
rbd_data.b904aebd3b3049.0000000000087e00
rbd_data.b904aebd3b3049.00000000000e8600
rbd_data.b904aebd3b3049.0000000000090e00
rbd_data.b904aebd3b3049.0000000000000109
rbd_data.b904aebd3b3049.00000000000b0c00
rbd_data.b904aebd3b3049.00000000000da800
rbd_data.b904aebd3b3049.0000000000120400
rbd_data.b904aebd3b3049.0000000000095c00
rbd_data.b904aebd3b3049.000000000011ac00
rbd_data.b904aebd3b3049.0000000000137600
rbd_data.b904aebd3b3049.000000000007e600
rbd_data.b904aebd3b3049.00000000000cf600
rbd_data.b904aebd3b3049.00000000000cec00
rbd_data.b904aebd3b3049.00000000000bc000
rbd_data.b904aebd3b3049.0000000000097000
rbd_data.b904aebd3b3049.0000000000041000
rbd_data.b904aebd3b3049.00000000000cb658
rbd_data.b904aebd3b3049.0000000000069200
rbd_data.b904aebd3b3049.0000000000023c00
rbd_data.b904aebd3b3049.000000000013b200
rbd_data.b904aebd3b3049.000000000017bc00
rbd_data.b904aebd3b3049.000000000015f400
rbd_data.b904aebd3b3049.0000000000032e00
rbd_data.b904aebd3b3049.0000000000000022

and

Code: 
root@pveclua:~# pveceph status
  cluster:
    id:     88cf2bbe-63d5-4fa1-80b2-96a59c7825e3
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum pveclua,pveclue,pvecluf (age 7h)
    mgr: pveclue(active, since 7h), standbys: pveclua, pvecluf
    mds: 1/1 daemons up, 2 standby
    osd: 30 osds: 30 up (since 6h), 30 in (since 11d)
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 833 pgs
    objects: 12.50M objects, 47 TiB
    usage:   140 TiB used, 247 TiB / 387 TiB avail
    pgs:     832 active+clean
             1   active+clean+scrubbing+deep
 
  io:
    client:   0 B/s rd, 1.1 KiB/s wr, 0 op/s rd, 0 op/s wr
 
ErkDog said:
OHHHH I think this might be a generic Linux issue. For root file systems over a certain size, you have to use LVM instead of standard partition stuff yeah?
Click to expand...
I think you're right: https://askubuntu.com/questions/867047/grub-attempt-to-read-or-write-outside-of-disk-hd0

Was the VM running before or did you reinstall it?

I would always recommend making a small OS disk and putting data on a second one. Then you always have them cleanly separated and can change the data disk at any time without having to laboriously install the system or having to copy.
 
It's a new VM I'm trying to setup. I'll redo the install using LVM and see if that fixes, will let everyone know.
 
  • Like
Reactions: sb-jw
You must log in or register to reply here.
Top Bottom