Linux VM on Ceph RDB Backed Disk won't boot

Hello

Could I see the output of lsblock, pvs, lvs and vgs please?

Regards
Philipp
 
Absolutely:

Code: 
root@pveclua:~# lsblk
NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                                                                                                     8:0    0  16.4T  0 disk
└─ceph--2b87cc00--81f8--4c5d--b5f1--7214f244ff22-osd--block--b31291c0--33a2--4b74--b544--51d68aaad424 253:0    0  16.4T  0 lvm 
sdb                                                                                                     8:16   0  16.4T  0 disk
└─ceph--bb18bfc4--f008--4504--88dd--4956d61313bf-osd--block--6360eadb--4785--4a3d--8c22--c6b8ea319ccc 253:1    0  16.4T  0 lvm 
sdc                                                                                                     8:32   0   1.8T  0 disk
├─sdc1                                                                                                  8:33   0  1007K  0 part
├─sdc2                                                                                                  8:34   0   512M  0 part
└─sdc3                                                                                                  8:35   0   1.8T  0 part
sdd                                                                                                     8:48   0  16.4T  0 disk
└─ceph--18b7e9d2--0fca--42bd--9d22--2bd0f42ff268-osd--block--1c2aaa3f--6ba3--4a2d--b092--4835a32ee5bc 253:2    0  16.4T  0 lvm 
zd0                                                                                                   230:0    0   256G  0 disk
├─zd0p1                                                                                               230:1    0   255G  0 part
├─zd0p2                                                                                               230:2    0     1K  0 part
└─zd0p5                                                                                               230:5    0   975M  0 part
zd16                                                                                                  230:16   0     4M  0 disk
zd32                                                                                                  230:32   0   630G  0 disk
├─zd32p1                                                                                              230:33   0     1G  0 part
└─zd32p2                                                                                              230:34   0   629G  0 part
zd48                                                                                                  230:48   0   256G  0 disk
├─zd48p1                                                                                              230:49   0   100M  0 part
├─zd48p2                                                                                              230:50   0    16M  0 part
├─zd48p3                                                                                              230:51   0 255.4G  0 part
└─zd48p4                                                                                              230:52   0   524M  0 part
zd64                                                                                                  230:64   0   256G  0 disk
├─zd64p1                                                                                              230:65   0   255G  0 part
├─zd64p2                                                                                              230:66   0     1K  0 part
└─zd64p5                                                                                              230:69   0   975M  0 part
zd80                                                                                                  230:80   0     1M  0 disk
zd96                                                                                                  230:96   0     4M  0 disk
nvme0n1                                                                                               259:0    0   1.8T  0 disk
├─nvme0n1p1                                                                                           259:1    0  1007K  0 part
├─nvme0n1p2                                                                                           259:2    0   512M  0 part
└─nvme0n1p3                                                                                           259:3    0   1.8T  0 part

Code: 
root@pveclua:~# pvs
  PV         VG                                        Fmt  Attr PSize  PFree
  /dev/sda   ceph-2b87cc00-81f8-4c5d-b5f1-7214f244ff22 lvm2 a--  16.37t    0
  /dev/sdb   ceph-bb18bfc4-f008-4504-88dd-4956d61313bf lvm2 a--  16.37t    0
  /dev/sdd   ceph-18b7e9d2-0fca-42bd-9d22-2bd0f42ff268 lvm2 a--  16.37t    0
root@pveclua:~#

Code: 
root@pveclua:~# lvs
  LV                                             VG                                        Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  osd-block-1c2aaa3f-6ba3-4a2d-b092-4835a32ee5bc ceph-18b7e9d2-0fca-42bd-9d22-2bd0f42ff268 -wi-ao---- 16.37t                                                   
  osd-block-b31291c0-33a2-4b74-b544-51d68aaad424 ceph-2b87cc00-81f8-4c5d-b5f1-7214f244ff22 -wi-ao---- 16.37t                                                   
  osd-block-6360eadb-4785-4a3d-8c22-c6b8ea319ccc ceph-bb18bfc4-f008-4504-88dd-4956d61313bf -wi-ao---- 16.37t                                                   
root@pveclua:~#
Code: 
root@pveclua:~# vgs
  VG                                        #PV #LV #SN Attr   VSize  VFree
  ceph-18b7e9d2-0fca-42bd-9d22-2bd0f42ff268   1   1   0 wz--n- 16.37t    0
  ceph-2b87cc00-81f8-4c5d-b5f1-7214f244ff22   1   1   0 wz--n- 16.37t    0
  ceph-bb18bfc4-f008-4504-88dd-4956d61313bf   1   1   0 wz--n- 16.37t    0
root@pveclua:~#

https://ss.ecansol.com/uploads/2023/11/09/chrome_2023-11-09_14-16-19.png
https://ss.ecansol.com/uploads/2023/11/09/chrome_2023-11-09_14-16-59.png
 
Hello

Can I see your /etc/pve/storage.cfg? Also, the config file of the effected vm /etc/pve/qemu-server/<vmid>.conf?
 
@Phillip Hufnagl

Code: 
root@pveclua:~# cat /etc/pve/storage.cfg 
dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

cephfs: mainceph
        path /mnt/pve/mainceph
        content vztmpl,iso,backup
        fs-name mainceph

zfspool: SixTBnVME
        pool SixTBnVME
        content rootdir,images
        mountpoint /SixTBnVME
        nodes pveclud

rbd: RDBBlock
        content images,rootdir
        krbd 0
        pool RDBBlock

cephfs: cephfs
        path /mnt/pve/cephfs
        content backup,vztmpl,iso
        fs-name cephfs

rbd: RDBLargeMedia
        content rootdir,images
        krbd 0
        pool RDBLargeMedia

rbd: RDBS3
        content rootdir,images
        krbd 0
        pool RDBS3

rbd: RDBRedMail
        content rootdir,images
        krbd 0
        pool RDBRedMail

and

Code: 
root@pveclua:~# cat /etc/pve/qemu-server/108.conf 
agent: 1
balloon: 0
boot: order=scsi0
cores: 7
cpu: x86-64-v2-AES
hotplug: 0
memory: 65536
meta: creation-qemu=8.0.2,ctime=1699386271
name: RedMailA
net0: virtio=02:72:D1:89:11:52,bridge=vmbr1
numa: 0
onboot: 1
ostype: l26
scsi0: RDBRedMail:vm-108-disk-0,backup=0,discard=on,iothread=1,size=6512G
scsihw: virtio-scsi-single
smbios1: uuid=b5b9b53e-5535-47d2-9ab4-c2577f380b28
sockets: 1
vmgenid: f42ea4a9-40ea-4f94-97e9-0aab42e6f336
 
OHHHH I think this might be a generic Linux issue. For root file systems over a certain size, you have to use LVM instead of standard partition stuff yeah?
 
No, I don't think that is the issue. I do not understand why the disk is not found. Are you sure it's on the storage?


Can you show me rados ls -p RDBRedMail and pveceph status?
 
the first command is a wall of text that continues with more of:
Code: 
root@pveclua:~# rados ls -p RDBRedMail|more
rbd_data.b904aebd3b3049.000000000003ae00
rbd_data.b904aebd3b3049.0000000000000123
rbd_data.b904aebd3b3049.0000000000135e00
rbd_data.b904aebd3b3049.00000000000e5c00
rbd_data.b904aebd3b3049.000000000007d200
rbd_data.b904aebd3b3049.0000000000121e00
rbd_data.b904aebd3b3049.0000000000038000
rbd_data.b904aebd3b3049.00000000000a9e00
rbd_data.b904aebd3b3049.0000000000000fa0
rbd_data.b904aebd3b3049.0000000000107c00
rbd_data.b904aebd3b3049.00000000000ec200
rbd_data.b904aebd3b3049.00000000000f3400
rbd_data.b904aebd3b3049.0000000000110e00
rbd_data.b904aebd3b3049.0000000000009400
rbd_data.b904aebd3b3049.000000000015d200
rbd_data.b904aebd3b3049.0000000000001e60
rbd_data.b904aebd3b3049.000000000014d800
rbd_data.b904aebd3b3049.0000000000015000
rbd_data.b904aebd3b3049.0000000000114c00
rbd_data.b904aebd3b3049.0000000000024000
rbd_data.b904aebd3b3049.0000000000074400
rbd_data.b904aebd3b3049.00000000000de800
rbd_data.b904aebd3b3049.0000000000087e00
rbd_data.b904aebd3b3049.00000000000e8600
rbd_data.b904aebd3b3049.0000000000090e00
rbd_data.b904aebd3b3049.0000000000000109
rbd_data.b904aebd3b3049.00000000000b0c00
rbd_data.b904aebd3b3049.00000000000da800
rbd_data.b904aebd3b3049.0000000000120400
rbd_data.b904aebd3b3049.0000000000095c00
rbd_data.b904aebd3b3049.000000000011ac00
rbd_data.b904aebd3b3049.0000000000137600
rbd_data.b904aebd3b3049.000000000007e600
rbd_data.b904aebd3b3049.00000000000cf600
rbd_data.b904aebd3b3049.00000000000cec00
rbd_data.b904aebd3b3049.00000000000bc000
rbd_data.b904aebd3b3049.0000000000097000
rbd_data.b904aebd3b3049.0000000000041000
rbd_data.b904aebd3b3049.00000000000cb658
rbd_data.b904aebd3b3049.0000000000069200
rbd_data.b904aebd3b3049.0000000000023c00
rbd_data.b904aebd3b3049.000000000013b200
rbd_data.b904aebd3b3049.000000000017bc00
rbd_data.b904aebd3b3049.000000000015f400
rbd_data.b904aebd3b3049.0000000000032e00
rbd_data.b904aebd3b3049.0000000000000022

and

Code: 
root@pveclua:~# pveceph status
  cluster:
    id:     88cf2bbe-63d5-4fa1-80b2-96a59c7825e3
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum pveclua,pveclue,pvecluf (age 7h)
    mgr: pveclue(active, since 7h), standbys: pveclua, pvecluf
    mds: 1/1 daemons up, 2 standby
    osd: 30 osds: 30 up (since 6h), 30 in (since 11d)
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 833 pgs
    objects: 12.50M objects, 47 TiB
    usage:   140 TiB used, 247 TiB / 387 TiB avail
    pgs:     832 active+clean
             1   active+clean+scrubbing+deep
 
  io:
    client:   0 B/s rd, 1.1 KiB/s wr, 0 op/s rd, 0 op/s wr
 
ErkDog said:
OHHHH I think this might be a generic Linux issue. For root file systems over a certain size, you have to use LVM instead of standard partition stuff yeah?
Click to expand...
I think you're right: https://askubuntu.com/questions/867047/grub-attempt-to-read-or-write-outside-of-disk-hd0

Was the VM running before or did you reinstall it?

I would always recommend making a small OS disk and putting data on a second one. Then you always have them cleanly separated and can change the data disk at any time without having to laboriously install the system or having to copy.
 
It's a new VM I'm trying to setup. I'll redo the install using LVM and see if that fixes, will let everyone know.
 
  • Like
Reactions: sb-jw
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom