TPM State problems with CEPH / Proxmox 9.2

[psalkiewicz]

Hi,

Proxmox version 9.2.3, CEPH Tentacle 20.2.1

I've came across an issue with CEPH and TPM State disks. I've created a VM that is located on ceph rbd pool. The VM has an additional 4MB disk with tpm state. The vm doesn't start with following error:

In some cases useful info is found in syslog - try "dmesg | tail".
rbd: sysfs write failed
TASK ERROR: start failed: can't map rbd volume vm-123-disk-2: rbd: sysfs write failed

CEPH is healthy and is running a hundred of other VMs (but with no TPM state).
When I migrate the ceph mini volume to local-lvm storage the VM starts with no issue.

I wonder if I miss something in the ceph config or just came across a bug in newest version?

 

[fiona]

Hi,
are you using dual-stack networking https://forum.proxmox.com/threads/rbd-sysfs-write-failed-on-tpm-disks.131233/ ?

Please share the storage configuration cat /etc/pve/storage.cfg, Ceph config /etc/pve/ceph.conf, VM config qm config 123 and the output of pveversion -v.

 

[psalkiewicz]

Yes, I'm using dual stack - my main network is ipv4 and the network for the ceph cluster is ipv6. This might be the similar issue as in the link you sent above. I'd very grateful if you give me any advice how to configure it better. Dual stack is always some kind of a problem

I have also problem deploying MDS servers for CEPHFS. It looks like it deploys, MDS servers get the ipv6 addresses but when I try to add CephFS at datacenter level it fails with timeout.

When I check the ports on the pve host MDS listens on ipv4 only:

tcp   LISTEN 0      512                  0.0.0.0:6867       0.0.0.0:*    users:(("ceph-mds",pid=907329,fd=21))                                                             
tcp   LISTEN 0      512                  0.0.0.0:6866       0.0.0.0:*    users:(("ceph-mds",pid=907329,fd=20))                                                             
tcp   LISTEN 0      512                        *:6865             *:*    users:(("ceph-mds",pid=907329,fd=19))                                                             
tcp   LISTEN 0      512                        *:6864             *:*    users:(("ceph-mds",pid=907329,fd=18))

ceph.conf

[global]
        auth_client_required = cephx
        auth_cluster_required = cephx
        auth_service_required = cephx
        fsid = 145705a8-549b-408e-8d55-7f7407781833
        mon_allow_pool_delete = true
        mon_host = fd69:beef:cafe::551 fd69:beef:cafe::552 fd69:beef:cafe::553 fd69:beef:cafe::555 fd69:beef:cafe::554
        ms_bind_ipv4 = true
        ms_bind_ipv6 = true
        osd_pool_default_min_size = 2
        osd_pool_default_size = 3

[client]
        keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
        keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.pv1]
        public_addr = fd69:beef:cafe::551

[mon.pv2]
        public_addr = fd69:beef:cafe::552

[mon.pv3]
        public_addr = fd69:beef:cafe::553

[mon.pv4]
        public_addr = fd69:beef:cafe::554

[mon.pv5]
        public_addr = fd69:beef:cafe::555

storage.cfg

dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

rbd: np03
        content rootdir,images
        data-pool np03-data
        krbd 0
        pool np03-metadata