[SOLVED] Windows 11 VM with CEPH storage fails to start

[GabrieleV]

Hello,
I've configured a Win 11 Vm as following:

Spoiler: VM CONFIG

agent: 1
bios: ovmf
boot: order=ide0;ide2;net0
cores: 2
efidisk0: CEPH0:vm-108-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: CEPH0:vm-108-disk-1,discard=on,size=80G
ide2: ISO:iso/Win11_22H2_Italian_x64v1.iso,media=cdrom,size=5341118K
machine: pc-q35-6.2
memory: 4096
meta: creation-qemu=6.2.0,ctime=1667986669
name: METADCONSOLE01
net0: virtio=0A:92:A9:66:FA:84,bridge=vmbr0
numa: 0
ostype: win11
scsihw: virtio-scsi-pci
smbios1: uuid=8e87ecc4-5df7-423b-aeb4-024555bc75b0
sockets: 1
tpmstate0: CEPH0:vm-108-disk-2,size=4M,version=v2.0
vmgenid: ca790979-57d2-4b99-ac11-6222ae951c18

Starting up the VM fails with this task log:

Spoiler: TASK LOG

did not load config file, using default settings.
2022-11-09 10:38:56.525 7fd6131d1080 -1 Errors while parsing config file!
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 Errors while parsing config file!
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
/dev/rbd0
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
swtpm_setup: Starting vTPM manufacturing as root:root @ Wed 09 Nov 2022 10:38:56 AM CET
swtpm_setup: TPM is listening on Unix socket.
swtpm_setup: Successfully created RSA 2048 EK with handle 0x81010001.
swtpm_setup: Invoking /usr/bin/swtpm_localca --type ek --ek dcf63101c9b9c1c922f179b44f9195db45da285d08bd9d3a042de373953b5e7c84e35464098ff6a06bac632f495e58bf49d65a95ad1e226a66db2ca650a319115064f206354b92db84e586d93eebf85906f8c861b95d24c16b03146e82c537891fb409711b34b28fa2336fee8e3ab974a5d321d892f5a457a451d98bb1c42d86ae196e4d76786dd12dc5bf32ae3a01608a02ccc115ee2ae14644f54658d3ba34b1b476909578af37ff331a26b80be118f4d4795cc83e13fac745c7074bd0b512d066cde08558bb01c12afc00f3552e6aac6aebd979ff6b7a4053fc69e524675857757dbbdaa04e7798c9bab84c33ce17f80d01d521aabd60d831b1d8775b1787 --dir /tmp/swtpm_setup.certs.H17XU1 --tpm-spec-family 2.0 --tpm-spec-level 0 --tpm-spec-revision 164 --tpm-manufacturer id:00001014 --tpm-model swtpm --tpm-version id:20191023 --tpm2 --configfile /etc/swtpm-localca.conf --optsfile /etc/swtpm-localca.options
swtpm_setup: swtpm_localca: Creating root CA and a local CA's signing key and issuer cert.
swtpm_setup: swtpm_localca: Successfully created EK certificate locally.
swtpm_setup: Invoking /usr/bin/swtpm_localca --type platform --ek dcf63101c9b9c1c922f179b44f9195db45da285d08bd9d3a042de373953b5e7c84e35464098ff6a06bac632f495e58bf49d65a95ad1e226a66db2ca650a319115064f206354b92db84e586d93eebf85906f8c861b95d24c16b03146e82c537891fb409711b34b28fa2336fee8e3ab974a5d321d892f5a457a451d98bb1c42d86ae196e4d76786dd12dc5bf32ae3a01608a02ccc115ee2ae14644f54658d3ba34b1b476909578af37ff331a26b80be118f4d4795cc83e13fac745c7074bd0b512d066cde08558bb01c12afc00f3552e6aac6aebd979ff6b7a4053fc69e524675857757dbbdaa04e7798c9bab84c33ce17f80d01d521aabd60d831b1d8775b1787 --dir /tmp/swtpm_setup.certs.H17XU1 --tpm-spec-family 2.0 --tpm-spec-level 0 --tpm-spec-revision 164 --tpm-manufacturer id:00001014 --tpm-model swtpm --tpm-version id:20191023 --tpm2 --configfile /etc/swtpm-localca.conf --optsfile /etc/swtpm-localca.options
swtpm_setup: swtpm_localca: Successfully created platform certificate locally.
swtpm_setup: Successfully created NVRAM area 0x1c00002 for RSA 2048 EK certificate.
swtpm_setup: Successfully created NVRAM area 0x1c08000 for platform certificate.
swtpm_setup: Successfully created ECC EK with handle 0x81010016.
swtpm_setup: Invoking /usr/bin/swtpm_localca --type ek --ek x=b9130497f7e91c7265ccfc5ae4897afb7ff6cbd699c684e2690284178e6b68eed01c03132514f3511466dbae6af11dd9,y=a09ae8b2046230388cafdba6238ebe8d57aeaabef0e15a1be0e4b104552e7d845900badb607ed096d859723021239e6b,id=secp384r1 --dir /tmp/swtpm_setup.certs.H17XU1 --tpm-spec-family 2.0 --tpm-spec-level 0 --tpm-spec-revision 164 --tpm-manufacturer id:00001014 --tpm-model swtpm --tpm-version id:20191023 --tpm2 --configfile /etc/swtpm-localca.conf --optsfile /etc/swtpm-localca.options
swtpm_setup: swtpm_localca: Successfully created EK certificate locally.
swtpm_setup: Successfully created NVRAM area 0x1c00016 for ECC EK certificate.
swtpm_setup: Successfully activated PCR banks sha256 among sha1,sha256,sha384,sha512.
swtpm_setup: Successfully authored TPM state.
swtpm_setup: Ending vTPM manufacturing @ Wed 09 Nov 2022 10:38:57 AM CET
kvm: -drive if=pflash,unit=1,cache=writeback,format=raw,id=drive-efidisk0,size=540672,file=rbd:CEPH0/vm-108-disk-0:mon_host=10.123.123.181;10.123.123.175;10.123.123.186:auth_supported=cephx:id=admin:keyring=/etc/pve/priv/ceph/CEPH0.keyring:rbd_cache_policy=writeback: invalid conf option rbd_cache_policy: No such file or directory
stopping swtpm instance (pid 2232456) due to QEMU startup error
did not load config file, using default settings.
2022-11-09 10:38:57.937 7fbf94463080 -1 Errors while parsing config file!
2022-11-09 10:38:57.937 7fbf94463080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:57.937 7fbf94463080 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:57.937 7fbf94463080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2022-11-09 10:38:57.937 7fbf94463080 -1 Errors while parsing config file!
2022-11-09 10:38:57.937 7fbf94463080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:57.937 7fbf94463080 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:57.937 7fbf94463080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
TASK ERROR: start failed: QEMU exited with code 1

What's wrong ?

Thank you,
GV

 

[Moayad]

Hello,

2022-11-09 10:38:56.525 7fd6131d1080 -1 Errors while parsing config file!
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 Errors while parsing config file!
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 10:38:56.525 7fd6131d1080 -1 parse_file: cannot open /.ceph/ceph.conf: (2) No such file or directory

Looks like the /etc/ceph/ceph.conf didn't exists.

Does other VM on the same storage "CEPH0" works fine?

 

[GabrieleV]

Does other VM on the same storage "CEPH0" works fine?

Yes, all other VM works fine. The message was always displayed.

 

[Moayad]

Hi,

Yes, all other VM works fine. The message was always displayed.

Do you see the same error when you do rbd -p <pool> ls -l?

Or you can test with the below command:

/usr/bin/rbd -p <pool-name> -m <mon-ips>  --auth_supported cephx -n client.admin --keyring /etc/pve/priv/ceph/<storage>.keyring ls -l --format json | jq

Note: if you don't have `jq` tool, you can get rid of '| jq' from the above command.

Can you please post the output of pveversion -v?

 

[GabrieleV]

Do you see the same error when you do rbd -p <pool> ls -l?

It fails:

Spoiler: rbd -p CEPH0 ls -l

did not load config file, using default settings.
2022-11-09 12:18:19.918 7f5b92962080 -1 Errors while parsing config file!
2022-11-09 12:18:19.918 7f5b92962080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:19.918 7f5b92962080 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:19.918 7f5b92962080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2022-11-09 12:18:19.918 7f5b92962080 -1 Errors while parsing config file!
2022-11-09 12:18:19.918 7f5b92962080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:19.918 7f5b92962080 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:19.918 7f5b92962080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
unable to get monitor info from DNS SRV with service name: ceph-mon
2022-11-09 12:18:19.922 7f5b92962080 -1 failed for service _ceph-mon._tcp
rbd: couldn't connect to the cluster!
2022-11-09 12:18:19.922 7f5b92962080 -1 monclient: get_monmap_and_config cannot identify monitors to contact
rbd: listing images failed: (2) No such file or directory

Or you can test with the below command:

/usr/bin/rbd -p <pool-name> -m <mon-ips>  --auth_supported cephx -n client.admin --keyring /etc/pve/priv/ceph/<storage>.keyring ls -l --format json | jq

It works:

Spoiler: /usr/bin/rbd -p CEPH0 -m 10.123.123.181 --auth_supported cephx -n client.admin --keyring /etc/pve/priv/ceph/CEPH0.keyring ls -l

did not load config file, using default settings.
2022-11-09 12:18:59.092 7fbbdb358080 -1 Errors while parsing config file!
2022-11-09 12:18:59.092 7fbbdb358080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:59.092 7fbbdb358080 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:59.092 7fbbdb358080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2022-11-09 12:18:59.092 7fbbdb358080 -1 Errors while parsing config file!
2022-11-09 12:18:59.092 7fbbdb358080 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:59.092 7fbbdb358080 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory
2022-11-09 12:18:59.092 7fbbdb358080 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
NAME SIZE PARENT FMT PROT LOCK
vm-101-disk-0 50 GiB 2 excl
vm-101-disk-1 300 GiB 2 excl
vm-102-disk-0 80 GiB 2 excl
vm-103-disk-0 32 GiB 2
vm-104-disk-0 80 GiB 2 excl
vm-105-disk-0 400 GiB 2 excl
vm-105-disk-1 120 GiB 2 excl
vm-106-disk-0 32 GiB 2 excl
vm-107-disk-0 512 GiB 2 excl
vm-107-disk-0@pre 512 GiB 2
vm-107-disk-1 80 GiB 2 excl
vm-107-disk-1@pre 80 GiB 2
vm-107-state-preupgrade 12 GiB 2 excl
vm-108-disk-0 1 MiB 2
vm-108-disk-1 80 GiB 2
vm-108-disk-2 4 MiB 2
vm-109-disk-0 160 GiB 2 excl
vm-109-disk-1 80 GiB 2 excl
vm-110-disk-0 512 GiB 2 excl
vm-110-disk-0@OK 512 GiB 2
vm-110-state-iniziale 32 GiB 2 excl
vm-111-disk-0 80 GiB 2 excl
vm-111-disk-1 160 GiB 2 excl
vm-112-disk-0 512 GiB 2 excl
vm-112-disk-1 80 GiB 2 excl
vm-115-disk-0 80 GiB 2
vm-115-disk-0@start 80 GiB 2
vm-116-disk-0 80 GiB 2
vm-116-disk-0@start 80 GiB 2
vm-118-disk-0 32 GiB 2 excl
vm-118-disk-0@start 32 GiB 2
vm-118-disk-0@prebackports 32 GiB 2
vm-119-disk-0 32 GiB 2
vm-119-disk-0@start 32 GiB 2
vm-120-disk-0 32 GiB 2 excl
vm-121-disk-0 32 GiB 2 excl
vm-121-disk-0@prebackports 32 GiB 2
vm-901-disk-0 32 GiB 2
vm-901-disk-1 32 GiB 2

Can you please post the output of pveversion -v?

Here it is:

Spoiler: pveversion -v

proxmox-ve: 7.2-1 (running kernel: 5.15.35-3-pve)
pve-manager: 7.2-5 (running version: 7.2-5/12f1e639)
pve-kernel-5.15: 7.2-5
pve-kernel-helper: 7.2-5
pve-kernel-5.15.35-3-pve: 5.15.35-6
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-5
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-10
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

PS: I tested another VM: moving the EFI disk out of a CEPH pool, let the machine starts.

 

[Moayad]

Hello,

I suppose from the output of pveversion -v that you have an external CEPH right?

PS: I tested another VM: moving the EFI disk out of a CEPH pool, let the machine starts.

Can you try to re-back the EFI disk and do the command qm rescan --vmid 108?



By the way, we recommend using SCSI for more performance than the IDE [0]. You can follow our wiki guide for Windows best practices [1]

[0] https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_hard_disk
[1] https://pve.proxmox.com/wiki/Windows_2022_guest_best_practices

 

[GabrieleV]

I suppose from the output of pveversion -v that you have an external CEPH right?

Yes, it's a cluster of 3 hypervisors only nodes and 3 ceph storage only nodes, all running PVE 7.2

Can you try to re-back the EFI disk and do the command qm rescan --vmid 108?

Done for both EFI and TPM state disks. Same error message.

I think the trouble is with:

kvm: -drive if=pflash,unit=1,cache=writeback,format=raw,id=drive-efidisk0,size=540672,file=rbd:CEPH0/vm-108-disk-0:mon_host=10.123.123.181;10.123.123.175;10.123.123.186:auth_supported=cephx:id=admin:keyring=/etc/pve/priv/ceph/CEPH0.keyring:rbd_cache_policy=writeback: invalid conf option rbd_cache_policy: No such file or directory
stopping swtpm instance (pid 2232456) due to QEMU startup error
did not load config file, using default settings.

By the way, we recommend using SCSI for more performance than the IDE [0]. You can follow our wiki guide for Windows best practices [1]

Thank you, when the machine starts, I will change to the optimum config.

 

[Moayad]

Hello,

I think the trouble is with:

Indeed, I found a similar issue where the upgrade ceph-fuse fix the issue [0]. I would simulate the situation before you update the ceph-fuse, by creating a PVE as a VM on your server with the same version, and adding the RBD storage, at the end create a VM with the same config as 108, if the start VM results the same message above, try upgrading the ceph-fuse and see if that did the trick.

Edit: In order to upgrade the `ceph-fuse` you have to use one of our repositories (it depends on the external Ceph version, e.g. if the external Ceph Quincy version you have to use the deb http://download.proxmox.com/debian/ceph-quincy bullseye main repo) as described in our wiki guide [1].

[0] https://forum.proxmox.com/threads/rbd_cache_policy-error-no-such-file-or-directory.99438/post-429286
[1] https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy#Preparation_on_each_Ceph_Cluster_Node

 

[GabrieleV]

Edit: In order to upgrade the `ceph-fuse` you have to use one of our repositories (it depends on the external Ceph version, e.g. if the external Ceph Quincy version you have to use the deb http://download.proxmox.com/debian/ceph-quincy bullseye main repo) as described in our wiki guide [1].

I have enabled the Proxmox Ceph Pacific apt reposisitory and upgraded the hypervisor.
Now the VM starts correctly.

Thank you !