[Feature request] Ability to enable rxbounce option in Ceph RBD storage for windows VMs

[itNGO]

What is a fast PBS in your words? NVMe / SSD only storage with sufficient bandwidth eg. 40G / 25G / 100G ?

10GBe to dedicated PBS with Kioxa SSDs in ZFS.

 

[fluxX04]

Also testing this and looks good so far - no more log spamming and performance on windows increased.

 

[itNGO]

Quick Update... we made the change in the file on about 10 3 Node Clusters now, all at all about 400 Windows VMs and some Linux VMs and Containers on these Hosts.... all looking stable and the Windows Guests benefit from the change, also the LinuxVMs and LXC did not show any draw-backs.... so I am still hoping this gets a "default" option or we will have to apply some guide for our default-setups to change this every time we install updates.

 

[daros]

Just to be sure, you need to update the file on the node. and then all the vm's use the rxbounce option?
And do you need to start stop the vm or only the restart the pvedaemon?

 

[mira]

The disk has to be mapped. This usually happens on VM start. So if you have the chance to live migrate between hosts, you shouldn't need to shutdown and start the VM. Instead simply move it between hosts.

 

[LnxBil]

First series of patches were applied.

 

[fweber]

I've sent a first version of a patch addressing this issue [1] (it's not applied yet though), see also Bugzilla [2].

Until a patch is applied and an updated package available, if you want to enable rxbounce without manual code changes, you can add it to the rbd_default_map_options [3] [4]. For example, the following command sets a pool-level configuration override that sets rbd_default_map_options to rxbounce:

rbd config pool set YOUR_POOL rbd_default_map_options rxbounce

This will take effect the next time a volume is mapped (so on VM start or live-migration, as mentioned by @mira).
You can unset the override with

rbd config pool rm YOUR_POOL rbd_default_map_options

[1] https://lore.proxmox.com/pve-devel/20241025111304.99680-2-f.weber@proxmox.com/T/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=5779
[3] https://docs.ceph.com/en/reef/man/8/rbd/#commands
[4] https://github.com/ceph/ceph/blob/b...26ac9b4c3/src/common/options/rbd.yaml.in#L507

 

[LnxBil]

I've sent a first version of a patch addressing this issue [1] (it's not applied yet though), see also Bugzilla [2].

That's very misleading on the mailinglist then. There is no branch information in the commit log so I thought that only applied commits generate the mail on pve-devel.

 

[fweber]

Hi,

That's very misleading on the mailinglist then. There is no branch information in the commit log so I thought that only applied commits generate the mail on pve-devel.

Not quite -- pve-devel is a public mailing list, and I sent a patch (in this case, a patch series) for review. If/when it (or a subsequent version) has been reviewed and gets applied, the patch will be committed to the corresponding git repositories [1]. I'll also update the status of the Bugzilla issue [2] then.
See the Developer Documentation for more information [3].

[1] https://git.proxmox.com/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=5779
[3] https://pve.proxmox.com/wiki/Developer_Documentation

 

[LnxBil]

Not quite -- pve-devel is a public mailing list, and I sent a patch (in this case, a patch series) for review. If/when it (or a subsequent version) has been reviewed and gets applied, the patch will be committed to the corresponding git repositories

I wasn't aware of that you guys also use it like this. It all seems so overcomplicated with respect to github or gitlab workflow, yet I know you like your git-email workflow.

 

[esi_y]

I wasn't aware of that you guys also use it like this. It all seems so overcomplicated with respect to github or gitlab workflow, yet I know you like your git-email workflow.

Yes and there's no documentation of any of these (desired) workflows for the public, this was my attempt to take on it in respect to (not only) BZ.

You have to subscribe and observe, guess, ask. And I still can't get over the fact that there's no guaranteed timeframe for review within some period even for staff contributions. The bump emails are pentiful.

 

[VoIP-Ninja]

I can also confirm, since this change - no more CRC sig Erros in the logs with KRBD.

 

[daros]

I tried the

I've sent a first version of a patch addressing this issue [1] (it's not applied yet though), see also Bugzilla [2].

Until a patch is applied and an updated package available, if you want to enable rxbounce without manual code changes, you can add it to the rbd_default_map_options [3] [4]. For example, the following command sets a pool-level configuration override that sets rbd_default_map_options to rxbounce:

rbd config pool set YOUR_POOL rbd_default_map_options rxbounce

This will take effect the next time a volume is mapped (so on VM start or live-migration, as mentioned by @mira).
You can unset the override with

rbd config pool rm YOUR_POOL rbd_default_map_options

[1] https://lore.proxmox.com/pve-devel/20241025111304.99680-2-f.weber@proxmox.com/T/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=5779
[3] https://docs.ceph.com/en/reef/man/8/rbd/#commands
[4] https://github.com/ceph/ceph/blob/b...26ac9b4c3/src/common/options/rbd.yaml.in#L507

I adjusted the settings, but I wasn't able to start any virtual machines afterward.

In some cases useful info is found in syslog - try "dmesg | tail".
rbd: sysfs write failed
TASK ERROR: can't map rbd volume vm-551-disk-2: rbd: sysfs write failed

 

[fweber]

I tried the


I adjusted the settings, but I wasn't able to start any virtual machines afterward.

In some cases useful info is found in syslog - try "dmesg | tail".
rbd: sysfs write failed
TASK ERROR: can't map rbd volume vm-551-disk-2: rbd: sysfs write failed

Can you provide the output of the following commands (fill in YOUR_POOL accordingly):

pveversion -v
rbd config pool list YOUR_POOL
dmesg | grep rbd

If you need to start your VMs, you can either try unsetting the config override again as described earlier [1], or temporarily switch to librbd by unchecking the "KRBD" option in your storage config.

[1] https://forum.proxmox.com/threads/155741/post-714951