Hi,
We're investigating a significant performance difference between Proxmox VE + Ceph and another KVM/OpenStack + Ceph environment. The issue only affects Windows guests.
The Ceph cluster is healthy:
Sequential file copy inside Windows is much slower than expected.
Copying a large file between two virtual disks inside the guest using Windows Explorer reaches only:
Running DiskSpd:
diskspd -c20G -b1M -d60 -o4 -t1 -w100 -Su -L
produces typical results of:
We have another infrastructure running:
with the same Windows Server 2022 guest.
Running the exact same DiskSpd command there produces:
Windows Explorer also copies the same files at approximately:
So this behavior is only observed on our Proxmox cluster.
Linux VMs on the same Proxmox cluster perform much better.
Sequential file copy inside Linux reaches approximately:
This suggests that Ceph itself is not bandwidth-limited.
We've already tried:
None of these changes produced any meaningful improvement.
During DiskSpd, rbd perf image iostat reports approximately:
At the same time:
Has anyone seen similar behaviour?
Specifically:
Is there anything specific in Proxmox, QEMU, libvirt, or the RBD implementation that could explain this difference?
Any ideas or suggestions would be greatly appreciated.
We're investigating a significant performance difference between Proxmox VE + Ceph and another KVM/OpenStack + Ceph environment. The issue only affects Windows guests.
Environment
- Proxmox VE 9.2
- Ceph Tentacle 20.2.2
- QEMU 11
- Windows Server 2022
- VirtIO drivers 0.1.285 (also tested an older version)
- Ceph RBD storage
- Tested with both virtio-scsi-single and virtio-blk
The Ceph cluster is healthy:
- No recovery or backfill operations
- ceph osd perf shows commit/apply latency mostly around 1–2 ms
Problem
Sequential file copy inside Windows is much slower than expected.
Copying a large file between two virtual disks inside the guest using Windows Explorer reaches only:
- 250–350 MB/s
Running DiskSpd:
diskspd -c20G -b1M -d60 -o4 -t1 -w100 -Su -L
produces typical results of:
- 100–170 MiB/s
- Average write latency 20–40 ms
Comparison
We have another infrastructure running:
- OpenStack
- KVM
- Ceph RBD
with the same Windows Server 2022 guest.
Running the exact same DiskSpd command there produces:
- ~373 MiB/s
- Average latency ~10.7 ms
Windows Explorer also copies the same files at approximately:
- 1–1.5 GB/s
So this behavior is only observed on our Proxmox cluster.
Linux comparison
Linux VMs on the same Proxmox cluster perform much better.
Sequential file copy inside Linux reaches approximately:
- ~1 GB/s
This suggests that Ceph itself is not bandwidth-limited.
What we've already tested
We've already tried:
- CPU type:
- x86-64-v2-AES
- host
- cache=none
- cache=writeback
- IOThread enabled/disabled
- virtio-scsi-single
- virtio-blk
- Latest VirtIO drivers
- Older VirtIO drivers
- Windows write cache policy
- CPU governor set to performance on Proxmox hosts
None of these changes produced any meaningful improvement.
Ceph observations
During DiskSpd, rbd perf image iostat reports approximately:
- 140–160 MiB/s writes
At the same time:
- OSD latency remains low
- No obvious bottleneck is visible inside Ceph
Question
Has anyone seen similar behaviour?
Specifically:
- Windows guests on Proxmox + Ceph
- Significantly higher sequential write latency than another KVM/OpenStack environment using Ceph RBD
- Linux guests perform well on the same cluster
- Windows guests are consistently much slower
Is there anything specific in Proxmox, QEMU, libvirt, or the RBD implementation that could explain this difference?
Any ideas or suggestions would be greatly appreciated.
Last edited: