Hello,
some time ago we switched from vmware to proxmox. Everything was smooth and fine at the beginning and we were very happy. But these days, we have about 120 VMs connected via iSCSI to the network storage that consist of two Dell PSM4110XS. For partitioning we use shared LVM. This type of storage has both SSD and HDD drives, so high IO rates are available. We expect throughput about 150-250MB/s from each VM. Dell Eqallogic show us that:
- max possible IOPS - 4667
- total IOPS - 752
- I/O load - low (so there is still some reserve)
- read write values - 30/70%
- iscsi connections - 34
- network usage - 0,5%
- total link speed - 2,35 GB/s
- TCP retransmmit < 0,1%
Each Blade server (node) in our chassis is connected via 10Gbit/s interface to the storage. So there is no bottleneck. We ran a few test with dd (read and write) and we got only around 8MB/s. During this test online storage monitoring shows us average queue depth in thousands, e.g. 100 000. When it was on the vmware, it showed max 10. We have tried changing network card drivers, tune TCP performance and iscsi parameters but nothing helped.
So we put online our another backup storage PS4100 (only 1Gbit/s connection) to eliminate actual real traffic load for more objective test results. So we have created new volume for performance tests only and ran same dd command (dd if=/dev/urandom of=/root/img bs=4M count=1024 status=progress) from vmware 6.5 and then from latest proxmox. vmware runs at 106MB/s (r/w interface limit) and proxmox again only 8MB/s. Both systems has one iscsi connection to the backup storage, same hw.
We have also experience with read only filesystem. Some VM randomly switch to readonly FS and we must reboot them. It is not so often, but it happens sometime. We think that this issue is related to iscsi problem too.
Where could be a problem?
We like very much the Proxmox interface and ecosystem around (Debian), but we are in a blind alley ... We don't know where we are making a mistake.
Thank you for any help, we don't want switch back to the VMware.
some time ago we switched from vmware to proxmox. Everything was smooth and fine at the beginning and we were very happy. But these days, we have about 120 VMs connected via iSCSI to the network storage that consist of two Dell PSM4110XS. For partitioning we use shared LVM. This type of storage has both SSD and HDD drives, so high IO rates are available. We expect throughput about 150-250MB/s from each VM. Dell Eqallogic show us that:
- max possible IOPS - 4667
- total IOPS - 752
- I/O load - low (so there is still some reserve)
- read write values - 30/70%
- iscsi connections - 34
- network usage - 0,5%
- total link speed - 2,35 GB/s
- TCP retransmmit < 0,1%
Each Blade server (node) in our chassis is connected via 10Gbit/s interface to the storage. So there is no bottleneck. We ran a few test with dd (read and write) and we got only around 8MB/s. During this test online storage monitoring shows us average queue depth in thousands, e.g. 100 000. When it was on the vmware, it showed max 10. We have tried changing network card drivers, tune TCP performance and iscsi parameters but nothing helped.
So we put online our another backup storage PS4100 (only 1Gbit/s connection) to eliminate actual real traffic load for more objective test results. So we have created new volume for performance tests only and ran same dd command (dd if=/dev/urandom of=/root/img bs=4M count=1024 status=progress) from vmware 6.5 and then from latest proxmox. vmware runs at 106MB/s (r/w interface limit) and proxmox again only 8MB/s. Both systems has one iscsi connection to the backup storage, same hw.
We have also experience with read only filesystem. Some VM randomly switch to readonly FS and we must reboot them. It is not so often, but it happens sometime. We think that this issue is related to iscsi problem too.
Where could be a problem?
We like very much the Proxmox interface and ecosystem around (Debian), but we are in a blind alley ... We don't know where we are making a mistake.
Thank you for any help, we don't want switch back to the VMware.
Last edited: