Something weird going on & could use help.
Dell R710, 2x E5645 CPU, 96gb ram, PERC 6i, 2x 128gb SAS raid1 for OS, 6x 300gb SAS raid5 for storage. Storage volume configured as single ZFS pool. PVE 5.3-7
Dec 2018: Used backup/restore process to import 1 Windows & 5 Linux VMs from a PVE 4.X cluster that we were retiring. Everything has been running great.
May 2019: Created a Windows 2019 VM to migrate 500GB email server from Win 2012 VM running on Hyper-v host. Performance appeared ok until we actually cutover last weekend then performance tanked. CPU hovering around 15-20% and iowait fluctuating 20-40% even after hours.
June 2019 (last night): Added NFS storage and migrated all VMs to it. Iowait immediately went to 0% and CPU running 2-5%. Deleted ZFS spool, reformatted storage volume as XFS, re-added as simple directory.
Cloned our Win2019 40GB golden image from NFS to the local XFS, saw iowait jump & hover to 15%. Started cloned VM and logged in, iowait stayed 15-20%. Shutdown VM.
Started moving the Dec2018 imported Windows VM from NFS to XFS, saw iowait jump to 15%, canceled move.
Started moving a Dec2018 imported Linux VM and iowait stayed at 0%. Moved the remaining four Dec2018 Linux VMs from NFS to XFS and iowait stays at 0% entire time.
As of right now all Linux VMs are running from local XFS and Windows VMs are running from NFS. CPU is around 8% and iowait is 0%.
Today I create a 12GB Linux VM on local XFS but don't power it on or do any install. I move it to NFS and it moves immediately with 0% iowait. I move it back from from NFS to local XFS and it takes 2-3 min and iowait jumps to 15%.
Just deleted/recreated the Linux VM on local XFS. Started VM to install CentOS, iowait jumped to 7%. During install iowait was 8-10%, fell to 0-1%, then back to 10% during post install setup tasks. After reboot, running yum update iowait is 8-10%. Moving from local XFS to NFS causes iowait to go to 6-8% and same moving from NFS back to XFS.
Just moved the powered off 40GB Windows golden image VM from NFS to XFS and it caued iowait to go to 30%.
So it appears that any new VMs we create perform badly on local storage but run fine on NFS and the VMs we imported in from our 4.X cluster run fine on both. We're using qcow2 for everything.
Dell R710, 2x E5645 CPU, 96gb ram, PERC 6i, 2x 128gb SAS raid1 for OS, 6x 300gb SAS raid5 for storage. Storage volume configured as single ZFS pool. PVE 5.3-7
Dec 2018: Used backup/restore process to import 1 Windows & 5 Linux VMs from a PVE 4.X cluster that we were retiring. Everything has been running great.
May 2019: Created a Windows 2019 VM to migrate 500GB email server from Win 2012 VM running on Hyper-v host. Performance appeared ok until we actually cutover last weekend then performance tanked. CPU hovering around 15-20% and iowait fluctuating 20-40% even after hours.
June 2019 (last night): Added NFS storage and migrated all VMs to it. Iowait immediately went to 0% and CPU running 2-5%. Deleted ZFS spool, reformatted storage volume as XFS, re-added as simple directory.
Cloned our Win2019 40GB golden image from NFS to the local XFS, saw iowait jump & hover to 15%. Started cloned VM and logged in, iowait stayed 15-20%. Shutdown VM.
Started moving the Dec2018 imported Windows VM from NFS to XFS, saw iowait jump to 15%, canceled move.
Started moving a Dec2018 imported Linux VM and iowait stayed at 0%. Moved the remaining four Dec2018 Linux VMs from NFS to XFS and iowait stays at 0% entire time.
As of right now all Linux VMs are running from local XFS and Windows VMs are running from NFS. CPU is around 8% and iowait is 0%.
Today I create a 12GB Linux VM on local XFS but don't power it on or do any install. I move it to NFS and it moves immediately with 0% iowait. I move it back from from NFS to local XFS and it takes 2-3 min and iowait jumps to 15%.
Just deleted/recreated the Linux VM on local XFS. Started VM to install CentOS, iowait jumped to 7%. During install iowait was 8-10%, fell to 0-1%, then back to 10% during post install setup tasks. After reboot, running yum update iowait is 8-10%. Moving from local XFS to NFS causes iowait to go to 6-8% and same moving from NFS back to XFS.
Just moved the powered off 40GB Windows golden image VM from NFS to XFS and it caued iowait to go to 30%.
So it appears that any new VMs we create perform badly on local storage but run fine on NFS and the VMs we imported in from our 4.X cluster run fine on both. We're using qcow2 for everything.