Proxmox and VM performance are too slow, Linux VM Taking 3-4 hours and Windows VM 7-8 hours to bootup

[akus]

Hello,
I installed a new setup in my server. (Unfortunately old one crashed completely). I had to rebuild it from scratch.
In capacity, it's 2tb HDD SATA + 500 GB SSD, with 64GB RAM.
I followed the steps as per documents. I selected Ext4 for my installation.
Surprisingly, once I loaded vm's. It start functioning very slow.
Linux VM are taking 3-4 hours to bootup :O. and window 7-8 hours.
Everything is damn slow. Earlier with same infra I was able to perform all vm smoothly.
Now ssh, GUI, VM NoVNC everything is slow. I write and wait for 30s- 5 min to get them reflect!!!!!!
I don't understand, where I did a mistake. Or do I need to perform some commands?
Do I Need to reinstall everything from scratch :'(

Please suggest, What to do!!!!

pveperf
CPU BOGOMIPS: 43117.86
REGEX/SECOND: 1461883
HD SIZE: 441.69 GB (/dev/mapper/pve-root)
BUFFERED READS: 4.27 MB/sec
AVERAGE SEEK TIME: 98.17 ms
FSYNCS/SECOND: 2.27
DNS EXT: 165.05 ms
DNS INT: 168.14 ms

root@akus:~# fdisk -l
Disk /dev/nvme0n1: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: GIGABYTE GP-AG4500G
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 484DCD64-7448-4889-93F1-A58FC3A45F24


Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: ST2000DM005-2CW1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 067AE0B6-24E6-4292-81A0-5295337C19BE

Device Start End Sectors Size Type
/dev/sdb1 34 2047 2014 1007K BIOS boot
/dev/sdb2 2048 1050623 1048576 512M EFI System
/dev/sdb3 1050624 3774873600 3773822977 1.8T Linux LVM

Partition 1 does not start on physical sector boundary.


Disk /dev/mapper/pve-swap: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/pve-root: 449.8 GiB, 482915385344 bytes, 943194112 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/pve-vm--101--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--101--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--101--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--101--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--103--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--103--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--103--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--103--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--106--disk--0: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 61923BA6-4ACF-4803-A63B-A842BC9E06B5

Device Start End Sectors Size Type
/dev/mapper/pve-vm--106--disk--0-part1 34 2047 2014 1007K BIOS boot
/dev/mapper/pve-vm--106--disk--0-part2 2048 1050623 1048576 512M EFI System
/dev/mapper/pve-vm--106--disk--0-part3 1050624 104857566 103806943 49.5G Linux LVM

Partition 1 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--104--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--104--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--104--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--104--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--105--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--105--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--105--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--105--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--107--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--107--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--107--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--107--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--153--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xcf0a2181

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--153--disk--0-part1 * 2048 1026047 1024000 500M 7 HPFS/NTFS/exFAT
/dev/mapper/pve-vm--153--disk--0-part2 1026048 209713151 208687104 99.5G 7 HPFS/NTFS/exFAT


Disk /dev/mapper/pve-vm--153--disk--1: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xcf0a2181

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--153--disk--1-part1 * 2048 1026047 1024000 500M 7 HPFS/NTFS/exFAT
/dev/mapper/pve-vm--153--disk--1-part2 1026048 209713151 208687104 99.5G 7 HPFS/NTFS/exFAT


Disk /dev/mapper/pve-vm--153--disk--2: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xcf0a2181

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--153--disk--2-part1 * 2048 1026047 1024000 500M 7 HPFS/NTFS/exFAT
/dev/mapper/pve-vm--153--disk--2-part2 1026048 209713151 208687104 99.5G 7 HPFS/NTFS/exFAT


Disk /dev/mapper/pve-vm--151--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0x295ecc9e

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--151--disk--0-part1 * 2048 206847 204800 100M 7 HPFS/NTFS/exFAT
/dev/mapper/pve-vm--151--disk--0-part2 206848 209713151 209506304 99.9G 7 HPFS/NTFS/exFAT


Disk /dev/mapper/pve-vm--130--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--130--disk--1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--152--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xcf0a2181

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--152--disk--0-part1 * 2048 1026047 1024000 500M 7 HPFS/NTFS/exFAT
/dev/mapper/pve-vm--152--disk--0-part2 1026048 209713151 208687104 99.5G 7 HPFS/NTFS/exFAT


Disk /dev/mapper/pve-vm--123--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--123--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--123--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--123--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--121--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--121--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--121--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--121--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

Partition 2 does not start on physical sector boundary.


Disk /dev/mapper/pve-vm--122--disk--0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: dos
Disk identifier: 0xe33545e8

Device Boot Start End Sectors Size Id Type
/dev/mapper/pve-vm--122--disk--0-part1 * 2048 1050623 1048576 512M b W95 FAT32
/dev/mapper/pve-vm--122--disk--0-part2 1052670 209713151 208660482 99.5G 5 Extended
/dev/mapper/pve-vm--122--disk--0-part5 1052672 209713151 208660480 99.5G 83 Linux

---

---

---

Moreover above output shows many Red alert of physical sector boundary.

 

[akus]

Hi @danielb

danielb said:
You should open a thread in the forum instead of a PM, as others would be able to share their advices too. Such slowliness might be due to faulty hardware, hard to tell.
It looks like you haven't used your SSD at all, and everything is on the SATA HDD, you should reinstall PVE on the SSD instead, and use the HDD only for VM storage

Thanks Danielb to sugggest .
I used HDD because it has good amount of size 2TB. SSD is 500Gb only. Which planned to take backups. How can I check whether it's faulty hardware, any idea?

 

[akus]

Hello guys,
Anyone who faced the same issue in past, please advise the solution!!

 

[need2gcm]

I know exactly what your problem is.

First and foremost, that Seagate ST2000DM005 is a Shingled Magnetic Recording (SMR) disk. These are only good for slow, mostly read only storage. They have terrible random write speeds due to the technology behind SMR. There is no way to fix this, there are no good mitigations. This disk must be replaced with a CMR disk. Western Digital got in trouble a bit ago for failing to properly label disks that were SMR, and its a shame that Seagate largely flew under the radar for similar practices. In the future, you should verify the hard drive model you are going to use for VM storage is Conventional Magnetic Recording (CMR).

On top of that, I suggest running the OS and VM storage on separate drives. The less you have trying to access a physical disk at the same time, the better.

And of course, if this is going to be production and not lab, some sort of disk redundancy is highly advised, but I am aware that adds complexity and expense. Not losing all data and function because of one failed disk however, can be worth a whole lot when the time comes.

 

[sherminator]

Hello guys,
Anyone who faced the same issue in past, please advise the solution!!

Hey akus,

some time ago I faced this problem - not with PVE but with KVM/Virt-Manager on a Linux desktop. Finally I discovered that I forgot to enable CPU virtualization features in BIOS. I'm not sure whether it's possible to install a PVE node without having CPU hardware virtualizaion enabled, but maybe you take a look at the output of egrep '(vmx|svm)' /proc/cpuinfo to check this.

Good luck and many greets
Stephan

 

[sherminator]

I know exactly what your problem is.

Of course this disk is far away from "the perfect PVE storage", but I don't think that this can explain the behavior described.

Greets
Stephan

 

[need2gcm]

He may have more than one issue, fair, but having accidentally ran VMs on that exact disk model before, I can say that my performance on an otherwise decent system was awful. The IO Wait those disks can pile up when you have a bunch of random writes will make them feel like a failed/failing drive. I've had those disks chug down to a few KB/s under VM load, which is basically unusable.

 

[akus]

I migrated on SSD+HDD Raid0 as advised by @danielb. Now they are tremendously fast and achieved beyond my expectations. My vm's are running like anything. Thanks a lot @danielb for the suggestion. Since, after first crash, I had important data of VM with me in separate way. It's a loss of course. And I had a backup of backup plan for that. But still I'm happy to fulfil requirement if dependents. Now speed is fastest. I can't explain my feelings. Thanks buddy @danielb.

 

[sherminator]

what the hell - thanks for sharing this experience! I knew that "bad hardware choice" can bring your performance down - but in such a way? Crazy...

 

[akus]

what the hell - thanks for sharing this experience! I knew that "bad hardware choice" can bring your performance down - but in such a way? Crazy...

@sherminator Sometimes, One choice makes your life complex and complicated. . One decision bring back to life again. Behavior of PVE was unpredictable but Yeah I had to face that.

 

[need2gcm]

@akus I'm glad the SSD cache is helping with that drive. Definitely keep a backup of what's stored on that array though, since I have seen those SMR drives cause issues with cached setups before, notably on my home machine when I tried to make some use of the extra SMR disks I had laying around.

 

[akus]

@akus I'm glad the SSD cache is helping with that drive. Definitely keep a backup of what's stored on that array though, since I have seen those SMR drives cause issues with cached setups before, notably on my home machine when I tried to make some use of the extra SMR disks I had laying around.

Yeah, I'm having backups vzdump with me safely now.