Proxmox fresh install and weird long times to execute commands

[franciscopaniskaseker]

Environment:
- fresh proxmox install
- proxmox installed under first ssd
- second ssd used as a cache of pve
- 4 x 2TB disks grouped in one volume

There is 6 centos 7 VMs almost doing nothing with I/O. The WA of Host is 0.0 and the WA of the VMs is n, with n 0.0>=n>1.0

screen command executed inside of the host:

root@pve:~# time screen -ls
No Sockets found in /run/screen/S-root.


real    0m0.002s
user    0m0.000s
sys    0m0.002s
root@pve:~#

Now inside of VM:

[root@dev-2 ~]# time screen -ls
No Sockets found in /var/run/screen/S-root.


real    0m0.431s
user    0m0.149s
sys    0m0.282s
[root@dev-2 ~]#

This is the first time that I am seeing that. I already configured this same setup at least 8 times and I never saw this. Ant two of them use exactly same server setup from same datacenter.

The host hardware/features/info:
- Dual Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (16c/32t)
- 256gb DDR3 1600mhz
- 2 x SSD (one to proxmox, another one to pve cache)
- 4x 2TB HDD (one volume under LVM). The chunksize is 64Kb (7.26Tb volume).

 

[aaron]

Hmm, what CPU did you configure for the VMs? Did you disable all power saving features in the BIOS?

 

[franciscopaniskaseker]

cpu type: kvm 64

When you talk about BIOS, are you asking about the Host or VM BIOS?

 

[aaron]

The hosts BIOS.
Is this a single node? Then you could also set the CPU for the VMs to "host" without worrying about other possible CPU types as in a cluster.

 

[franciscopaniskaseker]

Giving a feedback:
- disabling the powersaver gorvernor and set the performance one: no changes
- changing kvm64 to host: the time was divided by two for screen -ls command (0.5s to 0.25s). however the ideal number is 0.01s or less in normal situations

The IO use is very low. But commandas like "screen -ls" just check some structure in the memory, so it must be fast (unless you have any problem with screen -ls, but is not the case).

So amazing that kvm64->host increased the performance, but not seems that is the answer for my problem.

 

[franciscopaniskaseker]

pveperf results:

root@pve:~# pveperf
CPU BOGOMIPS:      141011.20
REGEX/SECOND:      1585748
HD SIZE:           54.63 GB (/dev/mapper/pve-root)
BUFFERED READS:    502.19 MB/sec
AVERAGE SEEK TIME: 0.13 ms
FSYNCS/SECOND:     3387.38
DNS EXT:           44.49 ms
DNS INT:           1.97 ms (xxx)
root@pve:~#

vm benchmarks:

[root@ ~]# sysbench --test=cpu --cpu-max-prime=20000 run
CPU speed:
    events per second:   289.64

General statistics:
    total time:                          10.0033s
    total number of events:              2898

Latency (ms):
         min:                                    3.24
         avg:                                    3.45
         max:                                    4.67
         95th percentile:                        3.75
         sum:                                 9998.30

Threads fairness:
    events (avg/stddev):           2898.0000/0.00
    execution time (avg/stddev):   9.9983/0.00

[root@ ~]# time sysbench --test=fileio --file-total-size=48G --file-num=24 prepare
51539607552 bytes written in 646.04 seconds (76.08 MiB/sec).

real    10m46.053s
user    0m1.820s
sys    1m32.177s

Impotant note: the same VM under SSD get better write MiB/sec value, but have exactly the same problem, so I do not think the storage performance is related.

 

[franciscopaniskaseker]

I have other server with same CPU, motherboard, memory etc (same datacenter server plan) configured with Proxmox in the same way:

[root@ip76 ~]#  time screen -ls
No Sockets found in /var/run/screen/S-root.


real    0m0.008s
user    0m0.001s
sys    0m0.007s

edit:

actually they have one difference: the proxmox without the problem has 4 disks and they are used as separated volumes. the proxmox with performannce problem, the same 4 disks were used to create just one big volume. but their benchmark about read/wirte performance get same resylts.

 

[franciscopaniskaseker]

Today I saw one thing different: We have ony one Ubuntu VM and there it is working perfect:

root@ubuntu:/home# time screen -ls
No Sockets found in /run/screen/S-root.


real  0m0.003s
user  0m0.003s
sys  0m0.000s

But is not with centos:

[root@centos home]# time screen -ls
No Sockets found in /var/run/screen/S-root.


real  0m0.278s
user  0m0.060s
sys  0m0.218s

Both VMs have same disk resources and they are in the same LVM volume. One is Ubuntu 20.04 and the other CentOS 7. Now I can confirm is not the server the problem.

The second difference is the Ubuntu VM has just 6 vCPU while CentOS has 16. But decrease the vCPU number did not fix.

My next try is the kernel of the CentOS 7.

 

[aaron]

thanks for keeping us updated