PVE 1.9 Unacceptable I/O delays

[apmuthu]

Extraordinary delay in creating OpenVZ VMs (normal templates complete in 13 to 18 seconds on good PVE 1.9x and in 35 seconds to 6 minutes variable on unacceptable ones). On identical hardware running same PVE 1.9 - 2TB WD Green Cavair (Good one was also a Seagate 2TB) XEON E1200 series:

# pveversion -v
pve-manager: 1.9-26 (pve-manager/1.9/6567)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 1.9-55+ovzfix-2
pve-kernel-2.6.32-7-pve: 2.6.32-55+ovzfix-2
qemu-server: 1.1-32
pve-firmware: 1.0-15
libpve-storage-perl: 1.0-19
vncterm: 0.9-2
vzctl: 3.0.29-3pve1
vzdump: 1.2-16
vzprocps: 2.0.11-2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.15.0-2
ksm-control-daemon: 1.0-6

Good ones running 7 OpenVZ VMs and 3 KVMs:

# pveperfs
-bash: pveperfs: command not found
dc02:~# pveperf
CPU BOGOMIPS:      51077.57
REGEX/SECOND:      1194366
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    125.93 MB/sec (upto 195.99 MB/sec)
AVERAGE SEEK TIME: 8.67 ms
FSYNCS/SECOND:     1456.75
DNS EXT:           4.94 ms
DNS INT:           261.88 ms (mydomain.com)

Bad ones running no containers (or 1 OpenVZ container):

# pveperf
CPU BOGOMIPS:      51193.09
REGEX/SECOND:      1237741
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    21.93 MB/sec
AVERAGE SEEK TIME: 11.14 ms
FSYNCS/SECOND:     221.91
DNS EXT:           239.44 ms
DNS INT:           265.92 ms (mydomain.com)

For exactly the same (2 x 8GB ECC Kingston) RAM, the dmidecode -t 17 command yielded 128 total bits for the bad one and 72 bits for the good one.

Handle 0x0014, DMI type 17, 34 bytes
Memory Device
        Array Handle: 0x0007
        Error Information Handle: 0x0015
        Total Width: 128 bits (72 bits for the good machine)
        Data Width: 64 bits
        Size: 8192 MB
        Form Factor: DIMM
        Set: None
        Locator: ChannelB-DIMM0
        Bank Locator: BANK 1
        Type: <OUT OF SPEC>
        Type Detail: Synchronous
        Speed: 1333 MHz (0.8 ns)
        Manufacturer: Kingston
        Serial Number: 232E15AB
        Asset Tag: 9876543210
        Part Number: 9965525-026.A00LF

 

[apmuthu]

A change of Hard Disk to a Seagate Barracuda 2TB with the latest firmware on the same SuperMicro board yielded:

# pveperf

CPU BOGOMIPS:      51193.06
REGEX/SECOND:      1269511
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    175.07 MB/sec
AVERAGE SEEK TIME: 8.43 ms
FSYNCS/SECOND:     1380.60
DNS EXT:           240.93 ms
DNS INT:           247.51 ms (mydomain.com)

The DNS EXT still seems way too much. What will fix it?

According to the help on the binary at /usr/bin/pveperf:
DNS EXT: average time to resolve an external DNS name