Fsyncs/sec problems 2x3TB mdadm raid 1 + lvm on VE 3.0

[geekcamp]

Hi fellow members,
Just installed my first Proxmox Server, played around alittle, but it just felt slow when making a winxp install, and was hanging every other second or so.

As you can see from the pveperf test below, some thing is wacked with my fsyncs/sec, the system has nothing running when the test where run, its just an i7 3770 @ 3,4ghz, with 16gb ram, and 2x3TB sata in software raid 1, with lvm on top, and then proxmox ve 3.0. as this is my first real take on vps's on linux other than vmware, it have no clue as to why the fsyncs/sec are so low.

root@s1 ~ # pveperf /homeCPU BOGOMIPS:      54401.92
REGEX/SECOND:      1736668
HD SIZE:           99.95 GB (/dev/mapper/vg0-home)
BUFFERED READS:    165.27 MB/sec
AVERAGE SEEK TIME: 9.10 ms
FSYNCS/SECOND:     15.34
DNS EXT:           48.75 ms

root@s1 ~ # pveperf /var/lib/vzCPU BOGOMIPS:      54401.92
REGEX/SECOND:      1761518
HD SIZE:           496.06 GB (/dev/mapper/vg0-root)
BUFFERED READS:    139.61 MB/sec
AVERAGE SEEK TIME: 11.63 ms
FSYNCS/SECOND:     9.89

 

[mir]

What is your disk alignment?

Am I right to assume your disks are Advanced format disks?

 

[geekcamp]

Well I was kinda reading upon how to check if the alignment was off, but havent found out how to check it.

I dident align the disks my self, they where installed by an auto install script from hetzner. but I will assume its not 512byte sectors aka they are 4096byte advanced disks, and that would clearly give bad writes, but it should still not be as bad as it is now, I'm up for a reinstall and see if I can poke the install script to align the partitions, but is it also possible that the lvm volume is also missaligned on top of the maybe misaligned md raid? oh man, misalign my a$$

 

[mir]

You should stick to gparted or fdisk and avoid the script from Hetzner. gparted and fdisk defaults to align the disk to 1MB which is the optimum for 512/4096 B disks.

Bad alignment can severely influence performance since LVM is configured to align PV's to 1MB and if your disks has a different alignment IOPS will drop significantly.

http://busybox.net/~aldot/mkfs_stride.html suggest something like this when creating the file system but try out different combinations and test with fio:
mkfs.ext3 -b 4096 -E stride=256,stripe-width=256

 

[geekcamp]

from hetzners faq. "When operating systems are installed via the Robot or via the installimage in the Rescue System, correct alignment is automatically carried out." So the partitions should be aligned, but looking at parted i get this from dev/sda

Model: ATA ST3000DM001-1CH1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt


Number  Start   End     Size    File system  Name  Flags
 3      1049kB  2097kB  1049kB                     bios_grub
 1      2097kB  539MB   537MB                      raid
 2      539MB   3001GB  3000GB                     raid

it looks aligned to me, can it be lvm?

 

[geekcamp]

You should stick to gparted or fdisk and avoid the script from Hetzner.

Kinda have alittle problem in that department... it aint as simple as slaping a disk in a dvd drive and power on the box...

 

[mir]

That looks completely wrong to me. It is not even aligned to a factor of 2. The correct alignment is 2048kB for 512/4096

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0004e37d


Device Boot Start End Blocks Id System
/dev/sdb1 3890251776 3907028991 8388608 82 Linux swap / Solaris
/dev/sdb2 2048 3890251775 1945124864 83 Linux

 

[mir]

Kinda have alittle problem in that department... it aint as simple as slaping a disk in a dvd drive and power on the box...

If you are not completely sure on yourself you could download the gparted or systemrecuecd live distribution and boot from this. They have a gui for parted and if I remember correctly gparted is also capable of creating the raid.

gparted: http://gparted.sourceforge.net/
systemrecuecd: http://www.sysresccd.org/SystemRescueCd_Homepage

I personally uses systemrecuecd and I cannot count the times it has saved by ass. I have it installed on a USB stick which I always carry around - you will never no when it will come in handy

 

[geekcamp]

The main problem with that, is the server is located at a DC in another country about 1200km from where i'm sitting, so I will just not swing by with a usb stick and poke around for a day. But I do have the option to put the server remotely in rescue mode via pxe boot, and thats also how I can reinstall the box for when I screw it all up.

 

[mir]

Are you not able to boot the server from an ISO image?

 

[DrStone]

Hi geekcamp.

I have the exact same setup (2x3TB mdadm raid 1 + lvm with hetzner installscript) but with VE 3.3 and the same problem - abysmal fsync/sec values:

root@abba:/# pveperf
CPU BOGOMIPS:      54402.00
REGEX/SECOND:      1581485
HD SIZE:           4.96 GB (/dev/mapper/vg0-abba_root)
BUFFERED READS:    174.91 MB/sec
AVERAGE SEEK TIME: 9.88 ms
FSYNCS/SECOND:     9.86
DNS EXT:           57.10 ms

the alignment should be OK:

root@abba:/# parted -s /dev/sda unit s print
Model: ATA TOSHIBA DT01ACA3 (scsi)
Disk /dev/sda: 5860533168s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start    End          Size         File system  Name  Flags
 3      2048s    4095s        2048s                           bios_grub
 1      4096s    528383s      524288s                         raid
 2      528384s  5860533134s  5860004751s                     raid

Here are my versiondump:

root@abba:/# pveversion -v
proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-20-pve)
pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-30-pve: 2.6.32-130
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.3-3
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-25
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

Please tell me you found a other solution than the alignment, I have no idea whats going wrong.


:thorsten