QuadCore 8400+3ware 9650-4LP very slow fsync

E

effem

Member
Oct 14, 2010
9
0
21
Hi,
I just read that fsync is very important for proxmox installation.

My hardware is a Quad Core Intel 8400 with 3ware 9650-4LP (write cache enable end storsave Performance)
4 WD 500Gb RE3 16M cache RAID 5

I have a debian lenny with proxmox installation.

Code: 
root@muletto:~# pveversion -v
pve-manager: 1.6-2 (pve-manager/1.6/5087)
running kernel: 2.6.32-4-pve
proxmox-ve-2.6.32: 1.6-19
pve-kernel-2.6.32-4-pve: 2.6.32-23
pve-kernel-2.6.35-1-pve: 2.6.35-6
qemu-server: 1.1-18
pve-firmware: 1.0-8
libpve-storage-perl: 1.0-14
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-7
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.5-1
ksm-control-daemon: 1.0-4

and pveperf
Code: 
root@muletto:~# pveperf 
CPU BOGOMIPS:      21372.16
REGEX/SECOND:      887659
HD SIZE:           9.31 GB (/dev/sda2)
BUFFERED READS:    332.42 MB/sec
AVERAGE SEEK TIME: 6.89 ms
FSYNCS/SECOND:     250.10
DNS EXT:           63.01 ms

I also configure

Code: 
# blockdev --setra 16384 /dev/sda
# echo "deadline" > /sys/block/sda/queue/scheduler
# echo "512" > /sys/block/sda/queue/nr_requests
[CODE]

I also tested 2.6.35 kernel (-3 and -6 version on test repository)

Fsync is very slow? Why?

Bye
francesco
 
do you run pveperf when the system was idle? try several times.
 
Thanks Tom,
no VM is installed and system is very idle.
With 2.6.35 fsync downgrade to 99 for second.

With this hardware, fsync should be a lot more? (over 1000?)
We need more information to resolve the problem?

Bye
Francesco
 
upgrade to the very latest packages (use the pvetest repository).

and make sure that you enable write back on the raid volume.
 
Sigh :-(

Nothing to do...

Summary of the configuration

Code: 
root@muletto:~# pveversion -v
pve-manager: 1.6-5 (pve-manager/1.6/5261)
running kernel: 2.6.35-1-pve
proxmox-ve-2.6.35: 1.6-6
pve-kernel-2.6.35-1-pve: 2.6.35-6
qemu-server: 1.1-22
pve-firmware: 1.0-9
libpve-storage-perl: 1.0-14
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-8
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.12.5-2
ksm-control-daemon: 1.0-4
Code: 
root@muletto:~# pveperf
CPU BOGOMIPS:      21372.91
REGEX/SECOND:      863708
HD SIZE:           9.31 GB (/dev/sda2)
BUFFERED READS:    274.59 MB/sec
AVERAGE SEEK TIME: 6.34 ms
FSYNCS/SECOND:     100.00
DNS EXT:           52.77 ms
Controller 3Ware

Code: 
//muletto/c2> show model
/c2 Model = 9650SE-4LPML
Code: 
//muletto/c2> show firmware
/c2 Firmware Version = FE9X 4.10.00.007
Code: 
//muletto/c2> show ctlbus
/c2 Controller Bus Type = PCIe
/c2 Controller Bus Width = 4 lanes
/c2 Controller Bus Speed = 2.5 Gbps/lane
Code: 
//muletto/c2> show driver
/c2 Driver Version = 2.26.02.014
Code: 
//muletto/c2> show

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    RAID-5    OK             -       -       256K    1396.95   RiW    ON     

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   465.76 GB SATA  0   -            WDC WD5002ABYS-02B1 
p1    OK             u0   465.76 GB SATA  1   -            WDC WD5002ABYS-02B1 
p2    OK             u0   465.76 GB SATA  2   -            WDC WD5002ABYS-02B1 
p3    OK             u0   465.76 GB SATA  3   -            WDC WD5002ABYS-02B1
The 3ware documentation

Code: 
[B]Note:[/B] The 'Cache' column in the unit summary differ between the older (up to 9550SX and 9650SE with Release 9.5.1 or earlier) and newer (9690SA and 9650SE  with Release 9.5.2 or later) controllers. 

[cut]
 Below is a summary of the possible settings in that column:
   W   - only the write cache is enabled  
Rb   - only read cache Basic is enabled  
Ri  - only read cache Intelligent is enabled  
RbW  - read cache Basic and write cache are both enabled  
RiW - read cache Intelligent and write cache are both enabled 
OFF - all read and write caches are disabled

The HOST machine is idle and no VM created.

Thanks for your help.

Bye
Francesco
 
I don't have BBU installed...but 3ware says that write cache is enabled (risk to me)...

If I switch off write cache...

Code: 
root@muletto:~# pveperf
CPU BOGOMIPS:      21372.91
REGEX/SECOND:      878367
HD SIZE:           9.31 GB (/dev/sda2)
BUFFERED READS:    327.96 MB/sec
AVERAGE SEEK TIME: 6.21 ms
FSYNCS/SECOND:     28.02
DNS EXT:           58.36 ms

Poor performance...for a RAID card.
Nobody with this card?
Bye
 
[RESOLVED]Re: QuadCore 8400+3ware 9650-4LP very slow fsync

Hi,
I understand the problem.

Code: 
effem@muletto:~$ df -h
Filesystem         Dimens. Usati Disp. Uso% Montato su
/dev/sda2             9,4G  959M  8,4G  11% /
tmpfs                 2,0G     0  2,0G   0% /lib/init/rw
udev                   10M  744K  9,3M   8% /dev
tmpfs                 2,0G     0  2,0G   0% /dev/shm
/dev/sda5              94G   33M   94G   1% /mnt/sda5
/dev/sda6              20G   33M   20G   1% /mnt/sda6

effem@muletto:~$ sudo blkid /dev/sda2
/dev/sda2: UUID="54f52fa1-1fef-425b-84b0-6e3a55db67d2" LABEL="sistema" TYPE="reiserfs" 
effem@muletto:~$ sudo blkid /dev/sda5
/dev/sda5: UUID="e75fd066-e78c-4cc6-a71e-731403c71c27" TYPE="xfs" 
effem@muletto:~$ sudo blkid /dev/sda6
/dev/sda6: UUID="dd5accb3-98b4-4b64-b4ad-da69a6225696" TYPE="reiserfs"
My pveperf...
Code: 
effem@muletto:~$ sudo pveperf
CPU BOGOMIPS:      21372.28
REGEX/SECOND:      863253
HD SIZE:           9.31 GB (/dev/sda2)
BUFFERED READS:    327.31 MB/sec
AVERAGE SEEK TIME: 6.25 ms
FSYNCS/SECOND:     99.37
DNS EXT:           53.28 ms

effem@muletto:~$ sudo pveperf /mnt/sda6
CPU BOGOMIPS:      21372.28
REGEX/SECOND:      843491
HD SIZE:           19.07 GB (/dev/sda6)
BUFFERED READS:    320.08 MB/sec
AVERAGE SEEK TIME: 6.69 ms
FSYNCS/SECOND:     100.13
DNS EXT:           51.02 ms

effem@muletto:~$ sudo pveperf /mnt/sda5
CPU BOGOMIPS:      21372.28
REGEX/SECOND:      892352
HD SIZE:           93.09 GB (/dev/sda5)
BUFFERED READS:    275.28 MB/sec
AVERAGE SEEK TIME: 7.74 ms
FSYNCS/SECOND:     5852.16
DNS EXT:           68.70 ms


Fsync depends on the filesystem?
If yes, when I have vgGroup for storage, which is the fsync?

Thanks to all.
best regards,
Francesco


 
Re: [RESOLVED]Re: QuadCore 8400+3ware 9650-4LP very slow fsync

so you are using a custom setup? would be nice if you tell this in advance ...

you should use the default ext3 file-system with Proxmox VE.
 
Re: [RESOLVED]Re: QuadCore 8400+3ware 9650-4LP very slow fsync

Sorry,
in my first post I wrote
I have a debian lenny with proxmox installation.
Click to expand...

Thanks
Francesco
 
Re: [RESOLVED]Re: QuadCore 8400+3ware 9650-4LP very slow fsync

yes, and a Debian Lenny uses always ext3 unless you select manually another file-system.

can you switch to ext3 and post your pveperf results?
 
Ok, I understand
I've already done the test with a xfs partition, and the result was nearly 6000 / s fsync.
My only question is why fsync varies depending on the filesystem?
The test (pveperf) not based on raw partitions?

Thanks
Francesco
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back