very slow IO, almost hangs time to time

J

Jae Lee

Renowned Member
Jul 22, 2012
70
0
71
Marina del Rey, California, United States
Hi there, I am experiencing a occasional system hangs/freeze ups w/ my new server which is:

SUPERMICRO 6017R-WRF
E5-2670
2 x Intel530 480gb SSD
2 x WD 3TB SATA3
On-board Controller

The HDDs are set to JBOD.

I have the proxmox installed on one of the WD3TB SATA3 with my other hard drives connected to.

It seemed fine at a glance but upon careful inspection, it appears that the proxmox host would occasionally freeze up, maybe for up to 5-10 minutes every hour or so. This happens even when server has ZERO load, no outside process, no cronjobs, running.

During this time, the Openvz containers inside created are fine (SSD) and shows no hang ups.

Now I have thought it was the hard drive defected so I switched the host w/ my other WD3TB and still experience same problem.
Moreover, the VM's created using WD3TB, also experience same hangups/slowness.

Here is pveperf done from the host:

root@server1:~# pveperf
CPU BOGOMIPS: 166389.92
REGEX/SECOND: 558502
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 0.17 MB/sec
AVERAGE SEEK TIME: 2540.09 ms
FSYNCS/SECOND: 10.33
DNS EXT: 95.44 ms
DNS INT: 187.68 ms

Which is VERY SLOW!

I even replaced the WD3TB hard drives thinking they were defective, but even w/ new drives, they are still slow!

Anyone have any ideas why this appears to be the case?
 
Last edited:
Yes, very low, I've no idea. When you state "the HDDs are set to JBOD" do you mean that you set a specific configuration on the controller? If so, try to remove it and use just like normal sata configuration, maybe it's interfering in some way (mine is just a shot in the dark...)
 
About of the freeze up, may be that your Bios Hardware have enabled energy savings, PVE works very bad with energy savings, change your Bios to performance.
Tested in several DELL servers.

About of your I/O performance, check your SATA cables (must be equal in speed to your SATA controller), and RAID performance in his own manual
 
Jae Lee said:
so i am seeing these in the host's syslog messages:
EXT4-fs (dm-5): Unaligned AIO/DIO on inode 67764229 by kvm; performance will be poor.
could it be due to this??? if so, what would be the cause for it (and fix)?
Click to expand...

You have a file system corrupt, the journaly of your file system is damaged, turn off your computer, and with a live cd check your file system (no the bad blocks, only the file system), and after, pray to GOD and start normally your system

Do you have a RAID by software?
 
I do not have RAID by software, it is just JBOD.
And FYI, smartctl checks out fine across all my hard drives, all test scans PASSED.
I basically did: smartctl -d ata -H /dev/sdb

But so here is one interesting thing I found.

First, I have:
WD3TB #1 (proxmox)
WD3TB #2 (extra)
IntelSSD480 #1 (extra)
IntelSSD480 #2 (extra)

All my VM's are on IntelSSD480 #1 and #2 right now, all OpenVZ.

Now, I noticed slowness once I created a KVM residing on the WD3TB #2.
By slowness, this affects not only the KVM, but also the proxmox host server on WD3TB #1.

They show the same "IO freeze", where I cant even ssh in for roughly 30 mins at a time, then works okay for another 5mins then freezes again.

So I shut off that KVM, and server is FAST again. I tried to restart that KVM and it won't start anymore, it says:

Unable to create lock file /var/lib/vz/lock/101.lckYrsGnN, use --skiplock option: Read-only file system
Unable to lock container
TASK ERROR: command 'vzctl start 101' failed: exit code 3

Apparently, my KVM is dead it looks like. But you see this happened exactly same before, and thus I prompted for a hard drive replacement, and now I am seeing the same behavior again.
How can this be?

Could it be that hard drive is actually fine and now has to do with something else entirely?
My first thought is the way I setup the LVM partition on the WD3TB #2, I used the whole device rather than creating a partition on it.

Also if only WD3TB #2 is failing, why does WD3TB #1 (proxmox) show slowness too, (I also did it previously on a partition w/ same result)?
So it just does not make sense and not sure if this a driver issue of some sort?
Does my server not like working in tandem w/ the SSDs?
Could this be a bug in proxmox end?
 
Jae Lee said:
so i am seeing these in the host's syslog messages:

EXT4-fs (dm-5): Unaligned AIO/DIO on inode 67764229 by kvm; performance will be poor.

could it be due to this??? if so, what would be the cause for it (and fix)?
Click to expand...

seem to be a wrong partition alignment somewhere (maybe in kvm guest)


can you provide a

fdisk -lu /dev/sdxxx of host and vm ?

(maybe also for /dev/dm-.. if you use raid)
 
spirit said:
seem to be a wrong partition alignment somewhere (maybe in kvm guest)


can you provide a

fdisk -lu /dev/sdxxx of host and vm ?

(maybe also for /dev/dm-.. if you use raid)
Click to expand...


Host is /dev/sda, VM is on /dev/sdb:


root@server1:~# fdisk -lu /dev/sda

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Device Boot Start End Blocks Id System
/dev/sda1 1 4294967295 2147483647+ ee GPT
Partition 1 does not start on physical sector boundary.


root@server1:~# fdisk -lu /dev/sdb


Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/sdb doesn't contain a valid partition table
 
Your /dev/sda has been formatted without partitions. This is clearly wrong and means that your disk is behaving badly due to an unaligned file system.
 
The /dev/sda is proxmox. It was a stock installation off the latest proxmox iso.
But I do see the various /dev/mapper/pve-root and /dev/mapper/pve-root, etc which are from /dev/sda.
Is there a probblem?

This is how it looks atm:

root@server1:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 26G 388K 26G 1% /run
/dev/mapper/pve-root 95G 1.8G 88G 2% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 51G 22M 51G 1% /run/shm
/dev/mapper/pve-data 2.4T 858G 1.5T 36% /var/lib/vz
/dev/sda2 494M 35M 434M 8% /boot
/dev/mapper/intel480gb-lvssd480gb 434G 312G 100G 76% /mnt/disks/intel480
/dev/mapper/intel480gb2-lvssd480gb2 434G 3.6G 408G 1% /mnt/disks/intel480gb2
/dev/fuse 30M 20K 30M 1% /etc/pve
/mnt/disks/intel480/private/102 400G 311G 90G 78% /var/lib/vz/root/102
/mnt/disks/intel480gb2/private/104 100G 3.5G 97G 4% /var/lib/vz/root/104
none 4.2G 4.0K 4.2G 1% /var/lib/vz/root/104/dev
none 4.0K 0 4.0K 0% /var/lib/vz/root/104/sys/fs/cgroup
none 852M 1.1M 851M 1% /var/lib/vz/root/104/run
none 5.0M 0 5.0M 0% /var/lib/vz/root/104/run/lock
none 4.2G 0 4.2G 0% /var/lib/vz/root/104/run/shm
none 100M 0 100M 0% /var/lib/vz/root/104/run/user
/var/lib/vz/private/105 100G 3.4G 97G 4% /var/lib/vz/root/105
none 2.3G 4.0K 2.3G 1% /var/lib/vz/root/105/dev
/dev/mapper/wd3tb-lvwd3tb 2.7T 470G 2.1T 19% /mnt/disks/wd3tb
/mnt/disks/intel480/private/100 4.0G 1.1G 3.0G 27% /var/lib/vz/root/100
none 256M 4.0K 256M 1% /var/lib/vz/root/100/dev
none 4.0K 0 4.0K 0% /var/lib/vz/root/100/sys/fs/cgroup
none 52M 1.1M 51M 2% /var/lib/vz/root/100/run
none 5.0M 0 5.0M 0% /var/lib/vz/root/100/run/lock
none 256M 0 256M 0% /var/lib/vz/root/100/run/shm
none 100M 0 100M 0% /var/lib/vz/root/100/run/user
/var/lib/vz/private/106 100G 30G 71G 30% /var/lib/vz/root/106
none 2.3G 4.0K 2.3G 1% /var/lib/vz/root/106/dev
/dev/mapper/intel480gb-vzsnap--server1--0 434G 286G 126G 70% /mnt/vzsnap0
/var/lib/vz/private/101 500G 337G 164G 68% /var/lib/vz/root/101
 
Ok. Then there is no problem since /dev/sda is used as a block device for LVM.

What output do you get from the following command: # pveperf
 
Hi, I wonder if the problem is not that you use 4K sector disks
I have already seen use reporting bad behaviour, mainly with cache=none or cache=writeback.

here my fdisk

# fdisk -lu /dev/sda


Disk /dev/sda: 72.7 GB, 72746008576 bytes
255 heads, 63 sectors/track, 8844 cylinders, total 142082048 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000bb0cd


Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1048575 523264 83 Linux
/dev/sda2 1048576 142082047 70516736 8e Linux LVM
Click to expand...

(sda1 begin at 2048, which is aligned with 512bytes sector.

I don't known of 4K disk, but you should align guest, and host partiitions to 4K sector.
 
So heres my pveperf:

root@server1:~# pveperf
CPU BOGOMIPS: 166389.92
REGEX/SECOND: 546091
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 30.60 MB/sec
AVERAGE SEEK TIME: 13.67 ms
FSYNCS/SECOND: 106.28
DNS EXT: 88.38 ms
DNS INT: 121.63 ms (whitehouse.gov)

And when I did the parted:

i did across partition 1 through 3 and they all came out saying "aligned"
(parted) align-check
alignment type(min/opt) [optimal]/minimal?
Partition number? 1
1 aligned
(parted) align-check
alignment type(min/opt) [optimal]/minimal?
Partition number? 2
2 aligned
(parted) align-check
alignment type(min/opt) [optimal]/minimal?
Partition number? 3
3 aligned
 
Jae Lee said:
So heres my pveperf:

root@server1:~# pveperf
CPU BOGOMIPS: 166389.92
REGEX/SECOND: 546091
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 30.60 MB/sec
AVERAGE SEEK TIME: 13.67 ms
FSYNCS/SECOND: 106.28
DNS EXT: 88.38 ms
DNS INT: 121.63 ms (whitehouse.gov)
Click to expand...
Alignment is not your problem since LVM by default aligns starting file system from 2048.

Your problem is that you have very low FSYNCS/SECOND. Recommended minimum for FSYNCS/SECOND is 700.
What kind of file system and mount options do you have?

This can be displayed using: cat /proc/mounts
 
Here is the cat /proc/mounts:

root@server1:~# cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=33012938,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=26412280k,mode=755 0 0
/dev/mapper/pve-root / ext3 rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=52824540k 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/mapper/pve-data /var/lib/vz ext3 ro,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered 0 0
/dev/sda2 /boot ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered 0 0
/dev/mapper/intel480gb-lvssd480gb /mnt/disks/intel480 ext4 rw,noatime,nodiratime,errors=remount-ro,barrier=1,data=ordered,discard 0 0
/dev/mapper/intel480gb2-lvssd480gb2 /mnt/disks/intel480gb2 ext4 rw,noatime,nodiratime,errors=remount-ro,barrier=1,data=ordered,discard 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
/dev/fuse /etc/pve fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
beancounter /proc/vz/beancounter cgroup rw,relatime,blkio,name=beancounter 0 0
container /proc/vz/container cgroup rw,relatime,freezer,devices,name=container 0 0
fairsched /proc/vz/fairsched cgroup rw,relatime,cpuacct,cpu,cpuset,name=fairsched 0 0
/mnt/disks/intel480/private/102 /var/lib/vz/root/102 simfs rw,relatime 0 0
proc /var/lib/vz/root/102/proc proc rw,relatime 0 0
sysfs /var/lib/vz/root/102/sys sysfs rw,relatime 0 0
none /var/lib/vz/root/102/dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0
/mnt/disks/intel480gb2/private/104 /var/lib/vz/root/104 simfs rw,relatime 0 0
proc /var/lib/vz/root/104/proc proc rw,relatime 0 0
sysfs /var/lib/vz/root/104/sys sysfs rw,relatime 0 0
none /var/lib/vz/root/104/dev devtmpfs rw,nosuid,noexec,relatime,size=4358144k,nr_inodes=1089536,mode=755 0 0
none /var/lib/vz/root/104/dev/pts devpts rw,nosuid,noexec,relatime,mode=600,ptmxmode=000 0 0
none /var/lib/vz/root/104/proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
none /var/lib/vz/root/104/sys/fs/cgroup tmpfs rw,relatime,size=4k,nr_inodes=1089536,mode=755 0 0
none /var/lib/vz/root/104/run tmpfs rw,nosuid,noexec,relatime,size=871632k,nr_inodes=1089536,mode=755 0 0
none /var/lib/vz/root/104/run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=1089536 0 0
none /var/lib/vz/root/104/run/shm tmpfs rw,relatime,size=4358144k,nr_inodes=1089536 0 0
none /var/lib/vz/root/104/run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,nr_inodes=1089536,mode=755 0 0
/var/lib/vz/private/105 /var/lib/vz/root/105 simfs rw,relatime 0 0
proc /var/lib/vz/root/105/proc proc rw,relatime 0 0
sysfs /var/lib/vz/root/105/sys sysfs rw,relatime 0 0
none /var/lib/vz/root/105/dev tmpfs rw,relatime,size=2310144k,nr_inodes=577536,mode=755 0 0
none /var/lib/vz/root/105/dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /var/lib/vz/root/105/proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/wd3tb-lvwd3tb /mnt/disks/wd3tb ext4 rw,relatime,barrier=1,data=ordered 0 0
/mnt/disks/intel480/private/100 /var/lib/vz/root/100 simfs rw,relatime 0 0
proc /var/lib/vz/root/100/proc proc rw,relatime 0 0
sysfs /var/lib/vz/root/100/sys sysfs rw,relatime 0 0
none /var/lib/vz/root/100/dev devtmpfs rw,nosuid,noexec,relatime,size=262144k,nr_inodes=65536,mode=755 0 0
none /var/lib/vz/root/100/dev/pts devpts rw,nosuid,noexec,relatime,mode=600,ptmxmode=000 0 0
none /var/lib/vz/root/100/proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
none /var/lib/vz/root/100/sys/fs/cgroup tmpfs rw,relatime,size=4k,nr_inodes=65536,mode=755 0 0
none /var/lib/vz/root/100/run tmpfs rw,nosuid,noexec,relatime,size=52432k,nr_inodes=65536,mode=755 0 0
none /var/lib/vz/root/100/run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=65536 0 0
none /var/lib/vz/root/100/run/shm tmpfs rw,relatime,size=262144k,nr_inodes=65536 0 0
none /var/lib/vz/root/100/run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,nr_inodes=65536,mode=755 0 0
/var/lib/vz/private/106 /var/lib/vz/root/106 simfs rw,relatime 0 0
proc /var/lib/vz/root/106/proc proc rw,relatime 0 0
sysfs /var/lib/vz/root/106/sys sysfs rw,relatime 0 0
none /var/lib/vz/root/106/dev tmpfs rw,relatime,size=2310144k,nr_inodes=577536,mode=755 0 0
none /var/lib/vz/root/106/dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /var/lib/vz/root/106/proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
/dev/mapper/intel480gb-vzsnap--server1--0 /mnt/vzsnap0 ext4 rw,relatime,barrier=1,data=ordered 0 0
/var/lib/vz/private/101 /var/lib/vz/root/101 simfs rw,relatime 0 0
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom