NFS share not working only for CTs (fresh install)

[kablubr]

I built a HA with 3 PROXMOX servers and a FreeBSD storage ZFS/NFS. (all last packages)
I am trying to run CTs and VMs.
The KVM VMs are working fine in the NFS storage with good performance.
But when I start any CT on a NFS share, it takes over 2 to 3 HOURS to come up.
When I enter the CT using "vzctl enter ID" just after starting it, I can see only 3 processes (main) loaded, then after 20 minutes I do "ps ax" again and can see one service more loaded, and one more after 20 min, so on...
Maybe there is a issue with ios limit or something like that?


Another strange detail, when looking into ioacct for that ct, there are no READ activity:

root@server01:~# cat /proc/bc/104/ioacct
read 0
write 151552
dirty 151552
cancel 0
missed 0
syncs_total 0
fsyncs_total 91
fdatasyncs_total 0
range_syncs_total 0
syncs_active 0
fsyncs_active 0
fdatasyncs_active 0
range_syncs_active 0
io_pbs 0
fuse_requests 0
fuse_bytes 0


Any ideas?

 

[ozgurerdogan]

I had similar issue, somehow nfs is slow. So I decided to SSHFS with automount to fstab and so far seems good. You may give it a try. And do not forget to add sshfs as directory. You may see steps here. But proxmox side is up to you. Not a hard work.

 

[kablubr]

I had similar issue, somehow nfs is slow. So I decided to SSHFS with automount to fstab and so far seems good. You may give it a try. And do not forget to add sshfs as directory. You may see steps here. But proxmox side is up to you. Not a hard work.

Hi thanks for the reply.

I did this test and it looses 50% performance.
NFS is copying at 99MB/s
SSHFS is copying at 45MB/s

if I start two DD at same time with nfs and sshfs
NFS 99MB/s
SSHFS 27MB/s

In this case performance is very important as I want to be running several VMs / CTs inside it.

I am just frustrated with this... why only CTs do this!?

Wondering if there exists any workarounds for this to work... I am melting my brain on internet for last days but could not find anything to fix this issue...

I am enjoying proxmox and only needs to fix this to start thinking in using it on prod env.

 

[ozgurerdogan]

I really have no idea. I saw some other thread in forum having nfs issues. But sshfs must not be slow as it is like ssh file transfer. What is your network speed? Any switch in between? Give scp file copy a try to see speed.

 

[kablubr]

scp is slower... 40MB/s most

 

[ozgurerdogan]

scp is slower... 40MB/s most

What is your ethernet card and harddisk? And storage model?

 

[kablubr]

What is your ethernet card and harddisk? And storage model?

ether card is an intel fiber 2 ports

storage model is freebsd ZFS (sync off) + NFS share

Anyway the performance should not be different from VM to CT right?

 

[mir]

Try enabling compression on ZFS pool.

How much RAM has the FreeBSD storage?
Number of CPU's and Mhz.
Disk type - SATA, SAS, SSD?
What kind of zpool to you have - mirror, raidz etc?
Number of vdevs and number of disks per vdev?

You should consider Omnios instead of FreeBSD.

 

[kablubr]

Try enabling compression on ZFS pool.

How much RAM has the FreeBSD storage?
Number of CPU's and Mhz.
Disk type - SATA, SAS, SSD?
What kind of zpool to you have - mirror, raidz etc?
Number of vdevs and number of disks per vdev?

You should consider Omnios instead of FreeBSD.

root@server1:~ # zpool status
  pool: mainpool
 state: ONLINE
  scan: scrub repaired 0 in 0h6m with 0 errors on Mon Oct 21 18:42:16 2013
config:

        NAME            STATE     READ WRITE CKSUM
        mainpool        ONLINE       0     0     0
          raidz1-0      ONLINE       0     0     0
            hast/disk1  ONLINE       0     0     0
            hast/disk2  ONLINE       0     0     0
            hast/disk3  ONLINE       0     0     0
            hast/disk4  ONLINE       0     0     0
            hast/disk5  ONLINE       0     0     0
            hast/disk6  ONLINE       0     0     0
            hast/disk7  ONLINE       0     0     0

errors: No known data errors
root@server1:~ # zfs get compression
NAME              PROPERTY     VALUE     SOURCE
mainpool          compression  lz4       local
mainpool/data  compression  lz4       inherited from mainpool

compression already set (tested with and without)
7 SAS 3GB disks
Dell 840
3Ghz cpu
4GB memory

 

[mir]

You should definitely add more RAM 8GB is an absolute minimum. I would recommend at least 16GB RAM.

You have also chosen the absolute slowest zpool layout provided by ZFS. I would recommend this:

mainpool
    mirror-0
        hast/disk1
        hast/disk2
    mirror-1
        hast/disk3
        hast/disk4
    mirror-2
        hast/disk5
        hast/disk6
spares
    hast/disk7

 

[kablubr]

You should definitely add more RAM 8GB is an absolute minimum. I would recommend at least 16GB RAM.

You have also chosen the absolute slowest zpool layout provided by ZFS. I would recommend this:

mainpool
    mirror-0
        hast/disk1
        hast/disk2
    mirror-1
        hast/disk3
        hast/disk4
    mirror-2
        hast/disk5
        hast/disk6
spares
    hast/disk7

I understand your point on adding more performance to ZFS filesystem. But with the already 100MB/s I have, it should be enough for me.
The problem is that VMs (KVM) can reach this 100MB/S performance on a storage, but CTs are taking forever to load/run, almost frozen.
I can start and stop a lot of VMs while the simple CT is still loading, on the same storage.

 

[mir]

You will soon see degraded performance when only a few VM's run concurrently which is generating IO's. Besides RAM is cheap these days.
I would personally never use a ZFS storage with your amount of RAM for anything else than a personal desktop.

 

[kablubr]

You will soon see degraded performance when only a few VM's run concurrently which is generating IO's. Besides RAM is cheap these days.
I would personally never use a ZFS storage with your amount of RAM for anything else than a personal desktop.

For production I will increase memory, but now I need to fix this CT problem, so I can get the funding approved to buy new servers.