NFS freezes Proxmox WebGUI and fails to backup

M

Maciej Bursztynowski

New Member
Jun 26, 2015
1
0
1
It's my first topic, so I want to say hello to everyone.

I've got weird issue with NFS.

I've installed Proxmox 3.4 on my OVH dedicated server. It works quite perfect, VMs works very fast but I've got weird issue with NFS.
When I connect NFS Share via WebGUI as NFS share or mount NFS share in console + adding as local storage It freezes Proxmox WebGui during backup task.
Sometimes it unfreezes, but I can't check summary and status of this share during backup.

Next problem - backup fails:
I've got 3 VMs. 2 of them have small disks ~20GB and backup to NFS storage finishes successfully every time, but when I want to backup VM which disk is approx. 256GB it fails over and over. If I choose local storage and try to backup it, backup is finished successfully.

Have anyone met with such a problem?
 
Proxmox 3.4 and nfs very bad idea. I saw too much forum posts with this bug. Why developers doesn't fixing this bug, i dont know.

i have same issue too.
i tried to mount nfs share with soft option, but this didnt help. Next i try mount remote server with SSFS (fuse driver) and this really works!!!

Now, i dont use and i will not use nfs in proxmox cluster, only sshfs
 
Hi Maciej,

I have a similar issue with OVH. When I backup a few GB there's no problem, but when I try to backup several VMs (about 300-400GB), sometimes the entire NFS Share (BackupStorage) becomes unresponsive, causing a high IO delay on the hypervisor, which leads to an unresponsive VM.

Sometimes, even the entire Proxmox web interface turns unresponsive, due to NFS drops.

I opened a ticket to OVH, but usually they say that there's no problem with Backup Storage service. I'm not happy with OVH. While dedicated servers and management interface is really good, mail/telephone customer service (and the BackupStorage service) leaves much to be desired, in my opinion. Anyone can provide an OVH alternative?

Regards,
 
Still facing such NFS issues with Proxmox VE 4.4.. :(

Code: 
Jun 17 01:40:05 vm-box-1 kernel: INFO: task lzop:1093 blocked for more than 120 seconds.
Jun 17 01:40:05 vm-box-1 kernel:       Tainted: P        W  O    4.4.49-1-pve #1
Jun 17 01:40:05 vm-box-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 17 01:40:05 vm-box-1 kernel: lzop            D ffff88014e10fa48     0  1093   1075 0x00000000
Jun 17 01:40:05 vm-box-1 kernel:  ffff88014e10fa48 0000000000017300 ffff88081aaaad00 ffff88010500bc00
Jun 17 01:40:05 vm-box-1 kernel:  ffff88014e110000 ffff88083ed17300 7fffffffffffffff ffffffff8185cb00
Jun 17 01:40:05 vm-box-1 kernel:  ffff88014e10fba8 ffff88014e10fa60 ffffffff8185c215 0000000000000000
Jun 17 01:40:05 vm-box-1 kernel: Call Trace:
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185cb00>] ? bit_wait_timeout+0xa0/0xa0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185c215>] schedule+0x35/0x80
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185f465>] schedule_timeout+0x235/0x2d0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185cb00>] ? bit_wait_timeout+0xa0/0xa0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185b70b>] io_schedule_timeout+0xbb/0x140
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185cb1b>] bit_wait_io+0x1b/0x70
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185c5cf>] __wait_on_bit+0x5f/0x90
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185cb00>] ? bit_wait_timeout+0xa0/0xa0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8185c681>] out_of_line_wait_on_bit+0x81/0xb0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff810c45d0>] ? autoremove_wake_function+0x40/0x40
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffffc0960c54>] nfs_wait_on_request+0x34/0x40 [nfs]
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffffc09659ed>] nfs_updatepage+0x1bd/0x9f0 [nfs]
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffffc0955dbf>] nfs_write_end+0x14f/0x4d0 [nfs]
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8140d97f>] ? iov_iter_copy_from_user_atomic+0x8f/0x230
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8118eb64>] generic_perform_write+0x114/0x1c0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff81190e06>] __generic_file_write_iter+0x1a6/0x1f0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8122c653>] ? touch_atime+0x33/0xd0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff81190f34>] generic_file_write_iter+0xe4/0x1e0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffffc095553a>] nfs_file_write+0x9a/0x160 [nfs]
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8120eafb>] new_sync_write+0x9b/0xe0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8120eb66>] __vfs_write+0x26/0x40
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8120f1d9>] vfs_write+0xa9/0x190
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff8120ffb5>] SyS_write+0x55/0xc0
Jun 17 01:40:05 vm-box-1 kernel:  [<ffffffff81860336>] entry_SYSCALL_64_fastpath+0x16/0x75
Jun 17 01:40:26 vm-box-1 pvestatd[1519]: status update time (105.333 seconds)
Jun 17 01:40:58 vm-box-1 pvestatd[1519]: got timeout

Code: 
root@vm-box-1:/var/log# pveversion -v
proxmox-ve: 4.4-86 (running kernel: 4.4.49-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.49-1-pve: 4.4.49-86
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-49
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-94
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-97
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80

Any chances to have this resolved?
 
carles89 said:
It seems to be an issue with your NFS server. Are you using OVH?
Click to expand...
No, I don't use OVH. My NFS server is CentOS 7 box with data storage on two drives managed by ZFS mirror. Proxmox and CentOS boxes are in the same 1Gb LAN.

NFS server:
Code: 
[root@filer-1 ~]# exportfs -v
/zmir/backup/images
                vm-box-*.dom.com(rw,async,wdelay,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

Proxmox clients:
Code: 
root@vm-box-1:/var/log# mount
...
filer-1.dom.com:/zmir/backup/images on /mnt/pve/filer-1-images type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.26.1.31,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=172.26.1.31)

I've just changed the scheduler to deadline (thought I've done that before, but it appears i haven't or it was reset to noop somehow). Not sure if it helps, though.

Code: 
[root@filer-1 etc]# cat /sys/block/sd{a,b}/queue/scheduler
noop [deadline] cfq
noop [deadline] cfq
 
Hi jazzl0ver,

It seems that NFS communication stops, hanging vzdump. It can be because of a bad network connection, or NFS' server bad performance. Have you tried with mdadm instead of zfs? ZFS uses lots of ram for caching, and if yor hardware is not enough it may be the bottleneck.

In my case, the problem was some kind of QoS applied to the network, after transferring X GB to the backup destination, network hanged after during some time, making everything unstable. We solved that by changing to a dedicated network and server for backups.
 
@carles89, thanks for you reply and wishing to help me!

I don't think it's a networking issue. The switch port is loaded for less than 100 Mbit/s during the backup (while it's 1Gb port). This indeed might be the NFS server issue. I haven't tried with mdadm, since ZFS seemed to me more preferred due to its compression and dedup capabilities.
The system has 16GB of RAM:
Code: 
[root@filer-1 etc]# free -m
              total        used        free      shared  buff/cache   available
Mem:          15847        8635        5436         239        1775        5618
Swap:             0           0           0
So, I don't think it's RAM issue.
However, the i/o raises up to 86% during the backup:
Code: 
09:00:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
...
09:40:01 PM     all      6.07      0.00     13.56     44.09      0.00     36.28
09:50:01 PM     all      6.22      0.00      7.33     48.03      0.00     38.42
10:00:01 PM     all      5.80      0.00      6.67     53.73      0.00     33.80
10:10:01 PM     all      6.91      0.00      8.92     52.28      0.00     31.90
10:20:01 PM     all      6.89      0.00      8.10     48.02      0.00     36.99
10:30:01 PM     all      6.66      0.00      8.75     45.83      0.00     38.76
10:40:01 PM     all      1.34      0.00      5.90     55.32      0.00     37.44
10:50:01 PM     all      0.97      0.00      5.79     64.41      0.00     28.83
11:00:01 PM     all      1.32      0.00      5.59     61.02      0.00     32.06
11:10:01 PM     all      1.19      0.00      5.41     63.69      0.00     29.72
11:20:01 PM     all      4.26      0.00      7.87     59.43      0.00     28.44
11:30:01 PM     all      3.21      0.00      4.40     86.98      0.00      5.41
11:40:01 PM     all      3.15      0.00      4.50     87.74      0.00      4.61
11:50:01 PM     all      4.15      0.00      5.46     86.19      0.00      4.19
which might lead to NFS server unresponsiveness. Wondering if there's a way to tune the NFS server somehow to avoid such issues..
 
@jazzlOver
1.
Having I/O wait just means: storage is the currently bottleneck for 44 % of you overall processes.
When you have processes, like backup jobs, who do more reading/writing that computing this processes will always be waiting for IO.

For instance this command will produce I/O wait on anystorage, simply because reading from RAM is faster that writing to a storage.
dd if=/dev/zero of=/root/bla bs=1M
(exept if your storage is RAM based of course ;))

2.
the dmesg you sent with "task blocked for more than 120s" is a warning that the kernel noticed that a process could not do the I/O operation it was scheduled to do in a 120s time slot.
It can either mean than the storage is extremely busy, or simply offline in the case of a NFS server.

if the nfs server is not online, pve detects this and will log it.
you can check if storages where noticed as not online with:
journalctl -u pvestatd | grep 'not online'

now for the extremely busy part, shutdown your VMs and benchmark the storage !
for comparison purpose I got 125MB/s sequential write of the NFS server of my testlab, meaning that I a saturating the Gigabit Ethernet Link here
 
Hi @manu. Thanks for your response.

The point is that NFS server is online it just seems to get frozen due to high i/o during backup and thus stops responding to nfs client's fsstat RPC command. So I'm wondering if there's a tweak that could tune up the server's responsiveness. This is not a Proxmox issue, I know.

Benchmarking ZFS is not simple, since it does not support directio - all workload appears to be cached:
Code: 
filebench> load singlestreamwrite
27895: 9.689: Single Stream Write Version 3.0 personality successfully loaded
filebench> set $dir=/zmir/legacy/bench
filebench> run 60
27895: 19.904: Creating/pre-allocating files and filesets
27895: 19.904: File largefile1: 0.000MB
27895: 19.906: Removed any existing file largefile1 in 1 seconds
27895: 19.906: making tree for filset /zmir/legacy/bench/largefile1
27895: 19.907: Creating file largefile1...
27895: 19.907: Preallocated 1 of 1 of file largefile1 in 1 seconds
27895: 19.907: waiting for fileset pre-allocation to finish
27903: 19.907: Starting 1 seqwrite instances
27904: 19.907: Starting 1 seqwrite threads
27895: 20.907: Running...
27895: 80.926: Run took 60 seconds...
27895: 80.926: Per-Operation Breakdown
seqwrite             239261ops     3986ops/s 3986.4mb/s      0.2ms/op      193us/op-cpu [0ms - 75ms]
27895: 80.926: IO Summary: 239261 ops, 3986.456 ops/s, (0/3986 r/w), 3986.4mb/s,    521us cpu/op,   0.2ms latency

filebench> load fileserver
27678: 11.207: File-server Version 3.0 personality successfully loaded
filebench> set $dir=/zmir/legacy/bench
filebench> run 60
27678: 55.093: Creating/pre-allocating files and filesets
27678: 55.103: Fileset bigfileset: 10000 files, 0 leafdirs, avg dir width = 20, avg dir depth = 3.1, 1240.757MB
27678: 55.105: Removed any existing fileset bigfileset in 1 seconds
27678: 55.105: making tree for filset /zmir/legacy/bench/bigfileset
27678: 55.118: Creating fileset bigfileset...
27678: 57.948: Preallocated 7979 of 10000 of fileset bigfileset in 3 seconds
27678: 57.948: waiting for fileset pre-allocation to finish
27681: 57.948: Starting 1 filereader instances
27682: 57.949: Starting 50 filereaderthread threads
27678: 58.983: Running...
27678: 118.987: Run took 60 seconds...
27678: 118.990: Per-Operation Breakdown
statfile1            108457ops     1808ops/s   0.0mb/s      0.0ms/op     2274us/op-cpu [0ms - 4ms]
deletefile1          108460ops     1808ops/s   0.0mb/s      0.3ms/op     2678us/op-cpu [0ms - 264ms]
closefile3           108465ops     1808ops/s   0.0mb/s      0.0ms/op     2273us/op-cpu [0ms - 1ms]
readfile1            108469ops     1808ops/s 237.7mb/s      0.5ms/op     3213us/op-cpu [0ms - 200ms]
openfile2            108473ops     1808ops/s   0.0mb/s      0.0ms/op     2232us/op-cpu [0ms - 11ms]
closefile2           108476ops     1808ops/s   0.0mb/s      0.0ms/op     2274us/op-cpu [0ms - 1ms]
appendfilerand1      108479ops     1808ops/s  14.1mb/s      0.4ms/op     3131us/op-cpu [0ms - 236ms]
openfile1            108481ops     1808ops/s   0.0mb/s      0.0ms/op     2262us/op-cpu [0ms - 11ms]
closefile1           108488ops     1808ops/s   0.0mb/s      0.0ms/op     2256us/op-cpu [0ms - 5ms]
wrtfile1             108495ops     1808ops/s 223.7mb/s      0.1ms/op     2410us/op-cpu [0ms - 98ms]
createfile1          108497ops     1808ops/s   0.0mb/s      0.3ms/op     2659us/op-cpu [0ms - 99ms]
27678: 118.990: IO Summary: 1193240 ops, 19886.104 ops/s, (1808/3616 r/w), 475.5mb/s,    437us cpu/op,   0.6ms latency
 
> This is not a Proxmox issue, I know

Thank you. Many people assume their storage problem to be Virtualization problems :)

I am benchmarking ZFS with fio and the following command line

fio --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting | grep bw

and I make sure I disable the cache before:
sudo zfs get all | grep cache
tank/backup primarycache none local
tank/backup secondarycache none local

but you should be doing the benchmark from the PVE side on the NFS mount

I don't know what you have on your ZFS server, but if with mechanical hard drives, you should think of a SSD slog
most ZFS performances I have seen was the mechanical hard drives could not cope syncing the content of the transaction log to the disks
you notice this on Linux ZFS boxes with tgx_sync process taking all the io bandwith with iotop
 
Here are the results when the cache is off (with and without directio):
Proxmox host:
Code: 
root@vm-box-1:/mnt/pve/filer-1-images# fio --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting | grep bw
  write: io=2352.7MB, bw=40141KB/s, iops=627, runt= 60016msec
    bw (KB  /s): min= 3984, max=65280, per=100.00%, avg=40415.02, stdev=20285.91

root@vm-box-1:/mnt/pve/filer-1-images# fio --size=9G --bs=64k --rw=write --runtime=60 --name=64kwrite --group_reporting | grep bw     write: io=3923.5MB, bw=66921KB/s, iops=1045, runt= 60035msec
    bw (KB  /s): min=  211, max=3027297, per=100.00%, avg=71287.87, stdev=333613.91

NFS server:
Code: 
[root@filer-1 legacy]# fio --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting | grep bw
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT

[root@filer-1 legacy]# fio --size=9G --bs=64k --rw=write --runtime=60 --name=64kwrite --group_reporting | grep bw
  write: io=1647.8MB, bw=26526KB/s, iops=414, runt= 63610msec
    bw (KB  /s): min=   11, max=2165248, per=100.00%, avg=52684.16, stdev=270808.40
 
Last edited:
26526KB ->25MB/s is really not that fast I got better throughput with a laptop disk connected over usb (35 MB/s)
 
Sorry for the delay. Yeah, it does not look speedy. It might be due to ZFS dedup and compression enabled. But this must not affect the latency when deadline scheduler is selected (if I'm not mistaken). Do you have any ideas how to make the NFS server to respond fast?
 
Just for the records. I forgot that the NFS server had a running Syncthing service, which started to sync the folder I used for fio (my bad). Here are the results after turning it off:
Code: 
root@vm-box-1:/mnt/pve/filer-1-images# fio --size=9G --bs=64k --rw=write --direct=1 --runtime=60 --name=64kwrite --group_reporting | grep bw
  write: io=2838.8MB, bw=48446KB/s, iops=756, runt= 60001msec
    bw (KB  /s): min=14464, max=64896, per=100.00%, avg=48617.77, stdev=15286.39

root@vm-box-1:/mnt/pve/filer-1-images# fio --size=9G --bs=64k --rw=write --runtime=60 --name=64kwrite --group_reporting | grep bw
  write: io=3809.2MB, bw=64959KB/s, iops=1014, runt= 60046msec
    bw (KB  /s): min=  427, max=3096047, per=100.00%, avg=68524.05, stdev=317124.94

[root@filer-1 legacy]# fio --size=9G --bs=64k --rw=write --runtime=60 --name=64kwrite --group_reporting | grep bw
  write: io=2718.6MB, bw=46370KB/s, iops=724, runt= 60035msec
    bw (KB  /s): min=   56, max=2086016, per=100.00%, avg=48791.51, stdev=225312.56
 
OK these numbers are a bit better but still not that expectional, so no wonder your backups are taking time
there is not magic making a fast NFS server apart from using SSDs und 10G/e ethernet :)

do you still have the messages "task blocked for more than 120s ? I would suggest to look at the timestamps when those happen and see if you can correlate that with an I/O spike on the NFS side
 
Well.. Actually I don't complain for backup speeds :)
Code: 
INFO: transferred 85899 MB in 480 seconds (178 MB/s)
INFO: archive file size: 260MB
INFO: Finished Backup of VM 107 (00:08:02)
The other day/time:
Code: 
INFO: transferred 311385 MB in 5490 seconds (56 MB/s)
INFO: archive file size: 27.85GB
INFO: Finished Backup of VM 124 (01:31:33)
So, the backups itself work pretty good. The issue arrives when I try to get disk counters by snmp or simple "df -h" command while backup is working - there's a significant delay before I get the output which I'd like to reduce.
The kernel's "INFO: task lzop:9045 blocked for more than 120 seconds" from my 1st post still happens but very seldom (last time it was on Jun 29th). However, I still observe "kernel: nfs: server filer-1 not responding" and "pvestatd: got timeout" errors during backups.
 
Just found out that the i/o scheduler had been reset to noop:
Code: 
[root@filer-1 ~]# cat /sys/block/sd{a,b}/queue/scheduler
[noop] deadline cfq
[noop] deadline cfq
Wondering how that's possible..

The only suggestion it was tuned - its profile was set to "balanced". Changed to "latency-performance".
It's ZFS. It uses its own scheduler and sets the system's one to noop (if the whole drive is used for ZFS). That's why the NFS server freezes during intensive i/o - ZFS doesn't share i/o with it.

Posted a FR: https://github.com/zfsonlinux/zfs/issues/6322
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back