[solved] NFS performance and mount options

m.ardito · Nov 26, 2013

Hi,

I have an old NFS sharing NAS which is performing good, since years, not excellent but good for its job.
Now I have a new NFS sharing NAS which is really slow...

old NAS specs: qnap ts809-rp, 8x 1.5tb SAMSUNG HD154UI 1AG0 in 1 raid5 vol, Core 2 Duo 2.8 GHz, 2GB DDRII RAM, 1GB eth MTU 1500
new NAS specs: qnap ts879-rp, 7x 2.0tb Hitachi HUA723020ALA640 MK7O in 1 raid5 vol, Quad Core Xeon® E3-1225 v2 Processor 3.2 GHz, 4 GB DDR3 ECC RAM, 1GB eth MTU 1500

i've run pveperf test on both, from two cluster nodes, and got these results:

Code:

# pveperf /mnt/pve/ts809/ (old NAS)
CPU BOGOMIPS:      72527.28
REGEX/SECOND:      947321
HD SIZE:           9617.06 GB (ts809:/myshare)
FSYNCS/SECOND:     [B]1740.50[/B]
DNS EXT:           57.34 ms
DNS INT:           1.33 ms 

# pveperf /mnt/pve/ts879/ (new NAS)
CPU BOGOMIPS:      72527.28
REGEX/SECOND:      948892
HD SIZE:           11081.12 GB (ts879:/myshare)
FSYNCS/SECOND:     [B]204.80[/B]
DNS EXT:           135.45 ms
DNS INT:           0.67 ms

this puzzled me a bit.. I know these are sw raid units, but I expected the new NAS to perform far better than the old...
so I looked at mount options (I mounted those units through pve gui)

Code:

IP_of_ts809:/iso_images on /mnt/pve/ts809 type nfs (rw
relatime
vers=3
[B]rsize=262144
wsize=262144
[/B]namlen=255
hard
proto=tcp
timeo=600
retrans=2
sec=sys
mountaddr=IP_of_ts809
mountvers=3
mountport=659
mountproto=udp
local_lock=none
addr=IP_of_ts809)

IP_of_ts879:/Download on /mnt/pve/ts879 type nfs (rw
relatime
vers=3
[B]rsize=32768
wsize=32768
[/B]namlen=255
hard
proto=tcp
timeo=600
retrans=2
sec=sys
mountaddr=IP_of_ts879
mountvers=3
mountport=59669
mountproto=udp
local_lock=none
addr=IP_of_ts879)

I think rsize and wsize are involved... but how can I change them (and to which values, how to find)?
and why pve mounted those in such different ways?
if it can make a difference:
- ts809 was mounted in cluster on a node with old, 2.x before upgrade it to 3.1,
- ts879 was mounted in cluster on another node 2.x, after upgrading it to 3.1

Have you any hints of what is happening and how to use at best the two units? other tests?

I can run tests from cli on both NAS, but that is a custom qnap linux, there's no apt-get, and I don't want to mess something there, if possibile...

Thanks, Marco

mir · Nov 26, 2013

Re: NFS performance and mount options

The settings for rsize and wsize are normally negotiated between server and client when attaching the mount. You can try to put it in manually in your storage.conf file. No guaranty it will work though. I have a Qnap myself which also is limited to the same speed you see.

m.ardito · Nov 26, 2013

Re: NFS performance and mount options

mir said:
You can try to put it in manually in your storage.conf file. I have a Qnap myself which also is limited to the same speed you see.

do you mean you have same rsize/wsize and get the slower fsyncs/sec, around 100-200?
I can test changing settings in storage.conf but after that is the storage remounted automatically? Or what should be done?

btw, how can I test iscsi speeds also ? I have doubts there, too...

[update]
I'm not expert about linux hd performance testing, but checking with hdparm, I feel the new system should do much better... ?

Code:

# hdparm -Tt /dev/sda (old NAS)
/dev/sda:
 Timing cached reads:   4232 MB in  2.00 seconds = 2115.73 MB/sec
 Timing buffered disk reads:  286 MB in  3.02 seconds =  94.83 MB/sec

# hdparm -Tt /dev/sda (new NAS)
/dev/sda:
 Timing cached reads:   17024 MB in  2.00 seconds = 8520.09 MB/sec
 Timing buffered disk reads:  432 MB in  3.01 seconds = 143.50 MB/sec

[update2]
looking for more info, i thought about schedulers:

- pve has deadline
- old NAS has deadline
- new NAS has CFQ

again, I don't know if and how this can show problems...

Thanks, Marco

mir · Nov 26, 2013

Re: NFS performance and mount options

From my Qnap RAID1:
# pveperf /mnt/pve/qnap_nfs/
CPU BOGOMIPS: 32551.14
REGEX/SECOND: 1305031
HD SIZE: 1373.87 GB (192.168.2.10:/vz)
FSYNCS/SECOND: 750.60
DNS EXT: 61.89 ms
DNS INT: 0.72 ms (datanom.net)

Timing buffered disk reads: 432 MB in 3.01 seconds = 143.50 MB/sec
This seems very low for a 5 disk RAID5

mir · Nov 26, 2013

Re: NFS performance and mount options

Try with these mount options in /etc/pve/storage.conf
options vers=3,tcp,nolock

m.ardito · Nov 26, 2013

Re: NFS performance and mount options

now storage.conf is:

Code:

nfs: ts879
        path /mnt/pve/ts879
        server IP_of_ts879
        export /myshare
        options vers=3
        content backup
        maxfiles 5

mir said:
Try with these mount options in /etc/pve/storage.conf
options vers=3,tcp,nolock

I will do that, but
- should I add one option per line in storage.conf ?
- after changing storage.conf, how to remount?

in the meanwhile, my backup test of a running ubuntu vm with a 32gb virtio disk (running on lvm/iscsi) over the two different NFS:

backup over older NAS takes 12 mins, on new NAS takes over 1 hour 7 mins... that's six times slower...

Code:

INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --storage ts809 --node pve1
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: update VM 101: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/ts809/dump/vzdump-qemu-101-2013_11_26-15_05_26.vma.lzo'
INFO: started backup task '156ad725-f082-4119-8fee-b132d2c62b6d'
INFO: status: 0% (169934848/34359738368), sparse 0% (10653696), duration 3, 56/53 MB/s
INFO: status: 1% (369033216/34359738368), sparse 0% (49627136), duration 6, 66/53 MB/s
INFO: status: 2% (751894528/34359738368), sparse 0% (53833728), duration 12, 63/63 MB/s
INFO: status: 3% (1070071808/34359738368), sparse 0% (56184832), duration 17, 63/63 MB/s
INFO: status: 4% (1382744064/34359738368), sparse 0% (58163200), duration 22, 62/62 MB/s
INFO: status: 5% (1749417984/34359738368), sparse 0% (60375040), duration 28, 61/60 MB/s
INFO: status: 6% (2094399488/34359738368), sparse 0% (61734912), duration 34, 57/57 MB/s
INFO: status: 7% (2467430400/34359738368), sparse 0% (84578304), duration 52, 20/19 MB/s
INFO: status: 8% (2760638464/34359738368), sparse 0% (84791296), duration 59, 41/41 MB/s
INFO: status: 9% (3129409536/34359738368), sparse 0% (85172224), duration 66, 52/52 MB/s
INFO: status: 10% (3462266880/34359738368), sparse 0% (85831680), duration 73, 47/47 MB/s
INFO: status: 11% (3792437248/34359738368), sparse 0% (86208512), duration 81, 41/41 MB/s
INFO: status: 12% (4153540608/34359738368), sparse 0% (87142400), duration 90, 40/40 MB/s
INFO: status: 13% (4514643968/34359738368), sparse 0% (94822400), duration 97, 51/50 MB/s
INFO: status: 14% (4865458176/34359738368), sparse 0% (124706816), duration 103, 58/53 MB/s
INFO: status: 15% (5168758784/34359738368), sparse 0% (125489152), duration 108, 60/60 MB/s
INFO: status: 16% (5541920768/34359738368), sparse 0% (127991808), duration 114, 62/61 MB/s
INFO: status: 17% (5873598464/34359738368), sparse 0% (131952640), duration 128, 23/23 MB/s
INFO: status: 18% (6185680896/34359738368), sparse 0% (133529600), duration 134, 52/51 MB/s
INFO: status: 19% (6545604608/34359738368), sparse 0% (146927616), duration 142, 44/43 MB/s
INFO: status: 20% (6906576896/34359738368), sparse 0% (184082432), duration 153, 32/29 MB/s
INFO: status: 21% (7218921472/34359738368), sparse 0% (184365056), duration 164, 28/28 MB/s
INFO: status: 22% (7580942336/34359738368), sparse 0% (187367424), duration 172, 45/44 MB/s
INFO: status: 23% (7944011776/34359738368), sparse 0% (190857216), duration 178, 60/59 MB/s
INFO: status: 24% (8313110528/34359738368), sparse 0% (298573824), duration 184, 61/43 MB/s
INFO: status: 25% (8608940032/34359738368), sparse 1% (594403328), duration 189, 59/0 MB/s
INFO: status: 26% (8940355584/34359738368), sparse 2% (900947968), duration 200, 30/2 MB/s
INFO: status: 27% (9280225280/34359738368), sparse 3% (1077559296), duration 207, 48/23 MB/s
INFO: status: 28% (9645391872/34359738368), sparse 4% (1442725888), duration 214, 52/0 MB/s
INFO: status: 29% (10010689536/34359738368), sparse 5% (1808023552), duration 221, 52/0 MB/s
INFO: status: 30% (10326245376/34359738368), sparse 6% (2123579392), duration 227, 52/0 MB/s
INFO: status: 31% (10672537600/34359738368), sparse 7% (2469871616), duration 233, 57/0 MB/s
INFO: status: 32% (11030036480/34359738368), sparse 8% (2820005888), duration 240, 51/1 MB/s
INFO: status: 33% (11347427328/34359738368), sparse 8% (2962325504), duration 246, 52/29 MB/s
INFO: status: 34% (11688673280/34359738368), sparse 9% (3303571456), duration 252, 56/0 MB/s
INFO: status: 35% (12037980160/34359738368), sparse 10% (3652878336), duration 258, 58/0 MB/s
INFO: status: 36% (12392923136/34359738368), sparse 11% (4007821312), duration 264, 59/0 MB/s
INFO: status: 37% (12720603136/34359738368), sparse 11% (4048818176), duration 272, 40/35 MB/s
INFO: status: 38% (13083672576/34359738368), sparse 11% (4053639168), duration 278, 60/59 MB/s
INFO: status: 39% (13453754368/34359738368), sparse 11% (4084056064), duration 284, 61/56 MB/s
INFO: status: 40% (13804568576/34359738368), sparse 11% (4084760576), duration 290, 58/58 MB/s
INFO: status: 41% (14107148288/34359738368), sparse 11% (4085248000), duration 295, 60/60 MB/s
INFO: status: 42% (14438301696/34359738368), sparse 11% (4086247424), duration 307, 27/27 MB/s
INFO: status: 43% (14778564608/34359738368), sparse 11% (4087103488), duration 321, 24/24 MB/s
INFO: status: 44% (15120138240/34359738368), sparse 12% (4181983232), duration 327, 56/41 MB/s
INFO: status: 45% (15495266304/34359738368), sparse 12% (4330348544), duration 333, 62/37 MB/s
INFO: status: 46% (15813181440/34359738368), sparse 12% (4347613184), duration 338, 63/60 MB/s
INFO: status: 47% (16187129856/34359738368), sparse 13% (4551356416), duration 344, 62/28 MB/s
INFO: status: 48% (16502554624/34359738368), sparse 13% (4552105984), duration 358, 22/22 MB/s
INFO: status: 49% (16851271680/34359738368), sparse 13% (4554588160), duration 371, 26/26 MB/s
INFO: status: 50% (17185832960/34359738368), sparse 13% (4560953344), duration 378, 47/46 MB/s
INFO: status: 51% (17584488448/34359738368), sparse 13% (4605497344), duration 385, 56/50 MB/s
INFO: status: 52% (17903517696/34359738368), sparse 13% (4606824448), duration 391, 53/52 MB/s
INFO: status: 53% (18232705024/34359738368), sparse 13% (4609425408), duration 403, 27/27 MB/s
INFO: status: 54% (18558812160/34359738368), sparse 13% (4619018240), duration 411, 40/39 MB/s
INFO: status: 55% (18911789056/34359738368), sparse 13% (4619542528), duration 421, 35/35 MB/s
INFO: status: 56% (19280297984/34359738368), sparse 13% (4619653120), duration 432, 33/33 MB/s
INFO: status: 57% (19594477568/34359738368), sparse 13% (4625125376), duration 439, 44/44 MB/s
INFO: status: 58% (19938279424/34359738368), sparse 13% (4665532416), duration 448, 38/33 MB/s
INFO: status: 59% (20324155392/34359738368), sparse 13% (4667133952), duration 455, 55/54 MB/s
INFO: status: 60% (20621885440/34359738368), sparse 13% (4667461632), duration 460, 59/59 MB/s
INFO: status: 61% (20993277952/34359738368), sparse 13% (4669468672), duration 469, 41/41 MB/s
INFO: status: 62% (21348679680/34359738368), sparse 13% (4670029824), duration 475, 59/59 MB/s
INFO: status: 63% (21663776768/34359738368), sparse 13% (4677591040), duration 482, 45/43 MB/s
INFO: status: 64% (22006136832/34359738368), sparse 13% (4786593792), duration 499, 20/13 MB/s
INFO: status: 65% (22354395136/34359738368), sparse 13% (4797280256), duration 507, 43/42 MB/s
INFO: status: 66% (22725984256/34359738368), sparse 13% (4798484480), duration 513, 61/61 MB/s
INFO: status: 67% (23045472256/34359738368), sparse 14% (4978364416), duration 519, 53/23 MB/s
INFO: status: 68% (23409917952/34359738368), sparse 14% (5013524480), duration 525, 60/54 MB/s
INFO: status: 69% (23747624960/34359738368), sparse 14% (5119418368), duration 536, 30/21 MB/s
INFO: status: 70% (24086511616/34359738368), sparse 15% (5319946240), duration 542, 56/23 MB/s
INFO: status: 71% (24441978880/34359738368), sparse 16% (5556514816), duration 548, 59/19 MB/s
INFO: status: 72% (24766513152/34359738368), sparse 17% (5881049088), duration 553, 64/0 MB/s
INFO: status: 73% (25104482304/34359738368), sparse 18% (6219018240), duration 561, 42/0 MB/s
INFO: status: 74% (25457328128/34359738368), sparse 19% (6571864064), duration 575, 25/0 MB/s
INFO: status: 75% (25786843136/34359738368), sparse 20% (6901379072), duration 581, 54/0 MB/s
INFO: status: 76% (26131234816/34359738368), sparse 21% (7216664576), duration 588, 49/4 MB/s
INFO: status: 77% (26485325824/34359738368), sparse 21% (7379742720), duration 594, 59/31 MB/s
INFO: status: 78% (26842497024/34359738368), sparse 22% (7736913920), duration 601, 51/0 MB/s
INFO: status: 79% (27176468480/34359738368), sparse 23% (8070885376), duration 611, 33/0 MB/s
INFO: status: 80% (27527610368/34359738368), sparse 24% (8422027264), duration 617, 58/0 MB/s
INFO: status: 81% (27881832448/34359738368), sparse 25% (8776249344), duration 623, 59/0 MB/s
INFO: status: 82% (28219998208/34359738368), sparse 26% (9094787072), duration 629, 56/3 MB/s
INFO: status: 83% (28573696000/34359738368), sparse 27% (9390829568), duration 635, 58/9 MB/s
INFO: status: 84% (28906225664/34359738368), sparse 28% (9723359232), duration 641, 55/0 MB/s
INFO: status: 85% (29210443776/34359738368), sparse 29% (10027577344), duration 646, 60/0 MB/s
INFO: status: 86% (29603201024/34359738368), sparse 30% (10420334592), duration 653, 56/0 MB/s
INFO: status: 87% (29904863232/34359738368), sparse 31% (10721996800), duration 658, 60/0 MB/s
INFO: status: 88% (30294999040/34359738368), sparse 32% (11112128512), duration 664, 65/0 MB/s
INFO: status: 89% (30611275776/34359738368), sparse 33% (11428405248), duration 669, 63/0 MB/s
INFO: status: 90% (30974541824/34359738368), sparse 34% (11791671296), duration 675, 60/0 MB/s
INFO: status: 91% (31295275008/34359738368), sparse 35% (12112404480), duration 680, 64/0 MB/s
INFO: status: 92% (31669944320/34359738368), sparse 36% (12487073792), duration 686, 62/0 MB/s
INFO: status: 93% (31974162432/34359738368), sparse 37% (12791291904), duration 691, 60/0 MB/s
INFO: status: 94% (32364167168/34359738368), sparse 38% (13181296640), duration 697, 65/0 MB/s
INFO: status: 95% (32651345920/34359738368), sparse 39% (13468475392), duration 702, 57/0 MB/s
INFO: status: 96% (33016905728/34359738368), sparse 40% (13834035200), duration 708, 60/0 MB/s
INFO: status: 97% (33330036736/34359738368), sparse 41% (14147166208), duration 713, 62/0 MB/s
INFO: status: 98% (33695137792/34359738368), sparse 42% (14512267264), duration 719, 60/0 MB/s
INFO: status: 99% (34069676032/34359738368), sparse 43% (14886805504), duration 725, 62/0 MB/s
INFO: status: 100% (34359738368/34359738368), sparse 44% (15176867840), duration 730, 58/0 MB/s
INFO: transferred 34359 MB in 730 seconds (47 MB/s)
INFO: archive file size: 9.53GB
INFO: Finished Backup of VM 101 (00:12:11)
INFO: Backup job finished successfully
TASK OK

INFO: starting new backup job: vzdump 101 --remove 0 --mode snapshot --compress lzo --storage ts879 --node pve1
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: update VM 101: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/ts879/dump/vzdump-qemu-101-2013_11_26-15_18_20.vma.lzo'
INFO: started backup task '4d3ed01c-2c76-421f-9334-dc4bdd1936a2'
INFO: status: 0% (181665792/34359738368), sparse 0% (11161600), duration 3, 60/56 MB/s
INFO: status: 1% (375390208/34359738368), sparse 0% (49651712), duration 6, 64/51 MB/s
INFO: status: 2% (746848256/34359738368), sparse 0% (53710848), duration 12, 61/61 MB/s
INFO: status: 3% (1085079552/34359738368), sparse 0% (56209408), duration 18, 56/55 MB/s
INFO: status: 4% (1399193600/34359738368), sparse 0% (58175488), duration 23, 62/62 MB/s
INFO: status: 5% (1771175936/34359738368), sparse 0% (60391424), duration 29, 61/61 MB/s
INFO: status: 6% (2074673152/34359738368), sparse 0% (61562880), duration 34, 60/60 MB/s
INFO: status: 7% (2465071104/34359738368), sparse 0% (84545536), duration 42, 48/45 MB/s
INFO: status: 8% (2749759488/34359738368), sparse 0% (84791296), duration 47, 56/56 MB/s
INFO: status: 9% (3116761088/34359738368), sparse 0% (85143552), duration 54, 52/52 MB/s
INFO: status: 10% (3448897536/34359738368), sparse 0% (85831680), duration 60, 55/55 MB/s
INFO: status: 11% (3782213632/34359738368), sparse 0% (86208512), duration 788, 0/0 MB/s
INFO: status: 12% (4157669376/34359738368), sparse 0% (87142400), duration 794, 62/62 MB/s
INFO: status: 13% (4525522944/34359738368), sparse 0% (95080448), duration 800, 61/59 MB/s
INFO: status: 14% (4815192064/34359738368), sparse 0% (124571648), duration 999, 1/1 MB/s
INFO: status: 15% (5182128128/34359738368), sparse 0% (125489152), duration 1005, 61/61 MB/s
INFO: status: 16% (5510397952/34359738368), sparse 0% (127807488), duration 1132, 2/2 MB/s
INFO: status: 17% (5884542976/34359738368), sparse 0% (131997696), duration 1192, 6/6 MB/s
INFO: status: 18% (6234177536/34359738368), sparse 0% (133906432), duration 1198, 58/57 MB/s
INFO: status: 19% (6583484416/34359738368), sparse 0% (153612288), duration 1204, 58/54 MB/s
INFO: status: 20% (6888488960/34359738368), sparse 0% (184041472), duration 1209, 61/54 MB/s
INFO: status: 21% (7258439680/34359738368), sparse 0% (184422400), duration 1215, 61/61 MB/s
INFO: status: 22% (7571439616/34359738368), sparse 0% (187113472), duration 1220, 62/62 MB/s
INFO: status: 23% (7933526016/34359738368), sparse 0% (188596224), duration 1573, 1/1 MB/s
INFO: status: 24% (8305639424/34359738368), sparse 0% (290709504), duration 1579, 62/45 MB/s
INFO: status: 25% (8623423488/34359738368), sparse 1% (608493568), duration 1584, 63/0 MB/s
INFO: status: 26% (8969977856/34359738368), sparse 2% (930177024), duration 1590, 57/4 MB/s
INFO: status: 27% (9278193664/34359738368), sparse 3% (1075134464), duration 1595, 61/32 MB/s
INFO: status: 28% (9672982528/34359738368), sparse 4% (1469923328), duration 1603, 49/0 MB/s
INFO: status: 29% (9971302400/34359738368), sparse 5% (1768243200), duration 1608, 59/0 MB/s
INFO: status: 30% (10360913920/34359738368), sparse 6% (2157854720), duration 1615, 55/0 MB/s
INFO: status: 31% (10676338688/34359738368), sparse 7% (2473279488), duration 1620, 63/0 MB/s
INFO: status: 32% (11015553024/34359738368), sparse 8% (2806964224), duration 1627, 48/0 MB/s
INFO: status: 33% (11386486784/34359738368), sparse 8% (3000926208), duration 1736, 3/1 MB/s
INFO: status: 34% (11697455104/34359738368), sparse 9% (3311894528), duration 1741, 62/0 MB/s
INFO: status: 35% (12042960896/34359738368), sparse 10% (3657400320), duration 1747, 57/0 MB/s
INFO: status: 36% (12414484480/34359738368), sparse 11% (4028923904), duration 1753, 61/0 MB/s
INFO: status: 37% (12714049536/34359738368), sparse 11% (4048769024), duration 1758, 59/55 MB/s
INFO: status: 38% (13081968640/34359738368), sparse 11% (4053639168), duration 1764, 61/60 MB/s
INFO: status: 39% (13412597760/34359738368), sparse 11% (4084047872), duration 1770, 55/50 MB/s
INFO: status: 40% (13782745088/34359738368), sparse 11% (4084760576), duration 1776, 61/61 MB/s
INFO: status: 41% (14088208384/34359738368), sparse 11% (4085231616), duration 1990, 1/1 MB/s
INFO: status: 42% (14459011072/34359738368), sparse 11% (4086255616), duration 2065, 4/4 MB/s
INFO: status: 43% (14797373440/34359738368), sparse 11% (4087160832), duration 2147, 4/4 MB/s
INFO: status: 44% (15180431360/34359738368), sparse 12% (4224233472), duration 2153, 63/40 MB/s
INFO: status: 45% (15505031168/34359738368), sparse 12% (4330348544), duration 2158, 64/43 MB/s
INFO: status: 46% (15819866112/34359738368), sparse 12% (4350861312), duration 2163, 62/58 MB/s
INFO: status: 47% (16154624000/34359738368), sparse 13% (4549423104), duration 2251, 3/1 MB/s
INFO: status: 48% (16499539968/34359738368), sparse 13% (4552060928), duration 2264, 26/26 MB/s
INFO: status: 49% (16858677248/34359738368), sparse 13% (4554588160), duration 2275, 32/32 MB/s
INFO: status: 50% (17225613312/34359738368), sparse 13% (4560953344), duration 2281, 61/60 MB/s
INFO: status: 51% (17532452864/34359738368), sparse 13% (4604465152), duration 2286, 61/52 MB/s
INFO: status: 52% (17893425152/34359738368), sparse 13% (4606795776), duration 2571, 1/1 MB/s
INFO: status: 53% (18213634048/34359738368), sparse 13% (4608970752), duration 2588, 18/18 MB/s
INFO: status: 54% (18561236992/34359738368), sparse 13% (4619018240), duration 2594, 57/56 MB/s
INFO: status: 55% (18942197760/34359738368), sparse 13% (4619608064), duration 2600, 63/63 MB/s
INFO: status: 56% (19253428224/34359738368), sparse 13% (4619653120), duration 2605, 62/62 MB/s
INFO: status: 57% (19587006464/34359738368), sparse 13% (4619964416), duration 2977, 0/0 MB/s
INFO: status: 58% (19972292608/34359738368), sparse 13% (4665556992), duration 3079, 3/3 MB/s
INFO: status: 59% (20290338816/34359738368), sparse 13% (4665966592), duration 3084, 63/63 MB/s
INFO: status: 60% (20676804608/34359738368), sparse 13% (4667572224), duration 3090, 64/64 MB/s
INFO: status: 61% (20999110656/34359738368), sparse 13% (4669468672), duration 3095, 64/64 MB/s
INFO: status: 62% (21346516992/34359738368), sparse 13% (4670029824), duration 3392, 1/1 MB/s
INFO: status: 63% (21650866176/34359738368), sparse 13% (4672712704), duration 3397, 60/60 MB/s
INFO: status: 64% (22020882432/34359738368), sparse 13% (4786593792), duration 3403, 61/42 MB/s
INFO: status: 65% (22351970304/34359738368), sparse 13% (4797280256), duration 3409, 55/53 MB/s
INFO: status: 66% (22729850880/34359738368), sparse 13% (4798529536), duration 3745, 1/1 MB/s
INFO: status: 67% (23078961152/34359738368), sparse 14% (5003071488), duration 3761, 21/9 MB/s
INFO: status: 68% (23399825408/34359738368), sparse 14% (5013524480), duration 3766, 64/62 MB/s
INFO: status: 69% (23723048960/34359738368), sparse 14% (5102993408), duration 3771, 64/46 MB/s
INFO: status: 70% (24120852480/34359738368), sparse 15% (5319954432), duration 3777, 66/30 MB/s
INFO: status: 71% (24450170880/34359738368), sparse 16% (5564706816), duration 3782, 65/16 MB/s
INFO: status: 72% (24774180864/34359738368), sparse 17% (5888716800), duration 3789, 46/0 MB/s
INFO: status: 73% (25094062080/34359738368), sparse 18% (6208598016), duration 3856, 4/0 MB/s
INFO: status: 74% (25469976576/34359738368), sparse 19% (6584512512), duration 3862, 62/0 MB/s
INFO: status: 75% (25798049792/34359738368), sparse 20% (6912585728), duration 3867, 65/0 MB/s
INFO: status: 76% (26122780672/34359738368), sparse 20% (7208210432), duration 3872, 64/5 MB/s
INFO: status: 77% (26523992064/34359738368), sparse 21% (7418408960), duration 3878, 66/31 MB/s
INFO: status: 78% (26842103808/34359738368), sparse 22% (7736520704), duration 3883, 63/0 MB/s
INFO: status: 79% (27155562496/34359738368), sparse 23% (8049979392), duration 3888, 62/0 MB/s
INFO: status: 80% (27514044416/34359738368), sparse 24% (8408461312), duration 3894, 59/0 MB/s
INFO: status: 81% (27887861760/34359738368), sparse 25% (8782278656), duration 3900, 62/0 MB/s
INFO: status: 82% (28196470784/34359738368), sparse 26% (9085407232), duration 3905, 61/1 MB/s
INFO: status: 83% (28559409152/34359738368), sparse 27% (9376542720), duration 3920, 24/4 MB/s
INFO: status: 84% (28874178560/34359738368), sparse 28% (9691312128), duration 3925, 62/0 MB/s
INFO: status: 85% (29253173248/34359738368), sparse 29% (10070306816), duration 3931, 63/0 MB/s
INFO: status: 86% (29569581056/34359738368), sparse 30% (10386714624), duration 3937, 52/0 MB/s
INFO: status: 87% (29943529472/34359738368), sparse 31% (10760663040), duration 3943, 62/0 MB/s
INFO: status: 88% (30264852480/34359738368), sparse 32% (11081981952), duration 3948, 64/0 MB/s
INFO: status: 89% (30644830208/34359738368), sparse 33% (11461959680), duration 3954, 63/0 MB/s
INFO: status: 90% (30963466240/34359738368), sparse 34% (11780595712), duration 3959, 63/0 MB/s
INFO: status: 91% (31273123840/34359738368), sparse 35% (12090253312), duration 3964, 61/0 MB/s
INFO: status: 92% (31649234944/34359738368), sparse 36% (12466364416), duration 3970, 62/0 MB/s
INFO: status: 93% (31963283456/34359738368), sparse 37% (12780412928), duration 3975, 62/0 MB/s
INFO: status: 94% (32311410688/34359738368), sparse 38% (13128540160), duration 3982, 49/0 MB/s
INFO: status: 95% (32655671296/34359738368), sparse 39% (13472800768), duration 3993, 31/0 MB/s
INFO: status: 96% (33024835584/34359738368), sparse 40% (13841965056), duration 4000, 52/0 MB/s
INFO: status: 97% (33377681408/34359738368), sparse 41% (14194810880), duration 4007, 50/0 MB/s
INFO: status: 98% (33717157888/34359738368), sparse 42% (14534287360), duration 4012, 67/0 MB/s
INFO: status: 99% (34043396096/34359738368), sparse 43% (14860525568), duration 4017, 65/0 MB/s
INFO: status: 100% (34359738368/34359738368), sparse 44% (15176867840), duration 4023, 52/0 MB/s
INFO: transferred 34359 MB in 4023 seconds (8 MB/s)
INFO: archive file size: 9.53GB
INFO: Finished Backup of VM 101 (01:07:05)
INFO: Backup job finished successfully
TASK OK

Thanks,
Marco

mir · Nov 26, 2013

Re: NFS performance and mount options

m.ardito said:
now storage.conf is:

Code:

nfs: ts879 path /mnt/pve/ts879 server IP_of_ts879 export /myshare options vers=3 content backup maxfiles 5

I will do that, but
- should I add one option per line in storage.conf ?
- after changing storage.conf, how to remount?

Simply replace each option line in question.
To remount simply unmount proxmox will auto mount when needed.

m.ardito · Nov 26, 2013

Re: NFS performance and mount options

mir said:
From my Qnap RAID1:
# pveperf /mnt/pve/qnap_nfs/
CPU BOGOMIPS: 32551.14
REGEX/SECOND: 1305031
HD SIZE: 1373.87 GB (192.168.2.10:/vz)
FSYNCS/SECOND: 750.60
DNS EXT: 61.89 ms
DNS INT: 0.72 ms (datanom.net)

Timing buffered disk reads: 432 MB in 3.01 seconds = 143.50 MB/sec
This seems very low for a 5 disk RAID5

Your result seems in the middle of mines, roughly...

that could be a low value, but the old one has even lower values (roughly half), and performs much much better...

the reason must be something else...

Marco

m.ardito · Nov 26, 2013

Re: NFS performance and mount options

mir said:
Simply replace each option line in question.
To remount simply unmount proxmox will auto mount when needed.

i did this:

Code:

nfs: ts879
        path /mnt/pve/ts879
        server ts879
        export /myshare
        options vers=3,tcp,nolock
        content images,iso,vztmpl,rootdir,backup
        maxfiles

it was immediately remounted, so now I have

Code:

ts879:/myshareon /mnt/pve/ts879 type nfs (rw
relatime
vers=3
rsize=32768
wsize=32768
namlen=255
hard
nolock
proto=tcp
timeo=600
retrans=2
sec=sys
mountaddr=ip_of_ts879
mountvers=3
mountport=38649
mountproto=tcp
local_lock=all
addr=ip_of_ts879)

but FSYNCS did not change that much (and rsize/wsize are identical as before)

Code:

# pveperf /mnt/pve/ts879/
CPU BOGOMIPS:      72527.28
REGEX/SECOND:      938304
HD SIZE:           11081.12 GB (ts879:/myshare)
FSYNCS/SECOND:    [B] 202.91[/B]
DNS EXT:           62.82 ms
DNS INT:           0.66 ms

tomorrow I will try to backup test again.
is there a way to force rsize/wsize in storage.conf ? and any risk? I would try that.

and how about the scheduler difference between the two NAS reported above? they also have very different kernels..

old NAS (using deadline):
# uname -a
Linux TS809 2.6.30.6 #1 SMP Tue Feb 9 02:42:31 CST 2010 i686 unknown

new NAS (using CFQ):
# uname -a
Linux TS879 3.4.6 #1 SMP Fri Apr 26 13:37:02 CST 2013 x86_64 unknown

Marco

dietmar · Nov 26, 2013

Re: NFS performance and mount options

m.ardito said:
tomorrow I will try to backup test again.
is there a way to force rsize/wsize in storage.conf ? and any risk? I would try that.

You can set any NFS option in the options tag "options vers=3,tcp,nolock,wsize=...,rsize=..."

see 'man mount.nfs' for available options.

mir · Nov 26, 2013

Re: NFS performance and mount options

Must I suspect the values for read and write in Qnap are hard coded somewhere. I have tried any option in the book but no matter what I do the rsize and wsize remains at 32768

m.ardito · Nov 27, 2013

Re: NFS performance and mount options

mir said:
Must I suspect the values for read and write in Qnap are hard coded somewhere. I have tried any option in the book but no matter what I do the rsize and wsize remains at 32768

that happens to me to... I suspect this unit has something wrong... any ideas?

Marco

m.ardito · Nov 27, 2013

Re: NFS performance and mount options

forget all... the fault was on the new qnap NAS eth side...
this eth was set on a 172.16 network, while pve is on a 192.168
putting the qnap eth on the same 192.168 solved the problem... it is now at least as fast as the old qnap NAS

now I have to find what network problem there is on that network...

Marco

Uzzi · Aug 14, 2020

Hi, I've all vm disks on qnap in nfs! I read in nfs qnap options "For virtualization environment, it is recommended to disable [write_cache] function." Now is it enabled. Can I disable during vm running?

Search

Search

[solved] NFS performance and mount options

m.ardito

Active Member

mir

Famous Member

m.ardito

Active Member

mir

Famous Member

mir

Famous Member

m.ardito

Active Member

mir

Famous Member

m.ardito

Active Member

m.ardito

Active Member

dietmar

Proxmox Staff Member

mir

Famous Member

m.ardito

Active Member

m.ardito

Active Member

Uzzi

Member