ZFS Pool local-zfs entfernen und Speicherplatz in ROOT nutzen

[HBO]

Ich hatte per ISO PVE 5 installiert, nun wurde hier automatisch der ZFS Pool "rpool" mit den Pools "rpool/ROOT" und "rpool/data" angelegt. Da alle VMs und LXC auf eigenen Pools sitzen, sowie Templates, ISOs und Backups auf einem NFS würde ich nun gerne das angelegte Storage "local-zfs" das auf den Pool "rpool/data" zeigt löschen und den Speicherplatz unter dem Pool "rpool/ROOT" nutzen.

Hintergrund ist folgender:
Hier ist eine Intel DC SSD im Einsatz und ich möchte diese gerne für vzdump als temporären Speicher nutzen. Nun wurde diese SSD aber in 2 100GB Pools gesplittet.

Wäre das grundsätzlich möglich? Was wäre genau zu tun?
Oder reichen in /var/lib/vz 80GB freier Speicher für vzdump als temp dir aus?

Hier mal die Ausgabe aus einer vzdump Log (ohne tmp dir) eines der größten Backups:

Spoiler: vzdump log

2017-08-28 08:11:29 INFO: Starting Backup of VM 194 (qemu)
2017-08-28 08:11:29 INFO: status = running
2017-08-28 08:11:30 INFO: update VM 194: -lock backup
2017-08-28 08:11:30 INFO: VM Name: config.xenonserver.de-K14163
2017-08-28 08:11:30 INFO: include disk 'scsi0' 'zfs-ssd:vm-194-disk-1' 150G
2017-08-28 08:11:30 INFO: include disk 'scsi1' 'zfs-ssd:vm-194-disk-2' 150G
2017-08-28 08:11:30 INFO: backup mode: snapshot
2017-08-28 08:11:30 INFO: ionice priority: 7
2017-08-28 08:11:30 INFO: creating archive '/mnt/pve/nas02-10GE/dump/vzdump-qemu-194-2017_08_28-08_11_29.vma.lzo'
2017-08-28 08:11:30 INFO: started backup task '69763bb6-0355-4617-8651-5cf33b824ec1'
2017-08-28 08:11:33 INFO: status: 0% (764018688/322122547200), sparse 0% (211894272), duration 3, read/write 254/184 MB/s
......
2017-08-28 08:42:50 INFO: status: 100% (322122547200/322122547200), sparse 70% (226731098112), duration 1880, read/write 598/63 MB/s
2017-08-28 08:42:50 INFO: transferred 322122 MB in 1880 seconds (171 MB/s)
2017-08-28 08:43:44 INFO: archive file size: 62.49GB
2017-08-28 08:43:44 INFO: delete old backup '/mnt/pve/nas02-10GE/dump/vzdump-qemu-194-2017_08_20-07_47_32.vma.lzo'
2017-08-28 08:43:45 INFO: Finished Backup of VM 194 (00:32:16)

 

[wolfgang]

Hi,

zfs hat nur bei zvols feste Größen.

rpool/data und rpool/ROOT sind aber subsets normalerweise den platz dynamisch verwenden.
Also wenn du da kein platz hast, dann hast du zu viele Platz den VM gegeben .
Damit meine ich wen du eine VM verschiebst oder löscht hast du sofort mehr platz am pool.

Das was wirklich frei ist siehst du mit

zfs list
[\CODE]

 

[HBO]

Ah wunderbar, das las sich so als ob die Subsets auch eigene Größen gehabt hätten.
Damit wäre das geklärt, danke.

Würden nun ca. 80GB maximal verfügbarer Speicher ausreichen um zbsp /var/lib/vz per Konfiguration in vzdump als "temp dir" zu nutzen?

 

[fabian]

Ah wunderbar, das las sich so als ob die Subsets auch eigene Größen gehabt hätten.
Damit wäre das geklärt, danke.

Würden nun ca. 80GB maximal verfügbarer Speicher ausreichen um zbsp /var/lib/vz per Konfiguration in vzdump als "temp dir" zu nutzen?

das hängt von der größe und art des backups ab, lässt sich also so pauschal nicht sagen..

 

[HBO]

Ich hatte oben ja im Eingangspost einen Log Output eines der größten Backups geschrieben.
Die Backups laufen alle auf die gleiche Art und Weise:
Basissystem ist ZFS, Backup per Snapshot Modus mit LZO auf externes NAS.

Die größte VM hat eine Disk Größe von 300GB.

 

[fabian]

Ich hatte oben ja im Eingangspost einen Log Output eines der größten Backups geschrieben.
Die Backups laufen alle auf die gleiche Art und Weise:
Basissystem ist ZFS, Backup per Snapshot Modus mit LZO auf externes NAS.

Die größte VM hat eine Disk Größe von 300GB.

für VMs wird das tmpdir nur für die VM und firewall configs verwendet, das backup des disk-inhalts erfolgt direkt auf den backup storage. bei containern gibt es einen modus (suspend), in welchem das tmpdir wirklich zum zwischenspeichern des backup-inhalts verwendet wird.

 

[HBO]

Ok, macht hier also nicht wirklich viel Sinn.
Mich wundert dann die Geschwindigkeit der Backups etwas, Modus wie gesagt überall gleich:
ZFS Pool mit SSDs im Mirror (3 Mirrors) bringen im Schnitt 500mb/s Übertragungsgeschwindigkeit.
ZFS Pool mit SAS 12G 10k rpm Platten im Mirror (4 Mirrors) kommen im Schnitt auf 50-60mb/s Übertragungsgeschwindigkeit

SSDs sind klar schneller, aber von SAS 12G 10k rpm Platten könnte man doch mehr Leistung erwarten?

*edit*

Spoiler: Backup einer VM im SAS Pool

INFO: Starting Backup of VM 176 (qemu)
INFO: status = running
INFO: update VM 176: -lock backup
INFO: VM Name: ........
INFO: include disk 'scsi0' 'zfs-sata:vm-176-disk-1' 25G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backups/dump/vzdump-qemu-176-2017_09_08-06_18_21.vma.lzo'
INFO: started backup task 'ed16a007-77af-4a3c-b119-6a1985bd67b4'
INFO: status: 0% (76021760/26843545600), sparse 0% (62656512), duration 4, read/write 19/3 MB/s
INFO: status: 1% (304087040/26843545600), sparse 0% (238190592), duration 8, read/write 57/13 MB/s
INFO: status: 2% (707002368/26843545600), sparse 1% (493240320), duration 11, read/write 134/49 MB/s
INFO: status: 3% (961675264/26843545600), sparse 2% (645312512), duration 15, read/write 63/25 MB/s
INFO: status: 6% (1687683072/26843545600), sparse 4% (1294807040), duration 18, read/write 242/25 MB/s
INFO: status: 9% (2463105024/26843545600), sparse 7% (2035699712), duration 21, read/write 258/11 MB/s
INFO: status: 12% (3360161792/26843545600), sparse 10% (2849824768), duration 25, read/write 224/20 MB/s
INFO: status: 17% (4606918656/26843545600), sparse 15% (4072566784), duration 28, read/write 415/8 MB/s
INFO: status: 18% (4869718016/26843545600), sparse 15% (4273299456), duration 31, read/write 87/20 MB/s
INFO: status: 24% (6701318144/26843545600), sparse 22% (6104285184), duration 34, read/write 610/0 MB/s
INFO: status: 25% (6898974720/26843545600), sparse 23% (6221864960), duration 38, read/write 49/20 MB/s
INFO: status: 26% (7081426944/26843545600), sparse 23% (6341607424), duration 41, read/write 60/20 MB/s
INFO: status: 27% (7248674816/26843545600), sparse 24% (6466670592), duration 51, read/write 16/4 MB/s
INFO: status: 28% (7567966208/26843545600), sparse 24% (6691958784), duration 65, read/write 22/6 MB/s
INFO: status: 29% (7784628224/26843545600), sparse 25% (6785892352), duration 90, read/write 8/4 MB/s
INFO: status: 30% (8073510912/26843545600), sparse 25% (6806634496), duration 127, read/write 7/7 MB/s
INFO: status: 31% (8335785984/26843545600), sparse 25% (6910451712), duration 152, read/write 10/6 MB/s
INFO: status: 32% (8620867584/26843545600), sparse 26% (6983254016), duration 175, read/write 12/9 MB/s
INFO: status: 33% (8898347008/26843545600), sparse 26% (7120379904), duration 187, read/write 23/11 MB/s
INFO: status: 34% (9160622080/26843545600), sparse 27% (7312482304), duration 193, read/write 43/11 MB/s
INFO: status: 35% (9445703680/26843545600), sparse 27% (7484768256), duration 208, read/write 19/7 MB/s
INFO: status: 36% (9730785280/26843545600), sparse 28% (7695319040), duration 211, read/write 95/24 MB/s
INFO: status: 37% (9998237696/26843545600), sparse 29% (7934160896), duration 216, read/write 53/5 MB/s
INFO: status: 38% (10274340864/26843545600), sparse 30% (8164225024), duration 219, read/write 92/15 MB/s
INFO: status: 39% (10491002880/26843545600), sparse 31% (8369246208), duration 222, read/write 72/3 MB/s
INFO: status: 40% (10745675776/26843545600), sparse 31% (8525869056), duration 227, read/write 50/19 MB/s
INFO: status: 41% (11046092800/26843545600), sparse 32% (8719536128), duration 233, read/write 50/17 MB/s
INFO: status: 42% (11285430272/26843545600), sparse 32% (8806735872), duration 273, read/write 5/3 MB/s
INFO: status: 43% (11692146688/26843545600), sparse 34% (9160421376), duration 276, read/write 135/17 MB/s
INFO: status: 45% (12182487040/26843545600), sparse 35% (9635700736), duration 280, read/write 122/3 MB/s
INFO: status: 46% (12452364288/26843545600), sparse 36% (9700163584), duration 286, read/write 44/34 MB/s
INFO: status: 47% (12634816512/26843545600), sparse 36% (9780457472), duration 295, read/write 20/11 MB/s
INFO: status: 50% (13467254784/26843545600), sparse 39% (10555363328), duration 299, read/write 208/14 MB/s
INFO: status: 51% (13695320064/26843545600), sparse 39% (10581344256), duration 327, read/write 8/7 MB/s
INFO: status: 52% (13968998400/26843545600), sparse 39% (10733383680), duration 356, read/write 9/4 MB/s
INFO: status: 53% (14227472384/26843545600), sparse 40% (10911522816), duration 365, read/write 28/8 MB/s
INFO: status: 54% (14497349632/26843545600), sparse 40% (10917482496), duration 443, read/write 3/3 MB/s
INFO: status: 55% (14786232320/26843545600), sparse 40% (10973028352), duration 532, read/write 3/2 MB/s
INFO: status: 56% (15181021184/26843545600), sparse 42% (11318059008), duration 541, read/write 43/5 MB/s
INFO: status: 57% (15337390080/26843545600), sparse 42% (11460804608), duration 544, read/write 52/4 MB/s
INFO: status: 58% (15601500160/26843545600), sparse 43% (11716984832), duration 548, read/write 66/1 MB/s
INFO: status: 62% (16813195264/26843545600), sparse 48% (12911353856), duration 553, read/write 242/3 MB/s
INFO: status: 64% (17446993920/26843545600), sparse 50% (13536325632), duration 557, read/write 158/2 MB/s
INFO: status: 66% (17724473344/26843545600), sparse 51% (13735133184), duration 562, read/write 55/15 MB/s
INFO: status: 72% (19467862016/26843545600), sparse 57% (15458971648), duration 566, read/write 435/4 MB/s
INFO: status: 80% (21734621184/26843545600), sparse 66% (17717125120), duration 573, read/write 323/1 MB/s
INFO: status: 81% (21894266880/26843545600), sparse 66% (17828220928), duration 577, read/write 39/12 MB/s
INFO: status: 82% (22038708224/26843545600), sparse 66% (17899417600), duration 583, read/write 24/12 MB/s
INFO: status: 88% (23882235904/26843545600), sparse 73% (19600551936), duration 594, read/write 167/12 MB/s
INFO: status: 89% (24057085952/26843545600), sparse 73% (19718569984), duration 597, read/write 58/18 MB/s
INFO: status: 90% (24197726208/26843545600), sparse 73% (19756703744), duration 600, read/write 46/34 MB/s
INFO: status: 95% (25641156608/26843545600), sparse 78% (21200134144), duration 603, read/write 481/0 MB/s
INFO: status: 96% (25866403840/26843545600), sparse 79% (21330632704), duration 608, read/write 45/18 MB/s
INFO: status: 97% (26060259328/26843545600), sparse 79% (21332824064), duration 615, read/write 27/27 MB/s
INFO: status: 98% (26352943104/26843545600), sparse 79% (21335748608), duration 629, read/write 20/20 MB/s
INFO: status: 99% (26577207296/26843545600), sparse 79% (21339951104), duration 638, read/write 24/24 MB/s
INFO: status: 100% (26843545600/26843545600), sparse 79% (21343428608), duration 646, read/write 33/32 MB/s
INFO: transferred 26843 MB in 646 seconds (41 MB/s)
INFO: archive file size: 2.47GB
INFO: Finished Backup of VM 176 (00:10:47)

Spoiler: Backup einer VM in SSD Pool

INFO: Starting Backup of VM 197 (qemu)
INFO: status = running
INFO: update VM 197: -lock backup
INFO: VM Name: ......
INFO: include disk 'scsi0' 'zfs-ssd:vm-197-disk-1' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backups/dump/vzdump-qemu-197-2017_09_08-06_29_47.vma.lzo'
INFO: started backup task 'cace6b94-0553-4906-a810-39d9128f8479'
INFO: status: 1% (638582784/34359738368), sparse 0% (313561088), duration 3, read/write 212/108 MB/s
INFO: status: 5% (1873936384/34359738368), sparse 3% (1269698560), duration 6, read/write 411/93 MB/s
INFO: status: 8% (2774794240/34359738368), sparse 5% (1759432704), duration 9, read/write 300/137 MB/s
INFO: status: 13% (4774166528/34359738368), sparse 10% (3542904832), duration 12, read/write 666/71 MB/s
INFO: status: 21% (7256276992/34359738368), sparse 17% (5878247424), duration 15, read/write 827/48 MB/s
INFO: status: 29% (10297344000/34359738368), sparse 25% (8856412160), duration 18, read/write 1013/20 MB/s
INFO: status: 38% (13063815168/34359738368), sparse 33% (11561283584), duration 21, read/write 922/20 MB/s
INFO: status: 44% (15320350720/34359738368), sparse 39% (13654036480), duration 24, read/write 752/54 MB/s
INFO: status: 51% (17770086400/34359738368), sparse 46% (15949848576), duration 27, read/write 816/51 MB/s
INFO: status: 54% (18807783424/34359738368), sparse 48% (16708329472), duration 30, read/write 345/93 MB/s
INFO: status: 56% (19431161856/34359738368), sparse 49% (16961871872), duration 33, read/write 207/123 MB/s
INFO: status: 58% (20107755520/34359738368), sparse 50% (17267879936), duration 36, read/write 225/123 MB/s
INFO: status: 61% (20997210112/34359738368), sparse 51% (17806540800), duration 39, read/write 296/116 MB/s
INFO: status: 64% (22023503872/34359738368), sparse 53% (18551025664), duration 42, read/write 342/93 MB/s
INFO: status: 66% (22836936704/34359738368), sparse 55% (19003600896), duration 45, read/write 271/120 MB/s
INFO: status: 70% (24167317504/34359738368), sparse 58% (20036464640), duration 48, read/write 443/99 MB/s
INFO: status: 74% (25649741824/34359738368), sparse 61% (21251018752), duration 51, read/write 494/89 MB/s
INFO: status: 80% (27777630208/34359738368), sparse 67% (23188426752), duration 54, read/write 709/63 MB/s
INFO: status: 87% (30145904640/34359738368), sparse 73% (25401323520), duration 57, read/write 789/51 MB/s
INFO: status: 100% (34359738368/34359738368), sparse 86% (29615157248), duration 60, read/write 1404/0 MB/s
INFO: transferred 34359 MB in 60 seconds (572 MB/s)
INFO: archive file size: 1.95GB
INFO: Finished Backup of VM 197 (00:01:01)

Spoiler: zpool status -v

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 111G 7.92G 103G - 14% 7% 1.00x ONLINE -
mirror 111G 7.92G 103G - 14% 7%
sdg2 - - - - - -
sdh2 - - - - - -
sata 5.44T 2.08T 3.36T - 36% 38% 1.00x ONLINE -
mirror 1.81T 711G 1.12T - 36% 38%
sda - - - - - -
sdb - - - - - -
mirror 1.81T 711G 1.12T - 36% 38%
sdc - - - - - -
sdd - - - - - -
mirror 1.81T 711G 1.12T - 36% 38%
sde - - - - - -
sdf - - - - - -
log - - - - - -
sdi2 19.9G 311M 19.6G - 84% 1%
cache - - - - - -
sdi6 207G 147G 60.2G - 0% 70%
ssd 888G 332G 556G - 37% 37% 1.00x ONLINE -
mirror 444G 166G 278G - 37% 37%
sdj - - - - - -
sdk - - - - - -
mirror 444G 166G 278G - 38% 37%
sdl - - - - - -
sdm - - - - - -
log - - - - - -
sdi1 19.9G 18.4M 19.9G - 36% 0%
cache - - - - - -
sdi5 200G 100G 99.8G - 0% 50%

 

[fabian]

sind die sonstigen pool/dataset properties ident? I/O last in den VMs kann das backup natürlich auch beeinflussen..

auffällig ist, dass bei der HDD VM schon das lesen extrem langsam ist.. (ich habe hier lokal mit einer SATA disk bessere werte!)

vielleicht lässt sich mittels 'zpool iostat' oder 'iostat' (letzteres aus dem paket sysstat) mehr herausfinden?

 

[HBO]

Das Backup aus dem SAS Pool betrifft zBsp einen einfachen Mailserver auf dem vor allem bei Backup Durchführung in der Zeit absolut nichts los ist, genauso bei allen anderen VMs. Dies sind in der Regel Mail-, Web-, oder DB Server.

Hier mal die Ausgaben:

Spoiler: iostat

Linux 4.10.15-1-pve (pve-node01) 09/08/2017 _x86_64_ (72 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle
2.61 0.00 1.10 2.19 0.00 94.10

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 161.23 955.20 1384.89 4626707140 6708019204
sdb 161.26 954.54 1384.89 4623539784 6708019336
sdc 152.68 955.23 1384.00 4626867076 6703697344
sdd 152.62 954.87 1384.00 4625109684 6703697004
sde 158.38 954.13 1382.52 4621552368 6696547684
sdf 158.47 955.22 1382.52 4626804172 6696547684
sdg 15.45 3.26 143.29 15805885 694037308
sdh 15.28 3.18 143.29 15405453 694037308
sdi 335.28 225.68 3647.91 1093145217 17669445954
sdj 62.68 372.86 522.08 1806013516 2528823180
sdk 62.58 372.08 522.08 1802231204 2528823180
sdl 60.80 372.69 521.19 1805201204 2524486988
sdm 60.77 372.17 521.19 1802701208 2524486988
zd0 0.74 0.53 2.42 2547484 11739404
zd16 11.72 541.36 404.11 2622207286 1957407777
zd32 32.10 348.13 108.78 1686261487 526904596
zd48 2.33 109.04 6.05 528152587 29306988
zd64 23.42 35.83 59.86 173548703 289965040
zd80 1.06 0.00 4.24 12 20532152
zd96 5.37 1.00 22.97 4828671 111237862
zd112 23.60 1.62 117.07 7861475 567076628
zd128 5.84 9.83 16.14 47631848 78199655
zd144 1.16 0.17 4.98 843478 24144129
zd160 4.28 216.55 6.93 1048906822 33544743
zd176 11.32 33.45 32.54 162023983 157610805
zd192 14.47 11.97 54.72 57956670 265066485
zd208 6.89 353.37 11.17 1711610488 54115248
zd224 7.53 346.42 13.89 1677985188 67286507
zd240 136.38 478.27 327.20 2316597996 1584855284
zd256 50.33 1065.74 214.92 5162141406 1040990659
zd272 37.14 1063.94 227.92 5153438994 1103993354
zd288 2.18 0.32 13.45 1556744 65157104
zd304 22.23 541.76 239.80 2624110954 1161544067
zd320 4.12 216.54 2.13 1048850518 10324924
zd336 3.06 0.81 12.53 3924640 60706589
zd352 0.11 0.02 0.42 88630 2021132
zd368 0.10 0.02 0.40 83882 1924636
zd384 3.25 0.27 13.65 1303158 66113717
zd400 2.65 0.26 10.64 1259153 51549516
zd416 20.15 1062.33 67.00 5145638857 324553416
zd432 2.11 0.00 8.45 20 40945716
zd448 48.86 141.17 156.79 683807800 759456710
zd464 18.55 530.41 61.88 2569146590 299719425
zd480 4.21 91.53 14.10 443361052 68310777
zd496 1.38 58.57 1.82 283688996 8827308
zd512 1.05 58.53 0.50 283491152 2440948
zd528 71.43 741.33 81.59 3590777064 395197221
zd544 8.20 25.30 48.43 122531833 234592965
zd560 10.26 0.29 42.96 1409506 208078328
zd576 1.97 3.70 15.32 17909475 74191716
zd592 17.55 66.96 59.21 324319375 286773270
zd608 0.51 0.00 2.05 12 9933852
zd624 25.77 111.20 87.06 538605966 421688217
zd656 20.83 118.69 76.58 574877302 370951325
zd672 30.39 214.23 26.91 1037677729 130332065
zd688 5.30 1.41 20.68 6818162 100179612
zd704 5.47 0.13 34.61 648551 167649445
zd720 3.11 0.78 12.70 3765553 61515145
zd736 0.69 0.00 2.76 12 13346500
zd752 1.41 0.46 5.77 2227129 27956772
zd768 3.73 1.18 14.34 5734963 69454000
zd784 23.59 693.51 131.34 3359176385 636166660
zd800 27.84 695.15 718.51 3367092533 3480248773
zd816 27.59 44.14 66.87 213797578 323909092
zd832 3.93 11.17 10.73 54083962 51996737
zd848 9.43 444.42 28.21 2152658320 136618142
zd864 16.48 888.00 71.05 4301236642 344169487
zd880 17.68 889.81 86.49 4309982970 418938568
zd896 16.78 385.21 120.16 1865870168 582028595
zd912 27.26 917.07 187.49 4442054136 908138603
zd928 127.66 147.20 479.28 712978888 2321491229
zd944 24.48 922.72 188.02 4469403072 910697072
zd960 5.32 146.05 38.44 707446262 186188087
zd976 68.07 915.86 472.79 4436173136 2290061176
zd992 12.07 0.01 48.28 63585 233836020
zd1008 9.22 0.00 36.89 14010 178663120
zd1024 17.43 3.62 69.75 17517711 337841391
zd1040 9.40 2.24 36.89 10863106 178663108
zd1056 1.96 0.02 19.32 74681 93573708
zd1072 1.74 0.05 7.00 225483 33926697
zd1088 2.11 0.02 20.24 79952 98019192
zd1104 10.32 8.31 390.07 40241487 1889402827
zd1120 19.73 683.14 53.84 3308937738 260772717
zd1136 17.25 239.26 54.59 1158909852 264418901
zd1152 15.76 465.00 28.33 2252315415 137236948
zd1184 39.40 954.70 304.64 4624293103 1475614490
zd1168 38.67 684.65 263.65 3316251340 1277026463
zd1200 19.56 696.25 107.54 3372443515 520880896
zd1216 2.83 0.06 23.00 273420 111398571
zd1232 4.75 221.72 11.06 1073949453 53572655
zd1248 14.06 40.72 98.07 197249278 475019407
zd1264 14.06 55.00 16.77 266380777 81238604
zd1280 1.51 0.49 5.65 2357022 27370576
zd1296 3.10 143.00 3.40 692631164 16470836
zd1312 1.80 11.92 6.42 57758007 31118632
zd1328 3.39 119.16 6.30 577198699 30505324
zd1344 2.62 23.91 8.99 115813746 43539680
zd1360 6.81 142.92 18.03 692286582 87324244
zd1376 27.31 75.35 50.11 364987119 242730288
zd1392 2.48 1.04 8.37 5030034 40522764
zd1408 3.87 119.75 17.75 580020707 85970821
zd1424 11.54 152.85 55.44 740375147 268547989

Spoiler: zpool iostat

capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
rpool 7.92G 103G 0 30 5.44K 131K
mirror 7.92G 103G 0 30 5.44K 131K
sdg2 - - 0 14 3.25K 143K
sdh2 - - 0 13 3.17K 143K
---------- ----- ----- ----- ----- ----- -----
sata 2.08T 3.35T 807 803 4.83M 5.88M
mirror 711G 1.12T 269 251 1.61M 1.34M
sda - - 39 122 955K 1.35M
sdb - - 39 122 955K 1.35M
mirror 711G 1.12T 269 251 1.61M 1.34M
sdc - - 39 114 955K 1.35M
sdd - - 39 114 955K 1.35M
mirror 711G 1.12T 269 251 1.61M 1.34M
sde - - 39 119 954K 1.35M
sdf - - 39 120 955K 1.35M
logs - - - - - -
sdi2 128M 19.8G 0 48 0 1.87M
cache - - - - - -
sdi6 147G 59.9G 49 131 167K 850K
---------- ----- ----- ----- ----- ----- -----
ssd 333G 555G 212 197 1.31M 1.69M
mirror 166G 278G 106 92 673K 514K
sdj - - 13 49 373K 522K
sdk - - 13 49 372K 522K
mirror 166G 278G 106 91 673K 513K
sdl - - 13 47 373K 521K
sdm - - 13 47 372K 521K
logs - - - - - -
sdi1 26.9M 19.8G 0 13 0 706K
cache - - - - - -
sdi5 100G 99.9G 14 28 58.8K 175K
---------- ----- ----- ----- ----- ----- -----

 

[fabian]

ich meinte während des backups auf die jeweiligen pools/disks

z.b. in 30s fenstern für den hdd pool:

zpool iostat -y -T d sata 30

und

iostat -dmxty /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdi2 /dev/sdi6 30

ergebnis bitte als anhang oder mit code-tags formatiert, nicht als spoiler (sonst stimmen die spalten nicht und es ist nicht lesbar)

 

[HBO]

Ok, werde ich dann am Wochenende mal anfertigen sofern auf dem Host entsprechend wenig los ist wie Nachts.

 

[HBO]

So und los gehts, zum einen habe ich mal ein Subvol "test" angelegt und FIO ausgeführt. Sehe ich das richtig, dass hier eigentlich so gar keine Leistung vorhanden ist? Weiterhin die Log Ausgabe von vzdump sowie iostats (als Dateianhang).

 fio --filename=/dev/zvol/sata/test --sync=1 --rw=randrw --bs=4k --numjobs=1 --iodepth=1 --runtime=120 --time_based --group_reporting --name=journal-test
journal-test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=psync, iodepth=1
fio-2.16
Starting 1 process
Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]
journal-test: (groupid=0, jobs=1): err= 0: pid=47519: Sat Sep  9 10:48:33 2017
  read : io=49720KB, bw=424199B/s, iops=103, runt=120022msec
    clat (usec): min=9, max=917140, avg=900.78, stdev=13039.03
     lat (usec): min=9, max=917140, avg=901.00, stdev=13039.03
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   16], 20.00th=[   23],
     | 30.00th=[   30], 40.00th=[  139], 50.00th=[  157], 60.00th=[  175],
     | 70.00th=[  197], 80.00th=[  229], 90.00th=[  410], 95.00th=[ 2832],
     | 99.00th=[10304], 99.50th=[16064], 99.90th=[80384], 99.95th=[195584],
     | 99.99th=[749568]
  write: io=49892KB, bw=425667B/s, iops=103, runt=120022msec
    clat (usec): min=159, max=1392.8K, avg=8718.55, stdev=53616.06
     lat (usec): min=159, max=1392.8K, avg=8718.88, stdev=53616.07
    clat percentiles (usec):
     |  1.00th=[  193],  5.00th=[  225], 10.00th=[  266], 20.00th=[  350],
     | 30.00th=[  406], 40.00th=[  458], 50.00th=[  532], 60.00th=[  684],
     | 70.00th=[ 3728], 80.00th=[ 6688], 90.00th=[ 9408], 95.00th=[14272],
     | 99.00th=[175104], 99.50th=[325632], 99.90th=[937984], 99.95th=[1056768],
     | 99.99th=[1384448]
    lat (usec) : 10=0.01%, 20=7.36%, 50=10.29%, 100=0.15%, 250=27.60%
    lat (usec) : 500=23.43%, 750=7.77%, 1000=0.81%
    lat (msec) : 2=1.41%, 4=4.65%, 10=12.01%, 20=2.40%, 50=0.80%
    lat (msec) : 100=0.45%, 250=0.48%, 500=0.24%, 750=0.05%, 1000=0.04%
    lat (msec) : 2000=0.04%
  cpu          : usr=0.11%, sys=1.76%, ctx=49254, majf=0, minf=178
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=12430/w=12473/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=49720KB, aggrb=414KB/s, minb=414KB/s, maxb=414KB/s, mint=120022msec, maxt=120022msec
  WRITE: io=49892KB, aggrb=415KB/s, minb=415KB/s, maxb=415KB/s, mint=120022msec, maxt=120022msec

()
Task viewer: Backup
Ausgabe
Status
Stopp
INFO: starting new backup job: vzdump 104 --mode snapshot --node pve-node01 --remove 0 --compress lzo --storage Backups
INFO: Starting Backup of VM 104 (qemu)
INFO: status = running
INFO: update VM 104: -lock backup
INFO: VM Name: galima-notenversand.de-K1001877
INFO: include disk 'ide0' 'zfs-sata:vm-104-disk-1' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backups/dump/vzdump-qemu-104-2017_09_09-10_18_21.vma.lzo'
INFO: started backup task '45fb326c-a174-444c-a385-6f5fbf1367a7'
INFO: status: 0% (3801088/64424509440), sparse 0% (20480), duration 3, read/write 1/1 MB/s
INFO: status: 1% (653787136/64424509440), sparse 0% (18882560), duration 28, read/write 25/25 MB/s
INFO: status: 2% (1341784064/64424509440), sparse 0% (41730048), duration 41, read/write 52/51 MB/s
INFO: status: 3% (1980366848/64424509440), sparse 0% (52224000), duration 49, read/write 79/78 MB/s
INFO: status: 4% (2584739840/64424509440), sparse 0% (62709760), duration 65, read/write 37/37 MB/s
INFO: status: 5% (3325952000/64424509440), sparse 0% (73183232), duration 78, read/write 57/56 MB/s
INFO: status: 6% (3934126080/64424509440), sparse 0% (92016640), duration 88, read/write 60/58 MB/s
INFO: status: 7% (4523294720/64424509440), sparse 0% (100405248), duration 102, read/write 42/41 MB/s
INFO: status: 8% (5158076416/64424509440), sparse 0% (110559232), duration 113, read/write 57/56 MB/s
INFO: status: 9% (5804261376/64424509440), sparse 0% (121077760), duration 128, read/write 43/42 MB/s
INFO: status: 10% (6442844160/64424509440), sparse 0% (128835584), duration 147, read/write 33/33 MB/s
INFO: status: 11% (7172653056/64424509440), sparse 0% (145592320), duration 158, read/write 66/64 MB/s
INFO: status: 12% (7750418432/64424509440), sparse 0% (153845760), duration 168, read/write 57/56 MB/s
INFO: status: 13% (8480227328/64424509440), sparse 0% (166428672), duration 182, read/write 52/51 MB/s
INFO: status: 14% (9076998144/64424509440), sparse 0% (174817280), duration 193, read/write 54/53 MB/s
INFO: status: 15% (9764995072/64424509440), sparse 0% (185303040), duration 203, read/write 68/67 MB/s
INFO: status: 16% (10384572416/64424509440), sparse 0% (195821568), duration 210, read/write 88/87 MB/s
INFO: status: 17% (10996547584/64424509440), sparse 0% (209637376), duration 222, read/write 50/49 MB/s
INFO: status: 18% (11711152128/64424509440), sparse 0% (222302208), duration 230, read/write 89/87 MB/s
INFO: status: 19% (12296519680/64424509440), sparse 0% (230686720), duration 239, read/write 65/64 MB/s
INFO: status: 20% (12946505728/64424509440), sparse 0% (241160192), duration 250, read/write 59/58 MB/s
INFO: status: 21% (13596491776/64424509440), sparse 0% (251633664), duration 260, read/write 64/63 MB/s
INFO: status: 22% (14238875648/64424509440), sparse 0% (262119424), duration 270, read/write 64/63 MB/s
INFO: status: 23% (14896463872/64424509440), sparse 0% (270512128), duration 280, read/write 65/64 MB/s
INFO: status: 24% (15497035776/64424509440), sparse 0% (280989696), duration 297, read/write 35/34 MB/s
INFO: status: 25% (16116613120/64424509440), sparse 0% (291475456), duration 303, read/write 103/101 MB/s
INFO: status: 26% (16755195904/64424509440), sparse 0% (299864064), duration 314, read/write 58/57 MB/s
INFO: status: 27% (17427988480/64424509440), sparse 0% (314523648), duration 327, read/write 51/50 MB/s
INFO: status: 28% (18119786496/64424509440), sparse 0% (325136384), duration 339, read/write 57/56 MB/s
INFO: status: 29% (18708955136/64424509440), sparse 0% (335609856), duration 347, read/write 73/72 MB/s
INFO: status: 30% (19434962944/64424509440), sparse 0% (346050560), duration 356, read/write 80/79 MB/s
INFO: status: 31% (20031733760/64424509440), sparse 0% (356536320), duration 366, read/write 59/58 MB/s
INFO: status: 32% (20681719808/64424509440), sparse 0% (367013888), duration 374, read/write 81/79 MB/s
INFO: status: 33% (21278490624/64424509440), sparse 0% (375398400), duration 383, read/write 66/65 MB/s
INFO: status: 34% (21985492992/64424509440), sparse 0% (385880064), duration 395, read/write 58/58 MB/s
INFO: status: 35% (22608871424/64424509440), sparse 0% (396394496), duration 410, read/write 41/40 MB/s
INFO: status: 36% (23209443328/64424509440), sparse 0% (404295680), duration 423, read/write 46/45 MB/s
INFO: status: 37% (23962058752/64424509440), sparse 0% (416845824), duration 431, read/write 94/92 MB/s
INFO: status: 38% (24494211072/64424509440), sparse 0% (424464384), duration 445, read/write 38/37 MB/s
INFO: status: 39% (25144197120/64424509440), sparse 0% (440381440), duration 468, read/write 28/27 MB/s
INFO: status: 40% (25816989696/64424509440), sparse 0% (456495104), duration 483, read/write 44/43 MB/s
INFO: status: 41% (26421362688/64424509440), sparse 0% (461479936), duration 500, read/write 35/35 MB/s
INFO: status: 42% (27154972672/64424509440), sparse 0% (471912448), duration 516, read/write 45/45 MB/s
INFO: status: 43% (27717533696/64424509440), sparse 0% (480321536), duration 526, read/write 56/55 MB/s
INFO: status: 44% (28435939328/64424509440), sparse 0% (490696704), duration 545, read/write 37/37 MB/s
INFO: status: 45% (28994699264/64424509440), sparse 0% (498634752), duration 557, read/write 46/45 MB/s
INFO: status: 46% (29724508160/64424509440), sparse 0% (509804544), duration 571, read/write 52/51 MB/s
INFO: status: 47% (30290870272/64424509440), sparse 0% (518279168), duration 579, read/write 70/69 MB/s
INFO: status: 48% (30937055232/64424509440), sparse 0% (528793600), duration 607, read/write 23/22 MB/s
INFO: status: 49% (31571836928/64424509440), sparse 0% (541896704), duration 670, read/write 10/9 MB/s
INFO: status: 50% (32225624064/64424509440), sparse 0% (551583744), duration 732, read/write 10/10 MB/s
INFO: status: 51% (32883212288/64424509440), sparse 0% (566214656), duration 782, read/write 13/12 MB/s
INFO: status: 52% (33525596160/64424509440), sparse 0% (576049152), duration 812, read/write 21/21 MB/s
INFO: status: 53% (34145173504/64424509440), sparse 0% (587161600), duration 835, read/write 26/26 MB/s
INFO: status: 54% (34817966080/64424509440), sparse 0% (600674304), duration 861, read/write 25/25 MB/s
INFO: status: 55% (35502161920/64424509440), sparse 0% (608378880), duration 877, read/write 42/42 MB/s
INFO: status: 56% (36155949056/64424509440), sparse 0% (615321600), duration 893, read/write 40/40 MB/s
INFO: status: 57% (36733714432/64424509440), sparse 0% (623075328), duration 910, read/write 33/33 MB/s
INFO: status: 58% (37402705920/64424509440), sparse 0% (635371520), duration 926, read/write 41/41 MB/s
INFO: status: 59% (38075498496/64424509440), sparse 1% (646500352), duration 994, read/write 9/9 MB/s
INFO: status: 60% (38698876928/64424509440), sparse 1% (660832256), duration 1033, read/write 15/15 MB/s
INFO: status: 61% (39299448832/64424509440), sparse 1% (671342592), duration 1046, read/write 46/45 MB/s
INFO: status: 62% (39945633792/64424509440), sparse 1% (681799680), duration 1108, read/write 10/10 MB/s
INFO: status: 63% (40648835072/64424509440), sparse 1% (694296576), duration 1122, read/write 50/49 MB/s
INFO: status: 64% (41291218944/64424509440), sparse 1% (715304960), duration 1132, read/write 64/62 MB/s
INFO: status: 65% (41899393024/64424509440), sparse 1% (757923840), duration 1155, read/write 26/24 MB/s
INFO: status: 66% (42545577984/64424509440), sparse 1% (766287872), duration 1165, read/write 64/63 MB/s
INFO: status: 67% (43206967296/64424509440), sparse 1% (776753152), duration 1172, read/write 94/92 MB/s
INFO: status: 68% (43917770752/64424509440), sparse 1% (789299200), duration 1202, read/write 23/23 MB/s
INFO: status: 69% (44529745920/64424509440), sparse 1% (797687808), duration 1211, read/write 67/67 MB/s
INFO: status: 70% (45134118912/64424509440), sparse 1% (803979264), duration 1226, read/write 40/39 MB/s
INFO: status: 71% (45761298432/64424509440), sparse 1% (810266624), duration 1239, read/write 48/47 MB/s
INFO: status: 72% (46468300800/64424509440), sparse 1% (827019264), duration 1251, read/write 58/57 MB/s
INFO: status: 73% (47114485760/64424509440), sparse 1% (835411968), duration 1257, read/write 107/106 MB/s
INFO: status: 74% (47867101184/64424509440), sparse 1% (845897728), duration 1262, read/write 150/148 MB/s
INFO: status: 75% (48482877440/64424509440), sparse 1% (856162304), duration 1268, read/write 102/100 MB/s
INFO: status: 76% (48965615616/64424509440), sparse 1% (862187520), duration 1284, read/write 30/29 MB/s
INFO: status: 77% (49801854976/64424509440), sparse 1% (877060096), duration 1290, read/write 139/136 MB/s
INFO: status: 78% (50265587712/64424509440), sparse 1% (883462144), duration 1294, read/write 115/114 MB/s
INFO: status: 79% (50900369408/64424509440), sparse 1% (893960192), duration 1386, read/write 6/6 MB/s
INFO: status: 80% (51550355456/64424509440), sparse 1% (904527872), duration 1405, read/write 34/33 MB/s
INFO: status: 81% (52223148032/64424509440), sparse 1% (915030016), duration 1413, read/write 84/82 MB/s
INFO: status: 82% (52835123200/64424509440), sparse 1% (923484160), duration 1420, read/write 87/86 MB/s
INFO: status: 83% (53542125568/64424509440), sparse 1% (933949440), duration 1426, read/write 117/116 MB/s
INFO: status: 84% (54260531200/64424509440), sparse 1% (946319360), duration 1432, read/write 119/117 MB/s
INFO: status: 85% (54937124864/64424509440), sparse 1% (956928000), duration 1441, read/write 75/73 MB/s
INFO: status: 86% (55609917440/64424509440), sparse 1% (967507968), duration 1446, read/write 134/132 MB/s
INFO: status: 87% (56130666496/64424509440), sparse 1% (975962112), duration 1451, read/write 104/102 MB/s
INFO: status: 88% (56795856896/64424509440), sparse 1% (986570752), duration 1456, read/write 133/130 MB/s
INFO: status: 89% (57539166208/64424509440), sparse 1% (993218560), duration 1479, read/write 32/32 MB/s
INFO: status: 90% (58103431168/64424509440), sparse 1% (1001705472), duration 1483, read/write 141/138 MB/s
INFO: status: 91% (58734411776/64424509440), sparse 1% (1012305920), duration 1489, read/write 105/103 MB/s
INFO: status: 92% (59399602176/64424509440), sparse 1% (1022914560), duration 1498, read/write 73/72 MB/s
INFO: status: 93% (60057190400/64424509440), sparse 1% (1033490432), duration 1505, read/write 93/92 MB/s
INFO: status: 94% (60589342720/64424509440), sparse 1% (1041977344), duration 1510, read/write 106/104 MB/s
INFO: status: 95% (61262135296/64424509440), sparse 1% (1052585984), duration 1515, read/write 134/132 MB/s
INFO: status: 96% (61957734400/64424509440), sparse 1% (1063247872), duration 1521, read/write 115/114 MB/s
INFO: status: 97% (62497488896/64424509440), sparse 1% (1071661056), duration 1526, read/write 107/106 MB/s
INFO: status: 98% (63215894528/64424509440), sparse 1% (1082134528), duration 1531, read/write 143/141 MB/s
INFO: status: 99% (63907692544/64424509440), sparse 1% (1094688768), duration 1536, read/write 138/135 MB/s
INFO: status: 100% (64424509440/64424509440), sparse 1% (1105895424), duration 1565, read/write 17/17 MB/s
INFO: transferred 64424 MB in 1565 seconds (41 MB/s)
INFO: archive file size: 8.04GB
INFO: Finished Backup of VM 104 (00:26:06)
INFO: Backup job finished successfully
TASK OK

 

[fabian]

das einzige was dabei auffällig ist, sind die recht stark ausgelasteten platten sde/sdf

 

[HBO]

Wie kann es nun also zu dieser unterirdischen Leistung kommen? Haben wir hier generell ggfl auch ein Kernel Problem?
Auf diversen VMs sind solche Fehler zu sehen:
[764149.028642] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:0:31229]
[764149.028644] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [vballoon:66]

proxmox-ve: 5.0-15 (running kernel: 4.10.15-1-pve)
pve-manager: 5.0-23 (running version: 5.0-23/af4267bf)
pve-kernel-4.10.15-1-pve: 4.10.15-15

Die Last im "Normalbetrief" verteilt sich recht moderat auf dem System.

Verbaut ist folgender HBA: http://www.supermicro.com/products/accessories/addon/AOC-S3008L-L8e.cfm

 

[fabian]

update auf den aktuellen 4.10.17-3-pve kernel schadet sicher nicht. solche kernel messages in VMs könnten ein indiz auf allgemeine überlastung oder scheduling-probleme sein..