ZFS - Restores

I've never seen this issue before but I'm doing a restore from a local spinner to a local ZFS volume.

It's taken the entire host node down to a crawl and while I can ping the node, I can't ssh into it.

I've never had this happen. I've seen some really slow behavior with ZFS reading and writing. What used to restore in 5 minutes over ext4 is now taking 40-50 minutes - or longer.

I'll have more information I believe when the restore is complete and I can get back into the machine - but right now it's killing me. The node and all of the vms just show a ? because the UI can't get any details.

Thoughts?

 

Here's the output - am I doing something wrong? 36 minutes for ~100 gigs... Worse - it took the entire server down. This is restoring from a local spinner to a local SSD.

Virtual Environment 5.2-5
Virtual Machine 100 (host6.x.com) on node 'mox2'
Logs
()
restore vma archive: lzop -d -c /home/backups//dump/vzdump-qemu-100-2018_07_12-18_30_02.vma.lzo | vma extract -v -r /var/tmp/vzdumptmp1215315.fifo - /var/tmp/vzdumptmp1215315
CFG: size: 572 name: qemu-server.conf
DEV: dev_id=1 size: 34359738368 devname: drive-virtio0
DEV: dev_id=2 size: 68719476736 devname: drive-virtio1
CTIME: Thu Jul 12 18:30:04 2018
new volume ID is 'Local-ZFS:vm-100-disk-1'
map 'drive-virtio0' to '/dev/zvol/Local-ZFS/vm-100-disk-1' (write zeros = 0)
new volume ID is 'Local-ZFS:vm-100-disk-2'
map 'drive-virtio1' to '/dev/zvol/Local-ZFS/vm-100-disk-2' (write zeros = 0)
progress 1% (read 1030815744 bytes, duration 6 sec)
progress 2% (read 2061631488 bytes, duration 10 sec)
progress 3% (read 3092381696 bytes, duration 15 sec)
progress 4% (read 4123197440 bytes, duration 21 sec)
progress 5% (read 5154013184 bytes, duration 26 sec)
progress 6% (read 6184763392 bytes, duration 31 sec)
progress 7% (read 7215579136 bytes, duration 35 sec)
progress 8% (read 8246394880 bytes, duration 40 sec)
progress 9% (read 9277145088 bytes, duration 47 sec)
progress 10% (read 10307960832 bytes, duration 53 sec)
progress 11% (read 11338776576 bytes, duration 59 sec)
progress 12% (read 12369526784 bytes, duration 65 sec)
progress 13% (read 13400342528 bytes, duration 71 sec)
progress 14% (read 14431092736 bytes, duration 79 sec)
progress 15% (read 15461908480 bytes, duration 85 sec)
progress 16% (read 16492724224 bytes, duration 91 sec)
progress 17% (read 17523474432 bytes, duration 97 sec)
progress 18% (read 18554290176 bytes, duration 103 sec)
progress 19% (read 19585105920 bytes, duration 109 sec)
progress 20% (read 20615856128 bytes, duration 116 sec)
progress 21% (read 21646671872 bytes, duration 123 sec)
progress 22% (read 22677487616 bytes, duration 130 sec)
progress 23% (read 23708237824 bytes, duration 136 sec)
progress 24% (read 24739053568 bytes, duration 143 sec)
progress 25% (read 25769803776 bytes, duration 169 sec)
progress 26% (read 26800619520 bytes, duration 202 sec)
progress 27% (read 27831435264 bytes, duration 235 sec)
progress 28% (read 28862185472 bytes, duration 263 sec)
progress 29% (read 29893001216 bytes, duration 291 sec)
progress 30% (read 30923816960 bytes, duration 291 sec)
progress 31% (read 31954567168 bytes, duration 291 sec)
progress 32% (read 32985382912 bytes, duration 291 sec)
progress 33% (read 34016198656 bytes, duration 291 sec)
progress 34% (read 35046948864 bytes, duration 310 sec)
progress 35% (read 36077764608 bytes, duration 356 sec)
progress 36% (read 37108580352 bytes, duration 382 sec)
progress 37% (read 38139330560 bytes, duration 423 sec)
progress 38% (read 39170146304 bytes, duration 445 sec)
progress 39% (read 40200896512 bytes, duration 474 sec)
progress 40% (read 41231712256 bytes, duration 486 sec)
progress 41% (read 42262528000 bytes, duration 521 sec)
progress 42% (read 43293278208 bytes, duration 554 sec)
progress 43% (read 44324093952 bytes, duration 583 sec)
progress 44% (read 45354909696 bytes, duration 609 sec)
progress 45% (read 46385659904 bytes, duration 653 sec)
progress 46% (read 47416475648 bytes, duration 678 sec)
progress 47% (read 48447291392 bytes, duration 703 sec)
progress 48% (read 49478041600 bytes, duration 738 sec)
progress 49% (read 50508857344 bytes, duration 761 sec)
progress 50% (read 51539607552 bytes, duration 792 sec)
progress 51% (read 52570423296 bytes, duration 819 sec)
progress 52% (read 53601239040 bytes, duration 868 sec)
progress 53% (read 54631989248 bytes, duration 896 sec)
progress 54% (read 55662804992 bytes, duration 922 sec)
progress 55% (read 56693620736 bytes, duration 961 sec)
progress 56% (read 57724370944 bytes, duration 986 sec)
progress 57% (read 58755186688 bytes, duration 1022 sec)
progress 58% (read 59786002432 bytes, duration 1046 sec)
progress 59% (read 60816752640 bytes, duration 1072 sec)
progress 60% (read 61847568384 bytes, duration 1106 sec)
progress 61% (read 62878384128 bytes, duration 1129 sec)
progress 62% (read 63909134336 bytes, duration 1165 sec)
progress 63% (read 64939950080 bytes, duration 1187 sec)
progress 64% (read 65970700288 bytes, duration 1213 sec)
progress 65% (read 67001516032 bytes, duration 1250 sec)
progress 66% (read 68032331776 bytes, duration 1276 sec)
progress 67% (read 69063081984 bytes, duration 1302 sec)
progress 68% (read 70093897728 bytes, duration 1325 sec)
progress 69% (read 71124713472 bytes, duration 1364 sec)
progress 70% (read 72155463680 bytes, duration 1376 sec)
progress 71% (read 73186279424 bytes, duration 1405 sec)
progress 72% (read 74217095168 bytes, duration 1469 sec)
progress 73% (read 75247845376 bytes, duration 1479 sec)
progress 74% (read 76278661120 bytes, duration 1512 sec)
progress 75% (read 77309411328 bytes, duration 1543 sec)
progress 76% (read 78340227072 bytes, duration 1575 sec)
progress 77% (read 79371042816 bytes, duration 1596 sec)
progress 78% (read 80401793024 bytes, duration 1629 sec)
progress 79% (read 81432608768 bytes, duration 1651 sec)
progress 80% (read 82463424512 bytes, duration 1689 sec)
progress 81% (read 83494174720 bytes, duration 1717 sec)
progress 82% (read 84524990464 bytes, duration 1743 sec)
progress 83% (read 85555806208 bytes, duration 1778 sec)
progress 84% (read 86586556416 bytes, duration 1805 sec)
progress 85% (read 87617372160 bytes, duration 1836 sec)
progress 86% (read 88648187904 bytes, duration 1859 sec)
progress 87% (read 89678938112 bytes, duration 1882 sec)
progress 88% (read 90709753856 bytes, duration 1914 sec)
progress 89% (read 91740504064 bytes, duration 1938 sec)
progress 90% (read 92771319808 bytes, duration 1959 sec)
progress 91% (read 93802135552 bytes, duration 1988 sec)
progress 92% (read 94832885760 bytes, duration 2022 sec)
progress 93% (read 95863701504 bytes, duration 2044 sec)
progress 94% (read 96894517248 bytes, duration 2075 sec)
progress 95% (read 97925267456 bytes, duration 2101 sec)
progress 96% (read 98956083200 bytes, duration 2124 sec)
progress 97% (read 99986898944 bytes, duration 2151 sec)
progress 98% (read 101017649152 bytes, duration 2151 sec)
progress 99% (read 102048464896 bytes, duration 2162 sec)
progress 100% (read 103079215104 bytes, duration 2162 sec)
total bytes read 103079215104, sparse bytes 9391480832 (9.11%)
space reduction due to 4K zero blocks 0.202%
TASK OK

 

Hi Nemesiz - not sure if this is helpful or what you are looking for - and I appreciate you reaching out and trying to help!

Just a single disk zpool RAID-0.

dir: local
path /var/lib/vz
content rootdir,images,vztmpl,iso
maxfiles 0

dir: Local-Backups
path /home/backups/
content iso,images,rootdir,vztmpl,backup
maxfiles 3

rbd: ILStore1
content rootdir,images
krbd 0
monhost 172.16.0.46 172.16.0.47 172.16.0.48
nodes mox3,mox0,mox1,mox2
pool ILStore1
username admin

zfspool: Local-ZFS
pool Local-ZFS
content rootdir,images
nodes mox1,mox3,mox0,mox2
sparse 1

dir: Local-SSD-ZFS
disable
path /Local-SSD-ZFS/storage
content iso,vztmpl,images,rootdir
nodes mox0
shared 0

zfspool: rpool-SSD
pool rpool
content images,rootdir
nodes mox0
sparse 1


pool: Local-ZFS
state: ONLINE
scan: scrub repaired 0B in 5h50m with 0 errors on Sun Jul 8 06:14:16 2018
config:

NAME STATE READ WRITE CKSUM
Local-ZFS ONLINE 0 0 0
sdb1 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0h4m with 0 errors on Sun Jul 8 00:28:28 2018
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
sda2 ONLINE 0 0 0

errors: No known data errors


NAME USED AVAIL REFER MOUNTPOINT
Local-ZFS 575G 324G 144K /Local-ZFS
Local-ZFS/dump 96K 324G 96K /Local-ZFS/dump
Local-ZFS/image 96K 324G 96K /Local-ZFS/image
Local-ZFS/iso 96K 324G 96K /Local-ZFS/iso
Local-ZFS/private 96K 324G 96K /Local-ZFS/private
Local-ZFS/storage 96K 324G 96K /Local-ZFS/storage
Local-ZFS/template 96K 324G 96K /Local-ZFS/template
Local-ZFS/vm-100-disk-1 27.8G 324G 27.0G -
Local-ZFS/vm-100-disk-2 61.0G 324G 61.0G -
Local-ZFS/vm-109-disk-1 32.3G 324G 32.2G -
Local-ZFS/vm-109-disk-2 406G 324G 404G -
Local-ZFS/vm-110-disk-1 23.8G 324G 23.8G -
Local-ZFS/vm-111-disk-1 23.8G 324G 23.8G -
rpool 38.8G 186G 96K /rpool
rpool/ROOT 7.77G 186G 96K /rpool/ROOT
rpool/ROOT/pve-1 7.77G 186G 7.77G /
rpool/swap 30.8G 188G 29.1G -


root@mox2:~# zpool status
pool: Local-ZFS
state: ONLINE
scan: scrub repaired 0B in 5h50m with 0 errors on Sun Jul 8 06:14:16 2018
config:

NAME STATE READ WRITE CKSUM
Local-ZFS ONLINE 0 0 0
sdb1 ONLINE 0 0 0

errors: No known data errors

zpool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0h4m with 0 errors on Sun Jul 8 00:28:28 2018
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
sda2 ONLINE 0 0 0

errors: No known data errors
root@mox2:~# zpool iostat
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
Local-ZFS 573G 355G 518 103 12.0M 6.98M
rpool 37.0G 195G 31 44 212K 597K
---------- ----- ----- ----- ----- ----- -----
root@mox2:~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
Local-ZFS 928G 573G 355G - 39% 61% 1.00x ONLINE -
rpool 232G 37.0G 195G - 53% 15% 1.00x ONLINE -

 

NAME PROPERTY VALUE SOURCE
rpool type filesystem -
rpool creation Thu Mar 3 10:11 2016 -
rpool used 38.8G -
rpool available 186G -
rpool referenced 96K -
rpool compressratio 1.14x -
rpool mounted yes -
rpool quota none default
rpool reservation none default
rpool recordsize 128K default
rpool mountpoint /rpool default
rpool sharenfs off default
rpool checksum on default
rpool compression lz4 local
rpool atime off local
rpool devices on default
rpool exec on default
rpool setuid on default
rpool readonly off default
rpool zoned off default
rpool snapdir hidden default
rpool aclinherit restricted default
rpool createtxg 1 -
rpool canmount on default
rpool xattr on default
rpool copies 1 default
rpool version 5 -
rpool utf8only off -
rpool normalization none -
rpool casesensitivity sensitive -
rpool vscan off default
rpool nbmand off default
rpool sharesmb off default
rpool refquota none default
rpool refreservation none default
rpool guid 9946285014865877177 -
rpool primarycache all default
rpool secondarycache all default
rpool usedbysnapshots 0B -
rpool usedbydataset 96K -
rpool usedbychildren 38.8G -
rpool usedbyrefreservation 0B -
rpool logbias latency default
rpool dedup off default
rpool mlslabel none default
rpool sync standard local
rpool dnodesize legacy default
rpool refcompressratio 1.00x -
rpool written 96K -
rpool logicalused 41.6G -
rpool logicalreferenced 40K -
rpool volmode default default
rpool filesystem_limit none default
rpool snapshot_limit none default
rpool filesystem_count none default
rpool snapshot_count none default
rpool snapdev hidden default
rpool acltype off default
rpool context none default
rpool fscontext none default
rpool defcontext none default
rpool rootcontext none default
rpool relatime off default
rpool redundant_metadata all default
rpool overlay off default

 

And then: Local-ZFS

root@mox2:~# zfs get all Local-ZFS
NAME PROPERTY VALUE SOURCE
Local-ZFS type filesystem -
Local-ZFS creation Tue Jun 19 23:55 2018 -
Local-ZFS used 573G -
Local-ZFS available 326G -
Local-ZFS referenced 144K -
Local-ZFS compressratio 1.00x -
Local-ZFS mounted yes -
Local-ZFS quota none default
Local-ZFS reservation none default
Local-ZFS recordsize 128K default
Local-ZFS mountpoint /Local-ZFS default
Local-ZFS sharenfs off default
Local-ZFS checksum on default
Local-ZFS compression off default
Local-ZFS atime on default
Local-ZFS devices on default
Local-ZFS exec on default
Local-ZFS setuid on default
Local-ZFS readonly off default
Local-ZFS zoned off default
Local-ZFS snapdir hidden default
Local-ZFS aclinherit restricted default
Local-ZFS createtxg 1 -
Local-ZFS canmount on default
Local-ZFS xattr on default
Local-ZFS copies 1 default
Local-ZFS version 5 -
Local-ZFS utf8only off -
Local-ZFS normalization none -
Local-ZFS casesensitivity sensitive -
Local-ZFS vscan off default
Local-ZFS nbmand off default
Local-ZFS sharesmb off default
Local-ZFS refquota none default
Local-ZFS refreservation none default
Local-ZFS guid 6229891795844742391 -
Local-ZFS primarycache all default
Local-ZFS secondarycache all default
Local-ZFS usedbysnapshots 0B -
Local-ZFS usedbydataset 144K -
Local-ZFS usedbychildren 573G -
Local-ZFS usedbyrefreservation 0B -
Local-ZFS logbias latency default
Local-ZFS dedup off default
Local-ZFS mlslabel none default
Local-ZFS sync standard default
Local-ZFS dnodesize legacy default
Local-ZFS refcompressratio 1.00x -
Local-ZFS written 144K -
Local-ZFS logicalused 570G -
Local-ZFS logicalreferenced 60.5K -
Local-ZFS volmode default default
Local-ZFS filesystem_limit none default
Local-ZFS snapshot_limit none default
Local-ZFS filesystem_count none default
Local-ZFS snapshot_count none default
Local-ZFS snapdev hidden default
Local-ZFS acltype off default
Local-ZFS context none default
Local-ZFS fscontext none default
Local-ZFS defcontext none default
Local-ZFS rootcontext none default
Local-ZFS relatime off default
Local-ZFS redundant_metadata all default
Local-ZFS overlay off default

 

Thanks Nemesiz... They are Samsumg 850 Pro's - but there are better/stronger/faster.

I read these threads yesterday:
https://forum.proxmox.com/threads/zfs-sync-disabled.37900/
https://forum.proxmox.com/threads/p...-ssd-drives-sync-parameter.31130/#post-155543

So basically on the pve host nodes I should enter:
zfs set sync=disabled

?

And no changes to vms and their caching correct? I typically use writethrough

 

Thanks - done. Does this take effect immediately or does the host or ZFS need to be reinitialized?

 

Thanks again for your help. I'm going to let things run like this for a day or two and do some testing tonight to see how things behave. Fingers crossed.