By upcoming I meant just the next version. As soon as the patch is merged to master I was planning to compile ZFS from source and make upgrade as right now I'm still unable to do any real backups and I need to resolve this ASAP.
Is there any danger in doing that or what would be your...
Actual problem seems to be described here:
https://github.com/zfsonlinux/zfs/issues/6171
@manu It seems as it could be fixed in upcoming 0.7.6 (maybe) so I would ask Proxmox staff to upgrade to this version ASAP because current version of ZFS is definitely not production ready.
Ok back to some another tests
Command:
rsync -r --progress /DP1/backups/ /DT3btrfs/backups/
Will cause system to start hanging (as usually) within 10 seconds after launch, I have to kill it pretty quickly.
sdd & sde are main data SSD (where data are copied from)
sdf & sdg are system SSD
sda...
@czechsys @manu I do not know why I didn't though about it before but I had idea to actually test speed from inside container VM/CT and what ever test I do from inside of container they are exactly what I would assume speed should be on this system:
# dd if=/dev/zero of=/tmp/test bs=1G count=10...
FIO on main storage SSD:
# full write pass - there is definitely problem - server slowing down
fio --name=writefile --size=10G --filesize=10G --filename=/DP1/test --bs=1M --nrfiles=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 --iodepth=200 --ioengine=libaio
writefile...
Even setting backup/copy to lowest possible priority and limiting speed does not help:
nice -n 19 ionice -c2 -n7 rsync --bwlimit=51200 -r --progress /DP1/backups/ /DT3btrfs/backups/
Setting:
vm.dirty_ratio = 5
vm.dirty_background_ ratio = 1
Seems to help slightly but it may as well be random...
I thing that I'm experiencing same unresolved problem:
https://forum.proxmox.com/threads/kvm-guests-freeze-hung-tasks-during-backup-restore-migrate.34362/
After many more tests and many more tuned values I'd given up on idea that It is FreeNAS problem completely, nothing I did on FreeNAS side changed the behavior.
Today I added two 500 GB spinning HDD to the Proxmox server and put them to RAID 1 (ZFS).
And surprise surprise problem is still there...
Thank you for tips, yes I read large portion of FreeNAS documentation and I have multiple FreeNAS boxes, it's true that only this one does not have ZIL on SSD, but I already tested transfers with disabled ZIL on given dataset to rule it out, and I also had NFS set to async.
And still why It...
I never thought about testing SSD because when I initially build the server and tested them they worked great:
dd if=/dev/zero of=/DP1/dump/test bs=1G count=2
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.26704 s, 1.7 GB/s
But as there is larger amount of data:
dd...
I'm running 6 x CT and 2 x VM.
What would be best way to do benchmark so It would give us proper/all informations? Just try to write bunch of data with dd?
Normally Server Load is around 1,5-2 so I didn't yet consider that there would be problem.
Thank you!
This is what happens during last two test transfers, system hanged so that graphs are not showing at all. The one that I mentioned in last post was around 13:00 and spike there is clearly visible, but I manage to kill it before it hang system as well.
I did:
echo 1073741824 >> /sys/module/zfs/parameters/zfs_arc_min
And It seems that dips are little lower (during dips instead of 2-10MB/s transfer speed I now get 25MB/s) system still feels very sluggish after few minutes of transfer.
Only thing that I noticed that wasn't there before (because...
When I do
ps faxl | grep " D "
Result is:
1 0 6613 2 20 0 0 0 io_sch D ? 0:02 \_ [txg_sync]
0 0 27891 6353 20 0 12788 936 pipe_w S+ pts/6 0:00 \_ grep D
But not always, txg_sync could be related to...
I do not have SLOG but I have 6x 4 TB disk in RAID 10 (3 groups with 2 discs) And according to usage graphs in Freenas they do not have problem to sustain the load, load drops down immediately as transfer starts loosing speed. The disc load doesn't go to more than 70 % and neither discs...
I will test that (ps faxl).
Yes I have lot of hang messages, sorry about not mentioned it earlier I totally forgot that I have that screenshot:
It reports folowing hang tasks (more than 120s):
kthreadd
khungepaged
kswapd
z_wr_iss
txg_sync
ksmtuned
usbhid-ups
cfs-loop
(it seems like all over the...
EDIT: As I find out that high IO will only result in freezing while performed on host, not guests please skip to post #19 as there are new findings.
I'm basically reposting issue that I have with backups to FreeNAS from this post...
Now I tried:
pct clone 102 101 -experimental true -storage vmContainer
(101 is not used)
and got error:
400 Parameter verification failed.
full: missing property - 'storage' requiers this property
What am I missing now?
So simply coping container to different location and changing config file of CT works - to certain degree.
Newly copied container will not aper in GUI, and in general Proxmox doesn't seam to know about this movement at all.
There is command "pct clone" but I can't find documentation for what...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.