Moving 120 GB disk to ZFS pool taking hours.

F

FuriousGeorge

Renowned Member
Sep 25, 2012
84
2
73
As the title suggests, it's taking a long time.

I read that ZFS destroy operations may take hours, but I did not expect this to.

I'm concerned that when the VM does finish moving it will be unuseable.

I also have no idea how the server will perform, since CPU usage is very high. If I cancel the operation will I render the system broken? I did not click the delete source box.
 
It's done. Users are reporting slowness.

Code: 
# zpool status
  pool: pool0
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        pool0       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0

[/quote]

I'm taking a backup now and I'm going to restart the VM.
 
The backup made the VM unusable. I shut it down and started it, and the issue appears to be resolved.

I'm going to try a backup in a bit and see how long it takes.
 
Disk performance still seems to be very slow.

Code: 
[LIST=1]
[*]# pveperf /pool0/ && pveperf
[*]CPU BOGOMIPS:      51199.92
[*]REGEX/SECOND:      1604685
[*]HD SIZE:           775.22 GB (pool0)
[*]FSYNCS/SECOND:     76.87
[*]DNS EXT:           79.13 ms
[*]DNS INT:           73.84 ms (domain.com)
[*]CPU BOGOMIPS:      51199.92
[*]REGEX/SECOND:      1569251
[*]HD SIZE:           94.37 GB (/dev/dm-0)
[*]BUFFERED READS:    187.26 MB/sec
[*]AVERAGE SEEK TIME: 10.06 ms
[*]FSYNCS/SECOND:     2275.87
[*]DNS EXT:           102.68 ms
[*]DNS INT:           71.12 ms (domain.com)
[/LIST]

I'm confused because originally I had put this pool on a HW raid 1. I read in several places that this was a bad idea, and performance was terrible, so I moved the VM back to local storage. This resolved the performance issue, but I was running my VM on my / partition.

I want to use zfs for a specific reason, so I turned the disks to JBOD,as was suggested in several places. The kvm is at least usable now, whereas it was not with ZFS/hw raid, but I have another two VMs to move onto this partition, and I don't think it's going to hold up.
 
I can't remember the model of the disks in the ZFS partition. They were "business class" SATA disks (larger cache, only 7,200 rpm) from Seagate.

The root partition is still a HW raid array, I expect it to be faster, but not by over an order of magnitude.
 
I have a 5 drive ssd raidz1 zpool.

restoring a backup was very slow ,

adding a write-log [ zil / log ] ssd made the restore speed 22 times faster.

spinning disks should have log and cache drive. here is zpool status from an old system with a 6 3TB sata drives , log and cache.. note the intel ssd has two partitions.
Code: 
zpool status tank
  pool: tank
state: ONLINE
  scan: scrub repaired 0 in 1h47m with 0 errors on Sun Apr 10 03:17:58 2016
config:

  NAME  STATE  READ WRITE CKSUM
  tank  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  scsi-35000c500561d1b23  ONLINE  0  0  0
  scsi-35000c500634e436f  ONLINE  0  0  0
  mirror-1  ONLINE  0  0  0
  scsi-35000c500724d5ff7  ONLINE  0  0  0
  scsi-35000c500724d562b  ONLINE  0  0  0
  mirror-2  ONLINE  0  0  0
  scsi-35000c5005887afdf  ONLINE  0  0  0
  scsi-35000c500588762f3  ONLINE  0  0  0
  logs
  ata-INTEL_SSDSC2BA200G3_BTTV422201TM200GGN-part2  ONLINE  0  0  0
  cache
  ata-INTEL_SSDSC2BA200G3_BTTV422201TM200GGN-part1  ONLINE  0  0  0

errors: No known data errors
backup and restores go about 60MB/s .

we have 3 all ssd systems . backup speed is around 225MB/s per pve backup logs.

some drives are a lot better for zfs then others. check pve wiki for zfs links .
 
Right now I am moving a disk from local to zfs. It's 8GB. It's taking about 45 minutes and maxes CPU. The KVM with the 120 GB disk in my zpool that I opened this thread about has been brought to a crawl. This is a quad core xeon with 8 gb of ram. The Windows server in the KVM currently has one user doing nothing.

When I am not backing up or moving disks they report the system seems fast enough, but considering what they use it for, that is setting the bar pretty low.

The only reason I am using zfs is to use pve-zsync. If I need to get two other disks to do it, then it is not a viable solution. The load on the server is very low. It runs a pbx, a router, and a win 2k server for 4 users. There should be more than enough hardware available to it.

I'm wondering if they may be some conflict between zfs and the controller card, despite being in JBOD mode for the two zfs disks.
 
The wiki seems to imply that JBOD is ok.

I have an identical system that I'll set up and try to replicate the problem. Then I'll see if IT mode resolves it.

A vzdump to the pool is currently going at about 3mb per second.
 
use a log drive in the new system, and it is easy to add on the existing one.

on board sata in most cases is better then a raid card.

data protection comes 1ST and zfs is doing as fast as you've set it up to do. 3mb/sec system is what you made is almost as bad as one of the systems I made 6 years ago.

with zfs it is easily fixable without any data loss.
 
I'll give it a try.

Good news is old system is able to make backups normally after a reboot. Seems things are slowly normalizing.
 
I solved this. It was my fault. The wiki says:

It is good to use max 50 percent (which is default) of the system memory for ZFS arc to prevent performance shortage of the host.

Use your preferred editor to change the config in /etc/modprobe.d/zfs.conf and insert:

options zfs zfs_arc_max=4299967296
Click to expand...

I miscalculated, and ZFS was competing violently for ram, gumming up the entire works.
 
You must log in or register to reply here.
Top Bottom