High (100%) ZVOL CPU usage when doing VM import from backup.

M

mailinglists

Renowned Member
Mar 14, 2012
641
68
93
Hi,

i did some testing yesterday with PM 5. to see how fast can I import some big VMs from PM 4.
So I setup a NFS VM in new cluster on a node where I will import VMs. I defined it as "exportVM" storage on PM 4 and PM 5 cluster concurrently.
I exported VMs (backup with GUI) from PM 4 to NFS VM. There were no issues here.
Then I started to import this backup vzdump made from this export storage (NFS VM on the same host as destination restore) via PM 5 GUI without any limits.
Quickly after the start, I noticed that two CPUs and all cores were at 100% most of the time, used by zvol kernel process with priority -20.
I guess I could manually reduce the number of cores ZFS can use, but I do not think this should happen anyway and is the right way to do it.
I will do another import right now and do some screenshots as well as host info to give you some more data.
 
So host is:
Code: 
24 x Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2 Sockets)
62.90 GiB RAM
Linux 4.15.18-16-pve #1 SMP PVE 4.15.18-41 (Tue, 18 Jun 2019 07:36:54 +0200)
pve-manager/5.4-7/fc10404a
2 x Intel DC S3510 Series 1.6TB
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda3    ONLINE       0     0     0
            sdb3    ONLINE       0     0     0
ARC Size:                95.72%    3.83    GiB
    Target Size: (Adaptive)        100.00%    4.00    GiB
    Min Size (Hard Limit):        97.67%    3.91    GiB
    Max Size (High Water):        1:1    4.00    GiB

On it runs just this NFS VM:
Code: 
agent: 1
bootdisk: scsi0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 1024
name: s203
net0: virtio=XXXX,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-141-disk-0,discard=on,iops_rd=333,iops_wr=333,mbps_rd=100,mbps_wr=100,size=15G
scsi1: local-zfs:vm-141-disk-1,backup=0,discard=on,size=666G
scsihw: virtio-scsi-pci
smbios1: uuid=c1232fce-0a3b-445e-ad7a-49fa7b43cf0f

So after I start the import from such a backup:
Code: 
scsi0: local-lvm-sata:vm-100-disk-2,cache=writeback,discard=on,size=52G
scsi1: local-lvm-sata:vm-100-disk-1,cache=writeback,discard=on,size=600G

This happens:

Zaslonska slika 2019-08-23 14-37-35.pngZaslonska slika 2019-08-23 14-36-57.pngZaslonska slika 2019-08-23 14-43-03.pngZaslonska slika 2019-08-23 14-42-35.png

From import log.
Code: 
DEV: dev_id=1 size: 55834574848 devname: drive-scsi0
DEV: dev_id=2 size: 644245094400 devname: drive-scsi1
CTIME: Thu Aug 22 15:16:56 2019
new volume ID is 'local-zfs:vm-142-disk-0'
map 'drive-scsi0' to '/dev/zvol/rpool/data/vm-142-disk-0' (write zeros = 0)
new volume ID is 'local-zfs:vm-142-disk-1'
map 'drive-scsi1' to '/dev/zvol/rpool/data/vm-142-disk-1' (write zeros = 0)
progress 1% (read 7000817664 bytes, duration 31 sec)
progress 2% (read 14001635328 bytes, duration 75 sec)
progress 3% (read 21002452992 bytes, duration 112 sec)
progress 4% (read 28003205120 bytes, duration 155 sec)
progress 5% (read 35004022784 bytes, duration 212 sec)
progress 6% (read 42004840448 bytes, duration 216 sec)
progress 7% (read 49005592576 bytes, duration 217 sec)
progress 8% (read 56006410240 bytes, duration 217 sec)
progress 9% (read 63007227904 bytes, duration 248 sec)

Besides high CPU usage, I notice high IO delay 23+%, but when IO delay is high, CPU usage is low.
Also load average is above 30+.

When talking about cores ZFS can use, I was talking about this parameter:
https://github.com/zfsonlinux/zfs/wiki/ZFS-on-Linux-Module-Parameters#zvol_threads

If I have the time, I will not try to decrease it to 12, to see what happens to CPU usage, but I would also have to do some additional benchmarks for which I certainly do not have the time.

Do you guys think this high ZVOL CPU usage is normal and expected and I should just try to limit import speed?
 
Hi,

Zvol is not so fast and performance could be bad. Try to increase zvolblocksize(default 8k) to a 16-64 k for vm with nfs.

I would not use nfs on the zfs. Try to use for your VM a zfs folder (using raw file format for vm).
 
Hmm... I don't think I understand correctly.
Do you think high CPU usage with ZVOL on restore is due to the fact that data disk of ZFS VM is actual zvol?
And that I should change data disk for this NFS VM from ZVOL to RAW file on ZFS dataset?

When I have some time and I put additional servers to the cluster, I will play with this a bit. For now, unless I do a import on the same host, I see no problems with high ZVOL CPU usage.

Thank you for your input @guletz , now i think the other ZFS "power user" :-) @udo should chip in with his opinion also.. :-)
 
mailinglists said:
Hmm... I don't think I understand correctly.
Do you think high CPU usage with ZVOL on restore is due to the fact that data disk of ZFS VM is actual zvol?
Click to expand...

Partial yes. 3 facts will help for this:

- small 8k volblocksize
- ... so many metadata(in fact this is your problem: until your zfs metadata is under zfs max metadata limit then your speed is ok, but after you are over this limit, the zfs will start to go down the seed, add anothers metadata in Ram and so on)
- many levels of storage

mailinglists said:
And that I should change data disk for this NFS VM from ZVOL to RAW file on ZF
Click to expand...

Yes. zfs datasets are lighting faster compared with zvol.

PS. Yes @udo have always a good knowledge about .... anything. And anybody can learn a lot from his posts (me included)
 
Last edited:
mailinglists said:
So host is:
Code: 
24 x Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2 Sockets)
62.90 GiB RAM
Linux 4.15.18-16-pve #1 SMP PVE 4.15.18-41 (Tue, 18 Jun 2019 07:36:54 +0200)
pve-manager/5.4-7/fc10404a
2 x Intel DC S3510 Series 1.6TB
        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda3    ONLINE       0     0     0
            sdb3    ONLINE       0     0     0
ARC Size:                95.72%    3.83    GiB
    Target Size: (Adaptive)        100.00%    4.00    GiB
    Min Size (Hard Limit):        97.67%    3.91    GiB
    Max Size (High Water):        1:1    4.00    GiB

On it runs just this NFS VM:
Code: 
agent: 1
bootdisk: scsi0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 1024
name: s203
net0: virtio=XXXX,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-141-disk-0,discard=on,iops_rd=333,iops_wr=333,mbps_rd=100,mbps_wr=100,size=15G
scsi1: local-zfs:vm-141-disk-1,backup=0,discard=on,size=666G
scsihw: virtio-scsi-pci
smbios1: uuid=c1232fce-0a3b-445e-ad7a-49fa7b43cf0f

So after I start the import from such a backup:
Code: 
scsi0: local-lvm-sata:vm-100-disk-2,cache=writeback,discard=on,size=52G
scsi1: local-lvm-sata:vm-100-disk-1,cache=writeback,discard=on,size=600G

This happens:

View attachment 11383View attachment 11384View attachment 11385View attachment 11386

From import log.
Code: 
DEV: dev_id=1 size: 55834574848 devname: drive-scsi0
DEV: dev_id=2 size: 644245094400 devname: drive-scsi1
CTIME: Thu Aug 22 15:16:56 2019
new volume ID is 'local-zfs:vm-142-disk-0'
map 'drive-scsi0' to '/dev/zvol/rpool/data/vm-142-disk-0' (write zeros = 0)
new volume ID is 'local-zfs:vm-142-disk-1'
map 'drive-scsi1' to '/dev/zvol/rpool/data/vm-142-disk-1' (write zeros = 0)
progress 1% (read 7000817664 bytes, duration 31 sec)
progress 2% (read 14001635328 bytes, duration 75 sec)
progress 3% (read 21002452992 bytes, duration 112 sec)
progress 4% (read 28003205120 bytes, duration 155 sec)
progress 5% (read 35004022784 bytes, duration 212 sec)
progress 6% (read 42004840448 bytes, duration 216 sec)
progress 7% (read 49005592576 bytes, duration 217 sec)
progress 8% (read 56006410240 bytes, duration 217 sec)
progress 9% (read 63007227904 bytes, duration 248 sec)

Besides high CPU usage, I notice high IO delay 23+%, but when IO delay is high, CPU usage is low.
Also load average is above 30+.

When talking about cores ZFS can use, I was talking about this parameter:
https://github.com/zfsonlinux/zfs/wiki/ZFS-on-Linux-Module-Parameters#zvol_threads

If I have the time, I will not try to decrease it to 12, to see what happens to CPU usage, but I would also have to do some additional benchmarks for which I certainly do not have the time.

Do you guys think this high ZVOL CPU usage is normal and expected and I should just try to limit import speed?
Click to expand...
Hi,
if I understand you right, your read the compredd data from the same zfs-pool (inside an VM), where you write the output?
If you look for the throughput: (read 14001635328 bytes, duration 75 sec - read 7000817664 bytes, duration 31 sec)/44/(1024*1024) you got 151MB/s. (tihs is readed, uncompressed and written - readed and written on the same disks)
Unfortunality ZOL is slower than zfs on Solaris/BSD, I think ths values are normal for an zraid-1 with DC S3510-SSDs.

You wrote about "big VMs" - how big are big?
To check, if it's more sync on write you can (only for testing!!) change sync to async like
Code: 
zfs get sync rpool/vm-900-disk-0
NAME                    PROPERTY  VALUE     SOURCE
testpool/vm-900-disk-0  sync      standard  default

zfs set sync=disabled rpool/vm-900-disk-0
Udo
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom