omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating ct)

kumarullal

Renowned Member
Jun 17, 2009
184
0
81
LA, USA
After I created an NFS share on the pool (named omnios). In order to work with openvz storage Idid the following


chmod 777 -R poolname/sharename
example
chmod 777 -R omnios/prox3ovz

This gives recursive permission to all the subfolders under omnios pool
Next I did the following


zfs set aclmode=passthrough omnios/prox3ovz

zfs set aclinherit=passthrough-x omnios/prox3ovs


zfs set sharenfs=rw,noaclfab,root_mapping=0,nosuid,root=@192.168.1.0/24 omnios/prox3ovz


KVM works just fine.
Openvz has issues. I created a CT from an ubuntu template (size 122MB). It took 35 minutes. The same template when stored on local storage it was 20 seconds.
Then I backed up the container. It tool 30 minutes.

WHen I backed up a KVM of 230 GB on nfs share it only tool 14 minutes. (Backup size of 26 GB)
So my question is why is it so slow when it comes to openvz stored on NFS share.
By the way. Once the container was created, it worked flawlessly form the NFS share. It is only the backup restore and creation of CT from template that takes extremely long.
Any ideas?
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

Found a partial answer in dramatically speedup vzdump for openvz.
All you do is edit the /etc/vzdump.conf file and enter this parameter.
tmpdir: /tmp

This will increase your vzdump speed for container dramatically.
With this settings, I reduced my backup time from 37 minutes to 20 seconds!!!!!
This is because
Proxmox rsyncs your local OpenVZ folder to the NFS target and zips it
afterwards. As the OpenVZ folder contains many many small files, this
takes quiet a long time.
So only the large gzip file will be transferred to your NFS share and not every small file.
The source of this solution is at https://dev-eth0.de/increase-performance-proxmox-backups-nfs-target/
However, the time to create a container is still taking a long time.
Any ideas?
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

Run the following script and get an idea of what is causing the slow creation of a VZ

Code:
#!/bin/bash


SHARE=$1


if [ -z $SHARE ]; then
        echo "Usage: $0 SHARE"
        echo "Eg. /mnt/pve/my_nfs_share then SHARE=my_nfs_share"
        echo "So: $0 my_nfs_share"
        exit 1
fi


four_k_files () {
START=$(date +%s)
for i in `seq 5000`
do
        dd if=/dev/zero of=/mnt/pve/$SHARE/vztmp/4k_$i.img bs=4k count=1 2> /dev/null
done
END=$(date +%s)
echo -n "Time: "
echo $((END-START)) | awk '{print int($1/60)":"int($1%60)}'
}

one_m_files () {
START=$(date +%s)
for i in `seq 20`
do
        dd if=/dev/zero of=/mnt/pve/$SHARE/vztmp/1m_$i.img bs=1M count=1 2> /dev/null
done
END=$(date +%s)
echo -n "Time: "
echo $((END-START)) | awk '{print int($1/60)":"int($1%60)}'
}


echo "Creating 20MB file in chunks of 4k"
START=$(date +%s)
dd if=/dev/zero of=/mnt/pve/$SHARE/vztmp/dd.img bs=4k count=5000 2> /dev/null
END=$(date +%s)
echo -n "Time: "
echo $((END-START)) | awk '{print int($1/60)":"int($1%60)}'
rm /mnt/pve/$SHARE/vztmp/dd.img

echo "Creating 20MB file in chunks of 1M"
START=$(date +%s)
dd if=/dev/zero of=/mnt/pve/$SHARE/vztmp/dd.img bs=1M count=20 2> /dev/null
END=$(date +%s)
echo -n "Time: "
echo $((END-START)) | awk '{print int($1/60)":"int($1%60)}'
rm /mnt/pve/$SHARE/vztmp/dd.img


echo "Creating 5000 files (20MB)of size 4k"
four_k_files
rm -f /mnt/pve/$SHARE/vztmp/4k_*.img


echo "Creating 20 files (20Mb) of size 1M"
one_m_files
rm -f /mnt/pve/$SHARE/vztmp/1m_*.img
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

Thanks, mir, for your reply. I will run the script and let you know.
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

This is what mine shows:
# sh dd_test.sh omnios_nfs
Creating 20MB file in chunks of 4k
Time: 0:0
Creating 20MB file in chunks of 1M
Time: 0:0
Creating 5000 files (20MB)of size 4k
Time: 0:32
Creating 20 files (20Mb) of size 1M
Time: 0:0
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

This is what mine looks like
./test.sh prox3ovz
Creating 20MB file in chunks of 4k
Time: 0:1
Creating 20MB file in chunks of 1M
Time: 0:1
Creating 5000 files (20MB)of size 4k
Time: 1:37
Creating 20 files (20Mb) of size 1M
Time: 0:1

What can be infferred from this result?
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

Since creating a CT requires copying a lot of small files from the proxmox server to the NFS server then creating a CT can be compared to the test case which creates 5000 files of size 4k. Eg. in your case creating one file of size 20MB in chunks of 4k is 97 times faster than creating 5000 files of size 4k.
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

So what do you suggest I do to increase the speed ?
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

The only way to increase speed is to add more disks and/or RAM to your Omnios system. Alternatively add a SSD log.
What kind of disks do you have? brand and model?

what is your sync setting? always, standard, disabled?

What does this show: zpool status
 
Last edited:
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

You mean on napp-it web interface? Yes, sync is set to standard.
My zpool status shows

zpool status
pool: omnios
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
omnios ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
c4t2d0 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0

errors: No known data errors
 
Last edited:
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

And What kind of disks do you have? brand and model?

How much RAM does the omnios have?
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

Are you aware of the fact that if you loose one of the disks in your pool you will loose the entire pool?
 
Re: omnios napp-it nfs share second thoughts. KVM ok, openvz tooooooo slow (Creating

I Have seagate 2 x 2TB 7200 rpm drives.
Yes I am aware since I have no mirror, I can loose the entire pool if I loose one disk.
One interesting finding. I have another storage on desktop, with 2 x 2TB WD Sata drive with 5400 rpm. On this one, I have openindiana and napp-it.
When I ran your script there, the results were much better
Here is the output.
./test.sh vzbu240
Creating 20MB file in chunks of 4k
Time: 0:1
Creating 20MB file in chunks of 1M
Time: 0:0
Creating 5000 files (20MB)of size 4k
Time: 1:25
Creating 20 files (20Mb) of size 1M
Time: 0:1


Isnt that strange? Both have identical configuration.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!