[SOLVED] Linux NFS or GlusterFS Server not creating sparse files

gbr

Active Member
May 13, 2012
125
2
38
Hi,

I have an Ubuntu 12.04.3 server as an NFS server. I use the kernel server. When I create a qcow2 image, the file file gets created and Proxmox times out waiting for a lock. I've tried with both NFSv3 and NFSv4. NFSV4 is considerably faster at the disk image creation, but not fast enough.

# ls -lh
total 33G
-rw-r--r-- 1 root root 33G Jan 6 17:09 vm-102-disk-1.qcow2

# du -h --apparent-size

33G .

# ls -lks

total 34079304
34079304 -rw-r--r-- 1 root root 33559808 Jan 6 17:09 vm-102-disk-1.qcow2


# du --block-size=1 vm-102-disk-1.qcow2
34897207296 vm-102-disk-1.qcow2

Is anyone able to use the Linux NFS kernel server and get sparse qcow2 files created succesfully? If so, how? What am I missing?

Gerald

NFS over XFS


# pveversion --verbose
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-24 (running version: 3.1-24/060bd5a6)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-9
libpve-access-control: 3.0-8
libpve-storage-perl: 3.0-18
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-6
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1
 
Last edited:

gbr

Active Member
May 13, 2012
125
2
38
Re: Linux NFS Server not creating sparse files

I just tested with

# dd if=/dev/zero of=test bs=1 count=0 seek=8G

on the Proxmox server using the NFS mount (/mnt/pve/Images), and the sparse file was created. Is Proxmox doing something weird and not creating a sparse file, or am I misunderstanding? If Proxmox is not creating a sparse file, then it needs to set it's timeout better, since my NFS server on a 1 gig network link gets pretty close to wire speed.
 

gbr

Active Member
May 13, 2012
125
2
38
Re: Linux NFS Server not creating sparse files

I created a VM with a raw image, and it was created as sparse. An 'ls' showed 32GB disk, and a du -s showed 0 bytes. The raw disk image is now growing as I use it.
 

gbr

Active Member
May 13, 2012
125
2
38
Re: Linux NFS Server not creating sparse files

Is anyone using qcow2 with and NFS server and Proxmox 3.1?
 

gbr

Active Member
May 13, 2012
125
2
38
Re: Linux NFS Server not creating sparse files

Yes. What would you like to know?

What are you using as an NFS Server? Do your qcow2 files get created sparse? Do you get lock timeouts when creating a qcow2 disk?

Gerald
 

mir

Famous Member
Apr 14, 2012
3,553
116
83
Copenhagen, Denmark
Re: Linux NFS Server not creating sparse files

I use Omnios (Solaris offspring) providing NFS through ZFS' NFS implementation. I use qcow2 over NFS and raw over iSCSI. qcow2 is created sparse without timeouts. My storage is configured as RAID10 connected through a LSI HBA.
 

gbr

Active Member
May 13, 2012
125
2
38
Re: Linux NFS Server not creating sparse files

This is not a Linux NFS issue (or a GlusterFS issue) at all. It's the underlying filesystem issue.

If you use NFS or GlusterFS on top of XFS, XFS needs special mount options to create qcow2 files quickly. Here is my fstab:

/dev/md2 /export/nfs/RAID-6 xfs noatime,inode64,nodiratime,allocsize=64k 0 2

The trick being allocsize.

XFS changed it's default pre-allocation size: http://serverfault.com/questions/40...-consuming-more-space-and-full-of-sparse-file

I now have a working system!

Gerald
 

malevolent

Member
Jun 1, 2012
24
1
23
Re: Linux NFS Server not creating sparse files

I did reach the same conclussion. XFS is the guilty. So I tried with ext4 and it worked out-of-the-box.

I fear that setting XFS allocsize to such a tiny value will help to fragment the discs, and when the occupied space exceeds the 85% can be really painful.
IMO, ext4 performs somehow better than XFS, and I do not see any benefeit to select XFS over ext4 on proxmox. Ext4 can handle easily severan tens of thousands of files and folders, and that is all what proxmox needs.

What do you think guys? About the use of ext4 over XFS, I mean.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!