[SOLVED] Linux NFS or GlusterFS Server not creating sparse files

[gbr]

Hi,

I have an Ubuntu 12.04.3 server as an NFS server. I use the kernel server. When I create a qcow2 image, the file file gets created and Proxmox times out waiting for a lock. I've tried with both NFSv3 and NFSv4. NFSV4 is considerably faster at the disk image creation, but not fast enough.

# ls -lh
total 33G
-rw-r--r-- 1 root root 33G Jan 6 17:09 vm-102-disk-1.qcow2

# du -h --apparent-size

33G .

# ls -lks

total 34079304
34079304 -rw-r--r-- 1 root root 33559808 Jan 6 17:09 vm-102-disk-1.qcow2


# du --block-size=1 vm-102-disk-1.qcow2
34897207296 vm-102-disk-1.qcow2

Is anyone able to use the Linux NFS kernel server and get sparse qcow2 files created succesfully? If so, how? What am I missing?

Gerald

NFS over XFS


# pveversion --verbose
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-24 (running version: 3.1-24/060bd5a6)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-9
libpve-access-control: 3.0-8
libpve-storage-perl: 3.0-18
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-6
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1

 

[gbr]

Re: Linux NFS Server not creating sparse files

I just tested with

# dd if=/dev/zero of=test bs=1 count=0 seek=8G

on the Proxmox server using the NFS mount (/mnt/pve/Images), and the sparse file was created. Is Proxmox doing something weird and not creating a sparse file, or am I misunderstanding? If Proxmox is not creating a sparse file, then it needs to set it's timeout better, since my NFS server on a 1 gig network link gets pretty close to wire speed.

 

[gbr]

Re: Linux NFS Server not creating sparse files

I created a VM with a raw image, and it was created as sparse. An 'ls' showed 32GB disk, and a du -s showed 0 bytes. The raw disk image is now growing as I use it.

 

[gbr]

Re: Linux NFS Server not creating sparse files

Is anyone using qcow2 with and NFS server and Proxmox 3.1?

 

[mir]

Re: Linux NFS Server not creating sparse files

Yes. What would you like to know?

 

[gbr]

Re: Linux NFS Server not creating sparse files

Yes. What would you like to know?

What are you using as an NFS Server? Do your qcow2 files get created sparse? Do you get lock timeouts when creating a qcow2 disk?

Gerald

 

[mir]

Re: Linux NFS Server not creating sparse files

I use Omnios (Solaris offspring) providing NFS through ZFS' NFS implementation. I use qcow2 over NFS and raw over iSCSI. qcow2 is created sparse without timeouts. My storage is configured as RAID10 connected through a LSI HBA.

 

[gbr]

Re: Linux NFS Server not creating sparse files

This is not a Linux NFS issue (or a GlusterFS issue) at all. It's the underlying filesystem issue.

If you use NFS or GlusterFS on top of XFS, XFS needs special mount options to create qcow2 files quickly. Here is my fstab:

/dev/md2 /export/nfs/RAID-6 xfs noatime,inode64,nodiratime,allocsize=64k 0 2

The trick being allocsize.

XFS changed it's default pre-allocation size: http://serverfault.com/questions/40...-consuming-more-space-and-full-of-sparse-file

I now have a working system!

Gerald

 

[malevolent]

Re: Linux NFS Server not creating sparse files

I did reach the same conclussion. XFS is the guilty. So I tried with ext4 and it worked out-of-the-box.

I fear that setting XFS allocsize to such a tiny value will help to fragment the discs, and when the occupied space exceeds the 85% can be really painful.
IMO, ext4 performs somehow better than XFS, and I do not see any benefeit to select XFS over ext4 on proxmox. Ext4 can handle easily severan tens of thousands of files and folders, and that is all what proxmox needs.

What do you think guys? About the use of ext4 over XFS, I mean.