Native ZFS for Linux on Proxmox

[bread-baker]

For testing I want to use extra disks with Proxmox for ZFS . We'll use this for backups for now.

I put my install notes here:
http://pve.proxmox.com/wiki/ZFS#Native_ZFS_for_Linux_on_Proxmox_2.0

ZFS is amazing .

 

[bmcdenmark]

Hi, I followed your instructions, and installed ZFS support without issue's, does anybody have any idea, why a kvm vm (with a virtio RAW image) won't start directly from a ZFS disk ?

Thanks

Michael, BMC.

 

[spirit]

It should works without any problem... What is your error message ?

 

[bread-baker]

we've been using zfs for iso and backups with out issues.

vz and kvm do not work when run directly on zfs .

vz would start but have issues with disk usage.

here are some notes:
------------------------
df in ct: , note 10G inaccurate. s/b 15G
8003-pro4 / # df
Filesystem Type Size Used Avail Use% Mounted on
/dev/simfs simfs 10G 10G 0 100% /


made disk 15G then:
8003-pro4 / # df
Filesystem Type Size Used Avail Use% Mounted on
/dev/simfs simfs 15G 15G 0 100% /

yet only one dir has more then a G :
8003-pro4 / # du -sh *|grep G
2.3G u

So I added to 20G then immediately checked df :
/dev/simfs simfs 20G 20G 0 100% /


KVM
- images do not start when on zfs
- they do work when on zvol / ext2

-----------------------------

So I've been using zfs for iso and backups.

zvol for kvm and vz .

 

[ernie]

I as about to post that an OpenVZ container doesn't work because ZFS handles quotas differently OpenVZ just doesn't work when quota's are enabled. That was my experience last month when I first tested things under ZFS on Linux rc6. I was able to get a container running by disabling disk quotas, just add the following to the containers *.conf file.

Disk_Quota=no

However I just installed two test container just to check if that was still the case and they both started just fine without any issues. I even increased the disk quota and that works now. I'm running the latest version with iscsi+smbfs patches found here: https://github.com/FransUrbo/zfs.

Keep in mind that ZFS on Linux is still about 10x slower than ext3 or ext4.

 

[bread-baker]

I've 3 test vz's . they are on lvm , zfs and zvol

On slowness - tests I've done inside vz's show no difference in speed. the test is to check the data integrity of our 2GB worth of pro4 files.

But rsyncs on zfs are a lot slower.

 

[Nemesiz]

Hi, I followed your instructions, and installed ZFS support without issue's, does anybody have any idea, why a kvm vm (with a virtio RAW image) won't start directly from a ZFS disk ?

Thanks

Michael, BMC.

To run KVM on ZFS you have to change cache to: Write back

OpenVZ run too but disk quota isn't counting (not tested with user quotas)

I was using ZFS-FUSE on server. Primarycache/secondarycache was using only for metadata cache. Worked great. After moved to native ZFS HDD reading was so slow like modem internet connection. Had to change primarycache to ALL to make native ZFS run more smoothly. Using arcstat script shows ARC grow up to 14GB. Had to limit to 4GB. So native ZFS cant run normally without ARC cache.

I tried to upgrade from ZFS-FUSE on my desktop computer using single HDD. Terrible weird ZFS become. CPU 100%, hdd 0%, 0 pkci/pkco !!! Had to boot OpenSolaris live CD to copy data and then recreate ZFS using ZFS-FUSE.

 

[bread-baker]

where is this done : ' change cache to Write Back' ?

 

[Nemesiz]

where is this done : ' change cache to Write Back' ?

You can do it using web GUI or manually.

Example:

ide0: data_zfs:100/vm-100-disk-1.raw,cache=writeback

 

[bread-baker]

regarding: "Had to change primarycache to ALL to make native ZFS run more smoothly"

Is that setting also needed ? if so where is it set?

 

[Nemesiz]

regarding: "Had to change primarycache to ALL to make native ZFS run more smoothly"

Is that setting also needed ? if so where is it set?

zfs set primarycache=all dataset_name

'primarycache' must be one of 'all | none | metadata'

 

[bread-baker]

I put that information to http://pve.proxmox.com/wiki/ZFS#kvm_tuning .

 

[ernie]

I put that information to http://pve.proxmox.com/wiki/ZFS#kvm_tuning .

@bread-baker, thanks for starting this thread there's a lot of useful information coming out of this. At least for those interested in ZFS on Linux.

 

[bread-baker]

using dkms to install zfs

see http://pve.proxmox.com/wiki/ZFS#dkms_install

 

[Nemesiz]

Re: using dkms to install zfs

Once again ZFS safe my day.

pool: data_zfs
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-9P
scan: scrub repaired 1020K in 0h50m with 0 errors on Mon May 14 22:33:18 2012
config:

NAME STATE READ WRITE CKSUM
data_zfs ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdd1 ONLINE 0 0 19
sdc1 ONLINE 0 0 0
sdb1 ONLINE 0 0 0
logs
sde1 ONLINE 0 0 0

errors: No known data errors

 

[Erk]

Repartition Proxmox to make a ZFS slice

I have a 1RU server that can only take two hard drives. I was thinking of using a 1.5TB boot drive and a 1TB second drive, making the 1.5TB into 500G for Proxmox and 1TB of ZFS which I will mirror with the 1TB second drive. When I install Proxmox it wants to use the whole 1.5TB with no option to change that. Assuming it's a fresh install with no VMs, how can I downsize the Proxmox so I can free up 1TB for the ZFS partition?

Also, does anyone know why when I do a pveperf on a mounted zfs volume, it doesn't display the BUFFERED READS: figure in the results?

 

[Nemesiz]

Re: Repartition Proxmox to make a ZFS slice

I have a 1RU server that can only take two hard drives. I was thinking of using a 1.5TB boot drive and a 1TB second drive, making the 1.5TB into 500G for Proxmox and 1TB of ZFS which I will mirror with the 1TB second drive. When I install Proxmox it wants to use the whole 1.5TB with no option to change that. Assuming it's a fresh install with no VMs, how can I downsize the Proxmox so I can free up 1TB for the ZFS partition?

Install with 'linux maxroot=SIZE' ( http://pve.proxmox.com/wiki/Debugging_Installation ) Then remove LVM and create ZFS.

Also, does anyone know why when I do a pveperf on a mounted zfs volume, it doesn't display the BUFFERED READS: figure in the results?

I guess ZFS file system works different then others like ext family.

 

[Erk]

Re: Repartition Proxmox to make a ZFS slice

Install with 'linux maxroot=SIZE' ( http://pve.proxmox.com/wiki/Debugging_Installation ) Then remove LVM and create ZFS.

Not sure I quite understand. The /root is part of the pve lvm group so I cant really delete that, if I could make it not be on LVM that would be good, but I don't see how to do that. Is there a point in the install in debug mode where I can delete /root and the LVM group and make a normal /root and have the installer continue to put the right files on /root?

 

[tom]

Re: Repartition Proxmox to make a ZFS slice

e.g. http://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Squeeze

 

[DaveQB]

Re: Repartition Proxmox to make a ZFS slice

How is this going for everyone?

I am VERY interested in this as my proxmox server doubles at the houses file server. I am interested/concerned about performance, but testing so far is not scaring me off (test box)

You could always create a block device in a zpool and them make that an LVM for KVM guests to use and then use zfs on the rest of the pool. If indeed guest VMs aren't liking being on a ZFS file system.
Then use a ZFS file system for storing backups etc.