Switch from KVM EXT 4 with HDD to ProxMox ZFS SSD

[Breymja]

Hello together,
i want to switch my current setup for my small private server to ProxMox.

At the moment i use:
- QEMU-KVM from commandline on Debian 8
- 2 TB HDD MDRAID 1
- QCOW2 Images

I want to use:
- ProxMox 5 on Debian 9
- 500 GB SSD ZFS-RAID1 with Dedup and Compression

I got 64 GB of RAM - that should be enough.

As far as i know i need to convert the .qcow2 Images to .raw, am i correct?
Will this have any impact on file size? I got three VMs at the moment, they are provisioned with 1 TB, 1 TB and 500 GB Disk - while together they are only using somewhat under 250 G real. (But were created bigger, if needed at any time) I now decided that even in the future i would not need more than 500 GB storage on this machine (which is the first to get the switch to ProxMox) and if i do, i could put in some HDD's, but for now i really need SSD for I/O Performance which sucks on my KVM System - and the pricetag was fine for this machine, as bigger SSDs are out of scope for this project. (Even added HDDs would cost a ton of money.)

As you can see those disks all use more space than my new SSDs got. I had the problem that .raw images take the complete space they were created with after moving to another server before, so i'm scared this will happen again - this time it would not work as of the SSD not having that much space. I also plan to shrink (if possible) those images later, but that will be a mess, too, i think.

It's important for me to overprovision as i don't know which of the VMs will use the space more - this could change at any time, therefore both VMs shall have the maximum available space provisioned. It's therefore important that only the space really used is used on the ZFS-RAID1. With Dedup and Compression i hope to achieve more storage as i indeed have many similar and good compressable files in use.

Is there anything else i have to pay attention to? What about TRIM - i still couldn't find out if ZFS does support it in the meanwhile. (At least, not for sure) Is it problematic to not have TRIM available, if so? Will unmap work for keeping the images little or do i no longer need this?

I just dived into ZFS a bit and had no experience with it before, the switch shall happen in three weeks, until then i want to try a test locally. I also got very little experience with proxmox. I used VMWare for years and than switched to kvm through console when i was no longer happy with the missing raid on vmware esxi without supported hardware (controllers). ProxMox seems to be the next logical step as with zfs i can add other servers to the cluster, got options for snapshots and kvm and finally do no longer need a vnc client *g*

Sorry for my beginners questions, i hope someone here has experience in a setup like this and could help me out with some tips.

 

[pabernethy]

As far as i know i need to convert the .qcow2 Images to .raw, am i correct?

Not necessarily. You can import the qcow2 directly into zvol.

Will this have any impact on file size?

Unlike QCOW2, which grows as usage increases, RAW allocates the complete disk space upfront. So a 1TB virtual disk will result in a 1TB file. But if you use ZFS that's not actually relevant. The zpool will be managed by the host. The VMs will live inside zvols. If you enable thin provisioning for the zpool only the ~250GB should be used. Just be sure to tell Proxmox that the volume is ZFS, not a directory.

What about TRIM

TRIM is not yet supported on ZFS on Linux. You can enable DISCARD for a VM's virtual disk, which allows qemu to accept the DISCARD command from the guest OS and pass it to the storage. Even with moderate losses to usage due to a lack of TRIM your SSDs should still be much faster than the fastest spinning rust.

 

[guletz]

Not necessarily. You can import the qcow2 directly into zvol.

It is true, but the performance will be bad. You can convert qcow image in a raw format.
Do not use deduplication if do not have knowledges about zfs. In most of the case it will hurt your perfomance.
For trim, I guessing that this year zfs will have this option.
Also pay attention on zfs RAM usage. By default, it use 1/2 of your RAM, but you can modify this.

 

[dcsapak]

It is true, but the performance will be bad. You can convert qcow image in a raw format.

the qm importdisk command is a convenience wrapper around qemu-img which converts the qcow2 to a zvol, so this should be of no concern

 

[guletz]

.... but qcow2(on zfs) it is a concern, isn't it?

 

[dcsapak]

.... but qcow2(on zfs) it is a concern, isn't it?

yes this is correct, but when you do an

qm importdisk 999 "pathto.qcow2" local-zfs

the tool converts the qcow2 to raw while writing it to the correct zvol for the vm '999' on storage 'local-zfs'

 

[guletz]

Thx, I was start to think that my memory was wrong

 

[Breymja]

Can ZVolumes be overprovisioned? How to create two 500 GB ZVols? As i said i need overprovisioning as i do not know which VM will use how much space at a given time. Is it neccessary at all?

 

[guletz]

Yes. Any zvol can be enlarged as you need and when you need if you have enough free space. You can do it from proxmox http interface. But you will need to manually extend the file-sistem partition who is on this zvol . You can do it with any iso live who can do this (gparted if I remember) in few minutes. Boot your VM with this kind of ISO, resize your fs and this is all.

 

[Breymja]

This seems like it's pretty inconvenient. Datausage can get bigger on one VM while i'm not there to handle any enlarging of anything. I need overprovisioning the kind of .qcow2. (Set max Usage for VM, that's it.) How - if - can i achieve to have 2x 500 GB Images at the same time for two VMS on my 500 GB SSD that do (and can!) only use the real used space, so i do not need to do anything as long as the usage of both together fits into the SSD? If i need to give 250 GB to each one and enlarge one if it needs more - i cannot do that.

I hope you understand what i mean.

Maybe i'm misunderstanding something? As you described, i will need to convert my .qcow2 image to raw, but directly into the zvol. Will i still have a "file" that sets the limit for the VM space like in .qcow2? Can i set two vms on one zvol? (So i could do a 4xx GB Volume with all the space i got left over aside from the OS and both VMs share this space)

I would be happy about any in-depth lecture to the workings of ZFS in virtualisation, also. I managed to learn what i need to know about configuring it, that dedup is no good for me, but i cannot really find anything that does explain how this will work in proxmox with vms and zvols and raw / qcow images.

I only know what i need: Two VMs, both with overprovisioned space of 500 GB where i do never need to intervene as long as both vms together stay under the limit of my SSD. If i need to adjust zvols or anything else when a vm get's much bigger than the other one - ZFS will not work out for me.

 

[guletz]

Will i still have a "file" that sets the limit for the VM space like in .qcow2?

No. You will have like virtual HDD(like /dev/sdX) with a maximum limit. Each VM have own zvol/virtual HDD.

 

[Breymja]

I see. Can this limit be higher as my SSD is big? And how i can backup these zvols if they are no files? How can i migrate them to other servers not using ZFS? (Remember i can NOT convert into .RAW as i haven't got enough space to do so.)

I also read about files removed not being removed from the place they take up on the zvol? Is this correct? Or does trim from the VM to the ZVOL work? (TRIM on system level does not work.)

When the ZVols can NOT be bigger than the SSD - what to do with my 1 TB RAW Images? I now got two of them, i need to convert them to ZVols. In this process they would be 1 TB Zvols - i do not have that much space on the SSD - what to do, if i need to do something, as i can't have 2 1 TB ZVols on my 500 GB SSD. Or am i able to convert the Images to 250 GB ZVol (as they don't take up that much in real usage) - what will happen with the filesystems in my VM that has 1 TB allocated and think the disk they got is 1 TB?

 

[pabernethy]

You can overprovision ZFS pools with thin provisioning enabled.

The pool has 4 GB but will happily give you a 32 GB zvol. So you can e.g. create 3 500 GB zvols in your 500 GB zpool and be fine, as long as all machines together use no more than 500 GB.

 

[Breymja]

Thank you, that is what i needed to know. Unfortunatley i'm stuck with my testinstallation on virtualbox as the latter doesn't want to work the way i need it to -.-

Seems goot until now. What about my question for backups / migration to a NON-ZFS Environment? How can i backup / import a ZVOL that has my vm stored onto a server without ZFS / using .qcow2 images? I think this could be a bit difficult if i need to convert it back to a .iso file, when my storage is full. With my .qcow2 files i could easily transfer them anywhere, as no storage was needed for conversion and every backup system supports image files.

Will the ZVol respect unmap and free space no longer used by the VM?

 

[pabernethy]

As mentioned before, you can import qcow2 to zvol with qm importdisk. For export you can either just use the integrated backup functionality or, if you want to do it manually, dd the zvol's block device to your desired location.

 

[Breymja]

How does the integrated backup functionality work? Does it create a .qcow2 image and can it do this onto an FTP space?
The problem is, i will never be able to convert a 1 TB zvol to an .raw or backup it as this, as there is not enough space.
Also, it would be pretty good if the backups are directly written onto my ftp storage. If they are created onto my SSD and then moved to the ftp this isn't possible as the SSD is too small. Could you might elaborate a little more on how the backup functionality works? I'm sorry for my dumb questions, but i need to check out everything before sometime in the future i'm really stuck with my data as there is no way to transfer the images as they are too big to be created on the server.

 

[pabernethy]

Containers make a file-level backup to a .tar.gz or .tar.lzo on whatever storage you specify. VMs use our own format, again with lzo or gzip compression. Size-wise it's roughly like raw, perhaps a bit larger.
But why would you backup to the same disk the machines run on? That would be pointless. If that disk fails the backups are gone, too. You have the HDDs you currently run your VMs on. Those are predestined to store your backups. The backup storage doesn't need to be particularly fast, as you don't want to access it as frequently as the live disk. Just keep them in the machine, in RAID 1 if you want. Ideally you want to backup to a different machine. That can just be a NAS using NFS.
The backup system automatically removes the oldest backup if the configured number has been reached. So if your 3 VMs use 400 GB together and all roughly the same amount you should have room for 2 backups per machine on 1 TB storage. If they use the full 500 GB you can only fit 1 backup per VM, as the backup routine requires space to write the backup before the old backup is deleted.
Why don't you just fire up a VM, install Proxmox on it and make yourself familiar with the system?

 

[Breymja]

Can i recreate an image out of these backups without needing to import them to proxmox first?

I would not want to backup them on the machine they run on, it was my question if it is possible to directly save the backups to another machine without them being created on the server they run on and being moved AFTERWARDS. (This is how backups work with the solution i have now.)

I cannot add any more storage to the server, so local backups are out anyways. (This is typical server of a big german hoster)

Also i now have 1 TB virtual disks - i would need to find out how to shrink them to 500 GB with the filesystem in the VMs staying functional - never did this before, as there was no need to do this. I will need to do this if backups are the size of .raw.

I'm already on trying it in a VM but have no room for VMs on my servers and VirtualBox on my desktop just doesn't want to accept my network configuration - so i can't reach the proxmox panel -.- Need some more testing :/

Thanks for your patience anyway, i appreciate it much!

 

[pabernethy]

Can i recreate an image out of these backups without needing to import them to proxmox first?

You can restore a machine from the backup. Proxmox doesn't make a difference between local and remote storage.

I would not want to backup them on the machine they run on, it was my question if it is possible to directly save the backups to another machine without them being created on the server they run on and being moved AFTERWARDS.

Proxmox doesn't make a difference between local and remote storage.

i would need to find out how to shrink them to 500 GB with the filesystem

No, you don't. I already told you that you can overprovision zvols.

I will need to do this if backups are the size of .raw.

VMs use our own format, again with lzo or gzip compression.

So the raw-sized image will be compressed. gzip is great at taking care of large swathes of zeros.

 

[Breymja]

My Question with those backups was for using them to start the image on a non proxmox server. So can i recreate a kvm-usable image from one of the backups to directly use with a non-proxmox server - that's the reasoning behind my question. I got several servers but will start to move only one to proxmox to see if it fits my workload. As i read, i also cannot use the cluster functionality for all machines, as they are not within the same ip-range but in totally different ones. (Otherwise it would have made sense to convert every host to proxmox immediatly)

Alright for the storage.

Why would i not need to shrink the VMs? You said, the backups are as big as .raw files, a .raw file of a 1 TB VM will be 1 TB big - this is too much, especially when the VM only uses 250 GB. Someone in this thread confirmed .raw images will take the complete space.

So the raw-sized image will be compressed. gzip is great at taking care of large swathes of zeros.

Does this mean a compressed backup will be nearly as small as a .qcow2 image of an VM, so my point above will be invalid?

Thank you again