Best practices question (Ext4/zfs/raw/qcow2)

[rcscny]

I am setting up a brand new Proxmox server running on a system with hardware raid. The server will host a mix of 2019, 2012 and 2008 Windows Server guests, It will be a production server. I plan on adding 2 servers and setting up fault tolerance in a few months after I have had some time to better acclimate.

The default format when you install Proxmox is Ext4. Most of the videos/articles I have read suggest changing from ext4 to zfs.
Now I am reading that you shouldn't put qcow2 files on zfs drives because of performance and stability issues.


My windows servers will be imaged via Disk2Vhd which creates vhdx files.
I then have to convert the vhdx images to either qcow or raw using qemu-img.exe before importing to Proxmox.


What is the recommended file format for my PVE Server? I do plan on performing regular backups and snapshots once I figure out how to best automate them.

ext4 with raw?
ext4 with qcow2

zfs with raw
zfs with qcow2?

Is there a problem with one format over the other if I want to implement fault tolerance at a later date?

 

[Dunuin]

What is the recommended file format for my PVE Server?

PVE is block storage centric. You usually don't work with image files on a filesystem to prevent the additional overhead and directly work with block devices like zvols or LVs. Some exceptions would be when working with NFS or in case you need more freedom when rolling back snapshots and you therefore want qcow2.

Is there a problem with one format over the other if I want to implement fault tolerance at a later date?

As you later want to run a cluster you should take that into account when choosing your storage. For that you need a shared storage like Ceph/NFS/ZFS-over-iSCSI or ZFS as local storage with replication.

 

[rcscny]

PVE is block storage centric. You usually don't work with image files on a filesystem to prevent the additional overhead and directly work with block devices like zvols or LVs. Some exceptions would be when working with NFS or in case you need more freedom when rolling back snapshots and you therefore want qcow2.


As you later want to run a cluster you should take that into account when choosing your storage. For that you need a shared storage like Ceph/NFS/ZFS-over-iSCSI or ZFS as local storage with replication.

Thank you, so your recommendation is zfs and my virtual vm drives as qcow2? I am converting the vhdx files on a windows pc, then once converted to qcow2 I am scp'ing them to the server /var/lib/vz/images/ folder ... then editing the vm to add the imported disks

 

[bbgeek17]

Thank you, so your recommendation is zfs and my virtual vm drives as qcow2?

No, you'd use ZFS as block storage - via zVols (zPool). No qcows involved. ZFS can accommodate both file and block storage. You do not want to put a copy-on-write qcow on copy-on-write ZFS.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Dunuin]

Thank you, so your recommendation is zfs and my virtual vm drives as qcow2?

Like said, you usually don't work with files. Default for ZFS would be to use a "raw" formated "zvol" block device for VM disks.

Thank you, so your recommendation is zfs and my virtual vm drives as qcow2?

With ZFS you will lose the last minute of writes in case HA triggers as it is not a shared storage. Depends if that fits your needs. Ceph requires 3+ Nodes and NFS or ZFS-over-iSCSI might be a bit silly for a single node unless you got some SAN.

 

[rcscny]

Sorry for being so new to this all ... I have been making practice runs on these servers all week trying to get the best methods down before applying to production servers.

Each server in the production environment has to be converted to a VM in Proxmox.
My procedure has been:

Set up Proxmox server, and change drive format to zfs in the initial setup window. Once Proxmox is configured, I copy my disk2vhd vhdx files to a temporary Windows pc to convert the vhdx files to something Proxmox can use.

My first test server is named Shop2k19


1) Windows pc I run this:
qemu-img.exe convert -p -f vhdx C:\NetworkShares\Images\Shop2k19.VHDX -O qcow2 C:\NetworkShares\Images\Shop2k19.qcow2

2) Then I upload the Shop2k19.qcow2 to my proxmox server via scp to /var/lib/vz/images/ on the proxmox server


3) Then from terminal window on the Proxmox server:
qm importdisk 100 /var/lib/vz/images/Shop2k19.qcow2 local-zfs

4) Edit the VM hardware and assign the hard disk to the VM


What should I change .. Everything right now is just test stuff, so I can start from scratch again (as I have 20 times already LOL).

Some screen shots attached for reference on my current setup. The help from these forums is fantastic, I really do appreciate it

 

[bbgeek17]

2) Then I upload the Shop2k19.qcow2 to my proxmox server via scp to /var/lib/vz/images/ on the proxmox server


3) Then from terminal window on the Proxmox server:
qm importdisk 100 /var/lib/vz/images/Shop2k19.qcow2 local-zfs

4) Edit the VM hardware and assign the hard disk to the VM


What should I change

You are doing fine and dont need to change anything at this point. You dont need to upload files to that specific location. The "impordisk" does not care where to import disks from. You have to provide absolute path.
The "local-zfs" by default is a ZVol based storage, so your files are automatically converted by importdisk.

However, if you want High Availability and Live Migration (without asynchronous replication), then you need to research and try different storage type. See comment #5 that covered it very succinctly.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox