Moving to Proxmox questions

[damiengm]

Hi All,

We run VMware with Essentials Plus license, and our renewal support license cost is x10 the previous one. No doubt I am looking at alternatives. So I'm researching here, and looking at Proxmox as an alternative, there are some things I found which I would like further clarification. If there are some pain points from migrating over to Proxmox please speak up.

Current Environment:
2 EXSi Hosts, 128GB RAM each. These hosts are EoL so will be replacing them as part of the move away from VMware
Synology NAS as iSCSI storage. This is new so its not going to be replaced anytime soon.
8-9 VMs
Veeam Backup & Restore

So, questions I have:
- Hosts = Nodes in Proxmox speak?
- VMware with its vMotion is great for live compute and storage migration. Proxmox looks to have the same capability? I used the vMotion live compute movement to help balance the host loads, and its so fast no-one ever knew I was doing it, and since the vm image files are shared it didn't need to worry about storage migration.
- Synology iSCSI: I need to setup LVM (is that one large LVM or is it per VM?) on top of the iSCSI? as there's no file management in LUNs in Synology side.
- Shared VM data storage: since I'm limited to iSCSI it looks like I can only have thick provisioned VM image files when I need to utilise HA Clustering because I need the image files shared across all nodes? The LUN on the Synology side I can make it thin provisioned, which only helps so much, but if all the image files are thick provisionedI can't see it helping much. Which means I might need to increase storage in the Synology?
- HA: I really need three nodes? VMware worked fine with a 2 host HA.

 

[LnxBil]

Synology NAS as iSCSI storage. This is new so its not going to be replaced anytime soon.

A single system with all of your data? What if the box breaks?

Hosts = Nodes in Proxmox speak?

For a cluster, you need 3 nodes (or two and a quorum device).

VMware with its vMotion is great for live compute and storage migration.

Why do you need storage live migration if you already have a SAN? That makes no sense to me. Nevertheless, it is possible with PVE.

Synology iSCSI: I need to setup LVM (is that one large LVM or is it per VM?) on top of the iSCSI? as there's no file management in LUNs in Synology side.

Unless Synology has a proper Proxmox VE integration, you will sadly have only thick-LVM without snaphots or thin provisioning.

HA: I really need three nodes? VMware worked fine with a 2 host HA.

As already said, two nodes and a quorum device will also work.

 

[IlNerd]

- Hosts = Nodes in Proxmox speak?

You can call is as you want.

- VMware with its vMotion is great for live compute and storage migration. Proxmox looks to have the same capability? I used the vMotion live compute movement to help balance the host loads, and its so fast no-one ever knew I was doing it, and since the vm image files are shared it didn't need to worry about storage migration.

Depend on the type of storage you want to use https://pve.proxmox.com/wiki/Storage, if you want the vMotion feature you need to implement the storage with shared and snapshot features, otherwise with 2 nodes you can use ZFS and replication for live migration of VMs and snapshot, take care that this kind of deployment require to double the storage. I think the better solution should be a 3 nodes with ceph as storage.

- Synology iSCSI: I need to setup LVM (is that one large LVM or is it per VM?) on top of the iSCSI? as there's no file management in LUNs in Synology side.

+1 with LnxBil

- Shared VM data storage: since I'm limited to iSCSI it looks like I can only have thick provisioned VM image files when I need to utilise HA Clustering because I need the image files shared across all nodes? The LUN on the Synology side I can make it thin provisioned, which only helps so much, but if all the image files are thick provisionedI can't see it helping much. Which means I might need to increase storage in the Synology?

Just one advice, as you said, you need to change your hardware, use the Synology as backup repository with PBS and deploy 3 new servers with ceph for computing and storage
https://pve.proxmox.com/pve-docs/chapter-pveceph.html

- HA: I really need three nodes? VMware worked fine with a 2 host HA.

You need to have the quorum (min 3 members) one can be a raspberry but i will never run this in production.

 

[SteveITS]

HA: I really need three nodes?

Technically you can set up HA with 2 nodes but if one is ever off (reboot, failure) then the other would not have over 50% of votes so no quorum (hence, the Qdevice). Or you can give one node 2 votes so there are 3 total votes, but of course then only the node with 1 vote can go offline. So overall two is not recommended for stability reasons.

 

[Johannes S]

Another Option might be to use NFS with your NAS, then you could use QCOW disk images with snapshot support. It's still a single-point of failure and you will still need a qdevice. You could reuse one of the old nodes for that though.

Depend on the type of storage you want to use https://pve.proxmox.com/wiki/Storage, if you want the vMotion feature you need to implement the storage with shared and snapshot features, otherwise with 2 nodes you can use ZFS and replication for live migration of VMs and snapshot, take care that this kind of deployment require to double the storage. I think the better solution should be a 3 nodes with ceph as storage.

Recommended reading for the OP: https://pve.proxmox.com/wiki/Storage_Replication

And this thread on small ceph clusters:

[TUTORIAL] Thread 'FabU: can I use Ceph in a _very_ small cluster?'

Dec 26, 2024

Ceph is great, but it needs some resources above the theoretical minimum to work reliably. My assumptions for the following text:

• you want to use Ceph because... why not?
• you want to use High Availability - which requires Shared Storage (note that a complete solution needs more things like a redundant network stack and power supplies)
• you want to start as small (and cheap) as possible, because this is... “only” a Homelab

You plan for three Nodes. Each node has s single dedicated disk for use as an “OSD”. This is the documented...

• UdoB
• ceph cluster mini pc minimize shared filesystem storage
• Replies: 25
• Forum: Proxmox VE: Installation and configuration

• 

 

[Johannes S]

One thing I forgot: If your synology allows hosting of VMs or docker containers you could also use the NAS as qdevice.
For generic information on qdevice:
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
And on hosting it inside a docker container:

B

Thread 'How to run a QDevice in Docker'

Feb 16, 2024

I'm looking into running a QDevice for my proxmox cluster within docker on my Synology nas. I've never used docker before so this is still new to me. I've downloaded a few "images' from the registry but while I can get at least one of them to build and run I can't seem to connect to it. I'm wondering if anyone here has experience running a qdevice in docker and may be able to point me in the right direction.

I've had the most success running this image: https://hub.docker.com/r/bcleonard/proxmox-qdevice/
This is the YAML config I've been using.

version: "3.5"
services...

• baron164
• docker proxmox 8.1.4 qdevice qnet synology
• Replies: 13
• Forum: Proxmox VE: Installation and configuration

 

[damiengm]

A single system with all of your data? What if the box breaks?

The Synology is a SA3200D, which has dual controllers in it. We did have 2 NASes, but this new box is I feel a better solution.

Why do you need storage live migration if you already have a SAN? That makes no sense to me. Nevertheless, it is possible with PVE.

Just to know its there. Previous IT person had 1 LUN per VM for some reason, I was glad when I got round to clearing that up to a single LUN I had live migration available. Though for a couple of VMs I did do offline migration. Its probably not much of an option needed now though.

Unless Synology has a proper Proxmox VE integration, you will sadly have only thick-LVM without snaphots or thin provisioning.

Is NFS not recommended for production environments? qcow thin-provisioning looks to be an promising option. snapshots is not important to me, never really used it and if I needed an instant backup I'd run a Veeam backup.

....Nodes with quorum devices...

Will research quorum devices more. we are not willing to spend more on infrastructure that in the end could cost the close to the same amount as just paying the new VMware licence cost.

 

[LnxBil]

Is NFS not recommended for production environments? qcow thin-provisioning looks to be an promising option. snapshots is not important to me, never really used it and if I needed an instant backup I'd run a Veeam backup.

Yes, it that works, it's fine. I assumed because you were talking about LUNs and iSCSI (block storage), NFS/CIFS is file storage is always fine and with QCOW2 you will have everything you need (snapshots, thin provisioning and live migration).

The Synology is a SA3200D, which has dual controllers in it. We did have 2 NASes, but this new box is I feel a better solution.

Thank you for clearing that up. I'm not familiar with Synology's lineup. Never seen one outside of the home market and also never a HA box. My enterprise customers are only running old school SAN vendors like 3par, Dell, Datacore, Fujitsu...

Just to know its there. Previous IT person had 1 LUN per VM for some reason, I was glad when I got round to clearing that up to a single LUN I had live migration available. Though for a couple of VMs I did do offline migration. Its probably not much of an option needed now though.

That works if you have a proper integration (Hypervisor <=> Storage) and if the hypervisor is allocating those. Having to manage that by hand is not fun

 

[Johannes S]

Is NFS not recommended for production environments? qcow thin-provisioning looks to be an promising option. snapshots is not important to me, never really used it and if I needed an instant backup I'd run a Veeam backup.

I really have no personal experience in a corporate environment since the vmware cluster at work is run by a different team (I'm just using ProxmoxVE at home) but in this case and if you can live without thin-provisoning it might be worth a shot to benchmark both variants (ISCSI versus NFS). In Theory using LVM on ISCSI as block storage should give better performance due to the missing overhead of one filesystem. Both protocols are from more innocent times and are not really good in terms of security so in that regard you are even if you have a dedicated network for communication between your compute and storage nodes (which you propably already have). I remeber I sat in a talk on VMWare Alternatives in a German Linux conference (FrosCon Sankt Augustin, great con btw) where the speaker explained that he usually recommended using NFS for small clusters. Now he might have been a little bit biased because his company Credativ was part of NetApp at that time but for me this indicates, that this is at least a solution which is often "good enough". Since then Credativ went through a management buyout though so his opinion might have changed
Funny thing you mention that you don't need snapshots due to using backups. This is exactly the workaround proposed in the official migration guidelines, although with Proxmox own backup product ProxmoxBackupServer: https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Alternatives_to_Snapshots

I also read several times in this forum that pros (which as said I'm definitively not) recommended this workaround to their customers who still had to continue using their existing storage hardware.


Another option could be to use ZFS over ISCSI (really the best of both world) but this needs support on the storage side and afik most storage hardware don't do it. The reference documentation has an overview on all storage options and their pros and cons: https://pve.proxmox.com/wiki/Storage

Will research quorum devices more. we are not willing to spend more on infrastructure that in the end could cost the close to the same amount as just paying the new VMware licence cost.

Understandable. The thing is that a qdevice doesn't need a lot (it's basically just a tiny service on a linux server), so even a Mini-PC or Raspberry would be enough and don't need much energy. It might be an issue in terms of compilance though. So for this reusing your existing hardware or the synology might be a cost-efficient way to save on budget.

 

[LnxBil]

if you can live without thin-provisoning

NFS does have thin-provisioning with qcow2, yet maintaining it over time is more complicated. You cannot trim on qcow2, you need to rewrite them.

 

[alexskysilk]

- Hosts = Nodes in Proxmox speak?

a node is a member of a cluster. Its not Proxmox speak.

VMware with its vMotion is great for live compute and storage migration. Proxmox looks to have the same capability?

Yes.

Synology iSCSI: I need to setup LVM (is that one large LVM or is it per VM?) on top of the iSCSI? as there's no file management in LUNs in Synology side.

I think this was already answered, but principally it will work either as iSCSI or NFS. there are caveats to both- see https://pve.proxmox.com/wiki/Storage

Synology iSCSI: I need to setup LVM (is that one large LVM or is it per VM?) on top of the iSCSI? as there's no file management in LUNs in Synology side.

iscsi is block storage, so it stands to reason the storage device has no visibility on its contents. There are two ways to consume iscsi in PVE- either as a lvm pool, in which case you will have it available to assign to vm workloads, or direct mapped to guest. There are caveats to both methods.

Shared VM data storage:

Your synology allows you to create nfs spaces alongside with iscsi luns. use NFS for this purpose.

HA: I really need three nodes? VMware worked fine with a 2 host HA.

Yes. the good news is the third node can be a lightweight witness node. see https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_corosync_external_vote_support

 

[alexskysilk]

but in this case and if you can live without thin-provisoning

The LUNs themselves can be thin provisioned on the storage; it would just not apparent to PVE (similarly to how a thin provisioned virtual disk is not apparent to the guest.)