proxmox deployment

K

kacper.adrianowicz

New Member
Sep 12, 2025
5
1
3
Hello everyone

Quick backstory. We have 2 main programs, both of them are using SQL server on windows. We used Hyper-V, but it wasn't very fast, nor secure enogh (periodic backups). Both of these systems are crucial for our company, so the best way would be to have some sort of continuity of working even in case of failure.

We had quite a few servers: 2 x R610, 2 x R630, and we recently bought 1 x R640 (All consumer grade SATA SSD) - we have other VM's there as well.

We decided to go with Proxmox, created fresh Proxmox installations, configure cluster. As we went deeper and deeper we tried ZFS storage, then Ceph, but these consumer Grade SSD's can't keep up with the load, and right now it is working probably worse that if it was just server with Windows install on it and scheduled backups.

Decision was made, we bought another R640 with HBA controller and fresh set of Enteprise Grade SAS drives, we will also replace controllers and drives for our R640 and one of R630.

So we will have cluster of: 2 x R640 and 1/2 x R630, all with SAS drives. All of them have 2 x 10Gb SFP ports and normal 1Gb RJ45 ports, 1 SFP port on each can be used for ceph.

My question is how to properly choose and configure storage (ZFS, Ceph, or something else), VM's and all other important parameters this time, so we will have flawless experience and safe deployment.

Thanks
Kacper
 
Enteprise Grade SAS drives -> SSDs?

If you decide to go the ceph route you should probably upgrade your network to 25G or faster if you have fast SSD's, otherwise the network could be a performance bottleneck.

Here are some official Ceph Benchmarks with different network configurations, could give you a hint what to expect.
 
  • Like
Reactions: UdoB
old R610s will go to other use, so I will have only 3 servers: 2 x R640 and 1 x R630. All with SAS drives. Should i use Ceph or something else? Will it work good with enterprise SAS SSD? I think I won't have budget for 25Gb cards right now

Kioxia Dell 1,92TB 2,5" SSD SAS 12Gb PM5-R KPM5XRUG1T92 0TDNP7 TDNP7​

 
In hindsight, you should have skipped the HBA and SAS and gone with NVMe.

When choosing between ZFS and Ceph, keep in mind that ZFS is a local filesystem, while Ceph is a form of distributed/shared storage. Each comes with its own set of pros and cons.

With only three servers, your Ceph performance will be somewhat limited, but it may still be the best use of your existing hardware. If you find that latency becomes a bottleneck for your database workload, then, given your current hardware and budget constraints, your only real option is to move to a local filesystem.

You may find this helpful: https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: UdoB and Johannes S
I would not use ceph with this setup because for a stable ceph cluster you should have at least 5 nodes (preferably all the same) with at least 2-4 drives on each node and 25G+ networking (LACP / MLAG) dedicated to ceph. You will need a dedicated network interface for cluster traffic (1G, corosync) and even more for the vm uplink (LACP / MLAG).

A 3 node ceph cluster is for home lab / evaluation, not really someting you would want for production.

The SSD's are certainly a good investment, consumer grade drives have some issues with ceph / zfs (fast wearout, no PLP, ...)
 
  • Like
Reactions: Johannes S
but is there a way to have immediate or near immediate failover without ceph? It is very important for our use case. I can do SFP aggregation to have 20Gb connection purely for Ceph storage, but with drives that I had i saw speeds in rebalancing 20/50MiB/s so nowhere near network bottleneck. Problem was that disks had so much latency that on VM disk avg response time was between 100s and 1000s ms.
 
You have basically three options together with the integrated high-availability solution (https://pve.proxmox.com/wiki/High_Availability ):

Please note, that storage replication (different than Ceph or dedicated storage hardware) are asynchronous: By default the data will be replicated every 15 minutes to the other nodes, you can reduce this time to one minute or up to several hours. Depending on your usecase this might still be enough or could be a show stopper.

Please note (see referenced wiki page) that like Ceph you can configure a dedicated network for the storage replication/migration traffic:
https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_migration_network
If this network is only used by your proxmox nodes you could even disable encryption on the transport layer for fully saturating the network connection.
This is especially useful for small clusters where Ceph isn't feasible (so 2-3 nodes), with more nodes storage replication doesn't scale any more compared to Ceph or shared storage.

How many RJ45 1GB ports do you have on the nodes? For a cluster you should have at least one dedicated link only used for cluster traffic see https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network
 
  • Like
Reactions: MarkusKo and UdoB
Do you need the 10Gb for the services (communictation of clients to the server)? If not I would set something up like this:

  • 10 Gbit: Two links for redundand storage migration network, I would also use this to connect the nodes to a ProxmoxBackupServer for vm and lxc backups.
  • 2 1Gb for outside access (again with redundancys)
  • 2 dedicated cluster communication networks

Investing in enterprise-SSDs is a good idea, to get most out of the cluster performancewise
 
kacper.adrianowicz said:
for backups i use NAS, so i dont think I need to use ProxmoxBackupserver.
Click to expand...

Well PBS will need less storage space and can be leveraged for Ransomware protection:
https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

It also allows live-restore, if i recall direct that's not possible the other way.
kacper.adrianowicz said:
Either way the best way to do it would be Ceph, correct?
Click to expand...
Yes but you could also use your NAS with NFS. You will need a fast network connection to it ( 10 gbit/s or more ) and it will be a single point of failure.
Depending on your usecase that might be ok. You would still need offsite backup, but that you will need anyhow
 
Last edited:
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom