Proxmox cluster architecture choice

[klass]

Hello,

My company wants to set up a proxmox infrastructure but we're hesitating between two choices (we can adjust the hardware configuration of each server). We want to have at least 5Tb of usable storage. Depending on the choice, we invest in a backup server (PBS or TrueNAS replication) a little later.

Infrastructure 1: 3 Proxmox VE nodes + Ceph

Each node (R640) would have the following configuration:

• 2* Intel 6148 20core @ 2.4Ghz
• 384Gb RAM
• 2* SSD 240G for Proxmox OS and ISO
• 5* SSD 1,92 Tb for Ceph
• No H730 because Zfs for OS and Ceph for data (confirm ?)
• Network card 4*1Gb Copper
• Network card 2*10Gb Copper

Network note: All would be connected via a Nexus 9382TX switch (48x10Gb)
Ceph note: What is the best Ceph configuration should be used?


Infrastructure 2: 3 Proxmox nodes + 1 NAS server (TrueNAS; NFS)

Each Proxmox node (R640) would have the following configuration:

• 2* Intel 6148 20core @ 2.4Ghz
• 384Gb RAM
• 2* SSD 240G for Proxmox OS and ISO
• No H730 because Zfs for OS and Ceph for data (confirm ?)
• Nework card 4*1Gb Copper
• Network card 2*10Gb Copper

The TrueNAS (R640 or R740) would have the following configuration:

• 2* Intel 5118 12core @ 2.3Ghz
• 128Gb RAM (sufficient?)
• 2* SSD 240G for TueNAS OS in ZFS Mirror
• 4 to 6 1.92Tb SSDs (Which best RaidZ ? 2?)
• No H730 because Zfs (confirm ?)
• Nework card 4*1Gb Copper
• Network card 2*10Gb Copper

Network note: Best Practice?
- Idea 1 : all interfaces would be connected via a Nexus 9382TX switch (48x10Gb) and to limit the problem of the limit on the number of 10GB PCIE cards on the TrueNAS server
- Idea 2 :
-> 1* 10Gb interfaces per Proxmox server connected directly to the NAS
-> The others interfaces on the Nexus

What would you recommend for performance,stable and resilient?


Thanks for your time

 

[jsterr]

Infrastructure 1: 3 Proxmox VE nodes + Ceph

Each node (R640) would have the following configuration:

• 2* Intel 6148 20core @ 2.4Ghz
• 384Gb RAM
• 2* SSD 240G for Proxmox OS and ISO
• 5* SSD 1,92 Tb for Ceph
• No H730 because Zfs for OS and Ceph for data (confirm ?)
• Network card 4*1Gb Copper
• Network card 2*10Gb Copper

Network note: All would be connected via a Nexus 9382TX switch (48x10Gb)
Ceph note: What is the best Ceph configuration should be used?

Hello, I would recommend:

• SSDs: 480 GB SSDs for OS because Ceph Logs can consume lot of disk-space later especially when you recover
• SSDs: If you have the possibility to switch to nvmes, I would recommend to do so
  • often the price gap between SATA SSDs and NVMe SSDs is not that big
  • but Performance compared to sata is big
• What are youj referring to with "H730"?
• Network: Network card 4*1Gb Copper: good for management and corosync
• Network: Network card 2*10Gb Copper: good for VM-Network (vmbr0)
• Network: I would recommend going for at least 25Gbit/s for Ceph
  • see https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
  • its a good start without the need for storage switches
• Ceph: 3:2 setup and all-nvme is the way to go, rest Proxmox VE 8.1 Defaults (Ceph 18.2 Reef)

Infrastructure 2: 3 Proxmox nodes + 1 NAS server (TrueNAS; NFS)

Each Proxmox node (R640) would have the following configuration:

• 2* Intel 6148 20core @ 2.4Ghz
• 384Gb RAM
• 2* SSD 240G for Proxmox OS and ISO
• No H730 because Zfs for OS and Ceph for data (confirm ?)
• Nework card 4*1Gb Copper
• Network card 2*10Gb Copper

The TrueNAS (R640 or R740) would have the following configuration:

• 2* Intel 5118 12core @ 2.3Ghz
• 128Gb RAM (sufficient?)
• 2* SSD 240G for TueNAS OS in ZFS Mirror
• 4 to 6 1.92Tb SSDs (Which best RaidZ ? 2?)
• No H730 because Zfs (confirm ?)
• Nework card 4*1Gb Copper
• Network card 2*10Gb Copper

As soon as I read NFS or iSCSI in combination with Proxmox VE I stop, you cant really Snapshot VMs in a good way with NFS or iSCSI. You will have some downsides. iSCSI cant Snapshot, NFS you cant use Snapshots on Containers, but on VMs. And it doesnt scale as good as ceph does: see https://pve.proxmox.com/pve-docs/chapter-pvesm.html

 

[bbgeek17]

As soon as I read NFS or iSCSI in combination with Proxmox VE I stop, you cant really Snapshot VMs in a good way with NFS or iSCSI. You will have some downsides. iSCSI cant Snapshot, NFS you cant use Snapshots on Containers, but on VMs. And it doesnt scale as good as ceph does

Hi @jsterr, overall I agree with your general sentiment, it is probably worth some clarification:

• One can snapshot VMs on NFS storage if using QCOW. Yes, there are downsides to NFS. The most notable issue is performance.
• iSCSI on Thick LVM (i.e., for shared storage) prevents you from taking snapshots. It also suffers performance issues at scale due to multiple hosts sharing a single LUN.
• There are solutions (including ours) that provide full-featured storage management using iSCSI/TCP and NVMe/TCP storage protocols (i.e., snaps, clones, rollback, data reduction, encryption, multipathing, etc.). These are typically for folks who need high availability and high performance.
• Ceph is probably the best option at Op's scale if cost efficiency is the primary goal.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox