Need suggetion for setup

[admin@mypracticalsupport]

We are planning to move our production servers to Proxmox. We have not published our web product, but once published, there will be a heavy load on web and database servers. So we are thinking of going with Dell or Lenovo as physical servers. Before we decide, we need advice from you guys on which one to pick. We don't want to buy all the modules for hardware as a starting point. We would buy as we grow. For example, we would go with a 1 GB NIC card as a starting point but need to have capacity to upgrade to a 10GB NIC.

We need advice on how we should set up a storage pool. Should we do it at the hardware level or in proxmox as a ZFS pool?

Which drive should we choose for creating raid: SSD, HDD, SAS SSD,NVME SSD?

Should we install VMs on a storage pool or on their own separate SSD?

 

[jdancer]

I use Dell and Supermicro in production. Supermicro's are less expensive than Dells.

These systems don't particular need high IOPS, so they are using SAS HDDs. They do have max RAM installed.

I won't bother with 1GbE. 10GbE or higher is what you want.

Since databases do require high IOPS if doing high-speed transactions, you'll want enterprise SAS SSD or NVMe with PLP (power-loss prevention).

I use ZFS on standalone servers and Ceph for clustered servers. All VMs share a storage pool. ZFS will use any unused RAM as a cache and will release it for other use.

Ceph can be high-performance but that requires lots of nodes, RAM, 40GbE or higher networking, and NVMe.

I won't bother with HW RAID. Software-defined storage is where it's at.

I've heard of the DB VMs using their own storage but you'll want to test that first.

 

[admin@mypracticalsupport]

I use Dell and Supermicro in production. Supermicro's are less expensive than Dells.

These systems don't particular need high IOPS, so they are using SAS HDDs. They do have max RAM installed.

I won't bother with 1GbE. 10GbE or higher is what you want.

Since databases do require high IOPS if doing high-speed transactions, you'll want enterprise SAS SSD or NVMe with PLP (power-loss prevention).

I use ZFS on standalone servers and Ceph for clustered servers. All VMs share a storage pool. ZFS will use any unused RAM as a cache and will release it for other use.

Ceph can be high-performance but that requires lots of nodes, RAM, 40GbE or higher networking, and NVMe.

I won't bother with HW RAID. Software-defined storage is where it's at.

I've heard of the DB VMs using their own storage but you'll want to test that first.

Thanks, My plan is to use old dell poweredge r740 with XEON GOLD 6140 18core processor as storage servers. I will install truenas as bare metal and passthrough SAS SSD to create Storage pool. On Poweredge r740, I will have two ssd as boot drive in mirror, and rest of them passthrough(SAS SSD)

If Truenas bare metal does not work then I will install proxmox on server and install Truenas as VM. But drives will be passthrough to Truenas to create pool. In this case, I will have two ssd on server for Proxmox in mirror then I will create another 2 ssd drive mirror pool for truenas installation from proxmox and rest of SAS SSD form server will passthrough to truenas. This server will have 2 10g nic aggregated. Good thing is other server node use this truenas nfs share datastore to store VM data, so I will have all VM data here and in any case any server dies then I can run VM on this server as data is just sitting in Truenas VM and I can spin VM in storage server as well for backup. Do you think this idea is good for storage server?

As server nodes I will also use dell poweredge r740 with 2 X XEON GOLD 6140 18core processor and 2 mirror SSD for proxmox installation and store VM on storage server which will be connected here as shared storage via nfs share. I will have this type of 3 server nodes and all using nfs storage for VM from storage server. Would I be able to setup high availability in this setup, or it would not work?

All nodes and storage servers will have 2 X 10g nic as aggregated link with connected through 10g switch.

Please share your thoughts.