Seeking advice on best practices

[TVKP]

Long-time Windows administrator making the jump into a VMWare shop that is looking to migrate out to PVE. I'm trying to get my head around Proxmox and am working on a few experimental servers so that I can make recommendations on the environment and long-term goals of the build. I'll probably be looking to build into a 10-15 node cluster. At present, the VMWare build is 15 stand-alone nodes (I've inherited a rat's nest of a datacenter) each with its own storage. At least starting off, a fair bit of this will involve bringing the initial cluster online, migrating VMs over, then repurposing the existing servers to join the cluster and take on more of the VMs.

So, a few questions:

1. Storage: My hyper-v stacks previously have been clusters of a handful of nodes with SAN storage. Would that be the best option, or is Ceph better? It reads like it's the storage on the VXRail system I administered a while back, distributed across nodes.
2. What about backups? I tried setting them up to target an iscsi target, but I seem to be missing a step and can't find the SAN as a target.
3. Any suggestions/warnings/advice on building out the cluster for fault tolerance?
4. What am I not even thinking about yet that I should know before I take this to production?

 

[bbgeek17]

Hi @TVKP , welcome to the forum.

All your questions are valid to ask. The answers, as usual, depend on variables that are too many to list them all.

1 Storage: the choices between centralized SAN and Distributed storage will be influenced by: your expertise, your comfort with technology, your need/want of reusing existing infrastructure, your budget, etc. There is no simple "this is better than that'.

2 Backup: Your tried to point what backup solution at iSCSI target that is presented as what?
Lets assume for a second that your tried to use PBS and point it to initialize an iSCSI disk. Last time I checked the PBS specifically filtered out iSCSI LUNs. You can certainly workaround that by preparing the DISK/FS manually.

3 Cluster: the cluster discussion is a weekly occurrence in the forum. There are several long and informative threads. Asking a specific technical question will be more productive than simply asking for suggestions. That said, I suggest starting with https://pve.proxmox.com/wiki/Cluster_Manager

4 You are likely not thinking that you need a Partner but you may actually benefit from one: https://proxmox.com/en/partners/find-partner/explore

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

You should also be aware that the vsphere portfolio doesnt overlap completely with PVE. I would begin by mapping out what your guests do, and what services do you depend on; do you use dswitches, vrops, storage plugins, nsx, etc.

1. Storage- @bbgeek17 answer is the literal truth, but be aware that the only FULLY supported clustered storage provided by PVE is ceph. Ceph has a steep learning curve, networking complexity, and storage capacity overhead in order to be both performant and dependable. To utilize other topologies such as iscsi block storage effectively you may need to contract software vendors such as Blockbridge.
2. backup is a bigger point then I think you are alluding to. What is the current solution? do you want/need to be able to restore from pre migration backup sets? do you need to integrate it with other infrastructure?
3. yes. dont mix networks (dont comingle traffic on the same interfaces.)
4. If there are things you arent thinking of, you havent done your due diligence. see preamble.

 

[guruevi]

Here is a good thing to keep in mind: Proxmox is a KVM-based solution. There are other KVM solutions, most of what you find about it is interchangeable.

As others said, you paid VMware a pretty penny, don’t be too stingy on support costs or talk to your solution vendor, they can often help you find what you need, including people. If you have licenses or supported hardware, may actually be worth selling it or getting a buy-back provided the licenses transfer. People are desperate to keep their stuff alive after getting the last upgrade quotes from Broadcom.

You inherited a rat’s nest, that’s not unusual, make migration plans, make them vendor and solution agnostic at first. Then find what you need. Proxmox is a lot more flexible than eg. VMware or Xen, but don’t go to deep into optimizations, and homelab solutions. Just because you can, doesn’t mean you should. Investing in slightly better hardware is probably better than spending hours optimizing existing outdated hardware. It’s great to learn on at home but not in production. Yes, there is a cool factor to setting up your SDN with BGP, but I work in big datacenter, I haven’t needed much more than the humble VLAN, unless I’m actually trying to isolate an outdated device, I don’t really use SDN.

Ceph is good, slam some good hardware on it and as long as you don’t stretch the limits, you’ll be fine with defaults. Can you squeeze 1M IOPS on 10/25G sure, do you need it, no, just get 100G if that’s your need. Buy enterprise-grade hardware and make everything redundant, there were a lot of VMware VAR squeezing by with limited investment to maximize their return, adjust your expectations accordingly. If you’re booting VMware from SD cards (was a thing at the low end) invest in some proper boot drives. There are other solutions and you can reuse that iSCSI array, some vendors are now actively supporting Proxmox, then again, is it better to sell your overpriced VMware-approved gadget and buy new ‘dumb’ storage?

 

[alexskysilk]

Can you squeeze 1M IOPS on 10/25G sure

the theoretical maximum 4k frames on a 10gb interface is 300938@1500mtu, 302294@9k. just sayin. in practice you're going to be limited more by the storage than by the network anyway, but that just comes back to the initial point about architecting the entire system to meet a defined performance criteria.