Proof of concept at my workplace

[Mike Toupin]

Good day,

I am setting up a proof of concept of various solution from Cloudstack, Openstack and also wanted to try Proxmox since it is a lot easier to maintain and deploy as the other 2 solutions.

We having 8 machines with Intel HT 8 Core 3.6Ghz, with 64G of RAM, 1 SSD(512G) and 1HDD 1T on each machines.

I was looking to setup a cluster of 8 nodes (8 machines) and make it High Availability but using only some nodes (3) for Ceph but I am not sure if this is the best approach. Do you think it is better to using 3 nodes for storage or should I be using all the 8 nodes and using only the 1T HDD for the Ceph with the 10G networking? I expect to have a total of 16 drives (likely 2T each in my final setup).

Switches are not a problem, I can purchase more with 10G or 1G connectivity. We don't expect to run much VMs (not more than 5-8) but combined with some lightweight containers. Some of our VMs will be heavy (Windows 10 with Tableau + Alteryx) and playing with large database.

Thanks for any feedback you can provide on the best way to deploy a 8 nodes Proxmox Cluster with HA/Ceph for storage.

Best regards,

 

[AlexLup]

The more nodes you have, the faster ceph is.
You will however also need SSDs/NVMe's for journals and/or writeback cache to get some speed.
10G switches are a must if you have NVMe's, if you don't have that its not really worth it as it only speeds recovery (rebuilding) and not normal transfer speeds since CEPH reads from one disk at the time having the others as failovers as opposed to torrent/raid like situation where all disks participate little by little.

That said, 10G network and Proxmox OS SSD will make HA much faster, milliseconds on my crappy setup

 

[Mike Toupin]

Thanks AlexLup. My only other question now is in regards of the Enterprise repos. My Proxmox Cluster will be a standalone cluster without any connection with the external network/Internet. As such updating Proxmox Enterprise will not be possible via Internet/Proxy. What I usually doing with our Windows and Linux distros, is I doing on another network downloading the repos (with Wget) and save the files on my HDD. I moving the HDD on my standalone network and rebuilding the repos with APT to operate those repos as local repos only and updating the distro Sources.list file. If Proxmox Enterprise repos permit me to copy the file with Wget then I will be quite fine with that despite it is taking a lot of my time to maintain such network.

Thanks if you can confirm this!

 

[AlexLup]

Never done offline install of .debs, but it sounds like you have a good grasp of that. Check the below.

You need to swap out the Enterprise repo to the free one like so:

# Swap to free distroupgrade
echo -e "deb http://ftp.se.debian.org/debian stretch main contrib\n\n# security updates\ndeb http://security.debian.org stretch/updates main contrib\ndeb http://download.proxmox.com/debian stretch pve-no-subscription" > /etc/apt/sources.list
echo "" > /etc/apt/sources.list.d/pve-enterprise.list

apt-get update
apt-get dist-upgrade -y

 

[Mike Toupin]

Yes, I knew already for the pve-no-subscription and this one is easy to fetch with wget. However this is the pve-enterprise that I need and I would like to know if Proxmox permit us to fetch the enterprise repos with wget or any other type of sync capability? The pve-no-subscription is pretty strait forward but I absolutely need the Enterprise packages in order to keep the cluster stable and reliable. Local repos is my only possible solution, if not I will have to go with OpenStack or Cloudstack.

Thanks again!

 

[AlexLup]

I understand. Since I am a bit cheap still until my proxmox/ceph NAS is done and ready I don't have a subscription (yet), so I wont be able to answer your question..

 

[mir]

Debian has tools to create your own local copy of proxmox enterprise repo:
https://wiki.debian.org/DebianRepository#Set_up_and_maintain_a_repository