Hello Proxmox users!
I am working on setting up a small personal "cloud" with a few servers. I am an enthusiastic infrastructure newbie, and I am hoping some friendly soul will push me in the right direction.
Background
My ambition is to use this infrastructure for a wide array of things - acting as backup for me and my extended family, running various things related to software development/hosting (kubernetes, ci-tools, git, web servers, databases) and maybe even a game server or two.
I initially looked at Cloudstack and Openstack, but I think that those technologies aren't all that suited for a small, personal deployment which has very real limitations such as only one WAN port in my apartment. I don't see any benefit in adding a lot of extra complexity since I won't be able to leverage the features they provide.
My current thoughts at hardware:
And so I arrived at Proxmox + Ceph. I found this video that demonstrated creating a Proxmox cluster that also ran Ceph on the same servers.
https://www.proxmox.com/en/training/video-tutorials/item/install-ceph-server-on-proxmox-ve
It seemed to be a great solution for me since I don't have room (or economy) for a large amount of servers. I am thinking about getting three of these machines:
CPU: Intel - Core i7-8700 3.2GHz 6-Core Processor
CPU Cooler: Cooler Master - Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler
Motherboard: MSI - Z370-A PRO ATX LGA1151 Motherboard
Memory: Corsair - Vengeance LPX 16GB (1 x 16GB) DDR4-2666 Memory
Storage: Western Digital - Green 120GB 2.5" Solid State Drive
Storage: 2 x Seagate - Barracuda 2TB 3.5" 7200RPM Internal Hard Drive
Case: Fractal Design - Define R5 (Black) ATX Mid Tower Case ($176.99 @ Amazon)
Power Supply: Corsair - RMx 550W 80+ Gold Certified Fully-Modular ATX Power Supply
Wired Network Adapter: 2 x Intel - EXPI9301CTBLK PCI-Express x1 10/100/1000 Mbps Network
Each HDD will be controlled by a Ceph OSD, and Ceph gets both of the extra NICs. Proxmox itself is installed on the SSD. I've chosen soundproofed cases to make the noise in my apartment as low as possible.
My questions
Should Proxmox be on it's own vLAN?
I will run Proxmox in my apartment, where I will also have other devices connected. Would it be prudent to separate my "casual" devices and the Proxmox cloud for security reasons? Or for some other reason?
Is it really possible to use a Raspberry as a third node?
I found this article, explaining how to use a raspberry to achieve quorum:
https://pve.proxmox.com/wiki/Rasberry_pi_as_third_node
It would be great if I could cut my costs by 1/3 naturally (Since I'll likely have more capacity than I need for a good while.), but I suppose that would make Ceph not work properly. Or can you achieve quorum using a Raspberry in Ceph as well?
Disk write speed vs network capacity?
The write speed of regular harddrives is quite low, seems to be around 200mb if you get a decent one (I realize the ones that are listed in the list are cheaper.). With two 1Gbit NICs for each ceph node, it seems I might barely reach the write capacity of even one of the HDDs in the machine. How should I setup/plan this part so that it works decently?
Do I need a new router for the corosync cluster network?
In the documentation here:
https://pve.proxmox.com/wiki/Cluster_Manager
I can read that it would be bad if storage (Ceph in my case) runs on the same network as corosync. Does this mean that it needs to run on a different router, or is it enough if it runs on it's own NIC (in my case, the one that is integrated into the motherboard)?
Can anyone recommend a router to buy for this lab?
And finally, do you have any suggestions on established ways of creating small clouds like the one I am describing?
Thank you in advance!
I am working on setting up a small personal "cloud" with a few servers. I am an enthusiastic infrastructure newbie, and I am hoping some friendly soul will push me in the right direction.
Background
My ambition is to use this infrastructure for a wide array of things - acting as backup for me and my extended family, running various things related to software development/hosting (kubernetes, ci-tools, git, web servers, databases) and maybe even a game server or two.
I initially looked at Cloudstack and Openstack, but I think that those technologies aren't all that suited for a small, personal deployment which has very real limitations such as only one WAN port in my apartment. I don't see any benefit in adding a lot of extra complexity since I won't be able to leverage the features they provide.
My current thoughts at hardware:
And so I arrived at Proxmox + Ceph. I found this video that demonstrated creating a Proxmox cluster that also ran Ceph on the same servers.
https://www.proxmox.com/en/training/video-tutorials/item/install-ceph-server-on-proxmox-ve
It seemed to be a great solution for me since I don't have room (or economy) for a large amount of servers. I am thinking about getting three of these machines:
CPU: Intel - Core i7-8700 3.2GHz 6-Core Processor
CPU Cooler: Cooler Master - Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler
Motherboard: MSI - Z370-A PRO ATX LGA1151 Motherboard
Memory: Corsair - Vengeance LPX 16GB (1 x 16GB) DDR4-2666 Memory
Storage: Western Digital - Green 120GB 2.5" Solid State Drive
Storage: 2 x Seagate - Barracuda 2TB 3.5" 7200RPM Internal Hard Drive
Case: Fractal Design - Define R5 (Black) ATX Mid Tower Case ($176.99 @ Amazon)
Power Supply: Corsair - RMx 550W 80+ Gold Certified Fully-Modular ATX Power Supply
Wired Network Adapter: 2 x Intel - EXPI9301CTBLK PCI-Express x1 10/100/1000 Mbps Network
Each HDD will be controlled by a Ceph OSD, and Ceph gets both of the extra NICs. Proxmox itself is installed on the SSD. I've chosen soundproofed cases to make the noise in my apartment as low as possible.
My questions
Should Proxmox be on it's own vLAN?
I will run Proxmox in my apartment, where I will also have other devices connected. Would it be prudent to separate my "casual" devices and the Proxmox cloud for security reasons? Or for some other reason?
Is it really possible to use a Raspberry as a third node?
I found this article, explaining how to use a raspberry to achieve quorum:
https://pve.proxmox.com/wiki/Rasberry_pi_as_third_node
It would be great if I could cut my costs by 1/3 naturally (Since I'll likely have more capacity than I need for a good while.), but I suppose that would make Ceph not work properly. Or can you achieve quorum using a Raspberry in Ceph as well?
Disk write speed vs network capacity?
The write speed of regular harddrives is quite low, seems to be around 200mb if you get a decent one (I realize the ones that are listed in the list are cheaper.). With two 1Gbit NICs for each ceph node, it seems I might barely reach the write capacity of even one of the HDDs in the machine. How should I setup/plan this part so that it works decently?
Do I need a new router for the corosync cluster network?
In the documentation here:
https://pve.proxmox.com/wiki/Cluster_Manager
I can read that it would be bad if storage (Ceph in my case) runs on the same network as corosync. Does this mean that it needs to run on a different router, or is it enough if it runs on it's own NIC (in my case, the one that is integrated into the motherboard)?
Can anyone recommend a router to buy for this lab?
And finally, do you have any suggestions on established ways of creating small clouds like the one I am describing?
Thank you in advance!