C
Chris Rivera
Guest
1. If you want to build the perfect proxmox cloud... start off by following the suggestions. Then don't over subscribe your nodes till you end up in some trouble. While you can oversubscribe proxmox over and over... the more you over subscribe your node the more issues you may encounter when you are in production.
Proxmox 8core cpu with 32gb hdd and a single hdd can run 100vms. While it can run 100 vms concurrently there are things you need to be aware of.
2. Do not use single Hdd... as simple as this sounds when you finally launch your production node and your on a single drive your asking for trouble. While proxmox can be recovered from minor corruptions you may end up with a loss of filesystem , client data which is not good , non bootable system.
Data loss = stress and time lost
Raid 1 = ok ( good in case 1 drive crashes, you have an extra copy of the data )
Raid 5 = better ( good space - less drives / performance increase )
Raid 10 = preferred (less space - more drives / performance increase)
** I've read that while raid 5 gives you more space... its a nightmare to fix if you ever need to fix it, also when 1 drive is gone performance is dropped dramatically, and will continue to be slow while building the new drive... Using Raid 10 best performance faster ability to recover faster, less impact when 1 drive is lost..
3. Keep your vm interface and proxmox interfaces seperate. For any reason if you keep both interfaces on the same port and a client vm gets DDoS you end up with not being able to manage your node from a GUI. While you can manage vms using CLI managing nodes with large amounts of vms will be equal to shooting your self in the face. ( something you don't want to do ). trying to track down whose being hit with a 6gb DDoS using CLI on a host node with 75+ vms is not fun or quick.
Keep your proxmox interface separate. In the case your client VM gets DDoS you can access the management interface using a private or public ip.
FYI: keep your vm interface and proxmox interface on 2 different vlans... Some DDoS attacks might saturate the whole vlan making all servers on the vlan either slow or make them inaccessible leaving you locked out if your accessing the server remotely.
**** This is VERY IMPORTANT for clusters
4. Have a KVM setup and ready. Lots of times i have found accessing the server to be slow or not responsive because of abusive client applications running. While remote connections to the server stop working, accessing a KVM you set up for cases like this will save you time. Using KVM you can access your box as you would if you were at the datacenter and you can manage the node from there.
5. By default when building a cluster start off with 2 servers. 1 for virtual hosting and 1 for network storage (raid 10). Don't provision on local then think later you will move it to a shared storage... this is not an easy task or some task you will wish to take on in the future when you wish you improve your infrastructure.
While you can choose to migrate your selected vm to another node you do not have the option to choose a different storage option available on that new node. Meaning if you have node1:vm100:localstorage you can choose to migrate it to node2 but the GUI will not allow you to choose the remote storage that is available on node2.
While this can all be done manually using CLI, this is not a point click task this will take some time.
6. Running a cluster is easy if you set it up right the first time. if you share the vm and proxmox interface on 1 nic you are asking for problems. Any vm obsessive data can delay or cause proxmox interface to loose sync / show red / or be considered offline. When this happens the pve filesystem which holds all .conf files becomes read only.
If any management things are done while a node is offline ( red ) you'll be sure to notice unexpected issues such as:
This is my list of dont dues that i have personally learned from... if you guys have more personal dont do's this might actually make for how to build a perfect cloud.
Proxmox 8core cpu with 32gb hdd and a single hdd can run 100vms. While it can run 100 vms concurrently there are things you need to be aware of.
- Boot process - containers boot 1 at a time.... booting 100 vms might take some time
- Shutdown process - containers shut down 1 at a time (with delay).... shutting down 100 vms will take some time
- Improper shutdown - containers improperly shut down, often need to have their space checked ( high i/o demand ) before the vms are able to boot... if you have 100 vms that need to be checked, you will experience very long wait time. normal vz boot time 30 seconds .... improper shutdown boot time may be over 2-5 minutes..
- Oversubscribing - CPU and RAM may be easy to oversubscribe but you will run into an issue with running so many vms on a drive.. each vm has so much time to read / write to a drive since its shared... adding alot of vms will cause i/o delay to raise... If you have 1 client downloading a 100meg file you will notice I/O raise... using RAID 10 would be a way to sustain more reads / writes since you have more drives that can read / write
2. Do not use single Hdd... as simple as this sounds when you finally launch your production node and your on a single drive your asking for trouble. While proxmox can be recovered from minor corruptions you may end up with a loss of filesystem , client data which is not good , non bootable system.
Data loss = stress and time lost
Raid 1 = ok ( good in case 1 drive crashes, you have an extra copy of the data )
Raid 5 = better ( good space - less drives / performance increase )
Raid 10 = preferred (less space - more drives / performance increase)
** I've read that while raid 5 gives you more space... its a nightmare to fix if you ever need to fix it, also when 1 drive is gone performance is dropped dramatically, and will continue to be slow while building the new drive... Using Raid 10 best performance faster ability to recover faster, less impact when 1 drive is lost..
3. Keep your vm interface and proxmox interfaces seperate. For any reason if you keep both interfaces on the same port and a client vm gets DDoS you end up with not being able to manage your node from a GUI. While you can manage vms using CLI managing nodes with large amounts of vms will be equal to shooting your self in the face. ( something you don't want to do ). trying to track down whose being hit with a 6gb DDoS using CLI on a host node with 75+ vms is not fun or quick.
Keep your proxmox interface separate. In the case your client VM gets DDoS you can access the management interface using a private or public ip.
FYI: keep your vm interface and proxmox interface on 2 different vlans... Some DDoS attacks might saturate the whole vlan making all servers on the vlan either slow or make them inaccessible leaving you locked out if your accessing the server remotely.
**** This is VERY IMPORTANT for clusters
4. Have a KVM setup and ready. Lots of times i have found accessing the server to be slow or not responsive because of abusive client applications running. While remote connections to the server stop working, accessing a KVM you set up for cases like this will save you time. Using KVM you can access your box as you would if you were at the datacenter and you can manage the node from there.
5. By default when building a cluster start off with 2 servers. 1 for virtual hosting and 1 for network storage (raid 10). Don't provision on local then think later you will move it to a shared storage... this is not an easy task or some task you will wish to take on in the future when you wish you improve your infrastructure.
While you can choose to migrate your selected vm to another node you do not have the option to choose a different storage option available on that new node. Meaning if you have node1:vm100:localstorage you can choose to migrate it to node2 but the GUI will not allow you to choose the remote storage that is available on node2.
While this can all be done manually using CLI, this is not a point click task this will take some time.
6. Running a cluster is easy if you set it up right the first time. if you share the vm and proxmox interface on 1 nic you are asking for problems. Any vm obsessive data can delay or cause proxmox interface to loose sync / show red / or be considered offline. When this happens the pve filesystem which holds all .conf files becomes read only.
If any management things are done while a node is offline ( red ) you'll be sure to notice unexpected issues such as:
- Deleting vm - vm data was deleted but the vm.conf file could not be deleted due to read only access... so you can see the vm as its on the cloud but it will not boot since all the files have been deleted. To fix this you will need to add the node back to the cluster then delete the file manually.
This is my list of dont dues that i have personally learned from... if you guys have more personal dont do's this might actually make for how to build a perfect cloud.