Orchestration Best Practices

[Optimus_sRex]

I am looking for some Orchestration Best Practices using Proxmox. My goal is to take a small/medium-sized business with some 30 servers in a mixed Windows/Linux environment and make them more portable using Proxmox and eventually migrate them to the cloud (Amazon/Azure/Google) using orchestration tools. The idea was to create a test environment via Proxmox and then use that as proof of concept and development playground, then begin transitioning them to a cloud service.

Originally I was looking to replicate the Packer/Terraform/Ansible model similar to this deployment but with Proxmox:
http://tech.equinox.com/engineering...deployment-with-packer-terraform-and-ansible/

But, after determining that Packer did not support multiple ISO images
https://github.com/hashicorp/packer/issues/7950

I determined that Ansible would be the best choice, given that this Windows template builder works pretty well:
https://github.com/clayshek/ans-pve-win-templ

At this point, I've started building an elaborate Ansible Playbook. My concern is that ultimately these ansible plays are built using a combination of pvesh, pct, lxc-attach, and qm. The problem is that this is not portable. The benefit of using Packer/Terraform/Ansible is that we aren't tied to one VM or cloud manufacturer. This orchestration technique ties us to one.
I am beginning to think that I need to simply replace Packer with Clay Shekleton's Ansible Windows Template system and then try to keep a similar structure to Wilmar Stephen's Packer/Terraform Ansible model.

Thank you for any thoughts you might have.

P.S. I will be posting some code in the next week and will be updating this post.

 

[vkosuri]

Hi @Optimus_sRex quite interesting any updates?

 

[hacman]

We've been deploying something similar lately, and what we have ultimately used is a combination of CloudInit, template VMs and Ansible. We then have an exteral IPAM system handling much of the rest.

 

[vkosuri]

thank you, to improve IOPS which storage system your preferring either cephFS or glusterFS

 

[hacman]

thank you, to improve IOPS which storage system your preferring either cephFS or glusterFS

We are currently using Ceph, though this decision was based around a desire for tight PMVE integration and a few other requirements, rather than raw IOPS.

We're using commodity hardware, with many, many, 300GB disks. Performance is more than adequate.

 

[Optimus_sRex]

I've gone down the path of trying to get my default template to have all the drivers and preinstalled software using Ansible code derived from ans-pve-win-templ . While I have the drivers locked down, I can't seem to get some default programs installed (Ansible, QM-Agent, and Cloudbase-init).

To that end, I think I am going to give up and just provide basic instructions for installing those base programs.

I have basic Terraform/Telmate working once those programs are installed. I haven't gotten to post-deployment configuration, but my expectation is that that should be easier.

I know I have promised code, but I also thought that autounattend.xml would be easier. But apparently Microsoft has made SSCM a necessary component for this type of preinstall.

 

[vkosuri]

We are currently using Ceph, though this decision was based around a desire for tight PMVE integration and a few other requirements, rather than raw IOPS.

We're using commodity hardware, with many, many, 300GB disks. Performance is more than adequate.

Wow. How does IP address and subnet management? Any documentation would help me great.

 

[hacman]

Wow. How does IP address and subnet management? Any documentation would help me great.

So for this aspect we've built an external IPAM system, based on PHP IPAM and a few other tools. We then have DHCP providing the IPs and keeping the DNS / hostnames updated based on that. We use DHCP relay servers on each site to pass these requests forward.

I don't really have much in the way of documentation I can share on this at present, as it's an ongoing internal project, but we can keep this thread going with questions and such.

When it's all completed I will try and put some of the information into a blog post or something, as I'm sure we're not the only ones who would benefit from this.