A Proxmox success story

[rorydog1]

I am not sure if this is the right thread. This is my first post to the forum and I would like to talk about a success story we had with Proxmox. I work for the STFC, one of the UK government's UKRI science and research councils. At out laboratory in Daresbury we design and implement control systems for, amongst other things, research accelerators. We use EPICS, an open source SCADA distributed control system framework under Linux.

We used to use racks of servers connected to all sorts of PLCs and other hardware via ethernet, Ethercat, directly to seral, Modbus etc. Everything is expected to run 24/7 with rock solid reliability. Some of the OS versions are ancient as they are frozen for stability with attached hardware. A number of years ago, I started experimenting with KVM then rapidly moved to Proxmix (v6 back then I think). I had all sorts of crazy hardware setups, complex Ethercat and Modbus coms and custom Linux versions thrown at my test rig and Proxmox has come through unscathed. Some systems have now ben running for over three years without problems.

I just wanted to mention this to the Proxmox community as it has proven itself as a solid SCADA system and has added a huge amount of flexibility to our control system. I wondered if anyone out there was virtualising hardware and SCADA and if they had tried Proxmox or is this an edge case

 

[finnzi]

Cool!

Can you go into some details regarding storage setups, etc in your environment? I'm very interested to hear about companies deploying Proxmox in production - I'm currently implementing a 3-node cluster with Ceph as the storage for a small company using SuperMicro 1U servers, and Mikrotik networking gear.

Bgrds,
Finnur

 

[rorydog1]

Sure, each rack contains a number of hardware units, two network switches and a Dell server, Each server has two or more NICs . The server runs Proxmox and inside, a number of VMs connect to both NICS which is running EPICS to connect hardware to the SCADA, However, these VMs are dumb as all the code they run is mounted from a central NFS server. Everything is fixed on a specific verion of Centos 7 with a copy of the yum repos stored locally on a Nexus server.

To wire all these racks up, in each rack one switch connects to ethernet from the hardware/PLCs and the other to the site's LAN where users can connect to EPICS and monitor everything. There are a load of these racks dotted over the site.

I tried Glusterfs and Ceph but these distributed file system were slow to compile on. Instead we stuck with NFS. There is a master Proxmox host that runs an NFS vm that all the rack Proxmox VMs mount from. That has a ZFS raid array. Software is compiled locally in that NFS host from other VMs as build hosts and distributed to rack VMs. There are different mount points for different OS versions. This master Proxmox server has several build servers as VMs, Centos, Alma, kernel tweaks etc. It runs as a cluster for redundancy incase we need to bring the NFS server down for upgrades. With migration, the VMs do not notice the NFS server changing. The cluster is using ZFS replication to keep the two servers in sync.

I tried HA, for the NFS server but it is better if things break as the only accidents have been human error and we don't want HA to hide that. Proxmox has been rock solid as an NFS server with a heavy duty UPS.

Everything is being driven by Ansible scripts that deploy new EPICS VMs. We use pure KVM, LXD has issues with kernels and NFS mounts and Docker didn't have SSH which everyone expects. Apart from the master server all other VMs are disposable and rapidly destroyed/rebuilt with Ansilbe scipts running a restore from a Proxmox Backup Server with master images.

We use cron scripts to sync data to a site backup service and Proxmox Backup server as well.

We probably threw away a lot of technology options but it gives a very stable setup that is close to a physical servers that users and engineers are used to.

 

[Chris]

I am not sure if this is the right thread. This is my first post to the forum and I would like to talk about a success story we had with Proxmox. I work for the STFC, one of the UK government's UKRI science and research councils. At out laboratory in Daresbury we design and implement control systems for, amongst other things, research accelerators. We use EPICS, an open source SCADA distributed control system framework under Linux.

We used to use racks of servers connected to all sorts of PLCs and other hardware via ethernet, Ethercat, directly to seral, Modbus etc. Everything is expected to run 24/7 with rock solid reliability. Some of the OS versions are ancient as they are frozen for stability with attached hardware. A number of years ago, I started experimenting with KVM then rapidly moved to Proxmix (v6 back then I think). I had all sorts of crazy hardware setups, complex Ethercat and Modbus coms and custom Linux versions thrown at my test rig and Proxmox has come through unscathed. Some systems have now ben running for over three years without problems.

I just wanted to mention this to the Proxmox community as it has proven itself as a solid SCADA system and has added a huge amount of flexibility to our control system. I wondered if anyone out there was virtualising hardware and SCADA and if they had tried Proxmox or is this an edge case

Glad to hear!
Having myself a background in Transmission Electron Microscopy I am pleased to say that I know the SuperSTEM facilities at Daresbury (I have no idea however if these are also somehow covered by your Proxmox setups) and had the pleasure to meet Prof. Ondrej Krivanek and Prof. Quentin Ramasse, especially during their frequent visits during the time of the installation of the (at the time) new Nion STEM at the University of Vienna.

Keep up the excellent science!