Migrating Production cluster to new HW

a_d

New Member
Jan 4, 2018
3
0
1
Hi,

I'm currently in the process of planing a migration of a production
PVE cluster to new hardware, which should be acomplished with minimal
(idealy no) downtime.

The old HW is:
* 3 Nodes
** VMs are stored on Ceph
** 2x Intel E5-2640 v2 / 64GB RRAM
** 2x 400GB SSD (System + Ceph Journal)
** 4x 300GB SAS 15kRPM OSD w/ SSD Journal (Filestore)
** 2 Port X520 10GBe Ethernet (Ceph + Cluster)
** 2 Port GBe (Frontend)

Given the Expirience over the 8 years running this configuration,
the new HW is slightly different:
* 3 Nodes:
** VMs are stored on Ceph (3 copies)
** 1x Intel Xeon Gold 6134/128 GB RAM
** 2x 400GB SSD (System + DB/WAL)
** 6x 1.8TB 10kRPM OSD (Bluestore)
** 2 Port X710 10GBe Ethernet (Ceph + Cluster)
** 2 Port GBe (Frontend)

My Idea would be (after burn in tests of the new HW):
* Install the new nodes (3 copies)
* add them to the cluster
* for each node:
** add the new Disks as OSDs
** Remove the OSDs from one old node
** wait until redundency is re-established
* HA-migrate the VMs to the new nodes
* Migrate (with restarts of course) the LXC Containers
* remove the old Nodes from the Cluster

In theory this should migrate everything with minimal downtime
(none for the VMs) and risk (as i still have redundency
for all data given that i'm starting 3 copies).

I still have some questions however:
* Will this be possible without getting addtional subscriptions for the new hosts?
(given that they only run in paralell for the mirgration, i would like to avoid that)
* Are there known problems live migrating between different CPU generations (older to newer)?

ps: i'm aware that the hardware configuration has changed quite a bit, over the 3 years the old
cluster was running, the requirements changed a bit
 
* Will this be possible without getting addtional subscriptions for the new hosts?
Yes you can migrate the subscription to the new nodes.
But be sure your system is updated.

* Are there known problems live migrating between different CPU generations (older to newer)?
You should avoid cpu type host because on the new Host you have an other cpu type.
It dependence on the VM OS how it will react.
KVM is fine with newer Intel CPU because they have in general always more CPU flags.
But I would set the CPU type of your VM to your old CPU type.

On point to your HW.
It is recommended to run WAL/DB/Log always on a dedicated SSD.
On a normal SATA SSD you can run up to 4 OSD.
In you setup I would recommend you an enterprise NVME or 2 SATA enterprise SSD.
 
Yes you can migrate the subscription to the new nodes.
But be sure your system is updated.
Thanks, i'm aware it can be moved (i have done that before when HW died)
My question was geared towards the situation where I would basically have the same
subscription in the same cluster for the short duration of migrating everthing, as i have to
enter the subscription key before joining the cluster if i read the documentation correctly?

You should avoid cpu type host because on the new Host you have an other cpu type.
It dependence on the VM OS how it will react.
KVM is fine with newer Intel CPU because they have in general always more CPU flags.
But I would set the CPU type of your VM to your old CPU type.
Thanks for the information...i will change the cpu type when i have to do system updates in the VMs the next time.

On point to your HW.
It is recommended to run WAL/DB/Log always on a dedicated SSD.
On a normal SATA SSD you can run up to 4 OSD.
In you setup I would recommend you an enterprise NVME or 2 SATA enterprise SSD.
I'm aware that that sharing the system with the WAL not 100% ideal, however since the system normaly only
produces only minimal io, so i had no real performance problems using some of the unmirrored storage space
at the end of the system ssds as journal for filestore, and the new ssds (enterprise grade sas ssds) should be
fast enough in a similar configuration, i'm aware of the drawbacks however
 
My question was geared towards the situation where I would basically have the same
subscription in the same cluster for the short duration of migrating everthing, as i have to
enter the subscription key before joining the cluster if i read the documentation correctly?
This will be no problem, if you reissue the subscription to a new node.
The only thing is you can't update the old node anymore.
The key of the old node will be invalid at the time you reissue the key.
 
My Idea would be (after burn in tests of the new HW):
* Install the new nodes (3 copies)
* add them to the cluster
* for each node:
** add the new Disks as OSDs
** Remove the OSDs from one old node
** wait until redundency is re-established
* HA-migrate the VMs to the new nodes
* Migrate (with restarts of course) the LXC Containers
* remove the old Nodes from the Cluster
Hi,
this part I understand not right.
What do "Install the new nodes (3 copies)" mean? Do you mean ceph replica of 3?!
Or simply three nodes, which you use to expand your existing Cluster?
I guess this is mean.
In this case it's works - but move the ceph-mons also.
And your OSDs are not numbered started with 0... (but you can reweight osd.0 to 0, remove them and then add your first osd on the new cluster (become osd.0), And further step by step.

Udo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!