Proxmox boot and Ceph on one disk possible?

[MartijnNL80]

Hi all,

I'm planning to make a low power HA cluster with shared ceph storage only for my services that need to be always up (Pihole, VPN, Unifi controller and Home Assistant).
My main Proxmox server has enough disks available for ceph but the two Intel Nucs I have laying around only can fit one nvme ssd and that is where I need to install Proxmox.
The disks are 1TB and I only need a very small amount for the Proxmox boot part.

Is it possible to use the unused storage of the nvme ssd for a ceph disk?
Or is there a way to make a Proxmox nvme boot disk that is already partitioned so that the unused space can be used for ceph?

Any help would be appreciated.

Kind regards,

Martijn

 

[jsterr]

Hello Martijn,

This is a maximum overkill and in the same time maximum complexity for home server, please use zfs replikation, this will be a lot easier with higher performance and also high availabilty, and little to no data loss, depending on how often you sync your vms storage with zfs replication in web-ui.

A useful ceph starts with 10gbit networking, multiple osds per node and separated osds (not mixed with os).

Greetings Jonas

 

[MartijnNL80]

Hello Martijn,

This is a maximum overkill and in the same time maximum complexity for home server, please use zfs replikation, this will be a lot easier with higher performance and also high availabilty, and little to no data loss, depending on how often you sync your vms storage with zfs replication in web-ui.

A useful ceph starts with 10gbit networking, multiple osds per node and separated osds (not mixed with os).

Greetings Jonas

Hi jsterr,

Thanks for your reply!

Im not looking for data or vm backup but for a way to keep my services running when my main server goes down.
I don't think zfs replication can move a proxmox lxc to another server and start so there is minimal downtime and no humen intervention needed.
Or am i mistaking?

Regards,
Martijn

 

[jsterr]

Hi jsterr,

Thanks for your reply!

Im not looking for data or vm backup but for a way to keep my services running when my main server goes down.
I don't think zfs replication can move a proxmox lxc to another server and start so there is minimal downtime and no humen intervention needed.
Or am i mistaking?

Regards,
Martijn

Thats totally possible, its the same as with ceph, but with the difference that the lxc container starts with the last sync that you did via replication. you can set the task to be done ever x-minutes, depending on how much data you write in a minute and what network you have, you can set the schedule as low as possible. If the node fails, your lxc (that is in ha mode) will automatically be started after 3-4 minutes on the second node.

You can also you livemigeration etc. and it will only need to "move" the data that has changed to the second server since the last sync. so you benefit in multiple ways:

- ZFS Replication for High Availability (ZFS Pool Name needs to be the same on both nodes)
- Quicker Livemigration (because of the already synced zfs)
- Automatic switch of the sync from node2 to node1, when node1 recovers from its failure (thats importan because node2 has the newer storage-state so proxmox does automatically switch the sync from n2 to n1 after n1 fails)

Greetings Jonas
Thomas-Krenn.AG

 

[MartijnNL80]

Thats totally possible, its the same as with ceph, but with the difference that the lxc container starts with the last sync that you did via replication. you can set the task to be done ever x-minutes, depending on how much data you write in a minute and what network you have, you can set the schedule as low as possible. If the node fails, your lxc (that is in ha mode) will automatically be started after 3-4 minutes on the second node.

You can also you livemigeration etc. and it will only need to "move" the data that has changed to the second server since the last sync. so you benefit in multiple ways:

- ZFS Replication for High Availability (ZFS Pool Name needs to be the same on both nodes)
- Quicker Livemigration (because of the already synced zfs)
- Automatic switch of the sync from node2 to node1, when node1 recovers from its failure (thats importan because node2 has the newer storage-state so proxmox does automatically switch the sync from n2 to n1 after n1 fails)

Greetings Jonas
Thomas-Krenn.AG

Hi Thomas,

Thanks for the feedback!

I ended up creating a HA 3 node cluster with the main Proxmox host being virtualized on my unraid server (wich has loads of horsepower) and the other two nodes running on two Lenovo mini pc's (Intel core I5 6500T's).
This setup works like a charm, whenever my main unraid server goes down the cluster picks up the most important containers and the most important services keep running. Love it!
If i manually stop my virtualized proxmox hosted on my unraid server it only takes 1 min and 15 secs for one of the HA nodes to pick up and let me have all my services back again