Home PVE planning

[FourtyTwo]

Hello!
I'm going to setup my home lab to play with VMs and store different files on it.

General idea:
1. Single node built from spare old hardware
2. Four different HDD's + Two SSD's
3. Ceph storage with journal on SSD or SSD cache-tier
4. Maximum storage Fault Tolerance that i can reach with this setup

Worst faults in my scenario is generally disk failures. I don't care too much about downtime if my cpu will die but i want to save my stored data in case of disk failure.


Best option i see for highest and easiest FT:
1. PVE initially installed on ZFS RAID 1 on SSD's
2. Ceph storage with 4 OSD's (osd pool default size = 2, osd crush chooseleaf type = 0)
3. Journal and block.db on ZFS RAID 1 on SSD's (for speed)
4. MON store.db on ZFS RAID 1 on SSD's (by default)

In this case on any disk failure i just need to replace disk with new one. Cause OS and MON's store.db are placed on ZFS RAID i don't even need to reinstall system and MON will not be affected. So i don't need second node to store MON data. Also journal and block.db are secured. Am i right?


The problem is... i can't initially install PVE on ZFS
So i need to choose another way. Maybe in future i will upgrade my MB but not now.


The question is:
1. PVE initially installed on ext4
2. Ceph storage with 4 OSD's (osd pool default size = 2, osd crush chooseleaf type = 0)
3. Partitions on SSD's + ZFS RAID 1 on that Partitions ??? +
a. + SSD journal, block.db... and so on ???
b. + SSD cache-tier... and so on ???
4. MON store.db ???

In this case on OS SSD failure i will need to reinstall OS. But MON store.db are stored on ext4 and my storage will be ruined anyway.
Please help me on how i can make this setup more failsafe? Move store.db? How?
Will ZFS be affected on OS disk failure?

Sorry for long post and thank you!

 

[damon1]

Hi FourtyTwo,

if it was me with a single node and your major concern is disk failure then I would get an empty NAS which will cost a few hundred dollars like this and put your HDD's into it. - If you have two the same size.
You save your VM's on the NAS and use the SSD to install Proxmox

https://www.amazon.com.au/Synology-...logy+DiskStation+DS218j&qid=1580352969&sr=8-1

Alternatively, use the motherboard if it has RAID built in and you have matching HDD's or SSD's then use the raid to set up a RAID 1 drive. Then you can put everything on it.
Use the other drives for "Backups" which are automated and work very well in Proxmox.

My 2c worth anyway.
thanks
Damon

 

[FourtyTwo]

Hi damon!
Yeah, i thinked about that too but NAS is not interesting at all. I can spend few hundred dollars to buy new MB that can boot to UEFI and use 1-st variant.

Integrated RAID on my MB works as a shit but automatic backups are good idea, i will think about it.

Isn't there any option to move MON db like journals and block.db? As i understand this is the main problem.

 

[damon1]

Hi FT,
I use CEPH on a 4 node set up. Each node has one SSD and two HDD. I use cache tiering as well.
The good thing about ceph is that if one node ( or two ) go down then the VM's will run on another node.
The other good thing is if there was a HDD or SDD failure then I can just replace the drive .

I am in no way an expert but I don't think ceph is supposed to run on less than 3 nodes.
Saying that I did find this which will allow you to set up ceph with multiple OSD'd on a single node

https://linoxide.com/linux-how-to/hwto-configure-single-node-ceph-cluster/

It should guide you toward what you are wanting to set up, at least an interesting read otherwise.

Thanks
Damon

 

[FourtyTwo]

Hi damon!
Thank you for answer, i'm already found info about chooseleaf osd mode before. It will make my single node ceph to run properly for sure.

But what to do with MON's store.db? After SSD failure it will be lost and i can't add back old OSD's without old store.db. I tried to search about changing db location but didn't found anything that may help me.

 

[damon1]

Hi FT,
are you sure your not over thinking it.
I have run CEPH for a year and lost OSD's and lost NODE's, somehow it works it out.

Give it a try. build your system and then pull the plug, drop a drive and replace it.
I'm reasonably confident that brighter minds than mine will ensure it fixes itself (mostly)...

if you get an error post them - i am 100% sure I have a fix. There are some rippers.

remembers Monitors should be on 3+ nodes, so maybe it isn't recommended for a reason.

PS. my system has 3 very nice units and 1 really crappy one. No all nodes have to have 16 cores and 32 threads.



Damon

 

[FourtyTwo]

Well, making some tests is reasonable. I think i can find some time to simulate my lab under VMware Workstation.

Thank you damon!