Challenge: setup something good with oldish HW for SMB use

[Dandan]

Hi all,

We are a team of two and work as content creators, got a new Workstation and want to reuse the old one for 2/3 functions:
first and foremost as a backup server second as a fileshare-server i.e use of truenas and maybe later try using pfsense.
I read this excellent article https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/ and still reading the proxmox and
truenas documentation. I am aware of 3,2,1 Backup strategies, but right now we are using an old WD MyMirror-Nas as Backup...
In the past I used VMware Workstation and loved it alot, but now i wish to use proxmox.
We have to balance how much to spend vs data security/integrity.

If i could get your feedback about my idea/setup on how to repurpose the old WS , id be grateful.
I already installed proxmox and truenas as VM - everything seams to work

My main concern is an "all eggs in one basket".

Some premises:
-we don't want to use "off-site paid cloud solutions",
-we want to spend as less as possible
-we would love to use zfs snapshot function in proxmox and truenas but don't really understand how
-some down time is not critical
-off site storage doesn't fit the bill: Natural disasters are highly unlikely, the worst that could happen is
a fire, but then again , if our office is burned down, we have bigger problems. Theft? So may God help us. We take that risk!
-we do value our data - who doesn't?

The old WS has
-i7-5930K CPU (3.50GHz, 12 cores)
-32 GB Ram
-4x 8TB HDD (new Seagate Ironwolf)
-4x 74GB HDD (old WD Raptors)
-2x 4TB HDD (oldish WD Black)
-2x 1Gbe Nic's

other HW lying around: the above mentioned NAS 2x3TB, an oldish Thinkpad Laptop W500, 3 oldish ssd


My setup up to now - One hypervisor and 2 Truenas VM's

1. Hypervisor
- I installed proxmox on 2 Raptors in mirror (zfs). Nothing else installed/stored (well also the truenas iso, but who cares)
Backup and Recovery plan:
I read here in the forum that if i backup the /etc/pve /etc/network/interfaces /etc/passwd and /etc/resolv.conf to external usb stick
i simply need to reinstall proxmox and restore the files from the external usb stick, keeping in mind that one hdd could fail (mirror)

-> Correct? this works? only this folder (/etc)?

What i basically want to accomplish is separate OS proxmox from data and the same for the truenas VM's, but here already i have some problems...

2. Backup-server
- I installed Truenas as a VM on a third Raptor and i "passed-through" the new 4x8TB to the VM. My problem here is if i use the proxmox
build-in backup solution, proxmox will _obviously_ backup the whole VM: the Raptor AND all 4 new 8TB Hdds. But I only want to backup the
OS Truenas similar to the Backup and Recovery plan above for the Hypervisor: simply restore the VM OS. My concern here is the failure of the
old Raptor
- inside the truenas VM i created a pool in raidz-2 with the 4x8TB Hdd's -> and this is my "backup of the backup recovery plan"! Raidz-2

On this server i wanted to backup the data of a) my new WS b) the other Laptops and PC's ANd b) the Fileshare Server of the second Truenas VM.
I was planing to use Truenas Urbackup plugin, simply because i don't understand how i could use the snapshot function of zfs in this case.
The new WS and all other PC's/Laptop's run on Windows or MacOS.

Probably its the elephant in the room i don't see - Any help/input here is greatly appreciated. How to backup the new WS,Pc's, Fileshare-server
using snapshots? (I think my VMware backround is messing with my head

3. "Fileshare-server"
- I installed a second truenas as a VM on the fourth and last Raptor and i "passed-through" the 2x4TB to the VM. Same problem as above: how do
i backup the OS Truenas and not also the data? Inside this second truenas-VM i created a pool in mirror with the 2x4TB Hdd's. I want to back up
this 4TB pool to the Backup-server... how? using the snapshot function? or the Urbackup plugin?
Since we are using big media files, are there any special zfs settings to be aware? Should i change the block size to 128k or higher?
Should i add a L2ARC to the pool with a ssd? (following the input of the above mentioned link). Any performance benefits for this case/big files?

So i am punching with these ideas/concepts, and would love to here your feedback.

- Separate OS from data --> usefull or not? , if yes , is my implementation correct? what could i do better? did i overshoot / overthink?
- Should i use only one truenas VM with two zfs pools, instead of two VMs with one pool each?
- It seems to me that i do have some redundancy (hypervisor is in mirror mode installed, Backup server in Raidz-2, File server in mirror mode, each OS on seperate HDD),
but is this a false sense of security?
- Do you see some evident perfomance issues? L2ARC, Blocksize, 1Gbe NIC, Other? Would a 10GbE NIC help? Would the HDD be too slow for the 10GbE NIC?
- How would you repurpose the old WS for this case?
- What is the _biggest_ problem of my setup?

- My biggest concern: since the old WS is well, oldish - What is my Backup and Recover plan if the Motherboard or the PSU or whatever fails?
How will i be able to access proxmox or the VM's provided that the HDD's/pools are still functioning/Online? - all eggs in one basket...

- Would a 16 TB external HDD as sort of Tape / Archive Backup solve all my concerns - with an all 3 months schedule?
- Do i need to throw away the raptors and simply make 3 partitions and install everything on a new ssd for the hypervisor/VM' OS?



Thank you if you did read my rumblings and if you will take your time to answer!!

 

[Dunuin]

-we do value our data - who doesn't?

If you do, you should do 3-2-1 backups with offsite backups. I would guess you are quite screwed when losing all raw footage as a content creator.
Its not just fire or theft...
-failing PSU electrocuting all hardware
-power outage without UPS
-lightning strike
-user error
-bad RAM
-melting cables
-software bugs
-ransomware
...So much that can kill entire disks/arrays/pools.

If you don't want to pay for a cloud storage, at least put a NAS for backups at one of your homes and connect it through VPN.

To backup the data of a TrueNAS I like to replicate it to another TrueNAS server (build in replication job of TrueNAS).

- It seems to me that i do have some redundancy (hypervisor is in mirror mode installed, Backup server in Raidz-2, File server in mirror mode, each OS on seperate HDD), but is this a false sense of security?

False sense of security. Lets say one of your RAM modules slowly fails and you only realize it after weeks when it get really bad with increasing amount of system crashes. Eveny bit of data touched by the server over the last few weeks then might be partially corrupted.

Had that 2 years ago corrupting hundreds of GBs of data on my TrueNAS server. I then had to delete every file of the TrueNAS the server that changed the last month and restore an 1 month old backup of these files from another server.

If you store everything on the same server with different VMs it would of cause corrupt the data of all those VMs and all disks. And you are screwed if you don't store revisions of your backups for months.

 

[Dandan]

Thanks Dunuin for the quick replay ,

As i said, no cloud. But I agree that a I could/should still use the old WD MyMirror as a second location for the Backup's
still onsite (no need to hassle with a VPN), because your list of
-failing PSU electrocuting all hardware
-power outage without UPS
-user error
-bad RAM
-melting cables
-software bugs
-user error
-ransomware
(as i see it) are all contained to the old WS / Backup Server / Box, and I am willing to take the risk of a lighting strike so powerful that not only it
struck us, but was even so powerful to destroy the old WS....nahh, my degree of balance between spending money and risking the data ends here

you say :

To backup the data of a TrueNAS I like to replicate it to another TrueNAS server (build in replication job of TrueNAS).

Then my setup with 2 VM TrueNAS server is reasonable using a replicate function in TrueNAS, Thanks , i will look into it!

you say:

Had that 2 years ago corrupting hundreds of GBs of data on my TrueNAS server. I then had to delete every file of the TrueNAS the server
that changed the last month and restore an 1 month old backup of these files from another server.

THIs is really interesting...so zfs is not as miraculous as i hoped - the "Self-healing RAID" , as defined by Aaron Toponce,
(see link provided above and see also https://openzfs.github.io/openzfs-docs/Getting Started/index.html - where he is mentioned as "excellent documentation")
can't combat the corruption of data - that's a shame
ZFS Documentation also says that ECC memory is really recommended- Did you use such memory?

Thanks for your inputs and thoughts!

 

[Dunuin]

Then my setup with 2 VM TrueNAS server is reasonable using a replicate function in TrueNAS, Thanks , i will look into it!

To another TrueNAs server, not TrueNAS VM on the same server

are all contained to the old WS / Backup Server / Box, and I am willing to take the risk of a lighting strike so powerful that not only it
struck us, but was even so powerful to destroy the old WS....nahh, my degree of balance between spending money and risking the data ends here

It doesn't need to hit your building. Would be enough to hit a powerline, lamp post or something similar nearby. And even if you got a surge protector, it probably can't handle 300 million volts.

THIs is really interesting...so zfs is not as miraculous as i hoped - the "Self-healing RAID" , as defined by Aaron Toponce,

It can only heal data that corrupted while on the disk. It won't help when data corrupts in the CPU or in RAM before writing it to the disks or after reading it from the disks.

ZFS Documentation also says that ECC memory is really recommended- Did you use such memory?

Yes but ECC RAM can only fix flipped bits to a small degree. If it corrupts too much (for example when entire RAM chips starts to fail) it can't correct it. And also keep in mind that EVERY device that is accessing the NFS/SMB shares needs to use ECC RAM. Lets say I got a consumer laptop without ECC RAM and want to edit a video that is stored on the NAS. The video is uncorrupted while on the NAS. I access it via SMB so it gets transfered to my laptop. While opening it it is stored in the laptops RAM and there it corrupts while I edit it and I don't notice this. Then I save my changes and these changes will be send to the NAS to be stored there overwriting the old file. Now that video on the NAS is corrupted and even a ZFS scrub will tell me that the file is healthy (no checksum errors will be detected) as ZFS can only tell me if the file is still the same as it was when I wrote it to the pool. So the bit rot protection won't help if the data was already corrupted when writing it to the NAS.

 

[Dandan]

And also keep in mind that EVERY device that is accessing the NFS/SMB shares needs to use ECC RAM.

I see from where you are coming from - protecting data from lightning bolts (FYI here we have lightning rods and trees,
or the need for all of our HW having ECC Ram, having 2 or maybe even better 3 servers, is all good and nice, but oops there comes reality,
and then one sees that we have to pay the bills - that's the challenge as per title

I hope that i can still get some more feedback about my setup with ONe Server and no Cloud - someone that can see from where i am coming from...
this is a proxmox forum - freeware. I wasn't looking for a Oracle or VMware Hypervisor - don't want to spend the money on that.

edit: added this link to show from where i am coming from. forget the cad part.
https://blog.grabcad.com/blog/2015/08/13/why-ecc-ram-matters/ - quoting the link

that statistically will only affect 8% of your hardware, and only if the hardware undergoes a server-like utilization.

 

[Dunuin]

I see from where you are coming from - protecting data from lightning bolts (FYI here we have lightning rods and trees,
or the need for all of our HW having ECC Ram, having 2 or maybe even better 3 servers, is all good and nice, but oops there comes reality,
and then one sees that we have to pay the bills - that's the challenge as per title

Your data, your choice. Just keep in mind that a lot of people here then got more secure private homeservers and backups for unimportant stuff like plex and smart home than you got for business data on a production system. Not that you later regret it after losing your data...especially already knowing why you should use the 3-2-1 rule...
Good backups aren't cheap, but still way cheaper than losing years of work. But you probably never lost important data yet...once you do (what you probably will do sooner or later) you won't take backups and redundancy that easy anymore.
Similar to insurances. Annoying to pay for a third party insurance without making use of it for many years but you are really screwed if you overrun someone and then have to pay him for decades.

 

[Dandan]

you should use the 3-2-1 rule...

In my setup i have 1 copy on the main production WS/Laptops, 1 copy on the VM Backupserver and as suggested by Dunuin i should add one NAS offsite connected with VPN for a total of 3. I still not sure if i will put the NAS offsite or not , but i will consider it. Same with using ECC Memory.
Thanks for the heads up!
But what about the other questions?

- Separate OS from data --> usefull or not? , if yes , is my implementation correct? what could i do better? did i overshoot / overthink?
- Should i use only one truenas VM with two zfs pools, instead of two VMs with one pool each?
- It seems to me that i do have some redundancy (hypervisor is in mirror mode installed, Backup server in Raidz-2, File server in mirror mode, each OS on seperate HDD),
but is this a false sense of security?
- Do you see some evident perfomance issues? L2ARC, Blocksize, 1Gbe NIC, Other? Would a 10GbE NIC help? Would the HDD be too slow for the 10GbE NIC?
- How would you repurpose the old WS for this case?
- What is the _biggest_ problem of my setup?

- My biggest concern: since the old WS is well, oldish - What is my Backup and Recover plan if the Motherboard or the PSU or whatever fails?
How will i be able to access proxmox or the VM's provided that the HDD's/pools are still functioning/Online? - all eggs in one basket...

- Would a 16 TB external HDD as sort of Tape / Archive Backup solve all my concerns - with an all 3 months schedule?
- Do i need to throw away the raptors and simply make 3 partitions and install everything on a new ssd for the hypervisor/VM' OS?

Looking forward to your replays - thanks