Need storage advice

[Stefan Pettersson]

So, I got a hold on a supermicro 4U chassis ( migrating to new hardware here at the firm ) with 16 2TB SAS 7200rpm disks and 2-3 120gb Intel 510 SSD MLC. Last two years it has been running with very low load due to migration away from it.

The processors and mb are a bit old but I will be moving my current hardware that is newer, motherboard, memory and procs to this box. The hardware in the new 4U chassi will be two E5-2643v2 and 64 gigs of RAM. In the box is also a LSI 2008 raid card and as I can understand it can do JBOD.

So, questions below is what I need answers on

So i was tinking of going with ZFS instead of hardware raid, whitch im runing right now.

Should i go with SSD cache with the two intel disks or not?

I will be running my own media library on the server, just like I am doing today. Plex is running as a container (ubuntu-server) and of course needs access to a large storage space. That large storage also need to be accessed by windows clients, so that I can update the library and add new files.
Should I setup a container and share everything with samba? Containers cant do nfs as I recall??

So, is it possible to setup a container that shares the "media-pool".
Or should I just setup a samba share on the plex server, install the plex on a small disk and then add a mount point on 10 TB??

Thats it fo now, more questions will shure come... But it's fun

 

[alexskysilk]

So i was tinking of going with ZFS instead of hardware raid, whitch im runing right now.

good idea.

Should i go with SSD cache with the two intel disks or not?

you can, but I dont know how useful it would be. There are a number of threads discussing this very subject at length, just search the forum. my suggestion follows in the answer to your next question.

I will be running my own media library on the server, just like I am doing today. Plex is running as a container (ubuntu-server) and of course needs access to a large storage space. That large storage also need to be accessed by windows clients, so that I can update the library and add new files.
Should I setup a container and share everything with samba? Containers cant do nfs as I recall??

for your setup, my suggestion is to create 2 pools:
1- your SSDs in a RAID1. Proxmox sits here, plus plex.
2- your HDDs in a RAIDZ2. your shared file system sits here, shared by proxmox.

In truth, based on what you're asking for, freenas/Nas4Free is probably the better way to go.

 

[Stefan Pettersson]

In truth, based on what you're asking for, freenas/Nas4Free is probably the better way to go.[/QUOTE]


Thanks for your answer! Maybee should have mentioned that im also running a nextcloud server, two centos lab servers, a w2k16 Domain controller and a debian desktop so Proxmox is the way to go Will do some experimenting today if I have the time.

Another question, in theory I should be able to install a Linux dist on the old hardware in the 4U box, create the zfs storage tank, copy the media files, change the hardware in the 4U box to the newer hardware, install proxmox and do a zfs scan and find the zfspool?? Doable??

 

[guletz]

So, I got a hold on a supermicro 4U chassis ( migrating to new hardware here at the firm ) with 16 2TB SAS 7200rpm disks and 2-3 120gb Intel 510 SSD MLC. Last two years it has been running with very low load due to migration away from it.

Hi,

ZFS part:

- because you have many (16) HDD, I would use 2 zfs pool raidz2(2 pool x 8 HDD), or if you want to be on the safe side, use 2 pool x 7 HDD, with 1 HDD as zfs spare disk(so in case of 1 HDD failure/pool you will have a auto-resilver)
- yes, is nice to have 1 giant pool(x16 HDD), but in case of one HDD failure, the resilver process will take several days, and because of this stress in many cases, you can loose another disk = game over(for raidz2)
- one of these 2 pool could be dedicated for media library, and in this case if you have the majority of files as BIG files, you can use 1 MiB record insted of default 128 KiB.

 

[mir]

[..] and because of this stress in many cases, you can loose another disk = game over(for raidz2)

raidz2 can survive loosing 2 disks.

 

[guletz]

raidz2 can survive loosing 2 disks.

... yes, it can survive, but not for long time. One single bad block and you are out! And you need a lot of luck to rebuild your 12 HDD pool(without any parity disk).

 

[mir]

... yes, it can survive, but not for long time. One single bad block and you are out! And you need a lot of luck to rebuild your 12 HDD pool(without any parity disk).

Please stick to the facts. raidz2 => you can loose 2 disks. Everything else is pure speculation and only serves the purpose of spreading FUD.

 

[guletz]

Everything else is pure speculation and only serves the purpose of spreading FUD.

Ok, let see if this is a speculation. Now let say(imagine for a moment) we have a raidZ2 pool, without any parity disk. OK, we can access any data in the pool. One little thing is the FACT that any HDD manufacturer said in their data-sheet that they have AT LEAST 4 unrecoverable blocks / approx. 6-7 TB data read(but many people say that is bigger then this figure). So let suppose that at 6 TB data failure read probability is 1. For a full 2 TB read for make a full resilver, the same probability is about 33 %(2*1/6). It is even worse, if you use bigger HDD.

Maybe you can find this link usefull

https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf

If you consider that this probability is a pure speculation, yes you are wright

 

[Stefan Pettersson]

Thanks for all the answers, nice to learn

After some googling I found out that the 4U chassi is not compatible with my motherboard from supermicro... It's an enhanced E-ATX... So I think I will stay with the hardware and install some NAS solution instead and share it to my Proxmox node, lika a SAN.

Now to decied what software to use, but thats for another forum Thanks again!

 

[guletz]

So I think I will stay with the hardware and install some NAS solution instead and share it to my Proxmox node, lika a SAN.

It is very common in enterprise landscape to have separate storage.

 

[Stefan Pettersson]

It is very common in enterprise landscape to have separate storage.

YesbI know, this is an Enterprise enviroment bit the projekt is private, the employes here att the company have a benefit, we have our own rack so all hardware that gets thrown away, we salvage

 

[alexskysilk]

Another question, in theory I should be able to install a Linux dist on the old hardware in the 4U box, create the zfs storage tank, copy the media files, change the hardware in the 4U box to the newer hardware, install proxmox and do a zfs scan and find the zfspool?? Doable??

Yes.

we have a raidZ2 pool, without any parity disk.

This is true for any RAIDset. the reason to create dual parity volumes is not to "loose 2 disks" but that you should never be degraded past a single disk- when a disk fails you replace it. The odds of two disks failing at the same time are astronomical, and if those odds are outside your acceptable risk tolerance you need a true storage cluster with a proper support contract.

- because you have many (16) HDD, I would use 2 zfs pool raidz2(2 pool x 8 HDD), or if you want to be on the safe side, use 2 pool x 7 HDD, with 1 HDD as zfs spare disk(so in case of 1 HDD failure/pool you will have a auto-resilver)

The nice thing about ZFS is its flexibility. determine your required usable space and risk tolerance and ZFS provides the means to get there. a pool with 2 RAIDZ2 vdevs? sure! a pool with 8 mirror vdevs? sure! any of these has its pros and cons. define the application and risk tolerance before designing your storage solution.

One little thing is the FACT that any HDD manufacturer said in their data-sheet that they have AT LEAST 4 unrecoverable blocks / approx. 6-7 TB data read

The reason ZFS RAID is superior to hardware RAID is precisely because IT IS IMMUNE to UREs; every read is checksummed and seamlessly corrected. never mind bad blocks, you can literally write garbage data to a physical disk in a ZFS parity volume and the the pool wouldn't miss a beat. your point, however, is well taken: should you allow a full file system on 16 2TB drives (~28TB) to become fully degraded, you'd need to read the whole 2TB from each of the survivors to rebuild; consumer (current) drives are rated a 1/10^14 or less, which makes it 1/12.5TB of data read or so- or ~1 in 6 chance you'll have a URE during rebuild PER MEMBER DISK. In a system with 14 disks thats virtually assured.

The lesson here is simple- if you value your data dont run the file system degraded.

 

[MajorD]

This is my understanding on ZFS:

Remember that with ZFS, a read/write error (bad sector with possibly unrecoverable error) is not the same thing as losing an entire disk!

ZFS will automatically fix bad read/write. The system will not run degraded.

If you have a bad disk (then you have a lot of bad sectors/unrecoverable errors), ZFS will mark the entire disk as bad and start using a spare it there is one. Until the resilvering process is done, the system will be degraded.

It is best not to wait until you have more than one bad disk and configure your pools so that resilvering does not take too long.

 

[alexskysilk]

ZFS will automatically fix bad read/write.

... if it can. "fixing" in this context requires sufficient data and parity blocks in excess of just the data. if we use RAIDZ2 as an example, a pool with 1 disk out is not fully degraded and still contains enough data and parity blocks to verify each read; 2 disks out would be insufficient to verify reads but there is still sufficient data to operate.

The system will not run degraded

sure it will. you SHOULDNT run a fully degraded pool, but thats up to the admin.

 

[MajorD]

I meant the system will not be degraded if it just has to fix bad reads/writes.