[SOLVED] BIG DATA ZFS (~200TB)

[Baader-IT]

We are only using it as filesystem to have the replication features. In the event that the controller has a hardware defect, we will get a new one from the manufacture and rebuild the ZFS Pool.

 

[morph027]

Yes, but in this case, the filesystem can get corrupted, because it does not have full controll and access to the disks, so it can't detect (and prevent) bit rot. And scrubbing on raid controller level might interfer with ZFS.

Ran into this once, we already had a spare controller ordered, but it had another firmware revision and was unable to read the data. Even the LSI support failed to help us. Since then, i only rely on ZFS.

Also, RAM is much cheaber than a decent hardware raid controller (with flash cache).

 

[Knuuut]

We know that the hardware RAID controller shouldn't be used but on all our systems we count on it.

Again, this IS a bad idea. Count on ZFS!
ZFS with HBA or RAID with anything else, make your descision.
You want to archive a lot of files over a long time, so BitRot IS an issue.
Do it, but do it right.

Cheers Knuuut

 

[Baader-IT]

Again, this IS a bad idea. Count on ZFS!
ZFS with HBA or RAID with anything else, make your descision.
You want to archive a lot of files over a long time, so BitRot IS an issue.
Do it, but do it right.

Cheers Knuuut

Please read this completely (also the comments) and forget the first enumeration from the second part of this article.
https://mangolassi.it/topic/12047/zfs-is-perfectly-safe-on-hardware-raid

This is exactly what I want to tell you.

We use ZFS as (any other) filesystem.
We just want to use the replication from proxmox.
We don't need any other feature from ZFS.

Greetings

 

[Baader-IT]

Hi,

das kann man nicht pauschal beantworten und liegt immer am HW Raidcontoller und dessen Implementierung(Firmware).
Im schlimmsten Fall verlierst du Daten.
Wie gesagt muss aber nicht sein.

P.S. man kann einige Dell Controller auch umflashen auf HBA mode.

https://forum.proxmox.com/threads/zfs-mit-hardware-raid.35671/
FYI - if anyone can speak german. (Google translator is an alternative for this problem :-D)

 

[Knuuut]

Please read this completely (also the comments) and forget the first enumeration from the second part of this article.
https://mangolassi.it/topic/12047/zfs-is-perfectly-safe-on-hardware-raid

I know this guy (not personally) and I don't agree him, not eyerything, but mostly. I don't want to start a new discussion on this and also I don't want to reinvent the wheel... ;-)
I can only tell you what is best practice and do some recommendations and I think you're old enough to be responsibel for your work. Feel free to do so or not to do so. But if you're asking here for advice and you don't like the answers you get, why do you ask?
I'm done.

Cheeers Knut

 

[guletz]

Please read this completely (also the comments) and forget the first enumeration from the second part of this article.
https://mangolassi.it/topic/12047/zfs-is-perfectly-safe-on-hardware-raid

Hi,

I read carefully this link without comments.
The author tell many things for his ideea support, but his argumentation is based on logical deduction(most part) but with almost no tehnical arguments. He mixes some true facts with many false facts. Based on this mix he make some statements and the readers must think that this is true.

In that post/post is a statement that I want to comment. He said that a mirror (software raid or hardware raid) is very safe. But is not in both situation (excluding zfs). Basicaly if the OS will try to write some data, any hw/software mirror system will wrote the same data on both disks who are in that mirror. And this is all. Now after some time we try to read the same data. Most of the time will be no problem. But somtime is possible that what we read is not the same with what we wrote. Take a look at ANY hdd manufacturer and you will see that they tell you that you can have read errors.
Also think that any raid controller have a firmware/software. It is complete unwise to think that this firmware/software is absolute free of errors. And this controller also have some electronics parts (RAM, chipset, battery, and so on), and we know very well that somtimes this electronics have the bad custom to malfunction.
For this resons I think that using a hardware raid is only a lottery. Most of the time you will win. But for sure you can also lose. And I do not say this because I read about this. No, I have lost my data in several occasions, but I was lucky(I have make some md5sum on some of the data, so I was able to know that my data was broken, even I was able to read it).
After this experience on my own skin(3 bad events), I start to use zfs (8 years ago).

So in the end you must think for yourself what you want: to play your data at lottery or not. @Knuuut also have show some other true arguments (for this he get a like from me).

I could easily tell you many many other arguments but this post is also too long.

Good luck with any decision that is ok for you.

 

[Baader-IT]

@Knuuut
Because I just wanted to know following

Should we place this vm on only one 80 TB disk or better use 4 disks with each 20 TB size.
Now I just want to know if there exists a maximum size of each virtual disk.
Are there any known performance issues?

 

[wolfgang]

Proxmox does not support ZFS on HW Raid cards.
So, if you have any problems you are on your own.

See
https://pve.proxmox.com/wiki/ZFS_on_Linux

 

[wolfgang]

Should we place this vm on only one 80 TB disk or better use 4 disks with each 20 TB size.
Now I just want to know if there exists a maximum size of each virtual disk.
Are there any known performance issues?

To be serious we have no experience with such large disks.
The largest running disk I know is about 16TB in a hight io environment.
But generally, Qemu is fine with this size.