Filesystem, storage configuration request

[ieronymous]

Hi
Once more I need a little guidance since I have read so many wiki,guides,propositions and I am still at zero in the matter of proper installation according to my needs

Still I am unable to decide which file system would be suitable for OS. Since I ll need your advice on that it would be wise to tell you my configuration. System is comprised of
2x intel xeon (12 cores, 24 threads) with 32gb for sure maybe 64 though if the budget allows it. 2x1gb nic onboard. Perc H310 controller is out, after reading the worst about it as performanceand lack of ssd trim support.
For storage i intent to use
-2x128gb (consumer ssd) for OS mirrored (but what is the point in using zfs when the vm's, containers,backups,snapshots will be stored in another storage media. Do you think zfs would help somehow?)
-2x480gb (I ll try for 960, consumer ssd) for Vm's and Containers mirrored
-2x480gb (possible again for 960gb) for backup and snapshots mirrored

The Os will be Win Serv2012 or 2016 with the role of DHCP, DNS, AD, SQL base 2014 (if I can recall) and 2-3 network printers (all these are already running on an ancient dual core x3130 xeon on a dell t300 or something - I m on vacation and forgot to take the notes with me).
Since a new ERP is going to be installed and I asked for a new SQL instance to work properly and a small win10 pc running 1-2 special services. Do you think that I should break the win serv in 2 or 3 vm's for all the workload. O fcourse then it would be the extra cost of licenses (By the way in VM the licenses types are the same as for the bare metal server?) Best possible solution is to use Linux OS for DHCP,DNS,SQL but I haven't done that in the past and that is beyond the scope of this post.

Since all the storage is going to be ssd what do you think about the filesystem in each of the above mirrored storage devices? I ve heard about trim isnt supported yet there is a read through something like that on zfs that makes trim not needed (still unsure about it ). Also even if Vm's are on ssd's do I really need ssd since the Vm's are used for services only and not being used directly? Same goes for backup purposes. Will ther be some gains using ssd's? It is going to be used a sas onboard intel controller(not in raid mode. I think there is an option that disabled raid yet I dont know if that makes the sas controller work in passthrough mode)

Thank you and I hope for answers

 

[wolfgang]

Hi,

Generall if you have a Perc H310 what is an HW raid controller you should go with Thin-LVM.
ZFS is not compatible with HWRaid.

 

[sb-jw]

Perc H310 controller is out, after reading the worst about it as performanceand lack of ssd trim support.

You can flash the H310 to LSI IT Firmware, after this you get a queue Depth of 600 instead of 25.
Im using it in 4x R710 with 4 OSDs per Node and there are not issues with CEPH or something other.

-2x128gb (consumer ssd) for OS mirrored (but what is the point in using zfs when the vm's, containers,backups,snapshots will be stored in another storage media. Do you think zfs would help somehow?)

I using ZFS too, it's easier to maintain after an failure and it is supported in PVE GUI.

-2x480gb (I ll try for 960, consumer ssd) for Vm's and Containers mirrored
-2x480gb (possible again for 960gb) for backup and snapshots mirrored

Really, do not store backups on the same server or Raid Controller. If you have one failure, all of your data will be lost. Buy a cheap NAS and use NFS to store backups.
I would not recommend to use Consumer Disks, the Samsung PM883 are very cheap at the moment.

(By the way in VM the licenses types are the same as for the bare metal server?)

Afaik no, it is not the same. You have to license all your Nodes in the cluster, if you want use the other nodes for windows VMs or not. You have to license each socket, so I would prefer to use the Datacenter License. Otherwise get in contact with a official License dealer.

 

[ieronymous]

Hi,

Generall if you have a Perc H310 what is an HW raid controller you should go with Thin-LVM.
ZFS is not compatible with HWRaid.

First of all thank you for taking the time answering me .

HW-Raid is the least concerns me right now first because I know about ZFS limitation-not working well in general and second because raid cards have life cycle themselves and are getting too hot (If I was to use though a hw-raid card it would be something with bbu (battery unit backup). So I wouldn't consider it as a solution. As for Lvm its what I have now in my test server on a usb stick (I didnt have at that time spare ssd but that is ok since I haven t yet decided the final configuration for the real server)

 

[ieronymous]

You can flash the H310 to LSI IT Firmware, after this you get a queue Depth of 600 instead of 25.
Im using it in 4x R710 with 4 OSDs per Node and there are not issues with CEPH or something other.

I using ZFS too, it's easier to maintain after an failure and it is supported in PVE GUI.

You mean as an OS file system? With what kind of storage, ssd's or hdd's? Just remembered why I wanted to use only ssd's because then there would be no strong reason for using 2 more ssd's for Zil and L2ARC even though using the same ssd for both OS and Zil seems kinds stretchy for the wear level of the ssd. Too many talks and argument about this subject but with no real conclusion except if you have a big budget and you dont care about replacing once in a while disks -not the case here of course.

Really, do not store backups on the same server or Raid Controller. If you have one failure, all of your data will be lost. Buy a cheap NAS and use NFS to store backups.
I would not recommend to use Consumer Disks, the Samsung PM883 are very cheap at the moment.

True but that's why I am going in dual mirrors for each purpose, OS - VM's&Container's - Back Up. As for the SSD's I ll try to convince boss for more enterprise models.

Afaik no, it is not the same. You have to license all your Nodes in the cluster, if you want use the other nodes for windows VMs or not. You have to license each socket, so I would prefer to use the Datacenter License. Otherwise get in contact with a official License dealer.

According to this, Standard license also have the same benefits as the Datacenter.

 

[sb-jw]

You mean as an OS file system?

Yes.

With what kind of storage, ssd's or hdd's?

SSDs, cheap WD Green 120G.

True but that's why I am going in dual mirrors for each purpose, OS - VM's&Container's - Back Up.

It do not change anything
If your Controller is failed in one Node and write only bullshit in the Backup, you will mirror it too. For Backups you need a seperate Systems which running standalone and do not have any real Connections to the productive system. In worst Case only the new and current Backup is corrupted but not the old one too.

 

[ieronymous]

If your Controller is failed in one Node and write only bullshit in the Backup, you will mirror it too. For Backups you need a seperate Systems which running standalone and do not have any real Connections to the productive system. In worst Case only the new and current Backup is corrupted but not the old one too.

Hm yes and no. If it fails it isnt going to write anything . Most probably it will output an error message (I hope ahah) But then again, if not, so you re right at the first point. But the old backup isnt accessed nor by the system if I had the backup hdds inside nor by the nas (I f I used one). when you have an old backup the only thing you do with that is restore. At least if I understood correctly what you mean.

So could you share you system configuration if you wouldnt mind? ... since you are very tight with letters and use them as less as possible

Since you are using ssd's also what method did you follow for the trim (discard process??) Meaning did you edit any files like fstab and added noatime,nodirtime ... did you program any cron scheduled tasks to run fstrim command daily,weekly... and how is that accomplished...
What about ZIL did you make any extra partition to the ssd for ZIL or no use at all
come on share the knowledge I wont eat your bred ...we re too far away for that

 

[sb-jw]

Hm yes and no. If it fails it isnt going to write anything . Most probably it will output an error message (I hope ahah) But then again, if not, so you re right at the first point.

I'm working in a Datacenter since 2015 and in this time i saw two failed Controllers from our Customers. One of the Controller failed not completly, no they write all the Data to the Disk, but corrupt. After a Reboot all the data was unrecoverable destroyed. This was the last action taken by the Customer, he was now broke and the lawyer of his Customer has visit our datacenter, they wanted to see and check the failed Controller.

The other one was a Hypervisor, the Controller failed not complelty but the Data are not complelty corrupt. It seems the Cache has done a good Job here, so we were are able to restore the VM from this Hypervisor, it takes about 3 days to restore the Data with an old stand.

But the old backup isnt accessed nor by the system if I had the backup hdds inside nor by the nas (I f I used one).

If you have the Disk insert in the Server, then you can access the Disk and can completly destroy the Data stored on the Disk. If you use a NAS with a Snapshot Function, it is not possible to completlty destroy the Data on it, if you do not have access to the NAS too. It can be a Hacker or a Crypto Trojaner - dont forget this
Best is you can only push Data to a Storage and can not view it with the same Credentials and you unmount the Backup Storage if you do not use it.

So could you share you system configuration if you wouldnt mind?

There is nothing really special
I have 4 Dell R710 with 2x X5650, 144GB DDR3 RAM, H310 and 4x 1GbE LACP. For the OS is have 2x 120GB SSDs and for CEPH i use 4x 1TB SSDs. My Backup System is a Dell C2100 with an H200 Controller and 9 Hard drives (old used ones). The Backup Server is running with PVE too but is not in the productive Cluster.

Since you are using ssd's also what method did you follow for the trim (discard process??)

I use HBA, so the OS have to do this or the CEPH itself or the VMs.

So thats my setup but its bit different from the Setup you want to use ;-)

 

[alexskysilk]

Hm yes and no. If it fails it isnt going to write anything . Most probably it will output an error message (I hope ahah) But then again, if not, so you re right at the first point. But the old backup isnt accessed nor by the system if I had the backup hdds inside nor by the nas (I f I used one). when you have an old backup the only thing you do with that is restore. At least if I understood correctly what you mean.

There is a website I've been referring to for MANY years when folks ask me why they should keep the backups seperate from the primary data. I will link it now http://www.taobackup.com/

I do have to ask though, if you're a full MS shop why are you trying to reinvent the wheel? If you're ponying up for Windows Server, both standard and DC versions come with virtualization entitlements already which means you could run a minimum of three servers (1 host plus 2 guests) with the license you already acquired.

 

[ieronymous]

@sb-jw this time you ve been very thorough and points about possible failures taken. Too bad for your client,

As for external backup storage, depends if i can convince my boss for extra costs.

Your set up is clearly build upon HA (High Avaliability) as I see it. But how come and you ve chosen CEPH as a filesystem?

I use HBA, so the OS have to do this or the CEPH itself or the VMs.

I think there is an escalation on this. All have to support it else it will fail to pass the trim commands. It begins with the drive itself, then the filesystem and last the OS.

 

[ieronymous]

I do have to ask though, if you're a full MS shop why are you trying to reinvent the wheel? If you're ponying up for Windows Server, both standard and DC versions come with virtualization entitlements already which means you could run a minimum of three servers (1 host plus 2 guests) with the license you already acquired.

Thank you for the link. As about virtualization upon windows server you mean hyper-v?