Filesystem, storage configuration request

Discussion in 'Proxmox VE: Installation and configuration' started by ieronymous, Apr 30, 2019.

  1. ieronymous

    ieronymous New Member

    Joined:
    Apr 1, 2019
    Messages:
    21
    Likes Received:
    0
    Hi
    Once more I need a little guidance since I have read so many wiki,guides,propositions and I am still at zero in the matter of proper installation according to my needs

    Still I am unable to decide which file system would be suitable for OS. Since I ll need your advice on that it would be wise to tell you my configuration. System is comprised of
    2x intel xeon (12 cores, 24 threads) with 32gb for sure maybe 64 though if the budget allows it. 2x1gb nic onboard. Perc H310 controller is out, after reading the worst about it as performanceand lack of ssd trim support.
    For storage i intent to use
    -2x128gb (consumer ssd) for OS mirrored (but what is the point in using zfs when the vm's, containers,backups,snapshots will be stored in another storage media. Do you think zfs would help somehow?)
    -2x480gb (I ll try for 960, consumer ssd) for Vm's and Containers mirrored
    -2x480gb (possible again for 960gb) for backup and snapshots mirrored

    The Os will be Win Serv2012 or 2016 with the role of DHCP, DNS, AD, SQL base 2014 (if I can recall) and 2-3 network printers (all these are already running on an ancient dual core x3130 xeon on a dell t300 or something - I m on vacation and forgot to take the notes with me).
    Since a new ERP is going to be installed and I asked for a new SQL instance to work properly and a small win10 pc running 1-2 special services. Do you think that I should break the win serv in 2 or 3 vm's for all the workload. O fcourse then it would be the extra cost of licenses (By the way in VM the licenses types are the same as for the bare metal server?) Best possible solution is to use Linux OS for DHCP,DNS,SQL but I haven't done that in the past and that is beyond the scope of this post.

    Since all the storage is going to be ssd what do you think about the filesystem in each of the above mirrored storage devices? I ve heard about trim isnt supported yet there is a read through something like that on zfs that makes trim not needed (still unsure about it ). Also even if Vm's are on ssd's do I really need ssd since the Vm's are used for services only and not being used directly? Same goes for backup purposes. Will ther be some gains using ssd's? It is going to be used a sas onboard intel controller(not in raid mode. I think there is an option that disabled raid yet I dont know if that makes the sas controller work in passthrough mode)

    Thank you and I hope for answers
     
  2. wolfgang

    wolfgang Proxmox Staff Member
    Staff Member

    Joined:
    Oct 1, 2014
    Messages:
    4,695
    Likes Received:
    310
    Hi,

    Generall if you have a Perc H310 what is an HW raid controller you should go with Thin-LVM.
    ZFS is not compatible with HWRaid.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. sb-jw

    sb-jw Active Member

    Joined:
    Jan 23, 2018
    Messages:
    445
    Likes Received:
    37
    You can flash the H310 to LSI IT Firmware, after this you get a queue Depth of 600 instead of 25.
    Im using it in 4x R710 with 4 OSDs per Node and there are not issues with CEPH or something other.

    I using ZFS too, it's easier to maintain after an failure and it is supported in PVE GUI.

    Really, do not store backups on the same server or Raid Controller. If you have one failure, all of your data will be lost. Buy a cheap NAS and use NFS to store backups.
    I would not recommend to use Consumer Disks, the Samsung PM883 are very cheap at the moment.

    Afaik no, it is not the same. You have to license all your Nodes in the cluster, if you want use the other nodes for windows VMs or not. You have to license each socket, so I would prefer to use the Datacenter License. Otherwise get in contact with a official License dealer.
     
  4. ieronymous

    ieronymous New Member

    Joined:
    Apr 1, 2019
    Messages:
    21
    Likes Received:
    0
    First of all thank you for taking the time answering me .

    HW-Raid is the least concerns me right now first because I know about ZFS limitation-not working well in general and second because raid cards have life cycle themselves and are getting too hot (If I was to use though a hw-raid card it would be something with bbu (battery unit backup). So I wouldn't consider it as a solution. As for Lvm its what I have now in my test server on a usb stick (I didnt have at that time spare ssd but that is ok since I haven t yet decided the final configuration for the real server)
     
    #4 ieronymous, Apr 30, 2019
    Last edited: Apr 30, 2019
  5. ieronymous

    ieronymous New Member

    Joined:
    Apr 1, 2019
    Messages:
    21
    Likes Received:
    0

    You mean as an OS file system? With what kind of storage, ssd's or hdd's? Just remembered why I wanted to use only ssd's because then there would be no strong reason for using 2 more ssd's for Zil and L2ARC even though using the same ssd for both OS and Zil seems kinds stretchy for the wear level of the ssd. Too many talks and argument about this subject but with no real conclusion except if you have a big budget and you dont care about replacing once in a while disks -not the case here of course.

    True but that's why I am going in dual mirrors for each purpose, OS - VM's&Container's - Back Up. As for the SSD's I ll try to convince boss for more enterprise models.

    According to this, Standard license also have the same benefits as the Datacenter.
     
  6. sb-jw

    sb-jw Active Member

    Joined:
    Jan 23, 2018
    Messages:
    445
    Likes Received:
    37
    Yes.

    SSDs, cheap WD Green 120G.

    It do not change anything :D
    If your Controller is failed in one Node and write only bullshit in the Backup, you will mirror it too. For Backups you need a seperate Systems which running standalone and do not have any real Connections to the productive system. In worst Case only the new and current Backup is corrupted but not the old one too.
     
  7. ieronymous

    ieronymous New Member

    Joined:
    Apr 1, 2019
    Messages:
    21
    Likes Received:
    0
    Hm yes and no. If it fails it isnt going to write anything . Most probably it will output an error message (I hope ahah) But then again, if not, so you re right at the first point. But the old backup isnt accessed nor by the system if I had the backup hdds inside nor by the nas (I f I used one). when you have an old backup the only thing you do with that is restore. At least if I understood correctly what you mean.

    So could you share you system configuration if you wouldnt mind? ... since you are very tight with letters and use them as less as possible :D

    Since you are using ssd's also what method did you follow for the trim (discard process??) Meaning did you edit any files like fstab and added noatime,nodirtime ... did you program any cron scheduled tasks to run fstrim command daily,weekly... and how is that accomplished...
    What about ZIL did you make any extra partition to the ssd for ZIL or no use at all
    come on share the knowledge I wont eat your bred ...we re too far away for that :D:D:D
     
    #7 ieronymous, May 1, 2019
    Last edited: May 1, 2019
  8. sb-jw

    sb-jw Active Member

    Joined:
    Jan 23, 2018
    Messages:
    445
    Likes Received:
    37
    I'm working in a Datacenter since 2015 and in this time i saw two failed Controllers from our Customers. One of the Controller failed not completly, no they write all the Data to the Disk, but corrupt. After a Reboot all the data was unrecoverable destroyed. This was the last action taken by the Customer, he was now broke and the lawyer of his Customer has visit our datacenter, they wanted to see and check the failed Controller.

    The other one was a Hypervisor, the Controller failed not complelty but the Data are not complelty corrupt. It seems the Cache has done a good Job here, so we were are able to restore the VM from this Hypervisor, it takes about 3 days to restore the Data with an old stand.

    If you have the Disk insert in the Server, then you can access the Disk and can completly destroy the Data stored on the Disk. If you use a NAS with a Snapshot Function, it is not possible to completlty destroy the Data on it, if you do not have access to the NAS too. It can be a Hacker or a Crypto Trojaner - dont forget this ;)
    Best is you can only push Data to a Storage and can not view it with the same Credentials and you unmount the Backup Storage if you do not use it.

    There is nothing really special ;)
    I have 4 Dell R710 with 2x X5650, 144GB DDR3 RAM, H310 and 4x 1GbE LACP. For the OS is have 2x 120GB SSDs and for CEPH i use 4x 1TB SSDs. My Backup System is a Dell C2100 with an H200 Controller and 9 Hard drives (old used ones). The Backup Server is running with PVE too but is not in the productive Cluster.

    I use HBA, so the OS have to do this or the CEPH itself or the VMs.

    So thats my setup but its bit different from the Setup you want to use ;-)
     
  9. alexskysilk

    alexskysilk Active Member

    Joined:
    Oct 16, 2015
    Messages:
    556
    Likes Received:
    58
    There is a website I've been referring to for MANY years when folks ask me why they should keep the backups seperate from the primary data. I will link it now :) http://www.taobackup.com/

    I do have to ask though, if you're a full MS shop why are you trying to reinvent the wheel? If you're ponying up for Windows Server, both standard and DC versions come with virtualization entitlements already which means you could run a minimum of three servers (1 host plus 2 guests) with the license you already acquired.
     
  10. ieronymous

    ieronymous New Member

    Joined:
    Apr 1, 2019
    Messages:
    21
    Likes Received:
    0
    @sb-jw this time you ve been very thorough and points about possible failures taken. Too bad for your client,

    As for external backup storage, depends if i can convince my boss for extra costs.

    Your set up is clearly build upon HA (High Avaliability) as I see it. But how come and you ve chosen CEPH as a filesystem?

    I think there is an escalation on this. All have to support it else it will fail to pass the trim commands. It begins with the drive itself, then the filesystem and last the OS.
     
  11. ieronymous

    ieronymous New Member

    Joined:
    Apr 1, 2019
    Messages:
    21
    Likes Received:
    0
    Thank you for the link. As about virtualization upon windows server you mean hyper-v?
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice