Skip to red text to skip all preamble and get to the issues I'm asking about.
So almost a year ago I installed Proxmox on my server to host some personal services like Plex and a Network shares. I had some issues with the installation of Proxmox since at the time I couldn't get the latest version (6.1) installed since it wouldn't boot, getting stuck on an error along the lines of "please insert DR-ROM". I installed version 5.4 and updated to the latest version. Recently I've had an issue with one of my SSDs and will most likely have to replace it and reinstall my VMs from backups (thankly I have the VM backups).
A bit about my setup and some of my questions:
My server is a Huawei RH1288 V3. The most notable thing about the platform is the fact that it's a 1U unit that has 8 2.5" bays on the front.
2x Intel Xeon E5-2620 v3 Obviously nothing lacking or requiring reinstalling here.
32GB ECC DDR4 Am not currently running out with my usage, and even if I was adding more would be no issue!
HERE
2x Kingston A400 240GB SSDsIN HW RAID 0.
I know I know RAID 0 is risky and I've now learned first hand the risk of it. I wouldn't be making this post had I used my drives in RAID 1 since my server would presumably be running fine still with only one of these two drives working.
I decided on RAID 0 because SSDs don't really fail that often (at least I thought so) and I wanted maximal speed for my VMs root storage, which I'm using this SSD array for.
As I've eluded to one of these drives seems to be failing. Suddenly It just errored out. The first time around it resolved itself after a reboot and I thought the disk might've just come loose because of vibration (even though I'm using the proper drive caddies for the server), but when this happened again, even after placing some paper in between the caddy and the frame to make sure vibration isn't setting it loose, I knew there's an issue with the drive. I've done the reboot thing maybe three times now and everytime it's started workingand the crapped on me a bit later. After first reboot it worked for maybe two weeks, but each reboot fix it's failed earlier and earlier (down to 24hrs last time), which to me seems like a pretty definitive sign and the drive is actually failing.
The management interface for the server doesn't tell me anything apart from "Drive failure". The RAID controller interface might be able to tell me something about the S.M.A.R.T status of the drive, but I don't remember if the LSI BIOS had an option for that and the last time I was around the server I didn't have a monitor and keyboard to check the BIOS out with.
2x Seagate BarraCuda 5 TB 2.5 Inch in ZFS RAID 1.
Got these a bit before corona raised prices for everything. I know a lot of people don't like Seagate and Barracudas aren't even recommended for NAS/RAID use, but they were quite affordable and high capacity so bought them for mass storage for my server.
If I recall correctly I decided to do ZFS RAID 1 partly because why not and also I think the RAID controller (specified below) cannot do more than one array at a time, which was already in use for the SSD RAID 0 array mentioned above.
My strategy for 'sharing' these drives across the VMs was to have a FreeNAS VM with the whole RAID1 Array attached to it as a virtual disk and then sharing the disk as a network share from FreeNAS. Though this didn't achieve all that good speeds it did work and was fast enough for my Plex media server instance for example. If I recall correctly (again) the speeds when downloading from the shares was somewhere around 40-50MB/s, even though at least according to the product page for the drives they should be capable of 140MB/s. So I'd expect at least 100MB/S sustained throughput in such a sequential read operation as reading big files from a network drive.
I'm wondering what's the best way to 'share' the storage across the different VMs? Is there a way for me to provision the array as virtual disks for all the VMs? I'd guess doing that would probably get rid of some overhead compared to the storage being attached virtually to a FreeNAS VM which is then sharing it through the network?
As a quick sidenote: For my VMs and containers, how can I make sure the virtual network interfaces and the bridge is 10Gbe to make sure the VM <-> VM communications are as fast as possible?
LSI SAS3008IR (RAID 0,1, 1E & 10 only; No battery backup)
This is the proper RAID version ("IR") of said controller. I found some online resources hinting that it could probably be flashed to an HBA only BIOS which I read would be better for ZFS as that would allow ZFS to control the cache on the drives. I don't think I want to try out flashing the RAID controller.
I figure I could just uninstall the RAID card and plug the SAS cable directly into the motherboard, which I presume has an integrated HBA only interface, which I (once again) presume would be my best option if I wanted to only use ZFS RAID in my server(?). Which brings me to the question I'm pondering the most:
What would be the best performing storage configuration for my server to be used with PVE?:
Even though the internet told me that with my RAID card (not HBA only) ZFS wouldn't work properly (because it wouldn't have access to the drives caches), I did get the 2 x 5TB drives ZFS RAID 1 pool working fine without any errors, even though the drives are attached to the LSI SAS3008IR which has a RAID firmware not an HBA firmware. Have I interpreted correctly that ZFS won't work properly/fully if the interface card is a proper RAID card or have I misinterpreted a source saying that drives in a RAID or RAID arrays cannot be used with ZFS??
Would it make sense for me to just dump the SAS3008IR on Ebay and use the (presumably) integrated HBA card to attach the drives, then make a 2x 240GB SSD ZFS RAID1 array and a 2x 5TB HDD ZFS RAID1 pool? Could this be faster or slower than the LSI RAID controller I'm using currently?
So the resulting storage config would become:
2x Kingston A400 240GB SSDsIN ZFS RAID 1.
2x Seagate BarraCuda 5 TB 2.5 Inch in ZFS RAID 1.
NO LSI SAS3008IR
PS. If you're wondering, I'm booting PVE from a USB stick currently. But doing research about the server for this post I found out it has two SD card slots (presumably for boot drives), so if I'm going to reinstall the server I think I'll swap out to using SD cards for boot drives!
So almost a year ago I installed Proxmox on my server to host some personal services like Plex and a Network shares. I had some issues with the installation of Proxmox since at the time I couldn't get the latest version (6.1) installed since it wouldn't boot, getting stuck on an error along the lines of "please insert DR-ROM". I installed version 5.4 and updated to the latest version. Recently I've had an issue with one of my SSDs and will most likely have to replace it and reinstall my VMs from backups (thankly I have the VM backups).
A bit about my setup and some of my questions:
My server is a Huawei RH1288 V3. The most notable thing about the platform is the fact that it's a 1U unit that has 8 2.5" bays on the front.
2x Intel Xeon E5-2620 v3 Obviously nothing lacking or requiring reinstalling here.
32GB ECC DDR4 Am not currently running out with my usage, and even if I was adding more would be no issue!
HERE
2x Kingston A400 240GB SSDsIN HW RAID 0.
I know I know RAID 0 is risky and I've now learned first hand the risk of it. I wouldn't be making this post had I used my drives in RAID 1 since my server would presumably be running fine still with only one of these two drives working.
I decided on RAID 0 because SSDs don't really fail that often (at least I thought so) and I wanted maximal speed for my VMs root storage, which I'm using this SSD array for.
As I've eluded to one of these drives seems to be failing. Suddenly It just errored out. The first time around it resolved itself after a reboot and I thought the disk might've just come loose because of vibration (even though I'm using the proper drive caddies for the server), but when this happened again, even after placing some paper in between the caddy and the frame to make sure vibration isn't setting it loose, I knew there's an issue with the drive. I've done the reboot thing maybe three times now and everytime it's started workingand the crapped on me a bit later. After first reboot it worked for maybe two weeks, but each reboot fix it's failed earlier and earlier (down to 24hrs last time), which to me seems like a pretty definitive sign and the drive is actually failing.
The management interface for the server doesn't tell me anything apart from "Drive failure". The RAID controller interface might be able to tell me something about the S.M.A.R.T status of the drive, but I don't remember if the LSI BIOS had an option for that and the last time I was around the server I didn't have a monitor and keyboard to check the BIOS out with.
2x Seagate BarraCuda 5 TB 2.5 Inch in ZFS RAID 1.
Got these a bit before corona raised prices for everything. I know a lot of people don't like Seagate and Barracudas aren't even recommended for NAS/RAID use, but they were quite affordable and high capacity so bought them for mass storage for my server.
If I recall correctly I decided to do ZFS RAID 1 partly because why not and also I think the RAID controller (specified below) cannot do more than one array at a time, which was already in use for the SSD RAID 0 array mentioned above.
My strategy for 'sharing' these drives across the VMs was to have a FreeNAS VM with the whole RAID1 Array attached to it as a virtual disk and then sharing the disk as a network share from FreeNAS. Though this didn't achieve all that good speeds it did work and was fast enough for my Plex media server instance for example. If I recall correctly (again) the speeds when downloading from the shares was somewhere around 40-50MB/s, even though at least according to the product page for the drives they should be capable of 140MB/s. So I'd expect at least 100MB/S sustained throughput in such a sequential read operation as reading big files from a network drive.
I'm wondering what's the best way to 'share' the storage across the different VMs? Is there a way for me to provision the array as virtual disks for all the VMs? I'd guess doing that would probably get rid of some overhead compared to the storage being attached virtually to a FreeNAS VM which is then sharing it through the network?
As a quick sidenote: For my VMs and containers, how can I make sure the virtual network interfaces and the bridge is 10Gbe to make sure the VM <-> VM communications are as fast as possible?
LSI SAS3008IR (RAID 0,1, 1E & 10 only; No battery backup)
This is the proper RAID version ("IR") of said controller. I found some online resources hinting that it could probably be flashed to an HBA only BIOS which I read would be better for ZFS as that would allow ZFS to control the cache on the drives. I don't think I want to try out flashing the RAID controller.
I figure I could just uninstall the RAID card and plug the SAS cable directly into the motherboard, which I presume has an integrated HBA only interface, which I (once again) presume would be my best option if I wanted to only use ZFS RAID in my server(?). Which brings me to the question I'm pondering the most:
What would be the best performing storage configuration for my server to be used with PVE?:
Even though the internet told me that with my RAID card (not HBA only) ZFS wouldn't work properly (because it wouldn't have access to the drives caches), I did get the 2 x 5TB drives ZFS RAID 1 pool working fine without any errors, even though the drives are attached to the LSI SAS3008IR which has a RAID firmware not an HBA firmware. Have I interpreted correctly that ZFS won't work properly/fully if the interface card is a proper RAID card or have I misinterpreted a source saying that drives in a RAID or RAID arrays cannot be used with ZFS??
Would it make sense for me to just dump the SAS3008IR on Ebay and use the (presumably) integrated HBA card to attach the drives, then make a 2x 240GB SSD ZFS RAID1 array and a 2x 5TB HDD ZFS RAID1 pool? Could this be faster or slower than the LSI RAID controller I'm using currently?
So the resulting storage config would become:
2x Kingston A400 240GB SSDsIN ZFS RAID 1.
2x Seagate BarraCuda 5 TB 2.5 Inch in ZFS RAID 1.
NO LSI SAS3008IR
PS. If you're wondering, I'm booting PVE from a USB stick currently. But doing research about the server for this post I found out it has two SD card slots (presumably for boot drives), so if I'm going to reinstall the server I think I'll swap out to using SD cards for boot drives!
