We are going to purchase 4 x Gigabyte R281-3C1 servers with CRA3338 array controller to create a Proxmox CEPH cluster. Has there been any issue while using this server or this array controller? Users who have used this server to host Proxmox VE, please let us know your experience with this server model or the array controller. Also mention your config if possible.
Proxmox VE on Gigabyte R281-3C1 with CRA3338 array controller
- Thread starter jsengupta
- Start date
Can u share more details about the exact HW config. So people might give you tips and optimizations in the hw config? When I see it correctly theres no possibility for NVMe? What your planning for the networking?
Here is the detailed configuration that we are going for. We are taking 4 of these for the cluster
Model | Description | Qty |
| R281-3C1 (rev. 200) | 2nd Gen. Intel® Xeon® Scalable and Intel® Xeon® Scalable Processors 6-Channel RDIMM/LRDIMM DDR4, 24 x DIMMs Supports Intel® Optane™ DC Persistent Memory Dual 1Gb/s LAN ports (Intel® I350-AM2) 1 x Dedicated management port 12 x 3.5" and 2 x 2.5" SATAIII hot-swappable HDD/SSD bays Onboard 12Gb/s SAS expander 8 x PCIe Gen3 expansion slots 2 x OCP Gen3 x16 mezzanine slots Aspeed® AST2500 remote management controller Dual 1200W 80 PLUS Platinum redundant power supply | 1 |
| CPU | Intel® Xeon® Silver 4210R (10 core 20 Thread) | 2 |
| Memory | 16 GB DDR4 2933MHz (192 GB) | 12 |
| HDD | 2TB 7.2K SATA 3.5'' (in JBOD mode) | 8 |
| HDD | 10TB 7.2K SATA 3.5'' (For temporary backup storage) | 1 |
| HDD | 1TB 7.2K SATA 3.5'' (For ISO image storage) | 1 |
| SSD | 240GB 2.5'' Sata (For OS) | 1 |
| SSD | 960Gb 2.5'' Sata (For Journal) | 2 |
| LAN Card | CLN4312 1G dual port (For Data traffic and Cluster traffic) | 1 |
| LAN Card | CLN4222 10G dual port (For Storage and Backup traffic) | 2 |
| HBA card | CRA3338 | 2 |
If I see it correctly from the SPECS CRA3338 is a Broadcom Controller flashed in the so called "IR" Mode. I'm not sure if it is really possible to expose the drives in JBOD mode.
For CEPH you need to expose the drives directly. So I would recommend to buy a version of the controller which is flashed with the "IT" Mode.
One more point:
in my experience backing the CEPH OSD's with a SSD is a pain in the ASS in case you need to replace a drive or even the SSD.
Also performance is not as good as you probably want to have it. It is of course much better as just with spinning drives, but at the lowest end you would like to see for VM Images.
Maybe you consider a recalculation with 2 TB Sata SSD for the OSD'S,
if there is a budget problem maybe you can create two pools -> one with large spinning disks and probably a SSD for journal and a fast pool for VM system images.
in my experience backing the CEPH OSD's with a SSD is a pain in the ASS in case you need to replace a drive or even the SSD.
Also performance is not as good as you probably want to have it. It is of course much better as just with spinning drives, but at the lowest end you would like to see for VM Images.
Maybe you consider a recalculation with 2 TB Sata SSD for the OSD'S,
if there is a budget problem maybe you can create two pools -> one with large spinning disks and probably a SSD for journal and a fast pool for VM system images.
I have tested CRA3338 with few drives and it is possible to not configure the drives with a software RAID as in the drives stay in Non Raid mode. But let me know if i can test whether the drives are actually in JBOD using proxmox cli (I only have access to the CLI).
ok if you see the drives in proxmox (usually as /dev/sd... ) then it is ok.
At a last resort it is easily possibly to reflash an IR mode controller to IT mode and vice versa (same hardware flashed with Megaraid code is another thing, it is often not possible to flash these beasts to IT/IR Mode)
Last edited:
Thank you for the quick response. The drives are showing up as /dev/sdx. I used smartctl /dev/sdX to check on all the connected drives. They diplayed all the drive details including drive serial and SMART status which was enabled. Also like you suggested, we are considering for an alternate config where the only difference would be 9 x 2TB SSDs (instead of HDDs) as OSD and we are skipping the journal.
Last edited: