Proxmox VE on Gigabyte R281-3C1 with CRA3338 array controller

jsengupta

Active Member
Feb 27, 2020
44
4
28
29
We are going to purchase 4 x Gigabyte R281-3C1 servers with CRA3338 array controller to create a Proxmox CEPH cluster. Has there been any issue while using this server or this array controller? Users who have used this server to host Proxmox VE, please let us know your experience with this server model or the array controller. Also mention your config if possible.
 
We are going to purchase 4 x Gigabyte R281-3C1 servers with CRA3338 array controller to create a Proxmox CEPH cluster. Has there been any issue while using this server or this array controller? Users who have used this server to host Proxmox VE, please let us know your experience with this server model or the array controller. Also mention your config if possible.

Can u share more details about the exact HW config. So people might give you tips and optimizations in the hw config? When I see it correctly theres no possibility for NVMe? What your planning for the networking?
 
Here is the detailed configuration that we are going for. We are taking 4 of these for the cluster
Model​
Description
Qty​
R281-3C1
(rev. 200)
2nd Gen. Intel® Xeon® Scalable and Intel® Xeon® Scalable Processors
6-Channel RDIMM/LRDIMM DDR4, 24 x DIMMs
Supports Intel® Optane™ DC Persistent Memory
Dual 1Gb/s LAN ports (Intel® I350-AM2)
1 x Dedicated management port
12 x 3.5" and 2 x 2.5" SATAIII hot-swappable HDD/SSD bays
Onboard 12Gb/s SAS expander
8 x PCIe Gen3 expansion slots
2 x OCP Gen3 x16 mezzanine slots
Aspeed® AST2500 remote management controller
Dual 1200W 80 PLUS Platinum redundant power supply
1​
CPUIntel® Xeon® Silver 4210R (10 core 20 Thread)
2​
Memory16 GB DDR4 2933MHz (192 GB)
12​
HDD2TB 7.2K SATA 3.5'' (in JBOD mode)
8​
HDD10TB 7.2K SATA 3.5'' (For temporary backup storage)
1​
HDD1TB 7.2K SATA 3.5'' (For ISO image storage)
1​
SSD240GB 2.5'' Sata (For OS)
1​
SSD960Gb 2.5'' Sata (For Journal)
2​
LAN CardCLN4312 1G dual port (For Data traffic and Cluster traffic)
1​
LAN CardCLN4222 10G dual port (For Storage and Backup traffic)
2​
HBA cardCRA3338
2​
 
We are going to purchase 4 x Gigabyte R281-3C1 servers with CRA3338 array controller to create a Proxmox CEPH cluster. Has there been any issue while using this server or this array controller? Users who have used this server to host Proxmox VE, please let us know your experience with this server model or the array controller. Also mention your config if possible.
If I see it correctly from the SPECS CRA3338 is a Broadcom Controller flashed in the so called "IR" Mode. I'm not sure if it is really possible to expose the drives in JBOD mode.

For CEPH you need to expose the drives directly. So I would recommend to buy a version of the controller which is flashed with the "IT" Mode.
 
One more point:

in my experience backing the CEPH OSD's with a SSD is a pain in the ASS in case you need to replace a drive or even the SSD.

Also performance is not as good as you probably want to have it. It is of course much better as just with spinning drives, but at the lowest end you would like to see for VM Images.

Maybe you consider a recalculation with 2 TB Sata SSD for the OSD'S,

if there is a budget problem maybe you can create two pools -> one with large spinning disks and probably a SSD for journal and a fast pool for VM system images.
 
If I see it correctly from the SPECS CRA3338 is a Broadcom Controller flashed in the so called "IR" Mode. I'm not sure if it is really possible to expose the drives in JBOD mode.

For CEPH you need to expose the drives directly. So I would recommend to buy a version of the controller which is flashed with the "IT" Mode.
I have tested CRA3338 with few drives and it is possible to not configure the drives with a software RAID as in the drives stay in Non Raid mode. But let me know if i can test whether the drives are actually in JBOD using proxmox cli (I only have access to the CLI).
 
I have tested CRA3338 with few drives and it is possible to not configure the drives with a software RAID as in the drives stay in Non Raid mode. But let me know if i can test whether the drives are actually in JBOD using proxmox cli (I only have access to the CLI).
ok if you see the drives in proxmox (usually as /dev/sd... ) then it is ok.

At a last resort it is easily possibly to reflash an IR mode controller to IT mode and vice versa (same hardware flashed with Megaraid code is another thing, it is often not possible to flash these beasts to IT/IR Mode)
 
Last edited:
ok if you see the drives in proxmox (usually as /dev/sd... ) then it is ok.

At a last resort it is easily possibly to reflash an IR mode controller to IT mode and vice versa (same hardware flashed with Megaraid code is another thing, it is often not possible to flash these beasts to IT/IR Mode)
Thank you for the quick response. The drives are showing up as /dev/sdx. I used smartctl /dev/sdX to check on all the connected drives. They diplayed all the drive details including drive serial and SMART status which was enabled. Also like you suggested, we are considering for an alternate config where the only difference would be 9 x 2TB SSDs (instead of HDDs) as OSD and we are skipping the journal.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!