-- Intro --
I have this situation, I have two servers, both with Proxmox on 4x NVME BTRFS RAID10 using a PCIEx Adapter + Bifurcation. They both form a cluster with a raspberry pi 3 as voting device.
I don't know if is the adapter, or the NVME, or BTRFS, but both servers are restarting and sometimes they hang on reboot(when that happen I need turn off and turn on).
I think the cause of the reboot is because sometimes a NVME stop showing up on Disk menu. After a reboot it show up.
Now the cause to be missing could be a fault NVME, or fault adapter, could be heat from the 10GB Fiber Adapter, which I already tried to move away from NVME, but with both servers are happening this.
-- Planning --
My initial plan is just reinstall Proxmox on a single SSD with ext4 in each server, but I have a zfs pool RAID6 8x8TB in one server and a zfs pool RAID5 3x2TB in the other.
I don't know if the pool can be just scanned and added back to Proxmox, this is my main issue, or lack of information, to been able reinstall Proxmox right now.
My second phase is turn my workstation on a server, creating 3 node cluster, removing the raspberry voting device, and instead use a zfs pool, use a Cephfs pool, buying more 4x 8TB, having 12x 8TB in total, and putting 4x 8TB in each server, the NVME probably I will use as ceph cache pool, since I can rely to much to use them as storage.
My currently hardware is:
I have this situation, I have two servers, both with Proxmox on 4x NVME BTRFS RAID10 using a PCIEx Adapter + Bifurcation. They both form a cluster with a raspberry pi 3 as voting device.
I don't know if is the adapter, or the NVME, or BTRFS, but both servers are restarting and sometimes they hang on reboot(when that happen I need turn off and turn on).
I think the cause of the reboot is because sometimes a NVME stop showing up on Disk menu. After a reboot it show up.
Now the cause to be missing could be a fault NVME, or fault adapter, could be heat from the 10GB Fiber Adapter, which I already tried to move away from NVME, but with both servers are happening this.
-- Planning --
My initial plan is just reinstall Proxmox on a single SSD with ext4 in each server, but I have a zfs pool RAID6 8x8TB in one server and a zfs pool RAID5 3x2TB in the other.
I don't know if the pool can be just scanned and added back to Proxmox, this is my main issue, or lack of information, to been able reinstall Proxmox right now.
My second phase is turn my workstation on a server, creating 3 node cluster, removing the raspberry voting device, and instead use a zfs pool, use a Cephfs pool, buying more 4x 8TB, having 12x 8TB in total, and putting 4x 8TB in each server, the NVME probably I will use as ceph cache pool, since I can rely to much to use them as storage.
My currently hardware is:
- Server 1:
- Supermicro X10DRi
- 2x Intel Xeon E5-2650L v4
- 4x 32GB RAM DDR4 2400Mhz RDIMM Samsumg
- 8x 8TB WD Enterprise SAS
- 4x M.2 NVMe 128GB Kingspec on adapter using bifurcation.
- Server 2:
- Supermicro X10DRi
- 2x Intel Xeon E5-2650L v4
- 6x 16GB RAM DDR4 2133Mhz RDIMM Hynix
- 3x 2TB WD Enterprise SATA
- 4x M.2 NVMe 128GB Kingspec on adapter using bifurcation.
- Server 3: (my actual workstation)
- Asus X99 Sabertooth
- 1x Intel Xeon E5-2697 v4
- 4x 8GB RAM DDR4 2400Mhz UDIMM Corsair
- 1x M.2 NVME 512GB Gigabyte PCIe 4.0 (slot is 3.0) on adapter.
- 1x M.2 AHCI 256GB Samsung PCIe 3.0
- Chassi:
- 3x Supermicro SC745(backplane SAS-2)
Last edited: