PROXMOX ON A BAREMETAL SERVER WITH 18 x 3.84TB NVMe (ZFS?)

[waltar]

Once setup 4 pve cluster, 8 same new ssd's, 1 raid1@p710 with ext4, 1 raid1@p710 with xfs, 2 with zfs mirror, the 2 hosts with p710 have 2% ssd lifetime used, the 2 with zfs have 36-40% lifetime used but we have 2 further p730 and will switch the zfs mirrors to hw-raid1 too later this year.

 

[alexskysilk]

ust readed, they talk about resilient replication, so i think its still the same periodically and simple zfs-send/receive.

Please provide a link. Truenas Enterprise has touted true HA since 2022; As I am not affiliated with them I cant speak for the technicals and would be glad for the education. Async replication has been and is available for the free product. its also available in pve directly or any CoW filesystem manually.

The other thing they are telling is dual controllers, which isnt interesting as it has nothing todo with HA.

What do you suppose these "controllers" are? also, what is HA? you might be arguing semantics.

With regards to Jovian, or hast, or drbd, or RSF1, or roll-your-own-pacemaker- There is no question that there are dyi clustering solutions. For the most part they offer a poor value proposition to an operator because they have severe technical limitation due to a lack of central metadata keeper making them susceptible to split brain condition. You made the choice (I assume after studying the matter carefully) of trusting Synology's implementation and that's fine for you, but others might not come to the same conclusion. The upside of buying a commercial product is that you have someone to yell at when shit hits the fan, and they would be obliged to help you recover and fix their product.

So had zfs fileserver with power outage, nothing help than full restore from second one and department cannot work for few days ...
In other place restore 16 TB hdd in raidz2 as 4x6 hdd, pool 349 TB, 114 TB which is 32% used, took 32 h for 32 % of the 16 TB = ~45 MB/s.

What conclusions do you draw from this? Given the large number of unknown variables in your report it could almost be anything.

Ceph subscriptions for ceph start in the high 5 digits and you can reach easily the 7 digit price range

I'm unsure of the context of the comment; I assume you mean to present RHCS/RHEV as a storage alternative, in which case you wouldn't really care what the underlying technology is any more then it matters that Synology uses BTRFS. You are buying a solution, not a storage technique.

 

[Ramalama]

Please provide a link. Truenas Enterprise has touted true HA since 2022; As I am not affiliated with them I cant speak for the technicals and would be glad for the education. Async replication has been and is available for the free product. its also available in pve directly or any CoW filesystem manually.


What do you suppose these "controllers" are? also, what is HA? you might be arguing semantics.

We buyed JovianDSS before 2022 and compared against TrueNas.
At that time there was definitive no true HA on TrueNas.
If things were different at that time, we would have gone TrueNas 100%. Especially because its surely cheaper.

And Dual controllers is not HA for me.
Ha is for me at least 2physical Servers, not 2 scsi controllers in one node, which tbh make only a little sense, since the chance of the failure of one Controller and not Something else in the node is less then 0,001% in reality.

True HA with 2 nodes, covers a failed controller on a node anyway. So 2 controllers is even more senseless in my opinion at least if you can have real HA, with at least 2 nodes.

Cheers

 

[alexskysilk]

If things were different at that time, we would have gone TrueNas 100%. Especially because its surely cheaper.

I doubt that. Truenas enterprise isnt "cheap."

Ha is for me at least 2physical Servers, not 2 scsi controllers in one node

Fair. I hope you understand that not many share your opinion. dual controllers with a single disk set is the gold standard for HA precisely BECAUSE it avoids the potential for split brains- there is only one set of disks. YOU might not consider it HA, but it fulfills the requirements of High Availability much more reliably then a two silo mirror.

edit on reflection, its also has the potential to be double as performant, as an active-active pair can provide two separate targets simultanously using both controllers independently. This is not really possible with a two silo mirror.

True HA with 2 nodes, covers a failed controller on a node anyway.

I think you're hung up on the mechanical layout. dual controllers ARE two nodes; this is no less valid than a blade "server," where multiple nodes share the same pairs of redundant services like power supplies, cooling, and switching.

 

[Ramalama]

You cant place dual controllers in 2 separate buildings.
Blades have at least a separate mainboard, separate memory and separate CPUs, compared to 2 controllers in the same node.

I hung up for the mechanical Layout, because thats how HA is handled in Companies.

TrueNas Enterprise is probably not cheap, but i believe, cheaper as JovianDSS. Thats what i said.
We compared multiple zfs Solutions that offers HA, 5years ago.

However, i dont want to spam this thread with nonsense.
If you believe that dual controllers and a single node is more reliable, then 2 nodes in HA, then i cant help you.

Cheers

 

[guruevi]

Replication is not HA, HA is where either node can fail and the system continues to work. Replication cannot recover when the replication hasn’t completed and the latency is not acceptable for most use cases to have HA across two datacenters. Snapshots and replication is a backup strategy. RAID is not a backup.

As with your other comments, you have no clue. All of 2-node HA works with shared disks and 2 controllers, it has to for consistency. Exactly how TrueNAS or Nexenta works, every hardware RAID (iSCSI from HP, Dell) and JBOD enclosure also has 2 controllers, shared disk. TrueNAS is pretty cheap, I’ve gotten actual quotes and they are basically badged SuperMicro servers and they charge a few % on top for the license. What you think is expensive did what they do properly: shared architecture, SAS controllers, redundancy, monitoring - the hardware on its own is expensive - more expensive than jamming bargain bin SSD in a chassis.

There are actually RAID solutions that put 2 controllers in 2 buildings and have it function, there are restrictions on how far the two locations can be from each other, but definitely tenable to be outside nuke range (~30 miles). It’s not going to be ‘fast’. There are also eventual consistency engines, provided you can sustain some data loss.

I’ve written code in the past to do the same on Solaris back in the day ZFS can definitely do it. You want expensive, go ask quotes for most NVMe scale-out solutions, $100/TB/month for software alone is not unusual.

What you’re thinking you need is scale-out, Ceph, VAST, Isilon, MinIO, but before you properly build those I’m sure you are going to compare them to the RAM in your homelab. BtrFS/ZFS and DRBD/snapshots is not scale-out, it’s calling your backup a HA solution.