Can you tell me how to do it better?

[alex821982]

We want to assemble the following virtualization scheme.



I need advice on the scheme itself and how to properly organize reservations.

It is clear that Proxmox BackUP Server is also missing here.

But it is not very clear to me how to act in case of failure of one of the VM Storage

If we have a backup server, then we can restore on the second VM Storage, but it takes a very long time.

There is a similar scheme on XCP-NG, and there we implemented it by simply doing continuous replication from one VM Storage to another using XOA every night. Won't it work on proxmox?

How to organize it correctly?

By replicating TrueNAS itself from one to another? Will this work correctly if all the machines are running on proxmox? And will proxmox normally see all these machines on the second storage? As far as I remember, there are some nuances when replicating TrueNAS with rights to objects where it is replicated, although I may be confusing something...

 

[sdettmer]

I have an instance where virtual machines share a "virtual NAS" (actually a few Samba Shares via privileged containers on ZFS data sets) and I used ZFS replication (every 5 minutes) for the file trees (windows document file trees). Was not a big installtion (IIRC 1 million files, 1 TB total, or so) and worked well, except sometimes the syncoid left some snapshots that I manually deleted a few times. The replica was on a different metal. The VM system image backups (just daily) were also ZFS send to the other metal, so restore of course took a few minutes, but at least was local as as fast as the SSDs, basically.

 

[SteveITS]

Is your goal uptime? You might consider a 3 node cluster using Ceph, and no NAS. Any VMs stay running if one node fails.

 

[alex821982]

I've just run through the description of the Ceph Cluster quickly.
As far as I understand it, it is much faster in disk performance, but more difficult to set up and the entry price is higher.
But the fact is that there is already one host and one NAS for such a scheme.
And now we need to launch all this gradually, and the second host and the second NAS are supposed to be added later...
So now we need to act within the framework of this scheme, so I'm thinking about options...

 

[Johannes S]

You will need a third host or qdevice for quorum/HA:
https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support

You could use a PBS for it or alternatively a small vm / docker instance on one of your TrueNAS hosts

 

[alexskysilk]

Truenas is available with real time sync. see https://www.truenas.com/truenas-enterprise/

If you mean to use the community version- no real time sync. You'd be better served by each of your pve nodes having local zfs storage and using pve-zsync for quasi HA. see https://pve.proxmox.com/wiki/PVE-zsync

 

[alex821982]

And please tell me about Ceph

I watched this little video on creating a cluster with Ceph

https://youtu.be/-qk_P9SKYK4?si=slmzBy1G1pHSb1Mx

But I didn't understand one thing.

Since I'm more familiar with XCP-NG, this video immediately reminded me of my pool on four XCP-PNG nodes with shared network storage. This is exactly the scheme above, which I wanted to implement on Proxmox as well.
But it just shows VM migration from one host to another, everything is clear about that.
But I did not understand how, if one of the hosts fails, how will I access the VM from the host that failed?
In the case of XCP-NG, it is clear that there is one network storage for all hosts and the VMs themselves are on it, in general, it works the same way on proxmox if we have a common network storage and VM migration between hosts, too, as I understand it.
What about here? Are all VMs distributed across each host's local storage? And how does it happen so fast if that's the case? After all, the entire volume of each machine should be on each host, shouldn't it?

Is it something similar to raidz from the drives on the hosts?

 

[david_tao]

Ceph needs 3 nodes at least, and will replicate data between all nodes in background(2 copies in 3 nodes cluster), it can also configured as erasure coding (in short, you may tread it as host level RAID5 or 6). So if any single host down, the VM disk in Ceph RBD is still available for read/write because you just lost a single copy. Following Red Paper may you can take a look at.
https://www.redbooks.ibm.com/redpapers/pdfs/redp5721.pdf

 

[alexskysilk]

But I did not understand how, if one of the hosts fails, how will I access the VM from the host that failed?

You need to seperate, logically, the compute cluster from the storage cluster, even if you intend to implement them hyperconverged (on the same hardware.)

The compute cluster functions as you expect- when a node is down or fails, the workload restarts on other nodes. This functionality depends on shared storage- so far so good.

A ceph cluster is a shared nothing storage cluster- the cluster can operate with any component out- up to and including a host; This also means you need a MINIMUM of 3 nodes and a MINIMUM of 3x your usable capacity- EG, if you plan on deploying a 12TB cluster, you will want at least 3 nodes each with 12TB of raw OSD (disk drive) deployed. a more supportable configuration would be 4 nodes which will allow for a full node fault to have rebuild capacity (the failed node hosted placement groups would simply be redeployed to available excess capacity on the surviving 3.)

 

[alexskysilk]

it can also configured as erasure coding

For the purpose of this conversation (virtual disk payload) pretend this isnt an option.

 

[alex821982]

Thanks, it seems generally understandable.
Still, it's very expensive.
I'll have to focus on the version that I drew in the first message.
For some reason, in various articles and discussions, this option is noticeably less common.
Although in all respects it is easier and cheaper to implement it.
Does the Ceph cluster have that much higher performance? Although it seems that considerable resources are required there for constant data replication to all hosts.
.
Are there any comparative tests of these configurations somewhere?

But I'll probably stick to my own scheme.
The only thing I don't understand yet is whether proxmox will correctly see and work with the shared storage that will be replicated by truenas to the second truenas. It is without additional manipulations on the second storage.

 

[alex821982]

There is also the problem of how to recreate the machines completely if we have a copy of the disks on the second NFS storage.Just connect these disks to an existing machine from the second storage, as far as I understood, it is impossible. Then replication between two NAS makes no sense. It only turns out to be a backup server from which you need to restore to a second storage...

 

[alexskysilk]

Still, it's very expensive.

Everything is relative. It is a fact of life that the more "9s" you seek the higher the cost goes up, exponentially.

For some reason, in various articles and discussions, this option is noticeably less common.
Although in all respects it is easier and cheaper to implement it.

Cant guess my what you are referring to.

Does the Ceph cluster have that much higher performance?

Possibly, but at smaller node/osd counts- no.

Are there any comparative tests of these configurations somewhere?

Plenty. but you're looking at it the wrong way; define your needs first- capacity, minimum acceptable performance per vm iops; total vm's to satisfy; feature requirements- thin provisioning, inline compression. snapshots, compatiblity with your backup solution, etc etc. The go about comparing the available solutions. The old admonition will likely stand- price, performance, and features- pick any two.

The only thing I don't understand yet is whether proxmox will correctly see and work with the shared storage that will be replicated by truenas to the second truenas.

You're having a hard time because there isn't a good way to do this. The two filers will be two seperate stores. The chances of creating split brained data is high and difficult to manage.

There IS a way to use such a configuration, and that is using in guest mirroring. In effect, you would map a vm two separate vdisks, one on each filer- and do software mirroring in guest. Probably the only sane way to get to what you're after.

 

[alex821982]

Cant guess my what you are referring to

Here I meant that there are mostly examples of configuration (various videos and articles on the Internet) either with repositories on the hosts themselves or with the Ceph organization. And with the setup and different usage features, there is almost no option with shared NFS storage. Although in my opinion this option is easy to set up and has good fault tolerance and functionality.
I haven't tried all the features of Proxmox yet, but I hope that using multiple nodes with one shared storage is as functional and simple here as on the XCP-NG I use.

Possibly, but at smaller node/osd counts- no.

That is, does the performance decrease with the growth of nodes?

Plenty. but you're looking at it the wrong way; define your needs first- capacity, minimum acceptable performance per vm iops; total vm's to satisfy; feature requirements- thin provisioning, inline compression. snapshots, compatiblity with your backup solution, etc etc. The go about comparing the available solutions. The old admonition will likely stand- price, performance, and features- pick any two.

The tasks here are very diverse. I don't even know in advance which services will need to be implemented, so I want to put together the most universal solution. Snapshots?, of course they are needed, how about without them The number of machines to start with is 25 pieces, linux and windows, they will also use mssql databases and others.

There IS a way to use such a configuration, and that is using in guest mirroring. In effect, you would map a vm two separate vdisks, one on each filer- and do software mirroring in guest. Probably the only sane way to get to what you're after.

Yes, here your option seems to be the only one, but it is quite laborious and, I would say, not very convenient for me. Too much extra stuff is needed.
In principle, I have already come to terms with the option of using Proxmox Backup Server, especially after conducting some tests even in a virtualized environment of all components, I got good speed results.
I think if there is a separate server with a fast read RAIDZ array and a fast network, you can get good results in terms of recovery speed from it to the second VM storage.
Although, as an XCP-NG user, it was a surprise to me that such a seemingly ordinary and simple function as Continuous Replication of a VM to any location and the subsequent ability to launch it from there at any time is not implemented here..

 

[Johannes S]

Although, as an XCP-NG user, it was a surprise to me that such a seemingly ordinary and simple function as Continuous Replication of a VM to any location and the subsequent ability to launch it from there at any time is not implemented here..

Isn't this what pve-zsync (mentioned by @alexskysilk earlier in the thread) is suppossed to do? https://pve.proxmox.com/wiki/PVE-zsync

 

[alex821982]

As far as I understand, this is not exactly the replication option that I am talking about and is not suitable for the option from my first message.