[SOLVED] What is possible with 2-node proxmox cluster?

S

swartz

Member
Oct 30, 2014
80
3
6
Short story: I got involved with assisting a non-profit. Over past 2 years I've consolidated all of their "servers" (just old workstations all over the office floor) into a single host running ProxmoxVE on an older Dell R200 server that they had.

Currently there are 5-7 VMs running their website, file server, accounting software, membership database, and some kind of app instances. The server is showing it's age, has a failing motherboard (won't boot with PCIe expansion card present), and it's has 12GB RAM already in 4 slots. They've also experienced data loss as a result of drive(s) failing before.

Through hardware donations, the organization obtained a pair of Dell R510 servers. The servers have identical CPUs, slightly different amounts of RAM, and different amount and sized drives.

The RAM and current drives configurations:
- Server1: 24GB RAM, 2x 146GB RAID1, 4x146GB in RAID10
- Server2: 16GB RAM, 2x 146GB RAID1, 2x300GB in RAID1

I've used Proxmox in the past for personal use but my experience is limited to one VM-host scenarios. They don't need a 24/7 HA environment. Since servers are used and already past their best-before date I would like to setup some kind of a redundant environment. If one VM host fails, let the other take the load. I'm OK with manual failover and few hours of downtime to make the switch if needed.

My first attempt was to setup hosts as separate Proxmox nodes. Small enough of environment. But how do I share/sync VM disks so if one host fails the other can run the VMs? Regularly copy snapshots via rsync or some other method? NFS share available to the other machine?

This turned into reading about Proxmox clusters. Seems like it would simplify management (from one web interface) and allows for scenarios as above. But there is no fence-device. Only 2 servers are available. The drives aren't identical.

I've attempted to setup a cluster environment but ran into a number of issues and have plenty of questions.

My idea:
Each server has local storage. The first pair of drives (RAID1) will be for Proxmox install, iso, templates, etc. These don't have to be sync-ed between hosts, right? Actual VM disks would be stored on the second RAID volume(s). RAID 10 should provide plenty of performance.

I would designate Server1 as primary. Server2 would be a hot-standby. No VMs would actually run on server2 unless server1 is down, right? Less important: Can I use server2 as test ground for other VMs that have no production use?

How do I sync the VM disk images from server1 to the second server? I've attempted to create the same dir/mounts on both servers and add these as "Directory" storage on each (non-shared). But in cluster mode I only ever see primary's storage on web interface. Secondary's storage disappears from web interface. I thought this meant these are now auto-synced as I saw mentions of corosync on screen at cluster setup. But that does not appear to be the case. I've tried creating files and VMs and these weren't syncing across machines' local storage volumes.

Also in cluster mode, secondary's web interface becomes inaccessible. So what would happen if primary goes down? I wont be able to access the web interface now?

Could someone advise me if what I am attempting is a possibility? If not, what is possible with what they have?

Thank you.
 
Last edited:
Hi,
I think the best solution for your Problem is ZFS available at V3.4.
you can use zfs send receive to replicate the VM disks.
 
wolfgang said:
Hi,
I think the best solution for your Problem is ZFS available at V3.4.
you can use zfs send receive to replicate the VM disks.
Click to expand...

Thanks for response, wolfgang. I will look into replication with ZFS.

Is there any way possible to append local storage to each machine such that when the two machines are clustered, the storage shows up separately for each machine? Any way to manually move VMs between nodes in this scenario (i.e. non-live migration)?
 
Zfs is a local storage, so each machine handle its Zpool by them self.
You can move a offline VM in a cluster with migration on any local storage
 
wolfgang said:
Zfs is a local storage, so each machine handle its Zpool by them self.
You can move a offline VM in a cluster with migration on any local storage
Click to expand...
Thank you again for responding!

I should've have been more clear in my question. After looking at ZFS, it seems a bit more of an administrative overhead. Concepts like ZFS pools, volumes and something like LVM on top of it all really making my head spin. I'm attempting to simplify system management especially if someone else takes over and has even less of a clue. :) I will continue to examine and test a ZFS scenario on boxes of my own.

In the mean time I think I should start with baby steps:
What I meant is how do I add simple local storage available on each node such that it's visible independently while in a cluster. Simple = ext3/4. Is that even possible? And how to make moving an offline VM between these 2 independent (non-synced) storages possible? Perhaps when I get this scenario down I can examine syncing the independent local storages.

What I have done so far on each node:
- created ext4 filesystem spanning entire available space of the secondary RAID volume (292GB RAID10 partition on server1 and 300GB RAID1 partition on server2)
- mount locally available filesystem as /mnt/vm_storage
- added "Directory"-type storage pointing to this newly created location

Ive attempted to create identical configuration for both nodes. They differ physically in the size of drives of course.

Having done this, I could see and add the local storage as "Directory" on each node as nodes were independently managed.

Now when I created a cluster out of these two nodes, the web interface only showed server1's local storage that I've added. Server2's storage listed server1's storage. Huh? So servers2's "Directory" storage simply disappeared from web interface. I saw references to corosync on screen. My first thought that this meant the similarly named storage locations were now automagically synced. I wish!
icon_smile.gif
But that was not the case as my further testing indicated. I now understand that it's just the /etc/pve/ that gets synced. Furthermore, I was unable to access server2's web interface.

What did I do wrong?
 
Last edited:
You could try using Bittorrent Sync of Syncthing to replicate the storage volumes. I've never tested it, but its an idea I've been thinking about. Also, I don't think ZFS will be as tough of an administrative task as you imagine, especially once its setup. There are also many tutorials on the web regarding clustering with DBRD
 
you could put them on a shared glusterfs volume. that way, if they get more hardware for their cluster, you could simply add to it. if you want to do HA, all you need is a third dummy box to make quorum easy. this third box only needs to sit on the network and be reasonably reliable.

theres also drbd. if you dont think youll expand storage, and are willing to do the extra work, its said to be faster(less latency) than glusterfs or ceph.
 
alatteri said:
You could try using Bittorrent Sync of Syncthing to replicate the storage volumes. I've never tested it, but its an idea I've been thinking about. Also, I don't think ZFS will be as tough of an administrative task as you imagine, especially once its setup. There are also many tutorials on the web regarding clustering with DBRD
Click to expand...

It's not so much the administration that scares me. I'm already taking steps at looking at ZFS among many other potential solutions. It's rather having to explain(and document) multiple layers of complexity and technologies involved. There are a few volunteers here just like me. I am full-time employed elsewhere so not always available to fix problems. In my absence I hope to delegate some/most of the maintenance tasks to others involved (e.g. if A doesn't work, check B, run C). I can't always expect to have someone competent to delegate to. I wouldn't even call myself competent given my questions here. The simpler the setup and the fewer pieces, the easier it is to document/explain what to do to someone at least partially competent. Worse, is someone completely new being brought up to speed in a pinch. Attempting to understand all of the moving pieces is hard enough and requires specialized knowledge. Now try wrapping your head around a particular deployment.

LOL. Now that my rant is over :).

It appears there is no shortage of possible solutions. I will be putting together my own setups to try these and see how well/easy these work:
- ZFS
- DRBD
- GlusterFS
- Ceph
- Bittorrent Sync now

In the mean time I'm still wondering if the the local-storage scenario is possible as described in my last post? And what happened to the servers2's local storage when I clustered the two nodes?
 
Last edited:
alatteri said:
well, in a cluster scenario aren't the hosts supposed to be sharing the same storage anyway? So it is no surprise that Proxmox thinks that 2 volumes with the same mount are actually the same single shared storage...
Click to expand...

If only they had a NAS box... which then becomes a single point of failure. So another NAS box for replication is needed.
Then again a true cluster requires 3 nodes. I'm just trying to make best of what they got.

It is a surprise to me. And possibly to others as I search through forums. If that's the case, it isn't documented or made clear. When I added the "Directory" storage type I didn't not check off "Shared". If I did that then it would make sense for Proxmox to assume that it's the same shared storage. I created two independent non-shared storage locations (different IDs, physical location, size) expecting them to appear independently under each node. So is this a bug or expect behavior?

If that's the expected behavior, then mounting the two volumes at different mount-point will resolve this issue?
 
Last edited:
Hi

Local storage cannot be mounted on two proxmox nodes at the same time.To do so, you need a clustered filesystem (like GFS for example).Ext3,Ext4,XFS etc are not clustered filesystems so they should be mounted on just one node.

In your scenario I would go with DRBD in dual Primary or Primary/Secondary setup (on top of RAID).
DRBD is easy to configure and manage.

Just be aware that in dual Primary setup you must configure fencing to avoid VM corruption.

Another option could be a NAS box which could store your VMs.
In that case though, NAS box is a single point of failure.
 
acidrop said:
Hi

Local storage cannot be mounted on two proxmox nodes at the same time.To do so, you need a clustered filesystem (like GFS for example).Ext3,Ext4,XFS etc are not clustered filesystems so they should be mounted on just one node.
Click to expand...

Wait... two independent (local to each node) storage locations cannot be mounted at the same time? Why not?
Each storage is mounted on just one node (locally to the node). I understand that ext3/4 isn't a clustered file system thus I'm not attempting to sync the data between the two nodes' local storages.
Right now I just want them to show up independently under each node's storage resources. I'm not trying to mount them remotely.
 
Hello again,

After carefully reading your previous posts I think I've managed to understand what you are talking about. :)

I have also 2 PVE nodes. On each PVE I have a local storage configured with the same path (/var/lib/vz).Now, If I navigate through the web interface to Datastore -> Storage I can see just one local storage defined (/var/lib/vz). Although it seems that this path is shared on both nodes, it doesn't. Each PVE node has it's own /var/lib/vz , where it saves it's own unique VMs.

Now, let's suppose that you create a VM on node1 located in it's /var/lib/vz (you can select the node you wish VM to be created, on create new vm wizard). After this is finished, you will end with a VM located on Node1 /var/lib/vz.

If you want to offline migrate this particular VM on the second node (Node2), then just stop the VM, right click on it, select migrate, select Node2, uncheck online box and PVE will initiate a rsync job in order to migrate the VM on the other Node. Is this what you want?

Don't forget to correctly setup two node cluster, otherwise you will have quorum issues which can break web interface. Check the following wiki https://pve.proxmox.com/wiki/Proxmox_VE_2.0_Cluster in particular section: https://pve.proxmox.com/wiki/Proxmox_VE_2.0_Cluster#Two_nodes_cluster_and_quorum_issues

P.S

On web interface, in the left side you can see the two PVE nodes. If you expand each of them you will notice that each node has it's own local storage showing up there.
 
Last edited:
acidrop said:
Hello again,

After carefully reading your previous posts I think I've managed to understand what you are talking about. :)
P.S

On web interface, in the left side you can see the two PVE nodes. If you expand each of them you will notice that each node has it's own local storage showing up there.
Click to expand...

We are not out of the woods yet. :)

I'm configuring 2 servers with the following RAID and mounts layout:
Server1: 2x 146GB RAID1 (OS and /var/lib/vz/), 4x146GB in RAID10 ( mounted as /vm_storage/)
Server2: 2x 146GB RAID1 (OS and /var/lib/vz), 2x300GB in RAID1 (mounted as /vm_storage/)

I have /vm_storage/ on each node as you can see. It's just an ext4 partition on each server. I've added it to each respective server as local "Directory" type storage. "Shared" checkbox was left empty.

So they are just independent partitions on each node. Not shared or sync-ed in any way.

When I clustered the two nodes, web interface only showed (local) and (vm_storage) from server1 under each of the nodes in web interface. It was as thought server2's /vm_storage/ and local storage did not exist.

Hope that makes sense.
 
Last edited:
When I clustered the two nodes, web interface only showed (local) and (vm_storage) from server1 under each of the nodes in web interface. It was as thought server2's /vm_storage/ and local storage did not exist.
Click to expand...

Then something must be wrong with your setup. Check how "local" storage appears in the following screenshot (marked with red circl).

Capture.JPG
 
I would keep it simple..

you said you could handle some downtime.. so here is my suggestion..

With wonderfull proxmox you also get a backup feature. SO

Make some shared NFS (NOT ZFS) storage where both nodes backup to. In the event of a systems failure you could restor from backup on the other node.. would that be ok? it would simple and any IT guy would understand it. Only problem is they would be rolled back one day but maybe you can make a backup more often that just one time a day .. i don't know

This is perhaps the most simple way i can think of but it does have its problems.
 
Just wanted to update this. Maybe it will come in handy for someone else who stumbled into the same problem. After some time debugging and testing I know what happened. I now got the local storage setup and working as I wanted.

The way I was going about it is setting up the nodes individually and configuring their storage. Then I was adding nodes to cluster. What happened is /etc/pve/cluster.conf would get overwritten as that mount point got synchronized across the cluster.

The take away is this: add nodes to cluster and only then add any (local) storage. This way changes to /etc/pve/cluster.conf aren't gone at cluster config time. I feel this important point should be in wiki.

Thanks for all those that helped me in this thread.

PS. Now I'm testing with glusterfs to see how well it performs.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back