pve-firewall prevents VMs to start from replicated (glustefs) volume

[swartz]

Proxmox v3.4 nodes in a cluster. I'm attempting to use pve-firewall to secure access to the nodes.

Nodes are on 10.10.10.1 and 10.10.10.2. I'm attempting to manage cluster from another machine on 10.10.2.1.

I've enabled the firewall as per https://pve.proxmox.com/wiki/Proxmox_VE_Firewall and wish to add the entire 10.10.0.0/16 to "management" IPSET.
So my cluster.fw look like this

[OPTIONS]
# enable firewall (cluster wide setting, default is disabled)
enable: 1

[IPSET management]
10.10.0.0/16

I'm able to use the web interface and SSH into all cluster nodes. But starting a VM fails. The VM icon changes to "white" and VM status is "running" but after a while the task fails with
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 ...[snipped]... failed: got timeout

Attempting to use NoVNC while a VM is being started causes its window to stall at "Starting VNC handshake" message. Notice that it doesn't say "Failed to connect to server (code: 1006)" so the VM is being started?

Accessing other, already running VMs, with NoVNC works just fine. I'm able to STOP already running VMs. Once I try to start them, same issue occurs.

Once I disable pve-firewall, I am able start VMs again.

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

I thought I'd bump the post as I've updated wording. By mistake said "node" when I meant VM in a few spots.

Also created a ticket:

https://bugzilla.proxmox.com/show_bug.cgi?id=620

Any ideas?

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Did some more investigating.

cluster.fw is now just this:

[OPTIONS] 
enable: 1

Web GUI is being accessed from laptop connected directly to the same switch that nodes are on. Laptop has static IP of 10.10.10.8.

Same issue. As soon as pve-firewall is enabled, unable to start VMs from web interface. Stopping already running VMs work fine.

#ipset list PVEFW-0-management-v4
Name: PVEFW-0-management-v4
Type: hash:net
Header: family inet hashsize 64 maxelem 64
Size in memory: 1424
References: 4
Members:
10.10.10.0/24

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

So this is getting just weird.

A VM on node's local storage can be started just fine with or without pve-firewall being enabled.

A VM on replicated storage (glusterfs) and pve-firewall disabled: I can start VM from web interface. All is well.

But a VM on replicated storage (glusterfs) and pve-firewall enabled, VM start does not work.
As soon as I attempt to start a VM I can see that there not one but TWO kvm processes being spawned.

root     1029174  0.0  0.0 258416 12780 ?        S    15:51   0:00 /usr/bin/kvm -id 100 [snip*]
root     1029175  0.0  0.0      0     0 ?        Zs   15:51   0:00 [kvm] <defunct>
root     1029176  0.0  0.0 430648 21820 ?        Sl   15:51   0:00 /usr/bin/kvm -id 100 [snip*]

The first process (pid 1029174) goes away after about 5-10sec. Then after 20-30 seconds the second kvm process (pid 1029176) exits as well. The second pid is the kvm PID that is reported as timing out via web interface (I looked up the PID as reported via JASON response).

I've tested this behaviour multiple times. It's consistent. This wasn't me hitting "Start" VM multiple times in a row.

* I snipped the full KVM params but they are identical for both processes (i even diff-ed them to be sure).

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

*crickets *

After posting here, creating an issue in the bugtracker, and posting to pve-users... no response?

 

[wolfgang]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Hi,
can you pleas send me you network config.
Do you use the 10.10.0.0/16 for cluster com and Glusterfs trafic?

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Hi,
can you pleas send me you network config.
Do you use the 10.10.0.0/16 for cluster com and Glusterfs trafic?

I use a public IP for cluster. And private network on separate interface.


Here's the config from first cluster node. The rest are exact the same except for IP addresses:

===========

# eth0 bridge
auto vmbr0
iface vmbr0 inet static
gateway w.x.y.1
network w.x.y.0
address w.x.y.200
netmask 255.255.255.0
bridge_ports eth0
bridge_stp off
bridge_fd 0

# eth3 private network for glusterfs
auto eth3
iface eth3 inet static
address 192.168.200.200
netmask 255.255.255.0
mtu 9000

 

[wolfgang]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Ok the double started Process is strange.
I will investigate why this is happens.
can it be that you block the traffic on the private net?
try to explicitly allow the gusterfs traffic on eth3.

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Ok the double started Process is strange.
I will investigate why this is happens.
can it be that you block the traffic on the private net?
try to explicitly allow the gusterfs traffic on eth3.

No, sir. No firewall for private network (eth3).
Wide open because it's a server-to-server link specifically for glusterfs.

 

[wolfgang]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Hi,
I investigate and it seems everything is ok here.
One question did you add a extra rule that allow all traffic on eth3?
if not it will block all traffic from glusterfs.
IPset allow only (PVE GUI, VNC, SPICE, SSH).

 

[swartz]

Re: pve-firewall prevents VM starts from web interface (also noVNC hangs)

Hi,
I investigate and it seems everything is ok here.
One question did you add a extra rule that allow all traffic on eth3?
if not it will block all traffic from glusterfs.
IPset allow only (PVE GUI, VNC, SPICE, SSH).

I've experimented with a number of firewall settings. Nothing prevented the issue from occurring as long as pve-firewall was enabled.
This is my current cluster.fw file:

=========================

[OPTIONS]
# enable firewall (cluster wide setting, default is disabled)
enable: 1

[IPSET management]
w.x.0.0/16
192.168.200.0/24

========================

 

[wolfgang]

As I wrote this setting do not allow glusterfs traffic.

 

[swartz]

As I wrote this setting do not allow glusterfs traffic.

Hmmmm glusterfs still working fine and replicating.
This may be due to the fact that connections are in the established state and get by firewall rules.

Still why would it start 2 VMs?

 

[nethfel]

are you hosting the glusterfs on the same servers as the proxmoxve hosts or separate servers?

 

[swartz]

are you hosting the glusterfs on the same servers as the proxmoxve hosts or separate servers?

Yes, i'm using the same two servers as both PVE and glusterfs hosts.

A bit late to the party, but thought I would update this in case someone encounters the same issue.
The issue was improperly configured firewall rules just as Proxmox staff indicated. Thank you for making me see the light

 

[swartz]

On a related note, if my machines are directly connected via eth3 ports to each other for glusterfs replication. And, once enabled, pve-firewall filters all interfaces.
What is the best way to minimize iptables overhead since there is no real need to filter traffic on the eth3 iface?

Is this as good as it gets?
IN ACCEPT -i eth3