[SOLVED] Proxmox WebGUI doesnt reply from original address

[michaldudek]

Hi there,
I tried few days to find any similar topic, but I cant find. So I hope, there is not duplicate thread.

My issue:
When I try to open proxmox WebGUI from local browser on address https://10.10.0.10:8006 I cannot reach. On router I see, sent syn packet to 10.10.0.10 tcp/8006 but, there is no response back. So in browser I get connection timeout, reaction was too slow.

So I tried curl https://10.10.0.10:8006 on proxmox itself. I get response: <title>pmx1 - Proxmox Virtual Environment</title>. Seems to be fine, and pveproxy respond, and serve right content.

So next I tried to see, what is going on on port 8006 by tcpdump: tcpdump -envi vmbr0 port 8006
And I get this:

15:14:08.890284 1c:69:7a:0f:0b:f7 > 3c:ec:ef:0e:9f:d4, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 49133, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.62848 > 10.20.0.10.8006: Flags [S], cksum 0x30a7 (correct), seq 1326631639, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
15:14:08.890314 3c:ec:ef:0e:9f:d4 > 48:a9:8a:4d:e3:3f, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 127, id 49133, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.62848 > 10.20.0.10.8006: Flags [S], cksum 0x30a7 (correct), seq 1326631639, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
15:14:08.890404 48:a9:8a:4d:e3:3f > 36:b4:d1:3f:95:61, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 126, id 49133, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.28813 > 10.20.0.10.8006: Flags [S], cksum 0xb59a (correct), seq 1326631639, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

I see the packet on port 8006 was delivered to my first VM with IP 10.20.0.10, but not to original vmbr0 pveproxy 10.10.0.10 address. So I thing, there is issue, why my local machine in same subnet, doesnt get response from pveproxy (syn-ack).

There are my configs:
/etc/hosts

127.0.0.1 localhost.localdomain localhost
10.10.0.10 krz-p-s-pmx1.skolakrizik.cz krz-p-s-pmx1

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

/etc/networks

default         0.0.0.0
loopback        127.0.0.0
link-local      169.254.0.0

/etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.10.0.10/24
        gateway 10.10.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface eno2 inet manual

route -v

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.10.0.1       0.0.0.0         UG    0      0        0 vmbr0
10.10.0.0       0.0.0.0         255.255.255.0   U     0      0        0 vmbr0

netstat -tupln

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:85            0.0.0.0:*               LISTEN      2934/pvedaemon
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2625/sshd: /usr/sbi
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      3136/spiceproxy
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2867/master
tcp        0      0 0.0.0.0:8006            0.0.0.0:*               LISTEN      3128/pveproxy
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/init
tcp6       0      0 :::22                   :::*                    LISTEN      2625/sshd: /usr/sbi
tcp6       0      0 :::111                  :::*                    LISTEN      1/init
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/init
udp        0      0 127.0.0.1:323           0.0.0.0:*                           2643/chronyd
udp6       0      0 :::111                  :::*                                1/init
udp6       0      0 ::1:323                 :::*                                2643/chronyd

Any others needed? Please tell me, I add them.

And if I look at log tail -n 20 /var/log/pveproxy/access.log
I get only:

10.10.0.10 - - [12/02/2024:12:38:59 +0100] "GET / HTTP/1.1" 200 2235

So again, my issue is, the pveproxy doest repond on original vmbr0 IP 10.10.0.10, but forward packet to 10.20.0.10 which is first IP of my first VM.
Please, can you help?
Thank you.

EDIT: AAAAAND- I'm sorry. I forgot. SSH on same IP (10.10.0.10:22) works fine. Only WebGUI.
EDIT 2: I change back my original IPs from config and remove fake one. I dont know why I done.. sorry :/

 

[_gabriel]

seems /etc/host has wrong ip

 

[michaldudek]

seems /etc/host has wrong ip

Sorry, I forgot to change IP from config to fake one. IP si correct, I edited my previous post.

 

[michaldudek]

I tried same tcpdump on another pmx2, and get same tcpdump result.

15:24:11.142797 1c:69:7a:0f:0b:f7 > 3c:ec:ef:0e:ad:2a, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 4955, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.62861 > 10.20.0.10.8006: Flags [S], cksum 0xa73b (correct), seq 3209290750, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
15:24:11.142853 3c:ec:ef:0e:ad:2a > 48:a9:8a:4d:e3:3f, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 127, id 4955, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.62861 > 10.20.0.10.8006: Flags [S], cksum 0xa73b (correct), seq 3209290750, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

So I think, there must be some issue with DNS, because IP 10.20.0.10 is my primary DNS server.
But I dont know why, if I use just IPv4.

 

[_gabriel]

is 10.20.0.10 a nested pve ?
have tried with 10.20.0.10 stopped ?

 

[michaldudek]

is 10.20.0.10 a nested pve ?

Nope 10.20.0.10 is static IPv4 address for virtual machine on pmx1. But also pmx2 tcpdump shows same result, and this VM isnt there.

have tried with 10.20.0.10 stopped ?

I tried shutdown VM id 100 with IPv4 10.20.0.10, and look up to tcpdump again, but same result.

tcpdump: listening on vmbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:03:49.499708 1c:69:7a:0f:0b:f7 > 3c:ec:ef:0e:9f:d4, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 49368, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.63635 > 10.20.0.10.8006: Flags [S], cksum 0x0d64 (correct), seq 1148515749, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:03:49.499733 3c:ec:ef:0e:9f:d4 > 48:a9:8a:4d:e3:3f, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 127, id 49368, offset 0, flags [DF], proto TCP (6), length 52)
    10.10.0.160.63635 > 10.20.0.10.8006: Flags [S], cksum 0x0d64 (correct), seq 1148515749, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
11:03:49.776800 1c:69:7a:0f:0b:f7 > 3c:ec:ef:0e:9f:d4, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 128, id 49369, offset 0, flags [DF], proto TCP (6), length 52)

 

[_gabriel]

i think it's expected as tcpdump show packets incoming into vmbr0 and vmbr0 act as switch for host and its VMs.
imo, tcpdump doesn't provide help here.

 

[michaldudek]

i think it's expected as tcpdump show packets incoming into vmbr0 and vmbr0 act as switch for host and its VMs.
imo, tcpdump doesn't provide help here.

You are right. If i try tcpdump without port 8006, there is all traffic for all VMs.
But why is forwarder traffic on port 8006 to this VM? I'm 100% sure on VM id 100 with IPv4 10.20.0.10 is no service with port 8006.

But, I'm Wrong!

OK, I found the reason why I'm forwarder from https://10.10.0.10:8006 to VM id 100 with IPv4 10.20.0.10:8006.
We have on our router set up DSTNAT for passive ports on FTPS in range 8000 - 8500. I reconfigure this FW rule, and WebGUI works on its IP https://10.10.0.10:8006. No more forward 8006 to FTPS service on 10.20.0.10.

Thank you _gabriel, your hint about "as switch" was heureka moment for me, and I realize, we have another service on VM id 100 with same ports range.

Problem solved. Thank you.