hello guys.
I am a sys admin currentyl being made humble again by (I think) proxmox's networking.
My setup:
WAN -> Router -> Physical Switches -> Proxmox VE (6.8.12-1) -> FreeBSD VM with WireGuard VPN
Whenever I connect with my VPN client (iPhone) to the WireGuard VPN server I experiance the following behavior:
1 (Expected) I can connect to any resource that is not hosted by the Proxmox VE.
For example: My router & Physical Switches have a Web interface, I can succesfully browse to those webpage.
I have multiple physical devices in my network, can access all of them fine just fine, as one would expect.
2 (Problem) I can not connect to any resource that is being hosted by proxmox VE.
For example: I have a webserver running on a 2nd VM within proxmox; can't access that webpage.
Also, I cant connect to the Proxmox webinterface.
To me it looks like Proxmox is unable to 'handle' the traffic on it's virtual switches (the way I would expect.)
So traffic from VM (Wireguard Free BSD) to other VM (webserver for example) is not handled right.
I ran a TCP Dump on my FreeBSD VM With Wireguard to troubleshoot this issue, and this is what I found:
Problem example
My iPhone is connected via the WireGuard IP: 10.96.100.2
And it try's to access a webinterface on 10.10.30.75:8123 witch is a VM hosted by the same proxmox VE server.
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
16:20:35.200498 IP 10.10.30.75.8123 > 10.96.100.2.54058: Flags [S.], seq 1645965074, ack 3563026976, win 65160, options [mss 1460,sackOK,TS val 16381042 ecr 3385564108,nop,wscale 7], length 0
16:20:37.760433 IP 10.10.30.75.8123 > 10.96.100.2.54066: Flags [S.], seq 703934273, ack 1988877869, win 65160, options [mss 1460,sackOK,TS val 16383602 ecr 1624876155,nop,wscale 7], length 0
16:20:41.605573 IP 10.96.100.2.54071 > 10.10.30.75.8123: Flags , seq 3662379520, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 247218524 ecr 0,sackOK,eol], length 0
16:20:41.605971 IP 10.10.30.75.8123 > 10.96.100.2.54071: Flags [S.], seq 4154632067, ack 3662379521, win 65160, options [mss 1460,sackOK,TS val 16387447 ecr 247218524,nop,wscale 7], length 0
16:20:41.741912 IP 10.96.100.2.54072 > 10.10.30.75.8123: Flags , seq 1832773481, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 2750693597 ecr 0,sackOK,eol], length 0
16:20:41.742240 IP 10.10.30.75.8123 > 10.96.100.2.54072: Flags [S.], seq 944871121, ack 1832773482, win 65160, options [mss 1460,sackOK,TS val 16387583 ecr 2750693597,nop,wscale 7], length 0
16:20:42.016015 IP 10.96.100.2.54073 > 10.10.30.75.8123: Flags , seq 1436938123, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 1952622307 ecr 0,sackOK,eol], length 0
16:20:42.016434 IP 10.10.30.75.8123 > 10.96.100.2.54073: Flags [S.], seq 2662819474, ack 1436938124, win 65160, options [mss 1460,sackOK,TS val 16387858 ecr 1952622307,nop,wscale 7], length 0
16:20:42.022667 IP 10.96.100.2.54074 > 10.10.30.75.8123: Flags , seq 399554212, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 180360513 ecr 0,sackOK,eol], length 0
16:20:42.022680 IP 10.96.100.2.54075 > 10.10.30.75.8123: Flags , seq 1908759648, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 1894941168 ecr 0,sackOK,eol], length 0
16:20:42.022944 IP 10.10.30.75.8123 > 10.96.100.2.54074: Flags [S.], seq 3436034975, ack 399554213, win 65160, options [mss 1460,sackOK,TS val 16387864 ecr 180360513,nop,wscale 7], length 0
16:20:42.022950 IP 10.10.30.75.8123 > 10.96.100.2.54075: Flags [S.], seq 2653459743, ack 1908759649, win 65160, options [mss 1460,sackOK,TS val 16387864 ecr 1894941168,nop,wscale 7], length 0
Working Exmaple
My iPhone is connected via the WireGuard IP: 10.96.100.2
And it try's to access a webinterface on 10.10.30.114:8080 witch is an physical appliance on the network, not hosted by Proxmox.
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
19:28:41.002091 IP 10.96.100.2.54120 > 10.10.30.114.http-alt: Flags , seq 1623234972, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 2851247756 ecr 0,sackOK,eol], length 0
19:28:41.003311 IP 10.10.30.114.http-alt > 10.96.100.2.54120: Flags [S.], seq 3721600261, ack 1623234973, win 64308, options [mss 1410,sackOK,TS val 333757866 ecr 2851247756,nop,wscale 6], length 0
19:28:41.018670 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags , seq 645618751, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 1622192869 ecr 0,sackOK,eol], length 0
19:28:41.019206 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [S.], seq 702451523, ack 645618752, win 64308, options [mss 1410,sackOK,TS val 333757882 ecr 1622192869,nop,wscale 6], length 0
19:28:41.029997 IP 10.96.100.2.54120 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 2851247815 ecr 333757866], length 0
19:28:41.048372 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192899 ecr 333757882], length 0
19:28:41.055224 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [P.], seq 1:385, ack 1, win 2057, options [nop,nop,TS val 1622192899 ecr 333757882], length 384: HTTP: GET / HTTP/1.1
19:28:41.055664 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], ack 385, win 1002, options [nop,nop,TS val 333757918 ecr 1622192899], length 0
19:28:41.057423 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 1461:2669, ack 385, win 1002, options [nop,nop,TS val 333757920 ecr 1622192899], length 1208: HTTP
19:28:41.057924 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 2669:3877, ack 385, win 1002, options [nop,nop,TS val 333757920 ecr 1622192899], length 1208: HTTP
19:28:41.058254 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 3877:5085, ack 385, win 1002, options [nop,nop,TS val 333757921 ecr 1622192899], length 1208: HTTP
19:28:41.058719 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 5085:6293, ack 385, win 1002, options [nop,nop,TS val 333757921 ecr 1622192899], length 1208: HTTP
19:28:41.059137 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 6293:7501, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.059149 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [P.], seq 7501:8709, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.059597 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 8709:9917, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.060015 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 9917:11125, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.084073 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192938 ecr 333757918,nop,nop,sack 1 {1461:2669}], length 0
19:28:41.084769 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 11125:12333, ack 385, win 1002, options [nop,nop,TS val 333757947 ecr 1622192938], length 1208: HTTP
19:28:41.099961 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192940 ecr 333757918,nop,nop,sack 1 {1461:5085}], length 0
19:28:41.099984 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192941 ecr 333757918,nop,nop,sack 1 {1461:7501}], length 0
19:28:41.099986 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192944 ecr 333757918,nop,nop,sack 1 {1461:11125}], length 0
19:28:41.100744 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 1:1209, ack 385, win 1002, options [nop,nop,TS val 333757963 ecr 1622192940], length 1208: HTTP: HTTP/1.1 200 OK
19:28:41.100750 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [P.], seq 1209:1461, ack 385, win 1002, options [nop,nop,TS val 333757963 ecr 1622192941], length 252: HTTP
Any sugggestion on where to go and look further would be amazing, thank you in advance!
If one need additional info for clarification, please ask me
Jacob
I am a sys admin currentyl being made humble again by (I think) proxmox's networking.
My setup:
WAN -> Router -> Physical Switches -> Proxmox VE (6.8.12-1) -> FreeBSD VM with WireGuard VPN
Whenever I connect with my VPN client (iPhone) to the WireGuard VPN server I experiance the following behavior:
1 (Expected) I can connect to any resource that is not hosted by the Proxmox VE.
For example: My router & Physical Switches have a Web interface, I can succesfully browse to those webpage.
I have multiple physical devices in my network, can access all of them fine just fine, as one would expect.
2 (Problem) I can not connect to any resource that is being hosted by proxmox VE.
For example: I have a webserver running on a 2nd VM within proxmox; can't access that webpage.
Also, I cant connect to the Proxmox webinterface.
To me it looks like Proxmox is unable to 'handle' the traffic on it's virtual switches (the way I would expect.)
So traffic from VM (Wireguard Free BSD) to other VM (webserver for example) is not handled right.
I ran a TCP Dump on my FreeBSD VM With Wireguard to troubleshoot this issue, and this is what I found:
Problem example
My iPhone is connected via the WireGuard IP: 10.96.100.2
And it try's to access a webinterface on 10.10.30.75:8123 witch is a VM hosted by the same proxmox VE server.
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
16:20:35.200498 IP 10.10.30.75.8123 > 10.96.100.2.54058: Flags [S.], seq 1645965074, ack 3563026976, win 65160, options [mss 1460,sackOK,TS val 16381042 ecr 3385564108,nop,wscale 7], length 0
16:20:37.760433 IP 10.10.30.75.8123 > 10.96.100.2.54066: Flags [S.], seq 703934273, ack 1988877869, win 65160, options [mss 1460,sackOK,TS val 16383602 ecr 1624876155,nop,wscale 7], length 0
16:20:41.605573 IP 10.96.100.2.54071 > 10.10.30.75.8123: Flags , seq 3662379520, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 247218524 ecr 0,sackOK,eol], length 0
16:20:41.605971 IP 10.10.30.75.8123 > 10.96.100.2.54071: Flags [S.], seq 4154632067, ack 3662379521, win 65160, options [mss 1460,sackOK,TS val 16387447 ecr 247218524,nop,wscale 7], length 0
16:20:41.741912 IP 10.96.100.2.54072 > 10.10.30.75.8123: Flags , seq 1832773481, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 2750693597 ecr 0,sackOK,eol], length 0
16:20:41.742240 IP 10.10.30.75.8123 > 10.96.100.2.54072: Flags [S.], seq 944871121, ack 1832773482, win 65160, options [mss 1460,sackOK,TS val 16387583 ecr 2750693597,nop,wscale 7], length 0
16:20:42.016015 IP 10.96.100.2.54073 > 10.10.30.75.8123: Flags , seq 1436938123, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 1952622307 ecr 0,sackOK,eol], length 0
16:20:42.016434 IP 10.10.30.75.8123 > 10.96.100.2.54073: Flags [S.], seq 2662819474, ack 1436938124, win 65160, options [mss 1460,sackOK,TS val 16387858 ecr 1952622307,nop,wscale 7], length 0
16:20:42.022667 IP 10.96.100.2.54074 > 10.10.30.75.8123: Flags , seq 399554212, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 180360513 ecr 0,sackOK,eol], length 0
16:20:42.022680 IP 10.96.100.2.54075 > 10.10.30.75.8123: Flags , seq 1908759648, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 1894941168 ecr 0,sackOK,eol], length 0
16:20:42.022944 IP 10.10.30.75.8123 > 10.96.100.2.54074: Flags [S.], seq 3436034975, ack 399554213, win 65160, options [mss 1460,sackOK,TS val 16387864 ecr 180360513,nop,wscale 7], length 0
16:20:42.022950 IP 10.10.30.75.8123 > 10.96.100.2.54075: Flags [S.], seq 2653459743, ack 1908759649, win 65160, options [mss 1460,sackOK,TS val 16387864 ecr 1894941168,nop,wscale 7], length 0
Working Exmaple
My iPhone is connected via the WireGuard IP: 10.96.100.2
And it try's to access a webinterface on 10.10.30.114:8080 witch is an physical appliance on the network, not hosted by Proxmox.
listening on wg0, link-type NULL (BSD loopback), snapshot length 262144 bytes
19:28:41.002091 IP 10.96.100.2.54120 > 10.10.30.114.http-alt: Flags , seq 1623234972, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 2851247756 ecr 0,sackOK,eol], length 0
19:28:41.003311 IP 10.10.30.114.http-alt > 10.96.100.2.54120: Flags [S.], seq 3721600261, ack 1623234973, win 64308, options [mss 1410,sackOK,TS val 333757866 ecr 2851247756,nop,wscale 6], length 0
19:28:41.018670 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags , seq 645618751, win 65535, options [mss 1220,nop,wscale 6,nop,nop,TS val 1622192869 ecr 0,sackOK,eol], length 0
19:28:41.019206 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [S.], seq 702451523, ack 645618752, win 64308, options [mss 1410,sackOK,TS val 333757882 ecr 1622192869,nop,wscale 6], length 0
19:28:41.029997 IP 10.96.100.2.54120 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 2851247815 ecr 333757866], length 0
19:28:41.048372 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192899 ecr 333757882], length 0
19:28:41.055224 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [P.], seq 1:385, ack 1, win 2057, options [nop,nop,TS val 1622192899 ecr 333757882], length 384: HTTP: GET / HTTP/1.1
19:28:41.055664 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], ack 385, win 1002, options [nop,nop,TS val 333757918 ecr 1622192899], length 0
19:28:41.057423 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 1461:2669, ack 385, win 1002, options [nop,nop,TS val 333757920 ecr 1622192899], length 1208: HTTP
19:28:41.057924 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 2669:3877, ack 385, win 1002, options [nop,nop,TS val 333757920 ecr 1622192899], length 1208: HTTP
19:28:41.058254 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 3877:5085, ack 385, win 1002, options [nop,nop,TS val 333757921 ecr 1622192899], length 1208: HTTP
19:28:41.058719 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 5085:6293, ack 385, win 1002, options [nop,nop,TS val 333757921 ecr 1622192899], length 1208: HTTP
19:28:41.059137 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 6293:7501, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.059149 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [P.], seq 7501:8709, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.059597 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 8709:9917, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.060015 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 9917:11125, ack 385, win 1002, options [nop,nop,TS val 333757922 ecr 1622192899], length 1208: HTTP
19:28:41.084073 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192938 ecr 333757918,nop,nop,sack 1 {1461:2669}], length 0
19:28:41.084769 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 11125:12333, ack 385, win 1002, options [nop,nop,TS val 333757947 ecr 1622192938], length 1208: HTTP
19:28:41.099961 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192940 ecr 333757918,nop,nop,sack 1 {1461:5085}], length 0
19:28:41.099984 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192941 ecr 333757918,nop,nop,sack 1 {1461:7501}], length 0
19:28:41.099986 IP 10.96.100.2.54121 > 10.10.30.114.http-alt: Flags [.], ack 1, win 2057, options [nop,nop,TS val 1622192944 ecr 333757918,nop,nop,sack 1 {1461:11125}], length 0
19:28:41.100744 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [.], seq 1:1209, ack 385, win 1002, options [nop,nop,TS val 333757963 ecr 1622192940], length 1208: HTTP: HTTP/1.1 200 OK
19:28:41.100750 IP 10.10.30.114.http-alt > 10.96.100.2.54121: Flags [P.], seq 1209:1461, ack 385, win 1002, options [nop,nop,TS val 333757963 ecr 1622192941], length 252: HTTP
Any sugggestion on where to go and look further would be amazing, thank you in advance!
If one need additional info for clarification, please ask me
Jacob
Last edited: