I can access to an application installed in k8s inside proxmox NAT

[Juliet]

HI.

i have an application running inside a clusert k8s (nested vms inside proxmox) i port-forward to the application service on port 443:

kubectl port-forward svc/argo-cd-argocd-server 30900:443 -n argocd

when i run curl 127.0.0.1:30900 i can see the app running :


so now i want to access the web ui from the browser..i put the : http://ip_proxmox_host:30900..but it doesnt work.

i have added the port on the nat :



can you help please ?

thanks

 

[_gabriel]

missing -s 'x.x.x.x/24' in PREROUTING rules.

 

[Juliet]

missing -s 'x.x.x.x/24' in PREROUTING rules.

where exactely please ?
what is the x.x.x.x/24 IP, proxmox host ip ?

 

[_gabriel]

same as your MASQUERADE rule.

do not add blank lines between "post-up/down", keeping one block is recommended.

+ add this recommended rule to fix some conntrack issue when firewall is enabled :
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1 post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

 

[Juliet]

same as your MASQUERADE rule.

do not add blank lines between "post-up/down", keeping one block is recommended.

+ add this recommended rule to fix some conntrack issue when firewall is enabled :
post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

i did what you suggest :


then

ifup vmbr2

but nothin happens. i'm still cant reach the UI.

 

[_gabriel]

fwbr+ rule was an extra.
you need to fix your PREROUTING DNAT rules, where missing the -s argument
don't forget to suppress blank lines between post- lines.
please skip screenshot and paste content into [CODE] tag

 

[Juliet]

hi gabriel,

i will post the code and show me please where i can add the -s :

auto vmbr2
auto vmbr2
#private sub network
iface vmbr2 inet static
        address  192.168.1.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '192.168.1.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '192.168.1.0/24' -o vmbr0 -j MASQUERADE

        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1
# redirection to the web server
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 32768 -j DNAT --to 192.168.1.2:22
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 32768 -j DNAT --to 192.168.1.2:22


        post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800
        post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800
        post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900
        post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900

thanks a lot for your help

 

[_gabriel]

I can't post the answer, because you need to understand things.
please read again, "-s" present in POSTROUTING rule need to be present in PREROUTING DNAT rules too.

EDIT: ifreload -a is recommended instead ifup

 

[Juliet]

thanks for your help , i'm trying to understand.
i tried :

ifup vmbr2
warning: vmbr2: post-up cmd 'iptables -t nat -A PREROUTING -s vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900' failed: returned 2 (iptables v1.8.9 (legacy): host/network `vmbr0' not found

but it gives the error above.

 

[_gabriel]

look at your POSTROUTING rule, then copy/paste -s '....../24' to your PREROUTING rules.

 

[Juliet]

look at your POSTROUTING rule, then copy/paste -s '....../24' to your PREROUTING rules.

just for your info, i have deployed nginx..i put the same rules and it works.

post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800

 

[Juliet]

just for your info, i have deployed nginx..i put the same rules and it works.

post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800

View attachment 71826

i dont know why with

post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900
        post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900

it doesnt work

 

[Juliet]

warning: vmbr2: post-up cmd 'iptables -t nat -A PREROUTING -i -s '192.168.1.0/24'  vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900' failed: returned 2 (Bad argument `192.168.1.0/24'
Try `iptables -h' or 'iptables --help' for more information.
)

it gives this error

 

[_gabriel]

iptables -t nat -A PREROUTING -i -s '192.168.1.0/24' vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900

interface need to be after -i

 

[Juliet]

interface need to be after -i

this time it gives no error ..but still doesnt work, i mean i dont see the web ui. i dont know why nginx work and argo-cd no.

post-up iptables -t nat -A PREROUTING -s '192.168.1.0/24' -i  vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900
        post-down iptables -t nat -D PREROUTING -s '192.168.1.0/24' -i  vmbr0 -p tcp --dport 30900 -j DNAT --to 192.168.1.2:30900

 

[_gabriel]

Désolé! I was wrong, -s seems isn't mandatory. I've not set in my rules.
Reboot host because current iptables rules is surely mixed up.
post iptables-save after host reboot.

 

[Juliet]

mabe i know whet is the issue..i mean why nginx work ..and argocd dont.

for nginx it is exposed as a nodeport :

nginx        NodePort    10.100.254.244   <none>        80:30800/TCP   25d

that's mean the 30800 is already exist in the worker nodes. and when i put the rules below it works :

post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800
             post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp --dport 30800 -j DNAT --to 192.168.1.2:30800

for argocd it has not a node port, so i have to forward the trafic like this :

kubectl port-forward svc/argo-cd-argocd-server 30900:443 -n argocd

Forwarding from 127.0.0.1:30900 -> 8080
Forwarding from [::1]:30900 -> 8080

but the port maybe does not exist in the worker node (the ufw is disables).

i dont know how to fix the issue , i have no idea

 

[_gabriel]

idk k8s, but it seems 30900 is only internal, and is exposed to external 443, --to 192.168.1.2:443 should work.

 

[Juliet]

192.168.1.2 it is not a public ip. i need to access the app using the @ip of proxmox host :

according to the doc To access the Web UI we have to port-forward to the argocd-server service on port 443

inside the vm when i put

 curl 127.0.0.1:30900

it work i can see the response..the question now how i access from the outside ?

 

[_gabriel]

idk Kubernetes world, seems there is another sub network.
but why it display 30900 -> 8080 ? if it's true, then you need iptables nat forward --to 192.168.1.2:8080 ...