LXC Container not acking traffic

kcrawford

Renowned Member
Nov 29, 2013
21
0
66
Hello all,

I have an LXC running the Gitea binary. (I am planning on running act_runner and docker alongside it). I also have an nginx LXC on a separate VLAN (same Proxmox server) that acts as a proxy and ssl termination point. This setup has been working with no issue for a month.

Last night, I was making some changes to a git repository and committing and pushing the changes. After making about 50+ changes, the Gitea Web UI stopped responding.

When running a packet capture in the container, I see syn packets come in from the nginx proxy but Gitea is not responding. The behavior is the same if I stop Gitea and start up netcat on port 3000. Port 3000 is the Web UI port.

If I target the Gitea container directly from my local machine, the connection to the Web UI works. So, my question is, where should I look on the Gitea LXC to determine why the traffic isn't reaching the binary listening on port 3000? Is there a conntrack table that is full? I built a new Gitea LXC with the same configuration and a different IP that is currently working as expected.



Packet Capture:

Code:
root@gitea:~# tcpdump -enni eth0 port 3000
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:19:48.894761 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.6.40302 > 192.168.10.122.3000: Flags [S], seq 3469146150, win 64240, options [mss 1460,sackOK,TS val 3644842041 ecr 0,nop,wscale 7], length 0
03:19:49.921417 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.6.40302 > 192.168.10.122.3000: Flags [S], seq 3469146150, win 64240, options [mss 1460,sackOK,TS val 3644843068 ecr 0,nop,wscale 7], length 0
03:19:51.933358 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.6.40302 > 192.168.10.122.3000: Flags [S], seq 3469146150, win 64240, options [mss 1460,sackOK,TS val 3644845080 ecr 0,nop,wscale 7], length 0
[CODE]

Gitea LXC:

[CODE]root@bigbear:~ # cat /etc/pve/lxc/122.conf
arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: gitea
memory: 2048
mp0: /dev/zvol/argon/vm-122-disk-1,mp=/var/lib/docker
net0: name=eth0,bridge=vmbr1,firewall=1,gw=192.168.10.1,hwaddr=76:E3:61:71:16:2A,ip=192.168.10.122/24,tag=10,type=veth
ostype: debian
rootfs: argon:122/vm-122-disk-0.raw,size=100G,mountoptions=noatime
swap: 512
unprivileged: 1

lsof output

Code:
root@gitea:~# lsof -i :3000
COMMAND PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
gitea   378  git   15u  IPv6 175428992      0t0  TCP *:3000 (LISTEN)
[CODE]


nginx Configuration:

[CODE]
server {
    listen 443 ssl http2;
    server_name git.<domain>;

    ssl_certificate /etc/letsencrypt/live/<domain>fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/<domain>/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    access_log /var/log/nginx/git-access.log;
    error_log /var/log/nginx/git-error.log warn;

    location / {

        proxy_pass http://192.168.10.122:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Real-IP $http_x_forwarded_for;
        client_max_body_size 400M;
     }

}
[CODE]

PVE Information:

[CODE]root@bigbear:~ # pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-11-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-18-pve: 6.2.16-18
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
proxmox-kernel-6.2.16-11-pve: 6.2.16-11
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.26-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.3-1
proxmox-backup-file-restore: 3.0.3-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1
 
If you start a nc on port 3000, you have the same situation? That sound to me that the forwarding from nginx is not working correctly?
Do you see any issues in the journal on the host and inside the LXC?

Are you trying to run Docker inside an (unprivileged) LXC?

You can upload the journal files to get a better picture.
Code:
journalctl --since "2023-11-10" > $(hostname)-journal.log

Also, does /var/log/nginx/git-error.log tell anything relevant?

In general, it's not recommended to run Docker inside LXC. If you setup a dedicated VM your setup will be more robust. Especially as you're planning to run act which heavily uses Docker.
 
Redoing the curl test with netcat listening:

Code:
root@gitea:~# systemctl stop docker.socket docker.service gitea.service
root@gitea:~# !nc
nc -l 3000
^C

Code:
root@nginx-proxy:~# curl -vk 192.168.10.122:3000
*   Trying 192.168.10.122:3000...
^C

Code:
root@gitea:~# !tcp
tcpdump -enni eth0 port 3000
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:18:18.654466 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.200.56878 > 192.168.10.122.3000: Flags [S], seq 4259599944, win 64240, options [mss 1460,sackOK,TS val 1016590414 ecr 0,nop,wscale 7], length 0
16:18:19.664086 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.200.56878 > 192.168.10.122.3000: Flags [S], seq 4259599944, win 64240, options [mss 1460,sackOK,TS val 1016591424 ecr 0,nop,wscale 7], length 0
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

Nothing strange in the journalctl logs. I do need to investigate the amount of bruteforce attempts I am seeing however.

I was able to resolve this for now by removing the interface for the gitea container in the Proxmox Web UI and re-adding the interface. If this does continue to happen in the future, I will report back here.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!