Hello all,
I have an LXC running the Gitea binary. (I am planning on running act_runner and docker alongside it). I also have an nginx LXC on a separate VLAN (same Proxmox server) that acts as a proxy and ssl termination point. This setup has been working with no issue for a month.
Last night, I was making some changes to a git repository and committing and pushing the changes. After making about 50+ changes, the Gitea Web UI stopped responding.
When running a packet capture in the container, I see syn packets come in from the nginx proxy but Gitea is not responding. The behavior is the same if I stop Gitea and start up netcat on port 3000. Port 3000 is the Web UI port.
If I target the Gitea container directly from my local machine, the connection to the Web UI works. So, my question is, where should I look on the Gitea LXC to determine why the traffic isn't reaching the binary listening on port 3000? Is there a conntrack table that is full? I built a new Gitea LXC with the same configuration and a different IP that is currently working as expected.
Packet Capture:
lsof output
I have an LXC running the Gitea binary. (I am planning on running act_runner and docker alongside it). I also have an nginx LXC on a separate VLAN (same Proxmox server) that acts as a proxy and ssl termination point. This setup has been working with no issue for a month.
Last night, I was making some changes to a git repository and committing and pushing the changes. After making about 50+ changes, the Gitea Web UI stopped responding.
When running a packet capture in the container, I see syn packets come in from the nginx proxy but Gitea is not responding. The behavior is the same if I stop Gitea and start up netcat on port 3000. Port 3000 is the Web UI port.
If I target the Gitea container directly from my local machine, the connection to the Web UI works. So, my question is, where should I look on the Gitea LXC to determine why the traffic isn't reaching the binary listening on port 3000? Is there a conntrack table that is full? I built a new Gitea LXC with the same configuration and a different IP that is currently working as expected.
Packet Capture:
Code:
root@gitea:~# tcpdump -enni eth0 port 3000
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:19:48.894761 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.6.40302 > 192.168.10.122.3000: Flags [S], seq 3469146150, win 64240, options [mss 1460,sackOK,TS val 3644842041 ecr 0,nop,wscale 7], length 0
03:19:49.921417 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.6.40302 > 192.168.10.122.3000: Flags [S], seq 3469146150, win 64240, options [mss 1460,sackOK,TS val 3644843068 ecr 0,nop,wscale 7], length 0
03:19:51.933358 c4:ad:34:17:8e:c2 > 76:e3:61:71:16:2a, ethertype IPv4 (0x0800), length 74: 192.168.20.6.40302 > 192.168.10.122.3000: Flags [S], seq 3469146150, win 64240, options [mss 1460,sackOK,TS val 3644845080 ecr 0,nop,wscale 7], length 0
[CODE]
Gitea LXC:
[CODE]root@bigbear:~ # cat /etc/pve/lxc/122.conf
arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: gitea
memory: 2048
mp0: /dev/zvol/argon/vm-122-disk-1,mp=/var/lib/docker
net0: name=eth0,bridge=vmbr1,firewall=1,gw=192.168.10.1,hwaddr=76:E3:61:71:16:2A,ip=192.168.10.122/24,tag=10,type=veth
ostype: debian
rootfs: argon:122/vm-122-disk-0.raw,size=100G,mountoptions=noatime
swap: 512
unprivileged: 1lsof output
Code:
root@gitea:~# lsof -i :3000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
gitea 378 git 15u IPv6 175428992 0t0 TCP *:3000 (LISTEN)
[CODE]
nginx Configuration:
[CODE]
server {
listen 443 ssl http2;
server_name git.<domain>;
ssl_certificate /etc/letsencrypt/live/<domain>fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/<domain>/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
access_log /var/log/nginx/git-access.log;
error_log /var/log/nginx/git-error.log warn;
location / {
proxy_pass http://192.168.10.122:3000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Real-IP $http_x_forwarded_for;
client_max_body_size 400M;
}
}
[CODE]
PVE Information:
[CODE]root@bigbear:~ # pveversion -v
proxmox-ve: 8.0.2 (running kernel: 6.2.16-11-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-18-pve: 6.2.16-18
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
proxmox-kernel-6.2.16-11-pve: 6.2.16-11
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.26-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.3-1
proxmox-backup-file-restore: 3.0.3-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1