After Upgrade to PBS4: pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream

R

roadrunner_rad

Well-Known Member
Aug 6, 2019
54
5
48
63
After Upgrading my pbs vm from 3 to 4 i get erros in /var/log/syslog and /var/log/user.log filling up the space in /var/log so that the server becomes unresponsive.

Code: 
root@pbs:/var/log# df -hl /root
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pbs-root  7.8G  7.6G     0 100% /


Code: 
root@pbs:/var/log# tail syslog
2025-10-22T20:29:39.402512+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402544+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402561+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402582+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402601+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402621+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402637+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402658+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402674+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:29:39.402690+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
root@pbs:/var/log# tail user.log
2025-10-22T20:30:00.223326+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223344+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223361+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223379+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223396+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223414+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223437+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223454+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223472+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
2025-10-22T20:30:00.223491+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream

PBS Server Versions:

Code: 
root@pbs:/var/log# proxmox-backup-manager versions --verbose
proxmox-backup                      4.0.0         running kernel: 6.14.11-4-pve
proxmox-backup-server               4.0.16-1      running version: 4.0.16
proxmox-kernel-helper               9.0.4
proxmox-kernel-6.14.11-4-pve-signed 6.14.11-4
proxmox-kernel-6.14                 6.14.11-4
proxmox-kernel-6.8                  6.8.12-16
proxmox-kernel-6.8.12-16-pve-signed 6.8.12-16
proxmox-kernel-6.8.12-14-pve-signed 6.8.12-14
ifupdown2                           3.3.0-1+pmx10
libjs-extjs                         7.0.0-5
proxmox-backup-docs                 4.0.16-1
proxmox-backup-client               4.0.16-1
proxmox-mail-forward                1.0.2
proxmox-mini-journalreader          1.6
proxmox-offline-mirror-helper       0.7.2
proxmox-widget-toolkit              5.0.6
pve-xtermjs                         5.5.0-2
smartmontools                       7.4-pve1
zfsutils-linux                      2.3.4-pve1

Client side backup log looks like this:

Code: 
INFO: notified via target `<backup@***********>`
TASK ERROR: could not activate storage 'BACKUPVM': BACKUPVM: error fetching datastores - 500 Can't connect to 192.168.111.148:8007 (Connection timed out

VM Cofiguration:

Code: 
agent: 1
balloon: 0
boot: order=scsi0
cores: 2
cpu: kvm64,flags=+aes
hostpci1: 0000:85:00.0
ide1: none,media=cdrom
memory: 32768
name: pbs
net0: virtio=A2:02:BF:7E:9D:EA,bridge=vmbr0
numa: 0
ostype: l26
scsi0: VMVGR6:vm-998-disk-0,aio=native,discard=on,size=10G
scsi1: BKUPSTOR:vm-998-disk-0,aio=native,backup=0,size=2900G
scsihw: virtio-scsi-pci
smbios1: uuid=56bd6f7d-1df5-46ca-a7c3-bb78caa56171
sockets: 2
vmgenid: 0ce923fe-9d56-47b0-b1a6-cc42d91771d8


Any idea what is going wrong here ?
 
What systemctl status proxmox-backup-proxy is saying?

Any clue from journalctl | grep proxmox-backup-proxy (other than couldn't set up TCP socket: error while accepting tcp stream )

Any clue from the logs under /var/log/proxmox-backup/ ?

What is the output of ss -lptn | grep 8007 ?

Edit: other than the above issue, the PBS is correctly accessible via network? You are logged to sshd, right?
 
Last edited:
can't find anything unusual in /var/log/proxmox-backup/api/ ...

Code: 
root@pbs:~# systemctl status proxmox-backup-proxy
● proxmox-backup-proxy.service - Proxmox Backup API Proxy Server
     Loaded: loaded (/usr/lib/systemd/system/proxmox-backup-proxy.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/proxmox-backup-proxy.service.d
             └─override.conf
     Active: active (running) since Wed 2025-10-22 21:53:35 CEST; 3s ago
 Invocation: 65225b94b8674151b56a805611da2690
   Main PID: 1423 (proxmox-backup-)
      Tasks: 8 (limit: 38430)
     Memory: 7.6M (peak: 8.1M)
        CPU: 58ms
     CGroup: /system.slice/proxmox-backup-proxy.service
             └─1423 /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy

Oct 22 21:53:35 pbs systemd[1]: Starting proxmox-backup-proxy.service - Proxmox Backup API Proxy Server...
Oct 22 21:53:35 pbs proxmox-backup-proxy[1423]: catching shutdown signal
Oct 22 21:53:35 pbs proxmox-backup-proxy[1423]: catching reload signal
Oct 22 21:53:35 pbs systemd[1]: Started proxmox-backup-proxy.service - Proxmox Backup API Proxy Server.
Oct 22 21:53:35 pbs proxmox-backup-proxy[1423]: applied rrd journal (452 entries in 0.043 seconds)
Oct 22 21:53:35 pbs proxmox-backup-proxy[1423]: rrd journal successfully committed (25 files in 0.135 seconds)
Code: 
root@pbs:~#  ss -lptn | grep 8007
LISTEN 0      1024               *:8007            *:*    users:(("proxmox-backup-",pid=1423,fd=17),("proxmox-backup-",pid=1423,fd=16))

journalctl shows lot's of No space left on device (os error 28) when the filesysten gets full due to syslog and user.log
 
roadrunner_rad said:
journalctl shows lot's of No space left on device (os error 28) when the filesysten gets full due to syslog and user.log
Click to expand...
I think we can start with making a room in the filesystem.
You are successfully connected to the PBS via sshd, right? Or have a physical access to the host?

If there is no other valuable data in /var/log/syslog and in /var/log/users.log you may zero them, e.g. with
cd /var/log
echo > syslog
echo > users.log
then restart the service with systemctl restart proxmox-backup-proxy
and see if there is some change.

If this doesn't help, you can try what the manual says:

For debugging, you can start the daemon in foreground using:

proxmox-backup-proxy

NOTE:
You need to stop the service before starting the daemon in foreground.
 
Last edited:
roadrunner_rad said:
2025-10-22T20:29:39.402512+02:00 pbs proxmox-backup-proxy[919]: couldn't set up TCP socket: error while accepting tcp stream
Click to expand...
These errors are logged by the proxy whenever a client establishes a connection, but disappears without sending any data. Such connections are typically performed by uptime monitoring tools or reverse proxys to check if the backend is reachable. Given the high frequency of such events in your case, the logs are spammed.

Is this PBS instance located behind a reverse proxy or do you even have it exposed to the internet somehow, which could explain the high frequency of such events?
 
Chris said:
These errors are logged by the proxy whenever a client establishes a connection, but disappears without sending any data. Such connections are typically performed by uptime monitoring tools or reverse proxys to check if the backend is reachable. Given the high frequency of such events in your case, the logs are spammed.

Is this PBS instance located behind a reverse proxy or do you even have it exposed to the internet somehow, which could explain the high frequency of such events?
Click to expand...
Hi Chris,

Thanks for your reply. Yes the pbs is exposed to the internet behind a reverse proxy (apache. But this situation appears when (host) backups are running. These backups are internal, means that they run towards the local address of the server directly (not traversing the reverse proxy.

To be sure I disable the path via reverse proxy to eliminate log spamming from outside for now and get back here.
 
You must log in or register to reply here.
Top Bottom