proxmox-backup-client Floating point exception(core dumped)

R

RuiSantos

New Member
Proxmox Subscriber
Aug 6, 2025
11
1
3
Hello,

I'm getting the following error when running:
Code: 
proxmox-backup-client status --repository "${REPO}"

Rich (BB code): 
systemd-coredump[87519]: [] Process 87509 (proxmox-backup-) of user 0 dumped core.
Stack trace of thread 87509:
#0  0x00007f4fb59731bb __libc_early_init (/usr/lib64/libc.so.6 + 0x1731bb)
#1  0x00007f4fb7451698 n/a (/usr/local/bin/proxmox-backup-client + 0x851698)
#2  0x00007f4fb744efb6 n/a (/usr/local/bin/proxmox-backup-client + 0x84efb6)
#3  0x00007f4fb7450855 n/a (/usr/local/bin/proxmox-backup-client + 0x850855)
#4  0x00007f4fb744efb6 n/a (/usr/local/bin/proxmox-backup-client + 0x84efb6)
#5  0x00007f4fb7450c17 n/a (/usr/local/bin/proxmox-backup-client + 0x850c17)
#6  0x00007f4fb74587b2 n/a (/usr/local/bin/proxmox-backup-client + 0x8587b2)
#7  0x00007f4fb744efb6 n/a (/usr/local/bin/proxmox-backup-client + 0x84efb6)
#8  0x00007f4fb744f04f n/a (/usr/local/bin/proxmox-backup-client + 0x84f04f)
#9  0x00007f4fb7458a30 n/a (/usr/local/bin/proxmox-backup-client + 0x858a30)
#10 0x00007f4fb7ab1085 n/a (/usr/local/bin/proxmox-backup-client + 0xeb1085)
#11 0x00007f4fb7ab1435 n/a (/usr/local/bin/proxmox-backup-client + 0xeb1435)
#12 0x00007f4fb7ab1608 n/a (/usr/local/bin/proxmox-backup-client + 0xeb1608)
#13 0x00007f4fb7aaa5f2 n/a (/usr/local/bin/proxmox-backup-client + 0xeaa5f2)
#14 0x00007f4fb7a8ae9a n/a (/usr/local/bin/proxmox-backup-client + 0xe8ae9a)
#15 0x00007f4fb7a8c7ce n/a (/usr/local/bin/proxmox-backup-client + 0xe8c7ce)
#16 0x00007f4fb718aa9d n/a (/usr/local/bin/proxmox-backup-client + 0x58aa9d)
#17 0x00007f4fb71b2919 n/a (/usr/local/bin/proxmox-backup-client + 0x5b2919)
#18 0x00007f4fb7212976 n/a (/usr/local/bin/proxmox-backup-client + 0x612976)
#19 0x00007f4fb7212563 n/a (/usr/local/bin/proxmox-backup-client + 0x612563)
#20 0x00007f4fb6e768d3 n/a (/usr/local/bin/proxmox-backup-client + 0x2768d3)
#21 0x00007f4fb6e8a309 n/a (/usr/local/bin/proxmox-backup-client + 0x28a309)
#22 0x00007f4fb6ea156d n/a (/usr/local/bin/proxmox-backup-client + 0x2a156d)
#23 0x00007f4fb6e44be9 n/a (/usr/local/bin/proxmox-backup-client + 0x244be9)
#24 0x00007f4fb7332ee6 n/a (/usr/local/bin/proxmox-backup-client + 0x732ee6)
#25 0x00007f4fb733448b n/a (/usr/local/bin/proxmox-backup-client + 0x73448b)
#26 0x00007f4fb6f6f9c1 n/a (/usr/local/bin/proxmox-backup-client + 0x36f9c1)
#27 0x00007f4fb6e5a2c6 n/a (/usr/local/bin/proxmox-backup-client + 0x25a2c6)
#28 0x00007f4fb6f49433 n/a (/usr/local/bin/proxmox-backup-client + 0x349433)
#29 0x00007f4fb7024899 n/a (/usr/local/bin/proxmox-backup-client + 0x424899)
#30 0x00007f4fb7a7e10d n/a (/usr/local/bin/proxmox-backup-client + 0xe7e10d)
#31 0x00007f4fb6e7717c n/a (/usr/local/bin/proxmox-backup-client + 0x27717c)
#32 0x00007f4fb73f1bf4 n/a (/usr/local/bin/proxmox-backup-client + 0x7f1bf4)
#33 0x00007f4fb73f3d9d n/a (/usr/local/bin/proxmox-backup-client + 0x7f3d9d)
#34 0x00007f4fb6dae001 n/a (/usr/local/bin/proxmox-backup-client + 0x1ae001)

This is a bit weird, as I'm not really sure if this is a proxmox-backup-client issue, as this happens when the job is triggered from systemd. When I trigger this particular command from the command line, all works fine, and the status is shown, along with the remaining backup process.

The jobs was however running fine until last August 22nd, where a bunch of updates were released for my OS, including kernel and glibc. From what I've read this static binary is sensitive on this.

Here's some of my current data:
  • openSUSE Slowroll (version_id 20250801)
  • proxmox-backup-client: 4.0.11 (static)
  • remote PBS Server: 4.0.14
  • glibc: 2.42-1
  • systemd 257 (257.7+suse.22.g835af70f4e)
  • Kernel 6.16.1-1.0.2.sr20250801-default

Let me know if you require any extra information, or need me to do any test.

As I've mentioned, this job works well on command line, but not from systemd service/timer.

PS: The $REPO above in the form of "<token>@<server ip address>:<datastore_name>"

Can you help on this?

Regards,
Rui
 
Last edited:
proxmox-backup-client is under /usr/bin/. and your error messages looks like trying to start /usr/local/bin/proxmox-backup-client where it's not (n/a = not available). Maybe you defined that wrong in your systemd unit ?!
 
Hi Walter, thanks you for your reply :)

The back client is however in that /usr/local/bin directory:
Bash: 
~> file /usr/local/bin/proxmox-backup-client
/usr/local/bin/proxmox-backup-client: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=c0b330782ac44169f9d295b978ddf1240ef9d4c0, for GNU/Linux 3.2.0, stripped

This is the static build, so I just copied the file there.
 
So systemd-coredump ... there's somethink wrong with your static selfbuild I assume ...
 
I haven't build it myself (although I haven't thought of that, and it could be an option too)
I've download it from Proxmox's official site:
Rich (BB code): 
http://download.proxmox.com/debian/pbs-client/dists/trixie/main/binary-amd64/proxmox-backup-client-static_4.0.11-1_amd64.deb
 
Hi,
RuiSantos said:
This is a bit weird, as I'm not really sure if this is a proxmox-backup-client issue, as this happens when the job is triggered from systemd. When I trigger this particular command from the command line, all works fine, and the status is shown, along with the remaining backup process.
Click to expand...
Can you check what's the output for ldd <path-to-executable> is for both cases and post it here? Further, please try to install the debug symbols build found at http://download.proxmox.com/debian/pbs-client/dists/trixie/main/binary-amd64/proxmox-backup-client-dbgsym_4.0.11-1_amd64.deb and check if this provides more context in the dump. Also, please share your systemd service.

Another option to debug this further would be by generating a backtrace via gdb.
 
  • Like
Reactions: waltar
Can you check what's the output for ldd <path-to-executable> is for both cases
Click to expand...
I don't know what you mean by "both cases" as the binary is just one, here's the ldd (it's also the static binary version):
Bash: 
# ldd /usr/local/bin/proxmox-backup-client
statically linked

please try to install the debug symbols build
Click to expand...
I did, but didn't get any symbols, as the binary build id's seem to not match the debugs on the package:
Bash: 
# find /usr/lib/debug/
/usr/lib/debug/
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/x86_64-linux-gnu
/usr/lib/debug/.dwz/x86_64-linux-gnu/proxmox-backup-client.debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/45
/usr/lib/debug/.build-id/45/e714b855f633e56fe9b0274f8edaae27fc29c6.debug
/usr/lib/debug/.build-id/7c
/usr/lib/debug/.build-id/7c/996d9a735520071fbb142364152a9c0cd61971.debug

# readelf -n /usr/local/bin/proxmox-backup-client | grep Build
Build ID: c0b330782ac44169f9d295b978ddf1240ef9d4c0
Seems the debug symbols are not for this static build

share your systemd service
Click to expand...
Sure, here you go:
Rich (BB code): 
# cat /etc/systemd/system/pbs-backup.service
[Unit]
Description=Backup to PBS
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=-/etc/sysconfig/pbs-backup

ExecStartPre=/usr/local/sbin/pbs-backup-wait.sh

ExecStart=/usr/local/sbin/pbs-backup.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
BTW, it crashes on the ExecStartPre part.

Another option to debug this further would be by generating a backtrace via gdb
Click to expand...
Since I was unable to make the symbols work, have a look at this strace, maybe it helps. Please check the attached traces
 

Attachments

Okay, I'm able to reproduce the floating point exception using the latest openSuse Slowroll ISO and the mentioned statically linked proxmox-backup-client version when triggered via a systemd service. From the core dump I could extract the following (relevant part only) backtrace with the debug symbols loaded
Code: 
#0  0x00007f74a7d731bb in __libc_early_init () from /lib64/libc.so.6
#1  0x00007f74c9a51698 in dl_open_worker_begin ()
#2  0x00007f74c9a4efb6 in _dl_catch_exception ()
#3  0x00007f74c9a50855 in dl_open_worker ()
#4  0x00007f74c9a4efb6 in _dl_catch_exception ()
#5  0x00007f74c9a50c17 in _dl_open ()
#6  0x00007f74c9a587b2 in do_dlopen ()
#7  0x00007f74c9a4efb6 in _dl_catch_exception ()
#8  0x00007f74c9a4f04f in _dl_catch_error ()
#9  0x00007f74c9a58a30 in __libc_dlopen_mode ()
#10 0x00007f74ca0b1085 in module_load ()
#11 0x00007f74ca0b1435 in __nss_module_get_function ()
#12 0x00007f74ca0b1608 in __nss_lookup ()
#13 0x00007f74ca0aa5f2 in getpwuid_r ()
#14 0x00007f74ca08ae9a in std::sys::pal::unix::os::home_dir::fallback () at library/std/src/sys/pal/unix/os.rs:818
#15 std::sys::pal::unix::os::home_dir::{closure#0} () at library/std/src/sys/pal/unix/os.rs:783
#16 core::option::Option<std::ffi::os_str::OsString>::or_else<std::ffi::os_str::OsString, std::sys::pal::unix::os::home_dir::{closure_env#0}> (self=..., f=...) at library/core/src/option.rs:1552
#17 std::sys::pal::unix::os::home_dir () at library/std/src/sys/pal/unix/os.rs:783
#18 0x00007f74ca08c7ce in std::env::home_dir () at library/std/src/env.rs:647
#19 0x00007f74c978aa9d in xdg::base_directories::BaseDirectories::with_env_impl<xdg::base_directories::{impl#7}::with_prefix::{closure_env#0}<&str>> (prefix=..., profile=..., env_var=<optimized out>) at /usr/share/cargo/registry/xdg-2.5.2/src/base_directories.rs:292
#20 xdg::base_directories::BaseDirectories::with_env<&str, &str, xdg::base_directories::{impl#7}::with_prefix::{closure_env#0}<&str>> (prefix=..., profile=..., env_var=0x0) at /usr/share/cargo/registry/xdg-2.5.2/src/base_directories.rs:257
#21 0x00007f74c97b140e in xdg::base_directories::BaseDirectories::with_prefix<&str> (prefix=...) at /usr/share/cargo/registry/xdg-2.5.2/src/base_directories.rs:218
#22 pbs_client::http_client::store_ticket_info (prefix=..., server=..., username=..., ticket=..., token=...) at pbs-client/src/http_client.rs:318
...

So this points at an incompatibility with nss modules, which are problematic for statically linked binaries. That is why we switched to a pure rust based DNS resolver implementation for the statically linked binary, see https://bugzilla.proxmox.com/show_bug.cgi?id=4788#c13.

So I'm afraid there is currently no workaround and we have to look for a custom implementation to avoid the getpwuid_r() call, which seems to cause the issue here. What is interesting though is that the issue does not appear under regular command invocation, indicating that differences in execution envrionment might play a role. (it's the not set HOME env variable). What is your /etc/nsswitch.config? On the reproducer VM here I have passwd: compat systemd, could very well be that the issue disappears if some modules are not being loaded.

Can you please open an issue at https://bugzilla.proxmox.com and link to this thread? Thanks.

Edit: Okay, so if I set the /etc/nsswitch.conf to passwd: files systemd the floating point exception dissapears and the proxmox-backup-client command is executed succesfully also in the systemd service context. You might consider setting this as workaround.

Edit2: Another workaround might be to explicitely set the HOME environment variable in the systemd unit file via Environment="HOME=/<path-to-users-home>", as this will skip the problematic fallback used to determine the current users home directory and is most likely the reason why normal command invocation works without issues.
 
Last edited:
  • Like
Reactions: waltar
Hi Chris!

Sorry for the late reply.

Workaround 2 did fix this particular issue, however a new issue arose:
Code: 
certificate validation failed - Certificate fingerprint was not confirmed.
Error: client error (Connect)
Caused by:
    0: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:2123:
    1: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:2123

Again this only happens when systemd executes the binary, as on plain terminal all works well.
I thought it could be a systemd thing, but if I run curl on the same setup, the chain is validated properly on systemd.
These are however self-signed certificates, in the form of CA -> subCA -> certificate

As per the Bugzilla, I've opened the ticket you've requested:
https://bugzilla.proxmox.com/show_bug.cgi?id=6765

Regards,
Rui
 
Chris, an additional info.

I've extracted the Trixie package, version 4.0.14, and checked it against ldd and all libraries are present on my OpenSUSE Slowroll (at least for now).
The entire script works pretty nice on systemd too, with this dynamic binary.

I guess this just helps confirming it's related to static version. Just FYI ;)
 
RuiSantos said:
Hi Chris!

Sorry for the late reply.

Workaround 2 did fix this particular issue, however a new issue arose:
Code: 
certificate validation failed - Certificate fingerprint was not confirmed.
Error: client error (Connect)
Caused by:
    0: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:2123:
    1: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:2123

Again this only happens when systemd executes the binary, as on plain terminal all works well.
I thought it could be a systemd thing, but if I run curl on the same setup, the chain is validated properly on systemd.
These are however self-signed certificates, in the form of CA -> subCA -> certificate

As per the Bugzilla, I've opened the ticket you've requested:
https://bugzilla.proxmox.com/show_bug.cgi?id=6765

Regards,
Rui
Click to expand...

are you specifying the fingerprint? else this might still be related, as the fingerprint that you interactively confirmed is cached locally, and that lookup might fail if environment variables are missing in systemd context.. or is the CA trusted by your system?
 
Hi Fabian,

are you specifying the fingerprint? else this might still be related, as the fingerprint that you interactively confirmed is cached locally, and that lookup might fail if environment variables are missing in systemd context.. or is the CA trusted by your system?
Click to expand...
The CA is trusted by the system. Like I mentioned above, if I run curl on that same systemd service, it works properly.
 
okay, then likely the culprit is openssl probing for the system trust store failing.. could you try once more with strace and either post it or see if it even tries to look into the system trust store? I am not super familiar with Suse, but openssl has an env var that you can point at the system trust store (SSL_CERT_DIR or SSL_CERT_FILE, the latter should point at the cert bundle), maybe setting that in the systemd context helps?
 
Hi Fabian.

From the strace I was able to find out that the binary is trying to open /usr/lib/ssl/cert.pem, I'm thinking it's predefined as part of the static linking. It fails with ENOENT, as that file is not present on OpenSUSE systems.

I was able to overcome the issue with the use of your mentioned variables. Let me just place them here for future reference for anyone getting this (this particular case was on OpenSUSE 15.6, but should apply to any):
Bash: 
export SSL_CERT_FILE="/etc/ssl/ca-bundle.pem"
or
Bash: 
export SSL_CERT_DIR="/etc/ssl/certs"
Any of these will work, both are not required.

In addition, on the strace, I was also able to see that the binary is trying to open ~/.config/proxmox-backup/fingerprints.
So I've created the file with:
Code: 
<PBS Server FDQN> <PBS Server fingerprint>
And it also worked.
NOTE: I'm using FDQN on the --repository.

I guess I can work with these to get all tasks done.

Thanks all :)
 
You must log in or register to reply here.
Top Bottom