proxmox-backup-client Floating point exception(core dumped)

[RuiSantos]

Hello,

I'm getting the following error when running:

proxmox-backup-client status --repository "${REPO}"

systemd-coredump[87519]: [] Process 87509 (proxmox-backup-) of user 0 dumped core.
Stack trace of thread 87509:
#0  0x00007f4fb59731bb __libc_early_init (/usr/lib64/libc.so.6 + 0x1731bb)
#1  0x00007f4fb7451698 n/a (/usr/local/bin/proxmox-backup-client + 0x851698)
#2  0x00007f4fb744efb6 n/a (/usr/local/bin/proxmox-backup-client + 0x84efb6)
#3  0x00007f4fb7450855 n/a (/usr/local/bin/proxmox-backup-client + 0x850855)
#4  0x00007f4fb744efb6 n/a (/usr/local/bin/proxmox-backup-client + 0x84efb6)
#5  0x00007f4fb7450c17 n/a (/usr/local/bin/proxmox-backup-client + 0x850c17)
#6  0x00007f4fb74587b2 n/a (/usr/local/bin/proxmox-backup-client + 0x8587b2)
#7  0x00007f4fb744efb6 n/a (/usr/local/bin/proxmox-backup-client + 0x84efb6)
#8  0x00007f4fb744f04f n/a (/usr/local/bin/proxmox-backup-client + 0x84f04f)
#9  0x00007f4fb7458a30 n/a (/usr/local/bin/proxmox-backup-client + 0x858a30)
#10 0x00007f4fb7ab1085 n/a (/usr/local/bin/proxmox-backup-client + 0xeb1085)
#11 0x00007f4fb7ab1435 n/a (/usr/local/bin/proxmox-backup-client + 0xeb1435)
#12 0x00007f4fb7ab1608 n/a (/usr/local/bin/proxmox-backup-client + 0xeb1608)
#13 0x00007f4fb7aaa5f2 n/a (/usr/local/bin/proxmox-backup-client + 0xeaa5f2)
#14 0x00007f4fb7a8ae9a n/a (/usr/local/bin/proxmox-backup-client + 0xe8ae9a)
#15 0x00007f4fb7a8c7ce n/a (/usr/local/bin/proxmox-backup-client + 0xe8c7ce)
#16 0x00007f4fb718aa9d n/a (/usr/local/bin/proxmox-backup-client + 0x58aa9d)
#17 0x00007f4fb71b2919 n/a (/usr/local/bin/proxmox-backup-client + 0x5b2919)
#18 0x00007f4fb7212976 n/a (/usr/local/bin/proxmox-backup-client + 0x612976)
#19 0x00007f4fb7212563 n/a (/usr/local/bin/proxmox-backup-client + 0x612563)
#20 0x00007f4fb6e768d3 n/a (/usr/local/bin/proxmox-backup-client + 0x2768d3)
#21 0x00007f4fb6e8a309 n/a (/usr/local/bin/proxmox-backup-client + 0x28a309)
#22 0x00007f4fb6ea156d n/a (/usr/local/bin/proxmox-backup-client + 0x2a156d)
#23 0x00007f4fb6e44be9 n/a (/usr/local/bin/proxmox-backup-client + 0x244be9)
#24 0x00007f4fb7332ee6 n/a (/usr/local/bin/proxmox-backup-client + 0x732ee6)
#25 0x00007f4fb733448b n/a (/usr/local/bin/proxmox-backup-client + 0x73448b)
#26 0x00007f4fb6f6f9c1 n/a (/usr/local/bin/proxmox-backup-client + 0x36f9c1)
#27 0x00007f4fb6e5a2c6 n/a (/usr/local/bin/proxmox-backup-client + 0x25a2c6)
#28 0x00007f4fb6f49433 n/a (/usr/local/bin/proxmox-backup-client + 0x349433)
#29 0x00007f4fb7024899 n/a (/usr/local/bin/proxmox-backup-client + 0x424899)
#30 0x00007f4fb7a7e10d n/a (/usr/local/bin/proxmox-backup-client + 0xe7e10d)
#31 0x00007f4fb6e7717c n/a (/usr/local/bin/proxmox-backup-client + 0x27717c)
#32 0x00007f4fb73f1bf4 n/a (/usr/local/bin/proxmox-backup-client + 0x7f1bf4)
#33 0x00007f4fb73f3d9d n/a (/usr/local/bin/proxmox-backup-client + 0x7f3d9d)
#34 0x00007f4fb6dae001 n/a (/usr/local/bin/proxmox-backup-client + 0x1ae001)

This is a bit weird, as I'm not really sure if this is a proxmox-backup-client issue, as this happens when the job is triggered from systemd. When I trigger this particular command from the command line, all works fine, and the status is shown, along with the remaining backup process.

The jobs was however running fine until last August 22nd, where a bunch of updates were released for my OS, including kernel and glibc. From what I've read this static binary is sensitive on this.

Here's some of my current data:

• openSUSE Slowroll (version_id 20250801)
• proxmox-backup-client: 4.0.11 (static)
• remote PBS Server: 4.0.14
• glibc: 2.42-1
• systemd 257 (257.7+suse.22.g835af70f4e)
• Kernel 6.16.1-1.0.2.sr20250801-default

Let me know if you require any extra information, or need me to do any test.

As I've mentioned, this job works well on command line, but not from systemd service/timer.

PS: The $REPO above in the form of "<token>@<server ip address>:<datastore_name>"

Can you help on this?

Regards,
Rui

 

[waltar]

proxmox-backup-client is under /usr/bin/. and your error messages looks like trying to start /usr/local/bin/proxmox-backup-client where it's not (n/a = not available). Maybe you defined that wrong in your systemd unit ?!

 

[RuiSantos]

Hi Walter, thanks you for your reply

The back client is however in that /usr/local/bin directory:

~> file /usr/local/bin/proxmox-backup-client
/usr/local/bin/proxmox-backup-client: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, BuildID[sha1]=c0b330782ac44169f9d295b978ddf1240ef9d4c0, for GNU/Linux 3.2.0, stripped

This is the static build, so I just copied the file there.

 

[waltar]

So systemd-coredump ... there's somethink wrong with your static selfbuild I assume ...

 

[RuiSantos]

I haven't build it myself (although I haven't thought of that, and it could be an option too)
I've download it from Proxmox's official site:

http://download.proxmox.com/debian/pbs-client/dists/trixie/main/binary-amd64/proxmox-backup-client-static_4.0.11-1_amd64.deb

 

[Chris]

Hi,

This is a bit weird, as I'm not really sure if this is a proxmox-backup-client issue, as this happens when the job is triggered from systemd. When I trigger this particular command from the command line, all works fine, and the status is shown, along with the remaining backup process.

Can you check what's the output for ldd <path-to-executable> is for both cases and post it here? Further, please try to install the debug symbols build found at http://download.proxmox.com/debian/pbs-client/dists/trixie/main/binary-amd64/proxmox-backup-client-dbgsym_4.0.11-1_amd64.deb and check if this provides more context in the dump. Also, please share your systemd service.

Another option to debug this further would be by generating a backtrace via gdb.

 

[RuiSantos]

Can you check what's the output for ldd <path-to-executable> is for both cases

I don't know what you mean by "both cases" as the binary is just one, here's the ldd (it's also the static binary version):

# ldd /usr/local/bin/proxmox-backup-client
statically linked

please try to install the debug symbols build

I did, but didn't get any symbols, as the binary build id's seem to not match the debugs on the package:

# find /usr/lib/debug/
/usr/lib/debug/
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/x86_64-linux-gnu
/usr/lib/debug/.dwz/x86_64-linux-gnu/proxmox-backup-client.debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/45
/usr/lib/debug/.build-id/45/e714b855f633e56fe9b0274f8edaae27fc29c6.debug
/usr/lib/debug/.build-id/7c
/usr/lib/debug/.build-id/7c/996d9a735520071fbb142364152a9c0cd61971.debug

# readelf -n /usr/local/bin/proxmox-backup-client | grep Build
Build ID: c0b330782ac44169f9d295b978ddf1240ef9d4c0

Seems the debug symbols are not for this static build

share your systemd service

Sure, here you go:

# cat /etc/systemd/system/pbs-backup.service
[Unit]
Description=Backup to PBS
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=-/etc/sysconfig/pbs-backup

ExecStartPre=/usr/local/sbin/pbs-backup-wait.sh

ExecStart=/usr/local/sbin/pbs-backup.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

BTW, it crashes on the ExecStartPre part.

Another option to debug this further would be by generating a backtrace via gdb

Since I was unable to make the symbols work, have a look at this strace, maybe it helps. Please check the attached traces

 

[Chris]

Okay, I'm able to reproduce the floating point exception using the latest openSuse Slowroll ISO and the mentioned statically linked proxmox-backup-client version when triggered via a systemd service. From the core dump I could extract the following (relevant part only) backtrace with the debug symbols loaded

#0  0x00007f74a7d731bb in __libc_early_init () from /lib64/libc.so.6
#1  0x00007f74c9a51698 in dl_open_worker_begin ()
#2  0x00007f74c9a4efb6 in _dl_catch_exception ()
#3  0x00007f74c9a50855 in dl_open_worker ()
#4  0x00007f74c9a4efb6 in _dl_catch_exception ()
#5  0x00007f74c9a50c17 in _dl_open ()
#6  0x00007f74c9a587b2 in do_dlopen ()
#7  0x00007f74c9a4efb6 in _dl_catch_exception ()
#8  0x00007f74c9a4f04f in _dl_catch_error ()
#9  0x00007f74c9a58a30 in __libc_dlopen_mode ()
#10 0x00007f74ca0b1085 in module_load ()
#11 0x00007f74ca0b1435 in __nss_module_get_function ()
#12 0x00007f74ca0b1608 in __nss_lookup ()
#13 0x00007f74ca0aa5f2 in getpwuid_r ()
#14 0x00007f74ca08ae9a in std::sys::pal::unix::os::home_dir::fallback () at library/std/src/sys/pal/unix/os.rs:818
#15 std::sys::pal::unix::os::home_dir::{closure#0} () at library/std/src/sys/pal/unix/os.rs:783
#16 core::option::Option<std::ffi::os_str::OsString>::or_else<std::ffi::os_str::OsString, std::sys::pal::unix::os::home_dir::{closure_env#0}> (self=..., f=...) at library/core/src/option.rs:1552
#17 std::sys::pal::unix::os::home_dir () at library/std/src/sys/pal/unix/os.rs:783
#18 0x00007f74ca08c7ce in std::env::home_dir () at library/std/src/env.rs:647
#19 0x00007f74c978aa9d in xdg::base_directories::BaseDirectories::with_env_impl<xdg::base_directories::{impl#7}::with_prefix::{closure_env#0}<&str>> (prefix=..., profile=..., env_var=<optimized out>) at /usr/share/cargo/registry/xdg-2.5.2/src/base_directories.rs:292
#20 xdg::base_directories::BaseDirectories::with_env<&str, &str, xdg::base_directories::{impl#7}::with_prefix::{closure_env#0}<&str>> (prefix=..., profile=..., env_var=0x0) at /usr/share/cargo/registry/xdg-2.5.2/src/base_directories.rs:257
#21 0x00007f74c97b140e in xdg::base_directories::BaseDirectories::with_prefix<&str> (prefix=...) at /usr/share/cargo/registry/xdg-2.5.2/src/base_directories.rs:218
#22 pbs_client::http_client::store_ticket_info (prefix=..., server=..., username=..., ticket=..., token=...) at pbs-client/src/http_client.rs:318
...

So this points at an incompatibility with nss modules, which are problematic for statically linked binaries. That is why we switched to a pure rust based DNS resolver implementation for the statically linked binary, see https://bugzilla.proxmox.com/show_bug.cgi?id=4788#c13.

So I'm afraid there is currently no workaround and we have to look for a custom implementation to avoid the getpwuid_r() call, which seems to cause the issue here. What is interesting though is that the issue does not appear under regular command invocation, indicating that differences in execution envrionment might play a role. (it's the not set HOME env variable). What is your /etc/nsswitch.config? On the reproducer VM here I have passwd: compat systemd, could very well be that the issue disappears if some modules are not being loaded.

Can you please open an issue at https://bugzilla.proxmox.com and link to this thread? Thanks.

Edit: Okay, so if I set the /etc/nsswitch.conf to passwd: files systemd the floating point exception dissapears and the proxmox-backup-client command is executed succesfully also in the systemd service context. You might consider setting this as workaround.

Edit2: Another workaround might be to explicitely set the HOME environment variable in the systemd unit file via Environment="HOME=/<path-to-users-home>", as this will skip the problematic fallback used to determine the current users home directory and is most likely the reason why normal command invocation works without issues.