Hi all,
I'm running the official Proxmox VE 9.2 arm64 build in a small 3-node lab, as VMs on an Apple Silicon Mac. Everything works (cluster, nested KVM), except the Ceph daemons: they die with SIGILL as soon as they start. I found the cause and a workaround, and I'm sharing them in case they help others, and in case the Proxmox team wants to look at it.
I know this is not supported hardware (best-effort for non-Grace arm64), so this is a report, not a complaint.
Environment
Symptom
The Ceph wizard installs the packages and writes the configuration, then fails creating the first monitor:
Even printing the version crashes. Client tools are fine:
Diagnosis
It is not a missing ISA extension in Ceph's own code. gdb shows a pointer authentication instruction failing inside libgcc's unwinder, called by tcmalloc while it records a stack trace during heap growth:
The CPU does implement PAC (paca/pacg), so autia1716 is a real authentication check. My reading, which I have not verified further: the unwinder fails to authenticate a return address while walking the stack, and the CPU raises a fault on the failed check, which Linux delivers as SIGILL. I don't know whether supported hardware (Grace) avoids this, or whether the same stack walk would fault there too.
Workaround
gperftools can select the stack capture method at runtime. Frame-pointer walking avoids libgcc's unwinder:
The variable has to reach the Ceph services and pvedaemon (which runs ceph-mon/ceph-osd --mkfs for the GUI), so I set it as systemd's default environment on every node and rebooted:
After a "pveceph purge" of the failed first attempt and running the wizard again, everything works: 3 MONs in quorum, 1 active MGR + 2 standby, 3 OSDs up/in, 129 PGs active+clean, HEALTH_OK.
Questions
Happy to run more tests if useful.
I'm running the official Proxmox VE 9.2 arm64 build in a small 3-node lab, as VMs on an Apple Silicon Mac. Everything works (cluster, nested KVM), except the Ceph daemons: they die with SIGILL as soon as they start. I found the cause and a workaround, and I'm sharing them in case they help others, and in case the Proxmox team wants to look at it.
I know this is not supported hardware (best-effort for non-Grace arm64), so this is a report, not a complaint.
Environment
- Host: Apple M3 Max, macOS 26.6.2, UTM 5.0.8 (Apple Virtualization backend, nested virtualization enabled)
- Guests: Proxmox VE 9.2 arm64 (ISO 9.2-1), upgraded to pve-manager 9.2.20, kernel 7.0.14-6-pve
- Ceph: Tentacle 20.2.4 from the ceph-tentacle no-subscription repository
- CPU features in the guest: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp flagm2 frint i8mm bf16 bti afp
Symptom
The Ceph wizard installs the packages and writes the configuration, then fails creating the first monitor:
Code:
TASK ERROR: command 'ceph-mon --mkfs -i pve1 --monmap /tmp/monmap --keyring /etc/pve/priv/ceph.mon.keyring' failed: got signal 4Even printing the version crashes. Client tools are fine:
Code:
# ceph-mon --version; echo "exit=$?"
Illegal instruction
exit=132
rados exit=0
rbd exit=0
ceph-osd Illegal instruction (exit=132)
ceph-mgr Illegal instruction (exit=132)Diagnosis
It is not a missing ISA extension in Ceph's own code. gdb shows a pointer authentication instruction failing inside libgcc's unwinder, called by tcmalloc while it records a stack trace during heap growth:
Code:
# gdb -q -batch -ex run -ex 'x/i $pc' -ex bt --args ceph-mon --version
Program received signal SIGILL, Illegal instruction.
0x0000fffff67271b0 in ?? () from /lib/aarch64-linux-gnu/libgcc_s.so.1
=> 0xfffff67271b0: autia1716
#0 0x0000fffff67271b0 in ?? () from /lib/aarch64-linux-gnu/libgcc_s.so.1
#1 0x0000fffff6727adc [PAC] in _Unwind_Backtrace () from /lib/aarch64-linux-gnu/libgcc_s.so.1
#2 0x0000fffff7182ef8 [PAC] in ?? () from /lib/aarch64-linux-gnu/libtcmalloc.so.4
#3 0x0000fffff71838c4 [PAC] in GetStackTrace(void**, int, int) () from /lib/aarch64-linux-gnu/libtcmalloc.so.4
#5 0x0000fffff7178820 [PAC] in tcmalloc::ThreadCachePtr::WithStacktraceScope(void (*)(bool, void*), void*) () from /lib/aarch64-linux-gnu/libtcmalloc.so.4
#7 0x0000fffff717386c [PAC] in tcmalloc::PageHeap::HandleUnlock(tcmalloc::PageHeap::LockingContext*) () from /lib/aarch64-linux-gnu/libtcmalloc.so.4
#8 0x0000fffff7175210 [PAC] in tcmalloc::PageHeap::NewWithSizeClass(unsigned long, unsigned int) () from /lib/aarch64-linux-gnu/libtcmalloc.so.4
#13 0x0000fffff773a6c8 [PAC] in Option::Option(Option const&) () from /usr/lib/ceph/libceph-common.so.2
#14 0x0000fffff77e8550 [PAC] in get_global_options() () from /usr/lib/ceph/libceph-common.so.2
Backtrace stopped: previous frame inner to this frame (corrupt stack?)The CPU does implement PAC (paca/pacg), so autia1716 is a real authentication check. My reading, which I have not verified further: the unwinder fails to authenticate a return address while walking the stack, and the CPU raises a fault on the failed check, which Linux delivers as SIGILL. I don't know whether supported hardware (Grace) avoids this, or whether the same stack walk would fault there too.
Workaround
gperftools can select the stack capture method at runtime. Frame-pointer walking avoids libgcc's unwinder:
Code:
# for m in generic_fp generic_fp_unsafe libunwind libgcc; do printf '%-18s ' $m; TCMALLOC_STACKTRACE_METHOD=$m ceph-mon --version >/dev/null 2>&1; echo "exit=$?"; done
generic_fp exit=0
generic_fp_unsafe exit=0
libunwind Illegal instruction (not built in, falls back to libgcc)
libgcc Illegal instruction
# TCMALLOC_STACKTRACE_METHOD_VERBOSE=t TCMALLOC_STACKTRACE_METHOD=generic_fp ceph-mon --version
Chosen stacktrace method is generic_fp
Supported methods:
* libgcc
* generic_fp
* generic_fp_unsafe
* nullThe variable has to reach the Ceph services and pvedaemon (which runs ceph-mon/ceph-osd --mkfs for the GUI), so I set it as systemd's default environment on every node and rebooted:
Code:
mkdir -p /etc/systemd/system.conf.d
cat > /etc/systemd/system.conf.d/tcmalloc-stacktrace.conf <<'EOF'
[Manager]
DefaultEnvironment=TCMALLOC_STACKTRACE_METHOD=generic_fp
EOF
echo 'TCMALLOC_STACKTRACE_METHOD=generic_fp' >> /etc/environment
rebootAfter a "pveceph purge" of the failed first attempt and running the wizard again, everything works: 3 MONs in quorum, 1 active MGR + 2 standby, 3 OSDs up/in, 129 PGs active+clean, HEALTH_OK.
Questions
- Is this known on other arm64 machines with PAC enabled?
- Would it make sense for the Ceph packages (or their systemd units) to default to the generic_fp method on arm64, or to build gperftools without the libgcc stack trace backend?
Happy to run more tests if useful.