segfault and systemctl timeout

[cagnulein]

My server has 20days of uptime and suddenly it starts to output these errors...

May 13 14:38:02 pve kernel: [1821444.558571] pvesr[10643]: segfault at 45a8bdf5ad18 ip 000055a8bb86d53d sp 00007ffe11ddfce0 error 4 in perl[55a8bb7b3000+15d000]
May 13 14:38:02 pve kernel: [1821444.558592] Code: f6 41 56 41 55 4c 8d 2d 99 9a 0b 00 41 54 55 53 48 89 fb 48 83 ec 28 48 8b 47 08 4c 8b 60 30 48 8d 87 50 01 00 00 48 89 04 24 <4d> 8b 3c 24 4c 89 a7 a0 00 00 00 4c 89 f8 83 e0 0f 48 83 f8 0d 0f
May 13 14:44:58 pve kernel: [1821860.608380] pvestatd[13139]: segfault at 457773b6f2a0 ip 00007f5cf16e0e0e sp 00007ffcff523000 error 4 in libc-2.28.so[7f5cf1683000+148000]
May 13 14:44:58 pve kernel: [1821860.608400] Code: 0f 1f 40 00 48 39 4f 60 74 52 4a 8d 34 11 f6 46 08 01 75 38 4c 01 d2 4c 39 16 75 68 48 8b 71 10 4c 8b 51 18 48 39 4e 18 75 4a <49> 39 4a 10 75 44 4c 89 56 18 49 89 72 10 49 81 f8 ff 03 00 00 0f
May 13 14:45:00 pve kernel: [1821862.485410] pvedaemon worke[30487]: segfault at 45fb0aae1838 ip 000055fb0259213d sp 00007ffd76eabec0 error 6 in perl[55fb024cb000+15d000]
May 13 14:45:00 pve kernel: [1821862.485432] Code: 02 0f 85 cd 04 00 00 44 89 44 24 0c e8 2c 87 fc ff 49 8b 14 24 44 8b 44 24 0c 48 8b 43 10 49 89 44 24 10 48 8b 03 48 8b 40 18 <48> 89 42 18 48 8b 13 49 8b 04 24 48 8b 52 10 48 89 50 10 41 81 64
May 13 14:45:00 pve kernel: [1821862.609717] pvesr[13160]: segfault at 45b727671678 ip 000055b724c14034 sp 00007fff3eba78c0 error 4 in perl[55b724b6d000+15d000]
May 13 14:45:00 pve kernel: [1821862.609742] Code: 48 8b 52 10 49 89 f7 41 89 cc 48 8d 04 c2 4c 8b 30 48 89 44 24 20 4d 85 f6 0f 84 93 00 00 00 4c 89 f3 0f 1f 40 00 48 8b 6b 08 <44> 39 65 00 75 76 4c 63 6d 04 4d 39 cd 75 6d 48 83 c5 08 4c 39 fd
May 13 14:45:07 pve kernel: [1821869.429288] pvedaemon worke[5368]: segfault at 45fb0ab0eea8 ip 000055fb0257525d sp 00007ffd76eabe90 error 4 in perl[55fb024cb000+15d000]
May 13 14:45:07 pve kernel: [1821869.429313] Code: 24 10 48 8b 50 18 4d 85 c9 0f 84 c6 09 00 00 49 8b 9f 60 08 00 00 23 94 24 d0 00 00 00 4d 8d 34 d1 48 85 db 0f 84 d3 09 00 00 <48> 8b 03 49 89 87 60 08 00 00 f6 45 0f 20 0f 85 ef 03 00 00 49 39
May 13 14:58:25 pve kernel: [1822667.353375] Initializing XFRM netlink socket
May 13 15:10:33 pve kernel: [1823395.750688] pvedaemon worke[8435]: segfault at 45fb0a419ad0 ip 000055fb025899ab sp 00007ffd76eabdb0 error 4 in perl[55fb024cb000+15d000]
May 13 15:10:33 pve kernel: [1823395.750710] Code: 4b 8d 0c ce 48 8b b1 60 08 00 00 48 89 32 48 89 91 60 08 00 00 48 39 eb 74 0b 8b 53 08 85 d2 0f 84 0c 02 00 00 48 85 c0 74 17 <8b> 50 08 85 d2 0f 84 1a 06 00 00 83 ea 01 89 50 08 0f 84 16 ff ff
May 13 15:10:36 pve kernel: [1823398.242110] pvedaemon worke[13176]: segfault at 45fb040f8588 ip 000055fb0260cede sp 00007ffd76eabdf0 error 4 in perl
May 13 15:10:36 pve kernel: [1823398.242133] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 90 41 57 41 56 41 55 49 89 fd 41 54 55 53 48 83 ec 08 48 85 f6 74 34 48 8b 06 48 85 c0 74 78 <48> 8b 40 08 48 85 c0 74 6b 48 8b 80 88 00 00 00 31 ed 48 85 c0 74
May 13 15:10:48 pve kernel: [1823410.529483] pvestatd[17917]: segfault at 4572851e81d0 ip 000055727f101334 sp 00007ffc4d114880 error 4 in perl[55727f02c000+15d000]
May 13 15:10:48 pve kernel: [1823410.529505] Code: 00 00 00 8b 46 0c 49 89 f4 48 89 fb 0f b6 c8 81 f9 ff 00 00 00 74 77 a9 00 00 20 00 75 60 48 8b ab 08 01 00 00 48 85 ed 74 44 <48> 8b 45 00 48 83 83 00 01 00 00 01 48 89 83 08 01 00 00 b9 10 00
May 13 16:09:08 pve kernel: [1826909.896673] pve-firewall[973]: segfault at 100000000008 ip 000055b9dfe13030 sp 00007ffdd202d270 error 4 in perl[55b9dfd6c000+15d000]
May 13 16:09:08 pve kernel: [1826909.896694] Code: 86 01 00 00 48 8b 52 10 49 89 f7 41 89 cc 48 8d 04 c2 4c 8b 30 48 89 44 24 20 4d 85 f6 0f 84 93 00 00 00 4c 89 f3 0f 1f 40 00 <48> 8b 6b 08 44 39 65 00 75 76 4c 63 6d 04 4d 39 cd 75 6d 48 83 c5

Now even systemctl gives me a timeout error...i think i should have to reboot the entire system, but before doing that, i would like to give you all the information that you need to understand the issue. Please, ask me if you need more info or test to do.

root@pve:~# uname -a
Linux pve 5.3.10-1-pve #1 SMP PVE 5.3.10-1 (Thu, 14 Nov 2019 10:43:13 +0100) x86_64 GNU/Linux

 

[Stoiko Ivanov]

hmm - On a hunch I would guess that this could be a broken memory module - try running memtest on the box

(segfaults in perl-code that runs on many more systems seems unlikely to be software related)

I hope this helps!