Proxmox crashes / restarts after UDP: bad checksum.

[Yuuka]

Hello,
since 2 days I am experiencing my Proxmox Host system crashing, going down for no reason.
After taking a look inside the kernel log, I saw this:

Jul 27 06:03:33 prox01 kernel: UDP: bad checksum. From 92.97.115.148:8466 to "my server ip":6881 ulen 70
Jul 27 06:05:29 prox01 kernel: imklog 4.6.4, log source = /proc/kmsg started.
Jul 27 06:05:29 prox01 kernel: Initializing cgroup subsys cpuset
Jul 27 06:05:29 prox01 kernel: Initializing cgroup subsys cpu

This happens always a few minutes before the server restarts / crashes.
I googled for this and found some posts about an exploit, so I guess that this is causing the Problem.
Is there any solution to get rid of this ?
I am using Proxmox v 2.3-13
Or might upgrading to the latest verion solce this problem?

I am thankful for any advice/help.

 

[dietmar]

please update (your version is very old).

 

[Yuuka]

Alright, I got the system now running on the latest version.
But it still keeps crashing/rebooting.

Jul 28 10:26:12 prox01 kernel: CT: 105: started
Jul 28 16:52:36 prox01 kernel: UDP: bad checksum. From 123.50.76.113:53414 to "my servers ip":6881 ulen 106

Jul 28 17:27:32 prox01 kernel: imklog 4.6.4, log source = /proc/kmsg started.

Jul 28 17:27:32 prox01 kernel: Initializing cgroup subsys cpuset
Jul 28 17:27:32 prox01 kernel: Initializing cgroup subsys cpu

 

[mir]

Could you try the following on your proxmox host:
iptables -A POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill

 

[Yuuka]

I am so frustrated, I did everything I was told by you guys.
Even made a Memtest to check the Ram, which is ok.
Now I dont see the UDP Stuff anymore in the kern.log, but the Server still randomly crashes.
I also unmounted 1 HDD that had an error displayed in the Kern.log, but still the same.
After unmounting the Host was running for about 7 hrs, then it crashed twice after ~ 1 hour between each crash.
Here is a short output from the kern.log

Jul 30 12:13:04 prox01 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jul 30 12:13:05 prox01 kernel: kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL does not work properly. Using workaround
Jul 30 12:13:05 prox01 kernel: tun: Universal TUN/TAP device driver, 1.6
Jul 30 12:13:05 prox01 kernel: tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Jul 30 12:13:05 prox01 kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
Jul 30 12:13:05 prox01 kernel: Enabling conntracks and NAT for ve0
Jul 30 12:13:05 prox01 kernel: nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
Jul 30 12:13:05 prox01 kernel: ploop_dev: module loaded
Jul 30 12:13:06 prox01 kernel: vmbr0: no IPv6 routers present
Jul 30 12:13:08 prox01 kernel: eth1: no IPv6 routers present
Jul 30 12:13:15 prox01 kernel: venet0: no IPv6 routers present
Jul 30 13:06:51 prox01 kernel: imklog 5.8.11, log source = /proc/kmsg started.
Jul 30 13:06:51 prox01 kernel: Initializing cgroup subsys cpuset
Jul 30 13:06:51 prox01 kernel: Initializing cgroup subsys cpu
Jul 30 13:06:51 prox01 kernel: Linux version 2.6.32-26-pve (root@lola) (gcc version 4.7.2 (Debian 4.7.2-5) ) #1 SMP Mon Oct 14 08:22:20 CEST 2013
Jul 30 13:06:51 prox01 kernel: Command line: BOOT_IMAGE=/vmlinuz-2.6.32-26-pve root=/dev/mapper/pve-root ro quiet
Jul 30 13:06:51 prox01 kernel: KERNEL supported cpus:
Jul 30 13:06:51 prox01 kernel:  Intel GenuineIntel
Jul 30 13:06:51 prox01 kernel:  AMD AuthenticAMD
Jul 30 13:06:51 prox01 kernel:  Centaur CentaurHauls
Jul 30 13:06:51 prox01 kernel: BIOS-provided physical RAM map:
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 00000000000e4c00 - 0000000000100000 (reserved)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 0000000000100000 - 00000000bf780000 (usable)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 00000000bf780000 - 00000000bf798000 (ACPI data)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 00000000bf798000 - 00000000bf7dc000 (ACPI NVS)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 00000000bf7dc000 - 00000000c0000000 (reserved)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 00000000ffe00000 - 0000000100000000 (reserved)
Jul 30 13:06:51 prox01 kernel: BIOS-e820: 0000000100000000 - 0000000640000000 (usable)
Jul 30 13:06:51 prox01 kernel: DMI 2.5 present.
Jul 30 13:06:51 prox01 kernel: SMBIOS version 2.5 @ 0xFBD10

 

[mir]

Is there a BIOS upgrade for your motherboard?

 

[Yuuka]

I dont know, but why should this change sth, the system was running fine for 2 years now.

 

[mir]

Changes in newer kernels may require support from BIOS which is only available in newer or newest BIOS. Or new functionality introduced in the kernel but current BIOS has bug for this new feature.

 

[Yuuka]

Well I dont think that this is the case, since I got this Problem with the previous Proxmox version.

 

[udo]

Well I dont think that this is the case, since I got this Problem with the previous Proxmox version.

Hi,
I had with one server frequently reboots, until I change the power supply. After that, all work well.

You can have many sources for issues (heat due old fans) - have you look in the server-log (ipmi/bios)?

Udo

 

[Yuuka]

Thank you for the tip, I'll check it out.
But checking the PSU is a bit difficult, since the server is in a datacenter.