Kernel panic: ip_set_hash_net

[Don Daniello]

We have a server that is typically running for 6 months between reboots to upgrade kernel. In other words, it's very stable and has been running for 4 years like that. Recently, we've had an unexpected hangup/crash-reboot. Upon that, a new kernel was loaded but the situation re-occured. Then a new kernel was loaded again (we install all apt upgrades including new PVE kernels very frequently, so whenever a reboot happens, new kernel is used).

Upon enabling journald persistance (it wasn't enabled at first for some reason - which may not be PVE's fault), we found that there are two symptoms:
1. Around 15 seconds before the crash/lockup, the "Proxmox VE replication runner" started. We don't use clustering/HA/replication, it's a standalone/single node.
2. The call trace for kernel panic typically refers to ipset

Some additional facts:
- We use firewall feature in PVE on this host but the number of rules is very log (20 total or so). Another host that does not use Firewall features does not crash.
- Load is always "healthy" (1 - 2). The host is never overloaded or attacked.
- All containers are unprivileged. 12 containers running, 1 KVM machine.

Here are a few crash logs, from most recent to oldest

root@pve:~# journalctl -b -1
Dec 26 18:47:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:47:01 pve CRON[28187]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 26 18:47:01 pve CRON[28188]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 26 18:47:01 pve CRON[28187]: pam_unix(cron:session): session closed for user root
Dec 26 18:47:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 26 18:47:24 pve kernel: watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ipset:26626]
Dec 26 18:47:24 pve kernel: Modules linked in: ip6t_rpfilter ebtable_nat ebtable_broute ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat_ipv4 nf
Dec 26 18:47:24 pve kernel:  cryptd snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ppdev eeepc_wmi asus_wmi wmi_bmof sparse_keymap intel_cstate intel_rapl_perf i915 snd_h
Dec 26 18:47:24 pve kernel:  megaraid_sas
Dec 26 18:47:24 pve kernel: CPU: 7 PID: 26626 Comm: ipset Tainted: P           O L   4.15.18-9-pve #1
Dec 26 18:47:24 pve kernel: Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1401 08/20/2013
Dec 26 18:47:24 pve kernel: RIP: 0010:ip_set_range_to_cidr+0x12/0x49 [ip_set]
Dec 26 18:47:24 pve kernel: RSP: 0018:ffffb30161067810 EFLAGS: 00000213 ORIG_RAX: ffffffffffffff11
Dec 26 18:47:24 pve kernel: RAX: 0000000080000000 RBX: ffffb30161067850 RCX: 000000000ab3973a
Dec 26 18:47:24 pve kernel: RDX: ffffb3016106784f RSI: 00000000ffffffff RDI: 000000000ab3973a
Dec 26 18:47:24 pve kernel: RBP: ffffb301610678c0 R08: ffffffffc0a51b00 R09: 0000000000000000
Dec 26 18:47:24 pve kernel: R10: 0000000000000021 R11: ffffb30161067848 R12: 0000000000000001
Dec 26 18:47:24 pve kernel: R13: 00000000ffffffff R14: ffff9818669b6780 R15: 000000000ab39739
Dec 26 18:47:24 pve kernel: FS:  00007fd1d0247740(0000) GS:ffff98199fbc0000(0000) knlGS:0000000000000000
Dec 26 18:47:24 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 26 18:47:24 pve kernel: CR2: 00007f4e382223a0 CR3: 00000007dbd90002 CR4: 00000000001606e0
Dec 26 18:47:24 pve kernel: Call Trace:
Dec 26 18:47:24 pve kernel:  ? hash_net4_uadt+0x200/0x330 [ip_set_hash_net]
Dec 26 18:47:24 pve kernel:  ? hash_net6_add+0x800/0x800 [ip_set_hash_net]
Dec 26 18:47:24 pve kernel:  call_ad+0xc4/0x220 [ip_set]
Dec 26 18:47:24 pve kernel:  ? hash_net6_uadt+0x200/0x200 [ip_set_hash_net]
Dec 26 18:47:24 pve kernel:  ? call_ad+0xc4/0x220 [ip_set]
Dec 26 18:47:24 pve kernel:  ? validate_nla+0x15b/0x240
Dec 26 18:47:24 pve kernel:  ? nla_parse+0xaa/0x140
Dec 26 18:47:24 pve kernel:  ip_set_uadd+0x23c/0x2a0 [ip_set]
Dec 26 18:47:24 pve kernel:  nfnetlink_rcv_msg+0x240/0x260 [nfnetlink]
Dec 26 18:47:24 pve kernel:  ? nfnetlink_net_exit_batch+0x70/0x70 [nfnetlink]
Dec 26 18:47:24 pve kernel:  netlink_rcv_skb+0xd9/0x110
Dec 26 18:47:24 pve kernel:  nfnetlink_rcv+0x6c/0x72a [nfnetlink]
Dec 26 18:47:24 pve kernel:  ? __slab_alloc+0x20/0x40
Dec 26 18:47:24 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 26 18:47:24 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 26 18:47:24 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 26 18:47:24 pve kernel:  netlink_unicast+0x19c/0x240
Dec 26 18:47:24 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 26 18:47:24 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 26 18:47:24 pve kernel:  SYSC_sendto+0x101/0x190
Dec 26 18:47:24 pve kernel:  ? __secure_computing+0x3f/0x100
Dec 26 18:47:24 pve kernel:  ? syscall_trace_enter+0xca/0x2e0
Dec 26 18:47:24 pve kernel:  SyS_sendto+0xe/0x10
Dec 26 18:47:24 pve kernel:  do_syscall_64+0x73/0x130
Dec 26 18:47:24 pve kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Dec 26 18:47:24 pve kernel: RIP: 0033:0x7fd1d034708b
Dec 26 18:47:24 pve kernel: RSP: 002b:00007ffe189e6578 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 26 18:47:24 pve kernel: RAX: ffffffffffffffda RBX: 000055cb33a5af70 RCX: 00007fd1d034708b
Dec 26 18:47:24 pve kernel: RDX: 00000000000004dc RSI: 000055cb33a58c78 RDI: 0000000000000003
Dec 26 18:47:24 pve kernel: RBP: 000055cb33a58c78 R08: 00007fd1d0412000 R09: 000000000000000c
Dec 26 18:47:24 pve kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
Dec 26 18:47:24 pve kernel: R13: 000055cb3308c137 R14: 000055cb3308c13f R15: 0000000000000000
Dec 26 18:47:24 pve kernel: Code: 48 8d 4d f7 0f b6 f2 48 89 c2 e8 4a fe ff ff eb c2 e8 13 b9 a3 fa 0f 1f 00 0f 1f 44 00 00 55 49 c7 c0 00 1b a5 c0 b8 00 00 00 80 <b9> 01 00 00 00 48 89 
Dec 26 18:48:03 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:48:21 pve CRON[28585]: pam_unix(cron:session): session opened for user root by (uid=0)

root@pve:~# journalctl -b -2
Dec 20 18:21:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 20 18:21:01 pve CRON[13307]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:21:01 pve CRON[13308]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:21:02 pve CRON[13307]: pam_unix(cron:session): session closed for user root
Dec 20 18:21:02 pve systemd[1]: Started Proxmox VE replication runner.
Dec 20 18:21:36 pve pvestatd[1836]: status update time (5.419 seconds)
Dec 20 18:22:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 20 18:22:01 pve CRON[13836]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:22:01 pve CRON[13837]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:22:05 pve CRON[13836]: pam_unix(cron:session): session closed for user root
Dec 20 18:22:29 pve pvestatd[1836]: got timeout
Dec 20 18:22:36 pve sshd[13958]: Did not receive identification string from 182.254.227.182 port 7109
Dec 20 18:23:12 pve pve-ha-lrm[1982]: loop take too long (57 seconds)
Dec 20 18:23:12 pve CRON[13963]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:23:18 pve CRON[13976]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:24:18 pve kernel: INFO: task ipset:13217 blocked for more than 120 seconds.
Dec 20 18:24:24 pve kernel:       Tainted: P           O L   4.15.18-9-pve #1
Dec 20 18:24:31 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 20 18:24:34 pve kernel: ipset           D    0 13217   1829 0x00000000
Dec 20 18:24:34 pve kernel: Call Trace:
Dec 20 18:24:34 pve kernel:  __schedule+0x3e0/0x870
Dec 20 18:24:34 pve kernel:  schedule+0x36/0x80
Dec 20 18:24:34 pve kernel:  schedule_preempt_disabled+0xe/0x10
Dec 20 18:24:34 pve kernel:  __mutex_lock.isra.2+0x2b1/0x4e0
Dec 20 18:24:34 pve kernel:  ? mem_cgroup_commit_charge+0x82/0x540
Dec 20 18:24:34 pve kernel:  ? lru_cache_add_active_or_unevictable+0x36/0xb0
Dec 20 18:24:34 pve kernel:  __mutex_lock_slowpath+0x13/0x20
Dec 20 18:24:47 pve kernel:  ? __mutex_lock_slowpath+0x13/0x20
Dec 20 18:24:48 pve kernel:  mutex_lock+0x2f/0x40
Dec 20 18:24:50 pve kernel:  nfnetlink_rcv_msg+0x1ab/0x260 [nfnetlink]
Dec 20 18:25:08 pve kernel:  ? nfnetlink_net_exit_batch+0x70/0x70 [nfnetlink]
Dec 20 18:25:13 pve kernel:  netlink_rcv_skb+0xd9/0x110
Dec 20 18:25:24 pve kernel:  nfnetlink_rcv+0x6c/0x72a [nfnetlink]
Dec 20 18:25:27 pve kernel:  ? walk_system_ram_range+0xb2/0xe0
Dec 20 18:25:28 pve kernel:  ? memcg_kmem_charge_memcg+0x7d/0xa0
Dec 20 18:25:40 pve kernel:  ? _cond_resched+0x1a/0x50
Dec 20 18:25:44 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 20 18:25:53 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 20 18:26:07 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 20 18:26:09 pve kernel:  netlink_unicast+0x19c/0x240
Dec 20 18:26:19 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 20 18:26:24 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 20 18:26:27 pve kernel:  SYSC_sendto+0x101/0x190

root@pve:~# journalctl -b -4
Dec 10 17:37:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:37:01 pve sshd[31389]: Connection closed by 182.254.227.182 port 10465 [preauth]
Dec 10 17:37:01 pve CRON[31398]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 10 17:37:01 pve CRON[31399]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 10 17:37:01 pve CRON[31398]: pam_unix(cron:session): session closed for user root
Dec 10 17:37:01 pve systemd[1]: Started Proxmox VE replication runner.
...skipping...
Dec 10 17:37:19 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 10 17:37:19 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 10 17:37:19 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 10 17:37:19 pve kernel:  netlink_unicast+0x19c/0x240
Dec 10 17:37:19 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 10 17:37:19 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 10 17:37:19 pve kernel:  SYSC_sendto+0x101/0x190
Dec 10 17:37:19 pve kernel:  ? __secure_computing+0x3f/0x100
Dec 10 17:37:19 pve kernel:  ? syscall_trace_enter+0xca/0x2e0
Dec 10 17:37:19 pve kernel:  SyS_sendto+0xe/0x10
Dec 10 17:37:19 pve kernel:  do_syscall_64+0x73/0x130
Dec 10 17:37:19 pve kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Dec 10 17:37:19 pve kernel: RIP: 0033:0x7f507c89108b
Dec 10 17:37:19 pve kernel: RSP: 002b:00007ffd935277f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 10 17:37:19 pve kernel: RAX: ffffffffffffffda RBX: 0000557cacbd5f70 RCX: 00007f507c89108b
Dec 10 17:37:19 pve kernel: RDX: 00000000000002c0 RSI: 0000557cacbd3c78 RDI: 0000000000000003
Dec 10 17:37:19 pve kernel: RBP: 0000557cacbd3c78 R08: 00007f507c95c000 R09: 000000000000000c
Dec 10 17:37:19 pve kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
Dec 10 17:37:19 pve kernel: R13: 0000557cab980137 R14: 0000557cab98013f R15: 0000000000000000
Dec 10 17:37:19 pve kernel: Code: 89 50 0e 41 0f b6 45 42 a8 01 74 1a 48 8b 45 c0 49 03 5d 68 8b 40 28 85 c0 0f 85 e2 02 00 00 48 c7 03 00 00 00 00 f0 4d 0f ab 27 <49> 83 f9 fe 74 23 48 8b 7d 98
Dec 10 17:37:26 pve sshd[31583]: Invalid user suraj from 40.68.187.59 port 41904
Dec 10 17:37:26 pve sshd[31583]: input_userauth_request: invalid user suraj [preauth]
Dec 10 17:37:26 pve sshd[31583]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:37:26 pve sshd[31583]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=40.68.187.59
Dec 10 17:37:28 pve sshd[31585]: Invalid user test3 from 191.179.16.4 port 44897
Dec 10 17:37:28 pve sshd[31585]: input_userauth_request: invalid user test3 [preauth]
Dec 10 17:37:28 pve sshd[31585]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:37:28 pve sshd[31585]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=191.179.16.4
Dec 10 17:37:28 pve sshd[31583]: Failed password for invalid user suraj from 40.68.187.59 port 41904 ssh2
Dec 10 17:37:28 pve sshd[31583]: Received disconnect from 40.68.187.59 port 41904:11: Bye Bye [preauth]
Dec 10 17:37:28 pve sshd[31583]: Disconnected from 40.68.187.59 port 41904 [preauth]
Dec 10 17:37:31 pve sshd[31585]: Failed password for invalid user test3 from 191.179.16.4 port 44897 ssh2
Dec 10 17:37:32 pve sshd[31585]: Received disconnect from 191.179.16.4 port 44897:11: Bye Bye [preauth]
Dec 10 17:37:32 pve sshd[31585]: Disconnected from 191.179.16.4 port 44897 [preauth]
Dec 10 17:37:53 pve sshd[31701]: Did not receive identification string from 190.0.10.138 port 58266
Dec 10 17:38:05 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:38:40 pve kernel: INFO: task ipset:30810 blocked for more than 120 seconds.
Dec 10 17:38:41 pve kernel:       Tainted: P           O L   4.15.18-9-pve #1
Dec 10 17:39:01 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 17:39:02 pve kernel: ipset           D    0 30810   1830 0x00000000
Dec 10 17:39:06 pve kernel: Call Trace:
Dec 10 17:39:17 pve kernel:  __schedule+0x3e0/0x870
Dec 10 17:39:20 pve kernel:  schedule+0x36/0x80
Dec 10 17:39:27 pve kernel:  schedule_preempt_disabled+0xe/0x10
Dec 10 17:39:30 pve kernel:  __mutex_lock.isra.2+0x2b1/0x4e0
Dec 10 17:39:35 pve kernel:  ? mem_cgroup_commit_charge+0x82/0x540
...skipping...
Dec 10 17:37:19 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 10 17:37:19 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 10 17:37:19 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 10 17:37:19 pve kernel:  netlink_unicast+0x19c/0x240
Dec 10 17:37:19 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 10 17:37:19 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 10 17:37:19 pve kernel:  SYSC_sendto+0x101/0x190
Dec 10 17:37:19 pve kernel:  ? __secure_computing+0x3f/0x100
Dec 10 17:37:19 pve kernel:  ? syscall_trace_enter+0xca/0x2e0
Dec 10 17:37:19 pve kernel:  SyS_sendto+0xe/0x10
Dec 10 17:37:19 pve kernel:  do_syscall_64+0x73/0x130
Dec 10 17:37:19 pve kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Dec 10 17:37:19 pve kernel: RIP: 0033:0x7f507c89108b
Dec 10 17:37:19 pve kernel: RSP: 002b:00007ffd935277f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 10 17:37:19 pve kernel: RAX: ffffffffffffffda RBX: 0000557cacbd5f70 RCX: 00007f507c89108b
Dec 10 17:37:19 pve kernel: RDX: 00000000000002c0 RSI: 0000557cacbd3c78 RDI: 0000000000000003
Dec 10 17:37:19 pve kernel: RBP: 0000557cacbd3c78 R08: 00007f507c95c000 R09: 000000000000000c
Dec 10 17:37:19 pve kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
Dec 10 17:37:19 pve kernel: R13: 0000557cab980137 R14: 0000557cab98013f R15: 0000000000000000
Dec 10 17:37:19 pve kernel: Code: 89 50 0e 41 0f b6 45 42 a8 01 74 1a 48 8b 45 c0 49 03 5d 68 8b 40 28 85 c0 0f 85 e2 02 00 00 48 c7 03 00 00 00 00 f0 4d 0f ab 27 <49> 83 f9 fe 74 23 48 8b 7d 98
Dec 10 17:37:26 pve sshd[31583]: Invalid user suraj from 40.68.187.59 port 41904
Dec 10 17:37:26 pve sshd[31583]: input_userauth_request: invalid user suraj [preauth]
Dec 10 17:37:26 pve sshd[31583]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:37:26 pve sshd[31583]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=40.68.187.59
Dec 10 17:37:28 pve sshd[31585]: Invalid user test3 from 191.179.16.4 port 44897
Dec 10 17:37:28 pve sshd[31585]: input_userauth_request: invalid user test3 [preauth]
Dec 10 17:37:28 pve sshd[31585]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:37:28 pve sshd[31585]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=191.179.16.4
Dec 10 17:37:28 pve sshd[31583]: Failed password for invalid user suraj from 40.68.187.59 port 41904 ssh2
Dec 10 17:37:28 pve sshd[31583]: Received disconnect from 40.68.187.59 port 41904:11: Bye Bye [preauth]
Dec 10 17:37:28 pve sshd[31583]: Disconnected from 40.68.187.59 port 41904 [preauth]
Dec 10 17:37:31 pve sshd[31585]: Failed password for invalid user test3 from 191.179.16.4 port 44897 ssh2
Dec 10 17:37:32 pve sshd[31585]: Received disconnect from 191.179.16.4 port 44897:11: Bye Bye [preauth]
Dec 10 17:37:32 pve sshd[31585]: Disconnected from 191.179.16.4 port 44897 [preauth]
Dec 10 17:37:53 pve sshd[31701]: Did not receive identification string from 190.0.10.138 port 58266
Dec 10 17:38:05 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:38:40 pve kernel: INFO: task ipset:30810 blocked for more than 120 seconds.
Dec 10 17:38:41 pve kernel:       Tainted: P           O L   4.15.18-9-pve #1
Dec 10 17:39:01 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 17:39:02 pve kernel: ipset           D    0 30810   1830 0x00000000
Dec 10 17:39:06 pve kernel: Call Trace:
Dec 10 17:39:17 pve kernel:  __schedule+0x3e0/0x870
Dec 10 17:39:20 pve kernel:  schedule+0x36/0x80
Dec 10 17:39:27 pve kernel:  schedule_preempt_disabled+0xe/0x10
Dec 10 17:39:30 pve kernel:  __mutex_lock.isra.2+0x2b1/0x4e0
Dec 10 17:39:35 pve kernel:  ? mem_cgroup_commit_charge+0x82/0x540

 

[mira]

Your BIOS is from 2013, please upgrade it first.

Could you provide additional information about your hardware?

 

[Don Daniello]

I'd love to upgrade BIOS but that may be impossible. The latest update is from 2015 anyway. It's an OVH dedicated server.

CPU: Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
Motherboard: ASUSTeK COMPUTER INC. Product Name: P8H77-M PRO Version: Rev X.0x

 

[mira]

It might be a problem with the ethernet controller. You could try https://packages.debian.org/stretch/r8168-dkms to see if it works.

 

[Don Daniello]

It might be a problem with the ethernet controller. You could try https://packages.debian.org/stretch/r8168-dkms to see if it works.

Thanks for suggestion but that does not seem to be working:

root@pve:~# apt install r8168-dkms 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  r8168-dkms
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.0 kB of archives.
After this operation, 1,135 kB of additional disk space will be used.
Get:1 http://debian.mirrors.ovh.net/debian stretch/non-free amd64 r8168-dkms all 8.043.02-1 [88.0 kB]
Fetched 88.0 kB in 0s (3,703 kB/s)
Selecting previously unselected package r8168-dkms.
(Reading database ... 121690 files and directories currently installed.)
Preparing to unpack .../r8168-dkms_8.043.02-1_all.deb ...
Unpacking r8168-dkms (8.043.02-1) ...
Setting up r8168-dkms (8.043.02-1) ...
Loading new r8168-8.043.02 DKMS files...
Building for 4.15.18-9-pve
Building initial module for 4.15.18-9-pve
Error! Bad return status for module build on kernel: 4.15.18-9-pve (x86_64)
Consult /var/lib/dkms/r8168/8.043.02/build/make.log for more information.
dpkg: error processing package r8168-dkms (--configure):
 subprocess installed post-installation script returned error exit status 10
Errors were encountered while processing:
 r8168-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@pve:~# tail  /var/lib/dkms/r8168/8.043.02/build/make.log
/var/lib/dkms/r8168/8.043.02/build/r8168_n.c: In function ‘rtl8168_rx_interrupt’:
/var/lib/dkms/r8168/8.043.02/build/r8168_n.c:25418:28: error: ‘struct net_device’ has no member named ‘last_rx’
                         dev->last_rx = jiffies;
                            ^~
cc1: some warnings being treated as errors
scripts/Makefile.build:324: recipe for target '/var/lib/dkms/r8168/8.043.02/build/r8168_n.o' failed
make[1]: *** [/var/lib/dkms/r8168/8.043.02/build/r8168_n.o] Error 1
Makefile:1551: recipe for target '_module_/var/lib/dkms/r8168/8.043.02/build' failed
make: *** [_module_/var/lib/dkms/r8168/8.043.02/build] Error 2
make: Leaving directory '/usr/src/linux-headers-4.15.18-9-pve'

 

[mira]

Looks like it's not compatible with our kernel.
After going through your logs again it seems they're not in the correct order (timestamps). Would be great if you could post them again (without skipping anything) in the correct order.

 

[Don Daniello]

Let me try that again. It looks like the journalctl "pager" is causing the "skipped". But still, the messages sometimes appear to be mangled in the journal itself.

Here's the oldest one (2018-12-10):

root@pve:~# journalctl -b -4 --no-pager --since '2018-12-10 17:35:00'     
-- Logs begin at Thu 2018-12-06 17:24:36 UTC, end at Fri 2018-12-28 16:28:02 UTC. --
Dec 10 17:35:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:35:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 10 17:35:01 pve sshd[30006]: Failed password for invalid user wildfly from 202.154.56.238 port 35978 ssh2
Dec 10 17:35:01 pve sshd[30006]: Received disconnect from 202.154.56.238 port 35978:11: Bye Bye [preauth]
Dec 10 17:35:01 pve sshd[30006]: Disconnected from 202.154.56.238 port 35978 [preauth]
Dec 10 17:35:01 pve CRON[30091]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 10 17:35:01 pve CRON[30092]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 10 17:35:02 pve CRON[30091]: pam_unix(cron:session): session closed for user root
Dec 10 17:36:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:36:01 pve CRON[30881]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 10 17:36:01 pve CRON[30882]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 10 17:36:01 pve CRON[30881]: pam_unix(cron:session): session closed for user root
Dec 10 17:36:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 10 17:36:28 pve sshd[31147]: Invalid user iulian from 49.206.196.254 port 43488
Dec 10 17:36:28 pve sshd[31147]: input_userauth_request: invalid user iulian [preauth]
Dec 10 17:36:28 pve sshd[31147]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:36:28 pve sshd[31147]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=49.206.196.254
Dec 10 17:36:30 pve sshd[31147]: Failed password for invalid user iulian from 49.206.196.254 port 43488 ssh2
Dec 10 17:36:30 pve sshd[31147]: Received disconnect from 49.206.196.254 port 43488:11: Bye Bye [preauth]
Dec 10 17:36:30 pve sshd[31147]: Disconnected from 49.206.196.254 port 43488 [preauth]
Dec 10 17:37:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:37:01 pve sshd[31389]: Connection closed by 182.254.227.182 port 10465 [preauth]
Dec 10 17:37:01 pve CRON[31398]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 10 17:37:01 pve CRON[31399]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 10 17:37:01 pve CRON[31398]: pam_unix(cron:session): session closed for user root
Dec 10 17:37:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 10 17:37:19 pve kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ipset:30807]
Dec 10 17:37:19 pve kernel: Modules linked in: binfmt_misc act_police cls_basic sch_ingress sch_htb veth ebtable_filter ebtables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_mac ipt_REJECT nf_reject_ipv4 xt_physdev xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_mark xt_set xt_addrtype xt_multiport xt_conntrack nf_conntrack ip_set_hash_net ip_set iptable_filter softdog nfnetlink_log nfnetlink xfs intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_hdmi irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic intel_cstate ppdev eeepc_wmi asus_wmi wmi_bmof sparse_keymap i915 intel_rapl_perf snd_hda_intel snd_hda_codec parport_pc
Dec 10 17:37:19 pve kernel:  drm_kms_helper snd_hda_core parport video snd_hwdep drm snd_pcm i2c_algo_bit fb_sys_fops syscopyarea snd_timer sysfillrect sysimgblt mei_me snd mac_hid shpchp soundcore mei ie31200_edac lpc_ich pcspkr wmi zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp sunrpc libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi i2c_i801 ahci libahci r8169 mii megaraid_sas
Dec 10 17:37:19 pve kernel: CPU: 4 PID: 30807 Comm: ipset Tainted: P           O L   4.15.18-9-pve #1
Dec 10 17:37:19 pve kernel: Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1401 08/20/2013
Dec 10 17:37:19 pve kernel: RIP: 0010:hash_net4_add+0x369/0x787 [ip_set_hash_net]
Dec 10 17:37:19 pve kernel: RSP: 0018:ffffba0dce1777a0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Dec 10 17:37:19 pve kernel: RAX: 0000000000000000 RBX: ffff9c7cb7d980a0 RCX: ffffffffffffffff
Dec 10 17:37:19 pve kernel: RDX: 0000000000000021 RSI: 0000000000000000 RDI: 0000000000000021
Dec 10 17:37:19 pve kernel: RBP: ffffba0dce177810 R08: 0000000000000000 R09: 0000000000000000
Dec 10 17:37:19 pve kernel: R10: 0000000000000021 R11: ffffba0dce177848 R12: 0000000000000000
Dec 10 17:37:19 pve kernel: R13: ffff9c7ba28606c0 R14: ffff9c7a3728b800 R15: ffff9c7cb7d98090
Dec 10 17:37:19 pve kernel: FS:  00007f507c791740(0000) GS:ffff9c81dfb00000(0000) knlGS:0000000000000000
Dec 10 17:37:19 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 10 17:37:19 pve kernel: CR2: 0000000000ecd220 CR3: 00000000a2fce004 CR4: 00000000001626e0
Dec 10 17:37:19 pve kernel: Call Trace:
Dec 10 17:37:19 pve kernel:  hash_net4_uadt+0x21f/0x330 [ip_set_hash_net]
Dec 10 17:37:19 pve kernel:  ? hash_net6_add+0x800/0x800 [ip_set_hash_net]
Dec 10 17:37:19 pve kernel:  call_ad+0xc4/0x220 [ip_set]
Dec 10 17:37:19 pve kernel:  ? hash_net6_uadt+0x200/0x200 [ip_set_hash_net]
Dec 10 17:37:19 pve kernel:  ? call_ad+0xc4/0x220 [ip_set]
Dec 10 17:37:19 pve kernel:  ? __update_load_avg_se.isra.38+0x1bc/0x1d0
Dec 10 17:37:19 pve kernel:  ? __update_load_avg_se.isra.38+0x1bc/0x1d0
Dec 10 17:37:19 pve kernel:  ? validate_nla+0x15b/0x240
Dec 10 17:37:19 pve kernel:  ? nla_parse+0xaa/0x140
Dec 10 17:37:19 pve kernel:  ip_set_uadd+0x23c/0x2a0 [ip_set]
Dec 10 17:37:19 pve kernel:  nfnetlink_rcv_msg+0x240/0x260 [nfnetlink]
Dec 10 17:37:19 pve kernel:  ? nfnetlink_net_exit_batch+0x70/0x70 [nfnetlink]
Dec 10 17:37:19 pve kernel:  netlink_rcv_skb+0xd9/0x110
Dec 10 17:37:19 pve kernel:  nfnetlink_rcv+0x6c/0x72a [nfnetlink]
Dec 10 17:37:19 pve kernel:  ? _cond_resched+0x1a/0x50
Dec 10 17:37:19 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 10 17:37:19 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 10 17:37:19 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 10 17:37:19 pve kernel:  netlink_unicast+0x19c/0x240
Dec 10 17:37:19 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 10 17:37:19 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 10 17:37:19 pve kernel:  SYSC_sendto+0x101/0x190
Dec 10 17:37:19 pve kernel:  ? __secure_computing+0x3f/0x100
Dec 10 17:37:19 pve kernel:  ? syscall_trace_enter+0xca/0x2e0
Dec 10 17:37:19 pve kernel:  SyS_sendto+0xe/0x10
Dec 10 17:37:19 pve kernel:  do_syscall_64+0x73/0x130
Dec 10 17:37:19 pve kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Dec 10 17:37:19 pve kernel: RIP: 0033:0x7f507c89108b
Dec 10 17:37:19 pve kernel: RSP: 002b:00007ffd935277f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 10 17:37:19 pve kernel: RAX: ffffffffffffffda RBX: 0000557cacbd5f70 RCX: 00007f507c89108b
Dec 10 17:37:19 pve kernel: RDX: 00000000000002c0 RSI: 0000557cacbd3c78 RDI: 0000000000000003
Dec 10 17:37:19 pve kernel: RBP: 0000557cacbd3c78 R08: 00007f507c95c000 R09: 000000000000000c
Dec 10 17:37:19 pve kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
Dec 10 17:37:19 pve kernel: R13: 0000557cab980137 R14: 0000557cab98013f R15: 0000000000000000
Dec 10 17:37:19 pve kernel: Code: 89 50 0e 41 0f b6 45 42 a8 01 74 1a 48 8b 45 c0 49 03 5d 68 8b 40 28 85 c0 0f 85 e2 02 00 00 48 c7 03 00 00 00 00 f0 4d 0f ab 27 <49> 83 f9 fe 74 23 48 8b 7d 98 48 8b 45 a0 4d 85 c9 48 8d 44 f8 
Dec 10 17:37:26 pve sshd[31583]: Invalid user suraj from 40.68.187.59 port 41904
Dec 10 17:37:26 pve sshd[31583]: input_userauth_request: invalid user suraj [preauth]
Dec 10 17:37:26 pve sshd[31583]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:37:26 pve sshd[31583]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=40.68.187.59
Dec 10 17:37:28 pve sshd[31585]: Invalid user test3 from 191.179.16.4 port 44897
Dec 10 17:37:28 pve sshd[31585]: input_userauth_request: invalid user test3 [preauth]
Dec 10 17:37:28 pve sshd[31585]: pam_unix(sshd:auth): check pass; user unknown
Dec 10 17:37:28 pve sshd[31585]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=191.179.16.4
Dec 10 17:37:28 pve sshd[31583]: Failed password for invalid user suraj from 40.68.187.59 port 41904 ssh2
Dec 10 17:37:28 pve sshd[31583]: Received disconnect from 40.68.187.59 port 41904:11: Bye Bye [preauth]
Dec 10 17:37:28 pve sshd[31583]: Disconnected from 40.68.187.59 port 41904 [preauth]
Dec 10 17:37:31 pve sshd[31585]: Failed password for invalid user test3 from 191.179.16.4 port 44897 ssh2
Dec 10 17:37:32 pve sshd[31585]: Received disconnect from 191.179.16.4 port 44897:11: Bye Bye [preauth]
Dec 10 17:37:32 pve sshd[31585]: Disconnected from 191.179.16.4 port 44897 [preauth]
Dec 10 17:37:53 pve sshd[31701]: Did not receive identification string from 190.0.10.138 port 58266
Dec 10 17:38:05 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 10 17:38:40 pve kernel: INFO: task ipset:30810 blocked for more than 120 seconds.
Dec 10 17:38:41 pve kernel:       Tainted: P           O L   4.15.18-9-pve #1
Dec 10 17:39:01 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 17:39:02 pve kernel: ipset           D    0 30810   1830 0x00000000
Dec 10 17:39:06 pve kernel: Call Trace:
Dec 10 17:39:17 pve kernel:  __schedule+0x3e0/0x870
Dec 10 17:39:20 pve kernel:  schedule+0x36/0x80
Dec 10 17:39:27 pve kernel:  schedule_preempt_disabled+0xe/0x10
Dec 10 17:39:30 pve kernel:  __mutex_lock.isra.2+0x2b1/0x4e0
Dec 10 17:39:35 pve kernel:  ? mem_cgroup_commit_charge+0x82/0x540

2018-12-20

root@pve:~# journalctl -b -2 --no-pager --since '2018-12-20 18:20:00'
-- Logs begin at Thu 2018-12-06 17:24:36 UTC, end at Fri 2018-12-28 16:30:10 UTC. --
Dec 20 18:20:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 20 18:20:00 pve systemd[1]: Started Proxmox VE replication runner.
Dec 20 18:20:01 pve CRON[12498]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:20:01 pve CRON[12499]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:20:01 pve CRON[12498]: pam_unix(cron:session): session closed for user root
Dec 20 18:21:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 20 18:21:01 pve CRON[13307]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:21:01 pve CRON[13308]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:21:02 pve CRON[13307]: pam_unix(cron:session): session closed for user root
Dec 20 18:21:02 pve systemd[1]: Started Proxmox VE replication runner.
Dec 20 18:21:36 pve pvestatd[1836]: status update time (5.419 seconds)
Dec 20 18:22:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 20 18:22:01 pve CRON[13836]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:22:01 pve CRON[13837]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:22:05 pve CRON[13836]: pam_unix(cron:session): session closed for user root
Dec 20 18:22:29 pve pvestatd[1836]: got timeout
Dec 20 18:22:36 pve sshd[13958]: Did not receive identification string from 182.254.227.182 port 7109
Dec 20 18:23:12 pve pve-ha-lrm[1982]: loop take too long (57 seconds)
Dec 20 18:23:12 pve CRON[13963]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 20 18:23:18 pve CRON[13976]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 20 18:24:18 pve kernel: INFO: task ipset:13217 blocked for more than 120 seconds.
Dec 20 18:24:24 pve kernel:       Tainted: P           O L   4.15.18-9-pve #1
Dec 20 18:24:31 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 20 18:24:34 pve kernel: ipset           D    0 13217   1829 0x00000000
Dec 20 18:24:34 pve kernel: Call Trace:
Dec 20 18:24:34 pve kernel:  __schedule+0x3e0/0x870
Dec 20 18:24:34 pve kernel:  schedule+0x36/0x80
Dec 20 18:24:34 pve kernel:  schedule_preempt_disabled+0xe/0x10
Dec 20 18:24:34 pve kernel:  __mutex_lock.isra.2+0x2b1/0x4e0
Dec 20 18:24:34 pve kernel:  ? mem_cgroup_commit_charge+0x82/0x540
Dec 20 18:24:34 pve kernel:  ? lru_cache_add_active_or_unevictable+0x36/0xb0
Dec 20 18:24:34 pve kernel:  __mutex_lock_slowpath+0x13/0x20
Dec 20 18:24:47 pve kernel:  ? __mutex_lock_slowpath+0x13/0x20
Dec 20 18:24:48 pve kernel:  mutex_lock+0x2f/0x40
Dec 20 18:24:50 pve kernel:  nfnetlink_rcv_msg+0x1ab/0x260 [nfnetlink]
Dec 20 18:25:08 pve kernel:  ? nfnetlink_net_exit_batch+0x70/0x70 [nfnetlink]
Dec 20 18:25:13 pve kernel:  netlink_rcv_skb+0xd9/0x110
Dec 20 18:25:24 pve kernel:  nfnetlink_rcv+0x6c/0x72a [nfnetlink]
Dec 20 18:25:27 pve kernel:  ? walk_system_ram_range+0xb2/0xe0
Dec 20 18:25:28 pve kernel:  ? memcg_kmem_charge_memcg+0x7d/0xa0
Dec 20 18:25:40 pve kernel:  ? _cond_resched+0x1a/0x50
Dec 20 18:25:44 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 20 18:25:53 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 20 18:26:07 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 20 18:26:09 pve kernel:  netlink_unicast+0x19c/0x240
Dec 20 18:26:19 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 20 18:26:24 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 20 18:26:27 pve kernel:  SYSC_sendto+0x101/0x190

2018-12-26

root@pve:~# journalctl -b -1 --no-pager --since '2018-12-26 18:43:00' 
-- Logs begin at Thu 2018-12-06 17:24:36 UTC, end at Fri 2018-12-28 16:31:01 UTC. --
Dec 26 18:43:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:43:00 pve systemd[1]: Started Proxmox VE replication runner.
Dec 26 18:43:01 pve CRON[22736]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 26 18:43:01 pve CRON[22737]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 26 18:43:01 pve CRON[22736]: pam_unix(cron:session): session closed for user root
Dec 26 18:43:21 pve sshd[23003]: Invalid user git from 145.239.76.62 port 54524
Dec 26 18:43:21 pve sshd[23003]: input_userauth_request: invalid user git [preauth]
Dec 26 18:43:21 pve sshd[23003]: pam_unix(sshd:auth): check pass; user unknown
Dec 26 18:43:21 pve sshd[23003]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=145.239.76.62
Dec 26 18:43:23 pve sshd[23003]: Failed password for invalid user git from 145.239.76.62 port 54524 ssh2
Dec 26 18:43:23 pve sshd[23003]: Received disconnect from 145.239.76.62 port 54524:11: Normal Shutdown, Thank you for playing [preauth]
Dec 26 18:43:23 pve sshd[23003]: Disconnected from 145.239.76.62 port 54524 [preauth]
Dec 26 18:43:38 pve sshd[23548]: Invalid user anish from 120.131.9.177 port 31099
Dec 26 18:43:38 pve sshd[23548]: input_userauth_request: invalid user anish [preauth]
Dec 26 18:43:38 pve sshd[23548]: pam_unix(sshd:auth): check pass; user unknown
Dec 26 18:43:38 pve sshd[23548]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=120.131.9.177
Dec 26 18:43:40 pve sshd[23548]: Failed password for invalid user anish from 120.131.9.177 port 31099 ssh2
Dec 26 18:43:40 pve sshd[23548]: Received disconnect from 120.131.9.177 port 31099:11: Bye Bye [preauth]
Dec 26 18:43:40 pve sshd[23548]: Disconnected from 120.131.9.177 port 31099 [preauth]
Dec 26 18:44:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:44:00 pve systemd[1]: Started Proxmox VE replication runner.
Dec 26 18:44:01 pve CRON[23914]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 26 18:44:01 pve CRON[23918]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 26 18:44:01 pve CRON[23914]: pam_unix(cron:session): session closed for user root
Dec 26 18:44:14 pve sshd[24455]: Invalid user git from 145.239.76.62 port 51631
Dec 26 18:44:14 pve sshd[24455]: input_userauth_request: invalid user git [preauth]
Dec 26 18:44:14 pve sshd[24455]: pam_unix(sshd:auth): check pass; user unknown
Dec 26 18:44:14 pve sshd[24455]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=145.239.76.62
Dec 26 18:44:16 pve sshd[24455]: Failed password for invalid user git from 145.239.76.62 port 51631 ssh2
Dec 26 18:44:16 pve sshd[24455]: Received disconnect from 145.239.76.62 port 51631:11: Normal Shutdown, Thank you for playing [preauth]
Dec 26 18:44:16 pve sshd[24455]: Disconnected from 145.239.76.62 port 51631 [preauth]
Dec 26 18:45:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:45:00 pve systemd[1]: Started Proxmox VE replication runner.
Dec 26 18:45:01 pve CRON[25432]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 26 18:45:01 pve CRON[25433]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 26 18:45:01 pve CRON[25432]: pam_unix(cron:session): session closed for user root
Dec 26 18:46:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:46:00 pve systemd[1]: Started Proxmox VE replication runner.
Dec 26 18:46:00 pve systemd[1]: Starting Cleanup of Temporary Directories...
Dec 26 18:46:00 pve systemd[1]: Started Cleanup of Temporary Directories.
Dec 26 18:46:01 pve CRON[26744]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 26 18:46:01 pve CRON[26745]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 26 18:46:02 pve CRON[26744]: pam_unix(cron:session): session closed for user root
Dec 26 18:46:26 pve sshd[27384]: Invalid user git from 145.239.76.62 port 60902
Dec 26 18:46:26 pve sshd[27384]: input_userauth_request: invalid user git [preauth]
Dec 26 18:46:26 pve sshd[27384]: pam_unix(sshd:auth): check pass; user unknown
Dec 26 18:46:26 pve sshd[27384]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=145.239.76.62
Dec 26 18:46:29 pve sshd[27384]: Failed password for invalid user git from 145.239.76.62 port 60902 ssh2
Dec 26 18:46:29 pve sshd[27384]: Received disconnect from 145.239.76.62 port 60902:11: Normal Shutdown, Thank you for playing [preauth]
Dec 26 18:46:29 pve sshd[27384]: Disconnected from 145.239.76.62 port 60902 [preauth]
Dec 26 18:47:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:47:01 pve CRON[28187]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 26 18:47:01 pve CRON[28188]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 26 18:47:01 pve CRON[28187]: pam_unix(cron:session): session closed for user root
Dec 26 18:47:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 26 18:47:24 pve kernel: watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ipset:26626]
Dec 26 18:47:24 pve kernel: Modules linked in: ip6t_rpfilter ebtable_nat ebtable_broute ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw binfmt_misc act_police cls_basic sch_ingress sch_htb veth ebtable_filter ebtables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_mac ipt_REJECT nf_reject_ipv4 xt_physdev xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_mark xt_set xt_addrtype xt_multiport xt_conntrack nf_conntrack ip_set_hash_net ip_set iptable_filter softdog nfnetlink_log nfnetlink xfs intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper
Dec 26 18:47:24 pve kernel:  cryptd snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ppdev eeepc_wmi asus_wmi wmi_bmof sparse_keymap intel_cstate intel_rapl_perf i915 snd_hda_intel snd_hda_codec drm_kms_helper snd_hda_core snd_hwdep parport_pc snd_pcm drm parport snd_timer i2c_algo_bit fb_sys_fops syscopyarea sysfillrect video sysimgblt snd mei_me soundcore shpchp ie31200_edac mei lpc_ich pcspkr mac_hid zfs(PO) wmi zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi i2c_i801 ahci libahci r8169 mii
Dec 26 18:47:24 pve kernel:  megaraid_sas
Dec 26 18:47:24 pve kernel: CPU: 7 PID: 26626 Comm: ipset Tainted: P           O L   4.15.18-9-pve #1
Dec 26 18:47:24 pve kernel: Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1401 08/20/2013
Dec 26 18:47:24 pve kernel: RIP: 0010:ip_set_range_to_cidr+0x12/0x49 [ip_set]
Dec 26 18:47:24 pve kernel: RSP: 0018:ffffb30161067810 EFLAGS: 00000213 ORIG_RAX: ffffffffffffff11
Dec 26 18:47:24 pve kernel: RAX: 0000000080000000 RBX: ffffb30161067850 RCX: 000000000ab3973a
Dec 26 18:47:24 pve kernel: RDX: ffffb3016106784f RSI: 00000000ffffffff RDI: 000000000ab3973a
Dec 26 18:47:24 pve kernel: RBP: ffffb301610678c0 R08: ffffffffc0a51b00 R09: 0000000000000000
Dec 26 18:47:24 pve kernel: R10: 0000000000000021 R11: ffffb30161067848 R12: 0000000000000001
Dec 26 18:47:24 pve kernel: R13: 00000000ffffffff R14: ffff9818669b6780 R15: 000000000ab39739
Dec 26 18:47:24 pve kernel: FS:  00007fd1d0247740(0000) GS:ffff98199fbc0000(0000) knlGS:0000000000000000
Dec 26 18:47:24 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 26 18:47:24 pve kernel: CR2: 00007f4e382223a0 CR3: 00000007dbd90002 CR4: 00000000001606e0
Dec 26 18:47:24 pve kernel: Call Trace:
Dec 26 18:47:24 pve kernel:  ? hash_net4_uadt+0x200/0x330 [ip_set_hash_net]
Dec 26 18:47:24 pve kernel:  ? hash_net6_add+0x800/0x800 [ip_set_hash_net]
Dec 26 18:47:24 pve kernel:  call_ad+0xc4/0x220 [ip_set]
Dec 26 18:47:24 pve kernel:  ? hash_net6_uadt+0x200/0x200 [ip_set_hash_net]
Dec 26 18:47:24 pve kernel:  ? call_ad+0xc4/0x220 [ip_set]
Dec 26 18:47:24 pve kernel:  ? validate_nla+0x15b/0x240
Dec 26 18:47:24 pve kernel:  ? nla_parse+0xaa/0x140
Dec 26 18:47:24 pve kernel:  ip_set_uadd+0x23c/0x2a0 [ip_set]
Dec 26 18:47:24 pve kernel:  nfnetlink_rcv_msg+0x240/0x260 [nfnetlink]
Dec 26 18:47:24 pve kernel:  ? nfnetlink_net_exit_batch+0x70/0x70 [nfnetlink]
Dec 26 18:47:24 pve kernel:  netlink_rcv_skb+0xd9/0x110
Dec 26 18:47:24 pve kernel:  nfnetlink_rcv+0x6c/0x72a [nfnetlink]
Dec 26 18:47:24 pve kernel:  ? __slab_alloc+0x20/0x40
Dec 26 18:47:24 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 26 18:47:24 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 26 18:47:24 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 26 18:47:24 pve kernel:  netlink_unicast+0x19c/0x240
Dec 26 18:47:24 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 26 18:47:24 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 26 18:47:24 pve kernel:  SYSC_sendto+0x101/0x190
Dec 26 18:47:24 pve kernel:  ? __secure_computing+0x3f/0x100
Dec 26 18:47:24 pve kernel:  ? syscall_trace_enter+0xca/0x2e0
Dec 26 18:47:24 pve kernel:  SyS_sendto+0xe/0x10
Dec 26 18:47:24 pve kernel:  do_syscall_64+0x73/0x130
Dec 26 18:47:24 pve kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Dec 26 18:47:24 pve kernel: RIP: 0033:0x7fd1d034708b
Dec 26 18:47:24 pve kernel: RSP: 002b:00007ffe189e6578 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 26 18:47:24 pve kernel: RAX: ffffffffffffffda RBX: 000055cb33a5af70 RCX: 00007fd1d034708b
Dec 26 18:47:24 pve kernel: RDX: 00000000000004dc RSI: 000055cb33a58c78 RDI: 0000000000000003
Dec 26 18:47:24 pve kernel: RBP: 000055cb33a58c78 R08: 00007fd1d0412000 R09: 000000000000000c
Dec 26 18:47:24 pve kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
Dec 26 18:47:24 pve kernel: R13: 000055cb3308c137 R14: 000055cb3308c13f R15: 0000000000000000
Dec 26 18:47:24 pve kernel: Code: 48 8d 4d f7 0f b6 f2 48 89 c2 e8 4a fe ff ff eb c2 e8 13 b9 a3 fa 0f 1f 00 0f 1f 44 00 00 55 49 c7 c0 00 1b a5 c0 b8 00 00 00 80 <b9> 01 00 00 00 48 89 e5 eb 0f 83 c1 01 80 f9 20 74 1e 41 8b 00 
Dec 26 18:48:03 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 26 18:48:21 pve CRON[28585]: pam_unix(cron:session): session opened for user root by (uid=0)

 

[Don Daniello]

And a new one from last night:

root@pve:~# journalctl -b -1 --no-pager --since '2018-12-29 07:00:00'
-- Logs begin at Thu 2018-12-06 17:24:36 UTC, end at Sat 2018-12-29 15:13:01 UTC. --
Dec 29 07:00:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 29 07:00:01 pve CRON[4976]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 29 07:00:01 pve CRON[4977]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 29 07:00:01 pve CRON[4976]: pam_unix(cron:session): session closed for user root
Dec 29 07:00:01 pve sshd[4953]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:00:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 29 07:00:03 pve sshd[4953]: Failed password for root from 218.92.1.175 port 26324 ssh2
Dec 29 07:00:05 pve pveproxy[32685]: worker exit
Dec 29 07:00:05 pve pveproxy[1972]: worker 32685 finished
Dec 29 07:00:05 pve pveproxy[1972]: starting 1 worker(s)
Dec 29 07:00:05 pve pveproxy[1972]: worker 5399 started
Dec 29 07:00:06 pve sshd[4953]: Failed password for root from 218.92.1.175 port 26324 ssh2
Dec 29 07:00:09 pve sshd[4953]: Failed password for root from 218.92.1.175 port 26324 ssh2
Dec 29 07:00:12 pve sshd[4953]: Failed password for root from 218.92.1.175 port 26324 ssh2
Dec 29 07:00:15 pve sshd[4953]: Failed password for root from 218.92.1.175 port 26324 ssh2
Dec 29 07:00:20 pve sshd[4953]: Failed password for root from 218.92.1.175 port 26324 ssh2
Dec 29 07:00:20 pve sshd[4953]: error: maximum authentication attempts exceeded for root from 218.92.1.175 port 26324 ssh2 [preauth]
Dec 29 07:00:20 pve sshd[4953]: Disconnecting: Too many authentication failures [preauth]
Dec 29 07:00:20 pve sshd[4953]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:00:20 pve sshd[4953]: PAM service(sshd) ignoring max retries; 6 > 3
Dec 29 07:00:24 pve sshd[5934]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:00:27 pve sshd[5934]: Failed password for root from 218.92.1.175 port 53042 ssh2
Dec 29 07:00:30 pve sshd[5934]: Failed password for root from 218.92.1.175 port 53042 ssh2
Dec 29 07:00:33 pve sshd[5934]: Failed password for root from 218.92.1.175 port 53042 ssh2
Dec 29 07:00:37 pve sshd[5934]: Failed password for root from 218.92.1.175 port 53042 ssh2
Dec 29 07:00:40 pve sshd[5934]: Failed password for root from 218.92.1.175 port 53042 ssh2
Dec 29 07:00:43 pve sshd[5934]: Failed password for root from 218.92.1.175 port 53042 ssh2
Dec 29 07:00:43 pve sshd[5934]: error: maximum authentication attempts exceeded for root from 218.92.1.175 port 53042 ssh2 [preauth]
Dec 29 07:00:43 pve sshd[5934]: Disconnecting: Too many authentication failures [preauth]
Dec 29 07:00:43 pve sshd[5934]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:00:43 pve sshd[5934]: PAM service(sshd) ignoring max retries; 6 > 3
Dec 29 07:00:47 pve pvestatd[1847]: status update time (5.893 seconds)
Dec 29 07:00:49 pve sshd[6183]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:00:50 pve sshd[6183]: Failed password for root from 218.92.1.175 port 15890 ssh2
Dec 29 07:00:53 pve sshd[6183]: Failed password for root from 218.92.1.175 port 15890 ssh2
Dec 29 07:00:54 pve sshd[6593]: Invalid user dennis from 95.58.194.148 port 43086
Dec 29 07:00:54 pve sshd[6593]: input_userauth_request: invalid user dennis [preauth]
Dec 29 07:00:54 pve sshd[6593]: pam_unix(sshd:auth): check pass; user unknown
Dec 29 07:00:54 pve sshd[6593]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=95.58.194.148
Dec 29 07:00:56 pve sshd[6183]: Failed password for root from 218.92.1.175 port 15890 ssh2
Dec 29 07:00:56 pve sshd[6593]: Failed password for invalid user dennis from 95.58.194.148 port 43086 ssh2
Dec 29 07:00:57 pve sshd[6593]: Received disconnect from 95.58.194.148 port 43086:11: Bye Bye [preauth]
Dec 29 07:00:57 pve sshd[6593]: Disconnected from 95.58.194.148 port 43086 [preauth]
Dec 29 07:00:59 pve sshd[6183]: Failed password for root from 218.92.1.175 port 15890 ssh2
Dec 29 07:01:00 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 29 07:01:01 pve systemd[1]: Started Proxmox VE replication runner.
Dec 29 07:01:01 pve CRON[6687]: pam_unix(cron:session): session opened for user root by (uid=0)
Dec 29 07:01:01 pve CRON[6688]: (root) CMD (/usr/local/rtm/bin/rtm 22 > /dev/null 2> /dev/null)
Dec 29 07:01:01 pve CRON[6687]: pam_unix(cron:session): session closed for user root
Dec 29 07:01:02 pve sshd[6183]: Failed password for root from 218.92.1.175 port 15890 ssh2
Dec 29 07:01:05 pve sshd[6183]: Failed password for root from 218.92.1.175 port 15890 ssh2
Dec 29 07:01:05 pve sshd[6183]: error: maximum authentication attempts exceeded for root from 218.92.1.175 port 15890 ssh2 [preauth]
Dec 29 07:01:05 pve sshd[6183]: Disconnecting: Too many authentication failures [preauth]
Dec 29 07:01:05 pve sshd[6183]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:01:05 pve sshd[6183]: PAM service(sshd) ignoring max retries; 6 > 3
Dec 29 07:01:08 pve sshd[6814]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:01:09 pve kernel: watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ipset:4830]
Dec 29 07:01:09 pve kernel: Modules linked in: tcp_diag inet_diag ip6t_rpfilter ebtable_nat ebtable_broute ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw binfmt_misc act_police cls_basic sch_ingress sch_htb veth ebtable_filter ebtables ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_mac ipt_REJECT nf_reject_ipv4 xt_physdev xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_tcpudp xt_mark xt_set xt_addrtype xt_multiport xt_conntrack nf_conntrack ip_set_hash_net ip_set iptable_filter softdog nfnetlink_log nfnetlink xfs intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd
Dec 29 07:01:09 pve kernel:  glue_helper cryptd snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic intel_cstate eeepc_wmi asus_wmi ppdev wmi_bmof sparse_keymap i915 snd_hda_intel snd_hda_codec drm_kms_helper snd_hda_core snd_hwdep intel_rapl_perf drm snd_pcm i2c_algo_bit fb_sys_fops snd_timer syscopyarea mei_me snd sysfillrect soundcore sysimgblt mei shpchp ie31200_edac lpc_ich pcspkr wmi parport_pc mac_hid video parport zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm ib_core sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear pata_acpi i2c_i801 ahci libahci
Dec 29 07:01:09 pve kernel:  r8169 mii megaraid_sas
Dec 29 07:01:09 pve kernel: CPU: 4 PID: 4830 Comm: ipset Tainted: P           O L   4.15.18-9-pve #1
Dec 29 07:01:09 pve kernel: Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1401 08/20/2013
Dec 29 07:01:09 pve kernel: RIP: 0010:hash_net4_add+0xdf/0x787 [ip_set_hash_net]
Dec 29 07:01:09 pve kernel: RSP: 0018:ffffa3678df577a0 EFLAGS: 00000206 ORIG_RAX: ffffffffffffff11
Dec 29 07:01:09 pve kernel: RAX: ffff88c4f02daec0 RBX: ffffa36801bb3000 RCX: 000000000000001b
Dec 29 07:01:09 pve kernel: RDX: 00000000be75cf71 RSI: 0000000007ffffff RDI: 000000000350fe6a
Dec 29 07:01:09 pve kernel: RBP: ffffa3678df57810 R08: 0000000084dbc809 R09: 0000000009c8db84
Dec 29 07:01:09 pve kernel: R10: 0000000000000021 R11: ffffa3678df57848 R12: 0000000000000001
Dec 29 07:01:09 pve kernel: R13: ffff88c27aff9b40 R14: ffff88c28d241800 R15: 0000000009c8db84
Dec 29 07:01:09 pve kernel: FS:  00007f193a339740(0000) GS:ffff88c9dfb00000(0000) knlGS:0000000000000000
Dec 29 07:01:09 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 29 07:01:09 pve kernel: CR2: 000056152a34fd30 CR3: 0000000033a80006 CR4: 00000000001626e0
Dec 29 07:01:09 pve kernel: Call Trace:
Dec 29 07:01:09 pve kernel:  hash_net4_uadt+0x21f/0x330 [ip_set_hash_net]
Dec 29 07:01:09 pve kernel:  ? hash_net6_add+0x800/0x800 [ip_set_hash_net]
Dec 29 07:01:09 pve kernel:  call_ad+0xc4/0x220 [ip_set]
Dec 29 07:01:09 pve kernel:  ? hash_net6_uadt+0x200/0x200 [ip_set_hash_net]
Dec 29 07:01:09 pve kernel:  ? call_ad+0xc4/0x220 [ip_set]
Dec 29 07:01:09 pve kernel:  ? _cond_resched+0x1a/0x50
Dec 29 07:01:09 pve kernel:  ? validate_nla+0x15b/0x240
Dec 29 07:01:09 pve kernel:  ? nla_parse+0xaa/0x140
Dec 29 07:01:09 pve kernel:  ip_set_uadd+0x23c/0x2a0 [ip_set]
Dec 29 07:01:09 pve kernel:  nfnetlink_rcv_msg+0x240/0x260 [nfnetlink]
Dec 29 07:01:09 pve kernel:  ? nfnetlink_net_exit_batch+0x70/0x70 [nfnetlink]
Dec 29 07:01:09 pve kernel:  netlink_rcv_skb+0xd9/0x110
Dec 29 07:01:09 pve kernel:  nfnetlink_rcv+0x6c/0x72a [nfnetlink]
Dec 29 07:01:09 pve kernel:  ? _cond_resched+0x1a/0x50
Dec 29 07:01:09 pve kernel:  ? __kmalloc_node_track_caller+0x1f8/0x2b0
Dec 29 07:01:09 pve kernel:  ? __alloc_skb+0x87/0x1c0
Dec 29 07:01:09 pve kernel:  ? __netlink_lookup+0xd9/0x140
Dec 29 07:01:09 pve kernel:  netlink_unicast+0x19c/0x240
Dec 29 07:01:09 pve kernel:  netlink_sendmsg+0x2d7/0x3d0
Dec 29 07:01:09 pve kernel:  sock_sendmsg+0x3e/0x50
Dec 29 07:01:09 pve kernel:  SYSC_sendto+0x101/0x190
Dec 29 07:01:09 pve kernel:  ? __secure_computing+0x3f/0x100
Dec 29 07:01:09 pve kernel:  ? syscall_trace_enter+0xca/0x2e0
Dec 29 07:01:09 pve kernel:  SyS_sendto+0xe/0x10
Dec 29 07:01:09 pve kernel:  do_syscall_64+0x73/0x130
Dec 29 07:01:09 pve kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Dec 29 07:01:09 pve kernel: RIP: 0033:0x7f193a43908b
Dec 29 07:01:09 pve kernel: RSP: 002b:00007ffcb28d76e8 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 29 07:01:09 pve kernel: RAX: ffffffffffffffda RBX: 00005596ca3bbf70 RCX: 00007f193a43908b
Dec 29 07:01:09 pve kernel: RDX: 00000000000004dc RSI: 00005596ca3b9c78 RDI: 0000000000000003
Dec 29 07:01:09 pve kernel: RBP: 00005596ca3b9c78 R08: 00007f193a504000 R09: 000000000000000c
Dec 29 07:01:09 pve kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
Dec 29 07:01:09 pve kernel: R13: 00005596c8b3b137 R14: 00005596c8b3b13f R15: 0000000000000000
Dec 29 07:01:09 pve kernel: Code: c6 04 29 f2 be 01 00 00 00 31 d1 c1 c2 0e 29 d1 31 c8 89 ca 0f b6 4b 08 c1 ca 08 29 d0 d3 e6 83 ee 01 89 f7 21 c7 48 8b 44 fb 10 <48> 89 7d 98 48 85 c0 48 89 45 d0 0f 84 b3 02 00 00 0f b6 50 19 
Dec 29 07:01:10 pve sshd[6814]: Failed password for root from 218.92.1.175 port 41940 ssh2
Dec 29 07:01:13 pve sshd[6814]: Failed password for root from 218.92.1.175 port 41940 ssh2
Dec 29 07:01:16 pve sshd[6814]: Failed password for root from 218.92.1.175 port 41940 ssh2
Dec 29 07:01:19 pve sshd[6814]: Failed password for root from 218.92.1.175 port 41940 ssh2
Dec 29 07:01:21 pve sshd[6814]: Failed password for root from 218.92.1.175 port 41940 ssh2
Dec 29 07:01:25 pve sshd[6814]: Failed password for root from 218.92.1.175 port 41940 ssh2
Dec 29 07:01:25 pve sshd[6814]: error: maximum authentication attempts exceeded for root from 218.92.1.175 port 41940 ssh2 [preauth]
Dec 29 07:01:25 pve sshd[6814]: Disconnecting: Too many authentication failures [preauth]
Dec 29 07:01:25 pve sshd[6814]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.92.1.175  user=root
Dec 29 07:01:25 pve sshd[6814]: PAM service(sshd) ignoring max retries; 6 > 3
Dec 29 07:01:38 pve sshd[6994]: Received disconnect from 218.92.1.175 port 1363:11:  [preauth]
Dec 29 07:01:38 pve sshd[6994]: Disconnected from 218.92.1.175 port 1363 [preauth]
Dec 29 07:02:04 pve lxcfs[916]: bindings.c: 2133: recv_creds: Timed out waiting for scm_cred: Success
Dec 29 07:02:05 pve systemd[1]: Starting Proxmox VE replication runner...
Dec 29 07:02:37 pve lxcfs[916]: bindings.c: 2087: send_creds: Failed at sendmsg: Connection refused.
Dec 29 07:03:05 pve kernel: INFO: task ipset:4944 blocked for more than 120 seconds.
Dec 29 07:03:16 pve kernel:       Tainted: P           O L   4.15.18-9-pve #1
Dec 29 07:03:25 pve kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 29 07:03:30 pve kernel: ipset           D    0  4944   1840 0x00000000
Dec 29 07:03:44 pve kernel: Call Trace:
Dec 29 07:03:48 pve kernel:  __schedule+0x3e0/0x870
Dec 29 07:03:59 pve kernel:  schedule+0x36/0x80
Dec 29 07:04:00 pve kernel:  schedule_preempt_disabled+0xe/0x10
Dec 29 07:04:10 pve kernel:  __mutex_lock.isra.2+0x2b1/0x4e0
Dec 29 07:04:21 pve kernel:  __mutex_lock_slowpath+0x13/0x20
Dec 29 07:04:27 pve kernel:  ? __mutex_lock_slowpath+0x13/0x20
Dec 29 07:04:42 pve kernel:  mutex_lock+0x2f/0x40
Dec 29 07:04:58 pve kernel:  nfnetlink_rcv_msg+0x1ab/0x260 [nfnetlink]

 

[Don Daniello]

I've tracked this down to a specific ipset command running in unprivileged LXC container.

On the host, when it's not crashing, it "looks" like this:

Message from syslogd@pve at Jan  1 19:25:29 ...
 kernel:[86763.415164] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ipset:11633]
Message from syslogd@pve at Jan  2 07:26:37 ...
 kernel:[130031.122370] watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ipset:5960]
Message from syslogd@pve at Jan  3 07:28:41 ...
 kernel:[216554.547819] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ipset:20348]
Message from syslogd@pve at Jan  3 19:29:49 ...
 kernel:[259822.259793] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ipset:28478]
Message from syslogd@pve at Jan  4 07:31:01 ...
 kernel:[303093.973464] watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ipset:11648]
Message from syslogd@pve at Jan  4 19:31:57 ...
 kernel:[346349.694019] watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ipset:30974]
Message from syslogd@pve at Jan  5 07:33:05 ...
 kernel:[389617.410331] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ipset:9968]
Message from syslogd@pve at Jan  5 19:34:21 ...
 kernel:[432893.135628] watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [ipset:17948]
Message from syslogd@pve at Jan  7 07:37:41 ...
 kernel:[562692.231819] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ipset:2220]

While the container gets this failure at the end of the long "lockup" cause by import of large ipset:

Jan 07 08:38:15 ct ipset[761]: ipset v7.0: Error in line 3139: Kernel error received: Cannot allocate memory

Details of ipset CLI:

# ipset --version
ipset v7.0, protocol version: 7
Warning: Kernel support protocol versions 6-6 while userspace supports protocol versions 6-7

It looks like there's a memory limit put on the ipset size of the container (not sure if this is intentional limit of LXC container or just an accidental hit of the host's capacity for ipsets) but in any case, it probably shouldn't be freezing that much and leading to a crash.
I noticed that the load increases 9 times (typically around 0.50, while ipset restore is running/hanging it's 4.5).

Is anything else useful in troubleshooting the kernel panic here?

 

[mira]

Hi, could you post the stacktrace of a stuck process? (cat /proc/<pid>/stack) Should be the same as the ones logged, but still might be of interest.
Which distribution are you running in the container?

 

[Don Daniello]

Hi, could you post the stacktrace of a stuck process? (cat /proc/<pid>/stack) Should be the same as the ones logged, but still might be of interest.

Don't really see much, that's all I see:

# cat /proc/27695/stack
[<0>] 0xffffffffffffffff

Which distribution are you running in the container?

Arch Linux.

I also noticed that on the host, the memory does in fact approach 100%. Then I have SWAP (half of RAM) so it doesn't crash right away.

The interesting thing is that I can't tell why the kernel did not panic the last few times that ran (it was originally a daily job). However, it's still a possible DoS attack if an unprivileged container can overload the host like that (all RAM, all CPU, potential panic). I didn't add anything custom to the container config either, no extra permissions.

 

[mira]

Could you provide us with the command you run and perhaps everything needed to reproduce it?

 

[Don Daniello]

Could you provide us with the command you run and perhaps everything needed to reproduce it?

This is the script: https://github.com/DevelopersPL/pkgbuild/blob/master/ipset-nuclear/ipset-nuclear

You only need ipset installed to run it. It's running on Arch Linux unprivileged container with 4 GB RAM, 2 GB SWAP, RAM is initially at 1.5 GB use (the specs shouldn't matter). Host is at 21/32 GB prior to starting the script in the container.

# pveversion --verbose
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-6 (running version: 5.3-6/37b3c8df)
pve-kernel-4.15: 5.2-12
pve-kernel-4.13: 5.2-2
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-8-pve: 4.15.18-28
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-5-pve: 4.15.18-24
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-2-pve: 4.15.18-21
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.13.16-4-pve: 4.13.16-51
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-4-pve: 4.13.13-35
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: not correctly installed
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-34
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

# uname -a
Linux pve 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 GNU/Linux

That should be enough to reproduce the RAM spike, kernel panic does not seem guaranteed now (maybe it needs to go in pair with certain type of network load to crash).

 

[mira]

Thank you, will try to reproduce it.

 

[mira]

I could reproduce it with your script and will report it to the kernel bugtracker. Is it okay if I link to your script in the bugreport?

 

[Don Daniello]

I could reproduce it with your script and will report it to the kernel bugtracker. Is it okay if I link to your script in the bugreport?

Sure, that is fine. Thanks!

 

[mira]

After further tests it can also be reproduced on the host directly with a 4.15.x kernel version. 4.13.16 seems to work. On the other hand i couldn't reproduce it on a 4.20 kernel (host).

 

[Don Daniello]

After further tests it can also be reproduced on the host directly with a 4.15.x kernel version. 4.13.16 seems to work. On the other hand i couldn't reproduce it on a 4.20 kernel (host).

There is a new problem on this same server that we had. Maybe related to this, maybe not.
We were running fine on 4.15.18-8-pve and when we upgraded to 4.15.18-9-pve, the network does not work at all - no errors in console or logs specific to network but there is absolutely no network connectivity working. Going back to 4.15.18-8-pve fixes the problem (it also works on 4.13 and other kernels).

Did anything change in 4.15.18-9-pve that could cause that?