Network drops if I disconnect the USB keyboard

[msangi]

Hello, yesterday I installed Proxmox on a Fujitsu Esprimo G5010 mini pc and I am seeing a weird behaviour. If the keyboard is plugged in everything works fine. As soon as I unplug it I see

Jun 16 11:02:19 pve kernel: usb 1-6: USB disconnect, device number 2

in the logs and connectivity drops. As far as I can tell the logs don't have any other relevant line and the system itself is otherwise healthy.
If I log in a bring the interface up again with

ifup enp3s0

connectivity comes back.

Has anyone seen a similar issue before? Happy to provide any further log or to perform any test that might be useful to diagnose the issue.

 

[shanreich]

Please provide the full syslogs during the time:

journalctl --since '1 day ago' > syslog.txt

 

[msangi]

Sure, I attached it. The system last booted at Jun 16 10:20:27 and there I unplugged the keyboard at Jun 16 11:02:19

 

[Spoonman2002]

Maybe your onboard nic is connected with the USB hub/controller on the motherboard.
A Raspberry Pi also has this construction.

Can you connect the keyboard to another USB port, if there are any free?
(at the front of the pc perhaps)

Or see what

lsusb

gives for output, you can see the USB ports here

 

[msangi]

Maybe your onboard nic is connected with the USB hub/controller on the motherboard.
A Raspberry Pi also has this construction.

Can you connect the keyboard to another USB port, if there are any free?
(at the front of the pc perhaps)

Or see what

lsusb

gives for output, you can see the USB ports here

I can definitely changing port a try. I was actually already using the front USB-C port but I can try using any of the others and see if it makes a difference.

In the meantime, this is the output of lsusb

root@pve:~# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 8087:0029 Intel Corp. AX200 Bluetooth
Bus 001 Device 006: ID 05ac:029c Apple, Inc. Magic Keyboard
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

I also copy here the output of lspci, which shows the ethernet controller:

root@pve:~# lspci
00:00.0 Host bridge: Intel Corporation Device 9b43 (rev 05)
00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:14.0 USB controller: Intel Corporation Comet Lake PCH-V USB Controller
00:14.2 Signal processing controller: Intel Corporation Comet Lake PCH-V Thermal Subsystem
00:15.0 Signal processing controller: Intel Corporation Device a3e0
00:15.1 Signal processing controller: Intel Corporation Device a3e1
00:16.0 Communication controller: Intel Corporation Device a3ba
00:17.0 SATA controller: Intel Corporation 400 Series Chipset Family SATA AHCI Controller
00:1c.0 PCI bridge: Intel Corporation Device a394 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Device a398 (rev f0)
00:1d.2 PCI bridge: Intel Corporation Device a39a (rev f0)
00:1d.3 PCI bridge: Intel Corporation Device a39b (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a3da
00:1f.2 Memory controller: Intel Corporation Memory controller
00:1f.3 Audio device: Intel Corporation Device a3f0
00:1f.4 SMBus: Intel Corporation Comet Lake PCH-V SMBus Host Controller
01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2018/SN750 / PC SN720 NVMe SSD
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
04:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
[CODE]

 

[Spoonman2002]

That Apple keyboard has Bluetooth connectivity (wireless/network related).
Maybe try a simple el cheapo USB keyboard, just for testing and plug it in an USB-A port?

 

[msangi]

Unfortunately I don't have any other keyboard around to do tests now. I tried using an USB-A to lightning adapter and that didn't really help, even if I tried all the other USB ports.

As an experiment I tried to boot without keyboard plugged in. For a while networking come up (as expected) and the machine responded to ping. Surprisingly that stopped working just right after I opened the Proxmox web UI (which worked) and tried to get a shell. I'm not sure if that was the event that triggered the disconnection or if it was just a coincidence.

Plugging the keyboard and monitor in and doing

ifdown enp3s
ifup enp3s

I cannot really decipher the syslog since boot (attached), but it seems it might have something relevant, or at least suspicious

Jun 16 17:14:37 pve kernel: ------------[ cut here ]------------
Jun 16 17:14:37 pve kernel: NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out
Jun 16 17:14:37 pve kernel: WARNING: CPU: 12 PID: 0 at net/sched/sch_generic.c:477 dev_watchdog+0x277/0x280
Jun 16 17:14:37 pve kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog nfnetlink_log nfnetlink btusb btrtl btbcm btintel bluetooth ecdh_generic ecc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_sof_pci_intel_cnl intel_rapl_msr intel_rapl_common snd_sof_intel_hda_common intel_tcc_cooling soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci x86_pkg_temp_thermal snd_sof_xtensa_dsp intel_powerclamp snd_sof coretemp snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match iwlmvm kvm_intel snd_soc_acpi soundwire_bus ledtrig_audio mac80211 snd_soc_core libarc4 i915 kvm snd_compress ac97_bus snd_pcm_dmaengine irqbypass crct10dif_pclmul ghash_clmulni_intel snd_hda_intel snd_intel_dspcfg aesni_intel ttm crypto_simd snd_intel_sdw_acpi cryptd snd_hda_codec drm_kms_helper snd_hda_core iwlwifi snd_hwdep cec mei_hdcp snd_pcm
Jun 16 17:14:37 pve kernel:  rapl snd_timer rc_core snd i2c_algo_bit fb_sys_fops intel_cstate wmi_bmof pcspkr efi_pstore cfg80211 soundcore syscopyarea mei_me sysfillrect sysimgblt mei fujitsu_laptop mac_hid acpi_pad sparse_keymap acpi_tad zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic xor zstd_compress raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c crc32_pclmul nvme xhci_pci xhci_pci_renesas i2c_i801 r8169 intel_lpss_pci ahci i2c_smbus realtek intel_lpss nvme_core xhci_hcd libahci idma64 wmi video pinctrl_sunrisepoint
Jun 16 17:14:37 pve kernel: CPU: 12 PID: 0 Comm: swapper/12 Tainted: P           O      5.15.102-1-pve #1
Jun 16 17:14:37 pve kernel: Hardware name: FUJITSU CLIENT COMPUTING LIMITED ESPRIMO G5010/D3804-A1, BIOS V5.0.0.17 R1.40.0 for D3804-A1x 04/27/2022
Jun 16 17:14:37 pve kernel: RIP: 0010:dev_watchdog+0x277/0x280
Jun 16 17:14:37 pve kernel: Code: eb 97 48 8b 5d d0 c6 05 d7 c1 4c 01 01 48 89 df e8 ee 50 f9 ff 44 89 e1 48 89 de 48 c7 c7 d0 ee 0a 91 48 89 c2 e8 79 9d 1c 00 <0f> 0b eb 80 e9 bb f5 25 00 0f 1f 44 00 00 55 49 89 ca 48 89 e5 41
Jun 16 17:14:37 pve kernel: RSP: 0018:ffffb06e80404e70 EFLAGS: 00010282
Jun 16 17:14:37 pve kernel: RAX: 0000000000000000 RBX: ffff9a4cd0038000 RCX: 000000000000083f
Jun 16 17:14:37 pve kernel: RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
Jun 16 17:14:37 pve kernel: RBP: ffffb06e80404ea8 R08: 0000000000000003 R09: 0000000000000001
Jun 16 17:14:37 pve kernel: R10: 000000000000000a R11: 572056454454454e R12: 0000000000000000
Jun 16 17:14:37 pve kernel: R13: ffff9a4cd0034080 R14: 0000000000000001 R15: ffff9a4cd00384c0
Jun 16 17:14:37 pve kernel: FS:  0000000000000000(0000) GS:ffff9a4e17500000(0000) knlGS:0000000000000000
Jun 16 17:14:37 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 16 17:14:37 pve kernel: CR2: 00005610d5835954 CR3: 0000000044810003 CR4: 00000000007706e0
Jun 16 17:14:37 pve kernel: PKRU: 55555554
Jun 16 17:14:37 pve kernel: Call Trace:
Jun 16 17:14:37 pve kernel:  <IRQ>
Jun 16 17:14:37 pve kernel:  ? pfifo_fast_enqueue+0x160/0x160
Jun 16 17:14:37 pve kernel:  call_timer_fn+0x29/0x120
Jun 16 17:14:37 pve kernel:  __run_timers.part.0+0x1e1/0x270
Jun 16 17:14:37 pve kernel:  ? ktime_get+0x43/0xc0
Jun 16 17:14:37 pve kernel:  ? lapic_next_deadline+0x2c/0x40
Jun 16 17:14:37 pve kernel:  ? clockevents_program_event+0xa8/0x130
Jun 16 17:14:37 pve kernel:  run_timer_softirq+0x2a/0x60
Jun 16 17:14:37 pve kernel:  __do_softirq+0xd6/0x2ea
Jun 16 17:14:37 pve kernel:  irq_exit_rcu+0x94/0xc0
Jun 16 17:14:37 pve kernel:  sysvec_apic_timer_interrupt+0x80/0x90
Jun 16 17:14:37 pve kernel:  </IRQ>
Jun 16 17:14:37 pve kernel:  <TASK>
Jun 16 17:14:37 pve kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jun 16 17:14:37 pve kernel: RIP: 0010:cpuidle_enter_state+0xd9/0x620
Jun 16 17:14:37 pve kernel: Code: 3d 04 78 be 6f e8 37 36 6d ff 49 89 c7 0f 1f 44 00 00 31 ff e8 78 43 6d ff 80 7d d0 00 0f 85 5e 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6a 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e5 03 00 00
Jun 16 17:14:37 pve kernel: RSP: 0018:ffffb06e80173e38 EFLAGS: 00000246
Jun 16 17:14:37 pve kernel: RAX: ffff9a4e17530bc0 RBX: ffffd06e7fd00b00 RCX: 0000000000000000
Jun 16 17:14:37 pve kernel: RDX: 000000000000000b RSI: 000000004041d0d2 RDI: 0000000000000000
Jun 16 17:14:37 pve kernel: RBP: ffffb06e80173e88 R08: 000000125f5f9858 R09: 00000000002f5530
Jun 16 17:14:37 pve kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffffffff918d4ca0
Jun 16 17:14:37 pve kernel: R13: 0000000000000003 R14: 0000000000000003 R15: 000000125f5f9858
Jun 16 17:14:37 pve kernel:  ? cpuidle_enter_state+0xc8/0x620
Jun 16 17:14:37 pve kernel:  cpuidle_enter+0x2e/0x50
Jun 16 17:14:37 pve kernel:  do_idle+0x20d/0x2b0
Jun 16 17:14:37 pve kernel:  cpu_startup_entry+0x20/0x30
Jun 16 17:14:37 pve kernel:  start_secondary+0x12a/0x180
Jun 16 17:14:37 pve kernel:  secondary_startup_64_no_verify+0xc2/0xcb
Jun 16 17:14:37 pve kernel:  </TASK>
Jun 16 17:14:37 pve kernel: ---[ end trace a7e5b44103064611 ]---

 

[msangi]

A follow up on this. I updated the kernel to 6.2 and I now seem to be able to unplug the keyboard without connection dropping.

 

[msangi]

Another follow up. When I did the kernel upgrade I got 6.2.11-2-pve and everything worked fine. Yesterday I updated to Proxmox VE 8, which upgraded the kernel again to 6.2.16-3-pve and the issue is back again.

I had some time to look into this and it looks like it might be an issue with my network card. I have a

3:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

and while the kernel ships with a module to support it (r8169) there is also a non-free module (r8168).

After I installed r8168-dkms and set these boot parameters (which I found here) I am back to everything working as expected

r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off loglevel=3

Out of curiosity, is there a way for me to see what changed between version 6.2.11-2-pve and 6.2.16-3-pve of the kernel?