Hello,
I have some problem at my Proxmox host since I move them from one SSD to two SSD's in ZFS Raid 1.
I allready try out what the write here, but it don't solve the problem also a reboot don't help, it came again after some days.
Here what happen at the Proxmox Host.
At the same time in the VM's (there are much more lines but then I have to much characters and because of this I delete some lines what I think are not important.
Maybe somebody can help or you need more details but I don't know what exactly so let me know what you need.
Thank you.
Mario
I have some problem at my Proxmox host since I move them from one SSD to two SSD's in ZFS Raid 1.
I allready try out what the write here, but it don't solve the problem also a reboot don't help, it came again after some days.
Here what happen at the Proxmox Host.
Code:
Jan 3 05:03:31 host pve-firewall[6029]: firewall update time (42.053 seconds)
Jan 3 05:03:32 host systemd[1]: Starting Daily PVE download activities...
Jan 3 05:03:33 host pvestatd[6028]: status update time (44.144 seconds)
Jan 3 05:03:33 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/201: -1
Jan 3 05:03:33 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/200: -1
Jan 3 05:03:33 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/203: -1
Jan 3 05:03:33 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/104: -1
Jan 3 05:03:34 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/ss2/backup: -1
Jan 3 05:03:34 host pmxcfs[5162]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-storage/ss2/backup: /var/lib/rrdcached/db/pve2-storage/ss2/backup: illegal attempt to update using time 1672718613 when last update time is 1672718613 (minimum one second step)
Jan 3 05:03:34 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/ss2/local-zfs: -1
Jan 3 05:03:34 host pmxcfs[5162]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/ss2/local: -1
Jan 3 05:03:36 host pveupdate[3365317]: <root@pam> starting task UPID:ss2:00335F57:02F72809:63B3A918:aptupdate::root@pam:
Jan 3 05:03:38 host pveupdate[3366743]: update new package list: /var/lib/pve-manager/pkgupdates
Jan 3 05:03:39 host pveupdate[3365317]: <root@pam> end task UPID:ss2:00335F57:02F72809:63B3A918:aptupdate::root@pam: OK
Jan 3 05:03:39 host systemd[1]: pve-daily-update.service: Succeeded.
Jan 3 05:03:39 host systemd[1]: Finished Daily PVE download activities.
Jan 3 05:03:39 host systemd[1]: pve-daily-update.service: Consumed 3.378s CPU time.At the same time in the VM's (there are much more lines but then I have to much characters and because of this I delete some lines what I think are not important.
Code:
Jan 3 05:03:33 Monitoring kernel: [497153.797661] watchdog: BUG: soft lockup - CPU#1 stuck for 42s! [jbd2/dm-2-8:378]
Jan 3 05:03:33 Monitoring kernel: [497153.799741] watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [irqbalance:461]
Jan 3 05:03:33 Monitoring kernel: [497154.842935] Modules linked in:
Jan 3 05:03:33 Monitoring kernel: [497154.843561] Modules linked in:
Jan 3 05:03:33 Monitoring kernel: [497154.845196] binfmt_misc
Jan 3 05:03:33 Monitoring kernel: [497154.845201] binfmt_misc
Jan 3 05:03:33 Monitoring kernel: [497154.847277] md4
Jan 3 05:03:33 Monitoring kernel: [497154.847281] md4
Jan 3 05:03:33 Monitoring kernel: [497154.848397] sha512_ssse3
Jan 3 05:03:33 Monitoring kernel: [497154.849753] sha512_ssse3 sha512_generic
Jan 3 05:03:33 Monitoring kernel: [497154.849784] sha512_generic cmac
Jan 3 05:03:33 Monitoring kernel: [497154.855286] cmac
Jan 3 05:03:33 Monitoring kernel: [497154.859783] nls_utf8
Jan 3 05:03:33 Monitoring kernel: [497154.859796] nls_utf8
Jan 3 05:03:33 Monitoring kernel: [497154.860148] cifs
Jan 3 05:03:33 Monitoring kernel: [497154.860159] cifs
Jan 3 05:03:33 Monitoring kernel: [497154.860550] libarc4
Jan 3 05:03:33 Monitoring kernel: [497154.860559] libarc4
Jan 3 05:03:33 Monitoring kernel: [497154.864890] dns_resolver
Jan 3 05:03:33 Monitoring kernel: [497154.864895] dns_resolver
Jan 3 05:03:33 Monitoring kernel: [497154.872756] fscache
Jan 3 05:03:33 Monitoring kernel: [497154.872763] fscache
Jan 3 05:03:33 Monitoring kernel: [497154.874593] libdes
Jan 3 05:03:33 Monitoring kernel: [497154.874639] libdes
Jan 3 05:03:33 Monitoring kernel: [497154.875285] bochs_drm
Jan 3 05:03:33 Monitoring kernel: [497154.875286] bochs_drm
Jan 3 05:03:33 Monitoring kernel: [497154.876257] drm_vram_helper drm_ttm_helper joydev evdev ttm sg serio_raw
Jan 3 05:03:33 Monitoring kernel: [497154.882171] drm_vram_helper drm_ttm_helper joydev evdev ttm sg serio_raw pcspkr
Jan 3 05:03:33 Monitoring kernel: [497154.886368] pcspkr virtio_balloon
Jan 3 05:03:33 Monitoring kernel: [497154.890799] virtio_balloon
Jan 3 05:03:33 Monitoring kernel: [497154.898496] drm_kms_helper
Jan 3 05:03:33 Monitoring kernel: [497154.898497] drm_kms_helper cec qemu_fw_cfg
Jan 3 05:03:33 Monitoring kernel: [497154.906181] cec qemu_fw_cfg button
Jan 3 05:03:33 Monitoring kernel: [497154.907039] button drm
Jan 3 05:03:33 Monitoring kernel: [497154.907486] drm
Jan 3 05:03:33 Monitoring kernel: [497154.911701] sunrpc fuse configfs ip_tables
Jan 3 05:03:33 Monitoring kernel: [497154.918178] sunrpc fuse configfs ip_tables x_tables autofs4
Jan 3 05:03:33 Monitoring kernel: [497154.923022] x_tables autofs4
Jan 3 05:03:33 Monitoring kernel: [497154.924446] ext4
Jan 3 05:03:33 Monitoring kernel: [497154.924451] ext4 crc16
Jan 3 05:03:33 Monitoring kernel: [497154.931184] crc16
Jan 3 05:03:33 Monitoring kernel: [497154.932335] mbcache jbd2 crc32c_generic ecb aes_generic
Jan 3 05:03:33 Monitoring kernel: [497154.938182] mbcache jbd2 crc32c_generic ecb aes_generic libaes
Jan 3 05:03:33 Monitoring kernel: [497154.938397] libaes crypto_simd
Jan 3 05:03:33 Monitoring kernel: [497154.939138] crypto_simd
Jan 3 05:03:33 Monitoring kernel: [497154.942400] cryptd
Jan 3 05:03:33 Monitoring kernel: [497154.965465] cryptd glue_helper
Jan 3 05:03:33 Monitoring kernel: [497154.965480] glue_helper xts
Jan 3 05:03:33 Monitoring kernel: [497154.976262] xts dm_crypt
Jan 3 05:03:33 Monitoring kernel: [497154.976266] dm_crypt dm_mod
Jan 3 05:03:33 Monitoring kernel: [497154.987637] dm_mod
Jan 3 05:03:33 Monitoring kernel: [497154.997781] hid_generic
Jan 3 05:03:33 Monitoring kernel: [497154.997783] hid_generic usbhid
Jan 3 05:03:33 Monitoring kernel: [497155.002177] usbhid
Jan 3 05:03:33 Monitoring kernel: [497155.013436] hid
Jan 3 05:03:33 Monitoring kernel: [497155.013437] hid sd_mod
Jan 3 05:03:33 Monitoring kernel: [497155.013439] sd_mod t10_pi
Jan 3 05:03:33 Monitoring kernel: [497155.020850] t10_pi
Jan 3 05:03:33 Monitoring kernel: [497155.028965] crc_t10dif
Jan 3 05:03:33 Monitoring kernel: [497155.030201] crc_t10dif crct10dif_generic
Jan 3 05:03:33 Monitoring kernel: [497155.033327] crct10dif_generic
Jan 3 05:03:33 Monitoring kernel: [497155.033873] virtio_net
Jan 3 05:03:33 Monitoring kernel: [497155.033876] virtio_net crct10dif_common
Jan 3 05:03:33 Monitoring kernel: [497155.038676] crct10dif_common
Jan 3 05:03:33 Monitoring kernel: [497155.053605] net_failover
Jan 3 05:03:33 Monitoring kernel: [497155.053607] net_failover failover
Jan 3 05:03:33 Monitoring kernel: [497155.063534] failover
Jan 3 05:03:33 Monitoring kernel: [497155.064700] ata_generic uhci_hcd ehci_hcd
Jan 3 05:03:33 Monitoring kernel: [497155.070191] ata_generic uhci_hcd ehci_hcd psmouse
Jan 3 05:03:33 Monitoring kernel: [497155.072973] psmouse virtio_pci
Jan 3 05:03:33 Monitoring kernel: [497155.075041] virtio_pci virtio_ring
Jan 3 05:03:33 Monitoring kernel: [497155.076818] virtio_ring
Jan 3 05:03:33 Monitoring kernel: [497155.077587] virtio
Jan 3 05:03:33 Monitoring kernel: [497155.077588] virtio usbcore i2c_piix4
Jan 3 05:03:33 Monitoring kernel: [497155.082182] usbcore i2c_piix4 ata_piix
Jan 3 05:03:33 Monitoring kernel: [497155.082857] ata_piix libata
Jan 3 05:03:33 Monitoring kernel: [497155.087604] libata usb_common scsi_mod floppy
Jan 3 05:03:33 Monitoring kernel: [497155.091534] CPU: 1 PID: 378 Comm: jbd2/dm-2-8 Tainted: G L 5.10.0-20-amd64 #1 Debian 5.10.158-2
Jan 3 05:03:33 Monitoring kernel: [497155.092743] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Jan 3 05:03:33 Monitoring kernel: [497155.097616] RIP: 0010:_raw_spin_unlock_irqrestore+0x11/0x20
Jan 3 05:03:33 Monitoring kernel: [497155.097620] Code: 7c ff 4c 29 e0 4c 39 f0 76 d8 80 4d 00 04 eb 8e cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 e8 da 42 7f ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 c3 cc cc cc cc 0f 1f 44 00 00 0f 1f 44 00 00 8b 07
Jan 3 05:03:33 Monitoring kernel: [497155.097622] RSP: 0018:ffffa603c00b8e98 EFLAGS: 00000246
Jan 3 05:03:33 Monitoring kernel: [497155.097625] RAX: 0000000000000001 RBX: ffffa603c00b8ea8 RCX: ffffffff86dc9580
Jan 3 05:03:33 Monitoring kernel: [497155.097626] RDX: 000000000000000c RSI: 0000000000000246 RDI: 0000000000000246
Jan 3 05:03:33 Monitoring kernel: [497155.097627] RBP: ffffa603c00b8ee0 R08: 6066a5f621210b9c R09: da8117716066a5f6
Jan 3 05:03:33 Monitoring kernel: [497155.097628] R10: 000000002741c4bf R11: 00000000d3c06953 R12: 0000000000000246
Jan 3 05:03:33 Monitoring kernel: [497155.097629] R13: 0000000080000001 R14: 0000000000000000 R15: ffff90fbbed24b40
Jan 3 05:03:33 Monitoring kernel: [497155.097632] FS: 0000000000000000(0000) GS:ffff90fbbed00000(0000) knlGS:0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.097633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 3 05:03:33 Monitoring kernel: [497155.097634] CR2: 00005582d13a2ec8 CR3: 0000000015bf0000 CR4: 00000000000006e0
Jan 3 05:03:33 Monitoring kernel: [497155.097638] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.097640] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 3 05:03:33 Monitoring kernel: [497155.097640] Call Trace:
Jan 3 05:03:33 Monitoring kernel: [497155.098185] usb_common scsi_mod floppy
Jan 3 05:03:33 Monitoring kernel: [497155.098192] CPU: 0 PID: 461 Comm: irqbalance Tainted: G L 5.10.0-20-amd64 #1 Debian 5.10.158-2
Jan 3 05:03:33 Monitoring kernel: [497155.098193] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Jan 3 05:03:33 Monitoring kernel: [497155.099913] RIP: 0010:__seccomp_filter+0x6f/0x680
Jan 3 05:03:33 Monitoring kernel: [497155.099917] Code: c0 fb 01 00 4c 8b b8 60 0b 00 00 48 c7 04 24 00 00 00 00 bd 00 00 ff 7f 4d 85 ff 0f 84 1c 04 00 00 4d 8b 77 18 0f 1f 44 00 00 <49> 8b 46 30 49 8d 76 38 4c 89 e7 e8 01 78 a8 00 89 c3 89 d9 89 ea
Jan 3 05:03:33 Monitoring kernel: [497155.099918] RSP: 0018:ffffa603c02e3e58 EFLAGS: 00010282
Jan 3 05:03:33 Monitoring kernel: [497155.099920] RAX: ffff90fba1809680 RBX: 0000000000000100 RCX: 0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.099921] RDX: ffff90fba1809680 RSI: 0000000000000000 RDI: 0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.101419] RBP: 000000007fff0000 R08: 000055d9153a3b05 R09: 00000000ffffff9c
Jan 3 05:03:33 Monitoring kernel: [497155.101421] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa603c02e3e98
Jan 3 05:03:33 Monitoring kernel: [497155.101422] R13: 0000000000000000 R14: ffffa603c02f5000 R15: ffff90fbb5a40d20
Jan 3 05:03:33 Monitoring kernel: [497155.101423] FS: 00007f782009f400(0000) GS:ffff90fbbec00000(0000) knlGS:0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.101424] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 3 05:03:33 Monitoring kernel: [497155.101425] CR2: 00007f876bd40430 CR3: 0000000008c20000 CR4: 00000000000006f0
Jan 3 05:03:33 Monitoring kernel: [497155.101427] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.101428] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 3 05:03:33 Monitoring kernel: [497155.101428] Call Trace:
Jan 3 05:03:33 Monitoring kernel: [497155.102065] <IRQ>
Jan 3 05:03:33 Monitoring kernel: [497155.117806] mix_interrupt_randomness+0xaf/0x100
Jan 3 05:03:33 Monitoring kernel: [497155.127143] ? __next_timer_interrupt+0x94/0x110
Jan 3 05:03:33 Monitoring kernel: [497155.127147] ? crng_reseed+0x100/0x100
Jan 3 05:03:33 Monitoring kernel: [497155.127149] call_timer_fn+0x29/0x100
Jan 3 05:03:33 Monitoring kernel: [497155.134818] syscall_trace_enter.constprop.0+0x89/0x1b0
Jan 3 05:03:33 Monitoring kernel: [497155.134822] __run_timers.part.0+0x1d9/0x250
Jan 3 05:03:33 Monitoring kernel: [497155.134843] ? kvm_clock_get_cycles+0xd/0x20
Jan 3 05:03:33 Monitoring kernel: [497155.134845] ? ktime_get+0x38/0xa0
Jan 3 05:03:33 Monitoring kernel: [497155.136479] ? native_apic_msr_write+0x2c/0x40
Jan 3 05:03:33 Monitoring kernel: [497155.136481] ? lapic_next_event+0x1d/0x30
Jan 3 05:03:33 Monitoring kernel: [497155.137747] ? clockevents_program_event+0x8d/0xf0
Jan 3 05:03:33 Monitoring kernel: [497155.138762] do_syscall_64+0xf/0x80
Jan 3 05:03:33 Monitoring kernel: [497155.138791] run_timer_softirq+0x26/0x50
Jan 3 05:03:33 Monitoring kernel: [497155.138805] __do_softirq+0xc5/0x279
Jan 3 05:03:33 Monitoring kernel: [497155.139559] entry_SYSCALL_64_after_hwframe+0x61/0xc6
Jan 3 05:03:33 Monitoring kernel: [497155.140065] asm_call_irq_on_stack+0x12/0x20
Jan 3 05:03:33 Monitoring kernel: [497155.140067] </IRQ>
Jan 3 05:03:33 Monitoring kernel: [497155.140072] do_softirq_own_stack+0x37/0x50
Jan 3 05:03:33 Monitoring kernel: [497155.163339] RIP: 0033:0x7f782049b564
Jan 3 05:03:33 Monitoring kernel: [497155.163351] Code: 84 00 00 00 00 00 44 89 54 24 0c e8 36 58 f9 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 68 58 f9 ff 8b 44
Jan 3 05:03:33 Monitoring kernel: [497155.163352] RSP: 002b:00007ffe760bdff0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
Jan 3 05:03:33 Monitoring kernel: [497155.163354] RAX: ffffffffffffffda RBX: 000055d9154b7400 RCX: 00007f782049b564
Jan 3 05:03:33 Monitoring kernel: [497155.163355] RDX: 0000000000000000 RSI: 000055d9153a3b05 RDI: 00000000ffffff9c
Jan 3 05:03:33 Monitoring kernel: [497155.163356] RBP: 000055d9153a3b05 R08: 0000000000000000 R09: 0000000000000001
Jan 3 05:03:33 Monitoring kernel: [497155.163357] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
Jan 3 05:03:33 Monitoring kernel: [497155.177953] irq_exit_rcu+0x92/0xc0
Jan 3 05:03:33 Monitoring kernel: [497155.177980] R13: 000055d9154b7400 R14: 0000000000000001 R15: 000055d9154b8140
Jan 3 05:03:33 Monitoring kernel: [497155.185517] sysvec_apic_timer_interrupt+0x36/0x80
Jan 3 05:03:33 Monitoring kernel: [497155.185531] asm_sysvec_apic_timer_interrupt+0x12/0x20
Jan 3 05:03:33 Monitoring kernel: [497155.201138] RIP: 0010:jbd2_journal_commit_transaction+0xef2/0x1ad0 [jbd2]
Jan 3 05:03:34 Monitoring kernel: [497155.201145] Code: 70 00 00 e9 fd fc ff ff c7 44 24 70 01 00 00 00 c7 84 24 88 00 00 00 01 00 00 00 e9 1b f8 ff ff f0 41 80 0e 02 e9 5e fc ff ff <f0> 41 80 4e 02 02 e9 48 fc ff ff 4c 8b 7c 24 78 48 8b 5c 24 70 48
Jan 3 05:03:34 Monitoring kernel: [497155.201151] RSP: 0018:ffffa603c0357d00 EFLAGS: 00010246
Jan 3 05:03:34 Monitoring kernel: [497155.201153] RAX: 0000000000000019 RBX: ffff90fb85484800 RCX: 0000000000000000
Jan 3 05:03:34 Monitoring kernel: [497155.201154] RDX: ffffffff86c06278 RSI: 0000000000000002 RDI: ffff90fbb2d45548
Jan 3 05:03:34 Monitoring kernel: [497155.201155] RBP: ffffa603c0357e98 R08: ffffffff86c06270 R09: 0000000000000000
Jan 3 05:03:34 Monitoring kernel: [497155.201156] R10: ffff90fb90b53000 R11: ffffa603c0357b98 R12: 0000000000000000
Jan 3 05:03:34 Monitoring kernel: [497155.201156] R13: ffff90fba87c4f00 R14: ffff90fbb2d45548 R15: ffff90fb82c4c100
Jan 3 05:03:34 Monitoring kernel: [497155.213947] ? sched_clock+0x5/0x10Maybe somebody can help or you need more details but I don't know what exactly so let me know what you need.
Thank you.
Mario
Last edited: