New to Proxmox not new to linux, been using it for some time, but so far I have had multiple issues where I have locked up the host doing testing at least twice now and I am testing what I consider normal stuff, but having these issues doesn't lead me to think things are pretty solid. Thought I would post here and see maybe what could be causing the problems.
Hardware is a Dell precision w/32g memory, boot drive is a raid1 instance and storage is 2 4tb HDD spinning rust in a ZFS pool (rpool), I have a couple of directories defined, one for backup and one for iso's. I have one win11 vm created and that is what I was doing the backup on to the local backup storage on the ZFS pool.
I mean I know spinning rust is not the fastest but I think ZFS should work fine but this leads me to believe differently. Any suggestions as to what is going on would be appreciated since I need to have confidence in this since I am needing in the future to migrate some VMware nodes and starting to think I should also look into xcp-ng.
Hardware is a Dell precision w/32g memory, boot drive is a raid1 instance and storage is 2 4tb HDD spinning rust in a ZFS pool (rpool), I have a couple of directories defined, one for backup and one for iso's. I have one win11 vm created and that is what I was doing the backup on to the local backup storage on the ZFS pool.
Code:
17:22:35 pve1 smartd[1286]: Device: /dev/sdc [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 115 to 117
Aug 09 17:23:07 pve1 pvedaemon[10509]: <root@pam> successful auth for user 'root@pam'
Aug 09 17:27:37 pve1 pvedaemon[10509]: <root@pam> starting task UPID:pve1:0007E713:010DA3DD:6897CB59:vzdump:100:root@pam:
Aug 09 17:27:37 pve1 pvedaemon[517907]: INFO: starting new backup job: vzdump 100 --remove 0 --node pve1 --compress zstd --notes-template 'Win11 latest' --notification-mode auto --mode>
Aug 09 17:27:37 pve1 pvedaemon[517907]: INFO: Starting Backup of VM 100 (qemu)
Aug 09 17:29:07 pve1 pveproxy[512535]: worker exit
Aug 09 17:29:07 pve1 pveproxy[1850]: worker 512535 finished
Aug 09 17:29:07 pve1 pveproxy[1850]: starting 1 worker(s)
Aug 09 17:29:07 pve1 pveproxy[1850]: worker 518265 started
Aug 09 17:29:11 pve1 kernel: perf: interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Aug 09 17:29:23 pve1 kernel: VERIFY3(hdr->b_type == type) failed (1 == 0)
Aug 09 17:29:23 pve1 kernel: PANIC at arc.c:1555:arc_buf_type()
Aug 09 17:29:23 pve1 kernel: Showing stack for process 517927
Aug 09 17:29:23 pve1 kernel: CPU: 0 PID: 517927 Comm: zvol_tq-2 Tainted: P O 6.8.12-13-pve #1
Aug 09 17:29:23 pve1 kernel: Hardware name: Dell Inc. Precision 3630 Tower/0NNNCT, BIOS 2.35.0 06/01/2025
Aug 09 17:29:23 pve1 kernel: Call Trace:
Aug 09 17:29:23 pve1 kernel: <TASK>
Aug 09 17:29:23 pve1 kernel: dump_stack_lvl+0x76/0xa0
Aug 09 17:29:23 pve1 kernel: dump_stack+0x10/0x20
Aug 09 17:29:23 pve1 kernel: spl_dumpstack+0x29/0x40 [spl]
Aug 09 17:29:23 pve1 kernel: spl_panic+0xfc/0x120 [spl]
Aug 09 17:29:23 pve1 kernel: ? arc_buf_fill+0x116/0xd00 [zfs]
Aug 09 17:29:23 pve1 kernel: ? arc_state_multilist_index_func+0x31/0x50 [zfs]
Aug 09 17:29:23 pve1 kernel: add_reference.constprop.0+0x102/0x110 [zfs]
Aug 09 17:29:23 pve1 kernel: arc_buf_alloc_impl.isra.0+0x76/0x330 [zfs]
Aug 09 17:29:23 pve1 kernel: arc_read+0x154d/0x17c0 [zfs]
Aug 09 17:29:23 pve1 kernel: ? __pfx_dbuf_read_done+0x10/0x10 [zfs]
Aug 09 17:29:23 pve1 kernel: ? spl_kmem_cache_alloc+0x71/0x680 [spl]
Aug 09 17:29:23 pve1 kernel: dbuf_read_impl.constprop.0+0x497/0x810 [zfs]
Aug 09 17:29:23 pve1 kernel: ? aggsum_add+0x1ad/0x1d0 [zfs]
Aug 09 17:29:23 pve1 kernel: dbuf_read+0x438/0x7c0 [zfs]
Aug 09 17:29:23 pve1 kernel: dmu_buf_hold_array_by_dnode+0x116/0x6b0 [zfs]
Aug 09 17:29:23 pve1 kernel: dmu_read_uio_dnode+0x5a/0x150 [zfs]
Aug 09 17:29:23 pve1 kernel: zvol_read+0x127/0x430 [zfs]
Aug 09 17:29:23 pve1 kernel: zvol_read_task+0x12/0x30 [zfs]
Aug 09 17:29:23 pve1 kernel: taskq_thread+0x27f/0x4c0 [spl]
Aug 09 17:29:23 pve1 kernel: ? __pfx_default_wake_function+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ? __pfx_zvol_read_task+0x10/0x10 [zfs]
Aug 09 17:29:23 pve1 kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 09 17:29:23 pve1 kernel: kthread+0xef/0x120
Aug 09 17:29:23 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ret_from_fork+0x44/0x70
Aug 09 17:29:23 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ret_from_fork_asm+0x1b/0x30
Aug 09 17:29:23 pve1 kernel: </TASK>
Aug 09 17:29:23 pve1 kernel: arc_read+0x154d/0x17c0 [zfs]
Aug 09 17:29:23 pve1 kernel: ? __pfx_dbuf_read_done+0x10/0x10 [zfs]
Aug 09 17:29:23 pve1 kernel: ? spl_kmem_cache_alloc+0x71/0x680 [spl]
Aug 09 17:29:23 pve1 kernel: dbuf_read_impl.constprop.0+0x497/0x810 [zfs]
Aug 09 17:29:23 pve1 kernel: ? aggsum_add+0x1ad/0x1d0 [zfs]
Aug 09 17:29:23 pve1 kernel: dbuf_read+0x438/0x7c0 [zfs]
Aug 09 17:29:23 pve1 kernel: dmu_buf_hold_array_by_dnode+0x116/0x6b0 [zfs]
Aug 09 17:29:23 pve1 kernel: dmu_read_uio_dnode+0x5a/0x150 [zfs]
Aug 09 17:29:23 pve1 kernel: zvol_read+0x127/0x430 [zfs]
Aug 09 17:29:23 pve1 kernel: zvol_read_task+0x12/0x30 [zfs]
Aug 09 17:29:23 pve1 kernel: taskq_thread+0x27f/0x4c0 [spl]
Aug 09 17:29:23 pve1 kernel: ? __pfx_default_wake_function+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ? __pfx_zvol_read_task+0x10/0x10 [zfs]
Aug 09 17:29:23 pve1 kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 09 17:29:23 pve1 kernel: kthread+0xef/0x120
Aug 09 17:29:23 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ret_from_fork+0x44/0x70
Aug 09 17:29:23 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ret_from_fork_asm+0x1b/0x30
Aug 09 17:29:23 pve1 kernel: </TASK>
Aug 09 17:29:23 pve1 kernel: arc_read+0x154d/0x17c0 [zfs]
Aug 09 17:29:23 pve1 kernel: ? __pfx_dbuf_read_done+0x10/0x10 [zfs]
Aug 09 17:29:23 pve1 kernel: ? spl_kmem_cache_alloc+0x71/0x680 [spl]
Aug 09 17:29:23 pve1 kernel: dbuf_read_impl.constprop.0+0x497/0x810 [zfs]
Aug 09 17:29:23 pve1 kernel: ? aggsum_add+0x1ad/0x1d0 [zfs]
Aug 09 17:29:23 pve1 kernel: dbuf_read+0x438/0x7c0 [zfs]
Aug 09 17:29:23 pve1 kernel: dmu_buf_hold_array_by_dnode+0x116/0x6b0 [zfs]
Aug 09 17:29:23 pve1 kernel: dmu_read_uio_dnode+0x5a/0x150 [zfs]
Aug 09 17:29:23 pve1 kernel: zvol_read+0x127/0x430 [zfs]
Aug 09 17:29:23 pve1 kernel: zvol_read_task+0x12/0x30 [zfs]
Aug 09 17:29:23 pve1 kernel: taskq_thread+0x27f/0x4c0 [spl]
Aug 09 17:29:23 pve1 kernel: ? __pfx_default_wake_function+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ? __pfx_zvol_read_task+0x10/0x10 [zfs]
Aug 09 17:29:23 pve1 kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 09 17:29:23 pve1 kernel: kthread+0xef/0x120
Aug 09 17:29:23 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ret_from_fork+0x44/0x70
Aug 09 17:29:23 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:29:23 pve1 kernel: ret_from_fork_asm+0x1b/0x30
Aug 09 17:29:23 pve1 kernel: </TASK>
17:30:33 pve1 pvedaemon[518391]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:30:35 pve1 pvestatd[1817]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:30:35 pve1 pvestatd[1817]: status update time (8.089 seconds)
Aug 09 17:30:45 pve1 pvestatd[1817]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:30:45 pve1 pvestatd[1817]: status update time (8.087 seconds)
Aug 09 17:30:55 pve1 pvestatd[1817]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:30:55 pve1 pvestatd[1817]: status update time (8.088 seconds)
Aug 09 17:30:59 pve1 pvedaemon[518391]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:31:05 pve1 pvestatd[1817]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:31:05 pve1 pvestatd[1817]: status update time (8.088 seconds)
Aug 09 17:31:15 pve1 pvestatd[1817]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:31:15 pve1 pvestatd[1817]: status update time (8.089 seconds)
Aug 09 17:31:25 pve1 pvedaemon[518391]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:31:25 pve1 pvestatd[1817]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 51 retries
Aug 09 17:31:25 pve1 pvestatd[1817]: status update time (8.090 seconds)
Aug 09 17:31:34 pve1 kernel: INFO: task zvol_tq-2:406 blocked for more than 122 seconds.
Aug 09 17:31:34 pve1 kernel: Tainted: P O 6.8.12-13-pve #1
Aug 09 17:31:34 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 17:31:34 pve1 kernel: task:zvol_tq-2 state:D stack:0 pid:406 tgid:406 ppid:2 flags:0x00004000
Aug 09 17:31:34 pve1 kernel: Call Trace:
Aug 09 17:31:34 pve1 kernel: <TASK>
Aug 09 17:31:34 pve1 kernel: __schedule+0x42b/0x1500
Aug 09 17:31:34 pve1 kernel: schedule+0x33/0x110
Aug 09 17:31:34 pve1 kernel: schedule_preempt_disabled+0x15/0x30
Aug 09 17:31:34 pve1 kernel: __mutex_lock.constprop.0+0x3f8/0x7a0
Aug 09 17:31:34 pve1 kernel: __mutex_lock_slowpath+0x13/0x20
Aug 09 17:31:34 pve1 kernel: mutex_lock+0x3c/0x50
Aug 09 17:31:34 pve1 kernel: arc_buf_destroy+0x62/0x100 [zfs]
Aug 09 17:31:34 pve1 kernel: dbuf_destroy+0x31/0x4a0 [zfs]
Aug 09 17:31:34 pve1 kernel: dbuf_evict_one+0x118/0x180 [zfs]
Aug 09 17:31:34 pve1 kernel: dbuf_rele_and_unlock+0x51f/0x550 [zfs]
Aug 09 17:31:34 pve1 kernel: dbuf_rele+0x3b/0x50 [zfs]
Aug 09 17:31:34 pve1 kernel: dmu_read_uio_dnode+0xec/0x150 [zfs]
Aug 09 17:31:34 pve1 kernel: zvol_read+0x127/0x430 [zfs]
Aug 09 17:31:34 pve1 kernel: zvol_read_task+0x12/0x30 [zfs]
Aug 09 17:31:34 pve1 kernel: taskq_thread+0x27f/0x4c0 [spl]
Aug 09 17:31:34 pve1 kernel: ? finish_task_switch.isra.0+0x8c/0x310
Aug 09 17:31:34 pve1 kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 09 17:31:34 pve1 kernel: ? __pfx_default_wake_function+0x10/0x10
Aug 09 17:31:34 pve1 kernel: ? __pfx_zvol_read_task+0x10/0x10 [zfs]
Aug 09 17:31:34 pve1 kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 09 17:31:34 pve1 kernel: kthread+0xef/0x120
Aug 09 17:31:34 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:31:34 pve1 kernel: ret_from_fork+0x44/0x70
Aug 09 17:31:34 pve1 kernel: ? __pfx_kthread+0x10/0x10
Aug 09 17:31:34 pve1 kernel: ret_from_fork_asm+0x1b/0x30
Aug 09 17:31:34 pve1 kernel: </TASK>
Aug 09 17:31:34 pve1 kernel: INFO: task dbuf_evict:412 blocked for more than 122 seconds.
Aug 09 17:31:34 pve1 kernel: Tainted: P O 6.8.12-13-pve #1
Aug 09 17:31:34 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 09 17:31:34 pve1 kernel: task:dbuf_evict state:D stack:0 pid:412 tgid:412 ppid:2 flags:0x00004000
and then I see this
Aug 09 17:34:22 pve1 zed[519233]: eid=6 class=deadman pool='rpool' vdev=ata-HGST_HUS726040ALA610_K7H99HZL-part1 size=131072 offset=864826679296 priority=3 err=0 flags=0x380080 bookmark>
Aug 09 17:34:22 pve1 zed[519235]: eid=7 class=deadman pool='rpool' vdev=ata-HGST_HUS726040ALA610_K7HB1N9L-part1 size=131072 offset=864826679296 priority=3 err=0 flags=0x180080 bookmarkI mean I know spinning rust is not the fastest but I think ZFS should work fine but this leads me to believe differently. Any suggestions as to what is going on would be appreciated since I need to have confidence in this since I am needing in the future to migrate some VMware nodes and starting to think I should also look into xcp-ng.
Last edited: