Hello everyone!
For some backstory, I upgraded my server, and I keep backups of all my containers and VMs on a NFS share. I have never had any issues restoring said backups, until today. After a fresh install of Proxmox 7.0-1, I went to restore my backups and it took longer than 15 minutes (the CT is only about 5gb), so I decided to look into it. It looks as if my NFS share is literally dropping as soon as the backup begins to start, as well as getting a bunch of errors related to processes timing out. I tried re-installing 6.4 to see if that would fix my issue, but it did not. Here is a copy of the syslog from that period of trying to back up:
I am using this NIC in both the TrueNAS as well as PVE host. I booted both into a live ubuntu enviroment and did some IPERF testing and it all came back at full 10gbe, so I know it is not the cards. I also put the IP and domain name of the truenas server in /etc/hosts. If anyone has any ideas on what to do, please let me know. Thank you in advance!
For some backstory, I upgraded my server, and I keep backups of all my containers and VMs on a NFS share. I have never had any issues restoring said backups, until today. After a fresh install of Proxmox 7.0-1, I went to restore my backups and it took longer than 15 minutes (the CT is only about 5gb), so I decided to look into it. It looks as if my NFS share is literally dropping as soon as the backup begins to start, as well as getting a bunch of errors related to processes timing out. I tried re-installing 6.4 to see if that would fix my issue, but it did not. Here is a copy of the syslog from that period of trying to back up:
Jul 27 14:15:05 pve1 pvestatd[1377]: got timeout
Jul 27 14:15:05 pve1 pvestatd[1377]: status update time (289.526 seconds)
Jul 27 14:16:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:16:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:16:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:17:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:17:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:17:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:17:01 pve1 CRON[5240]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 27 14:17:01 pve1 CRON[5241]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 27 14:17:01 pve1 CRON[5240]: pam_unix(cron:session): session closed for user root
Jul 27 14:17:46 pve1 kernel: INFO: task jbd2/loop0-8:2897 blocked for more than 120 seconds.
Jul 27 14:17:46 pve1 kernel: Tainted: P O 5.11.22-1-pve #1
Jul 27 14:17:46 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 27 14:17:46 pve1 kernel: task:jbd2/loop0-8 state:D stack: 0 pid: 2897 ppid: 2 flags:0x00004000
Jul 27 14:17:46 pve1 kernel: Call Trace:
Jul 27 14:17:46 pve1 kernel: ? bit_wait+0x70/0x70
Jul 27 14:17:46 pve1 kernel: __schedule+0x2ca/0x880
Jul 27 14:17:46 pve1 kernel: ? bit_wait+0x70/0x70
Jul 27 14:17:46 pve1 kernel: schedule+0x4f/0xc0
Jul 27 14:17:46 pve1 kernel: io_schedule+0x46/0x70
Jul 27 14:17:46 pve1 kernel: bit_wait_io+0x11/0x70
Jul 27 14:17:46 pve1 kernel: __wait_on_bit+0x33/0xa0
Jul 27 14:17:46 pve1 kernel: out_of_line_wait_on_bit+0x8d/0xb0
Jul 27 14:17:46 pve1 kernel: ? var_wake_function+0x30/0x30
Jul 27 14:17:46 pve1 kernel: __wait_on_buffer+0x34/0x40
Jul 27 14:17:46 pve1 kernel: jbd2_journal_commit_transaction+0x1554/0x1910
Jul 27 14:17:46 pve1 kernel: ? del_timer_sync+0x6c/0xb0
Jul 27 14:17:46 pve1 kernel: kjournald2+0xa9/0x280
Jul 27 14:17:46 pve1 kernel: ? wait_woken+0x80/0x80
Jul 27 14:17:46 pve1 kernel: kthread+0x12f/0x150
Jul 27 14:17:46 pve1 kernel: ? load_superblock.part.0+0xb0/0xb0
Jul 27 14:17:46 pve1 kernel: ? __kthread_bind_mask+0x70/0x70
Jul 27 14:17:46 pve1 kernel: ret_from_fork+0x22/0x30
Jul 27 14:18:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:18:01 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:18:01 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:19:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:19:01 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:19:01 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:19:24 pve1 pveproxy[1405]: worker exit
Jul 27 14:19:24 pve1 pveproxy[1407]: worker exit
Jul 27 14:19:24 pve1 pveproxy[1404]: worker 1407 finished
Jul 27 14:19:24 pve1 pveproxy[1404]: starting 1 worker(s)
Jul 27 14:19:24 pve1 pveproxy[1404]: worker 1405 finished
Jul 27 14:19:24 pve1 pveproxy[1404]: worker 5579 started
Jul 27 14:19:29 pve1 pveproxy[1404]: starting 1 worker(s)
Jul 27 14:19:29 pve1 pveproxy[1404]: worker 5602 started
Jul 27 14:20:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:20:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:20:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:20:39 pve1 pvestatd[1377]: got timeout
Jul 27 14:20:39 pve1 pvestatd[1377]: status update time (333.925 seconds)
Jul 27 14:20:41 pve1 pvestatd[1377]: got timeout
Jul 27 14:20:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:20:51 pve1 pvestatd[1377]: got timeout
Jul 27 14:20:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:21:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:21:01 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:21:01 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:21:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:21:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:21:11 pve1 pvestatd[1377]: got timeout
Jul 27 14:21:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:21:21 pve1 pvestatd[1377]: got timeout
Jul 27 14:21:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:21:31 pve1 pvestatd[1377]: got timeout
Jul 27 14:21:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:21:42 pve1 pvestatd[1377]: got timeout
Jul 27 14:21:42 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:21:51 pve1 pvestatd[1377]: got timeout
Jul 27 14:21:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:22:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:22:01 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:22:01 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:22:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:22:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:22:11 pve1 pvestatd[1377]: got timeout
Jul 27 14:22:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:22:21 pve1 pvestatd[1377]: got timeout
Jul 27 14:22:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:22:31 pve1 pvestatd[1377]: got timeout
Jul 27 14:22:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:22:41 pve1 pvestatd[1377]: got timeout
Jul 27 14:22:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:22:51 pve1 pvestatd[1377]: got timeout
Jul 27 14:22:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:23:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:23:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:23:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:23:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:23:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:23:12 pve1 pvestatd[1377]: got timeout
Jul 27 14:23:12 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:23:21 pve1 pvestatd[1377]: got timeout
Jul 27 14:23:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:23:31 pve1 pvestatd[1377]: got timeout
Jul 27 14:23:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:23:41 pve1 pvestatd[1377]: got timeout
Jul 27 14:23:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:23:52 pve1 pvestatd[1377]: got timeout
Jul 27 14:23:52 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:24:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:24:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:24:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:24:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:24:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:24:11 pve1 pvestatd[1377]: got timeout
Jul 27 14:24:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:24:21 pve1 pvestatd[1377]: got timeout
Jul 27 14:24:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:24:32 pve1 pvestatd[1377]: got timeout
Jul 27 14:24:32 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:24:41 pve1 pvestatd[1377]: got timeout
Jul 27 14:24:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:24:51 pve1 pvestatd[1377]: got timeout
Jul 27 14:24:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:25:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:25:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:25:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:25:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:25:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:25:11 pve1 pvestatd[1377]: got timeout
Jul 27 14:25:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:25:13 pve1 pvedaemon[1398]: <root@pam> successful auth for user 'root@pam'
Jul 27 14:25:22 pve1 pvestatd[1377]: got timeout
Jul 27 14:25:22 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:25:31 pve1 pvestatd[1377]: got timeout
Jul 27 14:25:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:25:41 pve1 pvestatd[1377]: got timeout
Jul 27 14:25:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:25:51 pve1 pvestatd[1377]: got timeout
Jul 27 14:25:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:26:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:26:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:26:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:26:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:26:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:26:11 pve1 pvestatd[1377]: got timeout
Jul 27 14:26:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:26:21 pve1 pvestatd[1377]: got timeout
Jul 27 14:26:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:26:31 pve1 pvestatd[1377]: got timeout
Jul 27 14:26:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:26:41 pve1 pvestatd[1377]: got timeout
Jul 27 14:26:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:26:51 pve1 pvestatd[1377]: got timeout
Jul 27 14:26:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:27:00 pve1 systemd[1]: Starting Proxmox VE replication runner...
Jul 27 14:27:00 pve1 systemd[1]: pvesr.service: Succeeded.
Jul 27 14:27:00 pve1 systemd[1]: Finished Proxmox VE replication runner.
Jul 27 14:27:01 pve1 pvestatd[1377]: got timeout
Jul 27 14:27:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:27:11 pve1 pvestatd[1377]: got timeout
Jul 27 14:27:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:27:21 pve1 pvestatd[1377]: got timeout
Jul 27 14:27:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:27:31 pve1 pvestatd[1377]: got timeout
Jul 27 14:27:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:27:42 pve1 pvestatd[1377]: got timeout
Jul 27 14:27:42 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable
Jul 27 14:27:50 pve1 kernel: INFO: task jbd2/loop0-8:2897 blocked for more than 120 seconds.I am using this NIC in both the TrueNAS as well as PVE host. I booted both into a live ubuntu enviroment and did some IPERF testing and it all came back at full 10gbe, so I know it is not the cards. I also put the IP and domain name of the truenas server in /etc/hosts. If anyone has any ideas on what to do, please let me know. Thank you in advance!