NFS share drops while attempting to restore backup

PVEboi69

New Member
Apr 29, 2020
6
0
1
24
Hello everyone!
For some backstory, I upgraded my server, and I keep backups of all my containers and VMs on a NFS share. I have never had any issues restoring said backups, until today. After a fresh install of Proxmox 7.0-1, I went to restore my backups and it took longer than 15 minutes (the CT is only about 5gb), so I decided to look into it. It looks as if my NFS share is literally dropping as soon as the backup begins to start, as well as getting a bunch of errors related to processes timing out. I tried re-installing 6.4 to see if that would fix my issue, but it did not. Here is a copy of the syslog from that period of trying to back up:


Jul 27 14:15:05 pve1 pvestatd[1377]: got timeout Jul 27 14:15:05 pve1 pvestatd[1377]: status update time (289.526 seconds) Jul 27 14:16:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:16:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:16:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:17:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:17:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:17:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:17:01 pve1 CRON[5240]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jul 27 14:17:01 pve1 CRON[5241]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jul 27 14:17:01 pve1 CRON[5240]: pam_unix(cron:session): session closed for user root Jul 27 14:17:46 pve1 kernel: INFO: task jbd2/loop0-8:2897 blocked for more than 120 seconds. Jul 27 14:17:46 pve1 kernel: Tainted: P O 5.11.22-1-pve #1 Jul 27 14:17:46 pve1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 27 14:17:46 pve1 kernel: task:jbd2/loop0-8 state:D stack: 0 pid: 2897 ppid: 2 flags:0x00004000 Jul 27 14:17:46 pve1 kernel: Call Trace: Jul 27 14:17:46 pve1 kernel: ? bit_wait+0x70/0x70 Jul 27 14:17:46 pve1 kernel: __schedule+0x2ca/0x880 Jul 27 14:17:46 pve1 kernel: ? bit_wait+0x70/0x70 Jul 27 14:17:46 pve1 kernel: schedule+0x4f/0xc0 Jul 27 14:17:46 pve1 kernel: io_schedule+0x46/0x70 Jul 27 14:17:46 pve1 kernel: bit_wait_io+0x11/0x70 Jul 27 14:17:46 pve1 kernel: __wait_on_bit+0x33/0xa0 Jul 27 14:17:46 pve1 kernel: out_of_line_wait_on_bit+0x8d/0xb0 Jul 27 14:17:46 pve1 kernel: ? var_wake_function+0x30/0x30 Jul 27 14:17:46 pve1 kernel: __wait_on_buffer+0x34/0x40 Jul 27 14:17:46 pve1 kernel: jbd2_journal_commit_transaction+0x1554/0x1910 Jul 27 14:17:46 pve1 kernel: ? del_timer_sync+0x6c/0xb0 Jul 27 14:17:46 pve1 kernel: kjournald2+0xa9/0x280 Jul 27 14:17:46 pve1 kernel: ? wait_woken+0x80/0x80 Jul 27 14:17:46 pve1 kernel: kthread+0x12f/0x150 Jul 27 14:17:46 pve1 kernel: ? load_superblock.part.0+0xb0/0xb0 Jul 27 14:17:46 pve1 kernel: ? __kthread_bind_mask+0x70/0x70 Jul 27 14:17:46 pve1 kernel: ret_from_fork+0x22/0x30 Jul 27 14:18:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:18:01 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:18:01 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:19:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:19:01 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:19:01 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:19:24 pve1 pveproxy[1405]: worker exit Jul 27 14:19:24 pve1 pveproxy[1407]: worker exit Jul 27 14:19:24 pve1 pveproxy[1404]: worker 1407 finished Jul 27 14:19:24 pve1 pveproxy[1404]: starting 1 worker(s) Jul 27 14:19:24 pve1 pveproxy[1404]: worker 1405 finished Jul 27 14:19:24 pve1 pveproxy[1404]: worker 5579 started Jul 27 14:19:29 pve1 pveproxy[1404]: starting 1 worker(s) Jul 27 14:19:29 pve1 pveproxy[1404]: worker 5602 started Jul 27 14:20:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:20:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:20:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:20:39 pve1 pvestatd[1377]: got timeout Jul 27 14:20:39 pve1 pvestatd[1377]: status update time (333.925 seconds) Jul 27 14:20:41 pve1 pvestatd[1377]: got timeout Jul 27 14:20:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:20:51 pve1 pvestatd[1377]: got timeout Jul 27 14:20:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:21:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:21:01 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:21:01 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:21:01 pve1 pvestatd[1377]: got timeout Jul 27 14:21:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:21:11 pve1 pvestatd[1377]: got timeout Jul 27 14:21:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:21:21 pve1 pvestatd[1377]: got timeout Jul 27 14:21:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:21:31 pve1 pvestatd[1377]: got timeout Jul 27 14:21:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:21:42 pve1 pvestatd[1377]: got timeout Jul 27 14:21:42 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:21:51 pve1 pvestatd[1377]: got timeout Jul 27 14:21:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:22:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:22:01 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:22:01 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:22:01 pve1 pvestatd[1377]: got timeout Jul 27 14:22:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:22:11 pve1 pvestatd[1377]: got timeout Jul 27 14:22:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:22:21 pve1 pvestatd[1377]: got timeout Jul 27 14:22:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:22:31 pve1 pvestatd[1377]: got timeout Jul 27 14:22:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:22:41 pve1 pvestatd[1377]: got timeout Jul 27 14:22:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:22:51 pve1 pvestatd[1377]: got timeout Jul 27 14:22:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:23:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:23:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:23:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:23:01 pve1 pvestatd[1377]: got timeout Jul 27 14:23:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:23:12 pve1 pvestatd[1377]: got timeout Jul 27 14:23:12 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:23:21 pve1 pvestatd[1377]: got timeout Jul 27 14:23:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:23:31 pve1 pvestatd[1377]: got timeout Jul 27 14:23:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:23:41 pve1 pvestatd[1377]: got timeout Jul 27 14:23:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:23:52 pve1 pvestatd[1377]: got timeout Jul 27 14:23:52 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:24:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:24:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:24:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:24:01 pve1 pvestatd[1377]: got timeout Jul 27 14:24:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:24:11 pve1 pvestatd[1377]: got timeout Jul 27 14:24:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:24:21 pve1 pvestatd[1377]: got timeout Jul 27 14:24:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:24:32 pve1 pvestatd[1377]: got timeout Jul 27 14:24:32 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:24:41 pve1 pvestatd[1377]: got timeout Jul 27 14:24:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:24:51 pve1 pvestatd[1377]: got timeout Jul 27 14:24:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:25:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:25:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:25:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:25:01 pve1 pvestatd[1377]: got timeout Jul 27 14:25:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:25:11 pve1 pvestatd[1377]: got timeout Jul 27 14:25:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:25:13 pve1 pvedaemon[1398]: <root@pam> successful auth for user 'root@pam' Jul 27 14:25:22 pve1 pvestatd[1377]: got timeout Jul 27 14:25:22 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:25:31 pve1 pvestatd[1377]: got timeout Jul 27 14:25:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:25:41 pve1 pvestatd[1377]: got timeout Jul 27 14:25:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:25:51 pve1 pvestatd[1377]: got timeout Jul 27 14:25:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:26:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:26:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:26:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:26:01 pve1 pvestatd[1377]: got timeout Jul 27 14:26:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:26:11 pve1 pvestatd[1377]: got timeout Jul 27 14:26:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:26:21 pve1 pvestatd[1377]: got timeout Jul 27 14:26:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:26:31 pve1 pvestatd[1377]: got timeout Jul 27 14:26:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:26:41 pve1 pvestatd[1377]: got timeout Jul 27 14:26:41 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:26:51 pve1 pvestatd[1377]: got timeout Jul 27 14:26:51 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:27:00 pve1 systemd[1]: Starting Proxmox VE replication runner... Jul 27 14:27:00 pve1 systemd[1]: pvesr.service: Succeeded. Jul 27 14:27:00 pve1 systemd[1]: Finished Proxmox VE replication runner. Jul 27 14:27:01 pve1 pvestatd[1377]: got timeout Jul 27 14:27:01 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:27:11 pve1 pvestatd[1377]: got timeout Jul 27 14:27:11 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:27:21 pve1 pvestatd[1377]: got timeout Jul 27 14:27:21 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:27:31 pve1 pvestatd[1377]: got timeout Jul 27 14:27:31 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:27:42 pve1 pvestatd[1377]: got timeout Jul 27 14:27:42 pve1 pvestatd[1377]: unable to activate storage 'PVE-CENT' - directory '/mnt/pve/PVE-CENT' does not exist or is unreachable Jul 27 14:27:50 pve1 kernel: INFO: task jbd2/loop0-8:2897 blocked for more than 120 seconds.

I am using this NIC in both the TrueNAS as well as PVE host. I booted both into a live ubuntu enviroment and did some IPERF testing and it all came back at full 10gbe, so I know it is not the cards. I also put the IP and domain name of the truenas server in /etc/hosts. If anyone has any ideas on what to do, please let me know. Thank you in advance!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!