Proxmox 6.0-11 doing a backup of VMs to a FreeNas 11.2-U6 NFS share
During a proxmox backup to a freenas NFS share, I see these errors popping up in the proxmox syslog:
Dec 23 16:31:15 mich1 pvestatd[2636]: unable to activate storage 'freenas_nfs' - directory '/mnt/pve/freenas_nfs' does not exist or is unreachable
When it happens, the proxmox backup status shows the transfer rate drop as follows, the backup hangs and it's taking forever for the backup to finish:
INFO: status: 7% (6077415424/85899345920), sparse 0% (199716864), duration 21, read/write 241/241 MB/s
INFO: status: 8% (7107444736/85899345920), sparse 0% (224649216), duration 25, read/write 257/251 MB/s
INFO: status: 9% (7940538368/85899345920), sparse 0% (232587264), duration 109, read/write 9/9 MB/s -> during the error
INFO: status: 10% (8731164672/85899345920), sparse 0% (239308800), duration 112, read/write 263/261 MB/s
The connection is a 10Gbe Mellanox connectx-3
I'm suspecting an NFS problem vs a network config problem, but I don't want to rule out that maybe I need to tune the kernel params on the mellanox interface (which I've not done). The mellanox is using all defaults.
Here's what's strange: when I go to the proxmox command line- I can copy large files all day long between the proxmox client and the freenas NFS share over he same 10Gbe link with no error. I only get the NFS errors (and hangs) when using proxmox backup. I can use scp at the command line with no errors and I can also use cp to copy files over the NFS mountpoint ( /mnt/pve/freenas_nfs ) with no problems.
Also, the Windows VMs on proxmox use veeam to backup across the same 10gbe network with no errors and speed seems decent (200-300Mb/s)
Here's the client side mount info on the proxmox side:
freenas2:/mnt/data/proxmox on /mnt/pve/freenas_nfs type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.60.1.20,mountvers=3,mountport=1023,mountproto=udp,local_lock=none,addr=10.60.1.20)
Finally, I also tried doing a proxmox backup over the 1GB link and saw NFS errors there too.
They always start to happen at the 6-7% stage of the backup.
Here's my /etc/network/interfaces file:
auto lo
iface lo inet loopback
iface enp7s0 inet manual
iface enp11s0f0 inet manual
iface enp11s0f1 inet manual
auto vmbr0
iface vmbr0 inet static
address 10.60.0.30
netmask 24
gateway 10.60.0.1
bridge-ports enp7s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#1Gbe port 0
auto vmbr2
iface vmbr2 inet static
address 10.60.1.30
netmask 24
bridge-ports enp11s0f0
bridge-stp off
bridge-fd 0
Any ideas?
During a proxmox backup to a freenas NFS share, I see these errors popping up in the proxmox syslog:
Dec 23 16:31:15 mich1 pvestatd[2636]: unable to activate storage 'freenas_nfs' - directory '/mnt/pve/freenas_nfs' does not exist or is unreachable
When it happens, the proxmox backup status shows the transfer rate drop as follows, the backup hangs and it's taking forever for the backup to finish:
INFO: status: 7% (6077415424/85899345920), sparse 0% (199716864), duration 21, read/write 241/241 MB/s
INFO: status: 8% (7107444736/85899345920), sparse 0% (224649216), duration 25, read/write 257/251 MB/s
INFO: status: 9% (7940538368/85899345920), sparse 0% (232587264), duration 109, read/write 9/9 MB/s -> during the error
INFO: status: 10% (8731164672/85899345920), sparse 0% (239308800), duration 112, read/write 263/261 MB/s
The connection is a 10Gbe Mellanox connectx-3
I'm suspecting an NFS problem vs a network config problem, but I don't want to rule out that maybe I need to tune the kernel params on the mellanox interface (which I've not done). The mellanox is using all defaults.
Here's what's strange: when I go to the proxmox command line- I can copy large files all day long between the proxmox client and the freenas NFS share over he same 10Gbe link with no error. I only get the NFS errors (and hangs) when using proxmox backup. I can use scp at the command line with no errors and I can also use cp to copy files over the NFS mountpoint ( /mnt/pve/freenas_nfs ) with no problems.
Also, the Windows VMs on proxmox use veeam to backup across the same 10gbe network with no errors and speed seems decent (200-300Mb/s)
Here's the client side mount info on the proxmox side:
freenas2:/mnt/data/proxmox on /mnt/pve/freenas_nfs type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.60.1.20,mountvers=3,mountport=1023,mountproto=udp,local_lock=none,addr=10.60.1.20)
Finally, I also tried doing a proxmox backup over the 1GB link and saw NFS errors there too.
They always start to happen at the 6-7% stage of the backup.
Here's my /etc/network/interfaces file:
auto lo
iface lo inet loopback
iface enp7s0 inet manual
iface enp11s0f0 inet manual
iface enp11s0f1 inet manual
auto vmbr0
iface vmbr0 inet static
address 10.60.0.30
netmask 24
gateway 10.60.0.1
bridge-ports enp7s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#1Gbe port 0
auto vmbr2
iface vmbr2 inet static
address 10.60.1.30
netmask 24
bridge-ports enp11s0f0
bridge-stp off
bridge-fd 0
Any ideas?
Last edited: