Hard disk corrupt after Backup

[taich]

I have a 3 Node PVE Cluster and a NFS-Server as storage. Every node has 2 nic's one (1GiB) for public net and one (10GiB) for NFS-storage connection. The backup server is connected via public lan (1 GiB). Everything runs smoothly as long as no backup runs. As soon as a backup job starts some, any or all VM's get corrupted hard disk's. I have to fsck every VM.

This is not very pleasant.
Does anyone have an idea why this happens?

I now have restricted the backup bandwidth and it seems as corruption has gone.
Since the backup is restricted with 1 GiB on public net I thought this wont be a problem on storage net.

Has anyone an explanation for this?

 

[dietmar]

Maybe the NFS server cant handle that much throughput ...

 

[taich]

Well NFS is connected via 10GiB internal lan, and backup is connected via 1GiB wan.

Here the test from NFS to pve node.

iperf -c 192.168.99.111 ------------------------------------------------------------ Client connecting to 192.168.99.111, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.99.104 port 44706 connected with 192.168.99.111 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 10.9 GBytes 9.35 Gbits/sec

 

[dietmar]

I am not talking about network. Maybe the disk is too slow. Or the NFS server is busy with other things?

Hard to say without any information about that server...

 

[taich]

This is the NFS Server

inxi -b
System:    Host: node4 Kernel: 4.15.0-143-generic x86_64 bits: 64 Console: tty 0 Distro: Ubuntu 18.04.5 LTS
Machine:   Device: desktop Mobo: Supermicro model: X9SCL/X9SCM v: 0123456789 serial: 0123456789
           BIOS: American Megatrends v: 2.3 date: 06/12/2018
CPU:       Quad core Intel Xeon E3-1230 V2 (-MT-MCP-) speed/max: 1600/3700 MHz
Network:   Card-1: Intel 82579LM Gigabit Network Connection (Lewisville) driver: e1000e
           Card-2: Intel 82598EB 10-Gigabit AF Dual Port Network Connection driver: ixgbe
           Card-3: Intel 82598EB 10-Gigabit AF Dual Port Network Connection driver: ixgbe
           Card-4: Intel 82574L Gigabit Network Connection driver: e1000e
Drives:    SSDD Total Size: 4250.8GB (0.1% used)
Info:      Processes: 186 Uptime: 2 days Memory: 508.5/7948.9MB Init: systemd runlevel: 5
           Client: Shell (bash) inxi: 2.3.56

It is hard to believe this machine is too slow. Especially during night is nothing else happening except backup.