[SOLVED] High I/O delay during vzdump Backup and issues Proxmox 8.0

M

mare19

Member
Jul 20, 2021
43
1
13
28
Hi,

I have installed the latest proxmox version 8.0, unfortunately the backup doesn't work. After 5% NFS or SMB cannot respond. This means that the VM is completely busy and Proxmox VE has no connection to the NFS or Samba/Cifs backup.

I did all Upgades on Proxmox but it didn't help.

PVE Version output:

proxmox-ve: 8.0.2 (running kernel: 6.2.16-15-pve)
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)
pve-kernel-6.2: 8.0.5
proxmox-kernel-helper: 8.0.3
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
proxmox-kernel-6.2: 6.2.16-15
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.26-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.5
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.9
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.5
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.3-1
proxmox-backup-file-restore: 3.0.3-1
proxmox-kernel-helper: 8.0.3
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.9
pve-cluster: 8.0.4
pve-container: 5.0.4
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.8-2
pve-ha-manager: 4.0.2
pve-i18n: 3.0.7
pve-qemu-kvm: 8.0.2-6
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.13-pve1

VM config:
agent: 1
boot: order=scsi0;ide0
cores: 8
ide0: local:iso/virtio-win-0.1.189.iso,media=cdrom,size=488766K
machine: pc-q35-5.1
memory: 24576
meta: creation-qemu=8.0.2,ctime=1697700466
name: EX01
net1: virtio=92:10:9B:73:EC:68,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win8
scsi0: Raid10-02:vm-1500-disk-1,size=600G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=1f36c08c-bce5-41d7-bd13-6650c49d4bb8
sockets: 1
vmgenid: c3ac03f7-b7b8-4e12-8b5a-3663098609a5

Running OS on the VM: Windows 2012 R2 (Windows Backup runs without Problems)

Any suggestions?

Thanks
 
Hi,

This looks like a network issue, did you check the monitor during the backup on the network traffic? What error you see in the syslog in the backup time of the VM 1500?

Can you also post the backup task job?
 
Moayad said:
Hi,

This looks like a network issue, did you check the monitor during the backup on the network traffic? What error you see in the syslog in the backup time of the VM 1500?

Can you also post the backup task job?
Click to expand...
Hi Moayad

Here is the output of the journal:

Oct 24 12:21:43 SSD-Server02 pvedaemon[253694]: INFO: starting new backup job: vzdump 1500 --remove 0 --notes-template '{{guestname}}' --compress zstd --storage Backup-Mo-Fr --mode snapshot --node SSD-Server02
Oct 24 12:21:43 SSD-Server02 pvedaemon[253694]: INFO: Starting Backup of VM 1500 (qemu)
Oct 24 12:21:56 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:22:14 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:22:14 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 00000000f0d90215 stuck for 15 seconds
Oct 24 12:22:14 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:22:14 SSD-Server02 pvestatd[3831]: status update time (19.686 seconds)
Oct 24 12:22:16 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:22:16 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:22:26 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:22:42 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:22:42 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 0000000029c2a01b stuck for 15 seconds
Oct 24 12:22:42 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:22:42 SSD-Server02 pvestatd[3831]: status update time (17.672 seconds)
Oct 24 12:22:44 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:22:44 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:22:44 SSD-Server02 pvedaemon[3859]: got timeout
Oct 24 12:22:53 SSD-Server02 pvedaemon[3860]: got timeout
Oct 24 12:22:54 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:23:05 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:23:05 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 00000000912fcacc stuck for 15 seconds
Oct 24 12:23:05 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:23:05 SSD-Server02 pvedaemon[3860]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:23:05 SSD-Server02 pvestatd[3831]: status update time (13.395 seconds)
Oct 24 12:23:08 SSD-Server02 pvedaemon[3862]: got timeout
Oct 24 12:23:08 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:23:13 SSD-Server02 pveproxy[3946]: proxy detected vanished client connection
Oct 24 12:23:32 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 000000009007f4f7 stuck for 15 seconds
Oct 24 12:23:32 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:23:32 SSD-Server02 pvestatd[3831]: status update time (26.600 seconds)
Oct 24 12:23:34 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:23:34 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:23:44 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:24:01 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:24:01 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 000000002f032e74 stuck for 15 seconds
Oct 24 12:24:01 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:24:01 SSD-Server02 pvestatd[3831]: status update time (18.478 seconds)
Oct 24 12:24:03 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:24:03 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:24:13 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:24:29 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:24:29 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 000000000f67d587 stuck for 15 seconds
Oct 24 12:24:29 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:24:29 SSD-Server02 pvestatd[3831]: status update time (18.486 seconds)
Oct 24 12:24:31 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:24:31 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:24:42 SSD-Server02 pvestatd[3831]: got timeout
Oct 24 12:24:53 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 sends on sock 000000002f032e74 stuck for 15 seconds
Oct 24 12:24:53 SSD-Server02 kernel: CIFS: VFS: \\192.168.12.249 Error -11 sending data on socket to server
Oct 24 12:24:53 SSD-Server02 pvestatd[3831]: unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Oct 24 12:24:53 SSD-Server02 pvestatd[3831]: status update time (13.387 seconds)
 
Code: 
unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Can you test by set bandwidth limits in your Proxmox VE node and test the backup (first make sure that the NFS storage is up and running). You can set the bandwidth limit by going to your Proxmox VE node Web UI then Datacenter->Options->Bandwidth Limits.
 
Moayad said:
Code: 
unable to activate storage 'Backup-Mo-Fr' - directory '/mnt/pve/Backup-Mo-Fr' does not exist or is unreachable
Can you test by set bandwidth limits in your Proxmox VE node and test the backup (first make sure that the NFS storage is up and running). You can set the bandwidth limit by going to your Proxmox VE node Web UI then Datacenter->Options->Bandwidth Limits.
Click to expand...
I got it resolved, there are two subnet for my backup-server and I took the wrong subnet. So the whole traffic went over our pfsense Firewall. :D
My bad, thanks anyways for your time!

Best regards,
Marko
 
  • Like
Reactions: Moayad
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back