problem with nfs storage

B

BillyIT

Member
Mar 6, 2022
30
1
13
44
Hi all,

i'm trying to solve an issue and i need help.

I have a nas that give nfs share to my proxmox cluster.
Some of my nodes can backup sometimes and sometimes can't.

I'd like to be able to find if it's nfs server problems, networks issue or proxmox related...

I will post everything needed like logs or others things i don't think about.

pveversion -v :
Code: 
pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.2-14 (running version: 7.2-14/65898fbc)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.39-3-pve: 5.15.39-3
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-7
libpve-guest-common-perl: 4.2-2
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.2
pve-cluster: 7.2-3
pve-container: 4.3-4
pve-docs: 7.2-3
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.1.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-10
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1
 
I'll try to give you more information but feel free to asks some tests or to point me on right direction.

/var/log/messages :
Code: 
[571637.268896] nfs: server 10.90.0.8 OK
[571637.268897] nfs: server 10.90.0.8 OK
[571655.546032] call_decode: 251 callbacks suppressed
[571655.546038] nfs: server 10.90.0.8 OK
[571655.546039] nfs: server 10.90.0.8 OK
[571655.546042] nfs: server 10.90.0.8 OK
[571655.546046] nfs: server 10.90.0.8 OK
[571655.546048] nfs: server 10.90.0.8 OK
[571655.546052] nfs: server 10.90.0.8 OK
[572114.709760] fwbr114i0: port 2(tap114i0) entered disabled state
[572114.725943] fwbr114i0: port 1(fwln114i0) entered disabled state
[572114.725995] vmbr0: port 2(fwpr114p0) entered disabled state
[572114.726539] device fwln114i0 left promiscuous mode
[572114.726544] fwbr114i0: port 1(fwln114i0) entered disabled state
[572114.751443] device fwpr114p0 left promiscuous mode
[572114.751446] vmbr0: port 2(fwpr114p0) entered disabled state
[572114.785212] device eno1 left promiscuous mode
[573587.908147] nfs: server 10.90.0.8 not responding, still trying
[573617.099726] nfs: server 10.90.0.8 OK
[573798.859143] nfs: server 10.90.0.8 not responding, still trying
[573805.954435] nfs: server 10.90.0.8 OK
[574159.319454] nfs: server 10.90.0.8 not responding, still trying
[574220.239362] nfs: server 10.90.0.8 OK
[574507.491265] nfs: server 10.90.0.8 not responding, still trying
[575004.148371] nfs: server 10.90.0.8 OK
[CODE]
 
Hi,

To narrow down the issue, first I would check:
1. I want to identify the node -> VM that has a backup issue. Is always happen on a specific VM on a specific PVE Node?
2. Did you see the monitor on the Proxmox VE (the network usage during the backup) - you can check for that in the PVE Web UI Datacente -> NodeName -> Network traffic?
3. Can you post the VM config and the Backup Task log and the syslog at the backup time?
- You can find the backup task log in PVE Web UI at the bottom [TASKS] section (double-click on the failed backup) to get the backup task.
- You can get the VM config using qm CLI qm config <VMID> if it's a CT the same command but instead qm do pct.
- To get the syslog with a specific time, you can use journalctl as the below command (you may change the date/time)
Bash: 
journalctl --since "2023-03-20 00:00" --until "2023-03-20 08:45" > /tmp/Syslog.log
3. Please provide us also with the output of pveversion -v and cat /etc/pve/storage.cfg commands.
 
ok, i'll try to do all that and come back

Code: 
mars 20 04:54:56 dauntlessi kernel: nfs: server 10.90.0.8 OK
mars 20 04:54:56 dauntlessi kernel: nfs: server 10.90.0.8 OK
mars 20 04:55:19 dauntlessi pvestatd[1491]: storage 'yt2400' is not online
mars 20 04:55:19 dauntlessi pvestatd[1491]: status update time (10.094 seconds)
mars 20 04:55:29 dauntlessi pvestatd[1491]: storage 'yt2400' is not online
mars 20 04:55:29 dauntlessi pvestatd[1491]: status update time (10.093 seconds)
mars 20 04:55:35 dauntlessi pvestatd[1491]: storage 'yt2400' is not online
mars 20 04:55:35 dauntlessi pvestatd[1491]: status update time (5.767 seconds)
mars 20 04:55:41 dauntlessi pvestatd[1491]: got timeout
mars 20 04:55:51 dauntlessi pvestatd[1491]: got timeout
mars 20 04:55:51 dauntlessi pvestatd[1491]: unable to activate storage 'yt2400' - directory '/mnt/pve/yt2400' does not exist or is unreachable
I have a lot of unable to activate storage / got timeout error

or this maybe can help :
Code: 
mars 20 03:04:44 dauntlessi pmxcfs[1032]: [dcdb] notice: data verification successful
mars 20 03:04:45 dauntlessi pvescheduler[155477]: INFO: Finished Backup of VM 100 (00:04:43)
mars 20 03:04:47 dauntlessi pvescheduler[155477]: INFO: Starting Backup of VM 101 (qemu)
mars 20 03:04:48 dauntlessi systemd[1]: Started 101.scope.
mars 20 03:04:49 dauntlessi systemd-udevd[156351]: Using default interface naming scheme 'v247'.
mars 20 03:04:49 dauntlessi systemd-udevd[156351]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
mars 20 03:04:50 dauntlessi kernel: device tap101i0 entered promiscuous mode
mars 20 03:04:50 dauntlessi kernel: vmbr1: port 3(tap101i0) entered blocking state
mars 20 03:04:50 dauntlessi kernel: vmbr1: port 3(tap101i0) entered disabled state
mars 20 03:04:50 dauntlessi kernel: vmbr1: port 3(tap101i0) entered blocking state
mars 20 03:04:50 dauntlessi kernel: vmbr1: port 3(tap101i0) entered forwarding state
mars 20 03:04:57 dauntlessi pvestatd[1491]: got timeout
mars 20 03:05:07 dauntlessi pvestatd[1491]: got timeout
mars 20 03:05:07 dauntlessi pvestatd[1491]: unable to activate storage 'yt2400' - directory '/mnt/pve/yt2400' does not exist or is unreachable
mars 20 03:05:22 dauntlessi pvestatd[1491]: status update time (6.235 seconds)
 
Answer to question :
1) No, it's not always the same node / vm that have the problem
2) my backup are at night but i'll change to make a backup after lunch to monitor it (but maybe it will hang all)
3) pveversion -v is on my first post

Code: 
cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content snippets,iso,rootdir,backup,images,vztmpl
        prune-backups keep-all=1

dir: d1-sdb
        path /mnt/sdb
        content backup,vztmpl,images,iso,snippets,rootdir
        nodes dauntlessi
        prune-backups keep-all=1
        shared 0

dir: d1-sdc
        path /mnt/sdc
        content iso,snippets,rootdir,backup,vztmpl,images
        nodes dauntlessi
        prune-backups keep-all=1
        shared 0

dir: d1-sdd
        path /mnt/sdd
        content backup,images,vztmpl,snippets,iso,rootdir
        nodes dauntlessi
        prune-backups keep-all=1
        shared 0

dir: d2-sdb
        path /mnt/sdb
        content iso,snippets,rootdir,backup,images,vztmpl
        nodes dauntlessii
        prune-backups keep-all=1
        shared 0

dir: d2-sdc
        path /mnt/sdc
        content rootdir,snippets,iso,images,vztmpl,backup
        nodes dauntlessii
        prune-backups keep-all=1
        shared 0

dir: d2-sdd
        path /mnt/sdd
        content vztmpl,images,backup,rootdir,snippets,iso
        nodes dauntlessii
        prune-backups keep-all=1
        shared 0

nfs: yt2400
        export /ldnas/backup
        path /mnt/pve/yt2400
        server 10.90.0.8
        content iso,snippets,rootdir,backup,images,vztmpl
        prune-backups keep-last=2

dir: h1-big1
        path /mnt/big1
        content vztmpl,images,backup,rootdir,iso,snippets
        nodes homeone
        prune-backups keep-last=2
        shared 0

dir: h1-big2
        path /mnt/big2
        content rootdir,iso,snippets,images,vztmpl,backup
        nodes homeone
        prune-backups keep-all=1
        shared 0

dir: h1-big3
        path /mnt/big3
        content rootdir,snippets,iso,images,vztmpl,backup
        nodes homeone
        prune-backups keep-all=1
        shared 0

dir: h1-sdb
        path /storage2
        content backup,vztmpl,images,snippets,iso,rootdir
        nodes homeone
        prune-backups keep-all=1
        shared 0

dir: h1-sdc
        path /storage3
        content images,rootdir,vztmpl,snippets,iso,backup
        nodes homeone
        prune-backups keep-all=1
        shared 0

dir: h1-sdd
        path /storage4
        content backup,vztmpl,images,iso,snippets,rootdir
        nodes homeone
        prune-backups keep-all=1
        shared 0
 
vm backup logs that was stucked this morning :
Code: 
INFO: starting new backup job: vzdump --all 1 --prune-backups 'keep-last=2' --mailnotification always --storage yt2400 --mode snapshot --quiet 1 --notes-template '{{cluster}}, {{guestname}}, {{node}}, {{vmid}}' --compress zstd
INFO: skip external VMs: 114, 115, 116
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2023-03-20 03:00:03
INFO: status = running
INFO: VM Name: tantiveII
INFO: include disk 'scsi0' 'local:100/vm-100-disk-0.qcow2' 20G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/yt2400/dump/vzdump-qemu-100-2023_03_20-03_00_02.vma.zst'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '5cddef17-e7f4-4b12-8bc6-0a848d8b5b0a'
INFO: resuming VM again
INFO:   6% (1.2 GiB of 20.0 GiB) in 3s, read: 422.9 MiB/s, write: 79.5 MiB/s
INFO:   8% (1.6 GiB of 20.0 GiB) in 6s, read: 137.5 MiB/s, write: 137.3 MiB/s
INFO:  10% (2.0 GiB of 20.0 GiB) in 9s, read: 133.2 MiB/s, write: 133.2 MiB/s
INFO:  11% (2.3 GiB of 20.0 GiB) in 12s, read: 103.4 MiB/s, write: 103.2 MiB/s
INFO:  13% (2.7 GiB of 20.0 GiB) in 15s, read: 136.3 MiB/s, write: 136.0 MiB/s
INFO:  15% (3.1 GiB of 20.0 GiB) in 18s, read: 138.4 MiB/s, write: 138.1 MiB/s
INFO:  17% (3.5 GiB of 20.0 GiB) in 21s, read: 135.5 MiB/s, write: 135.2 MiB/s
INFO:  19% (4.0 GiB of 20.0 GiB) in 24s, read: 148.1 MiB/s, write: 148.0 MiB/s
INFO:  21% (4.3 GiB of 20.0 GiB) in 27s, read: 103.5 MiB/s, write: 103.3 MiB/s
INFO:  23% (4.7 GiB of 20.0 GiB) in 30s, read: 130.1 MiB/s, write: 129.9 MiB/s
INFO:  25% (5.0 GiB of 20.0 GiB) in 33s, read: 130.4 MiB/s, write: 129.7 MiB/s
INFO:  27% (5.4 GiB of 20.0 GiB) in 36s, read: 136.7 MiB/s, write: 136.7 MiB/s
INFO:  28% (5.7 GiB of 20.0 GiB) in 39s, read: 102.8 MiB/s, write: 102.6 MiB/s
INFO:  30% (6.1 GiB of 20.0 GiB) in 42s, read: 132.9 MiB/s, write: 132.5 MiB/s
INFO:  32% (6.5 GiB of 20.0 GiB) in 45s, read: 130.5 MiB/s, write: 130.4 MiB/s
INFO:  34% (6.9 GiB of 20.0 GiB) in 48s, read: 130.3 MiB/s, write: 130.0 MiB/s
INFO:  36% (7.3 GiB of 20.0 GiB) in 51s, read: 133.1 MiB/s, write: 133.0 MiB/s
INFO:  38% (7.7 GiB of 20.0 GiB) in 54s, read: 151.5 MiB/s, write: 151.2 MiB/s
INFO:  40% (8.1 GiB of 20.0 GiB) in 57s, read: 139.7 MiB/s, write: 139.4 MiB/s
INFO:  42% (8.5 GiB of 20.0 GiB) in 1m 1s, read: 97.8 MiB/s, write: 97.4 MiB/s
INFO:  44% (8.9 GiB of 20.0 GiB) in 1m 4s, read: 130.7 MiB/s, write: 130.7 MiB/s
INFO:  46% (9.3 GiB of 20.0 GiB) in 1m 7s, read: 133.8 MiB/s, write: 133.4 MiB/s
INFO:  48% (9.7 GiB of 20.0 GiB) in 1m 10s, read: 124.7 MiB/s, write: 124.5 MiB/s
INFO:  50% (10.0 GiB of 20.0 GiB) in 1m 13s, read: 122.0 MiB/s, write: 121.6 MiB/s
INFO:  51% (10.4 GiB of 20.0 GiB) in 1m 16s, read: 119.8 MiB/s, write: 119.8 MiB/s
INFO:  53% (10.7 GiB of 20.0 GiB) in 1m 19s, read: 118.2 MiB/s, write: 117.8 MiB/s
INFO:  55% (11.0 GiB of 20.0 GiB) in 1m 22s, read: 113.6 MiB/s, write: 113.4 MiB/s
INFO:  56% (11.4 GiB of 20.0 GiB) in 1m 25s, read: 112.4 MiB/s, write: 112.4 MiB/s
INFO:  58% (11.7 GiB of 20.0 GiB) in 1m 28s, read: 112.4 MiB/s, write: 112.0 MiB/s
INFO:  60% (12.0 GiB of 20.0 GiB) in 1m 31s, read: 112.4 MiB/s, write: 112.0 MiB/s
INFO:  61% (12.4 GiB of 20.0 GiB) in 1m 34s, read: 112.2 MiB/s, write: 112.2 MiB/s
INFO:  63% (12.7 GiB of 20.0 GiB) in 1m 37s, read: 112.7 MiB/s, write: 111.6 MiB/s
INFO:  65% (13.0 GiB of 20.0 GiB) in 1m 40s, read: 121.9 MiB/s, write: 121.6 MiB/s
INFO:  66% (13.4 GiB of 20.0 GiB) in 1m 43s, read: 112.2 MiB/s, write: 112.2 MiB/s
INFO:  68% (13.7 GiB of 20.0 GiB) in 1m 46s, read: 112.5 MiB/s, write: 111.8 MiB/s
INFO:  70% (14.1 GiB of 20.0 GiB) in 1m 49s, read: 135.5 MiB/s, write: 135.3 MiB/s
INFO:  72% (14.4 GiB of 20.0 GiB) in 1m 52s, read: 111.7 MiB/s, write: 111.7 MiB/s
INFO:  73% (14.8 GiB of 20.0 GiB) in 1m 55s, read: 111.0 MiB/s, write: 110.7 MiB/s
INFO:  75% (15.1 GiB of 20.0 GiB) in 1m 58s, read: 111.8 MiB/s, write: 111.5 MiB/s
INFO:  77% (15.4 GiB of 20.0 GiB) in 2m 1s, read: 122.0 MiB/s, write: 122.0 MiB/s
INFO:  78% (15.8 GiB of 20.0 GiB) in 2m 4s, read: 110.7 MiB/s, write: 110.4 MiB/s
INFO:  80% (16.1 GiB of 20.0 GiB) in 2m 7s, read: 111.3 MiB/s, write: 111.1 MiB/s
INFO:  82% (16.4 GiB of 20.0 GiB) in 2m 10s, read: 111.5 MiB/s, write: 111.5 MiB/s
INFO:  84% (16.8 GiB of 20.0 GiB) in 2m 13s, read: 134.1 MiB/s, write: 133.9 MiB/s
INFO:  86% (17.2 GiB of 20.0 GiB) in 2m 16s, read: 139.0 MiB/s, write: 138.7 MiB/s
INFO:  87% (17.6 GiB of 20.0 GiB) in 2m 19s, read: 129.2 MiB/s, write: 129.2 MiB/s
INFO:  89% (17.9 GiB of 20.0 GiB) in 2m 22s, read: 110.0 MiB/s, write: 109.8 MiB/s
INFO:  91% (18.2 GiB of 20.0 GiB) in 2m 25s, read: 111.2 MiB/s, write: 111.0 MiB/s
INFO:  92% (18.6 GiB of 20.0 GiB) in 2m 28s, read: 111.0 MiB/s, write: 110.7 MiB/s
INFO:  94% (18.9 GiB of 20.0 GiB) in 2m 31s, read: 107.3 MiB/s, write: 107.3 MiB/s
INFO:  96% (19.2 GiB of 20.0 GiB) in 2m 34s, read: 109.7 MiB/s, write: 109.5 MiB/s
INFO:  99% (20.0 GiB of 20.0 GiB) in 2m 37s, read: 270.3 MiB/s, write: 102.0 MiB/s
INFO: 100% (20.0 GiB of 20.0 GiB) in 2m 38s, read: 1.8 MiB/s, write: 32.0 KiB/s
INFO: backup is sparse: 1.54 GiB (7%) total zero data
INFO: transferred 20.00 GiB in 158 seconds (129.6 MiB/s)
INFO: archive file size: 1.26GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=2
INFO: removing backup 'yt2400:backup/vzdump-qemu-100-2023_03_18-03_00_02.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 100 (00:04:43)
INFO: Backup finished at 2023-03-20 03:04:45
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-03-20 03:04:47
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: RogueShadow
INFO: include disk 'scsi0' 'yt2400:101/vm-101-disk-0.qcow2' 40G
INFO: creating vzdump archive '/mnt/pve/yt2400/dump/vzdump-qemu-101-2023_03_20-03_04_47.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '7d59b1f0-5fa0-43dd-9318-971a83379435'
INFO:   0% (20.0 MiB of 40.0 GiB) in 3s, read: 6.7 MiB/s, write: 1.4 MiB/s
INFO:   1% (415.0 MiB of 40.0 GiB) in 56s, read: 7.5 MiB/s, write: 6.4 MiB/s
INFO:   2% (821.0 MiB of 40.0 GiB) in 2m 10s, read: 5.5 MiB/s, write: 5.3 MiB/s
INFO:   3% (1.2 GiB of 40.0 GiB) in 3m 29s, read: 5.3 MiB/s, write: 5.1 MiB/s
INFO:   4% (1.6 GiB of 40.0 GiB) in 4m 6s, read: 11.3 MiB/s, write: 11.2 MiB/s
INFO:   5% (2.0 GiB of 40.0 GiB) in 4m 35s, read: 13.5 MiB/s, write: 13.5 MiB/s
INFO:   6% (2.4 GiB of 40.0 GiB) in 5m 34s, read: 7.0 MiB/s, write: 6.7 MiB/s
INFO:   7% (2.8 GiB of 40.0 GiB) in 6m 21s, read: 9.1 MiB/s, write: 9.0 MiB/s
INFO:   8% (3.2 GiB of 40.0 GiB) in 7m 2s, read: 9.5 MiB/s, write: 9.2 MiB/s
INFO:   9% (3.6 GiB of 40.0 GiB) in 7m 34s, read: 13.1 MiB/s, write: 12.4 MiB/s
INFO:  10% (4.0 GiB of 40.0 GiB) in 8m 22s, read: 8.8 MiB/s, write: 8.3 MiB/s
INFO:  11% (4.5 GiB of 40.0 GiB) in 8m 50s, read: 15.6 MiB/s, write: 15.3 MiB/s
INFO:  12% (4.8 GiB of 40.0 GiB) in 9m 17s, read: 14.1 MiB/s, write: 13.9 MiB/s
INFO:  13% (5.2 GiB of 40.0 GiB) in 9m 35s, read: 22.0 MiB/s, write: 19.0 MiB/s
INFO:  14% (5.6 GiB of 40.0 GiB) in 9m 57s, read: 20.1 MiB/s, write: 18.9 MiB/s
INFO:  15% (6.0 GiB of 40.0 GiB) in 10m 41s, read: 8.5 MiB/s, write: 8.4 MiB/s
INFO:  16% (6.4 GiB of 40.0 GiB) in 10m 58s, read: 25.0 MiB/s, write: 19.9 MiB/s
INFO:  17% (6.8 GiB of 40.0 GiB) in 11m 22s, read: 16.3 MiB/s, write: 13.7 MiB/s
INFO:  18% (7.2 GiB of 40.0 GiB) in 12m 21s, read: 7.2 MiB/s, write: 6.1 MiB/s
INFO:  19% (7.6 GiB of 40.0 GiB) in 12m 36s, read: 26.9 MiB/s, write: 11.8 MiB/s
INFO:  20% (8.0 GiB of 40.0 GiB) in 12m 53s, read: 23.5 MiB/s, write: 6.3 MiB/s
INFO:  21% (8.5 GiB of 40.0 GiB) in 14m 23s, read: 5.1 MiB/s, write: 4.9 MiB/s
INFO:  22% (8.9 GiB of 40.0 GiB) in 14m 28s, read: 84.1 MiB/s, write: 69.7 MiB/s
INFO:  23% (9.2 GiB of 40.0 GiB) in 14m 32s, read: 88.5 MiB/s, write: 3.0 MiB/s
INFO:  24% (9.7 GiB of 40.0 GiB) in 16m 29s, read: 4.2 MiB/s, write: 0 B/s
INFO:  25% (10.0 GiB of 40.0 GiB) in 16m 46s, read: 20.6 MiB/s, write: 8.4 MiB/s
INFO:  26% (10.4 GiB of 40.0 GiB) in 16m 51s, read: 85.6 MiB/s, write: 67.2 MiB/s
INFO:  27% (10.9 GiB of 40.0 GiB) in 20m 41s, read: 1.9 MiB/s, write: 1.2 MiB/s
INFO:  28% (11.3 GiB of 40.0 GiB) in 20m 45s, read: 112.5 MiB/s, write: 0 B/s
INFO:  29% (11.6 GiB of 40.0 GiB) in 20m 48s, read: 112.3 MiB/s, write: 0 B/s
INFO:  30% (12.0 GiB of 40.0 GiB) in 25m 58s, read: 1.2 MiB/s, write: 0 B/s
INFO:  31% (12.4 GiB of 40.0 GiB) in 26m 11s, read: 32.5 MiB/s, write: 31.3 MiB/s
INFO:  32% (12.8 GiB of 40.0 GiB) in 26m 17s, read: 68.7 MiB/s, write: 53.6 MiB/s
INFO:  33% (13.3 GiB of 40.0 GiB) in 26m 21s, read: 112.5 MiB/s, write: 0 B/s
INFO:  34% (13.7 GiB of 40.0 GiB) in 26m 25s, read: 112.2 MiB/s, write: 0 B/s
INFO:  35% (14.0 GiB of 40.0 GiB) in 26m 28s, read: 109.7 MiB/s, write: 4.3 MiB/s
INFO:  36% (14.4 GiB of 40.0 GiB) in 26m 34s, read: 67.2 MiB/s, write: 62.8 MiB/s
INFO:  37% (14.8 GiB of 40.0 GiB) in 26m 41s, read: 64.4 MiB/s, write: 40.2 MiB/s
INFO:  38% (15.3 GiB of 40.0 GiB) in 26m 45s, read: 112.2 MiB/s, write: 0 B/s
INFO:  39% (15.6 GiB of 40.0 GiB) in 26m 48s, read: 112.3 MiB/s, write: 0 B/s
INFO:  40% (16.0 GiB of 40.0 GiB) in 26m 52s, read: 106.8 MiB/s, write: 33.1 MiB/s
INFO:  41% (16.4 GiB of 40.0 GiB) in 26m 59s, read: 56.0 MiB/s, write: 55.8 MiB/s
INFO:  42% (16.8 GiB of 40.0 GiB) in 27m 6s, read: 59.9 MiB/s, write: 48.8 MiB/s
INFO:  43% (17.3 GiB of 40.0 GiB) in 27m 10s, read: 111.5 MiB/s, write: 0 B/s
INFO:  44% (17.7 GiB of 40.0 GiB) in 27m 14s, read: 112.0 MiB/s, write: 0 B/s
INFO:  45% (18.0 GiB of 40.0 GiB) in 27m 17s, read: 103.7 MiB/s, write: 42.7 MiB/s
INFO:  46% (18.4 GiB of 40.0 GiB) in 27m 21s, read: 106.1 MiB/s, write: 74.4 MiB/s
INFO:  47% (18.8 GiB of 40.0 GiB) in 27m 26s, read: 84.1 MiB/s, write: 84.0 MiB/s
INFO:  48% (19.3 GiB of 40.0 GiB) in 27m 31s, read: 89.2 MiB/s, write: 87.8 MiB/s
INFO:  49% (19.6 GiB of 40.0 GiB) in 28m 47s, read: 4.8 MiB/s, write: 4.6 MiB/s
INFO:  50% (20.0 GiB of 40.0 GiB) in 29m 9s, read: 17.7 MiB/s, write: 15.6 MiB/s
INFO:  51% (20.4 GiB of 40.0 GiB) in 29m 14s, read: 85.5 MiB/s, write: 78.5 MiB/s
INFO:  52% (20.8 GiB of 40.0 GiB) in 1h 1m 49s, read: 231.0 KiB/s, write: 230.7 KiB/s
INFO:  53% (21.2 GiB of 40.0 GiB) in 1h 2m 39s, read: 8.1 MiB/s, write: 8.1 MiB/s
INFO:  54% (21.6 GiB of 40.0 GiB) in 1h 3m 1s, read: 18.8 MiB/s, write: 18.7 MiB/s
INFO:  55% (22.0 GiB of 40.0 GiB) in 1h 3m 20s, read: 19.9 MiB/s, write: 18.9 MiB/s
INFO:  56% (22.4 GiB of 40.0 GiB) in 1h 3m 41s, read: 19.2 MiB/s, write: 18.5 MiB/s
INFO:  57% (22.8 GiB of 40.0 GiB) in 1h 3m 59s, read: 23.2 MiB/s, write: 22.9 MiB/s
INFO:  58% (23.2 GiB of 40.0 GiB) in 1h 4m 17s, read: 23.0 MiB/s, write: 17.0 MiB/s
INFO:  59% (23.6 GiB of 40.0 GiB) in 1h 4m 37s, read: 20.3 MiB/s, write: 17.4 MiB/s
INFO:  60% (24.0 GiB of 40.0 GiB) in 1h 4m 42s, read: 83.9 MiB/s, write: 70.2 MiB/s
INFO:  61% (24.4 GiB of 40.0 GiB) in 1h 5m 8s, read: 14.7 MiB/s, write: 12.3 MiB/s
INFO:  62% (24.8 GiB of 40.0 GiB) in 1h 10m 12s, read: 1.3 MiB/s, write: 1.3 MiB/s
INFO:  63% (25.3 GiB of 40.0 GiB) in 1h 10m 20s, read: 61.5 MiB/s, write: 56.9 MiB/s
INFO:  64% (25.6 GiB of 40.0 GiB) in 1h 13m 45s, read: 1.7 MiB/s, write: 1.7 MiB/s
INFO:  65% (26.0 GiB of 40.0 GiB) in 1h 15m 12s, read: 4.5 MiB/s, write: 4.4 MiB/s
INFO:  66% (26.4 GiB of 40.0 GiB) in 1h 17m 24s, read: 3.2 MiB/s, write: 2.3 MiB/s
INFO:  67% (26.8 GiB of 40.0 GiB) in 1h 17m 32s, read: 53.1 MiB/s, write: 51.2 MiB/s
INFO:  68% (27.2 GiB of 40.0 GiB) in 1h 17m 52s, read: 20.2 MiB/s, write: 19.3 MiB/s
INFO:  69% (27.6 GiB of 40.0 GiB) in 1h 22m 40s, read: 1.4 MiB/s, write: 1.3 MiB/s
INFO:  70% (28.0 GiB of 40.0 GiB) in 1h 23m 1s, read: 19.7 MiB/s, write: 18.5 MiB/s
INFO:  71% (28.4 GiB of 40.0 GiB) in 1h 23m 44s, read: 9.9 MiB/s, write: 7.4 MiB/s
INFO:  72% (28.8 GiB of 40.0 GiB) in 1h 50m, read: 244.3 KiB/s, write: 242.0 KiB/s

qm config 101 :
Code: 
qm config 101
agent: 1
boot: order=scsi0;net0
cores: 4
memory: 8192
meta: creation-qemu=7.0.0,ctime=1665225862
name: RogueShadow
net0: virtio=EA:82:9D:D7:24:E0,bridge=vmbr1
numa: 0
ostype: l26
scsi0: yt2400:101/vm-101-disk-0.qcow2,size=40G
scsihw: virtio-scsi-pci
smbios1: uuid=f8751120-0135-4ea5-a0bd-3031d723c159
sockets: 1
vcpus: 4
vmgenid: f82941f3-ec1b-44ac-bbd3-edd7a1331b7f
 
I think of something... Can it be because my nas is setup to be in raid3 ? i hope it is the cause. If it can be the root cause of my problem, i'll try to change from raid3 to another raid.
 
Can you test the backup with bandwidth limits? This will let us know if the issue is because an IO or huge network traffic during the backups.
 
What's the speed of the NAS connection to the network?

I had similar issues which were fixed by connecting the NAS to the network at higher speed, using LACP/bonding with multiple nics, and enabling NFS4 and multipathing. Zero problems since.
 
robertut said:
What's the speed of the NAS connection to the network?

I had similar issues which were fixed by connecting the NAS to the network at higher speed, using LACP/bonding with multiple nics, and enabling NFS4 and multipathing. Zero problems since.
Click to expand...
I have a problem with my nas. One of my disk just died. I try to change it for a better one but not compatible so i put it again inside my nas and... boom dead.
I'll try all you said later
there is a bonding with 2 gigabit to the nas.
I can't put nfs4, my nas is too old.
 
Yesterday I've tried CEPH just for fun. I added an NVMe SSD to each of my 3 nodes, and added a 2.5Gbit USB nic for the CEPH network. I gave it only about 60% chance that it will work, but it works! Performance is comparable to the old NFS (with 1Gbit switch only)!
 
Last edited:
You must log in or register to reply here.
Top Bottom