No VM/LXC can start after Proxmox update and Synology update to 7.2.1

[Chris]

Thanks for the logs and command outputs.
So there seems to be an issue with the communication to the NFS share your VM/CT disks are located on, e.g. in the output of ps we can see that setting up the loop device is blocked in an uninterruptible sleep state. Therefore the startup never finishes.

root       96123  0.0  0.0   2796  1536 ?        D    23:11   0:00      \_ losetup --show -f /mnt/pve/NFS-SYNO/images/103/vm-103-disk-0.raw

As a first step, please make sure the NFS share is mounted correctly and reachable. Try booting without autostarting any guests and only start the guests after the storage is reachable. Check if that solves your issue.

 

[Chris]

Also see here: https://forum.proxmox.com/threads/nfs-server-problem-führt-zur-nicht-erreichbarkeit-der-gui-von-pve.134290/

I think there is a correlation between the hanging of pvestatd, the pve-nfs-client and the last update of the Synology. Before 7.2.1 on Synology all went fine.

Hi,
thanks for the hint! Yes this definitely is an issue with the NFS share not being reachable/disconnecting during VM/CT startup, the question remains why.

 

[nebz]

thx all, i'll open a ticket at synology's

 

[BillboardsAreAllLearing]

Could you please link to the synology's ticket? Assuming the bug reporting system is public (no idea). And, could we rename this thread to add 'Synology' in the title to help others find this. I'm assuming this will affect quite a few people.

 

[BillboardsAreAllLearing]

Update: I've been able to enable successfully network drives from Synology NAS by manually setting the NFS version at 3 in PVE (Datacenter -> Storage -> Edit 'storage name' -> Advanced). I don't know if it will hold but I can start LXCs with root drives on the NAS.

Beforehand, I disabled all NFS storage mounts (pvesm set -disable 1 'name of storage from datacenter'), reboot nodes to remove the GUI errors, and tested first with manually mounting before changing it in Datacenter:

mount -v -t nfs -o vers=3 192.168.X.XX:/volume4/machine-backups /mnt/pve/synology-backup/

 

[nebz]

Assuming the bug reporting system is public (no idea).

no it's private

The support told me to try with another NFS version too, i'm trying atm

 

[nebz]

It's working on vers=3 ! (but in my case I'll still need 4.1 cause synology is a cluster, and for reconnect in case of balancing node, 4.1 is needed). but that saves me until there ! thx !

 

[nebz]

talked too fast... just working for LXC's... not for VM's ! strangely

 

[BillboardsAreAllLearing]

Glad it worked for LXCs for you too!
I have no VMs on PVE to test - Let us know if you find a workaround.

 

[nebz]

After this night, the NFS v3 felt too so no long time solution

 

[nebz]

I've moved all my CT and VM to lvm-local, and waiting for Synology Support to find a solution (downgrade or fix)

 

[BillboardsAreAllLearing]

It's still going on v3 with no issues on my end.

 

[nebz]

7.2.1-69057 Update 1 should solve the issue... installing​

 

[nebz]

problem solved with that new update !!! NFS 4.1 and working !!

 

[recalion]

I installed the complete OS update "/DSM_DS716%2BII_69057.pat", what finished at my DS716+II with version "DSM 7.2.1-69057 Update 1".

The NFS mount works, but I cannot create backups at the mounted folder.
At PBS the datastore based on an NFS share cannot be loaded.

Today I also got an Samba Update at the paket manager, but Samba shares are working.

Log from aborted backup (LXC):

INFO: starting new backup job: vzdump 104 --mode snapshot --remove 0 --storage ds716 --compress zstd --node proxmox --notes-template '{{guestname}}'
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2023-10-13 00:46:02
INFO: status = running
INFO: CT Name: iobroker
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
Logical volume "snap_vm-104-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tar.zst'
INFO: /bin/bash: line 1: /mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tar.dat: Permission denied
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-104-disk-0_vzdump" successfully removed.
ERROR: Backup of VM 104 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tar.dat' failed: exit code 1
INFO: Failed at 2023-10-13 00:46:03
cp: cannot create regular file '/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.log': Permission denied
INFO: Backup job finished with errors
TASK ERROR: job errors


I see the log file at the NFS share created by Proxmox. That's why I do not understand the "permission denied" info.

 

[BillboardsAreAllLearing]

What are your NFS nodes permissions? Do all your nodes have access to that specific shared drive (see NFS restrictions in Synology on that shared drive)?

 

[Chris]

I installed the complete OS update "/DSM_DS716%2BII_69057.pat", what finished at my DS716+II with version "DSM 7.2.1-69057 Update 1".

The NFS mount works, but I cannot create backups at the mounted folder.
At PBS the datastore based on an NFS share cannot be loaded.

Today I also got an Samba Update at the paket manager, but Samba shares are working.

Log from aborted backup (LXC):

INFO: starting new backup job: vzdump 104 --mode snapshot --remove 0 --storage ds716 --compress zstd --node proxmox --notes-template '{{guestname}}'
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2023-10-13 00:46:02
INFO: status = running
INFO: CT Name: iobroker
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
Logical volume "snap_vm-104-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tar.zst'
INFO: /bin/bash: line 1: /mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tar.dat: Permission denied
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-104-disk-0_vzdump" successfully removed.
ERROR: Backup of VM 104 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tmp' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | zstd '--threads=1' >/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.tar.dat' failed: exit code 1
INFO: Failed at 2023-10-13 00:46:03
cp: cannot create regular file '/mnt/pve/ds716/dump/vzdump-lxc-104-2023_10_13-00_46_02.log': Permission denied
INFO: Backup job finished with errors
TASK ERROR: job errors


I see the log file at the NFS share created by Proxmox. That's why I do not understand the "permission denied" info.

Your error is not related to the issues described in this thread. You have a permission problem on the NFS share site. Backups will be written to the share as the LXC user. So either setup the correct permissions on the share or use a tmp folder, which can be configured as described here https://pve.proxmox.com/pve-docs/pve-admin-guide.html#vzdump_configuration

 

[recalion]

I have a LXC, what makes a backup every night of a couple of files via cron job.
This worked last night as expected.

The failure only occurs with PBS and standard backup functin from Proxmox:

• But I also installed new version of PBS 3.0 at Proxmox itself, after upgrading the old VM from 2.x to 3.x at my NAS was faulty. After that I found out, that the PBS 3.x issue with the DSM 7.2 VM scenario is also known with other users.
  I agree, that there are obviously some permission isues with the PBS chunk folder.
• But this does not explain, why dump files from Proxmox standard backup cannot be saved, although the log file fro the backup task could be stored.

 

[recalion]

• But I also installed new version of PBS 3.0 at Proxmox itself, after upgrading the old VM from 2.x to 3.x at my NAS was faulty. After that I found out, that the PBS 3.x issue with the DSM 7.2 VM scenario is also known with other users.
  I agree, that there are obviously some permission isues with the PBS chunk folder.

I solved the issue with PBS 3.x and the old PBS 2.x directories via NFS share using a solution posted here:
https://forum.proxmox.com/threads/permssion-denied-os-error-13-on-nfs-share.74805/post-462248

On PBS:
Unmount NFS share

On Synology:
delete .lock file

On PBS:
mount -a
refresh dashboard at browser (F5)

>> Tested a Backup Scenario and this works.