mkdir /mnt/vzsnap0: Operation not permitted

RobertusIT

Member
Jul 13, 2021
44
0
6
36
Hi,

I have this error when i try to make a backup from my LXC container.
From my VM no problem.

I have an USB drive mounted in /mnt/USB500GB.

How can i fix it ?

I don't understand why proxmox do that. In the past i had the same usb mounted in /mnt, but now i have more than one usb drive, so i create a folder for every drives mounted, under /mnt.

This is my backup log.

Code:
Task viewer: VM/CT 101 - Backup

INFO: starting new backup job: vzdump 101 --mode snapshot --notes-template '{{guestname}}' --compress zstd --remove 0 --storage FRIGATE --node NUCi3
INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2022-06-30 08:40:46
INFO: status = running
INFO: CT Name: FRIGATE
INFO: including mount point rootfs ('/') in backup
ERROR: Backup of VM 101 failed - mkdir /mnt/vzsnap0: Operation not permitted at /usr/share/perl5/PVE/VZDump/LXC.pm line 178.
INFO: Failed at 2022-06-30 08:40:46
INFO: Backup job finished with errors
TASK ERROR: job errors

I use chattr +i in /mnt, to protect accidentally writes into an umounted folder, but now is mounted and doesn't works...
 
Last edited:

sterzy

Proxmox Staff Member
Staff member
Feb 1, 2022
328
72
28
Hi,

how did you configure the USB as storage on you PVE node? Can you maybe provide the file /etc/pve/storage.cfg?

Thanks!
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
Hi,

how did you configure the USB as storage on you PVE node? Can you maybe provide the file /etc/pve/storage.cfg?

Thanks!
I'm desperate, I edited /usr/share/perl5/PVE/VZDump/LXC.pm in mount point and I replace /mnt/vzsnap0 to mnt/USB500GB/vzsnap , but I rebooted and now i can see what's happened because i don't have here this machine.

I can fix it? How? ;(

I have access from SSH, and i have tried to change in
/usr/share/perl5/PVE/VZDump/LXC.pm
$default_mount_point
like default, so /mnt/vzsnap0

but file is read only and i cant edit now, i don't understand why is on it, because before edit the first time and reboot, this file was writable.

Anyway gui doesn't come up and also vm's doesn't come up, so maybe is this file the issue and need to fix?

Please can you help me?
 
Last edited:

RobertusIT

Member
Jul 13, 2021
44
0
6
36
This is the log, I can't do nothing
 

Attachments

  • Screenshot_20220630-202911.jpg
    Screenshot_20220630-202911.jpg
    960 KB · Views: 11

RobertusIT

Member
Jul 13, 2021
44
0
6
36
I have tried to shutdown and reboot, without success.

There is a way to fix it, from ssh ? ( i didn't have local access for now , i must get an airplane for that...... )
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,462
493
88
Hi,
your screenshot shows that the filesystem where /var/lib/apt/lists belongs to is read-only. If /mnt belongs to the same filesystem, that's likely the reason mkdir fails too. You need to find out why it's read-only. Cheking /var/log/syslog and the systemd journal might be a place to start.
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
Hi,
your screenshot shows that the filesystem where /var/lib/apt/lists belongs to is read-only. If /mnt belongs to the same filesystem, that's likely the reason mkdir fails too. You need to find out why it's read-only. Cheking /var/log/syslog and the systemd journal might be a place to start.
Here log, but I can fix it anyway from only SSH?
 

Attachments

  • 192.168.178.31_20220705122813.txt
    638 KB · Views: 1

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,462
493
88
For the Detected Hardware Unit Hang issue, please check out this thread. There's no solution, because it might be a hardware issue, but a reliable workaround.

For the read-only issue, I'd suggest you also check journalctl -b0. Use findmnt to find the relevant drive. I'd check the syslog for message about that, also from during boot.
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
For the Detected Hardware Unit Hang issue, please check out this thread. There's no solution, because it might be a hardware issue, but a reliable workaround.

For the read-only issue, I'd suggest you also check journalctl -b0. Use findmnt to find the relevant drive. I'd check the syslog for message about that, also from during boot.
 

Attachments

  • 192.168.178.31_20220705152725.txt
    696 KB · Views: 2

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,462
493
88
The output seems to have gotten messed up, and I'd guess earlier (i.e. closer after boot) messages might've been more telling. But there is one cut-off message, which might be the proper hint?
Code:
Jul 05 12:35:41 NUCi3 kernel: usb usb2-port4: Cannot enable. Maybe the USB cable i>Jul 05 12:35:41 NUCi3 spiceproxy[1840]: worker exit
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
The output seems to have gotten messed up, and I'd guess earlier (i.e. closer after boot) messages might've been more telling. But there is one cut-off message, which might be the proper hint?
Code:
Jul 05 12:35:41 NUCi3 kernel: usb usb2-port4: Cannot enable. Maybe the USB cable i>Jul 05 12:35:41 NUCi3 spiceproxy[1840]: worker exit
What you suggest ;(
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,462
493
88
Find out what the full error message is. Find out if the drive connected there is the one where /var lives (e.g. findmnt and maybe earlier parts of the logs help). If it really is the cable, I'm afraid that can't be fixed remotely.
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
Find out what the full error message is. Find out if the drive connected there is the one where /var lives (e.g. findmnt and maybe earlier parts of the logs help). If it really is the cable, I'm afraid that can't be fixed remotely.
Previous logs was var/log/syslog, where can I find "findmnt"? Which log file? Inside var/lig
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,462
493
88
findmnt is a command. If it's not installed, you can also use mount without any parameters.
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
findmnt is a command. If it's not installed, you can also use mount without any parameters.
Code:
Linux NUCi3 5.15.30-2-pve #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jun 30 08:10:07 2022 from 192.168.178.51
root@NUCi3:~# findmnt
TARGET                        SOURCE     FSTYPE  OPTIONS
/                             /dev/mapper/pve-root
│                                        ext4    ro,relatime
├─/sys                        sysfs      sysfs   rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security      securityfs securit rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup            cgroup2    cgroup2 rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/pstore            pstore     pstore  rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars efivarfs   efivarf rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf               bpf        bpf     rw,nosuid,nodev,noexec,relatime,mo│ ├─/sys/kernel/debug         debugfs    debugfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing       tracefs    tracefs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/fuse/connections  fusectl    fusectl rw,nosuid,nodev,noexec,relatime
│ └─/sys/kernel/config        configfs   configf rw,nosuid,nodev,noexec,relatime
├─/proc                       proc       proc    rw,relatime
│ ├─/proc/sys/fs/binfmt_misc  systemd-1  autofs  rw,relatime,fd=30,pgrp=1,timeout=0│ └─/proc/fs/nfsd             nfsd       nfsd    rw,relatime
├─/dev                        udev       devtmpf rw,nosuid,relatime,size=8091116k,n│ ├─/dev/pts                  devpts     devpts  rw,nosuid,noexec,relatime,gid=5,mo│ ├─/dev/shm                  tmpfs      tmpfs   rw,nosuid,nodev,inode64
│ ├─/dev/mqueue               mqueue     mqueue  rw,nosuid,nodev,noexec,relatime
│ └─/dev/hugepages            hugetlbfs  hugetlb rw,relatime,pagesize=2M
├─/run                        tmpfs      tmpfs   rw,nosuid,nodev,noexec,relatime,si│ ├─/run/lock                 tmpfs      tmpfs   rw,nosuid,nodev,noexec,relatime,si│ ├─/run/rpc_pipefs           sunrpc     rpc_pip rw,relatime
│ └─/run/user/0               tmpfs      tmpfs   rw,nosuid,nodev,relatime,size=1625├─/mnt/SSD128GB               /dev/sdb1  ext4    rw,relatime,errors=remount-ro
├─/boot/efi                   /dev/sda2  vfat    rw,relatime,fmask=0022,dmask=0022,├─/mnt/USB500GB               /dev/sdd1  ext4    rw,relatime,errors=remount-ro
└─/var/lib/lxcfs              lxcfs      fuse.lx rw,nosuid,nodev,relatime,user_id=0root@NUCi3:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=8091116k,nr_inodes=2022779,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1625096k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (ro,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20728)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sdb1 on /mnt/SSD128GB type ext4 (rw,relatime,errors=remount-ro)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sdd1 on /mnt/USB500GB type ext4 (rw,relatime,errors=remount-ro)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1625092k,nr_inodes=406273,mode=700,inode64)
root@NUCi3:~#

Anyway this issue read only and not web GUI and not any vm started, is when I edited /usr/share/perl5/PVE/VZDump/LXC.pm in mount point and I replace /mnt/vzsnap0 to mnt/USB500GB/vzsnap. I guess that when promos started, /mnt/USB500GB isn't mounted, so file system goes into read only.

Anyway I really hope to fix it remotely. Get an airplane for that is very hard :'(
 

fiona

Proxmox Staff Member
Staff member
Aug 1, 2019
2,462
493
88
The issue is that your root mount is read only (ro), which can be seen here /dev/mapper/pve-root on / type ext4 (ro,relatime). On which drive does the pve volume group live? You can use the command pvs to see which drives make up the volume group (VG) pve.

After you've determined that you can check if it's the same as the USB cable error. Otherwise please check the earlier parts of your journal/syslog for errors regarding the drive with the root mount.
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
The issue is that your root mount is read only (ro), which can be seen here /dev/mapper/pve-root on / type ext4 (ro,relatime). On which drive does the pve volume group live? You can use the command pvs to see which drives make up the volume group (VG) pve.

After you've determined that you can check if it's the same as the USB cable error. Otherwise please check the earlier parts of your journal/syslog for errors regarding the drive with the root mount.
root@NUCi3:~# pvs PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 232.38g <16.00g
root@NUCi3:~#

My root isn't in a usb drive anyway.

Is in the nvme inside my nuc
 

RobertusIT

Member
Jul 13, 2021
44
0
6
36
The issue is that your root mount is read only (ro), which can be seen here /dev/mapper/pve-root on / type ext4 (ro,relatime). On which drive does the pve volume group live? You can use the command pvs to see which drives make up the volume group (VG) pve.

After you've determined that you can check if it's the same as the USB cable error. Otherwise please check the earlier parts of your journal/syslog for errors regarding the drive with the root mount.
Needed other logs?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!