Backups failing after PVE 8 to 9 for specific VM templates (device is not writeable: Permission denied)

T

TimRex

New Member
Oct 22, 2024
15
1
3
Following up from original post, referenced here

please open a new thread pinging me with @fiona and providing the output of the following commands:

Code: 
cat /etc/pve/qemu-server/135.conf
apt install debsums && debsums -c qemu-server
pveversion -v

Does running systemctl reload-or-restart pvescheduler.service pvedaemon.service pveproxy.service pvestatd.service help?
Click to expand...


@fiona thanks for looking into this
Restarting those services does not help, neither does restarting the entire proxmox host (full reboot)

Here's the output of the requested commands
Code: 
# cat /etc/pve/qemu-server/135.conf
boot: order=ide1
cores: 1
cpu: x86-64-v2-AES
ide0: local:iso/virtio-win-0.1.262.iso,media=cdrom,size=708140K
ide1: local-lvm:base-135-disk-0,size=20G
ide2: local:iso/en_windows_vista_sp2_x86_dvd_342266.iso,media=cdrom,size=3167396K
machine: pc-i440fx-9.0
memory: 2048
meta: creation-qemu=9.0.2,ctime=1730623396
name: Windows-Vista-x86-Template
net0: virtio=BC:24:11:95:61:31,bridge=vmbr0,firewall=1
numa: 0
ostype: w2k8
scsihw: virtio-scsi-single
smbios1: uuid=9a34b906-faed-4ffe-a38b-9c07477d153d
sockets: 1
tags: windows
template: 1
vmgenid: a83ed537-40d5-4cee-a0fd-47bfda7ef015

Code: 
# apt install debsums && debsums -c qemu-server
Installing:
  debsums

Installing dependencies:
  libfile-fnmatch-perl

Summary:
  Upgrading: 0, Installing: 2, Removing: 0, Not Upgrading: 3
  Download size: 51.5 kB
  Space needed: 154 kB / 31.8 GB available

Continue? [Y/n] y
Get:1 http://ftp.au.debian.org/debian trixie/main amd64 libfile-fnmatch-perl amd64 0.02-3+b4 [10.3 kB]
Get:2 http://ftp.au.debian.org/debian trixie/main amd64 debsums all 3.0.2.3 [41.2 kB]
Fetched 51.5 kB in 0s (132 kB/s)
Selecting previously unselected package libfile-fnmatch-perl.
(Reading database ... 80694 files and directories currently installed.)
Preparing to unpack .../libfile-fnmatch-perl_0.02-3+b4_amd64.deb ...
Unpacking libfile-fnmatch-perl (0.02-3+b4) ...
Selecting previously unselected package debsums.
Preparing to unpack .../debsums_3.0.2.3_all.deb ...
Unpacking debsums (3.0.2.3) ...
Setting up libfile-fnmatch-perl (0.02-3+b4) ...
Setting up debsums (3.0.2.3) ...
Processing triggers for man-db (2.13.1-1) ...

Code: 
# debsums -c qemu-server
# echo $?
0

Code: 
# pveversion -v
proxmox-ve: 9.0.0 (running kernel: 6.14.8-2-pve)
pve-manager: 9.0.3 (running version: 9.0.3/025864202ebb6109)
proxmox-kernel-helper: 9.0.3
pve-kernel-5.15: 7.4-13
proxmox-kernel-6.14.8-2-pve-signed: 6.14.8-2
proxmox-kernel-6.14: 6.14.8-2
proxmox-kernel-6.8.12-13-pve-signed: 6.8.12-13
proxmox-kernel-6.8: 6.8.12-13
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 19.2.3-pve1
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.3.1-1+pve4
ifupdown2: 3.3.0-1+pmx9
intel-microcode: 3.20250512.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.1
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.3
libpve-apiclient-perl: 3.4.0
libpve-cluster-api-perl: 9.0.6
libpve-cluster-perl: 9.0.6
libpve-common-perl: 9.0.9
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.4
libpve-network-perl: 1.1.6
libpve-rs-perl: 0.10.10
libpve-storage-perl: 9.0.13
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2
lxc-pve: 6.0.4-2
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.0.11-1
proxmox-backup-file-restore: 4.0.11-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.1.1
proxmox-kernel-helper: 9.0.3
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-widget-toolkit: 5.0.5
pve-cluster: 9.0.6
pve-container: 6.0.9
pve-docs: 9.0.8
pve-edk2-firmware: 4.2025.02-4
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.3
pve-firmware: 3.16-3
pve-ha-manager: 5.0.4
pve-i18n: 3.5.2
pve-qemu-kvm: 10.0.2-4
pve-xtermjs: 5.5.0-2
qemu-server: 9.0.16
smartmontools: 7.4-pve1
spiceterm: 3.4.0
swtpm: 0.8.0+pve2
vncterm: 1.9.0
zfsutils-linux: 2.3.3-pve1




Original report here and here and here and here


The quick summary being, that a handful of VM templates are failing to backup since upgrading from PVE 8 to 9
The specific failure is

Code: 
INFO: starting new backup job: vzdump 135 --mode snapshot --notification-mode notification-system --storage PBS_Primary --notes-template '{{guestname}}' --node femputer --remove 0
INFO: Starting Backup of VM 135 (qemu)
INFO: Backup started at 2025-08-10 14:09:58
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Windows-Vista-x86-Template
INFO: include disk 'ide1' 'local-lvm:base-135-disk-0' 20G
INFO: creating Proxmox Backup Server archive 'vm/135/2025-08-10T04:09:58Z'
INFO: enabling encryption
INFO: starting kvm to execute backup task
kvm: -blockdev {"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/pve/base-135-disk-0","node-name":"ed82a0747b40061c861200835773fe9","read-only":false},"node-name":"fd82a0747b40061c861200835773fe9","read-only":false},"node-name":"drive-ide1","read-only":false,"throttle-group":"throttle-drive-ide1"}: The device is not writable: Permission denied
ERROR: start failed: QEMU exited with code 1
INFO: aborting backup job
ERROR: VM 135 not running
VM 135 not running
ERROR: Backup of VM 135 failed - start failed: QEMU exited with code 1
INFO: Failed at 2025-08-10 14:09:58
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Particularly intereting is:
kvm: <snipped> "filename":"/dev/pve/base-135-disk-0" <snipped>: The device is not writable: Permission denied


There have since been some smartd warnings about my drive, though I'm not convinced this is directly related..

  1. The backup errors only started occurring after upgrading PVE 8 to 9 (PBS backups run nightly, so.. ever since the upgrade)
  2. Those backup errors occur immediately on attempted backup, against 4 different templates but all in the same way (The device is not writable: Permission denied) while mounting the storage, rather than at some abritrary block during a read op
  3. The smartd warning didn't present until 3 days *after* upgrading, and *may* be unrelated
  4. Cloning the affected templates (full clone) to new VM's works without issue, indicating the storage for those templates may not be at issue (right?)
  5. I am able to restore all 4 of those templates from previous PBS backups (prior to the upgrade) without issue
  6. Subsequent backup attempts (following the restore) for those templates continue to fail in the same way
Click to expand...
 
Last edited:
What's somewhat interesting is..

Cloning that problematic template to create a new VM, the resulting VM can be backed up fine.
Converting that newly created VM into another template, and that new template suffers the same issue as the original (device not writable, permission denied)

I hadn't thought to try that previously, but hopefully that provides a semi-useful data point.
 
That was fast work, thank you for investigating and for proposing a solution so quickly. Much appreciated
 
So.. in the meantime, I took the hint that maybe my SSD is dying and rebuilt my PVE host from scratch - and restoring what I needed from a PBS backup

That all went exceedingly well (save for another issue I'll post separately about)
In fact, it went so well - those templated VM's that were giving me trouble are no longer complaning..

Has this patch already been released in the interim? Or have I circumvented the issue in some way?

The packages that have changed in the interim include:
Code: 
pve-manager: 9.0.4 --> 9.0.5
proxmox-backup-client: 4.0.12-1 --> 4.0.14-1
proxmox-backup-file-restore: 4.0.12-1 --> 4.0.14-1
qemu-server: 9.0.16 --> 9.0.17
 
  • Like
Reactions: Johannes S
TimRex said:
In fact, it went so well - those templated VM's that were giving me trouble are no longer complaning..
Has this patch already been released in the interim? Or have I circumvented the issue in some way?
Click to expand...
The patches to fix the issue was applied in git, but not packaged yet. Did you maybe change the underlying storage?
 
The issue only applies to certain storage types. A package with the fix, i.e. qemu-server=9.0.18, is on its way (currently going through internal testing).
 
You must log in or register to reply here.
Top Bottom