Problem in pve 8.1 and automatic backup tasks

[ymtnez]

Hello everyone,

I have a problem with my backup task scheduler (pvescheduler) that I have never had in previous versions of proxmox. It is something that I have not understood or resolved for days, and I assure you that I have reviewed what was said in the forums but perhaps I have not found the right one. I explain my testing environment:

I have 3 servers installed with updated Proxmox-VE version 8.1 and they are also clustered. In each of them I have a CT running with ubuntu-20.04 (photo1)

My problem is that when I create a backup task through the GUI Datacenter→Backup (photo2) it only executes the backup of the CTs of one node and the others of the other nodes do not, the curious thing is that the task is completed without errors. I am not new to proxmox as I have been using it for several years, I even have a cluster in production with version 7.2-7 with external NFS storage for the saves of all my CT/VM and everything works perfectly. But if I execute in “Datacenter→Backup→Run now” (photo3) then it does perform all the backups as I am used to seeing. I'll give you some information in case you need it.​

Excuse my English, I am Spanish speaking

 

[ymtnez]

root@pxmx8-1:~# pveversion -v

proxmox-ve: 8.1.0 (running kernel: 6.5.11-6-pve)
pve-manager: 8.1.3 (running version: 8.1.3/b46aac3b42da5d15)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5: 6.5.11-6
proxmox-kernel-6.5.11-6-pve-signed: 6.5.11-6
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx7
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.3
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-2
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.1.4
pve-qemu-kvm: 8.1.2-4
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve4

cat /etc/pve/jobs.cfg

vzdump: backup-a94d2061-4b13
    schedule 09:50
    compress zstd
    enabled 1
    mailnotification always
    mode snapshot
    notes-template {{guestname}}
    prune-backups keep-last=2
    storage local
    vmid 100,101,102

 

[spirit]

can you verify that service pvescheduler is running on others nodes ?

systemctl status pvescheduler.service ?



when you create a job, it's really create 1job, executed once on each server. (with the list of all vmid, on all servers).
That's why you see "skip externals vmid..." when a vm of the list is not on the node where the backup is running.

 

[ymtnez]

can you verify that service pvescheduler is running on others nodes ?

Hello spirit, this is the output on the three nodes

root@pxmx8-1:~# systemctl status pvescheduler.service
● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-12-06 08:08:07 CST; 2h 28min ago
   Main PID: 148466 (pvescheduler)
      Tasks: 1 (limit: 11844)
     Memory: 560.1M
        CPU: 55.540s
     CGroup: /system.slice/pvescheduler.service
             └─148466 pvescheduler

Dec 06 08:55:00 pxmx8-1 pvescheduler[156051]: <root@pam> starting task UPID:pxmx8-1:00026194:00597F83:65707D34:vzdump:100:root@pam:
Dec 06 08:55:00 pxmx8-1 pvescheduler[156052]: INFO: starting new backup job: vzdump 100 101 102 --mailnotification always --quiet 1 --compress zstd --storage Salvas --notes-template '{{guestname}}' --prune-backups 'keep-last=2' --mode sn>
Dec 06 08:55:01 pxmx8-1 pvescheduler[156052]: INFO: Starting Backup of VM 100 (lxc)
Dec 06 08:55:29 pxmx8-1 pvescheduler[156052]: INFO: Finished Backup of VM 100 (00:00:29)
Dec 06 08:55:29 pxmx8-1 pvescheduler[156052]: INFO: Backup job finished successfully
Dec 06 09:50:01 pxmx8-1 pvescheduler[165871]: <root@pam> starting task UPID:pxmx8-1:000287F0:005E88BE:65708A19:vzdump:100:root@pam:
Dec 06 09:50:01 pxmx8-1 pvescheduler[165872]: INFO: starting new backup job: vzdump 100 101 102 --mailnotification always --compress zstd --quiet 1 --notes-template '{{guestname}}' --storage local --prune-backups 'keep-last=2' --mode sna>
Dec 06 09:50:01 pxmx8-1 pvescheduler[165872]: INFO: Starting Backup of VM 100 (lxc)
Dec 06 09:50:31 pxmx8-1 pvescheduler[165872]: INFO: Finished Backup of VM 100 (00:00:30)
Dec 06 09:50:31 pxmx8-1 pvescheduler[165872]: INFO: Backup job finished successfully

root@pxmx8-2:~# systemctl status pvescheduler.service
● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-12-06 08:08:23 CST; 2h 28min ago
   Main PID: 148089 (pvescheduler)
      Tasks: 1 (limit: 11844)
     Memory: 341.0M
        CPU: 41.129s
     CGroup: /system.slice/pvescheduler.service
             └─148089 pvescheduler

Dec 06 08:12:00 pxmx8-2 pvescheduler[148666]: <root@pam> starting task UPID:pxmx8-2:000244BB:00558E70:65707320:vzdump:101:root@pam:
Dec 06 08:12:00 pxmx8-2 pvescheduler[148667]: INFO: starting new backup job: vzdump 100 101 102 --prune-backups 'keep-last=2' --quiet 1 --compress zstd --mailnotification always --notes-template '{{guestname}}' --storage local --mode sna>
Dec 06 08:12:00 pxmx8-2 pvescheduler[148667]: INFO: Starting Backup of VM 101 (lxc)
Dec 06 08:12:27 pxmx8-2 pvescheduler[148667]: INFO: Finished Backup of VM 101 (00:00:27)
Dec 06 08:12:27 pxmx8-2 pvescheduler[148667]: INFO: Backup job finished successfully
Dec 06 08:42:01 pxmx8-2 pvescheduler[153474]: <root@pam> starting task UPID:pxmx8-2:00025783:00584DB3:65707A29:vzdump:101:root@pam:
Dec 06 08:42:01 pxmx8-2 pvescheduler[153475]: INFO: starting new backup job: vzdump 100 101 102 --mailnotification always --notes-template '{{guestname}}' --storage local --prune-backups 'keep-last=2' --compress zstd --quiet 1 --mode sna>
Dec 06 08:42:01 pxmx8-2 pvescheduler[153475]: INFO: Starting Backup of VM 101 (lxc)
Dec 06 08:42:28 pxmx8-2 pvescheduler[153475]: INFO: Finished Backup of VM 101 (00:00:27)
Dec 06 08:42:28 pxmx8-2 pvescheduler[153475]: INFO: Backup job finished successfully

root@pxmx8-3:~# systemctl status pvescheduler.service
● pvescheduler.service - Proxmox VE scheduler
     Loaded: loaded (/lib/systemd/system/pvescheduler.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-12-06 08:08:11 CST; 2h 28min ago
   Main PID: 148197 (pvescheduler)
      Tasks: 1 (limit: 11844)
     Memory: 337.4M
        CPU: 25.972s
     CGroup: /system.slice/pvescheduler.service
             └─148197 pvescheduler

Dec 06 08:08:09 pxmx8-3 systemd[1]: Starting pvescheduler.service - Proxmox VE scheduler...
Dec 06 08:08:11 pxmx8-3 pvescheduler[148197]: starting server
Dec 06 08:08:11 pxmx8-3 systemd[1]: Started pvescheduler.service - Proxmox VE scheduler.
Dec 06 10:18:03 pxmx8-3 pvescheduler[170344]: <root@pam> starting task UPID:pxmx8-3:00029969:006116A2:657090AB:vzdump:102:root@pam:
Dec 06 10:18:03 pxmx8-3 pvescheduler[170345]: INFO: starting new backup job: vzdump 100 101 102 --mailnotification always --notes-template '{{guestname}}' --compress zstd --storage local --quiet 1 --mode snapshot --prune-backups 'keep-la>
Dec 06 10:18:03 pxmx8-3 pvescheduler[170345]: INFO: Starting Backup of VM 102 (lxc)
Dec 06 10:18:29 pxmx8-3 pvescheduler[170345]: INFO: Finished Backup of VM 102 (00:00:26)
Dec 06 10:18:29 pxmx8-3 pvescheduler[170345]: INFO: Backup job finished successfully

when you create a job, it's really create 1job, executed once on each server. (with the list of all vmid, on all servers).
That's why you see "skip externals vmid..." when a vm of the list is not on the node where the backup is running.

You are right my friend and I agree with you. I know that in a backup task each node will make backups of its vmids, the vmids of the other nodes will not and that is why it is "skip externals vmid...". But the point is that the task is not executed on the other nodes and for this reason the vm/ct of those nodes are not backed up. Look at photo 4 and you will see that in the task history it only backs up CT 100 of the pxmx8-1 node and finishes the task, it does not continue for the other nodes.

 

[ymtnez]

I show you two examples:
1-) A cluster with three servers in production with pve-7.2-7



2-) The case we are analyzing with pve-8.1.3

 

[ymtnez]

Hi, someone who can help me please!!

 

[ymtnez]

I have reinstalled the 3 servers with pve-8.1, updated them (pve-no-subscription) and only put them in a cluster. Still I have the same problem, the backup job only runs on a single node and does not continue for the other nodes