PDM missing backup jobs from PVE / Log retention

[devaux]

Hi everyone,

I am trying to simplify my backup monitoring and would like to use the Proxmox Datacenter Manager (PDM) for a centralized overview.

However, I noticed that PDM is not showing certain jobs, even though they are clearly visible and active in PVE. I have attached two images comparing the logs to show the discrepancy. Does anyone know why some jobs might be missing in PDM?

On a side note, is it possible to extend the log retention in PDM so that older logs aren't deleted so quickly?

Thanks for your help!

 

[Lukas Wagner]

However, I noticed that PDM is not showing certain jobs, even though they are clearly visible and active in PVE. I have attached two images comparing the logs to show the discrepancy. Does anyone know why some jobs might be missing in PDM?

Just to clarify since you mentioned 'active' jobs, are you talking about active as in running jobs? The task archive for remote tasks in PDM only shows tasks that have completed or running tasks that have been started by PDM itself, at least for now.

But judging from your screenshots it seems like the backup task is long completed and thus should indeed be present in the task archive.
Is this something that happens regularly or with some other form of regularity (e.g. always tasks from the same remotes or nodes, always the same backup tasks, anything)

Thanks!

 

[devaux]

Just to clarify since you mentioned 'active' jobs, are you talking about active as in running jobs? The task archive for remote tasks in PDM only shows tasks that have completed or running tasks that have been started by PDM itself, at least for now.

But judging from your screenshots it seems like the backup task is long completed and thus should indeed be present in the task archive.
Is this something that happens regularly or with some other form of regularity (e.g. always tasks from the same remotes or nodes, always the same backup tasks, anything)

Thanks!

These are jobs that have already been completed.
A backup (with 2 VMs) is configured on the PVE node. However, the backup is not logged in the PDM log under this PVE, even though I can see it locally on the PVE.
When I integrate the PBS into the PDM, the same backup is displayed correctly there.

I also have a second backup (only one VM) on this PVE. Strangely enough, I can see this in the overview.
I think I've noticed this behavior before, but I haven't paid any attention to it until now.

 

[carles89]

Same here, we've a PDM instance connected to several remotes, and in one of them we cannot see all completed backup tasks listed on PDM. It seems random, some tasks show, some tasks don't. All of them appear on PVE side (as completed).

PVE Version:
pve-manager/8.4.19/a68fb383814bb1e6

PDM Version:
proxmox-datacenter-manager-meta: 1.0.0 (running kernel: 6.17.2-2-pve)
proxmox-datacenter-manager: 1.0.3 (running version: 1.0.3)

We tried the following, with no luck:
- Removing and readding the remote
- Using an API Token with root permissions (we usually use a limited token with PVEAuditor permissions).

Any ideas?

Thanks!

 

[Lukas Wagner]

I've created an entry in our Bugzilla to track this issue, see here.

Could you tell me the 'Start Time' of the missing task, as well as send me the output of

ls -lah /var/cache/proxmox-datacenter-manager/remote-tasks

What is are the contents of the state file?

cat /var/cache/proxmox-datacenter-manager/remote-tasks/state

(please redact any sensitive information, such as the names of remotes)

Could you check if a

systemctl stop proxmox-datacenter-api.service
rm /var/cache/proxmox-datacenter-manager/remote-tasks/state
systemctl start proxmox-datacenter-api.service

fixes the issue? By removing the state file, the 'cutoff' timestamps are reset, ensuring that PDM will try to fetch all tasks from the last 7 days and merge them into the PDM-local task cache.

Thanks in advance!

 

[carles89]

Hi Lukas,

Before trying anything, I've upgraded to PDM 1.1, but the issue persisted.

systemctl stop proxmox-datacenter-api.service
rm /var/cache/proxmox-datacenter-manager/remote-tasks/state
systemctl start proxmox-datacenter-api.service

That did the trick! Tasks are showing now.

Here's the information you requested, gathered before removing the state file. "Remote3" was the one with issues.

root@pdm:~# ls -lah /var/cache/proxmox-datacenter-manager/remote-tasks
total 484K
drwxr-x--- 2 www-data www-data 4.0K May 28 16:14 .
drwxr-xr-x 3 www-data www-data 4.0K May 28 16:19 ..
-rw-r----- 1 www-data www-data    0 May 28 16:14 active
-rw-r----- 1 www-data www-data 2.4K May 28 15:23 archive.1779377080.zst
-rw-r----- 1 www-data www-data 2.6K May 28 15:23 archive.1779463640.zst
-rw-r----- 1 www-data www-data 2.6K May 28 15:23 archive.1779550190.zst
-rw-r----- 1 www-data www-data 1.5K May 28 15:23 archive.1779636770.zst
-rw-r----- 1 www-data www-data 3.0K May 28 15:23 archive.1779723330.zst
-rw-r----- 1 www-data www-data  23K May 28 15:23 archive.1779809890
-rw-r----- 1 www-data www-data  19K May 28 15:23 archive.1779896430
-rw-r--r-- 1 www-data www-data 408K May 28 16:14 journal
-rw-r----- 1 www-data www-data    0 May  5 15:59 .lock
-rw-r----- 1 www-data www-data 2.1K May 28 16:14 state


root@pdm:~# cat /var/cache/proxmox-datacenter-manager/remote-tasks/state
{
  "remote-state": {
    "remote1": {
      "node-state": {
        "pve1": {
          "cutoff": 1779938880
        }
      }
    },
    "remote2": {
      "node-state": {
        "srv1": {
          "cutoff": 1779934099
        },
        "srv3": {
          "cutoff": 1779940600
        },
        "srv2": {
          "cutoff": 1779940805
        }
      }
    },
    "remote3": {
      "node-state": {
        "px1": {
          "cutoff": 1779963666
        }
      }
    },
    "remote4": {
      "node-state": {
        "pve1": {
          "cutoff": 1779930932
        },
        "pve2": {
          "cutoff": 1779929566
        }
      }
    },
    "remote5": {
      "node-state": {
        "pve1": {
          "cutoff": 1779969987
        },
        "pve2": {
          "cutoff": 1779977158
        }
      }
    },
    "remote6": {
      "node-state": {
        "pve1": {
          "cutoff": 1779942604
        },
        "pve2": {
          "cutoff": 1779951398
        },
        "pve3": {
          "cutoff": 1779937834
        },
        "pve4": {
          "cutoff": 1779939000
        },
        "pve5": {
          "cutoff": 1779938818
        },
        "pve6": {
          "cutoff": 1779942602
        },
        "pve7": {
          "cutoff": 1779951600
        },
        "pve8": {
          "cutoff": 1779942602
        },
        "pve9": {
          "cutoff": 1779951601
        },
        "pve10": {
          "cutoff": 1779942600
        },
        "pve11": {
          "cutoff": 1779953403
        },
        "pve12": {
          "cutoff": 1779951600
        }
      }
    },
    "remote7": {
      "node-state": {
        "pve1": {
          "cutoff": 1779943803
        },
        "pve2": {
          "cutoff": 1779937372
        },
        "pve0": {
          "cutoff": 1779930569
        }
      }
    }
  },
  "tracked-tasks": []

Thank you!

 

[Lukas Wagner]

I could reproduce the issue and submitted a fix to the development mailing list.

https://lore.proxmox.com/pdm-devel/20260529133951.326103-1-l.wagner@proxmox.com/T/#t