pvestatd got timeout

[aGro]

Hallo,

ich habe seit ein paar Tagen das Problem, dass meine VM-Backup's übermäßig lange dauern.
Nachdem ich mir nun mal das Syslog angeschaut habe, sehe ich immer wieder folgende Fehlermeldungen:

May 07 08:38:24 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:39:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:39:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:39:03 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:39:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:39:54 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:39:54 mc-kvg-03 pvestatd[2455]: unable to activate storage 'backup-vm' - directory '/mnt/pve/backup-vm' does not exist or is unreachable
May 07 08:40:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:40:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:40:13 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:40:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:41:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:41:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:41:15 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:42:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:42:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:42:14 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:42:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:43:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:43:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:43:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:43:35 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:44:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:44:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:44:03 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:44:24 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:44:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 08:45:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:45:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 08:46:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 08:46:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.

Es existiert angeblich das Storage-Verzeichnis nicht bzw. ist nicht erreichbar und immer wieder der Timeoutfehler von pvestatd. Das Backup wird aber trotzdem ausgeführt, nur tut es das sehr zeitverzögert.
Ich hatte schon ein bisschen gegoogelt und einen Restart von

systemctl reload-or-restart pvedaemon.service pveproxy.service pvestatd.service

durchgeführt. Hat aber leider auch keine Verbesserung gebracht.
Was kann ich noch tun?

 

[fiona]

Hi,
ist das eine Netzwerk-Storage? Wie schaut die Last im Netzwerk und auf dem Storage-Server aus während das Backup läuft? Möglicherweise hilft es ein Ratelimit zu setzen, kann (bei Proxmox VE 8.2) beim Editieren vom Backup-Job unter Advanced gemacht werden oder für die ganze Node mit Editieren von /etc/vzdump.conf. Bei älteren Versionen muss es noch über CLI gemacht werden: https://forum.proxmox.com/threads/proxmox-ve-8-0-released.129320/post-626892

Wie schaut die Last auf dem Proxmox VE Server aus während das Backup läuft? Falls es VM-Backups sind und IO wait hoch ist, kannst Du versuchen die max_workers-Einstellung zu reduzieren.

 

[aGro]

Hallo Fiona,

ja es ist ein Netzwerk-Storage. Hier mal die Netzwerk-Last des heutigen Tages auf einem der Proxmox-Server...


Die VM-Backups starten um 0 Uhr. Ich denke mal, der Wert ist nicht der Rede wert.

Wo genau sehe ich den IO wait - Wert? Ist das IO Delay? Dieser befindet sich während des Backups zwischen 1-2 %.
Ansonsten liegen CPU- und Memory-Usage im unteren Drittel.
Diese verzögerten Backups treten ja auch erst seit ca. 3-4 Tagen auf. Es hat sich aber an der Infrastruktur nichts geändert.

 

[fiona]

Wo genau sehe ich den IO wait - Wert? Ist das IO Delay? Dieser befindet sich während des Backups zwischen 1-2 %.

Ja, das ist der Wert. Wie schaut die CPU/IO Last beim Storage-Server aus?

Ansonsten liegen CPU- und Memory-Usage im unteren Drittel.
Diese verzögerten Backups treten ja auch erst seit ca. 3-4 Tagen auf. Es hat sich aber an der Infrastruktur nichts geändert.

Wurden bestimmte Updates installiert z.B. Kernel oder QEMU? Kann in /var/log/apt/history.log nachgeschaut werden. Wurde an der Storage-Konfiguration etwas geändert? Nutzt etwas anderes das selbe Netzwerk zu der Zeit, was viel Traffic produziert?

Bitte auch den Task-Log von einem solchen Backup posten und die System-Logs während der Backups.

 

[aGro]

An der Storage-Konfiguration wurde nichts geändert. Die Backups landen auf einem separaten PBS, der mit sonst keinem Gerät kommuniziert.
Ich würde gerne die Last auf dem Proxmox-Backup-Server überprüfen, aber ich komme nicht mehr auf die Web Gui des PBS (warum auch immer). Per SSH komme ich drauf. Gibts auch einen CLI-Command, mit dem man die Last überprüfen kann?

Ich habe eben nochmal wegen der Updates geschaut....diese Verzögerungen fingen am 02.05. an. Geupdatet habe ich erst am 04.05., da die Subscription zu diesem Zeitpunkt ausgelaufen waren und erst am 04.05. wieder aktiv. Hier mal ein Auszug aus dem Tasklog....



Hier sieht man, dass bis zum 01.05. die Backups frühs 5 Uhr fertig waren. Seit dem 02.05. sind kvg03 und kvg06 stark verzögert. Ab dem 03.05. weißt der kvg04 die selben Anzeichen auf..



Hier der Task-Log von dem laufenden Backup...

INFO: starting new backup job: vzdump 101 --compress lzo --storage backup-vm --node mc-kvg-03 --remove 0 --mode snapshot
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2024-05-07 11:03:04
INFO: status = running
INFO: update VM 101: -lock backup
INFO: VM Name: unifi-video
INFO: include disk 'virtio0' 'ceph-pool1:vm-101-disk-0' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/backup-vm/dump/vzdump-qemu-101-2024_05_07-11_03_04.vma.lzo'
INFO: started backup task 'bc8bc23d-9426-46be-8ada-23f9828d741f'
INFO: status: 0% (50331648/53687091200), sparse 0% (8998912), duration 3, read/write 16/13 MB/s
INFO: status: 1% (552206336/53687091200), sparse 0% (36794368), duration 29, read/write 19/18 MB/s
INFO: status: 2% (1077018624/53687091200), sparse 0% (59281408), duration 75, read/write 11/10 MB/s
INFO: status: 3% (1622999040/53687091200), sparse 0% (77844480), duration 136, read/write 8/8 MB/s
INFO: status: 4% (2179268608/53687091200), sparse 0% (87195648), duration 182, read/write 12/11 MB/s
INFO: status: 5% (2717777920/53687091200), sparse 0% (231682048), duration 202, read/write 26/19 MB/s
INFO: status: 6% (3230859264/53687091200), sparse 0% (245850112), duration 280, read/write 6/6 MB/s
INFO: status: 7% (3769237504/53687091200), sparse 0% (254492672), duration 345, read/write 8/8 MB/s
INFO: status: 8% (4294967296/53687091200), sparse 0% (257363968), duration 395, read/write 10/10 MB/s
INFO: status: 9% (4857790464/53687091200), sparse 0% (288591872), duration 438, read/write 13/12 MB/s
INFO: status: 10% (5377163264/53687091200), sparse 0% (292024320), duration 495, read/write 9/9 MB/s
INFO: status: 11% (5906694144/53687091200), sparse 0% (292114432), duration 547, read/write 10/10 MB/s
INFO: status: 12% (6456082432/53687091200), sparse 0% (292278272), duration 591, read/write 12/12 MB/s
INFO: status: 13% (6980370432/53687091200), sparse 0% (310292480), duration 632, read/write 12/12 MB/s
INFO: status: 14% (7524253696/53687091200), sparse 0% (310591488), duration 697, read/write 8/8 MB/s
INFO: status: 15% (8058699776/53687091200), sparse 0% (310910976), duration 741, read/write 12/12 MB/s
INFO: status: 16% (8590852096/53687091200), sparse 0% (311062528), duration 791, read/write 10/10 MB/s
INFO: status: 17% (9146531840/53687091200), sparse 0% (322441216), duration 829, read/write 14/14 MB/s
INFO: status: 18% (9669967872/53687091200), sparse 0% (326139904), duration 877, read/write 10/10 MB/s
INFO: status: 19% (10201071616/53687091200), sparse 0% (327069696), duration 921, read/write 12/12 MB/s
INFO: status: 20% (10742464512/53687091200), sparse 0% (327639040), duration 971, read/write 10/10 MB/s
INFO: status: 21% (11283070976/53687091200), sparse 0% (340615168), duration 1014, read/write 12/12 MB/s
INFO: status: 22% (11823939584/53687091200), sparse 0% (342765568), duration 1069, read/write 9/9 MB/s
INFO: status: 23% (12349341696/53687091200), sparse 0% (346361856), duration 1114, read/write 11/11 MB/s
INFO: status: 24% (12888506368/53687091200), sparse 0% (346656768), duration 1165, read/write 10/10 MB/s
INFO: status: 25% (13423214592/53687091200), sparse 0% (369872896), duration 1203, read/write 14/13 MB/s
INFO: status: 26% (13961003008/53687091200), sparse 0% (370061312), duration 1265, read/write 8/8 MB/s
INFO: status: 27% (14497349632/53687091200), sparse 0% (370237440), duration 1310, read/write 11/11 MB/s
INFO: status: 28% (15041232896/53687091200), sparse 0% (370413568), duration 1356, read/write 11/11 MB/s
INFO: status: 29% (15583739904/53687091200), sparse 0% (394858496), duration 1395, read/write 13/13 MB/s
INFO: status: 30% (16108027904/53687091200), sparse 0% (395108352), duration 1442, read/write 11/11 MB/s
INFO: status: 31% (16643719168/53687091200), sparse 0% (395124736), duration 1493, read/write 10/10 MB/s
INFO: status: 32% (17182097408/53687091200), sparse 0% (399384576), duration 1548, read/write 9/9 MB/s
INFO: status: 33% (17720672256/53687091200), sparse 0% (445128704), duration 1612, read/write 8/7 MB/s
INFO: status: 34% (18257870848/53687091200), sparse 0% (445382656), duration 1660, read/write 11/11 MB/s
INFO: status: 35% (18792316928/53687091200), sparse 0% (448937984), duration 1716, read/write 9/9 MB/s
INFO: status: 36% (19399966720/53687091200), sparse 1% (553570304), duration 1766, read/write 12/10 MB/s
INFO: status: 37% (19872284672/53687091200), sparse 1% (610160640), duration 1802, read/write 13/11 MB/s
INFO: status: 38% (20405878784/53687091200), sparse 1% (613142528), duration 1865, read/write 8/8 MB/s
INFO: status: 39% (20947795968/53687091200), sparse 1% (616017920), duration 1927, read/write 8/8 MB/s
INFO: status: 40% (21475950592/53687091200), sparse 1% (616878080), duration 1991, read/write 8/8 MB/s
INFO: status: 41% (22017015808/53687091200), sparse 1% (676397056), duration 2040, read/write 11/9 MB/s
INFO: status: 42% (22552576000/53687091200), sparse 1% (676462592), duration 2096, read/write 9/9 MB/s
INFO: status: 43% (23091609600/53687091200), sparse 1% (680103936), duration 2151, read/write 9/9 MB/s
INFO: status: 44% (23623958528/53687091200), sparse 1% (680316928), duration 2207, read/write 9/9 MB/s
INFO: status: 45% (24167317504/53687091200), sparse 1% (770830336), duration 2260, read/write 10/8 MB/s
INFO: status: 46% (24706285568/53687091200), sparse 1% (773013504), duration 2322, read/write 8/8 MB/s
INFO: status: 47% (25237979136/53687091200), sparse 1% (781135872), duration 2376, read/write 9/9 MB/s
INFO: status: 48% (25811681280/53687091200), sparse 1% (903798784), duration 2441, read/write 8/6 MB/s
INFO: status: 49% (26312769536/53687091200), sparse 1% (1069719552), duration 2490, read/write 10/6 MB/s
INFO: status: 50% (26850885632/53687091200), sparse 1% (1070297088), duration 2536, read/write 11/11 MB/s
INFO: status: 51% (27428651008/53687091200), sparse 2% (1450852352), duration 2581, read/write 12/4 MB/s
INFO: status: 52% (27918008320/53687091200), sparse 3% (1851928576), duration 2600, read/write 25/4 MB/s
INFO: status: 53% (28458680320/53687091200), sparse 3% (1976721408), duration 2636, read/write 15/11 MB/s
INFO: status: 54% (28998500352/53687091200), sparse 3% (1976729600), duration 2722, read/write 6/6 MB/s
INFO: status: 55% (29532749824/53687091200), sparse 3% (1980452864), duration 2792, read/write 7/7 MB/s
INFO: status: 56% (30070407168/53687091200), sparse 3% (1980661760), duration 2853, read/write 8/8 MB/s
INFO: status: 57% (30602952704/53687091200), sparse 3% (2048753664), duration 2907, read/write 9/8 MB/s
INFO: status: 58% (31144476672/53687091200), sparse 3% (2050060288), duration 2970, read/write 8/8 MB/s
INFO: status: 59% (31680823296/53687091200), sparse 3% (2054156288), duration 3032, read/write 8/8 MB/s
INFO: status: 60% (32218021888/53687091200), sparse 3% (2055188480), duration 3092, read/write 8/8 MB/s
INFO: status: 61% (32758038528/53687091200), sparse 4% (2153148416), duration 3135, read/write 12/10 MB/s
INFO: status: 62% (33292419072/53687091200), sparse 4% (2153209856), duration 3205, read/write 7/7 MB/s
INFO: status: 63% (33843576832/53687091200), sparse 4% (2153213952), duration 3262, read/write 9/9 MB/s
INFO: status: 64% (34368716800/53687091200), sparse 4% (2153332736), duration 3323, read/write 8/8 MB/s
INFO: status: 65% (34915155968/53687091200), sparse 4% (2179919872), duration 3390, read/write 8/7 MB/s
INFO: status: 66% (35440623616/53687091200), sparse 4% (2180370432), duration 3457, read/write 7/7 MB/s
INFO: status: 67% (35978608640/53687091200), sparse 4% (2184089600), duration 3502, read/write 11/11 MB/s
INFO: status: 68% (36509450240/53687091200), sparse 4% (2184364032), duration 3550, read/write 11/11 MB/s
INFO: status: 69% (37050908672/53687091200), sparse 4% (2192850944), duration 3603, read/write 10/10 MB/s
INFO: status: 70% (37585158144/53687091200), sparse 4% (2192986112), duration 3656, read/write 10/10 MB/s
INFO: status: 71% (38126354432/53687091200), sparse 4% (2196738048), duration 3707, read/write 10/10 MB/s
INFO: status: 72% (38666174464/53687091200), sparse 4% (2197073920), duration 3758, read/write 10/10 MB/s
INFO: status: 73% (39201341440/53687091200), sparse 4% (2203623424), duration 3807, read/write 10/10 MB/s
INFO: status: 74% (39734149120/53687091200), sparse 4% (2203656192), duration 3862, read/write 9/9 MB/s
INFO: status: 75% (40280260608/53687091200), sparse 4% (2203906048), duration 3914, read/write 10/10 MB/s
INFO: status: 76% (40823685120/53687091200), sparse 4% (2203918336), duration 3962, read/write 11/11 MB/s
INFO: status: 77% (41344434176/53687091200), sparse 4% (2251796480), duration 4007, read/write 11/10 MB/s
INFO: status: 78% (41878224896/53687091200), sparse 4% (2251837440), duration 4048, read/write 13/13 MB/s
INFO: status: 79% (42418962432/53687091200), sparse 4% (2253877248), duration 4095, read/write 11/11 MB/s
INFO: status: 80% (42951311360/53687091200), sparse 4% (2254790656), duration 4151, read/write 9/9 MB/s

 

[aGro]

Und das dazugehörige Syslog...

May 07 11:34:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:35:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:35:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:35:14 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:36:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:36:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:37:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:37:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:37:09 mc-kvg-03 pmxcfs[2005]: [status] notice: received log
May 07 11:37:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:38:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:38:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:39:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:39:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:39:24 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:39:24 mc-kvg-03 pvestatd[2455]: unable to activate storage 'backup-vm' - directory '/mnt/pve/backup-vm' does not exist or is unreachable
May 07 11:40:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:40:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:40:33 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:41:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:41:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:41:34 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:42:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:42:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:42:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:43:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:43:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:44:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:44:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:44:33 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:44:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:45:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:45:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:45:03 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:45:03 mc-kvg-03 pvestatd[2455]: unable to activate storage 'backup-vm' - directory '/mnt/pve/backup-vm' does not exist or is unreachable
May 07 11:46:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:46:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:47:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:47:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:48:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:48:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:48:14 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:49:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:49:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:49:13 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:50:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:50:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:51:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:51:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:51:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:52:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:52:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:52:14 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:52:19 mc-kvg-03 pmxcfs[2005]: [status] notice: received log
May 07 11:53:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:53:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:53:43 mc-kvg-03 pmxcfs[2005]: [status] notice: received log
May 07 11:53:45 mc-kvg-03 pmxcfs[2005]: [status] notice: received log
May 07 11:53:45 mc-kvg-03 sshd[95566]: Accepted publickey for root from 10.2.1.52 port 37830 ssh2: RSA SHA256:KITpJOgZPbmjFVfk2OcBrUtY5+H4RbNsLm4hvVESglc
May 07 11:53:45 mc-kvg-03 sshd[95566]: pam_unix(sshd:session): session opened for user root by (uid=0)
May 07 11:53:45 mc-kvg-03 login[95574]: pam_unix(login:session): session opened for user root by root(uid=0)
May 07 11:53:45 mc-kvg-03 login[95579]: ROOT LOGIN  on '/dev/pts/0' from '10.2.1.52'
May 07 11:53:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:54:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:54:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:54:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:55:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:55:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:55:33 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:55:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:55:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:56:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:56:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:56:03 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:56:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:57:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:57:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:57:33 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:58:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:58:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:58:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:58:53 mc-kvg-03 pvestatd[2455]: unable to activate storage 'backup-vm' - directory '/mnt/pve/backup-vm' does not exist or is unreachable
May 07 11:59:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 11:59:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 11:59:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:59:34 mc-kvg-03 pvestatd[2455]: got timeout
May 07 11:59:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:00:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:00:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:00:04 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:00:04 mc-kvg-03 pvestatd[2455]: unable to activate storage 'backup-vm' - directory '/mnt/pve/backup-vm' does not exist or is unreachable
May 07 12:00:23 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:00:33 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:01:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:01:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:01:13 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:01:18 mc-kvg-03 smartd[1422]: Device: /dev/sda, failed to read Temperature
May 07 12:01:18 mc-kvg-03 smartd[1422]: Device: /dev/sdb, failed to read Temperature
May 07 12:01:18 mc-kvg-03 smartd[1422]: Device: /dev/sdc, failed to read Temperature
May 07 12:01:18 mc-kvg-03 smartd[1422]: Device: /dev/sdd, failed to read Temperature
May 07 12:01:18 mc-kvg-03 smartd[1422]: Device: /dev/sde, failed to read Temperature
May 07 12:01:19 mc-kvg-03 pmxcfs[2005]: [dcdb] notice: data verification successful
May 07 12:01:19 mc-kvg-03 rrdcached[1977]: flushing old values
May 07 12:01:19 mc-kvg-03 rrdcached[1977]: rotating journals
May 07 12:01:19 mc-kvg-03 rrdcached[1977]: started new journal /var/lib/rrdcached/journal/rrd.journal.1715076079.957794
May 07 12:01:19 mc-kvg-03 rrdcached[1977]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1715068879.957795
May 07 12:01:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:01:53 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:02:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:02:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:02:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:03:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:03:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:03:38 mc-kvg-03 pvestatd[2455]: status update time (6.288 seconds)
May 07 12:04:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:04:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:05:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:05:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:06:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:06:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:06:44 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:06:44 mc-kvg-03 pvestatd[2455]: unable to activate storage 'backup-vm' - directory '/mnt/pve/backup-vm' does not exist or is unreachable
May 07 12:07:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:07:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:07:19 mc-kvg-03 pmxcfs[2005]: [status] notice: received log
May 07 12:07:43 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:08:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:08:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:09:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:09:02 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.
May 07 12:09:33 mc-kvg-03 pvestatd[2455]: got timeout
May 07 12:10:00 mc-kvg-03 systemd[1]: Starting Proxmox VE replication runner...
May 07 12:10:01 mc-kvg-03 systemd[1]: Started Proxmox VE replication runner.

 

[aGro]

Ich habe nun zwischenzeitlich mal probiert, die VM-Backup's auf ein anderes Netzwerkstorage zu speichern. Dies funktioniert einwandfrei ohne Timeouts.
Es liegt also nicht an der Quelle, sondern am Ziel. Kommen wir also zum PBS.
Welchen Grund kann es geben, dass ich dort nicht mehr auf die Web Gui komme? Und was könnte es für Gründe geben, dass der PBS solche Fehler verursacht? Einen Neustart des Servers habe ich schon durchgeführt.

 

[fiona]

Am besten mal beim PBS schauen ob bei den System Logs etwas auffällig ist: journalctl -b oder auch

journalctl -b -u proxmox-backup.service -u proxmox-backup-proxy.service
systemctl status proxmox-backup.service proxmox-backup-proxy.service

Wurden am PBS Updates installiert oder an der Konfiguration geändert?

 

[aGro]

Ich habe mir mal das Log auf dem PBS von gestern während des Backups angeschaut und finde dort diese Fehler...

 

[fiona]

Bitte mal mit älteren Logs vergleichen bevor es die Probleme gab: journalctl -b-5 zeigt z.B. das Log von vor 5 Boots. Alternativ kann auch --since, --until benutzt werden. Ist der Backup-Storage ein Samba-Share? Falls ja, könnte das Problem auch beim Storage-Server bzw. Netzwerk zum Storage-Server liegen. Am besten für gute Performance wäre natürlich ein Storage, der lokal beim PBS ist.

 

[aGro]

Der Storage liegt lokal beim PBS.
Nachdem der PBS gestern Nachmittag stromlos gemacht wurde, sehe ich auch keine Logs mehr vor dieser Zeit. Die selben Fehler (wie im Bild oben) traten aber gestern Abend auch nochmal auf, ohne das ein Backup lief.