Lost Task!

V

vaschthestampede

Well-Known Member
Oct 21, 2020
139
8
58
39
With pdm I migrated 4 VMs and an LXC but I'm missing one in the Tasks list.

In order I migrated the 2002, 1000, 1001, 3004 and finaly 103 (the only LXC).
But in the migration tasks in the original node I only find the 2002, 1001, 3004 and 103, the 1000 is not present
1774513013039.png
In the qmtunnel log of the destination node I have all the migrations correctly.
1774513044820.png


PDM node:
Code: 
proxmox-datacenter-manager-meta: 1.0.0 (running kernel: 6.17.13-2-pve)
proxmox-datacenter-manager: 1.0.3 (running version: 1.0.3)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17: 6.17.13-2
proxmox-kernel-6.17.13-2-pve-signed: 6.17.13-2
proxmox-kernel-6.17.13-1-pve-signed: 6.17.13-1
proxmox-kernel-6.14: 6.14.11-6
proxmox-kernel-6.14.11-6-pve-signed: 6.14.11-6
proxmox-kernel-6.8: 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15
proxmox-kernel-6.8.12-5-pve-signed: 6.8.12-5
ifupdown2: 3.3.0-1+pmx12
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
pve-xtermjs: 5.5.0-3
zfsutils-linux: 2.4.1-pve1
 
hi,

was the vm 1000 located on another node perhaps? or is the 'pveEDGE' remote just a single node?

can you show what the 'pveEDGE' tasks (on the pve gui, not pdm) looks like?
 
Hello,

in addition to that, is there anything suspicious in the system logs on the PDM node, particularly in the time window when you migrated this particular guest journalctl -u proxmox-datacenter-manager-api -e ? Look out for anything that mentions 'task cache'.
 
dcsapak said:
was the vm 1000 located on another node perhaps?
Click to expand...
No, the 1000 was on the pveEDGE node.

dcsapak said:
the 'pveEDGE' remote just a single node?
Click to expand...
It is.

dcsapak said:
can you show what the 'pveEDGE' tasks (on the pve gui, not pdm) looks like?
Click to expand...
1774515313418.png

Lukas Wagner said:
journalctl -u proxmox-datacenter-manager-api -e
Click to expand...
Code: 
root@pdm:~# journalctl -u proxmox-datacenter-manager-api -e
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
-- No entries --

It seems strange to me...
 
Oh, sorry, I typed the command from memory without checking it first. It should rather be

Code: 
journalctl -u proxmox-datacenter-api.service -e
 
Try:
Code: 
journalctl -u proxmox-datacenter-api.service --since "2026-03-25 17:00:00"
 
Code: 
root@pdm:~# journalctl -u proxmox-datacenter-api.service --since "2026-03-25 17:00:00"
[...]
Mar 25 19:07:35 pdm proxmox-datacenter-api[26193]: remote migration requested
Mar 25 19:07:35 pdm proxmox-datacenter-api[26193]: forwarding remote migration requested
Mar 25 19:07:35 pdm proxmox-datacenter-api[26193]: migrating vm 1000 of node "pveEDGE"
Mar 25 19:18:21 pdm proxmox-datacenter-api[26193]: marking client 192.168.98.245:8006 as unreachable
Mar 25 19:18:21 pdm proxmox-datacenter-api[26193]: skipping host 192.168.98.245:8006 - marked unreachable
Mar 25 19:18:21 pdm proxmox-datacenter-api[26193]: client error on request /api2/extjs/cluster/resources, giving up - client error (SendRequest)
                                                  
                                                   Caused by:
                                                       connection closed before message completed
Mar 25 19:20:00 pdm proxmox-datacenter-api[26193]: marking client 192.168.98.245:8006 as unreachable
Mar 25 19:20:00 pdm proxmox-datacenter-api[26193]: skipping host 192.168.98.245:8006 - marked unreachable
Mar 25 19:20:00 pdm proxmox-datacenter-api[26193]: client error on request /api2/extjs/cluster/metrics/export?history=1&local%2Donly=0&start%2Dtime=1774462192, giving up - client error (SendRequest)
                                                  
                                                   Caused by:
                                                       connection closed before message completed
Mar 25 19:20:00 pdm proxmox-datacenter-api[26193]: coud not fetch metrics from 'pveEDGE': client error (SendRequest)
Mar 25 19:30:00 pdm proxmox-datacenter-api[26193]: rrd journal successfully committed (243 files in 0.077 seconds)
Mar 25 19:38:10 pdm proxmox-datacenter-api[26193]: applying task cache journal
Mar 25 19:38:10 pdm proxmox-datacenter-api[26193]: commited 893 tasks in 0.008.s to task cache archive
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: marking client 192.168.98.97:8006 as unreachable
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: marking client 192.168.98.245:8006 as unreachable
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: skipping host 192.168.98.245:8006 - marked unreachable
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: client error on request /api2/extjs/cluster/metrics/export?history=1&local%2Donly=0&start%2Dtime=1774463992, giving up - client error (Connect)
                                                  
                                                   Caused by:
                                                       error connecting to https://192.168.98.245:8006/ - tcp connect error: No route to host (os error 113)
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: coud not fetch metrics from 'pveEDGE': client error (Connect)
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: skipping host 192.168.98.97:8006 - marked unreachable
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: client error on request /api2/extjs/cluster/metrics/export?history=1&local%2Donly=0&start%2Dtime=1774463989, giving up - client error (Connect)
                                                  
                                                   Caused by:
                                                       error connecting to https://192.168.98.97:8006/ - tcp connect error: No route to host (os error 113)
Mar 25 19:50:05 pdm proxmox-datacenter-api[26193]: coud not fetch metrics from 'pveQNAP30bay': client error (Connect)
Mar 25 19:50:40 pdm proxmox-datacenter-api[26193]: client timed out on request /api2/extjs/nodes/pveEDGE/tasks/UPID%3ApveEDGE%3A0014C8B3%3A36A6EE08%3A69C42467%3Aqmigrate%3A1000%3Aroot%40pam%21pdm%2Dadmin%3A/status, no remotes reachable, giving up
Mar 25 19:50:40 pdm proxmox-datacenter-api[26193]: could not get status from remote: failed to perform API request: timed out
Mar 25 19:51:00 pdm proxmox-datacenter-api[26193]: marking client 192.168.154.50 as unreachable
Mar 25 19:51:00 pdm proxmox-datacenter-api[26193]: skipping host 192.168.154.50 - marked unreachable
Mar 25 19:51:00 pdm proxmox-datacenter-api[26193]: client timed out on request /api2/extjs/cluster/metrics/export?history=1&local%2Donly=0&start%2Dtime=1774463993, no remotes reachable, giving up
Mar 25 19:51:00 pdm proxmox-datacenter-api[26193]: coud not fetch metrics from 'pveISO': failed to perform API request: timed out
Mar 25 19:51:50 pdm proxmox-datacenter-api[26193]: skipping host 192.168.98.97:8006 - marked unreachable
Mar 25 19:51:50 pdm proxmox-datacenter-api[26193]: skipping host 192.168.98.245:8006 - marked unreachable
Mar 25 19:51:50 pdm proxmox-datacenter-api[26193]: skipping host 192.168.154.50 - marked unreachable
Mar 25 19:52:16 pdm proxmox-datacenter-api[26193]: coud not fetch metrics from 'pbsTEST': client error (Connect)
[...]

In fact there are errors but the node, I assure you, was up and reachable.
All nodes involved were up and reachable.
 
You must log in or register to reply here.
Top Bottom
Back