Network migration fail with "failed to parse api response" in network selection

bbgeek17

bbgeek17

Distinguished Member
Nov 20, 2020
5,887
2,069
258
Blockbridge
www.blockbridge.com
Splitting out this post into its own thread.

Summary: PDM was connected to multiple PVE clusters that are run behind NAT (i.e. hosts have 10.x.y.z/24 IPs and are externally accessible via 172.22.0.0/16)
Connection and discovery are successful, however migration option fails with the following error in "Target Network" field:
Error: api error (status = 400 Bad Request): api returned unexpected data - failed to parse api response

As an experiment I've moved the PDM to the same network as the hosts (10.x.y.z) and did the discovery via the 10.x.y.z network. In this case the connection between PDM and PVE is direct and NAT is not involved. However, that did not change the outcome - still getting the error.

All clusters are running the latest software.

Happy to supply logs as needed.
Thanks!

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
hi, thanks for the report

can you check the browser console and post which api call returns that data ?
(improving the error handling , e.g naming (e.g. including the source api call) is on our roadmap ;) )

there is probably some mismatch between what the PVE api returns and what the rust code expects (in javascript/perl land types are not that strict, so those mismatches don't appear there that often)
 
Hi Dominik,

I only have one error in the console and it seems to be unrelated:
src/widget/pve_migrate_mapping.rs:148 could not parse 'file' property of 'ide2

When initial Migate widget comes up the "Target Remote" points to itself. The "Target Network" only becomes available when I switch the remote.
At this time 4 API calls are logged in Dev Tools (2 of each):
https://172.22.2.5:8443/api2/extjs/pve/remotes/pve8r3q9-nvme/nodes/localhost/network?interface-type=any_bridge
https://172.22.2.5:8443/api2/extjs/...orage?enabled=true&format=true&content=images

The network one returns:
{
"errors": {},
"message": "api returned unexpected data - failed to parse api response",
"status": 400,
"success": false
}

curl 'https://172.22.2.5:8443/api2/extjs/pve/remotes/pve8r3q9-nvme/nodes/localhost/network?interface-type=any_bridge' \
> -H 'Accept: */*' \
> -H 'Cookie: PDMAuthCookie=PDM:root@pam:...' \
> -H 'Referer: https://172.22.2.5:8443/?' \
> -H 'Sec-Fetch-Dest: empty' \
> -H 'Sec-Fetch-Mode: cors' \
> -H 'Sec-Fetch-Site: same-origin' \
> -H 'cache-control: no-cache' \
> --insecure
{"errors":{},"message":"api returned unexpected data - failed to parse api response","status":400,"success":false}

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
thanks, that helps a bit, so it seems our parser for the network api is not correct currently.

can you share the output of the api calls from the pve remote itself?

e.g.
Code: 
pvesh get /nodes/<nodename>/network --type any_bridge --output-format json-pretty

you could also omit the '--type' so we get the output for all interfaces (maybe helps in the future)
also please censor the output if necessary ;)

thanks

EDIT: also the config of that vm would be interesting because of the 'could not parse 'file' property of 'ide2' error
 
dcsapak said:
EDIT: also the config of that vm would be interesting because of the 'could not parse 'file' property of 'ide2' error
Click to expand...
that's caused by empty cdrom drives IIRC..
 
dcsapak said:
you could also omit the '--type' so we get the output for all interfaces (maybe helps in the future)
Click to expand...
here you go, I guess the reason is clear - dhcp :-) As a reminder these are normally short-lived private cloud instances for CI/CD, hence the config.

pvesh get /nodes/pve8r3q9-host1-nvme/network --type any_bridge --output-format json-pretty
[
{
"active" : 1,
"autostart" : 1,
"bridge_fd" : "0",
"bridge_ports" : "ens3",
"bridge_stp" : "off",
"families" : [
"inet"
],
"iface" : "vmbr0",
"method" : "dhcp",
"method6" : "manual",
"priority" : 4,
"type" : "bridge"
}
]

pvesh get /nodes/pve8r3q9-host1-nvme/network --output-format json-pretty
[
{
"active" : 1,
"exists" : 1,
"families" : [
"inet"
],
"iface" : "ens7",
"method" : "manual",
"method6" : "manual",
"type" : "eth"
},
{
"active" : 1,
"autostart" : 1,
"bridge_fd" : "0",
"bridge_ports" : "ens3",
"bridge_stp" : "off",
"families" : [
"inet"
],
"iface" : "vmbr0",
"method" : "dhcp",
"method6" : "manual",
"priority" : 4,
"type" : "bridge"
},
{
"active" : 1,
"exists" : 1,
"families" : [
"inet"
],
"iface" : "ens3",
"method" : "manual",
"method6" : "manual",
"priority" : 3,
"type" : "eth"
}
]


qm config 100
boot: order=scsi0;ide2;net0
cores: 1
cpu: x86-64-v2-AES
ide2: none,media=cdrom
memory: 2048
meta: creation-qemu=9.0.2,ctime=1736180081
net0: virtio=02:BB:BB:7C:92:0B,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: bb-iscsi:vm-100-disk-0,iothread=1,size=1G
scsihw: virtio-scsi-single
smbios1: uuid=b1da7f16-77f8-40c1-a361-9ba43ea95452
sockets: 1
vmgenid: dc680872-1326-445c-9ded-fa53fb588bf9

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
Last edited:
Hello,

I'm also running into this API error, but not sure if it's already been patched.

Wanted to know how I could check / what info I could provide that'll help.
 
I have since updated to 0.1.11 and this issue persists. I've rebooted to ensure I'm on the latest kernel as well.

Anything I could do to help debug this / provide a fix?
 
I don't know if this has already been considered but the problem was that one of the two nodes was in DHCP, once the static IUP was set the problem disappeared, at least in my case.
 
oh god, I just spent all day wrestling with getting my stuff setup on DHCP.
Tried to migrate a VM, got this error and now I need a drink.

I didn't come here to whine, despite appearances, I'll see if I can add anything useful to the conversation.
 
So its July and this is still an issue. When setting up a DHCP scoped linux bridge on Proxmox PDM still throws "Error: api error (status = 400: api returned unexpected data - failed to parse api response)".

Wasn't this patched out around Jan 8th? if so code regression? Either way, very annoying when a DHCP enabled bridge is being used.
 
I am also facing this same problem still, this never seemed to be patched. I run my servers on DHCP, there is no reason for me to setup statics in my environment, nor is it actually possible without major risk to downtime in the event of a router reboot.

Is there a patch I can apply to fix this?
 
Hi, I'm using PDM 0.9.1.
I'm facing a similar issue, but looks different cause.
I don't use DHCP.

It looks, the error comes when a migration target VM is OFF.

* When VM is ON
https://(FQDN):8443/api2/extjs/pve/remotes/hci0x/qemu/104/status

{"data":{"cpu":0.103370922366178,"cpus":8.0,"diskread":110386688,"diskwrite":3036160,"ha":{"managed":0},"maxdisk":34359738368,"maxmem":8589934592,"mem":205795328,"memhost":623006720,"name":"vyos-140-02","netin":0,"netout":0,"pid":2004742,"pressurecpufull":0.0,"pressurecpusome":0.0,"pressureiofull":0.0,"pressureiosome":0.0,"pressurememoryfull":0.0,"pressurememorysome":0.0,"qmpstatus":"running","running-machine":"pc-i440fx-10.0+pve1","running-qemu":"10.0.2","status":"running","uptime":25,"vmid":104},"status":200,"success":true}

# pvesh get /nodes/hci02/qemu/104/status/current --output-format json-pretty
{
"balloon" : 8589934592,
"ballooninfo" : {
"actual" : 8589934592,
"free_mem" : 7578439680,
"last_update" : 1760954316,
"major_page_faults" : 502,
"max_mem" : 8589934592,
"mem_swapped_in" : 0,
"mem_swapped_out" : 0,
"minor_page_faults" : 719367,
"total_mem" : 8344281088
},
"blockstat" : {
"scsi0" : {
"account_failed" : true,
"account_invalid" : true,
"failed_flush_operations" : 0,
"failed_rd_operations" : 0,
"failed_unmap_operations" : 0,
"failed_wr_operations" : 0,
"failed_zone_append_operations" : 0,
"flush_operations" : 81,
"flush_total_time_ns" : 6980187,
"idle_time_ns" : 1013992856,
"invalid_flush_operations" : 0,
"invalid_rd_operations" : 0,
"invalid_unmap_operations" : 0,
"invalid_wr_operations" : 0,
"invalid_zone_append_operations" : 0,
"rd_bytes" : 199253504,
"rd_merged" : 0,
"rd_operations" : 7878,
"rd_total_time_ns" : 3241315950,
"timed_stats" : [],
"unmap_bytes" : 0,
"unmap_merged" : 0,
"unmap_operations" : 0,
"unmap_total_time_ns" : 0,
"wr_bytes" : 11523072,
"wr_highest_offset" : 15474405376,
"wr_merged" : 0,
"wr_operations" : 1014,
"wr_total_time_ns" : 4038342291,
"zone_append_bytes" : 0,
"zone_append_merged" : 0,
"zone_append_operations" : 0,
"zone_append_total_time_ns" : 0
}
},
"cpu" : 0,
"cpus" : 8,
"disk" : 0,
"diskread" : 199253504,
"diskwrite" : 11523072,
"freemem" : 7578439680,
"ha" : {
"managed" : 0
},
"maxdisk" : 34359738368,
"maxmem" : 8589934592,
"mem" : 765841408,
"memhost" : 1127136256,
"name" : "vyos-140-02",
"netin" : 0,
"netout" : 826,
"nics" : {
"tap104i0" : {
"netin" : 0,
"netout" : 826
}
},
"pid" : 2006786,
"pressurecpufull" : 0,
"pressurecpusome" : 0,
"pressureiofull" : 0,
"pressureiosome" : 0,
"pressurememoryfull" : 0,
"pressurememorysome" : 0,
"proxmox-support" : {
"backup-access-api" : true,
"backup-fleecing" : true,
"backup-max-workers" : true,
"pbs-dirty-bitmap" : true,
"pbs-dirty-bitmap-migration" : true,
"pbs-dirty-bitmap-savevm" : true,
"pbs-library-version" : "2.0.1 (2b8ebba0cffecd479f296aaad82d8bec89355599)",
"pbs-masterkey" : true,
"query-bitmap-info" : true
},
"qmpstatus" : "running",
"running-machine" : "pc-i440fx-10.0+pve1",
"running-qemu" : "10.0.2",
"status" : "running",
"uptime" : 41,
"vmid" : 104
}


* When VM is OFF

https://(FQDN):8443/api2/extjs/pve/remotes/hci0x/qemu/104/status

{"errors":{},"message":"api returned unexpected data - failed to parse api response","status":400,"success":false}


# pvesh get /nodes/hci02/qemu/104/status/current --output-format json-pretty

{
"cpu" : 0,
"cpus" : 8,
"disk" : 0,
"diskread" : null,
"diskwrite" : null,
"ha" : {
"managed" : 0
},
"maxdisk" : 34359738368,
"maxmem" : 8589934592,
"mem" : 0,
"memhost" : 0,
"name" : "vyos-140-02",
"netin" : 0,
"netout" : 0,
"qmpstatus" : "stopped",
"status" : "stopped",
"uptime" : 0,
"vmid" : 104
}
 

Attachments

  • ErrorOnMigrationClicked2.png
    ErrorOnMigrationClicked2.png
    120.8 KB · Views: 7
itaru said:
Hi, I'm using PDM 0.9.1.
I'm facing a similar issue, but looks different cause.
I don't use DHCP.

It looks, the error comes when a migration target VM is OFF.

* When VM is ON
https://(FQDN):8443/api2/extjs/pve/remotes/hci0x/qemu/104/status

{"data":{"cpu":0.103370922366178,"cpus":8.0,"diskread":110386688,"diskwrite":3036160,"ha":{"managed":0},"maxdisk":34359738368,"maxmem":8589934592,"mem":205795328,"memhost":623006720,"name":"vyos-140-02","netin":0,"netout":0,"pid":2004742,"pressurecpufull":0.0,"pressurecpusome":0.0,"pressureiofull":0.0,"pressureiosome":0.0,"pressurememoryfull":0.0,"pressurememorysome":0.0,"qmpstatus":"running","running-machine":"pc-i440fx-10.0+pve1","running-qemu":"10.0.2","status":"running","uptime":25,"vmid":104},"status":200,"success":true}

# pvesh get /nodes/hci02/qemu/104/status/current --output-format json-pretty
{
"balloon" : 8589934592,
"ballooninfo" : {
"actual" : 8589934592,
"free_mem" : 7578439680,
"last_update" : 1760954316,
"major_page_faults" : 502,
"max_mem" : 8589934592,
"mem_swapped_in" : 0,
"mem_swapped_out" : 0,
"minor_page_faults" : 719367,
"total_mem" : 8344281088
},
"blockstat" : {
"scsi0" : {
"account_failed" : true,
"account_invalid" : true,
"failed_flush_operations" : 0,
"failed_rd_operations" : 0,
"failed_unmap_operations" : 0,
"failed_wr_operations" : 0,
"failed_zone_append_operations" : 0,
"flush_operations" : 81,
"flush_total_time_ns" : 6980187,
"idle_time_ns" : 1013992856,
"invalid_flush_operations" : 0,
"invalid_rd_operations" : 0,
"invalid_unmap_operations" : 0,
"invalid_wr_operations" : 0,
"invalid_zone_append_operations" : 0,
"rd_bytes" : 199253504,
"rd_merged" : 0,
"rd_operations" : 7878,
"rd_total_time_ns" : 3241315950,
"timed_stats" : [],
"unmap_bytes" : 0,
"unmap_merged" : 0,
"unmap_operations" : 0,
"unmap_total_time_ns" : 0,
"wr_bytes" : 11523072,
"wr_highest_offset" : 15474405376,
"wr_merged" : 0,
"wr_operations" : 1014,
"wr_total_time_ns" : 4038342291,
"zone_append_bytes" : 0,
"zone_append_merged" : 0,
"zone_append_operations" : 0,
"zone_append_total_time_ns" : 0
}
},
"cpu" : 0,
"cpus" : 8,
"disk" : 0,
"diskread" : 199253504,
"diskwrite" : 11523072,
"freemem" : 7578439680,
"ha" : {
"managed" : 0
},
"maxdisk" : 34359738368,
"maxmem" : 8589934592,
"mem" : 765841408,
"memhost" : 1127136256,
"name" : "vyos-140-02",
"netin" : 0,
"netout" : 826,
"nics" : {
"tap104i0" : {
"netin" : 0,
"netout" : 826
}
},
"pid" : 2006786,
"pressurecpufull" : 0,
"pressurecpusome" : 0,
"pressureiofull" : 0,
"pressureiosome" : 0,
"pressurememoryfull" : 0,
"pressurememorysome" : 0,
"proxmox-support" : {
"backup-access-api" : true,
"backup-fleecing" : true,
"backup-max-workers" : true,
"pbs-dirty-bitmap" : true,
"pbs-dirty-bitmap-migration" : true,
"pbs-dirty-bitmap-savevm" : true,
"pbs-library-version" : "2.0.1 (2b8ebba0cffecd479f296aaad82d8bec89355599)",
"pbs-masterkey" : true,
"query-bitmap-info" : true
},
"qmpstatus" : "running",
"running-machine" : "pc-i440fx-10.0+pve1",
"running-qemu" : "10.0.2",
"status" : "running",
"uptime" : 41,
"vmid" : 104
}


* When VM is OFF

https://(FQDN):8443/api2/extjs/pve/remotes/hci0x/qemu/104/status

{"errors":{},"message":"api returned unexpected data - failed to parse api response","status":400,"success":false}


# pvesh get /nodes/hci02/qemu/104/status/current --output-format json-pretty

{
"cpu" : 0,
"cpus" : 8,
"disk" : 0,
"diskread" : null,
"diskwrite" : null,
"ha" : {
"managed" : 0
},
"maxdisk" : 34359738368,
"maxmem" : 8589934592,
"mem" : 0,
"memhost" : 0,
"name" : "vyos-140-02",
"netin" : 0,
"netout" : 0,
"qmpstatus" : "stopped",
"status" : "stopped",
"uptime" : 0,
"vmid" : 104
}
Click to expand...
I have the same error, same PDM version. When VM is offline, PDM throws a status 400 api error.
When VM is online, no issue in migrating.

I tried to migrate from a standalone host to a clustered host.
 
Last edited:
  • Like
Reactions: itaru
You must log in or register to reply here.
Top Bottom