Connections between Nodes stop when trying to Migrate Server

[LeiCraft_]

Hey,

if have a Proxmox Setup with two nodes:

Host03 and Host05 where host03 is the primary.
When i try migrating vms from host03 to host05 everything is fine. but when doing in reverse it fails. and with it the entire datacenter goes out of sync where they other node is showed as offline. trying to reboot or restart doesnt help with anything. but the connection will reastablish after an hour or so automaticly somehow.
this is what the logs are saying:

migration Log

2025-07-03 00:55:32 starting migration of VM 301 to node 'host03' (host03_ip)
2025-07-03 00:55:32 found local disk 'local:301/base-301-disk-0.raw' (attached)
2025-07-03 00:55:32 found generated disk 'local:301/vm-301-cloudinit.raw' (in current VM config)
2025-07-03 00:55:32 copying local disk images
2025-07-03 00:55:33 Formatting '/var/lib/vz/images/301/base-301-disk-0.raw', fmt=raw size=2147483648 preallocation=off
2025-07-03 00:55:37 360443904 bytes (360 MB, 344 MiB) copied, 3 s, 120 MB/s
2025-07-03 00:55:40 735469568 bytes (735 MB, 701 MiB) copied, 6 s, 123 MB/s
2025-07-03 00:55:50 1312784384 bytes (1.3 GB, 1.2 GiB) copied, 15 s, 87.5 MB/s
2025-07-03 00:55:50 1437351936 bytes (1.4 GB, 1.3 GiB) copied, 18 s, 81.2 MB/s
2025-07-03 00:55:51 1437356032 bytes (1.4 GB, 1.3 GiB) copied, 18 s, 81.2 MB/s
2025-07-03 00:55:52 1470369792 bytes (1.5 GB, 1.4 GiB) copied, 18 s, 81.7 MB/s
2025-07-03 00:56:01 2007277568 bytes (2.0 GB, 1.9 GiB) copied, 27 s, 73.2 MB/s
2025-07-03 00:56:01 524288+0 records in
2025-07-03 00:56:01 524288+0 records out
2025-07-03 00:56:01 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 28.6148 s, 75.0 MB/s
2025-07-03 00:56:02 961+66230 records in
2025-07-03 00:56:02 961+66230 records out
2025-07-03 00:56:02 2147483648 bytes (2.1 GB, 2.0 GiB) copied, 29.1773 s, 73.6 MB/s
2025-07-03 00:56:02 successfully imported 'local:301/base-301-disk-0.raw'
2025-07-03 00:56:02 volume 'local:301/base-301-disk-0.raw' is 'local:301/base-301-disk-0.raw' on the target
2025-07-03 00:56:02 kex_exchange_identification: read: Connection reset by peer
2025-07-03 00:56:02 Connection reset by {host03_ip} port 22
2025-07-03 00:56:03 command 'dd 'if=/var/lib/vz/images/301/vm-301-cloudinit.raw' 'bs=4k' 'status=progress'' failed: got signal 13
2025-07-03 00:56:03 ERROR: storage migration for 'local:301/vm-301-cloudinit.raw' to storage 'local' failed - command 'set -o pipefail && pvesm export local:301/vm-301-cloudinit.raw raw+size - -with-snapshots 0 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=host03' -o 'UserKnownHostsFile=/etc/pve/nodes/host03/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@{host03_ip} -- pvesm import local:301/vm-301-cloudinit.raw raw+size - -with-snapshots 0 -allow-rename 1' failed: exit code 255
2025-07-03 00:56:03 aborting phase 1 - cleanup resources
2025-07-03 00:56:03 ERROR: unable to open file '/etc/pve/nodes/host05/qemu-server/301.conf.tmp.47324' - Permission denied
2025-07-03 00:56:05 ERROR: migration aborted (duration 00:00:34): storage migration for 'local:301/vm-301-cloudinit.raw' to storage 'local' failed - command 'set -o pipefail && pvesm export local:301/vm-301-cloudinit.raw raw+size - -with-snapshots 0 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=host03' -o 'UserKnownHostsFile=/etc/pve/nodes/host03/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@{host03_ip} -- pvesm import local:301/vm-301-cloudinit.raw raw+size - -with-snapshots 0 -allow-rename 1' failed: exit code 255
TASK ERROR: migration aborted

also: at the moment the migration fails, my ssh session to host03 from terminus get terminated

 

[Magnus-mercer]

You can first add the following to /etc/ssh/sshd_config on both nodes:

ClientAliveInterval 60
ClientAliveCountMax 3

and then restart the sshd service—this prevents SSH from dropping the connection during large-file transfers.

Next, dedicate a separate network interface for migration traffic and assign it under “Datacenter → Options → Migration Network” in the Proxmox GUI—this keeps corosync and the /etc/pve mount from being disrupted.