The external network connection to/from a VM is not longer functionnal after a backup about 15% of the time. I am interested in any hint about how to avoid this.
The backup of a Virtual machine (CT) running in a LXC container on LVM (in /var/lib/vz) breaks the network and Proxmox VE t 5.2-6 . The machine itself is running Debian GNU/Linux 9.5 (stretch).
In this particular case, the backup took 3 min 32 sec . The service monitoring from another machine (every 3 minutes) reported failure at 03:03:26 while the backup started at 03:00:02. So failure is reported just before the backup actually finishes but well after the VM is online again.
Rebooting the VM does not help. Resetting the iptables does not help. Many other operations do not help. The only solution I have at this time is to reboot the physical machine.
Since a few weeks, the VM has its own dedicated IP which is statically assigned to the network device (veth) in the GUI as well as the MACADDR that the datacenter assigned to this IP.
Prior to that, the VM as well as the physical machine were sharing an IP supported by routing rules (iptables). The internal virutal network was still ok after backup, but it was impossible to connect to the public network from the VM.
Changing to a fixed IP about two weeks ago seemed to be a good workaround, but the problem re-appeared two days in a row now. The backup is daily. The network most often fails after Monday morning's backup but this time if failed on Tuesday's backup.
I have now defined a monitoring rule from another machine that will do a "ssh reboot" on the physical machine if the service is unavailable for 9 minutes.
This is the backup log if of any help:
INFO: starting new backup job: vzdump 100 --mailnotification always --node p1 --quiet 1 --compress lzo --mode suspend --storage local
INFO: Starting Backup of VM 100 (lxc)
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: connect
INFO: starting first sync /proc/2157/root// to /var/lib/vz/tmp_backup/vzdumptmp13100
INFO: Number of files: 120,165 (reg: 103,247, dir: 9,490, link: 7,396, dev: 2, special: 30)
INFO: Number of created files: 120,164 (reg: 103,247, dir: 9,489, link: 7,396, dev: 2, special: 30)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 103,237
INFO: Total file size: 12,120,705,976 bytes
INFO: Total transferred file size: 12,113,268,633 bytes
INFO: Literal data: 12,113,277,608 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 4,521,265
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 12,124,915,614
INFO: Total bytes received: 2,043,469
INFO: sent 12,124,915,614 bytes received 2,043,469 bytes 133,999,547.88 bytes/sec
INFO: total size is 12,120,705,976 speedup is 1.00
INFO: first sync finished (90 seconds)
INFO: suspend vm
INFO: starting final sync /proc/2157/root// to /var/lib/vz/tmp_backup/vzdumptmp13100
INFO: Number of files: 120,164 (reg: 103,246, dir: 9,490, link: 7,396, dev: 2, special: 30)
INFO: Number of created files: 0
INFO: Number of deleted files: 1 (reg: 1)
INFO: Number of regular files transferred: 718
INFO: Total file size: 12,120,748,157 bytes
INFO: Total transferred file size: 6,895,784,069 bytes
INFO: Literal data: 39,939,976 bytes
INFO: Matched data: 6,855,844,093 bytes
INFO: File list size: 458,709
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 45,075,392
INFO: Total bytes received: 1,557,739
INFO: sent 45,075,392 bytes received 1,557,739 bytes 923,428.34 bytes/sec
INFO: total size is 12,120,748,157 speedup is 259.92
INFO: final sync finished (50 seconds)
INFO: resume vm
INFO: vm is online again after 50 seconds
INFO: creating archive '/var/lib/vz/dump/vzdump-lxc-100-2018_08_21-03_00_02.tar.lzo'
INFO: Total bytes written: 11379374080 (11GiB, 157MiB/s)
INFO: archive file size: 4.86GB
INFO: delete old backup '/var/lib/vz/dump/vzdump-lxc-100-2018_08_18-03_00_02.tar.lzo'
INFO: Finished Backup of VM 100 (00:03:32)
INFO: Backup job finished successfully
TASK OK
The backup of a Virtual machine (CT) running in a LXC container on LVM (in /var/lib/vz) breaks the network and Proxmox VE t 5.2-6 . The machine itself is running Debian GNU/Linux 9.5 (stretch).
In this particular case, the backup took 3 min 32 sec . The service monitoring from another machine (every 3 minutes) reported failure at 03:03:26 while the backup started at 03:00:02. So failure is reported just before the backup actually finishes but well after the VM is online again.
Rebooting the VM does not help. Resetting the iptables does not help. Many other operations do not help. The only solution I have at this time is to reboot the physical machine.
Since a few weeks, the VM has its own dedicated IP which is statically assigned to the network device (veth) in the GUI as well as the MACADDR that the datacenter assigned to this IP.
Prior to that, the VM as well as the physical machine were sharing an IP supported by routing rules (iptables). The internal virutal network was still ok after backup, but it was impossible to connect to the public network from the VM.
Changing to a fixed IP about two weeks ago seemed to be a good workaround, but the problem re-appeared two days in a row now. The backup is daily. The network most often fails after Monday morning's backup but this time if failed on Tuesday's backup.
I have now defined a monitoring rule from another machine that will do a "ssh reboot" on the physical machine if the service is unavailable for 9 minutes.
This is the backup log if of any help:
INFO: starting new backup job: vzdump 100 --mailnotification always --node p1 --quiet 1 --compress lzo --mode suspend --storage local
INFO: Starting Backup of VM 100 (lxc)
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: connect
INFO: starting first sync /proc/2157/root// to /var/lib/vz/tmp_backup/vzdumptmp13100
INFO: Number of files: 120,165 (reg: 103,247, dir: 9,490, link: 7,396, dev: 2, special: 30)
INFO: Number of created files: 120,164 (reg: 103,247, dir: 9,489, link: 7,396, dev: 2, special: 30)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 103,237
INFO: Total file size: 12,120,705,976 bytes
INFO: Total transferred file size: 12,113,268,633 bytes
INFO: Literal data: 12,113,277,608 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 4,521,265
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 12,124,915,614
INFO: Total bytes received: 2,043,469
INFO: sent 12,124,915,614 bytes received 2,043,469 bytes 133,999,547.88 bytes/sec
INFO: total size is 12,120,705,976 speedup is 1.00
INFO: first sync finished (90 seconds)
INFO: suspend vm
INFO: starting final sync /proc/2157/root// to /var/lib/vz/tmp_backup/vzdumptmp13100
INFO: Number of files: 120,164 (reg: 103,246, dir: 9,490, link: 7,396, dev: 2, special: 30)
INFO: Number of created files: 0
INFO: Number of deleted files: 1 (reg: 1)
INFO: Number of regular files transferred: 718
INFO: Total file size: 12,120,748,157 bytes
INFO: Total transferred file size: 6,895,784,069 bytes
INFO: Literal data: 39,939,976 bytes
INFO: Matched data: 6,855,844,093 bytes
INFO: File list size: 458,709
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 45,075,392
INFO: Total bytes received: 1,557,739
INFO: sent 45,075,392 bytes received 1,557,739 bytes 923,428.34 bytes/sec
INFO: total size is 12,120,748,157 speedup is 259.92
INFO: final sync finished (50 seconds)
INFO: resume vm
INFO: vm is online again after 50 seconds
INFO: creating archive '/var/lib/vz/dump/vzdump-lxc-100-2018_08_21-03_00_02.tar.lzo'
INFO: Total bytes written: 11379374080 (11GiB, 157MiB/s)
INFO: archive file size: 4.86GB
INFO: delete old backup '/var/lib/vz/dump/vzdump-lxc-100-2018_08_18-03_00_02.tar.lzo'
INFO: Finished Backup of VM 100 (00:03:32)
INFO: Backup job finished successfully
TASK OK