Backup bricht bei einer VM ab

Maximilian

New Member
Apr 6, 2021
5
0
1
22
Guten Tag,

ich habe mir vor ca. 2 Wochen einen PBS aufgesetzt, welcher aktuell 2 Nodes (kein Cluster) backuped.

Insgesammt ca. 30 VM's.

Das Backup bricht allerdings immer bei einem Node bei der VM 105 ab (und nur da) egal ob ich es händisch starte, oder nicht:

PBS Logs:

2021-04-05T22:02:21+00:00: starting new backup on datastore 'backups2': "vm/105/2021-04-05T22:02:21Z"
2021-04-05T22:02:21+00:00: download 'index.json.blob' from previous backup.
2021-04-05T22:02:21+00:00: register chunks in 'drive-scsi0.img.fidx' from previous backup.
2021-04-05T22:02:21+00:00: download 'drive-scsi0.img.fidx' from previous backup.
2021-04-05T22:02:21+00:00: created new fixed index 1 ("vm/105/2021-04-05T22:02:21Z/drive-scsi0.img.fidx")
2021-04-05T22:02:21+00:00: add blob "/backups2/vm/105/2021-04-05T22:02:21Z/qemu-server.conf.blob" (333 bytes, comp: 333)
2021-04-05T22:16:07+00:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
2021-04-05T22:16:07+00:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
2021-04-05T22:16:07+00:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
2021-04-05T22:16:07+00:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
2021-04-05T22:16:07+00:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
2021-04-05T22:16:07+00:00: backup failed: connection error: Connection timed out (os error 110)
2021-04-05T22:16:07+00:00: removing failed backup
2021-04-05T22:16:07+00:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
2021-04-05T22:16:07+00:00: TASK ERROR: connection error: Connection timed out (os error 110)


PVE Logs:

INFO: started backup task '6e7cd067-1488-4e4b-929c-60aa13b6b134'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: OK (23.6 GiB of 35.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 23.6 GiB dirty of 35.0 GiB total
INFO: 1% (360.0 MiB of 23.6 GiB) in 3s, read: 120.0 MiB/s, write: 120.0 MiB/s
INFO: 2% (696.0 MiB of 23.6 GiB) in 6s, read: 112.0 MiB/s, write: 112.0 MiB/s
INFO: 4% (1.0 GiB of 23.6 GiB) in 9s, read: 114.7 MiB/s, write: 114.7 MiB/s
INFO: 5% (1.3 GiB of 23.6 GiB) in 12s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 7% (1.7 GiB of 23.6 GiB) in 15s, read: 109.3 MiB/s, write: 109.3 MiB/s
INFO: 8% (2.0 GiB of 23.6 GiB) in 18s, read: 106.7 MiB/s, write: 106.7 MiB/s
INFO: 9% (2.3 GiB of 23.6 GiB) in 21s, read: 100.0 MiB/s, write: 100.0 MiB/s
INFO: 11% (2.6 GiB of 23.6 GiB) in 24s, read: 116.0 MiB/s, write: 116.0 MiB/s
INFO: 12% (2.9 GiB of 23.6 GiB) in 27s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 13% (3.3 GiB of 23.6 GiB) in 30s, read: 117.3 MiB/s, write: 117.3 MiB/s
INFO: 15% (3.6 GiB of 23.6 GiB) in 33s, read: 102.7 MiB/s, write: 102.7 MiB/s
INFO: 16% (3.9 GiB of 23.6 GiB) in 36s, read: 102.7 MiB/s, write: 102.7 MiB/s
INFO: 17% (4.2 GiB of 23.6 GiB) in 39s, read: 105.3 MiB/s, write: 105.3 MiB/s
INFO: 19% (4.5 GiB of 23.6 GiB) in 42s, read: 108.0 MiB/s, write: 93.3 MiB/s
INFO: 20% (4.8 GiB of 23.6 GiB) in 45s, read: 113.3 MiB/s, write: 113.3 MiB/s
INFO: 21% (5.1 GiB of 23.6 GiB) in 48s, read: 106.7 MiB/s, write: 106.7 MiB/s
INFO: 23% (5.5 GiB of 23.6 GiB) in 51s, read: 112.0 MiB/s, write: 112.0 MiB/s
INFO: 24% (5.8 GiB of 23.6 GiB) in 54s, read: 113.3 MiB/s, write: 113.3 MiB/s
INFO: 25% (6.1 GiB of 23.6 GiB) in 57s, read: 112.0 MiB/s, write: 112.0 MiB/s
INFO: 26% (6.2 GiB of 23.6 GiB) in 58s, read: 68.0 MiB/s, write: 68.0 MiB/s
ERROR: backup write data failed: command error: broken pipe
INFO: aborting backup job
ERROR: Backup of VM 105 failed - backup write data failed: command error: broken pipe
INFO: Failed at 2021-04-06 09:19:37
INFO: Backup job finished with errors
TASK ERROR: job errors

Ich vermute jedoch kein Netzwerkproblem - es ist egal um welche Zeit ich das ganze starte, der Fehler tritt sporadisch auf.

Selbst wenn ich zwei VM's gleichzeitig backuppen lasse, funktioniert immer nur das Backup der VM105 nicht.

Hat dazu ggf. jemand eine Idee? :)
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,438
559
133
32
Vienna
was steht denn um diese zeit in den syslogs der beiden (pve/pbs) server?
 

Maximilian

New Member
Apr 6, 2021
5
0
1
22
Moin,

genau das gleiche:



Code:
Apr  6 04:31:47 pbs systemd[1]: Started Daily Proxmox Backup Server update and maintenance activities.
Apr  6 04:44:01 pbs CRON[25666]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Apr  6 05:44:01 pbs CRON[26688]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Apr  6 06:18:01 pbs CRON[27841]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Apr  6 06:22:52 pbs systemd[1]: Starting Daily apt upgrade and clean activities...
Apr  6 06:22:53 pbs systemd[1]: apt-daily-upgrade.service: Succeeded.
Apr  6 06:22:53 pbs systemd[1]: Started Daily apt upgrade and clean activities.
Apr  6 06:44:01 pbs CRON[28344]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Apr  6 07:18:39 pbs proxmox-backup-proxy[233]: starting new backup on datastore 'backups2': "vm/105/2021-04-06T07:18:39Z"
Apr  6 07:18:39 pbs proxmox-backup-proxy[233]: download 'index.json.blob' from previous backup.
Apr  6 07:18:39 pbs proxmox-backup-proxy[233]: register chunks in 'drive-scsi0.img.fidx' from previous backup.
Apr  6 07:18:39 pbs proxmox-backup-proxy[233]: download 'drive-scsi0.img.fidx' from previous backup.
Apr  6 07:18:39 pbs proxmox-backup-proxy[233]: created new fixed index 1 ("vm/105/2021-04-06T07:18:39Z/drive-scsi0.img.fidx")
Apr  6 07:18:39 pbs proxmox-backup-proxy[233]: add blob "/backups2/vm/105/2021-04-06T07:18:39Z/qemu-server.conf.blob" (333 bytes, comp: 333)
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: backup failed: connection error: Connection timed out (os error 110)
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: removing failed backup
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: broken pipe
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: removing backup snapshot "/backups2/vm/105/2021-04-06T07:18:39Z"
Apr  6 07:33:00 pbs proxmox-backup-proxy[233]: TASK ERROR: connection error: Connection timed out (os error 110)
Apr  6 07:44:01 pbs CRON[28804]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
 

Maximilian

New Member
Apr 6, 2021
5
0
1
22
Ah sorry, dass wäre dann das:


Apr 6 09:17:01 srv02 CRON[32057]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 6 09:17:55 srv02 pvedaemon[8947]: <root@pam> successful auth for user 'root@pam'
Apr 6 09:18:00 srv02 systemd[1]: Starting Proxmox VE replication runner...
Apr 6 09:18:00 srv02 systemd[1]: pvesr.service: Succeeded.
Apr 6 09:18:00 srv02 systemd[1]: Started Proxmox VE replication runner.
Apr 6 09:18:39 srv02 pvedaemon[8947]: <root@pam> starting task UPID:srv02:00000156:0C31D873:606C0B4F:vzdump:105:root@pam:
Apr 6 09:18:39 srv02 pvedaemon[342]: INFO: starting new backup job: vzdump 105 --mode snapshot --storage backups --node srv02 --remove 0
Apr 6 09:18:39 srv02 pvedaemon[342]: INFO: Starting Backup of VM 105 (qemu)
Apr 6 09:19:00 srv02 systemd[1]: Starting Proxmox VE replication runner...
Apr 6 09:19:00 srv02 systemd[1]: pvesr.service: Succeeded.
Apr 6 09:19:00 srv02 systemd[1]: Started Proxmox VE replication runner.
Apr 6 09:19:36 srv02 pvedaemon[8947]: worker exit
Apr 6 09:19:36 srv02 pvedaemon[1160]: worker 8947 finished
Apr 6 09:19:36 srv02 pvedaemon[1160]: starting 1 worker(s)
Apr 6 09:19:36 srv02 pvedaemon[1160]: worker 825 started
Apr 6 09:19:37 srv02 QEMU[7863]: HTTP/2.0 connection failed
Apr 6 09:19:37 srv02 pvedaemon[342]: ERROR: Backup of VM 105 failed - backup write data failed: command error: broken pipe
Apr 6 09:19:37 srv02 pvedaemon[342]: INFO: Backup job finished with errors
Apr 6 09:19:37 srv02 pvedaemon[342]: job errors
Apr 6 09:19:41 srv02 pvestatd[1135]: backups: error fetching datastores - 500 Can't connect to pbs.ZENSIERT:8007 (Connection reset by peer)
 
Last edited:

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,438
559
133
32
Vienna
Apr 6 09:19:37 srv02 QEMU[7863]: HTTP/2.0 connection failed
deutet doch auf ein netzwerk problem hin. ist vielleicht ein (reverse)proxy dazwischen?
 

Maximilian

New Member
Apr 6, 2021
5
0
1
22
Hallo,

nein, eben nicht, und ja, von den Logs her würde ich das gleiche sagen, aber warum immer nur bei VM 105?

Viele Grüße!
 

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
5,438
559
133
32
Vienna
ist mit der vm disk alles in ordnung?
vielleicht einmal fsck im gast laufen lassen, oder die ganze disk einmal lesen?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!