Hi,
Unfortunately I can't confirm this... production key - problem still existing.
Tobias
Error: http request timed out
- Thread starter iprigger
- Start date
Unfortunately I can't confirm this... production key - problem still existing.
Tobias
Note, the respective updates for this did only hit the no-subscription repositories, not yet any enterprise repository - so if you use enterprise that would explain it.
Hi Tim,
Thanks for the clarification - yes, I am using the enterprise repo... (production systems
)Any idea when this will be fixed in the enterprise repo?
Tobias
The last time I checked my name wasn't Tim, but from the quote I figure you meant meHi Tim,
Thanks for the clarification - yes, I am using the enterprise repo... (production systems
Any idea when this will be fixed in the enterprise repo?
Tobias
There were some updates pushed out today just a bit ago, including the proxmox-backup-client and qemu-server, both with a change to increase this timeout.
Hi,The last time I checked my name wasn't Tim, but from the quote I figure you meant me
There were some updates pushed out today just a bit ago, including the proxmox-backup-client and qemu-server, both with a change to increase this timeout.
I still get:
INFO: 99% (8.6 GiB of 8.6 GiB) in 27m 2s, read: 25.3 MiB/s, write: 25.3 MiB/s
INFO: 100% (8.6 GiB of 8.6 GiB) in 27m 9s, read: 4.0 MiB/s, write: 4.0 MiB/s
INFO: backup was done incrementally, reused 3.39 GiB (28%)
INFO: transferred 8.61 GiB in 1668 seconds (5.3 MiB/s)
ERROR: Backup of VM 730 failed - proxmox-backup-client failed: Error: http request timed out
... do I need to reboot the host systems?
Tobias
All...
Me again. This issue is killing my complete infrastructure over and over again. Http errors, stuck VMs, etc.
Basically: I have to babysit and pamper ALL backups ALL THE TIME. Honestly? This is not usable for production. Not if I don't want to reboot all host systems & VMs every day...
This used to be working in the beta...
What has gone wrong here?
Kindest
Tobias
Me again. This issue is killing my complete infrastructure over and over again. Http errors, stuck VMs, etc.
Basically: I have to babysit and pamper ALL backups ALL THE TIME. Honestly? This is not usable for production. Not if I don't want to reboot all host systems & VMs every day...
This used to be working in the beta...
What has gone wrong here?
Kindest
Tobias
Hi,
Below the requested output
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-4.15: 5.4-8
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.13-pve1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.13-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1
Yesterday, the backup killed most of my database servers... (reboot necessary).
Tobias
Below the requested output
proxmox-ve: 6.3-1 (running kernel: 5.4.73-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-4.15: 5.4-8
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.13-pve1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.13-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1
Yesterday, the backup killed most of my database servers... (reboot necessary).
Tobias
Hi Fabian,
Here you are... pbs server is a dedicated machine with 16G ram (i7) and multi 10G interfaces...
INFO: starting new backup job: vzdump --mailnotification always --storage zz_PBS01 --node burfell --all 1 --quiet 1 --mode snapshot
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2021-03-28 06:00:02
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a-core:103/vm-103-disk-0.raw' 16G
INFO: creating Proxmox Backup Server archive 'vm/103/2021-03-28T04:00:02Z'
INFO: starting kvm to execute backup task
INFO: started backup task '1a68d8b1-3553-441f-921b-9a71465b7515'
INFO: virtio0: dirty-bitmap status: created new
INFO: 3% (644.0 MiB of 16.0 GiB) in 3s, read: 214.7 MiB/s, write: 0 B/s
INFO: 98% (15.7 GiB of 16.0 GiB) in 2m 4s, read: 337.3 MiB/s, write: 0 B/s
INFO: 100% (16.0 GiB of 16.0 GiB) in 2m 6s, read: 142.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 13.99 GiB (87%) total zero data
INFO: backup was done incrementally, reused 16.00 GiB (100%)
INFO: transferred 16.00 GiB in 126 seconds (130.0 MiB/s)
INFO: stopping kvm after backup task
INFO: running 'proxmox-backup-client prune' for 'vm/103'
INFO: Finished Backup of VM 103 (00:02:11)
INFO: Backup finished at 2021-03-28 06:02:13
INFO: Starting Backup of VM 119 (qemu)
INFO: Backup started at 2021-03-28 06:02:13
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ***
[....]
INFO: Starting backup: ct/9031/2021-03-28T07:19:03Z
INFO: Client name: burfell
INFO: Starting backup protocol: Sun Mar 28 09:19:59 2021
INFO: Downloading previous manifest (Sun Mar 21 08:17:06 2021)
INFO: Upload config file '/var/tmp/vzdumptmp5828_9031/etc/vzdump/pct.conf' to 'pbscnr@pbs@172.29.233.254:8007bs01' as pct.conf.blob
INFO: Upload directory '/var/tmp/vzdumptmp5828_9031' to 'pbscnr@pbs@172.29.233.254:8007bs01' as root.pxar.didx
INFO: root.pxar: had to upload 2.00 GiB of 8.61 GiB in 465.92s, average speed 4.39 MiB/s).
INFO: root.pxar: backup was done incrementally, reused 6.61 GiB (76.8%)
INFO: Uploaded backup catalog (845.53 KiB)
INFO: Duration: 469.04s
INFO: End Time: Sun Mar 28 09:27:48 2021
ERROR: Backup of VM 9031 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 09:28:11
INFO: Starting Backup of VM 9115 (qemu)
INFO: Backup started at 2021-03-28 09:28:11
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:9115/vm-9115-disk-1.qcow2' 100G
INFO: include disk 'virtio1' 'esnx-a:9115/vm-9115-disk-1.qcow2' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9115/2021-03-28T07:28:11Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '6cbe2d3c-8801-4672-bb74-784617743cee'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (196.0 MiB of 132.0 GiB) in 3s, read: 65.3 MiB/s, write: 1.3 MiB/s
INFO: 1% (1.4 GiB of 132.0 GiB) in 20s, read: 71.5 MiB/s, write: 0 B/s
INFO: 99% (131.5 GiB of 132.0 GiB) in 1h 11m 51s, read: 1.4 GiB/s, write: 10.7 MiB/s
INFO: 100% (132.0 GiB of 132.0 GiB) in 1h 12m 22s, read: 17.0 MiB/s, write: 4.4 MiB/s
INFO: backup is sparse: 61.48 GiB (46%) total zero data
INFO: backup was done incrementally, reused 116.83 GiB (88%)
INFO: transferred 132.00 GiB in 4347 seconds (31.1 MiB/s)
ERROR: Backup of VM 9115 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 10:41:08
INFO: Starting Backup of VM 9117 (qemu)
INFO: Backup started at 2021-03-28 10:41:08
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a:9117/vm-9117-disk-0.raw' 12G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9117/2021-03-28T08:41:08Z'
INFO: started backup task 'dfde0caa-14da-4479-8751-d4bcccb0ada1'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 1% (184.0 MiB of 12.0 GiB) in 3s, read: 61.3 MiB/s, write: 5.3 MiB/s
INFO: 2% (264.0 MiB of 12.0 GiB) in 6s, read: 26.7 MiB/s, write: 0 B/s
INFO: 99% (11.9 GiB of 12.0 GiB) in 7m 20s, read: 32.0 MiB/s, write: 0 B/s
INFO: 100% (12.0 GiB of 12.0 GiB) in 7m 23s, read: 36.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 2.40 GiB (20%) total zero data
INFO: backup was done incrementally, reused 11.59 GiB (96%)
INFO: transferred 12.00 GiB in 443 seconds (27.7 MiB/s)
ERROR: Backup of VM 9117 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 10:48:58
INFO: Starting Backup of VM 9119 (qemu)
INFO: Backup started at 2021-03-28 10:48:58
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:9119/vm-9119-disk-0.raw' 32G
INFO: include disk 'virtio1' 'esnx-a:9119/vm-9119-disk-0.raw' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9119/2021-03-28T08:48:58Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '973688b6-195d-432d-8455-cba3cc350cb5'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (192.0 MiB of 64.0 GiB) in 3s, read: 64.0 MiB/s, write: 18.7 MiB/s
INFO: 1% (656.0 MiB of 64.0 GiB) in 42s, read: 11.9 MiB/s, write: 2.3 MiB/s
INFO: 99% (63.9 GiB of 64.0 GiB) in 11m 1s, read: 394.7 MiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 11m 2s, read: 144.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 47.57 GiB (74%) total zero data
INFO: backup was done incrementally, reused 61.96 GiB (96%)
INFO: transferred 64.00 GiB in 662 seconds (99.0 MiB/s)
ERROR: Backup of VM 9119 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 11:00:21
INFO: Starting Backup of VM 9200 (qemu)
INFO: Backup started at 2021-03-28 11:00:21
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:9200/vm-9200-disk-0.raw' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9200/2021-03-28T09:00:21Z'
INFO: started backup task '71e6afce-d5f4-4697-b69e-51187c630054'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 2% (672.0 MiB of 32.0 GiB) in 3s, read: 224.0 MiB/s, write: 0 B/s
INFO: 3% (1.2 GiB of 32.0 GiB) in 6s, read: 180.0 MiB/s, write: 2.7 MiB/s
INFO: 98% (31.5 GiB of 32.0 GiB) in 1m 55s, read: 448.0 MiB/s, write: 0 B/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 1m 57s, read: 232.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 29.32 GiB (91%) total zero data
INFO: backup was done incrementally, reused 31.86 GiB (99%)
INFO: transferred 32.00 GiB in 117 seconds (280.1 MiB/s)
INFO: running 'proxmox-backup-client prune' for 'vm/9200'
INFO: Finished Backup of VM 9200 (00:02:02)
INFO: Backup finished at 2021-03-28 11:02:23
INFO: Starting Backup of VM 30000 (qemu)
INFO: Backup started at 2021-03-28 11:02:23
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:30000/vm-30000-disk-1.qcow2' 32G
INFO: include disk 'virtio1' 'esnx-a:30000/vm-30000-disk-0.qcow2' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/30000/2021-03-28T09:02:23Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '8c063bc6-114e-45a5-b665-dc9c35d8d354'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (96.0 MiB of 64.0 GiB) in 3s, read: 32.0 MiB/s, write: 2.7 MiB/s
INFO: 1% (664.0 MiB of 64.0 GiB) in 25s, read: 25.8 MiB/s, write: 372.4 KiB/s
INFO: 98% (63.0 GiB of 64.0 GiB) in 7m 24s, read: 424.0 MiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 7m 27s, read: 344.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 52.23 GiB (81%) total zero data
INFO: backup was done incrementally, reused 62.71 GiB (97%)
INFO: transferred 64.00 GiB in 447 seconds (146.6 MiB/s)
ERROR: Backup of VM 30000 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 11:10:11
INFO: Starting Backup of VM 90010 (qemu)
INFO: Backup started at 2021-03-28 11:10:11
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a:90010/vm-90010-disk-0.raw' 64G
INFO: include disk 'virtio1' 'esnx-a:90010/vm-90010-disk-1.raw' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'esns-b:90010/vm-90010-disk-0.raw' (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/90010/2021-03-28T09:10:11Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '72846bcf-0cd5-4eee-b5ca-b74b31b253e2'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (116.0 MiB of 128.0 GiB) in 3s, read: 38.7 MiB/s, write: 8.0 MiB/s
INFO: 1% (1.3 GiB of 128.0 GiB) in 30s, read: 45.0 MiB/s, write: 2.5 MiB/s
INFO: 99% (126.7 GiB of 128.0 GiB) in 54m 30s, read: 406.7 MiB/s, write: 0 B/s
INFO: 100% (128.0 GiB of 128.0 GiB) in 54m 42s, read: 108.7 MiB/s, write: 0 B/s
INFO: backup is sparse: 65.29 GiB (51%) total zero data
INFO: backup was done incrementally, reused 117.14 GiB (91%)
INFO: transferred 128.00 GiB in 3282 seconds (39.9 MiB/s)
ERROR: Backup of VM 90010 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 12:05:17
INFO: Starting Backup of VM 90011 (qemu)
INFO: Backup started at 2021-03-28 12:05:17
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a:90011/vm-90011-disk-0.raw' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/90011/2021-03-28T10:05:17Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task 'e3f890b2-8239-4b4f-a4bb-23dbd845ce9c'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (56.0 MiB of 64.0 GiB) in 3s, read: 18.7 MiB/s, write: 2.7 MiB/s
INFO: 1% (664.0 MiB of 64.0 GiB) in 30s, read: 22.5 MiB/s, write: 6.5 MiB/s
INFO: 99% (63.5 GiB of 64.0 GiB) in 35m 5s, read: 254.7 MiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 35m 15s, read: 50.8 MiB/s, write: 0 B/s
INFO: backup is sparse: 18.79 GiB (29%) total zero data
INFO: backup was done incrementally, reused 58.07 GiB (90%)
INFO: transferred 64.00 GiB in 2115 seconds (31.0 MiB/s)
ERROR: Backup of VM 90011 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 12:40:56
INFO: Backup job finished with errors
TASK ERROR: job errors
Tobias
Here you are... pbs server is a dedicated machine with 16G ram (i7) and multi 10G interfaces...
INFO: starting new backup job: vzdump --mailnotification always --storage zz_PBS01 --node burfell --all 1 --quiet 1 --mode snapshot
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2021-03-28 06:00:02
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a-core:103/vm-103-disk-0.raw' 16G
INFO: creating Proxmox Backup Server archive 'vm/103/2021-03-28T04:00:02Z'
INFO: starting kvm to execute backup task
INFO: started backup task '1a68d8b1-3553-441f-921b-9a71465b7515'
INFO: virtio0: dirty-bitmap status: created new
INFO: 3% (644.0 MiB of 16.0 GiB) in 3s, read: 214.7 MiB/s, write: 0 B/s
INFO: 98% (15.7 GiB of 16.0 GiB) in 2m 4s, read: 337.3 MiB/s, write: 0 B/s
INFO: 100% (16.0 GiB of 16.0 GiB) in 2m 6s, read: 142.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 13.99 GiB (87%) total zero data
INFO: backup was done incrementally, reused 16.00 GiB (100%)
INFO: transferred 16.00 GiB in 126 seconds (130.0 MiB/s)
INFO: stopping kvm after backup task
INFO: running 'proxmox-backup-client prune' for 'vm/103'
INFO: Finished Backup of VM 103 (00:02:11)
INFO: Backup finished at 2021-03-28 06:02:13
INFO: Starting Backup of VM 119 (qemu)
INFO: Backup started at 2021-03-28 06:02:13
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ***
[....]
INFO: Starting backup: ct/9031/2021-03-28T07:19:03Z
INFO: Client name: burfell
INFO: Starting backup protocol: Sun Mar 28 09:19:59 2021
INFO: Downloading previous manifest (Sun Mar 21 08:17:06 2021)
INFO: Upload config file '/var/tmp/vzdumptmp5828_9031/etc/vzdump/pct.conf' to 'pbscnr@pbs@172.29.233.254:8007
bs01' as pct.conf.blobINFO: Upload directory '/var/tmp/vzdumptmp5828_9031' to 'pbscnr@pbs@172.29.233.254:8007
bs01' as root.pxar.didxINFO: root.pxar: had to upload 2.00 GiB of 8.61 GiB in 465.92s, average speed 4.39 MiB/s).
INFO: root.pxar: backup was done incrementally, reused 6.61 GiB (76.8%)
INFO: Uploaded backup catalog (845.53 KiB)
INFO: Duration: 469.04s
INFO: End Time: Sun Mar 28 09:27:48 2021
ERROR: Backup of VM 9031 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 09:28:11
INFO: Starting Backup of VM 9115 (qemu)
INFO: Backup started at 2021-03-28 09:28:11
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:9115/vm-9115-disk-1.qcow2' 100G
INFO: include disk 'virtio1' 'esnx-a:9115/vm-9115-disk-1.qcow2' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9115/2021-03-28T07:28:11Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '6cbe2d3c-8801-4672-bb74-784617743cee'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (196.0 MiB of 132.0 GiB) in 3s, read: 65.3 MiB/s, write: 1.3 MiB/s
INFO: 1% (1.4 GiB of 132.0 GiB) in 20s, read: 71.5 MiB/s, write: 0 B/s
INFO: 99% (131.5 GiB of 132.0 GiB) in 1h 11m 51s, read: 1.4 GiB/s, write: 10.7 MiB/s
INFO: 100% (132.0 GiB of 132.0 GiB) in 1h 12m 22s, read: 17.0 MiB/s, write: 4.4 MiB/s
INFO: backup is sparse: 61.48 GiB (46%) total zero data
INFO: backup was done incrementally, reused 116.83 GiB (88%)
INFO: transferred 132.00 GiB in 4347 seconds (31.1 MiB/s)
ERROR: Backup of VM 9115 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 10:41:08
INFO: Starting Backup of VM 9117 (qemu)
INFO: Backup started at 2021-03-28 10:41:08
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a:9117/vm-9117-disk-0.raw' 12G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9117/2021-03-28T08:41:08Z'
INFO: started backup task 'dfde0caa-14da-4479-8751-d4bcccb0ada1'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 1% (184.0 MiB of 12.0 GiB) in 3s, read: 61.3 MiB/s, write: 5.3 MiB/s
INFO: 2% (264.0 MiB of 12.0 GiB) in 6s, read: 26.7 MiB/s, write: 0 B/s
INFO: 99% (11.9 GiB of 12.0 GiB) in 7m 20s, read: 32.0 MiB/s, write: 0 B/s
INFO: 100% (12.0 GiB of 12.0 GiB) in 7m 23s, read: 36.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 2.40 GiB (20%) total zero data
INFO: backup was done incrementally, reused 11.59 GiB (96%)
INFO: transferred 12.00 GiB in 443 seconds (27.7 MiB/s)
ERROR: Backup of VM 9117 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 10:48:58
INFO: Starting Backup of VM 9119 (qemu)
INFO: Backup started at 2021-03-28 10:48:58
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:9119/vm-9119-disk-0.raw' 32G
INFO: include disk 'virtio1' 'esnx-a:9119/vm-9119-disk-0.raw' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9119/2021-03-28T08:48:58Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '973688b6-195d-432d-8455-cba3cc350cb5'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (192.0 MiB of 64.0 GiB) in 3s, read: 64.0 MiB/s, write: 18.7 MiB/s
INFO: 1% (656.0 MiB of 64.0 GiB) in 42s, read: 11.9 MiB/s, write: 2.3 MiB/s
INFO: 99% (63.9 GiB of 64.0 GiB) in 11m 1s, read: 394.7 MiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 11m 2s, read: 144.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 47.57 GiB (74%) total zero data
INFO: backup was done incrementally, reused 61.96 GiB (96%)
INFO: transferred 64.00 GiB in 662 seconds (99.0 MiB/s)
ERROR: Backup of VM 9119 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 11:00:21
INFO: Starting Backup of VM 9200 (qemu)
INFO: Backup started at 2021-03-28 11:00:21
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:9200/vm-9200-disk-0.raw' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/9200/2021-03-28T09:00:21Z'
INFO: started backup task '71e6afce-d5f4-4697-b69e-51187c630054'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 2% (672.0 MiB of 32.0 GiB) in 3s, read: 224.0 MiB/s, write: 0 B/s
INFO: 3% (1.2 GiB of 32.0 GiB) in 6s, read: 180.0 MiB/s, write: 2.7 MiB/s
INFO: 98% (31.5 GiB of 32.0 GiB) in 1m 55s, read: 448.0 MiB/s, write: 0 B/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 1m 57s, read: 232.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 29.32 GiB (91%) total zero data
INFO: backup was done incrementally, reused 31.86 GiB (99%)
INFO: transferred 32.00 GiB in 117 seconds (280.1 MiB/s)
INFO: running 'proxmox-backup-client prune' for 'vm/9200'
INFO: Finished Backup of VM 9200 (00:02:02)
INFO: Backup finished at 2021-03-28 11:02:23
INFO: Starting Backup of VM 30000 (qemu)
INFO: Backup started at 2021-03-28 11:02:23
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esns-b:30000/vm-30000-disk-1.qcow2' 32G
INFO: include disk 'virtio1' 'esnx-a:30000/vm-30000-disk-0.qcow2' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/30000/2021-03-28T09:02:23Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '8c063bc6-114e-45a5-b665-dc9c35d8d354'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (96.0 MiB of 64.0 GiB) in 3s, read: 32.0 MiB/s, write: 2.7 MiB/s
INFO: 1% (664.0 MiB of 64.0 GiB) in 25s, read: 25.8 MiB/s, write: 372.4 KiB/s
INFO: 98% (63.0 GiB of 64.0 GiB) in 7m 24s, read: 424.0 MiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 7m 27s, read: 344.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 52.23 GiB (81%) total zero data
INFO: backup was done incrementally, reused 62.71 GiB (97%)
INFO: transferred 64.00 GiB in 447 seconds (146.6 MiB/s)
ERROR: Backup of VM 30000 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 11:10:11
INFO: Starting Backup of VM 90010 (qemu)
INFO: Backup started at 2021-03-28 11:10:11
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a:90010/vm-90010-disk-0.raw' 64G
INFO: include disk 'virtio1' 'esnx-a:90010/vm-90010-disk-1.raw' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'esns-b:90010/vm-90010-disk-0.raw' (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/90010/2021-03-28T09:10:11Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task '72846bcf-0cd5-4eee-b5ca-b74b31b253e2'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: virtio1: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (116.0 MiB of 128.0 GiB) in 3s, read: 38.7 MiB/s, write: 8.0 MiB/s
INFO: 1% (1.3 GiB of 128.0 GiB) in 30s, read: 45.0 MiB/s, write: 2.5 MiB/s
INFO: 99% (126.7 GiB of 128.0 GiB) in 54m 30s, read: 406.7 MiB/s, write: 0 B/s
INFO: 100% (128.0 GiB of 128.0 GiB) in 54m 42s, read: 108.7 MiB/s, write: 0 B/s
INFO: backup is sparse: 65.29 GiB (51%) total zero data
INFO: backup was done incrementally, reused 117.14 GiB (91%)
INFO: transferred 128.00 GiB in 3282 seconds (39.9 MiB/s)
ERROR: Backup of VM 90010 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 12:05:17
INFO: Starting Backup of VM 90011 (qemu)
INFO: Backup started at 2021-03-28 12:05:17
INFO: status = running
INFO: VM Name: ***
INFO: include disk 'virtio0' 'esnx-a:90011/vm-90011-disk-0.raw' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/90011/2021-03-28T10:05:17Z'
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
INFO: started backup task 'e3f890b2-8239-4b4f-a4bb-23dbd845ce9c'
INFO: resuming VM again
INFO: virtio0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 0% (56.0 MiB of 64.0 GiB) in 3s, read: 18.7 MiB/s, write: 2.7 MiB/s
INFO: 1% (664.0 MiB of 64.0 GiB) in 30s, read: 22.5 MiB/s, write: 6.5 MiB/s
INFO: 99% (63.5 GiB of 64.0 GiB) in 35m 5s, read: 254.7 MiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 35m 15s, read: 50.8 MiB/s, write: 0 B/s
INFO: backup is sparse: 18.79 GiB (29%) total zero data
INFO: backup was done incrementally, reused 58.07 GiB (90%)
INFO: transferred 64.00 GiB in 2115 seconds (31.0 MiB/s)
ERROR: Backup of VM 90011 failed - proxmox-backup-client failed: Error: http request timed out
INFO: Failed at 2021-03-28 12:40:56
INFO: Backup job finished with errors
TASK ERROR: job errors
Tobias
Hallo Tobias,
entschuldige das ich deutsch antworte. Mein Englisch ist leider nicht gut.
Ich habe die gleichen Probleme wie Du seit Monaten, mal mehr mal weniger. Ich versuche immer auf dem aktuellen Stand von PVE und PBS zu sein. Siehe Dir meine Beiträge im Forum PBS und PVE an. Wenn es hier einen nachhaltigen funktionieren Lösungsvorschlag gibt, immer her damit. Deshalb sichere ich hauptsächlich mit PBS nur noch am Wochenende und unter meiner Aufsicht und in der Woche mit Veeam innerhalb der VM's um Probleme mit hängenden VM's rechtzeitig erkennen zu können.
Viele Grüße
crmspezi
entschuldige das ich deutsch antworte. Mein Englisch ist leider nicht gut.
Ich habe die gleichen Probleme wie Du seit Monaten, mal mehr mal weniger. Ich versuche immer auf dem aktuellen Stand von PVE und PBS zu sein. Siehe Dir meine Beiträge im Forum PBS und PVE an. Wenn es hier einen nachhaltigen funktionieren Lösungsvorschlag gibt, immer her damit. Deshalb sichere ich hauptsächlich mit PBS nur noch am Wochenende und unter meiner Aufsicht und in der Woche mit Veeam innerhalb der VM's um Probleme mit hängenden VM's rechtzeitig erkennen zu können.
Viele Grüße
crmspezi
Bei mir hat es geholfen das Verify der einzelnen VMs direkt nach dem Backup zu deaktivieren und stattdessen einen Verify Task einmal täglich laufen zu lassen. Seit dem ist der http request Timeout bei mir weg.
Obwohl mein PBS schon 128 GB RAM, 4 x 10 GBit, SAS Platten und SSD Cache hat war er einfach zu langsam beim abarbeiten der Menge an VMs die gesichert werden.
Im übrigen finde ich das ok das du in deutsch antwortest, das ist hier ja auch das deutsche Forum.
Danke für die Antwort.
Mein PBS hat nur 2x 10G, derzeit 32GB RAM, SSD Cache und 120TB im Raidz2, allerdings gibt es auch einen Second-PBS. Gesichert sollten 50VM's jede Nacht werden. Leider bisher keine Chance auf Zuverlässigkeit, ohne Stillstand irgendwann innerhalb der Woche. Ich hatte schon bei Acronis und Veeam wegen KVM angefragt. Die interessieren sich allerdings derzeit nicht dafür. Deshalb auch die Veeams innerhalb der VM's.
Ich will es sicher nicht umsonst, aber zuverlässig! Das ist leider seit der Beta nicht gegeben. Und das ein PBS Hardwareanforderungen für SSD Storage stellt und mir wirklich unerklärbar! Das ist so als ob ich zum Baumarkt mit dem Aston Martin fahren soll um Kabelkanäle zu holen!
Ich teste aber gern Deinen Vorschlag morgen. Danke für den Hinweis.
Wo stelle ich denn das Verify ab?
VG crmspezi
Last edited:
Auf dem Datastore unter Optionen den Punkt "neue Snapshots verifizieren" auf nein setzen und stattdessen einen Verifizierungsjob einrichten der täglich läuft und nur die neuen Snapshots verifiziert.
Wenn ich mir allerdings deine Konfiguration anschaue glaube ich das du etwas wenig RAM in deinem Server hast für deine Plattenkapazität.
Bei ZFS gilt, 4 GB für das System und 1 GB pro TB Plattenplatz. Das würde bei dir mindestens 128 GB RAM machen.
Ich habe nur 70 TB im Raidz1 und mein Server genehmigt sich bis zu 85% RAM von seinen 128 GB wenn er richtig unter Last ist.
Gut die Verschlüsselung auf den Datastores dürfte da auch eine Menge zu beitragen aber 32 GB sind bei 120 TB einfach viel zu wenig für ein performantes System.
Wenn ich mir allerdings deine Konfiguration anschaue glaube ich das du etwas wenig RAM in deinem Server hast für deine Plattenkapazität.
Bei ZFS gilt, 4 GB für das System und 1 GB pro TB Plattenplatz. Das würde bei dir mindestens 128 GB RAM machen.
Ich habe nur 70 TB im Raidz1 und mein Server genehmigt sich bis zu 85% RAM von seinen 128 GB wenn er richtig unter Last ist.
Gut die Verschlüsselung auf den Datastores dürfte da auch eine Menge zu beitragen aber 32 GB sind bei 120 TB einfach viel zu wenig für ein performantes System.
Vielen Dank für die Hilfe!.
Die 32GBb sind nicht so viel. Ich hatte auch schon 64GB drin, das hatte keine nennenswerten Performance Verbesserungen beim PBS gebracht. Die ZFS Regel kenne ich. Allerdings kann ich in die Freigaben (der PBS ist eine Openmedivault-NAS mit pbs Aufsatz und ZFS Datastore) locker mit über 400MByte/s reinspeichern (per Veeam). Den ZFS Speicher habe ich auf 24GB begrenzt (bei 64GB auf 56GB). Meinst Du der RAM wird beim Backup (Schreiben) bei async auch so benutzt? Sollte hier der RAM hier hauptsächlich nur beim Lesen eine Rollen spielen?
Ich probiere Deinen Ratschlag gleich heute Abend aus. Danke sehr!
Habe eben nachgeschaut, "neue Snapshots verifizieren" war bei mir schon aus.
Last edited:
Hi,
Das habe ich ebenfalls ausgeschaltet - keine wesentliche Veränderung.
Leider.
Gruss
Tobias
Hallo,
Eben... leider überhaupt nix merkwürziges auf dem PBS....
Ich hatte das Problem bereits auf der virtuellen PBS mal - danach habe ich mir nen physischen server installiert...
Gruss
Tobias
Eben... leider überhaupt nix merkwürziges auf dem PBS....
Ich hatte das Problem bereits auf der virtuellen PBS mal - danach habe ich mir nen physischen server installiert...
Gruss
Tobias
aber das backup wird auf PBS seite fertig angelegt (nur ohne client log)? kannst du mal bei einem backup mittels "atop" auf beiden seiten "zuschauen", ob beim finalisieren des backups die last nach oben schnellt? eventuell macht es sinn das backup haendisch in der shell anzustossen und in 'ts' (paket: moreutils) zu pipen (also
vzdump .... | ts), dann ist der output gleich mit timestamps versehen zur leichteren korrelierung.