[SOLVED] Backup dauert sehr lange. Optimierung möglich?

[showiproute]

Code contributions sind immer willkommen, jedoch sollten Implementationsdetails eventuell vorher diskutiert werden um allen beteiligten Zeit zu sparen. Aber ich sehe das issue wird ja bereits aktiv diskutiert.

Ich hatte mal vor ~8 Jahren Berührungen mit Java SE & EE - vermute aber, dass das recht wenig hilft. Jedoch unterstütze ich gerne beim Testen

 

[gno]

Ein Auslagern des LXC mountpoints mit Daten auf eine schnelle storage würde das Backup vermutlich beschleunigen, da vernünftige NVMe SSDs bessere IOPS und Datenraten bieten.

Ich habe die Daten des Fileservers und der Nextcloud jetzt auf einen Mirror aus 2 Samsung 980 Pro NVMe gelegt.
Die Backup Platte ist immer noch die WD Red Plus 4TB SATA HDD.
Außerdem wurden auch Prozessor und Board getauscht.

Die Dauer des gleichen Backups wie im Startpost ist von ca. 4,5 Stunden auf ca. 14 Minuten verringert worden.

Die IO-Verzögerung ist von ca. 50% auf maximal 6% zurückgegangen.

103: 2023-11-21 02:00:08 INFO: Starting Backup of VM 103 (lxc)
103: 2023-11-21 02:00:08 INFO: status = running
103: 2023-11-21 02:00:08 INFO: CT Name: srv
103: 2023-11-21 02:00:08 INFO: including mount point rootfs ('/') in backup
103: 2023-11-21 02:00:08 INFO: including mount point mp0 ('/mnt/srv_data0') in backup
103: 2023-11-21 02:00:08 INFO: backup mode: snapshot
103: 2023-11-21 02:00:08 INFO: ionice priority: 7
103: 2023-11-21 02:00:08 INFO: suspend vm to make snapshot
103: 2023-11-21 02:00:08 INFO: create storage snapshot 'vzdump'
103: 2023-11-21 02:00:08 INFO: resume vm
103: 2023-11-21 02:00:08 INFO: guest is online again after <1 seconds
103: 2023-11-21 02:00:08 INFO: creating Proxmox Backup Server archive 'ct/103/2023-11-21T01:00:08Z'
103: 2023-11-21 02:00:08 INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3640719_103/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./mnt/srv_data0 --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 103 --backup-time 1700528408 --repository root@pam@192.168.1.147:store1
103: 2023-11-21 02:00:09 INFO: Starting backup: ct/103/2023-11-21T01:00:08Z
103: 2023-11-21 02:00:09 INFO: Client name: pve2
103: 2023-11-21 02:00:09 INFO: Starting backup protocol: Tue Nov 21 02:00:09 2023
103: 2023-11-21 02:00:09 INFO: Downloading previous manifest (Sat Nov 18 02:00:19 2023)
103: 2023-11-21 02:00:09 INFO: Upload config file '/var/tmp/vzdumptmp3640719_103/etc/vzdump/pct.conf' to 'root@pam@192.168.1.147:8007:store1' as pct.conf.blob
103: 2023-11-21 02:00:09 INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@192.168.1.147:8007:store1' as root.pxar.didx
103: 2023-11-21 02:08:16 INFO: root.pxar: had to backup 450.946 MiB of 403.174 GiB (compressed 246.191 MiB) in 487.76s
103: 2023-11-21 02:08:16 INFO: root.pxar: average backup speed: 946.713 KiB/s
103: 2023-11-21 02:08:16 INFO: root.pxar: backup was done incrementally, reused 402.734 GiB (99.9%)
103: 2023-11-21 02:08:16 INFO: Uploaded backup catalog (6.235 MiB)
103: 2023-11-21 02:08:17 INFO: Duration: 488.29s
103: 2023-11-21 02:08:17 INFO: End Time: Tue Nov 21 02:08:17 2023
103: 2023-11-21 02:08:17 INFO: adding notes to backup
103: 2023-11-21 02:08:17 INFO: cleanup temporary 'vzdump' snapshot
103: 2023-11-21 02:08:17 INFO: Finished Backup of VM 103 (00:08:09)

105: 2023-11-21 02:08:17 INFO: Starting Backup of VM 105 (lxc)
105: 2023-11-21 02:08:17 INFO: status = running
105: 2023-11-21 02:08:17 INFO: CT Name: web01
105: 2023-11-21 02:08:17 INFO: including mount point rootfs ('/') in backup
105: 2023-11-21 02:08:17 INFO: including mount point mp0 ('/mnt/nextcloud_data') in backup
105: 2023-11-21 02:08:17 INFO: backup mode: snapshot
105: 2023-11-21 02:08:17 INFO: ionice priority: 7
105: 2023-11-21 02:08:17 INFO: suspend vm to make snapshot
105: 2023-11-21 02:08:17 INFO: create storage snapshot 'vzdump'
105: 2023-11-21 02:08:17 INFO: resume vm
105: 2023-11-21 02:08:17 INFO: guest is online again after <1 seconds
105: 2023-11-21 02:08:17 INFO: creating Proxmox Backup Server archive 'ct/105/2023-11-21T01:08:17Z'
105: 2023-11-21 02:08:17 INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3640719_105/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./mnt/nextcloud_data --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 105 --backup-time 1700528897 --repository root@pam@192.168.1.147:store1
105: 2023-11-21 02:08:17 INFO: Starting backup: ct/105/2023-11-21T01:08:17Z
105: 2023-11-21 02:08:17 INFO: Client name: pve2
105: 2023-11-21 02:08:17 INFO: Starting backup protocol: Tue Nov 21 02:08:17 2023
105: 2023-11-21 02:08:18 INFO: Downloading previous manifest (Sat Nov 18 04:16:20 2023)
105: 2023-11-21 02:08:18 INFO: Upload config file '/var/tmp/vzdumptmp3640719_105/etc/vzdump/pct.conf' to 'root@pam@192.168.1.147:8007:store1' as pct.conf.blob
105: 2023-11-21 02:08:18 INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@192.168.1.147:8007:store1' as root.pxar.didx
105: 2023-11-21 02:14:02 INFO: root.pxar: had to backup 1.102 GiB of 270.885 GiB (compressed 817.24 MiB) in 344.68s
105: 2023-11-21 02:14:02 INFO: root.pxar: average backup speed: 3.274 MiB/s
105: 2023-11-21 02:14:02 INFO: root.pxar: backup was done incrementally, reused 269.783 GiB (99.6%)
105: 2023-11-21 02:14:02 INFO: Uploaded backup catalog (7.679 MiB)
105: 2023-11-21 02:14:03 INFO: Duration: 345.78s
105: 2023-11-21 02:14:03 INFO: End Time: Tue Nov 21 02:14:03 2023
105: 2023-11-21 02:14:03 INFO: adding notes to backup
105: 2023-11-21 02:14:04 INFO: cleanup temporary 'vzdump' snapshot
105: 2023-11-21 02:14:04 INFO: Finished Backup of VM 105 (00:05:47)