Problem with (very) slow restore for CT

[hakim]

Hello and happy new year,

I just installed 2 servers with the last version of proxmox (last image + updates).

On these 2 servers, I am able to restore some CTs but for (at least) 2 other CTs :
- the restore process starts,
- the corresponding disks is filled quite quickly
- and suddenly, it gets very slow (20 Mb / 1 min),
and I cannot figure out why.

If I try to restore those 2 CTs on the proxmox server where they have been generated (v6.4-15), they are restored in 3 minutes, and if I had to wait to restore them on my new server, it would take something like 10 hours !

The backup source is on the same machine (on HD drives) so not a network issue.
The size of the CT is around 7 Go and the backup file around 4 Go.
During the restore process, the first 4,4Go are copied in about 1 minute, and then it start stucking always around 4,4 Go with about (20 Mb / 1 min) !

The 2 servers use ZFS as file system and have :
- 2 SSD (in raid 1)
- 2 HD (in raid 1)

The transfer of large data (many To) with "zfs send" was fast anb OK.

Normally, my CTs are stored on the SSD, but I have also tried to store them on the HDs, with the same problem.

When the restore is stucked, looking at Top shows the process "tar" at 100%

My backup use ZSTD as compression, and I also tried to backup without compression, and same problem.

For the faster server, I have :

root@prismox1:~# pveperf /tmp
CPU BOGOMIPS:      192005.44
REGEX/SECOND:      4 245 656
HD SIZE:           860.50 GB (rpool/ROOT/pve-1)
FSYNCS/SECOND:     15058.50
DNS EXT:           8.37 ms
DNS INT:           1.75 ms (iandi.fr)

Nothing special in journalctl

Here what I get in the GUI console when I try to restore :
Unable to restore CT 1095 - command 'set -o pipefail && cstream -t 0 | tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/1095/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: received interrupt

recovering backed-up configuration from 'backup:backup/vzdump-lxc-1095-2023_01_01-18_45_08.tar.zst'
restoring 'backup:backup/vzdump-lxc-1095-2023_01_01-18_45_08.tar.zst' now..
extracting archive '/backup/proxmox/dump/vzdump-lxc-1095-2023_01_01-18_45_08.tar.zst'
TASK ERROR: unable to restore CT 1095 - command 'set -o pipefail && cstream -t 0 | tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/1095/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: received interrupt

The "TASK ERROR" happen when I stop the process which take too long.

Any help, will be greatly appreciated,

Hakim

 

[hakim]

After further tests, I found the problem : in my VM I had a large cache with many little files.
By emptying the cache before the backup, I was then able to restore it.

But, what I do not understand is why I could restore this VM on my previous proxmox server (v6.4) and cannot restore it on my new server with Proxmox 7.3, and which is "stronger" (better hardware config) ?

 

[Christoph Weiler]

After further tests, I found the problem : in my VM I had a large cache with many little files.
By emptying the cache before the backup, I was then able to restore it.

But, what I do not understand is why I could restore this VM on my previous proxmox server (v6.4) and cannot restore it on my new server with Proxmox 7.3, and which is "stronger" (better hardware config) ?

What do you mean with "my VM I had a large cache"?
How do you emptying the cache?

 

[hakim]

>> What do you mean with "my VM I had a large cache"?
A PHP / Symfony and Git cache

>> How do you emptying the cache?
By deleting the files in these cache