List backups slow

R

rootnetworks

New Member
Feb 13, 2022
24
0
1
32
Hi guys,

I have add a NFS storage for backup but in GUI to list backup files very slow and sometimes got error 'Connection timed out (596)', But I also check in shell console to list nfs directory is very fast.

Screenshot at May 18 21-10-30.png
Screenshot at May 18 21-10-47.png
 
rootnetworks said:
Does running backup task will lock to list backup files?
Click to expand...
No, the backup lock is only a local one and it is unrelated to the storage.

rootnetworks said:
I have add a NFS storage for backup but in GUI to list backup files very slow and sometimes got error 'Connection timed out (596)',
Click to expand...
Can you also check the browser developer console, e.g., if there are any errors and how the storage GET request looks like, or if there's any more error details in the response there.

Also, is this a cluster? If so, do you see a difference if you open the storage entry of the node you're connected through the web interface vs. the one of some other cluster node?
 
t.lamprecht said:
No, the backup lock is only a local one and it is unrelated to the storage.


Can you also check the browser developer console, e.g., if there are any errors and how the storage GET request looks like, or if there's any more error details in the response there.

Also, is this a cluster? If so, do you see a difference if you open the storage entry of the node you're connected through the web interface vs. the one of some other cluster node?
Click to expand...
Hi Lamprecht,

List backup files get path is
Code: 
/api2/json/nodes/V-NODE-HND-1/storage/CK1/content?content=backup

I tried accessing this URL directly got the same problem
Code: 
Request URL: https://x.x.x.x:8006/api2/json/nodes/V-NODE-HND-1/storage/CK1/content?content=backup
Request Method: GET
Status Code: 596 Connection timed out
Remote Address: x.x.x.x:8006
Referrer Policy: strict-origin-when-cross-origin

If the request succeeds once, the next few seconds will succeed and the response will be very fast.


Also, is this a cluster? If so, do you see a difference if you open the storage entry of the node you're connected through the web interface vs. the one of some other cluster node?
Yes it's a cluster

Code: 
~#ansible hndpve -m shell -a 'cat /etc/pve/storage.cfg'
x.x.x3.3 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.6 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.2 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.4 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.5 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.7 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.8 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.9 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1
x.x.x3.10 | CHANGED | rc=0 >>
dir: local
    path /var/lib/vz
    content iso,snippets,vztmpl,rootdir,images
    prune-backups keep-all=1
    shared 0

nfs: CK1
    export /home/nfs
    path /mnt/pve/CK1
    server x.x.2.8
    content backup
    nodes V-NODE-HND-7,V-NODE-HND-2,V-NODE-HND-6,V-NODE-HND-5,V-NODE-HND-1,V-NODE-HND-3,V-NODE-HND-4,V-NODE-HND-8,V-NODE-HND-9
    options vers=4
    prune-backups keep-all=1

~#ansible hndpve -m shell -a 'pvesm status'
x.x.x3.4 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        36460480      7027552828    0.49%
x.x.x3.5 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        27795820      7036217488    0.37%
x.x.x3.3 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        38078696      7025934612    0.51%
x.x.x3.2 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480       694149196      6369864112    9.33%
x.x.x3.6 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        36418252      7027595056    0.49%
x.x.x3.7 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        33450376      7030562932    0.45%
x.x.x3.10 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        32368456      7031644852    0.44%
x.x.x3.9 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        36401880      7027611428    0.49%
x.x.x3.8 | CHANGED | rc=0 >>
Name         Type     Status           Total            Used       Available        %
CK1           nfs     active     52520721408       327603200     49556519936    0.62%
local         dir     active      7438992480        27999920      7036013388    0.38%


~# pvecm status
Cluster information
-------------------
Name:             v-cluster-hnd1
Config Version:   9
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu May 19 17:10:04 2022
Quorum provider:  corosync_votequorum
Nodes:            9
Node ID:          0x00000009
Ring ID:          1.145e
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   9
Highest expected: 9
Total votes:      9
Quorum:           5 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 x.x.x3.3
0x00000002          1 x.x.x3.2
0x00000003          1 x.x.x3.4
0x00000004          1 x.x.x3.5
0x00000005          1 x.x.x3.6
0x00000006          1 x.x.x3.7
0x00000007          1 x.x.x3.8
0x00000008          1 x.x.x3.9
0x00000009          1 x.x.x3.10 (local)





~#ansible hndpve -m shell -a 'ls /mnt/pve/CK1/dump/ | wc -l'
x.x.x3.3 | CHANGED | rc=0 >>
402
x.x.x3.2 | CHANGED | rc=0 >>
402
x.x.x3.6 | CHANGED | rc=0 >>
402
x.x.x3.5 | CHANGED | rc=0 >>
402
x.x.x3.4 | CHANGED | rc=0 >>
402
x.x.x3.7 | CHANGED | rc=0 >>
402
x.x.x3.10 | CHANGED | rc=0 >>
402
x.x.x3.8 | CHANGED | rc=0 >>
402
x.x.x3.9 | CHANGED | rc=0 >>
402
 
There have Hundreds of backup files when I list these files in console very fast by command
Code: 
ls  /mnt/pve/CK1/dump/
. but when I use
Code: 
du /mnt/pve/CK1/dump/*
command it's very slow

Code: 
startTime=`date +%Y%m%d-%H:%M:%S`
startTime_s=`date +%s`

ls /mnt/pve/CK1/dump/

endTime=`date +%Y%m%d-%H:%M:%S`
endTime_s=`date +%s`
sumTime=$[ $endTime_s - $startTime_s ]

echo "$startTime ---> $endTime" "Total:$sumTime seconds"
Total:0 seconds

Code: 
startTime=`date +%Y%m%d-%H:%M:%S`
startTime_s=`date +%s`

du /mnt/pve/CK1/dump/*

endTime=`date +%Y%m%d-%H:%M:%S`
endTime_s=`date +%s`
sumTime=$[ $endTime_s - $startTime_s ]
echo "$startTime ---> $endTime" "Total:$sumTime seconds"
Total:45 seconds

Does PVE to list backup files do something like du command due take a long time to get the size of the file then pveproxy time out?
 
I see this problem due connect to a remote NFS site. in my environment from PVE server to nfs server as 100ms latency.
so too hard to list a lot of file attributes such as du or ls -l command.
as I know NFS will cache files attributes but The default time is very short.

When I increase cache time to 3600s and working fine
actimeo=3600


Code: 
nfs: CK1
        export /home/nfs
        path /mnt/pve/CK1
        server x.x.2.8
        content backup
        nodes V-NODE-HND-5,V-NODE-HND-6,V-NODE-HND-2,V-NODE-HND-7,V-NODE-HND-3,V-NODE-HND-1,V-NODE-HND-4,V-NODE-HND-9,V-NODE-HND-8
        options actimeo=3600
        prune-backups keep-all=1

actimeoAbsolute time for which file and directory entries are kept in the file-attribute cache after an update. If specified, this value overrides the following *min and *max values, effectively setting them all to the actimeo value.acregminMinimum time after an update that file entries will be retained. The default is 3 seconds.acregmaxMaximum time after an update that file entries will be retained. The default is 60 seconds.acdirminMinimum time after an update that directory entries will be retained. The default is 30 seconds.acdirmaxMaximum time after an update that directory entries will be retained. The default is 60 seconds.
 
another word
When getting a backup of a specific VM on the GUI, actually the API gets all the backup files.
in GUI request
Code: 
https://x.x.x.x:8006/api2/json/nodes/V-NODE-HND-1/storage/CK1/content?content=backup

The correct one should be
Code: 
https://x.x.x.x:8006/api2/json/nodes/V-NODE-HND-1/storage/CK1/content?content=backup&vmid=10001

But I think the backend may also always list all files even request api2/json/nodes/V-NODE-HND-1/storage/CK1/content?content=backup&vmid=10001 just filter in response

In my environment I see ls -l is 50 times slower than the ls command.
 
Last edited:
Jumping in, we do have a PBS instance, rather powerful and well connected, yet the full listing takes 2-5s, while filtering by VMID is a lot quicker. The full list sometimes timeout.. there might be an improvement path there ;)
 
You must log in or register to reply here.
Top Bottom