Connection failure (0)

Rootinator

Member
May 1, 2019
23
0
21
54
Hello all,

I have a PBS VM. I have mounted the PBS as PBS storage in PVE.

Unfortunately I can't access anything on my backups "connection failure (0)".

But I can see the PBS load.

How can I fix this? :)
 

Attachments

  • Auslastung.PNG
    Auslastung.PNG
    371.4 KB · Views: 13
  • CommunicationFehler.PNG
    CommunicationFehler.PNG
    351.3 KB · Views: 13
  • PBS_Storage.PNG
    PBS_Storage.PNG
    419.5 KB · Views: 13
Sorry for my late reply!

Unfortunately, it loads with it infinitely long.


Code:
root@dedi1 ~ # proxmox-backup-client snapshot list --repository user@pbs@10.0.0.5:StorageBox0SMB
Password for "user@pbs": **************************
fingerprint: **:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**
Are you sure you want to continue connecting? (y/n): y
fingerprint: **:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**:**
Are you sure you want to continue connecting? (y/n): y
Error: http request timed out


The storage was mounted via SMB. However, a few weeks ago it worked, only now suddenly no longer.
 
seems that something in the network is not working correctly..
is there anything in the network that might interfere with the connection? firewall/proxy/etc ?
 
Firewall I use, yes. I disabled it completely for tests. But nothing happened

In the meantime, I was able to mount the storage successfully in PBS. I unmounted it and set the owner "backup:backup" and remounted it. I can now see and start the backups.

However, after a few % the backup aborts completely (about 20%) :confused:
 

Attachments

  • Unbenannt.PNG
    Unbenannt.PNG
    15 KB · Views: 5
  • log_backupjob.txt
    18.2 KB · Views: 3
since your log still shows lines with 'connection reset' etc. i still suspect something in the network
does a ping work reliably? can you run an iperf test between the nodes? any special in terms of network setup?
 
Ping works in both directions.

After a long time of trying I managed to create backups with the PBS again.

What have I done? I unhooked the storage, changed the owner to "backup:backup" and mounted the storage again.

That was it!

But now I still have a transfer speed problem. I mounted the storage via SMB. It is really slow (5 MiB/s). WebDAV is even slower.

I have a storage box at the red "H" from Germany. Here I have only a limited number of protocols available.

But even my storage at home is not faster.

I wanted to mount the storage via SSHFS, but then it is not recognized in PBS?

How can I increase the transfer speed? I have to backup almost 2 TiB, so it takes many hours.

Some time ago I backed up my backups to another server without PBS via SSHFS, there the backup was done within a few minutes. With PBS via SMB it takes hours...
 
How can I increase the transfer speed? I have to backup almost 2 TiB, so it takes many hours.
what is the transfer speed if you measure with e.g. iperf? also how fast are the storages (source and target) ?
we don't arbitrarily limit the throughput, so something (cpu/disk/network) should be the bottleneck
 
Here is an iperf3 speed test from PVE to PBS:

Code:
# iperf3 -c 10.0.0.5
Connecting to host 10.0.0.5, port 5201
[  5] local 10.0.0.1 port 48964 connected to 10.0.0.5 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.24 GBytes  36.4 Gbits/sec    0    713 KBytes
[  5]   1.00-2.00   sec  4.02 GBytes  34.6 Gbits/sec    0   1.22 MBytes
[  5]   2.00-3.00   sec  4.22 GBytes  36.3 Gbits/sec    0   1.40 MBytes
[  5]   3.00-4.00   sec  4.29 GBytes  36.8 Gbits/sec    0   1.52 MBytes
[  5]   4.00-5.00   sec  4.27 GBytes  36.7 Gbits/sec    0   1.52 MBytes
[  5]   5.00-6.00   sec  4.28 GBytes  36.8 Gbits/sec    0   1.65 MBytes
[  5]   6.00-7.00   sec  4.14 GBytes  35.5 Gbits/sec    0   1.73 MBytes
[  5]   7.00-8.00   sec  4.24 GBytes  36.4 Gbits/sec    0   1.73 MBytes
[  5]   8.00-9.00   sec  4.10 GBytes  35.2 Gbits/sec    0   1.91 MBytes
[  5]   9.00-10.00  sec  4.28 GBytes  36.8 Gbits/sec    0   1.91 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  42.1 GBytes  36.1 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  42.1 GBytes  36.1 Gbits/sec                  receiver

iperf Done.

And here is a hard disk speed test for external storage:

Code:
# dd if=/dev/zero of=/home/storagebox0/speedtest bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.3033 s, 58.7 MB/s

At least according to "dd" I have a transfer speed of just under 470 MBit
 
And here is a hard disk speed test for external storage:

Code:
# dd if=/dev/zero of=/home/storagebox0/speedtest bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.3033 s, 58.7 MB/s
dd is not a benchmark, especially not for the workload of pbs (normally writes/reads 2-4MiB chunks in random places)

see https://pve.proxmox.com/wiki/Benchmarking_Storage for some hints how you can benchmark your storages
also the read speed of the source and the write speed of the target matters
 
I have now performed a benchmark via "bonnie++".

The results from Bonnie++ are shown as a screenshot.
When writing, the PBS gets about 30 MB/s (~29 MiB/s).

However, backups are only written with ~5-8MiB/s.

Short excerpt from a backup from PVE to PBS:

Code:
[...]
INFO:  91% (8.2 GiB of 9.0 GiB) in 16m 14s, read: 9.2 MiB/s, write: 6.8 MiB/s
INFO:  92% (8.3 GiB of 9.0 GiB) in 16m 27s, read: 7.1 MiB/s, write: 4.6 MiB/s
INFO:  93% (8.4 GiB of 9.0 GiB) in 16m 38s, read: 8.4 MiB/s, write: 6.9 MiB/s
INFO:  94% (8.4 GiB of 9.0 GiB) in 16m 49s, read: 8.4 MiB/s, write: 6.5 MiB/s
INFO:  95% (8.5 GiB of 9.0 GiB) in 17m, read: 8.4 MiB/s, write: 6.5 MiB/s
INFO:  96% (8.6 GiB of 9.0 GiB) in 17m 14s, read: 6.6 MiB/s, write: 5.7 MiB/s
INFO:  97% (8.7 GiB of 9.0 GiB) in 17m 22s, read: 11.5 MiB/s, write: 8.5 MiB/s
INFO:  98% (8.8 GiB of 9.0 GiB) in 17m 33s, read: 8.7 MiB/s, write: 5.5 MiB/s
INFO:  99% (8.9 GiB of 9.0 GiB) in 17m 43s, read: 8.8 MiB/s, write: 4.8 MiB/s
INFO: 100% (9.0 GiB of 9.0 GiB) in 17m 55s, read: 7.7 MiB/s, write: 4.7 MiB/s
INFO: backup was done incrementally, reused 93.44 GiB (93%)
INFO: transferred 8.98 GiB in 1122 seconds (8.2 MiB/s)
INFO: Finished Backup of VM 105 (00:18:51)
 

Attachments

  • Unbenannt.PNG
    Unbenannt.PNG
    79.6 KB · Views: 2
Enclosed is the benchmark PVE to PBS:

Code:
# proxmox-backup-client benchmark --repository user@pbs@10.0.0.5:StorageBox0
Password for "user@pbs": ********
Uploaded 696 chunks in 5 seconds.
Time per request: 7218 microseconds.
TLS speed: 581.03 MB/s
SHA256 speed: 403.21 MB/s
Compression speed: 551.04 MB/s
Decompress speed: 892.07 MB/s
AES256/GCM speed: 1941.42 MB/s
Verify speed: 268.11 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 581.03 MB/s (47%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 403.21 MB/s (20%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 551.04 MB/s (73%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 892.07 MB/s (74%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 268.11 MB/s (35%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1941.42 MB/s (53%) │
└───────────────────────────────────┴────────────────────┘
 
ok the speed to the pbs looks reasonably fast. so i would conclude that your source storage is not fast enough to keep up..
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!