Search results

No longer have the test servers.

Apologies, was not meant as spam, there are forum users and track users, not everyone looks at the issue tracker.

https://bugzilla.proxmox.com/show_bug.cgi?id=4988 Is proxmox patched or vulnerable to the attack. See : CVE-2023-44487 https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/

https://bugzilla.proxmox.com/show_bug.cgi?id=4989

Conclusion : the problem is caused by latency, and how PBS uses TLS along with the small chunk size. Possible solutions: PBS to use Quic/http3 increasing the chunksize 10x Multi-threaded restore (user defined threads) Here is an actual real world comparison between 2 identical pbs backup...

Backups are run at around the 600Mbyte/s mark, restoring those backups is the issue.

@dcsapak @fabian any help/advice would be appreciated, is there a way we can increase the chunk size and/or to disable TLS to test ?

TLDR: Extremely slow restore speed between 2 dedicated PBS and PVE servers ( all NVME based + 10GBit + 16cores + 128GB ram ) restore of 1.2TB runs at 60Mbyte/s, where does the problem lie ? My thoughs are the TLS and small chunk size, has a high overhead which is limiting the restore speed...

its an error with the terminal / shell not having the correct locales. Below is the fix LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8"...

Unfortunately until now, proxmox has not allowed those using "atime" to purge the garbage quicker than those using "relatime"

Currently how does proxmox work around the usecase of an admin randomly enabling/disabling atime and relatime ? Or setting all the file times to -1200 days ? ;)

atime doesnt become set and uset at random, its an admin action. So if you check if atime is enabled and wait 25hours, and then atime is still enabled do the 1hour GC Every hour when GC is run, it can recheck if atime is enabled or not. If atime is no longer enabled, revert back to gc 25hours.

first GC run with atime enabled option, could wait the full 25hours for gc. The subsequent runs would be the 1hour gc

The suggestion was a hidden / expert option in the cfg, or a command on the command line. im taking pure nvme backup storage, where atime has a negligible impact. Having 20TB on nvme sitting idle and waiting for 25hours to be removed is expensive. It would be a simple check to see if atime...

^^we are all aware of that fact. This thread is about having a workable and reliable solution to clearing the garbage earlier. Large users (20+proxmox nodes, 100+TB of backups, quickly run into limitations with the current 25hour GC)

or an option to lower the cutoff time, especially with backups which are done every 1 hour

Licensing by core count would be more fair. 4x8core cpus vs 2x16core vs 1x32core cpu should have the same license cost. An example would be replace the socket count metric with a cpu core count metric, in multiples of 32cores.(or a lower/higher number) My point is a 128core single cpu server...

+1

hidden flag/setting in the datastore.cfg is the ideal solution. For perspective, one of the pbs servers: 2022-08-18T13:22:42+02:00: Removed garbage: 34.908 GiB 2022-08-18T13:22:42+02:00: Removed chunks: 27588 2022-08-18T13:22:42+02:00: Pending removals: 15.045 TiB (in 8600742 chunks)...

Unfortunately Snapshots are not remote and they prevent migrations between nodes. (local storage) I do think a good common ground would be to have an option to set a lower value than 24hours. (6hours, 12hours, etc) If atime is broken, surely relatime will also be broken ? From my estimation In...