Slow upload using nextcloud with proxmox (VM and LXC!) but download and SCP is fast!

[unholyhumorousunratedoat]

Hi guys,

I'm still pretty new to proxmox but im really happy I'm coming from a bare metal arch linux installation on my Dell PowerEdge T20 (for my nextcloud instance) - but now I'm having the following in use as proxmox hypervisor:

CPU(s) 12 x AMD Ryzen 5 1600 Six-Core Processor (1 Socket)
Kernel Version Linux 5.0.21-2-pve #1 SMP PVE 5.0.21-6 (Fri, 27 Sep 2019 17:17:02 +0200)
PVE Manager Version pve-manager/6.0-7/28984024
2x 8GB 3000Mhz DDR4 RAM (non-ECC)

In that system, there are 4 Disks:
1x 128GB NVMe M.2 SSD (proxmox installation on it, ZFS filesystem)
1x 120GB SATA Samsung SSD 850
1x 1TB Toshiba HDD
1x 3TB WD HDD

Communication to the server is over 1Gbit network interface.
For these tests I created a new directory "nc-test" (containing the 120GB Samsung SSD) and gave it the to nc-test container as root partition. Nothing else is using that.


I'm currently setting up my new server for the nextcloud. For testing purposes, I set up (meanwhile multiple) Debian LXC and installed nextcloud via SNAP package on it.
But bizarrely the upload to the nextcloud server is pretty slow.

Upload via SCP to the LXC is fast

Download from the nextcloud instance on the LXC is also pretty fast


But I just dont know why the upload is so slow.

I also set up for testing a VM (not LXC) and here is the Upload way faster - but still not as fast as it was with my bare metal nextcloud installation. There I had Upload 120MiB (like Download).




I don't understand that behavior. RAM Usage of the node is also while up- or downloading at 4-5GB of 16. (Using the VM 9-10GB)

Thank you in advance for your ideas

 

[Alwin]

On which disk is your nextcloud storage lying? And when you are doing the scp, use the same destination folder as nextcloud would do for comparison.

 

[unholyhumorousunratedoat]

Thank you for your answer!

That's a good point. The nextcloud data directory (in a snap installation) is here: /var/snap/nextcloud/common/nextcloud/data



LXC usage is also pretty ordinary while upload.


Here is an overview about the storage on the node:

The proxmox installation is on /dev/nvme0n1
The nextcloud installation is on /dev/sdb. I created a directory for this. And only this VM is using the Samsung SSD 850 EVO disk.



As I apprehended, uploading into the "data directory folder" via SCP is as fast as usual.

 

[Alwin]

This seems to me, more the overhead of the HTTP protocol on the upload through nextcloud. Also to note, that the container sure doesn't have the same resources allocated as the physical machine.

 

[unholyhumorousunratedoat]

Thank you for your answer!
Do you have any idea, how I cloud analyze this a little bit more? Maybe you can say suggest some tools or a strategy how you would try to solve that problem?

 

[ca_maer]

I have the feeling this is something related to PHP settings in your nextcloud
https://docs.nextcloud.com/server/s...pload_configuration.html?highlight=large file

 

[unholyhumorousunratedoat]

Thank you for that suggestion, but I can upload big files - it's just not as fast as the download.

Since I'm using the snap version, I followed the values here: https://github.com/nextcloud/nextcloud-snap#configuration

I executed the following commands, so I don't think there is a PHP problem.

snap set nextcloud php.memory-limit=-1
snap set nextcloud php.upload-max-filesize=16G
snap set nextcloud php.post-max-size=16G
snap set nextcloud php.max-input-time=-1
snap set nextcloud php.php.max-execution-time=-1

Nextcloud Overview also doesn't show any problems

 

[unholyhumorousunratedoat]

This looks really weird!

 

[dcsapak]

3 things come to mind:

* do you use a firewall (either in pve or elsewhere)?
* maybe nextcloud uses a tmpdir somewhere outside its upload dir?
* do you use any kind of rate limiting (disk/io/net) ?

 

[unholyhumorousunratedoat]

This upload behavior is also present, if I access it via the local IP in my subnet so I think this cant be a firewall issue. I don't use the PVE Firewall or any other internal firewall.

Yes, nextcloud is using a temporary directory - it is here /[path_to_data_directory]/username/uploads
But SCP into that "uploads" folder is as fast as expected 112,5MiByte/s

I dont think, that there is any limiting by disk, IO or network - in my opinion transfering through SCP is the proof for that.


This is the usage while uploading something to the nextcloud:


But thank you for your input Sir! If you have any other ideas - I'd be happy to hear it.
Maybe one more info on how I created that LXC. To use snap (in my unprivileged LXC) I did the things you wrote here:
https://forum.proxmox.com/threads/ubuntu-snaps-inside-lxc-container-on-proxmox.36463/#post-230060

Put this in /etc/pve/lxc/$vmid.conf
features: mount=fuse,nesting=1
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file 0 0

And inside the Debian Container I ran the following (before running "snap install nextcloud")
apt install snapd squashfuse fuse

 

[unholyhumorousunratedoat]

Just a quick update for you. Today I installed ESXi instead of Proxmox. I created a Debian VM and installed nextcloud via SNAP (on SSD, 8GB RAM). Same Upload and download behaviour.

Then I installed Debian directly on the hardware - and the same result. I'm sure the nextcloud installation via snap has some serious performace problems (probably integrated php is the problem).

 

[unholyhumorousunratedoat]

If I use Dolphin (my file explorer under Manjaro) for Uploading a file to the Server, the speed is perfect!

 

[unholyhumorousunratedoat]

While Uploading even via webDAV to my nextcloud-snap installation on my Arch Linux VM (SSD, 8GB RAM) ist pretty slow (and looks weird)

 

[unholyhumorousunratedoat]

So today I tried it one more time with Ubuntu LXC and Nextcloud Snap (8GB RAM, 8GB NVMe storage)

Short version: Download via Webbrowser ist pretty fast (~100MiB), but Upload is really slow. I tested the upload via Webbrowser and Nextcloud Sync Desktop application. It's between 2.5MiB and 15MiB. I'll test the same setup now but not using Ubuntu LXC but Ubuntu VM.

 

[Rothammel]

I have similar problems:
https://help.nextcloud.com/t/new-nextcloud-22-1-1-installation-slow-up-download-speed/123039