Troubleshooting: /dev/mapper/pve-root fills up when uploading ISO's. Confused.

[l0ktar]

Hey there~!

I'm very new to any kind of virtual environment and actually got the server I'm running proxmox VE on from a friend for free. I've been running like 2-3 VMS at a time and just decided to spin up an AD lab on it for pentest practice.

While uploading the Kali iso I wanna use it stopped and game me some "error code '0'" but I figured out it was because /dev/mapper/pve-root keeps filling up when I'm uploading ISO's. I'm not sure if I have the drive and stuff setup right so this could just be because of terrible setup on my part. Don't judge too hard lol.

Attached all possibly relevant pics.

TL,DR: /dev/mapper/pve-root fills up when uploading ISO's. Porque?

 

[t.lamprecht]

Hi,

TL,DR: /dev/mapper/pve-root fills up when uploading ISO's. Porque?

That's because uploads happen first to /var/tmp/pveupload-$RAND file first, there are some reason for this, but they aren't really that strong to justify having it hard-coded.
The biggest one would be that files can be uploaded to local storage of other nodes, so doing an intermediate copy makes it easier and also a bit more robust than streaming directly.

I think we can change this though, and be it only by making the base staging directory for uploads configurable, but we can probably do better and stream directly to other nodes or save the incoming upload directly to the target storage.

You could open an enhancement request over at https://bugzilla.proxmox.com/ for this, please include a link to this thread if you do so.

As workaround, you could make /var/tmp a mount (or symlink) on the filesystem that has more usable storage space.

 

[chrcoluk]

I will do the request if it hasnt been done already, I think its really wasteful of nand writes and adds an extra delay to use /var/tmp as a staging area for ISO's. in the mean time I might mount /var/tmp as a ram disk.

 

[jonaylton]

Hello, I tried both solutions (make /var/tmp a mount to /tank/tmp (or symlink /var/tmp to /tank/tmp)), rebooted pve host, but I can't get rid of the following error:

Error '0' occurred while receiving the document.

Here is the output for ls when using symlink

root@pve:~# ls -l /var/tmp lrwxrwxrwx 1 root root 9 Aug 27 13:46 /var/tmp -> /tank/tmp

journalctl -e shows the following:

Aug 27 01:46:14 pve pveproxy[1949]: unable to create temporary upload file '/var/tmp/pveupload-c3710511d73f11566420a021a9927079' at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1566. Aug 27 01:46:14 pve pveproxy[1949]: unable to create temporary upload file '/var/tmp/pveupload-4271a33c83c3dbedf506c7775f503ce8' at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1566. Aug 27 01:46:14 pve pveproxy[1949]: problem with client ::ffff:192.168.0.109; No space left on device

I have ZFS boot drive in mirror (raid-1) with 32 gb capacity, and another 2x 500GB WD-Blue SSDs (also in mirror (raid-1) ZFS) for ISOS, VMs and CTs, etc.
If I remove the mount/symlink, all works fine but uploads gets written first to /var/tmp (in local storage) then to /tank/isos destination, I want it to use /tank/tmp folder I created instead of /var/tmp

PS.: SCP/SFTP to /var/tmp -> /tank/tmp works fine too, also "Download from URL" option

 

[t.lamprecht]

No space left on device

Hmm, this error seems odd if it works when doing so manually..

Can you please post the output of:
findmnt --df
ls -la /tank/tmp

 

[jonaylton]

Hmm, this error seems odd if it works when doing so manually..

Can you please post the output of:
findmnt --df
ls -la /tank/tmp

Sure, here it is:

root@pve:~# findmnt --df
SOURCE                           FSTYPE     SIZE   USED  AVAIL USE% TARGET
udev                             devtmpfs   7.7G      0   7.7G   0% /dev
tmpfs                            tmpfs      1.5G   1.1M   1.5G   0% /run
rpool/ROOT/pve-1                 zfs       27.1G   4.2G  22.9G  16% /
tmpfs                            tmpfs      7.7G  48.8M   7.6G   1% /dev/shm
tmpfs                            tmpfs        5M      0     5M   0% /run/lock
rpool                            zfs       22.9G   128K  22.9G   0% /rpool
rpool/ROOT                       zfs       22.9G   128K  22.9G   0% /rpool/ROOT
rpool/data                       zfs       22.9G   128K  22.9G   0% /rpool/data
tank                             zfs      431.3G   128K 431.3G   0% /tank
tank/isos                        zfs      437.1G   5.8G 431.3G   1% /tank/isos
tank/backups                     zfs      435.3G     4G 431.3G   1% /tank/backups
tank/ct-drives                   zfs      431.3G   128K 431.3G   0% /tank/ct-drives
tank/media                       zfs         64G   2.2G  61.8G   3% /tank/media
tank/vm-drives                   zfs      431.3G   128K 431.3G   0% /tank/vm-drives
tank/ct-drives/subvol-101-disk-0 zfs          4G 960.3M   3.1G  23% /tank/ct-drives/subvol-101-disk-0
tank/ct-drives/subvol-100-disk-0 zfs          8G   5.4G   2.6G  67% /tank/ct-drives/subvol-100-disk-0
tmpfs                            tmpfs      1.5G      0   1.5G   0% /run/user/0

root@pve:~# ls -la /tank/tmp
total 14
drwxr-xr-x 4 root root  5 Aug 28 10:11 .
drwxr-xr-x 8 root root  8 Aug 27 12:06 ..
-rw-r--r-- 1 root root 16 Aug 28 10:11 pve-reserved-ports
drwx------ 3 root root  3 Aug 27 13:48 systemd-private-f5b5db065f004a6fb45d894d8dee040e-chrony.service-KVjEhD
drwx------ 3 root root  3 Aug 27 13:48 systemd-private-f5b5db065f004a6fb45d894d8dee040e-systemd-logind.service-16bHX0

Is it possible that the perl function that is trying to create the temp file is seeing the symlink size instead of actual destination directory size? For example, the symlink has 9 bytes in size as you can see bellow

root@pve:~# ls -lh /var/tmp
lrwxrwxrwx 1 root root 9 Aug 27 13:46 /var/tmp -> /tank/tmp

The perl function is:

root@pve:~# nano +1566 /usr/share/perl5/PVE/APIServer/AnyEvent.pm

            my $tmpfilename = get_upload_filename();
            my $outfh = IO::File->new($tmpfilename, O_RDWR|O_CREAT|O_EXCL, 0600) ||
                die "unable to create temporary upload file '$tmpfilename'";

root@pve:~# nano +106 /usr/share/perl5/PVE/APIServer/AnyEvent.pm

sub log_aborted_request {
    my ($self, $reqstate, $error) = @_;

    my $r = $reqstate->{request};
    return if !$r; # no active request

    if ($error) {
        syslog("err", "problem with client $reqstate->{peer_host}; $error");
    }

    $self->log_request($reqstate);
}

 

[t.lamprecht]

root@pve:~# ls -la /tank/tmp
total 14
drwxr-xr-x 4 root root 5 Aug 28 10:11 .

Hmm, above is the only thing that sticks out from a quick check, /var/tmp (or in your case /tank/tmp, as that's where the symlink points to) must be world writable and should also have the sticky bit set, so can you try:

chmod ugo+rwxt /tank/tmp

That would also fit to the "unable to create temporary upload" error (which I previously overlooked, only saw the "No space left on device" one)

 

[jonaylton]

Hmm, above is the only thing that sticks out from a quick check, /var/tmp (or in your case /tank/tmp, as that's where the symlink points to) must be world writable and should also have the sticky bit set, so can you try:

chmod ugo+rwxt /tank/tmp

That would also fit to the "unable to create temporary upload" error (which I previously overlooked, only saw the "No space left on device" one)

Awesome, that indeed solves the problem. Thank you!

 

[Neobin]

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=4935