ISO Upload Hangs pveproxy.service (>2GB)

[nicholas26]

Anyone else having issues with uploading ISO files in the GUI and if they are over 2GB the progress bar stops and the server appears to go offline?
Restarting the pveproxy.service (after getting a physical console on the server, since SSH seems to get hung up too) seems to fix it.
I get a console info: hung_task_timeout_secs

Seems to happen in multiple 6.0 versions. I'm currently 100% up-to-date with the no-subscription repo but it failed last week too before I applied the latest.

For now my workaround is to just SCP the ISO file to storage.

 

[dcsapak]

is your /var/tmp full? can you post the output of 'df -h' ?

 

[nicholas26]

`/tmp` has plenty of free space, currently 55G avail.
There are also five servers in the cluster and it will die on any of them. (unrelated, but shows it's not isolated to one specific server)
I've also tried uploading to a storage repo that's NFS, CEPH, and Local ZFS with the same results.
`/tmp` happens to be on RAID1 ZFS FWIW.

 

[warloxian]

I have same issue, no matter what the size of the files are. Here is my
df -h , I am running 6 nodes with more storage, ram and cpu power then I'll need for my cyber range. I am a NOOB and I'm sorry to take your time, but I need a little help. I assume I am not understanding how storage works on servers and Proxmox, but Im not sure. I am running Proxmox 7.0-13, but this was happening on last version as well??
Filesystem Size Used Avail Use% Mounted on
udev 24G 0 24G 0% /dev
tmpfs 4.8G 1.3M 4.8G 1% /run
/dev/mapper/pve-root 94G 19G 71G 21% /
tmpfs 24G 63M 24G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/fuse 128M 48K 128M 1% /etc/pve
tmpfs 4.8G 0 4.8G 0% /run/user/0

 

[dcsapak]

I have same issue, no matter what the size of the files are.

can you post the logs of the node at the time you try to upload (with 'dmesg' and 'journalctl' commands)

 

[dea]

Yes, can I confirm...

Proxmox CE 7.0.14+1, when upload 8 Gbyte ISO to share NFS through the node n.1, block transfer and the node become irresponsive (the node n.2 response correctly).

Luca

 

[dea]

OK, I've reproduced the problem exactly:

- Proxmox 7.0.14+1
- n.2 nodes NFS share 10 gbps to NetAPP datastore dedicated to NFS, 10 gbps trunk for VM and management. 1 gbps dedicated and isolated link for corosync.
- Upload in GUI 3 Gbyte ISO image
- I'm connected to node 1. After 35% process complete, node hang. No responsive via GUI, no responsive to SSH. All VM running on node 1 works correctly.
- Node 2 works fine, when I try to connect to cluster by node 2 I see the node 1 in state (?)
- After several minutes, node 1 works again, state green all connection works correctly.
- ... and yes... free space in node 1 is bigger than 50 Gbyte

Thanks
Luca

 

[warloxian]

My problem is with any method, scp, sftp, GUI etc...etc... The system keeps timing out. The only way I am able to get an ISO image on my system is to plug in my Ventoy USB drive to my server and copy the files over. That works fine. No other method works. Any answers??? I suspect, but am not sure that ths is related o the problem I am having with Ceph. No matter what Ceph commands I enter the system either timeouts at 300 or 500? I did have another node that I deleted and then I replaced it with a different node , diffeent name, different IP address. It would appear that I have some remnants of the old node still in some of my config files. Can anyone please take this NEWBY by the hand and guide me , step by step, in the process to fix CEPH? I can then see if it fixes my timeout isues with uploading, downloading or transferring ISO images to my PVE. Any help would be gratefully appreciated.

 

[ledufakademy]

same issue here with 7.1.10
the server become irresponsive

 

[oguz]

please post the logs like @dcsapak asked

 

[Jannoke]

It's probably somewhat offtopic but why are we uploading stuff to /tmp to copy them over to some storage later instead of just copying directly to storage. I have hung the proxmox many times because of big image file and not enough room on /tmp folder. Also after uploading it will take additional time to copy it from /tmp to real storage.

 

[warloxian]

Just for all of you. I have been dealing with this problem since day one. I have found that web UI, ssh and ftp all do the exact same thing 99% of the time and then randomly Proxmox would just accept one of these files? Anyway, about three updates ago, or so, the web Ui has started working perfectly. I can upload any size, so far, and it's working fine. I have made no other changes to files or directories. I don't know if this was something intentional that the builders of this fine system did or it ws n accident, but I want to say "YOUR DOING A F"N AWESOME JOB GUYS/GALS

 

[DmitryMigunov]

We are still observing this issue on the latest Proxmox 8.0.3:

Jun 28 14:24:01 prox-4 pvedaemon[1280869]: <root@pam> starting task UPID:prox-4:001DF912:008802CE:649C1851:download:2-ubuntu-23.04-desktop-amd64.iso:root@pam:
Jun 28 14:24:23 prox-4 pvedaemon[1190863]: worker exit
Jun 28 14:24:23 prox-4 pvedaemon[9990]: worker 1190863 finished
Jun 28 14:24:23 prox-4 pvedaemon[9990]: starting 1 worker(s)
Jun 28 14:24:23 prox-4 pvedaemon[9990]: worker 1973251 started
Jun 28 14:26:45 prox-4 sshd[1917396]: pam_unix(sshd:session): session closed for user root
Jun 28 14:26:45 prox-4 systemd-logind[4335]: Session 328 logged out. Waiting for processes to exit.
Jun 28 14:27:22 prox-4 sshd[1934413]: pam_unix(sshd:session): session closed for user root
Jun 28 14:27:22 prox-4 systemd[1]: session-328.scope: Deactivated successfully.
Jun 28 14:27:22 prox-4 sshd[1934413]: pam_systemd(sshd:session): Failed to release session: Interrupted system call
Jun 28 14:27:22 prox-4 systemd[1]: session-329.scope: Deactivated successfully.
Jun 28 14:27:22 prox-4 pveproxy[1438059]: proxy detected vanished client connection
Jun 28 14:27:22 prox-4 systemd[1]: session-329.scope: Consumed 28.808s CPU time.
Jun 28 14:27:22 prox-4 pveproxy[1438606]: proxy detected vanished client connection
Jun 28 14:27:22 prox-4 systemd-logind[4335]: Session 329 logged out. Waiting for processes to exit.
Jun 28 14:27:23 prox-4 pve-firewall[7893]: firewall update time (106.857 seconds)
Jun 28 14:27:22 prox-4 systemd-logind[4335]: Removed session 328.
Jun 28 14:27:22 prox-4 systemd-logind[4335]: Removed session 329.
Jun 28 14:27:23 prox-4 pmxcfs[7588]: [status] notice: received log
Jun 28 14:27:24 prox-4 pvestatd[10147]: status update time (106.875 seconds)
Jun 28 14:27:27 prox-4 pve-ha-crm[8595]: loop take too long (111 seconds)
Jun 28 14:27:28 prox-4 pve-ha-lrm[8433]: loop take too long (110 seconds)
Jun 28 14:29:45 prox-4 pmxcfs[7588]: [status] notice: received log
Jun 28 14:29:46 prox-4 pmxcfs[7588]: [status] notice: received log
Jun 28 14:29:46 prox-4 pve-firewall[7893]: firewall update time (92.376 seconds)
Jun 28 14:29:46 prox-4 pmxcfs[7588]: [status] notice: received log
Jun 28 14:29:46 prox-4 pvestatd[10147]: status update time (92.233 seconds)
Jun 28 14:29:50 prox-4 pve-ha-crm[8595]: loop take too long (93 seconds)
Jun 28 14:29:50 prox-4 pve-ha-lrm[8433]: loop take too long (97 seconds)
Jun 28 14:30:02 prox-4 pvedaemon[1280869]: <root@pam> end task UPID:prox-4:001DF912:008802CE:649C1851:download:2-ubuntu-23.04-desktop-amd64.iso:root@pam: OK

root@prox-4:~# df
Filesystem                                         1K-blocks       Used  Available Use% Mounted on
udev                                                49423648          0   49423648   0% /dev
tmpfs                                                9891596       2076    9889520   1% /run
rpool/ROOT/pve-1                                    93886464   38059264   55827200  41% /
tmpfs                                               49457968      64368   49393600   1% /dev/shm
tmpfs                                                   5120          0       5120   0% /run/lock
ssd-pool                                           522310528        256  522310272   1% /ssd-pool
rpool                                               55827328        128   55827200   1% /rpool
ssd-pool/subvol-101-disk-0                          52428800    4896640   47532160  10% /ssd-pool/subvol-101-disk-0
rpool/ROOT                                          55827328        128   55827200   1% /rpool/ROOT
ssd-pool/subvol-220-disk-0                         104857600   56969216   47888384  55% /ssd-pool/subvol-220-disk-0
rpool/data                                          55827328        128   55827200   1% /rpool/data
tmpfs                                                9891592          0    9891592   0% /run/user/0
/dev/fuse                                             131072        124     130948   1% /etc/pve

 

[EzrealC]

same issue on 8.0.3

 

[HPmoss]

Any solution in sight?

 

[itNGO]

Also on 8.0.4 this still happens....

 

[itNGO]

My problem is with any method, scp, sftp, GUI etc...etc... The system keeps timing out. The only way I am able to get an ISO image on my system is to plug in my Ventoy USB drive to my server and copy the files over. That works fine. No other method works. Any answers??? I suspect, but am not sure that ths is related o the problem I am having with Ceph. No matter what Ceph commands I enter the system either timeouts at 300 or 500? I did have another node that I deleted and then I replaced it with a different node , diffeent name, different IP address. It would appear that I have some remnants of the old node still in some of my config files. Can anyone please take this NEWBY by the hand and guide me , step by step, in the process to fix CEPH? I can then see if it fixes my timeout isues with uploading, downloading or transferring ISO images to my PVE. Any help would be gratefully appreciated.

This should be a separate thread....

Backup your VMs.... delete all CEPH and start from scratch....

 

[JohnBee]

I have the same problem - fresh install, all drives empty - plenty of space 1.7TB

I'm on 8.03 btw

 

[LibertyTalk]

Same issue. Fresh install

 

[JohnBee]

I have the same problem - fresh install, all drives empty - plenty of space 1.7TB

I'm on 8.03 btw

Just wanted to update and say that I was able to resolve my own issue, by turning-off all power saving features, as well as c-states in BIOS - hasn't happened since