Proxmox web UI accessible but shell returns Error 500: close (rename) atomic file....

[raiz3n]

Hello everyone,


I’m running into an issue with my Proxmox server that started about 2 hours ago and I’m a bit stuck.
The web interface is still accessible, but any attempt to open a shell (node shell or VM shell) returns an HTTP 500 error.
There was an unexpected stop / interruption earlier. After that, the virtual machines did restart automatically and they all seem up and running.

It looks like pveproxy / pveshell is not responding correctly, but I can’t confirm since I don’t have shell access anymore.
I still have access to the system logs via the web UI, and I can copy/paste them below.

What I’ve checked so far:

• Web UI is reachable and responsive
• No recent intentional configuration changes
• Issue appeared suddenly

Proxmox version:
9.1.4
Error message (HTTP 500 / shell):
Error 500: close (rename) atomic file '/var/log/pve/tasks/active' failed: Permission denied)

Relevant logs (journal / syslog):
Feb 09 12:11:02 proxmox pvestatd[2137]: status update time (10.748 seconds)
Feb 09 12:11:02 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:04 proxmox crontab[3982224]: (root) LIST (root)
Feb 09 12:11:04 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:04 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:04 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:04 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:10 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:10 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:10 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:10 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:11 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:11 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:11 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox crontab[3982251]: (root) LIST (root)
Feb 09 12:11:12 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:12 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:12 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:12 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:13 proxmox pvestatd[2137]: status update time (10.451 seconds)
Feb 09 12:11:15 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:18 proxmox crontab[3982286]: (root) LIST (root)
Feb 09 12:11:18 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:18 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:18 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:18 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:20 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:20 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:20 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:22 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:22 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:22 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:23 proxmox pvestatd[2137]: status update time (9.989 seconds)
Feb 09 12:11:23 proxmox crontab[3982332]: (root) LIST (root)
Feb 09 12:11:23 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:23 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:23 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:23 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:24 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:28 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:30 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:30 proxmox pvestatd[2137]: status update time (7.905 seconds)
Feb 09 12:11:31 proxmox crontab[3982368]: (root) LIST (root)
Feb 09 12:11:31 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:31 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:31 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:31 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:35 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:39 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:39 proxmox crontab[3982410]: (root) LIST (root)
Feb 09 12:11:39 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:39 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:39 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:39 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:41 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:41 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:41 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox systemd-logind[1377]: Power key pressed short.
Feb 09 12:11:42 proxmox systemd-logind[1377]: Powering off...
Feb 09 12:11:42 proxmox systemd-logind[1377]: System is powering down.

At this point, I’m wondering if a full reboot of the Proxmox host would be recommended — but I’m honestly a bit worried that the server might not come back up at all, given the current state.
Has anyone experienced something similar, or has advice on whether a reboot is safe or if there are checks I should do first?

Any help would be greatly appreciated. Thanks!

 

[Moayad]

Hi,

Are you logged in as the root user? Have you tried a different browser?

 

[raiz3n]

Yes i'm logged as root user.
and i tried with different browser, same thing.
I see that corosync service is with "dead" status
pvefw-logger => failed
systemd-journald => failed
all others services are running

 

[Moayad]

Could you please post the output of the following commands:

mount
df -h /

 

[raiz3n]

the retunr of the command @Moayad

root@proxmox:~# mount
df -h /
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=24578236k,nr_inodes=6144559,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=600,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4923800k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6554)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)
/dev/sdd2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
raid on /raid type zfs (rw,relatime,xattr,noacl,casesensitive)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/credentials/getty@tty1.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 68G 27G 38G 42% /

Thanks a lot for your help

 

[fba]

Has anyone experienced something similar, or has advice on whether a reboot is safe or if there are checks I should do first?

Seems like you're not the only one. No solution so far, unfortunately. https://forum.proxmox.com/threads/c...tasks-active-failed-permission-denied.180139/