Proxmox web UI accessible but shell returns Error 500: close (rename) atomic file....

R

raiz3n

New Member
Jul 8, 2024
8
3
3
Hello everyone,


I’m running into an issue with my Proxmox server that started about 2 hours ago and I’m a bit stuck.
The web interface is still accessible, but any attempt to open a shell (node shell or VM shell) returns an HTTP 500 error.
There was an unexpected stop / interruption earlier. After that, the virtual machines did restart automatically and they all seem up and running.

It looks like pveproxy / pveshell is not responding correctly, but I can’t confirm since I don’t have shell access anymore.
I still have access to the system logs via the web UI, and I can copy/paste them below.

What I’ve checked so far:
  • Web UI is reachable and responsive
  • No recent intentional configuration changes
  • Issue appeared suddenly
Proxmox version:
9.1.4
Error message (HTTP 500 / shell):
Error 500: close (rename) atomic file '/var/log/pve/tasks/active' failed: Permission denied)

Relevant logs (journal / syslog):
Feb 09 12:11:02 proxmox pvestatd[2137]: status update time (10.748 seconds)
Feb 09 12:11:02 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:04 proxmox crontab[3982224]: (root) LIST (root)
Feb 09 12:11:04 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:04 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:04 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:04 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:10 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:10 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:10 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:10 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:11 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:11 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:11 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:12 proxmox crontab[3982251]: (root) LIST (root)
Feb 09 12:11:12 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:12 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:12 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:12 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:12 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:13 proxmox pvestatd[2137]: status update time (10.451 seconds)
Feb 09 12:11:15 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:18 proxmox crontab[3982286]: (root) LIST (root)
Feb 09 12:11:18 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:18 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:18 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:18 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:20 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:20 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:20 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:21 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:22 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:22 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:22 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:23 proxmox pvestatd[2137]: status update time (9.989 seconds)
Feb 09 12:11:23 proxmox crontab[3982332]: (root) LIST (root)
Feb 09 12:11:23 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:23 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:23 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:23 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:24 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:28 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:30 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:30 proxmox pvestatd[2137]: status update time (7.905 seconds)
Feb 09 12:11:31 proxmox crontab[3982368]: (root) LIST (root)
Feb 09 12:11:31 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:31 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:31 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:31 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:35 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:39 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:39 proxmox crontab[3982410]: (root) LIST (root)
Feb 09 12:11:39 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:39 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:39 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:39 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:41 proxmox pvestatd[2137]: storage 'syno' is not online
Feb 09 12:11:41 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:41 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox kernel: usb 1-7.3: reset high-speed USB device number 4 using xhci_hcd
Feb 09 12:11:42 proxmox systemd-logind[1377]: Power key pressed short.
Feb 09 12:11:42 proxmox systemd-logind[1377]: Powering off...
Feb 09 12:11:42 proxmox systemd-logind[1377]: System is powering down.

At this point, I’m wondering if a full reboot of the Proxmox host would be recommended — but I’m honestly a bit worried that the server might not come back up at all, given the current state.
Has anyone experienced something similar, or has advice on whether a reboot is safe or if there are checks I should do first?

Any help would be greatly appreciated. Thanks!
 
Yes i'm logged as root user.
and i tried with different browser, same thing.
I see that corosync service is with "dead" status
pvefw-logger => failed
systemd-journald => failed
all others services are running
 
the retunr of the command @Moayad

root@proxmox:~# mount
df -h /
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=24578236k,nr_inodes=6144559,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=600,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4923800k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6554)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)
/dev/sdd2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
raid on /raid type zfs (rw,relatime,xattr,noacl,casesensitive)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/credentials/getty@tty1.service type tmpfs (ro,nosuid,nodev,noexec,relatime,nosymfollow,size=1024k,nr_inodes=1024,mode=700,inode64,noswap)
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 68G 27G 38G 42% /

Thanks a lot for your help
 
seriously? no solution ?
i tried also those commands, seen on other thread
chmod 777 on the file
and then
Bash:
systemctl stop pve-cluster
rm -f /var/log/pve/tasks/active*
rm -f /var/log/pve/tasks/index*
systemctl start pve-cluster[/CODE]

nothing..
file active doesn't exist anymore, but still the error 500, impossible to rename
nobody has another idea?
 
For me it's not resolved. and no others answers.
I'm a little bit disapointed about Proxmox
 
raiz3n said:
Feb 09 12:11:39 proxmox log[1284409]: cuowu: daai suers wenjan shbai: No such file or directory
Feb 09 12:11:39 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Feb 09 12:11:39 proxmox log[1284409]: Error: Could not protect file /var/log/log
Feb 09 12:11:39 proxmox log[1284409]: cuowu: shzhi wejian wuhong xngzhi sibai: Inappropriate ioctl for device
Click to expand...
you have other weird errors in your logs as well..

please double check the full journal for possible causes.
 
ShashwatUpadhyay said:
I am also facing the exact same issue since the day this thread was created.
Has anyone resolved it or found a solution?
Click to expand...
Do you have the same errors in your log?
cuowu: daai suers wenjan shbai
cuowu: shzhi wejian wuhong xngzhi sibai

fabian said:
you have other weird errors in your logs as well..

please double check the full journal for possible causes.
Click to expand...
ok,
i will double check
Thanks
 
raiz3n said:
cuowu: daai suers wenjan shbai
cuowu: shzhi wejian wuhong xngzhi sibai
Click to expand...
Is this Chinese as language? Using a translator it might be the case, but in total it doesn't make sense.
Error: The file failed to be saved.
Error: Even for the evil deeds of Wu Hongxia, that value is worth four hundred.
Click to expand...
 
raiz3n said:
Do you have the same errors in your log?
Click to expand...
yes i had the exact same error log.

I eventually decided to contact the person who set up the server a few months ago. After accessing the server, he noticed something suspicious — an application named rig-mining (or something similar) was running and pushing CPU usage to 99–100%. I hadn’t noticed that suspicious process earlier, so I force-restarted the server. After that, the error occurred, which completely failed the journald service. As a result, logs are no longer being created, and several services stopped working because of this issue.

We have now taken a backup of the disk and are planning to reinstall everything soon.
 
Hello,

Here is a short feedback regarding this issue.

Initial symptoms​

  • Web UI accessible
  • Shell access returned:

    apt upgrade failing with permission errors

    Error 500: close (rename) atomic file '/var/log/pve/tasks/active' failed: Permission denied

  • journald service failing
  • random segmentation faults
  • strange log entries
  • accessing /etc/ld.so.preload caused segfault

Root cause​


After investigation, /etc/ld.so.preload contained:

/usr/local/lib/sshdd.so

This is not legitimate and strongly indicates an LD_PRELOAD rootkit.
Even after manually cleaning /etc/ld.so.preload and removing the malicious .so file, the file was recreated and corrupted again after reboot.

At that point, system integrity could no longer be trusted.
Port 8006 had been exposed to WAN at the time.

Resolution​

I performed a clean reinstall of Proxmox on the OS disk.
My ZFS pool was on separate disks, so I was able to:

zpool import -f raid

All VMs were restored by recreating their configs manually using the existing ZFS volumes.
No data loss.

Conclusion​

If you see:
  • atomic rename permission errors
  • journald failing
  • segfault when reading /etc/ld.so.preload
  • unknown .so file referenced there
Do not try to “fix” the system in place.

I confirm reinstalling is the only one approach.

Everything is now stable after clean reinstall.
 
  • Like
Reactions: ShashwatUpadhyay and Onslow
raiz3n said:
For me it's not resolved. and no others answers.
I'm a little bit disapointed about Proxmox
Click to expand...
raiz3n said:

Root cause​


After investigation, /etc/ld.so.preload contained:

/usr/local/lib/sshdd.so

This is not legitimate and strongly indicates an LD_PRELOAD rootkit.
Even after manually cleaning /etc/ld.so.preload and removing the malicious .so file, the file was recreated and corrupted again after reboot.

At that point, system integrity could no longer be trusted.
Port 8006 had been exposed to WAN at the time.
Click to expand...

:)

This here is the official community forum. The fact that the Proxmox developers provide support here (for free) is a huge bonus, but nothing one can or should expect or even insist on.
If you need guaranteed support in a timely manner, you can get (a) subscription(s): [1] (at least: "basic") for your node(s) and then open a support ticket.

[1] https://proxmox.com/en/products/proxmox-virtual-environment/pricing
 
Neobin said:
:)

This here is the official community forum. The fact that the Proxmox developers provide support here (for free) is a huge bonus, but nothing one can or should expect or even insist on.
If you need guaranteed support in a timely manner, you can get (a) subscription(s): [1] (at least: "basic") for your node(s) and then open a support ticket.

[1] https://proxmox.com/en/products/proxmox-virtual-environment/pricing
Click to expand...


I fully agree with you.

I have always managed to solve my issues thanks to the community, and I would like to clarify that I was not disappointed with the help provided here — quite the opposite. The discussions have been constructive and helpful.

My frustration was more directed at Proxmox itself. From a Linux-based system, I was expecting a bit more robustness and perhaps more low-level options to troubleshoot and recover from this kind of issue without resorting to a full reinstallation.

€370 is still a significant amount in my current situation. And to be completely honest, I’m not entirely sure that official support would have provided a very different answer from the one given here — which ultimately was to reinstall everything.

That said, I fully understand the value of a subscription: it guarantees response times and a formal support framework, which the community understandably cannot promise.

In the end, everything worked out well. The reinstallation process turned out to be much smoother than I expected, and I was genuinely impressed by how easy it was to reinstall and re-import the ZFS RAID. On that point, it was remarkably well designed.

Overall, the experience remains positive, and I truly appreciate the help provided here.
 
  • Like
Reactions: miztaq
If this was really a rootkit, reinstallation is only valid solution. No idea what you expect Proxmox VE should do there against malicious software.
But I am glad you could resolve it and your system is up and running again.
 
miztaq said:
The HTTP 500 shell errors with messages like "close (rename) atomic file '/var/log/pve/tasks/active' failed: Permission denied" usually mean filesystem corruption or a compromised system. In this case, the file /etc/ld.so.preload contained a malicious sshdd.so, indicating a rootkit. The only safe solution is to back up your VMs and ZFS pools, do a clean reinstall of Proxmox on the OS disk, then re-import your ZFS pools and recreate VM configs. Trying to fix the system in place is unsafe. More details are in the Proxmox forum discussion here: https://forum.proxmox.com/threads/atomic-file-error-500-shell-access
Click to expand...
You created your account today and all of your 3 posts so far contain only AI Slop. What are your intentions at this point?
 
Last edited:
  • Like
Reactions: fba and Neobin
You must log in or register to reply here.
Top Bottom
Back