[SOLVED] Proxmox 7.0 Upgrade Errors

manjotsc

Active Member
Jul 2, 2020
73
6
28
Montreal, Quebec
Hi,

While upgrading to 7.0, in the end this is the output.

Code:
root@vms:~# apt upgrade
Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.
root@vms:~# df -i
Filesystem                  Inodes   IUsed     IFree IUse% Mounted on
udev                       3052638     742   3051896    1% /dev
tmpfs                      3068300  139840   2928460    5% /run
/dev/mapper/pve-root        966656  207986    758670   22% /
tmpfs                      3068300      95   3068205    1% /dev/shm
tmpfs                      3068300      30   3068270    1% /run/lock
tmpfs                      3068300      18   3068282    1% /sys/fs/cgroup
/dev/sdg1                305201152 1190911 304010241    1% /mnt/pve/TenTB
/dev/sdf1                 61054976    1080  61053896    1% /mnt/S1TB
/dev/sda2                        0       0         0     - /boot/efi
/dev/sdc1                 14655488      63  14655425    1% /mnt/pve/SSDStorage
/dev/sde1                244195328   86234 244109094    1% /mnt/4TB
/dev/sdd1                 61054976    5026  61049950    1% /mnt/pve/WDOneTB
/dev/sdb1                 61054976   11979  61042997    1% /mnt/pve/OneTB
//192.168.40.2/TempShare         0       0         0     - /mnt/pve/TempShare
tmpfs                      3068300      11   3068289    1% /run/user/0
/dev/fuse                   262144      73    262071    1% /etc/pve
root@vms:~#
 
@t.lamprecht @Nefariousparity Proxmox terminal is very slow over ssh and I not able to login to gui anymore. Login Falied Error. I scroll up the console and found flood of these.

Code:
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/bpck.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/dstr.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/epia.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pt.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/on26.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/friq.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/fit3.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pf.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/kbic.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/comm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pcd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/aoe/aoe.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/brd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/floppy.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/mtip32xx/mtip32xx.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/zram/zram.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-tcp.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/ib_core.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/ib_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/iw_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/rdma_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-rdma.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fc.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/target/nvmet.ko': No such file or directory
 
@t.lamprecht @Nefariousparity Found the problem /dev/mapper/pve-root was full in storage. But still can't login I am getting "

Connection failure. Network error or Proxmox VE services not running?"

Code:
root@vms:~# df
Filesystem                1K-blocks       Used  Available Use% Mounted on
udev                       12210552          0   12210552   0% /dev
tmpfs                       2454644     588312    1866332  24% /run
/dev/mapper/pve-root       15158232   15141848          0 100% /
tmpfs                      12273200      48772   12224428   1% /dev/shm
tmpfs                          5120          0       5120   0% /run/lock
tmpfs                      12273200          0   12273200   0% /sys/fs/cgroup
/dev/sdg1                9688383588 6723466588 2476578828  74% /mnt/pve/TenTB
/dev/sdf1                 960380648  910283528    1242660 100% /mnt/S1TB
/dev/sda2                    523248        312     522936   1% /boot/efi
/dev/sdc1                 229701520  136949532   81014104  63% /mnt/pve/SSDStorage
/dev/sde1                3844639728 2084415876 1564856636  58% /mnt/4TB
/dev/sdd1                 960380628  516056904  395469264  57% /mnt/pve/WDOneTB
/dev/sdb1                 960380628  460214484  451311684  51% /mnt/pve/OneTB
//192.168.40.2/TempShare   29439884    1921524   27518360   7% /mnt/pve/TempShare
tmpfs                       2454640          0    2454640   0% /run/user/0
/dev/fuse                    131072         36     131036   1% /etc/pve
 
So clearly no space left as a normal df call would show: df /

You need to cleanup some files to advance, if your lucky only some log file went crazy and takes up lots of space, those could be deleted relatively safely.

What I do in such cases is using "du" (disk usage) tool to find out the directories with the biggest usages, I normally start on root, tell du to not cross into other file-systems (-x option), output human-readable units (-h), and only report to 1 directory depth -level (-d1). e.g., first du -hxd1 / | sort -h and then if that shows /var as biggest user I use that as new start point: du -hxd1 /var/ | sort -h

If you're unsure about that I'd recommend posting the following output here, so we can help
Bash:
du -hxd2 / | sort -h | tail -20

(may need quite a while depending on IO speed)
 
I think they failed to start during an upgrade.

Check: systemctl list-units --failed

I'd guess that you need to restart at least pveproxy and pveproxy service:
systemctl try-reload-or-restart pveproxy.service pvedaemon.service

There may be other failed services requiring a restart too though.
 
Last edited:
@t.lamprecht This is what it's showing

Code:
root@vms:~# systemctl list-units --failed
  UNIT                     LOAD   ACTIVE SUB    DESCRIPTION
● dcservice.service        loaded failed failed Service for ManageEngine UEMS Agent
● nfs-blkmap.service       loaded failed failed pNFS block layout mapping daemon
● pve-daily-update.service loaded failed failed Daily PVE download activities
● pvesr.service            loaded failed failed Proxmox VE replication runner

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
4 loaded units listed.

Code:
root@vms:~# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-06-26 18:27:24 EDT; 2 weeks 2 days ago
    Process: 2825741 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
   Main PID: 1905 (pveproxy)
      Tasks: 4 (limit: 28618)
     Memory: 246.6M
     CGroup: /system.slice/pveproxy.service
             ├─   1905 pveproxy
             ├─2911473 pveproxy worker
             ├─2911504 pveproxy worker
             └─2913851 pveproxy worker

Jul 13 02:18:56 vms pveproxy[1905]: worker 2911504 started
Jul 13 02:19:26 vms pveproxy[2911443]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1795.
Jul 13 02:19:26 vms pveproxy[2911443]: error writing access log
Jul 13 02:26:31 vms pveproxy[2911443]: worker exit
Jul 13 02:26:31 vms pveproxy[1905]: worker 2911443 finished
Jul 13 02:26:31 vms pveproxy[1905]: starting 1 worker(s)
Jul 13 02:26:31 vms pveproxy[1905]: worker 2913851 started
Jul 13 02:37:03 vms pveproxy[2913851]: proxy detected vanished client connection
Jul 13 02:39:30 vms pveproxy[2911504]: proxy detected vanished client connection
Jul 13 02:47:44 vms pveproxy[2911473]: proxy detected vanished client connection

Code:
root@vms:~# systemctl status pvesr.service
● pvesr.service - Proxmox VE replication runner
     Loaded: loaded (/lib/systemd/system/pvesr.service; static)
     Active: failed (Result: timeout) since Tue 2021-07-13 02:43:15 EDT; 13min ago
TriggeredBy: ● pvesr.timer
   Main PID: 2832705
      Tasks: 1 (limit: 28618)
     Memory: 68.0M
     CGroup: /system.slice/pvesr.service
             └─2832705 /usr/bin/perl -T /usr/bin/pvesr run --mail 1

Jul 13 00:28:00 vms systemd[1]: Starting Proxmox VE replication runner...
Jul 13 02:38:45 vms systemd[1]: pvesr.service: State 'stop-sigterm' timed out. Killing.
Jul 13 02:38:45 vms systemd[1]: pvesr.service: Killing process 2832705 (pvesr) with signal SIGKILL.
Jul 13 02:40:15 vms systemd[1]: pvesr.service: Processes still around after SIGKILL. Ignoring.
Jul 13 02:41:45 vms systemd[1]: pvesr.service: State 'final-sigterm' timed out. Killing.
Jul 13 02:41:45 vms systemd[1]: pvesr.service: Killing process 2832705 (pvesr) with signal SIGKILL.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Processes still around after final SIGKILL. Entering failed mode.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Failed with result 'timeout'.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Unit process 2832705 (pvesr) remains running after unit stopped.
Jul 13 02:43:15 vms systemd[1]: Stopped Proxmox VE replication runner.
root@vms:~# systemctl start pvesr.service
Failed to start pvesr.service: Transaction for pvesr.service/start is destructive (cryptsetup.target has 'stop' job queued, but 'start' is included in transaction).
See system logs and 'systemctl status pvesr.service' for details.
root@vms:~# systemctl restart pvesr.service
 
Last edited:
systemctl start pvesr.service
pvesr (storage replication) is managed through the pvesr.timer unit, so it cannot be started directly - for it you can just reset the failed state (systemctl reset-failed pvesr.service.

Jul 13 02:18:56 vms pveproxy[1905]: worker 2911504 started
Jul 13 02:19:26 vms pveproxy[2911443]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1795.
Jul 13 02:19:26 vms pveproxy[2911443]: error writing access log
Jul 13 02:26:31 vms pveproxy[2911443]: worker exit
The pveproxy is the main API daemon, and it was definitively affected when space run out, does it work now after a full reload?

Else I'd recommend restarting pveproxy.service and pvedaemon.service via systemctl.
 
@t.lamprecht I was not able to reboot or shutdown the Proxmox server from terminal, It kept waiting for something to stop or something for hours. Then finally I hard rebooted it via power button(physical-one). Then Proxmox went into emergency mode, then I ran apt update and apt upgrade and reboot it and It's working now, all VMS and LXCs are up.
 
There were probably half finished upgrades applied when your host ran out of space, so repeating the apt update and apt full-upgrade (avoid a plain upgrade command, especially on major upgrades) was a good call.

Glad to hear that all is up again.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!