[SOLVED] Proxmox 7.0 Upgrade Errors

manjotsc · Jul 13, 2021

Hi,

While upgrading to 7.0, in the end this is the output.

Code:

root@vms:~# apt upgrade
Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.
root@vms:~# df -i
Filesystem                  Inodes   IUsed     IFree IUse% Mounted on
udev                       3052638     742   3051896    1% /dev
tmpfs                      3068300  139840   2928460    5% /run
/dev/mapper/pve-root        966656  207986    758670   22% /
tmpfs                      3068300      95   3068205    1% /dev/shm
tmpfs                      3068300      30   3068270    1% /run/lock
tmpfs                      3068300      18   3068282    1% /sys/fs/cgroup
/dev/sdg1                305201152 1190911 304010241    1% /mnt/pve/TenTB
/dev/sdf1                 61054976    1080  61053896    1% /mnt/S1TB
/dev/sda2                        0       0         0     - /boot/efi
/dev/sdc1                 14655488      63  14655425    1% /mnt/pve/SSDStorage
/dev/sde1                244195328   86234 244109094    1% /mnt/4TB
/dev/sdd1                 61054976    5026  61049950    1% /mnt/pve/WDOneTB
/dev/sdb1                 61054976   11979  61042997    1% /mnt/pve/OneTB
//192.168.40.2/TempShare         0       0         0     - /mnt/pve/TempShare
tmpfs                      3068300      11   3068289    1% /run/user/0
/dev/fuse                   262144      73    262071    1% /etc/pve
root@vms:~#

t.lamprecht · Jul 13, 2021

manjotsc said:
root@vms:~# df -i

And the actual space usage (i.e., the non-inode one?) df?

Nefariousparity · Jul 13, 2021

I wonder if it is freaking out over the NFS\SMB share there.

t.lamprecht · Jul 13, 2021

Would be strange, as we explicitly check the root mount point "/" only.

manjotsc · Jul 13, 2021

@t.lamprecht @Nefariousparity Proxmox terminal is very slow over ssh and I not able to login to gui anymore. Login Falied Error. I scroll up the console and found flood of these.

Code:

cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/bpck.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/dstr.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/epia.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pt.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/on26.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/friq.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/fit3.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pf.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/kbic.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/comm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pcd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/paride.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/paride/pd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/aoe/aoe.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/brd.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/floppy.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/mtip32xx/mtip32xx.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/block/zram/zram.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-tcp.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/ib_core.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/ib_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/iw_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/infiniband/core/rdma_cm.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-rdma.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fc.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/host/nvme-fabrics.ko': No such file or directory
mkdir: cannot create directory ‘/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme’: No space left on device
cp: cannot create regular file '/var/tmp/mkinitramfs_mqKJV7//usr/lib/modules/5.4.124-1-pve/kernel/drivers/nvme/target/nvmet.ko': No such file or directory

manjotsc · Jul 13, 2021

@t.lamprecht @Nefariousparity Found the problem /dev/mapper/pve-root was full in storage. But still can't login I am getting "

Connection failure. Network error or Proxmox VE services not running?"

Code:

root@vms:~# df
Filesystem                1K-blocks       Used  Available Use% Mounted on
udev                       12210552          0   12210552   0% /dev
tmpfs                       2454644     588312    1866332  24% /run
/dev/mapper/pve-root       15158232   15141848          0 100% /
tmpfs                      12273200      48772   12224428   1% /dev/shm
tmpfs                          5120          0       5120   0% /run/lock
tmpfs                      12273200          0   12273200   0% /sys/fs/cgroup
/dev/sdg1                9688383588 6723466588 2476578828  74% /mnt/pve/TenTB
/dev/sdf1                 960380648  910283528    1242660 100% /mnt/S1TB
/dev/sda2                    523248        312     522936   1% /boot/efi
/dev/sdc1                 229701520  136949532   81014104  63% /mnt/pve/SSDStorage
/dev/sde1                3844639728 2084415876 1564856636  58% /mnt/4TB
/dev/sdd1                 960380628  516056904  395469264  57% /mnt/pve/WDOneTB
/dev/sdb1                 960380628  460214484  451311684  51% /mnt/pve/OneTB
//192.168.40.2/TempShare   29439884    1921524   27518360   7% /mnt/pve/TempShare
tmpfs                       2454640          0    2454640   0% /run/user/0
/dev/fuse                    131072         36     131036   1% /etc/pve

t.lamprecht · Jul 13, 2021

So clearly no space left as a normal df call would show: df /

You need to cleanup some files to advance, if your lucky only some log file went crazy and takes up lots of space, those could be deleted relatively safely.

What I do in such cases is using "du" (disk usage) tool to find out the directories with the biggest usages, I normally start on root, tell du to not cross into other file-systems (-x option), output human-readable units (-h), and only report to 1 directory depth -level (-d1). e.g., first du -hxd1 / | sort -h and then if that shows /var as biggest user I use that as new start point: du -hxd1 /var/ | sort -h

If you're unsure about that I'd recommend posting the following output here, so we can help

Bash:

du -hxd2 / | sort -h | tail -20

(may need quite a while depending on IO speed)

t.lamprecht · Jul 13, 2021

manjotsc said:
Found the problem /dev/mapper/pve-root was full in storage. But still can't login I am getting "

Yeah, that's the root /, as indicated by pve6to7

"was" as in "I made some space"?

manjotsc · Jul 13, 2021

@t.lamprecht @Nefariousparity Thanks. It was /root filled NVIDIA driver files, as was trying pass gpu lxc month back. Forgot to cleanup. Any idea about login issue?

t.lamprecht · Jul 13, 2021

I think they failed to start during an upgrade.

Check: systemctl list-units --failed

I'd guess that you need to restart at least pveproxy and pveproxy service:
systemctl try-reload-or-restart pveproxy.service pvedaemon.service

There may be other failed services requiring a restart too though.

manjotsc · Jul 13, 2021

@t.lamprecht This is what it's showing

Code:

root@vms:~# systemctl list-units --failed
  UNIT                     LOAD   ACTIVE SUB    DESCRIPTION
● dcservice.service        loaded failed failed Service for ManageEngine UEMS Agent
● nfs-blkmap.service       loaded failed failed pNFS block layout mapping daemon
● pve-daily-update.service loaded failed failed Daily PVE download activities
● pvesr.service            loaded failed failed Proxmox VE replication runner

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
4 loaded units listed.

Code:

root@vms:~# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-06-26 18:27:24 EDT; 2 weeks 2 days ago
    Process: 2825741 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
   Main PID: 1905 (pveproxy)
      Tasks: 4 (limit: 28618)
     Memory: 246.6M
     CGroup: /system.slice/pveproxy.service
             ├─   1905 pveproxy
             ├─2911473 pveproxy worker
             ├─2911504 pveproxy worker
             └─2913851 pveproxy worker

Jul 13 02:18:56 vms pveproxy[1905]: worker 2911504 started
Jul 13 02:19:26 vms pveproxy[2911443]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1795.
Jul 13 02:19:26 vms pveproxy[2911443]: error writing access log
Jul 13 02:26:31 vms pveproxy[2911443]: worker exit
Jul 13 02:26:31 vms pveproxy[1905]: worker 2911443 finished
Jul 13 02:26:31 vms pveproxy[1905]: starting 1 worker(s)
Jul 13 02:26:31 vms pveproxy[1905]: worker 2913851 started
Jul 13 02:37:03 vms pveproxy[2913851]: proxy detected vanished client connection
Jul 13 02:39:30 vms pveproxy[2911504]: proxy detected vanished client connection
Jul 13 02:47:44 vms pveproxy[2911473]: proxy detected vanished client connection

Code:

root@vms:~# systemctl status pvesr.service
● pvesr.service - Proxmox VE replication runner
     Loaded: loaded (/lib/systemd/system/pvesr.service; static)
     Active: failed (Result: timeout) since Tue 2021-07-13 02:43:15 EDT; 13min ago
TriggeredBy: ● pvesr.timer
   Main PID: 2832705
      Tasks: 1 (limit: 28618)
     Memory: 68.0M
     CGroup: /system.slice/pvesr.service
             └─2832705 /usr/bin/perl -T /usr/bin/pvesr run --mail 1

Jul 13 00:28:00 vms systemd[1]: Starting Proxmox VE replication runner...
Jul 13 02:38:45 vms systemd[1]: pvesr.service: State 'stop-sigterm' timed out. Killing.
Jul 13 02:38:45 vms systemd[1]: pvesr.service: Killing process 2832705 (pvesr) with signal SIGKILL.
Jul 13 02:40:15 vms systemd[1]: pvesr.service: Processes still around after SIGKILL. Ignoring.
Jul 13 02:41:45 vms systemd[1]: pvesr.service: State 'final-sigterm' timed out. Killing.
Jul 13 02:41:45 vms systemd[1]: pvesr.service: Killing process 2832705 (pvesr) with signal SIGKILL.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Processes still around after final SIGKILL. Entering failed mode.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Failed with result 'timeout'.
Jul 13 02:43:15 vms systemd[1]: pvesr.service: Unit process 2832705 (pvesr) remains running after unit stopped.
Jul 13 02:43:15 vms systemd[1]: Stopped Proxmox VE replication runner.
root@vms:~# systemctl start pvesr.service
Failed to start pvesr.service: Transaction for pvesr.service/start is destructive (cryptsetup.target has 'stop' job queued, but 'start' is included in transaction).
See system logs and 'systemctl status pvesr.service' for details.
root@vms:~# systemctl restart pvesr.service

t.lamprecht · Jul 13, 2021

manjotsc said:
systemctl start pvesr.service

pvesr (storage replication) is managed through the pvesr.timer unit, so it cannot be started directly - for it you can just reset the failed state (systemctl reset-failed pvesr.service.

manjotsc said:
Jul 13 02:18:56 vms pveproxy[1905]: worker 2911504 started
Jul 13 02:19:26 vms pveproxy[2911443]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1795.
Jul 13 02:19:26 vms pveproxy[2911443]: error writing access log
Jul 13 02:26:31 vms pveproxy[2911443]: worker exit

The pveproxy is the main API daemon, and it was definitively affected when space run out, does it work now after a full reload?

Else I'd recommend restarting pveproxy.service and pvedaemon.service via systemctl.

manjotsc · Jul 15, 2021

@t.lamprecht I was not able to reboot or shutdown the Proxmox server from terminal, It kept waiting for something to stop or something for hours. Then finally I hard rebooted it via power button(physical-one). Then Proxmox went into emergency mode, then I ran apt update and apt upgrade and reboot it and It's working now, all VMS and LXCs are up.

t.lamprecht · Jul 15, 2021

There were probably half finished upgrades applied when your host ran out of space, so repeating the apt update and apt full-upgrade (avoid a plain upgrade command, especially on major upgrades) was a good call.

Glad to hear that all is up again.

Search

Search

[SOLVED] Proxmox 7.0 Upgrade Errors

manjotsc

Active Member

t.lamprecht

Proxmox Staff Member

Nefariousparity

Member

t.lamprecht

Proxmox Staff Member

manjotsc

Active Member

manjotsc

Active Member

t.lamprecht

Proxmox Staff Member

t.lamprecht

Proxmox Staff Member

manjotsc

Active Member

t.lamprecht

Proxmox Staff Member

manjotsc

Active Member

t.lamprecht

Proxmox Staff Member

manjotsc

Active Member

t.lamprecht

Proxmox Staff Member