[SOLVED] Deteled fstab

T

T.George

New Member
Jul 31, 2023
10
0
1
Hello guys,

I'm a really noob at Linux/VM and everything related.
Long story short: I ran a Proxmox machine with 3 VMs inside. Today i've got an error regarding lack of space, "error 500 no space left on device".
After some searches and because I'm a total noob, I've ended up with deleting the fstab file...

Now the Proxmox is booting, I can acces it via SSH, but the webGUI is not loading.
Tryed running "systemctl status pveproxy.service", get this:

Code: 
pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-07-31 13:31:32 EEST; 1h 55min ago
    Process: 787 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 788 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 789 (pveproxy)
      Tasks: 4 (limit: 14173)
     Memory: 203.1M
        CPU: 3min 49.359s
     CGroup: /system.slice/pveproxy.service
             ├─ 789 pveproxy
             ├─7018 pveproxy worker
             ├─7019 pveproxy worker
             └─7020 pveproxy worker

Jul 31 15:26:58 proxmox pveproxy[7015]: worker exit
Jul 31 15:26:58 proxmox pveproxy[789]: worker 7015 finished
Jul 31 15:26:58 proxmox pveproxy[789]: starting 1 worker(s)
Jul 31 15:26:58 proxmox pveproxy[789]: worker 7019 started
Jul 31 15:26:58 proxmox pveproxy[7019]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/API>
Jul 31 15:26:58 proxmox pveproxy[7016]: worker exit
Jul 31 15:26:58 proxmox pveproxy[789]: worker 7016 finished
Jul 31 15:26:58 proxmox pveproxy[789]: starting 1 worker(s)
Jul 31 15:26:58 proxmox pveproxy[789]: worker 7020 started
Jul 31 15:26:58 proxmox pveproxy[7020]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/API>

"systemctl status pve-cluster.service" is getting this:

Code: 
pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-07-31 14:13:42 EEST; 30min ago
    Process: 3107 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 9ms

Jul 31 14:13:42 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Jul 31 14:13:42 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 31 14:13:42 proxmox systemd[1]: pve-cluster.service: Start request repeated too quickly.
Jul 31 14:13:42 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 14:13:42 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.

I believe everything is related to that fstab file I deleted..
Also, seems like I cannot acces the backup solution, when I try "pct restore" I get this:

Code: 
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

Any ideas how I can get back my access?
Or I managed to kill it properly and I need to reinstall everything?
 
have you resolved the out-of-space issue since?
What does "df -h" look like?

This is a sample fstab however yours could be different, depending on your selection during installation:
Code: 
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
I'm having a hard time remembering when I screwed things up.

First, it was this command: "for i in /var/lib/vz/dump/*; do rm "$i"; done".
Seem like that folder was taking almoust all the space. From what I read, that folder should only containg temporary backup files and was safe to delete. After that, I ran " rm /etc/fstab". After this, the Proxmox was back again live, at the boot screen, when is shows the IP for webGUI.

If it helps, at the boot, I get all this errors, then the static page in the second picture.
 

Attachments

  • IMG_9252.jpeg
    IMG_9252.jpeg
    58.8 KB · Views: 17
  • IMG_9253.jpeg
    IMG_9253.jpeg
    29.7 KB · Views: 14
Please be careful removing files from which you do not know exactly what they are! For me it looks like you just deleted all your Backups!
 
Please post the output of df -h and lsblk so whe have a better overview about your system

Edit: if you want to see what is using up your memory you can run du / -h |sort -h. However please DO NOT delete anything you are not sure about what it is. If not sure you can always ask here first.
 
Last edited:
Sure.

"df - h":

Code: 
Filesystem            Size  Used Avail Use% Mounted on
udev                  5.8G     0  5.8G   0% /dev
tmpfs                 1.2G   16M  1.2G   2% /run
/dev/mapper/pve-root   94G   20G   70G  22% /
tmpfs                 5.8G     0  5.8G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 1.2G     0  1.2G   0% /run/user/0

"lsblk":

Code: 
Filesystem            Size  Used Avail Use% Mounted on
udev                  5.8G     0  5.8G   0% /dev
tmpfs                 1.2G   16M  1.2G   2% /run
/dev/mapper/pve-root   94G   20G   70G  22% /
tmpfs                 5.8G     0  5.8G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 1.2G     0  1.2G   0% /run/user/0
root@proxmox:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 447.1G  0 disk
|-sda1                         8:1    0  1007K  0 part
|-sda2                         8:2    0   512M  0 part
`-sda3                         8:3    0 446.6G  0 part
  |-pve-swap                 253:0    0     4G  0 lvm 
  |-pve-root                 253:1    0    96G  0 lvm  /
  |-pve-data_tmeta           253:2    0   3.3G  0 lvm 
  | `-pve-data-tpool         253:4    0   324G  0 lvm 
  |   |-pve-data             253:5    0   324G  1 lvm 
  |   |-pve-vm--100--disk--0 253:6    0     4M  0 lvm 
  |   |-pve-vm--100--disk--1 253:7    0    32G  0 lvm 
  |   |-pve-vm--101--disk--0 253:8    0     4M  0 lvm 
  |   |-pve-vm--101--disk--1 253:9    0    64G  0 lvm 
  |   `-pve-vm--103--disk--0 253:10   0    20G  0 lvm 
  `-pve-data_tdata           253:3    0   324G  0 lvm 
    `-pve-data-tpool         253:4    0   324G  0 lvm 
      |-pve-data             253:5    0   324G  1 lvm 
      |-pve-vm--100--disk--0 253:6    0     4M  0 lvm 
      |-pve-vm--100--disk--1 253:7    0    32G  0 lvm 
      |-pve-vm--101--disk--0 253:8    0     4M  0 lvm 
      |-pve-vm--101--disk--1 253:9    0    64G  0 lvm 
      `-pve-vm--103--disk--0 253:10   0    20G  0 lvm 
sdb                            8:16   0 931.5G  0 disk
`-sdb1                         8:17   0 931.5G  0 part

sba - the 512GB which holds Proxmox and all the configurations
sdb - 1TB HDD for storage, the backups should also be there, but I think there were not, given the initial error
 
Does pvecm updatecerts --force fix your pve-ssl.key issue?
Did you already regenerate the /etc/fstab?
 
Last edited:
No, I did not, because I have no ideea how to put it back.
If I use "ls" inside the /etc folder, I do not see any fstab there.

Running "pvecm updatecerts --force" gets this:

Code: 
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
 
Last edited:
Sorry I did not say. You have to execute it as root.

However first, since your installation seems to be the default ext4 installation I think you should copy the code bbgeek17 suggested to your /etc/fstab
bbgeek17 said:
This is a sample fstab however yours could be different, depending on your selection during installation:
Code: 
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

Click to expand...

Just execute (as root) nano /etc/fstab, copy it in and then execute mount -a

Did that work?
 
No, I get: "[ Error writing /etc/fstab: Read-only file system ]".
I'm loged in as root, at least so I think. Can I verify that?
 
T.George said:
ipcc_send_rec[1] failed: Connection refused ipcc_send_rec[2] failed: Connection refused ipcc_send_rec[3] failed: Connection refused
Click to expand...
Seems like the proxmox cluster filesystem is not running. Please post the output of journalctl -b -u pve-cluster.service Did you maybe delete also the file located under /var/lib/pve-cluster? Because that is where the pmxcfs backing sqlite DB is located.
 
Hi Chris,

Thanks for joining. I think the pve-cluster it's ok. If I cd into id, i can see a file called "config.db" there.
Below the resulst from running "journalctl -b -u pve-cluster.service":

Code: 
-- Journal begins at Sun 2022-10-23 11:44:02 EEST, ends at Mon 2023-07-31 16:35:24 EEST. --
Jul 31 15:43:55 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 31 15:43:55 proxmox pmxcfs[767]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:55 proxmox pmxcfs[767]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:55 proxmox pmxcfs[767]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:55 proxmox pmxcfs[767]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:55 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 31 15:43:55 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 15:43:55 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jul 31 15:43:55 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1.
Jul 31 15:43:55 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 31 15:43:56 proxmox pmxcfs[775]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[775]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[775]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox pmxcfs[775]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 15:43:56 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 2.
Jul 31 15:43:56 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 31 15:43:56 proxmox pmxcfs[776]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[776]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[776]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox pmxcfs[776]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 15:43:56 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 3.
Jul 31 15:43:56 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 31 15:43:56 proxmox pmxcfs[777]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[777]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[777]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox pmxcfs[777]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 15:43:56 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 4.
Jul 31 15:43:56 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 31 15:43:56 proxmox systemd[1]: Starting The Proxmox VE cluster filesystem...
Jul 31 15:43:56 proxmox pmxcfs[778]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[778]: [main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
Jul 31 15:43:56 proxmox pmxcfs[778]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox pmxcfs[778]: [main] notice: exit proxmox configuration filesystem (-1)
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
Jul 31 15:43:56 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 15:43:56 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Jul 31 15:43:57 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Jul 31 15:43:57 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 31 15:43:57 proxmox systemd[1]: pve-cluster.service: Start request repeated too quickly.
Jul 31 15:43:57 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 31 15:43:57 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
 
you can check it with whoami but probably your issue is that your system mounts your / as read only (since there is no /etc/fstab that specifies otherwise :) )

But lets confirm the suspicion by looking at the output of mount
 
  • Like
Reactions: Chris
Okay, so the DB is still there, that is good news. As @Philipp Hufnagl pointed out correctly, the issue is that the pmxcfs service cannot create the required lock file as the whole root filesystem seems to be mounted read-only.

Edit: Try running an fsck from a recovery stick, maybe you can share the full journal since boot to get a better understanding on why the filesystem is mounted as read-only (journalctl -b > journal.txt and attach the file here).
 
Last edited:
"whoami" returns "root", so I believe I'm good. I also do not have any other user. :)

From "mount" I get this.
I also assume that all the problems are from that missing fstab. I think it stores some important stuff and I'm stupid enough to delete it. :))

Code: 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=6047460k,nr_inodes=1511865,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1216236k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (ro,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17285)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1216232k,nr_inodes=304058,mode=700,inode64)
 
T.George said:
Code: 
/dev/mapper/pve-root on / type ext4 (ro,relatime)
Click to expand...
Yes. That confirms our suspicion :)

Lets remount it as rw with mount -o remount,rw /dev/mapper/pve-root / and then write a new /etc/fstab

Edit: This might also already fix your key issue. So reboot after and see if that is fixed too before you update certs
 
Last edited:
Ok. Now it worked, I created the fstab using nano.
After that, I ran "mount -a". Should something appearead? Because nothing happened, just waiting for a new comand.
 
T.George said:
I also assume that all the problems are from that missing fstab. I think it stores some important stuff and I'm stupid enough to delete it. :))
Click to expand...
The /etc/fstab tells your system how it should mount things
 
T.George said:
Ok. Now it worked, I created the fstab using nano.
After that, I ran "mount -a". Should something appearead? Because nothing happened, just waiting for a new comand.
Click to expand...
No. That should be good. I would recommend just rebooting your system at that point
 
Damn, the reboot worked and it's working again. 2 out of 3 VMs escaped, everything seems ok.
The last one it's working, but it contained an Openmediavault server insider, running Docker and Portainer. The configs seems to be lost, as some apps inside them show no data. Need to investigate date.

Now I'm more than happy that I fixed this. I'll investigate now how to properly made a backup on an external drive, just in case.

Thank you so much guys, you saved me. :d
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom