Cannot Access Web Interface - I hope that the data will still be saved

elipso · Dec 1, 2023

Hi there after power outage, cannot access web interface only SSH,

ERR_TIMED_OUT on 192.168.1.2

Here is journalctl -f (dont know if helps)

bbgeek17 · Dec 1, 2023

What does "mount" show?
What about "systemctl |grep pve" ?
How about "systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler" and then "journalctl -n 100" ?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

elipso · Dec 1, 2023

bbgeek17 said:
What does "mount" show?
What about "systemctl |grep pve" ?
How about "systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler" and then "journalctl -n 100" ?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Hi, here is:

Code:

root@proxmox:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32869640k,nr_inodes=8217410,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6580924k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=62619)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6580920k,nr_inodes=1645230,mode=700,inode64)
root@proxmox:~#

Here is systemctl |grep pve:

Code:

root@proxmox:~# systemctl |grep pve
● pve-cluster.service                                                                              loaded failed failed    The Proxmox VE cluster filesystem
● pve-firewall.service                                                                             loaded failed failed    Proxmox VE firewall
● pve-guests.service                                                                               loaded failed failed    PVE guests
● pve-ha-crm.service                                                                               loaded failed failed    PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service                                                                               loaded failed failed    PVE Local HA Resource Manager Daemon
  pve-lxc-syscalld.service                                                                         loaded active running   Proxmox VE LXC Syscall Daemon
  pvebanner.service                                                                                loaded active exited    Proxmox VE Login Banner
  pvedaemon.service                                                                                loaded active running   PVE API Daemon
  pvefw-logger.service                                                                             loaded active running   Proxmox VE firewall logger
  pvenetcommit.service                                                                             loaded active exited    Commit Proxmox VE network changes
  pveproxy.service                                                                                 loaded active running   PVE API Proxy Server
● pvescheduler.service                                                                             loaded failed failed    Proxmox VE scheduler
● pvestatd.service                                                                                 loaded failed failed    PVE Status Daemon
  dev-pve-swap.swap                                                                                loaded active active    /dev/pve/swap
  pve-storage.target                                                                               loaded active active    PVE Storage Target
  pve-daily-update.timer                                                                           loaded active waiting   Daily PVE download activities
root@proxmox:~#

And after systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler

bbgeek17 · Dec 1, 2023

Start by going through each failed service, ie:
systemctl status pvestatd
https://photobyte.org/troubleshooting-systemd-services/

Go back to logs from the boot time and analyze them: journalctl -b0

As you saw in systemctl output - things are not great.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

elipso · Dec 1, 2023

bbgeek17 said:
Start by going through each failed service, ie:
systemctl status pvestatd
https://photobyte.org/troubleshooting-systemd-services/

Go back to logs from the boot time and analyze them: journalctl -b0

As you saw in systemctl output - things are not great.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I fixed part of this issue with the following:

Code:

sqlite> .mode insert
sqlite> .output config.sql
sqlite> .dump
sqlite> .exit

root@proxmox:~# mv /var/lib/pve-cluster/config.db config.db.original
root@proxmox:~# sqlite3 /var/lib/pve-cluster/config.db < config.sql

root@proxmox:~# sqlite3 /var/lib/pve-cluster/config.db
sqlite> analyze;
sqlite> PRAGMA integrity_check;
ok

Now is web live. But there missing LVM-Thin disk and VM

elipso · Dec 1, 2023

Code:

root@proxmox:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32869660k,nr_inodes=8217415,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6580928k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=60538)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6580924k,nr_inodes=1645231,mode=700,inode64)
root@proxmox:~#

Here is: journalctl -n 100

Code:

root@proxmox:~# systemctl |grep pve
  etc-pve.mount                                                                                    loaded active     mounted   /etc/pve
  pve-cluster.service                                                                              loaded active     running   The Proxmox VE cluster filesystem
  pve-firewall.service                                                                             loaded active     running   Proxmox VE firewall
  pve-guests.service                                                                               loaded active     exited    PVE guests
  pve-ha-crm.service                                                                               loaded active     running   PVE Cluster HA Resource Manager Daemon
  pve-ha-lrm.service                                                                               loaded active     running   PVE Local HA Resource Manager Daemon
  pve-lxc-syscalld.service                                                                         loaded active     running   Proxmox VE LXC Syscall Daemon
  pvebanner.service                                                                                loaded active     exited    Proxmox VE Login Banner
  pvedaemon.service                                                                                loaded active     running   PVE API Daemon
  pvefw-logger.service                                                                             loaded active     running   Proxmox VE firewall logger
  pvenetcommit.service                                                                             loaded active     exited    Commit Proxmox VE network changes
  pveproxy.service                                                                                 loaded active     running   PVE API Proxy Server
  pvescheduler.service                                                                             loaded active     running   Proxmox VE scheduler
  pvestatd.service                                                                                 loaded active     running   PVE Status Daemon
  dev-pve-swap.swap                                                                                loaded active     active    /dev/pve/swap
  pve-storage.target                                                                               loaded active     active    PVE Storage Target
  pve-daily-update.timer                                                                           loaded active     waiting   Daily PVE download activities
root@proxmox:~#

Looks like VM are still on Disks, how to enable LVM-Thin disk and VM ?

elipso · Dec 1, 2023

Now added that disk but dont know if is right, and how enable VM

bbgeek17 · Dec 1, 2023

you will need to recreate VM config and point the disks to the one that already exist. Be careful to avoid overwriting or deleting the disks accidentally.
Make a backup.

You can use either UI or "qm" CLI to create VM.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

elipso · Dec 1, 2023

bbgeek17 said:
you will need to recreate VM config and point the disks to the one that already exist. Be careful to avoid overwriting or deleting the disks accidentally.
Make a backup.

You can use either UI or "qm" CLI to create VM.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Would you be able to show me some instructions, and especially how to backup safely.

elipso · Dec 1, 2023

I think there is missing ID 102

bbgeek17 · Dec 1, 2023

Your current disks are raw block LVM slices. Given that you are missing half the config required for PVE backup, your other choice is to "dd"/clone disk to another disk, dd/clone each LVM slice on its own, or find some other 3rd party backup program capable of handling LVM.

After that just create an appropriate VM, ie: qm create 100
Then scan the disks in qm disk rescan 100
The disks will be added as unknwon, you can use UI to assign them to scsi/ide/etc.
If you had any custom configuration in VM, you will need to add it as well.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

elipso · Dec 1, 2023

bbgeek17 said:
Your current disks are raw block LVM slices. Given that you are missing half the config required for PVE backup, your other choice is to "dd"/clone disk to another disk, dd/clone each LVM slice on its own, or find some other 3rd party backup program capable of handling LVM.

After that just create an appropriate VM, ie: qm create 100
Then scan the disks in qm disk rescan 100
The disks will be added as unknwon, you can use UI to assign them to scsi/ide/etc.
If you had any custom configuration in VM, you will need to add it as well.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Ok i create qm create 105

Code:

root@proxmox:~# qm disk rescan 105
400 too many arguments
qm disk rescan  [OPTIONS]
root@proxmox:~#

bbgeek17 · Dec 1, 2023

man qm
/ disk rescan

Code:

 qm disk rescan [OPTIONS]

       Rescan all storages and update disk sizes and unused disk images.

       --dryrun <boolean> (default = 0)
           Do not actually write changes out to VM config(s).

       --vmid <integer> (1 - N)
           The (unique) ID of the VM.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

elipso · Dec 1, 2023

bbgeek17 said:
man qm
/ disk rescan

Code:

qm disk rescan [OPTIONS] Rescan all storages and update disk sizes and unused disk images. --dryrun <boolean> (default = 0) Do not actually write changes out to VM config(s). --vmid <integer> (1 - N) The (unique) ID of the VM.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Code:

root@proxmox:~# qm disk rescan --vmid 105
rescan volumes...
VM 105 add unreferenced volume 'local-lvm:vm-105-disk-0' as 'unused0' to config
root@proxmox:~#

Says no bootable device

bbgeek17 · Dec 1, 2023

bbgeek17 said:
The disks will be added as unknwon, you can use UI to assign them to scsi/ide/etc.

did you do this? Look at your VM config. May be you didnt add boot device? May be you didnt set "boot order". Keep working at it.

Have a good weekend

Code:

man qm|grep boot
       --boot [[legacy=]<[acdn]{1,4}>] [,order=<device[;device...]>]
           Specify guest boot order. Use the order= sub-property as usage with no key or legacy= is deprecated.
       --bootdisk (ide|sata|scsi|virtio)\d+
           Enable booting from specified disk. Deprecated: Use boot: order=foo;bar instead.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

elipso · Dec 3, 2023

Some news:

I must go to Options Boot Order and disable and again enable boot device, then start working.

After system reboot kernel gut stuck and must choose older one with proxmox-boot-tool

Next:

I don't know why but vm-102-disk-0 raw is missing. I already say goodbye to them.

Next:

When i click on shell says: Unable to open Shell : Connection failed (Error 500: Timeout while waiting for port '5900' to get ready!)

There is missing 127.0.0.1 localhost inside /etc/hosts

Next:

I get some error in Syslog with:

Code:

[status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/pve/VMs: -1

Fixed with:

Code:

cd /var/lib/
systemctl stop rrdcached
mv rrdcached rrdcached.bck
systemctl start rrdcached
systemctl restart pve-cluster

Search

Search

Cannot Access Web Interface - I hope that the data will still be saved

elipso

New Member

bbgeek17

Distinguished Member

elipso

New Member

bbgeek17

Distinguished Member

elipso

New Member

elipso

New Member

elipso

New Member

bbgeek17

Distinguished Member

elipso

New Member

elipso

New Member

bbgeek17

Distinguished Member

elipso

New Member

bbgeek17

Distinguished Member

elipso

New Member

bbgeek17

Distinguished Member

elipso

New Member