[SOLVED] Problem on proxmox web interface

PaoloHolzl

New Member
Apr 27, 2023
29
2
3
I have a cluster of three machines.
Via the web I see the cluster but one of the three machines only shows "?"
I can easily log in into this machine via ssh and the VMs are active regularly.
I'm afraid to restart the machine because if the VMs don't restart the problem becomes serious.

This is end of my syslog ...
Can someone help me?
Code:
Nov  2 00:52:10 ProxMox02 smartd[747]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 29 to 30
Nov  2 01:01:18 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
Nov  2 01:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: Error while trying to create /var/lib/rrdcached/journal/rrd.journal.1698884531.869846 : Read-only file system
Nov  2 01:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: All values will be flushed at shutdown
Nov  2 01:52:10 ProxMox02 smartd[747]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 30 to 29
Nov  2 02:09:59 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
Nov  2 02:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: Error while trying to create /var/lib/rrdcached/journal/rrd.journal.1698888131.869804 : Read-only file system
....
system
Nov  2 06:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: All values will be flushed at shutdown
Nov  2 06:43:41 ProxMox02 systemd[1]: Starting Daily apt upgrade and clean activities...
Nov  2 06:43:41 ProxMox02 apt.systemd.daily[3520694]: /usr/lib/apt/apt.systemd.daily: 320: cannot create /var/lib/apt//daily_lock: Read-only file system
Nov  2 06:43:41 ProxMox02 systemd[1]: apt-daily-upgrade.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov  2 06:43:41 ProxMox02 systemd[1]: apt-daily-upgrade.service: Failed with result 'exit-code'.
Nov  2 06:43:41 ProxMox02 systemd[1]: Failed to start Daily apt upgrade and clean activities.
Nov  2 06:45:20 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
Nov  2 07:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: Error while trying to create /var/lib/rrdcached/journal/rrd.journal.1698906131.869781 : Read-only file system
Nov  2 07:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: All values will be flushed at shutdown
Nov  2 07:54:25 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
 
It seems your failed node is not part of the quorum partition of the cluster.

Can you log in to that node with the question mark and provide the output of pvecm status command?

Best regards,
Marco
 
When I click on quotation mark the result is Connection refused 595.
under ssh ...

Code:
pvecm status
ipcc_send_rec[1] failed: Resource temporarily unavailable
ipcc_send_rec[2] failed: Resource temporarily unavailable
ipcc_send_rec[3] failed: Resource temporarily unavailable
Unable to load access control list: Resource temporarily unavailable
 
Yes

Code:
 systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-09-14 09:22:13 CEST; 1 months 19 days ago
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview
   Main PID: 1088 (corosync)
      Tasks: 9 (limit: 38379)
     Memory: 3.0G
        CPU: 8h 8min 31.737s
     CGroup: /system.slice/corosync.service
             └─1088 /usr/sbin/corosync -f

Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] link: Resetting MTU for link 0 because host 3 joined
Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] Sync members[3]: 1 2 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] Sync joined[2]: 1 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [TOTEM ] A new membership (1.46bc) was formed. Members joined: 1 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] This node is within the primary component and will provide service.
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] Members[3]: 1 2 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [MAIN  ] Completed service synchronization, ready to provide service.
 
Hello,

The error is that `/var/lib/rrdcached/journal/rrd.journal.1698906131.869781` is in a read-only filesystem. I am not sure what can cause this at the moment. You should check it, chrony also failed on a similar error.

Could you please send us the ouput of `pvecm status` and of `cat /etc/pve/corosync.conf`. Please also check which directory has read-only is it `/var/lib`? Can you check if a reboot helps?
 
As all commands into directory /etc/pve freez so cannot do cat
Bash:
cat /etc/pve/corosync.conf
Have do the same command in an other machine from cluster

Code:
Linux ProxMox01 5.15.102-1-pve #1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Sep 29 08:46:48 2023
root@ProxMox01:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: ProxMox01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.64.92
  }
  node {
    name: ProxMox02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.64.70
  }
  node {
    name: ProxMox03
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.64.2
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: ClusterBondi
  config_version: 3
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

I waith to boot because if restarting VM don't stat my be a problem.
I'm trying to move virtual machine via external drive (internal move freez).
 
Please also post the output of the command dmesg. It'll have some points to why your rootfs is read-only. (best to attach a file if it's big)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!