[SOLVED] Problem on proxmox web interface

[PaoloHolzl]

I have a cluster of three machines.
Via the web I see the cluster but one of the three machines only shows "?"
I can easily log in into this machine via ssh and the VMs are active regularly.
I'm afraid to restart the machine because if the VMs don't restart the problem becomes serious.

This is end of my syslog ...
Can someone help me?

Nov  2 00:52:10 ProxMox02 smartd[747]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 29 to 30
Nov  2 01:01:18 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
Nov  2 01:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: Error while trying to create /var/lib/rrdcached/journal/rrd.journal.1698884531.869846 : Read-only file system
Nov  2 01:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: All values will be flushed at shutdown
Nov  2 01:52:10 ProxMox02 smartd[747]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 30 to 29
Nov  2 02:09:59 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
Nov  2 02:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: Error while trying to create /var/lib/rrdcached/journal/rrd.journal.1698888131.869804 : Read-only file system
....
system
Nov  2 06:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: All values will be flushed at shutdown
Nov  2 06:43:41 ProxMox02 systemd[1]: Starting Daily apt upgrade and clean activities...
Nov  2 06:43:41 ProxMox02 apt.systemd.daily[3520694]: /usr/lib/apt/apt.systemd.daily: 320: cannot create /var/lib/apt//daily_lock: Read-only file system
Nov  2 06:43:41 ProxMox02 systemd[1]: apt-daily-upgrade.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov  2 06:43:41 ProxMox02 systemd[1]: apt-daily-upgrade.service: Failed with result 'exit-code'.
Nov  2 06:43:41 ProxMox02 systemd[1]: Failed to start Daily apt upgrade and clean activities.
Nov  2 06:45:20 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system
Nov  2 07:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: Error while trying to create /var/lib/rrdcached/journal/rrd.journal.1698906131.869781 : Read-only file system
Nov  2 07:22:11 ProxMox02 rrdcached[979]: JOURNALING DISABLED: All values will be flushed at shutdown
Nov  2 07:54:25 ProxMox02 chronyd[908]: Could not open /var/lib/chrony/chrony.drift.tmp : Read-only file system

 

[mgabriel]

It seems your failed node is not part of the quorum partition of the cluster.

Can you log in to that node with the question mark and provide the output of pvecm status command?

Best regards,
Marco

 

[PaoloHolzl]

When I click on quotation mark the result is Connection refused 595.
under ssh ...

pvecm status
ipcc_send_rec[1] failed: Resource temporarily unavailable
ipcc_send_rec[2] failed: Resource temporarily unavailable
ipcc_send_rec[3] failed: Resource temporarily unavailable
Unable to load access control list: Resource temporarily unavailable

 

[mgabriel]

Is corosync running?

systemctl status corosync.service

 

[PaoloHolzl]

Yes

 systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
     Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-09-14 09:22:13 CEST; 1 months 19 days ago
       Docs: man:corosync
             man:corosync.conf
             man:corosync_overview
   Main PID: 1088 (corosync)
      Tasks: 9 (limit: 38379)
     Memory: 3.0G
        CPU: 8h 8min 31.737s
     CGroup: /system.slice/corosync.service
             └─1088 /usr/sbin/corosync -f

Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] link: Resetting MTU for link 0 because host 3 joined
Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] pmtud: PMTUD link change for host: 3 link: 0 from 469 to 1397
Sep 14 09:22:16 ProxMox02 corosync[1088]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] Sync members[3]: 1 2 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] Sync joined[2]: 1 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [TOTEM ] A new membership (1.46bc) was formed. Members joined: 1 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] This node is within the primary component and will provide service.
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [QUORUM] Members[3]: 1 2 3
Sep 14 09:22:17 ProxMox02 corosync[1088]:   [MAIN  ] Completed service synchronization, ready to provide service.

 

[Maximiliano]

Hello,

The error is that `/var/lib/rrdcached/journal/rrd.journal.1698906131.869781` is in a read-only filesystem. I am not sure what can cause this at the moment. You should check it, chrony also failed on a similar error.

Could you please send us the ouput of `pvecm status` and of `cat /etc/pve/corosync.conf`. Please also check which directory has read-only is it `/var/lib`? Can you check if a reboot helps?

 

[PaoloHolzl]

As all commands into directory /etc/pve freez so cannot do cat

cat /etc/pve/corosync.conf

Have do the same command in an other machine from cluster

Linux ProxMox01 5.15.102-1-pve #1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Fri Sep 29 08:46:48 2023
root@ProxMox01:~# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: ProxMox01
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.64.92
  }
  node {
    name: ProxMox02
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.64.70
  }
  node {
    name: ProxMox03
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.64.2
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: ClusterBondi
  config_version: 3
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

I waith to boot because if restarting VM don't stat my be a problem.
I'm trying to move virtual machine via external drive (internal move freez).

 

[LnxBil]

Please also post the output of the command dmesg. It'll have some points to why your rootfs is read-only. (best to attach a file if it's big)

 

[PaoloHolzl]

After reboot is started correctly.
I think it was an hardware failure.