Search results

  1. M

    QMP Socket timeout on console, backups etc... randomly

    The issue seems to be more contained to Windows VPS which are acting quite unstable. The VPS itself might be marked as "Started" but it seems to be like a ghost VPS - not accessible or manageable. Setting "KVM Hardware Virtualization" to "No" seems to slightly help the issue and make the time...
  2. M

    Question marks on all nodes but 1 in Proxmox VE 6

    Also, we're seeing this: root@hyp08:~# systemctl status pvestatd â pvestatd.service - PVE Status Daemon Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2019-08-08 18:31:19 CEST; 1 day 3h ago Main PID: 2217 (pvestatd)...
  3. M

    Question marks on all nodes but 1 in Proxmox VE 6

    Hi everyone, We recently installed a cluster with Proxmox VE 6 and restored hundreds of VPS on it. The cluster has 6 virtualization nodes and 4 storage nodes. It was running fine for about 20 hours, no issues at all - all nodes showing up with green checkmarks. All of a sudden, two of the...
  4. M

    Risks of upgrading Proxmox VE

    We have a cluster with a bunch of nodes running the latest Proxmox v4 in production. The services we host on this cluster can't suffer prolonged downtime. A few minutes of downtime (say, for a reboot) is the most we can allow really. We also have a Ceph cluster for storage of these VPS. We...
  5. M

    How do you monitor your Proxmox nodes?

    Hi there, We had some issues recently with processes failing in our Proxmox cluster and us not catching it until a few hours later. We want a monitoring solution that will monitor: Resource usage (CPU/load, RAM, storage (local and shared), network load) Log messages requiring attention Health...
  6. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Thank you all very much for your help. We ended up fixing this by enabling igmp snooping, giving the cluster some time and then following your recommended commands. If anyone finds this on Google, we recommend following the omping command to ensure multicast is set up properly and to kill the...
  7. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Quick note: The only consistent issue that I remember seeing throughout this whole thing is: Jun 19 01:16:18 hyp05 pmxcfs[105036]: [status] notice: cpg_send_message retried 100 times Jun 19 01:16:18 hyp05 pmxcfs[105036]: [status] crit: cpg_send_message failed: 6 Jun 19 01:16:19 hyp05...
  8. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Quick update: Now all services (corosync pve-cluster pveproxy pvedaemon pvestatd) are running on all nodes but still there are red Xs in Proxmox panel. The issues seem to be jumping from one node to the next. omping had perfect results except for 1% loss between hyp11 and nvme03 and between...
  9. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    The omping issue was a silly /etc/hosts file problem that is now fixed. Here's our current situation, no VPS are actually down but the red Xs are still visible when accessing All of the following services are Active on all nodes: corosync pve-cluster pveproxy pvedaemon pvestatd This worked...
  10. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Your reply got me thinking, so I run omping with all hypervisors but also all storage nodes. It seems like hyp11 (one of the hypervisors) and nvme04 (one of the storage nodes) can't communicate over omping: root@nvme04:~# omping -c 10000 -i 0.001 -F -q nvme04 hyp11 hyp11 : waiting for response...
  11. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Output of omping -c 10000 -i 0.001 -F -q hyp06 : unicast, xmt/rcv/%loss = 10000/10000/0%, min/avg/max/std-dev = 0.046/0.129/7.753/0.295 hyp06 : multicast, xmt/rcv/%loss = 10000/9996/0% (seq>=5 0%), min/avg/max/std-dev = 0.052/0.137/7.755/0.297 hyp07 : unicast, xmt/rcv/%loss =...
  12. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    igmp snooping command has been setup on all interfaces that are behind the cluster. Thanks RokaKen. Any ideas on what should be our next move with this rogue node?
  13. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Thanks! The command is not working, instead we must run this: 5.3-LAN01(config-vlan-512)#ip igmp snooping vlan 512 static 239.192.34.12 ? interface Specify interface But which interface should we add after the ip?
  14. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Thanks for your help! Here's what we got: root@hyp06-nl-ams:/etc/corosync# pvecm status pve configuration filesystem not mounted root@hyp06-nl-ams:/etc/corosync# journalctl -r -- Logs begin at Fri 2019-06-14 23:33:58 CEST, end at Tue 2019-06-18 20:04:33 CEST. -- Jun 18 20:04:33...
  15. M

    [SOLVED] Issue with cluster - showing red X'es and possible multicast problems

    Hey everyone, We run a cluster behind 2x Arista DCS-7050T-64-R switches, both in the same vlan (512). They are connected to each other also. The cluster has 9 nodes running Proxmox v4.4 (6 hypervisors, rest storage nodes). We had igmp snooping disabled in full on the vlan with command: no ip...
  16. M

    Possible multicast problems with Proxmox cluster

    Dear Proxmox community, We've been having crazy problems for the past two days with our VPS node cluster. What seems to happen is that at some point, all nodes show up with a red X through the management panel except for 1 node that has the green checkmark. The nodes that have the red X can't...
  17. M

    How would you find the Proxmox VPS that is getting DDoS attacked?

    How would you summarize that dump to something useable? We're seeing this in the /var/log/syslog: kernel: [ 8355.928291] nf_conntrack: table full, dropping packet pmxcfs[3283]: [status] notice: received log We believe that a VPS is getting attacked. Please advise.
  18. M

    How would you find the Proxmox VPS that is getting DDoS attacked?

    Hi there, We have a Proxmox cluster that seems to be getting DDoS attacked. How would you find the VPS that is getting attacked? Any and all help will be appreciated! Thank you.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!