[SOLVED] All guests and node show grey question mark

ResumeMan

New Member
Oct 31, 2024
2
0
1
Hello,

I'm pretty new to Proxmox. I've been using Linux on my own for a number of years, so I have the lay of the land but am certainly not an expert. This pertains to an informal home network. Single node, nothing fancy.

A month or so ago I bought a mini PC to complement my home-made NAS and serve as an app server (Plex, Subsonic clone, bittorrent, etc.). I ended up installing PVE (and later PBS) and fired up a bunch of guests. Even learned about Cloudflare tunnels and set up some of those. Overall it was going very well - till last night/this morning.

I've also recently been having trouble with my network, and ran into connection/DNS issues. Anyway, last night everything, net and PVE, seemed to be working fine. This morning I woke up to no internet. I rolled back my router/network setup to a couple steps back and got the internet working again. But now the node and all the guests, instead of having a green indicator and host name, have a grey question mark. (This has come up elsewhere, but none of the threads I saw really seem to match my situation). I've tried rebooting a couple times, removing and replacing my NFS shares, and a couple other things.

One interesting thing is, in a lot of ways the PVE is still working fine. I can still access most (not all) of the consoles and run commands. The services I have like Plex seem to work and serve up videos. But the thing is clearly not right. (it also takes a looong time to log in to the node)

Based on other things I've seen requested, here's some information.

I tried to run pveversion -v. I swear this worked earlier in the day but today I got
root@prox:~# pveversion -v
thread '<unnamed>' panicked at /usr/share/cargo/registry/proxmox-log-0.2.1/src/lib.rs:53:18:
Unable to open syslog: Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Aborted

Here's the output of journalctl for the last couple hours before everything crapped out. Looks like it seized up at 2:30 in the morning. Interestingly it looks like it's trying to access my gmail account. I *think* I had tried to set up notifications to send to my email, but I never got very far and pretty sure I wasn't trying to do it last night.

Oct 30 01:06:11 prox postfix/smtp[166567]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 01:06:36 prox postfix/smtp[166566]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.27]:25: Connection timed out
Oct 30 01:06:41 prox postfix/smtp[166565]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.27]:25: Connection timed out
Oct 30 01:06:41 prox postfix/smtp[166565]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 01:06:41 prox postfix/smtp[166567]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.27]:25: Connection timed out
Oct 30 01:06:41 prox postfix/smtp[166567]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 01:06:41 prox postfix/smtp[166567]: connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:4003:c04::1a]:25: Network is unreachable
Oct 30 01:06:41 prox postfix/smtp[166567]: 4FB3E20AAE: to=<xyz.xyzzy@gmail.com>, relay=none, delay=258853, delays=258787/0.02/66/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:4003:c04::1a]:25: Network is unreachable)
Oct 30 01:06:41 prox postfix/smtp[166564]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.27]:25: Connection timed out
Oct 30 01:07:06 prox postfix/smtp[166566]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.26]:25: Connection timed out
Oct 30 01:07:06 prox postfix/smtp[166566]: B839020B4B: to=<xyz.xyzzy@gmail.com>, relay=none, delay=169025, delays=168934/0.01/91/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.26]:25: Connection timed out)
Oct 30 01:07:11 prox postfix/smtp[166565]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.26]:25: Connection timed out
Oct 30 01:07:11 prox postfix/smtp[166565]: 7739620A12: to=<xyz.xyzzy@gmail.com>, relay=none, delay=248806, delays=248710/0.01/96/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.26]:25: Connection timed out)
Oct 30 01:07:11 prox postfix/smtp[166564]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.26]:25: Connection timed out
Oct 30 01:07:11 prox postfix/smtp[166564]: 3D27720A88: to=<xyz.xyzzy@gmail.com>, relay=none, delay=263033, delays=262937/0.01/96/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.26]:25: Connection timed out)
Oct 30 01:09:01 prox CRON[167817]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 30 01:09:01 prox CRON[167818]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 30 01:09:01 prox CRON[167817]: pam_unix(cron:session): session closed for user root
Oct 30 01:17:01 prox CRON[170831]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 30 01:17:01 prox CRON[170832]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 30 01:17:01 prox CRON[170831]: pam_unix(cron:session): session closed for user root
Oct 30 01:39:01 prox CRON[179063]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 30 01:39:01 prox CRON[179064]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 30 01:39:01 prox CRON[179063]: pam_unix(cron:session): session closed for user root
Oct 30 01:58:26 prox systemd[1]: Starting apt-daily.service - Daily apt download activities...
Oct 30 01:58:26 prox systemd[1]: apt-daily.service: Deactivated successfully.
Oct 30 01:58:26 prox systemd[1]: Finished apt-daily.service - Daily apt download activities.
Oct 30 02:09:01 prox CRON[190281]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 30 02:09:01 prox CRON[190282]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 30 02:09:01 prox CRON[190281]: pam_unix(cron:session): session closed for user root
Oct 30 02:15:35 prox postfix/qmgr[1193]: 3D27720A88: from=<root@pve.ccc>, size=2194, nrcpt=1 (queue active)
Oct 30 02:15:35 prox postfix/qmgr[1193]: 7739620A12: from=<root@pve.ccc>, size=7377, nrcpt=1 (queue active)
Oct 30 02:15:35 prox postfix/qmgr[1193]: B839020B4B: from=<root@pve.ccc>, size=132773, nrcpt=1 (queue active)
Oct 30 02:15:35 prox postfix/qmgr[1193]: 4FB3E20AAE: from=<root@pve.ccc>, size=147724, nrcpt=1 (queue active)
Oct 30 02:15:40 prox postfix/smtp[192922]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c05::1a]:25: Network is unreachable
Oct 30 02:16:05 prox postfix/smtp[192920]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 02:16:05 prox postfix/smtp[192920]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c05::1a]:25: Network is unreachable
Oct 30 02:16:05 prox postfix/smtp[192920]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:16:10 prox postfix/smtp[192921]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 02:16:10 prox postfix/smtp[192921]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c05::1a]:25: Network is unreachable
Oct 30 02:16:10 prox postfix/smtp[192922]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 02:16:10 prox postfix/smtp[192922]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:16:10 prox postfix/smtp[192919]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 02:16:10 prox postfix/smtp[192919]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c05::1a]:25: Network is unreachable
Oct 30 02:16:35 prox postfix/smtp[192920]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192921]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192921]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:16:40 prox postfix/smtp[192922]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192922]: connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:4003:c04::1a]:25: Network is unreachable
Oct 30 02:16:40 prox postfix/smtp[192922]: 4FB3E20AAE: to=<xyz.xyzzy@gmail.com>, relay=none, delay=263053, delays=262987/0.02/66/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:4003:c04::1a]:25: Network is unreachable)
Oct 30 02:16:40 prox postfix/smtp[192919]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192919]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:17:01 prox CRON[193541]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 30 02:17:01 prox CRON[193542]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 30 02:17:01 prox CRON[193541]: pam_unix(cron:session): session closed for user root
Oct 30 02:17:06 prox postfix/smtp[192920]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out
Oct 30 02:17:06 prox postfix/smtp[192920]: 7739620A12: to=<xyz.xyzzy@gmail.com>, relay=none, delay=253001, delays=252910/0.01/91/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out)
Oct 30 02:17:10 prox postfix/smtp[192921]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out
Oct 30 02:17:10 prox postfix/smtp[192921]: B839020B4B: to=<xyz.xyzzy@gmail.com>, relay=none, delay=173229, delays=173133/0.02/96/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out)
Oct 30 02:17:10 prox postfix/smtp[192919]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out
Oct 30 02:17:10 prox postfix/smtp[192919]: 3D27720A88: to=<xyz.xyzzy@gmail.com>, relay=none, delay=267233, delays=267137/0.01/96/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out)
Oct 30 02:19:49 prox kernel: _swap_info_get: Bad swap offset entry 3fffffffffeff
Oct 30 02:19:49 prox kernel: _swap_info_get: Bad swap offset entry 3fffffffffef

Here is some of the output of the system log in the Data center. Looks pretty much the same. I'm running into post size limitations so I'll just show the very end of this.
Oct 30 02:16:10 prox postfix/smtp[192922]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 02:16:10 prox postfix/smtp[192922]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:16:10 prox postfix/smtp[192919]: connect to gmail-smtp-in.l.google.com[173.194.203.27]:25: Connection timed out
Oct 30 02:16:10 prox postfix/smtp[192919]: connect to gmail-smtp-in.l.google.com[2607:f8b0:400e:c05::1a]:25: Network is unreachable
Oct 30 02:16:35 prox postfix/smtp[192920]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192921]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192921]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:16:40 prox postfix/smtp[192922]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192922]: connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:4003:c04::1a]:25: Network is unreachable
Oct 30 02:16:40 prox postfix/smtp[192922]: 4FB3E20AAE: to=<xyzzyn.xyzzy@gmail.com>, relay=none, delay=263053, delays=262987/0.02/66/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[2607:f8b0:4003:c04::1a]:25: Network is unreachable)
Oct 30 02:16:40 prox postfix/smtp[192919]: connect to alt1.gmail-smtp-in.l.google.com[142.250.115.26]:25: Connection timed out
Oct 30 02:16:40 prox postfix/smtp[192919]: connect to alt1.gmail-smtp-in.l.google.com[2607:f8b0:4023:1004::1b]:25: Network is unreachable
Oct 30 02:17:01 prox CRON[193541]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 30 02:17:01 prox CRON[193542]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Oct 30 02:17:01 prox CRON[193541]: pam_unix(cron:session): session closed for user root
Oct 30 02:17:06 prox postfix/smtp[192920]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out
Oct 30 02:17:06 prox postfix/smtp[192920]: 7739620A12: to=<xyzzyn.xyzzy@gmail.com>, relay=none, delay=253001, delays=252910/0.01/91/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out)
Oct 30 02:17:10 prox postfix/smtp[192921]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out
Oct 30 02:17:10 prox postfix/smtp[192921]: B839020B4B: to=<xyzzyn.xyzzy@gmail.com>, relay=none, delay=173229, delays=173133/0.02/96/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out)
Oct 30 02:17:10 prox postfix/smtp[192919]: connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out
Oct 30 02:17:10 prox postfix/smtp[192919]: 3D27720A88: to=<xyzzyn.xyzzy@gmail.com>, relay=none, delay=267233, delays=267137/0.01/96/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[108.177.104.27]:25: Connection timed out)
Oct 30 02:19:49 prox kernel: _swap_info_get: Bad swap offset entry 3fffffffffeff
Oct 30 02:19:49 prox kernel: _swap_info_get: Bad swap offset entry 3fffffffffeff

Any insights appreciated. Thanks in advance!
 
OH! OK, I didn't realize I had had anything set up along the lines of notifications. But I went back to the notification guide I had been looking at, and followed it to both the Datacenter/Users and Datacenter/Options spots and deleted my gmail address and - everything seems to be ok now.

Would still like to know what the hell was going on there. I'm pretty sure it was multiple days ago I had put those entries in there, so not sure why it decided to choke last night.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!