[SOLVED] Proxmox disconnecting from itself

[ConnorMcF]

Hi there, my Proxmox instance keeps disconnecting from itself (although it isn't the red icon, unknown?). It is setup as a single instance, not a cluster.



I've tried restarting services but the command just hangs when I attempt it in this state. A reboot resolves the problem temporarily but it occurs again a few hours later.

If there are any log files that could help figure this out, please let me know.

Best,
Connor

 

[Stoiko Ivanov]

* Check the system's journal (`journalctl -r` for the journal in reverse order)
* Is pvestatd running and without errors?

I hope this helps!

 

[ConnorMcF]

Hi, thanks for the reply! I had to wait a bit for this to happen again as I'd rebooted and I don't seem to see anything in particular in the logs.

● pvestatd.service - PVE Status Daemon
   Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-03-28 16:54:47 GMT; 2 days ago
  Process: 1492 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
 Main PID: 1506 (pvestatd)
    Tasks: 2 (limit: 4915)
   Memory: 99.0M
   CGroup: /system.slice/pvestatd.service
           ├─1506 pvestatd
           └─9225 /sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count

The only thing I see in the logs around the time of when this happened was this, it seemed to happen when I tried to add an external IP to a VM (possibly relevant?).

Mar 31 01:19:35 ix1vm01 pveproxy[1535]: worker 24928 started
Mar 31 01:19:35 ix1vm01 pveproxy[1535]: starting 1 worker(s)
Mar 31 01:19:35 ix1vm01 pveproxy[1535]: worker 19394 finished
Mar 31 01:19:35 ix1vm01 pveproxy[19394]: worker exit

Best,
Connor

 

[Stoiko Ivanov]

hmm - on a hunch - if it happens again - how long does the following command take:

/sbin/vgs --separator : --noheadings --units b --unbuffered --nosuffix --options vg_name,vg_size,vg_free,lv_count

might be that the vgs command hangs and this keeps pvestatd from updating everything else.

 

[ConnorMcF]

Yeah, looks like the command just hangs indefinitely on /dev/sdb.

stat("/dev/loop6", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0x6), ...}) = 0
openat(AT_FDCWD, "/dev/loop6", O_RDONLY|O_DIRECT|O_NOATIME) = 3
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0x6), ...}) = 0
ioctl(3, BLKGETSIZE64, [0])             = 0
close(3)                                = 0
stat("/dev/loop7", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0x7), ...}) = 0
openat(AT_FDCWD, "/dev/loop7", O_RDONLY|O_DIRECT|O_NOATIME) = 3
fstat(3, {st_mode=S_IFBLK|0660, st_rdev=makedev(0x7, 0x7), ...}) = 0
ioctl(3, BLKGETSIZE64, [0])             = 0
close(3)                                = 0
stat("/dev/sdb", {st_mode=S_IFBLK|0660, st_rdev=makedev(0x8, 0x10), ...}) = 0
openat(AT_FDCWD, "/dev/sdb", O_RDONLY|O_DIRECT|O_NOATIME

 

[Stoiko Ivanov]

hmm - that's odd - any messages in `dmesg`?
does running `udevadm trigger` change anything?

 

[ConnorMcF]

I don't see anything in dmesg and udevadm trigger didn't seem to help.

I've realised what /dev/sdb is though, it's a little USB stick which is probably dead or something. Perhaps it's trying to query it but it's just too far gone which is causing it to hang?

root@ix1vm01:~# udevadm info --query=all -n /dev/sdb | grep ID_BUS
E: ID_BUS=usb

 

[Stoiko Ivanov]

hmm - do you have anything important on that stick?
otherwise if you have physical access to the box you could simply unplug it
if you don't have access you can try to exclude it from being checked by the lvm-utils - edit '/etc/lvm/lvm.conf' (look for global_filter (and the explanation in the lines above it)

I hope this helps!

 

[ConnorMcF]

I don't have physical access right now but I was able to disable USB from the BIOS until I'm able to access it next.

Thanks for the help!