Broken Pipe - Cannot Access Server

[jeff.lariscy]

I have several servers in a cluster and one is no longer accessible via the GUI. It has a red circle with an X in it instead of the usual green circle with the check. When you click on it, a message pops up: Connection error 596: Broken pipe. We have had some power outages of late, but I think my UPS has held. No other configuration changes have been affected prior to the issue. The VMs appear to be running without issue, just inaccessible for maintenance. Any assistance would be greatly appreciated.

Respectfully,

Jeff Lariscy

 

[mariol]

Is the node with the red circle and the X still accessible somehow, i.e. when you call the WebUI directly or via SSH? Is a simple "ping" working? How is the cluster status [1] ?

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_quorum

 

[jeff.lariscy]

I was able to ping it and access the VMs. This morning I shut down the VMs and restarted the host and it came back. The message on the screen prior to reboot indicated that the journal was not clean and that it was mounting "read only". The reboot seems to have corrected all of the issues.

Thank you for your response!

Respectfully,

Jeff Lariscy

 

[jeff.lariscy]

Looks like I spoke too soon. The system came up and became available again but then went "offline" again. The VMs are working. On the screen of the console it stated:

[ 188.178167] EXT4-fs (dm-1): Delayed block allocation failed for inode 3804527 at logical offset 7 with max blocks 5 with error 30
[ 188.178237] EXT4-fs error (device dm-1) in ext4_do_writepages:2692: IO failure
[ 188.178806] EXT4-fs (dm-1): Remounting filesystem read-only
[ 188.181404] EXT4-fs (dm-1): ext4_do_writepages: jbd2_start: 13239 pages, ino 3804508; err -5

 

[jeff.lariscy]

I can also no longer connect to it via SSH/Putty.

 

[jeff.lariscy]

Finally, to note, commands at the console (shutdown, ls, etc.) are being responded to with the following message:

Input/output error.

 

[JFLariscy]

Looks like the root/boot partition is dying/dead. I can't issue any typical shell commands (has it fallen back to another shell?). The help command brings up a number of available commands but not the "normal" shell commands (e.g. shutdown, ls, mount, etc.).

 

[mariol]

Looks like there is something broken here. Please check your local storage drives with smartctl [1] and also your filesystem with fsck [2]. The file system can not be mounted during such a fsck check. To do this, please boot into rescue mode (if this still works) or use a live-system of your choice to run smartctl and fsck. Also check SATA/SAS cable and controller.

[1] https://linux.die.net/man/8/smartctl
[2] https://wiki.archlinux.org/title/Fsck