Lot of Crash cpus stall and filesystem Remounted read-only

[azkamstf]

i experiencing a strange behaviour on my proxmox, got stall CPUs task on rcu and disk on some VMs got mounted on Read-Only. i got no clue why this happen. I/O delay is quite high too. please someone give me a clue whats happening on my proxmox

Hardware : Oracle Server X7-2
Proxmox Version : 5.4-15

im using SAN storage and all drive is on a good condition

im new to this sysadmin stuff, my senior just left me with this. i didnt get a time to upgrade cause all VM on this server are production.

*im new to this forum

 

[esi_y]

I think you should check your drive and also edit the title to reflect your symptoms. Also provide information about the hardware.

 

[azkamstf]

I think you should check your drive and also edit the title to reflect your symptoms. Also provide information about the hardware.

im sorry, i provide more info for you to look out. thanks

 

[waltar]

Drive sda with ext4 has write errors and so goes readonly by kernel --> sda must be changed.
Is sda a virtual drive from a hw-raid1 or a single drive - see eg. with "parted -l" ?

 

[azkamstf]

Drive sda with ext4 has write errors and so goes readonly by kernel --> sda must be changed.
Is sda a virtual drive from a hw-raid1 or a single drive - see eg. with "parted -l" ?

its a raid from my SAN storage. i check on my DELL EMC there is no error on a drive or raid whatsoever.

 

[waltar]

parted -l ?

 

[azkamstf]

parted -l ?

root@px5:~# parted -l
Model: AVAGO MR9361-16i (scsi)
Disk /dev/sda: 3597GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2097kB  1049kB                     bios_grub
 2      2097kB  271MB   268MB   fat32              boot, esp
 3      271MB   3597GB  3597GB                     lvm


Model: DGC VRAID (scsi)
Disk /dev/sdc: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  7688GB  7688GB  ext4         primary


Model: DGC VRAID (scsi)
Disk /dev/sdd: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  11.5TB  11.5TB  ext4         primary


Error: /dev/sdf: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdf: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data: 3434GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  577MB   576MB   primary               boot
 2      577MB   64.4GB  63.8GB  primary


Model: Linux device-mapper (thin-pool) (dm)
Disk /dev/mapper/pve-data-tpool: 3434GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  577MB   576MB   primary               boot
 2      577MB   64.4GB  63.8GB  primary


Error: /dev/mapper/pve-data_tmeta: unrecognised disk label
Model: Linux device-mapper (linear) (dm)                                 
Disk /dev/mapper/pve-data_tmeta: 17.0GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data_tdata: 3434GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  577MB   576MB   primary               boot
 2      577MB   64.4GB  63.8GB  primary


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathd-part1: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  11.5TB  11.5TB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathh-part1: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  66.0TB  66.0TB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathg-part1: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  44.0TB  44.0TB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathc-part1: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  7688GB  7688GB  ext4


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathg: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  44.0TB  44.0TB  ext4         primary


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathh: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  66.0TB  66.0TB  ext4         primary


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathd: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  11.5TB  11.5TB  ext4         primary


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathc: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  7688GB  7688GB  ext4         primary


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-root: 103GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End    Size   File system  Flags
 1      0.00B  103GB  103GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-swap: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system     Flags
 1      0.00B  8590MB  8590MB  linux-swap(v1)


Error: /dev/sdy: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdy: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Unknown (unknown)
Disk /dev/nvme0n1: 6401GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6401GB  6401GB  btrfs


Model: DGC VRAID (scsi)
Disk /dev/sdo: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  7688GB  7688GB  ext4         primary


Error: /dev/sdm: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdm: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/sdu: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdu: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/sds: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sds: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/sdi: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdi: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Kingston XS2000 (scsi)
Disk /dev/sdz: 4097GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  4097GB  4097GB  ext4


Error: /dev/sdg: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdg: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: DGC VRAID (scsi)
Disk /dev/sdx: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  44.0TB  44.0TB  ext4         primary


Model: DGC VRAID (scsi)
Disk /dev/sdv: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  66.0TB  66.0TB  ext4         primary


Model: DGC VRAID (scsi)
Disk /dev/sdl: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  44.0TB  44.0TB  ext4         primary


Model: Unknown (unknown)
Disk /dev/nvme1n1: 6401GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6401GB  6401GB  btrfs


Model: DGC VRAID (scsi)
Disk /dev/sdj: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  66.0TB  66.0TB  ext4         primary


Error: /dev/sdr: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdr: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: DGC VRAID (scsi)
Disk /dev/sdp: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  11.5TB  11.5TB  ext4         primary

 

[waltar]

So sda is a virtual 4TB drive on internal LSI raid controller (mostly a raid1) with the pve installed on.
Do you have "storcli" (oder "MegaCli") installed on ? --> google storcli download, download from broadcom site,
unpack the download, go into the ubuntu dir and apt install <release>.deb
storcli /call show

 

[waltar]

You haven't got much time to consider long as with readonly sda your internal raid disks (assume raid1) are both defect and once again booting could fail until now !! With storcli you could find out which internal disk is more damaged and you could let blinking it. Exchange it online and wait for raid rebuild. After that exchange the other disk online and hopefully your very old pve installation is running on.
Otherwise you need new disks and a complete new pve installation with recovering all your vm and lxc configurations after to run further your virtual machines laying onto your emc.