Lot of Crash cpus stall and filesystem Remounted read-only

azkamstf · 2024-09-18T09:40:46+0200

i experiencing a strange behaviour on my proxmox, got stall CPUs task on rcu and disk on some VMs got mounted on Read-Only. i got no clue why this happen. I/O delay is quite high too. please someone give me a clue whats happening on my proxmox

Hardware : Oracle Server X7-2
Proxmox Version : 5.4-15

im using SAN storage and all drive is on a good condition

im new to this sysadmin stuff, my senior just left me with this. i didnt get a time to upgrade cause all VM on this server are production.

*im new to this forum

esi_y · 2024-09-18T11:13:52+0200

I think you should check your drive and also edit the title to reflect your symptoms. Also provide information about the hardware.

azkamstf · 2024-09-18T15:25:29+0200

esi_y said:
I think you should check your drive and also edit the title to reflect your symptoms. Also provide information about the hardware.

im sorry, i provide more info for you to look out. thanks

waltar · 2024-09-18T16:35:43+0200

Drive sda with ext4 has write errors and so goes readonly by kernel --> sda must be changed.
Is sda a virtual drive from a hw-raid1 or a single drive - see eg. with "parted -l" ?

azkamstf · 2024-09-18T17:29:05+0200

waltar said:
Drive sda with ext4 has write errors and so goes readonly by kernel --> sda must be changed.
Is sda a virtual drive from a hw-raid1 or a single drive - see eg. with "parted -l" ?

its a raid from my SAN storage. i check on my DELL EMC there is no error on a drive or raid whatsoever.

waltar · 2024-09-18T19:27:18+0200

parted -l ?

azkamstf · 2024-09-18T21:58:54+0200

waltar said:
parted -l ?

Bash:

root@px5:~# parted -l
Model: AVAGO MR9361-16i (scsi)
Disk /dev/sda: 3597GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  2097kB  1049kB                     bios_grub
 2      2097kB  271MB   268MB   fat32              boot, esp
 3      271MB   3597GB  3597GB                     lvm


Model: DGC VRAID (scsi)
Disk /dev/sdc: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  7688GB  7688GB  ext4         primary


Model: DGC VRAID (scsi)
Disk /dev/sdd: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  11.5TB  11.5TB  ext4         primary


Error: /dev/sdf: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdf: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data: 3434GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  577MB   576MB   primary               boot
 2      577MB   64.4GB  63.8GB  primary


Model: Linux device-mapper (thin-pool) (dm)
Disk /dev/mapper/pve-data-tpool: 3434GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  577MB   576MB   primary               boot
 2      577MB   64.4GB  63.8GB  primary


Error: /dev/mapper/pve-data_tmeta: unrecognised disk label
Model: Linux device-mapper (linear) (dm)                                 
Disk /dev/mapper/pve-data_tmeta: 17.0GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-data_tdata: 3434GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  577MB   576MB   primary               boot
 2      577MB   64.4GB  63.8GB  primary


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathd-part1: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  11.5TB  11.5TB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathh-part1: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  66.0TB  66.0TB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathg-part1: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  44.0TB  44.0TB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/mpathc-part1: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  7688GB  7688GB  ext4


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathg: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  44.0TB  44.0TB  ext4         primary


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathh: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  66.0TB  66.0TB  ext4         primary


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathd: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  11.5TB  11.5TB  ext4         primary


Model: Linux device-mapper (multipath) (dm)
Disk /dev/mapper/mpathc: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  7688GB  7688GB  ext4         primary


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-root: 103GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End    Size   File system  Flags
 1      0.00B  103GB  103GB  ext4


Model: Linux device-mapper (linear) (dm)
Disk /dev/mapper/pve-swap: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system     Flags
 1      0.00B  8590MB  8590MB  linux-swap(v1)


Error: /dev/sdy: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdy: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Unknown (unknown)
Disk /dev/nvme0n1: 6401GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6401GB  6401GB  btrfs


Model: DGC VRAID (scsi)
Disk /dev/sdo: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  7688GB  7688GB  ext4         primary


Error: /dev/sdm: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdm: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/sdu: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdu: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/sds: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sds: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Error: /dev/sdi: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdi: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: Kingston XS2000 (scsi)
Disk /dev/sdz: 4097GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  4097GB  4097GB  ext4


Error: /dev/sdg: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdg: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: DGC VRAID (scsi)
Disk /dev/sdx: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  44.0TB  44.0TB  ext4         primary


Model: DGC VRAID (scsi)
Disk /dev/sdv: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  66.0TB  66.0TB  ext4         primary


Model: DGC VRAID (scsi)
Disk /dev/sdl: 44.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  44.0TB  44.0TB  ext4         primary


Model: Unknown (unknown)
Disk /dev/nvme1n1: 6401GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6401GB  6401GB  btrfs


Model: DGC VRAID (scsi)
Disk /dev/sdj: 66.0TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      4194kB  66.0TB  66.0TB  ext4         primary


Error: /dev/sdr: unrecognised disk label
Model: DGC VRAID (scsi)                                                   
Disk /dev/sdr: 7688GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

Model: DGC VRAID (scsi)
Disk /dev/sdp: 11.5TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name     Flags
 1      33.6MB  11.5TB  11.5TB  ext4         primary

waltar · 2024-09-19T08:36:28+0200

So sda is a virtual 4TB drive on internal LSI raid controller (mostly a raid1) with the pve installed on.
Do you have "storcli" (oder "MegaCli") installed on ? --> google storcli download, download from broadcom site,
unpack the download, go into the ubuntu dir and apt install <release>.deb
storcli /call show

Search

Search

Lot of Crash cpus stall and filesystem Remounted read-only

azkamstf

New Member

Attachments

esi_y

Active Member

azkamstf

New Member

waltar

Member

azkamstf

New Member

waltar

Member

azkamstf

New Member

waltar

Member