Storage drive bricked!? Unable to mount a full disk

[imaginebricking]

I cannot mount my physical storage disk which had previously been mounted.
As soon as I run the mount command in Ubuntu terminal, my VM freezes and Proxmox shows "IO error". The drive is 100% full so it's not really surprising (not sure how that happened as I'd just made 200gb worth of free space).

The data on it isn't critical but I would rather not lose it all.

If I can't mount it then I have no idea how I can delete files from it to get it working again. Any ideas?

 

[BobhWasatch]

Does they disk have partitions? Maybe you should be mounting one of those instead of the full disk.

 

[imaginebricking]

There's no partitions. I have a m2 drive which runs proxmox and my vms, then the storage ssd drive which I have been mounting on one of the VMs. It was working fine before something gobbled up the remaining space

 

[BobhWasatch]

Being full won't prevent it mounting. The filesystem being corrupted might. Did you try running fsck on it? What filesystem is it anyway?

OTOH, an I/O error could mean the disk has gone bad.

ETA: Have you run SMART tests on it?

 

[imaginebricking]

SMART values all look goodL

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       0
  5 Reallocate_NAND_Blk_Cnt 0x0032   100   100   010    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       8127
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       792
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
173 Ave_Block-Erase_Count   0x0032   097   097   000    Old_age   Always       -       36
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age   Always       -       202
180 Unused_Reserve_NAND_Blk 0x0033   100   100   000    Pre-fail  Always       -       5
183 SATA_Interfac_Downshift 0x0032   100   100   000    Old_age   Always       -       0
184 Error_Correction_Count  0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   051   047   000    Old_age   Always       -       49 (Min/Max 12/53)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_ECC_Cnt 0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       903
202 Percent_Lifetime_Remain 0x0030   097   097   001    Old_age   Offline      -       3
206 Write_Error_Rate        0x000e   100   100   000    Old_age   Always       -       0
210 Success_RAIN_Recov_Cnt  0x0032   100   100   000    Old_age   Always       -       0
246 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       38621358051
247 Host_Program_Page_Count 0x0032   100   100   000    Old_age   Always       -       1206917439
248 FTL_Program_Page_Count  0x0032   100   100   000    Old_age   Always       -       1139246400
249 Unkn_CrucialMicron_Attr 0x0032   100   100   000    Old_age   Always       -       0
250 Read_Error_Retry_Rate   0x0032   100   100   000    Old_age   Always       -       0
251 Unkn_CrucialMicron_Attr 0x0032   100   100   000    Old_age   Always       -       1135291584
252 Unkn_CrucialMicron_Attr 0x0032   100   100   000    Old_age   Always       -       17
253 Unkn_CrucialMicron_Attr 0x0032   100   100   000    Old_age   Always       -       0
254 Unkn_CrucialMicron_Attr 0x0032   100   100   000    Old_age   Always       -       1497
223 Unkn_CrucialMicron_Attr 0x0032   100   100   000    Old_age   Always       -       14

It's XFS. I just ran xfs_repair from the proxmox shell and there were no issues and the problem persists.

 

[esi_y]

I would wonder what dmesg shows during that attempt to mount.

 

[imaginebricking]

Output of dmesg | grep sda:

[    1.401754] sd 2:0:0:0: [sda] 109051904 512-byte logical blocks: (55.8 GB/52.0 GiB)
[    1.402965] sd 2:0:0:0: [sda] Write Protect is off
[    1.403434] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08
[    1.403519] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.410511]  sda: sda1 sda2 sda3
[    1.411817] sd 2:0:0:0: [sda] Attached SCSI disk
[    4.624427] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

If I run blkid on the proxmox shell it shows the drive as:

/dev/sda1: UUID="4d0d8b33-28ac-44bf-9807-d4cfb39a0c18" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="f4a80284-f1ed-46e8-838d-fe6a3753e209"

Apologies, I'm a bit out of my depth now so please assume I know nothing

 

[esi_y]

Output of dmesg | grep sda:

[    1.401754] sd 2:0:0:0: [sda] 109051904 512-byte logical blocks: (55.8 GB/52.0 GiB)
[    1.402965] sd 2:0:0:0: [sda] Write Protect is off
[    1.403434] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08
[    1.403519] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    1.410511]  sda: sda1 sda2 sda3
[    1.411817] sd 2:0:0:0: [sda] Attached SCSI disk
[    4.624427] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

If I run blkid on the proxmox shell it shows the drive as:

/dev/sda1: UUID="4d0d8b33-28ac-44bf-9807-d4cfb39a0c18" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="f4a80284-f1ed-46e8-838d-fe6a3753e209"

Apologies, I'm a bit out of my depth now so please assume I know nothing

Wait a minute, you are talking "no partitions" etc, then you have /dev/sda (disk) going in and 3 partitions on it (sda1,2,3). The second partition gets mounted as ext4.

There's nothing about mounting partition 1, presumably xfs. How are you trying to mount sda1?

 

[waltar]

xfs_info /dev/sda1 ?
But anyway a sda with 3 partitions look as an os disk and not as an option disk ...