Problem with Disk on RAID6 more 10Tb

[cvhideki]

Hi everyone!
I have problem with my PVE 6.4-13/9f411e79 (running kernel: 5.4.128-1-pve) with Disk volume

Disk model: PERC H730 Mini (raid6 2Gb*8)
Disk /dev/sdd: 10.9 TiB
create file system in parted:

(parted) mklabel gpt
mkpart primary 1 90%

create file system ext4: mkfs.ext4 /dev/sdd1
done, I see my file system
/dev/sdd1 2048 21092106239 21092104192 9.8T Linux filesystem

mount /dev/sdd1 /data/hdd
add to /etc/fstab
mount to my folder and add in proxmox:
Datacenter->Storage->Add Directory ID:name1, folder: /data/hdd

everything is ok, all see and worked

cheeking in fio performers:
fio --ioengine=libaio --direct=1 --sync=1 --rw=readwrite --bs=1M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/sdd

Run status group 0 (all jobs):
   READ: bw=244MiB/s (256MB/s), 244MiB/s-244MiB/s (256MB/s-256MB/s), io=14.3GiB (15.4GB), run=60010-60010msec
  WRITE: bw=243MiB/s (255MB/s), 243MiB/s-243MiB/s (255MB/s-255MB/s), io=14.2GiB (15.3GB), run=60010-60010msec

Disk stats (read/write):
  sdd: ios=58609/63644, merge=0/41, ticks=109472/48335, in_queue=92312, util=78.79%

Then I start backup my virtual machines after I see problem with my file system

Open parted and see next:

(parted) p
Error: /dev/sdd: unrecognised disk label
Model: DELL PERC H730 Mini (scsi)
Disk /dev/sdd: 12.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown

log files:
/var/log/messages

kernel: [328804.360931] EXT4-fs warning (device sdd1): ext4_dirblock_csum_verify:370: inode #11: comm du: No space for directory leaf checksum. Please run e2fsck -D.

/var/log/messages

EXT4-fs warning (device sdd1): ext4_dirblock_csum_verify:370: inode #11: comm du: No space for directory leaf checksum. Please run e2fsck -D.

Please help me what could be the problem?
Raid controler working normal.
Thanks!

 

[czechsys]

Your disk/raid setup is strange.
1] /dev/sdb as raid6 on Perc
2] fio filename is /dev/sdd <<--- ???
3] errors are from /dev/sdd and sde <<--- ???

 

[cvhideki]

why is it strange ?
yes, /dev/sdd (typing mistake)
raid6 using 2 tb disk 8 (12 TB Available space 4 TB Used for protection)

all results on problems disk

 

[cvhideki]

pveversion -v

proxmox-ve: 6.4-1 (running kernel: 5.4.128-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-5.4: 6.4-6
pve-kernel-helper: 6.4-6
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.11-pve1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-1
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.5-pve1~bpo10+1

 

[cvhideki]

up

 

[Dunuin]

Did you first partition that disk + format it and then run fio afterwards?
If you use fio ... --filename=/dev/sdd you tell fio to directly write to your array on block level killing all data that was there before and so corrupting your partitions and filesystems. So that would be no wonder that ext4 complains that something isn'T working with your filesystem if you have randomly overwritten data using fio.

If you want fio not to destroy your data you need to tell it to use a file on a mountpoint like "--filename=/data/hdd/test.file" and not point to a disk/array itself using "--filename=/dev/sdd".

 

[cvhideki]

Benchmarking_Storage

 

[Dunuin]

That still don't answer in what order you did stuff. Like its written in your linked Wiki...

Make sure that you will not destroy any data if you perform a write test. best to double or tripple check the devices and files.

...you will loose all data on that array if doing a write test like you did, as I already explained. The fio tests in that wiki are only meant to be run on empty disks because it will destroy data while writing to it on block level.

 

[cvhideki]

@Dunuin Thanks for the explanation