VE 9.1.4 FreeBSD disk corruption

[mark0x01]

I'm having ongoing issues with disk corruption on multiple FreeBSD vm's withFreeBSD versions 13.0 up, all with UFS filesystems

The Proxmox Host is 9.1.4 on an Acer AT350 (Supermicro X8DTN+-F mb) with 132G ECC ram using zfs mirrored ST3600057SS for OS and ZFS mirror of 2 x Enterprise Toshiba SAS MG06SCA800E HDD with 2 x Enterprise SSD (HUSRL401) as zil log and cache log for the vm disks connected by two SAS2008 controllers with fw 20.00.07.00.

I get intermittent weird issues that show up like FreeBSD packages only partially installing -configuration files are missing.

Proxmox ZFS filesystem reports as clean, but when checking the vm disk, issues show up usually as journal corruption on the UFS file system.

I originally thought it was due to the ssd's dropping off, so I stopped mirroring them and that stopped, but the corruption continues.
Because it is intermittent, I can have no problems for days, but then a scripted build fails due to missing user (not created) or missing folder or config file from an application.

I was planning to migrate a physical box to this server, but no longer have confidence it will do the job.


So no underlying errors hardware errors show up on the host, just intermittently in the 9 or so FreeBSD vm's, which are the only OS I'm using on this at present.

Typical settings are:

boot: order=virtio0;ide2;net0
cores: 4
cpu: host
ide2: local:iso/FreeBSD-14.4-RELEASE-amd64-dvd1.iso,media=cdrom,size=4191088K
memory: 8192
meta: creation-qemu=10.1.2,ctime=1784361089
name: FreeBSD14.4
net0: virtio=BC:24:11:E8:29:C4,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=78917ad5-4661-4be9-8842-fe5dc701c62f
sockets: 1
virtio0: dpool1:vm-110-disk-0,iothread=1,size=300G

Any suggestions appreciated.

 

[g.fiore]

There's a pattern in your config that lines up with a known FreeBSD issue on Proxmox. Your disk is on virtio0 (VirtIO Block), not on the virtio-scsi-single controller you have configured. VirtIO Block on FreeBSD 13+ has documented intermittent corruption issues on write-heavy workloads like package installs — the FreeBSD virtio_blk driver is less mature than the Linux one and can lose flush ordering.. Your iothread=1 on VirtIO Block compounds this — that combo has been flagged as a race source.

I'd try in order:

1) Move one test VM's disk from virtio0 to scsi0 on the existing virtio-scsi-single controller. Detach, reattach as SCSI, fix /etc/fstab to use da0 instead of vtbd0. Reboot, re-run your scripted build a few times.

2) On the same VM also try iothread=0 and cache=writethrough.

3) Post pveversion -v and dmesg | grep -iE "vtbd|error|timeout" from a corrupted guest.

If steps 1-2 fix it, migrate the rest the same way. If not, we look at the storage layer next.

 

[mark0x01]

Thank you for the detailed response, it is very much appreciated.

The work load is quite high for the testing, but strangely it is more frequently the first package installed that breaks most often.

I'll try this and report back on the results.