Hello,
I've got a HP DL380 Gen 10 with 4 Intel SSDs and PVE 8.1.3. I have multiple containers and VMs on it, but I'm specifically having an issue with something that only sometimes happens. The containers in question are Gitlab runners.
The following script excerpt shows the issue I'm having. This portion of the script copies multiple files to sdcard.targetfs/. After the copies have completed the du command shows the directory is only 4 blocks in size. Waiting for 5 seconds to let the filesystem catch up then shows a second du command showing 22557 blocks being used. The reason this is a problem is because further down past what I pasted here a filesystem is crated of a size based on the du command which it runs itself. Eventually it fails because it thinks the filesystem only needs to be 4 blocks in size rather than 22557. It fails about 40% of the time leading me to believe this is a caching issue of some form, but the rest of the time it completes just fine.
# Assemble directory for SD card filesystem image, then make image
mkdir -p buildroot/output/images/sdcard.targetfs
cp buildroot/output/images/u-boot-emmc.imx buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/zImage buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/beltpack_v1.dtb buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/beltpack_v2.dtb buildroot/output/images/sdcard.targetfs/beltpack.dtb
cp buildroot/output/images/u-boot-emmc.env.bin buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/info.txt buildroot/output/images/sdcard.targetfs/
#
du -s -k buildroot/output/images/sdcard.targetfs/
4 buildroot/output/images/sdcard.targetfs/
sleep 5
du -s -k buildroot/output/images/sdcard.targetfs/
22557 buildroot/output/images/sdcard.targetfs/
I'm using ZFS.
root@pve-1:~# zpool status
pool: rpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:58:21 with 0 errors on Sun Dec 10 01:22:22 2023
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300QL3P8EGN-part3 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300T63P8EGN-part3 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300GP3P8EGN-part3 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300CR3P8EGN-part3 ONLINE 0 0 0
This feels like some sort of caching issie to me but I'm not very familiar with ZFS to know.
Does anyone have any ideas of what to check or set to resolve this?
Thanks,
Ryan Wilkins
I've got a HP DL380 Gen 10 with 4 Intel SSDs and PVE 8.1.3. I have multiple containers and VMs on it, but I'm specifically having an issue with something that only sometimes happens. The containers in question are Gitlab runners.
The following script excerpt shows the issue I'm having. This portion of the script copies multiple files to sdcard.targetfs/. After the copies have completed the du command shows the directory is only 4 blocks in size. Waiting for 5 seconds to let the filesystem catch up then shows a second du command showing 22557 blocks being used. The reason this is a problem is because further down past what I pasted here a filesystem is crated of a size based on the du command which it runs itself. Eventually it fails because it thinks the filesystem only needs to be 4 blocks in size rather than 22557. It fails about 40% of the time leading me to believe this is a caching issue of some form, but the rest of the time it completes just fine.
# Assemble directory for SD card filesystem image, then make image
mkdir -p buildroot/output/images/sdcard.targetfs
cp buildroot/output/images/u-boot-emmc.imx buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/zImage buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/beltpack_v1.dtb buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/beltpack_v2.dtb buildroot/output/images/sdcard.targetfs/beltpack.dtb
cp buildroot/output/images/u-boot-emmc.env.bin buildroot/output/images/sdcard.targetfs/
cp buildroot/output/images/info.txt buildroot/output/images/sdcard.targetfs/
#
du -s -k buildroot/output/images/sdcard.targetfs/
4 buildroot/output/images/sdcard.targetfs/
sleep 5
du -s -k buildroot/output/images/sdcard.targetfs/
22557 buildroot/output/images/sdcard.targetfs/
I'm using ZFS.
root@pve-1:~# zpool status
pool: rpool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 00:58:21 with 0 errors on Sun Dec 10 01:22:22 2023
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300QL3P8EGN-part3 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300T63P8EGN-part3 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300GP3P8EGN-part3 ONLINE 0 0 0
ata-INTEL_SSDSC2KB038T8_PHYF325300CR3P8EGN-part3 ONLINE 0 0 0
This feels like some sort of caching issie to me but I'm not very familiar with ZFS to know.
Does anyone have any ideas of what to check or set to resolve this?
Thanks,
Ryan Wilkins