Proxmox & NVME huge data written numbers in SMART

[catchdoe]

Hello,
anyone experiencing huge numbers in SMART? I'm running Proxmox 5.3-7 on server with 4 virtual machines running on Debian. I've checked the nvme smart and the Data Units Written are way too high. I've got two Intel SSDPE2MX450G7 running in raid 1.

Smart from 12 January

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning: 0x00
Temperature: 26 Celsius
Available Spare: 97%
Available Spare Threshold: 10%
Percentage Used: 20%
Data Units Read: 27,836,670 [14.2 TB]
Data Units Written: 724,552,635 [370 TB]
Host Read Commands: 320,954,705
Host Write Commands: 7,553,830,582
Controller Busy Time: 50
Power Cycles: 24
Power On Hours: 5,576
Unsafe Shutdowns: 3
Media and Data Integrity Errors: 0
Error Information Log Entries: 0

Today's check:

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        26 Celsius
Available Spare:                    98%
Available Spare Threshold:          10%
Percentage Used:                    21%
Data Units Read:                    29,830,146 [15.2 TB]
Data Units Written:                 765,801,832 [392 TB]
Host Read Commands:                 341,659,895
Host Write Commands:                8,047,955,688
Controller Busy Time:               1
Power Cycles:                       27
Power On Hours:                     5,887
Unsafe Shutdowns:                   0
Media and Data Integrity Errors:    0
Error Information Log Entries:      0

Is this a firmware bug? Because when I divide data written by number of power on seconds it would be ~20MB/s on average, which is not possible as the VMs are mostly idling.

zpool iostat rpool 60
capacity operations bandwidth
pool alloc free read write read write

rpool 342G 74.3G 0 91 10.0K 1.95M
rpool 342G 74.3G 0 90 7.80K 1.95M
rpool 342G 74.3G 0 107 7.60K 2.91M
rpool 342G 74.3G 0 85 22.1K 2.15M
rpool 342G 74.3G 0 92 8.47K 2.16M
rpool 342G 74.3G 0 90 6.67K 1.71M

 

[sb-jw]

Are you sure your VMs in idle? Have you check them with e.g. iotop?

 

[catchdoe]

I used idle as a mental shortcut. I've used the iotop and there's literally nothing special going on on the VMs. I've ran iostat -md 600 on every VM and the host itself and the numbers don't add up at all. For example:

55MB+125MB+88MB+90MB=358MB of writes on all VMs, but in the same time output on host shows 990MB of written data.

I echoed "1" into /proc/sys/vm//block_dump and there's the result of syslot:

Jan 25 16:56:19 proxmox kernel: [505463.283056] z_wr_int_2(438): WRITE block 310505368 on nvme0n1p2 (16 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283058] z_wr_int_0(436): WRITE block 575539312 on nvme1n1p2 (16 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283075] z_wr_int_1(437): WRITE block 315902632 on nvme0n1p2 (32 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283096] z_wr_int_4(562): WRITE block 460141312 on nvme0n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283108] z_wr_int_4(562): WRITE block 460141328 on nvme0n1p2 (16 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283271] z_null_iss(418): WRITE block 440 on nvme1n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283315] z_null_iss(418): WRITE block 952 on nvme1n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283348] z_null_iss(418): WRITE block 878030264 on nvme1n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283378] z_null_iss(418): WRITE block 878030776 on nvme1n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283409] z_null_iss(418): WRITE block 440 on nvme0n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283442] z_null_iss(418): WRITE block 952 on nvme0n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283472] z_null_iss(418): WRITE block 878030264 on nvme0n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.283502] z_null_iss(418): WRITE block 878030776 on nvme0n1p2 (8 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.289562] z_wr_iss(434): WRITE block 460808488 on nvme1n1p2 (24 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.289572] z_wr_iss(434): WRITE block 460808488 on nvme0n1p2 (24 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.457366] z_wr_iss(430): WRITE block 460808744 on nvme1n1p2 (24 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.457382] z_wr_iss(430): WRITE block 460808744 on nvme0n1p2 (24 sectors)
Jan 25 16:56:19 proxmox kernel: [505463.459003] z_wr_iss(431): WRITE block 460809000 on nvme1n1p2 (24 sectors)

I knew the ZFS likes to write, but that SMART values are just outrageous.

 

[fabian]

what's your pools asize value? what about your datasets/zvols and their recordsize/volblocksize values? you can check with zpool iostat (and its various flags) how requests of different sizes get aggregated (or not). most likely it's some sort of write amplification.

 

[catchdoe]

I've already benchmarked zpool with iostat and it doesn't show anything close to average write calculated out of total writes/power-on hours. All values are stock from proxmox zfs install as I didn't change that during the initial setup. I'm really noob when it comes to all this stuff

 

[LnxBil]

This looks like write amplification, doesn't it? You should try to account for all i/os to disk in a time frame and see what the smallest write is.

 

[catchdoe]

I was thinking about the amplification too. But how can I avoid this?