High SSD wear after a few days

You can use 4 large SSDs for VM storage. Let's call them /dev/sdc /dev/sdd /dev/sde /dev/sdf. You will create the pool like this for a "raid10" setup:
Code: 
# zpool create -o ashift=9 storage mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf
# zfs set atime=off storage
# zfs set compression=lz4 storage (optional)
# zpool export storage
# zpool import -d /dev/disk/by-id storage

No need for LVM on storage pool. When Proxmox creates a volume for a VM, it does it like this (assuming a 16G volume on the storage point "storage/kvm")

Code: 
zfs create -V 16g storage/kvm/vm-100-disk-1

ZFS on Linux will map this to a block device, like /dev/zd80 and KVM daemon will use this block device as its virtual disk, just like with qcow2. To back it up, here's a quick sequence:

Code: 
backup pool: backup
main storage pool: storage
Proxmox storage point for KVM: storage/kvm
VMID: 100

# zfs snapshot storage/kvm/vm-100-disk-1@snap1
# zfs send storage/kvm/vm-100-disk-1@snap1 | zfs recv backup

For incremental snapshots

# zfs snapshot storage/kvm/vm-100-disk-1@snap2
# zfs send -i storage/kvm/vm-100-disk-1@snap1 storage/kvm/vm-100-disk-1@snap2 | zfs recv backup

The big advantage is that incremental backups are very fast with ZFS.

Your automatic install gives you headaches, so it is not better than manual. The Promox install is still automatic. The SSD storage pool creation is done once, manually, because you have better control on how it is done.
 
I understand what you are writing, however, I do not understand when I am supposed to do that.

Are these steps correct:
1. Boot from Proxmox CD
2. Go into proxmox debug mode
3. create zpool as suggested
4. reboot and go into proxmox installer
5. when storage is shown, use the RAID10 mirror

Do I get this right? Do you have a how-to, that shows more steps?

Thank you!!!!

EDIT: I also need to say, that I only have space for 4 physical HDD's. I have no room for more, unless I do something with an expander and second case etc... (there is A LITTLE space int he 1U case, if I modify the power cable to add connectors to connect a SSD, I MAY be able to put one SSD inside the case, I do not think that I can get 2 in there)
 
Last edited:
What kind of 1U case you have? What server?

No, the steps presented above are not OK. The install should be done in standard mode on OS disks. Now that you've told me that you have only 4 slots, this is an issue. I do have a HP DL160 G6 with 4 front slots only, but there are 2 more SATA ports inside and enough space for 2 SSDs. There are also two spare Molex connectors that were connected to SSDs with a molex-to-SATA Y adapter.
 
I have the DELL C1100. It has 4x 3.5" bays in the front only (same as the HP). Inside the case, there is just enough room (maybe have to check, going from memory here), for one, if I am lucky, than 2 SSD's. However, there is no power for the internal SSD's, I would have to splice the 12V/5V (not a problem for me).

I will look this weekend.

So, assuming I can fit 2x SSD's inside the case, this is what I should do:

1. Partition and mirror 2 root SSD's (small) as ext4 (not sure what tool to use)
2. Partition and RAID10 ZFS on 4 big SSD's (as you suggested above) (not sure what tool to use)
3. Run Proxmox installer and select root partition
4. Add ZFS storage once proxmox has finished installing (via GUI)

To I understand it correctly now?

Thanks.
 
It looks almost exactly like DL160. I think there is space for 7mm SSD between the two iron sheets (above hard-drives): http://en.community.dell.com/cfs-fi...scussions-components-files/956/6153.c1100.jpg

I think you have plenty of space under the cables coming from power supply. No need to screw the drives in unless you want to play football with the server :)
You need to splice 5V only, because 2.5 drives don't use 12V.

I've put my SSD between fans and drive bays. I also see that there are 6 SATA ports onboard, so you have the two extra ports.
 
Yes I though right behind the power supply, I think it will work.

Ironically, the MB has approx 12 SATA ports ... go figure ...

One more thing, I am trying a proxmox install right now, as you have suggested. I am following this guide to setup the boot drives as mirror: https://www.howtoforge.com/proxmox-2-with-software-raid
After that, I will setup the zvool.

Am I on the right track??

Thank you for your help, if you would live in my city, than lunch would be on me ... ;)
 
No, you don't need "software raid". That is ZFS job. You boot the standard installer, pick the install type ZFS and pick two drives. It will mirror them and also install GRUB on both. I think it is almost exactly what you did before with 3 drives, but pick only 2.
After install do this:
Code: 
zfs create rpool/vms

In Proxmox GUI, select "Datacenter" on the left, "Storage" tab and add a "ZFS" type storage. At "ZFS Pool" pick rpool/vms.
From now everything is the same and select this new storage for VMs disks. This is ZVOL type.

For backups read what I've told you above about send/receive and to master it, Google is your friend.
 
I see, I was overcomplicating things ...

So, I did finish an install on proxmox as suggested (VMBox) and added the ZFS storage. That all worked nicely and raid10 also is setup, see here:

Code: 
root@pve:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:


        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0


errors: No known data errors


  pool: storage
 state: ONLINE
  scan: none requested
config:


        NAME                                       STATE     READ WRITE CKSUM
        storage                                    ONLINE       0     0     0
          mirror-0                                 ONLINE       0     0     0
            ata-VBOX_HARDDISK_VBaf2b8e67-3a757b87  ONLINE       0     0     0
            ata-VBOX_HARDDISK_VB6fb74c14-66fa68b3  ONLINE       0     0     0
          mirror-1                                 ONLINE       0     0     0
            ata-VBOX_HARDDISK_VBcd5dde03-0bfcce82  ONLINE       0     0     0
            ata-VBOX_HARDDISK_VB805e0a88-dba6f0ec  ONLINE       0     0     0


errors: No known data errors

Now (hopefully my last question), the suggested backup script, should run with cron, in order to have it happening at X intervals (whatever my need is)?
Will the full backup and snapshot create a file, that I can RSYNC to offsite server?

Thank you sigxcpu for your help!!! i learned a lot today!!!
 
You can create a file, but you will lose the (easy) incremental capabilities.
"zfs send" outputs a ZFS stream that you can redirect to a file, SSH or whatever.
The main issue is the zfs send -i part because you will output small .zfs files for incremental backups.
When restoring, you will need to apply the main file and all the small files in order.

The usual way is to "zfs send" over SSH to another server where you "zfs receive" in a dataset, therefore having an exact duplicate of the source filesystem.


Here's a small (maybe not 100% correct) example with files:
Code: 
zfs create storage/test
touch /storage/test/1
touch /storage/test/2
touch /storage/test/3
zfs snapshot storage/test@snap1
zfs send storage/test@snap1 > snap1.zfs
touch /storage/test/4
zfs snapshot storage/test@snap2
zfs send -i storage/test@snap1 storage/test@snap2 > snap1.zfs

... Now snap1.zfs is the full backup, snap2.zfs is diff between 1 and 2

So, to restore to a dataset /storage/test2, you can do this 

zfs recv /storage/test2 <snap1.zfs
ls /storage/test2 (should be 1 2 3)
zfs recv /storage/test2 <snap2.zfs
ls /storage/test2 (should be 1 2 3 4)

For local (same pool) backup/restore you can rollback to a previous snapshot, you can access files (useless for ZVOLs) in snapshots (special hidden .zfs/snapshot folder in a dataset) and so on.

Here's a quick-start on ZFS: https://www.freebsd.org/doc/handbook/zfs-quickstart.html

For periodic send/receive over SSH with both source and destination being ZFS, you can use zrep from here: http://www.bolthole.com/solaris/zrep/

It is easy to set-up and it "just works". You will need to passwordless (i.e. key) based authentication on the dest server to put it in cron.
 
Last edited:
Yes I wanted to do a direct replication of the filesystem.

My target system is a FreeNAS. I will have to play with that in my test setup first.

Thank you for all your help!!!


Sent from my iPhone using Tapatalk
 
Anyway, going back to the original issue: you need to replicate the mirrored setup on the real host, with SSDs and check again the writes (iostat and zpool iostat at the same time). Also a smartctl output 1-2 days apart to map.
 
That is the plan.

I will be getting 3 SSD's and will do the setup on Saturday. I will report the output of iostat and zpool iostat on Monday afternoon (after I get some data during business hours).

Would you have a recommendation on what SSD's to use for the boot mirror?


Sent from my iPhone using Tapatalk
 
Hi,

I got the system setup as suggested above. I will post some results later today, after it has been in production for a while.

I have one question.

I followed these steps to create the RAID10 pool.
Code: 
# zpool create -o ashift=9 storage mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf
# zfs set atime=off storage
# zfs set compression=lz4 storage (optional)
# zpool export storage
# zpool import -d /dev/disk/by-id storage

Than I went to the GUI and added "storage" as ZFS to the VM node.

I did not do:
Code: 
zfs create storage/vms

My question is, is this neccessary? Do I have to create "zfs create storage/vms"? Is it the same with and without, since both are ZFS?

Thanks.
 
Hi,

I like the split between KVM's and Containers. I will set that up and move the VM files over.

So, here is some data as requested (the office is already using the system for 4 hours, so this is production data):

First, ashift for the zfs pool's and lsblk -dt /dev/sd? (ashift for the mirror is 12, shouldn't it be 9 as well?):

Code: 
root@VMNode01:/# zdb | grep ashift
            ashift: 12
            ashift: 9
            ashift: 9


root@VMNode01:/# lsblk -dt /dev/sd?
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED    RQ-SIZE  RA WSAME
sda          0    512      0     512     512    0 deadline     128 128    0B
sdb          0    512      0     512     512    0 deadline     128 128    0B
sdc          0    512      0     512     512    0 noop         128 128    0B
sdd          0    512      0     512     512    0 noop         128 128    0B
sde          0    512      0     512     512    0 noop         128 128    0B
sdf          0    512      0     512     512    0 noop         128 128    0B
sdg          0    512      0     512     512    1 deadline     128 128    0B


Than the zpool layout:
Code: 
root@VMNode01:/# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:


        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0


errors: No known data errors


  pool: storage
 state: ONLINE
  scan: none requested
config:


        NAME                                               STATE     READ WRITE CKSUM
        storage                                            ONLINE       0     0     0
          mirror-0                                         ONLINE       0     0     0
            ata-Samsung_SSD_850_PRO_512GB_S250NWAG904013M  ONLINE       0     0     0
            ata-Samsung_SSD_850_PRO_512GB_S250NXAG936968X  ONLINE       0     0     0
          mirror-1                                         ONLINE       0     0     0
            ata-Samsung_SSD_850_PRO_512GB_S250NWAG905249J  ONLINE       0     0     0
            ata-Samsung_SSD_850_PRO_512GB_S250NWAG905234V  ONLINE       0     0     0


errors: No known data errors

IOSTAT for the ROOT mirror (uptime is ~4 hours):
Code: 
root@VMNode01:/# iostat sda sdb
Linux 4.2.3-2-pve (VMNode01)    11/30/2015      _x86_64_        (16 CPU)


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.15    0.00    2.29    3.27    0.00   91.29


Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              14.67        21.74       698.15     324024   10405524
sdb              14.41        21.29       698.15     317276   10405524

IOSTAT for the VM/Container storage:
Code: 
root@VMNode01:/# iostat sdc sdd sde sdf
Linux 4.2.3-2-pve (VMNode01)    11/30/2015      _x86_64_        (16 CPU)


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.17    0.00    2.30    3.28    0.00   91.25


Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdc              79.55      1060.99       976.00   15888113   14615415
sde              82.59      1068.12       969.83   15994848   14523057
sdd              79.41      1079.84       976.00   16170426   14615415
sdf              82.60      1072.26       969.83   16056907   14523057

Than zpool iostat:
Code: 
root@VMNode01:/# zpool iostat
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
rpool       50.6G  60.4G      5    114  40.6K   682K
storage     49.7G   902G    865    138  4.13M  1.90M
----------  -----  -----  -----  -----  -----  -----


And finally the SMART data for all 6 drives:
Code: 
root@VMNode01:/# smartctl -a /dev/sda
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.3-2-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Device Model:     ADATA SX930
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000a   100   100   000    Old_age   Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0013   100   100   050    Pre-fail  Always       -       0
  7 Unknown_SSD_Attribute   0x000b   100   100   050    Pre-fail  Always       -       0
  8 Unknown_SSD_Attribute   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       44
 10 Unknown_SSD_Attribute   0x0013   100   100   050    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0012   100   100   000    Old_age   Always       -       14
167 Unknown_Attribute       0x0022   100   100   000    Old_age   Always       -       0
168 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       0
169 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       262145
170 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       0
173 Unknown_Attribute       0x0012   199   199   000    Old_age   Always       -       4295229443
175 Program_Fail_Count_Chip 0x0013   100   100   010    Pre-fail  Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   100   100   020    Pre-fail  Always       -       1020
192 Power-Off_Retract_Count 0x0012   100   100   000    Old_age   Always       -       8
194 Temperature_Celsius     0x0022   071   071   030    Old_age   Always       -       29 (Min/Max 21/33)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
231 Temperature_Celsius     0x0033   100   100   005    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       772663296
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       667701632
240 Unknown_SSD_Attribute   0x0013   100   100   050    Pre-fail  Always       -       0
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       637564053
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       502969025





root@VMNode01:/# smartctl -a /dev/sdb
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.3-2-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Device Model:     ADATA SX930
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000a   100   100   000    Old_age   Always       -       0
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0013   100   100   050    Pre-fail  Always       -       0
  7 Unknown_SSD_Attribute   0x000b   100   100   050    Pre-fail  Always       -       0
  8 Unknown_SSD_Attribute   0x0005   100   100   050    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       44
 10 Unknown_SSD_Attribute   0x0013   100   100   050    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0012   100   100   000    Old_age   Always       -       11
167 Unknown_Attribute       0x0022   100   100   000    Old_age   Always       -       0
168 Unknown_Attribute       0x0012   100   100   000    Old_age   Always       -       0
169 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       262145
170 Unknown_Attribute       0x0013   100   100   010    Pre-fail  Always       -       0
173 Unknown_Attribute       0x0012   199   199   000    Old_age   Always       -       4295229443
175 Program_Fail_Count_Chip 0x0013   100   100   010    Pre-fail  Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033   100   100   020    Pre-fail  Always       -       1020
192 Power-Off_Retract_Count 0x0012   100   100   000    Old_age   Always       -       5
194 Temperature_Celsius     0x0022   071   071   030    Old_age   Always       -       29 (Min/Max 22/33)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
231 Temperature_Celsius     0x0033   100   100   005    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   100   100   000    Old_age   Always       -       771096064
234 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       665557248
240 Unknown_SSD_Attribute   0x0013   100   100   050    Pre-fail  Always       -       0
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       637570821
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       501735455


root@VMNode01:/# smartctl -a /dev/sdc
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.3-2-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 850 PRO 512GB
Firmware Version: EXM02B6Q
User Capacity:    512,110,190,592 bytes [512 GB]
Sector Size:      512 bytes logical/physical

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       192
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       11
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       15
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   071   069   000    Old_age   Always       -       29
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       6
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       2345870607


root@VMNode01:/# smartctl -a /dev/sdd
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.3-2-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 850 PRO 512GB
Firmware Version: EXM02B6Q
User Capacity:    512,110,190,592 bytes [512 GB]
Sector Size:      512 bytes logical/physical

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       192
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       10
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       15
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   071   069   000    Old_age   Always       -       29
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       5
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       2346678043

root@VMNode01:/# smartctl -a /dev/sde
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.3-2-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 850 PRO 512GB
Firmware Version: EXM02B6Q
User Capacity:    512,110,190,592 bytes [512 GB]
Sector Size:      512 bytes logical/physical

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       173
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       10
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       13
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   072   068   000    Old_age   Always       -       28
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       6
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       1844063776

root@VMNode01:/# smartctl -a /dev/sdf
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.2.3-2-pve] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 850 PRO 512GB
Firmware Version: EXM02B6Q
User Capacity:    512,110,190,592 bytes [512 GB]
Sector Size:      512 bytes logical/physical

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       45
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       7
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       1
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   069   067   000    Old_age   Always       -       31
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       4
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       293304224

When I installed and put the server into production (Sat Nov 28 @ 10pm), there are the SMART data (short):
Code: 
[TABLE="width: 590"]
[TR]
[TD="colspan: 7"]Sat Nov 28 2015 @ 10pm[/TD]
[/TR]
[TR]
[TD][/TD]
[TD]hours[/TD]
[TD]P/E cycle[/TD]
[TD]LBA Written[/TD]
[TD]P/E Cycle in GB[/TD]
[TD]LBA Written in GB[/TD]
[TD]GB wrtn/hour[/TD]
[/TR]
[TR]
[TD]sdc[/TD]
[TD="align: right"]152[/TD]
[TD="align: right"]13[/TD]
[TD="align: right"]2167117218[/TD]
[TD="align: right"]6656[/TD]
[TD="align: right"]1033[/TD]
[TD="align: right"]7[/TD]
[/TR]
[TR]
[TD]sdd[/TD]
[TD="align: right"]152[/TD]
[TD="align: right"]13[/TD]
[TD="align: right"]2167914517[/TD]
[TD="align: right"]6656[/TD]
[TD="align: right"]1034[/TD]
[TD="align: right"]7[/TD]
[/TR]
[TR]
[TD]sde[/TD]
[TD="align: right"]133[/TD]
[TD="align: right"]12[/TD]
[TD="align: right"]1652867613[/TD]
[TD="align: right"]6144[/TD]
[TD="align: right"]788[/TD]
[TD="align: right"]6[/TD]
[/TR]
[TR]
[TD]sdf[/TD]
[TD="align: right"]5[/TD]
[TD="align: right"]0[/TD]
[TD="align: right"]102093042[/TD]
[TD="align: right"]0[/TD]
[TD="align: right"]49[/TD]
[TD="align: right"]10[/TD]
[/TR]
[/TABLE]

What is confusing, is the ashift on the mirror. Also, according to the P/E Cycle, I am writing way to much.
I also have the Process Explorer running on one VM (the most used one) and my total write for the 4 hours is 1GB total.

I hope this helps.

Thanks.
 
Last edited:
That's too much data to grasp. What I would do is this.
1. Reboot the server
2. Take a snapshot (short SMART data)
3. Leave 12-24 hours
4. Take another snapshot
5. Do an "iostat -dm" (this will show you read & written data in MB since the last reboot)
6. Substract 2 from 4 and map it to 5 to find the write amplification.
 
Ok, I will post the short results on Wednesday morning.

What do you think about the ashift of 12 for the mirror? Shouldn't it be 9 as well?


Sent from my iPhone using Tapatalk
 
we are also experiencing similar massive write count increase (SMART and iostat). i started logging SMART-Outputs once per hour to have further data... we use 2 mirrored hard disks with 2 ssds as mirrored log and cache the KVM VMs use qcow files, the next cluster will use ZFS Volumes, perhaps that's better...
 
start by updating smartmon database

and 300 TBW isnt that much, even for this product class, there are other ssd's with greater tbw and they costs about the same.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back