[SOLVED] NVME disk "Available Spare" problem.

[gfngfn256]

Backup disc: /dev/sde is already SSD disc, connected via USB holder. I can mount it and use for saving backup there? It already contains VM backups. Is that ok or can it be a problem?

/dev/sde1 is a perfect location for the zipped image location (with ext4 fs) & I see its already mounted - so you can ignore the first mount command. Just locate its mountpoint in host node with lsblk command.

Please note that the above command to create the image will take some time (probably at least 30 minutes, maybe more) - this will depend on how much of that 500gb NVMe is actually occupied/used with data.

Make sure you have a enough space available on that /dev/sde1 to store the created compressed image file - it will be anything from about 8gb upwards (probably not more than 25gb IDK?) - again its size will depend on how much actual used data is on that NVMe.

 

[GazdaJezda]

I thought so. Great. There is 190GB free space, i think it will be enough. Otherwise I will delete on of daily backups it contains.

 

[gfngfn256]

There is 190GB free space

You should be good to go - Good Luck

 

[zzz09700]

Wait, if I'm reading this correct, you guys are killing a 500G SSD in three and a half years, cranking down it's available spare blocks to ~20%?

What kind of workload are you running on that thing? Like, almost one full disk write pre day, every day? We don't see that kind of load often, even with our primary DB disks.

 

[GazdaJezda]

Heh, nothing special. Load on that one is minimal, since there is only system on it. Corosync, pve-ha-crm, pve-ha-lrm, syslog services are stopped. Only thing I can think of can be a temperature. It can rise to 40 Celsius degrees (reported by device - S.M.A.R.T.). Other that that I really dont't know what would be so problematic.

Now I'm downloading Ubuntu 24.04 ISO for boot from it and made backup. Booting DSL image (Damn Small Linux) fail - reset, so I will try with Ubuntu. It apparently support Live (trial) mode.

 

[zzz09700]

Heh, nothing special. Load on that one is minimal, since there is only system on it. Corosync, pve-ha-crm, pve-ha-lrm, syslog services are stopped. Only thing I can think of can be a temperature. It can rise to 40 Celsius degrees (reported by device - S.M.A.R.T.). Other that that I really dont't know what would be so problematic.

Now I'm downloading Ubuntu 24.04 ISO for boot from it and made backup. Booting DSL image (Damn Small Linux) fail - reset, so I will try with Ubuntu. It apparently support Live (trial) mode.

That should never ever kill a SSD this fast. How many TB is written to the drive now? Temp around 40 is fine for consumer SSDs - heck even many data center SSDs are rated for 50 running 24x7.

I figure you could be looking at a faulty SSD that should be RMAed in the first year. Some of the cells are wearing out like after 10 wipes so the controller replaced those cells with spares.

Reduction in available spares should only happen when the SSD is VERY close to it's TBW limit, something like a few hundred TB for comsumer SSDs. If you don't have thay much TB written the you are looking at some typical Samsung crap.

 

[GazdaJezda]

That should never ever kill a SSD this fast. How many TB is written to the drive now? Temp around 40 is fine for consumer SSDs - heck even many data center SSDs are rated for 50 running 24x7.

I figure you could be looking at a faulty SSD that should be RMAed in the first year. Some of the cells are wearing out like after 10 wipes so the controller replaced those cells with spares.

Reduction in available spares should only happen when the SSD is VERY close to it's TBW limit, something like a few hundred TB for comsumer SSDs. If you don't have thay much TB written the you are looking at some typical Samsung crap.

SMART shows:
Data Units Read: 8,53 TB
Data Units Written: 778 GB

I guess I had a bad luck with that unit. I think it has a 5 year waranty, but yes, that doesn't help me much ATM. I ordered another one, same Samsung, just newer model. Will see how this one will cope with ProxMox, haha. But yes, I was hoping that since it is Samsung (their SSD's are not in the bottom of quality) current one will last longer...

 

[zzz09700]

There are bunch of "refurbished" (have their SMART data reset as new) Samsung SSDs out there and some shady vendors sell those as new.
The SMART data are cleared but the wear out on NAND flash is still there so those drives often fail with premature reduction of available spares.

Basically the controller thinks its looking at new NAND flash cells with like 300 wipes left but actually some cells only got 20 wipes left and grow unstable after like 10 wipes and then the controller decides it should be replaced by spares.

With something like ~800G written you should see expected life span at ~99% and available spares at 100% in SMART. Anything else is an indication of problematic (either broken from factory, or its SMART was tampered with) disk. It does matter what OS you use it with.

 

[zzz09700]

Attached is what SMART should look like after mild usage. Its from one of our spare part SATA SSD.
You can see with a few hundred hours of power on and a few TB written, available spares is still at 100%.

 

[GazdaJezda]

I see. Didn't know that. It was a brand new when i get it. Will see how the new one will perform. Better, i hope.

 

[zzz09700]

I see. Didn't know that. It was a brand new when i get it. Will see how the new one will perform. Better, i hope.

I would recommend stress test the drive and write like 2TB of data into it before putting it into use.
970Evo 500GB is rated for 300TBW, so the stress test only costs around 1% of the total lifespan but ensures you don't get a problematic drive.

 

[Kingneutron]

Thanks! Will use nvme0n1 for cloning / backing up current disc. I also order a replacement (SAMSUNG 970 EVO PLUS 500GB SSD instead of SAMSUNG 980 PRO 500GB SSD - this one is PCI-E v. 4, my board only support v. 3, my HW seller spot that). Next week will be here.

Best regards!

My first reaction was, Oh d--n it, he's going with EVO again--! - but I decided not to post.

I don't see EVO as good for anything serious, maybe desktop-class or gaming usage. For virtualization use a 500GB (or less) is likely going to wear out early unless you underprovision it - making a smaller final partition and leaving space -- but then you're wasting GBs of space on a small-ish drive.

https://www.tomshardware.com/reviews/samsung-970-evo-plus-ssd,5608.html

IDK, maybe the Plus will last longer if you put zfs on it, it's COW. Hope for the best, but IMHO for my expectations of performance and disk life would always recommend going with Pro over Evo even for homelab (and obv enterprise-class SSD is best, especially if you can get it on ebay for cheap)

YMMV, just saying don't expect great sustained performance out of it due to the SLC+TLC cache setup, it will throttle. If you have to suspend a (say) 32GB RAM VM for backup or Hibernate, it may take longer than you expect and you might start having I/O issues with shared VMs on the same storage if you're only running on a single disk. Not trying to cause waves or anything here, just trying to adjust expectations.

https://www.tweaktown.com/reviews/8879/samsung-970-evo-plus-500gb-ssd-review/index.html

 

[Ramalama]

All Samsung Consumer "NVME" drives are pure crap. The sata versions like the 870 Evo/plus are pretty good tho. (in reliability, not iops)
But other Brands can be even worse, or better in TBW but with less speed/worse latency.
So thats why im using myself something like 970/980/990 Pros/Evos xD

However thats an old discussion, but i do understand people pretty good why they buy Consumer SSD drives, especially something for OS with 500gb. The pricedifference is very steep from an 970/980/990 Pro/Evo 500GB to Enterprise Micron 7450/7500 or PM9a3.

And those people with home-servers don't have a lot pcie lanes, so often no mirror etc...
So in the end its all fine. They want a home-server, not enterprise grade ultra reliable 99,9999999 Uptime Server xD

 

[zzz09700]

I don't see EVO as good for anything serious, maybe desktop-class or gaming usage. For virtualization use a 500GB (or less) is likely going to wear out early unless you underprovision it - making a smaller final partition and leaving space -- but then you're wasting GBs of space on a small-ish drive.

The OP managed to wrote ~800GB to his SSD within three and a half years. So I figure any SSD will do (as long as they are genuine new SSD), he will be running out of controller chip life before exhausting write endurance of NAND flash.

Yeah in my place even spare parts that are mainly used, well, as temporary placeholder and test beds, has a few times more writes than OP.

We have a handful of Sandisk SSD Plus (Yeah the one that is a little slower than some 15K RPM HDD) and experices says those can be trusted for ~60TB write endurance (480GB model) and three years of 24x7 without heavy load.

But for anything serious - like someone gonna run a production DB within the VM - OP should go with real enterprise products, not just for write endurance but also power loss protection, controller cooling, flexible overpsovisioning space and many other advantages. Those drives are marketed as 1DWPD for a reason and most consumer SSDs won't survive a year 24x7 with heavy load due to controller overheating

For consumer SSDs there is simply no heat dispensing mechanism so no matter how much airflow you throw at the disk, it always overheats internally. Most enteprise NVME SSDs still use 2.5-inch form factor and SFF-8639 connector because its much easier to provide sufficient cooling for all internal components in this form factor then with M2 2280/22110.

 

[zzz09700]

And those people with home-servers don't have a lot pcie lanes, so often no mirror etc...

With EPYC flooding eBay I would recommend everyone give those a try. 128 PCIe lanes can render all SATA/SAS ports obsolete.
Back in the days we had to plan precisely how we gonna use PCIe lane for each server configuration and some of them had to be dual socket just for the extra PCIe lane, oh those were the days...

 

[GazdaJezda]

Hi!

I'm in the process of copying (backup-ing) my almost faulty 970 EVO. It starts ok, but after a while it returns reading errors (see pic below). Is this critical for a whole operation?

 

[zzz09700]

YHi!

I'm in the process of copying (backup-ing) my almost faulty 970 EVO. It starts ok, but after a while it returns reading errors (see pic below). Is this critical for a whole operation?

YES, IT IS.
Don't cancel the copying, you may not have a second chance. Consumer TLC SSDs in this state can be unstable even with reads-only operations (the cells needs to be recharged regularly, otherwise data is lost)

And you could end up with some broken/missing files. Just hope those files are from PVE install ISO or package repo and not from your VM config files. (Salvage what you can and hope for the best)

BTW I just saw one SSD with 135TB written and only 2% of available spares used. That one is almost half way through its write endurance, should survive whole advertised TBW I think.

 

[GazdaJezda]

Oh s**t. I will need to boot again from that disk, otherwise I'm cut from net and other services. Currently is on 82 GB, 78 GiB. Hopefully it will end soon.

 

[gfngfn256]

Currently is on 78 GiB. Hopefully it will end soon.

Its going to take time. This will depend on the size of the NVMe & how much real data (non-zero) is actually there. So you should see a speed improvement whenever dd is actually up to a non-zero data section of the disk.

Your errors don't look too good - but doesn't have to be a game-changer. Time will tell whether the compressed backup is usable.

 

[GazdaJezda]

Ok. Will wait till end. On 128GB...