PVE on SSDs?

A

achekalin

Member
Jun 17, 2011
74
0
6
Hello,

recently I got a chance to try PVE on server with SSD drives. The machine is equipped with Adaptec 5405 card with battery, and 2 x 500 Gb SATA HDDs + 2 x 256 Kingston SSDs. I planned to set up PVE on mirror made of HDDs (so PVE itself and backup will be there; 500 Gb storage is quite a lot for that), and use SSDs for VMs (we use KVM, so VM disks will be on SSDs).

Here is the point of decision, and I can:

1. build mirror of SSDs (and that's looking better idea since controller will care for h/w faults) and use it as singe volume, then put LVM over it, and create ext4 over LVM.

2. use both SSDs as different volumes (so RAID card will show it as 2 separate volumes, but card's cache and battery sill be in action for that disks), then build LVM over it (using LVM mirroring feature), then put ext4 over it.

3. as in case 2 above, use SSDs as different volumes, then use MD to create a mirror, the put LVM over it, then ext4 over LVM.

I'd prefer case 1, but that way LVM (and ext4 on top of it) won't use any SSD-specific optimizations. Even worst, I'm not even sure any TRIM support be in effect that way. RAID card firmware is the latest one, by the way.

So, instead of inventing wheel, I'd rather ask you, gurus: which way is the best while dealing with SSDs in PVE?

P.S. Once I even heard that PVE kernel version is a bit dated to have good SSD support (they said, I need at least 2.6.37 to use TRIM support on device mapper, but in stock PVE 2.1 kernel I was able to use "discard" mount option; even with it, I see rather slow SSD speed, so looks like I still need to find good way to use SSD).
 
Just curiosity, why did you not considered put LVM over SSD, create a new Volume Group and the use that storage directly as "LVM Group" (see http://pve.proxmox.com/wiki/Storage_Model#LVM_Groups_with_Local_Backing)? Is there something wrong with this kind of storage and a SSD (that needs special "trim" cares).
I would love to test SSD too, but I'm a bit confused... like is RAID controller a bottleneck to SSD I/O? Is correct try having fast raid1 or raid10 with ssh, with cheap controllers that don't have RAM cache (that should be almost useless) and so don't need expensive BBU? Any other better solution?
 
mmenaz said:
Just curiosity, why did you not considered put LVM over SSD
Click to expand...

In fact that's option 2 from my original post, I simple havn't tried it yet.

2 SSD in a mirror is a simple workaround against MLC SSD quick wear-out (and we use MLC due to reasons that I can't change). I can only connect SSDs directly to motherboard's SATA ports (and I appreciate you for pointing our that RAID itself may be the bottleneck!), but then the only way I can use it is to put it in LVM mirror, isn't it?

I just wonder if device mapper in out-of-box PVE trim (discard)-aware? As far I see that's not true, so that scheme seems to be not SSD optimized. That's my point of uncertainty :(.
 
Forgive me, but in option 2 of your original post, the "then put ext4 over it" seems like you want to create a new VG, then a LV that uses almost the entire space, format it with ext4 and use it as "directory based" storage to put VM as files (as it happens in local storage). The option I suggest to investigate instead lets your VM have a LV each, and inside the VM you will format it with the file system you want. In this way the SSD storage is used "directly" without an ext4 FS on it. With mechanical HD this brings performance advantages (the VM "talks" directly with a block device, sort of), without the middle layer of the host file system.
 
Sorry for that, you're right.

You see, I used to use VM as file concept due to its easiness for manipulate. Say, I can copy VM simple by copy its vdisks, etc. With LVMm, I'll have an advantage in speed and disadvantage in control.

But nevermind, I'll give it a try, thank you for pointing that!

Anyway, I have to do something with device-mapper (as of this, I need to have 2.6.33+ kernel). We're bound to PVE 2.1, but even in pvetest repository I see only 2.6.32-16...
 
pvetest, like "stable", now has 2.6.32-17 (not -16), but in any case, AFAIK, only the 2.3.32 is "long term support" so probably Red Hat will stay there, or jump to next long term support (3.4 I think). I'm not sure about it, but just want to raise this aspect to your attention. RH and Proxmox team do some backport, but probably, at least Proxmox team, only of device drivers. Also at the moment Proxmox wants to support OpenVZ, and so the choice is narrowed to future kernels that will support it too, or maybe Proxmox will decide to drop it completely.
 
Oh, I got another bad news here: "The good news is that in the 2.6.36 kernel, discard support was added to the dm layer! So any kernel 2.6.36 or later will support TRIM."

So I only can hope for the case that TRIM support in dm is also backported in PVE distro. And I simple don't know how to test it.
 
if you connect your SSD to a adaptec raid controller, trim cannot work by design (as the OS like Proxmox VE does not see the SSD)

if you want to use SSD on a raid controller, check the vendor website for perfect usage examples (ex. hybrid raid, etc.).

also check adaptec knowledge base for more details.
 
We've already disconnected SSDs from Adaptec and connected it directly to SATA ports on motherboard (we simply not sure if Adaptec card is SSD-aware). Now I need to choose how to build TRIM-enabled mirror with software tools.

I can:

1. use single SSD, put LVM on it, create VG and use that VG as storage for images. Then once an hour do a snapshot copy to second SSD. Very bad in terms of robutness, but simple to control and 2nd SSD won't be weared out that much.
2. use mdadm to create s/w raid1 over than two SSDs, create LVM on top of it etc.
3. use LVM to create mirrored storage (I think LVM can mirror only LVs, not VG, or it's me who's wrong?)
4. create raid1 with mdadm, ext4 over it (no LVM!), and put VM disks on it. No snapshots, no nice backups, but may work.

What to choose? I'm a bit confused which features are supported by PVE distribution keeping in mind I need TRIM support.
 
As I can see now, I need to know:

1. If kernel supports TRIM? (for PVE kernel the answer is "YES")
2. If device-mapper in PVE distro supports TRIM?
3. If LVM in PVE distro supports TRIM?
4. If mdadm in PVE distro supports TRIM?
What I sure about is the 1st line, but answer for 2..4 seems to be "NO". If it is so, how can I upgrade it to the version(s) which do support TRIM? When I try to do that trick with mdadm (via Debian unstable repos) the mdadm version seems to stay too old for TRIM support, if I try to do that for LVE, it won't upgrade due to dependency that involves upgrading PVE-related libs.
 
we do never recommend or support softraid, so if you think it make sense don´t expect too much help here.
see http://pve.proxmox.com/wiki/Software_RAID

if you still want to use such a custom setup you should just test it by yourself if it works as you want. no one here will test an unsupported custom setup for you.
and report your results.
 
device-mapper and LVM do support TRIM

/etc/lvm/lvm.conf
# Issue discards to a logical volumes's underlying physical volume(s) when
# the logical volume is no longer using the physical volumes' space (e.g.
# lvremove, lvreduce, etc). Discards inform the storage that a region is
# no longer in use. Storage that supports discards advertise the protocol
# specific way discards should be issued by the kernel (TRIM, UNMAP, or
# WRITE SAME with UNMAP bit set). Not all storage will support or benefit
# from discards but SSDs and thinly provisioned LUNs generally do. If set
# to 1, discards will only be issued if both the storage and kernel provide
# support.
# 1 enables; 0 disables.
issue_discards = 1

regards mdadm I think this is left over to the filesystem and what support the filesystem has for TRIM.
 
I got a chance and changed SSDs for SAS disks, which given me slower storage but at least I have no worry of 'if I do that right'.

Sad to tell but my choices was too fuzzy for any serious use: mdadm is unsupported by pve team (I've actually read that long ago, back in 1.8 era, and remember it; I mentioned it here just as last resort choice) and using it also adds up one more layer to storage stack. LVM mirroring is something better in terms of support but I've found not that many people uses it in production (and when in comes to production the old good raid with battery is a must, anyway). And as risking my data and VMs (dispute how much they cost) wasn't the primary goal, I decide to go with SAS.

I still really interested if it is really worth to use SSD to store virtual disks (ain't there too many layers already so speeding up one of it may not be very helpful?).
 
mir said:
device-mapper and LVM do support TRIM
...
regards mdadm I think this is left over to the filesystem and what support the filesystem has for TRIM.
Click to expand...

I saw that config but that's strange as version of dm and lvm is too old. Backported features? That's big and nice job!
 
SSD's will speed things up, but not make miracles. Most use you will get from SSD disk when zoning.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back