Problem new server with SSD and LVM thin volume

alain

Renowned Member
May 17, 2009
223
2
83
France/Paris
Hello all,

I would like to have some advice on a configuration problem with a new server we bought recently. It is a Dell PE R730 with two 200 GB SSDs in Raid 1, and 8 500 GB NLSAS drives in Raid 10.

We bought this configuration to use in the future this server in a Ceph cluster.as described in Proxmox Ceph Documentation. But for now, we don't have the three servers required to set up a miminal cluter, so we installed it as a normal proxmox server with local storage using last iso PVE 4.2.

The two SSDs in Raid 1 are seen as /dev/sda, and the 8 NLSAS in Raid 10 as /dev/sdb. During installation, PVE 4.2 was installed on /dev/sda. It went OK but after reboot, only a small 45 GB appeared as local storage under /var/lib/vz. The other Raid 10 volume with ~2 To of space was not available.

With PVE 4.2, there is the possibility to use Thin LVM, which enable snapshot and clone, so I thought it would be interesting to setup the additional storage as a thin LVM storage.

For those interested, I used these commands to achieve this, following the wiki:

Code:
# sgdisk -p /dev/sdb
# pvcreate --metadatasize 250k -y -ff /dev/sdb1
# vgcreate raid10-pve /dev/sdb1
# lvcreate -L 1.8T -T -n raid10-store raid10-pve

Then I was able to add this storage as thin LVM in proxmox web interface. So far, all is right.

But then, I added the server in an existing proxmox cluster, with all nodes using a local storage under /var/lib/vz. The thin LVM storage at first disappeared, but I was able to re-add it when connecting on this node.

But when I try to do an offline migration from one local storage on another node to this new node, the VM is migrated by default under the /var/lib/vz storage which has only 45 GB, and not to the thin LVM storage with 2 TB... And there is no option in the migration GUI (or in commande line), to specify the storage.

So my questions are :
- is there a possibility to offline migrate a VM from a standard /var/lib/vz storage to a thin LVM volume on another node ?
- in case it is, is it possible to define the thin LVM storage as the default one for migration ?
- Or alternatively, is it possible to remove the /var/lib/vz storage, so that only the thin LVM volume remains, and uses it for offline migration ?

I fear I cannot use thin LVM storage for a new node in a cluster which is configured with old local /var/lib/vz storage, and that I have to reconfigure the new server.

What are the best options in this case ?

Thank for your advices.

Alain
 
Last edited:
So my questions are :
- is there a possibility to offline migrate a VM from a standard /var/lib/vz storage to a thin LVM volume on another node ?
- in case it is, is it possible to define the thin LVM storage as the default one for migration ?
- Or alternatively, is it possible to remove the /var/lib/vz storage, so that only the thin LVM volume remains, and uses it for offline migration ?

unfortunately, the answer to those three is no. when migrating, proxmox expects the storage of each disk/volume/mountpoint to be available on the target node, and in case of a local disk/volume/mountpoint, it will use it as target of the disk/volume/mountpoint migration. for this reason it is advisable to use similar / identical storage setups on all nodes, or to use shared storage to decouple the hypervisor systems from the storage system(s)

I fear I cannot use thin LVM storage for a new node in a cluster which is configured with old local /var/lib/vz storage, and that I have to reconfigure the new server.

What are the best options in this case ?

you could also put the local storage (i.e., the directory /var/lib/vz) on top of the thin LV - but this only makes sense if you want to avoid repartitioning its disk once your cluster has moved to thin LVM or if you have VMs / containers that you don't need to migrate to the other nodes, but which would benefit from thin LVMs additional features. adapting either your old servers to use thin LVM or your new one to only use a local dir storage seems more straightforward to me.
 
Hi Fabian,

Thanks for your answer. Your advice comes in the way I was thinking. It's not very sustainable to add a new server with thin LVM, in a cluster where it has not been configured previously. I would like to keep the possibility to migrate VMs between new node and the older, so I think I have to recreate a new standard LVM volume on the Raid 10 volume, and format it, then change the mountpoint to this new directory.

It is a little bit disappointing, as I was interested to benefit from snapshot and clone by default with raw image (more efficient and no risk they will fill the storage when growing), but I will use qcow2 image format if I need it.
It is also a bit of work to change the configuration to go back to standard configuration.

Thnks a lot for your advice.

Alain
 
Hi again,

I think there is a problem with the default configuration of a Thin LVM volume with 4.2 iso, when you have an existing cluster with no Thin LVM already configured. When I introduced this new node in my cluster, I saw a little /var/lib/vz partition, of about 45 Go, that was created inside the root / directory of the SSD. The remaining space, 185 G, was used by the default LVM Thin volume group.

When I tried to migrate to this new node a VM on the local storage of another node, it was migrated to this little storage. Evidently, not all VMs fits 45 Go... So, it is not very good.

So I ended up by removing the Thin LVM volumes (anothe one was on a Raid10), then recreate a classic LVM volume, format it, and mount it under /var/lib/vz.

It is a little bit annoying to have to do all this after default (and fast) Proxmox installation, in order to join an existing cluster with no Thin LVM storage.

I think it should be better to add an option to the installer asking if you want classis LVM storage, or Thin LVM volume...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!