Advice seeked to configure node to be as resilient as possible

[lpallard]

Hello all,

I am seeking advices from experienced proxmox users in regards to reconfiguring my proxmox node to be the most resilient possible. I am not looking to upgrade my hardware and I am not looking to cluster, I will continue to use a single standalone node.

I am mainly interested in these three key elements:

Data reliability & integrity
Ease of backup and data protection
Capability to recover from disaster (doesnt include motherboard or power supply failure but mainly HDD and RAID controller failures)

My current setup is not very efficient and does not use the storage concepts so effectively. There are a lot of wasted/unused space, I am not using advanced filesystems such as ZFS and I am not using LVM effectively.

This is the current config:

Volume 1
2X 300GB SAS in RAID1 on a M5016 with BBU and cache.
Adapter 0 -- Virtual Drive Information:
Current Cache Policy: WriteBack, ReadAhead, Cached, No Write Cache if Bad BBU
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Is VD Cached: Yes
Cache Cade Type : Read Only

Volume 2
3X 2TB SATA in RAID5 on the M5016 with BBU and cache
Current Cache Policy: WriteBack, ReadAhead, Direct, No Write Cache if Bad BBU
Current Access Policy: Read/Write
Disk Cache Policy : Disabled
Is VD Cached: N/A
Cache Cade Type : N/A

Volume 3
6X 2TB SATA in RAID5 on a M1015 with NO BBU or cache

***
root@proxmox:~# df -H
Filesystem Size Used Avail Use% Mounted on
udev 11M 0 11M 0% /dev
tmpfs 6.8G 476k 6.8G 1% /run
/dev/mapper/pve-root 32G 1.5G 29G 5% /
tmpfs 5.3M 4.1k 5.3M 1% /run/lock
tmpfs 14G 23M 14G 1% /run/shm
/dev/mapper/pve-data 209G 171G 38G 83% /var/lib/vz
/dev/sda2 518M 86M 406M 18% /boot
/dev/mapper/datastore-backups 318G 272G 30G 91% /mnt/backups
/dev/fuse 32M 17k 32M 1% /etc/pve
***

Questions:

/ is currently 32GB and 5% (1.5GB) full which on expensive 15krpm SAS drives is a WASTE of precious data real-estate. Assuming I will never use / to save VMs or other stuff, can I safely reinstall proxmox and configure it to have / lets say set at 5GB?

All partitions on Volume 1 are using ext3 and LVM. Should I use LVM with ext3 for /boot and /dev/mapper/pve-root (since its default for proxmox) and use LVM+ZFS for /dev/mapper/pve-data to be able to do ZFS snapshots to backup my VMs???

I am currently storing data (databases and other files) in VM's instead of storing these files on volumes (partitions) and then passing these volumes to the VMs. This is clumsy and inefficient as backing up a 15GB VM actually backs up all data which may take up to 120GB of space. Should I do NO data storage inside the VM's?

What is the most reliable way to backup VM's? ZFS snapshots? Backup through the Proxmox WebUI?

My "mass" storage volume is volume 2 which (after parity) had 3.64TB of usable space. I currently have a large 3.34TB partition (LVM) with no filesystem on it and then passed to a specific VM to use for storage. Then at the VM level, I formatted the partition with ext4 and use it to store files. I dont like that since I feel something may happen as the partition is "managed" from the node and the VM at the same time. What would be the best way to take a partition and give it to a VM as if it were a standalone hard drive?

Thats enough for now! If you actually read everything to this point, please at least post a few ideas so I (and other begineers) can learn how to use proxmox the best possible way.

Thank you all!

 

[lpallard]

If you actually read everything to this point, please at least post a few ideas so I (and other begineers) can learn how to use proxmox the best possible way.

Ok I'm kindly bumping this thread because 60 views and not a single reply, I'd like to know why. Is this thread to general and vague and not specific enough? Too complicated to reply?

 

[Steijn van Essen]

Hello Ipallard,

As a ProxMox newbie myself I cannot advise you. But I can share some thoughts w u with respect to your three key elements:

- data reliability and integrity: IMO these elements refer to the protection from tampering of your data by someone. Protective measures for this may include encryption and signing of your data and access control to your data.
- when you're looking for data availability other words cross my mind, like reliable hardware and software (or "stable" at least and so far I feel quite comfortable with ProxMox ;-) the use of CRC (by default on most storage devices and network protocols) and RAID.
- RAID: I see the advantages of it and like using it myself. But be aware that it is not THE solution for availability of your data, for:
- a 2nd disc may fail before you've fully recovered the first
- at some point in time you might have no replacement disc left
- when using hardware (controller) based RAID you might not be able to read your discs anymore after a controller crash (for this reason I run two nodes w (almost) equal hw configuration, so worst case I can hot swap my RAID set to my other node for full data recovery)

 

[m.ardito]

I'm not an expert, but:

reconfiguring my proxmox node to be the most resilient possible. I am not looking to upgrade my hardware and I am not looking to cluster, I will continue to use a single standalone node.

this poses serious limitations to your scope as one single node gives you no means to recover permanent host issues. your node is the only hypervisor, has all configs, vm filesystems, all data... bad choice, in this respect, anyway I think you are aware of this.

My current setup is not very efficient and does not use the storage concepts so effectively. There are a lot of wasted/unused space, I am not using advanced filesystems such as ZFS and I am not using LVM effectively.
Volume 1 2X 300GB SAS in RAID1
Volume 2 3X 2TB SATA in RAID5
Volume 3 6X 2TB SATA in RAID5
/ is currently 32GB and 5% (1.5GB) full which on expensive 15krpm SAS drives is a WASTE of precious data real-estate. Assuming I will never use / to save VMs or other stuff, can I safely reinstall proxmox and configure it to have / lets say set at 5GB?

you have near 20TB of raw storage, are 17GB (x2) so important? anyway yes, you could reduce that, just use special install/boot config see wiki there is a page)

I am currently storing data (databases and other files) in VM's instead of storing these files on volumes (partitions) and then passing these volumes to the VMs. This is clumsy and inefficient as backing up a 15GB VM actually backs up all data which may take up to 120GB of space. Should I do NO data storage inside the VM's?

if that data is backed up in other ways (or if it shouldn't be), you can put it on a separate disk than the vm filesystem, and flag it to as "no backup" in the disk config.

What is the most reliable way to backup VM's? ZFS snapshots? Backup through the Proxmox WebUI?

I think the default pve backup. Not so efficient, since it is always a full (there are patches around, but they are not supported by pve team) but it is reliable. more reliable: test restore those backups. more reliable: use two (or more) different methods. expensive, but you'll have more recovery option. and test restores. and all backup storages,and if possible not store backups on your only node...

volume 2.. with no filesystem .. passed to a specific VM to use .. formatted the partition .. to store files. I dont like that since I feel something may happen as the partition is "managed" from the node and the VM at the same time. What would be the best way to take a partition and give it to a VM as if it were a standalone hard drive?

IMHO that is a standalone drive. If your node has not mounted it, only the vm has control of it, apart from raid controller (you could not raid this and do that in the vm (soft) so the vm has more control of it, since your concern is resilience and not performance).

Marco

 

[Steijn van Essen]

Sorry I was not able to finish my reply earlier, here it comes:

Since you only have one node you might consider sw RAID instead of hw RAID. At the cost of a small (?) performance penalty you'll be able to recreate your "software RAID controller" after it crashed.
Having said this I will not rely on RAID alone when it comes to my data. So before I started playing w ProxMox I (as a mere Windows guy) always made manual backups of my data and some of my system settings. Now w ProxMox I'll take a serious look at rsync for my data. Rsync combined w cron, nfs and some shell scripting probably could do the (backup) job for me.

 

[Steijn van Essen]

And some more:

File systems: since I want to integrate my real environment w my virtual environment I will use cifs/smb for my Windows clients and probably nfs for my ProxMox nodes, Linux clients (real and virtual) and nfs for rsync backups. I can't advise you on VM backups. I will have to build my own experience on that first. But I read about the qcow2 file system as a possible candidate for VM system volumes. It keeps track of changes to your virtual volumes and also allows you to revert to an older state of your VM installation in case you messed it up. I don't think, however, that I will use qcow2 for storage of virtual data volumes, unless my experience w rsync turn out to be a disappointment (which I don't expect).

Well, I'm not sure if the above is of any help to you, but at least you now have a serious reaction to your post

Steijn