Install on Soft Raid

[dietmar]

/etc/apt/sources.list sould look like:

# Standard debian repository
deb [URL]http://ftp.debian.org/debian[/URL] lenny main contrib
# PVE packages provided by proxmox.com
deb [URL]http://download.proxmox.com/debian[/URL] lenny pve
# security updates
deb [URL]http://security.debian.org/[/URL] lenny/updates main contrib

 

[mangoo]

I installed Proxmox VE 1.1 on software RAID manually.

And after an upgrade to 1.2 it didn't boot up with 2.6.24-5-pve kernel (kernel panic, couldn't find init).


The problem is that for some reason, on upgrade, the system creates the initrd with "mkinitrd". It produces an initrd which doesn't boot the system; at least my system (LVM on software RAID).

Running this command (from an old kernel):

update-initramfs -k 2.6.24-5-pve -v -u -t

fixes it and the system can boot with a new kernel.

 

[dietmar]

our kernel does not use the update-initramfs tolls and is thus incompatible with that (each kernel update will break your modification).

 

[dietmar]

We manage > 100 servers ...

Just curious if my reliability computation are correct. If you have 100 servers with (2 disks), you need to replace about 5 disks per year? Is it less or more?

 

[gijsbert]

Dietmar, each server has a Seagate and Western Digital (server)disk and last year we replaced approx. 10 disks; 9 of them where Western Digital. It seems we bought a bunch of WD-disks which were all bad; maybe the delivery boy dropped the box

In 12 months we only replaced 1 Seagate disk which was not even dead but showed some bad sectors. This year we only use Seagate serverdisks (although slower in writing than WD) and we didn't have any replacement so far. Under normal circumstances I would say it's 1 out of a 100.

Back to sw-raid vs hw-raid. There are man pros and cons for each situation. When a hw-raid controller dies we have to drive to our datacenter, replace the controller, make sure we have the right firmware and so on. It can take up to 1 day before our client is back in business. Beside that, some hw-raid controllers need to be in "bios-mode" when rebuilding. With sw-raid we do not have this problem at all; downtime is limited to powering down the system and replace the harddisk Performance is a different issue.

We're currently testing performance (read/write) of disks running CentOS and different virtual environments (proxmox and xenserver) in a soft- and hardware-raid-1 (3ware 9690). If you're interested in the results, please let me know.

We do have the knowledge to install proxmox on a sw-raid system, that's not the problem. But I believe that IF proxmox is supporting sw-raid your software can become a very hot item in the virtualization world. Check out the forums on vmware, xen and you'll see that a lot of people "begging" for a sw-raid solution. There's a big market out there

Gijsbert

 

[tom]

I have one questions for the software raid fans regarding hard drive cache.

If you use software raid, did you disable the hard drive cache?

If not, the operation system get feedback that data is written but its still in the cache (not protected by a BBU) - in the case of a power failure you loose data and in the worst scenario you cannot start any of your virtual machines on that host.

Hardware RAID:
You need to enable the cache on the hardware raid controller and DISABLE the cache on the hard disk. therefore you still get fast performance due to the fast controller cache and you are not in any risk of loosing data as this cache is protected by the BBU.

Am I wrong?

 

[gijsbert]

Tom, you're right. However, I have to admit that I never disable cache on harddisks. We're in business now for 12 years and have never experienced any problems. But in those 12 years we've had a power outage only once, so maybe we have just been lucky so far.

I will also test the read/write performance on a sw-raid with cache on the disks disabled. I think the performance will drop. I've seen this happen on a hw-raid controller with no bbu and cache disabled. Of course we only enable cache on a hw-raid controller when a bbu is installed. When disabling cache on the hw-raid controller performance decrease dramatically. Will post my results within a week in case you're interested.

Gijsbert

 

[dietmar]

A HW Raid Controller without BBU makes no sense for me.

 

[tom]

Tom, you're right. However, I have to admit that I never disable cache on harddisks. We're in business now for 12 years and have never experienced any problems. But in those 12 years we've had a power outage only once, so maybe we have just been lucky so far.

I will also test the read/write performance on a sw-raid with cache on the disks disabled. I think the performance will drop. I've seen this happen on a hw-raid controller with no bbu and cache disabled. Of course we only enable cache on a hw-raid controller when a bbu is installed. When disabling cache on the hw-raid controller performance decrease dramatically. Will post my results within a week in case you're interested.

Gijsbert

yes, i am interested in your results, also post your test setup.

and yes again, disabling cache leads to a very slow system. so its all about IO performance and I can´t wait to get SSD everywhere.

factory defaults/firmware of hard drives are also very different, also in the same series. some come with cache disable, most not. I got some hp re branded disks with cache disabled - so prepared for the use with hw raid with BBU. And yes, activation hardware raid cache without BBU is NEVER a good idea.

 

[mangoo]

I have one questions for the software raid fans regarding hard drive cache.

If you use software raid, did you disable the hard drive cache?

If not, the operation system get feedback that data is written but its still in the cache (not protected by a BBU) - in the case of a power failure you loose data and in the worst scenario you cannot start any of your virtual machines on that host.

If you mount your filesystem with barrier=1 option, data should be safe even if your system crashes (or is powered off brutally) with unsaved data in disk cache.

However, barriers are not supported when used with software RAID or LVM (2.6.30 kernel should be able to do it once it's released), so it's still better to disable disk cache.

Hardware RAID:
You need to enable the cache on the hardware raid controller and DISABLE the cache on the hard disk. therefore you still get fast performance due to the fast controller cache and you are not in any risk of loosing data as this cache is protected by the BBU.

Am I wrong?

In theory you're right, but I'm afraid it's not enough.

By default, Proxmox VE saves guest images as normal files. As such, they are subject to all caching done on the host side.

For example, on a host with lots of memory, try to do some heavy writes ("dd if=/dev/zero of=bigflie" usually is enough). As you do it, see how much unsaved data the system keeps in cache (cat /proc/meminfo - look for Dirty/Writeback). It can be hunderds of megabytes! How does that compare to tiny 16/32 MB disk cache?

Now, interrupt your heavy writing (ctrl+c) and do "time sync". Depending on fragmentation, other load, disk speed and cached data, it may take some time before it actually happens (several minutes to just "sync" is nothing unusual on some machines).


What does it all mean for Proxmox VE and its default settings? It makes no difference if you use hardware or software RAID; in both cases, if your host system fails (no electricity etc.) you may loose your data; worst case, not be able to start your guests anymore. Surprised?

To avoid potential data loss on guests with either software or hardware RAID, one should use cache=none option to kvm (see "man kvm" for details). Or, use block devices instead of files (i.e. each guest on a separate LVM volume).


To sum up, hardware RAID does not protect you much more than software RAID with default Proxmox VE settings (unless I'm mistaken somewhere). And, RAID is no backup replacement

 

[dietmar]

To sum up, hardware RAID does not protect you much more than software RAID with default Proxmox VE settings (unless I'm mistaken somewhere). And, RAID is no backup replacement

We will use lvm volumes by default (soon).

 

[tom]

another question to soft raid users:

is it possible to get hot plug back-planes working?

An example (hardware raid, hot plug back-planes):

I am used to get email notification from my server systems about a failed hard drive. Then I walk into the server room and I see 50 hard drives in different servers - 49 green lamps and 1 with a red lamp.

so its easy to find and replace the disk (we got hot spare drives and also a bunch of cold spares). its just one minute and the risk of doing something wrong is very limited.

how do you do this in your daily business with softraid? power off, open the server, how do you find the faulty disk in the case?

 

[gijsbert]

hw-raid without BBU makes no sense at all, you're absolutely right about that one. But sw-raid and cache on disks enabled should not be a real problem. There might be a very small chance that we will face a complete power outage and if that's the case and I loose data in such way that I cannot start the virtual machines, there's always a backup. Disabling cache on a harddisk is not really sw-raid related. If one is afraid of loosing data when a power outage occur they should _always_ disable cache, also on servers or client-PC's with a single disk.

As described in my first thread, there are many pros and cons for sw-raid as well as hw-raid and I can fully understand that proxmox is not going to (officially) support sw-raid. I only know there are many people looking for a sw-raid and from a commercial point of view it might be very interesting for supporting this.

In our situation we like 1 vm on 1 server for easy migration and upgrades in the future. I can simply install OS (sw-raid) and put openVZ-kernel on top of that. But proxmox has some very nice advantages over the CLI of openVZ. Adding 100+ hw-raidcontrollers with BBU on each server is not an option... I will do some more tests the nextcoming days and I will give you the results once I'm done.

Gijsbert

 

[thefool808]

Shouldn't there also be a very small chance you even see a complete power outage? When talking about reliability (and not having to go to your tapes to get data back), aren't we assuming UPSs with automatic shutdown?

 

[mangoo]

another question to soft raid users:

is it possible to get hot plug back-planes working?

You can do software RAID on about anything (i.e. one part local, one accessible through iSCSI or nbd), so it will depend on the underlying hardware.

But in short, with most SATA or SCSI controllers, yes.

There are some things you have to do manually:
- kick the faulty disk out of sw RAID
- replace the drive (of course)
- if hardware does not support hotplugging (but only coldplugging), reset the controller (echo - - - > /proc/...).
- add a drive to sw RAID

An example (hardware raid, hot plug back-planes):


I am used to get email notification from my server systems about a failed hard drive. Then I walk into the server room and I see 50 hard drives in different servers - 49 green lamps and 1 with a red lamp.


so its easy to find and replace the disk (we got hot spare drives and also a bunch of cold spares). its just one minute and the risk of doing something wrong is very limited.

Mind that not all hw controllers will have this shiny red lamp.

If the drive has no such indicator, it's usually enough to do something like:

dd if=/dev/faulty_drive of=/dev/null

and see which activity LED blinks most

how do you do this in your daily business with softraid? power off, open the server, how do you find the faulty disk in the case?

If you have LEDs, you don't have to do it - it is equally true for software and hardware RAID.

With hardware RAID controller which don't have this magic red/green indicator, it gets problematic, as you usually can't access the drive directly and make activity LEDs blinking.

 

[mangoo]

Shouldn't there also be a very small chance you even see a complete power outage? When talking about reliability (and not having to go to your tapes to get data back), aren't we assuming UPSs with automatic shutdown?

UPS devices fail too!

Some will have self-tests triggered from time to time to warn you if anything bad happens.

But yes, it happens very rarely - usually, in the most unexpected moment

 

[gijsbert]

another question to soft raid users:

is it possible to get hot plug back-planes working?

An example (hardware raid, hot plug back-planes):

I am used to get email notification from my server systems about a failed hard drive. Then I walk into the server room and I see 50 hard drives in different servers - 49 green lamps and 1 with a red lamp.

so its easy to find and replace the disk (we got hot spare drives and also a bunch of cold spares). its just one minute and the risk of doing something wrong is very limited.

how do you do this in your daily business with softraid? power off, open the server, how do you find the faulty disk in the case?

I get e-mailnotification (/etc/mdadm.conf). With /proc/mdstat I see which one is the faulty disk. The only thing I need to know is where servers are located in the server room, but that's a matter of administration.

 

[thefool808]

UPS devices fail too!

Some will have self-tests triggered from time to time to warn you if anything bad happens.

But yes, it happens very rarely - usually, in the most unexpected moment

Yeah, I only say that in response to people saying that losing data because of having cache enabled without battery backup is an ignorable edge case. Data safety is all about the edge case.

 

[tom]

Shouldn't there also be a very small chance you even see a complete power outage? When talking about reliability (and not having to go to your tapes to get data back), aren't we assuming UPSs with automatic shutdown?

UPS can fail, the mainboard, very often the power supply, etc.

 

[user100]

To sum up, hardware RAID does not protect you much more than software RAID

If a lonely hardware-raid-controller burn then it´s not just "not much more" - In that case the hardware-raid is much worse than a software-raid. If you know what you are doning software-raid can be a good thing. We have a backup of important servers on tape but I never needed it to reconstruct a hole server. Mostly I needed the backup if some user deleted some data by misstake (and there raid does not really help anyway).


Greetings,
user100