Is it safe to use wfc-timeout in DRBD configuration?

[giner]

Hello fluke, hello everybody,

Today I have an experiment with a real system and I got that I was exatly right. Here is my steps to reproduce.
1. Virtual Windows XP is working on virt1
2. I stop virt2.
3. Create directory test1 and file test1.txt on the "Virtual Windows XP".
4. Stop virt1.
5. Start virt2. DRBD become master (in 15 seconds after DRBD started).
6. Start virt1. DRBD synchronize with virt1 and become master.
7. Check my directory test1 and file test1.txt. RESULT: there are NO THESE FILES on VIRTUAL MACHINE.

Conclusion: wcf-timeout number must be much greater than 15 seconds (10 - 30 minutes) or equal zero.

 

[fluke]

Hello fluke, hello everybody,

Today I have an experiment with a real system and I got that I was exatly right. Here is my steps to reproduce.
1. Virtual Windows XP is working on virt1
2. I stop virt2.
3. Create directory test1 and file test1.txt on the "Virtual Windows XP".
4. Stop virt1.
5. Start virt2. DRBD become master (in 15 seconds after DRBD started).
6. Start virt1. DRBD synchronize with virt1 and become master.
7. Check my directory test1 and file test1.txt. RESULT: there are NO THESE FILES on VIRTUAL MACHINE.

That's interesting, unfortunately I can't test it, because my servers are production, but if someone else can test and share results, will be appreciated.

Conclusion: wcf-timeout number must be much greater than 15 seconds (10 - 30 minutes) or equal zero.

The safest way is to set it to zero.

 

[nmoas]

What magic did you expect? It is exactly what happens when you do this with mirrored disks (RAID-1):
- You have disk1 and disk2 in sync.
- You remove your hotpluggable disk1
- You write to disk0 the file helloworld.txt
- You kill the box by removing disk0 too
- You put in disk1 and reboot
- The file helloworld.txt is not (and was never) on disk1
- You put in your hotpluggable disk0 and start a resync
- Now your RAID-card will sync (copy each block) the mounted/running disk1 to the just insertet disk0 and this will delete helloworld.txt on your disk0 too.
- Yes: The file helloworld.txt is dead and gone

RAID and drbd do a simple block wise sync - not a file-wise or database-record-wise sync like organizers do with their files or addresses. Organizers use files and database records that have timestamps - blocks have no timestamps so the master does nothing but overwrite the slave. Even with disks you have to be verry carefull !!!

 

[giner]

What magic did you expect? It is exactly what happens when you do this with mirrored disks (RAID-1):
- You have disk1 and disk2 in sync.
- You remove your hotpluggable disk1
- You write to disk0 the file helloworld.txt
- You kill the box by removing disk0 too
- You put in disk1 and reboot
- The file helloworld.txt is not (and was never) on disk1
- You put in your hotpluggable disk0 and start a resync
- Now your RAID-card will sync (copy each block) the mounted/running disk1 to the just insertet disk0 and this will delete helloworld.txt on your disk0 too.
- Yes: The file helloworld.txt is dead and gone

RAID and drbd do a simple block wise sync - not a file-wise or database-record-wise sync like organizers do with their files or addresses. Organizers use files and database records that have timestamps - blocks have no timestamps so the master does nothing but overwrite the slave. Even with disks you have to be verry carefull !!!

I think you right. But there is one difference. When I use RAID-1 and do reboot/shutdown server all drives stop simultaneously. In the case of DRBD I can't start both servers at the same time exactly.

 

[nmoas]

If you did shutdown your servers correctly (sync drbd, stop writing to drbd, dismount drbd-volume, shutdown) - this isn't a problem: then both drbd-volumes have the same data (no split brain) and you needn't care about which one is master or slave.

If you find your systems powered off and you do not know what has happened then be carefull - (maybe split brain) - first you should probably test both systems in a split mode - then decide which one has the better/newer data (or maybe both might have valuable data) and make the better one the master.

But it is the same with RAID-1 disks - if you find them both, marked "out of sync", then you should first take a look which is the better one (or maybe both might have some valuable data) and then you may decide what should be done.

 

[giner]

If you did shutdown your servers correctly (sync drbd, stop writing to drbd, dismount drbd-volume, shutdown) - this isn't a problem: then both drbd-volumes have the same data (no split brain) and you needn't care about which one is master or slave.

If you find your systems powered off and you do not know what has happened then be carefull - (maybe split brain) - first you should probably test both systems in a split mode - then decide which one has the better/newer data (or maybe both might have valuable data) and make the better one the master.

But it is the same with RAID-1 disks - if you find them both, marked "out of sync", then you should first take a look which is the better one (or maybe both might have some valuable data) and then you may decide what should be done.

In that case what the reason to use wfc-timeout?

 

[nmoas]

In that case what the reason to use wfc-timeout?

See the drbd manual.

Without the wfc-timeout a drbd cluster will start to boot - but NOT become operable when one of the configured cluster-members (ressources) is not available - in other words: Without a wfc-timeout (or set to 0), the cluster will boot and be available ONLY if all configured nodes are there and happy. This prevents from getting split brained.

The risk is:
If you set the timeout (lets say to one minute) and have clients connected separately to both boxes and both boxes have no working interconnect, then on boot your systems will wait for one minute and try to find the other box - but without the working interconect, they won't find eachother - then, after one minute, both will boot and come up and serve their clients in a split brain condition.

The benefit is:
If one box is dead, the other will still boot and be able to serve your connected clients.

Warning:
With a wfc-timeout set, NEVER power up both Systems without a working interconnect.

 

[giner]

You are right.
When you use wfc-timeout=0 you can skip wcf-timeout interactively by pressing the key.
When you use wfc-timeout=15 you need to know which of servers was shut downed last and turn on it first. But sometimes you can't know what server was last.

 

[nmoas]

You are right. But sometimes you can't know what server was last.

Then be carefull !!! (maybe it is a split brain)
Then (first) you should test both systems in split mode (separately - with only one station connected - the one station you use for testing) and then decide which box has the better/newer data (or maybe both might have valuable data) and then make the better one the master.

 

[testbot]

would it be useful to have a cron that runs once per minute with 'date > /var/log/date' (use log rotate or something so it doesn't get huge) so if in the above situation you will have a date stamp on both systems and know which one was up last?

 

[giner]

Today's story

I have forgotten to set wfc-timeout to zero and we have lost two domain controllers.

What exactly has happened.

There are two servers are node1 and node2. Our DCs are on node1. VMs' images are on a dual-primary DRBD.
There was a planned power cut:
1) we have shut-downed our servers in advance (node2 first and then node1). Let's say node2 was stopped at the time_A and node1 at the time_B.
2) after power is back we started node2 first (it was our mistake) and then node1
3) first node2 (time_A) became primary in 60 seconds (wfc-timeout) and then node2 was synced to node1 when it came up.

So node1 became also time_A. A month later we've got a problem with our DCs, they just hadn't been working. Nobody can login, servers didn't work, etc. Only DNS worked quite well. We started to investigate and found out that DC controllers were rolled back a month ago and became out of scope. They hadn't been synchronized with other DCs anymore.

That is why we have to be very careful with non-zero wfc-timeout and I recommend to use set it only zero and nothing else.

 

[testbot]

Today's story

I have forgotten to set wfc-timeout to zero and we have lost two domain controllers.

What exactly has happened.

There are two servers are node1 and node2. Our DCs are on node1. VMs' images are on a dual-primary DRBD.
There was a planned power cut:
1) we have shut-downed our servers in advance (node2 first and then node1). Let's say node2 was stopped at the time_A and node1 at the time_B.
2) after power is back we started node2 first (it was our mistake) and then node1
3) first node2 (time_A) became primary in 60 seconds (wfc-timeout) and then node2 was synced to node1 when it came up.

So node1 became also time_A. A month later we've got a problem with our DCs, they just hadn't been working. Nobody can login, servers didn't work, etc. Only DNS worked quite well. We started to investigate and found out that DC controllers were rolled back a month ago and became out of scope. They hadn't been synchronized with other DCs anymore.

That is why we have to be very careful with non-zero wfc-timeout and I recommend to use set it only zero and nothing else.

that's scary.

the very thing you're trying to protect against becomes the cause.