lock for reading device state
Hi,
I am using drbd to implement xen block device migration. Right now I am parsing /proc/drbd to find out if the drives are synchronized and I can migrate them. Is there a way to obtain a lock while reading and processing this information and prevent other writes to the primary device? I need to be able to prevent writes while reading the state of the device from a script external to drbd.
In case there is no existing solution, can you please give me a few tips on how to start developing such a locking mechanism?
Thank you very much.
Cristian
Lars Ellenberg
maintainer/ 2006-12-06 17:22:43 +0000
\ Cristian Zamfir:
Quoted 7 lines
Hi,
I am using drbd to implement xen block device migration. Right now I am parsing
/proc/drbdto find out if the drives are synchronized and I can migrate them.
you talk about drbd state "Connected, Consistent", or what exactly are you parsing?
Quoted 2 lines
Is there a way to obtain a lock while reading and processing this information and prevent other writes to the primary device?
no. why?
Quoted 2 lines
I need to be able to prevent writes while reading the state of the device from a script external to drbd.
does not make sense to me yet?
Quoted 2 lines
In case there is no existing solution, can you please give me a few tips on how to start developing such a locking mechanism?
please give more details about your assumptions and reasoning, maybe its just that you have wrong expectations?
could also be that I'm just mentally block right now...
:)
Signature · 3 lines
: Lars Ellenberg Tel +43-1-8178292-55 : : LINBIT Information Technologies GmbH Fax +43-1-8178292-82 : : Vivenotgasse 48, A-1120 Vienna/Europe http://www.linbit.com :
Quoted 11 lines · Lars Ellenberg
/ 2006-12-06 17:22:43 +0000
\ Cristian Zamfir:Quoted 6 lines
Hi,
I am using drbd to implement xen block device migration. Right now I am parsing
/proc/drbdto find out if the drives are synchronized and I can migrate them.you talk about drbd state "Connected, Consistent", or what exactly are you parsing?
Yes, indeed, I am parsing these values: "cs:Connected st:Secondary/Primary ld:Consistent"
Quoted 6 lines
Quoted 3 lines
Is there a way to obtain a lock while reading and processing this information and prevent other writes to the primary device?
no. why?
I wrote a script that parses /proc/drbd on the primary node. While I am running this script, writes to the primary device are still allowed. If I find that the ld state is "Consistent" then I will make this node secondary and the peer will become primary. The problem is when writes happen while my script is making the peer node primary.
A race situation would be the following:
At moment X, I read /proc/drbd and see the ld state is consistent.
At moment X+1 a write arrives at /dev/drbd1 and the devices are not consistent any more. They start syncing but this may last longer, for instance until moment X+5. Now, at moment X+2, I wrongly believe that the state is still consistentand I decide to make the peer node primary and thus loose the write at moment X+1.
Are my assumptions correct so far?
I'm thinking that there are two solutions: One would be to prevent any writes from Xen's domUs by modifying Xen.
The other would be to be able to hold a lock that prevents writes from reaching /dev/drbdX and release it after the processing within the script finishes (that is while I switch the peer device from secondary to primary).
I haven't looked at drbd's source yet ( I am using 0.7.22 now) but I am considering implementing this lock within drbd if there is no other solution available.
As a future project, I am also interested if there is anyone working on implementing multiple secondary devices. I am interested in having multiple replicas of the primary node.
I hope this explains more my question.
Thank you very much for your help.
Cristian
Quoted 15 lines
Quoted 2 lines
I need to be able to prevent writes while reading the state of the device from a script external to drbd.
does not make sense to me yet?
Quoted 2 lines
In case there is no existing solution, can you please give me a few tips on how to start developing such a locking mechanism?
please give more details about your assumptions and reasoning, maybe its just that you have wrong expectations?
could also be that I'm just mentally block right now...
:)
Lars Ellenberg
maintainer/ 2006-12-07 13:52:15 +0000
\ Cristian Zamfir:
Quoted 35 lines
Quoted 10 lines · Lars Ellenberg
/ 2006-12-06 17:22:43 +0000
\ Cristian Zamfir:Quoted 6 lines
Hi,
I am using drbd to implement xen block device migration. Right now I am parsing
/proc/drbdto find out if the drives are synchronized and I can migrate them.you talk about drbd state "Connected, Consistent", or what exactly are you parsing?
Yes, indeed, I am parsing these values: "cs:Connected st:Secondary/Primary ld:Consistent"
Quoted 3 lines
Quoted 2 lines
Is there a way to obtain a lock while reading and processing this information and prevent other writes to the primary device?
no. why?
I wrote a script that parses
/proc/drbdon the primary node. While I am running this script, writes to the primary device are still allowed. If I find that the ld state is "Consistent" then I will make this node secondary and the peer will become primary.
The problem is when writes happen while my script is making the peer node primary.A race situation would be the following:
At moment X, I read/proc/drbdand see the ld state is consistent. At moment X+1 a write arrives at/dev/drbd1and the devices are not consistent any more. They start syncing but this may last longer, for instance until moment X+5.
Now, at moment X+2, I wrongly believe that the state is still consistentand I decide to make the peer node primary and thus loose the write at moment X+1.
Quoted 1 line
Are my assumptions correct so far?
no. you don't "become Inconsistent" because "some write".
"Consistent" in drbd speak is "not Inconsistent".
oh well.
so what is Inconsistent.
drbd starts as beeing "inconsistent" when the meta data is first initialized. then you force one side to think it is Consistent, to be able to make it Primary, and the initial full sync starts.
Once the sync is finished, the sync target becomes Connected Consistent. If the nodes now disconnect, they still are "Consistent" in the sense of "whatever data is on that disk, it is transactional consistent, though maybe it is not 'clean', i.e. you may have to replay some journal to get into 'clean' state."
You get into "Inconsistent" only by becoming SyncTarget after (re)establishing the connection to the Peer and the handshake determins that your data is different from the Peers, and the Peers is "better" (which typically means "newer").
Because the Resync copies changed blocks linearly over the device, while new writes get mirrored already, the data on the SyncTarget is "not Consistent" anymore during sync. Even if we had data journalling during degraded mode, and would replay that during Sync, the SyncTarget would stay Consistent but "outdated" until the Resync was completely done.
Quoted 6 lines
I'm thinking that there are two solutions: One would be to prevent any writes from Xen's domUs by modifying Xen. The other would be to be able to hold a lock that prevents writes from reaching
/dev/drbdXand release it after the processing within the script finishes (that is while I switch the peer device from secondary to primary).I haven't looked at drbd's source yet ( I am using 0.7.22 now) but I am considering implementing this lock within drbd if there is no other solution available.
That "lock" does not make sense to me,
and even if you could do it, it won't solve that "race", it would only move it to some other point in time.
Note that a device in Secondary state denies access.
Also note that you cannot make a device Primary if it sees its Peer as being Primary (unless you use drbd8, and explicitly allow "two-primaries").
And a device that knows it is "Inconsistent" cannot be made Primary, unless it is Connected, in which case it would be SyncTarget and get the good data from the SyncSource Peer.
So what you need to do for xen migration with drbd 0.7 is:
Start the migration, once you think you want to switch over, i.e.
** once you are done writing on nodeA **
** you switch nodeA to Secondary. **
now, both nodes are Secondary, and neither can write.
now you can check wether the target nodeB is still Connected, Consistent. if so, you make it Primary.
if not, you abort the migration.
"locking" the state of drbd or freezing io while it is Primary on migration source nodeA won't help you in any way.
Quoted 3 lines
As a future project, I am also interested if there is anyone working on implementing multiple secondary devices. I am interested in having multiple replicas of the primary node.
here at LINBIT we have some very nice concepts about how we'd implement multiple (> 2) nodes and other nice features. But don't ask about timelines.
Signature · 3 lines
: Lars Ellenberg Tel +43-1-8178292-55 : : LINBIT Information Technologies GmbH Fax +43-1-8178292-82 : : Vivenotgasse 48, A-1120 Vienna/Europe http://www.linbit.com :
Quoted 31 lines · Lars Ellenberg
/ 2006-12-07 13:52:15 +0000
\ Cristian Zamfir:Quoted 29 lines
Quoted 9 lines · Lars Ellenberg
/ 2006-12-06 17:22:43 +0000
\ Cristian Zamfir:Quoted 5 lines
Hi,
I am using drbd to implement xen block device migration. Right now I am parsing
/proc/drbdto find out if the drives are synchronized and I can migrate them.you talk about drbd state "Connected, Consistent", or what exactly are you parsing?
Yes, indeed, I am parsing these values: "cs:Connected st:Secondary/Primary ld:Consistent"
Quoted 3 lines
Quoted 2 lines
Is there a way to obtain a lock while reading and processing this information and prevent other writes to the primary device?
no. why?
I wrote a script that parses
/proc/drbdon the primary node. While I am running this script, writes to the primary device are still allowed. If I find that the ld state is "Consistent" then I will make this node secondary and the peer will become primary. The problem is when writes happen while my script is making the peer node primary.A race situation would be the following:
At moment X, I read/proc/drbdand see the ld state is consistent. At moment X+1 a write arrives at/dev/drbd1and the devices are not consistent any more. They start syncing but this may last longer, for instance until moment X+5.
Now, at moment X+2, I wrongly believe that the state is still consistentand I decide to make the peer node primary and thus loose the write at moment X+1.
Quoted 3 lines
Quoted 1 line
Are my assumptions correct so far?
no. you don't "become Inconsistent" because "some write".
Thank you very much for your answer. I guess what I assumed incorrectly was that writes would make the device inconsistent.
Quoted 31 lines
"Consistent" in drbd speak is "not Inconsistent".
oh well.
so what is Inconsistent.
drbd starts as beeing "inconsistent" when the meta data is first initialized. then you force one side to think it is Consistent, to be able to make it Primary, and the initial full sync starts.Once the sync is finished, the sync target becomes Connected Consistent. If the nodes now disconnect, they still are "Consistent" in the sense of "whatever data is on that disk, it is transactional consistent, though maybe it is not 'clean', i.e. you may have to replay some journal to get into 'clean' state."
You get into "Inconsistent" only by becoming SyncTarget after (re)establishing the connection to the Peer and the handshake determins that your data is different from the Peers, and the Peers is "better" (which typically means "newer").
Because the Resync copies changed blocks linearly over the device, while new writes get mirrored already, the data on the SyncTarget is "not Consistent" anymore during sync. Even if we had data journalling during degraded mode, and would replay that during Sync, the SyncTarget would stay Consistent but "outdated" until the Resync was completely done.
Quoted 4 lines
I'm thinking that there are two solutions: One would be to prevent any writes from Xen's domUs by modifying Xen.
The other would be to be able to hold a lock that prevents writes from reaching/dev/drbdXand release it after the processing within the script finishes (that is while I switch the peer device from secondary to primary).I haven't looked at drbd's source yet ( I am using 0.7.22 now) but I am considering implementing this lock within drbd if there is no other solution available.
Quoted 8 lines
That "lock" does not make sense to me,
and even if you could do it, it won't solve that "race", it would only move it to some other point in time.Note that a device in Secondary state denies access.
Also note that you cannot make a device Primary if it sees its Peer as being Primary (unless you use drbd8, and explicitly allow "two-primaries").
I assume that using drbd8 would make xen bloc device migration easier because both devices are primary. Am I right?
Quoted 4 lines
And a device that knows it is "Inconsistent" cannot be made Primary, unless it is Connected, in which case it would be SyncTarget and get the good data from the SyncSource Peer.
Quoted 8 lines
So what you need to do for xen migration with drbd 0.7 is:
Start the migration, once you think you want to switch over, i.e.
** once you are done writing on nodeA **
** you switch nodeA to Secondary. **
now, both nodes are Secondary, and neither can write.
now you can check wether the target nodeB is still Connected, Consistent. if so, you make it Primary.
if not, you abort the migration.
This is exactly what my code is doing now. I was worried that writes would make the drive inconsistent so that is why I needed the lock. Now it is clear that making the transition from primary to secondary is enough.
Quoted 11 lines
"locking" the state of drbd or freezing io while it is Primary on migration source nodeA won't help you in any way.
Quoted 3 lines
As a future project, I am also interested if there is anyone working on implementing multiple secondary devices. I am interested in having multiple replicas of the primary node.
here at LINBIT we have some very nice concepts about how we'd implement multiple (> 2) nodes and other nice features. But don't ask about timelines.
It is great that you are considering this because I will also start working on something similar in the near future.
Thanks,
Cristian