Skip to contentSkip to conversation

ocfs2_controld.cman

Ocfs2-devel New David Teigland 3 participants 10 messages Apr 8, 2009 – Apr 14, 2009 OCFS2

lore
Share & tools
lore
#0 lore raw

If I start ocfs2_controld.cman in parallel on a few nodes, only one of them starts up, the others exit with one of these errors:

call_section_read at 370: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) call_section_read at 387: Checkpoint "ocfs2:controld" does not have a section named "daemon_protocol"

call_section_read at 370: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) call_section_read at 397: Unable to read section "daemon_protocol" from checkpoint "ocfs2:controld": Object does not exist

It does work ok if I remove those two checks.

Another thing I noticed while looking in the code is that it assumes a single node will become the first member of a cpg on its own when a bunch of nodes join at once: daemon_joined(daemon_group.cg_member_count == 1);

This isn't a correct assumption. It's possible that two or more nodes joining at once will become initial members together. (I realize that it's a very convenient assumption to make after using it in previous pre-cpg programs, and it may take a fair amount of work to do without.)

Dave

#1 lore raw
Quoted 10 lines · David Teigland

If I start ocfs2_controld.cman in parallel on a few nodes, only one of them starts up, the others exit with one of these errors:

call_section_read at 370: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) call_section_read at 387: Checkpoint "ocfs2:controld" does not have a section named "daemon_protocol"

call_section_read at 370: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) call_section_read at 397: Unable to read section "daemon_protocol" from checkpoint "ocfs2:controld": Object does not exist

It does work ok if I remove those two checks.

These checks are required - otherwise you end up with unsync'd
daemons, which is crap.
I've changed the daemon to wait indefinitely, and that's
something lmb was testing. See the controld-fixes branch of ocfs2-tools.git. That should fix these problems.

Quoted 8 lines

Another thing I noticed while looking in the code is that it assumes a single node will become the first member of a cpg on its own when a bunch of nodes join at once: daemon_joined(daemon_group.cg_member_count == 1);

This isn't a correct assumption. It's possible that two or more nodes joining at once will become initial members together. (I realize that it's a very convenient assumption to make after using it in previous pre-cpg programs, and it may take a fair amount of work to do without.)

Well, this is going to be fun. I have to figure out which
daemon is the "first", and now it's just racy. I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

Joel

Signature · 9 lines
"Three o'clock is always too late or too early for anything you
 want to do."
        - Jean-Paul Sartre

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#2 lore raw
Quoted 5 lines · Joel Becker

? ? ? ?Well, this is going to be fun. ?I have to figure out which daemon is the "first", and now it's just racy. ?I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

"In order" does not necessarily imply "one node at a time".

I don't consider it unreasonable for two nodes starting (effectively) simultaneously to appear in the first membership.
I believe Heartbeat had the same property.

Why not just take a lock when you want to create the daemon_protocol section (and allow the second guy to fail gracefully)? Perhaps cpg even has something like this built in...

#3 lore raw
Quoted 7 lines · Andrew Beekhof
Quoted 5 lines · Joel Becker

? ? ? ?Well, this is going to be fun. ?I have to figure out which daemon is the "first", and now it's just racy. ?I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

Quoted 5 lines

"In order" does not necessarily imply "one node at a time".

I don't consider it unreasonable for two nodes starting (effectively) simultaneously to appear in the first membership.
I believe Heartbeat had the same property.

Right, confchg order is guaranteed, but doesn't imply one node per confchg.

Quoted 3 lines

Why not just take a lock when you want to create the daemon_protocol section (and allow the second guy to fail gracefully)? Perhaps cpg even has something like this built in...

You probably want to use cpg messages to order things. So, for example, everyone sends a message proposing that it create the section, and the node whose message arrives first does it. If you're coordinating things with messages like this anyway, it's not much more work to include protocol information in the message and eliminate checkpoints.

Dave

#4 lore raw
Quoted 8 lines · David Teigland
Quoted 7 lines · Andrew Beekhof
Quoted 5 lines · Joel Becker

? ? ? ?Well, this is going to be fun. ?I have to figure out which daemon is the "first", and now it's just racy. ?I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

Quoted 7 lines
Quoted 5 lines

"In order" does not necessarily imply "one node at a time".

I don't consider it unreasonable for two nodes starting (effectively) simultaneously to appear in the first membership.
I believe Heartbeat had the same property.

Right, confchg order is guaranteed, but doesn't imply one node per confchg.

I have no problem with more than one join at the same time, but
somehow I had the idea that the first joiner would be alone. Consider me corrected.

Quoted 9 lines
Quoted 3 lines

Why not just take a lock when you want to create the daemon_protocol section (and allow the second guy to fail gracefully)? Perhaps cpg even has something like this built in...

You probably want to use cpg messages to order things. So, for example, everyone sends a message proposing that it create the section, and the node whose message arrives first does it. If you're coordinating things with messages like this anyway, it's not much more work to include protocol information in the message and eliminate checkpoints.

Ugh ugh ugh. This code is already a complex world of states
that are hard to keep in your head. It only gets worse the more things in flight. Checkpoints give us a nice way to look up data about other nodes without this hassle - they only give us a little pain in this setup phase.

Joel

Signature · 9 lines
"I am working for the time when unqualified blacks, browns, and
 women join the unqualified men in running our overnment."
	- Sissy Farenthold

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#5 lore raw
In reply to Andrew Beekhof
Quoted 9 lines · Andrew Beekhof
Quoted 5 lines · Joel Becker

? ? ? ?Well, this is going to be fun. ?I have to figure out which daemon is the "first", and now it's just racy. ?I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

"In order" does not necessarily imply "one node at a time".

Quoted 7 lines

I don't consider it unreasonable for two nodes starting (effectively) simultaneously to appear in the first membership.
I believe Heartbeat had the same property.

Why not just take a lock when you want to create the daemon_protocol section (and allow the second guy to fail gracefully)? Perhaps cpg even has something like this built in...

I don't want to rely on dlm in this daemon. These control
daemons are complex enough, they are our only connection between the fs and the stack,and we need to make them correct.

Joel

Signature · 9 lines
"Ninety feet between bases is perhaps as close as man has ever come
 to perfection."
	- Red Smith

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#6 lore raw
In reply to Joel Becker
Quoted 8 lines · Joel Becker
Quoted 4 lines

This isn't a correct assumption. It's possible that two or more nodes joining at once will become initial members together. (I realize that it's a very convenient assumption to make after using it in previous pre-cpg programs, and it may take a fair amount of work to do without.)

Well, this is going to be fun. I have to figure out which daemon is the "first", and now it's just racy. I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

It may just work to have both race to create the checkpoint, the loser should get an error back from create (I haven't tried it, but I'd expect it to work that way.)

Dave

#7 lore raw
Quoted 6 lines · David Teigland
Quoted 5 lines · Joel Becker
Quoted 4 lines

This isn't a correct assumption. It's possible that two or more nodes joining at once will become initial members together. (I realize that it's a very convenient assumption to make after using it in previous pre-cpg programs, and it may take a fair amount of work to do without.)

Quoted 7 lines
Quoted 3 lines

Well, this is going to be fun. I have to figure out which daemon is the "first", and now it's just racy. I could swear that someone told me cpg would guarantee i see the joins in order, not at the same time.

It may just work to have both race to create the checkpoint, the loser should get an error back from create (I haven't tried it, but I'd expect it to work that way.)

If only OpenAIS wasn't so loose here. If my daemon dies and
restarts, the checkpoints I previously created might not have gone away yet. So I get EEXIST for a short while until CKPT is done disposing of them. ocfs2_controld handles this, but it means we can't rely on EEXIST.

Joel

Signature · 10 lines
"Baby, even the losers
 Get luck sometimes.
 Even the losers
 Keep a little bit of pride."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#8 lore raw
In reply to Joel Becker
Quoted 10 lines · Joel Becker
Quoted 9 lines · David Teigland

If I start ocfs2_controld.cman in parallel on a few nodes, only one of them starts up, the others exit with one of these errors:

call_section_read at 370: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) call_section_read at 387: Checkpoint "ocfs2:controld" does not have a section named "daemon_protocol"

call_section_read at 370: Reading from section "daemon_protocol" on checkpoint "ocfs2:controld" (try 1) call_section_read at 397: Unable to read section "daemon_protocol" from checkpoint "ocfs2:controld": Object does not exist

Quoted 7 lines
Quoted 1 line

It does work ok if I remove those two checks.

These checks are required - otherwise you end up with unsync'd
daemons, which is crap.
I've changed the daemon to wait indefinitely, and that's
something lmb was testing. See the controld-fixes branch of ocfs2-tools.git. That should fix these problems.

These changes are now in the master branch of ocfs2-tools.git.

Joel

Signature · 8 lines
"To fall in love is to create a religion that has a fallible god."
        -Jorge Luis Borges

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#9 lore raw

PATCH 1/1 ocfs2_controld: Handle simultaneous group join.

In reply to Joel Becker

More than one ocfs2_controld can join the cpg group at the same time, even for the first members of a group. So 'jouners == 1' is not a valid check to see if this controld is the first one.

We change the code to pick the controld with the lowest node number to set up the daemon control group.

---
 ocfs2_controld/cpg.c |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)
ocfs2_controld/cpg.c +12 -1
index a2d8ae1..5e166b0 100644
@@ -728,6 +728,8 @@ out:
 
 static void daemon_set_cgroup(struct cgroup *cg, void *user_data)
 {
+	int i, first = 1;
+
 	void (*daemon_joined)(int first) = user_data;
 
 	if (cg != &daemon_group) {
@@ -735,7 +737,16 @@ static void daemon_set_cgroup(struct cgroup *cg, void *user_data)
 		return;
 	}
 
-	daemon_joined(daemon_group.cg_member_count == 1);
+	if (daemon_group.cg_cb_member_count != daemon_group.cg_cb_joined_count)
+		first = 0;
+	else {
+		for (i = 0; i < daemon_group.cg_cb_joined_count; i++) {
+			if (daemon_group.cg_cb_joined[i].nodeid < our_nodeid)
+				first = 0;
+		}
+	}
+
+	daemon_joined(first);
 }
 
 int setup_cpg(void (*daemon_joined)(int first))
Signature · 13 lines
1.6.1.3

-- 

Life's Little Instruction Book #139

	"Never deprive someone of hope; it might be all they have."

Joel Becker
Principal Software Developer
Oracle
E-mail: [email protected]
Phone: (650) 506-8127