Skip to contentSkip to conversation

Fix pthread_atfork() behaviour

PATCH Mathieu Desnoyers 2 participants 7 messages Apr 17, 2014 – Apr 19, 2014 7 messages not in the archive

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 9 lines

From: "Keir Fraser" <[email protected]>
To: [email protected]
Sent: Monday, April 14, 2014 9:31:57 AM
Subject: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

In the process of integrating liburcu into a multi-threaded codebase with fork()s I found a couple of problems with liburcu that I could not work around without fixing the library. Hence I present the two required fixes here (as attachments, sorry!) with some background info about them.

This is interesting!

An initial question: Which URCU flavor are you using ?

Quoted 7 lines

After fork() the child process has no pthreads but the one that called fork(). Unfortunately call_rcu_after_fork_child() does not update URCU's thread registry to reflect this -- if fork() is called with any threads registered with URCU then the child process will inherit a corrupted registry containing a linked list through per-thread TLS state which is no longer valid allocated memory.

You appear to be using the mb/memb URCU flavor.

If we look at URCU README file:

Interaction with fork()

Special care must be taken for applications performing fork() without any following exec(). This is caused by the fact that Linux only clones the thread calling fork(), and thus never replicates any of the other parent thread into the child process. Most liburcu implementations require that all registrations (as reader, defer_rcu and call_rcu threads) should be released before a fork() is performed, except for the rather common scenario where fork() is immediately followed by exec() in the child process. The only implementation not subject to that rule is liburcu-bp, which is designed to handle fork() by calling rcu_bp_before_fork, rcu_bp_after_fork_parent and rcu_bp_after_fork_child.

Applications that use call_rcu() and that fork() without doing an immediate exec() must take special action. The parent must invoke call_rcu_before_fork() before the fork() and call_rcu_after_fork_parent() after the fork(). The child process must invoke call_rcu_after_fork_child(). Even though these three APIs are suitable for passing to pthread_atfork(), use of pthread_atfork() is *STRONGLY DISCOURAGED* for programs calling the glibc memory allocator (malloc(), calloc(), free(), ...) within call_rcu callbacks. This is due to limitations in the way glibc memory allocator handles calls to the memory allocator from concurrent threads while the pthread_atfork() handlers are executing. Combining e.g.:
* call to free() from callbacks executed within call_rcu worker threads,
* executing call_rcu atfork handlers within the glibc pthread atfork mechanism,
will sometimes trigger interesting process hangs. This usually hangs on a memory allocator lock within glibc.

Quoted 7 lines

Crash or hang soon after is the
result. Patch 1 therefore simply clears the registry list in the child process. Caveats here are that (a) the calling thread cannot be registered (it must unregister/re-register itself in the atfork handlers); and (b) some flavours of URCU may have more complex registries than a simple linked list and so this patch may not be sufficient for those. I only tested the memb/mb flavour myself.

Firstly, your use-case seems to be only supported by the urcu-bp flavor, and explicitly unsupported by other flavors. If you want to do this with other urcu flavors, you need to explicitly unregister all your other RCU threads before doing the fork().

Also, if we want to eventually support this in the other flavors, we would need to implement the flavor-specific teardown within each flavor implementation rather than in the call_rcu implementation.

Moreover, as stated in the README file, call_rcu() pre/post fork handlers don't behave well with glibc's pthread_atfork(), due to assumptions done within the glibc memory allocator.

Quoted 7 lines

A second problem is that although call_rcu threads are paused across fork(), the handshaking PAUSED flag is not cleared when their execution resumes. Hence a second fork() invocation in the original parent process will not spin-wait for call_rcu threads to quiesce (as the atfork handler will observe all PAUSED flags already set). Patch 2 fixes this with the appropriate clearing handshake on resume, post-fork.

I'd be very interested to see a test-case for this problem against the urcu-bp flavor. I think you have indeed caught a real bug here.

Quoted 4 lines

Please feel free to modify or rewrite these patches, or solve the described problems in a different way, as you see fit! Cc me on replies as I am not a subscriber to this list.

Could you provide a small test-case that we could add to the call_rcu regression (double fork) under tests/regression to make sure this behavior is covered in the future ? A small refactoring of test_urcu_fork.c to extend it should do the trick.

Thanks!

Mathieu

Quoted 9 lines

Regards,
Keir Fraser

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Quoted 1 line

From 3044b160c9e5614787db5891fb0ded8a4a1ed108 Mon Sep 17 00:00:00 2001

From: Keir Fraser <[email protected]>
Date: Mon, 7 Apr 2014 14:20:33 +0100
Subject: [PATCH 1/2] Clean the thread registry in the child of a fork.
No threads are registered there, and the registry is a linked list through non-existent per-thread state.

The caller of the pthread_atfork() handlers must not be an RCU-registered thread.

---
 urcu-call-rcu-impl.h | 3 +++
 1 file changed, 3 insertions(+)
urcu-call-rcu-impl.h +3 -0
index 10c4f3e..a050d41 100644
@@ -887,6 +887,9 @@ void call_rcu_after_fork_child(void)
 	/* Release the mutex. */
 	call_rcu_unlock(&call_rcu_mutex);
 
+	/* No RCU-registered threads at this point. Clean the registry. */
+	CDS_INIT_LIST_HEAD(&registry);
+
 	/* Do nothing when call_rcu() has not been used */
 	if (cds_list_empty(&call_rcu_data_list))
 		return;
Signature · 1 line
1.8.3.2
Quoted 1 line

From 7722952d181132856d71c929f5395e62a05f5c0c Mon Sep 17 00:00:00 2001

From: Keir <[email protected]>
Date: Mon, 7 Apr 2014 14:28:52 +0100
Subject: [PATCH 2/2] call_rcu threads should clear their PAUSED flag
when they unpause.

And call_rcu_after_fork_parent should spin-wait on this.

Otherwise a second fork in the parent will see the PAUSED flags already set and call_rcu_before_fork will not correctly wait for the call_rcu threads to quiesce on this second occasion.

---
 urcu-call-rcu-impl.h | 6 ++++++
 1 file changed, 6 insertions(+)
urcu-call-rcu-impl.h +6 -0
index a050d41..5c25a75 100644
@@ -308,6 +308,8 @@ static void *call_rcu_thread(void *arg)
 			uatomic_or(&crdp->flags, URCU_CALL_RCU_PAUSED);
 			while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_PAUSE) != 0)
 				poll(NULL, 0, 1);
+			uatomic_and(&crdp->flags, ~URCU_CALL_RCU_PAUSED);
+			cmm_smp_mb__after_uatomic_and();
 			rcu_register_thread();
 		}
 
@@ -872,6 +874,10 @@ void call_rcu_after_fork_parent(void)
 
 	cds_list_for_each_entry(crdp, &call_rcu_data_list, list)
 		uatomic_and(&crdp->flags, ~URCU_CALL_RCU_PAUSE);
+	cds_list_for_each_entry(crdp, &call_rcu_data_list, list) {
+		while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_PAUSED) != 0)
+			poll(NULL, 0, 1);
+	}
 	call_rcu_unlock(&call_rcu_mutex);
 }
 
Signature · 1 line
1.8.3.2

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#3 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 105 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "Keir Fraser" <[email protected]>
Cc: [email protected], "Paul E. McKenney" <[email protected]>
Sent: Thursday, April 17, 2014 8:20:00 AM
Subject: Re: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

----- Original Message -----

Quoted 9 lines

From: "Keir Fraser" <[email protected]>
To: [email protected]
Sent: Monday, April 14, 2014 9:31:57 AM
Subject: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

In the process of integrating liburcu into a multi-threaded codebase with fork()s I found a couple of problems with liburcu that I could not work around without fixing the library. Hence I present the two required fixes here (as attachments, sorry!) with some background info about them.

This is interesting!

An initial question: Which URCU flavor are you using ?

Quoted 7 lines

After fork() the child process has no pthreads but the one that called fork(). Unfortunately call_rcu_after_fork_child() does not update URCU's thread registry to reflect this -- if fork() is called with any threads registered with URCU then the child process will inherit a corrupted registry containing a linked list through per-thread TLS state which is no longer valid allocated memory.

You appear to be using the mb/memb URCU flavor.

If we look at URCU README file:

Interaction with fork()

Special care must be taken for applications performing fork() without any following exec(). This is caused by the fact that Linux only clones
the thread calling fork(), and thus never replicates any of the other parent thread into the child process. Most liburcu implementations require that all registrations (as reader, defer_rcu and call_rcu threads) should be released before a fork() is performed, except for the
rather common scenario where fork() is immediately followed by exec() in
the child process. The only implementation not subject to that rule is
liburcu-bp, which is designed to handle fork() by calling rcu_bp_before_fork, rcu_bp_after_fork_parent and rcu_bp_after_fork_child.

Applications that use call_rcu() and that fork() without doing an immediate exec() must take special action. The parent must invoke call_rcu_before_fork() before the fork() and call_rcu_after_fork_parent() after the fork(). The child process must invoke call_rcu_after_fork_child(). Even though these three APIs are suitable for passing to pthread_atfork(), use of pthread_atfork() is *STRONGLY DISCOURAGED* for programs calling the glibc memory allocator (malloc(), calloc(), free(), ...) within call_rcu callbacks. This is due to limitations in the way glibc memory allocator handles calls to the memory allocator from concurrent threads while the pthread_atfork() handlers are executing. Combining e.g.:
* call to free() from callbacks executed within call_rcu worker threads,
* executing call_rcu atfork handlers within the glibc pthread atfork mechanism,
will sometimes trigger interesting process hangs. This usually hangs on a memory allocator lock within glibc.

Quoted 7 lines

Crash or hang soon after is the
result. Patch 1 therefore simply clears the registry list in the child process. Caveats here are that (a) the calling thread cannot be registered (it must unregister/re-register itself in the atfork handlers); and (b) some flavours of URCU may have more complex registries than a simple linked list and so this patch may not be sufficient for those. I only tested the memb/mb flavour myself.

Firstly, your use-case seems to be only supported by the urcu-bp flavor, and explicitly unsupported by other flavors. If you want to do this with other urcu flavors, you need to explicitly unregister all your other RCU threads before doing the fork().

Also, if we want to eventually support this in the other flavors, we would need to implement the flavor-specific teardown within each flavor implementation rather than in the call_rcu implementation.

Moreover, as stated in the README file, call_rcu() pre/post fork handlers don't behave well with glibc's pthread_atfork(), due to assumptions done within the glibc memory allocator.

Quoted 7 lines

A second problem is that although call_rcu threads are paused across fork(), the handshaking PAUSED flag is not cleared when their execution resumes. Hence a second fork() invocation in the original parent process will not spin-wait for call_rcu threads to quiesce (as the atfork handler will observe all PAUSED flags already set). Patch 2 fixes this with the appropriate clearing handshake on resume, post-fork.

I'd be very interested to see a test-case for this problem against the urcu-bp flavor. I think you have indeed caught a real bug here.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Please feel free to modify or rewrite these patches, or solve the described problems in a different way, as you see fit! Cc me on replies as I am not a subscriber to this list.

Could you provide a small test-case that we could add to the call_rcu regression (double fork) under tests/regression to make sure this behavior is covered in the future ? A small refactoring of test_urcu_fork.c to extend it should do the trick.

I'm currently doing the test_urcu_fork.c refactoring, and testing your patch for call_rcu and double-fork. (letting you know so we don't duplicate the effort)

Thanks,

Mathieu

Quoted 25 lines

Thanks!

Mathieu

Quoted 9 lines

Regards,
Keir Fraser

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#5 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 121 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "Keir Fraser" <[email protected]>
Cc: [email protected], "Paul E. McKenney" <[email protected]>
Sent: Thursday, April 17, 2014 8:43:19 AM
Subject: Re: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

----- Original Message -----

Quoted 114 lines

From: "Mathieu Desnoyers" <[email protected]>
To: "Keir Fraser" <[email protected]>
Cc: [email protected], "Paul E. McKenney" <[email protected]>
Sent: Thursday, April 17, 2014 8:20:00 AM
Subject: Re: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

----- Original Message -----

Quoted 9 lines

From: "Keir Fraser" <[email protected]>
To: [email protected]
Sent: Monday, April 14, 2014 9:31:57 AM
Subject: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

In the process of integrating liburcu into a multi-threaded codebase with fork()s I found a couple of problems with liburcu that I could not work around without fixing the library. Hence I present the two required fixes here (as attachments, sorry!) with some background info about them.

This is interesting!

An initial question: Which URCU flavor are you using ?

Quoted 7 lines

After fork() the child process has no pthreads but the one that called fork(). Unfortunately call_rcu_after_fork_child() does not update URCU's thread registry to reflect this -- if fork() is called with any threads registered with URCU then the child process will inherit a corrupted registry containing a linked list through per-thread TLS state which is no longer valid allocated memory.

You appear to be using the mb/memb URCU flavor.

If we look at URCU README file:

Interaction with fork()

Special care must be taken for applications performing fork() without
any following exec(). This is caused by the fact that Linux only clones
the thread calling fork(), and thus never replicates any of the other
parent thread into the child process. Most liburcu implementations require that all registrations (as reader, defer_rcu and call_rcu threads) should be released before a fork() is performed, except for
the
rather common scenario where fork() is immediately followed by exec()
in
the child process. The only implementation not subject to that rule is
liburcu-bp, which is designed to handle fork() by calling rcu_bp_before_fork, rcu_bp_after_fork_parent and rcu_bp_after_fork_child.

Applications that use call_rcu() and that fork() without doing an immediate exec() must take special action. The parent must invoke call_rcu_before_fork() before the fork() and call_rcu_after_fork_parent() after the fork(). The child process must invoke call_rcu_after_fork_child(). Even though these three APIs are suitable for passing to pthread_atfork(), use of pthread_atfork() is *STRONGLY DISCOURAGED* for programs calling the glibc memory allocator (malloc(), calloc(), free(), ...) within call_rcu callbacks. This is due to limitations in the way glibc memory allocator handles calls to the memory allocator from concurrent threads while the pthread_atfork() handlers are executing. Combining e.g.:
* call to free() from callbacks executed within call_rcu worker threads,
* executing call_rcu atfork handlers within the glibc pthread atfork mechanism,
will sometimes trigger interesting process hangs. This usually hangs on a memory allocator lock within glibc.

Quoted 7 lines

Crash or hang soon after is the
result. Patch 1 therefore simply clears the registry list in the child process. Caveats here are that (a) the calling thread cannot be registered (it must unregister/re-register itself in the atfork handlers); and (b) some flavours of URCU may have more complex registries than a simple linked list and so this patch may not be sufficient for those. I only tested the memb/mb flavour myself.

Firstly, your use-case seems to be only supported by the urcu-bp flavor, and explicitly unsupported by other flavors. If you want to do this with other urcu flavors, you need to explicitly unregister all your other RCU threads before doing the fork().

Also, if we want to eventually support this in the other flavors, we would need to implement the flavor-specific teardown within each flavor implementation rather than in the call_rcu implementation.

Moreover, as stated in the README file, call_rcu() pre/post fork handlers don't behave well with glibc's pthread_atfork(), due to assumptions done within the glibc memory allocator.

Quoted 7 lines

A second problem is that although call_rcu threads are paused across fork(), the handshaking PAUSED flag is not cleared when their execution resumes. Hence a second fork() invocation in the original parent process will not spin-wait for call_rcu threads to quiesce (as the atfork handler will observe all PAUSED flags already set). Patch 2 fixes this with the appropriate clearing handshake on resume, post-fork.

I'd be very interested to see a test-case for this problem against the urcu-bp flavor. I think you have indeed caught a real bug here.

Quoted 4 lines

Please feel free to modify or rewrite these patches, or solve the described problems in a different way, as you see fit! Cc me on replies as I am not a subscriber to this list.

Quoted 8 lines
Quoted 4 lines

Could you provide a small test-case that we could add to the call_rcu regression (double fork) under tests/regression to make sure this behavior is covered in the future ? A small refactoring of test_urcu_fork.c to extend it should do the trick.

I'm currently doing the test_urcu_fork.c refactoring, and testing your patch for call_rcu and double-fork. (letting you know so we don't duplicate the effort)

Your fix for the call_rcu handling of double-fork is merged into master, stable-0.7, stable-0.8 branches.

I modified test_urcu_fork.c in master so it tests a sequence of 10 forks, 3 children deep (recursively). It seems to be solid with your patch. Without, it hangs very quickly.

I opened this bug tracker entry for this bug:

http://bugs.lttng.org/issues/786

Thanks!

Mathieu

Quoted 41 lines

Thanks,

Mathieu

Quoted 20 lines

Thanks!

Mathieu

Quoted 9 lines

Regards,
Keir Fraser

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#7 lore raw
In reply to a message not in the archive
Quoted 8 lines · Mathieu Desnoyers
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In the process of integrating liburcu into a multi-threaded codebase with fork()s I found a couple of problems with liburcu that I could not work around without fixing the library. Hence I present the two required fixes here (as attachments, sorry!) with some background info about them.

This is interesting!

An initial question: Which URCU flavor are you using ?

memb.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

After fork() the child process has no pthreads but the one that called fork(). Unfortunately call_rcu_after_fork_child() does not update URCU's thread registry to reflect this -- if fork() is called with any threads registered with URCU then the child process will inherit a corrupted registry containing a linked list through per-thread TLS state which is no longer valid allocated memory.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

You appear to be using the mb/memb URCU flavor.

If we look at URCU README file:
Firstly, your use-case seems to be only supported by the urcu-bp flavor, and explicitly unsupported by other flavors. If you want to do this with other urcu flavors, you need to explicitly unregister all your other RCU threads before doing the fork().

Well I should have read the manual it is true :) But even if I had, it is very hard to ensure that all threads de-register themselves in a large codebase. They would all need to be signalled to do so -- what a pita, almost impossible to do safely given the constraints of signal handler activities and usage of pthread_atfork I would say.

Basically I will have to switch to bp flavour or carry this patch. Actually de-registering and re-registering every thread is really untenable. So I strongly urge to consider giving the callers of this atfork() mechanism a break on this one. ;)

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also, if we want to eventually support this in the other flavors, we would need to implement the flavor-specific teardown within each flavor implementation rather than in the call_rcu implementation.

Yup.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Moreover, as stated in the README file, call_rcu() pre/post fork handlers don't behave well with glibc's pthread_atfork(), due to assumptions done within the glibc memory allocator.

Noted. :)

Quoted 27 lines
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines

A second problem is that although call_rcu threads are paused across fork(), the handshaking PAUSED flag is not cleared when their execution resumes. Hence a second fork() invocation in the original parent process will not spin-wait for call_rcu threads to quiesce (as the atfork handler will observe all PAUSED flags already set). Patch 2 fixes this with the appropriate clearing handshake on resume, post-fork.

I'd be very interested to see a test-case for this problem against the urcu-bp flavor. I think you have indeed caught a real bug here.

Quoted 3 lines

Please feel free to modify or rewrite these patches, or solve the described problems in a different way, as you see fit! Cc me on replies as I am not a subscriber to this list.

Quoted 7 lines
Quoted 4 lines

Could you provide a small test-case that we could add to the call_rcu regression (double fork) under tests/regression to make sure this behavior is covered in the future ? A small refactoring of test_urcu_fork.c to extend it should do the trick.

I'm currently doing the test_urcu_fork.c refactoring, and testing your patch for call_rcu and double-fork. (letting you know so we don't duplicate the effort)

Your fix for the call_rcu handling of double-fork is merged into master, stable-0.7, stable-0.8 branches.

I modified test_urcu_fork.c in master so it tests a sequence of 10 forks, 3 children deep (recursively). It seems to be solid with your patch. Without, it hangs very quickly.

Quoted 5 lines

I opened this bug tracker entry for this bug:

http://bugs.lttng.org/issues/786

Thanks!

Thanks for this one!

By the way, I also have a valgrind failure on rcu_barrier() due to uninitialised variable/field condition.futex. It gets decremented and passed to futex() syscall, but never actually initialised to zero in the first place. I believe. I haven't yet rolled a patch for that one but I'm sure you don't need one from me anyhow :) However I will send a separate email/patch to the list for this issue very soon unless you tell me that is not necessary.

Regards,
Keir

Quoted 39 lines

Mathieu

Quoted 36 lines

Thanks,

Mathieu

Quoted 17 lines

Thanks!

Mathieu

Quoted 8 lines

Regards,
Keir Fraser

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#9 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 8 lines

From: "Keir Fraser" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: [email protected], "Paul E. McKenney" <[email protected]>
Sent: Thursday, April 17, 2014 12:15:27 PM
Subject: Re: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

Mathieu Desnoyers wrote:

[...]

Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

After fork() the child process has no pthreads but the one that called fork(). Unfortunately call_rcu_after_fork_child() does not update URCU's thread registry to reflect this -- if fork() is called with any threads registered with URCU then the child process will inherit a corrupted registry containing a linked list through per-thread TLS state which is no longer valid allocated memory.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

You appear to be using the mb/memb URCU flavor.

If we look at URCU README file:
Firstly, your use-case seems to be only supported by the urcu-bp flavor, and explicitly unsupported by other flavors. If you want to do this with other urcu flavors, you need to explicitly unregister all your other RCU threads before doing the fork().

Well I should have read the manual it is true :) But even if I had, it is very hard to ensure that all threads de-register themselves in a large codebase. They would all need to be signalled to do so -- what a pita, almost impossible to do safely given the constraints of signal handler activities and usage of pthread_atfork I would say.

Quoted 4 lines

Basically I will have to switch to bp flavour or carry this patch. Actually de-registering and re-registering every thread is really untenable. So I strongly urge to consider giving the callers of this atfork() mechanism a break on this one. ;)

I'm perfectly fine with adding a nicer way to handle pthread atfork in RCU flavors besides urcu-bp. I just wanted to share with you the current state of what is supported, so we can agree that this would come as a new feature into master, and eventually be released in a urcu 0.9. Since it's a new feature, there won't be any associated bug entry, nor backports to 0.7 and 0.8 stable branches.

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Also, if we want to eventually support this in the other flavors, we would need to implement the flavor-specific teardown within each flavor implementation rather than in the call_rcu implementation.

Yup.

This would probably be similar to the rcu_bp_*fork*() family of functions. Basically, the pre/post fork functions would now become implemented for each urcu flavor. We'd have to think about how we want to handle relationship between the RCU pre/post fork handlers, and the call_rcu-specific pre/post fork handler, whether the order in which they are executed matters, and whether it still makes sense to expose a call_rcu pre/post fork handler when this handler could simply be called from within the RCU flavor pre/post fork handler. This would simplify handling of handler call order, but we have to think about API migration very carefully so we don't break existing users needlessly.

Quoted 44 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Moreover, as stated in the README file, call_rcu() pre/post fork handlers don't behave well with glibc's pthread_atfork(), due to assumptions done within the glibc memory allocator.

Noted. :)

Quoted 29 lines
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines

A second problem is that although call_rcu threads are paused across fork(), the handshaking PAUSED flag is not cleared when their execution resumes. Hence a second fork() invocation in the original parent process will not spin-wait for call_rcu threads to quiesce (as the atfork handler will observe all PAUSED flags already set). Patch 2 fixes this with the appropriate clearing handshake on resume, post-fork.

I'd be very interested to see a test-case for this problem against the urcu-bp flavor. I think you have indeed caught a real bug here.

Quoted 3 lines

Please feel free to modify or rewrite these patches, or solve the described problems in a different way, as you see fit! Cc me on replies as I am not a subscriber to this list.

Quoted 8 lines
Quoted 5 lines

Could you provide a small test-case that we could add to the call_rcu regression (double fork) under tests/regression to make sure this behavior
is covered in the future ? A small refactoring of test_urcu_fork.c to extend it should do the trick.

I'm currently doing the test_urcu_fork.c refactoring, and testing your patch for call_rcu and double-fork. (letting you know so we don't duplicate the effort)

Your fix for the call_rcu handling of double-fork is merged into master, stable-0.7, stable-0.8 branches.

I modified test_urcu_fork.c in master so it tests a sequence of 10 forks, 3 children deep (recursively). It seems to be solid with your patch. Without,
it hangs very quickly.

Quoted 5 lines

I opened this bug tracker entry for this bug:

http://bugs.lttng.org/issues/786

Thanks!

Thanks for this one!

Quoted 7 lines

By the way, I also have a valgrind failure on rcu_barrier() due to uninitialised variable/field condition.futex. It gets decremented and passed to futex() syscall, but never actually initialised to zero in the first place. I believe. I haven't yet rolled a patch for that one but I'm sure you don't need one from me anyhow :) However I will send a separate email/patch to the list for this issue very soon unless you tell me that is not necessary.

Urgh, yes. That's a very nice catch!! I'm preparing a fix right away. It will be tracked by: http://bugs.lttng.org/issues/787

Thanks!

Mathieu

Quoted 44 lines

Regards,
Keir

Quoted 39 lines

Mathieu

Quoted 36 lines

Thanks,

Mathieu

Quoted 17 lines

Thanks!

Mathieu

Quoted 8 lines

Regards,
Keir Fraser

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#11 lore raw
In reply to a message not in the archive
Quoted 14 lines · Mathieu Desnoyers

----- Original Message -----

Quoted 6 lines

From: "Keir Fraser"<[email protected]>
To: "Mathieu Desnoyers"<[email protected]>
Cc: [email protected], "Paul E. McKenney"<[email protected]>
Sent: Thursday, April 17, 2014 12:15:27 PM
Subject: Re: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

Quoted 6 lines

Mathieu Desnoyers wrote:

Basically I will have to switch to bp flavour or carry this patch. Actually de-registering and re-registering every thread is really untenable. So I strongly urge to consider giving the callers of this atfork() mechanism a break on this one. ;)

Quoted 6 lines

I'm perfectly fine with adding a nicer way to handle pthread atfork in RCU flavors besides urcu-bp. I just wanted to share with you the current state of what is supported, so we can agree that this would come as a new feature into master, and eventually be released in a urcu 0.9. Since it's a new feature, there won't be any associated bug entry, nor backports to 0.7 and 0.8 stable branches.

Okay, that makes sense, thanks.

Quoted 10 lines
Quoted 7 lines

By the way, I also have a valgrind failure on rcu_barrier() due to uninitialised variable/field condition.futex. It gets decremented and passed to futex() syscall, but never actually initialised to zero in the first place. I believe. I haven't yet rolled a patch for that one but I'm sure you don't need one from me anyhow :) However I will send a separate email/patch to the list for this issue very soon unless you tell me that is not necessary.

Urgh, yes. That's a very nice catch!! I'm preparing a fix right away. It will be tracked by: http://bugs.lttng.org/issues/787

Worse, while reading the rcu_barrier() code I found a more subtle bug. rcu_barrier() can return as soon as completion.barrier_count==0, which frees the completion struct along with the rest of its stack frame. But its call_rcu callbacks may yet try to read and write completion.futex via the wake_up function.

The completion struct should be calloc()ed, reference-counted, and freed by the thread that decs the count to 0, imo. I will be cooking up a patch to do just that.

There are other places in liburcu which protect a futex variable from premature freeing by the waiter polling a teardown flag, but I think refcounting is nicer :)

Regards,
Keir

Quoted 46 lines

Thanks!

Mathieu

Quoted 41 lines

Regards,
Keir

Quoted 38 lines

Mathieu

Quoted 36 lines

Thanks,

Mathieu

Quoted 17 lines

Thanks!

Mathieu

Quoted 8 lines

Regards,
Keir Fraser

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Quoted 5 lines

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

_______________________________________________
lttng-dev mailing list
[email protected]
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#13 lore raw
In reply to a message not in the archive

----- Original Message -----

Quoted 6 lines

From: "Keir Fraser" <[email protected]>
To: "Mathieu Desnoyers" <[email protected]>
Cc: [email protected], "Paul E. McKenney" <[email protected]>
Sent: Friday, April 18, 2014 1:42:50 PM
Subject: Re: [lttng-dev] [PATCH liburcu] Fix pthread_atfork() behaviour

[...]

Quoted 6 lines

Worse, while reading the rcu_barrier() code I found a more subtle bug. rcu_barrier() can return as soon as completion.barrier_count==0, which frees the completion struct along with the rest of its stack frame. But its call_rcu callbacks may yet try to read and write completion.futex via the wake_up function.

Good catch! I think I've fixed this kind of problematic pattern elsewhere already, but clearly rcu_barrier() slipped through.

Quoted 4 lines

The completion struct should be calloc()ed, reference-counted, and freed by the thread that decs the count to 0, imo. I will be cooking up a patch to do just that.

Yes, this should be a clean way to fix it.

Quoted 4 lines

There are other places in liburcu which protect a futex variable from premature freeing by the waiter polling a teardown flag, but I think refcounting is nicer :)

Makes sense to me,

Thanks!

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com