Skip to contentSkip to conversation

futex: Fault/error injection capabilities

PATCH In review Davidlohr Bueso 5 participants 11 messages Jun 30, 2015 – Jul 20, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover futex: Fault/error injection capabilities

Hello,

I've been wanting this for a while to improve overall futex testing. I must say that I checked rather late when I was nearly finished with patch 2 if something similar for futexes had already been proposed. Sure enough, in 2009 this was discussed[1].

Coincidently, I also took the natural approach of making use of our fault-injection machinery. I have no idea if perf nowadays does such things, and if it does not, I honestly don't have the bandwidth to do it Ingo's preferred way -- when there is nothing wrong with this approach, imho (0 overhead). Anyway, here's a working patch.

Patch 1 is merely a trivial add-on.

[1]: https://lwn.net/Articles/364742/

Thanks!

Davidlohr Bueso (2):
futex: Enhance comments in futex_lock_pi() for blocking paths
futex: Fault/error injection capabilities

 Documentation/fault-injection/fault-injection.txt |  11 +++
 kernel/futex.c                                    | 100 +++++++++++++++++++++-
 lib/Kconfig.debug                                 |   7 ++
 3 files changed, 114 insertions(+), 4 deletions(-)
Signature · 1 line
2.1.4
#1 lore raw

PATCH 1/2 futex: Enhance comments in futex_lock_pi() for blocking paths

... serves a bit better to clarify between blocking and non-blocking code paths.

---
 kernel/futex.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/futex.c +9 -2
index c4a182f..153eb22 100644
@@ -2268,8 +2268,11 @@ static long futex_wait_restart(struct restart_block *restart)
 /*
  * Userspace tried a 0 -> TID atomic transition of the futex value
  * and failed. The kernel side here does the whole locking operation:
- * if there are waiters then it will block, it does PI, etc. (Due to
- * races the kernel might see a 0 value of the futex too.)
+ * if there are waiters then it will block as a consequence of relying
+ * on rt-mutexes, it does PI, etc. (Due to races the kernel might see
+ * a 0 value of the futex too.).
+ *
+ * Also serves as futex trylock_pi()'ing, and due semantics.
  */
 static int futex_lock_pi(u32 __user *uaddr, unsigned int flags,
 			 ktime_t *time, int trylock)
@@ -2300,6 +2303,10 @@ retry_private:
 
 	ret = futex_lock_pi_atomic(uaddr, hb, &q.key, &q.pi_state, current, 0);
 	if (unlikely(ret)) {
+		/*
+		 * Atomic work succeeded and we got the lock,
+		 * or failed. Either way, we do _not_ block.
+		 */
 		switch (ret) {
 		case 1:
 			/* We got the lock. */
Signature · 1 line
2.1.4
#2 lore raw

[tip:locking/core] futex: Enhance comments in futex_lock_pi() for blocking paths

futex: Enhance comments in futex_lock_pi() for blocking paths
Commit-ID:  767f509ca11269c2bcd92e3972a93096f2173ac0
Gitweb:     http://git.kernel.org/tip/767f509ca11269c2bcd92e3972a93096f2173ac0
Author:     Davidlohr Bueso <[email protected]>
AuthorDate: Mon, 29 Jun 2015 23:26:01 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon, 20 Jul 2015 11:45:45 +0200

futex: Enhance comments in futex_lock_pi() for blocking paths

... serves a bit better to clarify between blocking and non-blocking code paths.

---
 kernel/futex.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/futex.c +9 -2
index c4a182f..153eb22 100644
@@ -2268,8 +2268,11 @@ static long futex_wait_restart(struct restart_block *restart)
 /*
  * Userspace tried a 0 -> TID atomic transition of the futex value
  * and failed. The kernel side here does the whole locking operation:
- * if there are waiters then it will block, it does PI, etc. (Due to
- * races the kernel might see a 0 value of the futex too.)
+ * if there are waiters then it will block as a consequence of relying
+ * on rt-mutexes, it does PI, etc. (Due to races the kernel might see
+ * a 0 value of the futex too.).
+ *
+ * Also serves as futex trylock_pi()'ing, and due semantics.
  */
 static int futex_lock_pi(u32 __user *uaddr, unsigned int flags,
 			 ktime_t *time, int trylock)
@@ -2300,6 +2303,10 @@ retry_private:
 
 	ret = futex_lock_pi_atomic(uaddr, hb, &q.key, &q.pi_state, current, 0);
 	if (unlikely(ret)) {
+		/*
+		 * Atomic work succeeded and we got the lock,
+		 * or failed. Either way, we do _not_ block.
+		 */
 		switch (ret) {
 		case 1:
 			/* We got the lock. */
#3 lore raw

Re: [tip:locking/core] futex: Enhance comments in futex_lock_pi() for blocking paths

Quoted 27 lines · tip-bot for Davidlohr Bueso
Commit-ID:  767f509ca11269c2bcd92e3972a93096f2173ac0
Gitweb:     http://git.kernel.org/tip/767f509ca11269c2bcd92e3972a93096f2173ac0
Author:     Davidlohr Bueso <[email protected]>
AuthorDate: Mon, 29 Jun 2015 23:26:01 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon, 20 Jul 2015 11:45:45 +0200

futex: Enhance comments in futex_lock_pi() for blocking paths

... serves a bit better to clarify between blocking and non-blocking code paths.

---
 kernel/futex.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/futex.c +0 -0
index c4a182f..153eb22 100644

kernel/futex.c static long futex_wait_restart(struct restart_block *restart) · line 2275

Quoted 10 lines
kernel/futex.c +5 -2
@@ -2268,8 +2268,11 @@ static long futex_wait_restart(struct restart_block *restart)
 /*
  * Userspace tried a 0 -> TID atomic transition of the futex value
  * and failed. The kernel side here does the whole locking operation:
- * if there are waiters then it will block, it does PI, etc. (Due to
- * races the kernel might see a 0 value of the futex too.)
+ * if there are waiters then it will block as a consequence of relying
+ * on rt-mutexes, it does PI, etc. (Due to races the kernel might see
+ * a 0 value of the futex too.).
+ *
+ * Also serves as futex trylock_pi()'ing, and due semantics.

Hrm, what does 'due' mean here?

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#4 lore raw

Re: [tip:locking/core] futex: Enhance comments in futex_lock_pi() for blocking paths

Quoted 32 lines · Darren Hart
Quoted 31 lines · tip-bot for Davidlohr Bueso
Commit-ID:  767f509ca11269c2bcd92e3972a93096f2173ac0
Gitweb:     http://git.kernel.org/tip/767f509ca11269c2bcd92e3972a93096f2173ac0
Author:     Davidlohr Bueso <[email protected]>
AuthorDate: Mon, 29 Jun 2015 23:26:01 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon, 20 Jul 2015 11:45:45 +0200

futex: Enhance comments in futex_lock_pi() for blocking paths

... serves a bit better to clarify between blocking and non-blocking code paths.

---
 kernel/futex.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/futex.c +0 -1
index c4a182f..153eb22 100644
@@ -2268,8 +2268,11 @@ static long futex_wait_restart(struct restart_block *restart)
 /*
  * Userspace tried a 0 -> TID atomic transition of the futex value
  * and failed. The kernel side here does the whole locking operation:
- * if there are waiters then it will block, it does PI, etc. (Due to
Quoted 8 lines

kernel/futex.c static long futex_wait_restart(struct restart_block *restart) · line 2275

Quoted 6 lines
kernel/futex.c +5 -1
- * races the kernel might see a 0 value of the futex too.)
+ * if there are waiters then it will block as a consequence of relying
+ * on rt-mutexes, it does PI, etc. (Due to races the kernel might see
+ * a 0 value of the futex too.).
+ *
+ * Also serves as futex trylock_pi()'ing, and due semantics.

Hrm, what does 'due' mean here?

By that I meant respective/corresponding.

#5 lore raw

Re: [tip:locking/core] futex: Enhance comments in futex_lock_pi() for blocking paths

Quoted 42 lines · Davidlohr Bueso
Quoted 32 lines · Darren Hart
Quoted 31 lines · tip-bot for Davidlohr Bueso
Commit-ID:  767f509ca11269c2bcd92e3972a93096f2173ac0
Gitweb:     http://git.kernel.org/tip/767f509ca11269c2bcd92e3972a93096f2173ac0
Author:     Davidlohr Bueso <[email protected]>
AuthorDate: Mon, 29 Jun 2015 23:26:01 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon, 20 Jul 2015 11:45:45 +0200

futex: Enhance comments in futex_lock_pi() for blocking paths

... serves a bit better to clarify between blocking and non-blocking code paths.

---
 kernel/futex.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/futex.c +0 -1
index c4a182f..153eb22 100644
@@ -2268,8 +2268,11 @@ static long futex_wait_restart(struct restart_block *restart)
 /*
  * Userspace tried a 0 -> TID atomic transition of the futex value
  * and failed. The kernel side here does the whole locking operation:
- * if there are waiters then it will block, it does PI, etc. (Due to
Quoted 8 lines

kernel/futex.c static long futex_wait_restart(struct restart_block *restart) · line 2275

Quoted 6 lines
kernel/futex.c +5 -1
- * races the kernel might see a 0 value of the futex too.)
+ * if there are waiters then it will block as a consequence of relying
+ * on rt-mutexes, it does PI, etc. (Due to races the kernel might see
+ * a 0 value of the futex too.).
+ *
+ * Also serves as futex trylock_pi()'ing, and due semantics.

Hrm, what does 'due' mean here?

Quoted 1 line

By that I meant respective/corresponding.

Right, OK, read awkwardly to me for some reason.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#6 lore raw

PATCH 2/2 futex: Fault/error injection capabilities

In reply to Davidlohr Bueso

Although futexes are well known for being a royal pita, we really have very little debugging capabilities - except for relying on tglx's eye half the time.

By simply making use of the existing fault-injection machinery, we can improve this situation, allowing generating artificial uaddress faults and deadlock scenarios. Of course, when this is disabled in production systems, the overhead for failure checks is practically zero -- so this is very cheap at the same time. Future work would be nice to now enhance trinity to make use of this.

There is a special tunable 'ignore-private', which can filter out private futexes. Given the tsk->make_it_fail filter and this option, pi futexes can be narrowed down pretty closely.

---
 Documentation/fault-injection/fault-injection.txt | 11 +++
 kernel/futex.c                                    | 89 ++++++++++++++++++++++-
 lib/Kconfig.debug                                 |  7 ++
 3 files changed, 105 insertions(+), 2 deletions(-)
Documentation/fault-injection/fault-injection.txt +11 -0
index 4cf1a2a..415484f 100644
@@ -15,6 +15,10 @@ o fail_page_alloc
 
   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
 
+o fail_futex
+
+  injects futex deadlock and uaddr fault errors.
+
 o fail_make_request
 
   injects disk IO errors on devices permitted by setting
@@ -113,6 +117,12 @@ configuration of fault-injection capabilities.
 	specifies the minimum page allocation order to be injected
 	failures.
 
+- /sys/kernel/debug/fail_futex/ignore-private:
+
+	Format: { 'Y' | 'N' }
+	default is 'N', setting it to 'Y' will disable failure injections
+	when dealing with private (address space) futexes.
+
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
@@ -121,6 +131,7 @@ use the boot option:
 	failslab=
 	fail_page_alloc=
 	fail_make_request=
+	fail_futex=
 	mmc_core.fail_request=<interval>,<probability>,<space>,<times>
 
 How to add new fault injection capability
kernel/futex.c +87 -2

161 lines not on this page · raw message

lib/Kconfig.debug +7 -0
index b908048..e772be6 100644
@@ -1537,6 +1537,13 @@ config FAIL_MMC_REQUEST
 	  and to test how the mmc host driver handles retries from
 	  the block device.
 
+config FAIL_FUTEX
+	bool "Fault-injection capability for futexes"
+	select DEBUG_FS
+	depends on FAULT_INJECTION && FUTEX
+	help
+	  Provide fault-injection capability for futexes.
+
 config FAULT_INJECTION_DEBUG_FS
 	bool "Debugfs entries for fault-injection capabilities"
 	depends on FAULT_INJECTION && SYSFS && DEBUG_FS
Signature · 1 line
2.1.4
#7 lore raw
futex: Fault/error injection capabilities
Commit-ID:  ab51fbab39d864f3223e44a2600fd951df261f0b
Gitweb:     http://git.kernel.org/tip/ab51fbab39d864f3223e44a2600fd951df261f0b
Author:     Davidlohr Bueso <[email protected]>
AuthorDate: Mon, 29 Jun 2015 23:26:02 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon, 20 Jul 2015 11:45:45 +0200

futex: Fault/error injection capabilities

Although futexes are well known for being a royal pita, we really have very little debugging capabilities - except for relying on tglx's eye half the time.

By simply making use of the existing fault-injection machinery, we can improve this situation, allowing generating artificial uaddress faults and deadlock scenarios. Of course, when this is disabled in production systems, the overhead for failure checks is practically zero -- so this is very cheap at the same time. Future work would be nice to now enhance trinity to make use of this.

There is a special tunable 'ignore-private', which can filter out private futexes. Given the tsk->make_it_fail filter and this option, pi futexes can be narrowed down pretty closely.

---
 Documentation/fault-injection/fault-injection.txt | 11 +++
 kernel/futex.c                                    | 89 ++++++++++++++++++++++-
 lib/Kconfig.debug                                 |  7 ++
 3 files changed, 105 insertions(+), 2 deletions(-)
Documentation/fault-injection/fault-injection.txt +11 -0
index 4cf1a2a..415484f 100644
@@ -15,6 +15,10 @@ o fail_page_alloc
 
   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
 
+o fail_futex
+
+  injects futex deadlock and uaddr fault errors.
+
 o fail_make_request
 
   injects disk IO errors on devices permitted by setting
@@ -113,6 +117,12 @@ configuration of fault-injection capabilities.
 	specifies the minimum page allocation order to be injected
 	failures.
 
+- /sys/kernel/debug/fail_futex/ignore-private:
+
+	Format: { 'Y' | 'N' }
+	default is 'N', setting it to 'Y' will disable failure injections
+	when dealing with private (address space) futexes.
+
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
@@ -121,6 +131,7 @@ use the boot option:
 	failslab=
 	fail_page_alloc=
 	fail_make_request=
+	fail_futex=
 	mmc_core.fail_request=<interval>,<probability>,<space>,<times>
 
 How to add new fault injection capability
kernel/futex.c +87 -2

161 lines not on this page · raw message

lib/Kconfig.debug +7 -0
index e2894b2..22554d6 100644
@@ -1542,6 +1542,13 @@ config FAIL_MMC_REQUEST
 	  and to test how the mmc host driver handles retries from
 	  the block device.
 
+config FAIL_FUTEX
+	bool "Fault-injection capability for futexes"
+	select DEBUG_FS
+	depends on FAULT_INJECTION && FUTEX
+	help
+	  Provide fault-injection capability for futexes.
+
 config FAULT_INJECTION_DEBUG_FS
 	bool "Debugfs entries for fault-injection capabilities"
 	depends on FAULT_INJECTION && SYSFS && DEBUG_FS
#8 lore raw
Quoted 13 lines · tip-bot for Davidlohr Bueso
Commit-ID:  ab51fbab39d864f3223e44a2600fd951df261f0b
Gitweb:     http://git.kernel.org/tip/ab51fbab39d864f3223e44a2600fd951df261f0b
Author:     Davidlohr Bueso <[email protected]>
AuthorDate: Mon, 29 Jun 2015 23:26:02 -0700
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Mon, 20 Jul 2015 11:45:45 +0200

futex: Fault/error injection capabilities

Although futexes are well known for being a royal pita, we really have very little debugging capabilities - except for relying on tglx's eye half the time.

Quoted 7 lines

By simply making use of the existing fault-injection machinery, we can improve this situation, allowing generating artificial uaddress faults and deadlock scenarios. Of course, when this is disabled in production systems, the overhead for failure checks is practically zero -- so this is very cheap at the same time. Future work would be nice to now enhance trinity to make use of this.

If you enable any of the fault injection modules, ie, like you demonstrate here:

Documentation/fault-injection/fault-injection.txt configuration of fault-injection capabilities. · line 125

Quoted 6 lines

+- /sys/kernel/debug/fail_futex/ignore-private:

Documentation/fault-injection/fault-injection.txt +5 -0
+
+	Format: { 'Y' | 'N' }
+	default is 'N', setting it to 'Y' will disable failure injections
+	when dealing with private (address space) futexes.
+

each pid gets a a make-it-fail file in /proc/self/. If present, trinity will set this to 1 for child processes.

So it should work today unless I've missed something.

Dave

#9 lore raw
In reply to Davidlohr Bueso

Ingo, any thoughts about this?

Thanks.

Quoted 28 lines · Davidlohr Bueso

Hello,

I've been wanting this for a while to improve overall futex testing. I must say that I checked rather late when I was nearly finished with patch 2 if something similar for futexes had already been proposed. Sure enough, in 2009 this was discussed[1].

Coincidently, I also took the natural approach of making use of our fault-injection machinery. I have no idea if perf nowadays does such things, and if it does not, I honestly don't have the bandwidth to do it Ingo's preferred way -- when there is nothing wrong with this approach, imho (0 overhead). Anyway, here's a working patch.

Patch 1 is merely a trivial add-on.

[1]: https://lwn.net/Articles/364742/

Thanks!

Davidlohr Bueso (2):
futex: Enhance comments in futex_lock_pi() for blocking paths
futex: Fault/error injection capabilities

 Documentation/fault-injection/fault-injection.txt |  11 +++
 kernel/futex.c                                    | 100 +++++++++++++++++++++-
 lib/Kconfig.debug                                 |   7 ++
 3 files changed, 114 insertions(+), 4 deletions(-)
#10 lore raw
Quoted 1 line · Davidlohr Bueso

Ingo, any thoughts about this?

It's on my review list....