Skip to contentSkip to conversation

selftests: add a new test driver for sysfs

PATCH In review Luis Chamberlain 2 participants 13 messages Jul 3, 2021 – Jul 23, 2021

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

v2 cover selftests: add a new test driver for sysfs

This v2 rebases onto the latest linux-next tag, next-20210701. A few changes were needed, namely:

1) changes kernfs_init_failure_injection() to return int instead of void. On the latest linux-next we have a new static build check for this, so this mistake was captured when building.

2) I made kernfs_init_failure_injection static

3) lib/test_sysfs.c moved to the new blk_alloc_disk() added by Christoph as direct queue allocation is no longer supported, ie, blk_alloc_queue() is no longer exported. This work was done by Christoph in preparation to help make add_disk*() callers eventually return an error code and make the error handling much saner. Because of this same change blk_cleanup_queue() is no longer needed so we embrace the shiny new blk_cleanup_disk().

I've put this up on my linux-next git tree [0] under the branch named 20210701-sysfs-fix-races-v2.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git/log/?h=20210701-sysfs-fix-races-v2

Luis Chamberlain (4):
selftests: add tests_sysfs module
kernfs: add initial failure injection support
test_sysfs: add support to use kernfs failure injection
test_sysfs: demonstrate deadlock fix

 .../fault-injection/fault-injection.rst       |   22 +
 MAINTAINERS                                   |    9 +-
 fs/kernfs/Makefile                            |    1 +
 fs/kernfs/failure-injection.c                 |   83 +
 fs/kernfs/file.c                              |   13 +
 fs/kernfs/kernfs-internal.h                   |   72 +
 include/linux/kernfs.h                        |    5 +
 lib/Kconfig.debug                             |   23 +
 lib/Makefile                                  |    1 +
 lib/test_sysfs.c                              | 1027 ++++++++++++
 tools/testing/selftests/sysfs/Makefile        |   12 +
 tools/testing/selftests/sysfs/config          |    5 +
 tools/testing/selftests/sysfs/sysfs.sh        | 1376 +++++++++++++++++
 13 files changed, 2648 insertions(+), 1 deletion(-)
 create mode 100644 fs/kernfs/failure-injection.c
 create mode 100644 lib/test_sysfs.c
 create mode 100644 tools/testing/selftests/sysfs/Makefile
 create mode 100644 tools/testing/selftests/sysfs/config
 create mode 100755 tools/testing/selftests/sysfs/sysfs.sh
Signature · 1 line
2.27.0
#1 lore raw

PATCH v2 1/4 selftests: add tests_sysfs module

This adds a new selftest module which can be used to test sysfs, which would otherwise require using an existing driver. This lets us muck with a template driver to test breaking things without affecting system behaviour or requiring the dependencies of a real device driver.

A series of 28 tests are added. Support for using two device types are supported:

* misc
* block

Contrary to sysctls, sysfs requires a full write to happen at once, and so we reduce the digit tests to single writes. Two main sysfs knobs are provided for testing reading/storing, one which doesn't inclur any delays and another which can incur programmed delays. What locks are held, if any, are configurable, at module load time, or through dynamic configuration at run time.

Since sysfs is a technically filesystem, but a pseudo one, which requires a kernel user, our test_sysfs module and respective test script embraces fstests format for tests in the kernel ring bufffer. Likewise, a scraper for kernel crashes is provided which matches what fstests does as well.

Two tests are kept disabled as they currently cause a deadlock, and so this provides a mechanism to easily show proof and demo how the deadlock can happen:

Demos the deadlock with a device specific lock
./tools/testing/selftests/sysfs/sysfs.sh -t 0027

Demos the deadlock with rtnl_lock()
./tools/testing/selftests/sysfs/sysfs.sh -t 0028

Two separate solutions to the deadlock issue have been proposed, and so now its a matter of either documenting this limitation or eventually adopting a generic fix.

This selftests will shortly be expanded upon with more tests which require further kernel changes in order to provide better test coverage.

---
 MAINTAINERS                            |    7 +
 lib/Kconfig.debug                      |   10 +
 lib/Makefile                           |    1 +
 lib/test_sysfs.c                       |  943 +++++++++++++++++++
 tools/testing/selftests/sysfs/Makefile |   12 +
 tools/testing/selftests/sysfs/config   |    2 +
 tools/testing/selftests/sysfs/sysfs.sh | 1202 ++++++++++++++++++++++++
 7 files changed, 2177 insertions(+)
 create mode 100644 lib/test_sysfs.c
 create mode 100644 tools/testing/selftests/sysfs/Makefile
 create mode 100644 tools/testing/selftests/sysfs/config
 create mode 100755 tools/testing/selftests/sysfs/sysfs.sh
MAINTAINERS +7 -0
index 66d047dc6880..fd369ed50040 100644
@@ -17958,6 +17958,13 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/mmc/host/sdhci-pci-dwc-mshc.c
 
+SYSFS TEST DRIVER
+M:	Luis Chamberlain <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	lib/test_sysfs.c
+F:	tools/testing/selftests/sysfs/
+
 SYSTEM CONFIGURATION (SYSCON)
 M:	Lee Jones <[email protected]>
 M:	Arnd Bergmann <[email protected]>
lib/Kconfig.debug +10 -0
index fb370c7c4756..568838ac1189 100644
@@ -2360,6 +2360,16 @@ config TEST_SYSCTL
 
 	  If unsure, say N.
 
+config TEST_SYSFS
+	tristate "sysfs test driver"
+	depends on SYSFS
+	help
+	  This builds the "test_sysfs" module. This driver enables to test the
+	  sysfs file system safely without affecting production knobs which
+	  might alter system functionality.
+
+	  If unsure, say N.
+
 config BITFIELD_KUNIT
 	tristate "KUnit test bitfield functions at runtime"
 	depends on KUNIT
lib/Makefile +1 -0
index 5efd1b435a37..effd1ef806f0 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_TEST_FIRMWARE) += test_firmware.o
 obj-$(CONFIG_TEST_BITOPS) += test_bitops.o
 CFLAGS_test_bitops.o += -Werror
 obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o
+obj-$(CONFIG_TEST_SYSFS) += test_sysfs.o
 obj-$(CONFIG_TEST_HASH) += test_hash.o test_siphash.o
 obj-$(CONFIG_TEST_IDA) += test_ida.o
 obj-$(CONFIG_KASAN_KUNIT_TEST) += test_kasan.o
lib/test_sysfs.c new file +943 -0

943 lines not on this page · raw message

tools/testing/selftests/sysfs/Makefile new file +12 -0
new file mode 100644
index 000000000000..fde99caa2338
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+# Makefile for sysfs selftests.
+
+# No binaries, but make sure arg-less "make" doesn't trigger "run_tests".
+all:
+
+TEST_PROGS := sysfs.sh
+
+include ../lib.mk
+
+# Nothing to clean up.
+clean:
tools/testing/selftests/sysfs/config new file +2 -0
new file mode 100644
index 000000000000..9196f452ecd5
@@ -0,0 +1,2 @@
+CONFIG_SYSFS=m
+CONFIG_TEST_SYSFS=m
tools/testing/selftests/sysfs/sysfs.sh new file +1202 -0

1,202 lines not on this page · raw message

Signature · 1 line
2.27.0
maintainer
#2 lore raw

Re: [PATCH v2 1/4] selftests: add tests_sysfs module

Quoted 35 lines · Luis Chamberlain

This adds a new selftest module which can be used to test sysfs, which would otherwise require using an existing driver. This lets us muck with a template driver to test breaking things without affecting system behaviour or requiring the dependencies of a real device driver.

A series of 28 tests are added. Support for using two device types are supported:

* misc
* block

Contrary to sysctls, sysfs requires a full write to happen at once, and so we reduce the digit tests to single writes. Two main sysfs knobs are provided for testing reading/storing, one which doesn't inclur any delays and another which can incur programmed delays. What locks are held, if any, are configurable, at module load time, or through dynamic configuration at run time.

Since sysfs is a technically filesystem, but a pseudo one, which requires a kernel user, our test_sysfs module and respective test script embraces fstests format for tests in the kernel ring bufffer. Likewise, a scraper for kernel crashes is provided which matches what fstests does as well.

Two tests are kept disabled as they currently cause a deadlock, and so this provides a mechanism to easily show proof and demo how the deadlock can happen:

Demos the deadlock with a device specific lock
./tools/testing/selftests/sysfs/sysfs.sh -t 0027

Demos the deadlock with rtnl_lock()
./tools/testing/selftests/sysfs/sysfs.sh -t 0028

Quoted 7 lines

Two separate solutions to the deadlock issue have been proposed, and so now its a matter of either documenting this limitation or eventually adopting a generic fix.

This selftests will shortly be expanded upon with more tests which require further kernel changes in order to provide better test coverage.

Why is this not using kunit? We should not be adding new in-kernel tests that are not using that api anymore.

Quoted 73 lines
---
 MAINTAINERS                            |    7 +
 lib/Kconfig.debug                      |   10 +
 lib/Makefile                           |    1 +
 lib/test_sysfs.c                       |  943 +++++++++++++++++++
 tools/testing/selftests/sysfs/Makefile |   12 +
 tools/testing/selftests/sysfs/config   |    2 +
 tools/testing/selftests/sysfs/sysfs.sh | 1202 ++++++++++++++++++++++++
 7 files changed, 2177 insertions(+)
 create mode 100644 lib/test_sysfs.c
 create mode 100644 tools/testing/selftests/sysfs/Makefile
 create mode 100644 tools/testing/selftests/sysfs/config
 create mode 100755 tools/testing/selftests/sysfs/sysfs.sh
MAINTAINERS +7 -0
index 66d047dc6880..fd369ed50040 100644
@@ -17958,6 +17958,13 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/mmc/host/sdhci-pci-dwc-mshc.c
 
+SYSFS TEST DRIVER
+M:	Luis Chamberlain <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	lib/test_sysfs.c
+F:	tools/testing/selftests/sysfs/
+
 SYSTEM CONFIGURATION (SYSCON)
 M:	Lee Jones <[email protected]>
 M:	Arnd Bergmann <[email protected]>
lib/Kconfig.debug +10 -0
index fb370c7c4756..568838ac1189 100644
@@ -2360,6 +2360,16 @@ config TEST_SYSCTL
 
 	  If unsure, say N.
 
+config TEST_SYSFS
+	tristate "sysfs test driver"
+	depends on SYSFS
+	help
+	  This builds the "test_sysfs" module. This driver enables to test the
+	  sysfs file system safely without affecting production knobs which
+	  might alter system functionality.
+
+	  If unsure, say N.
+
 config BITFIELD_KUNIT
 	tristate "KUnit test bitfield functions at runtime"
 	depends on KUNIT
lib/Makefile +1 -0
index 5efd1b435a37..effd1ef806f0 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_TEST_FIRMWARE) += test_firmware.o
 obj-$(CONFIG_TEST_BITOPS) += test_bitops.o
 CFLAGS_test_bitops.o += -Werror
 obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o
+obj-$(CONFIG_TEST_SYSFS) += test_sysfs.o
 obj-$(CONFIG_TEST_HASH) += test_hash.o test_siphash.o
 obj-$(CONFIG_TEST_IDA) += test_ida.o
 obj-$(CONFIG_KASAN_KUNIT_TEST) += test_kasan.o
lib/test_sysfs.c new file +0 -0
new file mode 100644
index 000000000000..bf43016d40b5

lib/test_sysfs.c line 1

Quoted 1 line
lib/test_sysfs.c new file +1 -0
@@ -0,0 +1,943 @@
+// SPDX-License-Identifier: GPL-2.0-or-later

This does not match your "boiler-plate" text below, sorry.

thanks,

greg k-h

#3 lore raw

Re: [PATCH v2 1/4] selftests: add tests_sysfs module

Quoted 7 lines · Greg KH
Quoted 3 lines · Luis Chamberlain

This selftests will shortly be expanded upon with more tests which require further kernel changes in order to provide better test coverage.

Why is this not using kunit? We should not be adding new in-kernel tests that are not using that api anymore.

No way. That cannot possibly be true. When was this decided? Did Shuah Khan, the maintainer of selftests, all of a sudden decide we are going to deprecate selftests in favor for trying to only use kunit? Did we have a conference where this was talked about and decided?

If so all these are huge news to me and I missed the memo!

If I would have been at such meeting I would have definitely yelled bloody murder!

kunit relies on UML and UML is a simple one core architecture, to start with. This means I cannot run tests for multicore with it, which is where many races do happen! Yes, you can run kunit on other architectures, but all that is new.

Second, I did help review kunit getting upstream, and suggested a few example tests, part of which were for sysctl to compare and contrast what is possible and what we cannot do.

Not everything we want to test should be written as a kunit test. No way.

In this case kunit is not ideal given I want to mimic something in userspace interaction, and expose races through error injection and if we can use as many cores to busy races out.

Trust me, I'm an advocate of kunit, and I'm even trying to see ideally what tests from fstests / blktests could be kunit'ified. But in this case, no. Using a selftests is much better target framework.

Quoted 9 lines

lib/test_sysfs.c line 1

Quoted 7 lines
lib/test_sysfs.c new file +1 -0
new file mode 100644
index 000000000000..bf43016d40b5
@@ -0,0 +1,943 @@
+// SPDX-License-Identifier: GPL-2.0-or-later

This does not match your "boiler-plate" text below, sorry.

Indeed, I'll fix it.

Luis

maintainer
#4 lore raw

Re: [PATCH v2 1/4] selftests: add tests_sysfs module

Quoted 2 lines · Luis Chamberlain

kunit relies on UML and UML is a simple one core architecture, to start with.

I thought the UML requirement was long gone, are you sure it is still present?

Quoted 3 lines

This means I cannot run tests for multicore with it, which is where many races do happen! Yes, you can run kunit on other architectures, but all that is new.

What do you mean by "new"? It should work today, in today's kernel tree, right?

Quoted 3 lines

In this case kunit is not ideal given I want to mimic something in userspace interaction, and expose races through error injection and if we can use as many cores to busy races out.

Can you not do that with kunit? If not, why not?

thanks,

greg k-h

#5 lore raw

Re: [PATCH v2 1/4] selftests: add tests_sysfs module

Quoted 6 lines · Greg KH
Quoted 2 lines · Luis Chamberlain

kunit relies on UML and UML is a simple one core architecture, to start with.

I thought the UML requirement was long gone, are you sure it is still present?

It's *the* way to run kunit.

Quoted 6 lines
Quoted 3 lines

This means I cannot run tests for multicore with it, which is where many races do happen! Yes, you can run kunit on other architectures, but all that is new.

What do you mean by "new"? It should work today, in today's kernel tree, right?

That was experimental. And I know no one using it.

Quoted 5 lines
Quoted 3 lines

In this case kunit is not ideal given I want to mimic something in userspace interaction, and expose races through error injection and if we can use as many cores to busy races out.

Can you not do that with kunit? If not, why not?

kunit requires codifying everything in C, what we need is a lot of flexibility to do all sorts of races in userspace, and actually use userspace tools.

I am *not* going to even try to rewrite my selftest as a kunit test. It is just not going to happen.

Luis

#6 lore raw

PATCH v2 2/4 kernfs: add initial failure injection support

In reply to Luis Chamberlain

This adds initial failure injection support to kernfs. We start off with debug knobs which when enabled allow test drivers, such as test_sysfs, to then make use of these to try to force certain difficult races to take place with a high degree of certainty.

This only adds runtime code *iff* the new bool CONFIG_FAIL_KERNFS_KNOBS is enabled in your kernel. If you don't have this enabled this provides no new functional. When CONFIG_FAIL_KERNFS_KNOBS is disabled the new routine kernfs_debug_should_wait() ends up being transformed to if (false), and so the compiler should optimize these out as dead code producing no new effective binary changes.

We start off with enabling failure injections in kernfs by allowing us to alter the way kernfs_fop_write_iter() behaves. We allow for the routine kernfs_fop_write_iter() to wait for a certain condition in the kernel to occur, after which it will sleep a predefined amount of time. This lets kernfs users to time exactly when it want kernfs_fop_write_iter() to complete, allowing for developing race conditions and test for correctness in kernfs.

You'd boot with this enabled on your kernel command line:

fail_kernfs_fop_write_iter=1,100,0,1

The values are <interval,probability,size,times>, we don't care for size, so for now we ignore it. The above ensures a failure will trigger only once.

*How* we allow for this routine to change behaviour is left to knobs we expose under debugfs:

 # ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/
wait_after_active
wait_after_mutex
wait_at_start
wait_before_mutex

A debugfs entry also exists to allow us to sleep a configurabler amount of time after the completion:

/sys/kernel/debug/kernfs/sleep_after_wait_ms

These two sets of knobs allow us to construct races and demonstrate how the kernfs active reference should suffice to project against races.

Enabling CONFIG_FAULT_INJECTION_DEBUG_FS enables us to configure the differnt fault injection parametres for the new fail_kernfs_fop_write_iter fault injection at run time:

ls -1 /sys/kernel/debug/kernfs/fail_kernfs_fop_write_iter/ interval
probability
space
task-filter
times
verbose
verbose_ratelimit_burst verbose_ratelimit_interval_ms

kernfs: fix for latest linux-next

---
 .../fault-injection/fault-injection.rst       | 22 +++++
 MAINTAINERS                                   |  2 +-
 fs/kernfs/Makefile                            |  1 +
 fs/kernfs/failure-injection.c                 | 83 +++++++++++++++++++
 fs/kernfs/file.c                              | 13 +++
 fs/kernfs/kernfs-internal.h                   | 72 ++++++++++++++++
 include/linux/kernfs.h                        |  5 ++
 lib/Kconfig.debug                             | 10 +++
 8 files changed, 207 insertions(+), 1 deletion(-)
 create mode 100644 fs/kernfs/failure-injection.c
Documentation/fault-injection/fault-injection.rst +22 -0
index f47d05ed0d94..d27620196ee8 100644
@@ -24,6 +24,28 @@ Available fault injection capabilities
 
   injects futex deadlock and uaddr fault errors.
 
+- fail_kernfs_fop_write_iter
+
+  Allows for failures to be enabled inside kernfs_fop_write_iter(). Enabling
+  this does not immediately enable any errors to occur. You must configure
+  how you want this routine to fail or change behaviour by using the debugfs
+  knobs for it:
+
+  # ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/
+  wait_after_active
+  wait_after_mutex
+  wait_at_start
+  wait_before_mutex
+
+  You can also configure how long to sleep after a wait under
+
+  /sys/kernel/debug/kernfs/sleep_after_wait_ms
+
+  If you enable CONFIG_FAULT_INJECTION_DEBUG_FS the fail_add_disk failure
+  injection parameters are placed under:
+
+  /sys/kernel/debug/kernfs/fail_kernfs_fop_write_iter/
+
 - fail_make_request
 
   injects disk IO errors on devices permitted by setting
MAINTAINERS +1 -1
index fd369ed50040..419405b2378a 100644
@@ -10246,7 +10246,7 @@ M:	Greg Kroah-Hartman <[email protected]>
 M:	Tejun Heo <[email protected]>
 S:	Supported
 T:	git git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core.git
-F:	fs/kernfs/
+F:	fs/kernfs/*
 F:	include/linux/kernfs.h
 
 KEXEC
fs/kernfs/Makefile +1 -0
index 4ca54ff54c98..bc5b32ca39f9 100644
@@ -4,3 +4,4 @@
 #
 
 obj-y		:= mount.o inode.o dir.o file.o symlink.o
+obj-$(CONFIG_FAIL_KERNFS_KNOBS)    += failure-injection.o
fs/kernfs/failure-injection.c new file +83 -0

83 lines not on this page · raw message

fs/kernfs/file.c +13 -0
index c75719312147..d45954600f11 100644
@@ -259,6 +259,9 @@ static ssize_t kernfs_fop_write_iter(struct kiocb *iocb, struct iov_iter *iter)
 	const struct kernfs_ops *ops;
 	char *buf;
 
+	if (kernfs_debug_should_wait(kernfs_fop_write_iter, at_start))
+		kernfs_debug_wait();
+
 	if (of->atomic_write_len) {
 		if (len > of->atomic_write_len)
 			return -E2BIG;
@@ -280,17 +283,27 @@ static ssize_t kernfs_fop_write_iter(struct kiocb *iocb, struct iov_iter *iter)
 	}
 	buf[len] = '\0';	/* guarantee string termination */
 
+	if (kernfs_debug_should_wait(kernfs_fop_write_iter, before_mutex))
+		kernfs_debug_wait();
+
 	/*
 	 * @of->mutex nests outside active ref and is used both to ensure that
 	 * the ops aren't called concurrently for the same open file.
 	 */
 	mutex_lock(&of->mutex);
+
+	if (kernfs_debug_should_wait(kernfs_fop_write_iter, after_mutex))
+		kernfs_debug_wait();
+
 	if (!kernfs_get_active(of->kn)) {
 		mutex_unlock(&of->mutex);
 		len = -ENODEV;
 		goto out_free;
 	}
 
+	if (kernfs_debug_should_wait(kernfs_fop_write_iter, after_active))
+		kernfs_debug_wait();
+
 	ops = kernfs_ops(of->kn);
 	if (ops->write)
 		len = ops->write(of, buf, len, iocb->ki_pos);
fs/kernfs/kernfs-internal.h +72 -0

82 lines not on this page · raw message

include/linux/kernfs.h +5 -0
index 9e8ca8743c26..0da36b8e18e1 100644
@@ -410,6 +410,11 @@ void kernfs_init(void);
 
 struct kernfs_node *kernfs_find_and_get_node_by_id(struct kernfs_root *root,
 						   u64 id);
+
+#ifdef CONFIG_FAIL_KERNFS_KNOBS
+extern struct completion kernfs_debug_wait_completion;
+#endif
+
 #else	/* CONFIG_KERNFS */
 
 static inline enum kernfs_node_type kernfs_type(struct kernfs_node *kn)
lib/Kconfig.debug +10 -0
index 568838ac1189..7dbfbfbd3cfc 100644
@@ -1930,6 +1930,16 @@ config FAULT_INJECTION_USERCOPY
 	  Provides fault-injection capability to inject failures
 	  in usercopy functions (copy_from_user(), get_user(), ...).
 
+config FAIL_KERNFS_KNOBS
+	bool "Fault-injection support in kernfs"
+	depends on FAULT_INJECTION
+	help
+	  Provide fault-injection capability for kernfs. This only enables
+	  the error injection functionality. To use it you must configure which
+	  which path you want to trigger on error on using debugfs under
+	  /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/. By
+	  default all of these are disabled.
+
 config FAIL_MAKE_REQUEST
 	bool "Fault-injection capability for disk IO"
 	depends on FAULT_INJECTION && BLOCK
Signature · 1 line
2.27.0
#7 lore raw

PATCH v2 3/4 test_sysfs: add support to use kernfs failure injection

In reply to Luis Chamberlain

This extends test_sysfs with support for using the failure injection wait completion and knobs to force a few race conditions which demonstrates that kernfs active reference protection is sufficient for kobject / device protection at higher layers.

This adds 4 new tests which tries to remove the device attribute store operation in 4 different situations:

1) at the start of kernfs_kernfs_fop_write_iter()
2) before the of->mutex is held in kernfs_kernfs_fop_write_iter()
3) after the of->mutex is held in kernfs_kernfs_fop_write_iter()
4) after the kernfs node active reference is taken

A write fails in call cases.

No null dereferences are reproduced, even though this has been observed in some complex testing cases [0]. If this issue really exists we should have enough tools on the sysfs_test toolbox now to try to reproduce this easily without having to poke around other drivers.

[0] https://lkml.kernel.org/r/[email protected]

---
 lib/Kconfig.debug                      |   3 +
 lib/test_sysfs.c                       |  31 +++++
 tools/testing/selftests/sysfs/config   |   3 +
 tools/testing/selftests/sysfs/sysfs.sh | 174 +++++++++++++++++++++++++
 4 files changed, 211 insertions(+)
lib/Kconfig.debug +3 -0
index 7dbfbfbd3cfc..717bc367a9db 100644
@@ -2373,6 +2373,9 @@ config TEST_SYSCTL
 config TEST_SYSFS
 	tristate "sysfs test driver"
 	depends on SYSFS
+	select FAULT_INJECTION
+	select FAULT_INJECTION_DEBUG_FS
+	select FAIL_KERNFS_KNOBS
 	help
 	  This builds the "test_sysfs" module. This driver enables to test the
 	  sysfs file system safely without affecting production knobs which
lib/test_sysfs.c +31 -0
index bf43016d40b5..f27ec0eab747 100644
@@ -38,6 +38,11 @@
 #include <linux/rtnetlink.h>
 #include <linux/genhd.h>
 #include <linux/blkdev.h>
+#include <linux/kernfs.h>
+
+#ifdef CONFIG_FAIL_KERNFS_KNOBS
+MODULE_IMPORT_NS(KERNFS_DEBUG_PRIVATE);
+#endif
 
 static bool enable_lock;
 module_param(enable_lock, bool_enable_only, 0644);
@@ -82,6 +87,13 @@ static bool enable_verbose_rmmod;
 module_param(enable_verbose_rmmod, bool_enable_only, 0644);
 MODULE_PARM_DESC(enable_verbose_rmmod, "enable verbose print messages on rmmod");
 
+#ifdef CONFIG_FAIL_KERNFS_KNOBS
+static bool enable_completion_on_rmmod;
+module_param(enable_completion_on_rmmod, bool_enable_only, 0644);
+MODULE_PARM_DESC(enable_completion_on_rmmod,
+		 "enable sending a kernfs completion on rmmod");
+#endif
+
 static int sysfs_test_major;
 
 /**
@@ -289,6 +301,12 @@ static ssize_t config_show(struct device *dev,
 			"enable_verbose_writes:\t%s\n",
 			enable_verbose_writes ? "true" : "false");
 
+#ifdef CONFIG_FAIL_KERNFS_KNOBS
+	len += snprintf(buf+len, PAGE_SIZE - len,
+			"enable_completion_on_rmmod:\t%s\n",
+			enable_completion_on_rmmod ? "true" : "false");
+#endif
+
 	test_dev_config_unlock(test_dev);
 
 	return len;
@@ -926,10 +944,23 @@ static int __init test_sysfs_init(void)
 }
 module_init(test_sysfs_init);
 
+#ifdef CONFIG_FAIL_KERNFS_KNOBS
+/* The goal is to race our device removal with a pending kernfs -> store call */
+static void test_sysfs_kernfs_send_completion_rmmod(void)
+{
+	if (!enable_completion_on_rmmod)
+		return;
+	complete(&kernfs_debug_wait_completion);
+}
+#else
+static inline void test_sysfs_kernfs_send_completion_rmmod(void) {}
+#endif
+
 static void __exit test_sysfs_exit(void)
 {
 	if (enable_debugfs)
 		debugfs_remove(debugfs_dir);
+	test_sysfs_kernfs_send_completion_rmmod();
 	if (delay_rmmod_ms)
 		msleep(delay_rmmod_ms);
 	unregister_test_dev_sysfs(first_test_dev);
tools/testing/selftests/sysfs/config +3 -0
index 9196f452ecd5..2876a229f95b 100644
@@ -1,2 +1,5 @@
 CONFIG_SYSFS=m
 CONFIG_TEST_SYSFS=m
+CONFIG_FAULT_INJECTION=y
+CONFIG_FAULT_INJECTION_DEBUG_FS=y
+CONFIG_FAIL_KERNFS_KNOBS=y
tools/testing/selftests/sysfs/sysfs.sh +174 -0

210 lines not on this page · raw message

Signature · 1 line
2.27.0
#8 lore raw

PATCH v2 4/4 test_sysfs: demonstrate deadlock fix

In reply to Luis Chamberlain

Two mechanisms have been proposed to fix the sysfs deadlock issue. The first approach proposed is by optionally allowing drivers to specify a module and augmenting attributes with module information [0]. A secondary approach is to use macros on drivers which needs this, in the meantime. This embraces the secondary approach, in lieu of agreement of a generic solution. This should be enough to allow for room for experimentation and demonstration of the issue.

This then also enables the two test cases which we have disabled as otherwise they would deadlock your system.

./tools/testing/selftests/sysfs/sysfs.sh -t 0027 Running test: sysfs_test_0027 - run #0
Test for possible rmmod deadlock while writing x ... ok

./tools/testing/selftests/sysfs/sysfs.sh -t 0028
Running test: sysfs_test_0028 - run #0
Test for possible rmmod deadlock using rtnl_lock while writing x ... ok

[0] https://lkml.kernel.org/r/[email protected]

---
 lib/test_sysfs.c                       | 71 ++++++++++++++++++++++----
 tools/testing/selftests/sysfs/sysfs.sh |  4 +-
 2 files changed, 64 insertions(+), 11 deletions(-)
lib/test_sysfs.c +62 -9
index f27ec0eab747..af14e992e1b8 100644
@@ -94,6 +94,59 @@ MODULE_PARM_DESC(enable_completion_on_rmmod,
 		 "enable sending a kernfs completion on rmmod");
 #endif
 
+#undef __ATTR_RO
+#undef __ATTR_RW
+#undef __ATTR_WO
+
+#define __ATTR_RO(_name) {						\
+	.attr	= { .name = __stringify(_name), .mode = 0444 },		\
+	.show	= module_##_name##_show,						\
+}
+#define __ATTR_RW(_name) __ATTR(_name, 0644, module_##_name##_show, module_##_name##_store)
+#define __ATTR_WO(_name) {						\
+	.attr	= { .name = __stringify(_name), .mode = 0200 },		\
+	.store	= module_##_name##_store,				\
+}
+
+#define MODULE_DEVICE_ATTR_FUNC_STORE(_name) \
+static ssize_t module_ ## _name ## _store(struct device *dev, \
+				   struct device_attribute *attr, \
+				   const char *buf, size_t len) \
+{ \
+	ssize_t __ret; \
+	if (!try_module_get(THIS_MODULE)) \
+		return -ENODEV; \
+	__ret = _name ## _store(dev, attr, buf, len); \
+	module_put(THIS_MODULE); \
+	return __ret; \
+}
+
+#define MODULE_DEVICE_ATTR_FUNC_SHOW(_name) \
+static ssize_t module_ ## _name ## _show(struct device *dev, \
+					 struct device_attribute *attr, \
+					 char *buf) \
+{ \
+	ssize_t __ret; \
+	if (!try_module_get(THIS_MODULE)) \
+		return -ENODEV; \
+	__ret = _name ## _show(dev, attr, buf); \
+	module_put(THIS_MODULE); \
+	return __ret; \
+}
+
+#define MODULE_DEVICE_ATTR_WO(_name) \
+MODULE_DEVICE_ATTR_FUNC_STORE(_name); \
+static DEVICE_ATTR_WO(_name)
+
+#define MODULE_DEVICE_ATTR_RW(_name) \
+MODULE_DEVICE_ATTR_FUNC_STORE(_name); \
+MODULE_DEVICE_ATTR_FUNC_SHOW(_name); \
+static DEVICE_ATTR_RW(_name)
+
+#define MODULE_DEVICE_ATTR_RO(_name) \
+MODULE_DEVICE_ATTR_FUNC_SHOW(_name); \
+static DEVICE_ATTR_RO(_name)
+
 static int sysfs_test_major;
 
 /**
@@ -311,7 +364,7 @@ static ssize_t config_show(struct device *dev,
 
 	return len;
 }
-static DEVICE_ATTR_RO(config);
+MODULE_DEVICE_ATTR_RO(config);
 
 static ssize_t reset_store(struct device *dev,
 			   struct device_attribute *attr,
@@ -336,7 +389,7 @@ static ssize_t reset_store(struct device *dev,
 
 	return count;
 }
-static DEVICE_ATTR_WO(reset);
+MODULE_DEVICE_ATTR_WO(reset);
 
 static void test_dev_busy_alloc(struct sysfs_test_device *test_dev)
 {
@@ -388,7 +441,7 @@ static ssize_t test_dev_x_show(struct device *dev,
 
 	return ret;
 }
-static DEVICE_ATTR_RW(test_dev_x);
+MODULE_DEVICE_ATTR_RW(test_dev_x);
 
 static ssize_t test_dev_y_store(struct device *dev,
 				struct device_attribute *attr,
@@ -432,7 +485,7 @@ static ssize_t test_dev_y_show(struct device *dev,
 
 	return ret;
 }
-static DEVICE_ATTR_RW(test_dev_y);
+MODULE_DEVICE_ATTR_RW(test_dev_y);
 
 static ssize_t config_enable_lock_store(struct device *dev,
 					struct device_attribute *attr,
@@ -477,7 +530,7 @@ static ssize_t config_enable_lock_show(struct device *dev,
 
 	return ret;
 }
-static DEVICE_ATTR_RW(config_enable_lock);
+MODULE_DEVICE_ATTR_RW(config_enable_lock);
 
 static ssize_t config_enable_lock_on_rmmod_store(struct device *dev,
 						 struct device_attribute *attr,
@@ -519,7 +572,7 @@ static ssize_t config_enable_lock_on_rmmod_show(struct device *dev,
 
 	return ret;
 }
-static DEVICE_ATTR_RW(config_enable_lock_on_rmmod);
+MODULE_DEVICE_ATTR_RW(config_enable_lock_on_rmmod);
 
 static ssize_t config_use_rtnl_lock_store(struct device *dev,
 					  struct device_attribute *attr,
@@ -558,7 +611,7 @@ static ssize_t config_use_rtnl_lock_show(struct device *dev,
 
 	return snprintf(buf, PAGE_SIZE, "%d\n", config->use_rtnl_lock);
 }
-static DEVICE_ATTR_RW(config_use_rtnl_lock);
+MODULE_DEVICE_ATTR_RW(config_use_rtnl_lock);
 
 static ssize_t config_write_delay_msec_y_store(struct device *dev,
 					       struct device_attribute *attr,
@@ -592,7 +645,7 @@ static ssize_t config_write_delay_msec_y_show(struct device *dev,
 
 	return snprintf(buf, PAGE_SIZE, "%d\n", config->write_delay_msec_y);
 }
-static DEVICE_ATTR_RW(config_write_delay_msec_y);
+MODULE_DEVICE_ATTR_RW(config_write_delay_msec_y);
 
 static ssize_t config_enable_busy_alloc_store(struct device *dev,
 					      struct device_attribute *attr,
@@ -626,7 +679,7 @@ static ssize_t config_enable_busy_alloc_show(struct device *dev,
 
 	return snprintf(buf, PAGE_SIZE, "%d\n", config->enable_busy_alloc);
 }
-static DEVICE_ATTR_RW(config_enable_busy_alloc);
+MODULE_DEVICE_ATTR_RW(config_enable_busy_alloc);
 
 #define TEST_SYSFS_DEV_ATTR(name)		(&dev_attr_##name.attr)
 
tools/testing/selftests/sysfs/sysfs.sh +2 -2
index f27ea61e0e95..2de9f37cb00b 100755
@@ -60,8 +60,8 @@ ALL_TESTS="$ALL_TESTS 0023:1:1:test_dev_y:block"
 ALL_TESTS="$ALL_TESTS 0024:1:1:test_dev_x:block"
 ALL_TESTS="$ALL_TESTS 0025:1:1:test_dev_y:block"
 ALL_TESTS="$ALL_TESTS 0026:1:1:test_dev_y:block"
-ALL_TESTS="$ALL_TESTS 0027:1:0:test_dev_x:block" # deadlock test
-ALL_TESTS="$ALL_TESTS 0028:1:0:test_dev_x:block" # deadlock test with rntl_lock
+ALL_TESTS="$ALL_TESTS 0027:1:1:test_dev_x:block" # deadlock test
+ALL_TESTS="$ALL_TESTS 0028:1:1:test_dev_x:block" # deadlock test with rntl_lock
 ALL_TESTS="$ALL_TESTS 0029:1:1:test_dev_x:block" # kernfs race removal of store
 ALL_TESTS="$ALL_TESTS 0030:1:1:test_dev_x:block" # kernfs race removal before mutex
 ALL_TESTS="$ALL_TESTS 0031:1:1:test_dev_x:block" # kernfs race removal after mutex
Signature · 1 line
2.27.0
maintainer
#9 lore raw

Re: [PATCH v2 4/4] test_sysfs: demonstrate deadlock fix

lib/test_sysfs.c MODULE_PARM_DESC(enable_completion_on_rmmod, · line 122

Quoted 12 lines · Luis Chamberlain
lib/test_sysfs.c +12 -0
+#define MODULE_DEVICE_ATTR_FUNC_STORE(_name) \
+static ssize_t module_ ## _name ## _store(struct device *dev, \
+				   struct device_attribute *attr, \
+				   const char *buf, size_t len) \
+{ \
+	ssize_t __ret; \
+	if (!try_module_get(THIS_MODULE)) \
+		return -ENODEV; \
+	__ret = _name ## _store(dev, attr, buf, len); \
+	module_put(THIS_MODULE); \
+	return __ret; \
+}

As I have pointed out before, doing try_module_get(THIS_MODULE) is racy and should not be added back to the kernel tree. We got rid of many instances of this "bad pattern" over the years, please do not encourage it to be added back as others will somehow think that it correct code.

I'll go over the rest of this after 5.14-rc1 is out, am busy until then.

thanks,

greg k-h

#10 lore raw

Re: [PATCH v2 4/4] test_sysfs: demonstrate deadlock fix

Quoted 10 lines · Greg KH
Quoted 9 lines · Luis Chamberlain
lib/test_sysfs.c +9 -0
+#define MODULE_DEVICE_ATTR_FUNC_STORE(_name) \
+static ssize_t module_ ## _name ## _store(struct device *dev, \
+				   struct device_attribute *attr, \
+				   const char *buf, size_t len) \
+{ \
+	ssize_t __ret; \
+	if (!try_module_get(THIS_MODULE)) \
+		return -ENODEV; \
+	__ret = _name ## _store(dev, attr, buf, len); \
Quoted 8 lines

lib/test_sysfs.c MODULE_PARM_DESC(enable_completion_on_rmmod, · line 122

Quoted 3 lines
lib/test_sysfs.c +3 -0
+	module_put(THIS_MODULE); \
+	return __ret; \
+}

As I have pointed out before, doing try_module_get(THIS_MODULE) is racy and should not be added back to the kernel tree. We got rid of many instances of this "bad pattern" over the years, please do not encourage it to be added back as others will somehow think that it correct code.

It is noted this is used in lieu of any agreed upon solution to *demonstrate* how this at least does fix it. In this case (and in the generic solution I also had suggested for kernfs a while ago), if the try fails, we give up. If it succeeds, we now know we can rely on the device pointer. If the refcount succeeds, can the module still not be present? Is try_module_get() racy in that way? In what way is it racy and where is this documented? Do we have a selftest to prove the race?

Luis

maintainer
#11 lore raw

Re: [PATCH v2 4/4] test_sysfs: demonstrate deadlock fix

Quoted 20 lines · Luis Chamberlain
Quoted 10 lines · Greg KH
Quoted 9 lines · Luis Chamberlain
lib/test_sysfs.c +9 -0
+#define MODULE_DEVICE_ATTR_FUNC_STORE(_name) \
+static ssize_t module_ ## _name ## _store(struct device *dev, \
+				   struct device_attribute *attr, \
+				   const char *buf, size_t len) \
+{ \
+	ssize_t __ret; \
+	if (!try_module_get(THIS_MODULE)) \
+		return -ENODEV; \
+	__ret = _name ## _store(dev, attr, buf, len); \
Quoted 8 lines

lib/test_sysfs.c MODULE_PARM_DESC(enable_completion_on_rmmod, · line 122

Quoted 3 lines
lib/test_sysfs.c +3 -0
+	module_put(THIS_MODULE); \
+	return __ret; \
+}

As I have pointed out before, doing try_module_get(THIS_MODULE) is racy and should not be added back to the kernel tree. We got rid of many instances of this "bad pattern" over the years, please do not encourage it to be added back as others will somehow think that it correct code.

Quoted 8 lines

It is noted this is used in lieu of any agreed upon solution to *demonstrate* how this at least does fix it. In this case (and in the generic solution I also had suggested for kernfs a while ago), if the try fails, we give up. If it succeeds, we now know we can rely on the device pointer. If the refcount succeeds, can the module still not be present? Is try_module_get() racy in that way? In what way is it racy and where is this documented? Do we have a selftest to prove the race?

As I say in the other email where you tried to add this, think about what happens if the module is removed _right before_ you make this call.

Or a few instructions before that. The race is still there, this fixes nothing except make the window smaller.

thanks,

greg k-h

#12 lore raw

Re: [PATCH v2 4/4] test_sysfs: demonstrate deadlock fix

Quoted 30 lines · Greg KH
Quoted 20 lines · Luis Chamberlain
Quoted 10 lines · Greg KH
Quoted 9 lines · Luis Chamberlain
lib/test_sysfs.c +9 -0
+#define MODULE_DEVICE_ATTR_FUNC_STORE(_name) \
+static ssize_t module_ ## _name ## _store(struct device *dev, \
+				   struct device_attribute *attr, \
+				   const char *buf, size_t len) \
+{ \
+	ssize_t __ret; \
+	if (!try_module_get(THIS_MODULE)) \
+		return -ENODEV; \
+	__ret = _name ## _store(dev, attr, buf, len); \
Quoted 8 lines

lib/test_sysfs.c MODULE_PARM_DESC(enable_completion_on_rmmod, · line 122

Quoted 3 lines
lib/test_sysfs.c +3 -0
+	module_put(THIS_MODULE); \
+	return __ret; \
+}

As I have pointed out before, doing try_module_get(THIS_MODULE) is racy and should not be added back to the kernel tree. We got rid of many instances of this "bad pattern" over the years, please do not encourage it to be added back as others will somehow think that it correct code.

Quoted 8 lines

It is noted this is used in lieu of any agreed upon solution to *demonstrate* how this at least does fix it. In this case (and in the generic solution I also had suggested for kernfs a while ago), if the try fails, we give up. If it succeeds, we now know we can rely on the device pointer. If the refcount succeeds, can the module still not be present? Is try_module_get() racy in that way? In what way is it racy and where is this documented? Do we have a selftest to prove the race?

Quoted 5 lines

As I say in the other email where you tried to add this, think about what happens if the module is removed _right before_ you make this call.

Or a few instructions before that. The race is still there, this fixes nothing except make the window smaller.

The kernfs active reference ensures that if the file is open the module must still exist. As such, the use within sysfs files should be safe as the module is the one in charge of removing the files.

Luis