firmware: selftest for request_firmware_into_buf
Subsystems (6)
cover firmware: selftest for request_firmware_into_buf
This patch series adds kernel selftest of request_firmware_into_buf. The API was added to the kernel previously untested.
Also included in this patch series is a fix for a race condition discovered while testing request_firmware_into_buf. Mutex may not be correct final solution but demonstrates a fix to a race condition new test exposes.
Scott Branden (3):
test_firmware: add support for request_firmware_into_buf
selftest: firmware: Add request_firmware_into_buf tests
firmware: add mutex fw_lock_fallback for race condition
drivers/base/firmware_loader/main.c | 15 +++++ lib/test_firmware.c | 50 +++++++++++++++- .../selftests/firmware/fw_filesystem.sh | 57 ++++++++++++++++++- tools/testing/selftests/firmware/fw_lib.sh | 11 ++++ 4 files changed, 129 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/3 test_firmware: add support for request_firmware_into_buf
Add test config into_buf to allow request_firmware_into_buf to be called instead of request_firmware/request_firmware_direct. The number of parameters differ calling request_firmware_into_buf and support has not been added to test such api in test_firmware until now.
--- lib/test_firmware.c | 50 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 48 insertions(+), 2 deletions(-)
lib/test_firmware.c +48 -2
@@ -18,6 +18,7 @@#include <linux/device.h> #include <linux/fs.h> #include <linux/miscdevice.h> +#include <linux/sizes.h> #include <linux/slab.h> #include <linux/uaccess.h> #include <linux/delay.h>
@@ -26,6 +27,7 @@#define TEST_FIRMWARE_NAME "test-firmware.bin"#define TEST_FIRMWARE_NUM_REQS 4 +#define TEST_FIRMWARE_BUF_SIZE SZ_1Kstatic DEFINE_MUTEX (test_fw_mutex);static const struct firmware *test_firmware;
@@ -45,6 +47,8 @@ struct test_batched_req {* test_config - represents configuration for the test for different triggers * * @name: the name of the firmware file to look for +* @into_buf: when the into_buf is used if this is true +* request_firmware_into_buf() will be used instead. * @sync_direct: when the sync trigger is used if this is true * request_firmware_direct() will be used instead. * @send_uevent: whether or not to send a uevent for async requests
@@ -83,6 +87,7 @@ struct test_batched_req {*/ struct test_config {char *name; +bool into_buf;bool sync_direct;bool send_uevent;u8 num_requests;
@@ -176,6 +181,7 @@ static int __test_firmware_config_init(void) test_fw_config->num_requests =TEST_FIRMWARE_NUM_REQS ; test_fw_config->send_uevent = true; + test_fw_config->into_buf = false; test_fw_config->sync_direct = false; test_fw_config->req_firmware = request_firmware; test_fw_config->test_result =0 ;
@@ -244,6 +250,9 @@ static ssize_t config_show(struct device *dev, test_fw_config->send_uevent ?"FW_ACTION_HOTPLUG" : "FW_ACTION_NOHOTPLUG"); + len +=scnprintf (buf+len,PAGE_SIZE - len, +"into_buf:\t\t%s\n" , + test_fw_config->into_buf ?"true" :"false" ); len +=scnprintf (buf+len,PAGE_SIZE - len,"sync_direct:\t\t%s\n" , test_fw_config->sync_direct ?"true" :"false" );
@@ -393,6 +402,23 @@ static ssize_t config_num_requests_show(struct device *dev, }static DEVICE_ATTR_RW (config_num_requests); +static ssize_t config_into_buf_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +return test_dev_config_update_bool (buf, + count, + &test_fw_config->into_buf); +} + +static ssize_t config_into_buf_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +return test_dev_config_show_bool (buf, test_fw_config->into_buf); +} +static DEVICE_ATTR_RW (config_into_buf); +static ssize_t config_sync_direct_store (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)
@@ -522,7 +548,7 @@ static ssize_t trigger_async_request_store(struct device *dev, rc = count; }else {pr_err ("failed to async load firmware\n" ); - rc = -ENODEV ; + rc = -ENOMEM ; } out:
@@ -585,7 +611,26 @@ static int test_fw_run_batch_request(void *data)return -EINVAL ; } - req->rc = test_fw_config->req_firmware (&req->fw, req->name, req->dev); +if (test_fw_config->into_buf) { +void *test_buf; + + test_buf =kzalloc (TEST_FIRMWARE_BUF_SIZE ,GFP_KERNEL ); +if (!test_buf) +return -ENOSPC ; + + req->rc =request_firmware_into_buf (&req->fw, + req->name, + req->dev, + test_buf, +TEST_FIRMWARE_BUF_SIZE ); +if (!req->fw) +kfree (test_buf); + }else { + req->rc = test_fw_config->req_firmware (&req->fw, + req->name, + req->dev); + } +if (req->rc) {pr_info ("#%u: batched sync load failed: %d\n" , req->idx, req->rc);
@@ -849,6 +894,7 @@ static struct attribute *test_dev_attrs[] = {TEST_FW_DEV_ATTR (config),TEST_FW_DEV_ATTR (config_name),TEST_FW_DEV_ATTR (config_num_requests), +TEST_FW_DEV_ATTR (config_into_buf),TEST_FW_DEV_ATTR (config_sync_direct),TEST_FW_DEV_ATTR (config_send_uevent), TEST_FW_DEV_ATTR(config_read_fw_idx),
Signature · 1 line
2.17.1
Re: [PATCH 1/3] test_firmware: add support for request_firmware_into_buf
Quoted 6 lines · Scott Branden
Add test config
into_bufto allowrequest_firmware_into_bufto be called instead of request_firmware/request_firmware_direct. The number of parameters differ callingrequest_firmware_into_bufand support has not been added to test such api intest_firmwareuntil now.Signed-off-by Scott Branden <[email protected]>
Thanks for the patch!
Luis
Re: [PATCH 1/3] test_firmware: add support for request_firmware_into_buf
Quoted 6 lines · Luis Chamberlain
Quoted 5 lines · Scott Branden
Add test config
into_bufto allowrequest_firmware_into_bufto be called instead of request_firmware/request_firmware_direct. The number of parameters differ callingrequest_firmware_into_bufand support has not been added to test such api intest_firmwareuntil now.
Quoted 8 lines
Quoted 1 line
Signed-off-by Scott Branden <[email protected]>Thanks for the patch!
Acked-by Luis Chamberlain <[email protected]>Luis
Greg! Pls let me know if you would like me to take this throough my tree. If not,
thanks,
-- Shuah
PATCH 2/3 selftest: firmware: Add request_firmware_into_buf tests
In reply to Scott BrandenAdd tests cases for checking request_firmware_into_buf api. API was introduced into kernel with no testing present previously.
--- .../selftests/firmware/fw_filesystem.sh | 57 ++++++++++++++++++- tools/testing/selftests/firmware/fw_lib.sh | 11 ++++ 2 files changed, 66 insertions(+), 2 deletions(-)
tools/testing/selftests/firmware/fw_filesystem.sh +55 -2
@@ -116,6 +116,16 @@ config_set_name() echo -n $1 > $DIR/config_name } +config_set_into_buf() +{ + echo 1 > $DIR/config_into_buf +} + +config_unset_into_buf() +{ + echo 0 > $DIR/config_into_buf +} + config_set_sync_direct() { echo 1 > $DIR/config_sync_direct
@@ -153,11 +163,14 @@ config_set_read_fw_idx() read_firmwares() { - if [ "$1" = "xzonly" ]; then - fwfile="${FW}-orig" + if [ "$(cat $DIR/config_into_buf)" == "1" ]; then + fwfile="$FW_INTO_BUF" else fwfile="$FW" fi + if [ "$1" = "xzonly" ]; then + fwfile="${fwfile}-orig" + fi for i in $(seq 0 3); do config_set_read_fw_idx $i # Verify the contents are what we expect.
@@ -194,6 +207,18 @@ test_batched_request_firmware_nofile() echo "OK" } +test_batched_request_firmware_into_buf_nofile() +{ + echo -n "Batched request_firmware_into_buf() nofile try #$1: " + config_reset + config_set_name nope-test-firmware.bin + config_set_into_buf + config_trigger_sync + read_firmwares_expect_nofile + release_all_firmware + echo "OK" +} + test_batched_request_firmware_direct_nofile() { echo -n "Batched request_firmware_direct() nofile try #$1: "
@@ -259,6 +284,18 @@ test_batched_request_firmware() echo "OK" } +test_batched_request_firmware_into_buf() +{ + echo -n "Batched request_firmware_into_buf() $2 try #$1: " + config_reset + config_set_name $TEST_FIRMWARE_INTO_BUF_FILENAME + config_set_into_buf + config_trigger_sync + read_firmwares $2 + release_all_firmware + echo "OK" +} + test_batched_request_firmware_direct() { echo -n "Batched request_firmware_direct() $2 try #$1: "
@@ -307,6 +344,10 @@ for i in $(seq 1 5); do test_batched_request_firmware $i normal done +for i in $(seq 1 5); do + test_batched_request_firmware_into_buf $i normal +done + for i in $(seq 1 5); do test_batched_request_firmware_direct $i normal done
@@ -327,6 +368,10 @@ for i in $(seq 1 5); do test_batched_request_firmware_nofile $i done +for i in $(seq 1 5); do + test_batched_request_firmware_into_buf_nofile $i +done + for i in $(seq 1 5); do test_batched_request_firmware_direct_nofile $i done
@@ -350,6 +395,10 @@ for i in $(seq 1 5); do test_batched_request_firmware $i both done +for i in $(seq 1 5); do + test_batched_request_firmware_into_buf $i both +done + for i in $(seq 1 5); do test_batched_request_firmware_direct $i both done
@@ -370,6 +419,10 @@ for i in $(seq 1 5); do test_batched_request_firmware $i xzonly done +for i in $(seq 1 5); do + test_batched_request_firmware_into_buf $i xzonly +done + for i in $(seq 1 5); do test_batched_request_firmware_direct $i xzonly done
tools/testing/selftests/firmware/fw_lib.sh +11 -0
@@ -9,6 +9,12 @@ DIR=/sys/devices/virtual/misc/test_firmware PROC_CONFIG="/proc/config.gz" TEST_DIR=$(dirname $0) +# We need to load a different file to test request_firmware_into_buf +# I believe the issue is firmware loaded cached vs. non-cached +# with same filename is bungled. +# To reproduce rename this to test-firmware.bin +TEST_FIRMWARE_INTO_BUF_FILENAME=test-firmware-into-buf.bin + # Kselftest framework requirement - SKIP code is 4. ksft_skip=4
@@ -108,6 +114,8 @@ setup_tmp_file() FWPATH=$(mktemp -d) FW="$FWPATH/test-firmware.bin" echo "ABCD0123" >"$FW" + FW_INTO_BUF="$FWPATH/$TEST_FIRMWARE_INTO_BUF_FILENAME" + echo "EFGH4567" >"$FW_INTO_BUF" NAME=$(basename "$FW") if [ "$TEST_REQS_FW_SET_CUSTOM_PATH" = "yes" ]; then echo -n "$FWPATH" >/sys/module/firmware_class/parameters/path
@@ -175,6 +183,9 @@ test_finish() if [ -f $FW ]; then rm -f "$FW" fi + if [ -f $FW_INTO_BUF ]; then + rm -f "$FW_INTO_BUF" + fi if [ -d $FWPATH ]; then rm -rf "$FWPATH" fi
Signature · 1 line
2.17.1
Re: [PATCH 2/3] selftest: firmware: Add request_firmware_into_buf tests
Quoted 4 lines · Scott Branden
Add tests cases for checking
request_firmware_into_bufapi. API was introduced into kernel with no testing present previously.Signed-off-by Scott Branden <[email protected]>
Luis
Re: [PATCH 2/3] selftest: firmware: Add request_firmware_into_buf tests
Quoted 10 lines · Luis Chamberlain
Quoted 4 lines · Scott Branden
Add tests cases for checking
request_firmware_into_bufapi. API was introduced into kernel with no testing present previously.Signed-off-by Scott Branden <[email protected]>Acked-by Luis Chamberlain <[email protected]>Luis
Greg! Pls let me know if you would like me to take this throough my tree. If not,
PATCH 3/3 firmware: add mutex fw_lock_fallback for race condition
In reply to Scott BrandenA race condition exists between _request_firmware_prepare checking if firmware is assigned and firmware_fallback_sysfs creating a sysfs entry (kernel trace below). To avoid such condition add a mutex fw_lock_fallback to protect against such condition.
misc test_firmware: Falling back to sysfs fallback for: nope-test-firmware.bin
sysfs: cannot create duplicate filename '/devices/virtual/misc/test_firmware/nope-test-firmware.bin'
CPU: 4 PID: 2059 Comm: test_firmware-3 Not tainted 5.3.0-rc4 #1 Hardware name: Dell Inc. OptiPlex 7010/0KRC95, BIOS A13 03/25/2013 Call Trace: dump_stack+0x67/0x90 sysfs_warn_dup.cold+0x17/0x24 sysfs_create_dir_ns+0xb3/0xd0 kobject_add_internal+0xa6/0x2a0 kobject_add+0x7e/0xb0 ? _cond_resched+0x15/0x30 device_add+0x121/0x670 firmware_fallback_sysfs+0x15c/0x3c9 _request_firmware+0x432/0x5a0 ? devres_find+0x63/0xc0 request_firmware_into_buf+0x63/0x80 test_fw_run_batch_request+0x96/0xe0 kthread+0xfb/0x130 ? reset_store+0x30/0x30 ? kthread_park+0x80/0x80 ret_from_fork+0x3a/0x50
kobject_add_internal failed for nope-test-firmware.bin with -EEXIST, don't try to register things with the same name in the same directory.
--- drivers/base/firmware_loader/main.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
drivers/base/firmware_loader/main.c +15 -0
@@ -88,6 +88,7 @@ static inline struct fw_priv *to_fw_priv(struct kref *ref)/* fw_lock could be moved to 'struct fw_sysfs' but since it is just * guarding for corner cases a global lock should be OK */ DEFINE_MUTEX (fw_lock); +DEFINE_MUTEX (fw_lock_fallback);static struct firmware_cache fw_cache;
@@ -758,6 +759,17 @@ _request_firmware(const struct firmware **firmware_p, const char *name,if (!firmware_p)return -EINVAL ; +/* +* There is a race condition between _request_firmware_prepare checking +* if firmware is assigned and firmware_fallback_sysfs creating sysfs +* entries with duplicate names. +* Yet, with this lock the firmware_test locks up with cache enabled +* and no event used during firmware test. +* This points to some very racy code I don't know how to entirely fix. +*/ +if (opt_flags &FW_OPT_NOCACHE ) +mutex_lock (&fw_lock_fallback); +if (!name || name[0 ] =='\0' ) { ret = -EINVAL ; goto out;
@@ -791,6 +803,9 @@ _request_firmware(const struct firmware **firmware_p, const char *name, fw =NULL ; } +if (opt_flags &FW_OPT_NOCACHE ) +mutex_unlock (&fw_lock_fallback); + *firmware_p = fw;return ret; }
Signature · 1 line
2.17.1
Luis Chamberlain
maintainerRe: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
Quoted 4 lines · Scott Branden
A race condition exists between
_request_firmware_preparechecking if firmware is assigned andfirmware_fallback_sysfscreating a sysfs entry (kernel trace below). To avoid such condition add a mutexfw_lock_fallbackto protect against such condition.
I am not buying this fix, and it seems sloppy. More below.
Quoted 1 line
misc
test_firmware: Falling back to sysfs fallback for: nope-test-firmware.bin
So the fallback kicks in with the file that is not there.
Quoted 1 line
sysfs: cannot create duplicate filename '/devices/virtual/misc/test_firmware/nope-test-firmware.bin'
And we have a duplicate entry, for the *device* created to allow us to create a file entry to allow us to copy the file. Your tests had a loop, so there is actually a race between two entries being created while one one failed.
Quoted 8 lines
CPU: 4 PID: 2059 Comm: test_firmware-3 Not tainted 5.3.0-rc4 #1 Hardware name: Dell Inc. OptiPlex 7010/0KRC95, BIOS A13 03/25/2013 Call Trace: dump_stack+0x67/0x90 sysfs_warn_dup.cold+0x17/0x24 sysfs_create_dir_ns+0xb3/0xd0 kobject_add_internal+0xa6/0x2a0 kobject_add+0x7e/0xb0
Note: kobject_add().
Quoted 11 lines
? _cond_resched+0x15/0x30 device_add+0x121/0x670 firmware_fallback_sysfs+0x15c/0x3c9 _request_firmware+0x432/0x5a0 ? devres_find+0x63/0xc0 request_firmware_into_buf+0x63/0x80 test_fw_run_batch_request+0x96/0xe0 kthread+0xfb/0x130 ? reset_store+0x30/0x30 ? kthread_park+0x80/0x80 ret_from_fork+0x3a/0x50
Quoted 1 line
kobject_add_internalfailed for nope-test-firmware.bin with-EEXIST, don't try to register things with the same name in the same directory.
So above it makes it even clearer, two kobjets with the same name.
Quoted 10 lines
Signed-off-by Scott Branden <[email protected]>--- drivers/base/firmware_loader/main.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)drivers/base/firmware_loader/main.c +0
-0
drivers/base/firmware_loader/main.c static inline struct fw_priv *to_fw_priv(struct kref *ref) · line 91
Quoted 4 lines
drivers/base/firmware_loader/main.c +1
-0@@ -88,6 +88,7 @@ static inline struct fw_priv *to_fw_priv(struct kref *ref)/* fw_lock could be moved to 'struct fw_sysfs' but since it is just * guarding for corner cases a global lock should be OK */ DEFINE_MUTEX (fw_lock); +DEFINE_MUTEX (fw_lock_fallback);
The reason I don't like this fix is that this mutex is named after ther fallback interface... but...
Quoted 7 lines
drivers/base/firmware_loader/main.c +0
-0static struct firmware_cache fw_cache;@@ -758,6 +759,17 @@ _request_firmware(const struct firmware **firmware_p, const char *name,if (!firmware_p)return -EINVAL ;
drivers/base/firmware_loader/main.c _request_firmware(const struct firmware **firmware_p, const char *name, · line 771
Quoted 10 lines
drivers/base/firmware_loader/main.c +10
-0+/* +* There is a race condition between _request_firmware_prepare checking +* if firmware is assigned and firmware_fallback_sysfs creating sysfs +* entries with duplicate names. +* Yet, with this lock the firmware_test locks up with cache enabled +* and no event used during firmware test. +* This points to some very racy code I don't know how to entirely fix. +*/ +if (opt_flags &FW_OPT_NOCACHE ) +mutex_lock (&fw_lock_fallback);
Whoa.. What does no-cache have anything to do with the fallback interface other than the fact we enable this feature for the fallback interface? We don't need to penalize non-fallback users who *also* may want to enable the no-cache feature.
So, the fix should be within the boundaries of the creation / deletion of the kobject, not this nocache feature. Can you please re-evaluate this code and look for a more compartamentalized solution to the fallback code only?
Luis
Re: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
Hi Luis,
Thanks for the review.
I did not think this patch would be the final solution either
as indicated in the original cover letter and code comment.
Some comments inline.
Quoted 79 lines · Luis Chamberlain
Quoted 4 lines · Scott Branden
A race condition exists between
_request_firmware_preparechecking if firmware is assigned andfirmware_fallback_sysfscreating a sysfs entry (kernel trace below). To avoid such condition add a mutexfw_lock_fallbackto protect against such condition.I am not buying this fix, and it seems sloppy. More below.
Quoted 1 line
misc
test_firmware: Falling back to sysfs fallback for: nope-test-firmware.binSo the fallback kicks in with the file that is not there.
Quoted 1 line
sysfs: cannot create duplicate filename '/devices/virtual/misc/test_firmware/nope-test-firmware.bin'
And we have a duplicate entry, for the *device* created to allow us to create a file entry to allow us to copy the file. Your tests had a loop, so there is actually a race between two entries being created while one one failed.
Quoted 8 lines
CPU: 4 PID: 2059 Comm: test_firmware-3 Not tainted 5.3.0-rc4 #1 Hardware name: Dell Inc. OptiPlex 7010/0KRC95, BIOS A13 03/25/2013 Call Trace: dump_stack+0x67/0x90 sysfs_warn_dup.cold+0x17/0x24 sysfs_create_dir_ns+0xb3/0xd0 kobject_add_internal+0xa6/0x2a0 kobject_add+0x7e/0xb0Note:
kobject_add().Quoted 11 lines
? _cond_resched+0x15/0x30 device_add+0x121/0x670 firmware_fallback_sysfs+0x15c/0x3c9 _request_firmware+0x432/0x5a0 ? devres_find+0x63/0xc0 request_firmware_into_buf+0x63/0x80 test_fw_run_batch_request+0x96/0xe0 kthread+0xfb/0x130 ? reset_store+0x30/0x30 ? kthread_park+0x80/0x80 ret_from_fork+0x3a/0x50Quoted 1 line
kobject_add_internalfailed for nope-test-firmware.bin with-EEXIST, don't try to register things with the same name in the same directory.So above it makes it even clearer, two kobjets with the same name.
Quoted 10 lines
Signed-off-by Scott Branden <[email protected]>--- drivers/base/firmware_loader/main.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)drivers/base/firmware_loader/main.c +0
-0drivers/base/firmware_loader/main.c static inline struct fw_priv *to_fw_priv(struct kref *ref) · line 91
Quoted 4 lines
drivers/base/firmware_loader/main.c +1
-0@@ -88,6 +88,7 @@ static inline struct fw_priv *to_fw_priv(struct kref *ref)/* fw_lock could be moved to 'struct fw_sysfs' but since it is just * guarding for corner cases a global lock should be OK */ DEFINE_MUTEX (fw_lock); +DEFINE_MUTEX (fw_lock_fallback);The reason I don't like this fix is that this mutex is named after ther fallback interface... but...
Quoted 7 lines
drivers/base/firmware_loader/main.c +0
-0static struct firmware_cache fw_cache;@@ -758,6 +759,17 @@ _request_firmware(const struct firmware **firmware_p, const char *name,if (!firmware_p)return -EINVAL ;drivers/base/firmware_loader/main.c _request_firmware(const struct firmware **firmware_p, const char *name, · line 771
Quoted 10 lines
drivers/base/firmware_loader/main.c +10
-0+/* +* There is a race condition between _request_firmware_prepare checking +* if firmware is assigned and firmware_fallback_sysfs creating sysfs +* entries with duplicate names. +* Yet, with this lock the firmware_test locks up with cache enabled +* and no event used during firmware test. +* This points to some very racy code I don't know how to entirely fix. +*/ +if (opt_flags &FW_OPT_NOCACHE ) +mutex_lock (&fw_lock_fallback);Whoa.. What does no-cache have anything to do with the fallback interface other than the fact we enable this feature for the fallback interface? We don't need to penalize non-fallback users who *also* may want to enable the no-cache feature.
Quoted 4 lines
So, the fix should be within the boundaries of the creation / deletion of the kobject, not this nocache feature. Can you please re-evaluate this code and look for a more compartamentalized solution to the fallback code only?
To be honest, I find the entire firmware code sloppy. I don't think the cache/no-cache feature is
implemented or tested properly nor fallback to begin with. I'm not claiming this patch is the final
solution and indicated such in the cover letter and the comment above.
I hope there is someone more familiar with this code to comment further and come up with a proper solution.
I have found numerous issues and race conditions with the firmware code (I simply added a test).
1) Try loading the same valid firmware using no-cache once it has already been loaded with cache.
It won't work, which is why I had to use a different filename in the test for request_firmware_into_buf.
2) Try removing the "if (opt_flags & FW_OPT_NOCACHE)" in my patch and always call the mutex.
The firmware test will lock up during a "no uevent" test. I am not familiar with the code to
know why such is true and what issue this exposes in the code.
3) I have a driver that uses request_firmware_into_buf and have multiple instances of the driver
loading the same firmware in parallel. Some of the data is not read correctly in each instance.
I haven't yet to reproduce this issue with the firmware test but currently have a mutex around the entire
call to request_firmware_into_buf in our driver.
Perhaps it is better at this point to add a mutex in request_firmware_into_buf to make is entirely safe?
(Perhaps even with every request_firmware functions as none seems to be tested properly.)
Or, add a new function called safe_request_firmware_into_buf with such mutex to protect the function.
The current racey request_firmware functions could then be left alone and those who want reliable
firmware loading can use the safe calls?
Quoted 2 lines
Luis
Luis Chamberlain
maintainerRe: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
Quoted 1 line · Scott Branden
To be honest, I find the entire firmware code sloppy.
And that is after years of cleanup on my part. Try going back to v4.1 for instance, check the code out then for an incredible horrific sight :)
Quoted 2 lines
I don't think the cache/no-cache feature is implemented or tested properly nor fallback to begin with.
I'm in total agreement! I *know* there must be holes in that code, and I acknowledge a few possible gotchas on the commit logs. For instance, I acknowledged that the firmware cache had a secondary purpose which was not well documented or understood through commit e44565f62a720 ("firmware: fix batched requests - wake all waiters"). The firmware cache allows for batching requests and sharing the same original request for multiple consecutive requests which *race against each other*. That's when I started having my doubts about the architecture of the firmware cache mechanism, it seemed too complex and perhaps overkill and considered killing it.
As I noted in that commit, the firmware cache is used for:
1) Addressing races with file lookups during the suspend/resume cycle by keeping firmware in memory during the suspend/resume cycle
2) Batched requests for the same file rely only on work from the first file lookup, which keeps the firmware in memory until the last release_firmware() is called
Also worth quoting from that commit as well:
"Batched requests *only* take effect if secondary requests come in prior to the first user calling release_firmware(). The devres name used for the internal firmware cache is used as a hint other pending requests are ongoing, the firmware buffer data is kept in memory until the last user of the buffer calls release_firmware(), therefore serializing requests and delaying the release until all requests are done."
Later we discovered that the firmware cache had a serious security issue since its inception through commit 422b3db2a503 ("firmware: Fix security issue with request_firmware_into_buf()"). Granted, exploiting this would require the ability to load kernel code, so the vector of exploitation is rather small.
The cache stuff cannot be removed as it *at least* resolves the fw suspend stuff, but still, this can likely use a revisit in rachitecture long term. The second implicit use case for batched requests however seems complex and not sure if its worth to maintain. I'll note that at least some drivers *do* their own firmware caching, iwlwifi, is one, so there is an example there to allow drivers to say "I actually don't need caching" for the future.
If you're volunteering to cleaning / testing the cache stuff I highly welcome that. That and the fallback stuff has been needing testing for years. Someoone was working on patches on the test case for cache stuff a while ago, from Intel, but they disappeared.
Quoted 2 lines
I'm not claiming this patch is the final
solution and indicated such in the cover letter and the comment above.
I missed that sorry.
Quoted 2 lines
I hope there is someone more familiar with this code to comment further and come up with a proper solution.
Alright, I'll dig in and take a look, and propose an alternative.
Quoted 2 lines
I have found numerous issues and race conditions with the firmware code (I simply added a test).
That is nothing compared to the amount of fixes I have found and actually fixed too, the code was a nightmare before I took on maintenance.
Quoted 2 lines
1) Try loading the same valid firmware using no-cache once it has already been loaded with cache.
:)
Quoted 2 lines
It won't work, which is why I had to use a different filename in the test for
request_firmware_into_buf.
Alright, I'll go try to fix this. Thanks for the report.
Quoted 7 lines
2) Try removing the "if (
opt_flags&FW_OPT_NOCACHE)" in my patch and always call the mutex.The firmware test will lock up during a "no uevent" test. I am not familiar with the code to
know why such is true and what issue this exposes in the code.
I hinted in my review of the oops what the issue was.
Quoted 2 lines
3) I have a driver that uses
request_firmware_into_bufand have multiple instances of the driver
Cool, is the driver upstream?
Quoted 2 lines
loading the same firmware in parallel. Some of the data is not read correctly in each instance.
Makes perfect sense considering the lack of testing I noted.
Quoted 1 line
I haven't yet to reproduce this issue with the firmware test
That's because of batched firmware request mechanism.
Quoted 3 lines
but currently
have a mutex around the entire
call torequest_firmware_into_bufin our driver.
I will take a look at this now.
Quoted 2 lines
Perhaps it is better at this point to add a mutex in
request_firmware_into_bufto make is entirely safe?
No, that is not sufficient, although it would also solve the issue.
Quoted 2 lines
(Perhaps even with every
request_firmwarefunctions as none seems to be tested properly.)
No, you are incorrect. The other firmware API calls *have* been elaborately tested. The firmware cache stuff *is a mess* however, since we *use and support it*, I've done my best to salvage it and document it.
I'll take a look at this and propose an alternative solution.
Luis
Re: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
Hi Luis,
I'm glad you are a subject expert in this area.
Some more comments inline.
Quoted 7 lines · Luis Chamberlain
Quoted 1 line · Scott Branden
To be honest, I find the entire firmware code sloppy.
And that is after years of cleanup on my part. Try going back to v4.1 for instance, check the code out then for an incredible horrific sight :)
Quoted 2 lines
I don't think the cache/no-cache feature is implemented or tested properly nor fallback to begin with.
Quoted 10 lines
I'm in total agreement! I *know* there must be holes in that code, and I acknowledge a few possible gotchas on the commit logs. For instance, I acknowledged that the firmware cache had a secondary purpose which was not well documented or understood through commit
e44565f62a720("firmware: fix batched requests - wake all waiters"). The firmware cache allows for batching requests and sharing the same original request for multiple consecutive requests which *race against each other*. That's when I started having my doubts about the architecture of the firmware cache mechanism, it seemed too complex and perhaps overkill and considered killing it.
Great (kill it!). I have no need for cached or batched requests.
The would remove a lot of problems.
Quoted 33 lines
As I noted in that commit, the firmware cache is used for:
1) Addressing races with file lookups during the suspend/resume cycle by keeping firmware in memory during the suspend/resume cycle
2) Batched requests for the same file rely only on work from the first file lookup, which keeps the firmware in memory until the last
release_firmware()is calledAlso worth quoting from that commit as well:
"Batched requests *only* take effect if secondary requests come in prior to the first user calling
release_firmware(). The devres name used for the internal firmware cache is used as a hint other pending requests are ongoing, the firmware buffer data is kept in memory until the last user of the buffer callsrelease_firmware(), therefore serializing requests and delaying the release until all requests are done."Later we discovered that the firmware cache had a serious security issue since its inception through commit
422b3db2a503("firmware: Fix security issue withrequest_firmware_into_buf()"). Granted, exploiting this would require the ability to load kernel code, so the vector of exploitation is rather small.The cache stuff cannot be removed as it *at least* resolves the fw suspend stuff, but still, this can likely use a revisit in rachitecture long term. The second implicit use case for batched requests however seems complex and not sure if its worth to maintain. I'll note that at least some drivers *do* their own firmware caching, iwlwifi, is one, so there is an example there to allow drivers to say "I actually don't need caching" for the future.
Quoted 2 lines
If you're volunteering to cleaning / testing the cache stuff I highly welcome that.
I would only volunteer to remove it, not test or support it.
Quoted 3 lines
That and the fallback stuff has been needing testing for
years. Someoone was working on patches on the test case for cache stuff a while ago, from Intel, but they disappeared.
Again, I would only volunteer to remove the fallback mechanism to remove added race conditions.
Quoted 14 lines
Quoted 2 lines
I'm not claiming this patch is the final
solution and indicated such in the cover letter and the comment above.I missed that sorry.
Quoted 2 lines
I hope there is someone more familiar with this code to comment further and come up with a proper solution.
Alright, I'll dig in and take a look, and propose an alternative.
Quoted 2 lines
I have found numerous issues and race conditions with the firmware code (I simply added a test).
That is nothing compared to the amount of fixes I have found and actually fixed too, the code was a nightmare before I took on maintenance.
Quoted 7 lines
Quoted 2 lines
1) Try loading the same valid firmware using no-cache once it has already been loaded with cache.
:)
Quoted 2 lines
It won't work, which is why I had to use a different filename in the test for
request_firmware_into_buf.Alright, I'll go try to fix this. Thanks for the report.
I think it's a minor issue compared to the race conditions present.
In reality I don't think anyone will load the same firmware using cache vs.
no-cache.
It's just something I stumbled upon when adding the test case and then had to avoid.
Quoted 9 lines
Quoted 8 lines
2) Try removing the "if (
opt_flags&FW_OPT_NOCACHE)" in my patch and always call the mutex.The firmware test will lock up during a "no uevent" test. I am not familiar with the code to
know why such is true and what issue this exposes in the code.
I hinted in my review of the oops what the issue was.
I don't know if it's the same bug for the "no uevent" test case though? The test
just hangs and the kernel oops is not present. It might be exposing another
underlying issue with the request_firmware code.
Quoted 4 lines
Quoted 3 lines
3) I have a driver that uses
request_firmware_into_bufand have multiple instances of the driverCool, is the driver upstream?
I'm working on cleaning up the driver right now to upstream.
First thing is I need the request_firmware_into_buf tests accepted upstream.
Then I can add my enhancement to request_firmware_into_buf to partial read the file (previous sent out but needed test case).
In order to do so Greg K-H required a test case for this but even the current API had no test.
In that patch series I can then add the new driver which requires my enhanced request_firmware_into_buf.
Quoted 7 lines
Quoted 3 lines
loading the same firmware in parallel. Some of the data is not read correctly in each instance.
Makes perfect sense considering the lack of testing I noted.
Quoted 1 line
I haven't yet to reproduce this issue with the firmware test
That's because of batched firmware request mechanism.
Is there a way to not use the batch firmware request mechanism when calling request_firmware_into_buf
to see if the problem doesn't happen?
Quoted 10 lines
Quoted 4 lines
but currently
have a mutex around the entire
call torequest_firmware_into_bufin our driver.I will take a look at this now.
Quoted 2 lines
Perhaps it is better at this point to add a mutex in
request_firmware_into_bufto make is entirely safe?No, that is not sufficient, although it would also solve the issue.
I don't have another solution with all the other mechanisms in
play in the current firmware code. For now I'll leave the mutex
in the driver I'm upstreaming so it works reliably.
Quoted 7 lines
Quoted 3 lines
(Perhaps even with every
request_firmwarefunctions as none seems to be tested properly.)No, you are incorrect. The other firmware API calls *have* been elaborately tested. The firmware cache stuff *is a mess* however, since we *use and support it*, I've done my best to salvage it and document it.
OK, I don't use any of the other mechanisms right now.
All I require is request_firmware_into_buf.
Quoted 4 lines
I'll take a look at this and propose an alternative solution.
Luis
Re: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
In reply to Luis ChamberlainQuoted 21 lines · Luis Chamberlain
Quoted 1 line · Scott Branden
To be honest, I find the entire firmware code sloppy.
And that is after years of cleanup on my part. Try going back to v4.1 for instance, check the code out then for an incredible horrific sight :)
Quoted 2 lines
I don't think the cache/no-cache feature is implemented or tested properly nor fallback to begin with.
I'm in total agreement! I *know* there must be holes in that code, and I acknowledge a few possible gotchas on the commit logs. For instance, I acknowledged that the firmware cache had a secondary purpose which was not well documented or understood through commit
e44565f62a720("firmware: fix batched requests - wake all waiters"). The firmware cache allows for batching requests and sharing the same original request for multiple consecutive requests which *race against each other*. That's when I started having my doubts about the architecture of the firmware cache mechanism, it seemed too complex and perhaps overkill and considered killing it.
Quoted 4 lines
As I noted in that commit, the firmware cache is used for:
1) Addressing races with file lookups during the suspend/resume cycle by keeping firmware in memory during the suspend/resume cycle
Right, this one is the significant need. And currently the fw loader core takes a complicated approach as:
- Store firmware name string in devres for each firmware
- Upon suspend, loop over all devices and associated firmware names, create a list, then loop over the list for loading the firmware files before sleeping.
- Upon resume, release the firmware files that have been loaded at suspend in a delayed manner.
So we have different level of lists there, which make the code quite hard to understand.
The reason of the above approach is because we didn't know which device driver would need the firmware at resume, so basically we do cache for all devices. Maybe it'd better to look for the exact drivers that require the firmware at resume, and handle only such ones instead of catch-all approach.
OTOH, I find it's not bad to keep the loaded firmware file names per device and expose e.g. via sysfs. Currently we have no way to look at which firmware files have been loaded afterwards; the only way to see it is enabling some debug option and read through kernel messages. (FWIW, I stumbled on this problem since I wanted to provide the split
kernel-firmware package on SUSE distro, and let the installer decide which package to pick up.)
Quoted 3 lines
2) Batched requests for the same file rely only on work from the first file lookup, which keeps the firmware in memory until the last
release_firmware()is called
IMO, this feature can be omitted if it makes things too complicated. I guess it were added because we handle the fw caching in anyway. There isn't a big need for this due to performance. If the performance matters, such driver should re-use its own firmware by itself.
(snip)
Quoted 18 lines
Quoted 2 lines
3) I have a driver that uses
request_firmware_into_bufand have multiple instances of the driverCool, is the driver upstream?
Quoted 2 lines
loading the same firmware in parallel. Some of the data is not read correctly in each instance.
Makes perfect sense considering the lack of testing I noted.
Quoted 1 line
I haven't yet to reproduce this issue with the firmware test
That's because of batched firmware request mechanism.
Quoted 3 lines
but currently
have a mutex around the entire
call torequest_firmware_into_bufin our driver.
Quoted 7 lines
I will take a look at this now.
Quoted 2 lines
Perhaps it is better at this point to add a mutex in
request_firmware_into_bufto make is entirely safe?No, that is not sufficient, although it would also solve the issue.
The mutex for request_firmware_into_buf() doesn't sound like a good approach. Basically the direct fw loading should work in parallel for the same firmware file. We might have some bug wrt cache stuff, but it can be fixed properly.
However, the fw loading in fallback mode can't run in parallel for the same file, per design -- no matter whether cached or not. So, if any, we'd need put a mutex around the fallback loader code. And, the mutex should be rather per device, not a global one.
Or we may trick it by appending the second parallel caller into the same wait queue, but the code will be more complex, so I don't think worth for it.
thanks,
Takashi
Luis Chamberlain
maintainerRe: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
Quoted 6 lines · Takashi Iwai
So, if any, we'd need put a mutex around the fallback loader code. And, the mutex should be rather per device, not a global one.
Or we may trick it by appending the second parallel caller into the same wait queue, but the code will be more complex, so I don't think worth for it.
For now I'm thinking of a new API with a devname prefix to the driver. I'll have to test if that works, but not sure if I'll get to it today before my vacation starts (today).
Luis
Re: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
Quoted 10 lines · Luis Chamberlain
Quoted 6 lines · Takashi Iwai
So, if any, we'd need put a mutex around the fallback loader code. And, the mutex should be rather per device, not a global one.
Or we may trick it by appending the second parallel caller into the same wait queue, but the code will be more complex, so I don't think worth for it.
For now I'm thinking of a new API with a devname prefix to the driver. I'll have to test if that works, but not sure if I'll get to it today before my vacation starts (today).
Have a good vacation.
Quoted 2 lines
Luis
Re: [PATCH 3/3] firmware: add mutex fw_lock_fallback for race condition
In reply to Takashi IwaiQuoted 34 lines · Takashi Iwai
Quoted 18 lines · Luis Chamberlain
Quoted 1 line · Scott Branden
To be honest, I find the entire firmware code sloppy.
And that is after years of cleanup on my part. Try going back to v4.1 for instance, check the code out then for an incredible horrific sight :)
Quoted 2 lines
I don't think the cache/no-cache feature is implemented or tested properly nor fallback to begin with.
I'm in total agreement! I *know* there must be holes in that code, and I acknowledge a few possible gotchas on the commit logs. For instance, I acknowledged that the firmware cache had a secondary purpose which was not well documented or understood through commit
e44565f62a720("firmware: fix batched requests - wake all waiters"). The firmware cache allows for batching requests and sharing the same original request for multiple consecutive requests which *race against each other*. That's when I started having my doubts about the architecture of the firmware cache mechanism, it seemed too complex and perhaps overkill and considered killing it.Quoted 4 lines
As I noted in that commit, the firmware cache is used for:
1) Addressing races with file lookups during the suspend/resume cycle by keeping firmware in memory during the suspend/resume cycle
Right, this one is the significant need. And currently the fw loader core takes a complicated approach as:
- Store firmware name string in devres for each firmware
- Upon suspend, loop over all devices and associated firmware names, create a list, then loop over the list for loading the firmware files before sleeping.
- Upon resume, release the firmware files that have been loaded at suspend in a delayed manner.
Quoted 8 lines
So we have different level of lists there, which make the code quite hard to understand.
The reason of the above approach is because we didn't know which device driver would need the firmware at resume, so basically we do cache for all devices. Maybe it'd better to look for the exact drivers that require the firmware at resume, and handle only such ones instead of catch-all approach.
Yes, that would be better. Or remove this cache mechanism entirely
and provide some helper functions of some sort to the limited
drivers that actually require such mechanism.
Quoted 9 lines
OTOH, I find it's not bad to keep the loaded firmware file names per device and expose e.g. via sysfs. Currently we have no way to look at which firmware files have been loaded afterwards; the only way to see it is enabling some debug option and read through kernel messages. (FWIW, I stumbled on this problem since I wanted to provide the split
kernel-firmware package on SUSE distro, and let the installer decide which package to pick up.)
Quoted 8 lines
Quoted 3 lines
2) Batched requests for the same file rely only on work from the first file lookup, which keeps the firmware in memory until the last
release_firmware()is calledIMO, this feature can be omitted if it makes things too complicated. I guess it were added because we handle the fw caching in anyway. There isn't a big need for this due to performance. If the performance matters, such driver should re-use its own firmware by itself.
Any simplifications would be appreciated.
I sure don't understand what the code is trying to do.
Quoted 27 lines
(snip)
Quoted 14 lines
Quoted 2 lines
3) I have a driver that uses
request_firmware_into_bufand have multiple instances of the driverCool, is the driver upstream?
Quoted 2 lines
loading the same firmware in parallel. Some of the data is not read correctly in each instance.
Makes perfect sense considering the lack of testing I noted.
Quoted 1 line
I haven't yet to reproduce this issue with the firmware test
That's because of batched firmware request mechanism.
Quoted 3 lines
but currently
have a mutex around the entire
call torequest_firmware_into_bufin our driver.Quoted 6 lines
I will take a look at this now.
Quoted 2 lines
Perhaps it is better at this point to add a mutex in
request_firmware_into_bufto make is entirely safe?No, that is not sufficient, although it would also solve the issue.
The mutex for
request_firmware_into_buf()doesn't sound like a good approach. Basically the direct fw loading should work in parallel for the same firmware file. We might have some bug wrt cache stuff, but it can be fixed properly.
Quoted 4 lines
However, the fw loading in fallback mode can't run in parallel for the same file, per design -- no matter whether cached or not. So, if any, we'd need put a mutex around the fallback loader code. And, the mutex should be rather per device, not a global one.
Sure, whatever solves the issue. All I wish to do is read
part of file into a buffer specified.
Quoted 9 lines
Or we may trick it by appending the second parallel caller into the same wait queue, but the code will be more complex, so I don't think worth for it.
thanks,
Takashi