firmware: Correct handling of fw_state_wait_timeout() return value
Subsystems (4)
PATCH v2 1/1 firmware: Correct handling of fw_state_wait_timeout() return value
Commit 5d47ec02c37e ("firmware: Correct handling of fw_state_wait() return value") made the assumption that any error returned from fw_state_wait_timeout() means FW load has to be aborted. This is incorrect, FW load only has to be aborted when load timed out or has been interrupted. Otherwise, if wait exited because of a wake up, the waking thread (in firmware_loading_store()) had already performed the clean up - deleted fw_buf from the pending list and set fw_priv->buf to NULL.
Example NULL dereference which occurs because requsted firmware could not be found by UMH:
nfp 0000:02:00.0: Direct firmware load for AMDA0081-0001.cat failed with error -2 nfp 0000:02:00.0: Falling back to user helper
BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 IP: _request_firmware+0x7fd/0x910
PGD 0
Oops: 0000 [#1] PREEMPT SMP CPU: 4 PID: 1981 Comm: insmod Tainted: G O 4.10.0-rc3-perf-00404-g575a284323c2 #78 Hardware name: Dell Inc. PowerEdge T630/0W9WXC, BIOS 1.2.10 03/10/2015 task: ffff919b81a99a80 task.stack: ffffb3bac5668000 RIP: 0010:_request_firmware+0x7fd/0x910 RSP: 0018:ffffb3bac566b978 EFLAGS: 00010246 RAX: 00000000fffffffe RBX: ffff919b9b2becc0 RCX: ffff919b9b2bece8 RDX: ffff919b81a99a80 RSI: 0000000000000206 RDI: 0000000000000000 RBP: ffffb3bac566b9e0 R08: ffff919b9f30f4b8 R09: 0000000000000002 R10: 00000120d2b47b48 R11: 0000000000000000 R12: ffffb3bac566ba18 R13: ffff919b823f4780 R14: ffff919b907c5000 R15: 0000000000003a98 FS: 00007fae4d803740(0000) GS:ffff919b9f300000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000038 CR3: 00000008507b8000 CR4: 00000000001406e0 Call Trace: request_firmware+0x37/0x50
...
---
v2:
- improve commit message.
--- drivers/base/firmware_class.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/base/firmware_class.c +1 -1
@@ -1020,7 +1020,7 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, } retval =fw_state_wait_timeout (&buf->fw_st, timeout); -if (retval <0 ) { +if (retval == -ETIMEDOUT || retval == -ERESTARTSYS ) {mutex_lock (&fw_lock);fw_load_abort (fw_priv);mutex_unlock (&fw_lock);
Signature · 1 line
2.11.0
Quoted 50 lines · Jakub Kicinski
Commit
5d47ec02c37e("firmware: Correct handling offw_state_wait()return value") made the assumption that any error returned fromfw_state_wait_timeout()means FW load has to be aborted. This is incorrect, FW load only has to be aborted when load timed out or has been interrupted. Otherwise, if wait exited because of a wake up, the waking thread (infirmware_loading_store()) had already performed the clean up - deletedfw_buffrom the pending list and setfw_priv->bufto NULL.Example NULL dereference which occurs because requsted firmware could not be found by UMH:
nfp 0000:02:00.0: Direct firmware load for AMDA0081-0001.cat failed with error -2 nfp 0000:02:00.0: Falling back to user helper
BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 IP: _request_firmware+0x7fd/0x910PGD 0
Oops: 0000 [#1] PREEMPT SMP CPU: 4 PID: 1981 Comm: insmod Tainted: G O 4.10.0-rc3-perf-00404-g575a284323c2 #78 Hardware name: Dell Inc. PowerEdge T630/0W9WXC, BIOS 1.2.10 03/10/2015 task: ffff919b81a99a80 task.stack: ffffb3bac5668000 RIP: 0010:_request_firmware+0x7fd/0x910 RSP: 0018:ffffb3bac566b978 EFLAGS: 00010246 RAX: 00000000fffffffe RBX: ffff919b9b2becc0 RCX: ffff919b9b2bece8 RDX: ffff919b81a99a80 RSI: 0000000000000206 RDI: 0000000000000000 RBP: ffffb3bac566b9e0 R08: ffff919b9f30f4b8 R09: 0000000000000002 R10: 00000120d2b47b48 R11: 0000000000000000 R12: ffffb3bac566ba18 R13: ffff919b823f4780 R14: ffff919b907c5000 R15: 0000000000003a98 FS: 00007fae4d803740(0000) GS:ffff919b9f300000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000038 CR3: 00000008507b8000 CR4: 00000000001406e0 Call Trace: request_firmware+0x37/0x50...
Fixes 5d47ec02c37e ("firmware: Correct handling of fw_state_wait() return value")Signed-off-by Jakub Kicinski <[email protected]>---
v2:
- improve commit message.--- drivers/base/firmware_class.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/base/firmware_class.c +0
-0
drivers/base/firmware_class.c static int _request_firmware_load(struct firmware_priv *fw_priv, · line 1026
Quoted 8 lines
drivers/base/firmware_class.c +1
-1@@ -1020,7 +1020,7 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, } retval =fw_state_wait_timeout (&buf->fw_st, timeout); -if (retval <0 ) { +if (retval == -ETIMEDOUT || retval == -ERESTARTSYS ) {mutex_lock (&fw_lock);fw_load_abort (fw_priv);mutex_unlock (&fw_lock);
This is a bit messy, two other similar issues were reported before and upon review I suggested Patrick Bruenn's fix with a better commit log seems best fit. Patrick sent a patch Jan 4, 2017 but never followed up despite my feedback on a small change on the commit log message [0]. Can you try that and if that fixes it can you adjust the commit log accordingly? Please note the preferred solution would be:
drivers/base/firmware_class.c +2 -0
@@ -542,6 +542,8 @@ static struct firmware_priv *to_firmware_priv(struct device *dev)static void __fw_load_abort (struct firmware_buf *buf) { +if (!buf) +return ;/* * There is a small window in which user can write to 'loading' * between loading done and disappearance of 'loading'
[0] https://http://lkml.kernel.org/r/[email protected]
Luis
Quoted 81 lines · Luis R. Rodriguez
Quoted 50 lines · Jakub Kicinski
Commit
5d47ec02c37e("firmware: Correct handling offw_state_wait()return value") made the assumption that any error returned fromfw_state_wait_timeout()means FW load has to be aborted. This is incorrect, FW load only has to be aborted when load timed out or has been interrupted. Otherwise, if wait exited because of a wake up, the waking thread (infirmware_loading_store()) had already performed the clean up - deletedfw_buffrom the pending list and setfw_priv->bufto NULL.Example NULL dereference which occurs because requsted firmware could not be found by UMH:
nfp 0000:02:00.0: Direct firmware load for AMDA0081-0001.cat failed with error -2 nfp 0000:02:00.0: Falling back to user helper
BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 IP: _request_firmware+0x7fd/0x910PGD 0
Oops: 0000 [#1] PREEMPT SMP CPU: 4 PID: 1981 Comm: insmod Tainted: G O 4.10.0-rc3-perf-00404-g575a284323c2 #78 Hardware name: Dell Inc. PowerEdge T630/0W9WXC, BIOS 1.2.10 03/10/2015 task: ffff919b81a99a80 task.stack: ffffb3bac5668000 RIP: 0010:_request_firmware+0x7fd/0x910 RSP: 0018:ffffb3bac566b978 EFLAGS: 00010246 RAX: 00000000fffffffe RBX: ffff919b9b2becc0 RCX: ffff919b9b2bece8 RDX: ffff919b81a99a80 RSI: 0000000000000206 RDI: 0000000000000000 RBP: ffffb3bac566b9e0 R08: ffff919b9f30f4b8 R09: 0000000000000002 R10: 00000120d2b47b48 R11: 0000000000000000 R12: ffffb3bac566ba18 R13: ffff919b823f4780 R14: ffff919b907c5000 R15: 0000000000003a98 FS: 00007fae4d803740(0000) GS:ffff919b9f300000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000038 CR3: 00000008507b8000 CR4: 00000000001406e0 Call Trace: request_firmware+0x37/0x50...
Fixes 5d47ec02c37e ("firmware: Correct handling of fw_state_wait() return value")Signed-off-by Jakub Kicinski <[email protected]>---
v2:
- improve commit message.--- drivers/base/firmware_class.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/base/firmware_class.c +0
-0drivers/base/firmware_class.c static int _request_firmware_load(struct firmware_priv *fw_priv, · line 1026
Quoted 8 lines
drivers/base/firmware_class.c +1
-1@@ -1020,7 +1020,7 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, } retval =fw_state_wait_timeout (&buf->fw_st, timeout); -if (retval <0 ) { +if (retval == -ETIMEDOUT || retval == -ERESTARTSYS ) {mutex_lock (&fw_lock);fw_load_abort (fw_priv);mutex_unlock (&fw_lock);This is a bit messy, two other similar issues were reported before and upon review I suggested Patrick Bruenn's fix with a better commit log seems best fit. Patrick sent a patch Jan 4, 2017 but never followed up despite my feedback on a small change on the commit log message [0]. Can you try that and if that fixes it can you adjust the commit log accordingly? Please note the preferred solution would be:
drivers/base/firmware_class.c +2
-0@@ -542,6 +542,8 @@ static struct firmware_priv *to_firmware_priv(struct device *dev)static void __fw_load_abort (struct firmware_buf *buf) { +if (!buf) +return ;/* * There is a small window in which user can write to 'loading' * between loading done and disappearance of 'loading'
Quoted 1 line
Actually Patrick did follow up with a new patch today but I just saw it was not as I recommended above. Patrick can you send a new one with your good commit log with the above change ? Can you both confirm it fixes the issue?
Luis
Quoted 21 lines · Luis R. Rodriguez
Quoted 21 lines
Review
Quoted 7 lines
retval = fw_state_wait_timeout(&buf->fw_st, timeout);(quoted diff) +1
-1- if (retval < 0) { + if (retval == -ETIMEDOUT || retval == -ERESTARTSYS) { mutex_lock(&fw_lock); fw_load_abort(fw_priv); mutex_unlock(&fw_lock);This is a bit messy, two other similar issues were reported before and upon review I suggested Patrick Bruenn's fix with a better commit log seems best fit. Patrick sent a patch Jan 4, 2017 but never followed up despite my feedback on a small change on the commit log message [0]. Can you try that and if that fixes it can you adjust the commit log accordingly? Please note the preferred solution would be:
drivers/base/firmware_class.c +0
-0@@ -542,6 +542,8 @@ static struct firmware_priv *to_firmware_priv(struct device *dev)
Quoted 4 lines
drivers/base/firmware_class.c static struct firmware_priv *to_firmware_priv(struct device *dev) · line 546
Quoted 4 lines
drivers/base/firmware_class.c +2
-0static void __fw_load_abort (struct firmware_buf *buf) { +if (!buf) +return ;
Allow me to try to persuade you one last time :) My patch makes the code more logical and easier to follow. The code says:
in case no wake up happened - finish the wait (otherwise the waking thread finishes it). Adding a NULL-check would just paper over the issue and can cause trouble down the line. If fw_state_wait_timeout() returned because someone woke it up - there is no reason to abort the wait. The wait is already finished. The buggy commit mixed up return codes from fw_state_wait_timeout() - mixed "nobody woke us up" with "we couldn't find the FW", that's why we need to check for specific error codes.
Quoted 22 lines · Jakub Kicinski
Quoted 21 lines · Luis R. Rodriguez
Quoted 21 lines
Review
Quoted 7 lines
retval = fw_state_wait_timeout(&buf->fw_st, timeout);(quoted diff) +1
-1- if (retval < 0) { + if (retval == -ETIMEDOUT || retval == -ERESTARTSYS) { mutex_lock(&fw_lock); fw_load_abort(fw_priv); mutex_unlock(&fw_lock);This is a bit messy, two other similar issues were reported before and upon review I suggested Patrick Bruenn's fix with a better commit log seems best fit. Patrick sent a patch Jan 4, 2017 but never followed up despite my feedback on a small change on the commit log message [0]. Can you try that and if that fixes it can you adjust the commit log accordingly? Please note the preferred solution would be:
drivers/base/firmware_class.c +0
-0@@ -542,6 +542,8 @@ static struct firmware_priv *to_firmware_priv(struct device *dev)
Quoted 9 lines
Quoted 4 lines
drivers/base/firmware_class.c static struct firmware_priv *to_firmware_priv(struct device *dev) · line 546
Quoted 4 lines
drivers/base/firmware_class.c +2
-0static void __fw_load_abort (struct firmware_buf *buf) { +if (!buf) +return ;Allow me to try to persuade you one last time :) My patch makes the code more logical and easier to follow. The code says:
in case no wake up happened - finish the wait (otherwise the waking thread finishes it).
Your patch is still wrong, as Patrick great commit log notes a null defer can also happen on a race with a case of -1 being sent and a -ENOENT error, so we'd have to adjust for when __fw_state_wait_common() returns also -ENOENT.
Quoted 2 lines
Adding a NULL-check would just paper over the
issue and can cause trouble down the line.
We typically bail on errors and use similar code to bail out, and we typically do these things. Here its no different. The *real* issue is the fact that we have a waiting timeout which can fail race against a user imposed error out on the sysfs interface. There is one catch:
We already lock with the big fw_lock and use this to be able to check for the status of the fw, so once aborted we technically should not have to abort again. A proper way to address then this would have been to check for the status of the fw prior to aborting again given we also lock on the big fw_lock. A problem with this though is the status is part of the buf which is set to NULL after we are done aborting.
Quoted 6 lines
If
fw_state_wait_timeout()
returned because someone woke it up - there is no reason to abort the wait. The wait is already finished. The buggy commit mixed up return codes fromfw_state_wait_timeout()- mixed "nobody woke us up" with "we couldn't find the FW", that's why we need to check for specific error codes.
Nope, sorry I still believe the check on buf is needed. The code not not ideal, but I welcome further changes to help shape it up, such as the changes Daniel has been helping with.
Patrick, if you can document what I mentioned about what I said about the catch in your commit log it could help further reviewers *why* we do it this way.
Luis
Quoted 33 lines · Luis R. Rodriguez
Quoted 22 lines · Jakub Kicinski
Quoted 21 lines · Luis R. Rodriguez
Quoted 21 lines
Review
Quoted 7 lines
retval = fw_state_wait_timeout(&buf->fw_st, timeout);(quoted diff) +1
-1- if (retval < 0) { + if (retval == -ETIMEDOUT || retval == -ERESTARTSYS) { mutex_lock(&fw_lock); fw_load_abort(fw_priv); mutex_unlock(&fw_lock);This is a bit messy, two other similar issues were reported before and upon review I suggested Patrick Bruenn's fix with a better commit log seems best fit. Patrick sent a patch Jan 4, 2017 but never followed up despite my feedback on a small change on the commit log message [0]. Can you try that and if that fixes it can you adjust the commit log accordingly? Please note the preferred solution would be:
drivers/base/firmware_class.c +0
-0@@ -542,6 +542,8 @@ static struct firmware_priv *to_firmware_priv(struct device *dev)Quoted 9 lines
Quoted 4 lines
drivers/base/firmware_class.c static struct firmware_priv *to_firmware_priv(struct device *dev) · line 546
Quoted 4 lines
drivers/base/firmware_class.c +2
-0static void __fw_load_abort (struct firmware_buf *buf) { +if (!buf) +return ;Allow me to try to persuade you one last time :) My patch makes the code more logical and easier to follow. The code says:
in case no wake up happened - finish the wait (otherwise the waking thread finishes it).
Quoted 4 lines
Your patch is still wrong, as Patrick great commit log notes a null defer can also happen on a race with a case of -1 being sent and a
-ENOENTerror, so we'd have to adjust for when__fw_state_wait_common()returns also-ENOENT.
Sorry, I don't follow. _Not_ calling abort on -ENOENT error is exactly what my patch does.
Quoted 8 lines
Quoted 2 lines
Adding a NULL-check would just paper over the
issue and can cause trouble down the line.We typically bail on errors and use similar code to bail out, and we typically do these things. Here its no different. The *real* issue is the fact that we have a waiting timeout which can fail race against a user imposed error out on the sysfs interface. There is one catch:
Quoted 6 lines
We already lock with the big
fw_lockand use this to be able to check for the status of the fw, so once aborted we technically should not have to abort again. A proper way to address then this would have been to check for the status of the fw prior to aborting again given we also lock on the bigfw_lock. A problem with this though is the status is part of the buf which is set to NULL after we are done aborting.
Yes, I've seen that too :\ This race seems to have been there prior to 4.9, though. I guess we could fix both issues with the NULL-check although I would prefer if we had both patches.
FWIW I think the NULL-check could be put in the existing conditional:
* There is a small window in which user can write to 'loading'
* between loading done and disappearance of 'loading' */
- if (fw_state_is_done(&buf->fw_st))
+ if (!buf || fw_state_is_done(&buf->fw_st))
return;
list_del_init(&buf->pending_list);Note that the comment above seems to be mentioning the race we're trying to solve.
Quoted 34 lines · Jakub Kicinski
Quoted 33 lines · Luis R. Rodriguez
Quoted 31 lines · Jakub Kicinski
Quoted 21 lines · Luis R. Rodriguez
Quoted 21 lines
Review
Quoted 7 lines
retval = fw_state_wait_timeout(&buf->fw_st, timeout);(quoted diff) +1
-1- if (retval < 0) { + if (retval == -ETIMEDOUT || retval == -ERESTARTSYS) { mutex_lock(&fw_lock); fw_load_abort(fw_priv); mutex_unlock(&fw_lock);This is a bit messy, two other similar issues were reported before and upon review I suggested Patrick Bruenn's fix with a better commit log seems best fit. Patrick sent a patch Jan 4, 2017 but never followed up despite my feedback on a small change on the commit log message [0]. Can you try that and if that fixes it can you adjust the commit log accordingly? Please note the preferred solution would be:
drivers/base/firmware_class.c +0
-0@@ -542,6 +542,8 @@ static struct firmware_priv *to_firmware_priv(struct device *dev)Quoted 4 lines
drivers/base/firmware_class.c static struct firmware_priv *to_firmware_priv(struct device *dev) · line 546
Quoted 4 lines
drivers/base/firmware_class.c +2
-0static void __fw_load_abort (struct firmware_buf *buf) { +if (!buf) +return ;Allow me to try to persuade you one last time :) My patch makes the code more logical and easier to follow. The code says:
in case no wake up happened - finish the wait (otherwise the waking thread finishes it).
Quoted 7 lines
Quoted 4 lines
Your patch is still wrong, as Patrick great commit log notes a null defer can also happen on a race with a case of -1 being sent and a
-ENOENTerror, so we'd have to adjust for when__fw_state_wait_common()returns also-ENOENT.Sorry, I don't follow. _Not_ calling abort on
-ENOENTerror is exactly what my patch does.
Yeah I see now what you mean. Your approach avoids the buf issue as well. Its still not addressing the real issue though, which is the chicken sloppy use of a status on the buf, which at one point gets set to NULL. This later practice makes it rather hard to make it correct to use a stateful check properly.
Quoted 30 lines
Quoted 8 lines
Quoted 2 lines
Adding a NULL-check would just paper over the
issue and can cause trouble down the line.We typically bail on errors and use similar code to bail out, and we typically do these things. Here its no different. The *real* issue is the fact that we have a waiting timeout which can fail race against a user imposed error out on the sysfs interface. There is one catch:
Quoted 6 lines
We already lock with the big
fw_lockand use this to be able to check for the status of the fw, so once aborted we technically should not have to abort again. A proper way to address then this would have been to check for the status of the fw prior to aborting again given we also lock on the bigfw_lock. A problem with this though is the status is part of the buf which is set to NULL after we are done aborting.Yes, I've seen that too :\ This race seems to have been there prior to 4.9, though. I guess we could fix both issues with the NULL-check although I would prefer if we had both patches.
FWIW I think the NULL-check could be put in the existing conditional:
* There is a small window in which user can write to 'loading'
* between loading done and disappearance of 'loading' */(quoted diff) +1
-1- if (fw_state_is_done(&buf->fw_st)) + if (!buf || fw_state_is_done(&buf->fw_st)) return; list_del_init(&buf->pending_list);
Quoted 2 lines
Note that the comment above seems to be mentioning the race we're trying to solve.
Right, I think another approach is to *enable* the state of the buf to be used to avoid further use on the sysfs iterface instead. Fortunately other sysfs interfaces already use fw_state_is_done() to bail out, so all that would be needed I think would be:
drivers/base/firmware_class.c +1 -4
@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv *fw_priv)struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);
@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device *dev,mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (!fw_buf ||fw_state_is_aborted (&fw_buf->fw_st)) goto out;switch (loading) {
Quoted 56 lines · Luis R. Rodriguez
Quoted 32 lines · Jakub Kicinski
Quoted 9 lines · Luis R. Rodriguez
Quoted 2 lines · Jakub Kicinski
Adding a NULL-check would just paper over the
issue and can cause trouble down the line.We typically bail on errors and use similar code to bail out, and we typically do these things. Here its no different. The *real* issue is the fact that we have a waiting timeout which can fail race against a user imposed error out on the sysfs interface. There is one catch:
Quoted 6 lines
We already lock with the big
fw_lockand use this to be able to check for the status of the fw, so once aborted we technically should not have to abort again. A proper way to address then this would have been to check for the status of the fw prior to aborting again given we also lock on the bigfw_lock. A problem with this though is the status is part of the buf which is set to NULL after we are done aborting.Yes, I've seen that too :\ This race seems to have been there prior to 4.9, though. I guess we could fix both issues with the NULL-check although I would prefer if we had both patches.
FWIW I think the NULL-check could be put in the existing conditional:
* There is a small window in which user can write to 'loading'
* between loading done and disappearance of 'loading' */(quoted diff) +1
-1- if (fw_state_is_done(&buf->fw_st)) + if (!buf || fw_state_is_done(&buf->fw_st)) return; list_del_init(&buf->pending_list);Quoted 2 lines
Note that the comment above seems to be mentioning the race we're trying to solve.
Right, I think another approach is to *enable* the state of the buf to be used to avoid further use on the sysfs iterface instead. Fortunately other sysfs interfaces already use
fw_state_is_done()to bail out, so all that would be needed I think would be:drivers/base/firmware_class.c +0
-3@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv *fw_priv)struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);
drivers/base/firmware_class.c static ssize_t firmware_loading_store(struct device *dev, · line 716
Quoted 8 lines
drivers/base/firmware_class.c +1
-1@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device *dev,mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (!fw_buf ||fw_state_is_aborted (&fw_buf->fw_st)) goto out;switch (loading) {
IMHO this one is nice! I think you can even drop the !fw_buf check in this case because AFAICS the only case where fw_buf is set to NULL is in the abort function.
I was initially thinking that this could be a slight change of behavior - note that if mapping pages failed the abort state is entered with fw_state_aborted() which does not unlink the buffer so in theory one could still restart the FW load by writing 0 to sysfs and retrying? But it would have to be done before the waiting thread gets woken so it's really a race condition rather then something user space can depend on. Or at least that's the case if I'm reading the code correctly.
Yet another way to change the condition in firmware_loading_store() would be to check if fw_priv->buf->pending_list is still hooked onto the pending_fw_head list.
Quoted 72 lines
From: Jakub Kicinski [mailto:[email protected]]
Sent: Dienstag, 17. Januar 2017 22:18Quoted 2 lines · Luis R. Rodriguez
Quoted 1 line · Jakub Kicinski
On Tue, Jan 17, 2017 at 9:30 AM, Luis R. Rodriguez <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines
Adding a NULL-check would just paper over the
issue and can cause trouble down the line.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We typically bail on errors and use similar code to bail out, and we typically do these things. Here its no different. The *real* issue is the fact that we have a waiting timeout which can fail race against a user imposed error out on the sysfs interface. There is one catch:
We already lock with the big
fw_lockand use this to be able to check for the status of the fw, so once aborted we technically should not have to abort again. A proper way to address then this would have been tocheck
drivers/base/firmware_class.c static void fw_load_abort(struct firmware_priv
Quoted 32 lines
Quoted 19 lines
Quoted 3 lines
for the status of the fw prior to aborting again given we also lock on the big
fw_lock. A problem with this though is the status is part of the buf which is set to NULL after we are done aborting.Yes, I've seen that too :\ This race seems to have been there prior to 4.9, though. I guess we could fix both issues with the NULL-check although I would prefer if we had both patches.
FWIW I think the NULL-check could be put in the existing conditional:
* There is a small window in which user can write to 'loading'
* between loading done and disappearance of 'loading' */(quoted diff) +1
-1- if (fw_state_is_done(&buf->fw_st)) + if (!buf || fw_state_is_done(&buf->fw_st)) return; list_del_init(&buf->pending_list);Quoted 2 lines
Note that the comment above seems to be mentioning the race we're trying to solve.
Right, I think another approach is to *enable* the state of the buf to be used to avoid further use on the sysfs iterface instead. Fortunately other sysfs interfaces already use
fw_state_is_done()to bail out, so all that would be needed I think would be:drivers/base/firmware_class.c +0
-0@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv
*fw_priv)drivers/base/firmware_class.c static ssize_t firmware_loading_store(struct device
Quoted 10 lines
drivers/base/firmware_class.c +0
-3struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device*dev,
Quoted 7 lines
drivers/base/firmware_class.c +1
-1mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (!fw_buf ||fw_state_is_aborted (&fw_buf->fw_st)) goto out;
Quoted 6 lines
drivers/base/firmware_class.c static ssize_t firmware_loading_store(struct device *dev, · line 716
Quoted 1 line
drivers/base/firmware_class.c +0
-0switch (loading) {IMHO this one is nice! I think you can even drop the !fw_buf check in this case because AFAICS the only case where
fw_bufis set to NULL is in the abort function.
I can confirm, that patch looks nice and is working for my setup, even without the !fw_buf. Feel free to grab everything you need from my commit log, if it helps. Unfortunately there is a crazy spam filter between us, so you can't rely on me.
Quoted 74 lines · linux-kernel-dev
Quoted 74 lines
From: Jakub Kicinski [mailto:[email protected]]
Sent: Dienstag, 17. Januar 2017 22:18Quoted 2 lines · Luis R. Rodriguez
Quoted 1 line · Jakub Kicinski
On Tue, Jan 17, 2017 at 9:30 AM, Luis R. Rodriguez <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Jakub Kicinski
Quoted 2 lines
Adding a NULL-check would just paper over the
issue and can cause trouble down the line.Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
We typically bail on errors and use similar code to bail out, and we typically do these things. Here its no different. The *real* issue is the fact that we have a waiting timeout which can fail race against a user imposed error out on the sysfs interface. There is one catch:
We already lock with the big
fw_lockand use this to be able to check for the status of the fw, so once aborted we technically should not have to abort again. A proper way to address then this would have been tocheck
drivers/base/firmware_class.c static void fw_load_abort(struct firmware_priv
Quoted 32 lines
Quoted 19 lines
Quoted 3 lines
for the status of the fw prior to aborting again given we also lock on the big
fw_lock. A problem with this though is the status is part of the buf which is set to NULL after we are done aborting.Yes, I've seen that too :\ This race seems to have been there prior to 4.9, though. I guess we could fix both issues with the NULL-check although I would prefer if we had both patches.
FWIW I think the NULL-check could be put in the existing conditional:
* There is a small window in which user can write to 'loading'
* between loading done and disappearance of 'loading' */(quoted diff) +1
-1- if (fw_state_is_done(&buf->fw_st)) + if (!buf || fw_state_is_done(&buf->fw_st)) return; list_del_init(&buf->pending_list);Quoted 2 lines
Note that the comment above seems to be mentioning the race we're trying to solve.
Right, I think another approach is to *enable* the state of the buf to be used to avoid further use on the sysfs iterface instead. Fortunately other sysfs interfaces already use
fw_state_is_done()to bail out, so all that would be needed I think would be:drivers/base/firmware_class.c +0
-0@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv
*fw_priv)drivers/base/firmware_class.c static ssize_t firmware_loading_store(struct device
Quoted 10 lines
drivers/base/firmware_class.c +0
-3struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device*dev,
Quoted 8 lines
drivers/base/firmware_class.c +1
-1mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (!fw_buf ||fw_state_is_aborted (&fw_buf->fw_st)) goto out;switch (loading) {
Quoted 7 lines
Quoted 4 lines
IMHO this one is nice! I think you can even drop the !fw_buf check in this case because AFAICS the only case where
fw_bufis set to NULL is in the abort function.I can confirm, that patch looks nice and is working for my setup, even without the !fw_buf. Feel free to grab everything you need from my commit log, if it helps. Unfortunately there is a crazy spam filter between us, so you can't rely on me.
OK I'll submit this version with both your Reported-and-Tested-by.
Luis
cover firmware: expand test units for fallback mechanism
A kernel crash has been reported by a few folks on cancelling the firmware fallback mechanism introduced by some new code changes in v4.10. Testing this is not easy as most distributions disable the option to always use the fallback mechanism by default (CONFIG_FW_LOADER_USER_HELPER_FALLBACK), and our test driver for the firmware does not test against the fallback mechanism.
If CONFIG_FW_LOADER_USER_HELPER_FALLBACK is disabled there are only two ways in which the kernel will use the fallback mechanism, on two drivers:
o drivers/firmware/dell_rbu.c
o drivers/leds/leds-lp55xx-common.c
We need a way to easily test the fallback mechanism then even if distributions disable CONFIG_FW_LOADER_USER_HELPER_FALLBACK. This series expands the test_firmware driver to always have a knob to trigger a request which always requests the fallback mechanism, so long as CONFIG_FW_LOADER_USER_HELPER=y is enabled (most distributions), this trigger should help test the fallback mechanism.
This series also fixes the issue reported. I've tested applying the test unit changes onto v4.9.5 and confirm the issue is not present there. The crash happens after commit 5d47ec02c37ea6 ("firmware: Correct handling of fw_state_wait() return value") but this is a fix for commit 5b029624948d ("firmware: do not use fw_lock for fw_state protection") which is a functional change introduced only on v4.10.
Moving forward I will ask all parties to use all test scripts before applying any changes. If any APIs are changed we need a respective test unit for it.
Luis R. Rodriguez (7):
test_firmware: move misc_device down
test_firmware: use device attribute groups
tools: firmware: check for distro fallback udev cancel rule
tools: firmware: rename fallback mechanism script
tools: firmware: add fallback cancelation testing
test_firmware: add test custom fallback trigger
firmware: firmware: fix NULL pointer dereference in __fw_load_abort()
drivers/base/firmware_class.c | 5 +- lib/test_firmware.c | 92 ++++++--- tools/testing/selftests/firmware/Makefile | 2 +- tools/testing/selftests/firmware/fw_fallback.sh | 224 ++++++++++++++++++++++ tools/testing/selftests/firmware/fw_userhelper.sh | 99 ---------- 5 files changed, 288 insertions(+), 134 deletions(-) create mode 100755 tools/testing/selftests/firmware/fw_fallback.sh delete mode 100755 tools/testing/selftests/firmware/fw_userhelper.sh
Signature · 1 line
2.11.0
PATCH 1/7 test_firmware: move misc_device down
This will make further changes easier to review.
--- lib/test_firmware.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
lib/test_firmware.c +6 -6
@@ -42,12 +42,6 @@ static const struct file_operations test_fw_fops = { .read = test_fw_misc_read, }; -static struct miscdevice test_fw_misc_device = { - .minor =MISC_DYNAMIC_MINOR , - .name ="test_firmware" , - .fops = &test_fw_fops, -}; -static ssize_t trigger_request_store (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)
@@ -132,6 +126,12 @@ static ssize_t trigger_async_request_store(struct device *dev, }static DEVICE_ATTR_WO (trigger_async_request); +static struct miscdevice test_fw_misc_device = { + .minor =MISC_DYNAMIC_MINOR , + .name ="test_firmware" , + .fops = &test_fw_fops, +}; +static int __init test_firmware_init (void ) {int rc;
Signature · 1 line
2.11.0
PATCH 2/7 test_firmware: use device attribute groups
In reply to Luis R. RodriguezThis simplifies init and exit.
--- lib/test_firmware.c | 35 +++++++++++------------------------ 1 file changed, 11 insertions(+), 24 deletions(-)
lib/test_firmware.c +11 -24
@@ -126,10 +126,21 @@ static ssize_t trigger_async_request_store(struct device *dev, }static DEVICE_ATTR_WO (trigger_async_request); +#define TEST_FW_DEV_ATTR (name) &dev_attr_##name.attr + +static struct attribute *test_dev_attrs[] = { +TEST_FW_DEV_ATTR (trigger_request), +TEST_FW_DEV_ATTR (trigger_async_request), +NULL , +}; + +ATTRIBUTE_GROUPS (test_dev); +static struct miscdevice test_fw_misc_device = { .minor =MISC_DYNAMIC_MINOR , .name ="test_firmware" , .fops = &test_fw_fops, + .groups = test_dev_groups, };static int __init test_firmware_init (void )
@@ -141,30 +152,10 @@ static int __init test_firmware_init(void)pr_err ("could not register misc device: %d\n" , rc);return rc; } - rc =device_create_file (test_fw_misc_device.this_device, - &dev_attr_trigger_request); -if (rc) { -pr_err ("could not create sysfs interface: %d\n" , rc); - goto dereg; - } - - rc =device_create_file (test_fw_misc_device.this_device, - &dev_attr_trigger_async_request); -if (rc) { -pr_err ("could not create async sysfs interface: %d\n" , rc); - goto remove_file; - }pr_warn ("interface ready\n" );return 0 ; - -remove_file: -device_remove_file (test_fw_misc_device.this_device, - &dev_attr_trigger_async_request); -dereg: -misc_deregister (&test_fw_misc_device); -return rc; }module_init (test_firmware_init);
@@ -172,10 +163,6 @@ module_init(test_firmware_init); static void __exit test_firmware_exit(void) {release_firmware (test_firmware); -device_remove_file (test_fw_misc_device.this_device, - &dev_attr_trigger_async_request); -device_remove_file (test_fw_misc_device.this_device, - &dev_attr_trigger_request);misc_deregister (&test_fw_misc_device);pr_warn ("removed interface\n" ); }
Signature · 1 line
2.11.0
PATCH 3/7 tools: firmware: check for distro fallback udev cancel rule
In reply to Luis R. RodriguezSome distributions (Debian, OpenSUSE) have a udev rule in place to cancel all fallback mechanism uevents immediately. This would obviously make it hard to test against the fallback mechanism test interface, so we need to check for this.
--- tools/testing/selftests/firmware/fw_userhelper.sh | 28 +++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
tools/testing/selftests/firmware/fw_userhelper.sh +26 -2
@@ -64,9 +64,33 @@ trap "test_finish" EXIT echo "ABCD0123" >"$FW" NAME=$(basename "$FW") +DEVPATH="$DIR"/"nope-$NAME"/loading + # Test failure when doing nothing (timeout works). -echo 1 >/sys/class/firmware/timeout -echo -n "$NAME" >"$DIR"/trigger_request +echo -n 2 >/sys/class/firmware/timeout +echo -n "nope-$NAME" >"$DIR"/trigger_request 2>/dev/null & + +# Give the kernel some time to load the loading file, must be less +# than the timeout above. +sleep 1 +if [ ! -f $DEVPATH ]; then + echo "$0: fallback mechanism immediately cancelled" + echo "" + echo "The file never appeared: $DEVPATH" + echo "" + echo "This might be a distribution udev rule setup by your distribution" + echo "to immediately cancel all fallback requests, this must be" + echo "removed before running these tests. To confirm look for" + echo "a firmware rule like /lib/udev/rules.d/50-firmware.rules" + echo "and see if you have something like this:" + echo "" + echo "SUBSYSTEM==\"firmware\", ACTION==\"add\", ATTR{loading}=\"-1\"" + echo "" + echo "If you do remove this file or comment out this line before" + echo "proceeding with these tests." + exit 1 +fi + if diff -q "$FW" /dev/test_firmware >/dev/null ; then echo "$0: firmware was not expected to match" >&2 exit 1
Signature · 1 line
2.11.0
PATCH 4/7 tools: firmware: rename fallback mechanism script
In reply to Luis R. RodriguezCalling it user mode helper only creates confusion.
---
tools/testing/selftests/firmware/Makefile | 2 +-
.../testing/selftests/firmware/{fw_userhelper.sh => fw_fallback.sh} | 5 +++--
2 files changed, 4 insertions(+), 3 deletions(-)
rename tools/testing/selftests/firmware/{fw_userhelper.sh => fw_fallback.sh} (96%)tools/testing/selftests/firmware/Makefile +1 -1
@@ -3,7 +3,7 @@ # No binaries, but make sure arg-less "make" doesn't trigger "run_tests" all: -TEST_PROGS := fw_filesystem.sh fw_userhelper.sh +TEST_PROGS := fw_filesystem.sh fw_fallback.sh include ../lib.mk
tools/testing/selftests/firmware/fw_fallback.sh renamed from tools/testing/selftests/firmware/fw_userhelper.sh +3 -2
@@ -1,5 +1,5 @@ #!/bin/sh -# This validates that the kernel will fall back to using the user helper +# This validates that the kernel will fall back to using the fallback mechanism # to load firmware it can't find on disk itself. We must request a firmware # that the kernel won't find, and any installed helper (e.g. udev) also # won't find so that we can do the load ourself manually.
@@ -117,7 +117,8 @@ if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then echo "$0: firmware was not loaded" >&2 exit 1 else - echo "$0: user helper firmware loading works" + echo "$0: fallback mechanism works" + fi exit 0
Signature · 1 line
2.11.0
PATCH 5/7 tools: firmware: add fallback cancelation testing
In reply to Luis R. RodriguezAdd a test case for cancelling the sync fallback mechanism.
--- tools/testing/selftests/firmware/fw_fallback.sh | 32 +++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
tools/testing/selftests/firmware/fw_fallback.sh +32 -0
@@ -58,6 +58,31 @@ load_fw() wait } +load_fw_cancel() +{ + local name="$1" + local file="$2" + + # This will block until our load (below) has finished. + echo -n "$name" >"$DIR"/trigger_request 2>/dev/null & + + # Give kernel a chance to react. + local timeout=10 + while [ ! -e "$DIR"/"$name"/loading ]; do + sleep 0.1 + timeout=$(( $timeout - 1 )) + if [ "$timeout" -eq 0 ]; then + echo "$0: firmware interface never appeared" >&2 + exit 1 + fi + done + + echo -1 >"$DIR"/"$name"/loading + + # Wait for request to finish. + wait +} + trap "test_finish" EXIT # This is an unlikely real-world firmware content. :)
@@ -118,7 +143,14 @@ if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then exit 1 else echo "$0: fallback mechanism works" +fi +load_fw_cancel "nope-$NAME" "$FW" +if diff -q "$FW" /dev/test_firmware >/dev/null ; then + echo "$0: firmware was expected to be cancelled" >&2 + exit 1 +else + echo "$0: cancelling fallback mechanism works" fi exit 0
Signature · 1 line
2.11.0
PATCH 6/7 test_firmware: add test custom fallback trigger
In reply to Luis R. RodriguezWe have no custom fallback mechanism test interface. Provide one. This tests both the custom fallback mechanism and cancelling the it.
--- lib/test_firmware.c | 45 ++++++++++++++++ tools/testing/selftests/firmware/fw_fallback.sh | 68 +++++++++++++++++++++++++ 2 files changed, 113 insertions(+)
lib/test_firmware.c +45 -0
@@ -126,11 +126,56 @@ static ssize_t trigger_async_request_store(struct device *dev, }static DEVICE_ATTR_WO (trigger_async_request); +static ssize_t trigger_custom_fallback_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +int rc; +char *name; + + name =kstrndup (buf, count,GFP_KERNEL ); +if (!name) +return -ENOSPC ; + +pr_info ("loading '%s' using custom fallback mechanism\n" , name); + +mutex_lock (&test_fw_mutex); +release_firmware (test_firmware); + test_firmware =NULL ; + rc =request_firmware_nowait (THIS_MODULE ,FW_ACTION_NOHOTPLUG , name, + dev,GFP_KERNEL ,NULL , + trigger_async_request_cb); +if (rc) { +pr_info ("async load of '%s' failed: %d\n" , name, rc); +kfree (name); + goto out; + } +/* Free 'name' ASAP, to test for race conditions */ +kfree (name); + +wait_for_completion (&async_fw_done); + +if (test_firmware) { +pr_info ("loaded: %zu\n" , test_firmware->size); + rc = count; + }else { +pr_err ("failed to async load firmware\n" ); + rc = -ENODEV ; + } + +out: +mutex_unlock (&test_fw_mutex); + +return rc; +} +static DEVICE_ATTR_WO (trigger_custom_fallback); +#define TEST_FW_DEV_ATTR (name) &dev_attr_##name.attrstatic struct attribute *test_dev_attrs[] = {TEST_FW_DEV_ATTR (trigger_request),TEST_FW_DEV_ATTR (trigger_async_request), +TEST_FW_DEV_ATTR (trigger_custom_fallback),NULL , };
tools/testing/selftests/firmware/fw_fallback.sh +68 -0
78 lines not on this page · raw message
Signature · 1 line
2.11.0
PATCH 7/7 firmware: firmware: fix NULL pointer dereference in __fw_load_abort()
In reply to Luis R. RodriguezSince commit 5d47ec02c37ea632398cb251c884e3a488dff794 ("firmware: Correct handling of fw_state_wait() return value") fw_load_abort(fw_priv) could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:
echo -n "-1" > /sys/$DEVPATH/loading
This will call fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian, OpenSUSE), they have:
$ cat /lib/udev/rules.d/50-firmware.rules
# stub for immediately telling the kernel that userspace firmware loading
# failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y
SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1This was done since udev removed the firmware fallback mechanism a while ago and a long standing misunderstood issues with the timeout (but now corrected). Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash: drivers/firmware/dell_rbu.c and the drivers/leds/leds-lp55xx-common.c driver.
Distributions enabling CONFIG_FW_LOADER_USER_HELPER_FALLBACK are clearly more exposed as every file not found through a firmware request will use the fallback mechanism.
The crash happens because after commit 5b029624948d ("firmware: do not use fw_lock for fw_state protection") and subsequent fix commit 5d47ec02c37ea6 ("firmware: Correct handling of fw_state_wait() return value") a race can happen between this cancelation and the firmware fw_state_wait_timeout() being woken up after a state change with which fw_load_abort() as that calls swake_up(). Upon error fw_state_wait_timeout() will also again call fw_load_abort() and trigger a null reference.
At first glance we could just fix this with a !buf check on fw_load_abort() before accessing buf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).
The firmware_class.c code is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.
An oops can be reproduced with the new fw_fallback.sh fallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.
mcgrof@piggy ~/linux-next/tools/testing/selftests/firmware (git::20170111-fw-fixes)$ sudo ./fw_fallback.sh
./fw_fallback.sh: timeout works
./fw_fallback.sh: firmware comparison works
./fw_fallback.sh: fallback mechanism works
[ this then sits here when it is trying the cancellation test ]
Kernel log:
[ 36.750521] test_firmware: loading 'nope-test-firmware.bin' [ 36.751144] misc test_firmware: Direct firmware load for nope-test-firmware.bin failed with error -2 [ 36.752034] misc test_firmware: Falling back to user helper [ 36.853324] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 36.854221] IP: _request_firmware+0xa27/0xad0 [ 36.854671] PGD 0 [ 36.854672] [ 36.855081] Oops: 0000 [#1] SMP [ 36.855433] Modules linked in: test_firmware(E) ... etc ... [ 36.857802] CPU: 1 PID: 1396 Comm: fw_fallback.sh Tainted: G W E 4.10.0-rc3-next-20170111+ #30 [ 36.857802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014 [ 36.857802] task: ffff9740b27f4340 task.stack: ffffbb15c0bc8000 [ 36.857802] RIP: 0010:_request_firmware+0xa27/0xad0 [ 36.857802] RSP: 0018:ffffbb15c0bcbd10 EFLAGS: 00010246 [ 36.857802] RAX: 00000000fffffffe RBX: ffff9740afe5aa80 RCX: 0000000000000000 [ 36.857802] RDX: ffff9740b27f4340 RSI: 0000000000000283 RDI: 0000000000000000 [ 36.857802] RBP: ffffbb15c0bcbd90 R08: ffffbb15c0bcbcd8 R09: 0000000000000000 [ 36.857802] R10: 0000000894a0d4b1 R11: 000000000000008c R12: ffffffffc0312480 [ 36.857802] R13: 0000000000000005 R14: ffff9740b1c32400 R15: 00000000000003e8 [ 36.857802] FS: 00007f8604422700(0000) GS:ffff9740bfc80000(0000) knlGS:0000000000000000 [ 36.857802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.857802] CR2: 0000000000000038 CR3: 000000012164c000 CR4: 00000000000006e0 [ 36.857802] Call Trace: [ 36.857802] request_firmware+0x37/0x50 [ 36.857802] trigger_request_store+0x79/0xd0 [test_firmware] [ 36.857802] dev_attr_store+0x18/0x30 [ 36.857802] sysfs_kf_write+0x37/0x40 [ 36.857802] kernfs_fop_write+0x110/0x1a0 [ 36.857802] __vfs_write+0x37/0x160 [ 36.857802] ? _cond_resched+0x1a/0x50 [ 36.857802] vfs_write+0xb5/0x1a0 [ 36.857802] SyS_write+0x55/0xc0 [ 36.857802] ? trace_do_page_fault+0x37/0xd0 [ 36.857802] entry_SYSCALL_64_fastpath+0x1e/0xad [ 36.857802] RIP: 0033:0x7f8603f49620 [ 36.857802] RSP: 002b:00007fff6287b788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 36.857802] RAX: ffffffffffffffda RBX: 000055c307b110a0 RCX: 00007f8603f49620 [ 36.857802] RDX: 0000000000000016 RSI: 000055c3084d8a90 RDI: 0000000000000001 [ 36.857802] RBP: 0000000000000016 R08: 000000000000c0ff R09: 000055c3084d6336 [ 36.857802] R10: 000055c307b108b0 R11: 0000000000000246 R12: 000055c307b13c80 [ 36.857802] R13: 000055c3084d6320 R14: 0000000000000000 R15: 00007fff6287b950 [ 36.857802] Code: 9f 64 84 e8 9c 61 fe ff b8 f4 ff ff ff e9 6b f9 ff ff 48 c7 c7 40 6b 8d 84 89 45 a8 e8 43 84 18 00 49 8b be 00 03 00 00 8b 45 a8 <83> 7f 38 02 74 08 e8 6e ec ff ff 8b 45 a8 49 c7 86 00 03 00 00 [ 36.857802] RIP: _request_firmware+0xa27/0xad0 RSP: ffffbb15c0bcbd10 [ 36.857802] CR2: 0000000000000038 [ 36.872685] ---[ end trace 6d94ac339c133e6f ]---
In above case the call hierarchy that causes the crash looks as follows:
lib/test_firmware.c request_firmware()
-> fw_load_from_user_helper()
-> _request_firmware_load()
-> call fw_state_wait_timeout()
Some time later firmware_loading_store() scans a control value of "-1" -> switch(loading) case -1: will call
-> fw_load_abort(fw_priv) which calls
-> __fw_load_abort(fw_priv->buf)
-> and set fw_priv->buf = NULL;
Upon being woken up via swake_up(), back in _request_firmware_load() fw_state_wait_timeout() returns -ENOENT
-> since mentioned commit
-> fw_load_abort(fw_priv) is called a second time
-> and this time it would call:
-> __fw_load_abort(NULL /* fw_priv->buf */)
-> and we get: NULL->fw_st.status
--- drivers/base/firmware_class.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/base/firmware_class.c +1 -4
@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv *fw_priv)struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);
@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device *dev,mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (fw_state_is_aborted (&fw_buf->fw_st)) goto out;switch (loading) {
Signature · 1 line
2.11.0
Greg KH
maintainerRe: [PATCH 7/7] firmware: firmware: fix NULL pointer dereference in __fw_load_abort()
Quoted 142 lines · Luis R. Rodriguez
Since commit
5d47ec02c37ea632398cb251c884e3a488dff794("firmware: Correct handling offw_state_wait()return value")fw_load_abort(fw_priv)could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:echo -n "-1" >
/sys/$DEVPATH/loadingThis will call
fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian, OpenSUSE), they have:$ cat /lib/udev/rules.d/50-firmware.rules # stub for immediately telling the kernel that userspace firmware loading # failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1This was done since udev removed the firmware fallback mechanism a while ago and a long standing misunderstood issues with the timeout (but now corrected). Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (
CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash:drivers/firmware/dell_rbu.cand thedrivers/leds/leds-lp55xx-common.cdriver.Distributions enabling
CONFIG_FW_LOADER_USER_HELPER_FALLBACKare clearly more exposed as every file not found through a firmware request will use the fallback mechanism.The crash happens because after commit
5b029624948d("firmware: do not usefw_lockforfw_stateprotection") and subsequent fix commit5d47ec02c37ea6("firmware: Correct handling offw_state_wait()return value") a race can happen between this cancelation and the firmwarefw_state_wait_timeout()being woken up after a state change with whichfw_load_abort()as that callsswake_up(). Upon errorfw_state_wait_timeout()will also again callfw_load_abort()and trigger a null reference.At first glance we could just fix this with a !buf check on
fw_load_abort()before accessingbuf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).The
firmware_class.ccode is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.An oops can be reproduced with the new
fw_fallback.shfallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.mcgrof@piggy ~/linux-next/tools/testing/selftests/firmware (git::20170111-fw-fixes)$ sudo ./fw_fallback.sh
./fw_fallback.sh: timeout works
./fw_fallback.sh: firmware comparison works
./fw_fallback.sh: fallback mechanism works[ this then sits here when it is trying the cancellation test ]
Kernel log:
[ 36.750521] test_firmware: loading 'nope-test-firmware.bin' [ 36.751144] misc test_firmware: Direct firmware load for nope-test-firmware.bin failed with error -2 [ 36.752034] misc test_firmware: Falling back to user helper [ 36.853324] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 36.854221] IP: _request_firmware+0xa27/0xad0 [ 36.854671] PGD 0 [ 36.854672] [ 36.855081] Oops: 0000 [#1] SMP [ 36.855433] Modules linked in: test_firmware(E) ... etc ... [ 36.857802] CPU: 1 PID: 1396 Comm: fw_fallback.sh Tainted: G W E 4.10.0-rc3-next-20170111+ #30 [ 36.857802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014 [ 36.857802] task: ffff9740b27f4340 task.stack: ffffbb15c0bc8000 [ 36.857802] RIP: 0010:_request_firmware+0xa27/0xad0 [ 36.857802] RSP: 0018:ffffbb15c0bcbd10 EFLAGS: 00010246 [ 36.857802] RAX: 00000000fffffffe RBX: ffff9740afe5aa80 RCX: 0000000000000000 [ 36.857802] RDX: ffff9740b27f4340 RSI: 0000000000000283 RDI: 0000000000000000 [ 36.857802] RBP: ffffbb15c0bcbd90 R08: ffffbb15c0bcbcd8 R09: 0000000000000000 [ 36.857802] R10: 0000000894a0d4b1 R11: 000000000000008c R12: ffffffffc0312480 [ 36.857802] R13: 0000000000000005 R14: ffff9740b1c32400 R15: 00000000000003e8 [ 36.857802] FS: 00007f8604422700(0000) GS:ffff9740bfc80000(0000) knlGS:0000000000000000 [ 36.857802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.857802] CR2: 0000000000000038 CR3: 000000012164c000 CR4: 00000000000006e0 [ 36.857802] Call Trace: [ 36.857802] request_firmware+0x37/0x50 [ 36.857802] trigger_request_store+0x79/0xd0 [test_firmware] [ 36.857802] dev_attr_store+0x18/0x30 [ 36.857802] sysfs_kf_write+0x37/0x40 [ 36.857802] kernfs_fop_write+0x110/0x1a0 [ 36.857802] __vfs_write+0x37/0x160 [ 36.857802] ? _cond_resched+0x1a/0x50 [ 36.857802] vfs_write+0xb5/0x1a0 [ 36.857802] SyS_write+0x55/0xc0 [ 36.857802] ? trace_do_page_fault+0x37/0xd0 [ 36.857802] entry_SYSCALL_64_fastpath+0x1e/0xad [ 36.857802] RIP: 0033:0x7f8603f49620 [ 36.857802] RSP: 002b:00007fff6287b788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 36.857802] RAX: ffffffffffffffda RBX: 000055c307b110a0 RCX: 00007f8603f49620 [ 36.857802] RDX: 0000000000000016 RSI: 000055c3084d8a90 RDI: 0000000000000001 [ 36.857802] RBP: 0000000000000016 R08: 000000000000c0ff R09: 000055c3084d6336 [ 36.857802] R10: 000055c307b108b0 R11: 0000000000000246 R12: 000055c307b13c80 [ 36.857802] R13: 000055c3084d6320 R14: 0000000000000000 R15: 00007fff6287b950 [ 36.857802] Code: 9f 64 84 e8 9c 61 fe ff b8 f4 ff ff ff e9 6b f9 ff ff 48 c7 c7 40 6b 8d 84 89 45 a8 e8 43 84 18 00 49 8b be 00 03 00 00 8b 45 a8 <83> 7f 38 02 74 08 e8 6e ec ff ff 8b 45 a8 49 c7 86 00 03 00 00 [ 36.857802] RIP: _request_firmware+0xa27/0xad0 RSP: ffffbb15c0bcbd10 [ 36.857802] CR2: 0000000000000038 [ 36.872685] ---[ end trace 6d94ac339c133e6f ]---In above case the call hierarchy that causes the crash looks as follows:
lib/test_firmware.crequest_firmware()
->fw_load_from_user_helper()
->_request_firmware_load()
-> callfw_state_wait_timeout()Some time later
firmware_loading_store()scans a control value of "-1" -> switch(loading) case -1: will call
->fw_load_abort(fw_priv)which calls
->__fw_load_abort(fw_priv->buf)
-> and setfw_priv->buf= NULL;Upon being woken up via
swake_up(), back in_request_firmware_load()fw_state_wait_timeout()returns-ENOENT
-> since mentioned commit
->fw_load_abort(fw_priv)is called a second time
-> and this time it would call:
->__fw_load_abort(NULL /* fw_priv->buf */)
-> and we get:NULL->fw_st.statusFixes 5d47ec02c37e ("firmware: Correct handling of fw_state_wait() return value")Reported-and-Tested-by Jakub Kicinski <[email protected]>Reported-and-Tested-by Patrick Bruenn <[email protected]>Reported-by Chris Wilson <[email protected]>CC <[email protected]> [3.10+]Signed-off-by Luis R. Rodriguez <[email protected]>
Quoted 3 lines
--- drivers/base/firmware_class.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
Why is this patch 7/7? Shouldn't it go into 4.10-final now? Why wait for 4.11-rc1?
thanks,
greg k-h
Re: [PATCH 7/7] firmware: firmware: fix NULL pointer dereference in __fw_load_abort()
Quoted 137 lines · Greg KH
Quoted 136 lines · Luis R. Rodriguez
Since commit
5d47ec02c37ea632398cb251c884e3a488dff794("firmware: Correct handling offw_state_wait()return value")fw_load_abort(fw_priv)could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:echo -n "-1" >
/sys/$DEVPATH/loadingThis will call
fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian, OpenSUSE), they have:$ cat /lib/udev/rules.d/50-firmware.rules # stub for immediately telling the kernel that userspace firmware loading # failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1This was done since udev removed the firmware fallback mechanism a while ago and a long standing misunderstood issues with the timeout (but now corrected). Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (
CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash:drivers/firmware/dell_rbu.cand thedrivers/leds/leds-lp55xx-common.cdriver.Distributions enabling
CONFIG_FW_LOADER_USER_HELPER_FALLBACKare clearly more exposed as every file not found through a firmware request will use the fallback mechanism.The crash happens because after commit
5b029624948d("firmware: do not usefw_lockforfw_stateprotection") and subsequent fix commit5d47ec02c37ea6("firmware: Correct handling offw_state_wait()return value") a race can happen between this cancelation and the firmwarefw_state_wait_timeout()being woken up after a state change with whichfw_load_abort()as that callsswake_up(). Upon errorfw_state_wait_timeout()will also again callfw_load_abort()and trigger a null reference.At first glance we could just fix this with a !buf check on
fw_load_abort()before accessingbuf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).The
firmware_class.ccode is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.An oops can be reproduced with the new
fw_fallback.shfallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.mcgrof@piggy ~/linux-next/tools/testing/selftests/firmware (git::20170111-fw-fixes)$ sudo ./fw_fallback.sh
./fw_fallback.sh: timeout works
./fw_fallback.sh: firmware comparison works
./fw_fallback.sh: fallback mechanism works[ this then sits here when it is trying the cancellation test ]
Kernel log:
[ 36.750521] test_firmware: loading 'nope-test-firmware.bin' [ 36.751144] misc test_firmware: Direct firmware load for nope-test-firmware.bin failed with error -2 [ 36.752034] misc test_firmware: Falling back to user helper [ 36.853324] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [ 36.854221] IP: _request_firmware+0xa27/0xad0 [ 36.854671] PGD 0 [ 36.854672] [ 36.855081] Oops: 0000 [#1] SMP [ 36.855433] Modules linked in: test_firmware(E) ... etc ... [ 36.857802] CPU: 1 PID: 1396 Comm: fw_fallback.sh Tainted: G W E 4.10.0-rc3-next-20170111+ #30 [ 36.857802] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014 [ 36.857802] task: ffff9740b27f4340 task.stack: ffffbb15c0bc8000 [ 36.857802] RIP: 0010:_request_firmware+0xa27/0xad0 [ 36.857802] RSP: 0018:ffffbb15c0bcbd10 EFLAGS: 00010246 [ 36.857802] RAX: 00000000fffffffe RBX: ffff9740afe5aa80 RCX: 0000000000000000 [ 36.857802] RDX: ffff9740b27f4340 RSI: 0000000000000283 RDI: 0000000000000000 [ 36.857802] RBP: ffffbb15c0bcbd90 R08: ffffbb15c0bcbcd8 R09: 0000000000000000 [ 36.857802] R10: 0000000894a0d4b1 R11: 000000000000008c R12: ffffffffc0312480 [ 36.857802] R13: 0000000000000005 R14: ffff9740b1c32400 R15: 00000000000003e8 [ 36.857802] FS: 00007f8604422700(0000) GS:ffff9740bfc80000(0000) knlGS:0000000000000000 [ 36.857802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.857802] CR2: 0000000000000038 CR3: 000000012164c000 CR4: 00000000000006e0 [ 36.857802] Call Trace: [ 36.857802] request_firmware+0x37/0x50 [ 36.857802] trigger_request_store+0x79/0xd0 [test_firmware] [ 36.857802] dev_attr_store+0x18/0x30 [ 36.857802] sysfs_kf_write+0x37/0x40 [ 36.857802] kernfs_fop_write+0x110/0x1a0 [ 36.857802] __vfs_write+0x37/0x160 [ 36.857802] ? _cond_resched+0x1a/0x50 [ 36.857802] vfs_write+0xb5/0x1a0 [ 36.857802] SyS_write+0x55/0xc0 [ 36.857802] ? trace_do_page_fault+0x37/0xd0 [ 36.857802] entry_SYSCALL_64_fastpath+0x1e/0xad [ 36.857802] RIP: 0033:0x7f8603f49620 [ 36.857802] RSP: 002b:00007fff6287b788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 36.857802] RAX: ffffffffffffffda RBX: 000055c307b110a0 RCX: 00007f8603f49620 [ 36.857802] RDX: 0000000000000016 RSI: 000055c3084d8a90 RDI: 0000000000000001 [ 36.857802] RBP: 0000000000000016 R08: 000000000000c0ff R09: 000055c3084d6336 [ 36.857802] R10: 000055c307b108b0 R11: 0000000000000246 R12: 000055c307b13c80 [ 36.857802] R13: 000055c3084d6320 R14: 0000000000000000 R15: 00007fff6287b950 [ 36.857802] Code: 9f 64 84 e8 9c 61 fe ff b8 f4 ff ff ff e9 6b f9 ff ff 48 c7 c7 40 6b 8d 84 89 45 a8 e8 43 84 18 00 49 8b be 00 03 00 00 8b 45 a8 <83> 7f 38 02 74 08 e8 6e ec ff ff 8b 45 a8 49 c7 86 00 03 00 00 [ 36.857802] RIP: _request_firmware+0xa27/0xad0 RSP: ffffbb15c0bcbd10 [ 36.857802] CR2: 0000000000000038 [ 36.872685] ---[ end trace 6d94ac339c133e6f ]---In above case the call hierarchy that causes the crash looks as follows:
lib/test_firmware.crequest_firmware()
->fw_load_from_user_helper()
->_request_firmware_load()
-> callfw_state_wait_timeout()Some time later
firmware_loading_store()scans a control value of "-1" -> switch(loading) case -1: will call
->fw_load_abort(fw_priv)which calls
->__fw_load_abort(fw_priv->buf)
-> and setfw_priv->buf= NULL;Upon being woken up via
swake_up(), back in_request_firmware_load()fw_state_wait_timeout()returns-ENOENT
-> since mentioned commit
->fw_load_abort(fw_priv)is called a second time
-> and this time it would call:
->__fw_load_abort(NULL /* fw_priv->buf */)
-> and we get:NULL->fw_st.status
Quoted 5 lines
Quoted 5 lines
Fixes 5d47ec02c37e ("firmware: Correct handling of fw_state_wait() return value")Reported-and-Tested-by Jakub Kicinski <[email protected]>Reported-and-Tested-by Patrick Bruenn <[email protected]>Reported-by Chris Wilson <[email protected]>CC <[email protected]> [3.10+]
Note: 3.10+
Quoted 6 lines
Quoted 4 lines
Signed-off-by Luis R. Rodriguez <[email protected]>--- drivers/base/firmware_class.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)Why is this patch 7/7?
Without the tests available on a development tree one cannot easily reproduce.
Quoted 2 lines
Shouldn't it go into 4.10-final now? Why wait
for 4.11-rc1?
Certainly, it should go into 4.10 now, sorry if it seemed otherwise.
Luis
Re: [PATCH 7/7] firmware: firmware: fix NULL pointer dereference in __fw_load_abort()
Quoted 11 lines · Luis R. Rodriguez
Quoted 10 lines · Greg KH
Quoted 9 lines · Luis R. Rodriguez
Since commit
5d47ec02c37ea632398cb251c884e3a488dff794("firmware: Correct handling offw_state_wait()return value")fw_load_abort(fw_priv)could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
echo -n "-1" >
/sys/$DEVPATH/loadingThis will call
fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian, OpenSUSE), they have:
I made a typo here, OpenSUSE in fact does not carry this. Its so far only Debian I am aware of.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
$ cat /lib/udev/rules.d/50-firmware.rules # stub for immediately telling the kernel that userspace firmware loading # failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1
Luis
Greg KH
maintainerRe: [PATCH 7/7] firmware: firmware: fix NULL pointer dereference in __fw_load_abort()
Quoted 12 lines · Luis R. Rodriguez
Quoted 11 lines · Luis R. Rodriguez
Quoted 10 lines · Greg KH
Quoted 9 lines · Luis R. Rodriguez
Since commit
5d47ec02c37ea632398cb251c884e3a488dff794("firmware: Correct handling offw_state_wait()return value")fw_load_abort(fw_priv)could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
echo -n "-1" >
/sys/$DEVPATH/loadingThis will call
fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian, OpenSUSE), they have:I made a typo here, OpenSUSE in fact does not carry this. Its so far only Debian I am aware of.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
$ cat /lib/udev/rules.d/50-firmware.rules # stub for immediately telling the kernel that userspace firmware loading # failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1
Please resend this as a single patch, to be applied to 4.10-final, with the above fix, and I will be glad to take it. Never mix patches in a series that are for -final and for -next.
thanks,
greg k-h
PATCH v2 1/1 firmware: fix NULL pointer dereference in __fw_load_abort()
Since commit 5d47ec02c37ea632398cb251c884e3a488dff794 ("firmware: Correct handling of fw_state_wait() return value") fw_load_abort(fw_priv) could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:
echo -n "-1" > /sys/$DEVPATH/loading
This will call fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian), they have:
$ cat /lib/udev/rules.d/50-firmware.rules
# stub for immediately telling the kernel that userspace firmware loading
# failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y
SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1This was done since udev removed the firmware fallback mechanism a while ago and a long standing misunderstood issues with the timeout (but now corrected). Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash: drivers/firmware/dell_rbu.c and the drivers/leds/leds-lp55xx-common.c driver.
The crash happens because after commit 5b029624948d ("firmware: do not use fw_lock for fw_state protection") and subsequent fix commit 5d47ec02c37ea6 ("firmware: Correct handling of fw_state_wait() return value") a race can happen between this cancelation and the firmware fw_state_wait_timeout() being woken up after a state change with which fw_load_abort() as that calls swake_up(). Upon error fw_state_wait_timeout() will also again call fw_load_abort() and trigger a null reference.
At first glance we could just fix this with a !buf check on fw_load_abort() before accessing buf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).
The firmware_class.c code is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.
An oops can be reproduced with the new fw_fallback.sh fallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.
mcgrof@piggy ~/linux-next/tools/testing/selftests/firmware (git::20170111-fw-fixes)$ sudo ./fw_fallback.sh
./fw_fallback.sh: timeout works
./fw_fallback.sh: firmware comparison works
./fw_fallback.sh: fallback mechanism works
[ this then sits here when it is trying the cancellation test ]
Kernel log:
test_firmware: loading 'nope-test-firmware.bin'
misc test_firmware: Direct firmware load for nope-test-firmware.bin failed with error -2 misc test_firmware: Falling back to user helper
BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 IP: _request_firmware+0xa27/0xad0
PGD 0
Oops: 0000 [#1] SMP Modules linked in: test_firmware(E) ... etc ... CPU: 1 PID: 1396 Comm: fw_fallback.sh Tainted: G W E 4.10.0-rc3-next-20170111+ #30 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014 task: ffff9740b27f4340 task.stack: ffffbb15c0bc8000 RIP: 0010:_request_firmware+0xa27/0xad0 RSP: 0018:ffffbb15c0bcbd10 EFLAGS: 00010246 RAX: 00000000fffffffe RBX: ffff9740afe5aa80 RCX: 0000000000000000 RDX: ffff9740b27f4340 RSI: 0000000000000283 RDI: 0000000000000000 RBP: ffffbb15c0bcbd90 R08: ffffbb15c0bcbcd8 R09: 0000000000000000 R10: 0000000894a0d4b1 R11: 000000000000008c R12: ffffffffc0312480 R13: 0000000000000005 R14: ffff9740b1c32400 R15: 00000000000003e8 FS: 00007f8604422700(0000) GS:ffff9740bfc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000038 CR3: 000000012164c000 CR4: 00000000000006e0 Call Trace: request_firmware+0x37/0x50 trigger_request_store+0x79/0xd0 [test_firmware] dev_attr_store+0x18/0x30 sysfs_kf_write+0x37/0x40 kernfs_fop_write+0x110/0x1a0 __vfs_write+0x37/0x160 ? _cond_resched+0x1a/0x50 vfs_write+0xb5/0x1a0 SyS_write+0x55/0xc0 ? trace_do_page_fault+0x37/0xd0 entry_SYSCALL_64_fastpath+0x1e/0xad RIP: 0033:0x7f8603f49620 RSP: 002b:00007fff6287b788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 000055c307b110a0 RCX: 00007f8603f49620 RDX: 0000000000000016 RSI: 000055c3084d8a90 RDI: 0000000000000001 RBP: 0000000000000016 R08: 000000000000c0ff R09: 000055c3084d6336 R10: 000055c307b108b0 R11: 0000000000000246 R12: 000055c307b13c80 R13: 000055c3084d6320 R14: 0000000000000000 R15: 00007fff6287b950 Code: 9f 64 84 e8 9c 61 fe ff b8 f4 ff ff ff e9 6b f9 ff ff 48 c7 c7 40 6b 8d 84 89 45 a8 e8 43 84 18 00 49 8b be 00 03 00 00 8b 45 a8 <83> 7f 38 02 74 08 e8 6e ec ff ff 8b 45 a8 49 c7 86 00 03 00 00 RIP: _request_firmware+0xa27/0xad0 RSP: ffffbb15c0bcbd10 CR2: 0000000000000038 ---[ end trace 6d94ac339c133e6f ]---
In above case the call hierarchy that causes the crash looks as follows:
lib/test_firmware.c request_firmware()
-> fw_load_from_user_helper()
-> _request_firmware_load()
-> call fw_state_wait_timeout()
Some time later firmware_loading_store() scans a control value of "-1" -> switch(loading) case -1: will call
-> fw_load_abort(fw_priv) which calls
-> __fw_load_abort(fw_priv->buf)
-> and set fw_priv->buf = NULL;
Upon being woken up via swake_up(), back in _request_firmware_load() fw_state_wait_timeout() returns -ENOENT
-> since mentioned commit
-> fw_load_abort(fw_priv) is called a second time
-> and this time it would call:
-> __fw_load_abort(NULL /* fw_priv->buf */)
-> and we get: NULL->fw_st.status
---
v2: adjust commit log to remove the OpenSUSE reference as it doesn't
carry the same udev rule which cancels all fallback requests.
drivers/base/firmware_class.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/base/firmware_class.c +1 -4
@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv *fw_priv)struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);
@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device *dev,mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (fw_state_is_aborted (&fw_buf->fw_st)) goto out;switch (loading) {
Signature · 1 line
2.11.0
Greg KH
maintainerRe: [PATCH v2] firmware: fix NULL pointer dereference in __fw_load_abort()
Quoted 43 lines · Luis R. Rodriguez
Since commit
5d47ec02c37ea632398cb251c884e3a488dff794("firmware: Correct handling offw_state_wait()return value")fw_load_abort(fw_priv)could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:echo -n "-1" >
/sys/$DEVPATH/loadingThis will call
fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian), they have:$ cat /lib/udev/rules.d/50-firmware.rules # stub for immediately telling the kernel that userspace firmware loading # failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1This was done since udev removed the firmware fallback mechanism a while ago and a long standing misunderstood issues with the timeout (but now corrected). Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (
CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash:drivers/firmware/dell_rbu.cand thedrivers/leds/leds-lp55xx-common.cdriver.The crash happens because after commit
5b029624948d("firmware: do not usefw_lockforfw_stateprotection") and subsequent fix commit5d47ec02c37ea6("firmware: Correct handling offw_state_wait()return value") a race can happen between this cancelation and the firmwarefw_state_wait_timeout()being woken up after a state change with whichfw_load_abort()as that callsswake_up(). Upon errorfw_state_wait_timeout()will also again callfw_load_abort()and trigger a null reference.At first glance we could just fix this with a !buf check on
fw_load_abort()before accessingbuf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).
Quoted 8 lines
The
firmware_class.ccode is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.An oops can be reproduced with the new
fw_fallback.shfallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.
You are still writing books here.
With crazy margins, pick one line width (72 columns), and stick with it please.
Can you reformat this and resend please?
thanks,
greg k-h-
Re: [PATCH v2] firmware: fix NULL pointer dereference in __fw_load_abort()
Quoted 49 lines · Greg KH
Quoted 48 lines · Luis R. Rodriguez
Since commit
5d47ec02c37ea632398cb251c884e3a488dff794("firmware: Correct handling offw_state_wait()return value")fw_load_abort(fw_priv)could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:echo -n "-1" >
/sys/$DEVPATH/loadingThis will call
fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian), they have:$ cat /lib/udev/rules.d/50-firmware.rules # stub for immediately telling the kernel that userspace firmware loading # failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1This was done since udev removed the firmware fallback mechanism a while ago and a long standing misunderstood issues with the timeout (but now corrected). Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (
CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash:drivers/firmware/dell_rbu.cand thedrivers/leds/leds-lp55xx-common.cdriver.The crash happens because after commit
5b029624948d("firmware: do not usefw_lockforfw_stateprotection") and subsequent fix commit5d47ec02c37ea6("firmware: Correct handling offw_state_wait()return value") a race can happen between this cancelation and the firmwarefw_state_wait_timeout()being woken up after a state change with whichfw_load_abort()as that callsswake_up(). Upon errorfw_state_wait_timeout()will also again callfw_load_abort()and trigger a null reference.At first glance we could just fix this with a !buf check on
fw_load_abort()before accessingbuf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).The
firmware_class.ccode is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.
Quoted 5 lines
Quoted 3 lines
An oops can be reproduced with the new
fw_fallback.shfallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.You are still writing books here.
Alright trimmed.
Quoted 4 lines
With crazy margins, pick one line width (72 columns), and stick with it please.
Can you reformat this and resend please?
Sure.
Luis
PATCH v3 1/1 firmware: fix NULL pointer dereference in __fw_load_abort()
In reply to Greg KHSince commit 5d47ec02c37ea6 ("firmware: Correct handling of fw_state_wait() return value") fw_load_abort() could be called twice and lead us to a kernel crash. This happens only when the firmware fallback mechanism (regular or custom) is used. The fallback mechanism exposes a sysfs interface for userspace to upload a file and notify the kernel when the file is loaded and ready, or to cancel an upload by echo'ing -1 into on the loading file:
echo -n "-1" > /sys/$DEVPATH/loading
This will call fw_load_abort(). Some distributions actually have a udev rule in place to *always* immediately cancel all firmware fallback mechanism requests (Debian), they have:
$ cat /lib/udev/rules.d/50-firmware.rules
# stub for immediately telling the kernel that userspace firmware loading
# failed; necessary to avoid long timeouts with CONFIG_FW_LOADER_USER_HELPER=y
SUBSYSTEM=="firmware", ACTION=="add", ATTR{loading}="-1Distributions with this udev rule would run into this crash only if the fallback mechanism is used. Since most distributions disable by default using the fallback mechanism (CONFIG_FW_LOADER_USER_HELPER_FALLBACK), this would typicaly mean only 2 drivers which *require* the fallback mechanism could typically incur a crash: drivers/firmware/dell_rbu.c and the drivers/leds/leds-lp55xx-common.c driver. Distributions enabling CONFIG_FW_LOADER_USER_HELPER_FALLBACK by default are obviously more exposed to this crash.
The crash happens because after commit 5b029624948d ("firmware: do not use fw_lock for fw_state protection") and subsequent fix commit 5d47ec02c37ea6 ("firmware: Correct handling of fw_state_wait() return value") a race can happen between this cancelation and the firmware fw_state_wait_timeout() being woken up after a state change with which fw_load_abort() as that calls swake_up(). Upon error fw_state_wait_timeout() will also again call fw_load_abort() and trigger a null reference.
At first glance we could just fix this with a !buf check on fw_load_abort() before accessing buf->fw_st, however there is a logical issue in having a state machine used for the fallback mechanism and preventing access from it once we abort as its inside the buf (buf->fw_st).
The firmware_class.c code is setting the buf to NULL to annotate an abort has occurred. Replace this mechanism by simply using the state check instead. All the other code in place already uses similar checks for aborting as well so no further changes are needed.
An oops can be reproduced with the new fw_fallback.sh fallback mechanism cancellation test. Either cancelling the fallback mechanism or the custom fallback mechanism triggers a crash.
mcgrof@piggy ~/linux-next/tools/testing/selftests/firmware (git::20170111-fw-fixes)$ sudo ./fw_fallback.sh
./fw_fallback.sh: timeout works
./fw_fallback.sh: firmware comparison works
./fw_fallback.sh: fallback mechanism works
[ this then sits here when it is trying the cancellation test ]
Kernel log:
test_firmware: loading 'nope-test-firmware.bin'
misc test_firmware: Direct firmware load for nope-test-firmware.bin failed with error -2 misc test_firmware: Falling back to user helper
BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 IP: _request_firmware+0xa27/0xad0
PGD 0
Oops: 0000 [#1] SMP Modules linked in: test_firmware(E) ... etc ... CPU: 1 PID: 1396 Comm: fw_fallback.sh Tainted: G W E 4.10.0-rc3-next-20170111+ #30 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014 task: ffff9740b27f4340 task.stack: ffffbb15c0bc8000 RIP: 0010:_request_firmware+0xa27/0xad0 RSP: 0018:ffffbb15c0bcbd10 EFLAGS: 00010246 RAX: 00000000fffffffe RBX: ffff9740afe5aa80 RCX: 0000000000000000 RDX: ffff9740b27f4340 RSI: 0000000000000283 RDI: 0000000000000000 RBP: ffffbb15c0bcbd90 R08: ffffbb15c0bcbcd8 R09: 0000000000000000 R10: 0000000894a0d4b1 R11: 000000000000008c R12: ffffffffc0312480 R13: 0000000000000005 R14: ffff9740b1c32400 R15: 00000000000003e8 FS: 00007f8604422700(0000) GS:ffff9740bfc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000038 CR3: 000000012164c000 CR4: 00000000000006e0 Call Trace: request_firmware+0x37/0x50 trigger_request_store+0x79/0xd0 [test_firmware] dev_attr_store+0x18/0x30 sysfs_kf_write+0x37/0x40 kernfs_fop_write+0x110/0x1a0 __vfs_write+0x37/0x160 ? _cond_resched+0x1a/0x50 vfs_write+0xb5/0x1a0 SyS_write+0x55/0xc0 ? trace_do_page_fault+0x37/0xd0 entry_SYSCALL_64_fastpath+0x1e/0xad RIP: 0033:0x7f8603f49620 RSP: 002b:00007fff6287b788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 000055c307b110a0 RCX: 00007f8603f49620 RDX: 0000000000000016 RSI: 000055c3084d8a90 RDI: 0000000000000001 RBP: 0000000000000016 R08: 000000000000c0ff R09: 000055c3084d6336 R10: 000055c307b108b0 R11: 0000000000000246 R12: 000055c307b13c80 R13: 000055c3084d6320 R14: 0000000000000000 R15: 00007fff6287b950 Code: 9f 64 84 e8 9c 61 fe ff b8 f4 ff ff ff e9 6b f9 ff ff 48 c7 c7 40 6b 8d 84 89 45 a8 e8 43 84 18 00 49 8b be 00 03 00 00 8b 45 a8 <83> 7f 38 02 74 08 e8 6e ec ff ff 8b 45 a8 49 c7 86 00 03 00 00 RIP: _request_firmware+0xa27/0xad0 RSP: ffffbb15c0bcbd10 CR2: 0000000000000038 ---[ end trace 6d94ac339c133e6f ]---
---
v3: shorten commit log and stick to 72 characters for margins on
commit log
drivers/base/firmware_class.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/base/firmware_class.c +1 -4
@@ -558,9 +558,6 @@ static void fw_load_abort(struct firmware_priv *fw_priv)struct firmware_buf *buf = fw_priv->buf;__fw_load_abort (buf); - -/* avoid user action after loading abort */ - fw_priv->buf =NULL ; }static LIST_HEAD (pending_fw_head);
@@ -713,7 +710,7 @@ static ssize_t firmware_loading_store(struct device *dev,mutex_lock (&fw_lock); fw_buf = fw_priv->buf; -if (!fw_buf) +if (fw_state_is_aborted (&fw_buf->fw_st)) goto out;switch (loading) {
Signature · 1 line
2.11.0