Hang on reboot in nand_get_device()
Subsystems (2)
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a
dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
It appears to be a hang in mtd/nand/nand_base.c:nand_get_device() waiting on chip->controller->active.
Shouldn't nand_shutdown(), or perhaps a special case in nand_get_device() for FL_SHUTDOWN, set chip->controller->active = NULL before returning?
This seems to fix the problem for me, but I don't know all the code well enough to know whether doing so is appropriate, or sufficient.
~~Andrew E. Mileski
+ others
Hi Andrew,
Sorry for the delay here. I overlooked this.
Quoted 3 lines · Andrew E. Mileski
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
Which driver?
Quoted 5 lines
It appears to be a hang in mtd/nand/nand_base.c:nand_get_device() waiting on
chip->controller->active.Shouldn't
nand_shutdown(), or perhaps a special case innand_get_device()forFL_SHUTDOWN, setchip->controller->active= NULL before returning?
I don't think that's exactly the right solution, but you're in the right ballpark I expect.
Quoted 2 lines
This seems to fix the problem for me, but I don't know all the code well enough to know whether doing so is appropriate, or sufficient.
I'm going to guess you're seeing two reboot handlers trying to 'get' the same controller structure. We could probably confirm that if we see your driver.
But I see this could be a problem with a wide class of drivers. Basically, any NAND driver that has multiple NAND chips attached will see multiple reboot handlers that point at the same controller lock. This will obviously deadlock, since only one of the chips will make it through the nand_shutdown() function successfully.
And now that I begin describing the problem and grepping through the source logs... I see that this problem was already resolved back in 2009, except for the FL_PM_SUSPENDED mode:
commit 6b0d9a84124937f048bcb8b21313152b23063978
Author: Li Yang <[email protected]> Date: Tue Nov 17 14:45:49 2009 -0800
mtd: nand: fix multi-chip suspend problem
I actually don't see why we can't just use nand_get_device(FL_PM_SUSPENDED) for the shutdown/reboot case, like this:
drivers/mtd/nand/nand_base.c +1 -1
@@ -3110,7 +3110,7 @@ static void nand_resume(struct mtd_info *mtd)*/ static void nand_shutdown (struct mtd_info *mtd) { -nand_get_device (mtd,FL_SHUTDOWN ); +nand_get_device (mtd,FL_PM_SUSPENDED ); }/* Set default functions */
It's also possible that this could be better solved in a proper refactor/rewrite of the NAND subsystem using a better controller/chip split, so there's only one reboot handler per NAND controller. Boris has been looking at that.
Anyway, if the above looks OK, I can send a proper patch and get your 'Tested-by's.
Brian
Hi Brian,
Quoted 13 lines · On Fri, 6 Nov 2015 10:00:52 -0800 Brian Norris <[email protected]>…
+ others
Hi Andrew,
Sorry for the delay here. I overlooked this.
Quoted 3 lines · Andrew E. Mileski
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
Which driver?
Quoted 8 lines
Quoted 5 lines
It appears to be a hang in mtd/nand/nand_base.c:nand_get_device() waiting on
chip->controller->active.Shouldn't
nand_shutdown(), or perhaps a special case innand_get_device()forFL_SHUTDOWN, setchip->controller->active= NULL before returning?I don't think that's exactly the right solution, but you're in the right ballpark I expect.
Hm, actually I find the idea of releasing the NAND controller when the flash is set in FL_SHUTDOWN state not that bad. Is there any reason we would want the NAND chip to stay active when we're shutting the system down.
I understand that this is needed for the suspended case because we want the system to restore the correct state when resuming (ie marking the right device as active), but shutdown should be simpler.
Quoted 7 lines
Quoted 3 lines
This seems to fix the problem for me, but I don't know all the code well enough to know whether doing so is appropriate, or sufficient.
I'm going to guess you're seeing two reboot handlers trying to 'get' the same controller structure. We could probably confirm that if we see your driver.
Yes, that's probably what's happening here.
Quoted 6 lines
But I see this could be a problem with a wide class of drivers. Basically, any NAND driver that has multiple NAND chips attached will see multiple reboot handlers that point at the same controller lock. This will obviously deadlock, since only one of the chips will make it through the
nand_shutdown()function successfully.
Indeed.
Quoted 9 lines
And now that I begin describing the problem and grepping through the source logs... I see that this problem was already resolved back in 2009, except for the
FL_PM_SUSPENDEDmode:commit
6b0d9a84124937f048bcb8b21313152b23063978Author: Li Yang <[email protected]> Date: Tue Nov 17 14:45:49 2009 -0800
Quoted 7 lines
mtd: nand: fix multi-chip suspend problem
I actually don't see why we can't just use
nand_get_device(FL_PM_SUSPENDED)for the shutdown/reboot case, like this:
That should work, but shouldn't we keep the appropriate state (FL_SHUTDOWN) and patch the nand_get_device() code instead?
drivers/mtd/nand/nand_base.c +3 -3
@@ -839,9 +839,9 @@ retry:spin_unlock (lock);return 0 ; } -if (new_state ==FL_PM_SUSPENDED ) { -if (chip->controller->active->state ==FL_PM_SUSPENDED ) { - chip->state =FL_PM_SUSPENDED ; +if (new_state ==FL_PM_SUSPENDED || new_state ==FL_SHUTDOWN ) { +if (chip->controller->active->state == new_state) { + chip->state = new_state;spin_unlock (lock);return 0 ; }
or
drivers/mtd/nand/nand_base.c +14 -0
@@ -830,6 +830,20 @@ nand_get_device(struct mtd_info *mtd, int new_state) retry:spin_lock (lock); +/* putting the NAND chip in shutdown state should always succeed. */ +if (new_state ==FL_SHUTDOWN ) { +/* +* release the controller if the chip put in shutdown state +* is the current active device. +*/ +if (chip->controller->active == chip) + chip->controller->active =NULL ; + + chip->state = new_state; +spin_unlock (lock); +return 0 ; + } +/* Hardware controller shared among independent devices */ if (!chip->controller->active) chip->controller->active = chip;
Quoted 16 lines
drivers/mtd/nand/nand_base.c +1
-1@@ -3110,7 +3110,7 @@ static void nand_resume(struct mtd_info *mtd)*/ static void nand_shutdown (struct mtd_info *mtd) { -nand_get_device (mtd,FL_SHUTDOWN ); +nand_get_device (mtd,FL_PM_SUSPENDED ); }/* Set default functions */
Quoted 4 lines
It's also possible that this could be better solved in a proper refactor/rewrite of the NAND subsystem using a better controller/chip split, so there's only one reboot handler per NAND controller. Boris has been looking at that.
Yes, but I haven't considered reworking the locking or the reboot notifier stuff :-). BTW, if you want to have a look at this work, it is in my nand/layering-rework branch (starting at commit 9da4cc22857c103010141bca4d403915ee103dbb).
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 15 lines · Boris Brezillon
Quoted 13 lines · On Fri, 6 Nov 2015 10:00:52 -0800 Brian Norris <[email protected]>…
Quoted 3 lines · Andrew E. Mileski
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
Which driver?
Quoted 5 lines
It appears to be a hang in mtd/nand/nand_base.c:nand_get_device() waiting on
chip->controller->active.Shouldn't
nand_shutdown(), or perhaps a special case innand_get_device()forFL_SHUTDOWN, setchip->controller->active= NULL before returning?
Quoted 7 lines
Quoted 2 lines
I don't think that's exactly the right solution, but you're in the right ballpark I expect.
Hm, actually I find the idea of releasing the NAND controller when the flash is set in
FL_SHUTDOWNstate not that bad. Is there any reason we would want the NAND chip to stay active when we're shutting the system down.
No, and that's the point of "getting" the device, without actually releasing it. (Andrew's suggestion is essentially a nand_release_device().) We don't want any other process picking up any I/O at this point. I suppose there really is no way to begin any new I/O, but it does seem possible for the last operation to still be in flight, at least according to Scott's reports.
Quoted 3 lines
I understand that this is needed for the suspended case because we want the system to restore the correct state when resuming (ie marking the right device as active), but shutdown should be simpler.
Is the SUSPENDED code really that complex? It's just allowing all FL_PM_SUSPENDED requests after the first one. It doesn't even do much fancy for the release path (and in fact, it seems slightly buggy, now that I'm looking at it; because we violated mutual exclusion, there can now be many chips that are releasing the same chip. So nand_get_device() won't really work right again until all chips have called nand_resume().)
Quoted 6 lines
Quoted 3 lines
I actually don't see why we can't just use
nand_get_device(FL_PM_SUSPENDED)for the shutdown/reboot case, like this:That should work, but shouldn't we keep the appropriate state (
FL_SHUTDOWN) and patch thenand_get_device()code instead?
The states mean only as much as we ascribe to them. If we really need to treat shutdown differently than suspend, then I suppose yes. But otherwise, I see no need.
Quoted 7 lines
drivers/mtd/nand/nand_base.c +0
-0@@ -839,9 +839,9 @@ retry:spin_unlock (lock);return 0 ;
drivers/mtd/nand/nand_base.c retry: · line 847
Quoted 10 lines
drivers/mtd/nand/nand_base.c +3
-3} -if (new_state ==FL_PM_SUSPENDED ) { -if (chip->controller->active->state ==FL_PM_SUSPENDED ) { - chip->state =FL_PM_SUSPENDED ; +if (new_state ==FL_PM_SUSPENDED || new_state ==FL_SHUTDOWN ) { +if (chip->controller->active->state == new_state) { + chip->state = new_state;spin_unlock (lock);return 0 ; }
I suppose that works, but I don't see it buying us much.
(Although, I suppose it makes it a little more obvious if we somehow mixed the SUSPENDED state with the SHUTDOWN state.)
Quoted 18 lines
or
drivers/mtd/nand/nand_base.c +7
-0@@ -830,6 +830,20 @@ nand_get_device(struct mtd_info *mtd, int new_state) retry:spin_lock (lock); +/* putting the NAND chip in shutdown state should always succeed. */ +if (new_state ==FL_SHUTDOWN ) { +/* +* release the controller if the chip put in shutdown state +* is the current active device. +*/ +if (chip->controller->active == chip)
drivers/mtd/nand/nand_base.c nand_get_device(struct mtd_info *mtd, int new_state) · line 849
Quoted 10 lines
drivers/mtd/nand/nand_base.c +7
-0+ chip->controller->active =NULL ; + + chip->state = new_state; +spin_unlock (lock); +return 0 ; + } +/* Hardware controller shared among independent devices */ if (!chip->controller->active) chip->controller->active = chip;
This looks a lot more subtle and potentially wrong. What exactly is the rationale here? It appears you're kind of unlocking the controller (any other flash on the same controller can still go ahead) but at the same time forcing no further users of this particular flash. In the end, I guess it accomplishes mostly the same thing as the SUSPENDED case, except that it's written differently, and has slightly different behaviors in corner cases (e.g., what if another nand_chip gets a call to nand_get_device(FL_WRITING)? with this patch, they can still write to the flash; with your first patch or with mine, they are locked out).
IOW, I'd rather share the implementation if they're really that similar, unless you really have a good reason for this one.
Quoted 7 lines
Quoted 4 lines
It's also possible that this could be better solved in a proper refactor/rewrite of the NAND subsystem using a better controller/chip split, so there's only one reboot handler per NAND controller. Boris has been looking at that.
Yes, but I haven't considered reworking the locking or the reboot notifier stuff :-).
Right, but once they're split, it might be easier to have more generic per-controller features (rather than per-flash). But maybe not. I'm definitely not suggesting we do that now; let's just fix the problem with the code as it stands now.
Brian
Quoted 7 lines · Brian Norris
No, and that's the point of "getting" the device, without actually releasing it. (Andrew's suggestion is essentially a
nand_release_device().) We don't want any other process picking up any I/O at this point. I suppose there really is no way to begin any new I/O, but it does seem possible for the last operation to still be in flight, at least according to Scott's reports.
That right there. is the very reason I questioned whether my approach was sufficient.
~~Andrew E. Mileski
Quoted 11 lines · Andrew E. Mileski
Quoted 7 lines · Brian Norris
No, and that's the point of "getting" the device, without actually releasing it. (Andrew's suggestion is essentially a
nand_release_device().) We don't want any other process picking up any I/O at this point. I suppose there really is no way to begin any new I/O, but it does seem possible for the last operation to still be in flight, at least according to Scott's reports.That right there. is the very reason I questioned whether my approach was sufficient.
Yes, the last operation can still be in flight - there are async cleanup threads running in the UBI/UBIFS layers which recreated this problem.
Quoted 2 lines
~~Andrew E. Mileski
Hi Brian,
Quoted 24 lines · On Mon, 9 Nov 2015 11:46:51 -0800 Brian Norris <[email protected]>…
Quoted 15 lines · Boris Brezillon
Quoted 13 lines · On Fri, 6 Nov 2015 10:00:52 -0800 Brian Norris <[email protected]>…
Quoted 3 lines · Andrew E. Mileski
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
Which driver?
Quoted 5 lines
It appears to be a hang in mtd/nand/nand_base.c:nand_get_device() waiting on
chip->controller->active.Shouldn't
nand_shutdown(), or perhaps a special case innand_get_device()forFL_SHUTDOWN, setchip->controller->active= NULL before returning?Quoted 7 lines
Quoted 2 lines
I don't think that's exactly the right solution, but you're in the right ballpark I expect.
Hm, actually I find the idea of releasing the NAND controller when the flash is set in
FL_SHUTDOWNstate not that bad. Is there any reason we would want the NAND chip to stay active when we're shutting the system down.
Quoted 6 lines
No, and that's the point of "getting" the device, without actually releasing it. (Andrew's suggestion is essentially a
nand_release_device().) We don't want any other process picking up any I/O at this point. I suppose there really is no way to begin any new I/O, but it does seem possible for the last operation to still be in flight, at least according to Scott's reports.
Hm, if we really want all I/O going through the NAND controller to be stopped, then that should definitely be handled at the controller level (as you suggested in your previous email).
My understanding was that the shutdown and suspended states were only attached to the device itself, not the controller behind this device (even if at some point all the devices will be marked as suspended or shut down and the controller will remain inactive).
Quoted 11 lines
Quoted 4 lines
I understand that this is needed for the suspended case because we want the system to restore the correct state when resuming (ie marking the right device as active), but shutdown should be simpler.
Is the SUSPENDED code really that complex? It's just allowing all
FL_PM_SUSPENDEDrequests after the first one. It doesn't even do much fancy for the release path (and in fact, it seems slightly buggy, now that I'm looking at it; because we violated mutual exclusion, there can now be many chips that are releasing the same chip. Sonand_get_device()won't really work right again until all chips have callednand_resume().)
No it's not that complex, but I find it weird to keep the lock on the controller even if the NAND chip attached to this controller has been suspended. And the problem you pointed out just shows how tricky it can be when you play this kind of game (keep the lock on the controller assigned to one NAND chip but consider other nand_get_device() calls as successful even if the controller is already taken by another chip in some specific cases).
BTW, I guess you already have the fix, but in case you haven't done it yet, this should do the trick:
drivers/mtd/nand/nand_base.c +4 -2
@@ -136,9 +136,11 @@ static void nand_release_device(struct mtd_info *mtd)/* Release the controller and the chip */ spin_lock (&chip->controller->lock); - chip->controller->active =NULL ; chip->state =FL_READY ; -wake_up (&chip->controller->wq); +if (chip->controller->active == chip) { + chip->controller->active =NULL ; +wake_up (&chip->controller->wq); + }spin_unlock (&chip->controller->lock); }
Quoted 32 lines
Quoted 7 lines
Quoted 4 lines
I actually don't see why we can't just use
nand_get_device(FL_PM_SUSPENDED)for the shutdown/reboot case, like this:That should work, but shouldn't we keep the appropriate state (
FL_SHUTDOWN) and patch thenand_get_device()code instead?The states mean only as much as we ascribe to them. If we really need to treat shutdown differently than suspend, then I suppose yes. But otherwise, I see no need.
Quoted 7 lines
drivers/mtd/nand/nand_base.c +0
-0@@ -839,9 +839,9 @@ retry:spin_unlock (lock);return 0 ;drivers/mtd/nand/nand_base.c retry: · line 847
Quoted 10 lines
drivers/mtd/nand/nand_base.c +3
-3} -if (new_state ==FL_PM_SUSPENDED ) { -if (chip->controller->active->state ==FL_PM_SUSPENDED ) { - chip->state =FL_PM_SUSPENDED ; +if (new_state ==FL_PM_SUSPENDED || new_state ==FL_SHUTDOWN ) { +if (chip->controller->active->state == new_state) { + chip->state = new_state;spin_unlock (lock);return 0 ; }I suppose that works, but I don't see it buying us much.
Quoted 2 lines
(Although, I suppose it makes it a little more obvious if we somehow mixed the SUSPENDED state with the SHUTDOWN state.)
Well, it's more about using the proper state in case we have to propagate it to NAND controller drivers someday.
Quoted 31 lines
Quoted 19 lines
or
drivers/mtd/nand/nand_base.c +7
-0@@ -830,6 +830,20 @@ nand_get_device(struct mtd_info *mtd, int new_state) retry:spin_lock (lock); +/* putting the NAND chip in shutdown state should always succeed. */ +if (new_state ==FL_SHUTDOWN ) { +/* +* release the controller if the chip put in shutdown state +* is the current active device. +*/ +if (chip->controller->active == chip)drivers/mtd/nand/nand_base.c nand_get_device(struct mtd_info *mtd, int new_state) · line 849
Quoted 10 lines
drivers/mtd/nand/nand_base.c +7
-0+ chip->controller->active =NULL ; + + chip->state = new_state; +spin_unlock (lock); +return 0 ; + } +/* Hardware controller shared among independent devices */ if (!chip->controller->active) chip->controller->active = chip;This looks a lot more subtle and potentially wrong. What exactly is the
Quoted 8 lines
rationale here? It appears you're kind of unlocking the controller (any other flash on the same controller can still go ahead) but at the same time forcing no further users of this particular flash. In the end, I guess it accomplishes mostly the same thing as the SUSPENDED case, except that it's written differently, and has slightly different behaviors in corner cases (e.g., what if another
nand_chipgets a call tonand_get_device(FL_WRITING)? with this patch, they can still write to the flash; with your first patch or with mine, they are locked out).
Yes, that's pretty much was I said above: I was seeing FL_XXX states as attached to the NAND chips not to the controller used to access those chips. Not sure this is possible, but what if we had a way to use runtime PM at the chip level, not the controller level. That would mean preventing every other NAND chip under the same controller to work until this chip enter the resumed/ready state.
Quoted 12 lines
IOW, I'd rather share the implementation if they're really that similar, unless you really have a good reason for this one.
Quoted 5 lines
Quoted 4 lines
It's also possible that this could be better solved in a proper refactor/rewrite of the NAND subsystem using a better controller/chip split, so there's only one reboot handler per NAND controller. Boris has been looking at that.
Quoted 7 lines
Quoted 2 lines
Yes, but I haven't considered reworking the locking or the reboot notifier stuff :-).
Right, but once they're split, it might be easier to have more generic per-controller features (rather than per-flash). But maybe not. I'm definitely not suggesting we do that now; let's just fix the problem with the code as it stands now.
Yep.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Hi again,
Just want to add that this discussion shouldn't prevent your fix from being applied. The main reason I'm arguing here is because I want to understand the rationale behind the current handling of FL_PM_SUSPENDED and FL_SHUTDOWN.
Quoted 24 lines · On Mon, 9 Nov 2015 21:55:08 +0100 Boris Brezillon…
Quoted 24 lines
Quoted 24 lines
drivers/mtd/nand/nand_base.c +14
-0@@ -830,6 +830,20 @@ nand_get_device(struct mtd_info *mtd, int new_state) retry:spin_lock (lock); +/* putting the NAND chip in shutdown state should always succeed. */ +if (new_state ==FL_SHUTDOWN ) { +/* +* release the controller if the chip put in shutdown state +* is the current active device. +*/ +if (chip->controller->active == chip) + chip->controller->active =NULL ; + + chip->state = new_state; +spin_unlock (lock); +return 0 ; + } +/* Hardware controller shared among independent devices */
Quoted 8 lines
Quoted 8 lines
drivers/mtd/nand/nand_base.c nand_get_device(struct mtd_info *mtd, int new_state) · line 849
Quoted 3 lines
drivers/mtd/nand/nand_base.c +0
-0if (!chip->controller->active) chip->controller->active = chip;This looks a lot more subtle and potentially wrong. What exactly is the rationale here? It appears you're kind of unlocking the controller (any other flash on the same controller can still go ahead) but at the same time forcing no further users of this particular flash.
It's even worst: I'm not waiting for the chip to become ready, so I'm potentially re-introducing the bug Scott was trying to solve with his reboot notifier.
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 4 lines · Boris Brezillon
Just want to add that this discussion shouldn't prevent your fix from being applied. The main reason I'm arguing here is because I want to understand the rationale behind the current handling of
FL_PM_SUSPENDEDandFL_SHUTDOWN.
Sure, that's reasonable. I'd also like to touch this code only once (or very close to that) in the near future, so it's best if we get to a good understanding.
I'll send this as a proper patch, if that sounds OK:
http://patchwork.ozlabs.org/patch/541065/
Quoted 36 lines
Quoted 24 lines · On Mon, 9 Nov 2015 21:55:08 +0100 Boris Brezillon…
Quoted 24 lines
Quoted 24 lines
drivers/mtd/nand/nand_base.c +14
-0@@ -830,6 +830,20 @@ nand_get_device(struct mtd_info *mtd, int new_state) retry:spin_lock (lock); +/* putting the NAND chip in shutdown state should always succeed. */ +if (new_state ==FL_SHUTDOWN ) { +/* +* release the controller if the chip put in shutdown state +* is the current active device. +*/ +if (chip->controller->active == chip) + chip->controller->active =NULL ; + + chip->state = new_state; +spin_unlock (lock); +return 0 ; + } +/* Hardware controller shared among independent devices */ Quoted 8 lines
Quoted 8 lines
drivers/mtd/nand/nand_base.c nand_get_device(struct mtd_info *mtd, int new_state) · line 849
Quoted 3 lines
drivers/mtd/nand/nand_base.c +0
-0if (!chip->controller->active) chip->controller->active = chip;This looks a lot more subtle and potentially wrong. What exactly is the rationale here? It appears you're kind of unlocking the controller (any other flash on the same controller can still go ahead) but at the same time forcing no further users of this particular flash.
Quoted 3 lines
It's even worst: I'm not waiting for the chip to become ready, so I'm potentially re-introducing the bug Scott was trying to solve with his reboot notifier.
Ah, I see! Good catch. My distaste for duplication pays off, then :)
Brian
Hi Brian,
I'm confused as to what the outcome is here and what the final patch is.
Will this affect the previous fixes we made such that shutdown is
called on reboot so that MTD operations to the controller are not in progress on reboot?
Thanks,
Scott
Quoted 58 lines · Brian Norris
Quoted 4 lines · Boris Brezillon
Just want to add that this discussion shouldn't prevent your fix from being applied. The main reason I'm arguing here is because I want to understand the rationale behind the current handling of
FL_PM_SUSPENDEDandFL_SHUTDOWN.Sure, that's reasonable. I'd also like to touch this code only once (or very close to that) in the near future, so it's best if we get to a good understanding.
I'll send this as a proper patch, if that sounds OK:
http://patchwork.ozlabs.org/patch/541065/
Quoted 36 lines
Quoted 24 lines · On Mon, 9 Nov 2015 21:55:08 +0100 Boris Brezillon…
Quoted 24 lines
Quoted 24 lines
drivers/mtd/nand/nand_base.c +14
-0@@ -830,6 +830,20 @@ nand_get_device(struct mtd_info *mtd, int new_state) retry:spin_lock (lock); +/* putting the NAND chip in shutdown state should always succeed. */ +if (new_state ==FL_SHUTDOWN ) { +/* +* release the controller if the chip put in shutdown state +* is the current active device. +*/ +if (chip->controller->active == chip) + chip->controller->active =NULL ; + + chip->state = new_state; +spin_unlock (lock); +return 0 ; + } +/* Hardware controller shared among independent devices */ Quoted 8 lines
Quoted 8 lines
drivers/mtd/nand/nand_base.c nand_get_device(struct mtd_info *mtd, int new_state) · line 849
Quoted 3 lines
drivers/mtd/nand/nand_base.c +0
-0if (!chip->controller->active) chip->controller->active = chip;This looks a lot more subtle and potentially wrong. What exactly is the rationale here? It appears you're kind of unlocking the controller (any other flash on the same controller can still go ahead) but at the same time forcing no further users of this particular flash.
Quoted 3 lines
It's even worst: I'm not waiting for the chip to become ready, so I'm potentially re-introducing the bug Scott was trying to solve with his reboot notifier.
Ah, I see! Good catch. My distaste for duplication pays off, then :)
Brian
Hi Scott,
Quoted 4 lines · Scott Branden
I'm confused as to what the outcome is here and what the final patch is. Will this affect the previous fixes we made such that shutdown is called on reboot so that MTD operations to the controller are not in progress on reboot?
There is no "final outcome" yet, but the plan is below:
Quoted 4 lines · Brian Norris
Quoted 3 lines · Boris Brezillon
Quoted 2 lines
Just want to add that this discussion shouldn't prevent your fix from being applied.
[...]
Quoted 3 lines
Quoted 3 lines
I'll send this as a proper patch, if that sounds OK:
^^ I plan to resend that patch as an independent thread, and give people a chance to test/ack/nak anything there. I'll CC you, so you can ensure it doesn't break anything you did previously. (I don't see how it would.)
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
On Mon, 9 Nov 2015 21:55:08 +0100
Boris Brezillon <[email protected]> wrote:It's even worst: I'm not waiting for the chip to become ready, so I'm potentially re-introducing the bug Scott was trying to solve with his reboot notifier.
Ah, I see! Good catch. My distaste for duplication pays off, then :)
In case this hunk was confusing: Boris was providing another reason to reject one of his suggested alternative patches, and I was agreeing with him.
Stay tuned.
Brian
P.S. Richard and I had some discussion on IRC, and I think there was a rough agreement that the whole reboot handler dance really doesn't belong (exclusively) in the MTD layer. Your patch was inspired by problems with UBI, and (at least for non-initrd cases) we think UBI should probably learn how to clean up after itself before we reboot. So in the long term, there may be an attempt to fix up UBI and drop the MTD reboot handlers. But that's probably not going to happen today.
Quoted 7 lines · Brian Norris
Quoted 3 lines · Andrew E. Mileski
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
Which driver?
Sorry, I wasn't specific enough: fsl_elbc_nand.c
Currently using the 4.2.0 Linux kernel version, and the problem persists.
I've actually made a small hack to it to enable software ECC, but it isn't fit for distribution.
The stock driver is rather ugly in my opinion, and makes some overly broad assumptions about the hardware; the driver really is only intended to support hardware ECC. I've taken a few stabs at cleaning-up the stock driver, but they all end-up being complete re-writes (for multi-die support), which is a bit more than I can chew.
The driver otherwise is usable as-is, albeit with only hardware ECC (that is insufficient for many modern NAND devices), and exhibits the same problem.
Quoted 39 lines
Quoted 2 lines
It appears to be a hang in mtd/nand/nand_base.c:nand_get_device() waiting on
chip->controller->active.And now that I begin describing the problem and grepping through the source logs... I see that this problem was already resolved back in 2009, except for the
FL_PM_SUSPENDEDmode:commit
6b0d9a84124937f048bcb8b21313152b23063978Author: Li Yang <[email protected]> Date: Tue Nov 17 14:45:49 2009 -0800mtd: nand: fix multi-chip suspend problem
I actually don't see why we can't just use
nand_get_device(FL_PM_SUSPENDED)for the shutdown/reboot case, like this:drivers/mtd/nand/nand_base.c +1
-1@@ -3110,7 +3110,7 @@ static void nand_resume(struct mtd_info *mtd)*/ static void nand_shutdown (struct mtd_info *mtd) { -nand_get_device (mtd,FL_SHUTDOWN ); +nand_get_device (mtd,FL_PM_SUSPENDED ); }/* Set default functions */ It's also possible that this could be better solved in a proper refactor/rewrite of the NAND subsystem using a better controller/chip split, so there's only one reboot handler per NAND controller. Boris has been looking at that.
Quoted 4 lines
Anyway, if the above looks OK, I can send a proper patch and get your 'Tested-by's.
Brian
That seems to work too! Thanks!
~~Andrew E. Mileski
Quoted 22 lines · Andrew E. Mileski
Quoted 6 lines · Brian Norris
Quoted 3 lines · Andrew E. Mileski
I'm experiencing a hang on reboot with a Freescale P1022 PowerPC system, with a dual chip-select NAND part (specified in the device tree as two separate devices), and kernel v4.0.6.
Which driver?
Sorry, I wasn't specific enough:
fsl_elbc_nand.cCurrently using the 4.2.0 Linux kernel version, and the problem persists.
I've actually made a small hack to it to enable software ECC, but it isn't fit for distribution.
The stock driver is rather ugly in my opinion, and makes some overly broad assumptions about the hardware; the driver really is only intended to support hardware ECC. I've taken a few stabs at cleaning-up the stock driver, but they all end-up being complete re-writes (for multi-die support), which is a bit more than I can chew.
Quoted 3 lines
The driver otherwise is usable as-is, albeit with only hardware ECC (that is insufficient for many modern NAND devices), and exhibits the same problem.
Yikes, I wish I hadn't actually taken a closer look at that driver :( It could really use some rewrites for proper multi-chip/multi-die support.
...
Quoted 6 lines
Quoted 4 lines
Anyway, if the above looks OK, I can send a proper patch and get your 'Tested-by's.
Brian
That seems to work too! Thanks!
Great! I'll review Boris's suggestions to see if there's a better option.
Brian