scsi: lpfc: Add shutdown method for kexec
Subsystems (2)
PATCH 1/1 scsi: lpfc: Add shutdown method for kexec
From: Anton Blanchard <[email protected]>
We see lpfc devices regularly fail during kexec. Fix this by adding a shutdown method which mirrors the remove method.
--- drivers/scsi/lpfc/lpfc_init.c | 1 + 1 file changed, 1 insertion(+)
drivers/scsi/lpfc/lpfc_init.c +1 -0
@@ -11447,6 +11447,7 @@ static struct pci_driver lpfc_driver = { .id_table = lpfc_id_table, .probe = lpfc_pci_probe_one, .remove = lpfc_pci_remove_one, + .shutdown = lpfc_pci_remove_one, .suspend = lpfc_pci_suspend_one, .resume = lpfc_pci_resume_one, .err_handler = &lpfc_err_handler,
Signature · 1 line
2.9.3
Quoted 4 lines · Anton Blanchard
From: Anton Blanchard <[email protected]>
We see lpfc devices regularly fail during kexec. Fix this by adding a shutdown method which mirrors the remove method.
Or instead finally do what I've been advocating for years (and even sent patches for) which is to have kexec call remove instead of shutdown.
Shutdown is and has *always* been the wrong thing to do.
Quoted 18 lines
Signed-off-by Anton Blanchard <[email protected]>---
drivers/scsi/lpfc/lpfc_init.c | 1 +1 file changed, 1 insertion(+)drivers/scsi/lpfc/lpfc_init.c +1
-0@@ -11447,6 +11447,7 @@ static struct pci_driver lpfc_driver = { .id_table = lpfc_id_table, .probe = lpfc_pci_probe_one, .remove = lpfc_pci_remove_one, + .shutdown = lpfc_pci_remove_one, .suspend = lpfc_pci_suspend_one, .resume = lpfc_pci_resume_one, .err_handler = &lpfc_err_handler,
Hi Ben,
Quoted 11 lines · On Mon, 13 Feb 2017 10:14:54 +1100 Benjamin Herrenschmidt…
Quoted 4 lines · Anton Blanchard
From: Anton Blanchard <[email protected]>
We see lpfc devices regularly fail during kexec. Fix this by adding a shutdown method which mirrors the remove method.
Or instead finally do what I've been advocating for years (and even sent patches for) which is to have kexec call remove instead of shutdown.
Shutdown is and has *always* been the wrong thing to do.
Good point, at the very least we should call remove if shutdown doesn't exist. Eric: could we make the changes Ben suggests?
Anton
Quoted 19 lines
Quoted 18 lines
Signed-off-by Anton Blanchard <[email protected]>---
drivers/scsi/lpfc/lpfc_init.c | 1 +1 file changed, 1 insertion(+)drivers/scsi/lpfc/lpfc_init.c +1
-0@@ -11447,6 +11447,7 @@ static struct pci_driver lpfc_driver = { .id_table = lpfc_id_table, .probe = lpfc_pci_probe_one, .remove = lpfc_pci_remove_one, + .shutdown = lpfc_pci_remove_one, .suspend = lpfc_pci_suspend_one, .resume = lpfc_pci_resume_one, .err_handler = &lpfc_err_handler,
Quoted 14 lines · Anton Blanchard
Hi Ben,
Quoted 9 lines · On Mon, 13 Feb 2017 10:14:54 +1100 Benjamin Herrenschmidt…
Quoted 4 lines · Anton Blanchard
From: Anton Blanchard <[email protected]>
We see lpfc devices regularly fail during kexec. Fix this by adding a shutdown method which mirrors the remove method.
Or instead finally do what I've been advocating for years (and even sent patches for) which is to have kexec call remove instead of shutdown.
Ben what happened to those patches. I don't remember them?
Quoted 1 line
Quoted 1 line
Shutdown is and has *always* been the wrong thing to do.
I won't argue that.
Quoted 2 lines
Good point, at the very least we should call remove if shutdown doesn't exist. Eric: could we make the changes Ben suggests?
Definitely. That was the original design of the kexec interface but people were worried about calling remove during reboot might introduce regressions on the reboot functionality. So shutdown was added to be remove without the cleaning up the kernel data structures.
I am all for changing the core to call remove. That seems to be a more tested code path (because remove tends to be part of the development path for modules) and thus much more likely to work in practice.
The challenge with changes in this area is that when the infrastructure is changed for everyone someone needs to baby sit it until all of the unexpected issues are resolved. I was hoping a couple of years ago that Ben could be that person.
Eric
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines
Good point, at the very least we should call remove if shutdown doesn't exist. Eric: could we make the changes Ben suggests?
Definitely. That was the original design of the kexec interface but people were worried about calling remove during reboot might introduce regressions on the reboot functionality. So shutdown was added to be remove without the cleaning up the kernel data structures.
Right. Part of the problem was also that remove was dependent on CONFIG_HOTPLUG though that's no longer the case anymore.
The problem is that a bunch of drivers either don't have a shutdown or worse, have one that actually shuts the HW down rather than "idle" it which puts it in a state that the new kernel can't deal with.
Quoted 8 lines
I am all for changing the core to call remove. That seems to be a more tested code path (because remove tends to be part of the development path for modules) and thus much more likely to work in practice.
The challenge with changes in this area is that when the infrastructure is changed for everyone someone needs to baby sit it until all of the unexpected issues are resolved. I was hoping a couple of years ago that Ben could be that person.
Correct. And I lack time. But since we are a platform that uses kexec daily as our main booting mechanism we should probably be the ones driving that.
I had a patch at least to fallback to remove in absence of shutdown which I can try to dig out.
We can add a config option to make it do the other way around that we should start testing internally. I'm sure we will find 1 or 2 regressions as drivers do weird things.
Cheers,
Ben.
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines
Good point, at the very least we should call remove if shutdown doesn't exist. Eric: could we make the changes Ben suggests?
Definitely. That was the original design of the kexec interface but people were worried about calling remove during reboot might introduce regressions on the reboot functionality. So shutdown was added to be remove without the cleaning up the kernel data structures.
Quoted 6 lines
Right. Part of the problem was also that remove was dependent on
CONFIG_HOTPLUGthough that's no longer the case anymore.The problem is that a bunch of drivers either don't have a shutdown or worse, have one that actually shuts the HW down rather than "idle" it which puts it in a state that the new kernel can't deal with.
If we do transition to use remove rather than shutdown, I think we want some way for a device driver to know whether we are doing kexec or not. A RAID adapter with a write cache is going to want to flush its write cache on a PCI hotplug remove, but for a kexec, its going to want to skip that so the kexec is faster. Today, since kexec looks like a reboot, rather than a shutdown, we can skip the flush on a reboot, since its technically not needed there either.
-Brian
Signature · 3 lines
Brian King Power Linux I/O IBM Linux Technology Center
Quoted 17 lines · Brian King
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 7 lines · Eric W. Biederman
Quoted 2 lines
Good point, at the very least we should call remove if shutdown doesn't exist. Eric: could we make the changes Ben suggests?
Definitely. That was the original design of the kexec interface but people were worried about calling remove during reboot might introduce regressions on the reboot functionality. So shutdown was added to be remove without the cleaning up the kernel data structures.
Quoted 6 lines
Right. Part of the problem was also that remove was dependent on
CONFIG_HOTPLUGthough that's no longer the case anymore.The problem is that a bunch of drivers either don't have a shutdown or worse, have one that actually shuts the HW down rather than "idle" it which puts it in a state that the new kernel can't deal with.
Quoted 7 lines
If we do transition to use remove rather than shutdown, I think we want some way for a device driver to know whether we are doing kexec or not. A RAID adapter with a write cache is going to want to flush its write cache on a PCI hotplug remove, but for a kexec, its going to want to skip that so the kexec is faster. Today, since kexec looks like a reboot, rather than a shutdown, we can skip the flush on a reboot, since its technically not needed there either.
This doesn't make any sense.
With a PCI hotplug remove you can't do anything because by the time you get the event the hardware is gone. I think there are optional interlocks but nothing as good as the old 3.5" floppy disk button, or the button on CD drives.
The only difference ever that should exist between shutdown and remove is do you clean up kernel data structures. The shutdown method is allowed to skip the cleanup up kernel data structures that the remove method needs to make.
Assuming the kernel does not have corrupted data structures I don't see any practical reason for distinguishing between the two. There may be some real world gotchas we have to deal with but semantically I don't see any concerns.
Eric
Quoted 9 lines · Eric W. Biederman
The only difference ever that should exist between shutdown and remove is do you clean up kernel data structures. The shutdown method is allowed to skip the cleanup up kernel data structures that the remove method needs to make.
Assuming the kernel does not have corrupted data structures I don't see any practical reason for distinguishing between the two. There may be some real world gotchas we have to deal with but semantically I don't see any concerns.
We had historical additions to shutdown actually shutting things down, like spinning platters down to park disk heads, switching devices into D3 on some systems, etc... that remove never had (and we don't want for kexec).
Cheers,
Ben.
Quoted 10 lines · Brian King
If we do transition to use remove rather than shutdown, I think we want
some way for a device driver to know whether we are doing kexec or not.
A RAID adapter with a write cache is going to want to flush its write cache on a PCI hotplug remove, but for a kexec, its going to want to skip
that so the kexec is faster. Today, since kexec looks like a reboot, rather than a shutdown, we can skip the flush on a reboot, since its technically not needed there either.
What happens if a non-flushed adapter gets a PERST ?
I wouldn't trust that 'don't have to flush' magic ...
Cheers,
Ben.
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 10 lines · Brian King
If we do transition to use remove rather than shutdown, I think we want
some way for a device driver to know whether we are doing kexec or not.
A RAID adapter with a write cache is going to want to flush its write cache on a PCI hotplug remove, but for a kexec, its going to want to skip
that so the kexec is faster. Today, since kexec looks like a reboot, rather than a shutdown, we can skip the flush on a reboot, since its technically not needed there either.
Quoted 1 line
What happens if a non-flushed adapter gets a PERST ?
It depends on the adapter, so it really needs to be a policy in the device driver. For any adapter that has a non volatile write cache, data must preserved, otherwise the write cache is not truly non volatile. For adapters with a volatile write cache, then its subject to the adapter hardware. For ipr adapters that have a volatile cache, they do guarantee the data is preserved across a PERST.
Quoted 1 line
I wouldn't trust that 'don't have to flush' magic ...
I really don't think we want to force RAID adapters that have gigabytes of *non volatile* write cache to flush their cache when we are merely performing a kexec. This can take several minutes in the worst case scenarios.
-Brian
Signature · 3 lines
Brian King Power Linux I/O IBM Linux Technology Center
Quoted 22 lines · Brian King
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 10 lines · Brian King
If we do transition to use remove rather than shutdown, I think we want
some way for a device driver to know whether we are doing kexec or not.
A RAID adapter with a write cache is going to want to flush its write cache on a PCI hotplug remove, but for a kexec, its going to want to skip
that so the kexec is faster. Today, since kexec looks like a reboot, rather than a shutdown, we can skip the flush on a reboot, since its technically not needed there either.Quoted 1 line
What happens if a non-flushed adapter gets a PERST ?
It depends on the adapter, so it really needs to be a policy in the device driver. For any adapter that has a non volatile write cache, data must preserved, otherwise the write cache is not truly non volatile. For adapters with a volatile write cache, then its subject to the adapter hardware. For ipr adapters that have a volatile cache, they do guarantee the data is preserved across a PERST.
Quoted 5 lines
Quoted 1 line
I wouldn't trust that 'don't have to flush' magic ...
I really don't think we want to force RAID adapters that have gigabytes of *non volatile* write cache to flush their cache when we are merely performing a kexec. This can take several minutes in the worst case scenarios.
It is very very simple. kexec does not guarantee what comes after it.
An optimization that skips flushing because the hardware write cache can be trusted is fine. An optimization that skips flushing because some later kernel will take care of it is not.
I really don't see any room in there for distinguishing between which kind of shutdown and/or kexec we are doing. Either the hardware handles it or it doesn't.
If the hardware doesn't handle everything than we need to handle it in software.
Eric
Quoted 2 lines · Anton Blanchard
We see lpfc devices regularly fail during kexec. Fix this by adding a shutdown method which mirrors the remove method.
@mkp, @jejb: may you please flag this patch for stable? Thank you.
This patch fixes a problem of some adapter ports going offline after reboot on the following adapter, at least:
[10df:f100] Saturn-X: LightPulse Fibre Channel Host Adapter
# dmesg | grep lpfc
...
[ 61.676728] lpfc 0003:12:00.0: 6:0442 Adapter failed to init,
mbxCmd x88 CONFIG_PORT, mbxStatus x0 Data: x0
[ 92.766717] lpfc 0003:12:00.0: 6:0442 Adapter failed to init,
mbxCmd x88 CONFIG_PORT, mbxStatus x0 Data: x0
[ 123.856717] lpfc 0003:12:00.0: 6:0442 Adapter failed to init, mbxCmd x88 CONFIG_PORT, mbxStatus x0 Data: x0
...
Signature · 2 lines
Mauricio Faria de Oliveira IBM Linux Technology Center
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Mauricio" == Mauricio Faria de Oliveira <[email protected]> writes:
Quoted 2 lines · Anton Blanchard
Quoted 2 lines
We see lpfc devices regularly fail during kexec. Fix this by adding a shutdown method which mirrors the remove method.
Mauricio> @mkp, @jejb: may you please flag this patch for stable? Thank Mauricio> you.
I don't recall a consensus being reached on this patch.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 8 lines · Martin K. Petersen
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
"Mauricio" == Mauricio Faria de Oliveira <[email protected] et.ibm.com> writes:
Quoted 3 lines · Anton Blanchard
Quoted 3 lines
We see lpfc devices regularly fail during kexec. Fix this by adding a
shutdown method which mirrors the remove method.
Quoted 5 lines
Mauricio> @mkp, @jejb: may you please flag this patch for stable? Thank
Mauricio> you.I don't recall a consensus being reached on this patch.
What would be the opposition ? Without it kexec breaks. With it, it works ...
Now we all seem to agree that kexec should be overhauled to not use shutdown but instead unbind drivers, but that's a more long term project that nobody yet had a chance to tackle.
In the meantime, some systems need a functioning kexec to boot.
Cheers,
Ben.
Martin,
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 1 line · Martin K. Petersen
I don't recall a consensus being reached on this patch.
What would be the opposition ? Without it kexec breaks. With it, it works ...
That is the argument I'd present/ask for consideration too.
I think I should have included this in the tested-by tag email, for documentation/evidence: no regression observed in system shutdown path.
Thanks,
Signature · 2 lines
Mauricio Faria de Oliveira IBM Linux Technology Center
Martin K. Petersen
AppliedQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Mauricio" == Mauricio Faria de Oliveira <[email protected]> writes:
Mauricio> I think I should have included this in the tested-by tag Mauricio> email, for documentation/evidence: no regression observed in Mauricio> system shutdown path.
Applied to 4.11/scsi-fixes.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering