Why does kexec use device_shutdown rather than ubind them
Subsystems (2)
Hi Folks !
Sorry for the semi-random CC list, not sure who owns kexec nowadays. So we are working on a new crop of power servers for which the bootloader is going to be using kexec.
As expected, we've been chasing a number of reliability issues mostly due to drivers not behaving properly, such as leaving devices DMA'ing or in a state that upsets the new kernel etc...
So far our approach has been to fix the drivers one by one, adding the shutdown() method when it's missing, etc...
But that lead me to wonder ... why shutdown() in the first place ? The semantic of shutdown() is that we are going to power the machine off. In some cases, that method will actively participate in the shutdown, powering things off, spinning disk down, etc....
It doesn't have the semantic of "put the device into a clean state for a new driver to pick up". It's also rarely implemented.
On the other hand, the remove() routine is almost everywhere, and is already well understood as needing to leave the device in a clean state, as it's often used for rmmod (often by the driver developer him/herself), more likely to be tested in a condition that doesn't involve having the machine off immediately afterward but on the contrare in a condition where a new driver can come and try to pick the device up.
Additionally, remove() is also what KVM does when assigning devices to guest, ie, the original driver is unbound from the host, and VFIO is bound in its place.
So we have common purpose with kexec (somewhat) and possibly common (and better) testing coverage with remove() than with shutdown().
I plan to experiment a bit in our bootloader see if that makes a difference, maybe doing a first pass of unbind for anything that can be unbound, and shutdown for the rest.
(I'll probably also sneak it a PCIe hot reset at the end but that's more platform specific).
Any opinion ?
Cheers,
Ben.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 36 lines · Benjamin Herrenschmidt
Hi Folks !
Sorry for the semi-random CC list, not sure who owns kexec nowadays. So we are working on a new crop of power servers for which the bootloader is going to be using kexec.
As expected, we've been chasing a number of reliability issues mostly due to drivers not behaving properly, such as leaving devices DMA'ing or in a state that upsets the new kernel etc...
So far our approach has been to fix the drivers one by one, adding the
shutdown()method when it's missing, etc...But that lead me to wonder ... why
shutdown()in the first place ? The semantic ofshutdown()is that we are going to power the machine off. In some cases, that method will actively participate in the shutdown, powering things off, spinning disk down, etc....It doesn't have the semantic of "put the device into a clean state for a new driver to pick up". It's also rarely implemented.
On the other hand, the
remove()routine is almost everywhere, and is already well understood as needing to leave the device in a clean state, as it's often used for rmmod (often by the driver developer him/herself), more likely to be tested in a condition that doesn't involve having the machine off immediately afterward but on the contrare in a condition where a new driver can come and try to pick the device up.Additionally,
remove()is also what KVM does when assigning devices to guest, ie, the original driver is unbound from the host, and VFIO is bound in its place.So we have common purpose with kexec (somewhat) and possibly common (and better) testing coverage with
remove()than withshutdown().
Quoted 8 lines
I plan to experiment a bit in our bootloader see if that makes a difference, maybe doing a first pass of unbind for anything that can be unbound, and shutdown for the rest.
(I'll probably also sneak it a PCIe hot reset at the end but that's more platform specific).
Any opinion ?
When I was young and kexec was a new idea and the device model was just getting methods. An argument was made that using remove in the reboot path was a bad idea because it does more than is necessary so we needed shutdown.
I wanted to use remove at the time.
There is a lot of silliness now, but my inclination is to remove shutdown entirely and replace it with remove. I certainly would not have heart burn with doing so just for kexec but getting the whole reboot path at this point would make me very happy and then we could kill a little used rarely implemented method from the kernel.
Given that we have found emperically that flipping the bus master enable bit to off helps enormously for stability, but doing so generically occassionally has unfortunate consequences why not.
I think we have largely survied until now because kdump is so popular and kdump winds up having to reinitialize devices from any random state.
But like I said I am all for reducing the burden on device driver developers.
Eric
Quoted 4 lines · Eric W. Biederman
I think we have largely survied until now because kdump is so popular and kdump winds up having to reinitialize devices from any random state.
kdump also doesn't care too much if the device is still DMA'ing to the old kernel memory :-)
Quoted 2 lines
But like I said I am all for reducing the burden on device driver developers.
Right. I'm experimenting with a variant of device_shutdown() that tries remove() first and if it doesn't exist and shutdown() does, call that (is that ever the case ?). I'm keeping this kexec-specific for now.
I'll try to hammer that on some of our machines see if it breaks anything, I think it's a much better approach for kexec.
As for actual machine shutdown, we *might* have some corner cases where shutdown is actually different from remove for good reasons, so that will have to be investigated a bit more in depth.
I'll post my results when I have them.
Cheers,
Ben.
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 4 lines · Eric W. Biederman
I think we have largely survied until now because kdump is so popular and kdump winds up having to reinitialize devices from any random state.
kdump also doesn't care too much if the device is still DMA'ing to the old kernel memory :-)
In principle kdump does not care about ongoing DMAs but in practice it is giving us some headaches with IOMMU. Various kind of issues crop up during IOMMU intialization in second kernel while DMA is ongoing and unfortunately no good solution has made into upstream yet.
Well, ongoing DMA and IOMMU seems to be orthogonal to using ->remove() in kexec. So I will stop here. :-)
Thanks
Vivek
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Vivek, Ben, Eric,
Please take a look at a proposed patch to intel-iommu: "[PATCHv3 0/6] Crashdump Accepting Active IOMMU"
This is specifically for kdump; however, would some small variation of this technique be applicable to kexec ?
Thanks,
Bill
-----Original Message-----
From: kexec [mailto:[email protected]] On Behalf Of Vivek Goyal
Sent: Friday, January 17, 2014 8:13 AM
To: Benjamin Herrenschmidt
Cc: Matthew Garrett; [email protected]; [email protected]; Eric W. Biederman; Andrew Morton; Linus Torvalds
Subject: Re: [Q] Why does kexec use device_shutdown rather than ubind them
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 4 lines · Eric W. Biederman
I think we have largely survied until now because kdump is so popular and kdump winds up having to reinitialize devices from any random state.
kdump also doesn't care too much if the device is still DMA'ing to the old kernel memory :-)
In principle kdump does not care about ongoing DMAs but in practice it is giving us some headaches with IOMMU. Various kind of issues crop up during IOMMU intialization in second kernel while DMA is ongoing and unfortunately no good solution has made into upstream yet.
Well, ongoing DMA and IOMMU seems to be orthogonal to using ->remove() in kexec. So I will stop here. :-)
Thanks
Vivek
_______________________________________________
kexec mailing list [email protected]
http://lists.infradead.org/mailman/listinfo/kexec
Quoted 10 lines · Vivek Goyal
Quoted 8 lines · Benjamin Herrenschmidt
Quoted 4 lines · Eric W. Biederman
I think we have largely survied until now because kdump is so popular and kdump winds up having to reinitialize devices from any random state.
kdump also doesn't care too much if the device is still DMA'ing to the old kernel memory :-)
Quoted 7 lines
In principle kdump does not care about ongoing DMAs but in practice it is giving us some headaches with IOMMU. Various kind of issues crop up during IOMMU intialization in second kernel while DMA is ongoing and unfortunately no good solution has made into upstream yet.
Well, ongoing DMA and IOMMU seems to be orthogonal to using ->remove() in kexec. So I will stop here. :-)
Right, it's an orthogonal problem. I think hot resetting the bus might solve it too though. It's even worse on ppc because the resulting iommu errors trigger those "EEH freeze" that we have here blocking the devices out etc...
Ben.
Quoted 7 lines
Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec