In place kexec

We are getting a claim that the qla driver corrupts memory after a kexec, apparently due to a DMA engine left running in the before-kernel.

For an in-place kexec (as opposed to a crash dump kexec, where we switch into dedicated memory), what shutdown paths get executed?

Quoted 3 lines · H. Peter Anvin

We are getting a claim that the qla driver corrupts memory after a kexec, apparently due to a DMA engine left running in the before-kernel.

Quoted 2 lines

For an in-place kexec (as opposed to a crash dump kexec, where we switch into dedicated memory), what shutdown paths get executed?

It is the normal reboot path, so the device shutdown method gets executed.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Quoted 11 lines · Eric W. Biederman

Quoted 5 lines · H. Peter Anvin

We are getting a claim that the qla driver corrupts memory after a kexec, apparently due to a DMA engine left running in the before-kernel.

For an in-place kexec (as opposed to a crash dump kexec, where we switch into dedicated memory), what shutdown paths get executed?

It is the normal reboot path, so the device shutdown method gets executed.

Quoted 2 lines

Eric

Check your iommu. We've had lots of problems with them in the past, and in the crash path we explicity leave the iommu on now, whereas the normal shutdown path turns it off. If some other dma-capable device doesn't shut down properly and keeps dma operations going, you're liable to get memory corruption when the iommu re-initalizes.

Quoted 12 lines · Neil Horman

Quoted 11 lines · Eric W. Biederman

Quoted 5 lines · H. Peter Anvin

We are getting a claim that the qla driver corrupts memory after a kexec, apparently due to a DMA engine left running in the before-kernel.

For an in-place kexec (as opposed to a crash dump kexec, where we switch into dedicated memory), what shutdown paths get executed?

It is the normal reboot path, so the device shutdown method gets executed.

Quoted 8 lines

Quoted 2 lines

Eric

Check your iommu. We've had lots of problems with them in the past, and in the crash path we explicity leave the iommu on now, whereas the normal shutdown path turns it off. If some other dma-capable device doesn't shut down properly and keeps dma operations going, you're liable to get memory corruption when the iommu re-initalizes.

Sorry, can we keep the discussions of kexec-on-crash and kexec-in-place clearly separated, please? The qla driver issue is supposed to be kexec-in-place, and it sounds like you're talking about kexec-on-crash.

-hpa

Signature · 2 lines

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Quoted 5 lines

Sorry, can we keep the discussions of kexec-on-crash and kexec-in-place clearly separated, please? The qla driver issue is supposed to be kexec-in-place, and it sounds like you're talking about kexec-on-crash.

No, I'm just indicating a difference between the two paths, and I'm doing so because we used to have simmilar dma problems in the crash path, which we resolved by not turning of the iommu during shutdown, which is different from the in-place path. Just trying to give you some thoughts about where to look

Quoted 5 lines · Neil Horman

Quoted 5 lines

Sorry, can we keep the discussions of kexec-on-crash and kexec-in-place clearly separated, please? The qla driver issue is supposed to be kexec-in-place, and it sounds like you're talking about kexec-on-crash.

Quoted 4 lines

No, I'm just indicating a difference between the two paths, and I'm doing so because we used to have simmilar dma problems in the crash path, which we resolved by not turning of the iommu during shutdown, which is different from the in-place path. Just trying to give you some thoughts about where to look

Fair enough... just wanted to flag this as a problem, because it has already been the source of a lot of confusion.

Thinking about this I am a bit surprised that you would find DMA left on from a disk driver. Historically disks have been pretty good about shutting off in this scenario.

Added to that typically we unmount all filesystems.

Calling rmmod on the driver before the final kexec --exec could be interesting, and drivers much more reliably implement .remove than .shutdown.

Network drivers are more likely to be a problem, but we should be downing all of the network interfaces before something happens.

All of which is to say kexec-in-place has generally been a lot less hassle, because it is so similar to the normal case.

Quoted 14 lines · Eric W. Biederman

Thinking about this I am a bit surprised that you would find DMA left on from a disk driver. Historically disks have been pretty good about shutting off in this scenario.

Added to that typically we unmount all filesystems.

Calling rmmod on the driver before the final kexec --exec could be interesting, and drivers much more reliably implement .remove than .shutdown.

Network drivers are more likely to be a problem, but we should be downing all of the network interfaces before something happens.

Quoted 3 lines

All of which is to say kexec-in-place has generally been a lot less hassle, because it is so similar to the normal case.

In particular, the supposed corruption comes from the "firmware logging" feature in the qla2xxx driver. I'd really like to understand if this is a kexec problem or a qla2xxx problem.

Quoted 15 lines · H. Peter Anvin

Quoted 14 lines · Eric W. Biederman

Thinking about this I am a bit surprised that you would find DMA left on from a disk driver. Historically disks have been pretty good about shutting off in this scenario.

Added to that typically we unmount all filesystems.

Calling rmmod on the driver before the final kexec --exec could be interesting, and drivers much more reliably implement .remove than .shutdown.

Network drivers are more likely to be a problem, but we should be downing all of the network interfaces before something happens.

Quoted 8 lines

Quoted 3 lines

All of which is to say kexec-in-place has generally been a lot less hassle, because it is so similar to the normal case.

In particular, the supposed corruption comes from the "firmware logging" feature in the qla2xxx driver. I'd really like to understand if this is a kexec problem or a qla2xxx problem.

kernel_kexec()
kernel_restart_prepare()
device_shutdown()

I would suspect it to be a qla2xxx driver problem that it did not shut down the device properly.

Quoted 29 lines · Vivek Goyal

Quoted 15 lines · H. Peter Anvin

Quoted 14 lines · Eric W. Biederman

Thinking about this I am a bit surprised that you would find DMA left on from a disk driver. Historically disks have been pretty good about shutting off in this scenario.

Added to that typically we unmount all filesystems.

Calling rmmod on the driver before the final kexec --exec could be interesting, and drivers much more reliably implement .remove than .shutdown.

Network drivers are more likely to be a problem, but we should be downing all of the network interfaces before something happens.

Quoted 8 lines

Quoted 3 lines

All of which is to say kexec-in-place has generally been a lot less hassle, because it is so similar to the normal case.

In particular, the supposed corruption comes from the "firmware logging" feature in the qla2xxx driver. I'd really like to understand if this is a kexec problem or a qla2xxx problem.

kernel_kexec()
kernel_restart_prepare()
device_shutdown()

Quoted 2 lines

I would suspect it to be a qla2xxx driver problem that it did not shut down the device properly.

And device_shutdown calls every drivers .shutdown method.

Things like this are always a driver problem.

Quoted 4 lines

And device_shutdown calls every drivers .shutdown method.

Things like this are always a driver problem.

so is there a default .shutdown method for drivers that do not specify one?

like the qla2xxx driver does not.

Quoted 5 lines · Randy Dunlap

Quoted 4 lines

And device_shutdown calls every drivers .shutdown method.

Things like this are always a driver problem.

Quoted 4 lines

so is there a default .shutdown method for drivers that do not specify one?

like the qla2xxx driver does not.

If it doesn't, even if bus mastering gets shut off at the core level, there is a risk that is clobbers data when it turns it back on if the initialization sequence is problematic.

There isn't a bus master shut off at the core level. When we did the original analysis it turned out that the bus mastering bit was implemented on a lot of devices in advisory way, so it didn't make sense to count on it.

That said it looks like the code to do the shutdown is in qla2x00_remove_one so it should be too hard if someone cared to extract just the hardware bits.

Quoted 6 lines · Eric W. Biederman

There isn't a bus master shut off at the core level. When we did the original analysis it turned out that the bus mastering bit was implemented on a lot of devices in advisory way, so it didn't make sense to count on it.

But does it make sense to not flip the bit for the cases where it is implemented properly?

Quoted 3 lines

That said it looks like the code to do the shutdown is in qla2x00_remove_one so it should be too hard if someone cared to extract just the hardware bits.

Charming. Code is there, just not hooked up.

Quoted 6 lines

Quoted 4 lines

That said it looks like the code to do the shutdown is in qla2x00_remove_one so it should be too hard if someone cared to extract just the hardware bits.

Charming. Code is there, just not hooked up.

Using the .remove method in reboot is a fight a lost long ago.

Quoted 5 lines · Eric W. Biederman

Quoted 5 lines

Quoted 4 lines

That said it looks like the code to do the shutdown is in qla2x00_remove_one so it should be too hard if someone cared to extract just the hardware bits.

Quoted 4 lines

Quoted 1 line

Charming. Code is there, just not hooked up.

Using the .remove method in reboot is a fight a lost long ago.

Could you elucidate, please?

Quoted 6 lines · H. Peter Anvin

Quoted 5 lines · Eric W. Biederman

Quoted 5 lines

Quoted 4 lines

That said it looks like the code to do the shutdown is in qla2x00_remove_one so it should be too hard if someone cared to extract just the hardware bits.

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

Charming. Code is there, just not hooked up.

Using the .remove method in reboot is a fight a lost long ago.

Could you elucidate, please?

My original proposal was for device_shutdown to call the .remove methods as those are well exercised and tested in development. aka rmmod.

It was argued (with some merit) that for a system reboot we don't want to perform all of the subsystem registration work, to make it more likely that reboot -f will reboot even if there is a kernel oops.

What I proposed and unfortunately failed to write the patch for at the time is was to have the device remove path call shutdown before calling remove, so drivers wouldn't have to code it all up twice.

A lot of the disk drivers implement .shutdown these days and there aren't may bug reports about kexec failing. So I would be reluctant to change things other than on a driver by driver basis unless I had a lot of time for testing etc.

It might be worth playing with adding a pci_clear_master in pci_device_shutdown. It has the potential to break things like usb keyboards, so I would be careful. If it doesn't break fundamental things like usb a pci_clear_master when shutting down devices should improve reliability somewhat.

And of course there is the old staple of work arounds: "rmmod <driver>" before calling kexec --exec.

Looking through all these emails, what's the upshot here? Is the expectation, for all storage drivers to starting to implement some 'minimal' level of shutdown with the hardware/firmware during the .shutdown callback?

Quoted 6 lines · Andrew Vasquez

Looking through all these emails, what's the upshot here? Is the expectation, for all storage drivers to starting to implement some 'minimal' level of shutdown with the hardware/firmware during the .shutdown callback?

I believe so. It seems to be a fundamental requirement for kexec to function.

There isn't a bus master shut off at the core level.

Quoted 1 line · Eric W. Biederman

There isn't a bus master shut off at the core level.

Effectively, there is if you have an IOMMU.

Quoted 4 lines · David Woodhouse

Quoted 1 line · Eric W. Biederman

There isn't a bus master shut off at the core level.

Effectively, there is if you have an IOMMU.

Depends on the IOMMU. There are several dinky IOMMUs that when you shut them off DMA simply goes around them, and is not stopped.

Quoted 5 lines · Eric W. Biederman

Quoted 3 lines · David Woodhouse

Quoted 1 line · Eric W. Biederman

There isn't a bus master shut off at the core level.

Quoted 4 lines

Quoted 1 line

Effectively, there is if you have an IOMMU.

Depends on the IOMMU. There are several dinky IOMMUs that when you shut them off DMA simply goes around them, and is not stopped.

I think last time we were discussing this for AMD IOMMU where if you disable IOMMU, it just kind of become pass through with 1:1 mapping of addresses.

Quoted 11 lines · Vivek Goyal

Quoted 9 lines · Eric W. Biederman

Quoted 4 lines · David Woodhouse

Quoted 1 line · Eric W. Biederman

There isn't a bus master shut off at the core level.

Effectively, there is if you have an IOMMU.

Depends on the IOMMU. There are several dinky IOMMUs that when you shut them off DMA simply goes around them, and is not stopped.

Quoted 3 lines

I think last time we were discussing this for AMD IOMMU where if you disable IOMMU, it just kind of become pass through with 1:1 mapping of addresses.

Yeah, don't do that. The IOMMU should be *on*, but without any active mappings set up. Which is exactly how Linux will set it up at boot.

So what happens if we tear down the mapping while DMA is on.

Quoted 1 line · Vivek Goyal

So what happens if we tear down the mapping while DMA is on.

The DMA gets blocked, and you don't have to worry about whether the device was shut down cleanly or not. The device may be unhappy, but when the new kernel's driver loads and reinitialises it, all should be forgiven.

Quoted 3 lines · David Woodhouse

The DMA gets blocked, and you don't have to worry about whether the device was shut down cleanly or not. The device may be unhappy, but when the new kernel's driver loads and reinitialises it, all should be forgiven.

Assuming IOMMU page faults don't cause pain. I seem to remember that also being a nasty issue.

Quoted 7 lines · Eric W. Biederman

Quoted 3 lines · David Woodhouse

The DMA gets blocked, and you don't have to worry about whether the device was shut down cleanly or not. The device may be unhappy, but when the new kernel's driver loads and reinitialises it, all should be forgiven.

Assuming IOMMU page faults don't cause pain. I seem to remember that also being a nasty issue.

Only if the driver (or the hardware) is so broken that it can't reccover. There's very little excuse for a driver to have that problem even at runtime (and fail to recover from such an error)... for a driver to fail to initialise the hardware even when that driver is first being loaded is *entirely* fucked.

Not that it doesn't happen, of course. But do we care? I lump those broken drivers is the same class as the ones which only work after a warm start from Windows or Mac OS.

The issue is what happens if you take an IOMMU page fault during between shutdown and restart. I seem to remember an IOMMU page fault triggering a machine check on AMD cpus. So maybe it works but my gut impression is simply leaving the IOMMU in a state that is on but not responding could actually make a reboot or kexec less stable than having on-going DMAs stomping on memory. If you can leave it on, without translations and not trapping to software that is a different story.

- 'on with page tables cleared', in which case you are safe but some devices might get upset when their DMA is aborted, so their driver needs not to be a pile of shit, and needs to recover from that.

Quoted 1 line · Eric W. Biederman

There isn't a bus master shut off at the core level.

Effectively, there is if you have an IOMMU.

Quoted 4 lines · David Woodhouse

Quoted 1 line · Eric W. Biederman

There isn't a bus master shut off at the core level.

Effectively, there is if you have an IOMMU.

With the "core level" I meant Linux kernel code, as opposed to hardware level which is slightly different; I meant it would make sense to at least set the bus master control bit (PCI_COMMAND_MASTER) to zero before kexec.

There were some discussions regarding this and this code was removed by the time it was merged into mainline kernel. I can't remember the details of why. I remember one report of kernel hang on kexec that seemed to happen in device_shootdown(). I will look for any discussion threads I can find.

I would appreciate if others could test it and give a thumbs up or thumbs down before I send it to LKML.

Quoted 5 lines

kernel_kexec()
kernel_restart_prepare()
device_shutdown()

Quoted 2 lines

I would suspect it to be a qla2xxx driver problem that it did not shut down the device properly.

And device_shutdown calls every drivers .shutdown method.

Things like this are always a driver problem.