firewire: fw-ohci: sync AT dma buffer before use
PATCH 1/1 firewire: fw-ohci: sync AT dma buffer before use
Per DMA API docs, when using dma_map_single(), DMA_TO_DEVICE synchronization must be done after the last modification of the memory region by software before it can be handed off to the device and safely read. Such a sync is currently missing from firewire_ohci:at_context_queue_packet().
At least on my setup, where I could within seconds reliably reproduce a panic in handle_at_packet() by simply dd'ing from two drives on different controllers, the panic is gone.
See http://bugzilla.kernel.org/show_bug.cgi?id=9617
---
drivers/firewire/fw-ohci.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-)
drivers/firewire/fw-ohci.c +4 -0
@@ -780,6 +780,10 @@ at_context_queue_packet(struct context *ctx, struct fw_packet *packet)context_append (ctx, d, z,4 - z); +/* Sync the DMA buffer up for the device to read from */ +dma_sync_single_for_device (ohci->card.device, payload_bus, + packet->payload_length,DMA_TO_DEVICE ); +/* If the context isn't already running, start it up. */ reg =reg_read (ctx->ohci,CONTROL_SET (ctx->regs));if ((reg &CONTEXT_RUN ) ==0 )
Signature · 2 lines
Jarod Wilson [email protected]
Quoted 6 lines · Jarod Wilson
drivers/firewire/fw-ohci.c +0
-0
drivers/firewire/fw-ohci.c at_context_queue_packet(struct context *ctx, struct fw_packet *packet) · line 789
Quoted 10 lines
drivers/firewire/fw-ohci.c +4
-0@@ -780,6 +780,10 @@ at_context_queue_packet(struct context *ctx, struct fw_packet *packet)context_append (ctx, d, z,4 - z); +/* Sync the DMA buffer up for the device to read from */ +dma_sync_single_for_device (ohci->card.device, payload_bus, + packet->payload_length,DMA_TO_DEVICE ); +/* If the context isn't already running, start it up. */ reg =reg_read (ctx->ohci,CONTROL_SET (ctx->regs));if ((reg &CONTEXT_RUN ) ==0 )
Two things:
1. The dma_sync should probably be done before the context_append because the controller could in theory start reading the data as soon as context_append is called right?
2. As an optimization, we should attempt to allocate the payload in the lower 32-bits of physical memory, to prevent extra memcopies on x86_64. I think this can be done by adding GFP_DMA32 to kmalloc where the payload was allocated in fw-cdev.c. There might be other places where we would benefit from GFP_DMA32 also. Of course, these optimizations are probably better saved for another patch.
-David
Quoted 3 lines · David Moore
1. The
dma_syncshould probably be done before thecontext_appendbecause the controller could in theory start reading the data as soon ascontext_appendis called right?
Yes, but its syncing is superfluous anyway, as mentioned in the other post. *If* it is doing something for Jarod's setup, then only because of side effects like memory barrier properties.
Quoted 6 lines
2. As an optimization, we should attempt to allocate the payload in the lower 32-bits of physical memory, to prevent extra memcopies on
x86_64. I think this can be done by addingGFP_DMA32to kmalloc where the payload was allocated infw-cdev.c. There might be other places where we would benefit fromGFP_DMA32also. Of course, these optimizations are probably better saved for another patch.
I think so too. GFP_DMA32 would be appropriate on machines with "software IOMMU" alias swiotlb.
Does GFP_DMA32 have adverse affects on machines with a real IOMMU?
Signature · 3 lines
Stefan Richter -=====-==--- --== -==-- http://arcgraph.de/sr/
Quoted 5 lines · Jarod Wilson
At least on my setup, where I could within seconds reliably reproduce a panic in
handle_at_packet()by simply dd'ing from two drives on different controllers, the panic is gone.
Alas the panic from comment #10 is still there, i.e. instant crash when plugging in an LSI based CD-RW (shortly after SCSI inquiry) --- but only if CONFIG_DEBUG_PAGEALLOC=y.
Jarod, did your crashes happen with CONFIG_DEBUG_PAGEALLOC=n?
Quoted 4 lines
drivers/firewire/fw-ohci.c at_context_queue_packet(struct context *ctx, struct fw_packet *packet) · line 789
Quoted 10 lines
drivers/firewire/fw-ohci.c +4
-0@@ -780,6 +780,10 @@ at_context_queue_packet(struct context *ctx, struct fw_packet *packet)context_append (ctx, d, z,4 - z); +/* Sync the DMA buffer up for the device to read from */ +dma_sync_single_for_device (ohci->card.device, payload_bus, + packet->payload_length,DMA_TO_DEVICE ); +/* If the context isn't already running, start it up. */ reg =reg_read (ctx->ohci,CONTROL_SET (ctx->regs));if ((reg &CONTEXT_RUN ) ==0 )
The dma_sync_single_ call should be conditional for packet->payload_length > 0. You would have noticed that if my patch "firewire: fw-ohci: shut up false compiler warning on PPC32" wouldn't have shadowed the corresponding compiler warning, which would be for real after your patch. And, as David wrote, the call should come before context_append.
However, we actually don't need it at all.
The dma_map_single(...) already syncs the payload for the device, and we don't access the payload after that anymore. So this patch shouldn't do anything, except that it inserts a call which happens to have barrier characteristics on some platforms.
What we rather have to check is:
- Are we really writing into the context program the order that we need to? This includes ordering WRT MMIO writes.
- Are we writing the branch address atomically? (No, we don't enforce an atomic access at the moment, although it is very likely that the compiler uses an atomic access.)
(We have to expect that the controller reads a descriptor while we write into it.)
- Is there a use-after-free problem somewhere?
(A pattern in the original report and in a crash that you mentioned looked like use of freed memory: "Faulting instruction address: 0x6b6b6b68" in comment #1.)
Signature · 3 lines
Stefan Richter -=====-==--- --== -==-- http://arcgraph.de/sr/
Quoted 5 lines · Stefan Richter
Quoted 4 lines · Jarod Wilson
At least on my setup, where I could within seconds reliably reproduce a panic in
handle_at_packet()by simply dd'ing from two drives on different controllers, the panic is gone.
Quoted 7 lines
Quoted 1 line
Alas the panic from comment #10 is still there, i.e. instant crash when plugging in an LSI based CD-RW (shortly after SCSI inquiry) --- but only if
CONFIG_DEBUG_PAGEALLOC=y.Jarod, did your crashes happen with
CONFIG_DEBUG_PAGEALLOC=n?
No, they're with it turned on (and still on w/this change where it doesn't panic). If I run with CONFIG_DEBUG_PAGEALLOC=n, no panic.
Quoted 22 lines
Quoted 13 lines
drivers/firewire/fw-ohci.c +4
-0@@ -780,6 +780,10 @@ at_context_queue_packet(struct context *ctx, struct fw_packet *packet) context_append(ctx, d, z, 4 - z); +/* Sync the DMA buffer up for the device to read from */ +dma_sync_single_for_device (ohci->card.device, payload_bus, + packet->payload_length,DMA_TO_DEVICE ); +/* If the context isn't already running, start it up. */ reg =reg_read (ctx->ohci,CONTROL_SET (ctx->regs));Quoted 1 line
if ((reg &
CONTEXT_RUN) == 0)The
dma_sync_single_call should be conditional forpacket->payload_length> 0. You would have noticed that if my patch "firewire: fw-ohci: shut up false compiler warning on PPC32" wouldn't have shadowed the corresponding compiler warning, which would be for real after your patch. And, as David wrote, the call should come beforecontext_append.
Quoted 4 lines
However, we actually don't need it at all.
The
dma_map_single(...)already syncs the payload for the device, and we don't access the payload after that anymore.
D'oh, got overzealous reading dma docs, missed the fact that we didn't actually touch it on the host side, so nothing to actually sync...
Quoted 3 lines
So this patch shouldn't do
anything, except that it inserts a call which happens to have barrier characteristics on some platforms.
...but got lucky in that it actually helps this particular setup (x86_64 kernel, dual quad-core opteron, 8G RAM, 3 FireWire controllers). Hrm.
Quoted 11 lines
What we rather have to check is:
- Are we really writing into the context program the order that we need to? This includes ordering WRT MMIO writes.
- Are we writing the branch address atomically? (No, we don't enforce an atomic access at the moment, although it is very likely that the compiler uses an atomic access.)
(We have to expect that the controller reads a descriptor while we write into it.)
More investigative fun for tomorrow...
Quoted 5 lines
- Is there a use-after-free problem somewhere?
(A pattern in the original report and in a crash that you mentioned looked like use of freed memory: "Faulting instruction address:
0x6b6b6b68" in comment #1.)
Seems both of the ones that looked like slab poison were on PowerPC. I'll have to spin up a ppc kernel on the old powerbook and poke around more.
Signature · 2 lines
Jarod Wilson [email protected]
Quoted 10 lines · Jarod Wilson
Quoted 7 lines · Stefan Richter
Quoted 1 line · Jarod Wilson
Alas the panic from comment #10 is still there, i.e. instant crash when plugging in an LSI based CD-RW (shortly after SCSI inquiry) --- but only if
CONFIG_DEBUG_PAGEALLOC=y.Jarod, did your crashes happen with
CONFIG_DEBUG_PAGEALLOC=n?No, they're with it turned on (and still on w/this change where it doesn't panic). If I run with
CONFIG_DEBUG_PAGEALLOC=n, no panic.
So then that's like "my" panic.
The other thing that I see here is that it only happens with my two LSI based devices (both CD-RWs), always quickly after SCSI inquiry.
I shall test with my Prolific based DVD-RW to find out whether it is about some requests that are sent to CD/DVD-RWs or about the split transactions that I get from the LSI bridges. (See bugzilla.)
Quoted 2 lines
Quoted 2 lines
So this patch shouldn't do anything, except that it inserts a call which happens to have barrier characteristics on some platforms.
...and potentially delays execution.
Quoted 1 line
...but got lucky in that it actually helps this particular setup (
x86_64kernel, dual quad-core opteron, 8G RAM, 3 FireWire controllers). Hrm.
Unless you or I spot the real solution earlier, you could also try replacing your dma_sync_ with mb() and with mdelay() respectively to see what aspect of the dma_sync_ is fixing your setup. Also move the mb() to other interesting places of the involved code.
Signature · 3 lines
Stefan Richter -=====-==--- --== -==-= http://arcgraph.de/sr/
Quoted 6 lines · Stefan Richter
Quoted 3 lines
Quoted 3 lines
So this patch shouldn't do
anything, except that it inserts a call which happens to have barrier characteristics on some platforms....and potentially delays execution.
Quoted 7 lines
Quoted 2 lines
...but got lucky in that it actually helps this particular setup (
x86_64kernel, dual quad-core opteron, 8G RAM, 3 FireWire controllers). Hrm.Unless you or I spot the real solution earlier, you could also try replacing your
dma_sync_withmb()and withmdelay()respectively to see what aspect of thedma_sync_is fixing your setup. Also move themb()to other interesting places of the involved code.
So it would seem I screwed up something in my testing, and failed to reproduce the panic after adding '[PATCH] firewire: fw-ohci: use dma_alloc_coherent for ar_buffer' to the mix. Best as I can tell now, the panic was actually resolved by that patch, as backing out the sync altogether now works just as well as with the sync, with an mb and with an mdelay.
This sort of makes some degree of sense, since this is another x86_64 system w/>= 4GB of RAM. It would appear possible that at some higher layer, where AT and AR transactions are coordinated with one another, we were getting an AT packet into a bad state due to its corresponding AR traffic being stuck in a non-coherent buffer we couldn't read. But this is only semi-informed speculation...
Signature · 2 lines
Jarod Wilson [email protected]