Skip to contentSkip to conversation

b44 driver causes panic when using swiotlb

DISCUSSION Chuck Ebbert 5 participants 17 messages Jan 31, 2011 – Feb 15, 2011 Library code

lore
Share & tools
lore
#0 lore raw

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Placing 64MB software IO TLB between ffff8800db400000 - ffff8800df400000 software IO TLB at phys 0xdb400000 - 0xdf400000

#1 lore raw
Quoted 7 lines · Chuck Ebbert

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

b44 needs to use GFP_DMA then and do its own custom bouncing. The standard pci_map_* bounce buffering is only designed for at least 32bit capable devices.

BTW I'm pretty sure this worked at some point, must have regressed somehow.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#2 lore raw
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Chuck Ebbert

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

Quoted 6 lines
Quoted 2 lines

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

b44 needs to use GFP_DMA then and do its own custom bouncing. The standard pci_map_* bounce buffering is only designed for at least 32bit capable devices.

That seems wrong - it's a documented API and that restriction isn't documented. Either it should comply with the request or return a failure if it can't accomodate it, not just blow up internally. There's no reason the driver should have to deal with this on its own.

In this case the DMA mapping code should really be falling back to GFP_DMA automatically if the IOMMU aperture is outside the DMA mask of the device.

Quoted 3 lines

BTW I'm pretty sure this worked at some point, must have regressed somehow.

#3 lore raw
Quoted 14 lines · On Mon, 31 Jan 2011 18:54:21 -0600 Robert Hancock <[email protected]> wrote:
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Chuck Ebbert

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

Quoted 6 lines
Quoted 2 lines

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

b44 needs to use GFP_DMA then and do its own custom bouncing. The standard pci_map_* bounce buffering is only designed for at least 32bit capable devices.

Quoted 8 lines

That seems wrong - it's a documented API and that restriction isn't documented. Either it should comply with the request or return a failure if it can't accomodate it, not just blow up internally. There's no reason the driver should have to deal with this on its own.

In this case the DMA mapping code should really be falling back to GFP_DMA automatically if the IOMMU aperture is outside the DMA mask of the device.

swiotlb allocates the bounce buffer when a system boots up. We can't allocate much in GFP_DMA. swiotlb uses somewhere under 4GB. So it can't help devices that have odd dma_mask (that is, except for 4GB).

Unfortunately, Such device needs to do own custom bouncing or needs their subsystem to does that.

Some ideas to implement something that works for such device were discussed. Seems that the conclusion is that it's doesn't worth making the common code complicated for such minor and insane devices.

#4 lore raw
Quoted 30 lines · FUJITA Tomonori
Quoted 14 lines · On Mon, 31 Jan 2011 18:54:21 -0600 Robert Hancock<[email protected]> wrote:
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Chuck Ebbert

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

Quoted 6 lines
Quoted 2 lines

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

b44 needs to use GFP_DMA then and do its own custom bouncing. The standard pci_map_* bounce buffering is only designed for at least 32bit capable devices.

Quoted 8 lines

That seems wrong - it's a documented API and that restriction isn't documented. Either it should comply with the request or return a failure if it can't accomodate it, not just blow up internally. There's no reason the driver should have to deal with this on its own.

In this case the DMA mapping code should really be falling back to GFP_DMA automatically if the IOMMU aperture is outside the DMA mask of the device.

swiotlb allocates the bounce buffer when a system boots up. We can't allocate much in GFP_DMA. swiotlb uses somewhere under 4GB. So it can't help devices that have odd dma_mask (that is, except for 4GB).

Quoted 6 lines

Unfortunately, Such device needs to do own custom bouncing or needs their subsystem to does that.

Some ideas to implement something that works for such device were discussed. Seems that the conclusion is that it's doesn't worth making the common code complicated for such minor and insane devices.

I don't think this is the only device that has sub-32-bit DMA restrictions, this will just lead to a bunch of duplicated code. In particular, how is LPC DMA supposed to work?

At the very least we should be allowing the driver to deal with the failure instead of panicing the system. Otherwise we are just leaving a land mine for people to trip over.

#5 lore raw
Quoted 6 lines · On Mon, 31 Jan 2011 21:22:23 -0600 Robert Hancock <[email protected]> wrote:
Quoted 3 lines

Some ideas to implement something that works for such device were discussed. Seems that the conclusion is that it's doesn't worth making the common code complicated for such minor and insane devices.

I don't think this is the only device that has sub-32-bit DMA restrictions, this will just lead to a bunch of duplicated code.

Yeah, not only device but not many.

The block layer has the own bouncing mechanism. Some network drivers have the similar bouncing code. I don't know if there are other kinds of drivers that have the own bouncing code.

I thought that we can make mm/bounce.c (used for block drivers now) work any drivers without complicating it. We could make swiotlb to do but it's too complicated and it doesn't worth.

Quoted 2 lines

In
particular, how is LPC DMA supposed to work?

LPC DMA can't do 32bit dma?

Quoted 3 lines

At the very least we should be allowing the driver to deal with the failure instead of panicing the system. Otherwise we are just leaving a land mine for people to trip over.

Agreed. swiotlb shouldn't panic in this case. I'll take care of it.

#6 lore raw
Quoted 21 lines · FUJITA Tomonori
Quoted 6 lines · On Mon, 31 Jan 2011 21:22:23 -0600 Robert Hancock<[email protected]> wrote:
Quoted 3 lines

Some ideas to implement something that works for such device were discussed. Seems that the conclusion is that it's doesn't worth making the common code complicated for such minor and insane devices.

I don't think this is the only device that has sub-32-bit DMA restrictions, this will just lead to a bunch of duplicated code.

Yeah, not only device but not many.

The block layer has the own bouncing mechanism. Some network drivers have the similar bouncing code. I don't know if there are other kinds of drivers that have the own bouncing code.

I thought that we can make mm/bounce.c (used for block drivers now) work any drivers without complicating it. We could make swiotlb to do but it's too complicated and it doesn't worth.

Quoted 4 lines
Quoted 2 lines

In
particular, how is LPC DMA supposed to work?

LPC DMA can't do 32bit dma?

At least not if it's using ISA-style 3rd-party DMA. Some devices may do bus-mastering and be able to do 32-bit DMA, but I've never seen one, at least.

Quoted 7 lines
Quoted 5 lines

At the very least we should be allowing the driver to deal with the failure instead of panicing the system. Otherwise we are just leaving a land mine for people to trip over.

Agreed. swiotlb shouldn't panic in this case. I'll take care of it.

#7 lore raw
In reply to FUJITA Tomonori
Quoted 7 lines · On Tue, 1 Feb 2011 10:28:00 +0900 FUJITA Tomonori…

swiotlb allocates the bounce buffer when a system boots up. We can't allocate much in GFP_DMA. swiotlb uses somewhere under 4GB. So it can't help devices that have odd dma_mask (that is, except for 4GB).

Unfortunately, Such device needs to do own custom bouncing or needs their subsystem to does that.

I think we're chasing the wrong problem here.

swiotlb uses alloc_bootmem_low_pages() to try to get buffers as low in memory as possible. I asked someone who is hitting this bug to try 2.6.36 and he reports the buffers really are low there:

2.6.36: 5c00000
2.6.37: db600000

So something happened very early in the 2.6.37-rc cycle that changed this behavior. I tried looking at the bootmem code but could not see the problem. The only related option I could find in .config was this:

# CONFIG_NO_BOOTMEM is not set

It was set this way in both .36 and .37.

#8 lore raw
Quoted 16 lines · On Tue, 1 Feb 2011 16:18:59 -0500 Chuck Ebbert <[email protected]> wrote:
Quoted 6 lines

swiotlb allocates the bounce buffer when a system boots up. We can't allocate much in GFP_DMA. swiotlb uses somewhere under 4GB. So it can't help devices that have odd dma_mask (that is, except for 4GB).

Unfortunately, Such device needs to do own custom bouncing or needs their subsystem to does that.

I think we're chasing the wrong problem here.

swiotlb uses alloc_bootmem_low_pages() to try to get buffers as low in memory as possible. I asked someone who is hitting this bug to try 2.6.36 and he reports the buffers really are low there:

2.6.36: 5c00000
2.6.37: db600000

Quoted 7 lines

So something happened very early in the 2.6.37-rc cycle that changed this behavior. I tried looking at the bootmem code but could not see the problem. The only related option I could find in .config was this:

# CONFIG_NO_BOOTMEM is not set

It was set this way in both .36 and .37.

I don't think this matters because we can't guarantee that swiotlb can't allocate ZONE_DMA. Obviously, the default swiotlb bouncing buffer size, 64MB, is too large for ZONE_DMA.

If you are lucky, you sometimes can allocate lower memory if swiotlb allocates lower memory at the boot time. But you can't assume that. swiotlb isn't designed to handle such.

Can you try the patch that I sent you? As I wrote, b44 has the own bouncing mechanism, so we should be fine if swiotlb returns an appropriate error instead of just panicking.

#9 lore raw
Quoted 23 lines · On Wed, 2 Feb 2011 09:06:34 +0900 FUJITA Tomonori…
Quoted 20 lines · On Tue, 1 Feb 2011 16:18:59 -0500 Chuck Ebbert <[email protected]> wrote:
Quoted 6 lines

swiotlb allocates the bounce buffer when a system boots up. We can't allocate much in GFP_DMA. swiotlb uses somewhere under 4GB. So it can't help devices that have odd dma_mask (that is, except for 4GB).

Unfortunately, Such device needs to do own custom bouncing or needs their subsystem to does that.

I think we're chasing the wrong problem here.

swiotlb uses alloc_bootmem_low_pages() to try to get buffers as low in memory as possible. I asked someone who is hitting this bug to try 2.6.36 and he reports the buffers really are low there:

2.6.36: 5c00000
2.6.37: db600000

So something happened very early in the 2.6.37-rc cycle that changed this behavior. I tried looking at the bootmem code but could not see the problem. The only related option I could find in .config was this:

Quoted 6 lines
Quoted 3 lines

# CONFIG_NO_BOOTMEM is not set

It was set this way in both .36 and .37.

I don't think this matters because we can't guarantee that swiotlb can't allocate ZONE_DMA.

Oops, I meant that, we can't guarantee that swiotlb allocates ZONE_DMA.

Quoted 15 lines

Obviously, the default swiotlb bouncing
buffer size, 64MB, is too large for ZONE_DMA.

If you are lucky, you sometimes can allocate lower memory if swiotlb allocates lower memory at the boot time. But you can't assume that. swiotlb isn't designed to handle such.

Can you try the patch that I sent you? As I wrote, b44 has the own bouncing mechanism, so we should be fine if swiotlb returns an appropriate error instead of just panicking.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#10 lore raw
In reply to Robert Hancock
Quoted 7 lines · Robert Hancock
Quoted 6 lines · Andi Kleen
Quoted 5 lines · Chuck Ebbert

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

b44 needs to use GFP_DMA then and do its own custom bouncing. The standard pci_map_* bounce buffering is only designed for at least 32bit capable devices.

That seems wrong - it's a documented API and that restriction isn'

Please read the documentation. When PCI DMA cannot handle it it returns failure. That is what happened here.

Besides historically PCI-DMA never handled < 4GB.

Quoted 2 lines

documented. Either it should comply with the request or return a failure if it can't accomodate it, not just blow up internally.

It does return failure, that is what breaks the driver because it doesn't handle it.

Quoted 2 lines

There's no reason the driver should have to deal with this on its own.

Even if swiotlb could handle it, other implementations of PCI-DMA can not

Quoted 3 lines

In this case the DMA mapping code should really be falling back to GFP_DMA automatically if the IOMMU aperture is outside the DMA mask of the device.

No, no -- you cannot allocate memory in DMA API. It's forbidden. Some paths that use it cannot tolerate it.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#11 lore raw
Quoted 11 lines · Andi Kleen
Quoted 10 lines · Robert Hancock
Quoted 9 lines · Andi Kleen
Quoted 7 lines · Chuck Ebbert

The b44 driver is triggering this panic in swiotlb_map_page():

        if (!dma_capable(dev, dev_addr, size)) panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

b44 needs to use GFP_DMA then and do its own custom bouncing. The standard pci_map_* bounce buffering is only designed for at least 32bit capable devices.

That seems wrong - it's a documented API and that restriction isn'

Please read the documentation. When PCI DMA cannot handle it it returns failure. That is what happened here.

Apparently not, given that it hit a panic inside the swiotlb code..

#12 lore raw
In reply to Chuck Ebbert
Quoted 7 lines · On Mon, 31 Jan 2011 10:54:12 -0500 Chuck Ebbert <[email protected]> wrote:

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

Can you try this? b44 has the own bouncing mechanism so if swiotlb returns an error, the driver can allocate an appropriate buffer.

lib/swiotlb.c +4 -2
index c47bbe1..93ca08b 100644
@@ -686,8 +686,10 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	/*
 	 * Ensure that the address returned is DMA'ble
 	 */
-	if (!dma_capable(dev, dev_addr, size))
-		panic("map_single: bounce buffer is not DMA'ble");
+	if (!dma_capable(dev, dev_addr, size)) {
+		swiotlb_tbl_unmap_single(dev, map, size, dir);
+		dev_addr = swiotlb_virt_to_bus(dev, io_tlb_overflow_buffer);
+	}
 
 	return dev_addr;
 }
#13 lore raw
Quoted 21 lines · On Tue, 1 Feb 2011 22:54:56 +0900 FUJITA Tomonori…
Quoted 7 lines · On Mon, 31 Jan 2011 10:54:12 -0500 Chuck Ebbert <[email protected]> wrote:

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

Can you try this? b44 has the own bouncing mechanism so if swiotlb returns an error, the driver can allocate an appropriate buffer.

lib/swiotlb.c +0 -0
index c47bbe1..93ca08b 100644
@@ -686,8 +686,10 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	/*
 	 * Ensure that the address returned is DMA'ble

lib/swiotlb.c dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, · line 695

Quoted 10 lines
lib/swiotlb.c +4 -2
 	 */
-	if (!dma_capable(dev, dev_addr, size))
-		panic("map_single: bounce buffer is not DMA'ble");
+	if (!dma_capable(dev, dev_addr, size)) {
+		swiotlb_tbl_unmap_single(dev, map, size, dir);
+		dev_addr = swiotlb_virt_to_bus(dev, io_tlb_overflow_buffer);
+	}
 
 	return dev_addr;
 }

Ping, any chance to try this?

Thanks,

#14 lore raw
Quoted 17 lines · FUJITA Tomonori
Quoted 14 lines · On Tue, 1 Feb 2011 22:54:56 +0900 FUJITA Tomonori…
Quoted 7 lines · On Mon, 31 Jan 2011 10:54:12 -0500 Chuck Ebbert <[email protected]> wrote:

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

Can you try this? b44 has the own bouncing mechanism so if swiotlb returns an error, the driver can allocate an appropriate buffer.

Quoted 14 lines
Quoted 5 lines
lib/swiotlb.c +0 -0
index c47bbe1..93ca08b 100644

lib/swiotlb.c dma_addr_t swiotlb_map_page(struct device *dev, · line 692

Quoted 9 lines
lib/swiotlb.c +3 -2
@@ -686,8 +686,10 @@ dma_addr_t swiotlb_map_page(struct device *dev,
 struct page *page,
 /*
 * Ensure that the address returned is DMA'ble
 */
-	if (!dma_capable(dev, dev_addr, size))
-		panic("map_single: bounce buffer is not DMA'ble");
+	if (!dma_capable(dev, dev_addr, size)) {
+		swiotlb_tbl_unmap_single(dev, map, size, dir);
+		dev_addr = swiotlb_virt_to_bus(dev, 

io_tlb_overflow_buffer);

Quoted 6 lines
Quoted 4 lines
lib/swiotlb.c +1 -0
+	}
 
 return dev_addr;

}

Ping, any chance to try this?

Friend machine oopsed on network with b44 on 2.6.37, so we tried this patch and network card was detected nicely and was actually usable. The oops started to happen after he upgraded memory from 2GB to 4GB in his dell vostro 1500 notebook.

03:00.0 Ethernet controller: Broadcom Corporation BCM4401-B0 100Base-TX (rev
02)

# grep b44 dmesg.log
[    8.524297] b44 0000:03:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    8.576838] b44: b44.c:v2.0
[    8.593920] b44 ssb1:0: eth0: Broadcom 44xx/47xx 10/100BaseT Ethernet 
00:1c:23:9b:3f:5f
[   55.350228] b44 ssb1:0: eth0: Link is up at 100 Mbps, full duplex
[   55.350233] b44 ssb1:0: eth0: Flow control is off for TX and off for RX
Quoted 1 line

Thanks,

Signature · 2 lines
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/
#15 lore raw
Quoted 36 lines · On Mon, 14 Feb 2011 17:41:54 +0100 Arkadiusz Mi$(D+\(Bkiewicz…
Quoted 17 lines · FUJITA Tomonori
Quoted 14 lines · On Tue, 1 Feb 2011 22:54:56 +0900 FUJITA Tomonori…
Quoted 7 lines · On Mon, 31 Jan 2011 10:54:12 -0500 Chuck Ebbert <[email protected]> wrote:

The b44 driver is triggering this panic in swiotlb_map_page():

if (!dma_capable(dev, dev_addr, size))
panic("map_single: bounce buffer is not DMA'ble");

The kernel log says the bounce buffers are at 0xdb400000, but b44 can only do DMA to the first 1GB of memory:

Can you try this? b44 has the own bouncing mechanism so if swiotlb returns an error, the driver can allocate an appropriate buffer.

Quoted 14 lines
Quoted 5 lines
lib/swiotlb.c +0 -0
index c47bbe1..93ca08b 100644

lib/swiotlb.c dma_addr_t swiotlb_map_page(struct device *dev, · line 692

Quoted 9 lines
lib/swiotlb.c +3 -2
@@ -686,8 +686,10 @@ dma_addr_t swiotlb_map_page(struct device *dev,
 struct page *page,
 /*
 * Ensure that the address returned is DMA'ble
 */
-	if (!dma_capable(dev, dev_addr, size))
-		panic("map_single: bounce buffer is not DMA'ble");
+	if (!dma_capable(dev, dev_addr, size)) {
+		swiotlb_tbl_unmap_single(dev, map, size, dir);
+		dev_addr = swiotlb_virt_to_bus(dev, 

io_tlb_overflow_buffer);

Quoted 2 lines
Quoted 2 lines
lib/swiotlb.c +1 -0
+	}
 
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
lib/swiotlb.c +0 -0
 return dev_addr;

}

Ping, any chance to try this?

Friend machine oopsed on network with b44 on 2.6.37, so we tried this patch and network card was detected nicely and was actually usable. The oops

Thanks for the confirmation.

Quoted 2 lines

started to happen after he upgraded memory from 2GB to 4GB in his dell vostro 1500 notebook.

Yeah, a system with 2GB memory doesn't use swiotlb so you don't hit this bug.

Thanks,

#16 lore raw
In reply to FUJITA Tomonori
Quoted 18 lines · On Mon, 14 Feb 2011 20:59:43 +0900 FUJITA Tomonori…
Quoted 18 lines

Can you try this? b44 has the own bouncing mechanism so if swiotlb returns an error, the driver can allocate an appropriate buffer.

lib/swiotlb.c +4 -2
index c47bbe1..93ca08b 100644
@@ -686,8 +686,10 @@ dma_addr_t swiotlb_map_page(struct device *dev, struct page *page,
 	/*
 	 * Ensure that the address returned is DMA'ble
 	 */
-	if (!dma_capable(dev, dev_addr, size))
-		panic("map_single: bounce buffer is not DMA'ble");
+	if (!dma_capable(dev, dev_addr, size)) {
+		swiotlb_tbl_unmap_single(dev, map, size, dir);
+		dev_addr = swiotlb_virt_to_bus(dev, io_tlb_overflow_buffer);
+	}
 
Quoted 4 lines

lib/swiotlb.c dma_addr_t swiotlb_map_page(struct device *dev, struct page *page, · line 695

Quoted 2 lines
lib/swiotlb.c +0 -0
 	return dev_addr;
 }

Ping, any chance to try this?

The user who reported the bug tried it and says that it works.