r8169 DMA failure with iommu=off

This only fixes the Rx side, what about Tx?

I think maybe just removing the whole use_dac flag completely?

Subject: r8169: allocate rx memory in correct region

This fixes failure when using r8169 with IOMMU defined but not enabled. Reproduced on a 16G machine with LOM r8169 and kernel set to 'iommu=off'.

This driver has two dma modes. By default use_dac module parameter is zero and therefore the driver uses only has a 32 bit DMA mask. But since it calls kmalloc_node() without setting DMA32 flag the receive buffer maybe above 4G and the dma_map_single will fail.

In either case since this is a receive DMA buffer, it should set the appropriate GFP_DMA since that may matter on some platforms.

This an old bug was introduced by:

commit 6f0333b8fde44b8c04a53b2461504f0e8f1cebe6

Author: Eric Dumazet <[email protected]>
Date:   Mon Oct 11 11:17:47 2010 +0000

r8169: use 50% less ram for RX ring

Using standard skb allocations in r8169 leads to order-3 allocations (if PAGE_SIZE=4096), because NIC needs 16383 bytes, and skb overhead makes this bigger than 16384 -> 32768 bytes per "skb"

Using kmalloc() permits to reduce memory requirements of one r8169 nic by 4Mbytes. (256 frames * 16Kbytes). This is fine since a hardware bug requires us to copy incoming frames, so we build real skb when doing this copy.

@@ -5721,14 +5721,16 @@ static struct sk_buff *rtl8169_alloc_rx_
 	struct device *d = &tp->pci_dev->dev;
 	struct net_device *dev = tp->dev;
 	int node = dev->dev.parent ? dev_to_node(dev->dev.parent) : -1;
+	gfp_t flags = GFP_KERNEL;
 
-	data = kmalloc_node(rx_buf_sz, GFP_KERNEL, node);
+	flags |= (dev->features & NETIF_F_HIGHDMA) ? GFP_DMA : GFP_DMA32;
+	data = kmalloc_node(rx_buf_sz, flags, node);
 	if (!data)
 		return NULL;
 
 	if (rtl8169_align(data) != data) {
 		kfree(data);
-		data = kmalloc_node(rx_buf_sz + 15, GFP_KERNEL, node);
+		data = kmalloc_node(rx_buf_sz + 15, flags, node);
 		if (!data)
 			return NULL;
 	}

Stephen Hemminger <[email protected]> :

Something simple would be welcome but I doubt it should be that simple.

The problems that the use_dac comment (MODULE_PARM_DESC) relates to are 2003 ~ 2004, plain old PCI 8169 stuff. I see no problem exchanging the use_dac test for something else (Config2.BIT(3) test for bus width or old PCI chipsets blacklist).

I lack explicit reports to tell if high dma is safe with the PCI-E chipsets but it's worth testing. The DAC bit in the CPlusCmd register is indirectly documented in my old 8168c datasheet through the 64 bit address capable bit of the MSI message control word (it's missing in the section dedicated to the CPlusCmd register :o/). Expect it to initially mirror some eeprom value.

Other than that I am mostly unqualified when it comes to high DMA paths in the upper parts of the transmit stack.

Ueimor

From: Stephen Hemminger <[email protected]>
Date: Wed, 29 Apr 2015 16:58:13 -0700

Plain GFP_DMA is really only appropriate when used for ISA DMA situations where we must have 24-bit addresses or whatever that restriction was.

Anyways, it is dma_map_single()'s job to restrict DMA addressing if necessary. Otherwise, the transmit path would not work at all.