4.3+: Atheros ethernet fails after resume from s2ram, due to order 4 allocation

Ok, so what went on is easy.. any ideas how to fix it ?

Hi!

...and dmesg tells us what is going on:

[ 6961.550240] NetworkManager: page allocation failure: order:4,

mode:0x2080020

[ 6961.550249] CPU: 0 PID: 2590 Comm: NetworkManager Tainted: G
W       4.3.0+ #124
[ 6961.550250] Hardware name: Acer Aspire 5732Z/Aspire 5732Z, BIOS
V3.07 02/10/2010
[ 6961.550252]  00000000 00000000 f2ad1a04 c42ba5b8 00000000 f2ad1a2c
c40d650a c4d3ee1c
[ 6961.550260]  f34ef600 00000004 02080020 c4eeef40 00000000 00000010
00000000 f2ad1ac8
[ 6961.550266]  c40d8caa 02080020 00000004 00000000 00000070 f34ef200
00000060 00000010
[ 6961.550272] Call Trace:
...[ 6961.550299]  [<c4006811>] dma_generic_alloc_coherent+0x71/0x120
[ 6961.550301]  [<c40067a0>] ? via_no_dac+0x30/0x30
[ 6961.550307]  [<c465b16e>] atl1c_open+0x29e/0x300
[ 6961.550313]  [<c48b96f5>] ? call_netdevice_notifiers_info+0x25/0x50
[ 6961.550316]  [<c48c081b>] __dev_open+0x7b/0xf0
[ 6961.550318]  [<c48c0ac9>] __dev_change_flags+0x89/0x140
[ 6961.550320]  [<c48c0ba3>] dev_change_flags+0x23/0x60
[ 6961.550325]  [<c48ce416>] do_setlink+0x286/0x7b0
[ 6961.550328]  [<c42ded02>] ? nla_parse+0x22/0xd0
[ 6961.550330]  [<c48cf906>] rtnl_newlink+0x5d6/0x860
[ 6961.550336]  [<c407f8a1>] ? __lock_acquire.isra.24+0x3a1/0xc80
[ 6961.550342]  [<c4047ae2>] ? ns_capable+0x22/0x60
[ 6961.550345]  [<c48e7c5d>] ? __netlink_ns_capable+0x2d/0x40
[ 6961.550351]  [<c49c9c54>] ? xprt_transmit+0x94/0x220
[ 6961.550354]  [<c48cd9e6>] rtnetlink_rcv_msg+0x76/0x1f0
[ 6961.550356]  [<c48cd970>] ? rtnetlink_rcv+0x30/0x30
[ 6961.550359]  [<c48eb35e>] netlink_rcv_skb+0x8e/0xb0
...
[ 6961.550412] Mem-Info:
[ 6961.550417] active_anon:30319 inactive_anon:25075 isolated_anon:0
 active_file:327764 inactive_file:152179 isolated_file:16
  unevictable:0 dirty:6 writeback:0 unstable:0
   slab_reclaimable:149091 slab_unreclaimable:18973
    mapped:18100 shmem:4847 pagetables:1538 bounce:0
     free:57732 free_pcp:10 free_cma:0
...
[ 6961.550492] 485897 total pagecache pages
[ 6961.550494] 1086 pages in swap cache
[ 6961.550496] Swap cache stats: add 16738, delete 15652, find
6708/8500
[ 6961.550497] Free swap  = 1656440kB
[ 6961.550498] Total swap = 1681428kB
[ 6961.550499] 785914 pages RAM
[ 6961.550500] 557663 pages HighMem/MovableOnly
[ 6961.550501] 12639 pages reserved
[ 6961.550506] atl1c 0000:05:00.0: pci_alloc_consistend failed
[ 6962.148358] psmouse serio1: synaptics: queried max coordinates: x

[..5772], y [..5086]

Order 4 allocation... probably doable during boot, but not really suitable during resume.

I'm not sure how repeatable it is, but it definitely happened more than once.

        /*                                                                      
         * real ring DMA buffer                                                 
         * each ring/block may need up to 8 bytes for alignment, hence the      
         * additional bytes tacked onto the end.                                
         */
        ring_header->size = size =
                sizeof(struct atl1c_tpd_desc) * tpd_ring->count * 2 +
                sizeof(struct atl1c_rx_free_desc) * rx_desc_count +
                sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
                8 * 4;

        ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
                                &ring_header->dma);

        if (unlikely(!ring_header->desc)) {
                dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
                goto err_nomem;
        }

(Note the typo in dev_err... at least it is easy to grep).

Ok, so what went on is easy.. any ideas how to fix it?

Quoted 11 lines

/*
* real ring DMA buffer
* each ring/block may need up to 8 bytes for alignment, hence the
* additional bytes tacked onto the end.
*/
ring_header->size = size =
sizeof(struct atl1c_tpd_desc) * tpd_ring->count * 2 + sizeof(struct atl1c_rx_free_desc) * rx_desc_count + sizeof(struct atl1c_recv_ret_status) * rx_desc_count + 8 * 4;

Quoted 2 lines

        ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
                                &ring_header->dma);

Why is pci_alloc_consistent doing an unconditional GFP_ATOMIC allocation? atl1_setup_ring_resources already does GFP_KERNEL allocation in the same function so this should be sleepable context. I think we should either add pci_alloc_consistent_gfp if there are no explicit reasons to not do so or you can workaround

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +0 -0

index 2795d6d..afb71e0 100644

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +3 -3

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 
-	ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
-				&ring_header->dma);
+	ring_header->desc = dma_alloc_coherent(&pdev->dev, ring_header->size,
+					       &ring_header->dma, GFP_KERNEL);
 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memmory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

Signature · 3 lines

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +0 -0

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 

Quoted 4 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

It is not clear why GFP_KERNEL can replace GFP_ATOMIC safely neither from the changelog nor from the patch context.

Quoted 14 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +0 -0

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 11 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +3 -3

-	ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
-				&ring_header->dma);
+	ring_header->desc = dma_alloc_coherent(&pdev->dev, ring_header->size,
+					       &ring_header->dma, GFP_KERNEL);
 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memmory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

It seems there is a missed opportunity to get rid of the memset() here, by adding __GFP_ZERO to the dma_alloc_coherent() GFP_KERNEL mask, or simply using dma_zalloc_coherent()

From: Michal Hocko <[email protected]>
Date: Mon, 30 Nov 2015 14:21:29 +0100

Quoted 6 lines · Pavel Machek

Quoted 4 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

Quoted 2 lines

It is not clear why GFP_KERNEL can replace GFP_ATOMIC safely neither from the changelog nor from the patch context.

Earlier in the function we do a GFP_KERNEL kmalloc so:

¯\_(ツ)_/¯

It should be fine.

Quoted 16 lines · David Miller

From: Michal Hocko <[email protected]>
Date: Mon, 30 Nov 2015 14:21:29 +0100

Quoted 8 lines · Pavel Machek

Quoted 4 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

It is not clear why GFP_KERNEL can replace GFP_ATOMIC safely neither from the changelog nor from the patch context.

Earlier in the function we do a GFP_KERNEL kmalloc so:

¯\_(ツ)_/¯

Quoted 2 lines

It should be fine.

AFAICT, the people who benefit from GFP_ATOMIC are the people running all their storage over NFS/iSCSI who are suspending their machines while they're so busy they don't have any clean order 4 pagecache to drop, and want the machine to panic rather than hang. The people who benefit from

Quoted 16 lines · David Miller

From: Michal Hocko <[email protected]>
Date: Mon, 30 Nov 2015 14:21:29 +0100

Quoted 8 lines · Pavel Machek

Quoted 4 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

It is not clear why GFP_KERNEL can replace GFP_ATOMIC safely neither from the changelog nor from the patch context.

Earlier in the function we do a GFP_KERNEL kmalloc so:

¯\_(ツ)_/¯

Quoted 2 lines

It should be fine.

AFAICT, the people who benefit from GFP_ATOMIC are the people running all their storage over NFS/iSCSI who are suspending their machines while they're so busy they don't have any clean order 4 pagecache to drop, and want the machine to panic rather than hang.

From: Eric Dumazet <[email protected]>
Date: Mon, 30 Nov 2015 09:58:23 -0800

Quoted 15 lines · Pavel Machek

Quoted 14 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +0 -0

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 

Quoted 15 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 11 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +3 -3

-	ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
-				&ring_header->dma);
+	ring_header->desc = dma_alloc_coherent(&pdev->dev, ring_header->size,
+					       &ring_header->dma, GFP_KERNEL);
 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memmory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

It seems there is a missed opportunity to get rid of the memset() here, by adding __GFP_ZERO to the dma_alloc_coherent() GFP_KERNEL mask, or simply using dma_zalloc_coherent()

Also, the Subject line needs to be adjusted. The proper format for the Subject line is:

[PATCH $TREE] $subsystem: $description.

Where "$TREE" is either 'net' or 'net-next', $subsystem is the lowercase name of the driver (here 'atl1c') and then a colon, and then a space, and then the single-line description.

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

atl1c_setup_ring_resources() is called from .open() function, and already uses GFP_KERNEL, so this change is safe.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +3 -3

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 
-	ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
-				&ring_header->dma);
+	ring_header->desc = dma_alloc_coherent(&pdev->dev, ring_header->size,
+					       &ring_header->dma, GFP_KERNEL);
 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

Signature · 3 lines

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

atl1c_setup_ring_resources() is called from .open() function, and already uses GFP_KERNEL, so this change is safe.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +0 -0

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 

Quoted 19 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

atl1c_setup_ring_resources() is called from .open() function, and already uses GFP_KERNEL, so this change is safe.

Signed-off-by Pavel Machek <[email protected]>

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +0 -0

index 2795d6d..afb71e0 100644
@@ -1016,10 +1016,10 @@ static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter)
 		sizeof(struct atl1c_recv_ret_status) * rx_desc_count +
 		8 * 4;
 

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 10 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +3 -3

-	ring_header->desc = pci_alloc_consistent(pdev, ring_header->size,
-				&ring_header->dma);
+	ring_header->desc = dma_alloc_coherent(&pdev->dev, ring_header->size,
+					       &ring_header->dma, GFP_KERNEL);
 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

So this memset() will really require a different patch to get removed ?

Sigh, not sure why I review patches.

Quoted 1 line · Chris Snook

On Tue, Dec 1, 2015 at 12:35 PM David Miller <[email protected]>

Quoted 17 lines · wrote:

Quoted 17 lines

From: Michal Hocko <[email protected]>
Date: Mon, 30 Nov 2015 14:21:29 +0100

Quoted 8 lines · Pavel Machek

Quoted 4 lines

atl1c driver is doing order-4 allocation with GFP_ATOMIC priority. That often breaks networking after resume. Switch to GFP_KERNEL. Still not ideal, but should be significantly better.

It is not clear why GFP_KERNEL can replace GFP_ATOMIC safely neither from the changelog nor from the patch context.

Earlier in the function we do a GFP_KERNEL kmalloc so:

¯\_(ツ)_/¯

Quoted 7 lines

Quoted 2 lines

It should be fine.

AFAICT, the people who benefit from GFP_ATOMIC are the people running all their storage over NFS/iSCSI who are suspending their machines while they're so busy they don't have any clean order 4 pagecache to drop, and want the machine to panic rather than hang.

Why would GFP_KERNEL order-4 allocation hang? It will fail if there are not >=4 order pages available even after reclaim and/or compaction. GFP_ATOMIC allocations should be used only when an access to memory reserves is really required. If the allocation just doesn't want to invoke direct reclaim then GFP_NOWAIT is a more suitable alternative.

Quoted 11 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 8 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +1 -1

 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

So this memset() will really require a different patch to get removed ?

Quoted 1 line

Sigh, not sure why I review patches.

Agreed, please use dma_zalloc_coherent() and kill that memset().

Quoted 11 lines

Quoted 11 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 8 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +1 -1

 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

So this memset() will really require a different patch to get removed ?

Quoted 3 lines

Quoted 1 line

Sigh, not sure why I review patches.

Agreed, please use dma_zalloc_coherent() and kill that memset().

Ok, updated. I'll also add cc: stable, because it makes notebooks with affected chipset unusable.

From: Pavel Machek <[email protected]>
Date: Fri, 4 Dec 2015 09:11:27 +0100

Quoted 13 lines

Quoted 13 lines

Quoted 12 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 8 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +1 -1

 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

So this memset() will really require a different patch to get removed ?

Sigh, not sure why I review patches.

Quoted 4 lines

Quoted 1 line

Agreed, please use dma_zalloc_coherent() and kill that memset().

Ok, updated. I'll also add cc: stable, because it makes notebooks with affected chipset unusable.

Networking patches do not use CC: stable, instead you simply ask me to queue it up and then I batch submit networking fixes to -stable periodically myself.

Quoted 21 lines

From: Pavel Machek <[email protected]>
Date: Fri, 4 Dec 2015 09:11:27 +0100

Quoted 17 lines

Quoted 11 lines

Quoted 11 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c static int atl1c_setup_ring_resources(struct atl1c_adapter *adapter) · line 1025

Quoted 8 lines

drivers/net/ethernet/atheros/atl1c/atl1c_main.c +1 -1

 	if (unlikely(!ring_header->desc)) {
-		dev_err(&pdev->dev, "pci_alloc_consistend failed\n");
+		dev_err(&pdev->dev, "could not get memory for DMA buffer\n");
 		goto err_nomem;
 	}
 	memset(ring_header->desc, 0, ring_header->size);

So this memset() will really require a different patch to get removed ?

Quoted 3 lines

Quoted 1 line

Sigh, not sure why I review patches.

Agreed, please use dma_zalloc_coherent() and kill that memset().

Ok, updated. I'll also add cc: stable, because it makes notebooks with affected chipset unusable.

Quoted 3 lines

Networking patches do not use CC: stable, instead you simply ask me to queue it up and then I batch submit networking fixes to -stable periodically myself.

Ok, can you take the patch and ignore the Cc, or should I do one more iteration?