igb: cope with large MAX_SKB_FRAGS
Subsystems (3)
PATCH v2 1/1 igb: cope with large MAX_SKB_FRAGS
From: Paolo Abeni <[email protected]>
Sabrina reports that the igb driver does not cope well with large MAX_SKB_FRAG values: setting MAX_SKB_FRAG to 45 causes payload corruption on TX.
An easy reproducer is to run ssh to connect to the machine. With MAX_SKB_FRAGS=17 it works, with MAX_SKB_FRAGS=45 it fails.
The root cause of the issue is that the driver does not take into account properly the (possibly large) shared info size when selecting the ring layout, and will try to fit two packets inside the same 4K page even when the 1st fraglist will trump over the 2nd head.
Address the issue forcing the driver to fit a single packet per page, leaving there enough room to store the (currently) largest possible skb_shared_info.
---
v2: fix subject, add a simple reproducer
drivers/net/ethernet/intel/igb/igb_main.c | 1 + 1 file changed, 1 insertion(+)
drivers/net/ethernet/intel/igb/igb_main.c +1 -0
@@ -4833,6 +4833,7 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter,#if (PAGE_SIZE <8192 )if (adapter->max_frame_size >IGB_MAX_FRAME_BUILD_SKB || +SKB_HEAD_ALIGN (adapter->max_frame_size) > (PAGE_SIZE /2 ) ||rd32 (E1000_RCTL ) &E1000_RCTL_SBP )set_ring_uses_large_buffer (rx_ring);#endif
Signature · 1 line
2.44.0
Quoted 37 lines · Corinna Vinschen
From: Paolo Abeni <[email protected]>
Sabrina reports that the igb driver does not cope well with large
MAX_SKB_FRAGvalues: settingMAX_SKB_FRAGto 45 causes payload corruption on TX.An easy reproducer is to run ssh to connect to the machine. With MAX_SKB_FRAGS=17 it works, with MAX_SKB_FRAGS=45 it fails.
The root cause of the issue is that the driver does not take into account properly the (possibly large) shared info size when selecting the ring layout, and will try to fit two packets inside the same 4K page even when the 1st fraglist will trump over the 2nd head.
Address the issue forcing the driver to fit a single packet per page, leaving there enough room to store the (currently) largest possible
skb_shared_info.Fixes 3948b05950fd ("net: introduce a config option to tweak MAX_SKB_FRAG")Reported-by Jan Tluka <[email protected]>Reported-by Jirka Hladky <[email protected]>Reported-by Sabrina Dubroca <[email protected]>Tested-by Sabrina Dubroca <[email protected]>Tested-by Corinna Vinschen <[email protected]>Signed-off-by Paolo Abeni <[email protected]>---
v2: fix subject, add a simple reproducerdrivers/net/ethernet/intel/igb/igb_main.c | 1 + 1 file changed, 1 insertion(+)drivers/net/ethernet/intel/igb/igb_main.c +0
-0
drivers/net/ethernet/intel/igb/igb_main.c static void igb_set_rx_buffer_len(struct igb_adapter *adapter, · line 4836
Quoted 4 lines
drivers/net/ethernet/intel/igb/igb_main.c +1
-0@@ -4833,6 +4833,7 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter,#if (PAGE_SIZE <8192 ) if (adapter->max_frame_size > IGB_MAX_FRAME_BUILD_SKB || + SKB_HEAD_ALIGN(adapter->max_frame_size) > (PAGE_SIZE / 2) ||
I wonder if adding IGB_SKB_PAD would be needed ?
adapter->max_frame_size does not seem to include it.
I would try using all mtus between 1200 and 1280 to make sure this works.
Thanks !
Quoted 6 lines
drivers/net/ethernet/intel/igb/igb_main.c +0
-1rd32(E1000_RCTL) & E1000_RCTL_SBP)set_ring_uses_large_buffer (rx_ring);#endif --2.44.0
Hi Eric,
Quoted 45 lines · Eric Dumazet
Quoted 37 lines · Corinna Vinschen
From: Paolo Abeni <[email protected]>
Sabrina reports that the igb driver does not cope well with large
MAX_SKB_FRAGvalues: settingMAX_SKB_FRAGto 45 causes payload corruption on TX.An easy reproducer is to run ssh to connect to the machine. With MAX_SKB_FRAGS=17 it works, with MAX_SKB_FRAGS=45 it fails.
The root cause of the issue is that the driver does not take into account properly the (possibly large) shared info size when selecting the ring layout, and will try to fit two packets inside the same 4K page even when the 1st fraglist will trump over the 2nd head.
Address the issue forcing the driver to fit a single packet per page, leaving there enough room to store the (currently) largest possible
skb_shared_info.Fixes 3948b05950fd ("net: introduce a config option to tweak MAX_SKB_FRAG")Reported-by Jan Tluka <[email protected]>Reported-by Jirka Hladky <[email protected]>Reported-by Sabrina Dubroca <[email protected]>Tested-by Sabrina Dubroca <[email protected]>Tested-by Corinna Vinschen <[email protected]>Signed-off-by Paolo Abeni <[email protected]>---
v2: fix subject, add a simple reproducerdrivers/net/ethernet/intel/igb/igb_main.c | 1 + 1 file changed, 1 insertion(+)drivers/net/ethernet/intel/igb/igb_main.c +0
-0drivers/net/ethernet/intel/igb/igb_main.c static void igb_set_rx_buffer_len(struct igb_adapter *adapter, · line 4836
Quoted 4 lines
drivers/net/ethernet/intel/igb/igb_main.c +1
-0@@ -4833,6 +4833,7 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter,#if (PAGE_SIZE <8192 ) if (adapter->max_frame_size > IGB_MAX_FRAME_BUILD_SKB || + SKB_HEAD_ALIGN(adapter->max_frame_size) > (PAGE_SIZE / 2) ||I wonder if adding
IGB_SKB_PADwould be needed ?
Quoted 3 lines
adapter->max_frame_sizedoes not seem to include it.I would try using all mtus between 1200 and 1280 to make sure this works.
Erm... did you mean between 1500 and 1580 by any chance? 1200 doesn't really seem to make sense...
I tested this patch now with mtu 1500, 1540 and 1580 successfully.
Either way, I'm just heading into vacation, so I guess I'll pick this up again when I'm back, unless Paolo takes another look during my absence.
Thanks,
Corinna
Quoted 49 lines · Corinna Vinschen
Hi Eric,
Quoted 45 lines · Eric Dumazet
Quoted 37 lines · Corinna Vinschen
From: Paolo Abeni <[email protected]>
Sabrina reports that the igb driver does not cope well with large
MAX_SKB_FRAGvalues: settingMAX_SKB_FRAGto 45 causes payload corruption on TX.An easy reproducer is to run ssh to connect to the machine. With MAX_SKB_FRAGS=17 it works, with MAX_SKB_FRAGS=45 it fails.
The root cause of the issue is that the driver does not take into account properly the (possibly large) shared info size when selecting the ring layout, and will try to fit two packets inside the same 4K page even when the 1st fraglist will trump over the 2nd head.
Address the issue forcing the driver to fit a single packet per page, leaving there enough room to store the (currently) largest possible
skb_shared_info.Fixes 3948b05950fd ("net: introduce a config option to tweak MAX_SKB_FRAG")Reported-by Jan Tluka <[email protected]>Reported-by Jirka Hladky <[email protected]>Reported-by Sabrina Dubroca <[email protected]>Tested-by Sabrina Dubroca <[email protected]>Tested-by Corinna Vinschen <[email protected]>Signed-off-by Paolo Abeni <[email protected]>---
v2: fix subject, add a simple reproducerdrivers/net/ethernet/intel/igb/igb_main.c | 1 + 1 file changed, 1 insertion(+)drivers/net/ethernet/intel/igb/igb_main.c +0
-0drivers/net/ethernet/intel/igb/igb_main.c static void igb_set_rx_buffer_len(struct igb_adapter *adapter, · line 4836
Quoted 4 lines
drivers/net/ethernet/intel/igb/igb_main.c +1
-0@@ -4833,6 +4833,7 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter,#if (PAGE_SIZE <8192 ) if (adapter->max_frame_size > IGB_MAX_FRAME_BUILD_SKB || + SKB_HEAD_ALIGN(adapter->max_frame_size) > (PAGE_SIZE / 2) ||I wonder if adding
IGB_SKB_PADwould be needed ?
Quoted 6 lines
Quoted 3 lines
adapter->max_frame_sizedoes not seem to include it.I would try using all mtus between 1200 and 1280 to make sure this works.
Erm... did you mean between 1500 and 1580 by any chance? 1200 doesn't really seem to make sense...
No, I meant 1200 to 1280 . IPv4 should accept these MTU .
1200 + 768 = 1968
1280 + 768 = 2048 (2 KB)
I am worried of some padding that would cross 2048 bytes boundary, while SKB_HEAD_ALIGN(adapter->max_frame_size) could still be < 2048
Quoted 6 lines
I tested this patch now with mtu 1500, 1540 and 1580 successfully.
Either way, I'm just heading into vacation, so I guess I'll pick this up again when I'm back, unless Paolo takes another look during my absence.
I guess your patch is better than nothing, this can be refined if necessary later.
Hi Eric,
sorry for the very late reply, I was offline for quite some time and then other things took over.
Quoted 41 lines · Eric Dumazet
Quoted 23 lines · Corinna Vinschen
Hi Eric,
Quoted 19 lines · Eric Dumazet
Quoted 11 lines · Corinna Vinschen
From: Paolo Abeni <[email protected]>
Sabrina reports that the igb driver does not cope well with large
MAX_SKB_FRAGvalues: settingMAX_SKB_FRAGto 45 causes payload corruption on TX.An easy reproducer is to run ssh to connect to the machine. With MAX_SKB_FRAGS=17 it works, with MAX_SKB_FRAGS=45 it fails. [...]
drivers/net/ethernet/intel/igb/igb_main.c static void igb_set_rx_buffer_len(struct igb_adapter *adapter, · line 4836
Quoted 4 lines
drivers/net/ethernet/intel/igb/igb_main.c +1
-0@@ -4833,6 +4833,7 @@ static void igb_set_rx_buffer_len(struct igb_adapter *adapter,#if (PAGE_SIZE <8192 ) if (adapter->max_frame_size > IGB_MAX_FRAME_BUILD_SKB || + SKB_HEAD_ALIGN(adapter->max_frame_size) > (PAGE_SIZE / 2) ||I wonder if adding
IGB_SKB_PADwould be needed ?Quoted 6 lines
Quoted 3 lines
adapter->max_frame_sizedoes not seem to include it.I would try using all mtus between 1200 and 1280 to make sure this works.
Erm... did you mean between 1500 and 1580 by any chance? 1200 doesn't really seem to make sense...
No, I meant 1200 to 1280 . IPv4 should accept these MTU .
1200 + 768 = 1968
1280 + 768 = 2048 (2 KB)I am worried of some padding that would cross 2048 bytes boundary, while
SKB_HEAD_ALIGN(adapter->max_frame_size)could still be < 2048Quoted 1 line
Quoted 8 lines
Quoted 5 lines
I tested this patch now with mtu 1500, 1540 and 1580 successfully.
Either way, I'm just heading into vacation, so I guess I'll pick this up again when I'm back, unless Paolo takes another look during my absence.
I guess your patch is better than nothing, this can be refined if necessary later.
I now tested the patch with all MTUs between 1200 and 1280 and it works as desired. I just sent a v3.
Thanks,
Corinna
Simon Horman
reviewerQuoted 14 lines · Corinna Vinschen
From: Paolo Abeni <[email protected]>
Sabrina reports that the igb driver does not cope well with large
MAX_SKB_FRAGvalues: settingMAX_SKB_FRAGto 45 causes payload corruption on TX.An easy reproducer is to run ssh to connect to the machine. With MAX_SKB_FRAGS=17 it works, with MAX_SKB_FRAGS=45 it fails.
The root cause of the issue is that the driver does not take into account properly the (possibly large) shared info size when selecting the ring layout, and will try to fit two packets inside the same 4K page even when the 1st fraglist will trump over the 2nd head.
Quoted 5 lines
Address the issue forcing the driver to fit a single packet per page, leaving there enough room to store the (currently) largest possible
skb_shared_info.Fixes 3948b05950fd ("net: introduce a config option to tweak MAX_SKB_FRAG")
nit: The trailing "S" in the subject for the fixes tag seems to have been lost.
Quoted 6 lines
Reported-by Jan Tluka <[email protected]>Reported-by Jirka Hladky <[email protected]>Reported-by Sabrina Dubroca <[email protected]>Tested-by Sabrina Dubroca <[email protected]>Tested-by Corinna Vinschen <[email protected]>Signed-off-by Paolo Abeni <[email protected]>
...