ENA driver bug fixes
Subsystems (3)
[email protected]
maintainercover ENA driver bug fixes
From: David Arinzon <[email protected]>
ENA driver bug fixes
David Arinzon (7):
net: ena: Fix toeplitz initial hash value
net: ena: Don't register memory info on XDP exchange
net: ena: Account for the number of processed bytes in XDP
net: ena: Use bitmask to indicate packet redirection
net: ena: Fix rx_copybreak value update
net: ena: Set default value for RX interrupt moderation
net: ena: Update NUMA TPH hint register upon NUMA node update
drivers/net/ethernet/amazon/ena/ena_com.c | 29 ++----- drivers/net/ethernet/amazon/ena/ena_ethtool.c | 6 +- drivers/net/ethernet/amazon/ena/ena_netdev.c | 83 ++++++++++++++----- drivers/net/ethernet/amazon/ena/ena_netdev.h | 17 +++- 4 files changed, 85 insertions(+), 50 deletions(-)
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 1/7 net: ena: Fix toeplitz initial hash value
From: David Arinzon <[email protected]>
On driver initialization, RSS hash initial value is set to zero, instead of the default value. This happens because we pass NULL as the RSS key parameter, which caused us to never initialize the RSS hash value.
This patch fixes it by making sure the initial value is set, no matter what the value of the RSS key is.
--- drivers/net/ethernet/amazon/ena/ena_com.c | 29 +++++++---------------- 1 file changed, 9 insertions(+), 20 deletions(-)
drivers/net/ethernet/amazon/ena/ena_com.c +9 -20
@@ -2400,29 +2400,18 @@ int ena_com_fill_hash_function(struct ena_com_dev *ena_dev,return -EOPNOTSUPP ; } -switch (func) { -case ENA_ADMIN_TOEPLITZ : -if (key) { -if (key_len !=sizeof (hash_key->key)) { -netdev_err (ena_dev->net_device, -"key len (%u) doesn't equal the supported size (%zu)\n" , - key_len,sizeof (hash_key->key)); -return -EINVAL ; - } -memcpy (hash_key->key, key, key_len); - rss->hash_init_val = init_val; - hash_key->key_parts = key_len /sizeof (hash_key->key[0 ]); +if ((func ==ENA_ADMIN_TOEPLITZ ) && key) { +if (key_len !=sizeof (hash_key->key)) { +netdev_err (ena_dev->net_device, +"key len (%u) doesn't equal the supported size (%zu)\n" , + key_len,sizeof (hash_key->key)); +return -EINVAL ; } -break ; -case ENA_ADMIN_CRC32 : - rss->hash_init_val = init_val; -break ; -default : -netdev_err (ena_dev->net_device,"Invalid hash function (%d)\n" , - func); -return -EINVAL ; +memcpy (hash_key->key, key, key_len); + hash_key->key_parts = key_len /sizeof (hash_key->key[0 ]); } + rss->hash_init_val = init_val; old_func = rss->hash_func; rss->hash_func = func; rc =ena_com_set_hash_function (ena_dev);
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 2/7 net: ena: Don't register memory info on XDP exchange
In reply to [email protected]From: David Arinzon <[email protected]>
Since the queues aren't destroyed when we only exchange XDP programs, there's no need to re-register them again.
--- drivers/net/ethernet/amazon/ena/ena_netdev.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +5 -3
@@ -512,16 +512,18 @@ static void ena_xdp_exchange_program_rx_in_range(struct ena_adapter *adapter,struct bpf_prog *prog, int first, int count) { + struct bpf_prog *old_bpf_prog;struct ena_ring *rx_ring;int i =0 ;for (i = first; i < count; i++) { rx_ring = &adapter->rx_ring[i]; -xchg (&rx_ring->xdp_bpf_prog, prog); -if (prog) { + old_bpf_prog =xchg (&rx_ring->xdp_bpf_prog, prog); + +if (!old_bpf_prog && prog) {ena_xdp_register_rxq_info (rx_ring); rx_ring->rx_headroom =XDP_PACKET_HEADROOM ; - }else { + }else if (old_bpf_prog && !prog) {ena_xdp_unregister_rxq_info (rx_ring); rx_ring->rx_headroom =NET_SKB_PAD ; }
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 3/7 net: ena: Account for the number of processed bytes in XDP
In reply to [email protected]From: David Arinzon <[email protected]>
The size of packets that were forwarded or dropped by XDP wasn't added to the total processed bytes statistic.
--- drivers/net/ethernet/amazon/ena/ena_netdev.c | 1 + 1 file changed, 1 insertion(+)
drivers/net/ethernet/amazon/ena/ena_netdev.c +1 -0
@@ -1719,6 +1719,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi, }if (xdp_verdict !=XDP_PASS ) { xdp_flags |= xdp_verdict; + total_len += ena_rx_ctx.ena_bufs[0 ].len; res_budget--;continue ; }
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 4/7 net: ena: Use bitmask to indicate packet redirection
In reply to [email protected]From: David Arinzon <[email protected]>
Redirecting packets with XDP Redirect is done in two phases:
1. A packet is passed by the driver to the kernel using xdp_do_redirect().
2. After finishing polling for new packets the driver lets the kernel know that it can now process the redirected packet using xdp_do_flush_map().
The packets' redirection is handled in the napi context of the queue that called xdp_do_redirect()
To avoid calling xdp_do_flush_map() each time the driver first checks whether any packets were redirected, using
xdp_flags |= xdp_verdict; and if (xdp_flags & XDP_REDIRECT) xdp_do_flush_map()
essentially treating XDP instructions as a bitmask, which isn't the case:
enum xdp_action {
XDP_ABORTED = 0,
XDP_DROP,
XDP_PASS,
XDP_TX,
XDP_REDIRECT,
};Given the current possible values of xdp_action, the current design doesn't have a bug (since XDP_REDIRECT = 100b), but it is still flawed.
This patch makes the driver use a bitmask instead, to avoid future issues.
--- drivers/net/ethernet/amazon/ena/ena_netdev.c | 26 ++++++++++++-------- drivers/net/ethernet/amazon/ena/ena_netdev.h | 9 +++++++ 2 files changed, 25 insertions(+), 10 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +16 -10
@@ -374,9 +374,9 @@ static int ena_xdp_xmit(struct net_device *dev, int n,static int ena_xdp_execute (struct ena_ring *rx_ring,struct xdp_buff *xdp) { +u32 verdict =ENA_XDP_PASS ;struct bpf_prog *xdp_prog;struct ena_ring *xdp_ring; -u32 verdict =XDP_PASS ;struct xdp_frame *xdpf;u64 *xdp_stat;
@@ -393,7 +393,7 @@ static int ena_xdp_execute(struct ena_ring *rx_ring, struct xdp_buff *xdp)if (unlikely (!xdpf)) {trace_xdp_exception (rx_ring->netdev, xdp_prog, verdict); xdp_stat = &rx_ring->rx_stats.xdp_aborted; - verdict =XDP_ABORTED ; + verdict =ENA_XDP_DROP ;break ; }
@@ -409,29 +409,35 @@ static int ena_xdp_execute(struct ena_ring *rx_ring, struct xdp_buff *xdp)spin_unlock (&xdp_ring->xdp_tx_lock); xdp_stat = &rx_ring->rx_stats.xdp_tx; + verdict =ENA_XDP_TX ;break ;case XDP_REDIRECT :if (likely (!xdp_do_redirect (rx_ring->netdev, xdp, xdp_prog))) { xdp_stat = &rx_ring->rx_stats.xdp_redirect; + verdict =ENA_XDP_REDIRECT ;break ; }trace_xdp_exception (rx_ring->netdev, xdp_prog, verdict); xdp_stat = &rx_ring->rx_stats.xdp_aborted; - verdict =XDP_ABORTED ; + verdict =ENA_XDP_DROP ;break ;case XDP_ABORTED :trace_xdp_exception (rx_ring->netdev, xdp_prog, verdict); xdp_stat = &rx_ring->rx_stats.xdp_aborted; + verdict =ENA_XDP_DROP ;break ;case XDP_DROP : xdp_stat = &rx_ring->rx_stats.xdp_drop; + verdict =ENA_XDP_DROP ;break ;case XDP_PASS : xdp_stat = &rx_ring->rx_stats.xdp_pass; + verdict =ENA_XDP_PASS ;break ;default :bpf_warn_invalid_xdp_action (rx_ring->netdev, xdp_prog, verdict); xdp_stat = &rx_ring->rx_stats.xdp_invalid; + verdict =ENA_XDP_DROP ; }ena_increase_stat (xdp_stat,1 , &rx_ring->syncp);
@@ -1621,12 +1627,12 @@ static int ena_xdp_handle_buff(struct ena_ring *rx_ring, struct xdp_buff *xdp)* we expect, then we simply drop it */ if (unlikely (rx_ring->ena_bufs[0 ].len >ENA_XDP_MAX_MTU )) -return XDP_DROP ; +return ENA_XDP_DROP ; ret =ena_xdp_execute (rx_ring, xdp);/* The xdp program might expand the headers */ -if (ret ==XDP_PASS ) { +if (ret ==ENA_XDP_PASS ) { rx_info->page_offset = xdp->data - xdp->data_hard_start; rx_ring->ena_bufs[0 ].len = xdp->data_end - xdp->data; }
@@ -1665,7 +1671,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,xdp_init_buff (&xdp,ENA_PAGE_SIZE , &rx_ring->xdp_rxq); do { - xdp_verdict = XDP_PASS; + xdp_verdict = ENA_XDP_PASS; skb = NULL; ena_rx_ctx.ena_bufs = rx_ring->ena_bufs; ena_rx_ctx.max_bufs = rx_ring->sgl_size;
@@ -1693,7 +1699,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi, xdp_verdict =ena_xdp_handle_buff (rx_ring, &xdp);/* allocate skb and fill it */ -if (xdp_verdict ==XDP_PASS ) +if (xdp_verdict ==ENA_XDP_PASS ) skb = ena_rx_skb(rx_ring, rx_ring->ena_bufs, ena_rx_ctx.descs,
@@ -1711,13 +1717,13 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,/* Packets was passed for transmission, unmap it * from RX side. */ -if (xdp_verdict ==XDP_TX || xdp_verdict ==XDP_REDIRECT ) { +if (xdp_verdict &ENA_XDP_FORWARDED ) {ena_unmap_rx_buff (rx_ring, &rx_ring->rx_buffer_info[req_id]); rx_ring->rx_buffer_info[req_id].page =NULL ; } } -if (xdp_verdict !=XDP_PASS ) { +if (xdp_verdict !=ENA_XDP_PASS ) { xdp_flags |= xdp_verdict; total_len += ena_rx_ctx.ena_bufs[0 ].len; res_budget--;
@@ -1763,7 +1769,7 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,ena_refill_rx_bufs (rx_ring, refill_required); } -if (xdp_flags &XDP_REDIRECT ) +if (xdp_flags &ENA_XDP_REDIRECT )xdp_do_flush_map ();return work_done;
drivers/net/ethernet/amazon/ena/ena_netdev.h +9 -0
@@ -409,6 +409,15 @@ enum ena_xdp_errors_t { ENA_XDP_NO_ENOUGH_QUEUES, }; +enum ENA_XDP_ACTIONS { +ENA_XDP_PASS =0 , +ENA_XDP_TX =BIT (0 ), +ENA_XDP_REDIRECT =BIT (1 ), +ENA_XDP_DROP =BIT (2 ) +}; + +#define ENA_XDP_FORWARDED (ENA_XDP_TX | ENA_XDP_REDIRECT) +static inline bool ena_xdp_present (struct ena_adapter *adapter) {return !!adapter->xdp_bpf_prog;
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 6/7 net: ena: Set default value for RX interrupt moderation
In reply to [email protected]From: David Arinzon <[email protected]>
RX ring can be NULL in XDP use cases where only TX queues are configured. In this scenario, the RX interrupt moderation value sent to the device remains in its default value of 0.
In this change, setting the default value of the RX interrupt moderation to be the same as of the TX.
--- drivers/net/ethernet/amazon/ena/ena_netdev.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +2 -1
@@ -1823,8 +1823,9 @@ static void ena_adjust_adaptive_rx_intr_moderation(struct ena_napi *ena_napi)static void ena_unmask_interrupt (struct ena_ring *tx_ring,struct ena_ring *rx_ring) { +u32 rx_interval = tx_ring->smoothed_interval;struct ena_eth_io_intr_reg intr_reg; -u32 rx_interval =0 ; +/* Rx ring can be NULL when for XDP tx queues which don't have an * accompanying rx_ring pair. */
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 7/7 net: ena: Update NUMA TPH hint register upon NUMA node update
In reply to [email protected]From: David Arinzon <[email protected]>
The device supports a PCIe optimization hint, which indicates on which NUMA the queue is currently processed. This hint is utilized by PCIe in order to reduce its access time by accessing the correct NUMA resources and maintaining cache coherence.
The driver calls the register update for the hint (called TPH - TLP Processing Hint) during the NAPI loop.
Though the update is expected upon a NUMA change (when a queue is moved from one NUMA to the other), the current logic performs a register update when the queue is moved to a different CPU, but the CPU is not necessarily in a different NUMA.
The changes include:
1. Performing the TPH update only when the queue has switched a NUMA node.
2. Moving the TPH update call to be triggered only when NAPI was scheduled from interrupt context, as opposed to a busy-polling loop. This is due to the fact that during busy-polling, the frequency of CPU switches for a particular queue is significantly higher, thus, the likelihood to switch NUMA is much higher. Therefore, providing the frequent updates to the device upon a NUMA update are unlikely to be beneficial.
--- drivers/net/ethernet/amazon/ena/ena_netdev.c | 27 +++++++++++++------- drivers/net/ethernet/amazon/ena/ena_netdev.h | 6 +++-- 2 files changed, 22 insertions(+), 11 deletions(-)
drivers/net/ethernet/amazon/ena/ena_netdev.c +18 -9
@@ -680,6 +680,7 @@ static void ena_init_io_rings_common(struct ena_adapter *adapter, ring->ena_dev = adapter->ena_dev; ring->per_napi_packets =0 ; ring->cpu =0 ; + ring->numa_node =0 ; ring->no_interrupt_event_cnt =0 ;u64_stats_init (&ring->syncp); }
@@ -783,6 +784,7 @@ static int ena_setup_tx_resources(struct ena_adapter *adapter, int qid) tx_ring->next_to_use =0 ; tx_ring->next_to_clean =0 ; tx_ring->cpu = ena_irq->cpu; + tx_ring->numa_node = node;return 0 ; err_push_buf_intermediate_buf:
@@ -915,6 +917,7 @@ static int ena_setup_rx_resources(struct ena_adapter *adapter, rx_ring->next_to_clean =0 ; rx_ring->next_to_use =0 ; rx_ring->cpu = ena_irq->cpu; + rx_ring->numa_node = node;return 0 ; }
@@ -1863,20 +1866,27 @@ static void ena_update_ring_numa_node(struct ena_ring *tx_ring,if (likely (tx_ring->cpu == cpu)) goto out; + tx_ring->cpu = cpu; +if (rx_ring) + rx_ring->cpu = cpu; + numa_node =cpu_to_node (cpu); + +if (likely (tx_ring->numa_node == numa_node)) + goto out; +put_cpu ();if (numa_node !=NUMA_NO_NODE ) {ena_com_update_numa_node (tx_ring->ena_com_io_cq, numa_node); -if (rx_ring) + tx_ring->numa_node = numa_node; +if (rx_ring) { + rx_ring->numa_node = numa_node;ena_com_update_numa_node (rx_ring->ena_com_io_cq, numa_node); + } } - tx_ring->cpu = cpu; -if (rx_ring) - rx_ring->cpu = cpu; -return ; out:put_cpu ();
@@ -1997,11 +2007,10 @@ static int ena_io_poll(struct napi_struct *napi, int budget)if (ena_com_get_adaptive_moderation_enabled (rx_ring->ena_dev))ena_adjust_adaptive_rx_intr_moderation (ena_napi); +ena_update_ring_numa_node (tx_ring, rx_ring);ena_unmask_interrupt (tx_ring, rx_ring); } -ena_update_ring_numa_node (tx_ring, rx_ring); - ret = rx_work_done; }else { ret = budget;
@@ -2386,7 +2395,7 @@ static int ena_create_io_tx_queue(struct ena_adapter *adapter, int qid) ctx.mem_queue_type = ena_dev->tx_mem_queue_type; ctx.msix_vector = msix_vector; ctx.queue_size = tx_ring->ring_size; - ctx.numa_node =cpu_to_node (tx_ring->cpu); + ctx.numa_node = tx_ring->numa_node; rc =ena_com_create_io_queue (ena_dev, &ctx);if (rc) {
@@ -2454,7 +2463,7 @@ static int ena_create_io_rx_queue(struct ena_adapter *adapter, int qid) ctx.mem_queue_type =ENA_ADMIN_PLACEMENT_POLICY_HOST ; ctx.msix_vector = msix_vector; ctx.queue_size = rx_ring->ring_size; - ctx.numa_node =cpu_to_node (rx_ring->cpu); + ctx.numa_node = rx_ring->numa_node; rc =ena_com_create_io_queue (ena_dev, &ctx);if (rc) {
drivers/net/ethernet/amazon/ena/ena_netdev.h +4 -2
@@ -262,9 +262,11 @@ struct ena_ring {bool disable_meta_caching;u16 no_interrupt_event_cnt; -/* cpu for TPH */ +/* cpu and NUMA for TPH */ int cpu; -/* number of tx/rx_buffer_info's entries */ +int numa_node; + +/* number of tx/rx_buffer_info's entries */ int ring_size;enum ena_admin_placement_policy_type tx_mem_queue_type;
Signature · 1 line
2.38.1
[email protected]
maintainerPATCH 5/7 net: ena: Fix rx_copybreak value update
In reply to [email protected]From: David Arinzon <[email protected]>
Make the upper bound on rx_copybreak tighter, by making sure it is smaller than the minimum of mtu and ENA_PAGE_SIZE. With the current upper bound of mtu, rx_copybreak can be larger than a page. Such large rx_copybreak will not bring any performance benefit to the user and therefore makes no sense.
In addition, the value update was only reflected in the adapter structure, but not applied for each ring, causing it to not take effect.
--- drivers/net/ethernet/amazon/ena/ena_ethtool.c | 6 +----- drivers/net/ethernet/amazon/ena/ena_netdev.c | 18 ++++++++++++++++++ drivers/net/ethernet/amazon/ena/ena_netdev.h | 2 ++ 3 files changed, 21 insertions(+), 5 deletions(-)
drivers/net/ethernet/amazon/ena/ena_ethtool.c +1 -5
@@ -887,11 +887,7 @@ static int ena_set_tunable(struct net_device *netdev,switch (tuna->id) {case ETHTOOL_RX_COPYBREAK : len = *(u32 *)data; -if (len > adapter->netdev->mtu) { - ret = -EINVAL ; -break ; - } - adapter->rx_copybreak = len; + ret =ena_set_rx_copybreak (adapter, len);break ;default : ret = -EINVAL ;
drivers/net/ethernet/amazon/ena/ena_netdev.c +18 -0
@@ -2814,6 +2814,24 @@ int ena_update_queue_sizes(struct ena_adapter *adapter,return dev_was_up ?ena_up (adapter) :0 ; } +int ena_set_rx_copybreak (struct ena_adapter *adapter,u32 rx_copybreak) +{ +struct ena_ring *rx_ring; +int i; + +if (rx_copybreak >min_t (u16, adapter->netdev->mtu,ENA_PAGE_SIZE )) +return -EINVAL ; + + adapter->rx_copybreak = rx_copybreak; + +for (i =0 ; i < adapter->num_io_queues; i++) { + rx_ring = &adapter->rx_ring[i]; + rx_ring->rx_copybreak = rx_copybreak; + } + +return 0 ; +} +int ena_update_queue_count (struct ena_adapter *adapter,u32 new_channel_count) {struct ena_com_dev *ena_dev = adapter->ena_dev;
drivers/net/ethernet/amazon/ena/ena_netdev.h +2 -0
@@ -392,6 +392,8 @@ int ena_update_queue_sizes(struct ena_adapter *adapter,int ena_update_queue_count (struct ena_adapter *adapter,u32 new_channel_count); +int ena_set_rx_copybreak (struct ena_adapter *adapter,u32 rx_copybreak); +int ena_get_sset_count (struct net_device *netdev,int sset); static inline void ena_reset_device(struct ena_adapter *adapter,
Signature · 1 line
2.38.1
This series was applied to netdev/net.git (master) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net.git (master) by David S. Miller <[email protected]>:
Quoted 13 lines · On Thu, 29 Dec 2022 07:30:04 +0000 you wrote:
From: David Arinzon <[email protected]>
ENA driver bug fixes
David Arinzon (7):
net: ena: Fix toeplitz initial hash value
net: ena: Don't register memory info on XDP exchange
net: ena: Account for the number of processed bytes in XDP
net: ena: Use bitmask to indicate packet redirection
net: ena: Fixrx_copybreakvalue update
net: ena: Set default value for RX interrupt moderation
net: ena: Update NUMA TPH hint register upon NUMA node update
Quoted 1 line
[...]
Here is the summary with links:
- [V1,net,1/7] net: ena: Fix toeplitz initial hash value
https://git.kernel.org/netdev/net/c/332b49ff637d - [V1,net,2/7] net: ena: Don't register memory info on XDP exchange https://git.kernel.org/netdev/net/c/9c9e539956fa - [V1,net,3/7] net: ena: Account for the number of processed bytes in XDP https://git.kernel.org/netdev/net/c/c7f5e34d9063 - [V1,net,4/7] net: ena: Use bitmask to indicate packet redirection https://git.kernel.org/netdev/net/c/59811faa2c54 - [V1,net,5/7] net: ena: Fix rx_copybreak value update https://git.kernel.org/netdev/net/c/c7062aaee099 - [V1,net,6/7] net: ena: Set default value for RX interrupt moderation https://git.kernel.org/netdev/net/c/e712f3e4920b - [V1,net,7/7] net: ena: Update NUMA TPH hint register upon NUMA node update https://git.kernel.org/netdev/net/c/a8ee104f986e
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html