net: stmmac: improve driver statistics
Subsystems (5)
v5 cover net: stmmac: improve driver statistics
improve the stmmac driver statistics:
1. don't clear network driver statistics in .ndo_close() and .ndo_open() cycle
2. avoid some network driver statistics overflow on 32 bit platforms
3. use per-queue statistics where necessary to remove frequent cacheline ping pongs.
NOTE: v1 and v2 are back ported from an internal LTS tree, I made some mistakes when backporting and squashing. Now, net-next + v3 has been well tested with 'ethtool -s' and 'ip -s link show'.
Since v4:
- rebased on the latest net-next
Since v3:
- coding style pointed out by Simon, I.E reverse xmas tree for local variable declarations and so on.
- put the counters in queue structs, I.E per-queue rather than per-cpu
- use _irqsave() variant where necessary.
Since v2:
- fix ethtool .get_sset_count, .get_strings and per queue stats couting.
- fix .ndo_get_stats64 only counts the last cpu's pcpu stats.
- fix typo: s/iff/if in commit msg.
- remove unnecessary if statement brackets since we have removed one LoC.
Since v1:
- rebase on net-next
- fold two original patches into one patch
- fix issues found by lkp
- update commit msg
Jisheng Zhang (2):
net: stmmac: don't clear network statistics in .ndo_open()
net: stmmac: use per-queue 64 bit statistics where necessary
drivers/net/ethernet/stmicro/stmmac/common.h | 39 ++-- .../net/ethernet/stmicro/stmmac/dwmac-sun8i.c | 12 +- .../ethernet/stmicro/stmmac/dwmac100_dma.c | 7 +- .../ethernet/stmicro/stmmac/dwmac4_descs.c | 16 +- .../net/ethernet/stmicro/stmmac/dwmac4_lib.c | 15 +- .../net/ethernet/stmicro/stmmac/dwmac_lib.c | 12 +- .../ethernet/stmicro/stmmac/dwxgmac2_descs.c | 6 +- .../ethernet/stmicro/stmmac/dwxgmac2_dma.c | 14 +- .../net/ethernet/stmicro/stmmac/enh_desc.c | 20 +- drivers/net/ethernet/stmicro/stmmac/hwif.h | 12 +- .../net/ethernet/stmicro/stmmac/norm_desc.c | 15 +- drivers/net/ethernet/stmicro/stmmac/stmmac.h | 2 + .../ethernet/stmicro/stmmac/stmmac_ethtool.c | 123 ++++++++--- .../net/ethernet/stmicro/stmmac/stmmac_main.c | 206 ++++++++++++++---- 14 files changed, 337 insertions(+), 162 deletions(-)
Signature · 1 line
2.40.0
PATCH v5 1/2 net: stmmac: don't clear network statistics in .ndo_open()
FWICT, the common style in other network drivers: the network statistics are not cleared since initialization, follow the common style for stmmac.
--- drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +2 -4
@@ -3829,10 +3829,6 @@ static int __stmmac_open(struct net_device *dev, } } -/* Extra statistics */ -memset (&priv->xstats,0 ,sizeof (struct stmmac_extra_stats )); - priv->xstats.threshold = tc; - priv->rx_copybreak =STMMAC_RX_COPYBREAK ; buf_sz = dma_conf->dma_buf_sz;
@@ -7322,6 +7318,8 @@ int stmmac_dvr_probe(struct device *device,#endif priv->msg_enable =netif_msg_init (debug, default_msg_level); + priv->xstats.threshold = tc; +/* Initialize RSS */ rxq = priv->plat->rx_queues_to_use;netdev_rss_key_fill (priv->rss.key,sizeof (priv->rss.key));
Signature · 1 line
2.40.0
PATCH v5 2/2 net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Jisheng ZhangCurrently, there are two major issues with stmmac driver statistics First of all, statistics in stmmac_extra_stats, stmmac_rxq_stats and stmmac_txq_stats are 32 bit variables on 32 bit platforms. This can cause some stats to overflow after several minutes of high traffic, for example rx_pkt_n, tx_pkt_n and so on.
Secondly, if HW supports multiqueues, there are frequent cacheline ping pongs on some driver statistic vars, for example, normal_irq_n, tx_pkt_n and so on. What's more, frequent cacheline ping pongs on normal_irq_n happens in ISR, this makes the situation worse.
To improve the driver, we convert those statistics to 64 bit, implement ndo_get_stats64 and update .get_ethtool_stats implementation accordingly. We also use per-queue statistics where necessary to remove the cacheline ping pongs as much as possible to make multiqueue operations faster. Those statistics which are not possible to overflow and not frequently updated are kept as is.
--- drivers/net/ethernet/stmicro/stmmac/common.h | 39 ++-- .../net/ethernet/stmicro/stmmac/dwmac-sun8i.c | 12 +- .../ethernet/stmicro/stmmac/dwmac100_dma.c | 7 +- .../ethernet/stmicro/stmmac/dwmac4_descs.c | 16 +- .../net/ethernet/stmicro/stmmac/dwmac4_lib.c | 15 +- .../net/ethernet/stmicro/stmmac/dwmac_lib.c | 12 +- .../ethernet/stmicro/stmmac/dwxgmac2_descs.c | 6 +- .../ethernet/stmicro/stmmac/dwxgmac2_dma.c | 14 +- .../net/ethernet/stmicro/stmmac/enh_desc.c | 20 +- drivers/net/ethernet/stmicro/stmmac/hwif.h | 12 +- .../net/ethernet/stmicro/stmmac/norm_desc.c | 15 +- drivers/net/ethernet/stmicro/stmmac/stmmac.h | 2 + .../ethernet/stmicro/stmmac/stmmac_ethtool.c | 123 ++++++++--- .../net/ethernet/stmicro/stmmac/stmmac_main.c | 200 ++++++++++++++---- 14 files changed, 335 insertions(+), 158 deletions(-)
drivers/net/ethernet/stmicro/stmmac/common.h +21 -18
@@ -59,13 +59,25 @@/* #define FRAME_FILTER_DEBUG */ struct stmmac_txq_stats { -unsigned long tx_pkt_n; -unsigned long tx_normal_irq_n; +u64 tx_bytes; +u64 tx_packets; +u64 tx_pkt_n; +u64 tx_normal_irq_n; +u64 napi_poll; +u64 tx_clean; +u64 tx_set_ic_bit; +u64 tx_tso_frames; +u64 tx_tso_nfrags; +struct u64_stats_sync syncp; };struct stmmac_rxq_stats { -unsigned long rx_pkt_n; -unsigned long rx_normal_irq_n; +u64 rx_bytes; +u64 rx_packets; +u64 rx_pkt_n; +u64 rx_normal_irq_n; +u64 napi_poll; +struct u64_stats_sync syncp; };/* Extra statistic and debug information exposed by ethtool */
@@ -81,6 +93,7 @@ struct stmmac_extra_stats {unsigned long tx_frame_flushed;unsigned long tx_payload_error;unsigned long tx_ip_header_error; +unsigned long tx_collision;/* Receive errors */ unsigned long rx_desc;unsigned long sa_filter_fail;
@@ -113,14 +126,6 @@ struct stmmac_extra_stats {/* Tx/Rx IRQ Events */ unsigned long rx_early_irq;unsigned long threshold; -unsigned long tx_pkt_n; -unsigned long rx_pkt_n; -unsigned long normal_irq_n; -unsigned long rx_normal_irq_n; -unsigned long napi_poll; -unsigned long tx_normal_irq_n; -unsigned long tx_clean; -unsigned long tx_set_ic_bit;unsigned long irq_receive_pmt_irq_n;/* MMC info */ unsigned long mmc_tx_irq_n;
@@ -190,18 +195,16 @@ struct stmmac_extra_stats {unsigned long mtl_rx_fifo_ctrl_active;unsigned long mac_rx_frame_ctrl_fifo;unsigned long mac_gmii_rx_proto_engine; -/* TSO */ -unsigned long tx_tso_frames; -unsigned long tx_tso_nfrags;/* EST */ unsigned long mtl_est_cgce;unsigned long mtl_est_hlbs;unsigned long mtl_est_hlbf;unsigned long mtl_est_btre;unsigned long mtl_est_btrlm; -/* per queue statistics */ -struct stmmac_txq_stats txq_stats[MTL_MAX_TX_QUEUES ]; -struct stmmac_rxq_stats rxq_stats[MTL_MAX_RX_QUEUES ]; +unsigned long rx_dropped; +unsigned long rx_errors; +unsigned long tx_dropped; +unsigned long tx_errors; };/* Safety Feature statistics exposed by ethtool */
drivers/net/ethernet/stmicro/stmmac/dwmac-sun8i.c +9 -3
@@ -440,8 +440,10 @@ static int sun8i_dwmac_dma_interrupt(struct stmmac_priv *priv, struct stmmac_extra_stats *x, u32 chan, u32 dir) { - u32 v; + struct stmmac_rx_queue *rx_q = &priv->dma_conf.rx_queue[chan]; +struct stmmac_tx_queue *tx_q = &priv->dma_conf.tx_queue[chan];int ret =0 ; +u32 v; v =readl (ioaddr +EMAC_INT_STA );
@@ -452,7 +454,9 @@ static int sun8i_dwmac_dma_interrupt(struct stmmac_priv *priv,if (v &EMAC_TX_INT ) { ret |= handle_tx; - x->tx_normal_irq_n++; +u64_stats_update_begin (&tx_q->txq_stats.syncp); + tx_q->txq_stats.tx_normal_irq_n++; +u64_stats_update_end (&tx_q->txq_stats.syncp); }if (v &EMAC_TX_DMA_STOP_INT )
@@ -474,7 +478,9 @@ static int sun8i_dwmac_dma_interrupt(struct stmmac_priv *priv,if (v &EMAC_RX_INT ) { ret |= handle_rx; - x->rx_normal_irq_n++; +u64_stats_update_begin (&rx_q->rxq_stats.syncp); + rx_q->rxq_stats.rx_normal_irq_n++; +u64_stats_update_end (&rx_q->rxq_stats.syncp); }if (v &EMAC_RX_BUF_UA_INT )
drivers/net/ethernet/stmicro/stmmac/dwmac100_dma.c +1 -6
@@ -82,29 +82,24 @@ static void dwmac100_dump_dma_regs(struct stmmac_priv *priv, }/* DMA controller has two counters to track the number of the missed frames. */ -static void dwmac100_dma_diagnostic_fr (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static void dwmac100_dma_diagnostic_fr (struct stmmac_extra_stats *x,void __iomem *ioaddr) {u32 csr8 =readl (ioaddr +DMA_MISSED_FRAME_CTR );if (unlikely (csr8)) {if (csr8 &DMA_MISSED_FRAME_OVE ) { - stats->rx_over_errors +=0x800 ; x->rx_overflow_cntr +=0x800 ; }else {unsigned int ove_cntr; ove_cntr = ((csr8 &DMA_MISSED_FRAME_OVE_CNTR ) >>17 ); - stats->rx_over_errors += ove_cntr; x->rx_overflow_cntr += ove_cntr; }if (csr8 &DMA_MISSED_FRAME_OVE_M ) { - stats->rx_missed_errors +=0xffff ; x->rx_missed_cntr +=0xffff ; }else {unsigned int miss_f = (csr8 &DMA_MISSED_FRAME_M_CNTR ); - stats->rx_missed_errors += miss_f; x->rx_missed_cntr += miss_f; } }
drivers/net/ethernet/stmicro/stmmac/dwmac4_descs.c +5 -11
@@ -13,8 +13,7 @@#include "dwmac4.h" #include "dwmac4_descs.h" -static int dwmac4_wrback_get_tx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int dwmac4_wrback_get_tx_status (struct stmmac_extra_stats *x,struct dma_desc *p,void __iomem *ioaddr) {
@@ -40,15 +39,13 @@ static int dwmac4_wrback_get_tx_status(struct net_device_stats *stats, x->tx_frame_flushed++;if (unlikely (tdes3 &TDES3_LOSS_CARRIER )) { x->tx_losscarrier++; - stats->tx_carrier_errors++; }if (unlikely (tdes3 &TDES3_NO_CARRIER )) { x->tx_carrier++; - stats->tx_carrier_errors++; }if (unlikely ((tdes3 &TDES3_LATE_COLLISION ) || (tdes3 &TDES3_EXCESSIVE_COLLISION ))) - stats->collisions += + x->tx_collision += (tdes3 &TDES3_COLLISION_COUNT_MASK ) >>TDES3_COLLISION_COUNT_SHIFT ;
@@ -73,8 +70,7 @@ static int dwmac4_wrback_get_tx_status(struct net_device_stats *stats,return ret; } -static int dwmac4_wrback_get_rx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int dwmac4_wrback_get_rx_status (struct stmmac_extra_stats *x,struct dma_desc *p) {unsigned int rdes1 =le32_to_cpu (p->des1);
@@ -93,7 +89,7 @@ static int dwmac4_wrback_get_rx_status(struct net_device_stats *stats,if (unlikely (rdes3 &RDES3_ERROR_SUMMARY )) {if (unlikely (rdes3 &RDES3_GIANT_PACKET )) - stats->rx_length_errors++; + x->rx_length++;if (unlikely (rdes3 &RDES3_OVERFLOW_ERROR )) x->rx_gmac_overflow++;
@@ -103,10 +99,8 @@ static int dwmac4_wrback_get_rx_status(struct net_device_stats *stats,if (unlikely (rdes3 &RDES3_RECEIVE_ERROR )) x->rx_mii++; -if (unlikely (rdes3 &RDES3_CRC_ERROR )) { +if (unlikely (rdes3 &RDES3_CRC_ERROR )) x->rx_crc_errors++; - stats->rx_crc_errors++; - }if (unlikely (rdes3 &RDES3_DRIBBLE_ERROR )) x->dribbling_bit++;
drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c +9 -6
@@ -171,6 +171,8 @@ int dwmac4_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr,const struct dwmac4_addrs *dwmac4_addrs = priv->plat->dwmac4_addrs;u32 intr_status =readl (ioaddr +DMA_CHAN_STATUS (dwmac4_addrs, chan));u32 intr_en =readl (ioaddr +DMA_CHAN_INTR_ENA (dwmac4_addrs, chan)); +struct stmmac_rx_queue *rx_q = &priv->dma_conf.rx_queue[chan]; +struct stmmac_tx_queue *tx_q = &priv->dma_conf.tx_queue[chan];int ret =0 ;if (dir ==DMA_DIR_RX )
@@ -198,18 +200,19 @@ int dwmac4_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr, } }/* TX/RX NORMAL interrupts */ -if (likely (intr_status &DMA_CHAN_STATUS_NIS )) - x->normal_irq_n++;if (likely (intr_status &DMA_CHAN_STATUS_RI )) { - x->rx_normal_irq_n++; - x->rxq_stats[chan].rx_normal_irq_n++; +u64_stats_update_begin (&rx_q->rxq_stats.syncp); + rx_q->rxq_stats.rx_normal_irq_n++; +u64_stats_update_end (&rx_q->rxq_stats.syncp); ret |= handle_rx; }if (likely (intr_status &DMA_CHAN_STATUS_TI )) { - x->tx_normal_irq_n++; - x->txq_stats[chan].tx_normal_irq_n++; +u64_stats_update_begin (&tx_q->txq_stats.syncp); + tx_q->txq_stats.tx_normal_irq_n++; +u64_stats_update_end (&tx_q->txq_stats.syncp); ret |= handle_tx; } +if (unlikely (intr_status &DMA_CHAN_STATUS_TBU )) ret |= handle_tx;if (unlikely (intr_status &DMA_CHAN_STATUS_ERI ))
drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c +9 -3
@@ -10,6 +10,7 @@#include <linux/iopoll.h> #include "common.h" #include "dwmac_dma.h" +#include "stmmac.h" #define GMAC_HI_REG_AE 0x80000000
@@ -161,6 +162,8 @@ static void show_rx_process_state(unsigned int status)int dwmac_dma_interrupt (struct stmmac_priv *priv,void __iomem *ioaddr,struct stmmac_extra_stats *x,u32 chan,u32 dir) { +struct stmmac_rx_queue *rx_q = &priv->dma_conf.rx_queue[chan]; +struct stmmac_tx_queue *tx_q = &priv->dma_conf.tx_queue[chan];int ret =0 ;/* read the status register (CSR5) */ u32 intr_status =readl (ioaddr +DMA_STATUS );
@@ -208,17 +211,20 @@ int dwmac_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr, }/* TX/RX NORMAL interrupts */ if (likely (intr_status &DMA_STATUS_NIS )) { - x->normal_irq_n++;if (likely (intr_status &DMA_STATUS_RI )) {u32 value =readl (ioaddr +DMA_INTR_ENA );/* to schedule NAPI on real RIE event. */ if (likely (value &DMA_INTR_ENA_RIE )) { - x->rx_normal_irq_n++; +u64_stats_update_begin (&rx_q->rxq_stats.syncp); + rx_q->rxq_stats.rx_normal_irq_n++; +u64_stats_update_end (&rx_q->rxq_stats.syncp); ret |= handle_rx; } }if (likely (intr_status &DMA_STATUS_TI )) { - x->tx_normal_irq_n++; +u64_stats_update_begin (&tx_q->txq_stats.syncp); + tx_q->txq_stats.tx_normal_irq_n++; +u64_stats_update_end (&tx_q->txq_stats.syncp); ret |= handle_tx; }if (unlikely (intr_status &DMA_STATUS_ERI ))
drivers/net/ethernet/stmicro/stmmac/dwxgmac2_descs.c +2 -4
@@ -8,8 +8,7 @@#include "common.h" #include "dwxgmac2.h" -static int dwxgmac2_get_tx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int dwxgmac2_get_tx_status (struct stmmac_extra_stats *x,struct dma_desc *p,void __iomem *ioaddr) {unsigned int tdes3 =le32_to_cpu (p->des3);
@@ -23,8 +22,7 @@ static int dwxgmac2_get_tx_status(struct net_device_stats *stats,return ret; } -static int dwxgmac2_get_rx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int dwxgmac2_get_rx_status (struct stmmac_extra_stats *x,struct dma_desc *p) {unsigned int rdes3 =le32_to_cpu (p->des3);
drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c +8 -6
@@ -337,6 +337,8 @@ static int dwxgmac2_dma_interrupt(struct stmmac_priv *priv, struct stmmac_extra_stats *x, u32 chan, u32 dir) { + struct stmmac_rx_queue *rx_q = &priv->dma_conf.rx_queue[chan]; +struct stmmac_tx_queue *tx_q = &priv->dma_conf.tx_queue[chan];u32 intr_status =readl (ioaddr +XGMAC_DMA_CH_STATUS (chan));u32 intr_en =readl (ioaddr +XGMAC_DMA_CH_INT_EN (chan));int ret =0 ;
@@ -364,16 +366,16 @@ static int dwxgmac2_dma_interrupt(struct stmmac_priv *priv,/* TX/RX NORMAL interrupts */ if (likely (intr_status &XGMAC_NIS )) { - x->normal_irq_n++; -if (likely (intr_status &XGMAC_RI )) { - x->rx_normal_irq_n++; - x->rxq_stats[chan].rx_normal_irq_n++; +u64_stats_update_begin (&rx_q->rxq_stats.syncp); + rx_q->rxq_stats.rx_normal_irq_n++; +u64_stats_update_end (&rx_q->rxq_stats.syncp); ret |= handle_rx; }if (likely (intr_status & (XGMAC_TI |XGMAC_TBU ))) { - x->tx_normal_irq_n++; - x->txq_stats[chan].tx_normal_irq_n++; +u64_stats_update_begin (&tx_q->txq_stats.syncp); + tx_q->txq_stats.tx_normal_irq_n++; +u64_stats_update_end (&tx_q->txq_stats.syncp); ret |= handle_tx; } }
drivers/net/ethernet/stmicro/stmmac/enh_desc.c +7 -13
@@ -12,8 +12,7 @@#include "common.h" #include "descs_com.h" -static int enh_desc_get_tx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int enh_desc_get_tx_status (struct stmmac_extra_stats *x,struct dma_desc *p,void __iomem *ioaddr) {unsigned int tdes0 =le32_to_cpu (p->des0);
@@ -38,15 +37,13 @@ static int enh_desc_get_tx_status(struct net_device_stats *stats,if (unlikely (tdes0 &ETDES0_LOSS_CARRIER )) { x->tx_losscarrier++; - stats->tx_carrier_errors++; }if (unlikely (tdes0 &ETDES0_NO_CARRIER )) { x->tx_carrier++; - stats->tx_carrier_errors++; }if (unlikely ((tdes0 &ETDES0_LATE_COLLISION ) || (tdes0 &ETDES0_EXCESSIVE_COLLISIONS ))) - stats->collisions += + x->tx_collision += (tdes0 &ETDES0_COLLISION_COUNT_MASK ) >>3 ;if (unlikely (tdes0 &ETDES0_EXCESSIVE_DEFERRAL ))
@@ -117,8 +114,7 @@ static int enh_desc_coe_rdes0(int ipc_err, int type, int payload_err)return ret; } -static void enh_desc_get_ext_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static void enh_desc_get_ext_status (struct stmmac_extra_stats *x,struct dma_extended_desc *p) {unsigned int rdes0 =le32_to_cpu (p->basic.des0);
@@ -182,8 +178,7 @@ static void enh_desc_get_ext_status(struct net_device_stats *stats, } } -static int enh_desc_get_rx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int enh_desc_get_rx_status (struct stmmac_extra_stats *x,struct dma_desc *p) {unsigned int rdes0 =le32_to_cpu (p->des0);
@@ -193,14 +188,14 @@ static int enh_desc_get_rx_status(struct net_device_stats *stats,return dma_own;if (unlikely (!(rdes0 &RDES0_LAST_DESCRIPTOR ))) { - stats->rx_length_errors++; + x->rx_length++;return discard_frame; }if (unlikely (rdes0 &RDES0_ERROR_SUMMARY )) {if (unlikely (rdes0 &RDES0_DESCRIPTOR_ERROR )) { x->rx_desc++; - stats->rx_length_errors++; + x->rx_length++; }if (unlikely (rdes0 &RDES0_OVERFLOW_ERROR )) x->rx_gmac_overflow++;
@@ -209,7 +204,7 @@ static int enh_desc_get_rx_status(struct net_device_stats *stats,pr_err ("\tIPC Csum Error/Giant frame\n" );if (unlikely (rdes0 &RDES0_COLLISION )) - stats->collisions++; + x->rx_collision++;if (unlikely (rdes0 &RDES0_RECEIVE_WATCHDOG )) x->rx_watchdog++;
@@ -218,7 +213,6 @@ static int enh_desc_get_rx_status(struct net_device_stats *stats,if (unlikely (rdes0 &RDES0_CRC_ERROR )) { x->rx_crc_errors++; - stats->rx_crc_errors++; } ret = discard_frame; }
drivers/net/ethernet/stmicro/stmmac/hwif.h +4 -8
@@ -57,8 +57,7 @@ struct stmmac_desc_ops {/* Last tx segment reports the transmit status */ int (*get_tx_ls)(struct dma_desc *p);/* Return the transmit status looking at the TDES1 */ -int (*tx_status)(struct net_device_stats *stats, -struct stmmac_extra_stats *x, +int (*tx_status)(struct stmmac_extra_stats *x,struct dma_desc *p,void __iomem *ioaddr);/* Get the buffer size from the descriptor */ int (*get_tx_len)(struct dma_desc *p);
@@ -67,11 +66,9 @@ struct stmmac_desc_ops {/* Get the receive frame size */ int (*get_rx_frame_len)(struct dma_desc *p,int rx_coe_type);/* Return the reception status looking at the RDES1 */ -int (*rx_status)(struct net_device_stats *stats, -struct stmmac_extra_stats *x, +int (*rx_status)(struct stmmac_extra_stats *x,struct dma_desc *p); -void (*rx_extended_status)(struct net_device_stats *stats, -struct stmmac_extra_stats *x, +void (*rx_extended_status)(struct stmmac_extra_stats *x,struct dma_extended_desc *p);/* Set tx timestamp enable bit */ void (*enable_tx_timestamp) (struct dma_desc *p);
@@ -191,8 +188,7 @@ struct stmmac_dma_ops {void (*dma_tx_mode)(struct stmmac_priv *priv,void __iomem *ioaddr,int mode,u32 channel,int fifosz,u8 qmode);/* To track extra statistic (if supported) */ -void (*dma_diagnostic_fr)(struct net_device_stats *stats, -struct stmmac_extra_stats *x, +void (*dma_diagnostic_fr)(struct stmmac_extra_stats *x,void __iomem *ioaddr);void (*enable_dma_transmission) (void __iomem *ioaddr); void (*enable_dma_irq)(struct stmmac_priv *priv, void __iomem *ioaddr,
drivers/net/ethernet/stmicro/stmmac/norm_desc.c +4 -11
@@ -12,8 +12,7 @@#include "common.h" #include "descs_com.h" -static int ndesc_get_tx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int ndesc_get_tx_status (struct stmmac_extra_stats *x,struct dma_desc *p,void __iomem *ioaddr) {unsigned int tdes0 =le32_to_cpu (p->des0);
@@ -31,15 +30,12 @@ static int ndesc_get_tx_status(struct net_device_stats *stats,if (unlikely (tdes0 &TDES0_ERROR_SUMMARY )) {if (unlikely (tdes0 &TDES0_UNDERFLOW_ERROR )) { x->tx_underflow++; - stats->tx_fifo_errors++; }if (unlikely (tdes0 &TDES0_NO_CARRIER )) { x->tx_carrier++; - stats->tx_carrier_errors++; }if (unlikely (tdes0 &TDES0_LOSS_CARRIER )) { x->tx_losscarrier++; - stats->tx_carrier_errors++; } if (unlikely((tdes0 & TDES0_EXCESSIVE_DEFERRAL) || (tdes0 & TDES0_EXCESSIVE_COLLISIONS) ||
@@ -47,7 +43,7 @@ static int ndesc_get_tx_status(struct net_device_stats *stats,unsigned int collisions; collisions = (tdes0 &TDES0_COLLISION_COUNT_MASK ) >>3 ; - stats->collisions += collisions; + x->tx_collision += collisions; } ret = tx_err; }
@@ -70,8 +66,7 @@ static int ndesc_get_tx_len(struct dma_desc *p)* and, if required, updates the multicast statistics. * In case of success, it returns good_frame because the GMAC device * is supposed to be able to compute the csum in HW. */ -static int ndesc_get_rx_status (struct net_device_stats *stats, -struct stmmac_extra_stats *x, +static int ndesc_get_rx_status (struct stmmac_extra_stats *x,struct dma_desc *p) {int ret = good_frame;
@@ -81,7 +76,7 @@ static int ndesc_get_rx_status(struct net_device_stats *stats,return dma_own;if (unlikely (!(rdes0 &RDES0_LAST_DESCRIPTOR ))) { - stats->rx_length_errors++; + x->rx_length++;return discard_frame; }
@@ -96,11 +91,9 @@ static int ndesc_get_rx_status(struct net_device_stats *stats, x->ipc_csum_error++;if (unlikely (rdes0 &RDES0_COLLISION )) { x->rx_collision++; - stats->collisions++; }if (unlikely (rdes0 &RDES0_CRC_ERROR )) { x->rx_crc_errors++; - stats->rx_crc_errors++; } ret = discard_frame; }
drivers/net/ethernet/stmicro/stmmac/stmmac.h +2 -0
@@ -77,6 +77,7 @@ struct stmmac_tx_queue {dma_addr_t dma_tx_phy;dma_addr_t tx_tail_addr;u32 mss; +struct stmmac_txq_stats txq_stats; };struct stmmac_rx_buffer {
@@ -121,6 +122,7 @@ struct stmmac_rx_queue {unsigned int len;unsigned int error; } state; +struct stmmac_rxq_stats rxq_stats; };struct stmmac_channel {
drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c +98 -25
187 lines not on this page · raw message
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +156 -44
467 lines not on this page · raw message
Signature · 1 line
2.40.0
Jakub Kicinski
maintainerQuoted 11 lines · On Tue, 18 Jul 2023 00:06:28 +0800 Jisheng Zhang wrote:
improve the stmmac driver statistics:
1. don't clear network driver statistics in .ndo_close() and .ndo_open() cycle
2. avoid some network driver statistics overflow on 32 bit platforms
3. use per-queue statistics where necessary to remove frequent cacheline ping pongs.NOTE: v1 and v2 are back ported from an internal LTS tree, I made some mistakes when backporting and squashing. Now, net-next + v3 has been well tested with 'ethtool -s' and 'ip -s link show'.
Giuseppe, please take a look.
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 9 lines · On Tue, 18 Jul 2023 00:06:28 +0800 you wrote:
improve the stmmac driver statistics:
1. don't clear network driver statistics in .ndo_close() and .ndo_open() cycle
2. avoid some network driver statistics overflow on 32 bit platforms
3. use per-queue statistics where necessary to remove frequent cacheline ping pongs.[...]
Here is the summary with links:
- [net-next,v5,1/2] net: stmmac: don't clear network statistics in .ndo_open()
https://git.kernel.org/netdev/net-next/c/2eb85b750512 - [net-next,v5,2/2] net: stmmac: use per-queue 64 bit statistics where necessary https://git.kernel.org/netdev/net-next/c/133466c3bbe1
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
PATCH v5 2/2 net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Jisheng ZhangHello,
this patch became commit 133466c3bbe171f826294161db203f7670bb30c8 and is part of v6.6-rc1.
On my arm/stm32mp157 based machine using NFS root this commit makes the following appear in the kernel log:
INFO: trying to register non-static key. The code is fine but needs lockdep annotation, or maybe you didn't initialize this object before use? turning off the locking correctness validator. CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00449-g133466c3bbe1-dirty #21 Hardware name: STM32 (Device Tree Support) unwind_backtrace from show_stack+0x18/0x1c show_stack from dump_stack_lvl+0x60/0x90 dump_stack_lvl from register_lock_class+0x98c/0x99c register_lock_class from __lock_acquire+0x74/0x293c __lock_acquire from lock_acquire+0x134/0x398 lock_acquire from stmmac_get_stats64+0x2ac/0x2fc stmmac_get_stats64 from dev_get_stats+0x44/0x130 dev_get_stats from rtnl_fill_stats+0x38/0x120 rtnl_fill_stats from rtnl_fill_ifinfo+0x834/0x17f4 rtnl_fill_ifinfo from rtmsg_ifinfo_build_skb+0xc0/0x144 rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x50/0x88 rtmsg_ifinfo from __dev_notify_flags+0xc0/0xec __dev_notify_flags from dev_change_flags+0x50/0x5c dev_change_flags from ip_auto_config+0x2f4/0x1260 ip_auto_config from do_one_initcall+0x70/0x35c do_one_initcall from kernel_init_freeable+0x2ac/0x308 kernel_init_freeable from kernel_init+0x1c/0x138 kernel_init from ret_from_fork+0x14/0x2c Exception stack(0xe0815fb0 to 0xe0815ff8) 5fa0: 00000000 00000000 00000000 00000000 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 dwc2 49000000.usb-otg: new device is high-speed
I didn't try understand this problem, it's too close to quitting time :-)
Best regards
Uwe
Signature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
Quoted 38 lines · Uwe Kleine-König
Hello,
this patch became commit
133466c3bbe171f826294161db203f7670bb30c8and is part of v6.6-rc1.On my arm/stm32mp157 based machine using NFS root this commit makes the following appear in the kernel log:
INFO: trying to register non-static key. The code is fine but needs lockdep annotation, or maybe you didn't initialize this object before use? turning off the locking correctness validator. CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00449-g133466c3bbe1-dirty #21 Hardware name: STM32 (Device Tree Support) unwind_backtrace from show_stack+0x18/0x1c show_stack from dump_stack_lvl+0x60/0x90 dump_stack_lvl from register_lock_class+0x98c/0x99c register_lock_class from __lock_acquire+0x74/0x293c __lock_acquire from lock_acquire+0x134/0x398 lock_acquire from stmmac_get_stats64+0x2ac/0x2fc stmmac_get_stats64 from dev_get_stats+0x44/0x130 dev_get_stats from rtnl_fill_stats+0x38/0x120 rtnl_fill_stats from rtnl_fill_ifinfo+0x834/0x17f4 rtnl_fill_ifinfo from rtmsg_ifinfo_build_skb+0xc0/0x144 rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x50/0x88 rtmsg_ifinfo from __dev_notify_flags+0xc0/0xec __dev_notify_flags from dev_change_flags+0x50/0x5c dev_change_flags from ip_auto_config+0x2f4/0x1260 ip_auto_config from do_one_initcall+0x70/0x35c do_one_initcall from kernel_init_freeable+0x2ac/0x308 kernel_init_freeable from kernel_init+0x1c/0x138 kernel_init from ret_from_fork+0x14/0x2c Exception stack(0xe0815fb0 to 0xe0815ff8) 5fa0: 00000000 00000000 00000000 00000000 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 dwc2 49000000.usb-otg: new device is high-speed
Quoted 2 lines
I didn't try understand this problem, it's too close to quitting time :-)
Thanks for the bug report, I'm checking the code.
Quoted 7 lines
Best regards
UweSignature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Uwe Kleine-KönigQuoted 5 lines · Uwe Kleine-König
Hello,
this patch became commit
133466c3bbe171f826294161db203f7670bb30c8and is part of v6.6-rc1.
Quoted 8 lines
On my arm/stm32mp157 based machine using NFS root this commit makes the following appear in the kernel log:
INFO: trying to register non-static key.
The code is fine but needs lockdep annotation, or maybe
you didn't initialize this object before use?
turning off the locking correctness validator.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00449-g133466c3bbe1-dirty #21
Hi,
Which kernel version are you using? The latest linus tree? But why here say 6.5.0-rc1?
Thanks
Quoted 34 lines
Hardware name: STM32 (Device Tree Support) unwind_backtrace from show_stack+0x18/0x1c show_stack from dump_stack_lvl+0x60/0x90 dump_stack_lvl from register_lock_class+0x98c/0x99c register_lock_class from __lock_acquire+0x74/0x293c __lock_acquire from lock_acquire+0x134/0x398 lock_acquire from stmmac_get_stats64+0x2ac/0x2fc stmmac_get_stats64 from dev_get_stats+0x44/0x130 dev_get_stats from rtnl_fill_stats+0x38/0x120 rtnl_fill_stats from rtnl_fill_ifinfo+0x834/0x17f4 rtnl_fill_ifinfo from rtmsg_ifinfo_build_skb+0xc0/0x144 rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x50/0x88 rtmsg_ifinfo from __dev_notify_flags+0xc0/0xec __dev_notify_flags from dev_change_flags+0x50/0x5c dev_change_flags from ip_auto_config+0x2f4/0x1260 ip_auto_config from do_one_initcall+0x70/0x35c do_one_initcall from kernel_init_freeable+0x2ac/0x308 kernel_init_freeable from kernel_init+0x1c/0x138 kernel_init from ret_from_fork+0x14/0x2c Exception stack(0xe0815fb0 to 0xe0815ff8) 5fa0: 00000000 00000000 00000000 00000000 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 dwc2 49000000.usb-otg: new device is high-speedI didn't try understand this problem, it's too close to quitting time :-)
Best regards
UweSignature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Jisheng ZhangAm Dienstag, dem 12.09.2023 um 16:01 +0800 schrieb Jisheng Zhang:
Quoted 39 lines · Uwe Kleine-König
Quoted 38 lines
Hello,
this patch became commit
133466c3bbe171f826294161db203f7670bb30c8and is part of v6.6-rc1.On my arm/stm32mp157 based machine using NFS root this commit makes the following appear in the kernel log:
INFO: trying to register non-static key. The code is fine but needs lockdep annotation, or maybe you didn't initialize this object before use? turning off the locking correctness validator. CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00449-g133466c3bbe1-dirty #21 Hardware name: STM32 (Device Tree Support) unwind_backtrace from show_stack+0x18/0x1c show_stack from dump_stack_lvl+0x60/0x90 dump_stack_lvl from register_lock_class+0x98c/0x99c register_lock_class from __lock_acquire+0x74/0x293c __lock_acquire from lock_acquire+0x134/0x398 lock_acquire from stmmac_get_stats64+0x2ac/0x2fc stmmac_get_stats64 from dev_get_stats+0x44/0x130 dev_get_stats from rtnl_fill_stats+0x38/0x120 rtnl_fill_stats from rtnl_fill_ifinfo+0x834/0x17f4 rtnl_fill_ifinfo from rtmsg_ifinfo_build_skb+0xc0/0x144 rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x50/0x88 rtmsg_ifinfo from __dev_notify_flags+0xc0/0xec __dev_notify_flags from dev_change_flags+0x50/0x5c dev_change_flags from ip_auto_config+0x2f4/0x1260 ip_auto_config from do_one_initcall+0x70/0x35c do_one_initcall from kernel_init_freeable+0x2ac/0x308 kernel_init_freeable from kernel_init+0x1c/0x138 kernel_init from ret_from_fork+0x14/0x2c Exception stack(0xe0815fb0 to 0xe0815ff8) 5fa0: 00000000 00000000 00000000 00000000 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 dwc2 49000000.usb-otg: new device is high-speed
Quoted 4 lines
Quoted 2 lines
I didn't try understand this problem, it's too close to quitting time :-)
Thanks for the bug report, I'm checking the code.
The newly added "struct u64_stats_sync syncp" uses a seqlock internally, which is broken into multiple words on 32bit machines, and needs to be initialized properly. You need to call u64_stats_init on syncp before first usage.
Regards,
Lucas
Quoted 10 lines
Quoted 7 lines
Best regards
UweSignature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
Hello,
Quoted 43 lines · Lucas Stach
Am Dienstag, dem 12.09.2023 um 16:01 +0800 schrieb Jisheng Zhang:
Quoted 42 lines · Uwe Kleine-König
Quoted 40 lines
Hello,
this patch became commit
133466c3bbe171f826294161db203f7670bb30c8and is part of v6.6-rc1.On my arm/stm32mp157 based machine using NFS root this commit makes the following appear in the kernel log:
INFO: trying to register non-static key. The code is fine but needs lockdep annotation, or maybe you didn't initialize this object before use? turning off the locking correctness validator. CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00449-g133466c3bbe1-dirty #21 Hardware name: STM32 (Device Tree Support) unwind_backtrace from show_stack+0x18/0x1c show_stack from dump_stack_lvl+0x60/0x90 dump_stack_lvl from register_lock_class+0x98c/0x99c register_lock_class from __lock_acquire+0x74/0x293c __lock_acquire from lock_acquire+0x134/0x398 lock_acquire from stmmac_get_stats64+0x2ac/0x2fc stmmac_get_stats64 from dev_get_stats+0x44/0x130 dev_get_stats from rtnl_fill_stats+0x38/0x120 rtnl_fill_stats from rtnl_fill_ifinfo+0x834/0x17f4 rtnl_fill_ifinfo from rtmsg_ifinfo_build_skb+0xc0/0x144 rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x50/0x88 rtmsg_ifinfo from __dev_notify_flags+0xc0/0xec __dev_notify_flags from dev_change_flags+0x50/0x5c dev_change_flags from ip_auto_config+0x2f4/0x1260 ip_auto_config from do_one_initcall+0x70/0x35c do_one_initcall from kernel_init_freeable+0x2ac/0x308 kernel_init_freeable from kernel_init+0x1c/0x138 kernel_init from ret_from_fork+0x14/0x2c Exception stack(0xe0815fb0 to 0xe0815ff8) 5fa0: 00000000 00000000 00000000 00000000 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 dwc2 49000000.usb-otg: new device is high-speedI didn't try understand this problem, it's too close to quitting time :-)
Quoted 6 lines
Quoted 1 line
Thanks for the bug report, I'm checking the code.
The newly added "struct
u64_stats_syncsyncp" uses a seqlock internally, which is broken into multiple words on 32bit machines, and needs to be initialized properly. You need to callu64_stats_initon syncp before first usage.
This is done. The problematic thing is that in stmmac_open() -> __stmmac_open() the syncp initialized before is overwritten by
memcpy(&priv->dma_conf, dma_conf, sizeof(*dma_conf));
Do I need to point out that this is ugly?
Together with Johannes Berg I debugged that in #netdev and we came up with:
drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +8 -1
@@ -3893,7 +3893,7 @@ static int stmmac_open(struct net_device *dev) {struct stmmac_priv *priv =netdev_priv (dev);struct stmmac_dma_conf *dma_conf; -int ret; +int ret, i; dma_conf =stmmac_setup_dma_desc (priv, dev->mtu);if (IS_ERR (dma_conf))
@@ -3904,6 +3904,13 @@ static int stmmac_open(struct net_device *dev)free_dma_desc_resources (priv, dma_conf);kfree (dma_conf); + + dma_conf = &priv->dma_conf; +for (i =0 ; i <MTL_MAX_RX_QUEUES ; i++) +u64_stats_init (&dma_conf->rx_queue[i].rxq_stats.syncp); +for (i =0 ; i <MTL_MAX_TX_QUEUES ; i++) +u64_stats_init (&dma_conf->tx_queue[i].txq_stats.syncp); +return ret; }
which works around the problem. Note however that the u64_stats_init() calls must not be deleted from stmmac_dvr_probe() because it's used there once before __stmmac_open() overwrites it.
Best regards
Uwe
Signature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Jisheng ZhangQuoted 15 lines · Jisheng Zhang
Quoted 5 lines · Uwe Kleine-König
Hello,
this patch became commit
133466c3bbe171f826294161db203f7670bb30c8and is part of v6.6-rc1.Quoted 8 lines
On my arm/stm32mp157 based machine using NFS root this commit makes the following appear in the kernel log:
INFO: trying to register non-static key.
The code is fine but needs lockdep annotation, or maybe
you didn't initialize this object before use?
turning off the locking correctness validator.
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.5.0-rc1-00449-g133466c3bbe1-dirty #21
Quoted 4 lines
Hi,
Which kernel version are you using? The latest linus tree? But why here say 6.5.0-rc1?
This is the kernel from the last bisection test. I.e. 133466c3bbe171f826294161db203f7670bb30c8 (plus some minor unrelated changes to work around another problem). This commit is based on 6.5-rc1.
Best regards
Uwe
Signature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Uwe Kleine-KönigQuoted 6 lines · Uwe Kleine-König
Quoted 5 lines
The newly added "struct
u64_stats_syncsyncp" uses a seqlock internally, which is broken into multiple words on 32bit machines, and needs to be initialized properly. You need to callu64_stats_initon syncp before first usage.
Quoted 6 lines
This is done. The problematic thing is that in
stmmac_open()->__stmmac_open()the syncp initialized before is overwritten bymemcpy(&priv->dma_conf, dma_conf, sizeof(*dma_conf));Do I need to point out that this is ugly?
I think it also leaks the (lockdep) state since it reinits the syncp (and a lot of other state) doing this. This is also called when the MTU changes.
Also, I couldn't convince myself that it's even race-free? Even if it is, it's not really obvious, IMHO.
So it seems to me that really this needs to be split into data that actually should be reinitialized, and data that shouldn't, or just not use memcpy() here but copy only the relevant state?
But anyway, I have no skin in this game - just reviewing this because I was trying to help out Uwe.
johannes
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
Quoted 11 lines · Johannes Berg
Quoted 10 lines · Uwe Kleine-König
Quoted 5 lines
The newly added "struct
u64_stats_syncsyncp" uses a seqlock internally, which is broken into multiple words on 32bit machines, and needs to be initialized properly. You need to callu64_stats_initon syncp before first usage.This is done. The problematic thing is that in
stmmac_open()->__stmmac_open()the syncp initialized before is overwritten bymemcpy(&priv->dma_conf, dma_conf, sizeof(*dma_conf));
Thank Johannes and Uwe for pointing out the issue.
Quoted 7 lines
Quoted 2 lines
Do I need to point out that this is ugly?
I think it also leaks the (lockdep) state since it reinits the syncp (and a lot of other state) doing this. This is also called when the MTU changes.
Quoted 6 lines
Also, I couldn't convince myself that it's even race-free? Even if it is, it's not really obvious, IMHO.
So it seems to me that really this needs to be split into data that actually should be reinitialized, and data that shouldn't, or just not use
memcpy()here but copy only the relevant state?
Since we are in rc1, I need to fix the bug with as small changes as possible. so another solution could be: replace rx/tx stats structure with pointers, then setup pointers in the new allocated dma_conf with the old one as current code did for dma_tx_size/dma_rx_size in stmmac_setup_dma_desc():
dma_conf->dma_tx_size = priv->dma_conf.dma_tx_size
Is it acceptable?
Thanks
Quoted 5 lines
But anyway, I have no skin in this game - just reviewing this because I was trying to help out Uwe.
johannes
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
Quoted 10 lines · Jisheng Zhang
Since we are in rc1, I need to fix the bug with as small changes as possible. so another solution could be: replace rx/tx stats structure with pointers, then setup pointers in the new allocated
dma_confwith the old one as current code did for dma_tx_size/dma_rx_size instmmac_setup_dma_desc():
dma_conf->dma_tx_size=priv->dma_conf.dma_tx_sizeIs it acceptable?
I'm not sure who you're asking of all the people in this thread, but I honestly don't understand much about this driver other than what I gleaned in the few minutes looking at it with Uwe ... so I don't think I'm able to answer that question :)
You could also just move the stats out of the structure entirely, I guess? And perhaps even start some new structure where more things might move in the future that shouldn't be reallocated?
I'm all for not moving things now that haven't caused problems, but I guess doing a minimal fix just for the sake of being minimal in -rc1 wouldn't be my preference.
johannes
Re: [REGRESSION] [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Jisheng ZhangHello,
Quoted 35 lines · Jisheng Zhang
Quoted 11 lines · Johannes Berg
Quoted 10 lines · Uwe Kleine-König
Quoted 5 lines
The newly added "struct
u64_stats_syncsyncp" uses a seqlock internally, which is broken into multiple words on 32bit machines, and needs to be initialized properly. You need to callu64_stats_initon syncp before first usage.This is done. The problematic thing is that in
stmmac_open()->__stmmac_open()the syncp initialized before is overwritten bymemcpy(&priv->dma_conf, dma_conf, sizeof(*dma_conf));Thank Johannes and Uwe for pointing out the issue.
Quoted 7 lines
Quoted 2 lines
Do I need to point out that this is ugly?
I think it also leaks the (lockdep) state since it reinits the syncp (and a lot of other state) doing this. This is also called when the MTU changes.
Quoted 6 lines
Also, I couldn't convince myself that it's even race-free? Even if it is, it's not really obvious, IMHO.
So it seems to me that really this needs to be split into data that actually should be reinitialized, and data that shouldn't, or just not use
memcpy()here but copy only the relevant state?Since we are in rc1, I need to fix the bug with as small changes as possible. so another solution could be: replace rx/tx stats structure with pointers, then setup pointers in the new allocated
dma_confwith the old one as current code did for dma_tx_size/dma_rx_size instmmac_setup_dma_desc():
Quoted 3 lines
dma_conf->dma_tx_size=priv->dma_conf.dma_tx_sizeIs it acceptable?
I wondered if you can just initialize the data directly in *priv, instead of setting up a local copy, initialize that one + copy it over?!
Best regards
Uwe
Signature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
In reply to Jisheng ZhangHi,
Quoted 11 lines · Jisheng Zhang
Currently, there are two major issues with stmmac driver statistics First of all, statistics in
stmmac_extra_stats,stmmac_rxq_statsandstmmac_txq_statsare 32 bit variables on 32 bit platforms. This can cause some stats to overflow after several minutes of high traffic, for examplerx_pkt_n,tx_pkt_nand so on.Secondly, if HW supports multiqueues, there are frequent cacheline ping pongs on some driver statistic vars, for example,
normal_irq_n,tx_pkt_nand so on. What's more, frequent cacheline ping pongs onnormal_irq_nhappens in ISR, this makes the situation worse.
Quoted 8 lines
To improve the driver, we convert those statistics to 64 bit, implement
ndo_get_stats64and update .get_ethtool_stats implementation accordingly. We also use per-queue statistics where necessary to remove the cacheline ping pongs as much as possible to make multiqueue operations faster. Those statistics which are not possible to overflow and not frequently updated are kept as is.Signed-off-by Jisheng Zhang <[email protected]>
Your patch results in lockdep splats. This is with the orangepi-pc emulation in qemu.
[ 11.126950] dwmac-sun8i 1c30000.ethernet eth0: PHY [mdio_mux-0.1:01] driver [Generic PHY] (irq=POLL) [ 11.127912] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found [ 11.128294] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available [ 11.128511] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW [ 11.138990] dwmac-sun8i 1c30000.ethernet eth0: configuring for phy/mii link mode [ 11.144387] INFO: trying to register non-static key. [ 11.144483] The code is fine but needs lockdep annotation, or maybe [ 11.144568] you didn't initialize this object before use? [ 11.144640] turning off the locking correctness validator. [ 11.144845] CPU: 2 PID: 688 Comm: ip Tainted: G N 6.6.0-rc2 #1 [ 11.144956] Hardware name: Allwinner sun8i Family [ 11.145137] unwind_backtrace from show_stack+0x10/0x14 [ 11.145610] show_stack from dump_stack_lvl+0x68/0x90 [ 11.145692] dump_stack_lvl from register_lock_class+0x99c/0x9b0 [ 11.145779] register_lock_class from __lock_acquire+0x6c/0x2244 [ 11.145861] __lock_acquire from lock_acquire+0x11c/0x368 [ 11.145938] lock_acquire from stmmac_get_stats64+0x350/0x374 [ 11.146021] stmmac_get_stats64 from dev_get_stats+0x3c/0x160 [ 11.146101] dev_get_stats from rtnl_fill_stats+0x30/0x118 [ 11.146179] rtnl_fill_stats from rtnl_fill_ifinfo.constprop.0+0x82c/0x1770 [ 11.146273] rtnl_fill_ifinfo.constprop.0 from rtmsg_ifinfo_build_skb+0xac/0x138 [ 11.146370] rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x44/0x7c [ 11.146452] rtmsg_ifinfo from __dev_notify_flags+0xac/0xd8 [ 11.146531] __dev_notify_flags from dev_change_flags+0x48/0x54 [ 11.146612] dev_change_flags from do_setlink+0x244/0xe6c [ 11.146689] do_setlink from rtnl_newlink+0x514/0x838 [ 11.146761] rtnl_newlink from rtnetlink_rcv_msg+0x170/0x5b0 [ 11.146841] rtnetlink_rcv_msg from netlink_rcv_skb+0xb4/0x10c [ 11.146925] netlink_rcv_skb from netlink_unicast+0x190/0x254 [ 11.147006] netlink_unicast from netlink_sendmsg+0x1dc/0x460 [ 11.147086] netlink_sendmsg from ____sys_sendmsg+0xa0/0x2a0 [ 11.147168] ____sys_sendmsg from ___sys_sendmsg+0x68/0x94 [ 11.147245] ___sys_sendmsg from sys_sendmsg+0x4c/0x88 [ 11.147329] sys_sendmsg from ret_fast_syscall+0x0/0x1c [ 11.147439] Exception stack(0xf23edfa8 to 0xf23edff0) [ 11.147558] dfa0: 00000000 00000000 00000003 bef9a8d8 00000000 00000000 [ 11.147668] dfc0: 00000000 00000000 ffffffff 00000128 00000001 00000002 bef9af4a bef9af4d [ 11.147769] dfe0: bef9a868 bef9a858 b6f9ddac b6f9d228 [ 11.150020] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
My apologies for the noise if this has already been reported.
Guenter
Re: [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
Hello Guenter,
Quoted 64 lines · Guenter Roeck
Quoted 11 lines · Jisheng Zhang
Currently, there are two major issues with stmmac driver statistics First of all, statistics in
stmmac_extra_stats,stmmac_rxq_statsandstmmac_txq_statsare 32 bit variables on 32 bit platforms. This can cause some stats to overflow after several minutes of high traffic, for examplerx_pkt_n,tx_pkt_nand so on.Secondly, if HW supports multiqueues, there are frequent cacheline ping pongs on some driver statistic vars, for example,
normal_irq_n,tx_pkt_nand so on. What's more, frequent cacheline ping pongs onnormal_irq_nhappens in ISR, this makes the situation worse.Quoted 8 lines
To improve the driver, we convert those statistics to 64 bit, implement
ndo_get_stats64and update .get_ethtool_stats implementation accordingly. We also use per-queue statistics where necessary to remove the cacheline ping pongs as much as possible to make multiqueue operations faster. Those statistics which are not possible to overflow and not frequently updated are kept as is.Signed-off-by Jisheng Zhang <[email protected]>Your patch results in lockdep splats. This is with the orangepi-pc emulation in qemu.
[ 11.126950] dwmac-sun8i 1c30000.ethernet eth0: PHY [mdio_mux-0.1:01] driver [Generic PHY] (irq=POLL) [ 11.127912] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found [ 11.128294] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available [ 11.128511] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW [ 11.138990] dwmac-sun8i 1c30000.ethernet eth0: configuring for phy/mii link mode [ 11.144387] INFO: trying to register non-static key. [ 11.144483] The code is fine but needs lockdep annotation, or maybe [ 11.144568] you didn't initialize this object before use? [ 11.144640] turning off the locking correctness validator. [ 11.144845] CPU: 2 PID: 688 Comm: ip Tainted: G N 6.6.0-rc2 #1 [ 11.144956] Hardware name: Allwinner sun8i Family [ 11.145137] unwind_backtrace from show_stack+0x10/0x14 [ 11.145610] show_stack from dump_stack_lvl+0x68/0x90 [ 11.145692] dump_stack_lvl from register_lock_class+0x99c/0x9b0 [ 11.145779] register_lock_class from __lock_acquire+0x6c/0x2244 [ 11.145861] __lock_acquire from lock_acquire+0x11c/0x368 [ 11.145938] lock_acquire from stmmac_get_stats64+0x350/0x374 [ 11.146021] stmmac_get_stats64 from dev_get_stats+0x3c/0x160 [ 11.146101] dev_get_stats from rtnl_fill_stats+0x30/0x118 [ 11.146179] rtnl_fill_stats from rtnl_fill_ifinfo.constprop.0+0x82c/0x1770 [ 11.146273] rtnl_fill_ifinfo.constprop.0 from rtmsg_ifinfo_build_skb+0xac/0x138 [ 11.146370] rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x44/0x7c [ 11.146452] rtmsg_ifinfo from __dev_notify_flags+0xac/0xd8 [ 11.146531] __dev_notify_flags from dev_change_flags+0x48/0x54 [ 11.146612] dev_change_flags from do_setlink+0x244/0xe6c [ 11.146689] do_setlink from rtnl_newlink+0x514/0x838 [ 11.146761] rtnl_newlink from rtnetlink_rcv_msg+0x170/0x5b0 [ 11.146841] rtnetlink_rcv_msg from netlink_rcv_skb+0xb4/0x10c [ 11.146925] netlink_rcv_skb from netlink_unicast+0x190/0x254 [ 11.147006] netlink_unicast from netlink_sendmsg+0x1dc/0x460 [ 11.147086] netlink_sendmsg from ____sys_sendmsg+0xa0/0x2a0 [ 11.147168] ____sys_sendmsg from ___sys_sendmsg+0x68/0x94 [ 11.147245] ___sys_sendmsg from sys_sendmsg+0x4c/0x88 [ 11.147329] sys_sendmsg from ret_fast_syscall+0x0/0x1c [ 11.147439] Exception stack(0xf23edfa8 to 0xf23edff0) [ 11.147558] dfa0: 00000000 00000000 00000003 bef9a8d8 00000000 00000000 [ 11.147668] dfc0: 00000000 00000000 ffffffff 00000128 00000001 00000002 bef9af4a bef9af4d [ 11.147769] dfe0: bef9a868 bef9a858 b6f9ddac b6f9d228 [ 11.150020] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Quoted 1 line
My apologies for the noise if this has already been reported.
This seems to be the issue I reported earlier. So you might want to test the patch that fixed it for me:
https://lore.kernel.org/netdev/[email protected]/
Best regards
Uwe
Signature · 2 lines
Pengutronix e.K. | Uwe Kleine-König | Industrial Linux Solutions | https://www.pengutronix.de/ |
Re: [PATCH net-next v5 2/2] net: stmmac: use per-queue 64 bit statistics where necessary
Quoted 67 lines · Uwe Kleine-König
Hello Guenter,
Quoted 64 lines · Guenter Roeck
Quoted 11 lines · Jisheng Zhang
Currently, there are two major issues with stmmac driver statistics First of all, statistics in
stmmac_extra_stats,stmmac_rxq_statsandstmmac_txq_statsare 32 bit variables on 32 bit platforms. This can cause some stats to overflow after several minutes of high traffic, for examplerx_pkt_n,tx_pkt_nand so on.Secondly, if HW supports multiqueues, there are frequent cacheline ping pongs on some driver statistic vars, for example,
normal_irq_n,tx_pkt_nand so on. What's more, frequent cacheline ping pongs onnormal_irq_nhappens in ISR, this makes the situation worse.Quoted 8 lines
To improve the driver, we convert those statistics to 64 bit, implement
ndo_get_stats64and update .get_ethtool_stats implementation accordingly. We also use per-queue statistics where necessary to remove the cacheline ping pongs as much as possible to make multiqueue operations faster. Those statistics which are not possible to overflow and not frequently updated are kept as is.Signed-off-by Jisheng Zhang <[email protected]>Your patch results in lockdep splats. This is with the orangepi-pc emulation in qemu.
[ 11.126950] dwmac-sun8i 1c30000.ethernet eth0: PHY [mdio_mux-0.1:01] driver [Generic PHY] (irq=POLL) [ 11.127912] dwmac-sun8i 1c30000.ethernet eth0: No Safety Features support found [ 11.128294] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management Counters available [ 11.128511] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW [ 11.138990] dwmac-sun8i 1c30000.ethernet eth0: configuring for phy/mii link mode [ 11.144387] INFO: trying to register non-static key. [ 11.144483] The code is fine but needs lockdep annotation, or maybe [ 11.144568] you didn't initialize this object before use? [ 11.144640] turning off the locking correctness validator. [ 11.144845] CPU: 2 PID: 688 Comm: ip Tainted: G N 6.6.0-rc2 #1 [ 11.144956] Hardware name: Allwinner sun8i Family [ 11.145137] unwind_backtrace from show_stack+0x10/0x14 [ 11.145610] show_stack from dump_stack_lvl+0x68/0x90 [ 11.145692] dump_stack_lvl from register_lock_class+0x99c/0x9b0 [ 11.145779] register_lock_class from __lock_acquire+0x6c/0x2244 [ 11.145861] __lock_acquire from lock_acquire+0x11c/0x368 [ 11.145938] lock_acquire from stmmac_get_stats64+0x350/0x374 [ 11.146021] stmmac_get_stats64 from dev_get_stats+0x3c/0x160 [ 11.146101] dev_get_stats from rtnl_fill_stats+0x30/0x118 [ 11.146179] rtnl_fill_stats from rtnl_fill_ifinfo.constprop.0+0x82c/0x1770 [ 11.146273] rtnl_fill_ifinfo.constprop.0 from rtmsg_ifinfo_build_skb+0xac/0x138 [ 11.146370] rtmsg_ifinfo_build_skb from rtmsg_ifinfo+0x44/0x7c [ 11.146452] rtmsg_ifinfo from __dev_notify_flags+0xac/0xd8 [ 11.146531] __dev_notify_flags from dev_change_flags+0x48/0x54 [ 11.146612] dev_change_flags from do_setlink+0x244/0xe6c [ 11.146689] do_setlink from rtnl_newlink+0x514/0x838 [ 11.146761] rtnl_newlink from rtnetlink_rcv_msg+0x170/0x5b0 [ 11.146841] rtnetlink_rcv_msg from netlink_rcv_skb+0xb4/0x10c [ 11.146925] netlink_rcv_skb from netlink_unicast+0x190/0x254 [ 11.147006] netlink_unicast from netlink_sendmsg+0x1dc/0x460 [ 11.147086] netlink_sendmsg from ____sys_sendmsg+0xa0/0x2a0 [ 11.147168] ____sys_sendmsg from ___sys_sendmsg+0x68/0x94 [ 11.147245] ___sys_sendmsg from sys_sendmsg+0x4c/0x88 [ 11.147329] sys_sendmsg from ret_fast_syscall+0x0/0x1c [ 11.147439] Exception stack(0xf23edfa8 to 0xf23edff0) [ 11.147558] dfa0: 00000000 00000000 00000003 bef9a8d8 00000000 00000000 [ 11.147668] dfc0: 00000000 00000000 ffffffff 00000128 00000001 00000002 bef9af4a bef9af4d [ 11.147769] dfe0: bef9a868 bef9a858 b6f9ddac b6f9d228 [ 11.150020] dwmac-sun8i 1c30000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Quoted 6 lines
Quoted 1 line
My apologies for the noise if this has already been reported.
This seems to be the issue I reported earlier. So you might want to test the patch that fixed it for me:
https://lore.kernel.org/netdev/[email protected]/
That just showed up in mainline and, yes, of course it fixes the problem. As I said, sorry for the noise.
Guenter