net/cxgbe: updates and bug fixes
cover net/cxgbe: updates and bug fixes
This series of patches add the following updates and bug fixes to the cxgbe PMD.
Patch 1 fixes an issue with wrong port id being filled in mbufs allocated in Rx path.
Patch 2 fixes an issue with Txq getting stuck when trying to coalesce mbufs with chains.
Patch 3 reworks and simplifies the code for posting mbufs and their payload buffer sizes to hardware in Rx path.
Patch 4 adds support to track packets in Rx path dropped due to congestion at Rx queues that may not get propagated to the rest of the Rx pipeline.
Patch 5 adds support to read firmware configuration file from /lib/firmware/cxgb4/ to allow changing firmware parameters without having to flash the configuration file onto the adapter.
Thanks,
Rahul
Rahul Lakkireddy (5):
net/cxgbe: fill correct port info in mbufs for Rx net/cxgbe: fix Tx queue stuck with mbuf chain coalescing net/cxgbe: simplify Rx payload buffer size posting net/cxgbe: track packets dropped by TP due to congestion net/cxgbe: read firmware configuration file from filesystem
drivers/net/cxgbe/base/adapter.h | 6 +- drivers/net/cxgbe/base/common.h | 4 +- drivers/net/cxgbe/base/t4_hw.c | 57 +--- drivers/net/cxgbe/base/t4_hw.h | 1 + drivers/net/cxgbe/base/t4_regs.h | 4 + drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/base/t4vf_hw.c | 40 --- drivers/net/cxgbe/cxgbe_ethdev.c | 11 + drivers/net/cxgbe/cxgbe_main.c | 332 ++++++++++++++++-------- drivers/net/cxgbe/sge.c | 296 +++++++-------------- 10 files changed, 351 insertions(+), 401 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/5 net/cxgbe: fill correct port info in mbufs for Rx
Fill the correct DPDK ethdev port_id, instead of local adapter physical port_id in mbufs allocated for Rx.
--- drivers/net/cxgbe/sge.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/net/cxgbe/sge.c +1 -1
@@ -1910,7 +1910,7 @@ int t4_sge_alloc_rxq(struct adapter *adap, struct sge_rspq *iq, bool fwevtq, iq->stat = (void *)&iq->desc[iq->size *8 ]; iq->eth_dev = eth_dev; iq->handler = hnd; - iq->port_id = pi->pidx; + iq->port_id = eth_dev->data->port_id; iq->mb_pool = mp;/* set offset to -1 to distinguish ingress queues without FL */
Signature · 1 line
2.27.0
PATCH 2/5 net/cxgbe: fix Tx queue stuck with mbuf chain coalescing
In reply to Rahul LakkireddyWhen trying to coalesce mbufs with chain on Tx side, it is possible to get stuck during queue wrap around. When coalescing this mbuf chain fails, the Tx path returns EBUSY and when the same packet is retried again, it couldn't get coalesced again, and the loop repeats. Fix by pushing the packet through the normal Tx path. Also use FW_ETH_TX_PKTS_WR to handle mbufs with chain for FW to optimize.
--- drivers/net/cxgbe/sge.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-)
drivers/net/cxgbe/sge.c +23 -15
@@ -789,9 +789,9 @@ static inline void txq_advance(struct sge_txq *q, unsigned int n)#define MAX_COALESCE_LEN 64000 -static inline int wraps_around (struct sge_txq *q,int ndesc) +static inline bool wraps_around (struct sge_txq *q,int ndesc) { -return (q->pidx + ndesc) > q->size ?1 :0 ; +return (q->pidx + ndesc) > q->size ? true : false; }static void tx_timer_cb (void *data)
@@ -842,7 +842,6 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap,/* fill the pkts WR header */ wr = (void *)&q->desc[q->pidx]; - wr->op_pkd =htonl (V_FW_WR_OP (FW_ETH_TX_PKTS2_WR )); vmwr = (void *)&q->desc[q->pidx]; wr_mid =V_FW_WR_LEN16 (DIV_ROUND_UP (q->coalesce.flits,2 ));
@@ -852,8 +851,11 @@ static inline void ship_tx_pkt_coalesce_wr(struct adapter *adap, wr->npkt = q->coalesce.idx; wr->r3 =0 ;if (is_pf4 (adap)) { - wr->op_pkd =htonl (V_FW_WR_OP (FW_ETH_TX_PKTS2_WR )); wr->type = q->coalesce.type; +if (likely (wr->type !=0 )) + wr->op_pkd =htonl (V_FW_WR_OP (FW_ETH_TX_PKTS2_WR )); +else + wr->op_pkd =htonl (V_FW_WR_OP (FW_ETH_TX_PKTS_WR )); }else { wr->op_pkd =htonl (V_FW_WR_OP (FW_ETH_TX_PKTS_VM_WR )); vmwr->r4 =0 ;
@@ -932,13 +934,16 @@ static inline int should_tx_packet_coalesce(struct sge_eth_txq *txq, ndesc =DIV_ROUND_UP (q->coalesce.flits + flits,8 ); credits =txq_avail (q) - ndesc; +if (unlikely (wraps_around (q, ndesc))) +return 0 ; +/* If we are wrapping or this is last mbuf then, send the * already coalesced mbufs and let the non-coalesce pass * handle the mbuf. */ -if (unlikely (credits <0 ||wraps_around (q, ndesc))) { +if (unlikely (credits <0 )) {ship_tx_pkt_coalesce_wr (adap, txq); -return 0 ; +return -EBUSY ; }/* If the max coalesce len or the max WR len is reached
@@ -962,8 +967,12 @@ static inline int should_tx_packet_coalesce(struct sge_eth_txq *txq, ndesc =flits_to_desc (q->coalesce.flits + flits); credits =txq_avail (q) - ndesc; -if (unlikely (credits <0 ||wraps_around (q, ndesc))) +if (unlikely (wraps_around (q, ndesc)))return 0 ; + +if (unlikely (credits <0 )) +return -EBUSY ; + q->coalesce.flits += wr_size /sizeof (__be64); q->coalesce.type = type; q->coalesce.ptr = (unsigned char *)&q->desc[q->pidx] +
@@ -1106,7 +1115,7 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf,unsigned int flits, ndesc, cflits;int l3hdr_len, l4hdr_len, eth_xtra_len;int len, last_desc; -int credits; +int should_coal, credits;u32 wr_mid;u64 cntrl, *end;bool v6;
@@ -1138,9 +1147,9 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf,/* align the end of coalesce WR to a 512 byte boundary */ txq->q.coalesce.max = (8 - (txq->q.pidx &7 )) *8 ; -if (!((m->ol_flags &RTE_MBUF_F_TX_TCP_SEG ) || - m->pkt_len >RTE_ETHER_MAX_LEN )) { -if (should_tx_packet_coalesce (txq, mbuf, &cflits, adap)) { +if ((m->ol_flags &RTE_MBUF_F_TX_TCP_SEG ) ==0 ) { + should_coal =should_tx_packet_coalesce (txq, mbuf, &cflits, adap); +if (should_coal >0 ) {if (unlikely (map_mbuf (mbuf, addr) <0 )) {dev_warn (adap,"%s: mapping err for coalesce\n" , __func__);
@@ -1149,8 +1158,8 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf, }return tx_do_packet_coalesce (txq, mbuf, cflits, adap, pi, addr, nb_pkts); - }else { -return -EBUSY ; + }else if (should_coal <0 ) { +return should_coal; } }
@@ -1197,8 +1206,7 @@ int t4_eth_xmit(struct sge_eth_txq *txq, struct rte_mbuf *mbuf, end = (u64 *)vmwr + flits; } - len =0 ; - len +=sizeof (*cpl); + len =sizeof (*cpl);/* Coalescing skipped and we send through normal path */ if (!(m->ol_flags &RTE_MBUF_F_TX_TCP_SEG )) {
Signature · 1 line
2.27.0
PATCH 3/5 net/cxgbe: simplify Rx payload buffer size posting
In reply to Rahul LakkireddyMatch the closest supported Rx payload buffer size with the mempool data size and program it for the Rx queue. This removes unnecessary need for handling additional padding, packing, and alignment, when posting Rx buffers to hardware.
--- drivers/net/cxgbe/base/adapter.h | 6 +- drivers/net/cxgbe/base/common.h | 2 - drivers/net/cxgbe/base/t4_hw.c | 43 ------ drivers/net/cxgbe/base/t4_hw.h | 1 + drivers/net/cxgbe/base/t4vf_hw.c | 40 ----- drivers/net/cxgbe/cxgbe_main.c | 3 +- drivers/net/cxgbe/sge.c | 256 ++++++++++--------------------- 7 files changed, 81 insertions(+), 270 deletions(-)
drivers/net/cxgbe/base/adapter.h +2 -4
@@ -98,6 +98,7 @@ struct sge_fl { /* SGE free-buffer queue state */unsigned long alloc_failed;/* # of times buffer allocation failed */ unsigned long low;/* # of times momentarily starving */ +u8 fl_buf_size_idx;/* Selected SGE_FL_BUFFER_SIZE index */ };#define MAX_MBUF_FRAGS (16384 / 512 + 2)
@@ -110,7 +111,6 @@ struct pkt_gl {void *va;/* virtual address of first byte */ unsigned int nfrags;/* # of fragments */ unsigned int tot_len;/* total length of fragments */ -bool usembufs;/* use mbufs for fragments */ }; typedef int (*rspq_handler_t)(struct sge_rspq *q, const __be64 *rsp,
@@ -160,7 +160,6 @@ struct sge_eth_rxq { /* a SW Ethernet Rx queue */struct sge_rspq rspq;struct sge_fl fl;struct sge_eth_rx_stats stats; -bool usembufs;/* one ingress packet per mbuf FL buffer */ } __rte_cache_aligned;/*
@@ -286,9 +285,8 @@ struct sge {u16 timer_val[SGE_NTIMERS ];u8 counter_val[SGE_NCOUNTERS ]; -u32 fl_align;/* response queue message alignment */ -u32 fl_pg_order;/* large page allocation size */ u32 fl_starve_thres;/* Free List starvation threshold */ +u32 fl_buffer_size[SGE_FL_BUFFER_SIZE_NUM ];/* Free List buffer sizes */ };/*
drivers/net/cxgbe/base/common.h +0 -2
@@ -343,8 +343,6 @@ int t4_fw_reset(struct adapter *adap, unsigned int mbox, int reset);int t4vf_fw_reset (struct adapter *adap);int t4_fw_halt (struct adapter *adap,unsigned int mbox,int reset);int t4_fw_restart (struct adapter *adap,unsigned int mbox,int reset); -int t4_fl_pkt_align (struct adapter *adap); -int t4vf_fl_pkt_align (struct adapter *adap,u32 sge_control,u32 sge_control2);int t4vf_get_vfres (struct adapter *adap); int t4_fixup_host_params_compat(struct adapter *adap, unsigned int page_size, unsigned int cache_line_size,
drivers/net/cxgbe/base/t4_hw.c +0 -43
49 lines not on this page · raw message
drivers/net/cxgbe/base/t4_hw.h +1 -0
@@ -28,6 +28,7 @@ enum {enum {SGE_NTIMERS =6 ,/* # of interrupt holdoff timer values */ SGE_NCOUNTERS =4 ,/* # of interrupt packet counter values */ +SGE_FL_BUFFER_SIZE_NUM =16 ,/* # of freelist buffser size regs */ };/* PCI-e memory window access */
drivers/net/cxgbe/base/t4vf_hw.c +0 -40
@@ -468,46 +468,6 @@ int t4vf_set_params(struct adapter *adapter, unsigned int nparams,return t4vf_wr_mbox (adapter, &cmd,sizeof (cmd),NULL ); } -/** -* t4vf_fl_pkt_align - return the fl packet alignment -* @adapter: the adapter -* -* T4 has a single field to specify the packing and padding boundary. -* T5 onwards has separate fields for this and hence the alignment for -* next packet offset is maximum of these two. -*/ -int t4vf_fl_pkt_align (struct adapter *adapter,u32 sge_control, -u32 sge_control2) -{ -unsigned int ingpadboundary, ingpackboundary, fl_align, ingpad_shift; - -/* T4 uses a single control field to specify both the PCIe Padding and -* Packing Boundary. T5 introduced the ability to specify these -* separately. The actual Ingress Packet Data alignment boundary -* within Packed Buffer Mode is the maximum of these two -* specifications. -*/ -if (CHELSIO_CHIP_VERSION (adapter->params.chip) <=CHELSIO_T5 ) - ingpad_shift =X_INGPADBOUNDARY_SHIFT ; -else - ingpad_shift =X_T6_INGPADBOUNDARY_SHIFT ; - - ingpadboundary =1 << (G_INGPADBOUNDARY (sge_control) + ingpad_shift); - - fl_align = ingpadboundary; -if (!is_t4 (adapter->params.chip)) { - ingpackboundary =G_INGPACKBOUNDARY (sge_control2); -if (ingpackboundary ==X_INGPACKBOUNDARY_16B ) - ingpackboundary =16 ; -else - ingpackboundary =1 << (ingpackboundary + -X_INGPACKBOUNDARY_SHIFT ); - - fl_align =max (ingpadboundary, ingpackboundary); - } -return fl_align; -} -unsigned int t4vf_get_pf_from_vf (struct adapter *adapter) {u32 whoami;
drivers/net/cxgbe/cxgbe_main.c +1 -2
@@ -623,8 +623,7 @@ int cxgbe_cfg_queues(struct rte_eth_dev *eth_dev)struct sge_eth_txq *t = &s->ethtxq[i];init_rspq (adap, &r->rspq,5 ,32 ,1024 ,64 ); - r->usembufs =1 ; - r->fl.size = (r->usembufs ?1024 :72 ); + r->fl.size =1024 ; t->q.size =1024 ; }
drivers/net/cxgbe/sge.c +77 -179
395 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH 4/5 net/cxgbe: track packets dropped by TP due to congestion
In reply to Rahul LakkireddyRx packets can get dropped at TP due to congestion and this info will not get propagated to MPS. Track these Rx dropped packets in imissed counter. Also add xstats for these counters.
--- drivers/net/cxgbe/base/common.h | 2 ++ drivers/net/cxgbe/base/t4_hw.c | 14 ++++++++++++-- drivers/net/cxgbe/base/t4_regs.h | 4 ++++ drivers/net/cxgbe/cxgbe_ethdev.c | 11 +++++++++++ 4 files changed, 29 insertions(+), 2 deletions(-)
drivers/net/cxgbe/base/common.h +2 -0
@@ -101,6 +101,8 @@ struct port_stats {u64 rx_trunc1;/* buffer-group 1 truncated packets */ u64 rx_trunc2;/* buffer-group 2 truncated packets */ u64 rx_trunc3;/* buffer-group 3 truncated packets */ + +u64 rx_tp_tnl_cong_drops[NCHAN ];/* TP frame drops due to congestion */ };struct sge_params {
drivers/net/cxgbe/base/t4_hw.c +12 -2
@@ -3040,8 +3040,10 @@ unsigned int t4_get_tp_ch_map(struct adapter *adapter, unsigned int pidx)*/ void t4_get_port_stats (struct adapter *adap,int idx,struct port_stats *p) { -u32 bgmap =t4_get_mps_bg_map (adap, idx);u32 stat_ctl =t4_read_reg (adap,A_MPS_STAT_CTL ); +u32 bgmap =t4_get_mps_bg_map (adap, idx); +u32 val[NCHAN ] = {0 }; +u8 i;#define GET_STAT (name) \ t4_read_reg64(adap, \
@@ -3129,6 +3131,11 @@ void t4_get_port_stats(struct adapter *adap, int idx, struct port_stats *p) p->rx_trunc2 = (bgmap &4 ) ?GET_STAT_COM (RX_BG_2_MAC_TRUNC_FRAME ) :0 ; p->rx_trunc3 = (bgmap &8 ) ?GET_STAT_COM (RX_BG_3_MAC_TRUNC_FRAME ) :0 ; +t4_read_indirect (adap,A_TP_MIB_INDEX ,A_TP_MIB_DATA , &val[idx],1 , +A_TP_MIB_TNL_CNG_DROP_0 + idx); + +for (i =0 ; i <NCHAN ; i++) + p->rx_tp_tnl_cong_drops[i] = val[i];#undef GET_STAT#undef GET_STAT_COM }
@@ -3163,9 +3170,10 @@ void t4_get_port_stats_offset(struct adapter *adap, int idx,*/ void t4_clr_port_stats (struct adapter *adap,int idx) { -unsigned int i;u32 bgmap =t4_get_mps_bg_map (adap, idx);u32 port_base_addr; +unsigned int i; +u32 val =0 ;if (is_t4 (adap->params.chip)) port_base_addr =PORT_BASE (idx);
@@ -3187,6 +3195,8 @@ void t4_clr_port_stats(struct adapter *adap, int idx)A_MPS_STAT_RX_BG_0_MAC_TRUNC_FRAME_L + i * 8, 0); } +t4_write_indirect (adap,A_TP_MIB_INDEX ,A_TP_MIB_DATA , + &val,1 ,A_TP_MIB_TNL_CNG_DROP_0 + idx); }/**
drivers/net/cxgbe/base/t4_regs.h +4 -0
@@ -525,6 +525,8 @@#define A_TP_PIO_ADDR 0x7e40#define A_TP_PIO_DATA 0x7e44 +#define A_TP_MIB_INDEX 0x7e50 +#define A_TP_MIB_DATA 0x7e54#define A_TP_RSS_SECRET_KEY0 0x40
@@ -587,6 +589,8 @@#define S_RM_OVLAN 9#define V_RM_OVLAN (x) ((x) << S_RM_OVLAN) +#define A_TP_MIB_TNL_CNG_DROP_0 0x18 +/* registers for module MA */ #define A_MA_EDRAM0_BAR 0x77c0
drivers/net/cxgbe/cxgbe_ethdev.c +11 -0
@@ -709,6 +709,9 @@ static int cxgbe_dev_stats_get(struct rte_eth_dev *eth_dev, ps.rx_ovflow2 + ps.rx_ovflow3 + ps.rx_trunc0 + ps.rx_trunc1 + ps.rx_trunc2 + ps.rx_trunc3; +for (i =0 ; i <NCHAN ; i++) + eth_stats->imissed += ps.rx_tp_tnl_cong_drops[i]; + eth_stats->ierrors = ps.rx_symbol_err + ps.rx_fcs_err + ps.rx_jabber + ps.rx_too_long + ps.rx_runt + ps.rx_len_err;
@@ -851,6 +854,14 @@ static const struct cxgbe_dev_xstats_name_off cxgbe_dev_port_stats_strings[] = { {"rx_bg1_truncated_packets", offsetof(struct port_stats, rx_trunc1)}, {"rx_bg2_truncated_packets", offsetof(struct port_stats, rx_trunc2)}, {"rx_bg3_truncated_packets", offsetof(struct port_stats, rx_trunc3)}, + {"rx_tp_tnl_cong_drops0", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[0])}, + {"rx_tp_tnl_cong_drops1", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[1])}, + {"rx_tp_tnl_cong_drops2", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[2])}, + {"rx_tp_tnl_cong_drops3", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[3])}, }; static const struct cxgbe_dev_xstats_name_off
Signature · 1 line
2.27.0
Re: [PATCH 4/5] net/cxgbe: track packets dropped by TP due to congestion
Quoted 5 lines · Rahul Lakkireddy
Rx packets can get dropped at TP due to congestion and this info will not get propagated to MPS. Track these Rx dropped packets in imissed counter. Also add xstats for these counters.
Signed-off-by Rahul Lakkireddy<[email protected]>
What 'TP' stands for? As far as I understand it is a kind of FW (TP Microcode) but I didn't able to find any reference to it.
Re: [PATCH 4/5] net/cxgbe: track packets dropped by TP due to congestion
Quoted 9 lines · Ferruh Yigit
Quoted 5 lines · Rahul Lakkireddy
Rx packets can get dropped at TP due to congestion and this info will not get propagated to MPS. Track these Rx dropped packets in imissed counter. Also add xstats for these counters.
Signed-off-by Rahul Lakkireddy<[email protected]>What 'TP' stands for? As far as I understand it is a kind of FW (TP Microcode) but I didn't able to find any reference to it.
Transport Processor (TP) on the NIC delivers the incoming packets from the wire to NIC's DMA engine to place the packets in Rx buffers. TP sends signal towards the Multi-Port Switch (MPS) near the MAC when the Rxqs run out of Rx buffers posted by driver. These MPS buffer drop stats are already accounted for in imissed counters. However, if a large number of Rxqs run out of Rx buffers simultaneously, then the TP can start dropping packets by itself without informing to the MPS if there is heavy congestion on the channel. These packets dropped by TP that could not be informed to MPS are now being accounted for in imissed counters in this patch.
Thanks,
Rahul
Re: [PATCH 4/5] net/cxgbe: track packets dropped by TP due to congestion
Quoted 11 lines · Rahul Lakkireddy
Quoted 9 lines · Ferruh Yigit
Quoted 5 lines · Rahul Lakkireddy
Rx packets can get dropped at TP due to congestion and this info will not get propagated to MPS. Track these Rx dropped packets in imissed counter. Also add xstats for these counters.
Signed-off-by Rahul Lakkireddy<[email protected]>What 'TP' stands for? As far as I understand it is a kind of FW (TP Microcode) but I didn't able to find any reference to it.
Quoted 11 lines
Transport Processor (TP) on the NIC delivers the incoming packets from the wire to NIC's DMA engine to place the packets in Rx buffers. TP sends signal towards the Multi-Port Switch (MPS) near the MAC when the Rxqs run out of Rx buffers posted by driver. These MPS buffer drop stats are already accounted for in imissed counters. However, if a large number of Rxqs run out of Rx buffers simultaneously, then the TP can start dropping packets by itself without informing to the MPS if there is heavy congestion on the channel. These packets dropped by TP that could not be informed to MPS are now being accounted for in imissed counters in this patch.
Thanks Rahul, this is clearer explanation than the original commit log.
Can you please put what abbreviations stands for (as you did in above paragraph) in the commit log in next version? Like:
'Transport Processor (TP)', 'Multi-Port Switch (MPS)'
PATCH v2 1/1 net/cxgbe: track packets dropped by TP due to congestion
In reply to Rahul LakkireddyTransport Processor (TP) on the NIC delivers the incoming packets from the wire to NIC's DMA engine to place the packets in Rx buffers. TP sends signal towards the Multi-Port Switch (MPS) near the MAC when the Rxqs run out of Rx buffers posted by driver. These MPS buffer drop stats are already accounted for in imissed counters. However, if a large number of Rxqs run out of Rx buffers simultaneously, then the TP can start dropping packets by itself when there is heavy congestion on the channel and hence could not inform to the MPS. So, track these packets dropped by TP in imissed counters. Also add xstats for these counters.
---
v2:
- Update commit message with more info about the TP counters.
drivers/net/cxgbe/base/common.h | 2 ++ drivers/net/cxgbe/base/t4_hw.c | 14 ++++++++++++-- drivers/net/cxgbe/base/t4_regs.h | 4 ++++ drivers/net/cxgbe/cxgbe_ethdev.c | 11 +++++++++++ 4 files changed, 29 insertions(+), 2 deletions(-)
drivers/net/cxgbe/base/common.h +2 -0
@@ -101,6 +101,8 @@ struct port_stats {u64 rx_trunc1;/* buffer-group 1 truncated packets */ u64 rx_trunc2;/* buffer-group 2 truncated packets */ u64 rx_trunc3;/* buffer-group 3 truncated packets */ + +u64 rx_tp_tnl_cong_drops[NCHAN ];/* TP frame drops due to congestion */ };struct sge_params {
drivers/net/cxgbe/base/t4_hw.c +12 -2
@@ -3040,8 +3040,10 @@ unsigned int t4_get_tp_ch_map(struct adapter *adapter, unsigned int pidx)*/ void t4_get_port_stats (struct adapter *adap,int idx,struct port_stats *p) { -u32 bgmap =t4_get_mps_bg_map (adap, idx);u32 stat_ctl =t4_read_reg (adap,A_MPS_STAT_CTL ); +u32 bgmap =t4_get_mps_bg_map (adap, idx); +u32 val[NCHAN ] = {0 }; +u8 i;#define GET_STAT (name) \ t4_read_reg64(adap, \
@@ -3129,6 +3131,11 @@ void t4_get_port_stats(struct adapter *adap, int idx, struct port_stats *p) p->rx_trunc2 = (bgmap &4 ) ?GET_STAT_COM (RX_BG_2_MAC_TRUNC_FRAME ) :0 ; p->rx_trunc3 = (bgmap &8 ) ?GET_STAT_COM (RX_BG_3_MAC_TRUNC_FRAME ) :0 ; +t4_read_indirect (adap,A_TP_MIB_INDEX ,A_TP_MIB_DATA , &val[idx],1 , +A_TP_MIB_TNL_CNG_DROP_0 + idx); + +for (i =0 ; i <NCHAN ; i++) + p->rx_tp_tnl_cong_drops[i] = val[i];#undef GET_STAT#undef GET_STAT_COM }
@@ -3163,9 +3170,10 @@ void t4_get_port_stats_offset(struct adapter *adap, int idx,*/ void t4_clr_port_stats (struct adapter *adap,int idx) { -unsigned int i;u32 bgmap =t4_get_mps_bg_map (adap, idx);u32 port_base_addr; +unsigned int i; +u32 val =0 ;if (is_t4 (adap->params.chip)) port_base_addr =PORT_BASE (idx);
@@ -3187,6 +3195,8 @@ void t4_clr_port_stats(struct adapter *adap, int idx)A_MPS_STAT_RX_BG_0_MAC_TRUNC_FRAME_L + i * 8, 0); } +t4_write_indirect (adap,A_TP_MIB_INDEX ,A_TP_MIB_DATA , + &val,1 ,A_TP_MIB_TNL_CNG_DROP_0 + idx); }/**
drivers/net/cxgbe/base/t4_regs.h +4 -0
@@ -525,6 +525,8 @@#define A_TP_PIO_ADDR 0x7e40#define A_TP_PIO_DATA 0x7e44 +#define A_TP_MIB_INDEX 0x7e50 +#define A_TP_MIB_DATA 0x7e54#define A_TP_RSS_SECRET_KEY0 0x40
@@ -587,6 +589,8 @@#define S_RM_OVLAN 9#define V_RM_OVLAN (x) ((x) << S_RM_OVLAN) +#define A_TP_MIB_TNL_CNG_DROP_0 0x18 +/* registers for module MA */ #define A_MA_EDRAM0_BAR 0x77c0
drivers/net/cxgbe/cxgbe_ethdev.c +11 -0
@@ -709,6 +709,9 @@ static int cxgbe_dev_stats_get(struct rte_eth_dev *eth_dev, ps.rx_ovflow2 + ps.rx_ovflow3 + ps.rx_trunc0 + ps.rx_trunc1 + ps.rx_trunc2 + ps.rx_trunc3; +for (i =0 ; i <NCHAN ; i++) + eth_stats->imissed += ps.rx_tp_tnl_cong_drops[i]; + eth_stats->ierrors = ps.rx_symbol_err + ps.rx_fcs_err + ps.rx_jabber + ps.rx_too_long + ps.rx_runt + ps.rx_len_err;
@@ -851,6 +854,14 @@ static const struct cxgbe_dev_xstats_name_off cxgbe_dev_port_stats_strings[] = { {"rx_bg1_truncated_packets", offsetof(struct port_stats, rx_trunc1)}, {"rx_bg2_truncated_packets", offsetof(struct port_stats, rx_trunc2)}, {"rx_bg3_truncated_packets", offsetof(struct port_stats, rx_trunc3)}, + {"rx_tp_tnl_cong_drops0", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[0])}, + {"rx_tp_tnl_cong_drops1", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[1])}, + {"rx_tp_tnl_cong_drops2", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[2])}, + {"rx_tp_tnl_cong_drops3", + offsetof(struct port_stats, rx_tp_tnl_cong_drops[3])}, }; static const struct cxgbe_dev_xstats_name_off
Signature · 1 line
2.27.0
Ferruh Yigit
AppliedRe: [PATCH v2] net/cxgbe: track packets dropped by TP due to congestion
Quoted 11 lines · Rahul Lakkireddy
Transport Processor (TP) on the NIC delivers the incoming packets from the wire to NIC's DMA engine to place the packets in Rx buffers. TP sends signal towards the Multi-Port Switch (MPS) near the MAC when the Rxqs run out of Rx buffers posted by driver. These MPS buffer drop stats are already accounted for in imissed counters. However, if a large number of Rxqs run out of Rx buffers simultaneously, then the TP can start dropping packets by itself when there is heavy congestion on the channel and hence could not inform to the MPS. So, track these packets dropped by TP in imissed counters. Also add xstats for these counters.
Quoted 1 line
Signed-off-by Rahul Lakkireddy <[email protected]>
Applied to dpdk-next-net/main, thanks.
PATCH 5/5 net/cxgbe: read firmware configuration file from filesystem
In reply to Rahul LakkireddyAdd support to read firmware configuration file from /lib/firmware/cxgb4/ path in the filesystem.
--- drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/cxgbe_main.c | 329 ++++++++++++++++-------- 2 files changed, 217 insertions(+), 113 deletions(-)
drivers/net/cxgbe/base/t4fw_interface.h +1 -0
@@ -697,6 +697,7 @@ enum fw_params_param_dev {*/ FW_PARAMS_PARAM_DEV_FWREV =0x0B ,/* fw version */ FW_PARAMS_PARAM_DEV_TPREV =0x0C ,/* tp version */ +FW_PARAMS_PARAM_DEV_CF =0x0D ,FW_PARAMS_PARAM_DEV_ULPTX_MEMWRITE_DSGL =0x17 ,FW_PARAMS_PARAM_DEV_FILTER2_WR =0x1D , FW_PARAMS_PARAM_DEV_OPAQUE_VIID_SMT_EXTN = 0x27,
drivers/net/cxgbe/cxgbe_main.c +216 -113
371 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
Quoted 3 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem.
Hi Rahul,
Can you please document the FW config file in the driver documentation?
Please add:
- Path of the config file
- Content of the config file. As far as I can see from the code the config file directly sent to the FW, does this mean config file is binary?
- What happens when config file is not found
Also are these values overlap with the devargs that PMD has? If so what happens in that case, which one is used, devargs one or config file one?
Previously there was 'cxgbtool' tool to send the config file, is this method replacing it? Why not keep using 'cxgbtool'?
Thanks,
ferruh
Quoted 1 line
Signed-off-by Rahul Lakkireddy<[email protected]>
<...>
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
Quoted 14 lines · Ferruh Yigit
Quoted 3 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem.Hi Rahul,
Can you please document the FW config file in the driver documentation? Please add:
- Path of the config file
- Content of the config file. As far as I can see from the code the config file directly sent to the FW, does this mean config file is binary?
- What happens when config file is not found
Quoted 4 lines
Also are these values overlap with the devargs that PMD has? If so what happens in that case, which one is used, devargs one or config file one?
Previously there was 'cxgbtool' tool to send the config file, is this method replacing it? Why not keep using 'cxgbtool'?
cc'ed more folks.
This patch introduces a userspace config file for runtime FW config.
What do you think about this approach?
Should we formalize this method more, like introducing an ethdev level config option to hold the config file, which can be used for driver and/or FW. And perhaps with a defined syntax (yaml?). Can this be an alternative to PMD devargs?
Cheers,
ferruh
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
Quoted 16 lines · Ferruh Yigit
Quoted 15 lines · Ferruh Yigit
Quoted 3 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem.Hi Rahul,
Can you please document the FW config file in the driver documentation? Please add:
- Path of the config file
- Content of the config file. As far as I can see from the code the config file directly sent to the FW, does this mean config file is binary?
- What happens when config file is not found
Quoted 6 lines
Quoted 6 lines
Also are these values overlap with the devargs that PMD has? If so what happens in that case, which one is used, devargs one or config file one?
Previously there was 'cxgbtool' tool to send the config file, is this method replacing it? Why not keep using 'cxgbtool'?
The Chelsio FW config file contains a list of register=value pairs to change configuration of the NIC before firmware is initialized. It closely resembles the INI file format. It is mainly used to aid in debugging FW initialization failures and to optimally partition NIC hardware resources for specific requirements. Partitioning generally involves moving resources on unused Physical Functions (PFs) to the main PF, like redistributing queues, hardware TCAMs, etc., before firmware begins initialization. Once the configuration looks good, then the final FW config file is flashed onto the NIC using the cxgbtool. The FW config file can then be removed from the /lib/firmware/cxgb4/ directory and the FW will begin initializing with the flashed FW config file on the NIC from next time onwards.
With this patch, the FW config file is selected in following order.
1) Check and select FW config file present in /lib/firmware/cxgb4/ directory.
2) Otherwise, check and select FW config file flashed onto the NIC.
3) Otherwise, select the default FW config file embedded within the FW binary on the NIC.
Since this is pretty low-level hardware configuration, the users are not expected to change this file without expert guidance. So, exporting such a low-level configuration via devargs API does not feel like the right fit for this specific requirement.
Once FW is up and running with the FW config file, some of the driver and FW runtime features can be enabled/disabled via devargs API during driver probe.
To summarize, the FW config file is intended to debug FW initialization failures and/or aid in resources partitioning before FW starts initialization. Once the FW is running, the whole or smaller slices of these partitioned resources can be further redistributed across the multiple physical ports controlled by the same underlying PF.
Quoted 5 lines
cc'ed more folks.
This patch introduces a userspace config file for runtime FW config.
Quoted 8 lines
What do you think about this approach?
Should we formalize this method more, like introducing an ethdev level config option to hold the config file, which can be used for driver and/or FW. And perhaps with a defined syntax (yaml?).
Can this be an alternative to PMD devargs?Cheers,
ferruh
Thanks,
Rahul
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
06/05/2022 13:36, Rahul Lakkireddy:
Quoted 33 lines
The Chelsio FW config file contains a list of register=value pairs to change configuration of the NIC before firmware is initialized. It closely resembles the INI file format. It is mainly used to aid in debugging FW initialization failures and to optimally partition NIC hardware resources for specific requirements. Partitioning generally involves moving resources on unused Physical Functions (PFs) to the main PF, like redistributing queues, hardware TCAMs, etc., before firmware begins initialization. Once the configuration looks good, then the final FW config file is flashed onto the NIC using the cxgbtool. The FW config file can then be removed from the
/lib/firmware/cxgb4/directory and the FW will begin initializing with the flashed FW config file on the NIC from next time onwards.With this patch, the FW config file is selected in following order.
1) Check and select FW config file present in
/lib/firmware/cxgb4/directory.2) Otherwise, check and select FW config file flashed onto the NIC.
3) Otherwise, select the default FW config file embedded within the FW binary on the NIC.
Since this is pretty low-level hardware configuration, the users are not expected to change this file without expert guidance. So, exporting such a low-level configuration via devargs API does not feel like the right fit for this specific requirement.
Once FW is up and running with the FW config file, some of the driver and FW runtime features can be enabled/disabled via devargs API during driver probe.
Quoted 5 lines
To summarize, the FW config file is intended to debug FW initialization failures and/or aid in resources partitioning before FW starts initialization. Once the FW is running, the whole or smaller slices of these partitioned resources can be further redistributed across the multiple physical ports controlled by the same underlying PF.
Sorry it is not clear to me.
The FW file is used by cxgbtool to flash it, right? The PMD may use the same FW file for debug diagnostics? How is it used by the kernel?
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
Quoted 40 lines · Thomas Monjalon
06/05/2022 13:36, Rahul Lakkireddy:
Quoted 33 lines
The Chelsio FW config file contains a list of register=value pairs to change configuration of the NIC before firmware is initialized. It closely resembles the INI file format. It is mainly used to aid in debugging FW initialization failures and to optimally partition NIC hardware resources for specific requirements. Partitioning generally involves moving resources on unused Physical Functions (PFs) to the main PF, like redistributing queues, hardware TCAMs, etc., before firmware begins initialization. Once the configuration looks good, then the final FW config file is flashed onto the NIC using the cxgbtool. The FW config file can then be removed from the
/lib/firmware/cxgb4/directory and the FW will begin initializing with the flashed FW config file on the NIC from next time onwards.With this patch, the FW config file is selected in following order.
1) Check and select FW config file present in
/lib/firmware/cxgb4/directory.2) Otherwise, check and select FW config file flashed onto the NIC.
3) Otherwise, select the default FW config file embedded within the FW binary on the NIC.
Since this is pretty low-level hardware configuration, the users are not expected to change this file without expert guidance. So, exporting such a low-level configuration via devargs API does not feel like the right fit for this specific requirement.
Once FW is up and running with the FW config file, some of the driver and FW runtime features can be enabled/disabled via devargs API during driver probe.
Quoted 5 lines
To summarize, the FW config file is intended to debug FW initialization failures and/or aid in resources partitioning before FW starts initialization. Once the FW is running, the whole or smaller slices of these partitioned resources can be further redistributed across the multiple physical ports controlled by the same underlying PF.
Quoted 3 lines
Sorry it is not clear to me.
The FW file is used by cxgbtool to flash it, right? The PMD may use the same FW file for debug diagnostics?
There are 2 FW related files in /lib/firmware/cxgb4 directory that can be written into the NIC flash (at different locations in the flash) using cxgbtool:
1) FW config file (t6-config.txt): This file contains register=value pairs to configure some HW registers before initializing FW (t6fw.bin). These register=value pairs are very low level and specific to HW. The FW config file can be used to enable/disable some paths in HW to debug FW initialization issues. Since it can disable some paths in HW, it can also be used to partition and redistribute resources from other unused PFs to main PF, before FW initialization.
2) FW binary (t6fw.bin): This contains the actual FW. This binary also has an embedded "default" FW config file that will be used if no FW config file (t6-config.txt) is present in NIC "flash" or in the "/lib/firmware/cxgb4/" directory.
Quoted 2 lines
How is it used by the kernel?
In kernel cxgb4 driver also, we follow the same sequence when initializing FW. If any FW config file (t6-config.txt) is present in "/lib/firmware/cxgb4" directory, then it is picked up first. Otherwise, a FW config file that has been "flashed" onto the NIC using cxgbtool, is picked up next. Otherwise, FW will init with the "default" FW config file embedded inside the FW binary (t6fw.bin).
Currently, the DPDK cxgbe PMD only picks up the FW config file from NIC "flash" or the "default" one inside the FW binary. This patch adds a way to pick FW config file from "lib/firmware/cxgb4" directory when a FW config file is placed there.
Thanks,
Rahul
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
10/05/2022 16:11, Rahul Lakkireddy:
Quoted 74 lines · Thomas Monjalon
Quoted 40 lines
06/05/2022 13:36, Rahul Lakkireddy:
Quoted 33 lines
The Chelsio FW config file contains a list of register=value pairs to change configuration of the NIC before firmware is initialized. It closely resembles the INI file format. It is mainly used to aid in debugging FW initialization failures and to optimally partition NIC hardware resources for specific requirements. Partitioning generally involves moving resources on unused Physical Functions (PFs) to the main PF, like redistributing queues, hardware TCAMs, etc., before firmware begins initialization. Once the configuration looks good, then the final FW config file is flashed onto the NIC using the cxgbtool. The FW config file can then be removed from the
/lib/firmware/cxgb4/directory and the FW will begin initializing with the flashed FW config file on the NIC from next time onwards.With this patch, the FW config file is selected in following order.
1) Check and select FW config file present in
/lib/firmware/cxgb4/directory.2) Otherwise, check and select FW config file flashed onto the NIC.
3) Otherwise, select the default FW config file embedded within the FW binary on the NIC.
Since this is pretty low-level hardware configuration, the users are not expected to change this file without expert guidance. So, exporting such a low-level configuration via devargs API does not feel like the right fit for this specific requirement.
Once FW is up and running with the FW config file, some of the driver and FW runtime features can be enabled/disabled via devargs API during driver probe.
Quoted 5 lines
To summarize, the FW config file is intended to debug FW initialization failures and/or aid in resources partitioning before FW starts initialization. Once the FW is running, the whole or smaller slices of these partitioned resources can be further redistributed across the multiple physical ports controlled by the same underlying PF.
Quoted 3 lines
Sorry it is not clear to me.
The FW file is used by cxgbtool to flash it, right? The PMD may use the same FW file for debug diagnostics?There are 2 FW related files in
/lib/firmware/cxgb4directory that can be written into the NIC flash (at different locations in the flash) using cxgbtool:1) FW config file (t6-config.txt): This file contains register=value pairs to configure some HW registers before initializing FW (t6fw.bin). These register=value pairs are very low level and specific to HW. The FW config file can be used to enable/disable some paths in HW to debug FW initialization issues. Since it can disable some paths in HW, it can also be used to partition and redistribute resources from other unused PFs to main PF, before FW initialization.
2) FW binary (t6fw.bin): This contains the actual FW. This binary also has an embedded "default" FW config file that will be used if no FW config file (t6-config.txt) is present in NIC "flash" or in the "
/lib/firmware/cxgb4/" directory.Quoted 2 lines
How is it used by the kernel?
In kernel cxgb4 driver also, we follow the same sequence when initializing FW. If any FW config file (t6-config.txt) is present in "
/lib/firmware/cxgb4" directory, then it is picked up first. Otherwise, a FW config file that has been "flashed" onto the NIC using cxgbtool, is picked up next. Otherwise, FW will init with the "default" FW config file embedded inside the FW binary (t6fw.bin).
Quoted 4 lines
Currently, the DPDK cxgbe PMD only picks up the FW config file from NIC "flash" or the "default" one inside the FW binary. This patch adds a way to pick FW config file from "
lib/firmware/cxgb4" directory when a FW config file is placed there.
OK, so both Linux and DPDK drivers read the FW config file to provide better diagnostics. Am I right?
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
Quoted 75 lines · Thomas Monjalon
10/05/2022 16:11, Rahul Lakkireddy:
Quoted 74 lines · Thomas Monjalon
Quoted 40 lines
06/05/2022 13:36, Rahul Lakkireddy:
Quoted 33 lines
The Chelsio FW config file contains a list of register=value pairs to change configuration of the NIC before firmware is initialized. It closely resembles the INI file format. It is mainly used to aid in debugging FW initialization failures and to optimally partition NIC hardware resources for specific requirements. Partitioning generally involves moving resources on unused Physical Functions (PFs) to the main PF, like redistributing queues, hardware TCAMs, etc., before firmware begins initialization. Once the configuration looks good, then the final FW config file is flashed onto the NIC using the cxgbtool. The FW config file can then be removed from the
/lib/firmware/cxgb4/directory and the FW will begin initializing with the flashed FW config file on the NIC from next time onwards.With this patch, the FW config file is selected in following order.
1) Check and select FW config file present in
/lib/firmware/cxgb4/directory.2) Otherwise, check and select FW config file flashed onto the NIC.
3) Otherwise, select the default FW config file embedded within the FW binary on the NIC.
Since this is pretty low-level hardware configuration, the users are not expected to change this file without expert guidance. So, exporting such a low-level configuration via devargs API does not feel like the right fit for this specific requirement.
Once FW is up and running with the FW config file, some of the driver and FW runtime features can be enabled/disabled via devargs API during driver probe.
Quoted 5 lines
To summarize, the FW config file is intended to debug FW initialization failures and/or aid in resources partitioning before FW starts initialization. Once the FW is running, the whole or smaller slices of these partitioned resources can be further redistributed across the multiple physical ports controlled by the same underlying PF.
Quoted 3 lines
Sorry it is not clear to me.
The FW file is used by cxgbtool to flash it, right? The PMD may use the same FW file for debug diagnostics?There are 2 FW related files in
/lib/firmware/cxgb4directory that can be written into the NIC flash (at different locations in the flash) using cxgbtool:1) FW config file (t6-config.txt): This file contains register=value pairs to configure some HW registers before initializing FW (t6fw.bin). These register=value pairs are very low level and specific to HW. The FW config file can be used to enable/disable some paths in HW to debug FW initialization issues. Since it can disable some paths in HW, it can also be used to partition and redistribute resources from other unused PFs to main PF, before FW initialization.
2) FW binary (t6fw.bin): This contains the actual FW. This binary also has an embedded "default" FW config file that will be used if no FW config file (t6-config.txt) is present in NIC "flash" or in the "
/lib/firmware/cxgb4/" directory.Quoted 2 lines
How is it used by the kernel?
In kernel cxgb4 driver also, we follow the same sequence when initializing FW. If any FW config file (t6-config.txt) is present in "
/lib/firmware/cxgb4" directory, then it is picked up first. Otherwise, a FW config file that has been "flashed" onto the NIC using cxgbtool, is picked up next. Otherwise, FW will init with the "default" FW config file embedded inside the FW binary (t6fw.bin).
Quoted 8 lines
Quoted 4 lines
Currently, the DPDK cxgbe PMD only picks up the FW config file from NIC "flash" or the "default" one inside the FW binary. This patch adds a way to pick FW config file from "
lib/firmware/cxgb4" directory when a FW config file is placed there.OK, so both Linux and DPDK drivers read the FW config file to provide better diagnostics. Am I right?
Yes, both use the FW config file to enable/disable specific paths in HW to get better diagnostics for FW init failures.
Thanks,
Rahul
Re: [PATCH 5/5] net/cxgbe: read firmware configuration file from filesystem
10/05/2022 17:05, Rahul Lakkireddy:
Quoted 81 lines · Thomas Monjalon
Quoted 80 lines
10/05/2022 16:11, Rahul Lakkireddy:
Quoted 78 lines · Thomas Monjalon
Quoted 40 lines
06/05/2022 13:36, Rahul Lakkireddy:
Quoted 33 lines
The Chelsio FW config file contains a list of register=value pairs to change configuration of the NIC before firmware is initialized. It closely resembles the INI file format. It is mainly used to aid in debugging FW initialization failures and to optimally partition NIC hardware resources for specific requirements. Partitioning generally involves moving resources on unused Physical Functions (PFs) to the main PF, like redistributing queues, hardware TCAMs, etc., before firmware begins initialization. Once the configuration looks good, then the final FW config file is flashed onto the NIC using the cxgbtool. The FW config file can then be removed from the
/lib/firmware/cxgb4/directory and the FW will begin initializing with the flashed FW config file on the NIC from next time onwards.With this patch, the FW config file is selected in following order.
1) Check and select FW config file present in
/lib/firmware/cxgb4/directory.2) Otherwise, check and select FW config file flashed onto the NIC.
3) Otherwise, select the default FW config file embedded within the FW binary on the NIC.
Since this is pretty low-level hardware configuration, the users are not expected to change this file without expert guidance. So, exporting such a low-level configuration via devargs API does not feel like the right fit for this specific requirement.
Once FW is up and running with the FW config file, some of the driver and FW runtime features can be enabled/disabled via devargs API during driver probe.
Quoted 5 lines
To summarize, the FW config file is intended to debug FW initialization failures and/or aid in resources partitioning before FW starts initialization. Once the FW is running, the whole or smaller slices of these partitioned resources can be further redistributed across the multiple physical ports controlled by the same underlying PF.
Quoted 3 lines
Sorry it is not clear to me.
The FW file is used by cxgbtool to flash it, right? The PMD may use the same FW file for debug diagnostics?There are 2 FW related files in
/lib/firmware/cxgb4directory that can be written into the NIC flash (at different locations in the flash) using cxgbtool:1) FW config file (t6-config.txt): This file contains register=value pairs to configure some HW registers before initializing FW (t6fw.bin). These register=value pairs are very low level and specific to HW. The FW config file can be used to enable/disable some paths in HW to debug FW initialization issues. Since it can disable some paths in HW, it can also be used to partition and redistribute resources from other unused PFs to main PF, before FW initialization.
2) FW binary (t6fw.bin): This contains the actual FW. This binary also has an embedded "default" FW config file that will be used if no FW config file (t6-config.txt) is present in NIC "flash" or in the "
/lib/firmware/cxgb4/" directory.Quoted 2 lines
How is it used by the kernel?
In kernel cxgb4 driver also, we follow the same sequence when initializing FW. If any FW config file (t6-config.txt) is present in "
/lib/firmware/cxgb4" directory, then it is picked up first. Otherwise, a FW config file that has been "flashed" onto the NIC using cxgbtool, is picked up next. Otherwise, FW will init with the "default" FW config file embedded inside the FW binary (t6fw.bin).Currently, the DPDK cxgbe PMD only picks up the FW config file from NIC "flash" or the "default" one inside the FW binary. This patch adds a way to pick FW config file from "
lib/firmware/cxgb4" directory when a FW config file is placed there.
Quoted 6 lines
Quoted 3 lines
OK, so both Linux and DPDK drivers read the FW config file to provide better diagnostics. Am I right?
Yes, both use the FW config file to enable/disable specific paths in HW to get better diagnostics for FW init failures.
OK thanks for the explanation.
The idea looks OK.
PATCH v2 1/1 net/cxgbe: read firmware configuration file from filesystem
In reply to Rahul LakkireddyAdd support to read firmware configuration file from /lib/firmware/cxgb4/ path in the filesystem. The firmware config file is used to enable or disable NIC features before firmware initialization to help retrieve better debug data to analyze firmware init failures. The config file can also be used to redistribute resources, like queues, TCAMs, etc., from disabled physical functions (PFs) to main PF, before firmware init.
---
v2:
- Update cxgbe pmd doc about the firmware config file
- Update commit message to explain more about firmware config file
doc/guides/nics/cxgbe.rst | 28 ++ drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/cxgbe_main.c | 329 ++++++++++++++++-------- 3 files changed, 245 insertions(+), 113 deletions(-)
doc/guides/nics/cxgbe.rst +28 -0
@@ -838,3 +838,31 @@ to configure the mtu of all the ports with a single command. testpmd> port stop all testpmd> port config all max-pkt-len 9000 + +Hardware Configuration and Debugging +------------------------------------ + +Firmware Configuration File +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To enable or disable Chelsio NIC features before firmware initialization, +the Chelsio firmware configuration file can be placed in following +directory. The CXGBE PMD will search and pick up the firmware +configuration file during the Chelsio NIC probe, before initializing +the firmware. + +.. code-block:: console + + cp <path_to_config_file>/t6-config.txt /lib/firmware/cxgb4/t6-config.txt + +The firmware configuration file is mainly intended to be used to debug +firmware initialization failures. It can also be used to redistribute +NIC resources from disabled physical functions (PFs) to main PF before +initializing firmware. + +.. warning:: + + Note that the Chelsio firmware configuration file contains very low + level details that is specific to the Chelsio NIC. Hence, the + firmware configuration file must not be modified without expert + guidance from Chelsio support team.
drivers/net/cxgbe/base/t4fw_interface.h +1 -0
@@ -697,6 +697,7 @@ enum fw_params_param_dev {*/ FW_PARAMS_PARAM_DEV_FWREV =0x0B ,/* fw version */ FW_PARAMS_PARAM_DEV_TPREV =0x0C ,/* tp version */ +FW_PARAMS_PARAM_DEV_CF =0x0D ,FW_PARAMS_PARAM_DEV_ULPTX_MEMWRITE_DSGL =0x17 ,FW_PARAMS_PARAM_DEV_FILTER2_WR =0x1D , FW_PARAMS_PARAM_DEV_OPAQUE_VIID_SMT_EXTN = 0x27,
drivers/net/cxgbe/cxgbe_main.c +216 -113
371 lines not on this page · raw message
Signature · 1 line
2.27.0
Re: [PATCH v2] net/cxgbe: read firmware configuration file from filesystem
Quoted 8 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem. The firmware config file is used to enable or disable NIC features before firmware initialization to help retrieve better debug data to analyze firmware init failures. The config file can also be used to redistribute resources, like queues, TCAMs, etc., from disabled physical functions (PFs) to main PF, before firmware init.
Hi Rahul,
Please find comments below.
Also can you please send a new version for both 4/5 and 5/5 (this patch) from original series?
Quoted 23 lines
Signed-off-by Rahul Lakkireddy <[email protected]>---
v2:
- Update cxgbe pmd doc about the firmware config file
- Update commit message to explain more about firmware config filedoc/guides/nics/cxgbe.rst | 28 ++ drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/cxgbe_main.c | 329 ++++++++++++++++-------- 3 files changed, 245 insertions(+), 113 deletions(-)doc/guides/nics/cxgbe.rst +3
-0@@ -838,3 +838,31 @@ to configure the mtu of all the ports with a single command. testpmd> port stop all testpmd> port config all max-pkt-len 9000 + +Hardware Configuration and Debugging +------------------------------------
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 853
Quoted 10 lines
doc/guides/nics/cxgbe.rst +10
-0+ +Firmware Configuration File +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To enable or disable Chelsio NIC features before firmware initialization, +the Chelsio firmware configuration file can be placed in following +directory. The CXGBE PMD will search and pick up the firmware +configuration file during the Chelsio NIC probe, before initializing +the firmware. +
Does it worth to mention what happens if the FW config file doesn't exist? Or mention from FW config file load order, as you described in mail list, to understand the relation with 'cxgbtool'?
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 857
Quoted 4 lines
doc/guides/nics/cxgbe.rst +4
-0+.. code-block:: console + + cp <path_to_config_file>/t6-config.txt /lib/firmware/cxgb4/t6-config.txt +
There is also 't5-config.txt' in the code.
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 868
Quoted 11 lines
doc/guides/nics/cxgbe.rst +11
-0+The firmware configuration file is mainly intended to be used to debug +firmware initialization failures. It can also be used to redistribute +NIC resources from disabled physical functions (PFs) to main PF before +initializing firmware. + +.. warning:: + + Note that the Chelsio firmware configuration file contains very low + level details that is specific to the Chelsio NIC. Hence, the + firmware configuration file must not be modified without expert + guidance from Chelsio support team.
Will it be too much detail to document what config can be changed via this FW config?
<...>
Re: [PATCH v2] net/cxgbe: read firmware configuration file from filesystem
Quoted 10 lines · Ferruh Yigit
Quoted 8 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem. The firmware config file is used to enable or disable NIC features before firmware initialization to help retrieve better debug data to analyze firmware init failures. The config file can also be used to redistribute resources, like queues, TCAMs, etc., from disabled physical functions (PFs) to main PF, before firmware init.
Quoted 7 lines
Hi Rahul,
Please find comments below.
Also can you please send a new version for both 4/5 and 5/5 (this patch) from original series?
I had already posted v2 for 4/5 from original series at below location.
Do you want me to send a new patch series with 4/5 and 5/5 in a single patch series?
Quoted 28 lines
Quoted 28 lines
Signed-off-by Rahul Lakkireddy <[email protected]>---
v2:
- Update cxgbe pmd doc about the firmware config file
- Update commit message to explain more about firmware config filedoc/guides/nics/cxgbe.rst | 28 ++ drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/cxgbe_main.c | 329 ++++++++++++++++-------- 3 files changed, 245 insertions(+), 113 deletions(-)doc/guides/nics/cxgbe.rst +9
-0@@ -838,3 +838,31 @@ to configure the mtu of all the ports with a single command. testpmd> port stop all testpmd> port config all max-pkt-len 9000 + +Hardware Configuration and Debugging +------------------------------------ + +Firmware Configuration File +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To enable or disable Chelsio NIC features before firmware initialization, +the Chelsio firmware configuration file can be placed in following
Quoted 9 lines
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 853
Quoted 4 lines
doc/guides/nics/cxgbe.rst +4
-0+directory. The CXGBE PMD will search and pick up the firmware +configuration file during the Chelsio NIC probe, before initializing +the firmware. +Does it worth to mention what happens if the FW config file doesn't exist? Or mention from FW config file load order, as you described in mail list, to understand the relation with 'cxgbtool'?
Will do.
Quoted 7 lines
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 857
Quoted 4 lines
doc/guides/nics/cxgbe.rst +4
-0+.. code-block:: console + + cp <path_to_config_file>/t6-config.txt /lib/firmware/cxgb4/t6-config.txt +There is also 't5-config.txt' in the code.
Yes, t5-config.txt is for Chelsio T5 NIC series and t6-config.txt is for Chelsio T6 NIC series. Will update t5-config.txt as well.
Quoted 6 lines
Quoted 6 lines
doc/guides/nics/cxgbe.rst +6
-0+The firmware configuration file is mainly intended to be used to debug +firmware initialization failures. It can also be used to redistribute +NIC resources from disabled physical functions (PFs) to main PF before +initializing firmware. + +.. warning::
Quoted 9 lines
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 868
Quoted 5 lines
doc/guides/nics/cxgbe.rst +5
-0+ + Note that the Chelsio firmware configuration file contains very low + level details that is specific to the Chelsio NIC. Hence, the + firmware configuration file must not be modified without expert + guidance from Chelsio support team.Will it be too much detail to document what config can be changed via this FW config?
Yes, this will be difficult to document. The config file just looks like register=value pairs and are often dependent on each other based on the selected configuration. A portion of sample config file is given below. I'm not sure how to document this kind of info.
[global]
reg[0x100c] = 0x22222222
reg[0x10a0] = 0x01040810
reg[0x1044] = 4096
reg[0x1048] = 65536
reg[0x104c] = 1536
reg[0x1050] = 9024
reg[0x1054] = 9216
reg[0x1058] = 2048
reg[0x105c] = 128
reg[0x1060] = 8192
reg[0x1064] = 16384[...]
Thanks,
Rahul
Re: [PATCH v2] net/cxgbe: read firmware configuration file from filesystem
Quoted 19 lines · Rahul Lakkireddy
Quoted 10 lines · Ferruh Yigit
Quoted 8 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem. The firmware config file is used to enable or disable NIC features before firmware initialization to help retrieve better debug data to analyze firmware init failures. The config file can also be used to redistribute resources, like queues, TCAMs, etc., from disabled physical functions (PFs) to main PF, before firmware init.Quoted 7 lines
Hi Rahul,
Please find comments below.
Also can you please send a new version for both 4/5 and 5/5 (this patch) from original series?
Quoted 7 lines
I had already posted v2 for 4/5 from original series at below location.
Do you want me to send a new patch series with 4/5 and 5/5 in a single patch series?
Nope, I missed above patch, it is good as separate patches.
Quoted 84 lines
Quoted 28 lines
Quoted 28 lines
Signed-off-by Rahul Lakkireddy <[email protected]>---
v2:
- Update cxgbe pmd doc about the firmware config file
- Update commit message to explain more about firmware config filedoc/guides/nics/cxgbe.rst | 28 ++ drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/cxgbe_main.c | 329 ++++++++++++++++-------- 3 files changed, 245 insertions(+), 113 deletions(-)doc/guides/nics/cxgbe.rst +9
-0@@ -838,3 +838,31 @@ to configure the mtu of all the ports with a single command. testpmd> port stop all testpmd> port config all max-pkt-len 9000 + +Hardware Configuration and Debugging +------------------------------------ + +Firmware Configuration File +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To enable or disable Chelsio NIC features before firmware initialization, +the Chelsio firmware configuration file can be placed in followingQuoted 9 lines
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 853
Quoted 4 lines
doc/guides/nics/cxgbe.rst +4
-0+directory. The CXGBE PMD will search and pick up the firmware +configuration file during the Chelsio NIC probe, before initializing +the firmware. +Does it worth to mention what happens if the FW config file doesn't exist? Or mention from FW config file load order, as you described in mail list, to understand the relation with 'cxgbtool'?
Will do.
Quoted 7 lines
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 857
Quoted 4 lines
doc/guides/nics/cxgbe.rst +4
-0+.. code-block:: console + + cp <path_to_config_file>/t6-config.txt /lib/firmware/cxgb4/t6-config.txt +There is also 't5-config.txt' in the code.
Yes, t5-config.txt is for Chelsio T5 NIC series and t6-config.txt is for Chelsio T6 NIC series. Will update t5-config.txt as well.
Quoted 6 lines
Quoted 6 lines
doc/guides/nics/cxgbe.rst +6
-0+The firmware configuration file is mainly intended to be used to debug +firmware initialization failures. It can also be used to redistribute +NIC resources from disabled physical functions (PFs) to main PF before +initializing firmware. + +.. warning::Quoted 9 lines
doc/guides/nics/cxgbe.rst to configure the mtu of all the ports with a single command. · line 868
Quoted 5 lines
doc/guides/nics/cxgbe.rst +5
-0+ + Note that the Chelsio firmware configuration file contains very low + level details that is specific to the Chelsio NIC. Hence, the + firmware configuration file must not be modified without expert + guidance from Chelsio support team.Will it be too much detail to document what config can be changed via this FW config?
Yes, this will be difficult to document. The config file just looks like register=value pairs and are often dependent on each other based on the selected configuration. A portion of sample config file is given below. I'm not sure how to document this kind of info.
[global]
reg[0x100c] = 0x22222222 reg[0x10a0] = 0x01040810 reg[0x1044] = 4096 reg[0x1048] = 65536 reg[0x104c] = 1536 reg[0x1050] = 9024 reg[0x1054] = 9216 reg[0x1058] = 2048 reg[0x105c] = 128 reg[0x1060] = 8192 reg[0x1064] = 16384
Quoted 2 lines
[...]
Above won't be much useful to user anyway (without description of the values), if there is already some documentation a link to that can be good, otherwise OK to skip this.
PATCH v3 1/1 net/cxgbe: read firmware configuration file from filesystem
In reply to Rahul LakkireddyAdd support to read firmware configuration file from /lib/firmware/cxgb4/ path in the filesystem. The firmware config file is used to enable or disable NIC features before firmware initialization to help retrieve better debug data to analyze firmware init failures. The config file can also be used to redistribute resources, like queues, TCAMs, etc., from disabled physical functions (PFs) to main PF, before firmware init.
---
v3:
- Update cxgbe pmd doc about copying t5-config.txt to /lib/firmware/cxgb4/ directory.
- Update cxgbe pmd doc about firmware config file search and selection order.
v2:
- Update cxgbe pmd doc about the firmware config file
- Update commit message to explain more about firmware config file
doc/guides/nics/cxgbe.rst | 48 ++++ drivers/net/cxgbe/base/t4fw_interface.h | 1 + drivers/net/cxgbe/cxgbe_main.c | 329 ++++++++++++++++-------- 3 files changed, 265 insertions(+), 113 deletions(-)
doc/guides/nics/cxgbe.rst +48 -0
@@ -838,3 +838,51 @@ to configure the mtu of all the ports with a single command. testpmd> port stop all testpmd> port config all max-pkt-len 9000 + +Hardware Configuration and Debugging +------------------------------------ + +Firmware Configuration File +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To enable or disable Chelsio NIC features before firmware initialization, +the Chelsio firmware configuration file can be placed in following +directory. + +.. code-block:: console + + # For Chelsio T5 NIC series + cp <path_to_config_file>/t5-config.txt /lib/firmware/cxgb4/t5-config.txt + + # For Chelsio T6 NIC series + cp <path_to_config_file>/t6-config.txt /lib/firmware/cxgb4/t6-config.txt + +The firmware configuration file is mainly intended to be used to debug +firmware initialization failures. It can also be used to redistribute +NIC resources from disabled physical functions (PFs) to main PF before +initializing firmware. + +The CXGBE PMD will search and pick up the firmware configuration file +during the Chelsio NIC probe, in following order. + +#. If the firmware configuration file is present in /lib/firmware/cxgb4/ + directory, then this file is downloaded to temporary location in + NIC's on-chip RAM. When firmware is initializing, it picks up the + file from the temporary on-chip RAM location. + +#. Otherwise, if the firmware configuration file has been written + onto the NIC persistent flash area using cxgbtool, then this + file is picked up from the persistent flash area during + firmware initialization. + +#. If no firmware configuration file is found at /lib/firmware/cxgb4/ + directory or on the NIC persistent flash area, then the firmware + will initialize with the default configuration file embedded inside + the firmware binary. + +.. warning:: + + Note that the Chelsio firmware configuration file contains very low + level details that is specific to the Chelsio NIC. Hence, the + firmware configuration file must not be modified without expert + guidance from Chelsio support team.
drivers/net/cxgbe/base/t4fw_interface.h +1 -0
@@ -697,6 +697,7 @@ enum fw_params_param_dev {*/ FW_PARAMS_PARAM_DEV_FWREV =0x0B ,/* fw version */ FW_PARAMS_PARAM_DEV_TPREV =0x0C ,/* tp version */ +FW_PARAMS_PARAM_DEV_CF =0x0D ,FW_PARAMS_PARAM_DEV_ULPTX_MEMWRITE_DSGL =0x17 ,FW_PARAMS_PARAM_DEV_FILTER2_WR =0x1D , FW_PARAMS_PARAM_DEV_OPAQUE_VIID_SMT_EXTN = 0x27,
drivers/net/cxgbe/cxgbe_main.c +216 -113
371 lines not on this page · raw message
Signature · 1 line
2.27.0
Ferruh Yigit
AppliedRe: [PATCH v3] net/cxgbe: read firmware configuration file from filesystem
Quoted 10 lines · Rahul Lakkireddy
Add support to read firmware configuration file from
/lib/firmware/cxgb4/path in the filesystem. The firmware config file is used to enable or disable NIC features before firmware initialization to help retrieve better debug data to analyze firmware init failures. The config file can also be used to redistribute resources, like queues, TCAMs, etc., from disabled physical functions (PFs) to main PF, before firmware init.Signed-off-by Rahul Lakkireddy <[email protected]>
Applied to dpdk-next-net/main, thanks.
Quoted 23 lines · Rahul Lakkireddy
This series of patches add the following updates and bug fixes to the cxgbe PMD.
Patch 1 fixes an issue with wrong port id being filled in mbufs allocated in Rx path.
Patch 2 fixes an issue with Txq getting stuck when trying to coalesce mbufs with chains.
Patch 3 reworks and simplifies the code for posting mbufs and their payload buffer sizes to hardware in Rx path.
Patch 4 adds support to track packets in Rx path dropped due to congestion at Rx queues that may not get propagated to the rest of the Rx pipeline.
Patch 5 adds support to read firmware configuration file from
/lib/firmware/cxgb4/to allow changing firmware parameters without having to flash the configuration file onto the adapter.Thanks,
Rahul
Quoted 7 lines
Rahul Lakkireddy (5):
net/cxgbe: fill correct port info in mbufs for Rx net/cxgbe: fix Tx queue stuck with mbuf chain coalescing net/cxgbe: simplify Rx payload buffer size posting net/cxgbe: track packets dropped by TP due to congestion net/cxgbe: read firmware configuration file from filesystem
Except from 4/5 & 5/5,
Series applied to dpdk-next-net/main, thanks.
Please check comments for 4/5 & 5/5. Since patches are independent, I think it would be OK to send a new series for remaining patches.
Thanks,
ferruh