net: qed/qede: critical hw error handling
Subsystems (3)
v2 cover net: qed/qede: critical hw error handling
FastLinQ devices as a complex systems may observe various hardware level error conditions, both severe and recoverable.
Driver is able to detect and report this, but so far it only did trace/dmesg based reporting.
Here we implement an extended hw error detection, service task handler captures a dump for the later analysis.
I also resubmit a patch from Denis Bolotin on tx timeout handler, addressing David's comment regarding recovery procedure as an extra reaction on this event.
v2:
Removing the patch with ethtool dump and udev magic. Its quite isolated, I'm working on devlink based logic for this separately.
v1:
https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
Denis Bolotin (1):
net: qede: Implement ndo_tx_timeout
Igor Russkikh (10):
net: qed: adding hw_err states and handling
net: qede: add hw err scheduled handler
net: qed: invoke err notify on critical areas
net: qed: critical err reporting to management firmware
net: qed: cleanup debug related declarations
net: qed: attention clearing properties
net: qede: optional hw recovery procedure
net: qed: introduce critical fan failure handler
net: qed: introduce critical hardware error handler
net: qed: fix bad formatting
drivers/net/ethernet/qlogic/qed/qed.h | 16 +- drivers/net/ethernet/qlogic/qed/qed_debug.c | 26 +- drivers/net/ethernet/qlogic/qed/qed_dev.c | 4 +- drivers/net/ethernet/qlogic/qed/qed_hsi.h | 49 +++- drivers/net/ethernet/qlogic/qed/qed_hw.c | 42 ++- drivers/net/ethernet/qlogic/qed/qed_hw.h | 15 ++ drivers/net/ethernet/qlogic/qed/qed_int.c | 40 ++- drivers/net/ethernet/qlogic/qed/qed_int.h | 11 + drivers/net/ethernet/qlogic/qed/qed_main.c | 34 +++ drivers/net/ethernet/qlogic/qed/qed_mcp.c | 254 ++++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_mcp.h | 28 ++ drivers/net/ethernet/qlogic/qed/qed_spq.c | 16 +- drivers/net/ethernet/qlogic/qede/qede.h | 14 +- .../net/ethernet/qlogic/qede/qede_ethtool.c | 24 ++ drivers/net/ethernet/qlogic/qede/qede_main.c | 147 +++++++++- include/linux/qed/qed_if.h | 26 +- 16 files changed, 700 insertions(+), 46 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/11 net: qed: adding hw_err states and handling
Here we introduce qed device error tracking flags and error types.
qed_hw_err_notify is an entrace point to report errors. It'll notify higher level drivers (qede/qedr/etc) to handle and recover the error.
List of posible errors comes from hardware interfaces, but could be extended in future.
--- drivers/net/ethernet/qlogic/qed/qed.h | 2 ++ drivers/net/ethernet/qlogic/qed/qed_hw.c | 32 ++++++++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_hw.h | 15 ++++++++++ drivers/net/ethernet/qlogic/qed/qed_main.c | 29 ++++++++++++++++++++ include/linux/qed/qed_if.h | 12 ++++++++ 5 files changed, 90 insertions(+)
drivers/net/ethernet/qlogic/qed/qed.h +2 -0
@@ -1020,6 +1020,8 @@ u32 qed_unzip_data(struct qed_hwfn *p_hwfn, u32 input_len, u8 *input_buf, u32 max_size, u8 *unzip_buf);void qed_schedule_recovery_handler (struct qed_hwfn *p_hwfn); +void qed_hw_error_occurred (struct qed_hwfn *p_hwfn, +enum qed_hw_err_type err_type);void qed_get_protocol_stats (struct qed_dev *cdev,enum qed_mcp_protocol_type type,union qed_mcp_protocol_stats *stats);
drivers/net/ethernet/qlogic/qed/qed_hw.c +32 -0
@@ -837,6 +837,38 @@ int qed_dmae_host2host(struct qed_hwfn *p_hwfn,return rc; } +void qed_hw_err_notify (struct qed_hwfn *p_hwfn, +struct qed_ptt *p_ptt, +enum qed_hw_err_type err_type,char *fmt, ...) +{ +char buf[QED_HW_ERR_MAX_STR_SIZE ]; +va_list vl; +int len; + +if (fmt) { +va_start (vl, fmt); + len =vsnprintf (buf,QED_HW_ERR_MAX_STR_SIZE , fmt, vl); +va_end (vl); + +if (len >QED_HW_ERR_MAX_STR_SIZE -1 ) + len =QED_HW_ERR_MAX_STR_SIZE -1 ; + +DP_NOTICE (p_hwfn,"%s" , buf); + } + +/* Fan failure cannot be masked by handling of another HW error */ +if (p_hwfn->cdev->recov_in_prog && + err_type !=QED_HW_ERR_FAN_FAIL ) { +DP_VERBOSE (p_hwfn, +NETIF_MSG_DRV , +"Recovery is in progress. Avoid notifying about HW error %d.\n" , + err_type); +return ; + } + +qed_hw_error_occurred (p_hwfn, err_type); +} +int qed_dmae_sanity (struct qed_hwfn *p_hwfn,struct qed_ptt *p_ptt,const char *phase) {
drivers/net/ethernet/qlogic/qed/qed_hw.h +15 -0
@@ -315,4 +315,19 @@ int qed_init_fw_data(struct qed_dev *cdev,int qed_dmae_sanity (struct qed_hwfn *p_hwfn,struct qed_ptt *p_ptt,const char *phase); +#define QED_HW_ERR_MAX_STR_SIZE 256 + +/** +* @brief qed_hw_err_notify - Notify upper layer driver and management FW +* about a HW error. +* +* @param p_hwfn +* @param p_ptt +* @param err_type +* @param fmt - debug data buffer to send to the MFW +* @param ... - buffer format args +*/ +void qed_hw_err_notify (struct qed_hwfn *p_hwfn, +struct qed_ptt *p_ptt, +enum qed_hw_err_type err_type,char *fmt, ...);#endif
drivers/net/ethernet/qlogic/qed/qed_main.c +29 -0
@@ -2468,6 +2468,35 @@ void qed_schedule_recovery_handler(struct qed_hwfn *p_hwfn) ops->schedule_recovery_handler (cookie); } +char *qed_hw_err_type_descr[] = { + [QED_HW_ERR_FAN_FAIL ] ="Fan Failure" , + [QED_HW_ERR_MFW_RESP_FAIL ] ="MFW Response Failure" , + [QED_HW_ERR_HW_ATTN ] ="HW Attention" , + [QED_HW_ERR_DMAE_FAIL ] ="DMAE Failure" , + [QED_HW_ERR_RAMROD_FAIL ] ="Ramrod Failure" , + [QED_HW_ERR_FW_ASSERT ] ="FW Assertion" , + [QED_HW_ERR_LAST ] ="Unknown" , +}; + +void qed_hw_error_occurred (struct qed_hwfn *p_hwfn, +enum qed_hw_err_type err_type) +{ +struct qed_common_cb_ops *ops = p_hwfn->cdev->protocol_ops.common; +void *cookie = p_hwfn->cdev->ops_cookie; +char *err_str; + +if (err_type >QED_HW_ERR_LAST ) + err_type =QED_HW_ERR_LAST ; + err_str = qed_hw_err_type_descr[err_type]; + +DP_NOTICE (p_hwfn,"HW error occurred [%s]\n" , err_str); + +/* Call the HW error handler of the protocol driver +*/ +if (ops && ops->schedule_hw_err_handler) + ops->schedule_hw_err_handler (cookie, err_type); +} +static int qed_set_coalesce (struct qed_dev *cdev,u16 rx_coal,u16 tx_coal,void *handle) {
include/linux/qed/qed_if.h +12 -0
@@ -607,6 +607,16 @@ struct qed_sb_info {struct qed_dev *cdev; }; +enum qed_hw_err_type { +QED_HW_ERR_FAN_FAIL , +QED_HW_ERR_MFW_RESP_FAIL , +QED_HW_ERR_HW_ATTN , +QED_HW_ERR_DMAE_FAIL , +QED_HW_ERR_RAMROD_FAIL , +QED_HW_ERR_FW_ASSERT , +QED_HW_ERR_LAST , +}; +enum qed_dev_type {QED_DEV_TYPE_BB ,QED_DEV_TYPE_AH ,
@@ -814,6 +824,8 @@ struct qed_common_cb_ops {void (*link_update)(void *dev,struct qed_link_output *link);void (*schedule_recovery_handler)(void *dev); +void (*schedule_hw_err_handler)(void *dev, +enum qed_hw_err_type err_type);void (*dcbx_aen)(void *dev,struct qed_dcbx_get *get,u32 mib_type);void (*get_generic_tlv_data)(void *dev,struct qed_generic_tlvs *data);void (*get_protocol_tlv_data)(void *dev,void *data);
Signature · 1 line
2.17.1
PATCH v2 2/11 net: qede: add hw err scheduled handler
In reply to Igor Russkikhqede (ethernet level driver) registers a callback handler. This handler maintains eth dev state flags/bits to track error processing.
It implements in place processing part for nonsleeping context (WARN_ON trigger), and a deferred (delayed work) part which triggers recovery process for recoverable errors.
In later patches this atomic handler will come with more meat.
We introduce err_flags on ethdevice structure, its being used to record error handling properties.
--- drivers/net/ethernet/qlogic/qede/qede.h | 13 ++- drivers/net/ethernet/qlogic/qede/qede_main.c | 95 +++++++++++++++++++- 2 files changed, 106 insertions(+), 2 deletions(-)
drivers/net/ethernet/qlogic/qede/qede.h +12 -1
@@ -278,6 +278,14 @@ struct qede_dev {struct qede_rdma_dev rdma_info;struct bpf_prog *xdp_prog; + +unsigned long err_flags; +#define QEDE_ERR_IS_HANDLED 31 +#define QEDE_ERR_ATTN_CLR_EN 0 +#define QEDE_ERR_GET_DBG_INFO 1 +#define QEDE_ERR_IS_RECOVERABLE 2 +#define QEDE_ERR_WARN 3 +struct qede_dump_info dump_info; };
@@ -485,12 +493,15 @@ struct qede_fastpath {#define QEDE_SP_RECOVERY 0#define QEDE_SP_RX_MODE 1 +#define QEDE_SP_RSVD1 2 +#define QEDE_SP_RSVD2 3 +#define QEDE_SP_HW_ERR 4 +#define QEDE_SP_ARFS_CONFIG 5#define QEDE_SP_AER 7#ifdef CONFIG_RFS_ACCEL int qede_rx_flow_steer (struct net_device *dev,const struct sk_buff *skb,u16 rxq_index,u32 flow_id); -#define QEDE_SP_ARFS_CONFIG 4#define QEDE_SP_TASK_POLL_DELAY (5 * HZ)#endif
drivers/net/ethernet/qlogic/qede/qede_main.c +94 -1
122 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v2 3/11 net: qed: invoke err notify on critical areas
In reply to Igor RusskikhIn a number of critical places not only debug trace should be printed, but the appropriate hw error condition should be raised and error handling/recovery should start.
Introduce our new qed_hw_err_notify invocation in these places to record and indicate critical error conditions in hardware.
--- drivers/net/ethernet/qlogic/qed/qed_dev.c | 4 +++- drivers/net/ethernet/qlogic/qed/qed_hw.c | 7 ++++--- drivers/net/ethernet/qlogic/qed/qed_int.c | 20 ++++++++++++++++---- drivers/net/ethernet/qlogic/qed/qed_mcp.c | 2 ++ drivers/net/ethernet/qlogic/qed/qed_spq.c | 16 ++++++++++------ 5 files changed, 35 insertions(+), 14 deletions(-)
drivers/net/ethernet/qlogic/qed/qed_dev.c +3 -1
@@ -3085,7 +3085,9 @@ int qed_hw_init(struct qed_dev *cdev, struct qed_hw_init_params *p_params) rc =qed_final_cleanup (p_hwfn, p_hwfn->p_main_ptt, p_hwfn->rel_pf_id, false);if (rc) { -DP_NOTICE (p_hwfn,"Final cleanup failed\n" ); +qed_hw_err_notify (p_hwfn, p_hwfn->p_main_ptt, +QED_HW_ERR_RAMROD_FAIL , +"Final cleanup failed\n" ); goto load_err; } }
drivers/net/ethernet/qlogic/qed/qed_hw.c +4 -3
@@ -762,9 +762,10 @@ static int qed_dmae_execute_command(struct qed_hwfn *p_hwfn, dst_type, length_cur);if (qed_status) { -DP_NOTICE (p_hwfn, -"qed_dmae_execute_sub_operation Failed with error 0x%x. source_addr 0x%llx, destination addr 0x%llx, size_in_dwords 0x%x\n" , - qed_status, src_addr, dst_addr, length_cur); +qed_hw_err_notify (p_hwfn, p_ptt,QED_HW_ERR_DMAE_FAIL , +"qed_dmae_execute_sub_operation Failed with error 0x%x. source_addr 0x%llx, destination addr 0x%llx, size_in_dwords 0x%x\n" , + qed_status, src_addr, + dst_addr, length_cur);break ; } }
drivers/net/ethernet/qlogic/qed/qed_int.c +16 -4
@@ -363,6 +363,14 @@ static int qed_pglueb_rbc_attn_cb(struct qed_hwfn *p_hwfn)return qed_pglueb_rbc_attn_handler (p_hwfn, p_hwfn->p_dpc_ptt); } +static int qed_fw_assertion (struct qed_hwfn *p_hwfn) +{ +qed_hw_err_notify (p_hwfn, p_hwfn->p_dpc_ptt,QED_HW_ERR_FW_ASSERT , +"FW assertion!\n" ); + +return -EINVAL ; +} +#define QED_DORQ_ATTENTION_REASON_MASK (0xfffff)#define QED_DORQ_ATTENTION_OPAQUE_MASK (0xffff)#define QED_DORQ_ATTENTION_OPAQUE_SHIFT (0x0)
@@ -606,7 +614,8 @@ static struct aeu_invert_reg aeu_descs[NUM_ATTN_REGS] = { { {/* After Invert 4 */ {"General Attention 32" ,ATTENTION_SINGLE , - NULL, MAX_BLOCK_ID}, + qed_fw_assertion, + MAX_BLOCK_ID}, {"General Attention %d", (2 <<ATTENTION_LENGTH_SHIFT ) | (33 << ATTENTION_OFFSET_SHIFT), NULL, MAX_BLOCK_ID},
@@ -927,9 +936,12 @@ qed_int_deassertion_aeu_bit(struct qed_hwfn *p_hwfn,qed_int_attn_print (p_hwfn, p_aeu->block_index,ATTN_TYPE_INTERRUPT , !b_fatal); - -/* If the attention is benign, no need to prevent it */ -if (!rc) +/* Reach assertion if attention is fatal */ +if (b_fatal) +qed_hw_err_notify (p_hwfn, p_hwfn->p_dpc_ptt,QED_HW_ERR_HW_ATTN , +"`%s': Fatal attention\n" , + p_bit_name); +else /* If the attention is benign, no need to prevent it */ goto out;/* Prevent this Attention from being asserted in the future */
drivers/net/ethernet/qlogic/qed/qed_mcp.c +2 -0
@@ -575,6 +575,8 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,if (!QED_MB_FLAGS_IS_SET (p_mb_params,AVOID_BLOCK ))qed_mcp_cmd_set_blocking (p_hwfn, true); +qed_hw_err_notify (p_hwfn, p_ptt, +QED_HW_ERR_MFW_RESP_FAIL ,NULL );return -EAGAIN ; }
drivers/net/ethernet/qlogic/qed/qed_spq.c +10 -6
@@ -160,12 +160,16 @@ static int qed_spq_block(struct qed_hwfn *p_hwfn,return 0 ; } err: -DP_NOTICE (p_hwfn, -"Ramrod is stuck [CID %08x cmd %02x protocol %02x echo %04x]\n" , -le32_to_cpu (p_ent->elem.hdr.cid), - p_ent->elem.hdr.cmd_id, - p_ent->elem.hdr.protocol_id, -le16_to_cpu (p_ent->elem.hdr.echo)); + p_ptt =qed_ptt_acquire (p_hwfn); +if (!p_ptt) +return -EBUSY ; +qed_hw_err_notify (p_hwfn, p_ptt,QED_HW_ERR_RAMROD_FAIL , +"Ramrod is stuck [CID %08x cmd %02x protocol %02x echo %04x]\n" , +le32_to_cpu (p_ent->elem.hdr.cid), + p_ent->elem.hdr.cmd_id, + p_ent->elem.hdr.protocol_id, +le16_to_cpu (p_ent->elem.hdr.echo)); +qed_ptt_release (p_hwfn, p_ptt);return -EBUSY ; }
Signature · 1 line
2.17.1
PATCH v2 4/11 net: qed: critical err reporting to management firmware
In reply to Igor RusskikhOn various critical errors, notification handler should also report the err information into the management firmware.
MFW can interact with server/motherboard backend agents - these are used by server manufacturers to monitor server HW health.
Thus, it is important for driver to report on any faulty conditions
--- drivers/net/ethernet/qlogic/qed/qed_hsi.h | 19 ++++ drivers/net/ethernet/qlogic/qed/qed_hw.c | 3 + drivers/net/ethernet/qlogic/qed/qed_mcp.c | 125 ++++++++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_mcp.h | 15 +++ 4 files changed, 162 insertions(+)
drivers/net/ethernet/qlogic/qed/qed_hsi.h +19 -0
@@ -12492,6 +12492,8 @@ struct public_drv_mb {#define DRV_MSG_CODE_GET_ENGINE_CONFIG 0x00370000#define DRV_MSG_CODE_GET_PPFID_BITMAP 0x43000000 +#define DRV_MSG_CODE_DEBUG_DATA_SEND 0xc0040000 +#define RESOURCE_CMD_REQ_RESC_MASK 0x0000001F#define RESOURCE_CMD_REQ_RESC_SHIFT 0#define RESOURCE_CMD_REQ_OPCODE_MASK 0x000000E0
@@ -12626,6 +12628,17 @@ struct public_drv_mb {#define DRV_MB_PARAM_FEATURE_SUPPORT_PORT_EEE 0x00000002#define DRV_MB_PARAM_FEATURE_SUPPORT_FUNC_VLINK 0x00010000 +/* DRV_MSG_CODE_DEBUG_DATA_SEND parameters */ +#define DRV_MSG_CODE_DEBUG_DATA_SEND_SIZE_OFFSET 0 +#define DRV_MSG_CODE_DEBUG_DATA_SEND_SIZE_MASK 0xFF + +/* Driver attributes params */ +#define DRV_MB_PARAM_ATTRIBUTE_KEY_OFFSET 0 +#define DRV_MB_PARAM_ATTRIBUTE_KEY_MASK 0x00FFFFFF +#define DRV_MB_PARAM_ATTRIBUTE_CMD_OFFSET 24 +#define DRV_MB_PARAM_ATTRIBUTE_CMD_MASK 0xFF000000 + +#define DRV_MB_PARAM_NVM_CFG_OPTION_ID_OFFSET 0#define DRV_MB_PARAM_NVM_CFG_OPTION_ID_SHIFT 0#define DRV_MB_PARAM_NVM_CFG_OPTION_ID_MASK 0x0000FFFF#define DRV_MB_PARAM_NVM_CFG_OPTION_ALL_SHIFT 16
@@ -12678,6 +12691,12 @@ struct public_drv_mb {#define FW_MSG_CODE_DRV_CFG_PF_VFS_MSIX_DONE 0x00870000#define FW_MSG_SEQ_NUMBER_MASK 0x0000ffff +#define FW_MSG_CODE_DEBUG_DATA_SEND_INV_ARG 0xb0070000 +#define FW_MSG_CODE_DEBUG_DATA_SEND_BUF_FULL 0xb0080000 +#define FW_MSG_CODE_DEBUG_DATA_SEND_NO_BUF 0xb0090000 +#define FW_MSG_CODE_DEBUG_NOT_ENABLED 0xb00a0000 +#define FW_MSG_CODE_DEBUG_DATA_SEND_OK 0xb00b0000 +u32 fw_mb_param;#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_MASK 0xFFFF0000#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_SHIFT 16
drivers/net/ethernet/qlogic/qed/qed_hw.c +3 -0
@@ -868,6 +868,9 @@ void qed_hw_err_notify(struct qed_hwfn *p_hwfn, }qed_hw_error_occurred (p_hwfn, err_type); + +if (fmt) +qed_mcp_send_raw_debug_data (p_hwfn, p_ptt, buf, len); } int qed_dmae_sanity(struct qed_hwfn *p_hwfn,
drivers/net/ethernet/qlogic/qed/qed_mcp.c +125 -0
128 lines not on this page · raw message
drivers/net/ethernet/qlogic/qed/qed_mcp.h +15 -0
@@ -685,6 +685,18 @@ int qed_mcp_bist_nvm_get_image_att(struct qed_hwfn *p_hwfn,*/ int qed_mfw_process_tlv_req (struct qed_hwfn *p_hwfn,struct qed_ptt *p_ptt); +/** +* @brief Send raw debug data to the MFW +* +* @param p_hwfn +* @param p_ptt +* @param p_buf - raw debug data buffer +* @param size - buffer size +*/ +int +qed_mcp_send_raw_debug_data (struct qed_hwfn *p_hwfn, +struct qed_ptt *p_ptt,u8 *p_buf,u32 size); +/* Using hwfn number (and not pf_num) is required since in CMT mode, * same pf_num may be used by two different hwfn * TODO - this shouldn't really be in .h file, but until all fields
@@ -731,6 +743,9 @@ struct qed_mcp_info {/* Capabilties negotiated with the MFW */ u32 capabilities; + +/* S/N for debug data mailbox commands */ +atomic_t dbg_data_seq; };struct qed_mcp_mb_params {
Signature · 1 line
2.17.1
PATCH v2 5/11 net: qed: cleanup debug related declarations
In reply to Igor RusskikhThats probably a legacy code had double declaration of some fields. Cleanup this, removing copy and fixing references.
--- drivers/net/ethernet/qlogic/qed/qed.h | 11 +++------ drivers/net/ethernet/qlogic/qed/qed_debug.c | 26 ++++++++++----------- 2 files changed, 16 insertions(+), 21 deletions(-)
drivers/net/ethernet/qlogic/qed/qed.h +3 -8
@@ -740,12 +740,6 @@ struct qed_dbg_feature {u32 dumped_dwords; }; -struct qed_dbg_params { -struct qed_dbg_feature features[DBG_FEATURE_NUM ]; -u8 engine_for_debug; -bool print_data; -}; -struct qed_dev {u32 dp_module;u8 dp_level;
@@ -872,17 +866,18 @@ struct qed_dev { } protocol_ops;void *ops_cookie; -struct qed_dbg_params dbg_params; -#ifdef CONFIG_QED_LL2 struct qed_cb_ll2_info *ll2;u8 ll2_mac_address[ETH_ALEN ];#endif struct qed_dbg_feature dbg_features[DBG_FEATURE_NUM ]; +u8 engine_for_debug;bool disable_ilt_dump;DECLARE_HASHTABLE (connections,10 );const struct firmware *firmware; +bool print_dbg_data; +u32 rdma_max_sge;u32 rdma_max_inline;u32 rdma_max_srq_sge;
drivers/net/ethernet/qlogic/qed/qed_debug.c +13 -13
@@ -7453,7 +7453,7 @@ static enum dbg_status format_feature(struct qed_hwfn *p_hwfn, enum qed_dbg_features feature_idx) {struct qed_dbg_feature *feature = - &p_hwfn->cdev->dbg_params.features[feature_idx]; + &p_hwfn->cdev->dbg_features[feature_idx];u32 text_size_bytes, null_char_pos, i;enum dbg_status rc;char *text_buf;
@@ -7502,7 +7502,7 @@ static enum dbg_status format_feature(struct qed_hwfn *p_hwfn, text_buf[i] ='\n' ;/* Dump printable feature to log */ -if (p_hwfn->cdev->dbg_params.print_data) +if (p_hwfn->cdev->print_dbg_data)qed_dbg_print_feature (text_buf, text_size_bytes);/* Free the old dump_buf and point the dump_buf to the newly allocagted
@@ -7523,7 +7523,7 @@ static enum dbg_status qed_dbg_dump(struct qed_hwfn *p_hwfn, enum qed_dbg_features feature_idx) {struct qed_dbg_feature *feature = - &p_hwfn->cdev->dbg_params.features[feature_idx]; + &p_hwfn->cdev->dbg_features[feature_idx];u32 buf_size_dwords;enum dbg_status rc;
@@ -7648,7 +7648,7 @@ static int qed_dbg_nvm_image(struct qed_dev *cdev, void *buffer, enum qed_nvm_images image_id) {struct qed_hwfn *p_hwfn = - &cdev->hwfns[cdev->dbg_params.engine_for_debug]; + &cdev->hwfns[cdev->engine_for_debug];u32 len_rounded, i;__be32 val;int rc;
@@ -7780,7 +7780,7 @@ int qed_dbg_all_data(struct qed_dev *cdev, void *buffer) {u8 cur_engine, omit_engine =0 , org_engine;struct qed_hwfn *p_hwfn = - &cdev->hwfns[cdev->dbg_params.engine_for_debug]; + &cdev->hwfns[cdev->engine_for_debug];struct dbg_tools_data *dev_data = &p_hwfn->dbg_info;int grc_params[MAX_DBG_GRC_PARAMS ], i;u32 offset =0 , feature_size;
@@ -8000,7 +8000,7 @@ int qed_dbg_all_data(struct qed_dev *cdev, void *buffer)int qed_dbg_all_data_size (struct qed_dev *cdev) {struct qed_hwfn *p_hwfn = - &cdev->hwfns[cdev->dbg_params.engine_for_debug]; + &cdev->hwfns[cdev->engine_for_debug];u32 regs_len =0 , image_len =0 , ilt_len =0 , total_ilt_len =0 ;u8 cur_engine, org_engine;
@@ -8059,9 +8059,9 @@ int qed_dbg_feature(struct qed_dev *cdev, void *buffer, enum qed_dbg_features feature, u32 *num_dumped_bytes) { struct qed_hwfn *p_hwfn = - &cdev->hwfns[cdev->dbg_params.engine_for_debug]; + &cdev->hwfns[cdev->engine_for_debug];struct qed_dbg_feature *qed_feature = - &cdev->dbg_params.features[feature]; + &cdev->dbg_features[feature];enum dbg_status dbg_rc;struct qed_ptt *p_ptt;int rc =0 ;
@@ -8084,7 +8084,7 @@ int qed_dbg_feature(struct qed_dev *cdev, void *buffer,DP_VERBOSE (cdev,QED_MSG_DEBUG ,"copying debugfs feature to external buffer\n" );memcpy (buffer, qed_feature->dump_buf, qed_feature->buf_size); - *num_dumped_bytes = cdev->dbg_params.features[feature].dumped_dwords * + *num_dumped_bytes = cdev->dbg_features[feature].dumped_dwords *4 ; out:
@@ -8095,7 +8095,7 @@ int qed_dbg_feature(struct qed_dev *cdev, void *buffer,int qed_dbg_feature_size (struct qed_dev *cdev,enum qed_dbg_features feature) {struct qed_hwfn *p_hwfn = - &cdev->hwfns[cdev->dbg_params.engine_for_debug]; + &cdev->hwfns[cdev->engine_for_debug];struct qed_dbg_feature *qed_feature = &cdev->dbg_features[feature];struct qed_ptt *p_ptt =qed_ptt_acquire (p_hwfn);u32 buf_size_dwords;
@@ -8120,14 +8120,14 @@ int qed_dbg_feature_size(struct qed_dev *cdev, enum qed_dbg_features feature)u8 qed_get_debug_engine (struct qed_dev *cdev) { -return cdev->dbg_params.engine_for_debug; +return cdev->engine_for_debug; }void qed_set_debug_engine (struct qed_dev *cdev,int engine_number) {DP_VERBOSE (cdev,QED_MSG_DEBUG ,"set debug engine to %d\n" , engine_number); - cdev->dbg_params.engine_for_debug = engine_number; + cdev->engine_for_debug = engine_number; }void qed_dbg_pf_init (struct qed_dev *cdev)
@@ -8146,7 +8146,7 @@ void qed_dbg_pf_init(struct qed_dev *cdev) }/* Set the hwfn to be 0 as default */ - cdev->dbg_params.engine_for_debug =0 ; + cdev->engine_for_debug =0 ; }void qed_dbg_pf_exit (struct qed_dev *cdev)
Signature · 1 line
2.17.1
PATCH v2 6/11 net: qed: attention clearing properties
In reply to Igor RusskikhOn different hardware events we have to respond differently, on some of hardware indications hw attention (error condition) should be cleared by the driver to continue normal functioning.
Here we introduce attention clear flags, and put them on some important events (in aeu_descs).
--- drivers/net/ethernet/qlogic/qed/qed.h | 3 +++ drivers/net/ethernet/qlogic/qed/qed_int.c | 22 ++++++++++++++++---- drivers/net/ethernet/qlogic/qed/qed_int.h | 11 ++++++++++ drivers/net/ethernet/qlogic/qed/qed_main.c | 7 ++++++- drivers/net/ethernet/qlogic/qede/qede_main.c | 6 ++++++ include/linux/qed/qed_if.h | 9 ++++++++ 6 files changed, 53 insertions(+), 5 deletions(-)
drivers/net/ethernet/qlogic/qed/qed.h +3 -0
@@ -838,6 +838,9 @@ struct qed_dev {/* Recovery */ bool recov_in_prog; +/* Indicates whether should prevent attentions from being reasserted */ +bool attn_clr_en; +/* LLH info */ u8 ppfid_bitmap;struct qed_llh_info *p_llh_info;
drivers/net/ethernet/qlogic/qed/qed_int.c +18 -4
@@ -96,6 +96,7 @@ struct aeu_invert_reg_bit {#define ATTENTION_BB (value) (value << ATTENTION_BB_SHIFT)#define ATTENTION_BB_DIFFERENT BIT(23) +#define ATTENTION_CLEAR_ENABLE BIT(28)unsigned int flags;/* Callback to call if attention will be triggered */
@@ -371,6 +372,13 @@ static int qed_fw_assertion(struct qed_hwfn *p_hwfn)return -EINVAL ; } +static int qed_general_attention_35 (struct qed_hwfn *p_hwfn) +{ +DP_INFO (p_hwfn,"General attention 35!\n" ); + +return 0 ; +} +#define QED_DORQ_ATTENTION_REASON_MASK (0xfffff)#define QED_DORQ_ATTENTION_OPAQUE_MASK (0xffff)#define QED_DORQ_ATTENTION_OPAQUE_SHIFT (0x0)
@@ -613,14 +621,15 @@ static struct aeu_invert_reg aeu_descs[NUM_ATTN_REGS] = { { {/* After Invert 4 */ - {"General Attention 32" ,ATTENTION_SINGLE , - qed_fw_assertion, + {"General Attention 32" ,ATTENTION_SINGLE | +ATTENTION_CLEAR_ENABLE , qed_fw_assertion, MAX_BLOCK_ID}, {"General Attention %d", (2 <<ATTENTION_LENGTH_SHIFT ) | (33 << ATTENTION_OFFSET_SHIFT), NULL, MAX_BLOCK_ID}, - {"General Attention 35", ATTENTION_SINGLE, - NULL, MAX_BLOCK_ID}, + {"General Attention 35", ATTENTION_SINGLE | +ATTENTION_CLEAR_ENABLE , qed_general_attention_35, + MAX_BLOCK_ID}, {"NWS Parity",ATTENTION_PAR |ATTENTION_BB_DIFFERENT | ATTENTION_BB(AEU_INVERT_REG_SPECIAL_CNIG_0),
@@ -2361,6 +2370,11 @@ void qed_int_disable_post_isr_release(struct qed_dev *cdev) cdev->hwfns[i].b_int_requested = false; } +void qed_int_attn_clr_enable (struct qed_dev *cdev,bool clr_enable) +{ + cdev->attn_clr_en = clr_enable; +} +int qed_int_set_timer_res (struct qed_hwfn *p_hwfn,struct qed_ptt *p_ptt,u8 timer_res,u16 sb_id,bool tx) {
drivers/net/ethernet/qlogic/qed/qed_int.h +11 -0
@@ -190,6 +190,17 @@ void qed_int_get_num_sbs(struct qed_hwfn *p_hwfn,*/ void qed_int_disable_post_isr_release (struct qed_dev *cdev); +/** +* @brief qed_int_attn_clr_enable - sets whether the general behavior is +* preventing attentions from being reasserted, or following the +* attributes of the specific attention. +* +* @param cdev +* @param clr_enable +* +*/ +void qed_int_attn_clr_enable (struct qed_dev *cdev,bool clr_enable); +/** * @brief - Doorbell Recovery handler. * Run doorbell recovery in case of PF overflow (and flush DORQ if
drivers/net/ethernet/qlogic/qed/qed_main.c +6 -1
@@ -2491,10 +2491,14 @@ void qed_hw_error_occurred(struct qed_hwfn *p_hwfn,DP_NOTICE (p_hwfn,"HW error occurred [%s]\n" , err_str); -/* Call the HW error handler of the protocol driver +/* Call the HW error handler of the protocol driver. +* If it is not available - perform a minimal handling of preventing +* HW attentions from being reasserted. */ if (ops && ops->schedule_hw_err_handler) ops->schedule_hw_err_handler (cookie, err_type); +else +qed_int_attn_clr_enable (p_hwfn->cdev, true); } static int qed_set_coalesce(struct qed_dev *cdev, u16 rx_coal, u16 tx_coal,
@@ -2718,6 +2722,7 @@ const struct qed_common_ops qed_common_ops_pass = { .set_led = &qed_set_led, .recovery_process = &qed_recovery_process, .recovery_prolog = &qed_recovery_prolog, + .attn_clr_enable = &qed_int_attn_clr_enable, .update_drv_state = &qed_update_drv_state, .update_mac = &qed_update_mac, .update_mtu = &qed_update_mtu,
drivers/net/ethernet/qlogic/qede/qede_main.c +6 -0
@@ -2516,6 +2516,8 @@ static void qede_recovery_handler(struct qede_dev *edev)static void qede_atomic_hw_err_handler (struct qede_dev *edev) { +struct qed_dev *cdev = edev->cdev; +DP_NOTICE (edev,"Generic non-sleepable HW error handling started - err_flags 0x%lx\n" , edev->err_flags);
@@ -2523,6 +2525,10 @@ static void qede_atomic_hw_err_handler(struct qede_dev *edev)/* Get a call trace of the flow that led to the error */ WARN_ON (test_bit (QEDE_ERR_WARN , &edev->err_flags)); +/* Prevent HW attentions from being reasserted */ +if (test_bit (QEDE_ERR_ATTN_CLR_EN , &edev->err_flags)) + edev->ops->common->attn_clr_enable (cdev, true); +DP_NOTICE (edev,"Generic non-sleepable HW error handling is done\n" ); }
include/linux/qed/qed_if.h +9 -0
@@ -1046,6 +1046,15 @@ struct qed_common_ops {*/ int (*set_led)(struct qed_dev *cdev,enum qed_led_mode mode); + +/** +* @brief attn_clr_enable - Prevent attentions from being reasserted +* +* @param cdev +* @param clr_enable +*/ +void (*attn_clr_enable)(struct qed_dev *cdev,bool clr_enable); +/** * @brief db_recovery_add - add doorbell information to the doorbell * recovery mechanism.
Signature · 1 line
2.17.1
PATCH v2 7/11 net: qede: optional hw recovery procedure
In reply to Igor RusskikhDriver has an ability to initiate a recovery process as a reaction to detected errors. But the codepath (recovery_process) was disabled and never active.
Here we add ethtool private flag to allow user have the recovery procedure activated.
We still do not enable this by default though, since in some configurations this is not desirable. E.g. this may impact other PFs/VFs.
--- .../net/ethernet/qlogic/qede/qede_ethtool.c | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+)
drivers/net/ethernet/qlogic/qede/qede_ethtool.c +24 -0
@@ -190,12 +190,14 @@ static const struct {enum {QEDE_PRI_FLAG_CMT ,QEDE_PRI_FLAG_SMART_AN_SUPPORT ,/* MFW supports SmartAN */ +QEDE_PRI_FLAG_RECOVER_ON_ERROR ,QEDE_PRI_FLAG_LEN , };static const char qede_private_arr[QEDE_PRI_FLAG_LEN ][ETH_GSTRING_LEN ] = {"Coupled-Function" ,"SmartAN capable" , +"Recover on error" , };enum qede_ethtool_tests {
@@ -417,9 +419,30 @@ static u32 qede_get_priv_flags(struct net_device *dev)if (edev->dev_info.common.smart_an) flags |=BIT (QEDE_PRI_FLAG_SMART_AN_SUPPORT ); +if (edev->err_flags &BIT (QEDE_ERR_IS_RECOVERABLE )) + flags |=BIT (QEDE_PRI_FLAG_RECOVER_ON_ERROR ); +return flags; } +static int qede_set_priv_flags (struct net_device *dev,u32 flags) +{ +struct qede_dev *edev =netdev_priv (dev); +u32 cflags =qede_get_priv_flags (dev); +u32 dflags = flags ^ cflags; + +/* can only change RECOVER_ON_ERROR flag */ +if (dflags & ~BIT (QEDE_PRI_FLAG_RECOVER_ON_ERROR )) +return -EINVAL ; + +if (flags &BIT (QEDE_PRI_FLAG_RECOVER_ON_ERROR )) +set_bit (QEDE_ERR_IS_RECOVERABLE , &edev->err_flags); +else +clear_bit (QEDE_ERR_IS_RECOVERABLE , &edev->err_flags); + +return 0 ; +} +struct qede_link_mode_mapping {u32 qed_link_mode;u32 ethtool_link_mode;
@@ -2098,6 +2121,7 @@ static const struct ethtool_ops qede_ethtool_ops = { .set_phys_id = qede_set_phys_id, .get_ethtool_stats = qede_get_ethtool_stats, .get_priv_flags = qede_get_priv_flags, + .set_priv_flags = qede_set_priv_flags, .get_sset_count = qede_get_sset_count, .get_rxnfc = qede_get_rxnfc, .set_rxnfc = qede_set_rxnfc,
Signature · 1 line
2.17.1
PATCH v2 8/11 net: qede: Implement ndo_tx_timeout
In reply to Igor RusskikhFrom: Denis Bolotin <[email protected]>
Upon tx timeout detection we do disable carrier and print TX queue info on TX timeout. We then raise hw error condition and trigger service task to handle this.
This handler will capture extra debug info and then optionally trigger recovery procedure to try restore function.
--- drivers/net/ethernet/qlogic/qede/qede.h | 1 - drivers/net/ethernet/qlogic/qede/qede_main.c | 46 ++++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-)
drivers/net/ethernet/qlogic/qede/qede.h +0 -1
@@ -533,7 +533,6 @@ u16 qede_select_queue(struct net_device *dev, struct sk_buff *skb,netdev_features_t qede_features_check (struct sk_buff *skb,struct net_device *dev,netdev_features_t features); -void qede_tx_log_print (struct qede_dev *edev,struct qede_fastpath *fp);int qede_alloc_rx_buffer (struct qede_rx_queue *rxq,bool allow_lazy);int qede_free_tx_pkt (struct qede_dev *edev,struct qede_tx_queue *txq,int *len);
drivers/net/ethernet/qlogic/qede/qede_main.c +46 -0
58 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v2 9/11 net: qed: introduce critical fan failure handler
In reply to Igor RusskikhFan failure is sent by firmware, driver reacts on this error with newly introduced notification path. It will collect dump and shut down the device to prevent physical breakage
--- drivers/net/ethernet/qlogic/qed/qed_hsi.h | 2 +- drivers/net/ethernet/qlogic/qed/qed_mcp.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
drivers/net/ethernet/qlogic/qed/qed_hsi.h +1 -1
@@ -12761,7 +12761,7 @@ enum MFW_DRV_MSG_TYPE {MFW_DRV_MSG_GET_FCOE_STATS ,MFW_DRV_MSG_GET_ISCSI_STATS ,MFW_DRV_MSG_GET_RDMA_STATS , -MFW_DRV_MSG_BW_UPDATE10 , +MFW_DRV_MSG_FAILURE_DETECTED ,MFW_DRV_MSG_TRANSCEIVER_STATE_CHANGE ,MFW_DRV_MSG_BW_UPDATE11 , MFW_DRV_MSG_RESERVED,
drivers/net/ethernet/qlogic/qed/qed_mcp.c +14 -0
@@ -1706,6 +1706,17 @@ static void qed_mcp_update_stag(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt) &resp, ¶m); } +static void qed_mcp_handle_fan_failure (struct qed_hwfn *p_hwfn, +struct qed_ptt *p_ptt) +{ +/* A single notification should be sent to upper driver in CMT mode */ +if (p_hwfn !=QED_LEADING_HWFN (p_hwfn->cdev)) +return ; + +qed_hw_err_notify (p_hwfn, p_ptt,QED_HW_ERR_FAN_FAIL , +"Fan failure was detected on the network interface card and it's going to be shut down.\n" ); +} +void qed_mcp_read_ufp_config (struct qed_hwfn *p_hwfn,struct qed_ptt *p_ptt) {struct public_func shmem_info;
@@ -1852,6 +1863,9 @@ int qed_mcp_handle_events(struct qed_hwfn *p_hwfn,case MFW_DRV_MSG_S_TAG_UPDATE :qed_mcp_update_stag (p_hwfn, p_ptt);break ; +case MFW_DRV_MSG_FAILURE_DETECTED : +qed_mcp_handle_fan_failure (p_hwfn, p_ptt); +break ;case MFW_DRV_MSG_GET_TLV_REQ :qed_mfw_tlv_req (p_hwfn);break ;
Signature · 1 line
2.17.1
PATCH v2 10/11 net: qed: introduce critical hardware error handler
In reply to Igor RusskikhMCP may signal driver about generic critical failure. Driver has to collect mdump information (get_retain), it pushes that to logs and triggers generic notification on "hardware attention" event.
--- drivers/net/ethernet/qlogic/qed/qed_hsi.h | 28 +++++- drivers/net/ethernet/qlogic/qed/qed_mcp.c | 113 ++++++++++++++++++++++ drivers/net/ethernet/qlogic/qed/qed_mcp.h | 13 +++ 3 files changed, 153 insertions(+), 1 deletion(-)
drivers/net/ethernet/qlogic/qed/qed_hsi.h +27 -1
@@ -12400,6 +12400,13 @@ struct load_rsp_stc {#define LOAD_RSP_FLAGS0_DRV_EXISTS (0x1 << 0) }; +struct mdump_retain_data_stc { +u32 valid; +u32 epoch; +u32 pf; +u32 status; +}; +union drv_union_data {u32 ver_str[MCP_DRV_VER_STR_SIZE_DWORD ];struct mcp_mac wol_mac;
@@ -12488,6 +12495,8 @@ struct public_drv_mb {#define DRV_MSG_CODE_BIST_TEST 0x001e0000#define DRV_MSG_CODE_SET_LED_MODE 0x00200000#define DRV_MSG_CODE_RESOURCE_CMD 0x00230000 +/* Send crash dump commands with param[3:0] - opcode */ +#define DRV_MSG_CODE_MDUMP_CMD 0x00250000#define DRV_MSG_CODE_GET_TLV_DONE 0x002f0000#define DRV_MSG_CODE_GET_ENGINE_CONFIG 0x00370000#define DRV_MSG_CODE_GET_PPFID_BITMAP 0x43000000
@@ -12519,6 +12528,21 @@ struct public_drv_mb {#define RESOURCE_DUMP 0 +/* DRV_MSG_CODE_MDUMP_CMD parameters */ +#define MDUMP_DRV_PARAM_OPCODE_MASK 0x0000000f +#define DRV_MSG_CODE_MDUMP_ACK 0x01 +#define DRV_MSG_CODE_MDUMP_SET_VALUES 0x02 +#define DRV_MSG_CODE_MDUMP_TRIGGER 0x03 +#define DRV_MSG_CODE_MDUMP_GET_CONFIG 0x04 +#define DRV_MSG_CODE_MDUMP_SET_ENABLE 0x05 +#define DRV_MSG_CODE_MDUMP_CLEAR_LOGS 0x06 +#define DRV_MSG_CODE_MDUMP_GET_RETAIN 0x07 +#define DRV_MSG_CODE_MDUMP_CLR_RETAIN 0x08 + +#define DRV_MSG_CODE_HW_DUMP_TRIGGER 0x0a +#define DRV_MSG_CODE_MDUMP_GEN_MDUMP2 0x0b +#define DRV_MSG_CODE_MDUMP_FREE_MDUMP2 0x0c +#define DRV_MSG_CODE_GET_PF_RDMA_PROTOCOL 0x002b0000#define DRV_MSG_CODE_OS_WOL 0x002e0000
@@ -12697,6 +12721,8 @@ struct public_drv_mb {#define FW_MSG_CODE_DEBUG_NOT_ENABLED 0xb00a0000#define FW_MSG_CODE_DEBUG_DATA_SEND_OK 0xb00b0000 +#define FW_MSG_CODE_MDUMP_INVALID_CMD 0x00030000 +u32 fw_mb_param;#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_MASK 0xFFFF0000#define FW_MB_PARAM_RESOURCE_ALLOC_VERSION_MAJOR_SHIFT 16
@@ -12763,7 +12789,7 @@ enum MFW_DRV_MSG_TYPE {MFW_DRV_MSG_GET_RDMA_STATS ,MFW_DRV_MSG_FAILURE_DETECTED ,MFW_DRV_MSG_TRANSCEIVER_STATE_CHANGE , -MFW_DRV_MSG_BW_UPDATE11 , +MFW_DRV_MSG_CRITICAL_ERROR_OCCURRED ,MFW_DRV_MSG_RESERVED ,MFW_DRV_MSG_GET_TLV_REQ , MFW_DRV_MSG_OEM_CFG_UPDATE,
drivers/net/ethernet/qlogic/qed/qed_mcp.c +113 -0
125 lines not on this page · raw message
drivers/net/ethernet/qlogic/qed/qed_mcp.h +13 -0
@@ -1016,6 +1016,19 @@ int __qed_configure_pf_min_bandwidth(struct qed_hwfn *p_hwfn,int qed_mcp_mask_parities (struct qed_hwfn *p_hwfn,struct qed_ptt *p_ptt,u32 mask_parities); +/* @brief - Gets the mdump retained data from the MFW. +* +* @param p_hwfn +* @param p_ptt +* @param p_mdump_retain +* +* @param return 0 upon success. +*/ +int +qed_mcp_mdump_get_retain (struct qed_hwfn *p_hwfn, +struct qed_ptt *p_ptt, +struct mdump_retain_data_stc *p_mdump_retain); +/** * @brief - Sets the MFW's max value for the given resource *
Signature · 1 line
2.17.1
PATCH v2 11/11 net: qed: fix bad formatting
In reply to Igor RusskikhOn some adjacent code, fix bad code formatting
--- include/linux/qed/qed_if.h | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
include/linux/qed/qed_if.h +2 -3
@@ -821,12 +821,11 @@ enum qed_nvm_flash_cmd {struct qed_common_cb_ops {void (*arfs_filter_op)(void *dev,void *fltr,u8 fw_rc); -void (*link_update)(void *dev, -struct qed_link_output *link); +void (*link_update)(void *dev,struct qed_link_output *link);void (*schedule_recovery_handler)(void *dev);void (*schedule_hw_err_handler)(void *dev,enum qed_hw_err_type err_type); -void (*dcbx_aen)(void *dev,struct qed_dcbx_get *get,u32 mib_type); +void (*dcbx_aen)(void *dev,struct qed_dcbx_get *get,u32 mib_type);void (*get_generic_tlv_data)(void *dev,struct qed_generic_tlvs *data);void (*get_protocol_tlv_data)(void *dev,void *data); };
Signature · 1 line
2.17.1
Jakub Kicinski
maintainerQuoted 13 lines · On Thu, 14 May 2020 12:57:16 +0300 Igor Russkikh wrote:
FastLinQ devices as a complex systems may observe various hardware level error conditions, both severe and recoverable.
Driver is able to detect and report this, but so far it only did trace/dmesg based reporting.
Here we implement an extended hw error detection, service task handler captures a dump for the later analysis.
I also resubmit a patch from Denis Bolotin on tx timeout handler, addressing David's comment regarding recovery procedure as an extra reaction on this event.
Quoted 8 lines
v2:
Removing the patch with ethtool dump and udev magic. Its quite isolated, I'm working on devlink based logic for this separately.
v1:
https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
I'm not 100% happy that the debug data gets reported to the management FW before the devlink health code is in place. For the Linux community, I think, having standard Linux interfaces implemented first is the priority.
Quoted 5 lines
I'm not 100% happy that the debug data gets reported to the management FW before the devlink health code is in place. For the Linux community, I think, having standard Linux interfaces implemented first is the priority.
Hi Jakub,
Thanks for the comment. I feel these two are a bit separate. We try to push important messages to MFW, not debug data. And all these messages are as well perfectly being reported on device level error printouts, they are not kind of lost.
And for devlink, we anyway will need all the above infrastructure, to eventually implement devlink dumps and other features.
Or, may be I didn't get your point?
Thanks,
Igor
Jakub Kicinski
maintainerQuoted 12 lines · On Thu, 14 May 2020 22:40:12 +0300 Igor Russkikh wrote:
Quoted 4 lines
I'm not 100% happy that the debug data gets reported to the management FW before the devlink health code is in place. For the Linux community, I think, having standard Linux interfaces implemented first is the priority.
Hi Jakub,
Thanks for the comment. I feel these two are a bit separate. We try to push important messages to MFW, not debug data. And all these messages are as well perfectly being reported on device level error printouts, they are not kind of lost.
Quoted 4 lines
And for devlink, we anyway will need all the above infrastructure, to eventually implement devlink dumps and other features.
Or, may be I didn't get your point?
That's fine, I'm just saying - I hope the devlink part doesn't take too long to implement :)
Quoted 9 lines · Jakub Kicinski
That's fine, I'm just saying - I hope the devlink part doesn't take too long to implement :)
I'm only applying this series because I trust that you will actually do the
devlink work, and you will have it done and submitted in a reasonable amount
of ti me.
I see. Thanks Jakub, David. Doing hard already on devlink side.
Quoted 4 lines
Also, patch #4 had trailing empty lines added to a file, which is warned about by 'git' when I apply your patches. I fixed it up, but this is the kind of thing you should have sorted out before you submit changes to the list.
Sorry for that miss, will do in future.
Thanks
Igor
David Miller
maintainerFrom: Igor Russkikh <[email protected]>
Date: Thu, 14 May 2020 12:57:16 +0300
Quoted 13 lines
FastLinQ devices as a complex systems may observe various hardware level error conditions, both severe and recoverable.
Driver is able to detect and report this, but so far it only did trace/dmesg based reporting.
Here we implement an extended hw error detection, service task handler captures a dump for the later analysis.
I also resubmit a patch from Denis Bolotin on tx timeout handler, addressing David's comment regarding recovery procedure as an extra reaction on this event.
Quoted 8 lines
v2:
Removing the patch with ethtool dump and udev magic. Its quite isolated, I'm working on devlink based logic for this separately.
v1:
https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
I'm only applying this series because I trust that you will actually do the devlink work, and you will have it done and submitted in a reasonable amount of ti me.
Also, patch #4 had trailing empty lines added to a file, which is warned about by 'git' when I apply your patches. I fixed it up, but this is the kind of thing you should have sorted out before you submit changes to the list.
Thank you.
David Miller
maintainerFrom: David Miller <[email protected]>
Date: Thu, 14 May 2020 13:01:59 -0700 (PDT)
Quoted 25 lines
From: Igor Russkikh <[email protected]>
Date: Thu, 14 May 2020 12:57:16 +0300Quoted 13 lines
FastLinQ devices as a complex systems may observe various hardware level error conditions, both severe and recoverable.
Driver is able to detect and report this, but so far it only did trace/dmesg based reporting.
Here we implement an extended hw error detection, service task handler captures a dump for the later analysis.
I also resubmit a patch from Denis Bolotin on tx timeout handler, addressing David's comment regarding recovery procedure as an extra reaction on this event.
Quoted 8 lines
v2:
Removing the patch with ethtool dump and udev magic. Its quite isolated, I'm working on devlink based logic for this separately.
v1:
https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
Quoted 8 lines
I'm only applying this series because I trust that you will actually do the devlink work, and you will have it done and submitted in a reasonable amount of ti me.
Also, patch #4 had trailing empty lines added to a file, which is warned about by 'git' when I apply your patches. I fixed it up, but this is the kind of thing you should have sorted out before you submit changes to the list.
Actually, I had to revert, please fix these warnings (with gcc-10.1.1 on Fedora)_:
drivers/net/ethernet/qlogic/qed/qed_dev.c: In function ‘qed_llh_add_mac_filter’:./include/linux/printk.h:303:2: warning: ‘abs_ppfid’ may be used uninitialized in this function [-Wmaybe-uninitialized]
303 | printk(KERN_NOTICE pr_fmt(fmt), ##__VA_ARGS__)
| ^~~~~~
drivers/net/ethernet/qlogic/qed/qed_dev.c:983:17: note: ‘abs_ppfid’ was declared here
983 | u8 filter_idx, abs_ppfid;
| ^~~~~~~~~David Miller
maintainerFrom: David Miller <[email protected]>
Date: Thu, 14 May 2020 13:09:52 -0700 (PDT)
Quoted 9 lines
Actually, I had to revert, please fix these warnings (with gcc-10.1.1 on Fedora)_:
drivers/net/ethernet/qlogic/qed/qed_dev.c: In function ‘qed_llh_add_mac_filter’:./include/linux/printk.h:303:2: warning: ‘abs_ppfid’ may be used uninitialized in this function [-Wmaybe-uninitialized]303 | printk(KERN_NOTICE pr_fmt(fmt), ##__VA_ARGS__) | ^~~~~~ drivers/net/ethernet/qlogic/qed/qed_dev.c:983:17: note: ‘abs_ppfid’ was declared here 983 | u8 filter_idx, abs_ppfid; | ^~~~~~~~~
Hmm, this seems to actually be an existing warning, sorry.
I'll reapply this.