ibmvnic: Allow extra failures before disabling
PATCH 1/4 ibmvnic: Allow extra failures before disabling
If auto-priority-failover (APF) is enabled and there are at least two backing devices of different priorities, some resets like fail-over, change-param etc can cause at least two back to back failovers. (Failover from high priority backing device to lower priority one and then back to the higher priority one if that is still functional).
Depending on the timimg of the two failovers it is possible to trigger a "hard" reset and for the hard reset to fail due to failovers. When this occurs, the driver assumes that the network is unstable and disables the VNIC for a 60-second "settling time". This in turn can cause the ethtool command to fail with "No such device" while the vnic automatically recovers a little while later.
Given that it's possible to have two back to back failures, allow for extra failures before disabling the vnic for the settling time.
--- drivers/net/ethernet/ibm/ibmvnic.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)
drivers/net/ethernet/ibm/ibmvnic.c +17 -4
@@ -2598,6 +2598,7 @@ static void __ibmvnic_reset(struct work_struct *work)struct ibmvnic_rwi *rwi;unsigned long flags;u32 reset_state; +int num_fails =0 ;int rc =0 ; adapter = container_of(work, struct ibmvnic_adapter, ibmvnic_reset);
@@ -2651,11 +2652,23 @@ static void __ibmvnic_reset(struct work_struct *work) rc =do_hard_reset (adapter, rwi, reset_state);rtnl_unlock (); } -if (rc) { -/* give backing device time to settle down */ +if (rc) + num_fails++; +else + num_fails =0 ; + +/* If auto-priority-failover is enabled we can get +* back to back failovers during resets, resulting +* in at least two failed resets (from high-priority +* backing device to low-priority one and then back) +* If resets continue to fail beyond that, give the +* adapter some time to settle down before retrying. +*/ +if (num_fails >=3 ) {netdev_dbg (adapter->netdev, -"[S:%s] Hard reset failed, waiting 60 secs\n" , -adapter_state_to_string (adapter->state)); +"[S:%s] Hard reset failed %d times, waiting 60 secs\n" , +adapter_state_to_string (adapter->state), + num_fails);set_current_state (TASK_UNINTERRUPTIBLE );schedule_timeout (60 *HZ ); }
Signature · 1 line
2.27.0
PATCH 2/4 ibmvnic: init ->running_cap_crqs early
We use ->running_cap_crqs to determine when the ibmvnic_tasklet() should send out the next protocol message type. i.e when we get back responses to all our QUERY_CAPABILITY CRQs we send out REQUEST_CAPABILITY crqs. Similiary, when we get responses to all the REQUEST_CAPABILITY crqs, we send out the QUERY_IP_OFFLOAD CRQ.
We currently increment ->running_cap_crqs as we send out each CRQ and have the ibmvnic_tasklet() send out the next message type, when this running_cap_crqs count drops to 0.
This assumes that all the CRQs of the current type were sent out before the count drops to 0. However it is possible that we send out say 6 CRQs, get preempted and receive all the 6 responses before we send out the remaining CRQs. This can result in ->running_cap_crqs count dropping to zero before all messages of the current type were sent and we end up sending the next protocol message too early.
Instead initialize the ->running_cap_crqs upfront so the tasklet will only send the next protocol message after all responses are received.
Use the cap_reqs local variable to also detect any discrepancy (either now or in future) in the number of capability requests we actually send.
Currently only send_query_cap() is affected by this behavior (of sending next message early) since it is called from the worker thread (during reset) and from application thread (during ->ndo_open()) and they can be preempted. send_request_cap() is only called from the tasklet which processes CRQ responses sequentially, is not be affected. But to maintain the existing symmtery with send_query_capability() we update send_request_capability() also.
--- drivers/net/ethernet/ibm/ibmvnic.c | 106 +++++++++++++++++++---------- 1 file changed, 71 insertions(+), 35 deletions(-)
drivers/net/ethernet/ibm/ibmvnic.c +71 -35
@@ -3849,11 +3849,25 @@ static void send_request_cap(struct ibmvnic_adapter *adapter, int retry)struct device *dev = &adapter->vdev->dev;union ibmvnic_crq crq;int max_entries; +int cap_reqs; + +/* We send out 6 or 7 REQUEST_CAPABILITY CRQs below (depending on +* the PROMISC flag). Initialize this count upfront. When the tasklet +* receives a response to all of these, it will send the next protocol +* message (QUERY_IP_OFFLOAD). +*/ +if (!(adapter->netdev->flags &IFF_PROMISC ) || + adapter->promisc_supported) + cap_reqs =7 ; +else + cap_reqs =6 ;if (!retry) {/* Sub-CRQ entries are 32 byte long */ int entries_page =4 *PAGE_SIZE / (sizeof (u64) *4 ); +atomic_set (&adapter->running_cap_crqs, cap_reqs); +if (adapter->min_tx_entries_per_subcrq > entries_page || adapter->min_rx_add_entries_per_subcrq > entries_page) {dev_err (dev,"Fatal, invalid entries per sub-crq\n" );
@@ -3914,44 +3928,45 @@ static void send_request_cap(struct ibmvnic_adapter *adapter, int retry) adapter->opt_rx_comp_queues; adapter->req_rx_add_queues = adapter->max_rx_add_queues; + }else { +atomic_add (cap_reqs, &adapter->running_cap_crqs); } -memset (&crq,0 ,sizeof (crq)); crq.request_capability.first =IBMVNIC_CRQ_CMD ; crq.request_capability.cmd =REQUEST_CAPABILITY ; crq.request_capability.capability =cpu_to_be16 (REQ_TX_QUEUES ); crq.request_capability.number =cpu_to_be64 (adapter->req_tx_queues); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_RX_QUEUES ); crq.request_capability.number =cpu_to_be64 (adapter->req_rx_queues); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_RX_ADD_QUEUES ); crq.request_capability.number =cpu_to_be64 (adapter->req_rx_add_queues); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_TX_ENTRIES_PER_SUBCRQ ); crq.request_capability.number =cpu_to_be64 (adapter->req_tx_entries_per_subcrq); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_RX_ADD_ENTRIES_PER_SUBCRQ ); crq.request_capability.number =cpu_to_be64 (adapter->req_rx_add_entries_per_subcrq); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_MTU ); crq.request_capability.number =cpu_to_be64 (adapter->req_mtu); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq);if (adapter->netdev->flags &IFF_PROMISC ) {
@@ -3959,16 +3974,21 @@ static void send_request_cap(struct ibmvnic_adapter *adapter, int retry) crq.request_capability.capability =cpu_to_be16 (PROMISC_REQUESTED ); crq.request_capability.number =cpu_to_be64 (1 ); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); } }else { crq.request_capability.capability =cpu_to_be16 (PROMISC_REQUESTED ); crq.request_capability.number =cpu_to_be64 (0 ); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); } + +/* Keep at end to catch any discrepancy between expected and actual +* CRQs sent. +*/ +WARN_ON (cap_reqs !=0 ); } static int pending_scrq(struct ibmvnic_adapter *adapter,
@@ -4362,118 +4382,132 @@ static void send_query_map(struct ibmvnic_adapter *adapter)static void send_query_cap (struct ibmvnic_adapter *adapter) {union ibmvnic_crq crq; +int cap_reqs; + +/* We send out 25 QUERY_CAPABILITY CRQs below. Initialize this count +* upfront. When the tasklet receives a response to all of these, it +* can send out the next protocol messaage (REQUEST_CAPABILITY). +*/ + cap_reqs =25 ; + +atomic_set (&adapter->running_cap_crqs, cap_reqs); -atomic_set (&adapter->running_cap_crqs,0 );memset (&crq,0 ,sizeof (crq)); crq.query_capability.first =IBMVNIC_CRQ_CMD ; crq.query_capability.cmd =QUERY_CAPABILITY ; crq.query_capability.capability =cpu_to_be16 (MIN_TX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_RX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_RX_ADD_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_TX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_RX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_RX_ADD_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_TX_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_RX_ADD_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_TX_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_RX_ADD_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (TCP_IP_OFFLOAD ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (PROMISC_SUPPORTED ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_MTU ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_MTU ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_MULTICAST_FILTERS ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (VLAN_HEADER_INSERTION ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (RX_VLAN_HEADER_INSERTION ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_TX_SG_ENTRIES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (RX_SG_SUPPORTED ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_TX_COMP_SUB_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_RX_COMP_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_RX_BUFADD_Q_PER_RX_COMP_Q ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_TX_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_RXBA_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (TX_RX_DESC_REQ ); -atomic_inc (&adapter->running_cap_crqs); +ibmvnic_send_crq (adapter, &crq); + cap_reqs--; + +/* Keep at end to catch any discrepancy between expected and actual +* CRQs sent. +*/ +WARN_ON (cap_reqs !=0 ); }static void send_query_ip_offload (struct ibmvnic_adapter *adapter)
@@ -4777,6 +4811,8 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq,char *name;atomic_dec (&adapter->running_cap_crqs); +netdev_dbg (adapter->netdev,"Outstanding request-caps: %d\n" , +atomic_read (&adapter->running_cap_crqs));switch (be16_to_cpu (crq->request_capability_rsp.capability)) {case REQ_TX_QUEUES : req_value = &adapter->req_tx_queues;
Signature · 1 line
2.27.0
PATCH 3/4 ibmvnic: don't spin in tasklet
In reply to Sukadev Bhattiproluibmvnic_tasklet() continuously spins waiting for responses to all capability requests. It does this to avoid encountering an error during initialization of the vnic. However if there is a bug in the VIOS and we do not receive a response to one or more queries the tasklet ends up spinning continuously leading to hard lock ups.
If we fail to receive a message from the VIOS it is reasonable to timeout the login attempt rather than spin indefinitely in the tasklet.
--- drivers/net/ethernet/ibm/ibmvnic.c | 6 ------ 1 file changed, 6 deletions(-)
drivers/net/ethernet/ibm/ibmvnic.c +0 -6
@@ -5491,12 +5491,6 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)ibmvnic_handle_crq (crq, adapter); crq->generic.first =0 ; } - -/* remain in tasklet until all -* capabilities responses are received -*/ -if (!adapter->wait_capability) - done = true; }/* if capabilities CRQ's were sent in this tasklet, the following * tasklet must wait until all responses are received
Signature · 1 line
2.27.0
PATCH 4/4 ibmvnic: remove unused ->wait_capability
In reply to Sukadev BhattiproluWith previous bug fix, ->wait_capability flag is no longer needed and can be removed.
--- drivers/net/ethernet/ibm/ibmvnic.c | 38 +++++++++++------------------- drivers/net/ethernet/ibm/ibmvnic.h | 1 - 2 files changed, 14 insertions(+), 25 deletions(-)
drivers/net/ethernet/ibm/ibmvnic.c +14 -24
@@ -4876,10 +4876,8 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq, }/* Done receiving requested capabilities, query IP offload support */ -if (atomic_read (&adapter->running_cap_crqs) ==0 ) { - adapter->wait_capability = false; +if (atomic_read (&adapter->running_cap_crqs) ==0 )send_query_ip_offload (adapter); - } } static int handle_login_rsp(union ibmvnic_crq *login_rsp_crq,
@@ -5177,10 +5175,8 @@ static void handle_query_cap_rsp(union ibmvnic_crq *crq, } out: -if (atomic_read (&adapter->running_cap_crqs) ==0 ) { - adapter->wait_capability = false; +if (atomic_read (&adapter->running_cap_crqs) ==0 )send_request_cap (adapter,0 ); - } }static int send_query_phys_parms (struct ibmvnic_adapter *adapter)
@@ -5476,27 +5472,21 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)struct ibmvnic_crq_queue *queue = &adapter->crq;union ibmvnic_crq *crq;unsigned long flags; -bool done = false;spin_lock_irqsave (&queue->lock, flags); -while (!done) { -/* Pull all the valid messages off the CRQ */ -while ((crq =ibmvnic_next_crq (adapter)) !=NULL ) { -/* This barrier makes sure ibmvnic_next_crq()'s -* crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded -* before ibmvnic_handle_crq()'s -* switch(gen_crq->first) and switch(gen_crq->cmd). -*/ -dma_rmb (); -ibmvnic_handle_crq (crq, adapter); - crq->generic.first =0 ; - } + +/* Pull all the valid messages off the CRQ */ +while ((crq =ibmvnic_next_crq (adapter)) !=NULL ) { +/* This barrier makes sure ibmvnic_next_crq()'s +* crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded +* before ibmvnic_handle_crq()'s +* switch(gen_crq->first) and switch(gen_crq->cmd). +*/ +dma_rmb (); +ibmvnic_handle_crq (crq, adapter); + crq->generic.first =0 ; } -/* if capabilities CRQ's were sent in this tasklet, the following -* tasklet must wait until all responses are received -*/ -if (atomic_read (&adapter->running_cap_crqs) !=0 ) - adapter->wait_capability = true; +spin_unlock_irqrestore (&queue->lock, flags); }
drivers/net/ethernet/ibm/ibmvnic.h +0 -1
@@ -921,7 +921,6 @@ struct ibmvnic_adapter {int login_rsp_buf_sz;atomic_t running_cap_crqs; -bool wait_capability;struct ibmvnic_sub_crq_queue **tx_scrq____cacheline_aligned ;struct ibmvnic_sub_crq_queue **rx_scrq____cacheline_aligned ;
Signature · 1 line
2.27.0
Quoted 13 lines · Sukadev Bhattiprolu
If auto-priority-failover (APF) is enabled and there are at least two backing devices of different priorities, some resets like fail-over, change-param etc can cause at least two back to back failovers. (Failover from high priority backing device to lower priority one and then back to the higher priority one if that is still functional).
Depending on the timimg of the two failovers it is possible to trigger a "hard" reset and for the hard reset to fail due to failovers. When this occurs, the driver assumes that the network is unstable and disables the VNIC for a 60-second "settling time". This in turn can cause the ethtool command to fail with "No such device" while the vnic automatically recovers a little while later.
Quoted 5 lines
Given that it's possible to have two back to back failures, allow for extra failures before disabling the vnic for the settling time.
Fixes f15fde9d47b8 ("ibmvnic: delay next reset if hard reset fails")Signed-off-by Sukadev Bhattiprolu <[email protected]>
Quoted 47 lines
--- drivers/net/ethernet/ibm/ibmvnic.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)drivers/net/ethernet/ibm/ibmvnic.c +1
-0@@ -2598,6 +2598,7 @@ static void __ibmvnic_reset(struct work_struct *work)struct ibmvnic_rwi *rwi;unsigned long flags;u32 reset_state; +int num_fails =0 ;int rc =0 ;drivers/net/ethernet/ibm/ibmvnic.c +16
-4adapter = container_of(work, struct ibmvnic_adapter, ibmvnic_reset);@@ -2651,11 +2652,23 @@ static void __ibmvnic_reset(struct work_struct *work) rc =do_hard_reset (adapter, rwi, reset_state);rtnl_unlock (); } -if (rc) { -/* give backing device time to settle down */ +if (rc) + num_fails++; +else + num_fails =0 ; + +/* If auto-priority-failover is enabled we can get +* back to back failovers during resets, resulting +* in at least two failed resets (from high-priority +* backing device to low-priority one and then back) +* If resets continue to fail beyond that, give the +* adapter some time to settle down before retrying. +*/ +if (num_fails >=3 ) {netdev_dbg (adapter->netdev, -"[S:%s] Hard reset failed, waiting 60 secs\n" , -adapter_state_to_string (adapter->state)); +"[S:%s] Hard reset failed %d times, waiting 60 secs\n" , +adapter_state_to_string (adapter->state), + num_fails);set_current_state (TASK_UNINTERRUPTIBLE );schedule_timeout (60 *HZ );}
Re: [PATCH net 4/4] ibmvnic: remove unused ->wait_capability
In reply to Sukadev BhattiproluQuoted 6 lines · Sukadev Bhattiprolu
With previous bug fix, ->wait_capability flag is no longer needed and can be removed.
Fixes 249168ad07cd ("ibmvnic: Make CRQ interrupt tasklet wait for allcapabilities crqs")
Signed-off-by Sukadev Bhattiprolu <[email protected]>
Quoted 90 lines
--- drivers/net/ethernet/ibm/ibmvnic.c | 38 +++++++++++------------------- drivers/net/ethernet/ibm/ibmvnic.h | 1 - 2 files changed, 14 insertions(+), 25 deletions(-)drivers/net/ethernet/ibm/ibmvnic.c +1
-3@@ -4876,10 +4876,8 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq, }/* Done receiving requested capabilities, query IP offload support */ -if (atomic_read (&adapter->running_cap_crqs) ==0 ) { - adapter->wait_capability = false; +if (atomic_read (&adapter->running_cap_crqs) ==0 )send_query_ip_offload (adapter); - } }static int handle_login_rsp(union
ibmvnic_crq*login_rsp_crq,drivers/net/ethernet/ibm/ibmvnic.c +1
-3@@ -5177,10 +5175,8 @@ static void handle_query_cap_rsp(union ibmvnic_crq *crq, } out: -if (atomic_read (&adapter->running_cap_crqs) ==0 ) { - adapter->wait_capability = false; +if (atomic_read (&adapter->running_cap_crqs) ==0 )send_request_cap (adapter,0 ); - } }static int
send_query_phys_parms(struct ibmvnic_adapter *adapter)drivers/net/ethernet/ibm/ibmvnic.c +12
-18@@ -5476,27 +5472,21 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)struct ibmvnic_crq_queue *queue = &adapter->crq;union ibmvnic_crq *crq;unsigned long flags; -bool done = false;spin_lock_irqsave (&queue->lock, flags); -while (!done) { -/* Pull all the valid messages off the CRQ */ -while ((crq =ibmvnic_next_crq (adapter)) !=NULL ) { -/* This barrier makes sure ibmvnic_next_crq()'s -* crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded -* before ibmvnic_handle_crq()'s -* switch(gen_crq->first) and switch(gen_crq->cmd). -*/ -dma_rmb (); -ibmvnic_handle_crq (crq, adapter); - crq->generic.first =0 ; - } + +/* Pull all the valid messages off the CRQ */ +while ((crq =ibmvnic_next_crq (adapter)) !=NULL ) { +/* This barrier makes sure ibmvnic_next_crq()'s +* crq->generic.first & IBMVNIC_CRQ_CMD_RSP is loaded +* before ibmvnic_handle_crq()'s +* switch(gen_crq->first) and switch(gen_crq->cmd). +*/ +dma_rmb (); +ibmvnic_handle_crq (crq, adapter); + crq->generic.first =0 ; } -/* if capabilities CRQ's were sent in this tasklet, the following -* tasklet must wait until all responses are received -*/ -if (atomic_read (&adapter->running_cap_crqs) !=0 ) - adapter->wait_capability = true; +spin_unlock_irqrestore (&queue->lock, flags); }drivers/net/ethernet/ibm/ibmvnic.h +0
-1@@ -921,7 +921,6 @@ struct ibmvnic_adapter {int login_rsp_buf_sz;atomic_t running_cap_crqs; -bool wait_capability;struct ibmvnic_sub_crq_queue **tx_scrq____cacheline_aligned ;struct ibmvnic_sub_crq_queue **rx_scrq____cacheline_aligned ;
Re: [PATCH net 2/4] ibmvnic: init ->running_cap_crqs early
In reply to Sukadev BhattiproluQuoted 31 lines · Sukadev Bhattiprolu
We use ->running_cap_crqs to determine when the
ibmvnic_tasklet()should send out the next protocol message type. i.e when we get back responses to all ourQUERY_CAPABILITYCRQs we send outREQUEST_CAPABILITYcrqs. Similiary, when we get responses to all theREQUEST_CAPABILITYcrqs, we send out theQUERY_IP_OFFLOADCRQ.We currently increment ->running_cap_crqs as we send out each CRQ and have the
ibmvnic_tasklet()send out the next message type, when thisrunning_cap_crqscount drops to 0.This assumes that all the CRQs of the current type were sent out before the count drops to 0. However it is possible that we send out say 6 CRQs, get preempted and receive all the 6 responses before we send out the remaining CRQs. This can result in ->running_cap_crqs count dropping to zero before all messages of the current type were sent and we end up sending the next protocol message too early.
Instead initialize the ->running_cap_crqs upfront so the tasklet will only send the next protocol message after all responses are received.
Use the
cap_reqslocal variable to also detect any discrepancy (either now or in future) in the number of capability requests we actually send.Currently only
send_query_cap()is affected by this behavior (of sending next message early) since it is called from the worker thread (during reset) and from application thread (during ->ndo_open()) and they can be preempted.send_request_cap()is only called from the tasklet which processes CRQ responses sequentially, is not be affected. But to maintain the existing symmtery withsend_query_capability()we updatesend_request_capability()also.
Quoted 3 lines
Fixes 249168ad07cd ("ibmvnic: Make CRQ interrupt tasklet wait for allcapabilities crqs")
Signed-off-by Sukadev Bhattiprolu <[email protected]>
Quoted 290 lines
--- drivers/net/ethernet/ibm/ibmvnic.c | 106 +++++++++++++++++++---------- 1 file changed, 71 insertions(+), 35 deletions(-)drivers/net/ethernet/ibm/ibmvnic.c +5
-0@@ -3849,11 +3849,25 @@ static void send_request_cap(struct ibmvnic_adapter *adapter, int retry)struct device *dev = &adapter->vdev->dev;union ibmvnic_crq crq;int max_entries; +int cap_reqs; + +/* We send out 6 or 7 REQUEST_CAPABILITY CRQs below (depending on +* the PROMISC flag). Initialize this count upfront. When the tasklet +* receives a response to all of these, it will send the next protocol
drivers/net/ethernet/ibm/ibmvnic.c +17
-7+* message (QUERY_IP_OFFLOAD). +*/ +if (!(adapter->netdev->flags &IFF_PROMISC ) || + adapter->promisc_supported) + cap_reqs =7 ; +else + cap_reqs =6 ;if (!retry) {/* Sub-CRQ entries are 32 byte long */ int entries_page =4 *PAGE_SIZE / (sizeof (u64) *4 ); +atomic_set (&adapter->running_cap_crqs, cap_reqs); +if (adapter->min_tx_entries_per_subcrq > entries_page || adapter->min_rx_add_entries_per_subcrq > entries_page) {dev_err (dev,"Fatal, invalid entries per sub-crq\n" );@@ -3914,44 +3928,45 @@ static void send_request_cap(struct ibmvnic_adapter *adapter, int retry) adapter->opt_rx_comp_queues; adapter->req_rx_add_queues = adapter->max_rx_add_queues; + }else { +atomic_add (cap_reqs, &adapter->running_cap_crqs); } -memset (&crq,0 ,sizeof (crq)); crq.request_capability.first =IBMVNIC_CRQ_CMD ; crq.request_capability.cmd =REQUEST_CAPABILITY ; crq.request_capability.capability =cpu_to_be16 (REQ_TX_QUEUES ); crq.request_capability.number =cpu_to_be64 (adapter->req_tx_queues); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_RX_QUEUES ); crq.request_capability.number =cpu_to_be64 (adapter->req_rx_queues); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_RX_ADD_QUEUES ); crq.request_capability.number =cpu_to_be64 (adapter->req_rx_add_queues); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_TX_ENTRIES_PER_SUBCRQ ); crq.request_capability.number =cpu_to_be64 (adapter->req_tx_entries_per_subcrq); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_RX_ADD_ENTRIES_PER_SUBCRQ ); crq.request_capability.number =cpu_to_be64 (adapter->req_rx_add_entries_per_subcrq); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); crq.request_capability.capability =cpu_to_be16 (REQ_MTU ); crq.request_capability.number =cpu_to_be64 (adapter->req_mtu); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq);drivers/net/ethernet/ibm/ibmvnic.c +7
-2if (adapter->netdev->flags &IFF_PROMISC ) {@@ -3959,16 +3974,21 @@ static void send_request_cap(struct ibmvnic_adapter *adapter, int retry) crq.request_capability.capability =cpu_to_be16 (PROMISC_REQUESTED ); crq.request_capability.number =cpu_to_be64 (1 ); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); } }else { crq.request_capability.capability =cpu_to_be16 (PROMISC_REQUESTED ); crq.request_capability.number =cpu_to_be64 (0 ); -atomic_inc (&adapter->running_cap_crqs); + cap_reqs--;ibmvnic_send_crq (adapter, &crq); } + +/* Keep at end to catch any discrepancy between expected and actual +* CRQs sent. +*/ +WARN_ON (cap_reqs !=0 ); }static int pending_scrq(struct
ibmvnic_adapter*adapter,drivers/net/ethernet/ibm/ibmvnic.c +40
-26@@ -4362,118 +4382,132 @@ static void send_query_map(struct ibmvnic_adapter *adapter)static void send_query_cap (struct ibmvnic_adapter *adapter) {union ibmvnic_crq crq; +int cap_reqs; + +/* We send out 25 QUERY_CAPABILITY CRQs below. Initialize this count +* upfront. When the tasklet receives a response to all of these, it +* can send out the next protocol messaage (REQUEST_CAPABILITY). +*/ + cap_reqs =25 ; + +atomic_set (&adapter->running_cap_crqs, cap_reqs); -atomic_set (&adapter->running_cap_crqs,0 );memset (&crq,0 ,sizeof (crq)); crq.query_capability.first =IBMVNIC_CRQ_CMD ; crq.query_capability.cmd =QUERY_CAPABILITY ; crq.query_capability.capability =cpu_to_be16 (MIN_TX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_RX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_RX_ADD_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_TX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_RX_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_RX_ADD_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_TX_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_RX_ADD_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_TX_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_RX_ADD_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (TCP_IP_OFFLOAD ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (PROMISC_SUPPORTED ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MIN_MTU ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_MTU ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_MULTICAST_FILTERS ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (VLAN_HEADER_INSERTION ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (RX_VLAN_HEADER_INSERTION ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (MAX_TX_SG_ENTRIES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (RX_SG_SUPPORTED ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_TX_COMP_SUB_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_RX_COMP_QUEUES ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_RX_BUFADD_Q_PER_RX_COMP_Q ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_TX_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (OPT_RXBA_ENTRIES_PER_SUBCRQ ); -atomic_inc (&adapter->running_cap_crqs);ibmvnic_send_crq (adapter, &crq); + cap_reqs--; crq.query_capability.capability =cpu_to_be16 (TX_RX_DESC_REQ ); -atomic_inc (&adapter->running_cap_crqs); +ibmvnic_send_crq (adapter, &crq); + cap_reqs--; + +/* Keep at end to catch any discrepancy between expected and actual +* CRQs sent. +*/ +WARN_ON (cap_reqs !=0 );}
static void
send_query_ip_offload(struct ibmvnic_adapter *adapter)drivers/net/ethernet/ibm/ibmvnic.c +2
-0@@ -4777,6 +4811,8 @@ static void handle_request_cap_rsp(union ibmvnic_crq *crq, char *name;atomic_dec (&adapter->running_cap_crqs); +netdev_dbg (adapter->netdev,"Outstanding request-caps: %d\n" , +atomic_read (&adapter->running_cap_crqs));switch (be16_to_cpu (crq->request_capability_rsp.capability)) {case REQ_TX_QUEUES :
req_value=&adapter->req_tx_queues;
Re: [PATCH net 3/4] ibmvnic: don't spin in tasklet
In reply to Sukadev BhattiproluQuoted 6 lines · Sukadev Bhattiprolu
ibmvnic_tasklet()continuously spins waiting for responses to all capability requests. It does this to avoid encountering an error during initialization of the vnic. However if there is a bug in the VIOS and we do not receive a response to one or more queries the tasklet ends up spinning continuously leading to hard lock ups.
Quoted 6 lines
If we fail to receive a message from the VIOS it is reasonable to timeout the login attempt rather than spin indefinitely in the tasklet.
Fixes 249168ad07cd ("ibmvnic: Make CRQ interrupt tasklet wait for allcapabilities crqs")
Signed-off-by Sukadev Bhattiprolu <[email protected]>
Quoted 23 lines
--- drivers/net/ethernet/ibm/ibmvnic.c | 6 ------ 1 file changed, 6 deletions(-)drivers/net/ethernet/ibm/ibmvnic.c +0
-6@@ -5491,12 +5491,6 @@ static void ibmvnic_tasklet(struct tasklet_struct *t)ibmvnic_handle_crq (crq ,adapter ); crq->generic.first =0 ; } - -/* remain in tasklet until all -* capabilities responses are received -*/ -if (!adapter->wait_capability) - done = true; }/* if capabilities CRQ's were sent in this tasklet, the following * tasklet must wait until all responses are received
This series was applied to netdev/net.git (master) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net.git (master) by David S. Miller <[email protected]>:
Quoted 6 lines · On Fri, 21 Jan 2022 18:59:18 -0800 you wrote:
If auto-priority-failover (APF) is enabled and there are at least two backing devices of different priorities, some resets like fail-over, change-param etc can cause at least two back to back failovers. (Failover from high priority backing device to lower priority one and then back to the higher priority one if that is still functional).
Quoted 8 lines
Depending on the timimg of the two failovers it is possible to trigger a "hard" reset and for the hard reset to fail due to failovers. When this occurs, the driver assumes that the network is unstable and disables the VNIC for a 60-second "settling time". This in turn can cause the ethtool command to fail with "No such device" while the vnic automatically recovers a little while later.
[...]
Here is the summary with links:
- [net,1/4] ibmvnic: Allow extra failures before disabling
https://git.kernel.org/netdev/net/c/db9f0e8bf79e - [net,2/4] ibmvnic: init ->running_cap_crqs early https://git.kernel.org/netdev/net/c/151b6a5c06b6 - [net,3/4] ibmvnic: don't spin in tasklet https://git.kernel.org/netdev/net/c/48079e7fdd02 - [net,4/4] ibmvnic: remove unused ->wait_capability https://git.kernel.org/netdev/net/c/3a5d9db7fbdf
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html