ice: wait for EMP reset after firmware flash
Subsystems (4)
PATCH 1/1 ice: wait for EMP reset after firmware flash
We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this wait fw_activate let card in inconsistent state recoverable only by second flash/activate
Reproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/net/ethernet/intel/ice/ice_main.c +3 -0
@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)dev_dbg (dev,"rebuilding PF after reset_type=%d\n" , reset_type); +#define ICE_EMP_RESET_SLEEP 5000if (reset_type ==ICE_RESET_EMPR ) {/* If an EMP reset has occurred, any previously pending flash * update will have completed. We no longer know whether or * not the NVM update EMP reset is restricted. */ pf->fw_emp_reset_disabled = false; + +msleep (ICE_EMP_RESET_SLEEP ); } err =ice_init_all_ctrlq (hw);
Signature · 1 line
2.35.1
[Cc: +Jakob, +regressions at lists.linux.dev]
Dear Petr,
Thank you for your patch for the regression.
#regzbot ^introduced 399e27dbbd9e94
Am 12.04.22 um 12:27 schrieb Petr Oros:
Please mention the time in the commit message summary:
ice: Wait 5 s for EMP reset after firmware flash
Quoted 2 lines
We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this waitfw_activatelet
Which OOT driver exactly?
Quoted 1 line
card in inconsistent state recoverable only by second flash/activate
Please reflow for 75 characters per line, and add a dot/period to the end of sentences.
Quoted 25 lines
Reproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0dmesg after flash:
Quoted 10 lines
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
So, what is the error? Get PHY capabilities failed status = -5?
What firmware version did the network card have before and after the update?
Quoted 1 line
Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
s/don't/doesn?t/
Quoted 21 lines
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
Quoted 2 lines
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0Did you try anything less than five seconds? i40e uses one second, cf. commit 9b13bd53134c (i40e: Increase delay to 1 s after global EMP reset).
Quoted 11 lines
Fixes 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")Signed-off-by Petr Oros <[email protected]>--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)drivers/net/ethernet/intel/ice/ice_main.c +0
-0
drivers/net/ethernet/intel/ice/ice_main.c static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) · line 6934
Quoted 4 lines
drivers/net/ethernet/intel/ice/ice_main.c +1
-0@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)dev_dbg (dev,"rebuilding PF after reset_type=%d\n" , reset_type); +#define ICE_EMP_RESET_SLEEP 5000
Please append the unit to the macro name.
drivers/net/ethernet/intel/ice/ice_main.c static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) · line 6945
Quoted 11 lines
drivers/net/ethernet/intel/ice/ice_main.c +2
-0if (reset_type ==ICE_RESET_EMPR ) {/* If an EMP reset has occurred, any previously pending flash * update will have completed. We no longer know whether or * not the NVM update EMP reset is restricted. */ pf->fw_emp_reset_disabled = false; + +msleep (ICE_EMP_RESET_SLEEP ); } err =ice_init_all_ctrlq (hw);
Kind regards,
Paul
Re: [Intel-wired-lan] [PATCH] ice: wait for EMP reset after firmware flash #forregzbot
TWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.
#regzbot fixed-by: b537752e6cbf0e
Quoted 146 lines · Paul Menzel
[Cc: +Jakob, +regressions at lists.linux.dev]
Dear Petr,
Thank you for your patch for the regression.
#regzbot ^introduced
399e27dbbd9e94Am 12.04.22 um 12:27 schrieb Petr Oros:
Please mention the time in the commit message summary:
ice: Wait 5 s for EMP reset after firmware flash
Quoted 2 lines
We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this waitfw_activateletWhich OOT driver exactly?
Quoted 1 line
card in inconsistent state recoverable only by second flash/activate
Please reflow for 75 characters per line, and add a dot/period to the end of sentences.
Quoted 35 lines
Reproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
????fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
???? link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
???? altname enp202s0f0dmesg after flash:
[?? 55.120788] ice: Copyright (c) 2018, Intel Corporation. [?? 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
[?? 55.569797] ice 0000:ca:00.0: The DDP package was successfully
loaded: ICE OS Default Package version 1.3.28.0
[?? 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [?? 55.608951] ice 0000:ca:00.0:ice_init_nvm_phy_typefailed: -5 [?? 55.647348] ice 0000:ca:00.0: PTP init successfulQuoted 8 lines
[?? 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[?? 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[?? 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [?? 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)So, what is the error?
Get PHY capabilities failed status = -5?What firmware version did the network card have before and after the update?
Quoted 2 lines
Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
s/don't/doesn?t/
Quoted 17 lines
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reloadQuoted 8 lines
[root at host ~]# devlink dev reload pci/0000:ca:00.0 action
fw_activatereload_actions_performed:
????fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
???? link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
???? altname enp202s0f0Did you try anything less than five seconds? i40e uses one second, cf. commit
9b13bd53134c(i40e: Increase delay to 1 s after global EMP reset).Quoted 15 lines
Fixes 399e27dbbd9e94 ("ice: support immediate firmware activation viadevlink reload")
Signed-off-by Petr Oros <[email protected]>---
?drivers/net/ethernet/intel/ice/ice_main.c| 3 +++ ? 1 file changed, 3 insertions(+)drivers/net/ethernet/intel/ice/ice_main.c +0
-0@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) ? ????? dev_dbg(dev, "rebuilding PF after reset_type=%d\n", reset_type);Quoted 1 line
? +#define
ICE_EMP_RESET_SLEEP5000Please append the unit to the macro name.
Quoted 10 lines
????? if (
reset_type==ICE_RESET_EMPR) {
????????? /* If an EMP reset has occurred, any previously pending flash ?????????? * update will have completed. We no longer know whether or ?????????? * not the NVM update EMP reset is restricted. ?????????? */
?????????pf->fw_emp_reset_disabled= false;
+
+???????msleep(ICE_EMP_RESET_SLEEP);
????? }
? ????? err =ice_init_all_ctrlq(hw);Kind regards,
Paul
From: Petr Oros <[email protected]>
Date: Tue, 12 Apr 2022 12:27:53 +0200
Quoted 76 lines
We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this waitfw_activatelet card in inconsistent state recoverable only by second flash/activateReproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0Fixes 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")Signed-off-by Petr Oros <[email protected]>--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)drivers/net/ethernet/intel/ice/ice_main.c +0
-0
drivers/net/ethernet/intel/ice/ice_main.c static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) · line 6934
Quoted 4 lines
drivers/net/ethernet/intel/ice/ice_main.c +1
-0@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)dev_dbg (dev,"rebuilding PF after reset_type=%d\n" , reset_type); +#define ICE_EMP_RESET_SLEEP 5000
Ooof, 5 sec is a lot! Is there any way to poll the device readiness? Does it really need the whole 5 sec?
Quoted 13 lines
drivers/net/ethernet/intel/ice/ice_main.c +2
-0if (reset_type ==ICE_RESET_EMPR ) {/* If an EMP reset has occurred, any previously pending flash * update will have completed. We no longer know whether or * not the NVM update EMP reset is restricted. */ pf->fw_emp_reset_disabled = false; + +msleep (ICE_EMP_RESET_SLEEP ); } err =ice_init_all_ctrlq (hw);Signature · 1 line
2.35.1
Thanks,
Al
PATCH 1/1 ice: wait for EMP reset after firmware flash
Quoted 80 lines · Alexander Lobakin
From: Petr Oros <[email protected]>
Date: Tue, 12 Apr 2022 12:27:53 +0200Quoted 77 lines
We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this waitfw_activatelet card in inconsistent state recoverable only by second flash/activateReproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0Fixes 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")Signed-off-by Petr Oros <[email protected]>--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)drivers/net/ethernet/intel/ice/ice_main.c +0
-0@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)
Quoted 7 lines
drivers/net/ethernet/intel/ice/ice_main.c static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) · line 6934
Quoted 3 lines
drivers/net/ethernet/intel/ice/ice_main.c +1
-0dev_dbg (dev,"rebuilding PF after reset_type=%d\n" , reset_type); +#define ICE_EMP_RESET_SLEEP 5000Ooof, 5 sec is a lot! Is there any way to poll the device readiness? Does it really need the whole 5 sec?
This came from a workaround we shipped with our sourceforge out-of-tree release. So far, I don't think we have any data on precisely how long we need to wait, or how to automatically detect this situation.
The issue appears to be caused by collisions with firmware finishing its own internal recovery of EMP.
Quoted 20 lines
Quoted 13 lines
drivers/net/ethernet/intel/ice/ice_main.c +2
-0if (reset_type ==ICE_RESET_EMPR ) {/* If an EMP reset has occurred, any previously pending flash * update will have completed. We no longer know whether or * not the NVM update EMP reset is restricted. */ pf->fw_emp_reset_disabled = false; + +msleep (ICE_EMP_RESET_SLEEP ); } err =ice_init_all_ctrlq (hw);Signature · 1 line
2.35.1Thanks,
Al
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 4 lines · Alexander Lobakin
Quoted 1 line
+#define
ICE_EMP_RESET_SLEEP5000Ooof, 5 sec is a lot! Is there any way to poll the device readiness? Does it really need the whole 5 sec?
The problem with resets is usually you don't want to disturb the hardware with PCIe pokes while it's resetting. In this case, this code only runs if it was indicated by the update status that the firmware could internally reset itself to pick up new updates, avoiding a reboot.
Yes the five seconds is less than ideal, but it was the value we found that offered the most stability / likelihood of reset completing. In this case the 5 seconds is a tradeoff vs a reboot.
Thank you Ivan for chasing down this bug!
Quoted 62 lines · Petr Oros
We need to wait for EMP reset after firmware flash.
Code was extracted from OOT driver and without this waitfw_activatelet card in inconsistent state recoverable only by second flash/activateReproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent state
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
Quoted 3 lines
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0Ahh.. good find. I checked a bunch of places, but didn't check here for differences. :(
For what its worth, I checked the source history of the out-of-tree driver this came from. It appears to be a workaround added for fixing a similar issue.
I haven't been able to dig up the full details yet. It appeares to be a collision with firmware finalizing recovery after the EMP reset.
Still trying to dig for any more information I can find.
Quoted 26 lines
Fixes 399e27dbbd9e94 ("ice: support immediate firmware activation via devlink reload")Signed-off-by Petr Oros <[email protected]>--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)drivers/net/ethernet/intel/ice/ice_main.c +3
-0@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)dev_dbg (dev,"rebuilding PF after reset_type=%d\n" , reset_type); +#define ICE_EMP_RESET_SLEEP 5000if (reset_type ==ICE_RESET_EMPR ) {/* If an EMP reset has occurred, any previously pending flash * update will have completed. We no longer know whether or * not the NVM update EMP reset is restricted. */ pf->fw_emp_reset_disabled = false; + +msleep (ICE_EMP_RESET_SLEEP ); } err =ice_init_all_ctrlq (hw);
Jacob Keller p??e v ?t 12. 04. 2022 v 09:58 -0700:
Quoted 96 lines · Petr Oros
Quoted 74 lines
We need to wait for EMP reset after firmware flash. Code was extracted from OOT driver and without this wait
fw_activatelet
card in inconsistent state recoverable only by second flash/activateReproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bi n
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
???fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
??? link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff ??? altname enp202s0f0dmesg after flash:
[?? 55.120788] ice: Copyright (c) 2018, Intel Corporation. [?? 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway
[?? 55.569797] ice 0000:ca:00.0: The DDP package was successfully
loaded: ICE OS Default Package version 1.3.28.0
[?? 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [?? 55.608951] ice 0000:ca:00.0:ice_init_nvm_phy_typefailed: -5 [?? 55.647348] ice 0000:ca:00.0: PTP init successful
[?? 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[?? 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[?? 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware
[?? 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link) Reboot don't help, only second flash/activate with OOT or patched driver put card back in consistent stateAfter patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bi n
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionQuoted 9 lines
fw_activatereload_actions_performed:
???fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
??? link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff ??? altname enp202s0f0Ahh.. good find. I checked a bunch of places, but didn't check here for
differences. :(For what its worth, I checked the source history of the out-of-tree driver this came from. It appears to be a workaround added for fixing a
similar issue.
Quoted 5 lines
I haven't been able to dig up the full details yet. It appeares to be a
collision with firmware finalizing recovery after the EMP reset.Still trying to dig for any more information I can find.
Interesting time frame could be around this commit:08771bce330036 ("ice: Continue probe on link/PHY errors")
Petr
Quoted 34 lines
Quoted 33 lines
Fixes 399e27dbbd9e94 ("ice: support immediate firmware activationvia devlink reload")
Signed-off-by Petr Oros <[email protected]>---
?drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ ?1 file changed, 3 insertions(+)drivers/net/ethernet/intel/ice/ice_main.c +0
-0@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)?
????????dev_dbg(dev, "rebuilding PF after reset_type=%d\n",reset_type);
?
+#defineICE_EMP_RESET_SLEEP5000
????????if (reset_type==ICE_RESET_EMPR) { ????????????????/* If an EMP reset has occurred, any previously pending flash
???????????????? * update will have completed. We no longer know whether or
???????????????? * not the NVM update EMP reset is restricted. ???????????????? */ ????????????????pf->fw_emp_reset_disabled = false;
+
+???????????????msleep(ICE_EMP_RESET_SLEEP);
????????}
?
????????err =ice_init_all_ctrlq(hw);
PATCH v2 1/1 ice: wait 5 s for EMP reset after firmware flash
In reply to Petr OrosWe need to wait 5 s for EMP reset after firmware flash. Code was extracted from OOT driver (ice v1.8.3 downloaded from sourceforge). Without this wait, fw_activate let card in inconsistent state and recoverable only by second flash/activate. Flash was tested on these fw's:
From -> To
3.00 -> 3.10/3.20
3.10 -> 3.00/3.20
3.20 -> 3.00/3.10
Reproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
Reboot doesn?t help, only second flash/activate with OOT or patched driver put card back in consistent state.
After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_8000AD7B.bin Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 action fw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff
altname enp202s0f0v2 changes:
- fixed format issues
- added info about fw and OOT driver versions
- added time in the commit message summary
- appended the unit to the macro name
--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/net/ethernet/intel/ice/ice_main.c +3 -0
@@ -6931,12 +6931,15 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type)dev_dbg (dev,"rebuilding PF after reset_type=%d\n" , reset_type); +#define ICE_EMP_RESET_SLEEP_MS 5000if (reset_type ==ICE_RESET_EMPR ) {/* If an EMP reset has occurred, any previously pending flash * update will have completed. We no longer know whether or * not the NVM update EMP reset is restricted. */ pf->fw_emp_reset_disabled = false; + +msleep (ICE_EMP_RESET_SLEEP_MS ); } err =ice_init_all_ctrlq (hw);
Signature · 1 line
2.35.1
RE: [Intel-wired-lan] [PATCH v2] ice: wait 5 s for EMP reset after firmware flash
Quoted 101 lines
-----Original Message-----
From: Intel-wired-lan <[email protected]> On Behalf Of Petr Oros
Sent: Wednesday, April 13, 2022 9:08 PM
To: netdev at vger.kernel.org
Cc: ivecera <[email protected]>; pmenzel at molgen.mpg.de; intel-wired- lan at lists.osuosl.org; linux-kernel at vger.kernel.org; kuba at kernel.org; pabeni at redhat.com; davem at davemloft.net
Subject: [Intel-wired-lan] [PATCH v2] ice: wait 5 s for EMP reset after firmware flashWe need to wait 5 s for EMP reset after firmware flash. Code was extracted from OOT driver (ice v1.8.3 downloaded from sourceforge). Without this wait,
fw_activatelet card in inconsistent state and recoverable only by second flash/activate. Flash was tested on these fw's:
From -> To
3.00 -> 3.10/3.20
3.10 -> 3.00/3.20
3.20 -> 3.00/3.10Reproducer:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_80 00AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
71: ens7f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0dmesg after flash:
[ 55.120788] ice: Copyright (c) 2018, Intel Corporation. [ 55.274734] ice 0000:ca:00.0: Get PHY capabilities failed status = -5, continuing anyway [ 55.569797] ice 0000:ca:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.28.0 [ 55.603629] ice 0000:ca:00.0: Get PHY capability failed. [ 55.608951] ice 0000:ca:00.0: ice_init_nvm_phy_type failed: -5 [ 55.647348] ice 0000:ca:00.0: PTP init successful [ 55.675536] ice 0000:ca:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8 [ 55.685365] ice 0000:ca:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode. [ 55.692179] ice 0000:ca:00.0: Commit DCB Configuration to the hardware [ 55.701382] ice 0000:ca:00.0: 126.024 Gb/s available PCIe bandwidth, limitedby 16.0 GT/s PCIe x8 link at 0000:c9:02.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
Reboot doesn?t help, only second flash/activate with OOT or patched driver put card back in consistent state.After patch:
[root at host ~]# devlink dev flash pci/0000:ca:00.0 file E810_XXVDA4_FH_O_SEC_FW_1p6p1p9_NVM_3p10_PLDMoMCTP_0.11_80 00AD7B.bin
Preparing to flash
[fw.mgmt] Erasing
[fw.mgmt] Erasing done
[fw.mgmt] Flashing 100%
[fw.mgmt] Flashing done 100%
[fw.undi] Erasing
[fw.undi] Erasing done
[fw.undi] Flashing 100%
[fw.undi] Flashing done 100%
[fw.netlist] Erasing
[fw.netlist] Erasing done
[fw.netlist] Flashing 100%
[fw.netlist] Flashing done 100%
Activate new firmware by devlink reload
[root at host ~]# devlink dev reload pci/0000:ca:00.0 actionfw_activatereload_actions_performed:
fw_activate
[root at host ~]# ip link show ens7f0
19: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000link/ether b4:96:91:dc:72:e0 brd ff:ff:ff:ff:ff:ff altname enp202s0f0v2 changes:
- fixed format issues
- added info about fw and OOT driver versions
- added time in the commit message summary
- appended the unit to the macro name
Quoted 7 lines
Fixes 399e27dbbd9e94 ("ice: support immediate firmware activation viadevlink reload")
Signed-off-by Petr Oros <[email protected]>--- drivers/net/ethernet/intel/ice/ice_main.c | 3 +++ 1 file changed, 3 insertions(+)