zfcp: fix reaction on bit error theshold notification with adapter close
Subsystems (4)
PATCH 1/1 zfcp: fix reaction on bit error theshold notification with adapter close
Kernel message explanation:
* Description:
* The FCP channel reported that its bit error threshold has been exceeded.
* These errors might result from a problem with the physical components
* of the local fibre link into the FCP channel.
* The problem might be damage or malfunction of the cable or
* cable connection between the FCP channel and
* the adjacent fabric switch port or the point-to-point peer.
* Find details about the errors in the HBA trace for the FCP device.
* The zfcp device driver closed down the FCP device
* to limit the performance impact from possible I/O command timeouts.
* User action:
* Check for problems on the local fibre link, ensure that fibre optics are
* clean and functional, and all cables are properly plugged.
* After the repair action, you can manually recover the FCP device by
* writing "0" into its "failed" sysfs attribute.
* If recovery through sysfs is not possible, set the CHPID of the device
* offline and back online on the service element.
---
Martin, James,
an important zfcp fix for v5.4-rc.
It applies to Martin's 5.4/scsi-fixes or to James' fixes branch.
drivers/s390/scsi/zfcp_fsf.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
drivers/s390/scsi/zfcp_fsf.c +13 -3
@@ -27,6 +27,11 @@struct kmem_cache *zfcp_fsf_qtcb_cache; +static bool ber_stop = true; +module_param (ber_stop, bool,0600 ); +MODULE_PARM_DESC (ber_stop, +"Shuts down FCP devices for FCP channels that report a bit-error count in excess of its threshold (default on)" ); +static void zfcp_fsf_request_timeout_handler (struct timer_list *t) {struct zfcp_fsf_req *fsf_req =from_timer (fsf_req, t, timer);
@@ -236,10 +241,15 @@ static void zfcp_fsf_status_read_handler(struct zfcp_fsf_req *req)case FSF_STATUS_READ_SENSE_DATA_AVAIL :break ;case FSF_STATUS_READ_BIT_ERROR_THRESHOLD : -dev_warn (&adapter->ccw_device->dev, -"The error threshold for checksum statistics " -"has been exceeded\n" );zfcp_dbf_hba_bit_err ("fssrh_3" , req); +if (ber_stop) { +dev_warn (&adapter->ccw_device->dev, +"All paths over this FCP device are disused because of excessive bit errors\n" ); +zfcp_erp_adapter_shutdown (adapter,0 ,"fssrh_b" ); + }else { +dev_warn (&adapter->ccw_device->dev, +"The error threshold for checksum statistics has been exceeded\n" ); + }break ;case FSF_STATUS_READ_LINK_DOWN :zfcp_fsf_status_read_link_down (req);
Signature · 1 line
2.17.1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 17 lines · Sasha Levin
[This is an automated email]
This commit has been processed because it contains a "Fixes:" tag, fixing commit:
1da177e4c3f4Linux-2.6.12-rc2.The bot has tested the following trees: v5.3.1, v5.2.17, v4.19.75, v4.14.146, v4.9.194, v4.4.194.
v5.3.1: Build OK!
v5.2.17: Build OK!
v4.19.75: Build OK!
v4.14.146: Failed to apply! Possible dependencies:75492a51568b ("s390/scsi: Convert timers to use timer_setup()")v4.9.194: Failed to apply! Possible dependencies:
75492a51568b("s390/scsi: Convert timers to usetimer_setup()")bc46427e807e("scsi: zfcp: usesetup_timerinstead ofinit_timer")
Quoted 8 lines
v4.4.194: Failed to apply! Possible dependencies:
75492a51568b("s390/scsi: Convert timers to usetimer_setup()")bc46427e807e("scsi: zfcp: usesetup_timerinstead ofinit_timer")NOTE: The patch will not be queued to stable trees until it is upstream.
How should we proceed with this patch?
It's sufficient to have the fix in those more recent stable trees where it applies (and builds). My fixes tag formally indicates since when it was at least broken but I don't expect all stable or longterm kernels to get the fix. If I happen to find out we need the fix in a kernel where it does not apply, I'll send a backport to stable when the time is right.
Showing the possible dependencies is awesome!
Signature · 11 lines
Mit freundlichen Gruessen / Kind regards Steffen Maier Linux on IBM Z Development https://www.ibm.com/privacy/us/en/ IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschaeftsfuehrung: Dirk Wittkopp Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Steffen,
Quoted 11 lines
Kernel message explanation:
* Description:
* The FCP channel reported that its bit error threshold has been exceeded.
* These errors might result from a problem with the physical components
* of the local fibre link into the FCP channel.
* The problem might be damage or malfunction of the cable or
* cable connection between the FCP channel and
* the adjacent fabric switch port or the point-to-point peer.
* Find details about the errors in the HBA trace for the FCP device.
* The zfcp device driver closed down the FCP device
Quoted 8 lines
* to limit the performance impact from possible I/O command timeouts.
* User action:
* Check for problems on the local fibre link, ensure that fibre optics are
* clean and functional, and all cables are properly plugged.
* After the repair action, you can manually recover the FCP device by
* writing "0" into its "failed" sysfs attribute.
* If recovery through sysfs is not possible, set the CHPID of the device
* offline and back online on the service element.
This commentary does not read like a patch description. It makes no mention of the actual kernel changes and the introduced module parameter.
Quoted 5 lines
drivers/s390/scsi/zfcp_fsf.c +5
-0+static bool ber_stop = true; +module_param (ber_stop, bool,0600 ); +MODULE_PARM_DESC (ber_stop, +"Shuts down FCP devices for FCP channels that report a bit-error count in excess of its threshold (default on)" ); +
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
PATCH v2 1/1 zfcp: fix reaction on bit error theshold notification with adapter close
On excessive bit errors for the FCP channel ingress fibre path, the channel notifies us. Previously, we only emitted a kernel message and a trace record. Since performance can become suboptimal with I/O timeouts due to bit errors, we now stop using an FCP device by default on channel notification so multipath on top can timely failover to other paths. A new module parameter zfcp.ber_stop can be used to get zfcp old behavior.
User explanation of new kernel message:
* Description:
* The FCP channel reported that its bit error threshold has been exceeded.
* These errors might result from a problem with the physical components
* of the local fibre link into the FCP channel.
* The problem might be damage or malfunction of the cable or
* cable connection between the FCP channel and
* the adjacent fabric switch port or the point-to-point peer.
* Find details about the errors in the HBA trace for the FCP device.
* The zfcp device driver closed down the FCP device
* to limit the performance impact from possible I/O command timeouts.
* User action:
* Check for problems on the local fibre link, ensure that fibre optics are
* clean and functional, and all cables are properly plugged.
* After the repair action, you can manually recover the FCP device by
* writing "0" into its "failed" sysfs attribute.
* If recovery through sysfs is not possible, set the CHPID of the device
* offline and back online on the service element.
---
Martin, James,
an important zfcp fix for v5.4-rc.
It applies to Martin's 5.4/scsi-fixes or to James' fixes branch.
Changes since v1:
* Martin's review comments: describe code change and new module parameter
drivers/s390/scsi/zfcp_fsf.c | 16 +++++++++++++--- 2 files changed, 37 insertions(+), 3 deletions(-)
drivers/s390/scsi/zfcp_fsf.c +13 -3
@@ -29,6 +29,11 @@struct kmem_cache *zfcp_fsf_qtcb_cache; +static bool ber_stop = true; +module_param (ber_stop, bool,0600 ); +MODULE_PARM_DESC (ber_stop, +"Shuts down FCP devices for FCP channels that report a bit-error count in excess of its threshold (default on)" ); +static void zfcp_fsf_request_timeout_handler (struct timer_list *t) {struct zfcp_fsf_req *fsf_req =from_timer (fsf_req, t, timer);
@@ -238,10 +243,15 @@ static void zfcp_fsf_status_read_handler(struct zfcp_fsf_req *req)case FSF_STATUS_READ_SENSE_DATA_AVAIL :break ;case FSF_STATUS_READ_BIT_ERROR_THRESHOLD : -dev_warn (&adapter->ccw_device->dev, -"The error threshold for checksum statistics " -"has been exceeded\n" );zfcp_dbf_hba_bit_err ("fssrh_3" , req); +if (ber_stop) { +dev_warn (&adapter->ccw_device->dev, +"All paths over this FCP device are disused because of excessive bit errors\n" ); +zfcp_erp_adapter_shutdown (adapter,0 ,"fssrh_b" ); + }else { +dev_warn (&adapter->ccw_device->dev, +"The error threshold for checksum statistics has been exceeded\n" ); + }break ;case FSF_STATUS_READ_LINK_DOWN :zfcp_fsf_status_read_link_down (req);
Signature · 1 line
2.17.1
Greg KH
maintainerQuoted 6 lines · Steffen Maier
On excessive bit errors for the FCP channel ingress fibre path, the channel notifies us. Previously, we only emitted a kernel message and a trace record. Since performance can become suboptimal with I/O timeouts due to bit errors, we now stop using an FCP device by default on channel notification so multipath on top can timely failover to other paths. A new module parameter zfcp.ber_stop can be used to get zfcp old behavior.
Ugh, module parameters? This isn't the 1990's anymore :(
Why not just make this a dynamic sysfs variable, that way you properly can set this on whatever device you want, not just "all or nothing"?
thanks,
greg k-h
Quoted 8 lines · Greg KH
Quoted 6 lines · Steffen Maier
On excessive bit errors for the FCP channel ingress fibre path, the channel notifies us. Previously, we only emitted a kernel message and a trace record. Since performance can become suboptimal with I/O timeouts due to bit errors, we now stop using an FCP device by default on channel notification so multipath on top can timely failover to other paths. A new module parameter zfcp.ber_stop can be used to get zfcp old behavior.
Quoted 4 lines
Ugh, module parameters? This isn't the 1990's anymore :(
Why not just make this a dynamic sysfs variable, that way you properly can set this on whatever device you want, not just "all or nothing"?
Since we can see many more (virtual) FCP devices than we want to actually use, we defer probing. It means, we only start allocating structures and sysfs entries on setting an FCP "online" for the first time. Setting online works through another sysfs attribute owned by our ccw bus code component called "cio". IIRC, setting online does not emit a uevent. On setting online, the (add) uevent of hot-/coldplug of an FCP device had already happened, so we could not easily have end users craft udev rules to automatically/persistently configure a new sysfs attribute (which is FCP-device-specific and appears late) to disable the new code behavior.
Not sure if that could ever become a problem for end users: Even if we were to write into a new sysfs attribute, the attribute only appears during setting online so this might race with starting to actually use the FCP device with the new default behavior and could potentially disable I/O paths before the sysfs attribute write could become effective to disable the new behavor.
Signature · 11 lines
Mit freundlichen Gruessen / Kind regards Steffen Maier Linux on IBM Z Development https://www.ibm.com/privacy/us/en/ IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschaeftsfuehrung: Dirk Wittkopp Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Greg KH
maintainerQuoted 24 lines · Steffen Maier
Quoted 8 lines · Greg KH
Quoted 6 lines · Steffen Maier
On excessive bit errors for the FCP channel ingress fibre path, the channel notifies us. Previously, we only emitted a kernel message and a trace record. Since performance can become suboptimal with I/O timeouts due to bit errors, we now stop using an FCP device by default on channel notification so multipath on top can timely failover to other paths. A new module parameter zfcp.ber_stop can be used to get zfcp old behavior.
Quoted 4 lines
Ugh, module parameters? This isn't the 1990's anymore :(
Why not just make this a dynamic sysfs variable, that way you properly can set this on whatever device you want, not just "all or nothing"?
Since we can see many more (virtual) FCP devices than we want to actually use, we defer probing. It means, we only start allocating structures and sysfs entries on setting an FCP "online" for the first time. Setting online works through another sysfs attribute owned by our ccw bus code component called "cio". IIRC, setting online does not emit a uevent. On setting online, the (add) uevent of hot-/coldplug of an FCP device had already happened, so we could not easily have end users craft udev rules to automatically/persistently configure a new sysfs attribute (which is FCP-device-specific and appears late) to disable the new code behavior.
Quoted 6 lines
Not sure if that could ever become a problem for end users: Even if we were to write into a new sysfs attribute, the attribute only appears during setting online so this might race with starting to actually use the FCP device with the new default behavior and could potentially disable I/O paths before the sysfs attribute write could become effective to disable the new behavor.
Ok, then why make this a module option that you will have to support for the next 20+ years anyway if you feel this fix is the correct way that it should be done instead?
module options are tough to manage and support, only add them as a very last thing, when all other options have been ruled out.
thanks,
greg k-h
Greg,
Quoted 3 lines
Ok, then why make this a module option that you will have to support for the next 20+ years anyway if you feel this fix is the correct way that it should be done instead?
I agree.
Why not just shut FCP down unconditionally on excessive bit errors? What's the benefit of allowing things to continue? Are you hoping things will eventually recover in a single-path scenario?
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Quoted 9 lines · Martin K. Petersen
Quoted 3 lines
Ok, then why make this a module option that you will have to support for the next 20+ years anyway if you feel this fix is the correct way that it should be done instead?
I agree.
Why not just shut FCP down unconditionally on excessive bit errors? What's the benefit of allowing things to continue? Are you hoping things will eventually recover in a single-path scenario?
Experience told me that there will be an unforeseen end user scenario where I need a quick switch to let even shaky paths survive.
Signature · 11 lines
Mit freundlichen Gruessen / Kind regards Steffen Maier Linux on IBM Z Development https://www.ibm.com/privacy/us/en/ IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschaeftsfuehrung: Dirk Wittkopp Sitz der Gesellschaft: Boeblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Martin K. Petersen
AppliedSteffen,
Quoted 6 lines
Quoted 3 lines
Why not just shut FCP down unconditionally on excessive bit errors? What's the benefit of allowing things to continue? Are you hoping things will eventually recover in a single-path scenario?
Experience told me that there will be an unforeseen end user scenario where I need a quick switch to let even shaky paths survive.
Can't say I like it. But it's your driver.
Applied to 5.4/scsi-fixes. Thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering