s390/dasd: add dasd autoquiesce feature
Subsystems (3)
Stefan Haberland
maintainercover s390/dasd: add dasd autoquiesce feature
Hi Jens,
please apply the following patchset that introduces an autoquiesce feature for DASD devices.
Quiesce and resume are functions that tell Linux to stop/resume issuing I/Os to a specific DASD.
The DASD driver allows a manual quiesce/resume via ioctl.
Autoquiesce will define an amount of triggers that will lead to an automatic quiesce if a certain event occurs.
There is no automatic resume.
The autoquiesce feature is useful for devices in a copy relation where different events should lead to an automatic quiesce of a device. This prevents IO errors, unnecessary load and gives the controlling agent time to sort out the situation and trigger a swap of copy pairs if needed before it will resume device operation.
Stefan Haberland (7):
s390/dasd: remove unused DASD EER defines s390/dasd: add autoquiesce feature
s390/dasd: add aq_mask sysfs attribute s390/dasd: add aq_requeue sysfs attribute s390/dasd: add aq_timeouts autoquiesce trigger s390/dasd: add autoquiesce event for start IO error s390/dasd: fix hanging blockdevice after request requeue
arch/s390/include/uapi/asm/dasd.h | 2 + drivers/s390/block/dasd.c | 75 +++++++++++++----- drivers/s390/block/dasd_devmap.c | 126 ++++++++++++++++++++++++++++++ drivers/s390/block/dasd_eckd.c | 1 + drivers/s390/block/dasd_eer.c | 1 + drivers/s390/block/dasd_int.h | 32 ++++---- 6 files changed, 204 insertions(+), 33 deletions(-)
Signature · 1 line
2.37.2
Stefan Haberland
maintainerPATCH 1/7 s390/dasd: remove unused DASD EER defines
Remove definitions that have never been used.
--- drivers/s390/block/dasd_int.h | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-)
drivers/s390/block/dasd_int.h +1 -11
@@ -444,17 +444,7 @@ struct dasd_discipline {extern struct dasd_discipline *dasd_diag_discipline_pointer; -/* -* Notification numbers for extended error reporting notifications: -* The DASD_EER_DISABLE notification is sent before a dasd_device (and it's -* eer pointer) is freed. The error reporting module needs to do all necessary -* cleanup steps. -* The DASD_EER_TRIGGER notification sends the actual error reports (triggers). -*/ -#define DASD_EER_DISABLE 0 -#define DASD_EER_TRIGGER 1 - -/* Trigger IDs for extended error reporting DASD_EER_TRIGGER notification */ +/* Trigger IDs for extended error reporting */ #define DASD_EER_FATALERROR 1#define DASD_EER_NOPATH 2#define DASD_EER_STATECHANGE 3
Signature · 1 line
2.37.2
Re: [PATCH 1/7] s390/dasd: remove unused DASD EER defines
Quoted 4 lines · On Wed, 5 Apr 2023 16:20:11 +0200 Stefan Haberland <[email protected]> wrote:
Remove definitions that have never been used.
Signed-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
Stefan Haberland
maintainerPATCH 2/7 s390/dasd: add autoquiesce feature
In reply to Stefan HaberlandAdd the internal logic to check for autoquiesce triggers and handle them.
Quiesce and resume are functions that tell Linux to stop/resume issuing I/Os to a specific DASD.
The DASD driver allows a manual quiesce/resume via ioctl.
Autoquiesce will define an amount of triggers that will lead to an automatic quiesce if a certain event occurs.
There is no automatic resume.
All events will be reported via DASD Extended Error Reporting (EER) if configured.
--- arch/s390/include/uapi/asm/dasd.h | 2 ++ drivers/s390/block/dasd.c | 60 ++++++++++++++++++++++--------- drivers/s390/block/dasd_eer.c | 1 + drivers/s390/block/dasd_int.h | 2 ++ 4 files changed, 48 insertions(+), 17 deletions(-)
arch/s390/include/uapi/asm/dasd.h +2 -0
@@ -78,6 +78,7 @@ typedef struct dasd_information2_t {* 0x040: give access to raw eckd data * 0x080: enable discard support * 0x100: enable autodisable for IFCC errors (default) +* 0x200: enable requeue of all requests on autoquiesce */ #define DASD_FEATURE_READONLY 0x001#define DASD_FEATURE_USEDIAG 0x002
@@ -88,6 +89,7 @@ typedef struct dasd_information2_t {#define DASD_FEATURE_USERAW 0x040#define DASD_FEATURE_DISCARD 0x080#define DASD_FEATURE_PATH_AUTODISABLE 0x100 +#define DASD_FEATURE_REQUEUEQUIESCE 0x200#define DASD_FEATURE_DEFAULT DASD_FEATURE_PATH_AUTODISABLE#define DASD_PARTN_BITS 2
drivers/s390/block/dasd.c +43 -17
112 lines not on this page · raw message
drivers/s390/block/dasd_eer.c +1 -0
@@ -387,6 +387,7 @@ void dasd_eer_write(struct dasd_device *device, struct dasd_ccw_req *cqr,break ;case DASD_EER_NOPATH :case DASD_EER_NOSPC : +case DASD_EER_AUTOQUIESCE :dasd_eer_write_standard_trigger (device,NULL , id);break ;case DASD_EER_STATECHANGE :
drivers/s390/block/dasd_int.h +2 -0
@@ -450,6 +450,7 @@ extern struct dasd_discipline *dasd_diag_discipline_pointer;#define DASD_EER_STATECHANGE 3#define DASD_EER_PPRCSUSPEND 4#define DASD_EER_NOSPC 5 +#define DASD_EER_AUTOQUIESCE 31/* DASD path handling */
@@ -627,6 +628,7 @@ struct dasd_device {struct dasd_format_entry format_entry;struct kset *paths_info;struct dasd_copy_relation *copy; +unsigned long aq_mask; };struct dasd_block {
Signature · 1 line
2.37.2
Re: [PATCH 2/7] s390/dasd: add autoquiesce feature
Quoted 11 lines · On Wed, 5 Apr 2023 16:20:12 +0200 Stefan Haberland <[email protected]> wrote:
Add the internal logic to check for autoquiesce triggers and handle them.
Quiesce and resume are functions that tell Linux to stop/resume issuing I/Os to a specific DASD.
The DASD driver allows a manual quiesce/resume via ioctl.Autoquiesce will define an amount of triggers that will lead to an automatic quiesce if a certain event occurs.
There is no automatic resume.
Quoted 5 lines
All events will be reported via DASD Extended Error Reporting (EER) if configured.
Signed-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
[..]
Stefan Haberland
maintainerPATCH 3/7 s390/dasd: add aq_mask sysfs attribute
In reply to Stefan HaberlandAdd sysfs attribute that controls the DASD autoquiesce feature. The autoquiesce is disabled when 0 is echoed to the attribute.
A value greater than 0 will enable the feature.
The aq_mask attribute will accept an unsigned integer and the value will be interpreted as bitmask defining the trigger events that will lead to an automatic quiesce.
The following autoquiesce triggers will currently be available:
DASD_EER_FATALERROR 1 - any final I/O errorDASD_EER_NOPATH 2 - no remaining paths for the deviceDASD_EER_STATECHANGE 3 - a state change interrupt occurred DASD_EER_PPRCSUSPEND 4 - the device is PPRC suspended
DASD_EER_NOSPC 5 - there is no space remaining on an ESE device DASD_EER_TIMEOUT 6 - a certain amount of timeouts occurred DASD_EER_STARTIO 7 - the IO start function encountered an error
The currently supported maximum value is 255.
Bit 31 is reserved for internal usage.
Bit 0 is not used.
Example:
- deactivate autoquiesce
$ echo 0 > /sys/bus/ccw/0.0.1234/aq_mask
- enable autoquiesce for FATALERROR, NOPATH and TIMEOUT (0000 0000 0000 0000 0000 0000 0100 0110 => 70)
$ echo 70 > /sys/bus/ccw/0.0.1234/aq_mask
--- drivers/s390/block/dasd_devmap.c | 43 ++++++++++++++++++++++++++++++++ drivers/s390/block/dasd_int.h | 23 +++++++++++------ 2 files changed, 59 insertions(+), 7 deletions(-)
drivers/s390/block/dasd_devmap.c +43 -0
@@ -50,6 +50,7 @@ struct dasd_devmap {unsigned short features;struct dasd_device *device;struct dasd_copy_relation *copy; +unsigned int aq_mask; };/*
@@ -1475,6 +1476,47 @@ dasd_eer_store(struct device *dev, struct device_attribute *attr, static DEVICE_ATTR(eer_enabled, 0644, dasd_eer_show, dasd_eer_store); +/* +* aq_mask controls if the DASD should be quiesced on certain triggers +* The aq_mask attribute is interpreted as bitmap of the DASD_EER_* triggers. +*/ +static ssize_t dasd_aq_mask_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct dasd_devmap *devmap; +unsigned int aq_mask =0 ; + + devmap =dasd_find_busid (dev_name (dev)); +if (!IS_ERR (devmap)) + aq_mask = devmap->aq_mask; + +return sysfs_emit (buf,"%d\n" , aq_mask); +} + +static ssize_t dasd_aq_mask_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t count) +{ +struct dasd_devmap *devmap; +unsigned int val; + +if (kstrtouint (buf,0 , &val) || val >DASD_EER_VALID ) +return -EINVAL ; + + devmap =dasd_devmap_from_cdev (to_ccwdev (dev)); +if (IS_ERR (devmap)) +return PTR_ERR (devmap); + +spin_lock (&dasd_devmap_lock); + devmap->aq_mask = val; +if (devmap->device) + devmap->device->aq_mask = devmap->aq_mask; +spin_unlock (&dasd_devmap_lock); + +return count; +} + +static DEVICE_ATTR(aq_mask, 0644, dasd_aq_mask_show, dasd_aq_mask_store); +/* * expiration time for default requests */
@@ -2324,6 +2366,7 @@ static struct attribute * dasd_attrs[] = { &dev_attr_copy_pair.attr, &dev_attr_copy_role.attr, &dev_attr_ping.attr, + &dev_attr_aq_mask.attr, NULL, };
drivers/s390/block/dasd_int.h +16 -7
@@ -444,13 +444,22 @@ struct dasd_discipline {extern struct dasd_discipline *dasd_diag_discipline_pointer; -/* Trigger IDs for extended error reporting */ -#define DASD_EER_FATALERROR 1 -#define DASD_EER_NOPATH 2 -#define DASD_EER_STATECHANGE 3 -#define DASD_EER_PPRCSUSPEND 4 -#define DASD_EER_NOSPC 5 -#define DASD_EER_AUTOQUIESCE 31 +/* Trigger IDs for extended error reporting DASD EER and autoquiesce */ +enum eer_trigger { +DASD_EER_FATALERROR =1 , +DASD_EER_NOPATH , +DASD_EER_STATECHANGE , +DASD_EER_PPRCSUSPEND , +DASD_EER_NOSPC , +DASD_EER_TIMEOUTS , +DASD_EER_STARTIO , + +/* enum end marker, only add new trigger above */ +DASD_EER_MAX , +DASD_EER_AUTOQUIESCE =31 ,/* internal only */ +}; + +#define DASD_EER_VALID ((1U << DASD_EER_MAX) - 1)/* DASD path handling */
Signature · 1 line
2.37.2
Re: [PATCH 3/7] s390/dasd: add aq_mask sysfs attribute
Quoted 26 lines · On Wed, 5 Apr 2023 16:20:13 +0200 Stefan Haberland <[email protected]> wrote:
Add sysfs attribute that controls the DASD autoquiesce feature. The autoquiesce is disabled when 0 is echoed to the attribute.
A value greater than 0 will enable the feature.
Theaq_maskattribute will accept an unsigned integer and the value will be interpreted as bitmask defining the trigger events that will lead to an automatic quiesce.The following autoquiesce triggers will currently be available:
DASD_EER_FATALERROR1 - any final I/O errorDASD_EER_NOPATH2 - no remaining paths for the deviceDASD_EER_STATECHANGE3 - a state change interrupt occurredDASD_EER_PPRCSUSPEND4 - the device is PPRC suspendedDASD_EER_NOSPC 5 - there is no space remaining on an ESE device DASD_EER_TIMEOUT 6 - a certain amount of timeouts occurred DASD_EER_STARTIO 7 - the IO start function encountered an errorThe currently supported maximum value is 255.
Bit 31 is reserved for internal usage.
Bit 0 is not used.Example:
- deactivate autoquiesce
Quoted 8 lines
$ echo 0 > /sys/bus/ccw/0.0.1234/aq_mask- enable autoquiesce for FATALERROR, NOPATH and TIMEOUT (0000 0000 0000 0000 0000 0000 0100 0110 => 70)
$ echo 70 > /sys/bus/ccw/0.0.1234/aq_maskSigned-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
[..]
Stefan Haberland
maintainerPATCH 4/7 s390/dasd: add aq_requeue sysfs attribute
In reply to Stefan HaberlandAdd a sysfs attribute to control if all IO requests will be requeued to the blocklayer in case of an autoquiesce event or not.
A value of 1 means that in case of an autoquiesce event all IO requests will be requeued to the blocklayer.
A value of 0 means that the device will only be stopped.
--- drivers/s390/block/dasd_devmap.c | 36 ++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
drivers/s390/block/dasd_devmap.c +36 -0
@@ -1517,6 +1517,41 @@ static ssize_t dasd_aq_mask_store(struct device *dev, struct device_attribute *a static DEVICE_ATTR(aq_mask, 0644, dasd_aq_mask_show, dasd_aq_mask_store); +/* +* aq_requeue controls if requests are returned to the blocklayer on quiesce +* or if requests are only not started +*/ +static ssize_t dasd_aqr_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct dasd_devmap *devmap; +int flag; + + devmap =dasd_find_busid (dev_name (dev)); +if (!IS_ERR (devmap)) + flag = (devmap->features &DASD_FEATURE_REQUEUEQUIESCE ) !=0 ; +else + flag = (DASD_FEATURE_DEFAULT & +DASD_FEATURE_REQUEUEQUIESCE ) !=0 ; +return sysfs_emit (buf,"%d\n" , flag); +} + +static ssize_t dasd_aqr_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t count) +{ +bool val; +int rc; + +if (kstrtobool (buf, &val)) +return -EINVAL ; + + rc =dasd_set_feature (to_ccwdev (dev),DASD_FEATURE_REQUEUEQUIESCE , val); + +return rc ? : count; +} + +static DEVICE_ATTR(aq_requeue, 0644, dasd_aqr_show, dasd_aqr_store); +/* * expiration time for default requests */
@@ -2367,6 +2402,7 @@ static struct attribute * dasd_attrs[] = { &dev_attr_copy_role.attr, &dev_attr_ping.attr, &dev_attr_aq_mask.attr, + &dev_attr_aq_requeue.attr, NULL, };
Signature · 1 line
2.37.2
Re: [PATCH 4/7] s390/dasd: add aq_requeue sysfs attribute
Quoted 10 lines · On Wed, 5 Apr 2023 16:20:14 +0200 Stefan Haberland <[email protected]> wrote:
Add a sysfs attribute to control if all IO requests will be requeued to the blocklayer in case of an autoquiesce event or not.
A value of 1 means that in case of an autoquiesce event all IO requests will be requeued to the blocklayer.
A value of 0 means that the device will only be stopped.
Signed-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
[..]
Stefan Haberland
maintainerPATCH 5/7 s390/dasd: add aq_timeouts autoquiesce trigger
In reply to Stefan HaberlandAdd a sysfs attribute aq_timeouts that controls after how many timeouts a autoquiesce event might be triggered.
The default value is 32768 which is the maximum number of retries for the DASD device driver DASD_RETRIES_MAX. This means that the timeout trigger will never happen.
The default value for DASD retries is 255.
Setting the value to below 255 will trigger the timeout autoquiesce event before an IO error is generated.
Also add the check for the configured amount of timeouts and trigger an autoquiesce event if exceeded.
--- drivers/s390/block/dasd.c | 11 ++++++++ drivers/s390/block/dasd_devmap.c | 47 ++++++++++++++++++++++++++++++++ drivers/s390/block/dasd_eckd.c | 1 + drivers/s390/block/dasd_int.h | 1 + 4 files changed, 60 insertions(+)
drivers/s390/block/dasd.c +11 -0
@@ -1953,6 +1953,16 @@ static void __dasd_device_process_final_queue(struct dasd_device *device, } } +/* +* check if device should be autoquiesced due to too many timeouts +*/ +static void __dasd_device_check_autoquiesce_timeout (struct dasd_device *device, +struct dasd_ccw_req *cqr) +{ +if ((device->default_retries - cqr->retries) >= device->aq_timeouts) +dasd_handle_autoquiesce (device, cqr,DASD_EER_TIMEOUTS ); +} +/* * Take a look at the first request on the ccw queue and check * if it reached its expire time. If so, terminate the IO.
@@ -1987,6 +1997,7 @@ static void __dasd_device_check_expire(struct dasd_device *device)"remaining\n" , cqr, (cqr->expires/HZ ), cqr->retries); } +__dasd_device_check_autoquiesce_timeout (device, cqr); } }
drivers/s390/block/dasd_devmap.c +47 -0
59 lines not on this page · raw message
drivers/s390/block/dasd_eckd.c +1 -0
@@ -2109,6 +2109,7 @@ dasd_eckd_check_characteristics(struct dasd_device *device) device->default_retries =DASD_RETRIES ; device->path_thrhld =DASD_ECKD_PATH_THRHLD ; device->path_interval =DASD_ECKD_PATH_INTERVAL ; + device->aq_timeouts =DASD_RETRIES_MAX ;if (private->conf.gneq) { value =1 ;
drivers/s390/block/dasd_int.h +1 -0
@@ -638,6 +638,7 @@ struct dasd_device {struct kset *paths_info;struct dasd_copy_relation *copy;unsigned long aq_mask; +unsigned int aq_timeouts; };struct dasd_block {
Signature · 1 line
2.37.2
Re: [PATCH 5/7] s390/dasd: add aq_timeouts autoquiesce trigger
Quoted 11 lines · On Wed, 5 Apr 2023 16:20:15 +0200 Stefan Haberland <[email protected]> wrote:
Add a sysfs attribute
aq_timeoutsthat controls after how many timeouts a autoquiesce event might be triggered.The default value is 32768 which is the maximum number of retries for the DASD device driver
DASD_RETRIES_MAX. This means that the timeout trigger will never happen.The default value for DASD retries is 255.
Setting the value to below 255 will trigger the timeout autoquiesce event before an IO error is generated.
Quoted 5 lines
Also add the check for the configured amount of timeouts and trigger an autoquiesce event if exceeded.
Signed-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
[..]
Stefan Haberland
maintainerPATCH 6/7 s390/dasd: add autoquiesce event for start IO error
In reply to Stefan HaberlandAdd a check for errors in the start_io function that signal a not working device. Trigger an autoquiesce event in that case.
--- drivers/s390/block/dasd.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/s390/block/dasd.c +2 -0
@@ -1452,6 +1452,8 @@ int dasd_start_IO(struct dasd_ccw_req *cqr)case -ENODEV :DBF_DEV_EVENT (DBF_WARNING , device,"%s" ,"start_IO: -ENODEV device gone, retry" ); +/* this is equivalent to CC=3 for SSCH report this to EER */ +dasd_handle_autoquiesce (device, cqr,DASD_EER_STARTIO );break ;case -EIO : DBF_DEV_EVENT(DBF_WARNING, device, "%s",
Signature · 1 line
2.37.2
Re: [PATCH 6/7] s390/dasd: add autoquiesce event for start IO error
Quoted 5 lines · On Wed, 5 Apr 2023 16:20:16 +0200 Stefan Haberland <[email protected]> wrote:
Add a check for errors in the
start_iofunction that signal a not working device. Trigger an autoquiesce event in that case.Signed-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
[..]
Stefan Haberland
maintainerPATCH 7/7 s390/dasd: fix hanging blockdevice after request requeue
In reply to Stefan HaberlandThe DASD driver does not kick the requeue list when requeuing IO requests to the blocklayer. This might lead to hanging blockdevice when there is no other trigger for this.
Fix by automatically kick the requeue list when requeuing DASD requests to the blocklayer.
--- drivers/s390/block/dasd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/s390/block/dasd.c +1 -1
@@ -2953,7 +2953,7 @@ static int _dasd_requeue_request(struct dasd_ccw_req *cqr)return 0 ;spin_lock_irq (&cqr->dq->lock); req = (struct request *) cqr->callback_data; -blk_mq_requeue_request (req, false); +blk_mq_requeue_request (req, true);spin_unlock_irq (&cqr->dq->lock);return 0 ;
Signature · 1 line
2.37.2
Re: [PATCH 7/7] s390/dasd: fix hanging blockdevice after request requeue
Quoted 11 lines · On Wed, 5 Apr 2023 16:20:17 +0200 Stefan Haberland <[email protected]> wrote:
The DASD driver does not kick the requeue list when requeuing IO requests to the blocklayer. This might lead to hanging blockdevice when there is no other trigger for this.
Fix by automatically kick the requeue list when requeuing DASD requests to the blocklayer.
Fixes e443343e509a ("s390/dasd: blk-mq conversion")CC [email protected] # 4.14+Signed-off-by Stefan Haberland <[email protected]>Reviewed-by Jan Hoeppner <[email protected]>
[..]
Quoted 8 lines · Stefan Haberland
please apply the following patchset that introduces an autoquiesce feature for DASD devices.
Quiesce and resume are functions that tell Linux to stop/resume issuing I/Os to a specific DASD.
The DASD driver allows a manual quiesce/resume via ioctl.[...]
Applied, thanks!
[1/7] s390/dasd: remove unused DASD EER defines
commit: 861d53dbed4cad8cf1bbef692111f2215e02c38e
[2/7] s390/dasd: add autoquiesce feature
commit: 1cee2975bbabd89df1097c354867192106b058ea
[3/7] s390/dasd: add aq_mask sysfs attribute
commit: 9558a8e9d4a681e67b3abe9cabf3e3d8825af57e
[4/7] s390/dasd: add aq_requeue sysfs attribute
commit: bdac94e29564bab9f24c2700f16ff11f31af7c11
[5/7] s390/dasd: add aq_timeouts autoquiesce trigger
commit: 0c1a14748133024a33aa8ffd763ca7f5c03bb27e
[6/7] s390/dasd: add autoquiesce event for start IO error
commit: d9ee2bee4a63844cd9d1e0d00b1e3c49eacd1c2f
[7/7] s390/dasd: fix hanging blockdevice after request requeue
commit: d8898ee50edecacdf0141f26fd90acf43d7e9cd7Best regards,
Signature · 1 line
Jens Axboe