zfcp: cleanups, refactorings and features for 5.11
Subsystems (3)
cover zfcp: cleanups, refactorings and features for 5.11
Hello James, Martin,
here is a series of changes for our zfcp driver for 5.11.
Other than 2 smaller cleanups and clarifications for maintainability we have a refactoring of how zfcp uses s390's qdio layer, and we have a small feature improving our handling of out-of-band version changes to our adapters (or firmware).
Especially the refactoring ("zfcp: lift Input Queue tasklet from qdio") would be nice to have, because we have other patches queued internally that depend on this, and because qdio patches go via the s390 tree, this creates a dependency for Heiko and Vasily.
As always, feedback and reviews are appreciated :-)
Julian Wiedmann (4):
zfcp: lift Input Queue tasklet from qdio
zfcp: clarify & assert the stat_lock locking in zfcp_qdio_send()
zfcp: process Version Change events
zfcp: handle event-lost notification for Version Change events
Vasily Gorbik (1):
zfcp: remove orphaned function declarations
drivers/s390/scsi/zfcp_aux.c | 11 ++++++++ drivers/s390/scsi/zfcp_def.h | 1 + drivers/s390/scsi/zfcp_ext.h | 2 -- drivers/s390/scsi/zfcp_fsf.c | 19 ++++++++++++++ drivers/s390/scsi/zfcp_fsf.h | 11 ++++++++ drivers/s390/scsi/zfcp_qdio.c | 47 +++++++++++++++++++++++++++++++++++ drivers/s390/scsi/zfcp_qdio.h | 2 ++ 7 files changed, 91 insertions(+), 2 deletions(-)
Signature · 1 line
2.26.2
PATCH 1/5 zfcp: lift Input Queue tasklet from qdio
From: Julian Wiedmann <[email protected]>
Shift the IRQ tasklet processing from the qdio layer into zfcp. This will allow for a good amount of cleanups in qdio, and provides future opportunity to improve the IRQ processing inside zfcp.
We continue to use the qdio layer's internal tasklet/timer mechanism (ie. scan_threshold etc) to check for Request Queue completions. Initially we planned to check for such completions after inspecting the Response Queue - this should typically work, but there's a theoretical race where the device only presents the Request Queue completions _after_ all Response Queue processing has finished. If the Request Queue is then also _completely_ full, we could send no further IOs and thus get no interrupt that would trigger an inspection of the Request Queue.
So for now stick to the old model, where we can trust that such a race would be recovered by qdio's internal timer.
Code-flow wise, this establishes two levels of control:
1. the qdio layer will only deliver IRQs to the device driver if the QDIO_IRQ_DISABLED flag is cleared. zfcp manages this through qdio_start_irq() / qdio_stop_irq(). The initial state is DISABLED, and zfcp_qdio_open() schedules zfcp's IRQ tasklet once during startup to explicitly enable IRQ delivery.
2. the zfcp tasklet is initialized with tasklet_disable(), and only gets enabled once we open the qdio device.
When closing the qdio device, we must disable the tasklet _before_ disabling IRQ delivery (otherwise a concurrently running tasklet could re-enable IRQ delivery after we disabled it).
A final tasklet_kill() during teardown ensures that no lingering tasklet_schedule() is still accessing the tasklet structure.
--- drivers/s390/scsi/zfcp_qdio.c | 39 +++++++++++++++++++++++++++++++++++ drivers/s390/scsi/zfcp_qdio.h | 2 ++ 2 files changed, 41 insertions(+)
drivers/s390/scsi/zfcp_qdio.c +39 -0
@@ -131,6 +131,33 @@ static void zfcp_qdio_int_resp(struct ccw_device *cdev, unsigned int qdio_err,zfcp_erp_adapter_reopen (qdio->adapter,0 ,"qdires2" ); } +static void zfcp_qdio_irq_tasklet (struct tasklet_struct *tasklet) +{ +struct zfcp_qdio *qdio =from_tasklet (qdio, tasklet, irq_tasklet); +struct ccw_device *cdev = qdio->adapter->ccw_device; +unsigned int start, error; +int completed; + +/* Check the Response Queue, and kick off the Request Queue tasklet: */ + completed =qdio_get_next_buffers (cdev,0 , &start, &error); +if (completed <0 ) +return ; +if (completed >0 ) +zfcp_qdio_int_resp (cdev, error,0 , start, completed, + (unsigned long ) qdio); + +if (qdio_start_irq (cdev)) +/* More work pending: */ +tasklet_schedule (&qdio->irq_tasklet); +} + +static void zfcp_qdio_poll (struct ccw_device *cdev,unsigned long data) +{ +struct zfcp_qdio *qdio = (struct zfcp_qdio *) data; + +tasklet_schedule (&qdio->irq_tasklet); +} +static struct qdio_buffer_element *zfcp_qdio_sbal_chain (struct zfcp_qdio *qdio,struct zfcp_qdio_req *q_req) {
@@ -332,6 +359,8 @@ void zfcp_qdio_close(struct zfcp_qdio *qdio)wake_up (&qdio->req_q_wq); +tasklet_disable (&qdio->irq_tasklet); +qdio_stop_irq (adapter->ccw_device);qdio_shutdown (adapter->ccw_device,QDIO_FLAG_CLEANUP_USING_CLEAR );/* cleanup used outbound sbals */
@@ -387,6 +416,7 @@ int zfcp_qdio_open(struct zfcp_qdio *qdio) init_data.no_output_qs =1 ; init_data.input_handler = zfcp_qdio_int_resp; init_data.output_handler = zfcp_qdio_int_req; + init_data.irq_poll = zfcp_qdio_poll; init_data.int_parm = (unsigned long ) qdio; init_data.input_sbal_addr_array = input_sbals; init_data.output_sbal_addr_array = output_sbals;
@@ -433,6 +463,11 @@ int zfcp_qdio_open(struct zfcp_qdio *qdio)atomic_set (&qdio->req_q_free,QDIO_MAX_BUFFERS_PER_Q );atomic_or (ZFCP_STATUS_ADAPTER_QDIOUP , &qdio->adapter->status); +/* Enable processing for QDIO interrupts: */ +tasklet_enable (&qdio->irq_tasklet); +/* This results in a qdio_start_irq(): */ +tasklet_schedule (&qdio->irq_tasklet); +zfcp_qdio_shost_update (adapter, qdio);return 0 ;
@@ -450,6 +485,8 @@ void zfcp_qdio_destroy(struct zfcp_qdio *qdio)if (!qdio)return ; +tasklet_kill (&qdio->irq_tasklet); +if (qdio->adapter->ccw_device)qdio_free (qdio->adapter->ccw_device);
@@ -475,6 +512,8 @@ int zfcp_qdio_setup(struct zfcp_adapter *adapter)spin_lock_init (&qdio->req_q_lock);spin_lock_init (&qdio->stat_lock); +tasklet_setup (&qdio->irq_tasklet, zfcp_qdio_irq_tasklet); +tasklet_disable (&qdio->irq_tasklet); adapter->qdio = qdio;return 0 ;
drivers/s390/scsi/zfcp_qdio.h +2 -0
@@ -10,6 +10,7 @@#ifndef ZFCP_QDIO_H #define ZFCP_QDIO_H +#include <linux/interrupt.h> #include <asm/qdio.h> #define ZFCP_QDIO_SBALE_LEN PAGE_SIZE
@@ -44,6 +45,7 @@ struct zfcp_qdio {u64 req_q_util;atomic_t req_q_full;wait_queue_head_t req_q_wq; +struct tasklet_struct irq_tasklet;struct zfcp_adapter *adapter;u16 max_sbale_per_sbal;u16 max_sbale_per_req;
Signature · 1 line
2.26.2
PATCH 2/5 zfcp: remove orphaned function declarations
In reply to Benjamin BlockFrom: Vasily Gorbik <[email protected]>
drivers/s390/scsi/zfcp_ext.h: zfcp_sg_free_table - only declaration left after commit 58f3ead54752 ("scsi: zfcp: move SG table helper from aux to fc and make them static")
drivers/s390/scsi/zfcp_ext.h: zfcp_sg_setup_table - only declaration left after commit 58f3ead54752 ("scsi: zfcp: move SG table helper from aux to fc and make them static")
--- drivers/s390/scsi/zfcp_ext.h | 2 -- 1 file changed, 2 deletions(-)
drivers/s390/scsi/zfcp_ext.h +0 -2
@@ -20,8 +20,6 @@ extern struct zfcp_port *zfcp_get_port_by_wwpn(struct zfcp_adapter *, u64);extern struct zfcp_adapter *zfcp_adapter_enqueue (struct ccw_device *);extern struct zfcp_port *zfcp_port_enqueue (struct zfcp_adapter *,u64 ,u32 ,u32 ); -extern void zfcp_sg_free_table (struct scatterlist *,int ); -extern int zfcp_sg_setup_table (struct scatterlist *,int );extern void zfcp_adapter_release (struct kref *);extern void zfcp_adapter_unregister (struct zfcp_adapter *);
Signature · 1 line
2.26.2
PATCH 3/5 zfcp: clarify & assert the stat_lock locking in zfcp_qdio_send()
In reply to Benjamin BlockFrom: Julian Wiedmann <[email protected]>
Explain why the plain spin_lock() suffices in current code, even when the stat_lock is also used by zfcp_qdio_int_req() in tasklet context.
We could also promote the spin_lock() to a spin_lock_irqsave(), but that would just obfuscate the locking even further.
--- drivers/s390/scsi/zfcp_qdio.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/s390/scsi/zfcp_qdio.c +8 -0
@@ -10,6 +10,7 @@#define KMSG_COMPONENT "zfcp"#define pr_fmt (fmt) KMSG_COMPONENT ": " fmt +#include <linux/lockdep.h> #include <linux/slab.h> #include <linux/module.h> #include "zfcp_ext.h"
@@ -283,6 +284,13 @@ int zfcp_qdio_send(struct zfcp_qdio *qdio, struct zfcp_qdio_req *q_req)int retval;u8 sbal_number = q_req->sbal_number; +/* +* This should actually be a spin_lock_bh(stat_lock), to protect against +* zfcp_qdio_int_req() in tasklet context. +* But we can't do so (and are safe), as we always get called with IRQs +* disabled by spin_lock_irq[save](req_q_lock). +*/ +lockdep_assert_irqs_disabled ();spin_lock (&qdio->stat_lock);zfcp_qdio_account (qdio);spin_unlock (&qdio->stat_lock);
Signature · 1 line
2.26.2
PATCH 4/5 zfcp: process Version Change events
In reply to Benjamin BlockFrom: Julian Wiedmann <[email protected]>
Handle notifications for a concurrent change of the FCP Channel firmware.
Update the relevant user-visible fields to provide accurate data.
--- drivers/s390/scsi/zfcp_fsf.c | 16 ++++++++++++++++ drivers/s390/scsi/zfcp_fsf.h | 10 ++++++++++ 2 files changed, 26 insertions(+)
drivers/s390/scsi/zfcp_fsf.c +16 -0
@@ -242,6 +242,19 @@ static void zfcp_fsf_status_read_link_down(struct zfcp_fsf_req *req) } } +static void +zfcp_fsf_status_read_version_change (struct zfcp_adapter *adapter, +struct fsf_status_read_buffer *sr_buf) +{ +if (sr_buf->status_subtype ==FSF_STATUS_READ_SUB_LIC_CHANGE ) { +u32 version = sr_buf->payload.version_change.current_version; + +WRITE_ONCE (adapter->fsf_lic_version, version); +snprintf (fc_host_firmware_version (adapter->scsi_host), +FC_VERSION_STRING_SIZE ,"%#08x" , version); + } +} +static void zfcp_fsf_status_read_handler (struct zfcp_fsf_req *req) {struct zfcp_adapter *adapter = req->adapter;
@@ -300,6 +313,9 @@ static void zfcp_fsf_status_read_handler(struct zfcp_fsf_req *req)case FSF_STATUS_READ_FEATURE_UPDATE_ALERT : adapter->adapter_features = sr_buf->payload.word[0 ];break ; +case FSF_STATUS_READ_VERSION_CHANGE : +zfcp_fsf_status_read_version_change (adapter, sr_buf); +break ; }mempool_free (virt_to_page (sr_buf), adapter->pool.sr_data);
drivers/s390/scsi/zfcp_fsf.h +10 -0
@@ -134,6 +134,7 @@#define FSF_STATUS_READ_LINK_UP 0x00000006#define FSF_STATUS_READ_NOTIFICATION_LOST 0x00000009#define FSF_STATUS_READ_FEATURE_UPDATE_ALERT 0x0000000C +#define FSF_STATUS_READ_VERSION_CHANGE 0x0000000D/* status subtypes for link down */ #define FSF_STATUS_READ_SUB_NO_PHYSICAL_LINK 0x00000000
@@ -143,6 +144,9 @@/* status subtypes for unsolicited status notification lost */ #define FSF_STATUS_READ_SUB_INCOMING_ELS 0x00000001 +/* status subtypes for version change */ +#define FSF_STATUS_READ_SUB_LIC_CHANGE 0x00000001 +/* topologie that is detected by the adapter */ #define FSF_TOPO_P2P 0x00000001#define FSF_TOPO_FABRIC 0x00000002
@@ -226,6 +230,11 @@ struct fsf_link_down_info {u8 vendor_specific_code; } __attribute__ ((packed)); +struct fsf_version_change { +u32 current_version; +u32 previous_version; +} __packed; +struct fsf_status_read_buffer {u32 status_type;u32 status_subtype;
@@ -242,6 +251,7 @@ struct fsf_status_read_buffer {u32 word[FSF_STATUS_READ_PAYLOAD_SIZE /sizeof (u32)];struct fsf_link_down_info link_down_info;struct fsf_bit_error_payload bit_error; +struct fsf_version_change version_change; } payload; } __attribute__ ((packed));
Signature · 1 line
2.26.2
PATCH 5/5 zfcp: handle event-lost notification for Version Change events
In reply to Benjamin BlockFrom: Julian Wiedmann <[email protected]>
As recovery for a lost Version Change event, trigger an Exchange Config Data cmd to retrieve the current FW version.
Doing so requires process context (as eg. zfcp_qdio_sbal_get() might need to sleep), so defer from tasklet context into a work item.
--- drivers/s390/scsi/zfcp_aux.c | 11 +++++++++++ drivers/s390/scsi/zfcp_def.h | 1 + drivers/s390/scsi/zfcp_fsf.c | 3 +++ drivers/s390/scsi/zfcp_fsf.h | 1 + 4 files changed, 16 insertions(+)
drivers/s390/scsi/zfcp_aux.c +11 -0
@@ -292,6 +292,14 @@ static void _zfcp_status_read_scheduler(struct work_struct *work) stat_work)); } +static void zfcp_version_change_lost_work (struct work_struct *work) +{ + struct zfcp_adapter *adapter = container_of(work, struct zfcp_adapter, + version_change_lost_work); + +zfcp_fsf_exchange_config_data_sync (adapter->qdio,NULL ); +} +static void zfcp_print_sl (struct seq_file *m,struct service_level *sl) { struct zfcp_adapter *adapter =
@@ -353,6 +361,8 @@ struct zfcp_adapter *zfcp_adapter_enqueue(struct ccw_device *ccw_device)INIT_WORK (&adapter->stat_work, _zfcp_status_read_scheduler);INIT_DELAYED_WORK (&adapter->scan_work, zfcp_fc_scan_ports);INIT_WORK (&adapter->ns_up_work, zfcp_fc_sym_name_update); +INIT_WORK (&adapter->version_change_lost_work, + zfcp_version_change_lost_work); adapter->next_port_scan = jiffies;
@@ -429,6 +439,7 @@ void zfcp_adapter_unregister(struct zfcp_adapter *adapter)cancel_delayed_work_sync (&adapter->scan_work);cancel_work_sync (&adapter->stat_work);cancel_work_sync (&adapter->ns_up_work); +cancel_work_sync (&adapter->version_change_lost_work);zfcp_destroy_adapter_work_queue (adapter);zfcp_fc_wka_ports_force_offline (adapter->gs);
drivers/s390/scsi/zfcp_def.h +1 -0
@@ -201,6 +201,7 @@ struct zfcp_adapter {struct zfcp_fc_events events;unsigned long next_port_scan;struct zfcp_diag_adapter *diagnostics; +struct work_struct version_change_lost_work; };struct zfcp_port {
drivers/s390/scsi/zfcp_fsf.c +3 -0
@@ -309,6 +309,9 @@ static void zfcp_fsf_status_read_handler(struct zfcp_fsf_req *req)case FSF_STATUS_READ_NOTIFICATION_LOST :if (sr_buf->status_subtype &FSF_STATUS_READ_SUB_INCOMING_ELS )zfcp_fc_conditional_port_scan (adapter); +if (sr_buf->status_subtype &FSF_STATUS_READ_SUB_VERSION_CHANGE ) +queue_work (adapter->work_queue, + &adapter->version_change_lost_work);break ;case FSF_STATUS_READ_FEATURE_UPDATE_ALERT : adapter->adapter_features = sr_buf->payload.word[0 ];
drivers/s390/scsi/zfcp_fsf.h +1 -0
@@ -143,6 +143,7 @@/* status subtypes for unsolicited status notification lost */ #define FSF_STATUS_READ_SUB_INCOMING_ELS 0x00000001 +#define FSF_STATUS_READ_SUB_VERSION_CHANGE 0x00000100/* status subtypes for version change */ #define FSF_STATUS_READ_SUB_LIC_CHANGE 0x00000001
Signature · 1 line
2.26.2
Martin K. Petersen
AppliedBenjamin,
Quoted 1 line
here is a series of changes for our zfcp driver for 5.11.
Applied to 5.11/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Martin K. Petersen
AppliedQuoted 8 lines · Benjamin Block
here is a series of changes for our zfcp driver for 5.11.
Other than 2 smaller cleanups and clarifications for maintainability we have a refactoring of how zfcp uses s390's qdio layer, and we have a small feature improving our handling of out-of-band version changes to our adapters (or firmware).
[...]
Applied to 5.11/scsi-queue, thanks!
[1/5] scsi: zfcp: Lift Input Queue tasklet from qdio
https://git.kernel.org/mkp/scsi/c/0b524abc2dd1 [2/5] scsi: zfcp: Remove orphaned function declarations https://git.kernel.org/mkp/scsi/c/84e7b4169f94 [3/5] scsi: zfcp: Clarify & assert the stat_lock locking in zfcp_qdio_send() https://git.kernel.org/mkp/scsi/c/efd321768d2e [4/5] scsi: zfcp: Process Version Change events https://git.kernel.org/mkp/scsi/c/a6c37abe6988 [5/5] scsi: zfcp: Handle event-lost notification for Version Change events https://git.kernel.org/mkp/scsi/c/d90196317484
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering