Add support for storage element depopulation
Subsystems (7)
Damien Le Moal
maintainerv10 cover Add support for storage element depopulation
Jens,
These patches define a new set of block device operations for generically using from the block layer the storage element depopulation feature of zoned block devices. Support for this feature is added to the SCSI disk driver and an emulation of this feature added to the zloop driver.
The management operations can be accessed using block layer API, which is intended for file systems (e.g. zonefs and XFS), as well as using ioctls for users using zoned disks directly from user space.
Changes from v9:
- Addressed Sashiko comments on patch 6. This change redesigns the ordered zone append handling to simplify the code and avoid locking difficulties.
- Addressed Sashiko comments on patch 8.
Changes from v8:
- Addressed Sashiko comment on patch 7 by adding patch 6. Patch 7 is unmodified.
- Addressed Sashiko comments on patch 8 and patch 12
- Added review tags to unmodified pacthes
Changes from v7:
A significant amount of changes in this version to address most of the Sashiko review comments and to simplify the code. In more details, the changes are:
- Added patches 3, 6, and 9.
- Dropped former patch 5 as the use of irq-disabling spinlocks in zloop is not necessary.
- Reworked patch 4 by migrating the idepop operation wait loops to the device drivers. This simplifies the implementation in zloop without making the sd driver implementation more difficult. Also fixed problems with errors during restoration and zone write plugs state.
- Reworked patch 7 and 11 to be consistent with the changes in patch 4.
Changes from v6:
- Addressed the nit pointed out by Christoph in patch 4
- Added review tags
Changes from v5:
- Rebased on latest block/for-next
- Addressed Christoph's comments on patch 3
- Added patch 5 to address a potential deadlock in zloop (signaled by sashiko)
- Reworked the restoration of storage elements in patch 6 to avoid an inconsistent state of storage elements in case of error. Also added protection against disk removal during storage element removal and restoration execution.
- Added review tags
Changes from v4:
- Rebased on latest block/for-next
- Added review tags to patch 1 and 2
- Reworked patch 3, 4 and 5 to address Christoph's and sashiko comments
Changes from v3:
- Reworked patch 1 to only provide the disk_zone_is_offline() helper function. Failing of read operations to offline zones is handled directly by the zloop driver and the new patch 7 is added to handle failing early read operations to offline zones of ZBC SCSI disks.
- Added patch 2
- Addressed further valid comments from Sashiko
The biggest changes are in patch 2 to save the number of storage elements of a disk in struct gendisk and to serialize the execution of the storage element management operations using the zone revalidation mutex.
- Rebased on latest block/for-next
Changes from v2:
- Addressed further valid comments from Sashiko
- Rebased on latest block/for-next
Changes from v1:
- Addressed the many valid comments that Sashiko had
- Rebased on latest block/for-next
Damien Le Moal (13):
block: introduce disk_zone_is_offline()
block: refactor blk_revalidate_disk_zones()
block: allow dropping all zone write plugs on revalidation
block: introduce storage element management
block: add storage element management ioctls
zloop: fix ordered zone append handling
zloop: fix zone reporting consistency
zloop: add storage element emulation
zloop: add degrade_element control command
scsi: scsi_proto: refine the definition of storage element health
scsi: sd_zbc: always revalidate zones for disks supporting head depopulation
scsi: sd_zbc: define storage element management operations
scsi: sd: fail read commands to offline zones early
.../admin-guide/blockdev/zoned_loop.rst | 33 +- block/blk-zoned.c | 450 ++++++++- block/blk.h | 14 + block/genhd.c | 4 + block/ioctl.c | 4 + drivers/block/zloop.c | 924 ++++++++++++++++-- drivers/scsi/sd.c | 23 +- drivers/scsi/sd.h | 5 + drivers/scsi/sd_zbc.c | 538 +++++++++- include/linux/blkdev.h | 64 ++ include/scsi/scsi_proto.h | 6 +- include/uapi/linux/blkzoned.h | 106 ++ include/uapi/linux/fs.h | 1 + 13 files changed, 2052 insertions(+), 120 deletions(-)
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 1/13 block: introduce disk_zone_is_offline()
Introduce the helper function disk_zone_is_offline() for testing if a zone of a gendisk has the offline condition. The zone to test is identified using a sector number that must belong to the target zone to check.
--- block/blk-zoned.c | 18 ++++++++++++++++++ include/linux/blkdev.h | 9 +++++++++ 2 files changed, 27 insertions(+)
block/blk-zoned.c +18 -0
@@ -292,6 +292,24 @@ static bool disk_zone_is_seq(struct gendisk *disk, sector_t sector)return !blk_zstate_is_conv (zs); } +/** +* disk_zone_is_offline - check if a sector belongs to an offline zone +* @disk: gendisk to check +* @sector: sector number +* +* Check if @sector on @disk is contained in an offline zone. +* This function can be used only when the bdev queue was entered (BIOs +* submitted but not yet completed) or when holding zone_revalidate_mutex. +*/ +bool disk_zone_is_offline (struct gendisk *disk,sector_t sector) +{ +if (!blk_queue_is_zoned (disk->queue)) +return false; + +return disk_zone_get_cond (disk, sector) ==BLK_ZONE_COND_OFFLINE ; +} +EXPORT_SYMBOL_GPL (disk_zone_is_offline); +/** * bdev_zone_is_seq - check if a sector belongs to a sequential write zone * @bdev: block device to check
include/linux/blkdev.h +9 -0
@@ -940,6 +940,9 @@ static inline unsigned int disk_zone_capacity(struct gendisk *disk,return disk->last_zone_capacity;return disk->zone_capacity; } + +bool disk_zone_is_offline (struct gendisk *disk,sector_t sector); +static inline unsigned int bdev_zone_capacity (struct block_device *bdev,sector_t pos) {
@@ -954,6 +957,12 @@ static inline unsigned int disk_nr_zones(struct gendisk *disk)return 0 ; } +static inline bool disk_zone_is_offline (struct gendisk *disk, +sector_t sector) +{ +return false; +} +static inline bool bdev_zone_is_seq (struct block_device *bdev,sector_t sector) {return false;
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 2/13 block: refactor blk_revalidate_disk_zones()
In reply to Damien Le MoalSplit out most of the code of blk_revalidate_disk_zones() into the internal function disk_revalidate_zones(). This will allow calling this new helper together with other code under the zone revalidation mutex. No functional change.
--- block/blk-zoned.c | 63 +++++++++++++++++++++++++---------------------- 1 file changed, 34 insertions(+), 29 deletions(-)
block/blk-zoned.c +34 -29
@@ -2552,20 +2552,7 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,return ret; } -/** -* blk_revalidate_disk_zones - (re)allocate and initialize zone write plugs -* @disk: Target disk -* -* Helper function for low-level device drivers to check, (re) allocate and -* initialize resources used for managing zoned disks. This function should -* normally be called by blk-mq based drivers when a zoned gendisk is probed -* and when the zone configuration of the gendisk changes (e.g. after a format). -* Before calling this function, the device driver must already have set the -* device zone size (chunk_sector limit) and the max zone append limit. -* BIO based drivers can also use this function as long as the device queue -* can be safely frozen. -*/ -int blk_revalidate_disk_zones (struct gendisk *disk) +static int disk_revalidate_zones (struct gendisk *disk) {struct blk_revalidate_zone_args args = { };struct blk_report_zones_args rep_args = {
@@ -2576,19 +2563,10 @@ int blk_revalidate_disk_zones(struct gendisk *disk)int retries =2 ;int ret; -if (WARN_ON_ONCE (!blk_queue_is_zoned (disk->queue))) -return -EIO ; - -/* -* Serialize calls to this function so that we can safely look at and -* eventually change the disk zone information. -*/ -mutex_lock (&disk->zone_revalidate_mutex); - again: ret =disk_revalidate_capacity (disk, &args);if (ret) - goto unlock; +return ret;/* * Allocate zone resources if they are needed and we have not done
@@ -2601,14 +2579,14 @@ int blk_revalidate_disk_zones(struct gendisk *disk) ret =disk_alloc_zone_resources (disk, args.capacity);if (ret) {memalloc_noio_restore (noio_flag); - goto unlock; +return ret; } } ret =disk_init_revalidate_args (disk, &args);if (ret) {memalloc_noio_restore (noio_flag); - goto unlock; +return ret; } ret = disk->fops->report_zones (disk,0 ,UINT_MAX , &rep_args);
@@ -2625,8 +2603,6 @@ int blk_revalidate_disk_zones(struct gendisk *disk)if (ret) goto free_args; -mutex_unlock (&disk->zone_revalidate_mutex); -return 0 ; free_args:
@@ -2642,7 +2618,36 @@ int blk_revalidate_disk_zones(struct gendisk *disk) }pr_warn ("%s: failed to revalidate zones\n" , disk->disk_name); -unlock: + +return ret; +} + +/** +* blk_revalidate_disk_zones - (re)allocate and initialize zone write plugs +* @disk: Target disk +* +* Helper function for low-level device drivers to check, (re) allocate and +* initialize resources used for managing zoned disks. This function should +* normally be called by blk-mq based drivers when a zoned gendisk is probed +* and when the zone configuration of the gendisk changes (e.g. after a format). +* Before calling this function, the device driver must already have set the +* device zone size (chunk_sector limit) and the max zone append limit. +* BIO based drivers can also use this function as long as the device queue +* can be safely frozen. +*/ +int blk_revalidate_disk_zones (struct gendisk *disk) +{ +int ret; + +if (WARN_ON_ONCE (!blk_queue_is_zoned (disk->queue))) +return -EIO ; + +/* +* Serialize calls to this function so that we can safely look at and +* eventually change the disk zone information. +*/ +mutex_lock (&disk->zone_revalidate_mutex); + ret =disk_revalidate_zones (disk);mutex_unlock (&disk->zone_revalidate_mutex);return ret;
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 3/13 block: allow dropping all zone write plugs on revalidation
In reply to Damien Le MoalIn preparation for supporting storage element depopulation, modify the function disk_revalidate_zones() to take an additional boolean parameter drop_all_zwplugs to force removing all zone write plugs from a disk hash table. Removing all zone write plugs is done in the same manner as for a capacity change, using the function disk_drop_zone_wplug() applied to all hashed zone write plugs.
Calls to disk_revalidate_zones() with the argument drop_all_zwplugs set to true will be used when restoring depopulated storage elements as such operation clears all sequential zones, modifying them to be empty.
--- block/blk-zoned.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
block/blk-zoned.c +5 -2
@@ -2552,7 +2552,7 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,return ret; } -static int disk_revalidate_zones (struct gendisk *disk) +static int disk_revalidate_zones (struct gendisk *disk,bool drop_all_zwplugs) {struct blk_revalidate_zone_args args = { };struct blk_report_zones_args rep_args = {
@@ -2563,6 +2563,9 @@ static int disk_revalidate_zones(struct gendisk *disk)int retries =2 ;int ret; +if (drop_all_zwplugs) +disk_for_all_zone_wplugs (disk, disk_drop_zone_wplug,NULL ); + again: ret =disk_revalidate_capacity (disk, &args);if (ret)
@@ -2647,7 +2650,7 @@ int blk_revalidate_disk_zones(struct gendisk *disk)* eventually change the disk zone information. */ mutex_lock (&disk->zone_revalidate_mutex); - ret =disk_revalidate_zones (disk); + ret =disk_revalidate_zones (disk, false);mutex_unlock (&disk->zone_revalidate_mutex);return ret;
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 4/13 block: introduce storage element management
In reply to Damien Le MoalRecent SCSI (SBC) and ATA (ACS) standards define the storage element depopulation feature. This feature is intended for managing hard-disks heads, either combined read-write heads or pairs of read and write heads, allowing to keep disks with defective heads longer in production by allowing "depopulating" (removing) defective heads.
This feature comes in two different flavors:
- A destructive version which removes a head and reformats the disk at a lower capacity point, restoring a fully functional contiguous LBA address space.
- A data preserving version restricted to host-managed zoned disks, which marks the zones served by a removed head as offline or read-only.
In preparation for using the data-preserving flavor of the depopulation feature in file systems natively supporting zoned block devices, introduce a set of storage element management operations and functions to define generic calls into block device drivers for managing storage elements.
The set of operations is defined with struct blk_se_ops and includes three operations:
- report_elements: get information on a device storage elements state reported with the new struct blk_se.
- remove_element: depopulate a defective storage element, blocking until the operation completes.
- restore_elements: restore depopulated storage elements, blocking until the operation completes.
Each operation is called from the functions bdev_report_storage_elements(), bdev_remove_storage_element() and bdev_restore_storage_elements() respectively.
Removing a healthy storage element from a device is possible and useful for testing. The storage element restoration operation restore_elements allows repopulating such healthy element. Repopulating defective storage elements is generally not allowed by devices.
The remove_element and restore_elements operations are executed under the zone revalidation lock so that they are serialized against each other and also serialized against user initiated device revalidation, thus allowing to always have a consistent view of the device zone conditions after executing these operations. In case of failure, zone revalidation is still executed, which may give an unstable view of the zone states. In such case, stable zone states will be recovered with zone access error handling or later zone revalidation.
The device drivers of zoned block devices can indicate support for the storage element depopulation feature by specifying the storage element management operations with the se_ops field of struct block_device_operations.
--- block/blk-zoned.c | 205 ++++++++++++++++++++++++++++++++++ block/blk.h | 5 + block/genhd.c | 4 + include/linux/blkdev.h | 55 +++++++++ include/uapi/linux/blkzoned.h | 65 +++++++++++ 5 files changed, 334 insertions(+)
block/blk-zoned.c +205 -0
@@ -2736,3 +2736,208 @@ int queue_zone_wplugs_show(void *data, struct seq_file *m) }#endif + +/** +* bdev_report_storage_elements - report the storage elements of a zoned +* block device +* @bdev: the zoned block device to check +* @elements: array of storage element descriptors to fill +* @nr_elements: maximum size of @elements +* +* Fill at most @nr_elements storage element descriptors in the array +* @elements. The number of storage elements filled in the array is returned +* using @nr_elements. If @elements is NULL, only @nr_elements is returned to +* indicate the maximum number of storage elements of @bdev. This function may +* sleep. +* +* Returns 0 on success and a negative error code on failure. +*/ +int bdev_report_storage_elements (struct block_device *bdev, +struct blk_se *elements, +unsigned int *nr_elements) +{ +struct gendisk *disk = bdev->bd_disk; + +if (test_bit (GD_DEAD , &disk->state)) +return -ENODEV ; + +if (!blk_queue_is_zoned (disk->queue) || !disk->nr_storage_elements) +return -EOPNOTSUPP ; + +if (!nr_elements) +return -EINVAL ; + +if (!elements) { + *nr_elements = disk->nr_storage_elements; +return 0 ; + } + + *nr_elements =min (*nr_elements, disk->nr_storage_elements); +if (!*nr_elements) +return -EINVAL ; + +return disk->fops->se_ops->report_elements (disk, elements, + nr_elements); +} +EXPORT_SYMBOL_GPL (bdev_report_storage_elements); + +/** +* bdev_remove_storage_element - Remove (depopulate) a storage element of a +* zoned block device +* @bdev: block device to operate on +* @element_id: ID of the storage element to remove +* +* Remove (depopulate) the storage element identified by @element_id from the +* block device @bdev. The caller is responsible for taking care of any +* necessary dirty page and device write cache flush and invalidation of cached +* data. This function will block until the storage element is removed or an +* error occurs. +* +* Returns 0 on success and a negative error code on failure. +*/ +int bdev_remove_storage_element (struct block_device *bdev, +unsigned int element_id) +{ +struct gendisk *disk = bdev->bd_disk; +unsigned int memflags; +int ret, err; + +if (test_bit (GD_DEAD , &disk->state)) +return -ENODEV ; + +if (!blk_queue_is_zoned (disk->queue) || !disk->nr_storage_elements) +return -EOPNOTSUPP ; + +/* Zero is not a valid storage element ID. */ +if (!element_id) +return -EINVAL ; + +/* +* Serialize against concurrent revalidation and concurrent storage +* element management operations so that we can see consistent zone +* condition changes. +*/ +mutex_lock (&disk->zone_revalidate_mutex); + +/* +* Freeze and unfreeze the queue to flush any outstanding command. +* The caller is responsible for not queuing up more I/Os by higher +* level means. +*/ + memflags =blk_mq_freeze_queue (disk->queue); +blk_mq_unfreeze_queue (disk->queue, memflags); + +/* Initiate the storage element removal and wait for its completion. */ + ret = disk->fops->se_ops->remove_element (disk, element_id); + +/* +* Always revalidate zones, even in the case of an error as we do not +* know what condition the zones of the block device are in. +*/ + err =disk_revalidate_zones (disk, false); +if (err && !ret) + ret = err; + +mutex_unlock (&disk->zone_revalidate_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (bdev_remove_storage_element); + +/** +* bdev_restore_storage_elements - Restore all depopulated storage elements of +* a zoned block device +* @bdev: block device to operate on +* +* Restore all storage elements of @bdev that have been depopulated. Not all +* elements may be restored by this operation. This operation is destructive and +* will result in all sequential zones to be transitioned to the empty condition. +* The caller is responsible for taking care of any necessary invalidation of +* cached data. This function will block until the storage elements are restored +* or an error occurs. +* +* Returns 0 on success and a negative error code on failure. +*/ +int bdev_restore_storage_elements (struct block_device *bdev) +{ +struct gendisk *disk = bdev->bd_disk; +unsigned int memflags; +int ret, err; + +if (test_bit (GD_DEAD , &disk->state)) +return -ENODEV ; + +if (!blk_queue_is_zoned (disk->queue) || !disk->nr_storage_elements) +return -EOPNOTSUPP ; + +/* +* Serialize against concurrent revalidation and concurrent storage +* element management operations so that we can see consistent zone +* condition changes. +*/ +mutex_lock (&disk->zone_revalidate_mutex); + +/* +* Freeze and unfreeze the queue to flush any outstanding commands. +* The caller is responsible for not queuing up more I/O by higher +* level means. +*/ + memflags =blk_mq_freeze_queue (disk->queue); +blk_mq_unfreeze_queue (disk->queue, memflags); + +/* +* Initiate the storage elements restoration and wait for its +* completion. +*/ + ret = disk->fops->se_ops->restore_elements (disk); + +/* +* Always revalidate zones, even in the case of an error as we do not +* know what condition the zones of the block device are in. Note that +* since all sequential zones should have been transitioned to the +* empty condition, we can drop all zone write plugs. If an error +* occurred and not all sequential zones are empty, the revalidation +* will reinstate the write plugs for these zones. +*/ + err =disk_revalidate_zones (disk, true); +if (err && !ret) + ret = err; + +mutex_unlock (&disk->zone_revalidate_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (bdev_restore_storage_elements); + +int disk_init_storage_element_mgmt (struct gendisk *disk) +{ +int ret; + +/* +* Storage element operations are for zoned disks only and are +* optional... +*/ +if (!blk_queue_is_zoned (disk->queue) || !disk->fops->se_ops) +return 0 ; + +/* ...but if defined, all operation callbacks must be defined. */ +if (!disk->fops->se_ops->report_elements || + !disk->fops->se_ops->remove_element || + !disk->fops->se_ops->restore_elements) { +pr_err ("%s: invalid storage element operations\n" , + disk->disk_name); +return -EINVAL ; + } + +/* +* Get the number of storage elements of the disk. If this fails, +* do not report an error as the failure may be because the zoned disk +* does not support storage element management. +*/ + ret = disk->fops->se_ops->report_elements (disk,NULL , + &disk->nr_storage_elements); +if (ret) + disk->nr_storage_elements =0 ; + +return 0 ; +}
block/blk.h +5 -0
@@ -543,6 +543,7 @@ static inline void ioc_clear_queue(struct request_queue *q)#endif /* CONFIG_BLK_ICQ */ #ifdef CONFIG_BLK_DEV_ZONED +int disk_init_storage_element_mgmt (struct gendisk *disk);void disk_init_zone_resources (struct gendisk *disk);void disk_release_zone_resources (struct gendisk *disk);static inline bool bio_zone_write_plugging (struct bio *bio)
@@ -595,6 +596,10 @@ int blkdev_zone_mgmt_ioctl(struct block_device *bdev, blk_mode_t mode, unsigned int cmd, unsigned long arg);bool bdev_zone_mgmt_allowed (struct block_device *bdev,sector_t sector);#else /* CONFIG_BLK_DEV_ZONED */ +static inline int disk_init_storage_element_mgmt (struct gendisk *disk) +{ +return 0 ; +}static inline void disk_init_zone_resources (struct gendisk *disk) { }
block/genhd.c +4 -0
@@ -435,6 +435,10 @@ static int __add_disk(struct device *parent, struct gendisk *disk,if (WARN_ON_ONCE (bdev_nr_sectors (disk->part0) >BLK_DEV_MAX_SECTORS ))return -EINVAL ; + ret =disk_init_storage_element_mgmt (disk); +if (ret) +return ret; +if (queue_is_mq (disk->queue)) {/* * ->submit_bio and ->poll_bio are bypassed for blk-mq drivers.
include/linux/blkdev.h +55 -0
85 lines not on this page · raw message
include/uapi/linux/blkzoned.h +65 -0
69 lines not on this page · raw message
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 5/13 block: add storage element management ioctls
In reply to Damien Le MoalDefine the new ioctl commands BLKREPORTSTORELEMS, BLKREMOVESTORELEM, and BLKRESTORESTORELEMS to provide users accessing zoned block devices directly with an interface to the storage elements management operations of the block layer.
The ioctl BLKREPORTSTORELEMS allows to get the total number of storage elements of a zoned block device and to get an array of storage element descriptors (struct blk_se) describing the current state of the device storage elements.
The ioctl BLKREMOVESTORELEM can be used to remove (depopulate) a storage element, in particular storage elements that have a degraded status (i.e. BLK_SE_STS_DEGRADED). But healthy elements (BLK_SE_STS_OK) can also be removed, and later restored.
Finally, the BLKRESTORESTORELEMS ioctl interfaces with bdev_restore_storage_elements() to restore the depopulated storage elements of a zoned device.
--- block/blk-zoned.c | 163 ++++++++++++++++++++++++++++++++++ block/blk.h | 9 ++ block/ioctl.c | 4 + include/uapi/linux/blkzoned.h | 41 +++++++++ include/uapi/linux/fs.h | 1 + 5 files changed, 218 insertions(+)
block/blk-zoned.c +163 -0
@@ -18,6 +18,7 @@#include <linux/mempool.h> #include <linux/kthread.h> #include <linux/freezer.h> +#include <linux/uaccess.h> #include <trace/events/block.h>
@@ -2781,6 +2782,53 @@ int bdev_report_storage_elements(struct block_device *bdev, }EXPORT_SYMBOL_GPL (bdev_report_storage_elements); +static int blkdev_ioctl_report_storage_elements (struct block_device *bdev, +void __user *argp) +{ +struct gendisk *disk = bdev->bd_disk; +struct blk_se_report *rep; +int ret =0 ; + +/* +* Allocate the report header with an array of elements, assuming that +* we need the maximum number of storage elements. In practice, we may +* need less than that, but this potentially unnecessary preallocation +* simplifies the code. +*/ + rep =kzalloc_flex (*rep, elements, disk->nr_storage_elements); +if (!rep) +return -ENOMEM ; + +if (copy_from_user (rep, argp,sizeof (struct blk_se_report ))) { + ret = -EFAULT ; + goto free_rep; + } + +if (rep->reserved) { + ret = -EINVAL ; + goto free_rep; + } + + rep->nr_reported_elements = +min (rep->nr_elements, disk->nr_storage_elements); + rep->nr_elements = disk->nr_storage_elements; + +if (rep->nr_reported_elements) { + ret =bdev_report_storage_elements (bdev, rep->elements, + &rep->nr_reported_elements); +if (ret) + goto free_rep; + } + +if (copy_to_user (argp, rep, +struct_size (rep, elements, rep->nr_reported_elements))) + ret = -EFAULT ; + +free_rep: +kfree (rep); +return ret; +} +/** * bdev_remove_storage_element - Remove (depopulate) a storage element of a * zoned block device
@@ -2844,6 +2892,62 @@ int bdev_remove_storage_element(struct block_device *bdev, }EXPORT_SYMBOL_GPL (bdev_remove_storage_element); +static int blkdev_ioctl_remove_storage_element (struct block_device *bdev, +blk_mode_t mode, +unsigned int __user *argp) +{ +unsigned int element_id; +int ret; + +/* +* Note: we require an O_EXCL open here so that a user cannot trigger +* a storage element removal from user space under a mounted file system. +* An FS that wants to manage storage element can do so using the +* function bdev_remove_storage_element() from kernel context. +*/ +if (!(mode &BLK_OPEN_EXCL )) +return -EBADF ; +if (!(mode &BLK_OPEN_WRITE )) +return -EBADF ; +if (bdev_read_only (bdev)) +return -EPERM ; + +if (get_user (element_id, argp)) +return -EFAULT ; + +/* +* Sync the device dirty data, flush the volatile write cache, and +* invalidate all cached data so that reads do not return old data for +* zones that went offline. +* Since a user can only successfully write to sequential zones of a +* zoned devices using direct IOs, the sync should only handle dirty +* data from buffered writes to conventional zones, for devices that +* have such zones. +*/ + ret =sync_blockdev (bdev); +if (ret) +return ret; + +inode_lock (bdev->bd_mapping->host); +filemap_invalidate_lock (bdev->bd_mapping); + + ret =blkdev_issue_flush (bdev->bd_disk->part0); +if (ret) + goto unlock; + + ret =truncate_bdev_range (bdev, mode,0 ,bdev_nr_bytes (bdev) -1 ); +if (ret) + goto unlock; + + ret =bdev_remove_storage_element (bdev, element_id); + +unlock: +filemap_invalidate_unlock (bdev->bd_mapping); +inode_unlock (bdev->bd_mapping->host); + +return ret; +} +/** * bdev_restore_storage_elements - Restore all depopulated storage elements of * a zoned block device
@@ -2909,6 +3013,42 @@ int bdev_restore_storage_elements(struct block_device *bdev) }EXPORT_SYMBOL_GPL (bdev_restore_storage_elements); +static int blkdev_ioctl_restore_storage_elements (struct block_device *bdev, +blk_mode_t mode) +{ +int ret; + +/* +* Note: because restoring storage elements is a destructive operation +* equivalent to reformatting the device, we require an O_EXCL open here +* so that a user cannot trigger with another active user (e.g. a +* mounted file system). +*/ +if (!(mode &BLK_OPEN_EXCL )) +return -EBADF ; +if (!(mode &BLK_OPEN_WRITE )) +return -EBADF ; +if (bdev_read_only (bdev)) +return -EPERM ; + +inode_lock (bdev->bd_mapping->host); +filemap_invalidate_lock (bdev->bd_mapping); + +/* +* Storage element restoration is a destructive operation that will +* reset all zones. So invalidate all cached data that we may have so +* that old data cannot be read once the operation completes. +*/ + ret =truncate_bdev_range (bdev, mode,0 ,bdev_nr_bytes (bdev) -1 ); +if (!ret) + ret =bdev_restore_storage_elements (bdev); + +filemap_invalidate_unlock (bdev->bd_mapping); +inode_unlock (bdev->bd_mapping->host); + +return ret; +} +int disk_init_storage_element_mgmt (struct gendisk *disk) {int ret;
@@ -2941,3 +3081,26 @@ int disk_init_storage_element_mgmt(struct gendisk *disk)return 0 ; } + +int blk_ioctl_zone_storage_elements (struct block_device *bdev, +blk_mode_t mode,unsigned int cmd, +void __user *argp) +{ +struct gendisk *disk = bdev->bd_disk; + +if (!disk->nr_storage_elements) +return -ENOTTY ; + +switch (cmd) { +case BLKREPORTSTORELEMS : +return blkdev_ioctl_report_storage_elements (bdev, argp); +case BLKREMOVESTORELEM : +return blkdev_ioctl_remove_storage_element (bdev, mode, argp); +case BLKRESTORESTORELEMS : +return blkdev_ioctl_restore_storage_elements (bdev, mode); +default : +break ; + } + +return -ENOTTY ; +}
block/blk.h +9 -0
@@ -595,6 +595,9 @@ int blkdev_report_zones_ioctl(struct block_device *bdev, unsigned int cmd,int blkdev_zone_mgmt_ioctl (struct block_device *bdev,blk_mode_t mode,unsigned int cmd,unsigned long arg);bool bdev_zone_mgmt_allowed (struct block_device *bdev,sector_t sector); +int blk_ioctl_zone_storage_elements (struct block_device *bdev, +blk_mode_t mode,unsigned int cmd, +void __user *argp);#else /* CONFIG_BLK_DEV_ZONED */ static inline int disk_init_storage_element_mgmt (struct gendisk *disk) {
@@ -646,6 +649,12 @@ static inline bool bdev_zone_mgmt_allowed(struct block_device *bdev, {return false; } +static inline int blk_ioctl_zone_storage_elements (struct block_device *bdev, +blk_mode_t mode,unsigned int cmd, +void __user *argp) +{ +return -ENOTTY ; +}#endif /* CONFIG_BLK_DEV_ZONED */ struct block_device *bdev_alloc (struct gendisk *disk,u8 partno);
block/ioctl.c +4 -0
@@ -679,6 +679,10 @@ static int blkdev_common_ioctl(struct block_device *bdev, blk_mode_t mode,return put_uint (argp,bdev_zone_sectors (bdev));case BLKGETNRZONES :return put_uint (argp,bdev_nr_zones (bdev)); +case BLKREPORTSTORELEMS : +case BLKREMOVESTORELEM : +case BLKRESTORESTORELEMS : +return blk_ioctl_zone_storage_elements (bdev, mode, cmd, argp);case BLKROGET :return put_int (argp,bdev_read_only (bdev) !=0 );case BLKSSZGET :/* get block device logical block size */
include/uapi/linux/blkzoned.h +41 -0
45 lines not on this page · raw message
include/uapi/linux/fs.h +1 -0
@@ -309,6 +309,7 @@ struct file_attr {/* 130-136 and 142 are used by zoned block device ioctls (uapi/linux/blkzoned.h) */ /* 137-141 are used by blk-crypto ioctls (uapi/linux/blk-crypto.h) */ #define BLKTRACESETUP2 _IOWR(0x12, 142, struct blk_user_trace_setup2) +/* 143-145 are used by storage element management for zoned block devices. */ #define BMAP_IOCTL 1/* obsolete - kept for compatibility */ #define FIBMAP _IO(0x00,1)/* bmap access */
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 6/13 zloop: fix ordered zone append handling
In reply to Damien Le MoalFor oredered zone append commands, the function zloop_set_zone_append_sector() increments a zone write pointer without checking if the zone is already open (implicitly or explicitly). This check is done only when the command is later executed using a work item, and the zone is only then implicitly open if needed. This creates a short time window where a report zone can show an empty zone with a write pointer that is not equal to the zone start sector, which is incorrect.
Calling zloop_do_open_zone() in zloop_set_zone_append_sector() is required but introduces locking complexity as the zone mutex cannot be taken in the context of zloop_queue_rq(). Instead introduce the ordered workqueue append_workqueue to handle zone append commands when ordered zone append is enabled. This ensure that all zone append operations are executed in the order of there submission. This removes the need for setting the command sector in the context of zloop_queue_rq() and we can rely on the reguar command execution done by zloop_rw(). With this change, zloop_seq_write_prep() is simplified and zloop_set_zone_append_sector() is removed entirely.
This change does reduce the command execution concurrency and impact performance with ordered zone append enabled for file systems using zone append. This is however not a critical option and the default unordered zone append handling is unaffected.
--- drivers/block/zloop.c | 115 +++++++++++++++++------------------------- 1 file changed, 47 insertions(+), 68 deletions(-)
drivers/block/zloop.c +47 -68
@@ -129,6 +129,7 @@ struct zloop_device {struct gendisk *disk;struct workqueue_struct *workqueue; +struct workqueue_struct *append_workqueue;bool buffered_io;bool zone_append;bool ordered_zone_append;
@@ -590,32 +591,24 @@ static int zloop_seq_write_prep(struct zloop_cmd *cmd)* when submitted. */ if (is_append) { -/* -* If ordered zone append is in use, we already checked and set -* the target sector in zloop_queue_rq(). -*/ -if (!zlo->ordered_zone_append) { -if (zone->cond ==BLK_ZONE_COND_FULL || - zone->wp + nr_sectors > zone_end) { - ret = -EIO ; - goto out_unlock; - } - cmd->sector = zone->wp; - } - }else { -if (cmd->sector != zone->wp) { -if (zone->wp ==ULLONG_MAX ) -pr_err ("%pg: Zone %u: write to %s zone: sect %llu\n" , - zlo->disk->part0, zone_no, -blk_zone_cond_str (zone->cond), - cmd->sector); -else -pr_err ("%pg: Zone %u: unaligned write: sect %llu, wp %llu\n" , - zlo->disk->part0, zone_no, cmd->sector, - zone->wp); +if (zone->cond ==BLK_ZONE_COND_FULL || + zone->wp + nr_sectors > zone_end) { ret = -EIO ; goto out_unlock; } + cmd->sector = zone->wp; + }else if (cmd->sector != zone->wp) { +if (zone->wp ==ULLONG_MAX ) +pr_err ("%pg: Zone %u: write to %s zone: sect %llu\n" , + zlo->disk->part0, zone_no, +blk_zone_cond_str (zone->cond), + cmd->sector); +else +pr_err ("%pg: Zone %u: unaligned write: sect %llu, wp %llu\n" , + zlo->disk->part0, zone_no, cmd->sector, + zone->wp); + ret = -EIO ; + goto out_unlock; }/* Implicitly open the target zone. */
@@ -625,15 +618,13 @@ static int zloop_seq_write_prep(struct zloop_cmd *cmd) }/* -* Advance the write pointer, unless ordered zone append is in use. If -* the write fails, the write pointer position will be corrected when -* the next I/O starts execution. +* Advance the write pointer. If the write fails, the write pointer +* position will be corrected when the next I/O starts execution. */ -if (!is_append || !zlo->ordered_zone_append) { - zone->wp += nr_sectors; -if (zone->wp == zone_end) -zloop_mark_full (zlo, zone); - } + zone->wp += nr_sectors; +if (zone->wp == zone_end) +zloop_mark_full (zlo, zone); + out_unlock:spin_unlock (&zone->wp_lock);return ret;
@@ -885,58 +876,27 @@ static void zloop_complete_rq(struct request *rq)blk_mq_end_request (rq, sts); } -static bool zloop_set_zone_append_sector (struct request *rq) -{ -struct zloop_device *zlo = rq->q->queuedata; -unsigned int zone_no =rq_zone_no (rq); -struct zloop_zone *zone = &zlo->zones[zone_no]; -sector_t zone_end = zone->start + zlo->zone_capacity; -sector_t nr_sectors =blk_rq_sectors (rq); - -spin_lock (&zone->wp_lock); - -if (zone->cond ==BLK_ZONE_COND_FULL || - zone->wp + nr_sectors > zone_end) { -spin_unlock (&zone->wp_lock); -return false; - } - - rq->__sector = zone->wp; - zone->wp +=blk_rq_sectors (rq); -if (zone->wp >= zone_end) -zloop_mark_full (zlo, zone); - -spin_unlock (&zone->wp_lock); - -return true; -} -static blk_status_t zloop_queue_rq (struct blk_mq_hw_ctx *hctx,const struct blk_mq_queue_data *bd) {struct request *rq = bd->rq;struct zloop_cmd *cmd =blk_mq_rq_to_pdu (rq);struct zloop_device *zlo = rq->q->queuedata; +struct workqueue_struct *wq;if (data_race (READ_ONCE (zlo->state)) == Zlo_deleting) { rq->rq_flags |=RQF_QUIET ;return BLK_STS_IOERR ; } -/* -* If we need to strongly order zone append operations, set the request -* sector to the zone write pointer location now instead of when the -* command work runs. -*/ -if (zlo->ordered_zone_append &&req_op (rq) ==REQ_OP_ZONE_APPEND ) { -if (!zloop_set_zone_append_sector (rq)) -return BLK_STS_IOERR ; - } -blk_mq_start_request (rq);INIT_WORK (&cmd->work, zloop_cmd_workfn); -queue_work (zlo->workqueue, &cmd->work); +if (zlo->ordered_zone_append &&req_op (rq) ==REQ_OP_ZONE_APPEND ) + wq = zlo->append_workqueue; +else + wq = zlo->workqueue; +queue_work (wq, &cmd->work);return BLK_STS_OK ; }
@@ -1029,6 +989,8 @@ static void zloop_free_disk(struct gendisk *disk)fput (zlo->data_dir);destroy_workqueue (zlo->workqueue); +if (zlo->append_workqueue) +destroy_workqueue (zlo->append_workqueue);kfree (zlo->base_dir);kvfree (zlo); }
@@ -1285,6 +1247,21 @@ static int zloop_ctl_add(struct zloop_options *opts) goto out_free_idr; } +/* +* Use an ordered workqueue for strongly ordered zone append operations +* so that we can keep processing the operations in the same order as +* they are submitted. +*/ +if (zlo->ordered_zone_append) { + zlo->append_workqueue = +alloc_ordered_workqueue ("zloop%d_append" ,WQ_FREEZABLE , + zlo->id); +if (!zlo->append_workqueue) { + ret = -ENOMEM ; + goto out_destroy_workqueue; + } + } +if (opts->base_dir) zlo->base_dir =kstrdup (opts->base_dir,GFP_KERNEL );else
@@ -1391,6 +1368,8 @@ static int zloop_ctl_add(struct zloop_options *opts)kfree (zlo->base_dir); out_destroy_workqueue:destroy_workqueue (zlo->workqueue); +if (zlo->append_workqueue) +destroy_workqueue (zlo->append_workqueue); out_free_idr:mutex_lock (&zloop_ctl_mutex);idr_remove (&zloop_index_idr, zlo->id);
Signature · 1 line
2.56.0
Re: [PATCH v10 06/13] zloop: fix ordered zone append handling
drivers/block/zloop.c static void zloop_complete_rq(struct request *rq) · line 899
Quoted 7 lines · Damien Le Moal
drivers/block/zloop.c +5
-1INIT_WORK (&cmd->work, zloop_cmd_workfn); -queue_work (zlo->workqueue, &cmd->work); +if (zlo->ordered_zone_append &&req_op (rq) ==REQ_OP_ZONE_APPEND ) + wq = zlo->append_workqueue; +else + wq = zlo->workqueue; +queue_work (wq, &cmd->work);
I'd do away with the wq variable here and just have separate queue_work calls, which would be a bit simpler.
Otherwise this looks good:
Damien Le Moal
maintainerPATCH v10 7/13 zloop: fix zone reporting consistency
In reply to Damien Le MoalIn zloop_report_zones(), filling the write pointer and condition of a zone descriptor separately outside of the zone wp_lock can lead to an inconsistent zone report with a write pointer potentially not being consistent with the condition being reported. Avoid such inconsistency by filling both fields while holding the zone wp_lock.
--- drivers/block/zloop.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/zloop.c +1 -1
@@ -952,8 +952,8 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector, blkz.len = zlo->zone_size;spin_lock (&zone->wp_lock); blkz.wp = zone->wp; -spin_unlock (&zone->wp_lock); blkz.cond = zone->cond; +spin_unlock (&zone->wp_lock);if (test_bit (ZLOOP_ZONE_CONV , &zone->flags)) { blkz.type =BLK_ZONE_TYPE_CONVENTIONAL ; blkz.capacity = zlo->zone_size;
Signature · 1 line
2.56.0
Re: [PATCH v10 07/13] zloop: fix zone reporting consistency
Quoted 5 lines · Damien Le Moal
In
zloop_report_zones(), filling the write pointer and condition of a zone descriptor separately outside of the zonewp_lockcan lead to an inconsistent zone report with a write pointer potentially not being consistent with the condition being reported. Avoid such inconsistency by filling both fields while holding the zonewp_lock.
Looks good:
Damien Le Moal
maintainerPATCH v10 8/13 zloop: add storage element emulation
In reply to Damien Le MoalEmulate the storage element depopulation feature of SMR disks in the zloop driver. This emulation is controlled using the new stor_elements option. This option can take several values:
- ZLOOP_STOR_ELEMENTS_NONE (0): no emulation (default)
- ZLOOP_STOR_ELEMENTS_RDWR (1): emulate all access storage elements (e.g. read+write heads)
- ZLOOP_STOR_ELEMENTS_PAIRS (2): emulate fractional access storage elements (e.g. pairs of read and write heads)
If enabled with the value 1 or 2, the number of storage elements, or of pairs of fractional access storage elements, is automatically calculated based on the number of zones of the device so that we have at least 2 and at most 32 storage elements or pairs of fractional access storage elements.
The mapping of zones to storage elements is defined simply as the modulo of a zone number with the number of storage elements for the all access storage elements emulation and with the number of storage element pairs for the fractional access storage element emulation case. E.g, removing a storage element from a set of 4 all access storage elements will offline 1 zone every 4 zones.
The storage element management operations are specified with the structure zloop_se_ops (struct blk_se_ops).
--- .../admin-guide/blockdev/zoned_loop.rst | 11 +- drivers/block/zloop.c | 731 +++++++++++++++++- 2 files changed, 715 insertions(+), 27 deletions(-)
Documentation/admin-guide/blockdev/zoned_loop.rst +10 -1
@@ -61,7 +61,7 @@ The options available for the add command can be listed by reading the /dev/zloop-control device:: $ cat /dev/zloop-control - add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache + add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache,stor_elements=%u remove id=%d In more details, the options that can be used with the "add" command are as
@@ -113,6 +113,15 @@ discard_write_cache Discard all data that was not explicitly persisted using a each zone file to the size recorded during the last flush operation. This simulates power fail events where uncommitted data is lost. +stor_elements Control storage element emulation. The default value is + 0, indicating no emulation. A value of 1 indicates that + storage elements are emulated as full access physical + elements, which is equivalent to combined read+write heads + of a hard-disk. A value of 2 enables storage element + emulation as fractional access physical storage elements, + which is equivalent to pairs of read and write heads of a + hard-disk. The health state of storage elements is not + preserved across a zloop device remove and add. =================== ========================================================= 3) Deleting a Zoned Device
drivers/block/zloop.c +705 -26
956 lines not on this page · raw message
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH v10 08/13] zloop: add storage element emulation
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [Medium] zloop: stale zone state on partial failure in zloop_execute_restore_elements() - [Medium] zloop…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [Medium] zloop: stale zone state on partial failure in zloop_execute_restore_elements()
- [Medium] zloop: zloop_rdwr_is_ok() blocks ZLOOP_ZONE_SEQ_ERROR recovery on full zones
- [Medium] zloop: race between storage element ops and zloop_ctl_remove()
--
commit 8e8c9e4aeea297b926b2b865b9d40a12a001bcbe
Author: Damien Le Moal <[email protected]>
zloop: add storage element emulation
This commit adds storage element depopulation emulation to the zloop driver, controlled via the new stor_elements option and implemented through zloop_se_ops.
Quoted 4 lines
drivers/block/zloop.c +0
-0
[ ... ]
Quoted 61 lines
drivers/block/zloop.c +50
-0@@ -876,6 +988,43 @@ static void zloop_complete_rq(struct request *rq)blk_mq_end_request (rq, sts); } +static bool zloop_rdwr_is_ok (struct zloop_device *zlo,struct request *rq) +{ +struct zloop_zone *zone = &zlo->zones[rq_zone_no (rq)]; +bool is_write =op_is_write (req_op (rq)); +enum blk_se_status sts =zloop_zone_element_status (zlo, zone, is_write); +bool ret; + +/* +* Fail read and write operations if the storage element serving the +* target zone is not healthy. +*/ +if (sts >=BLK_SE_STS_DEGRADED ) +return false; + +/* +* Check the zone condition: do not allow any read and write operations +* to offline zones, and allow only read operations to full and +* read-only zones. +*/ +spin_lock (&zone->wp_lock); +switch (zone->cond) { +case BLK_ZONE_COND_OFFLINE : + ret = false; +break ; +case BLK_ZONE_COND_FULL : +case BLK_ZONE_COND_READONLY : + ret = !is_write; +break ; +default : + ret = true; +break ; + } +spin_unlock (&zone->wp_lock); + +return ret; +} +static blk_status_t zloop_queue_rq (struct blk_mq_hw_ctx *hctx,const struct blk_mq_queue_data *bd) {@@ -889,6 +1038,30 @@ static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,return BLK_STS_IOERR ; } +if (zlo->nr_elements) { +/* +* Do not allow commands while the restoration of storage +* elements is in progress. +*/ +if (READ_ONCE (zlo->restore_in_progress)) +return BLK_STS_IOERR ; + +/* +* Check if we can accept data access commands based on the +* health state of the storage element serving the target zone. +*/ +switch (req_op (rq)) {
drivers/block/zloop.c static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx, · line 1063
Quoted 10 lines
drivers/block/zloop.c +10
-0+case REQ_OP_READ : +case REQ_OP_WRITE : +case REQ_OP_ZONE_APPEND : +if (!zloop_rdwr_is_ok (zlo, rq)) +return BLK_STS_IOERR ; +break ; +default : +break ; + } + }
[Severity: Medium]
Does rejecting writes to BLK_ZONE_COND_FULL zones in zloop_rdwr_is_ok() prevent ZLOOP_ZONE_SEQ_ERROR recovery?
zloop_seq_write_prep() advances zone->wp and calls zloop_set_zone_full() when zone->wp reaches zone_end before zloop_do_rw() writes to the backing file. If that write fails, zloop_complete_rq() sets ZLOOP_ZONE_SEQ_ERROR so the next I/O will resynchronize zone->wp and zone->cond from the file size in zloop_rw() -> zloop_update_seq_zone(). When zlo->nr_elements is non-zero, however, zloop_queue_rq() calls zloop_rdwr_is_ok() before queueing cmd->work, so subsequent writes or zone appends to that zone fail immediately with BLK_STS_IOERR and never reach zloop_rw().
[ ... ]
Quoted 25 lines
drivers/block/zloop.c +25
-0+static int zloop_execute_restore_elements (struct zloop_device *zlo) +{ +struct gendisk *disk = zlo->disk; +struct zloop_zone *zone; +struct blk_se *se; +enum blk_se_status sts; +unsigned int memflags, i; +int ret =0 ; + +/* Wait for all IOs to complete. */ + memflags =blk_mq_freeze_queue (disk->queue); +blk_mq_unfreeze_queue (disk->queue, memflags); + +/* First, truncate all files for sequential zones. */ +for (i =0 , zone = zlo->zones; i < zlo->nr_zones; i++, zone++) { + ret =zloop_restore_zone_file (zlo, zone); +if (ret) { +pr_err ("zloop%d: restore zone %u file failed %d\n" , + zlo->id, i, ret); +break ; + } + } + +/* +* Now, resetore the condition of all zones, unless we failed to
drivers/block/zloop.c static int zloop_report_zones(struct gendisk *disk, sector_t sector, · line 1439
Quoted 10 lines
drivers/block/zloop.c +10
-0+* truncate all files, in which case we report a restoration error +* without changing the zone conditions. +*/ +if (!ret) { +for (i =0 , zone = zlo->zones; i < zlo->nr_zones; i++, zone++) +zloop_restore_zone_cond (zlo, zone); + sts =BLK_SE_STS_OK ; + }else { + sts =BLK_SE_STS_RESTORE_ERROR ; + }
[Severity: Medium]
If zloop_restore_zone_file() fails on zone i, do sequential zones before i keep stale in-memory zone conditions and write pointers?
Those earlier sequential zones have already had their backing files truncated to 0 bytes by vfs_truncate(), but zloop_restore_zone_cond() is skipped when ret is non-zero and ZLOOP_ZONE_SEQ_ERROR is only set on the zone that failed. Earlier non-empty zones are left with 0-byte backing files while retaining their old zone->cond, non-zero zone->wp, and open_zones_lru_list accounting, which zloop_report_zones() will then report as-is.
Quoted 15 lines
drivers/block/zloop.c +15
-0+ +/* +* Update the status of all elements being restored based on the zone +* restoration result. +*/ +for (i =0 , se = zlo->elements; i < zlo->nr_elements; i++, se++) { +if (READ_ONCE (se->status) ==BLK_SE_STS_RESTORE_IN_PROGRESS ) +WRITE_ONCE (se->status, sts); + } + +return ret; +} + +static int zloop_restore_elements (struct gendisk *disk) +{
[ ... ]
Quoted 6 lines
drivers/block/zloop.c +6
-0+msleep (5000 ); + +mutex_lock (&zlo->stor_elements_lock); + +/* +* Re-check the device as its removal may have been initiated
drivers/block/zloop.c static int zloop_report_zones(struct gendisk *disk, sector_t sector, · line 1517
Quoted 10 lines
drivers/block/zloop.c +10
-0+* while we were sleeping. +*/ +if (test_bit (GD_DEAD , &disk->state) || +data_race (READ_ONCE (zlo->state)) == Zlo_deleting) { + ret = -ENODEV ; + goto clear_restore; + } + +/* Restore the elements. */ + ret =zloop_execute_restore_elements (zlo);
[Severity: Medium]
Can zloop_restore_elements() and zloop_remove_element() race with zloop_ctl_remove() right after the post-sleep check?
zloop_ctl_remove() sets zlo->state to Zlo_deleting under zloop_ctl_mutex without synchronizing with zlo->stor_elements_lock:
CPU0 CPU1zloop_restore_elements()
mutex_lock(&zlo->stor_elements_lock)
/* checks GD_DEAD and zlo->state */
zloop_ctl_remove()
mutex_lock_killable(&zloop_ctl_mutex)
WRITE_ONCE(zlo->state, Zlo_deleting)
mutex_unlock(&zloop_ctl_mutex)
del_gendisk(zlo->disk)
zloop_execute_restore_elements()
zloop_restore_zone_file()
vfs_truncate(..., 0)
zloop_forget_cache()
/* sees active zone->cond / wp */
vfs_truncate(..., old_wp - start)
zloop_restore_zone_cond()Because zloop_execute_restore_elements() truncates all sequential zone files to 0 in a first pass before resetting zone->cond and zone->wp in a second pass, a concurrent zloop_forget_cache() still sees active zone conditions and extends the truncated zone files back to old_wp.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Damien Le Moal
maintainerPATCH v10 9/13 zloop: add degrade_element control command
In reply to Damien Le MoalAllow users to mark storage elements of a zloop device as degraded using the new "degrade_element" control command. The element to degrade is indicated using the element_id option. Example:
echo "degrade_element id=0,element_id=2" > /dev/zloop-control
If the element ID identifies an all access storage element, both read and write operations targeting a zone served by the degraded element are failed. In the case of a partial access storage element, any write access to a zone whose write element is degraded will fail, and read accesses to a zone whose read element is degraded will result in an error.
The zoned_loop.rst documentation file is also updated to explain this new feature.
--- .../admin-guide/blockdev/zoned_loop.rst | 22 +++- drivers/block/zloop.c | 100 +++++++++++++++++- 2 files changed, 119 insertions(+), 3 deletions(-)
Documentation/admin-guide/blockdev/zoned_loop.rst +21 -1
@@ -63,6 +63,7 @@ The options available for the add command can be listed by reading the $ cat /dev/zloop-control add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache,stor_elements=%u remove id=%d + degrade_element id=%d,element_id=%u In more details, the options that can be used with the "add" command are as follows.
@@ -144,7 +145,26 @@ the zoned device will not be created. To fully delete a zoned device, after executing the remove operation, the device base directory containing the backing files of the device zones must be deleted. -4) Example +4) Degrading a storage element +------------------------------ + +When storage element emulation is enabled, storage elements can be set in a +degraded state to emulate defective storage elements (e.g. defective heads of a +hard disk). This is done by issuing the "degrade_element" command to +/dev/zloop-control, specifying the ID of the target device and the ID of the +storage element to degrade:: + + $ echo "degrade_element id=X,element_id=Y" > /dev/zloop-control + +Depending on the type of the storage element specified, a read or write access +to sectors of a zone served by a degraded storage element will result in a +failure. For partial access storage elements, write accesses are always failed +if the write element serving the zone is degraded. Likewise, read accesses are +failed if the read element serving the zone is degraded. When storage elements +are emulated as all access physical elements, both read and write operations are +failed when the storage element serving the zones being accessed is degraded. + +5) Example ---------- The following sequence of commands creates a 2GB zoned device with zones of 64
drivers/block/zloop.c +98 -2
184 lines not on this page · raw message
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 10/13 scsi: scsi_proto: refine the definition of storage element health
In reply to Damien Le MoalA disk may report the health of a storage element as being within or outside of the manufacturer's specification limits using a range of values for the health field. However, in scsi_proto.h, we only define the start of each range using the SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS and SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS enum value.
Define the full range by renaming SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS to SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MIN and by defining the healthy range end with SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MAX. The same treatment is done for the outside range with the renaming of SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS to SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MIN, and the definition of the range upper value with SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MAX.
--- include/scsi/scsi_proto.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
include/scsi/scsi_proto.h +4 -2
@@ -408,9 +408,11 @@ enum scsi_phys_element_type {/* SCSI physical element health. */ enum scsi_phys_element_health {SCSI_PHYS_ELEM_HEALTH_NOT_REPORTED =0x00 , -SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS =0x01 , +SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MIN =0x01 , +SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MAX =0x63 ,SCSI_PHYS_ELEM_HEALTH_AT_SPEC_LIMITS =0x64 , -SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS =0x65 , +SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MIN =0x65 , +SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MAX =0xCF ,SCSI_PHYS_ELEM_HEALTH_DEPOP_REVOKE_ERR =0xFB ,SCSI_PHYS_ELEM_HEALTH_DEPOP_REVOKE_IN_PROGRESS =0xFC ,SCSI_PHYS_ELEM_HEALTH_DEPOP_ERR =0xFD ,
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 11/13 scsi: sd_zbc: always revalidate zones for disks supporting head depopulation
In reply to Damien Le Moalsd_zbc_revalidate_zones() skips revalidating the zones of a ZBC device if the zone size and total number of zones of the disk has not changed. This is to avoid the sometimes slow report zones for the entire disk that blk_revalidate_disk_zones() will generate.
For ZBC devices that support the storage element depopulation feature, a revalidation after executing a passthrough REMOVE ELEMENT AND TRUNCATE command will thus correctly trigger a zone revalidation as the disk capacity and number of zones changed.
However, for devices supporting the data preserving head depopulation variant of the depopulation feature (REMOVE ELEMENT AND MODIFY ZONES command), a disk capacity and number of zones does not change after a head is depopulated. Only the conditions of the zones served by the head that was depopulated change. These zones are transitioned to either read-only or offline.
While the block layer storage element management functions always call blk_revalidate_disk_zones() to catch these changes, a user scan, libata device rescan after EH or a depopulation or restoration action executed through SG_IO would not. In these cases, not calling blk_revalidate_disk_zones() prevents the block layer from taking appropriate actions on the zone write plugs of the disk.
Avoid any issue with the block layer view of the zone conditions by not skipping the call to blk_revalidate_disk_zones() for disks that support both the GET PHYSICAL ELEMENTS STATUS and REMOVE ELEMENT AND MODIFY ZONES commands. This check is done from sd_zbc_read_zones() using the helper function sd_zbc_check_depop_supported(). The new SCSI disk flags depop_report_supported and depop_modify_zones_supported are defined to remember the result of this check.
This change will slow down a little SMR disk revalidation. However, user forced revalidations are normally rare, and other revalidations triggered by udev only happen when the disk device files is not in use anymore (closed). So this slow down should in practice not be an issue.
--- drivers/scsi/sd.h | 2 ++ drivers/scsi/sd_zbc.c | 47 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 47 insertions(+), 2 deletions(-)
drivers/scsi/sd.h +2 -0
@@ -156,6 +156,8 @@ struct scsi_disk { unsigned ignore_medium_access_errors : 1; unsigned rscs : 1; /* reduced stream control support */ unsigned use_atomic_write_boundary : 1; + unsigned depop_report_supported : 1; + unsigned depop_modify_zones_supported : 1; };#define to_scsi_disk (obj) container_of(obj, struct scsi_disk, disk_dev)
drivers/scsi/sd_zbc.c +45 -2
81 lines not on this page · raw message
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v10 12/13 scsi: sd_zbc: define storage element management operations
In reply to Damien Le MoalDefine the storage element management operations using struct blk_se_ops. These operations are valid only on SMR disks supporting the storage element depopulation featu, with support for the GET PHYSICAL ELEMENTS STATUS command being mandatory. The support for the modify_zones operation is already indicated by the depop_modify_zones_supported SCSI disk flag. sd_zbc_check_depop_supported() is modified to add detection for the command RESTORE ELEMENTS AND REBUILD, which is indicated with the new SCSI disk flag depop_restore_supported.
Both the remove_element and restore_elements operations will block until the operation completes or an error is detected.
--- drivers/scsi/sd.c | 3 + drivers/scsi/sd.h | 3 + drivers/scsi/sd_zbc.c | 493 +++++++++++++++++++++++++++++++++++++++++- 3 files changed, 498 insertions(+), 1 deletion(-)
drivers/scsi/sd.c +3 -0
@@ -3939,6 +3939,9 @@ static const struct block_device_operations sd_fops = { .get_unique_id = sd_get_unique_id, .free_disk = scsi_disk_free_disk, .pr_ops = &sd_pr_ops, +#ifdef CONFIG_BLK_DEV_ZONED + .se_ops = &sd_zbc_se_ops, +#endif };/**
drivers/scsi/sd.h +3 -0
@@ -158,6 +158,7 @@ struct scsi_disk { unsigned use_atomic_write_boundary : 1; unsigned depop_report_supported : 1; unsigned depop_modify_zones_supported : 1; + unsigned depop_restore_supported : 1; };#define to_scsi_disk (obj) container_of(obj, struct scsi_disk, disk_dev)
@@ -244,6 +245,8 @@ unsigned int sd_zbc_complete(struct scsi_cmnd *cmd, unsigned int good_bytes,int sd_zbc_report_zones (struct gendisk *disk,sector_t sector,unsigned int nr_zones,struct blk_report_zones_args *args); +extern const struct blk_se_ops sd_zbc_se_ops; +#else /* CONFIG_BLK_DEV_ZONED */ static inline int sd_zbc_read_zones(struct scsi_disk *sdkp,
drivers/scsi/sd_zbc.c +492 -1
529 lines not on this page · raw message
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH v10 12/13] scsi: sd_zbc: define storage element management operations
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [Medium] scsi: sd_zbc: missed restore error in sd_zbc_restore_storage_elements() - [Low] scsi: sd_zbc…
Thank you for your contribution! Sashiko AI review found 2 potential issues to consider:
- [Medium] scsi: sd_zbc: missed restore error in sd_zbc_restore_storage_elements()
- [Low] scsi: sd_zbc: missing ULLONG_MAX check in sd_zbc_parse_storage_element()
--
commit 41edcf895fef1b022cae55f553c74d5d5da41fa8
Author: Damien Le Moal <[email protected]>
scsi: sd_zbc: define storage element management operations
This commit defines struct blk_se_ops callbacks in sd_zbc to report, remove, and restore storage elements on SMR SCSI disks that support depopulation.
Quoted 4 lines
drivers/scsi/sd_zbc.c +0
-0
[ ... ]
Quoted 25 lines
drivers/scsi/sd_zbc.c +21
-0@@ -560,6 +573,484 @@ static void sd_zbc_print_zones(struct scsi_disk *sdkp) sdkp->zone_info.zone_blocks); } +static void sd_zbc_parse_storage_element (struct scsi_disk *sdkp,u8 *desc, +struct blk_se *element) +{ +struct scsi_device *sdp = sdkp->device; +sector_t zone_sectors =sd_zbc_zone_sectors (sdkp); +u64 capacity; + +memset (element,0 ,sizeof (*element)); + + element->id =get_unaligned_be32 (&desc[4 ]); + element->restore_allowed = desc[13 ] &0x01 ; + +switch (desc[14 ]) { +case SCSI_PHYS_ELEM_TYPE_ALL_ACCESS_STORAGE : + element->type =BLK_SE_TYPE_RDWR ; + capacity =get_unaligned_be64 (&desc[16 ]); +if (!zone_sectors || capacity ==ULLONG_MAX ) + element->nr_zones =0 ; +else + element->nr_zones = + logical_to_sectors(sdp, capacity) >>
drivers/scsi/sd_zbc.c static void sd_zbc_print_zones(struct scsi_disk *sdkp) · line 606
Quoted 10 lines
drivers/scsi/sd_zbc.c +10
-0+ilog2 (zone_sectors); +break ; +case SCSI_PHYS_ELEM_TYPE_FRAC_ACCESS_STORAGE : + element->paired_id =get_unaligned_be32 (&desc[16 ]); +if (desc[20 ] &0x01 ) + element->type =BLK_SE_TYPE_READ ; +else + element->type =BLK_SE_TYPE_WRITE ; + element->nr_zones =get_unaligned_be64 (&desc[24 ]); +break ;
[Severity: Low]
Should SCSI_PHYS_ELEM_TYPE_FRAC_ACCESS_STORAGE also normalize a reported count of ULLONG_MAX to element->nr_zones = 0, matching the handling for SCSI_PHYS_ELEM_TYPE_ALL_ACCESS_STORAGE above? When a device sets the number of associated zones field to 0xffffffffffffffff to indicate that the count is not reported, callers of bdev_report_storage_elements() and BLKREPORTSTORELEMS will otherwise receive ULLONG_MAX instead of 0.
[ ... ]
Quoted 6 lines
drivers/scsi/sd_zbc.c +6
-0+static int sd_zbc_wait_for_se_restore (struct scsi_disk *sdkp, +struct sd_zbc_se_report *rep, +unsigned int in_error_orig) +{ +unsigned int i, in_error, in_progress; +int ret;
[ ... ]
Quoted 16 lines
drivers/scsi/sd_zbc.c +16
-0+if (!in_progress) { +if (in_error > in_error_orig) +return -EIO ; +return 0 ; + } + }while (!fatal_signal_pending (current)); + +return -EINTR ; +} + +static int sd_zbc_restore_storage_elements (struct gendisk *disk) +{ +struct scsi_disk *sdkp =scsi_disk (disk); +unsigned int i, in_error_orig =0 ; +struct sd_zbc_se_report rep; +int ret;
[ ... ]
drivers/scsi/sd_zbc.c static void sd_zbc_print_zones(struct scsi_disk *sdkp) · line 1029
Quoted 4 lines
drivers/scsi/sd_zbc.c +4
-0+for (i =0 ; i < rep.nr_reported_elements; i++) { +if (rep.elements[i].status ==BLK_SE_STS_RESTORE_ERROR ) + in_error_orig++; + }
[Severity: Medium]
Can sd_zbc_restore_storage_elements() return 0 when a retried element fails restoration again?
When an element is in BLK_SE_STS_RESTORE_ERROR with restore_allowed set, RESTORE ELEMENTS AND REBUILD attempts to restore it again. Because that element is counted in in_error_orig before the command is issued, if its restoration fails again (or if it succeeds while another element fails), in_error will not exceed in_error_orig in sd_zbc_wait_for_se_restore() and the function will return 0 instead of -EIO.
Only elements in BLK_SE_STS_RESTORE_ERROR with !restore_allowed are ignored by the restore command and should be counted in in_error_orig.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Damien Le Moal
maintainerPATCH v10 13/13 scsi: sd: fail read commands to offline zones early
In reply to Damien Le MoalIssuing read commands to offline zones of a ZBC disk will result in failures. So do not issue such command and fail them early in sd_setup_read_write_cmnd(). This change also checks write commands, which is fine but should actually never happen as the block layer zone write plugging code already fails write BIOs to offline zones early in the submission path.
As the code comment explains, since disk_zone_is_offline() uses the block layer cached zone condition, this is inherently racy with zone revalidation updating the zone condition. However, this will never prevent valid read operations as the only case where we could proceed is if the read is received after a restoration of the disk storage element completes but before the zones are revalidated. And in that case, all zones are empty and thus have no data to read.
--- drivers/scsi/sd.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
drivers/scsi/sd.c +19 -1
@@ -1424,7 +1424,8 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd) {struct request *rq =scsi_cmd_to_rq (cmd);struct scsi_device *sdp = cmd->device; -struct scsi_disk *sdkp =scsi_disk (rq->q->disk); +struct gendisk *disk = rq->q->disk; +struct scsi_disk *sdkp =scsi_disk (disk);sector_t lba =sectors_to_logical (sdp,blk_rq_pos (rq));sector_t threshold;unsigned int nr_blocks =sectors_to_logical (sdp,blk_rq_sectors (rq));
@@ -1456,6 +1457,23 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd) goto fail; } +/* +* The block layer already prevents writes to offline zones of zoned +* block devices. But there are no checks for reads, so check this here. +* Since disk_zone_is_offline() uses the block layer cached zone +* condition, this is inherently racy with zone revalidation updating +* the zone condition. However, this will never prevent valid read +* operations as the only case where we could proceed is if the read is +* received after a restoration of the disk storage element completes +* but before the zones are revalidated. And in that case, all zones are +* empty and thus have no data to read. +* Note that we are quiet about the error (no message) to avoid flooding +* the kernel log. +*/ +if (sdkp->device->type ==TYPE_ZBC && +disk_zone_is_offline (disk,blk_rq_pos (rq))) + goto fail; +/* * Some SD card readers can't handle accesses which touch the * last one or two logical blocks. Split accesses as needed.
Signature · 1 line
2.56.0