Add support for storage element depopulation
Subsystems (7)
Damien Le Moal
maintainerv9 cover Add support for storage element depopulation
Jens,
These patches define a new set of block device operations for generically using from the block layer the storage element depopulation feature of zoned block devices. Support for this feature is added to the SCSI disk driver and an emulation of this feature added to the zloop driver.
The management operations can be accessed using block layer API, which is intended for file systems (e.g. zonefs and XFS), as well as using ioctls for users using zoned disks directly from user space.
Changes from v8:
- Addressed Sashiko comment on patch 7 by adding patch 6. Patch 7 is unmodified.
- Addressed Sashiko comments on patch 8 and patch 12
- Added review tags to unmodified pacthes
Changes from v7:
A significant amount of changes in this version to address most of the Sashiko review comments and to simplify the code. In more details, the changes are:
- Added patches 3, 6, and 9.
- Dropped former patch 5 as the use of irq-disabling spinlocks in zloop is not necessary.
- Reworked patch 4 by migrating the idepop operation wait loops to the device drivers. This simplifies the implementation in zloop without making the sd driver implementation more difficult. Also fixed problems with errors during restoration and zone write plugs state.
- Reworked patch 7 and 11 to be consistent with the changes in patch 4.
Changes from v6:
- Addressed the nit pointed out by Christoph in patch 4
- Added review tags
Changes from v5:
- Rebased on latest block/for-next
- Addressed Christoph's comments on patch 3
- Added patch 5 to address a potential deadlock in zloop (signaled by sashiko)
- Reworked the restoration of storage elements in patch 6 to avoid an inconsistent state of storage elements in case of error. Also added protection against disk removal during storage element removal and restoration execution.
- Added review tags
Changes from v4:
- Rebased on latest block/for-next
- Added review tags to patch 1 and 2
- Reworked patch 3, 4 and 5 to address Christoph's and sashiko comments
Changes from v3:
- Reworked patch 1 to only provide the disk_zone_is_offline() helper function. Failing of read operations to offline zones is handled directly by the zloop driver and the new patch 7 is added to handle failing early read operations to offline zones of ZBC SCSI disks.
- Added patch 2
- Addressed further valid comments from Sashiko
The biggest changes are in patch 2 to save the number of storage elements of a disk in struct gendisk and to serialize the execution of the storage element management operations using the zone revalidation mutex.
- Rebased on latest block/for-next
Changes from v2:
- Addressed further valid comments from Sashiko
- Rebased on latest block/for-next
Changes from v1:
- Addressed the many valid comments that Sashiko had
- Rebased on latest block/for-next
Damien Le Moal (13):
block: introduce disk_zone_is_offline()
block: refactor blk_revalidate_disk_zones()
block: allow dropping all zone write plugs on revalidation
block: introduce storage element management
block: add storage element management ioctls
zloop: keep zone write pointer and condition consistent
zloop: fix zone reporting consistency
zloop: add storage element emulation
zloop: add degrade_element control command
scsi: scsi_proto: refine the definition of storage element health
scsi: sd_zbc: always revalidate zones for disks supporting head depopulation
scsi: sd_zbc: define storage element management operations
scsi: sd: fail read commands to offline zones early
.../admin-guide/blockdev/zoned_loop.rst | 33 +- block/blk-zoned.c | 450 +++++++++- block/blk.h | 14 + block/genhd.c | 4 + block/ioctl.c | 4 + drivers/block/zloop.c | 841 +++++++++++++++++- drivers/scsi/sd.c | 23 +- drivers/scsi/sd.h | 5 + drivers/scsi/sd_zbc.c | 538 ++++++++++- include/linux/blkdev.h | 64 ++ include/scsi/scsi_proto.h | 6 +- include/uapi/linux/blkzoned.h | 106 +++ include/uapi/linux/fs.h | 1 + 13 files changed, 2007 insertions(+), 82 deletions(-)
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 1/13 block: introduce disk_zone_is_offline()
Introduce the helper function disk_zone_is_offline() for testing if a zone of a gendisk has the offline condition. The zone to test is identified using a sector number that must belong to the target zone to check.
--- block/blk-zoned.c | 18 ++++++++++++++++++ include/linux/blkdev.h | 9 +++++++++ 2 files changed, 27 insertions(+)
block/blk-zoned.c +18 -0
@@ -292,6 +292,24 @@ static bool disk_zone_is_seq(struct gendisk *disk, sector_t sector)return !blk_zstate_is_conv (zs); } +/** +* disk_zone_is_offline - check if a sector belongs to an offline zone +* @disk: gendisk to check +* @sector: sector number +* +* Check if @sector on @disk is contained in an offline zone. +* This function can be used only when the bdev queue was entered (BIOs +* submitted but not yet completed) or when holding zone_revalidate_mutex. +*/ +bool disk_zone_is_offline (struct gendisk *disk,sector_t sector) +{ +if (!blk_queue_is_zoned (disk->queue)) +return false; + +return disk_zone_get_cond (disk, sector) ==BLK_ZONE_COND_OFFLINE ; +} +EXPORT_SYMBOL_GPL (disk_zone_is_offline); +/** * bdev_zone_is_seq - check if a sector belongs to a sequential write zone * @bdev: block device to check
include/linux/blkdev.h +9 -0
@@ -940,6 +940,9 @@ static inline unsigned int disk_zone_capacity(struct gendisk *disk,return disk->last_zone_capacity;return disk->zone_capacity; } + +bool disk_zone_is_offline (struct gendisk *disk,sector_t sector); +static inline unsigned int bdev_zone_capacity (struct block_device *bdev,sector_t pos) {
@@ -954,6 +957,12 @@ static inline unsigned int disk_nr_zones(struct gendisk *disk)return 0 ; } +static inline bool disk_zone_is_offline (struct gendisk *disk, +sector_t sector) +{ +return false; +} +static inline bool bdev_zone_is_seq (struct block_device *bdev,sector_t sector) {return false;
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 2/13 block: refactor blk_revalidate_disk_zones()
In reply to Damien Le MoalSplit out most of the code of blk_revalidate_disk_zones() into the internal function disk_revalidate_zones(). This will allow calling this new helper together with other code under the zone revalidation mutex. No functional change.
--- block/blk-zoned.c | 63 +++++++++++++++++++++++++---------------------- 1 file changed, 34 insertions(+), 29 deletions(-)
block/blk-zoned.c +34 -29
@@ -2552,20 +2552,7 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,return ret; } -/** -* blk_revalidate_disk_zones - (re)allocate and initialize zone write plugs -* @disk: Target disk -* -* Helper function for low-level device drivers to check, (re) allocate and -* initialize resources used for managing zoned disks. This function should -* normally be called by blk-mq based drivers when a zoned gendisk is probed -* and when the zone configuration of the gendisk changes (e.g. after a format). -* Before calling this function, the device driver must already have set the -* device zone size (chunk_sector limit) and the max zone append limit. -* BIO based drivers can also use this function as long as the device queue -* can be safely frozen. -*/ -int blk_revalidate_disk_zones (struct gendisk *disk) +static int disk_revalidate_zones (struct gendisk *disk) {struct blk_revalidate_zone_args args = { };struct blk_report_zones_args rep_args = {
@@ -2576,19 +2563,10 @@ int blk_revalidate_disk_zones(struct gendisk *disk)int retries =2 ;int ret; -if (WARN_ON_ONCE (!blk_queue_is_zoned (disk->queue))) -return -EIO ; - -/* -* Serialize calls to this function so that we can safely look at and -* eventually change the disk zone information. -*/ -mutex_lock (&disk->zone_revalidate_mutex); - again: ret =disk_revalidate_capacity (disk, &args);if (ret) - goto unlock; +return ret;/* * Allocate zone resources if they are needed and we have not done
@@ -2601,14 +2579,14 @@ int blk_revalidate_disk_zones(struct gendisk *disk) ret =disk_alloc_zone_resources (disk, args.capacity);if (ret) {memalloc_noio_restore (noio_flag); - goto unlock; +return ret; } } ret =disk_init_revalidate_args (disk, &args);if (ret) {memalloc_noio_restore (noio_flag); - goto unlock; +return ret; } ret = disk->fops->report_zones (disk,0 ,UINT_MAX , &rep_args);
@@ -2625,8 +2603,6 @@ int blk_revalidate_disk_zones(struct gendisk *disk)if (ret) goto free_args; -mutex_unlock (&disk->zone_revalidate_mutex); -return 0 ; free_args:
@@ -2642,7 +2618,36 @@ int blk_revalidate_disk_zones(struct gendisk *disk) }pr_warn ("%s: failed to revalidate zones\n" , disk->disk_name); -unlock: + +return ret; +} + +/** +* blk_revalidate_disk_zones - (re)allocate and initialize zone write plugs +* @disk: Target disk +* +* Helper function for low-level device drivers to check, (re) allocate and +* initialize resources used for managing zoned disks. This function should +* normally be called by blk-mq based drivers when a zoned gendisk is probed +* and when the zone configuration of the gendisk changes (e.g. after a format). +* Before calling this function, the device driver must already have set the +* device zone size (chunk_sector limit) and the max zone append limit. +* BIO based drivers can also use this function as long as the device queue +* can be safely frozen. +*/ +int blk_revalidate_disk_zones (struct gendisk *disk) +{ +int ret; + +if (WARN_ON_ONCE (!blk_queue_is_zoned (disk->queue))) +return -EIO ; + +/* +* Serialize calls to this function so that we can safely look at and +* eventually change the disk zone information. +*/ +mutex_lock (&disk->zone_revalidate_mutex); + ret =disk_revalidate_zones (disk);mutex_unlock (&disk->zone_revalidate_mutex);return ret;
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 3/13 block: allow dropping all zone write plugs on revalidation
In reply to Damien Le MoalIn preparation for supporting storage element depopulation, modify the function disk_revalidate_zones() to take an additional boolean parameter drop_all_zwplugs to force removing all zone write plugs from a disk hash table. Removing all zone write plugs is done in the same manner as for a capacity change, using the function disk_drop_zone_wplug() applied to all hashed zone write plugs.
Calls to disk_revalidate_zones() with the argument drop_all_zwplugs set to true will be used when restoring depopulated storage elements as such operation clears all sequential zones, modifying them to be empty.
--- block/blk-zoned.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
block/blk-zoned.c +5 -2
@@ -2552,7 +2552,7 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,return ret; } -static int disk_revalidate_zones (struct gendisk *disk) +static int disk_revalidate_zones (struct gendisk *disk,bool drop_all_zwplugs) {struct blk_revalidate_zone_args args = { };struct blk_report_zones_args rep_args = {
@@ -2563,6 +2563,9 @@ static int disk_revalidate_zones(struct gendisk *disk)int retries =2 ;int ret; +if (drop_all_zwplugs) +disk_for_all_zone_wplugs (disk, disk_drop_zone_wplug,NULL ); + again: ret =disk_revalidate_capacity (disk, &args);if (ret)
@@ -2647,7 +2650,7 @@ int blk_revalidate_disk_zones(struct gendisk *disk)* eventually change the disk zone information. */ mutex_lock (&disk->zone_revalidate_mutex); - ret =disk_revalidate_zones (disk); + ret =disk_revalidate_zones (disk, false);mutex_unlock (&disk->zone_revalidate_mutex);return ret;
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 4/13 block: introduce storage element management
In reply to Damien Le MoalRecent SCSI (SBC) and ATA (ACS) standards define the storage element depopulation feature. This feature is intended for managing hard-disks heads, either combined read-write heads or pairs of read and write heads, allowing to keep disks with defective heads longer in production by allowing "depopulating" (removing) defective heads.
This feature comes in two different flavors:
- A destructive version which removes a head and reformats the disk at a lower capacity point, restoring a fully functional contiguous LBA address space.
- A data preserving version restricted to host-managed zoned disks, which marks the zones served by a removed head as offline or read-only.
In preparation for using the data-preserving flavor of the depopulation feature in file systems natively supporting zoned block devices, introduce a set of storage element management operations and functions to define generic calls into block device drivers for managing storage elements.
The set of operations is defined with struct blk_se_ops and includes three operations:
- report_elements: get information on a device storage elements state reported with the new struct blk_se.
- remove_element: depopulate a defective storage element, blocking until the operation completes.
- restore_elements: restore depopulated storage elements, blocking until the operation completes.
Each operation is called from the functions bdev_report_storage_elements(), bdev_remove_storage_element() and bdev_restore_storage_elements() respectively.
Removing a healthy storage element from a device is possible and useful for testing. The storage element restoration operation restore_elements allows repopulating such healthy element. Repopulating defective storage elements is generally not allowed by devices.
The remove_element and restore_elements operations are executed under the zone revalidation lock so that they are serialized against each other and also serialized against user initiated device revalidation, thus allowing to always have a consistent view of the device zone conditions after executing these operations. In case of failure, zone revalidation is still executed, which may give an unstable view of the zone states. In such case, stable zone states will be recovered with zone access error handling or later zone revalidation.
The device drivers of zoned block devices can indicate support for the storage element depopulation feature by specifying the storage element management operations with the se_ops field of struct block_device_operations.
--- block/blk-zoned.c | 205 ++++++++++++++++++++++++++++++++++ block/blk.h | 5 + block/genhd.c | 4 + include/linux/blkdev.h | 55 +++++++++ include/uapi/linux/blkzoned.h | 65 +++++++++++ 5 files changed, 334 insertions(+)
block/blk-zoned.c +205 -0
@@ -2736,3 +2736,208 @@ int queue_zone_wplugs_show(void *data, struct seq_file *m) }#endif + +/** +* bdev_report_storage_elements - report the storage elements of a zoned +* block device +* @bdev: the zoned block device to check +* @elements: array of storage element descriptors to fill +* @nr_elements: maximum size of @elements +* +* Fill at most @nr_elements storage element descriptors in the array +* @elements. The number of storage elements filled in the array is returned +* using @nr_elements. If @elements is NULL, only @nr_elements is returned to +* indicate the maximum number of storage elements of @bdev. This function may +* sleep. +* +* Returns 0 on success and a negative error code on failure. +*/ +int bdev_report_storage_elements (struct block_device *bdev, +struct blk_se *elements, +unsigned int *nr_elements) +{ +struct gendisk *disk = bdev->bd_disk; + +if (test_bit (GD_DEAD , &disk->state)) +return -ENODEV ; + +if (!blk_queue_is_zoned (disk->queue) || !disk->nr_storage_elements) +return -EOPNOTSUPP ; + +if (!nr_elements) +return -EINVAL ; + +if (!elements) { + *nr_elements = disk->nr_storage_elements; +return 0 ; + } + + *nr_elements =min (*nr_elements, disk->nr_storage_elements); +if (!*nr_elements) +return -EINVAL ; + +return disk->fops->se_ops->report_elements (disk, elements, + nr_elements); +} +EXPORT_SYMBOL_GPL (bdev_report_storage_elements); + +/** +* bdev_remove_storage_element - Remove (depopulate) a storage element of a +* zoned block device +* @bdev: block device to operate on +* @element_id: ID of the storage element to remove +* +* Remove (depopulate) the storage element identified by @element_id from the +* block device @bdev. The caller is responsible for taking care of any +* necessary dirty page and device write cache flush and invalidation of cached +* data. This function will block until the storage element is removed or an +* error occurs. +* +* Returns 0 on success and a negative error code on failure. +*/ +int bdev_remove_storage_element (struct block_device *bdev, +unsigned int element_id) +{ +struct gendisk *disk = bdev->bd_disk; +unsigned int memflags; +int ret, err; + +if (test_bit (GD_DEAD , &disk->state)) +return -ENODEV ; + +if (!blk_queue_is_zoned (disk->queue) || !disk->nr_storage_elements) +return -EOPNOTSUPP ; + +/* Zero is not a valid storage element ID. */ +if (!element_id) +return -EINVAL ; + +/* +* Serialize against concurrent revalidation and concurrent storage +* element management operations so that we can see consistent zone +* condition changes. +*/ +mutex_lock (&disk->zone_revalidate_mutex); + +/* +* Freeze and unfreeze the queue to flush any outstanding command. +* The caller is responsible for not queuing up more I/Os by higher +* level means. +*/ + memflags =blk_mq_freeze_queue (disk->queue); +blk_mq_unfreeze_queue (disk->queue, memflags); + +/* Initiate the storage element removal and wait for its completion. */ + ret = disk->fops->se_ops->remove_element (disk, element_id); + +/* +* Always revalidate zones, even in the case of an error as we do not +* know what condition the zones of the block device are in. +*/ + err =disk_revalidate_zones (disk, false); +if (err && !ret) + ret = err; + +mutex_unlock (&disk->zone_revalidate_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (bdev_remove_storage_element); + +/** +* bdev_restore_storage_elements - Restore all depopulated storage elements of +* a zoned block device +* @bdev: block device to operate on +* +* Restore all storage elements of @bdev that have been depopulated. Not all +* elements may be restored by this operation. This operation is destructive and +* will result in all sequential zones to be transitioned to the empty condition. +* The caller is responsible for taking care of any necessary invalidation of +* cached data. This function will block until the storage elements are restored +* or an error occurs. +* +* Returns 0 on success and a negative error code on failure. +*/ +int bdev_restore_storage_elements (struct block_device *bdev) +{ +struct gendisk *disk = bdev->bd_disk; +unsigned int memflags; +int ret, err; + +if (test_bit (GD_DEAD , &disk->state)) +return -ENODEV ; + +if (!blk_queue_is_zoned (disk->queue) || !disk->nr_storage_elements) +return -EOPNOTSUPP ; + +/* +* Serialize against concurrent revalidation and concurrent storage +* element management operations so that we can see consistent zone +* condition changes. +*/ +mutex_lock (&disk->zone_revalidate_mutex); + +/* +* Freeze and unfreeze the queue to flush any outstanding commands. +* The caller is responsible for not queuing up more I/O by higher +* level means. +*/ + memflags =blk_mq_freeze_queue (disk->queue); +blk_mq_unfreeze_queue (disk->queue, memflags); + +/* +* Initiate the storage elements restoration and wait for its +* completion. +*/ + ret = disk->fops->se_ops->restore_elements (disk); + +/* +* Always revalidate zones, even in the case of an error as we do not +* know what condition the zones of the block device are in. Note that +* since all sequential zones should have been transitioned to the empty +* condition, we can drop all zone write plugs. If an error occured and +* not all sequential zones are empty, the revalidation will reinstate +* the write plugs for these zones. +*/ + err =disk_revalidate_zones (disk, true); +if (err && !ret) + ret = err; + +mutex_unlock (&disk->zone_revalidate_mutex); + +return ret; +} +EXPORT_SYMBOL_GPL (bdev_restore_storage_elements); + +int disk_init_storage_element_mgmt (struct gendisk *disk) +{ +int ret; + +/* +* Storage element operations are for zoned disks only and are +* optional... +*/ +if (!blk_queue_is_zoned (disk->queue) || !disk->fops->se_ops) +return 0 ; + +/* ...but if defined, all operation callbacks must be defined. */ +if (!disk->fops->se_ops->report_elements || + !disk->fops->se_ops->remove_element || + !disk->fops->se_ops->restore_elements) { +pr_err ("%s: invalid storage element operations\n" , + disk->disk_name); +return -EINVAL ; + } + +/* +* Get the number of storage elements of the disk. If this fails, +* do not report an error as the failure may be because the zoned disk +* does not support storage element management. +*/ + ret = disk->fops->se_ops->report_elements (disk,NULL , + &disk->nr_storage_elements); +if (ret) + disk->nr_storage_elements =0 ; + +return 0 ; +}
block/blk.h +5 -0
@@ -543,6 +543,7 @@ static inline void ioc_clear_queue(struct request_queue *q)#endif /* CONFIG_BLK_ICQ */ #ifdef CONFIG_BLK_DEV_ZONED +int disk_init_storage_element_mgmt (struct gendisk *disk);void disk_init_zone_resources (struct gendisk *disk);void disk_release_zone_resources (struct gendisk *disk);static inline bool bio_zone_write_plugging (struct bio *bio)
@@ -595,6 +596,10 @@ int blkdev_zone_mgmt_ioctl(struct block_device *bdev, blk_mode_t mode, unsigned int cmd, unsigned long arg);bool bdev_zone_mgmt_allowed (struct block_device *bdev,sector_t sector);#else /* CONFIG_BLK_DEV_ZONED */ +static inline int disk_init_storage_element_mgmt (struct gendisk *disk) +{ +return 0 ; +}static inline void disk_init_zone_resources (struct gendisk *disk) { }
block/genhd.c +4 -0
@@ -435,6 +435,10 @@ static int __add_disk(struct device *parent, struct gendisk *disk,if (WARN_ON_ONCE (bdev_nr_sectors (disk->part0) >BLK_DEV_MAX_SECTORS ))return -EINVAL ; + ret =disk_init_storage_element_mgmt (disk); +if (ret) +return ret; +if (queue_is_mq (disk->queue)) {/* * ->submit_bio and ->poll_bio are bypassed for blk-mq drivers.
include/linux/blkdev.h +55 -0
85 lines not on this page · raw message
include/uapi/linux/blkzoned.h +65 -0
69 lines not on this page · raw message
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 5/13 block: add storage element management ioctls
In reply to Damien Le MoalDefine the new ioctl commands BLKREPORTSTORELEMS, BLKREMOVESTORELEM, and BLKRESTORESTORELEMS to provide users accessing zoned block devices directly with an interface to the storage elements management operations of the block layer.
The ioctl BLKREPORTSTORELEMS allows to get the total number of storage elements of a zoned block device and to get an array of storage element descriptors (struct blk_se) describing the current state of the device storage elements.
The ioctl BLKREMOVESTORELEM can be used to remove (depopulate) a storage element, in particular storage elements that have a degraded status (i.e. BLK_SE_STS_DEGRADED). But healthy elements (BLK_SE_STS_OK) can also be removed, and later restored.
Finally, the BLKRESTORESTORELEMS ioctl interfaces with bdev_restore_storage_elements() to restore the depopulated storage elements of a zoned device.
--- block/blk-zoned.c | 163 ++++++++++++++++++++++++++++++++++ block/blk.h | 9 ++ block/ioctl.c | 4 + include/uapi/linux/blkzoned.h | 41 +++++++++ include/uapi/linux/fs.h | 1 + 5 files changed, 218 insertions(+)
block/blk-zoned.c +163 -0
@@ -18,6 +18,7 @@#include <linux/mempool.h> #include <linux/kthread.h> #include <linux/freezer.h> +#include <linux/uaccess.h> #include <trace/events/block.h>
@@ -2781,6 +2782,53 @@ int bdev_report_storage_elements(struct block_device *bdev, }EXPORT_SYMBOL_GPL (bdev_report_storage_elements); +static int blkdev_ioctl_report_storage_elements (struct block_device *bdev, +void __user *argp) +{ +struct gendisk *disk = bdev->bd_disk; +struct blk_se_report *rep; +int ret =0 ; + +/* +* Allocate the report header with an array of elements, assuming that +* we need the maximum number of storage elements. In practice, we may +* need less than that, but this potentially unnecessary preallocation +* simplifies the code. +*/ + rep =kzalloc_flex (*rep, elements, disk->nr_storage_elements); +if (!rep) +return -ENOMEM ; + +if (copy_from_user (rep, argp,sizeof (struct blk_se_report ))) { + ret = -EFAULT ; + goto free_rep; + } + +if (rep->reserved) { + ret = -EINVAL ; + goto free_rep; + } + + rep->nr_reported_elements = +min (rep->nr_elements, disk->nr_storage_elements); + rep->nr_elements = disk->nr_storage_elements; + +if (rep->nr_reported_elements) { + ret =bdev_report_storage_elements (bdev, rep->elements, + &rep->nr_reported_elements); +if (ret) + goto free_rep; + } + +if (copy_to_user (argp, rep, +struct_size (rep, elements, rep->nr_reported_elements))) + ret = -EFAULT ; + +free_rep: +kfree (rep); +return ret; +} +/** * bdev_remove_storage_element - Remove (depopulate) a storage element of a * zoned block device
@@ -2844,6 +2892,62 @@ int bdev_remove_storage_element(struct block_device *bdev, }EXPORT_SYMBOL_GPL (bdev_remove_storage_element); +static int blkdev_ioctl_remove_storage_element (struct block_device *bdev, +blk_mode_t mode, +unsigned int __user *argp) +{ +unsigned int element_id; +int ret; + +/* +* Note: we require an O_EXCL open here so that a user cannot trigger +* a storage element removal from user space under a mounted file system. +* An FS that wants to manage storage element can do so using the +* function bdev_remove_storage_element() from kernel context. +*/ +if (!(mode &BLK_OPEN_EXCL )) +return -EBADF ; +if (!(mode &BLK_OPEN_WRITE )) +return -EBADF ; +if (bdev_read_only (bdev)) +return -EPERM ; + +if (get_user (element_id, argp)) +return -EFAULT ; + +/* +* Sync the device dirty data, flush the volatile write cache, and +* invalidate all cached data so that reads do not return old data for +* zones that went offline. +* Since a user can only successfully write to sequential zones of a +* zoned devices using direct IOs, the sync should only handle dirty +* data from buffered writes to conventional zones, for devices that +* have such zones. +*/ + ret =sync_blockdev (bdev); +if (ret) +return ret; + +inode_lock (bdev->bd_mapping->host); +filemap_invalidate_lock (bdev->bd_mapping); + + ret =blkdev_issue_flush (bdev->bd_disk->part0); +if (ret) + goto unlock; + + ret =truncate_bdev_range (bdev, mode,0 ,bdev_nr_bytes (bdev) -1 ); +if (ret) + goto unlock; + + ret =bdev_remove_storage_element (bdev, element_id); + +unlock: +filemap_invalidate_unlock (bdev->bd_mapping); +inode_unlock (bdev->bd_mapping->host); + +return ret; +} +/** * bdev_restore_storage_elements - Restore all depopulated storage elements of * a zoned block device
@@ -2909,6 +3013,42 @@ int bdev_restore_storage_elements(struct block_device *bdev) }EXPORT_SYMBOL_GPL (bdev_restore_storage_elements); +static int blkdev_ioctl_restore_storage_elements (struct block_device *bdev, +blk_mode_t mode) +{ +int ret; + +/* +* Note: because restoring storage elements is a destructive operation +* equivalent to reformatting the device, we require an O_EXCL open here +* so that a user cannot trigger with another active user (e.g. a +* mounted file system). +*/ +if (!(mode &BLK_OPEN_EXCL )) +return -EBADF ; +if (!(mode &BLK_OPEN_WRITE )) +return -EBADF ; +if (bdev_read_only (bdev)) +return -EPERM ; + +inode_lock (bdev->bd_mapping->host); +filemap_invalidate_lock (bdev->bd_mapping); + +/* +* Storage element restoration is a destructive operation that will +* reset all zones. So invalidate all cached data that we may have so +* that old data cannot be read once the operation completes. +*/ + ret =truncate_bdev_range (bdev, mode,0 ,bdev_nr_bytes (bdev) -1 ); +if (!ret) + ret =bdev_restore_storage_elements (bdev); + +filemap_invalidate_unlock (bdev->bd_mapping); +inode_unlock (bdev->bd_mapping->host); + +return ret; +} +int disk_init_storage_element_mgmt (struct gendisk *disk) {int ret;
@@ -2941,3 +3081,26 @@ int disk_init_storage_element_mgmt(struct gendisk *disk)return 0 ; } + +int blk_ioctl_zone_storage_elements (struct block_device *bdev, +blk_mode_t mode,unsigned int cmd, +void __user *argp) +{ +struct gendisk *disk = bdev->bd_disk; + +if (!disk->nr_storage_elements) +return -ENOTTY ; + +switch (cmd) { +case BLKREPORTSTORELEMS : +return blkdev_ioctl_report_storage_elements (bdev, argp); +case BLKREMOVESTORELEM : +return blkdev_ioctl_remove_storage_element (bdev, mode, argp); +case BLKRESTORESTORELEMS : +return blkdev_ioctl_restore_storage_elements (bdev, mode); +default : +break ; + } + +return -ENOTTY ; +}
block/blk.h +9 -0
@@ -595,6 +595,9 @@ int blkdev_report_zones_ioctl(struct block_device *bdev, unsigned int cmd,int blkdev_zone_mgmt_ioctl (struct block_device *bdev,blk_mode_t mode,unsigned int cmd,unsigned long arg);bool bdev_zone_mgmt_allowed (struct block_device *bdev,sector_t sector); +int blk_ioctl_zone_storage_elements (struct block_device *bdev, +blk_mode_t mode,unsigned int cmd, +void __user *argp);#else /* CONFIG_BLK_DEV_ZONED */ static inline int disk_init_storage_element_mgmt (struct gendisk *disk) {
@@ -646,6 +649,12 @@ static inline bool bdev_zone_mgmt_allowed(struct block_device *bdev, {return false; } +static inline int blk_ioctl_zone_storage_elements (struct block_device *bdev, +blk_mode_t mode,unsigned int cmd, +void __user *argp) +{ +return -ENOTTY ; +}#endif /* CONFIG_BLK_DEV_ZONED */ struct block_device *bdev_alloc (struct gendisk *disk,u8 partno);
block/ioctl.c +4 -0
@@ -679,6 +679,10 @@ static int blkdev_common_ioctl(struct block_device *bdev, blk_mode_t mode,return put_uint (argp,bdev_zone_sectors (bdev));case BLKGETNRZONES :return put_uint (argp,bdev_nr_zones (bdev)); +case BLKREPORTSTORELEMS : +case BLKREMOVESTORELEM : +case BLKRESTORESTORELEMS : +return blk_ioctl_zone_storage_elements (bdev, mode, cmd, argp);case BLKROGET :return put_int (argp,bdev_read_only (bdev) !=0 );case BLKSSZGET :/* get block device logical block size */
include/uapi/linux/blkzoned.h +41 -0
45 lines not on this page · raw message
include/uapi/linux/fs.h +1 -0
@@ -309,6 +309,7 @@ struct file_attr {/* 130-136 and 142 are used by zoned block device ioctls (uapi/linux/blkzoned.h) */ /* 137-141 are used by blk-crypto ioctls (uapi/linux/blk-crypto.h) */ #define BLKTRACESETUP2 _IOWR(0x12, 142, struct blk_user_trace_setup2) +/* 143-145 are used by storage element management for zoned block devices. */ #define BMAP_IOCTL 1/* obsolete - kept for compatibility */ #define FIBMAP _IO(0x00,1)/* bmap access */
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 6/13 zloop: keep zone write pointer and condition consistent
In reply to Damien Le MoalFor oredered zone append commands, the function zloop_set_zone_append_sector() increments a zone write pointer without checking if the zone is already open (implicitly or explicitly). This check is done only when the command is later executed using a work item, and the zone is implicitly open if needed. This creates a short time window where a report zone can show an empty zone with a write pointer that is not equal to the zone start sector, which is incorrect.
Fix this by calling zloop_do_open_zone() in zloop_set_zone_append_sector() to make sure that a zone is always first implicitly open if needed. This requires changing zloop_set_zone_append_sector() to return a blk_status_t so that potential zone resource errors can also be returned correctly.
--- drivers/block/zloop.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
drivers/block/zloop.c +15 -6
@@ -885,20 +885,26 @@ static void zloop_complete_rq(struct request *rq)blk_mq_end_request (rq, sts); } -static bool zloop_set_zone_append_sector (struct request *rq) +static blk_status_t zloop_set_zone_append_sector (struct request *rq) {struct zloop_device *zlo = rq->q->queuedata;unsigned int zone_no =rq_zone_no (rq);struct zloop_zone *zone = &zlo->zones[zone_no];sector_t zone_end = zone->start + zlo->zone_capacity;sector_t nr_sectors =blk_rq_sectors (rq); +static blk_status_t ret =BLK_STS_OK ;spin_lock (&zone->wp_lock);if (zone->cond ==BLK_ZONE_COND_FULL || zone->wp + nr_sectors > zone_end) { -spin_unlock (&zone->wp_lock); -return false; + ret =BLK_STS_IOERR ; + goto unlock; + } + +if (!zloop_do_open_zone (zlo, zone, false)) { + ret =BLK_STS_ZONE_OPEN_RESOURCE ; + goto unlock; } rq->__sector = zone->wp;
@@ -906,9 +912,10 @@ static bool zloop_set_zone_append_sector(struct request *rq)if (zone->wp >= zone_end)zloop_mark_full (zlo, zone); + unlock:spin_unlock (&zone->wp_lock); -return true; +return ret; } static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,
@@ -917,6 +924,7 @@ static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,struct request *rq = bd->rq;struct zloop_cmd *cmd =blk_mq_rq_to_pdu (rq);struct zloop_device *zlo = rq->q->queuedata; +blk_status_t ret;if (data_race (READ_ONCE (zlo->state)) == Zlo_deleting) { rq->rq_flags |=RQF_QUIET ;
@@ -929,8 +937,9 @@ static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,* command work runs. */ if (zlo->ordered_zone_append &&req_op (rq) ==REQ_OP_ZONE_APPEND ) { -if (!zloop_set_zone_append_sector (rq)) -return BLK_STS_IOERR ; + ret =zloop_set_zone_append_sector (rq); +if (ret !=BLK_STS_OK ) +return ret; }blk_mq_start_request (rq);
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH v9 06/13] zloop: keep zone write pointer and condition consistent
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider: - [High] zloop: static ret in zloop_set_zone_append_sector() causes sticky errors - [High] zloop: redundant…
Thank you for your contribution! Sashiko AI review found 3 potential issues to consider:
- [High] zloop: static ret in zloop_set_zone_append_sector() causes sticky errors
- [High] zloop: redundant zloop_do_open_zone() in zloop_seq_write_prep() fails I/O
- [Medium] zloop: race on zone->cond between zloop_set_zone_append_sector() and open
--
commit f1c0873a72a5832aa13e65548e6e03362a25c8cb
Author: Damien Le Moal <[email protected]>
zloop: keep zone write pointer and condition consistent
This commit calls zloop_do_open_zone() in zloop_set_zone_append_sector() and changes its return type to blk_status_t so ordered zone append commands implicitly open the target zone when the write pointer is updated.
Quoted 7 lines
drivers/block/zloop.c +0
-0@@ -885,20 +885,26 @@ static void zloop_complete_rq(struct request *rq)blk_mq_end_request (rq, sts); }
drivers/block/zloop.c static void zloop_complete_rq(struct request *rq) · line 895
Quoted 10 lines
drivers/block/zloop.c +2
-1-static bool zloop_set_zone_append_sector (struct request *rq) +static blk_status_t zloop_set_zone_append_sector (struct request *rq) {struct zloop_device *zlo = rq->q->queuedata;unsigned int zone_no =rq_zone_no (rq);struct zloop_zone *zone = &zlo->zones[zone_no];sector_t zone_end = zone->start + zlo->zone_capacity;sector_t nr_sectors =blk_rq_sectors (rq); +static blk_status_t ret =BLK_STS_OK ;
[Severity: High]
Should ret be a non-static local variable?
Because ret is declared static and is never reset to BLK_STS_OK on the success path, once any call fails, every subsequent ordered zone append opens the target zone, advances zone->wp, and still returns the stale error to zloop_queue_rq(), aborting the request and leaving zone->wp out of sync with the backing file. Concurrent callers on different zones also race on reading and writing ret.
A later patch in the series, commit "zloop: add storage element emulation", refactors zloop_set_zone_append_sector() and removes this static variable, but this commit carries a Fixes: tag and introduces the regression for bisection and backports.
Quoted 5 lines
drivers/block/zloop.c +0
-0spin_lock (&zone->wp_lock);if (zone->cond ==BLK_ZONE_COND_FULL || zone->wp + nr_sectors > zone_end) {
drivers/block/zloop.c static void zloop_complete_rq(struct request *rq) · line 908
Quoted 10 lines
drivers/block/zloop.c +7
-2-spin_unlock (&zone->wp_lock); -return false; + ret =BLK_STS_IOERR ; + goto unlock; + } + +if (!zloop_do_open_zone (zlo, zone, false)) { + ret =BLK_STS_ZONE_OPEN_RESOURCE ; + goto unlock; }
[Severity: Medium]
Can calling zloop_do_open_zone() under only zone->wp_lock race with zloop_open_zone() on the same zone?
zloop_open_zone() calls zloop_do_open_zone() while holding only zone->lock, without taking zone->wp_lock:
drivers/block/zloop.c:zloop_open_zone() {
...
mutex_lock(&zone->lock);
...
if (!zloop_do_open_zone(zlo, zone, true))
ret = -EIO;
...
}When a zone is in BLK_ZONE_COND_EMPTY or BLK_ZONE_COND_CLOSED, an ordered zone append and an explicit zone open can enter zloop_do_open_zone() concurrently:
CPU0 CPU1zloop_queue_rq() zloop_open_zone()
zloop_set_zone_append_sector() mutex_lock(&zone->lock)
spin_lock(&zone->wp_lock) zloop_do_open_zone()
zloop_do_open_zone() switch (zone->cond)
switch (zone->cond) zloop_open_closed_or_empty_zone()
zloop_open_closed_or_empty_zone() spin_lock(&zlo->open_zones_lock)
zlo->nr_open_zones++;
list_add_tail(&zone->open_zone_entry, ...)
spin_unlock(&zlo->open_zones_lock)
spin_lock(&zlo->open_zones_lock)
zlo->nr_open_zones++;
list_add_tail(&zone->open_zone_entry, ...) <-- list corruptionBoth paths add zone->open_zone_entry to zlo->open_zones_lru_list and increment zlo->nr_open_zones, corrupting the LRU list and open zone accounting.
drivers/block/zloop.c static bool zloop_set_zone_append_sector(struct request *rq) · line 913
Quoted 5 lines
drivers/block/zloop.c +0
-0rq->__sector = zone->wp;@@ -906,9 +912,10 @@ static bool zloop_set_zone_append_sector(struct request *rq)if (zone->wp >= zone_end)zloop_mark_full (zlo, zone);
[Severity: High]
Does calling zloop_do_open_zone() again in zloop_seq_write_prep() fail ordered zone appends that fill a zone?
When an ordered zone append reaches zone_end here, zloop_mark_full() sets zone->cond to BLK_ZONE_COND_FULL before the command work item runs. When the work item later executes zloop_rw() -> zloop_seq_write_prep(), it skips the capacity check and write pointer update for ordered_zone_append, but still calls zloop_do_open_zone() unconditionally:
drivers/block/zloop.c:zloop_seq_write_prep() {
...
/* Implicitly open the target zone. */
if (!zloop_do_open_zone(zlo, zone, false)) {
ret = -EIO;
goto out_unlock;
}
if (!is_append || !zlo->ordered_zone_append) {
zone->wp += nr_sectors;
if (zone->wp == zone_end)
zloop_mark_full(zlo, zone);
}
...
}Because zloop_do_open_zone() returns false when zone->cond is BLK_ZONE_COND_FULL, the zone-filling append (and any earlier in-flight ordered append to that zone whose work item runs after zloop_mark_full()) fails with -EIO and sets ZLOOP_ZONE_SEQ_ERROR. Skipping zloop_do_open_zone() in zloop_seq_write_prep() when is_append and zlo->ordered_zone_append are set would avoid reopening a zone that was already opened and possibly marked full at submission time.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [PATCH v9 06/13] zloop: keep zone write pointer and condition consistent
In reply to Damien Le MoalQuoted 13 lines · Damien Le Moal
For oredered zone append commands, the function
zloop_set_zone_append_sector()increments a zone write pointer without checking if the zone is already open (implicitly or explicitly). This check is done only when the command is later executed using a work item, and the zone is implicitly open if needed. This creates a short time window where a report zone can show an empty zone with a write pointer that is not equal to the zone start sector, which is incorrect.Fix this by calling
zloop_do_open_zone()inzloop_set_zone_append_sector()to make sure that a zone is always first implicitly open if needed. This requires changingzloop_set_zone_append_sector()to return ablk_status_tso that potential zone resource errors can also be returned correctly.
Quoted 6 lines
Fixes fcc6eaa3a03a ("zloop: introduce the ordered_zone_append configuration parameter")Signed-off-by Damien Le Moal <[email protected]>--- drivers/block/zloop.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Damien Le Moal
maintainerPATCH v9 7/13 zloop: fix zone reporting consistency
In reply to Damien Le MoalIn zloop_report_zones(), filling the write pointer and condition of a zone descriptor separately outside of the zone wp_lock can lead to an inconsistent zone report with a write pointer potentially not being consistent with the condition being reported. Avoid such inconsistency by filling both fields while holding the zone wp_lock.
--- drivers/block/zloop.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/zloop.c +1 -1
@@ -1001,8 +1001,8 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector, blkz.len = zlo->zone_size;spin_lock (&zone->wp_lock); blkz.wp = zone->wp; -spin_unlock (&zone->wp_lock); blkz.cond = zone->cond; +spin_unlock (&zone->wp_lock);if (test_bit (ZLOOP_ZONE_CONV , &zone->flags)) { blkz.type =BLK_ZONE_TYPE_CONVENTIONAL ; blkz.capacity = zlo->zone_size;
Signature · 1 line
2.56.0
Re: [PATCH v9 07/13] zloop: fix zone reporting consistency
Quoted 18 lines · Damien Le Moal
In
zloop_report_zones(), filling the write pointer and condition of a zone descriptor separately outside of the zonewp_lockcan lead to an inconsistent zone report with a write pointer potentially not being consistent with the condition being reported. Avoid such inconsistency by filling both fields while holding the zonewp_lock.Fixes fcc6eaa3a03a ("zloop: introduce the ordered_zone_append configuration parameter")Signed-off-by Damien Le Moal <[email protected]>Reviewed-by Hannes Reinecke <[email protected]>--- drivers/block/zloop.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/block/zloop.c +0
-0
drivers/block/zloop.c static int zloop_report_zones(struct gendisk *disk, sector_t sector, · line 1008
Quoted 9 lines
drivers/block/zloop.c +1
-1@@ -1001,8 +1001,8 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector, blkz.len = zlo->zone_size;spin_lock (&zone->wp_lock); blkz.wp = zone->wp; -spin_unlock (&zone->wp_lock); blkz.cond = zone->cond; +spin_unlock (&zone->wp_lock);if (test_bit (ZLOOP_ZONE_CONV , &zone->flags)) { blkz.type =BLK_ZONE_TYPE_CONVENTIONAL ; blkz.capacity = zlo->zone_size;
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Damien Le Moal
maintainerPATCH v9 8/13 zloop: add storage element emulation
In reply to Damien Le MoalEmulate the storage element depopulation feature of SMR disks in the zloop driver. This emulation is controlled using the new stor_elements option. This option can take several values:
- ZLOOP_STOR_ELEMENTS_NONE (0): no emulation (default)
- ZLOOP_STOR_ELEMENTS_RDWR (1): emulate all access storage elements (e.g. read+write heads)
- ZLOOP_STOR_ELEMENTS_PAIRS (2): emulate fractional access storage elements (e.g. pairs of read and write heads)
If enabled with the value 1 or 2, the number of storage elements, or of pairs of fractional access storage elements, is automatically calculated based on the number of zones of the device so that we have at least 2 and at most 32 storage elements or pairs of fractional access storage elements.
The mapping of zones to storage elements is defined simply as the modulo of a zone number with the the number of storage elements for the all access storage elements emulation and with the number of storage element pairs for the fractional access storage element emulation case. E.g, removing a storage element from a set of 4 all access storage elements will offline 1 zone every 4 zones.
The storage element management operations are specified with the zloop_se_ops (struct blk_se_ops).
--- .../admin-guide/blockdev/zoned_loop.rst | 11 +- drivers/block/zloop.c | 738 ++++++++++++++++-- 2 files changed, 700 insertions(+), 49 deletions(-)
Documentation/admin-guide/blockdev/zoned_loop.rst +10 -1
@@ -61,7 +61,7 @@ The options available for the add command can be listed by reading the /dev/zloop-control device:: $ cat /dev/zloop-control - add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache + add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache,stor_elements=%u remove id=%d In more details, the options that can be used with the "add" command are as
@@ -113,6 +113,15 @@ discard_write_cache Discard all data that was not explicitly persisted using a each zone file to the size recorded during the last flush operation. This simulates power fail events where uncommitted data is lost. +stor_elements Control storage element emulation. The default value is + 0, indicating no emulation. A value of 1 indicates that + storage elements are emulated as full access physical + elements, which is equivalent to combined read+write heads + of a hard-disk. A value of 2 enables storage element + emulation as fractional access physical storage elements, + which is equivalent to pairs of read and write heads of a + hard-disk. The health state of storage elements is not + preserved across a zloop device remove and add. =================== ========================================================= 3) Deleting a Zoned Device
drivers/block/zloop.c +690 -48
977 lines not on this page · raw message
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH v9 08/13] zloop: add storage element emulation
Thank you for your contribution! Sashiko AI review found 4 potential issues to consider: - [High] zloop: inverted zone check and missing unlock in zloop_restore_zone_file() - [Medium] zloop: bool…
Thank you for your contribution! Sashiko AI review found 4 potential issues to consider:
- [High] zloop: inverted zone check and missing unlock in zloop_restore_zone_file()
- [Medium] zloop: bool return type on zloop_prep_rq() corrupts blk_status_t error codes
- [Medium] zloop: broken write element removal check and zone cond transition
- [Low] zloop: wrong enum blk_se_status return type on zloop_zone_element_is_ok()
--
commit add2167f11ea8b6280aa470bbfffca31f313cf06
Author: Damien Le Moal <[email protected]>
zloop: add storage element emulation
This commit adds storage element depopulation and restoration emulation to the zloop driver, controlled via the new stor_elements option and exposed through zloop_se_ops.
Quoted 4 lines
drivers/block/zloop.c +0
-0
[ ... ]
Quoted 1 line
drivers/block/zloop.c +0
-0@@ -289,22 +314,80 @@ static bool zloop_do_open_zone(struct zloop_device *zlo,
[ ... ]
drivers/block/zloop.c static bool zloop_do_open_zone(struct zloop_device *zlo, · line 368
Quoted 9 lines
drivers/block/zloop.c +9
-0+static enum blk_se_status zloop_zone_element_is_ok (struct zloop_device *zlo, +struct zloop_zone *zone, +bool for_write) +{ +enum blk_se_status sts = +zloop_zone_element_status (zlo, zone, for_write); + +return sts <BLK_SE_STS_DEGRADED ; +}
[Severity: Low]
Should zloop_zone_element_is_ok() have a return type of bool instead of enum blk_se_status, since it returns a boolean comparison result?
[ ... ]
Quoted 1 line
drivers/block/zloop.c +0
-0@@ -885,34 +1007,65 @@ static void zloop_complete_rq(struct request *rq)
[ ... ]
Quoted 31 lines
drivers/block/zloop.c +31
-0+static bool zloop_prep_rq (struct zloop_device *zlo,struct request *rq) +{ +struct zloop_zone *zone = &zlo->zones[rq_zone_no (rq)]; +bool is_write =op_is_write (req_op (rq)); +blk_status_t ret =BLK_STS_OK ; + +if (!zlo->nr_elements && + (!zlo->ordered_zone_append ||req_op (rq) !=REQ_OP_ZONE_APPEND )) +return BLK_STS_OK ; + +spin_lock (&zone->wp_lock); + +/* +* If storage element emulation is enabled, check the health state of +* the storage element serving the zone and the zone condition: do not +* allow any operation to zones served by an unusable storage element, +* operations to offline zones, and allow only read operations to +* read-only zones. +*/ +if (zlo->nr_elements) { +if (!zloop_zone_element_is_ok (zlo, zone, is_write) || + zone->cond ==BLK_ZONE_COND_OFFLINE || + (zone->cond ==BLK_ZONE_COND_READONLY && is_write)) { + ret =BLK_STS_IOERR ; + goto unlock; + } + } + +/* +* If we need to strongly order zone append operations, set the request +* sector to the zone write pointer location now instead of when the
drivers/block/zloop.c static void zloop_complete_rq(struct request *rq)
Quoted 10 lines
drivers/block/zloop.c +9
-0+* command work runs. +*/ +if (zlo->ordered_zone_append &&req_op (rq) ==REQ_OP_ZONE_APPEND ) + ret =zloop_set_zone_append_sector (zlo, zone, rq); + +unlock: +spin_unlock (&zone->wp_lock); + +return ret; }
[Severity: Medium]
Does the bool return type of zloop_prep_rq() turn non-zero blk_status_t values into 1 (BLK_STS_NOTSUPP)? When zloop_prep_rq() fails with BLK_STS_IOERR or BLK_STS_ZONE_OPEN_RESOURCE, zloop_queue_rq() returns BLK_STS_NOTSUPP (-EOPNOTSUPP) to blk-mq instead.
[ ... ]
Quoted 1 line
drivers/block/zloop.c +0
-0@@ -1021,6 +1188,354 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector,
[ ... ]
Quoted 99 lines
drivers/block/zloop.c +99
-0+static int zloop_execute_remove_element (struct zloop_device *zlo) +{ +struct gendisk *disk = zlo->disk; +struct blk_se *se, *paired_se =NULL ; +enum blk_zone_cond cond; +struct zloop_zone *zone; +unsigned int memflags, i; +int ret =0 ; + +/* Wait for all IOs to complete. */ + memflags =blk_mq_freeze_queue (disk->queue); +blk_mq_unfreeze_queue (disk->queue, memflags); + +/* +* If the element to remove is a read element, zones must go offline and +* the associated write element is also removed. +*/ + se = &zlo->elements[zlo->remove_element_id -1 ]; +switch (se->type) { +case BLK_SE_TYPE_RDWR : + cond =BLK_ZONE_COND_OFFLINE ; +break ; +case BLK_SE_TYPE_READ : + cond =BLK_ZONE_COND_OFFLINE ; + paired_se = &zlo->elements[se->paired_id -1 ]; +break ; +case BLK_SE_TYPE_WRITE : + cond =BLK_ZONE_COND_READONLY ; +break ; +default : +WARN_ON_ONCE (1 ); +return -EINVAL ; + } + +/* +* Change the condition of the zones owned by the (pair of) elements +* being removed and mark the elements removed. +*/ +for (i =0 , zone = zlo->zones; i < zlo->nr_zones; i++, zone++) { +if (zone->wr_se_id != se->id && zone->rd_se_id != se->id) +continue ; +mutex_lock (&zone->lock); +spin_lock (&zone->wp_lock); +zloop_set_zone_cond (zlo, zone, cond); +clear_bit (ZLOOP_ZONE_SEQ_ERROR , &zone->flags); +spin_unlock (&zone->wp_lock); +mutex_unlock (&zone->lock); + } + +WRITE_ONCE (se->status,BLK_SE_STS_REMOVED ); +if (paired_se &&READ_ONCE (paired_se->status) !=BLK_SE_STS_REMOVED ) +WRITE_ONCE (paired_se->status,BLK_SE_STS_REMOVED ); + +return ret; +} + +static int zloop_remove_element (struct gendisk *disk,unsigned int element_id) +{ +struct zloop_device *zlo = disk->private_data; +struct blk_se *se, *rse, *rpse =NULL ; +unsigned int i, nr_se_remaining =0 ; +int ret =0 ; + +if (test_bit (GD_DEAD , &disk->state) || +data_race (READ_ONCE (zlo->state)) == Zlo_deleting) +return -ENODEV ; + +if (zlo->stor_elements ==ZLOOP_STOR_ELEMENTS_NONE ) +return -EOPNOTSUPP ; + +if (!element_id || element_id > zlo->nr_elements) +return -EINVAL ; + +mutex_lock (&zlo->stor_elements_lock); + +/* If a remove or restore operation is already in progress, bail out. */ +if (zlo->remove_element_id ||READ_ONCE (zlo->restore_in_progress)) { + ret = -EBUSY ; + goto unlock; + } + +/* +* Get the element to remove. If it is a read element, the associated +* write element must also be removed. +*/ + rse = &zlo->elements[element_id -1 ]; +if (rse->type ==BLK_SE_TYPE_READ ) + rpse = &zlo->elements[rse->paired_id -1 ]; + +/* If the element is already removed, we have nothing to do. */ +if (READ_ONCE (rse->status) ==BLK_SE_STS_REMOVED ) + goto unlock; + +/* +* Make sure we do not allow removing all storage elements as that does +* not make any sense. This is consistent with the device advertized +* limit of SCSI and ATA devices supporting the storage element +* depopulation feature. +*/
drivers/block/zloop.c static int zloop_report_zones(struct gendisk *disk, sector_t sector, · line 1340
Quoted 10 lines
drivers/block/zloop.c +10
-0+for (i =0 , se = zlo->elements; i < zlo->nr_elements; i++, se++) { +if (se == rse || se == rpse) +continue ; +if (READ_ONCE (se->status) <=BLK_SE_STS_DEGRADED ) + nr_se_remaining++; + } +if (!nr_se_remaining) { + ret = -EINVAL ; + goto unlock; + }
[Severity: Medium]
Can zloop_remove_element() allow removing every write element in ZLOOP_STOR_ELEMENTS_PAIRS mode? When removing a BLK_SE_TYPE_WRITE element, rpse is NULL and the paired BLK_SE_TYPE_READ elements remain in BLK_SE_STS_OK and are counted in nr_se_remaining, so nr_se_remaining is always non-zero and all write elements on the device can be removed. Counting BLK_SE_STS_DEGRADED elements in nr_se_remaining also allows removing the last healthy element when zloop_zone_element_is_ok() rejects I/O to degraded elements (sts < BLK_SE_STS_DEGRADED).
In addition, if a removed read+write element pair fails restoration (leaving both elements in BLK_SE_STS_RESTORE_ERROR and their zones in BLK_ZONE_COND_OFFLINE), removing the BLK_SE_TYPE_WRITE element in zloop_execute_remove_element() unconditionally sets those zones to BLK_ZONE_COND_READONLY even though the paired read element cannot serve reads.
[ ... ]
Quoted 12 lines
drivers/block/zloop.c +12
-0+static int zloop_restore_zone_file (struct zloop_device *zlo, +struct zloop_zone *zone) +{ +int ret =0 ; + +if (!test_bit (ZLOOP_ZONE_CONV , &zone->flags)) +return 0 ; + +mutex_lock (&zone->lock); + + ret =vfs_truncate (&zone->file->f_path,0 ); +if (ret)
drivers/block/zloop.c static int zloop_report_zones(struct gendisk *disk, sector_t sector, · line 1404
Quoted 10 lines
drivers/block/zloop.c +10
-0+set_bit (ZLOOP_ZONE_SEQ_ERROR , &zone->flags); + +return ret; +} + +static void zloop_restore_zone_cond (struct zloop_device *zlo, +struct zloop_zone *zone) +{ +mutex_lock (&zone->lock); +spin_lock (&zone->wp_lock);
[Severity: High]
Is the ZLOOP_ZONE_CONV check in zloop_restore_zone_file() inverted, and is mutex_unlock(&zone->lock) missing before return?
As written, zloop_restore_zone_file() skips sequential zones, truncates conventional zone backing files to 0 bytes, and returns with zone->lock held, which immediately deadlocks when zloop_execute_restore_elements() calls zloop_restore_zone_cond() on the same zone.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Damien Le Moal
maintainerPATCH v9 9/13 zloop: add degrade_element control command
In reply to Damien Le MoalAllow users to mark storage elements of a zloop device as degraded using the new "degrade_element" control command. The element to degrade is indicated using the element_id option. Example:
echo "degrade_element id=0,element_id=2" > /dev/zloop-control
If the element ID identifies an all access storage element, both read and write operations targeting a zone served by the degraded element are failed. In the case of a partial access storage element, any write access to a zone whose write element is degraded will fail, and read accesses to a zone whose read element is degraded will result in an error.
The zoned_loop.rst documentation file is also updated to explain this new feature.
--- .../admin-guide/blockdev/zoned_loop.rst | 22 +++- drivers/block/zloop.c | 100 +++++++++++++++++- 2 files changed, 119 insertions(+), 3 deletions(-)
Documentation/admin-guide/blockdev/zoned_loop.rst +21 -1
@@ -63,6 +63,7 @@ The options available for the add command can be listed by reading the $ cat /dev/zloop-control add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache,stor_elements=%u remove id=%d + degrade_element id=%d,element_id=%u In more details, the options that can be used with the "add" command are as follows.
@@ -144,7 +145,26 @@ the zoned device will not be created. To fully delete a zoned device, after executing the remove operation, the device base directory containing the backing files of the device zones must be deleted. -4) Example +4) Degrading a storage element +------------------------------ + +When storage element emulation is enabled, storage elements can be set in a +degraded state to emulate defective storage elements (e.g. defective heads of a +hard disk). This is done by issuing the "degrade_element" command to +/dev/zloop-control, specifying the ID of the target device and the ID of the +storage element to degrade:: + + $ echo "degrade_element id=X,element_id=Y" > /dev/zloop-control + +Depending on the type of the storage element specified, a read or write access +to sectors of a zone served by a degraded storage element will result in a +failure. For partial access storage elements, write accesses are always failed +if the write element serving the zone is degraded. Likewise, read accesses are +failed if the read element serving the zone is degraded. When storage elements +are emulated as all access physical elements, both read and write operations are +failed when the storage element serving the zones being accessed is degraded. + +5) Example ---------- The following sequence of commands creates a 2GB zoned device with zones of 64
drivers/block/zloop.c +98 -2
184 lines not on this page · raw message
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 10/13 scsi: scsi_proto: refine the definition of storage element health
In reply to Damien Le MoalA disk may report the health of a storage element as being within or outside of the manufacturer's specification limits using a range of values for the health field. However, in scsi_proto.h, we only define the start of each range using the SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS and SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS enum value.
Define the full range by renaming SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS to SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MIN and by defining the healthy range end with SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MAX. The same treatment is done for the outside range with the renaming of SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS to SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MIN, and the definition of the range upper value with SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MAX.
--- include/scsi/scsi_proto.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
include/scsi/scsi_proto.h +4 -2
@@ -408,9 +408,11 @@ enum scsi_phys_element_type {/* SCSI physical element health. */ enum scsi_phys_element_health {SCSI_PHYS_ELEM_HEALTH_NOT_REPORTED =0x00 , -SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS =0x01 , +SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MIN =0x01 , +SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MAX =0x63 ,SCSI_PHYS_ELEM_HEALTH_AT_SPEC_LIMITS =0x64 , -SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS =0x65 , +SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MIN =0x65 , +SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MAX =0xCF ,SCSI_PHYS_ELEM_HEALTH_DEPOP_REVOKE_ERR =0xFB ,SCSI_PHYS_ELEM_HEALTH_DEPOP_REVOKE_IN_PROGRESS =0xFC ,SCSI_PHYS_ELEM_HEALTH_DEPOP_ERR =0xFD ,
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 11/13 scsi: sd_zbc: always revalidate zones for disks supporting head depopulation
In reply to Damien Le Moalsd_zbc_revalidate_zones() skips revalidating the zones of a ZBC device if the zone size and total number of zones of the disk has not changed. This is to avoid the sometimes slow report zones for the entire disk that blk_revalidate_disk_zones() will generate.
For ZBC devices that support the storage element depopulation feature, a revalidation after executing a passthrough REMOVE ELEMENT AND TRUNCATE command will thus correctly trigger a zone revalidation as the disk capacity and number of zones changed.
However, for devices supporting the data preserving head depopulation variant of the depopulation feature (REMOVE ELEMENT AND MODIFY ZONES command), a disk capacity and number of zones does not change after a head is depopulated. Only the conditions of the zones served by the head that was depopulated change. These zones are transitioned to either read-only or offline.
While the block layer storage element management functions always call blk_revalidate_disk_zones() to catch these changes, a user scan, libata device rescan after EH or a depopulation or restoration action executed through SG_IO would not. In these cases, not calling blk_revalidate_disk_zones() prevents the block layer from taking appropriate actions on the zone write plugs of the disk.
Avoid any issue with the block layer view of the zone conditions by not skipping the call to blk_revalidate_disk_zones() for disks that support both the GET PHYSICAL ELEMENTS STATUS and REMOVE ELEMENT AND MODIFY ZONES commands. This check is done from sd_zbc_read_zones() using the helper function sd_zbc_check_depop_supported(). The new SCSI disk flags depop_report_supported and depop_modify_zones_supported are defined to remember the result of this check.
This change will slow down a little SMR disk revalidation. However, user forced revalidations are normally rare, and other revalidations triggered by udev only happen when the disk device files is not in use anymore (closed). So this slow down should in practice not be an issue.
--- drivers/scsi/sd.h | 2 ++ drivers/scsi/sd_zbc.c | 47 +++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 47 insertions(+), 2 deletions(-)
drivers/scsi/sd.h +2 -0
@@ -156,6 +156,8 @@ struct scsi_disk { unsigned ignore_medium_access_errors : 1; unsigned rscs : 1; /* reduced stream control support */ unsigned use_atomic_write_boundary : 1; + unsigned depop_report_supported : 1; + unsigned depop_modify_zones_supported : 1; };#define to_scsi_disk (obj) container_of(obj, struct scsi_disk, disk_dev)
drivers/scsi/sd_zbc.c +45 -2
81 lines not on this page · raw message
Signature · 1 line
2.56.0
Damien Le Moal
maintainerPATCH v9 12/13 scsi: sd_zbc: define storage element management operations
In reply to Damien Le MoalDefine the storage element management operations using struct blk_se_ops. These operations are valid only on SMR disks supporting the storage element depopulation featu, with support for the GET PHYSICAL ELEMENTS STATUS command being mandatory. The support for the modify_zones operation is already indicated by the depop_modify_zones_supported SCSI disk flag. sd_zbc_check_depop_supported() is modified to add detection for the command RESTORE ELEMENTS AND REBUILD, which is indicated with the new SCSI disk flag depop_restore_supported.
Both the remove_element and restore_elements operations will block until the operation completes or an error is detected.
--- drivers/scsi/sd.c | 3 + drivers/scsi/sd.h | 3 + drivers/scsi/sd_zbc.c | 493 +++++++++++++++++++++++++++++++++++++++++- 3 files changed, 498 insertions(+), 1 deletion(-)
drivers/scsi/sd.c +3 -0
@@ -3939,6 +3939,9 @@ static const struct block_device_operations sd_fops = { .get_unique_id = sd_get_unique_id, .free_disk = scsi_disk_free_disk, .pr_ops = &sd_pr_ops, +#ifdef CONFIG_BLK_DEV_ZONED + .se_ops = &sd_zbc_se_ops, +#endif };/**
drivers/scsi/sd.h +3 -0
@@ -158,6 +158,7 @@ struct scsi_disk { unsigned use_atomic_write_boundary : 1; unsigned depop_report_supported : 1; unsigned depop_modify_zones_supported : 1; + unsigned depop_restore_supported : 1; };#define to_scsi_disk (obj) container_of(obj, struct scsi_disk, disk_dev)
@@ -244,6 +245,8 @@ unsigned int sd_zbc_complete(struct scsi_cmnd *cmd, unsigned int good_bytes,int sd_zbc_report_zones (struct gendisk *disk,sector_t sector,unsigned int nr_zones,struct blk_report_zones_args *args); +extern const struct blk_se_ops sd_zbc_se_ops; +#else /* CONFIG_BLK_DEV_ZONED */ static inline int sd_zbc_read_zones(struct scsi_disk *sdkp,
drivers/scsi/sd_zbc.c +492 -1
529 lines not on this page · raw message
Signature · 1 line
2.56.0
Re: [PATCH v9 12/13] scsi: sd_zbc: define storage element management operations
Quoted 12 lines · Damien Le Moal
Define the storage element management operations using struct
blk_se_ops. These operations are valid only on SMR disks supporting the storage element depopulation featu, with support for the GET PHYSICAL ELEMENTS STATUS command being mandatory. The support for themodify_zonesoperation is already indicated by thedepop_modify_zones_supportedSCSI disk flag.sd_zbc_check_depop_supported()is modified to add detection for the command RESTORE ELEMENTS AND REBUILD, which is indicated with the new SCSI disk flagdepop_restore_supported.Both the
remove_elementandrestore_elementsoperations will block until the operation completes or an error is detected.
Quoted 7 lines
Signed-off-by Damien Le Moal <[email protected]>--- drivers/scsi/sd.c | 3 + drivers/scsi/sd.h | 3 + drivers/scsi/sd_zbc.c | 493 +++++++++++++++++++++++++++++++++++++++++- 3 files changed, 498 insertions(+), 1 deletion(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Damien Le Moal
maintainerPATCH v9 13/13 scsi: sd: fail read commands to offline zones early
In reply to Damien Le MoalIssuing read commands to offline zones of a ZBC disk will result in failures. So do not issue such command and fail them early in sd_setup_read_write_cmnd(). This change also checks write commands, which is fine but should actually never happen as the block layer zone write plugging code already fails write BIOs to offline zones early in the submission path.
As the code comment explains, since disk_zone_is_offline() uses the block layer cached zone condition, this is inherently racy with zone revalidation updating the zone condition. However, this will never prevent valid read operations as the only case where we could proceed is if the read is received after a restoration of the disk storage element completes but before the zones are revalidated. And in that case, all zones are empty and thus have no data to read.
--- drivers/scsi/sd.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
drivers/scsi/sd.c +19 -1
@@ -1424,7 +1424,8 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd) {struct request *rq =scsi_cmd_to_rq (cmd);struct scsi_device *sdp = cmd->device; -struct scsi_disk *sdkp =scsi_disk (rq->q->disk); +struct gendisk *disk = rq->q->disk; +struct scsi_disk *sdkp =scsi_disk (disk);sector_t lba =sectors_to_logical (sdp,blk_rq_pos (rq));sector_t threshold;unsigned int nr_blocks =sectors_to_logical (sdp,blk_rq_sectors (rq));
@@ -1456,6 +1457,23 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd) goto fail; } +/* +* The block layer already prevents writes to offline zones of zoned +* block devices. But there are no checks for reads, so check this here. +* Since disk_zone_is_offline() uses the block layer cached zone +* condition, this is inherently racy with zone revalidation updating +* the zone condition. However, this will never prevent valid read +* operations as the only case where we could proceed is if the read is +* received after a restoration of the disk storage element completes +* but before the zones are revalidated. And in that case, all zones are +* empty and thus have no data to read. +* Note that we are quiet about the error (no message) to avoid flooding +* the kernel log. +*/ +if (sdkp->device->type ==TYPE_ZBC && +disk_zone_is_offline (disk,blk_rq_pos (rq))) + goto fail; +/* * Some SD card readers can't handle accesses which touch the * last one or two logical blocks. Split accesses as needed.
Signature · 1 line
2.56.0
Damien Le Moal
maintainerQuoted 10 lines · Damien Le Moal
Jens,
These patches define a new set of block device operations for generically using from the block layer the storage element depopulation feature of zoned block devices. Support for this feature is added to the SCSI disk driver and an emulation of this feature added to the zloop driver.
The management operations can be accessed using block layer API, which is intended for file systems (e.g. zonefs and XFS), as well as using ioctls for users using zoned disks directly from user space.
I had a -ENEEDMORECOFFEE error on this one. Sashiko pointed out my mistakes on patches 6 and 8. Will send v10 with the corrections. Please ignore this version. My apologies for the noise.
Signature · 2 lines
Damien Le Moal Western Digital Research