Skip to contentSkip to conversation

Add support for storage element depopulation

PATCH Superseded Damien Le Moal 3 participants 21 messages Nov 16 – Nov 19

Subsystems (7)
Documentation VFS SCSI Block layer +3 more subsystems

v11 of 12 Superseded Latest is v12 →

lore
Share & tools
lore

Damien Le Moal

author maintainer
#0 lore raw

v11 cover Add support for storage element depopulation

Jens,

These patches define a new set of block device operations for generically using from the block layer the storage element depopulation feature of zoned block devices. Support for this feature is added to the SCSI disk driver and an emulation of this feature added to the zloop driver.

The management operations can be accessed using block layer API, which is intended for file systems (e.g. zonefs and XFS), as well as using ioctls for users using zoned disks directly from user space.

Changes from v10:
- Addressed sashiko comment on patch 8 and 12.
- Added review tags

Changes from v9:
- Addressed Sashiko comments on patch 6. This change redesigns the ordered zone append handling to simplify the code and avoid locking difficulties.
- Addressed Sashiko comments on patch 8.

Changes from v8:
- Addressed Sashiko comment on patch 7 by adding patch 6. Patch 7 is unmodified.
- Addressed Sashiko comments on patch 8 and patch 12
- Added review tags to unmodified pacthes

Changes from v7:
A significant amount of changes in this version to address most of the Sashiko review comments and to simplify the code. In more details, the changes are:
- Added patches 3, 6, and 9.
- Dropped former patch 5 as the use of irq-disabling spinlocks in zloop is not necessary.
- Reworked patch 4 by migrating the idepop operation wait loops to the device drivers. This simplifies the implementation in zloop without making the sd driver implementation more difficult. Also fixed problems with errors during restoration and zone write plugs state.
- Reworked patch 7 and 11 to be consistent with the changes in patch 4.

Changes from v6:
- Addressed the nit pointed out by Christoph in patch 4
- Added review tags

Changes from v5:
- Rebased on latest block/for-next
- Addressed Christoph's comments on patch 3
- Added patch 5 to address a potential deadlock in zloop (signaled by sashiko)
- Reworked the restoration of storage elements in patch 6 to avoid an inconsistent state of storage elements in case of error. Also added protection against disk removal during storage element removal and restoration execution.
- Added review tags

Changes from v4:
- Rebased on latest block/for-next
- Added review tags to patch 1 and 2
- Reworked patch 3, 4 and 5 to address Christoph's and sashiko comments

Changes from v3:
- Reworked patch 1 to only provide the disk_zone_is_offline() helper function. Failing of read operations to offline zones is handled directly by the zloop driver and the new patch 7 is added to handle failing early read operations to offline zones of ZBC SCSI disks.
- Added patch 2
- Addressed further valid comments from Sashiko
The biggest changes are in patch 2 to save the number of storage elements of a disk in struct gendisk and to serialize the execution of the storage element management operations using the zone revalidation mutex.
- Rebased on latest block/for-next

Changes from v2:
- Addressed further valid comments from Sashiko
- Rebased on latest block/for-next

Changes from v1:
- Addressed the many valid comments that Sashiko had
- Rebased on latest block/for-next

Damien Le Moal (13):
block: introduce disk_zone_is_offline()
block: refactor blk_revalidate_disk_zones()
block: allow dropping all zone write plugs on revalidation
block: introduce storage element management
block: add storage element management ioctls
zloop: fix ordered zone append handling
zloop: fix zone reporting consistency
zloop: add storage element emulation
zloop: add degrade_element control command
scsi: scsi_proto: refine the definition of storage element health
scsi: sd_zbc: always revalidate zones for disks supporting head depopulation
scsi: sd_zbc: define storage element management operations
scsi: sd: fail read commands to offline zones early

 .../admin-guide/blockdev/zoned_loop.rst       |  33 +-
 block/blk-zoned.c                             | 450 ++++++++-
 block/blk.h                                   |  14 +
 block/genhd.c                                 |   4 +
 block/ioctl.c                                 |   4 +
 drivers/block/zloop.c                         | 917 ++++++++++++++++--
 drivers/scsi/sd.c                             |  23 +-
 drivers/scsi/sd.h                             |   5 +
 drivers/scsi/sd_zbc.c                         | 534 +++++++++-
 include/linux/blkdev.h                        |  64 ++
 include/scsi/scsi_proto.h                     |   6 +-
 include/uapi/linux/blkzoned.h                 | 106 ++
 include/uapi/linux/fs.h                       |   1 +
 13 files changed, 2033 insertions(+), 128 deletions(-)
Signature · 1 line
2.56.1
author maintainer
#1 lore raw

PATCH v11 1/13 block: introduce disk_zone_is_offline()

Introduce the helper function disk_zone_is_offline() for testing if a zone of a gendisk has the offline condition. The zone to test is identified using a sector number that must belong to the target zone to check.

---
 block/blk-zoned.c      | 18 ++++++++++++++++++
 include/linux/blkdev.h |  9 +++++++++
 2 files changed, 27 insertions(+)
block/blk-zoned.c +18 -0
index 4c5889296191..07d504287024 100644
@@ -292,6 +292,24 @@ static bool disk_zone_is_seq(struct gendisk *disk, sector_t sector)
 	return !blk_zstate_is_conv(zs);
 }
 
+/**
+ * disk_zone_is_offline - check if a sector belongs to an offline zone
+ * @disk:	gendisk to check
+ * @sector:	sector number
+ *
+ * Check if @sector on @disk is contained in an offline zone.
+ * This function can be used only when the bdev queue was entered (BIOs
+ * submitted but not yet completed) or when holding zone_revalidate_mutex.
+ */
+bool disk_zone_is_offline(struct gendisk *disk, sector_t sector)
+{
+	if (!blk_queue_is_zoned(disk->queue))
+		return false;
+
+	return disk_zone_get_cond(disk, sector) == BLK_ZONE_COND_OFFLINE;
+}
+EXPORT_SYMBOL_GPL(disk_zone_is_offline);
+
 /**
  * bdev_zone_is_seq - check if a sector belongs to a sequential write zone
  * @bdev:       block device to check
include/linux/blkdev.h +9 -0
index 215753171cef..a2c36b4e6308 100644
@@ -940,6 +940,9 @@ static inline unsigned int disk_zone_capacity(struct gendisk *disk,
 		return disk->last_zone_capacity;
 	return disk->zone_capacity;
 }
+
+bool disk_zone_is_offline(struct gendisk *disk, sector_t sector);
+
 static inline unsigned int bdev_zone_capacity(struct block_device *bdev,
 					      sector_t pos)
 {
@@ -954,6 +957,12 @@ static inline unsigned int disk_nr_zones(struct gendisk *disk)
 	return 0;
 }
 
+static inline bool disk_zone_is_offline(struct gendisk *disk,
+					sector_t sector)
+{
+	return false;
+}
+
 static inline bool bdev_zone_is_seq(struct block_device *bdev, sector_t sector)
 {
 	return false;
Signature · 1 line
2.56.1
author maintainer
#2 lore raw

PATCH v11 2/13 block: refactor blk_revalidate_disk_zones()

In reply to Damien Le Moal

Split out most of the code of blk_revalidate_disk_zones() into the internal function disk_revalidate_zones(). This will allow calling this new helper together with other code under the zone revalidation mutex. No functional change.

---
 block/blk-zoned.c | 63 +++++++++++++++++++++++++----------------------
 1 file changed, 34 insertions(+), 29 deletions(-)
block/blk-zoned.c +34 -29
index 07d504287024..600d6bd008ae 100644
@@ -2552,20 +2552,7 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,
 	return ret;
 }
 
-/**
- * blk_revalidate_disk_zones - (re)allocate and initialize zone write plugs
- * @disk:	Target disk
- *
- * Helper function for low-level device drivers to check, (re) allocate and
- * initialize resources used for managing zoned disks. This function should
- * normally be called by blk-mq based drivers when a zoned gendisk is probed
- * and when the zone configuration of the gendisk changes (e.g. after a format).
- * Before calling this function, the device driver must already have set the
- * device zone size (chunk_sector limit) and the max zone append limit.
- * BIO based drivers can also use this function as long as the device queue
- * can be safely frozen.
- */
-int blk_revalidate_disk_zones(struct gendisk *disk)
+static int disk_revalidate_zones(struct gendisk *disk)
 {
 	struct blk_revalidate_zone_args args = { };
 	struct blk_report_zones_args rep_args = {
@@ -2576,19 +2563,10 @@ int blk_revalidate_disk_zones(struct gendisk *disk)
 	int retries = 2;
 	int ret;
 
-	if (WARN_ON_ONCE(!blk_queue_is_zoned(disk->queue)))
-		return -EIO;
-
-	/*
-	 * Serialize calls to this function so that we can safely look at and
-	 * eventually change the disk zone information.
-	 */
-	mutex_lock(&disk->zone_revalidate_mutex);
-
 again:
 	ret = disk_revalidate_capacity(disk, &args);
 	if (ret)
-		goto unlock;
+		return ret;
 
 	/*
 	 * Allocate zone resources if they are needed and we have not done
@@ -2601,14 +2579,14 @@ int blk_revalidate_disk_zones(struct gendisk *disk)
 		ret = disk_alloc_zone_resources(disk, args.capacity);
 		if (ret) {
 			memalloc_noio_restore(noio_flag);
-			goto unlock;
+			return ret;
 		}
 	}
 
 	ret = disk_init_revalidate_args(disk, &args);
 	if (ret) {
 		memalloc_noio_restore(noio_flag);
-		goto unlock;
+		return ret;
 	}
 
 	ret = disk->fops->report_zones(disk, 0, UINT_MAX, &rep_args);
@@ -2625,8 +2603,6 @@ int blk_revalidate_disk_zones(struct gendisk *disk)
 	if (ret)
 		goto free_args;
 
-	mutex_unlock(&disk->zone_revalidate_mutex);
-
 	return 0;
 
 free_args:
@@ -2642,7 +2618,36 @@ int blk_revalidate_disk_zones(struct gendisk *disk)
 	}
 
 	pr_warn("%s: failed to revalidate zones\n", disk->disk_name);
-unlock:
+
+	return ret;
+}
+
+/**
+ * blk_revalidate_disk_zones - (re)allocate and initialize zone write plugs
+ * @disk:	Target disk
+ *
+ * Helper function for low-level device drivers to check, (re) allocate and
+ * initialize resources used for managing zoned disks. This function should
+ * normally be called by blk-mq based drivers when a zoned gendisk is probed
+ * and when the zone configuration of the gendisk changes (e.g. after a format).
+ * Before calling this function, the device driver must already have set the
+ * device zone size (chunk_sector limit) and the max zone append limit.
+ * BIO based drivers can also use this function as long as the device queue
+ * can be safely frozen.
+ */
+int blk_revalidate_disk_zones(struct gendisk *disk)
+{
+	int ret;
+
+	if (WARN_ON_ONCE(!blk_queue_is_zoned(disk->queue)))
+		return -EIO;
+
+	/*
+	 * Serialize calls to this function so that we can safely look at and
+	 * eventually change the disk zone information.
+	 */
+	mutex_lock(&disk->zone_revalidate_mutex);
+	ret = disk_revalidate_zones(disk);
 	mutex_unlock(&disk->zone_revalidate_mutex);
 
 	return ret;
Signature · 1 line
2.56.1
author maintainer
#3 lore raw

PATCH v11 3/13 block: allow dropping all zone write plugs on revalidation

In reply to Damien Le Moal

In preparation for supporting storage element depopulation, modify the function disk_revalidate_zones() to take an additional boolean parameter drop_all_zwplugs to force removing all zone write plugs from a disk hash table. Removing all zone write plugs is done in the same manner as for a capacity change, using the function disk_drop_zone_wplug() applied to all hashed zone write plugs.

Calls to disk_revalidate_zones() with the argument drop_all_zwplugs set to true will be used when restoring depopulated storage elements as such operation clears all sequential zones, modifying them to be empty.

---
 block/blk-zoned.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
block/blk-zoned.c +5 -2
index 600d6bd008ae..3761e634650e 100644
@@ -2552,7 +2552,7 @@ static int blk_revalidate_zone_cb(struct blk_zone *zone, unsigned int idx,
 	return ret;
 }
 
-static int disk_revalidate_zones(struct gendisk *disk)
+static int disk_revalidate_zones(struct gendisk *disk, bool drop_all_zwplugs)
 {
 	struct blk_revalidate_zone_args args = { };
 	struct blk_report_zones_args rep_args = {
@@ -2563,6 +2563,9 @@ static int disk_revalidate_zones(struct gendisk *disk)
 	int retries = 2;
 	int ret;
 
+	if (drop_all_zwplugs)
+		disk_for_all_zone_wplugs(disk, disk_drop_zone_wplug, NULL);
+
 again:
 	ret = disk_revalidate_capacity(disk, &args);
 	if (ret)
@@ -2647,7 +2650,7 @@ int blk_revalidate_disk_zones(struct gendisk *disk)
 	 * eventually change the disk zone information.
 	 */
 	mutex_lock(&disk->zone_revalidate_mutex);
-	ret = disk_revalidate_zones(disk);
+	ret = disk_revalidate_zones(disk, false);
 	mutex_unlock(&disk->zone_revalidate_mutex);
 
 	return ret;
Signature · 1 line
2.56.1
author maintainer
#4 lore raw

PATCH v11 4/13 block: introduce storage element management

In reply to Damien Le Moal

Recent SCSI (SBC) and ATA (ACS) standards define the storage element depopulation feature. This feature is intended for managing hard-disks heads, either combined read-write heads or pairs of read and write heads, allowing to keep disks with defective heads longer in production by allowing "depopulating" (removing) defective heads.

This feature comes in two different flavors:
- A destructive version which removes a head and reformats the disk at a lower capacity point, restoring a fully functional contiguous LBA address space.
- A data preserving version restricted to host-managed zoned disks, which marks the zones served by a removed head as offline or read-only.

In preparation for using the data-preserving flavor of the depopulation feature in file systems natively supporting zoned block devices, introduce a set of storage element management operations and functions to define generic calls into block device drivers for managing storage elements.

The set of operations is defined with struct blk_se_ops and includes three operations:
- report_elements: get information on a device storage elements state reported with the new struct blk_se.
- remove_element: depopulate a defective storage element, blocking until the operation completes.
- restore_elements: restore depopulated storage elements, blocking until the operation completes.

Each operation is called from the functions bdev_report_storage_elements(), bdev_remove_storage_element() and bdev_restore_storage_elements() respectively.

Removing a healthy storage element from a device is possible and useful for testing. The storage element restoration operation restore_elements allows repopulating such healthy element. Repopulating defective storage elements is generally not allowed by devices.

The remove_element and restore_elements operations are executed under the zone revalidation lock so that they are serialized against each other and also serialized against user initiated device revalidation, thus allowing to always have a consistent view of the device zone conditions after executing these operations. In case of failure, zone revalidation is still executed, which may give an unstable view of the zone states. In such case, stable zone states will be recovered with zone access error handling or later zone revalidation.

The device drivers of zoned block devices can indicate support for the storage element depopulation feature by specifying the storage element management operations with the se_ops field of struct block_device_operations.

---
 block/blk-zoned.c             | 205 ++++++++++++++++++++++++++++++++++
 block/blk.h                   |   5 +
 block/genhd.c                 |   4 +
 include/linux/blkdev.h        |  55 +++++++++
 include/uapi/linux/blkzoned.h |  65 +++++++++++
 5 files changed, 334 insertions(+)
block/blk-zoned.c +205 -0
index 3761e634650e..5b27a639cd2c 100644
@@ -2736,3 +2736,208 @@ int queue_zone_wplugs_show(void *data, struct seq_file *m)
 }
 
 #endif
+
+/**
+ * bdev_report_storage_elements - report the storage elements of a zoned
+ *				  block device
+ * @bdev:	the zoned block device to check
+ * @elements:	array of storage element descriptors to fill
+ * @nr_elements: maximum size of @elements
+ *
+ * Fill at most @nr_elements storage element descriptors in the array
+ * @elements. The number of storage elements filled in the array is returned
+ * using @nr_elements. If @elements is NULL, only @nr_elements is returned to
+ * indicate the maximum number of storage elements of @bdev. This function may
+ * sleep.
+ *
+ * Returns 0 on success and a negative error code on failure.
+ */
+int bdev_report_storage_elements(struct block_device *bdev,
+				 struct blk_se *elements,
+				 unsigned int *nr_elements)
+{
+	struct gendisk *disk = bdev->bd_disk;
+
+	if (test_bit(GD_DEAD, &disk->state))
+		return -ENODEV;
+
+	if (!blk_queue_is_zoned(disk->queue) || !disk->nr_storage_elements)
+		return -EOPNOTSUPP;
+
+	if (!nr_elements)
+		return -EINVAL;
+
+	if (!elements) {
+		*nr_elements = disk->nr_storage_elements;
+		return 0;
+	}
+
+	*nr_elements = min(*nr_elements, disk->nr_storage_elements);
+	if (!*nr_elements)
+		return -EINVAL;
+
+	return disk->fops->se_ops->report_elements(disk, elements,
+						   nr_elements);
+}
+EXPORT_SYMBOL_GPL(bdev_report_storage_elements);
+
+/**
+ * bdev_remove_storage_element - Remove (depopulate) a storage element of a
+ *				 zoned block device
+ * @bdev:	block device to operate on
+ * @element_id:	ID of the storage element to remove
+ *
+ * Remove (depopulate) the storage element identified by @element_id from the
+ * block device @bdev. The caller is responsible for taking care of any
+ * necessary dirty page and device write cache flush and invalidation of cached
+ * data. This function will block until the storage element is removed or an
+ * error occurs.
+ *
+ * Returns 0 on success and a negative error code on failure.
+ */
+int bdev_remove_storage_element(struct block_device *bdev,
+				unsigned int element_id)
+{
+	struct gendisk *disk = bdev->bd_disk;
+	unsigned int memflags;
+	int ret, err;
+
+	if (test_bit(GD_DEAD, &disk->state))
+		return -ENODEV;
+
+	if (!blk_queue_is_zoned(disk->queue) || !disk->nr_storage_elements)
+		return -EOPNOTSUPP;
+
+	/* Zero is not a valid storage element ID. */
+	if (!element_id)
+		return -EINVAL;
+
+	/*
+	 * Serialize against concurrent revalidation and concurrent storage
+	 * element management operations so that we can see consistent zone
+	 * condition changes.
+	 */
+	mutex_lock(&disk->zone_revalidate_mutex);
+
+	/*
+	 * Freeze and unfreeze the queue to flush any outstanding command.
+	 * The caller is responsible for not queuing up more I/Os by higher
+	 * level means.
+	 */
+	memflags = blk_mq_freeze_queue(disk->queue);
+	blk_mq_unfreeze_queue(disk->queue, memflags);
+
+	/* Initiate the storage element removal and wait for its completion. */
+	ret = disk->fops->se_ops->remove_element(disk, element_id);
+
+	/*
+	 * Always revalidate zones, even in the case of an error as we do not
+	 * know what condition the zones of the block device are in.
+	 */
+	err = disk_revalidate_zones(disk, false);
+	if (err && !ret)
+		ret = err;
+
+	mutex_unlock(&disk->zone_revalidate_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(bdev_remove_storage_element);
+
+/**
+ * bdev_restore_storage_elements - Restore all depopulated storage elements of
+ *				   a zoned block device
+ * @bdev:	block device to operate on
+ *
+ * Restore all storage elements of @bdev that have been depopulated. Not all
+ * elements may be restored by this operation. This operation is destructive and
+ * will result in all sequential zones to be transitioned to the empty condition.
+ * The caller is responsible for taking care of any necessary invalidation of
+ * cached data. This function will block until the storage elements are restored
+ * or an error occurs.
+ *
+ * Returns 0 on success and a negative error code on failure.
+ */
+int bdev_restore_storage_elements(struct block_device *bdev)
+{
+	struct gendisk *disk = bdev->bd_disk;
+	unsigned int memflags;
+	int ret, err;
+
+	if (test_bit(GD_DEAD, &disk->state))
+		return -ENODEV;
+
+	if (!blk_queue_is_zoned(disk->queue) || !disk->nr_storage_elements)
+		return -EOPNOTSUPP;
+
+	/*
+	 * Serialize against concurrent revalidation and concurrent storage
+	 * element management operations so that we can see consistent zone
+	 * condition changes.
+	 */
+	mutex_lock(&disk->zone_revalidate_mutex);
+
+	/*
+	 * Freeze and unfreeze the queue to flush any outstanding commands.
+	 * The caller is responsible for not queuing up more I/O by higher
+	 * level means.
+	 */
+	memflags = blk_mq_freeze_queue(disk->queue);
+	blk_mq_unfreeze_queue(disk->queue, memflags);
+
+	/*
+	 * Initiate the storage elements restoration and wait for its
+	 * completion.
+	 */
+	ret = disk->fops->se_ops->restore_elements(disk);
+
+	/*
+	 * Always revalidate zones, even in the case of an error as we do not
+	 * know what condition the zones of the block device are in. Note that
+	 * since all sequential zones should have been transitioned to the
+	 * empty condition, we can drop all zone write plugs. If an error
+	 * occurred and not all sequential zones are empty, the revalidation
+	 * will reinstate the write plugs for these zones.
+	 */
+	err = disk_revalidate_zones(disk, true);
+	if (err && !ret)
+		ret = err;
+
+	mutex_unlock(&disk->zone_revalidate_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(bdev_restore_storage_elements);
+
+int disk_init_storage_element_mgmt(struct gendisk *disk)
+{
+	int ret;
+
+	/*
+	 * Storage element operations are for zoned disks only and are
+	 * optional...
+	 */
+	if (!blk_queue_is_zoned(disk->queue) || !disk->fops->se_ops)
+		return 0;
+
+	/* ...but if defined, all operation callbacks must be defined. */
+	if (!disk->fops->se_ops->report_elements ||
+	    !disk->fops->se_ops->remove_element ||
+	    !disk->fops->se_ops->restore_elements) {
+		pr_err("%s: invalid storage element operations\n",
+		       disk->disk_name);
+		return -EINVAL;
+	}
+
+	/*
+	 * Get the number of storage elements of the disk. If this fails,
+	 * do not report an error as the failure may be because the zoned disk
+	 * does not support storage element management.
+	 */
+	ret = disk->fops->se_ops->report_elements(disk, NULL,
+						  &disk->nr_storage_elements);
+	if (ret)
+		disk->nr_storage_elements = 0;
+
+	return 0;
+}
block/blk.h +5 -0
index 0e38e4b88cf5..05e2692b80d5 100644
@@ -543,6 +543,7 @@ static inline void ioc_clear_queue(struct request_queue *q)
 #endif /* CONFIG_BLK_ICQ */
 
 #ifdef CONFIG_BLK_DEV_ZONED
+int disk_init_storage_element_mgmt(struct gendisk *disk);
 void disk_init_zone_resources(struct gendisk *disk);
 void disk_release_zone_resources(struct gendisk *disk);
 static inline bool bio_zone_write_plugging(struct bio *bio)
@@ -595,6 +596,10 @@ int blkdev_zone_mgmt_ioctl(struct block_device *bdev, blk_mode_t mode,
 		unsigned int cmd, unsigned long arg);
 bool bdev_zone_mgmt_allowed(struct block_device *bdev, sector_t sector);
 #else /* CONFIG_BLK_DEV_ZONED */
+static inline int disk_init_storage_element_mgmt(struct gendisk *disk)
+{
+	return 0;
+}
 static inline void disk_init_zone_resources(struct gendisk *disk)
 {
 }
block/genhd.c +4 -0
index 69204d44adae..251a5b07baff 100644
@@ -435,6 +435,10 @@ static int __add_disk(struct device *parent, struct gendisk *disk,
 	if (WARN_ON_ONCE(bdev_nr_sectors(disk->part0) > BLK_DEV_MAX_SECTORS))
 		return -EINVAL;
 
+	ret = disk_init_storage_element_mgmt(disk);
+	if (ret)
+		return ret;
+
 	if (queue_is_mq(disk->queue)) {
 		/*
 		 * ->submit_bio and ->poll_bio are bypassed for blk-mq drivers.
include/linux/blkdev.h +55 -0

85 lines not on this page · raw message

include/uapi/linux/blkzoned.h +65 -0

69 lines not on this page · raw message

Signature · 1 line
2.56.1
author maintainer
#5 lore raw

PATCH v11 5/13 block: add storage element management ioctls

In reply to Damien Le Moal

Define the new ioctl commands BLKREPORTSTORELEMS, BLKREMOVESTORELEM, and BLKRESTORESTORELEMS to provide users accessing zoned block devices directly with an interface to the storage elements management operations of the block layer.

The ioctl BLKREPORTSTORELEMS allows to get the total number of storage elements of a zoned block device and to get an array of storage element descriptors (struct blk_se) describing the current state of the device storage elements.

The ioctl BLKREMOVESTORELEM can be used to remove (depopulate) a storage element, in particular storage elements that have a degraded status (i.e. BLK_SE_STS_DEGRADED). But healthy elements (BLK_SE_STS_OK) can also be removed, and later restored.

Finally, the BLKRESTORESTORELEMS ioctl interfaces with bdev_restore_storage_elements() to restore the depopulated storage elements of a zoned device.

---
 block/blk-zoned.c             | 163 ++++++++++++++++++++++++++++++++++
 block/blk.h                   |   9 ++
 block/ioctl.c                 |   4 +
 include/uapi/linux/blkzoned.h |  41 +++++++++
 include/uapi/linux/fs.h       |   1 +
 5 files changed, 218 insertions(+)
block/blk-zoned.c +163 -0
index 5b27a639cd2c..e5147a4d2101 100644
@@ -18,6 +18,7 @@
 #include <linux/mempool.h>
 #include <linux/kthread.h>
 #include <linux/freezer.h>
+#include <linux/uaccess.h>
 
 #include <trace/events/block.h>
 
@@ -2781,6 +2782,53 @@ int bdev_report_storage_elements(struct block_device *bdev,
 }
 EXPORT_SYMBOL_GPL(bdev_report_storage_elements);
 
+static int blkdev_ioctl_report_storage_elements(struct block_device *bdev,
+						void __user *argp)
+{
+	struct gendisk *disk = bdev->bd_disk;
+	struct blk_se_report *rep;
+	int ret = 0;
+
+	/*
+	 * Allocate the report header with an array of elements, assuming that
+	 * we need the maximum number of storage elements. In practice, we may
+	 * need less than that, but this potentially unnecessary preallocation
+	 * simplifies the code.
+	 */
+	rep = kzalloc_flex(*rep, elements, disk->nr_storage_elements);
+	if (!rep)
+		return -ENOMEM;
+
+	if (copy_from_user(rep, argp, sizeof(struct blk_se_report))) {
+		ret = -EFAULT;
+		goto free_rep;
+	}
+
+	if (rep->reserved) {
+		ret = -EINVAL;
+		goto free_rep;
+	}
+
+	rep->nr_reported_elements =
+		min(rep->nr_elements, disk->nr_storage_elements);
+	rep->nr_elements = disk->nr_storage_elements;
+
+	if (rep->nr_reported_elements) {
+		ret = bdev_report_storage_elements(bdev, rep->elements,
+						   &rep->nr_reported_elements);
+		if (ret)
+			goto free_rep;
+	}
+
+	if (copy_to_user(argp, rep,
+			struct_size(rep, elements, rep->nr_reported_elements)))
+		ret = -EFAULT;
+
+free_rep:
+	kfree(rep);
+	return ret;
+}
+
 /**
  * bdev_remove_storage_element - Remove (depopulate) a storage element of a
  *				 zoned block device
@@ -2844,6 +2892,62 @@ int bdev_remove_storage_element(struct block_device *bdev,
 }
 EXPORT_SYMBOL_GPL(bdev_remove_storage_element);
 
+static int blkdev_ioctl_remove_storage_element(struct block_device *bdev,
+					       blk_mode_t mode,
+					       unsigned int __user *argp)
+{
+	unsigned int element_id;
+	int ret;
+
+	/*
+	 * Note: we require an O_EXCL open here so that a user cannot trigger
+	 * a storage element removal from user space under a mounted file system.
+	 * An FS that wants to manage storage element can do so using the
+	 * function bdev_remove_storage_element() from kernel context.
+	 */
+	if (!(mode & BLK_OPEN_EXCL))
+		return -EBADF;
+	if (!(mode & BLK_OPEN_WRITE))
+		return -EBADF;
+	if (bdev_read_only(bdev))
+		return -EPERM;
+
+	if (get_user(element_id, argp))
+		return -EFAULT;
+
+	/*
+	 * Sync the device dirty data, flush the volatile write cache, and
+	 * invalidate all cached data so that reads do not return old data for
+	 * zones that went offline.
+	 * Since a user can only successfully write to sequential zones of a
+	 * zoned devices using direct IOs, the sync should only handle dirty
+	 * data from buffered writes to conventional zones, for devices that
+	 * have such zones.
+	 */
+	ret = sync_blockdev(bdev);
+	if (ret)
+		return ret;
+
+	inode_lock(bdev->bd_mapping->host);
+	filemap_invalidate_lock(bdev->bd_mapping);
+
+	ret = blkdev_issue_flush(bdev->bd_disk->part0);
+	if (ret)
+		goto unlock;
+
+	ret = truncate_bdev_range(bdev, mode, 0, bdev_nr_bytes(bdev) - 1);
+	if (ret)
+		goto unlock;
+
+	ret = bdev_remove_storage_element(bdev, element_id);
+
+unlock:
+	filemap_invalidate_unlock(bdev->bd_mapping);
+	inode_unlock(bdev->bd_mapping->host);
+
+	return ret;
+}
+
 /**
  * bdev_restore_storage_elements - Restore all depopulated storage elements of
  *				   a zoned block device
@@ -2909,6 +3013,42 @@ int bdev_restore_storage_elements(struct block_device *bdev)
 }
 EXPORT_SYMBOL_GPL(bdev_restore_storage_elements);
 
+static int blkdev_ioctl_restore_storage_elements(struct block_device *bdev,
+						 blk_mode_t mode)
+{
+	int ret;
+
+	/*
+	 * Note: because restoring storage elements is a destructive operation
+	 * equivalent to reformatting the device, we require an O_EXCL open here
+	 * so that a user cannot trigger with another active user (e.g. a
+	 * mounted file system).
+	 */
+	if (!(mode & BLK_OPEN_EXCL))
+		return -EBADF;
+	if (!(mode & BLK_OPEN_WRITE))
+		return -EBADF;
+	if (bdev_read_only(bdev))
+		return -EPERM;
+
+	inode_lock(bdev->bd_mapping->host);
+	filemap_invalidate_lock(bdev->bd_mapping);
+
+	/*
+	 * Storage element restoration is a destructive operation that will
+	 * reset all zones. So invalidate all cached data that we may have so
+	 * that old data cannot be read once the operation completes.
+	 */
+	ret = truncate_bdev_range(bdev, mode, 0, bdev_nr_bytes(bdev) - 1);
+	if (!ret)
+		ret = bdev_restore_storage_elements(bdev);
+
+	filemap_invalidate_unlock(bdev->bd_mapping);
+	inode_unlock(bdev->bd_mapping->host);
+
+	return ret;
+}
+
 int disk_init_storage_element_mgmt(struct gendisk *disk)
 {
 	int ret;
@@ -2941,3 +3081,26 @@ int disk_init_storage_element_mgmt(struct gendisk *disk)
 
 	return 0;
 }
+
+int blk_ioctl_zone_storage_elements(struct block_device *bdev,
+				    blk_mode_t mode, unsigned int cmd,
+				    void __user *argp)
+{
+	struct gendisk *disk = bdev->bd_disk;
+
+	if (!disk->nr_storage_elements)
+		return -ENOTTY;
+
+	switch (cmd) {
+	case BLKREPORTSTORELEMS:
+		return blkdev_ioctl_report_storage_elements(bdev, argp);
+	case BLKREMOVESTORELEM:
+		return blkdev_ioctl_remove_storage_element(bdev, mode, argp);
+	case BLKRESTORESTORELEMS:
+		return blkdev_ioctl_restore_storage_elements(bdev, mode);
+	default:
+		break;
+	}
+
+	return -ENOTTY;
+}
block/blk.h +9 -0
index 05e2692b80d5..44876b148237 100644
@@ -595,6 +595,9 @@ int blkdev_report_zones_ioctl(struct block_device *bdev, unsigned int cmd,
 int blkdev_zone_mgmt_ioctl(struct block_device *bdev, blk_mode_t mode,
 		unsigned int cmd, unsigned long arg);
 bool bdev_zone_mgmt_allowed(struct block_device *bdev, sector_t sector);
+int blk_ioctl_zone_storage_elements(struct block_device *bdev,
+				    blk_mode_t mode, unsigned int cmd,
+				    void __user *argp);
 #else /* CONFIG_BLK_DEV_ZONED */
 static inline int disk_init_storage_element_mgmt(struct gendisk *disk)
 {
@@ -646,6 +649,12 @@ static inline bool bdev_zone_mgmt_allowed(struct block_device *bdev,
 {
 	return false;
 }
+static inline int blk_ioctl_zone_storage_elements(struct block_device *bdev,
+					blk_mode_t mode, unsigned int cmd,
+					void __user *argp)
+{
+	return -ENOTTY;
+}
 #endif /* CONFIG_BLK_DEV_ZONED */
 
 struct block_device *bdev_alloc(struct gendisk *disk, u8 partno);
block/ioctl.c +4 -0
index fb2f4b4d2505..0f043363048c 100644
@@ -679,6 +679,10 @@ static int blkdev_common_ioctl(struct block_device *bdev, blk_mode_t mode,
 		return put_uint(argp, bdev_zone_sectors(bdev));
 	case BLKGETNRZONES:
 		return put_uint(argp, bdev_nr_zones(bdev));
+	case BLKREPORTSTORELEMS:
+	case BLKREMOVESTORELEM:
+	case BLKRESTORESTORELEMS:
+		return blk_ioctl_zone_storage_elements(bdev, mode, cmd, argp);
 	case BLKROGET:
 		return put_int(argp, bdev_read_only(bdev) != 0);
 	case BLKSSZGET: /* get block device logical block size */
include/uapi/linux/blkzoned.h +41 -0

45 lines not on this page · raw message

include/uapi/linux/fs.h +1 -0
index a46c33692aa2..5b6f8248a479 100644
@@ -309,6 +309,7 @@ struct file_attr {
 /* 130-136 and 142 are used by zoned block device ioctls (uapi/linux/blkzoned.h) */
 /* 137-141 are used by blk-crypto ioctls (uapi/linux/blk-crypto.h) */
 #define BLKTRACESETUP2 _IOWR(0x12, 142, struct blk_user_trace_setup2)
+/* 143-145 are used by storage element management for zoned block devices. */
 
 #define BMAP_IOCTL 1		/* obsolete - kept for compatibility */
 #define FIBMAP	   _IO(0x00,1)	/* bmap access */
Signature · 1 line
2.56.1
author maintainer
#6 lore raw

PATCH v11 6/13 zloop: fix ordered zone append handling

In reply to Damien Le Moal

For oredered zone append commands, the function zloop_set_zone_append_sector() increments a zone write pointer without checking if the zone is already open (implicitly or explicitly). This check is done only when the command is later executed using a work item, and the zone is only then implicitly open if needed. This creates a short time window where a report zone can show an empty zone with a write pointer that is not equal to the zone start sector, which is incorrect.

Calling zloop_do_open_zone() in zloop_set_zone_append_sector() is required but introduces locking complexity as the zone mutex cannot be taken in the context of zloop_queue_rq(). Instead introduce the ordered workqueue append_workqueue to handle zone append commands when ordered zone append is enabled. This ensure that all zone append operations are executed in the order of there submission. This removes the need for setting the command sector in the context of zloop_queue_rq() and we can rely on the reguar command execution done by zloop_rw(). With this change, zloop_seq_write_prep() is simplified and zloop_set_zone_append_sector() is removed entirely.

This change does reduce the command execution concurrency and impact performance with ordered zone append enabled for file systems using zone append. This is however not a critical option and the default unordered zone append handling is unaffected.

Fixes fcc6eaa3a03a ("zloop: introduce the ordered_zone_append configuration parameter")
---
 drivers/block/zloop.c | 113 +++++++++++++++++-------------------------
 1 file changed, 45 insertions(+), 68 deletions(-)
drivers/block/zloop.c +45 -68
index 96666742523b..19949ed3824f 100644
@@ -129,6 +129,7 @@ struct zloop_device {
 	struct gendisk		*disk;
 
 	struct workqueue_struct *workqueue;
+	struct workqueue_struct *append_workqueue;
 	bool			buffered_io;
 	bool			zone_append;
 	bool			ordered_zone_append;
@@ -590,32 +591,24 @@ static int zloop_seq_write_prep(struct zloop_cmd *cmd)
 	 * when submitted.
 	 */
 	if (is_append) {
-		/*
-		 * If ordered zone append is in use, we already checked and set
-		 * the target sector in zloop_queue_rq().
-		 */
-		if (!zlo->ordered_zone_append) {
-			if (zone->cond == BLK_ZONE_COND_FULL ||
-			    zone->wp + nr_sectors > zone_end) {
-				ret = -EIO;
-				goto out_unlock;
-			}
-			cmd->sector = zone->wp;
-		}
-	} else {
-		if (cmd->sector != zone->wp) {
-			if (zone->wp == ULLONG_MAX)
-				pr_err("%pg: Zone %u: write to %s zone: sect %llu\n",
-				       zlo->disk->part0, zone_no,
-				       blk_zone_cond_str(zone->cond),
-				       cmd->sector);
-			else
-				pr_err("%pg: Zone %u: unaligned write: sect %llu, wp %llu\n",
-				       zlo->disk->part0, zone_no, cmd->sector,
-				       zone->wp);
+		if (zone->cond == BLK_ZONE_COND_FULL ||
+		    zone->wp + nr_sectors > zone_end) {
 			ret = -EIO;
 			goto out_unlock;
 		}
+		cmd->sector = zone->wp;
+	} else if (cmd->sector != zone->wp) {
+		if (zone->wp == ULLONG_MAX)
+			pr_err("%pg: Zone %u: write to %s zone: sect %llu\n",
+			       zlo->disk->part0, zone_no,
+			       blk_zone_cond_str(zone->cond),
+			       cmd->sector);
+		else
+			pr_err("%pg: Zone %u: unaligned write: sect %llu, wp %llu\n",
+			       zlo->disk->part0, zone_no, cmd->sector,
+			       zone->wp);
+		ret = -EIO;
+		goto out_unlock;
 	}
 
 	/* Implicitly open the target zone. */
@@ -625,15 +618,13 @@ static int zloop_seq_write_prep(struct zloop_cmd *cmd)
 	}
 
 	/*
-	 * Advance the write pointer, unless ordered zone append is in use. If
-	 * the write fails, the write pointer position will be corrected when
-	 * the next I/O starts execution.
+	 * Advance the write pointer. If the write fails, the write pointer
+	 * position will be corrected when the next I/O starts execution.
 	 */
-	if (!is_append || !zlo->ordered_zone_append) {
-		zone->wp += nr_sectors;
-		if (zone->wp == zone_end)
-			zloop_mark_full(zlo, zone);
-	}
+	zone->wp += nr_sectors;
+	if (zone->wp == zone_end)
+		zloop_mark_full(zlo, zone);
+
 out_unlock:
 	spin_unlock(&zone->wp_lock);
 	return ret;
@@ -885,32 +876,6 @@ static void zloop_complete_rq(struct request *rq)
 	blk_mq_end_request(rq, sts);
 }
 
-static bool zloop_set_zone_append_sector(struct request *rq)
-{
-	struct zloop_device *zlo = rq->q->queuedata;
-	unsigned int zone_no = rq_zone_no(rq);
-	struct zloop_zone *zone = &zlo->zones[zone_no];
-	sector_t zone_end = zone->start + zlo->zone_capacity;
-	sector_t nr_sectors = blk_rq_sectors(rq);
-
-	spin_lock(&zone->wp_lock);
-
-	if (zone->cond == BLK_ZONE_COND_FULL ||
-	    zone->wp + nr_sectors > zone_end) {
-		spin_unlock(&zone->wp_lock);
-		return false;
-	}
-
-	rq->__sector = zone->wp;
-	zone->wp += blk_rq_sectors(rq);
-	if (zone->wp >= zone_end)
-		zloop_mark_full(zlo, zone);
-
-	spin_unlock(&zone->wp_lock);
-
-	return true;
-}
-
 static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,
 				   const struct blk_mq_queue_data *bd)
 {
@@ -923,20 +888,13 @@ static blk_status_t zloop_queue_rq(struct blk_mq_hw_ctx *hctx,
 		return BLK_STS_IOERR;
 	}
 
-	/*
-	 * If we need to strongly order zone append operations, set the request
-	 * sector to the zone write pointer location now instead of when the
-	 * command work runs.
-	 */
-	if (zlo->ordered_zone_append && req_op(rq) == REQ_OP_ZONE_APPEND) {
-		if (!zloop_set_zone_append_sector(rq))
-			return BLK_STS_IOERR;
-	}
-
 	blk_mq_start_request(rq);
 
 	INIT_WORK(&cmd->work, zloop_cmd_workfn);
-	queue_work(zlo->workqueue, &cmd->work);
+	if (zlo->ordered_zone_append && req_op(rq) == REQ_OP_ZONE_APPEND)
+		queue_work(zlo->append_workqueue, &cmd->work);
+	else
+		queue_work(zlo->workqueue, &cmd->work);
 
 	return BLK_STS_OK;
 }
@@ -1029,6 +987,8 @@ static void zloop_free_disk(struct gendisk *disk)
 
 	fput(zlo->data_dir);
 	destroy_workqueue(zlo->workqueue);
+	if (zlo->append_workqueue)
+		destroy_workqueue(zlo->append_workqueue);
 	kfree(zlo->base_dir);
 	kvfree(zlo);
 }
@@ -1285,6 +1245,21 @@ static int zloop_ctl_add(struct zloop_options *opts)
 		goto out_free_idr;
 	}
 
+	/*
+	 * Use an ordered workqueue for strongly ordered zone append operations
+	 * so that we can keep processing the operations in the same order as
+	 * they are submitted.
+	 */
+	if (zlo->ordered_zone_append) {
+		zlo->append_workqueue =
+			alloc_ordered_workqueue("zloop%d_append", WQ_FREEZABLE,
+						zlo->id);
+		if (!zlo->append_workqueue) {
+			ret = -ENOMEM;
+			goto out_destroy_workqueue;
+		}
+	}
+
 	if (opts->base_dir)
 		zlo->base_dir = kstrdup(opts->base_dir, GFP_KERNEL);
 	else
@@ -1391,6 +1366,8 @@ static int zloop_ctl_add(struct zloop_options *opts)
 	kfree(zlo->base_dir);
 out_destroy_workqueue:
 	destroy_workqueue(zlo->workqueue);
+	if (zlo->append_workqueue)
+		destroy_workqueue(zlo->append_workqueue);
 out_free_idr:
 	mutex_lock(&zloop_ctl_mutex);
 	idr_remove(&zloop_index_idr, zlo->id);
Signature · 1 line
2.56.1
author maintainer
#8 lore raw

PATCH v11 7/13 zloop: fix zone reporting consistency

In reply to Damien Le Moal

In zloop_report_zones(), filling the write pointer and condition of a zone descriptor separately outside of the zone wp_lock can lead to an inconsistent zone report with a write pointer potentially not being consistent with the condition being reported. Avoid such inconsistency by filling both fields while holding the zone wp_lock.

Fixes fcc6eaa3a03a ("zloop: introduce the ordered_zone_append configuration parameter")
---
 drivers/block/zloop.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/block/zloop.c +1 -1
index 19949ed3824f..6bb68022ab30 100644
@@ -950,8 +950,8 @@ static int zloop_report_zones(struct gendisk *disk, sector_t sector,
 		blkz.len = zlo->zone_size;
 		spin_lock(&zone->wp_lock);
 		blkz.wp = zone->wp;
-		spin_unlock(&zone->wp_lock);
 		blkz.cond = zone->cond;
+		spin_unlock(&zone->wp_lock);
 		if (test_bit(ZLOOP_ZONE_CONV, &zone->flags)) {
 			blkz.type = BLK_ZONE_TYPE_CONVENTIONAL;
 			blkz.capacity = zlo->zone_size;
Signature · 1 line
2.56.1
author maintainer
#9 lore raw

PATCH v11 8/13 zloop: add storage element emulation

In reply to Damien Le Moal

Emulate the storage element depopulation feature of SMR disks in the zloop driver. This emulation is controlled using the new stor_elements option. This option can take several values:
- ZLOOP_STOR_ELEMENTS_NONE (0): no emulation (default)
- ZLOOP_STOR_ELEMENTS_RDWR (1): emulate all access storage elements (e.g. read+write heads)
- ZLOOP_STOR_ELEMENTS_PAIRS (2): emulate fractional access storage elements (e.g. pairs of read and write heads)

If enabled with the value 1 or 2, the number of storage elements, or of pairs of fractional access storage elements, is automatically calculated based on the number of zones of the device so that we have at least 2 and at most 32 storage elements or pairs of fractional access storage elements.

The mapping of zones to storage elements is defined simply as the modulo of a zone number with the number of storage elements for the all access storage elements emulation and with the number of storage element pairs for the fractional access storage element emulation case. E.g, removing a storage element from a set of 4 all access storage elements will offline 1 zone every 4 zones.

The storage element management operations are specified with the structure zloop_se_ops (struct blk_se_ops).

---
 .../admin-guide/blockdev/zoned_loop.rst       |  11 +-
 drivers/block/zloop.c                         | 710 +++++++++++++++++-
 2 files changed, 694 insertions(+), 27 deletions(-)
Documentation/admin-guide/blockdev/zoned_loop.rst +10 -1
index 64277494fb36..80a564836c1b 100644
@@ -61,7 +61,7 @@ The options available for the add command can be listed by reading the
 /dev/zloop-control device::
 
 	$ cat /dev/zloop-control
-        add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache
+        add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache,stor_elements=%u
         remove id=%d
 
 In more details, the options that can be used with the "add" command are as
@@ -113,6 +113,15 @@ discard_write_cache   Discard all data that was not explicitly persisted using a
                       each zone file to the size recorded during the last flush
                       operation. This simulates power fail events where
                       uncommitted data is lost.
+stor_elements         Control storage element emulation. The default value is
+                      0, indicating no emulation. A value of 1 indicates that
+                      storage elements are emulated as full access physical
+                      elements, which is equivalent to combined read+write heads
+                      of a hard-disk. A value of 2 enables storage element
+                      emulation as fractional access physical storage elements,
+                      which is equivalent to pairs of read and write heads of a
+                      hard-disk. The health state of storage elements is not
+                      preserved across a zloop device remove and add.
 ===================   =========================================================
 
 3) Deleting a Zoned Device
drivers/block/zloop.c +684 -26

953 lines not on this page · raw message

Signature · 1 line
2.56.1
author maintainer
#13 lore raw

PATCH v11 9/13 zloop: add degrade_element control command

In reply to Damien Le Moal

Allow users to mark storage elements of a zloop device as degraded using the new "degrade_element" control command. The element to degrade is indicated using the element_id option. Example:

echo "degrade_element id=0,element_id=2" > /dev/zloop-control

If the element ID identifies an all access storage element, both read and write operations targeting a zone served by the degraded element are failed. In the case of a partial access storage element, any write access to a zone whose write element is degraded will fail, and read accesses to a zone whose read element is degraded will result in an error.

The zoned_loop.rst documentation file is also updated to explain this new feature.

---
 .../admin-guide/blockdev/zoned_loop.rst       |  22 +++-
 drivers/block/zloop.c                         | 100 +++++++++++++++++-
 2 files changed, 119 insertions(+), 3 deletions(-)
Documentation/admin-guide/blockdev/zoned_loop.rst +21 -1
index 80a564836c1b..d378fbdc1983 100644
@@ -63,6 +63,7 @@ The options available for the add command can be listed by reading the
 	$ cat /dev/zloop-control
         add id=%d,capacity_mb=%u,zone_size_mb=%u,zone_capacity_mb=%u,conv_zones=%u,max_open_zones=%u,base_dir=%s,nr_queues=%u,queue_depth=%u,buffered_io,zone_append=%u,ordered_zone_append,discard_write_cache,stor_elements=%u
         remove id=%d
+        degrade_element id=%d,element_id=%u
 
 In more details, the options that can be used with the "add" command are as
 follows.
@@ -144,7 +145,26 @@ the zoned device will not be created.
 To fully delete a zoned device, after executing the remove operation, the device
 base directory containing the backing files of the device zones must be deleted.
 
-4) Example
+4) Degrading a storage element
+------------------------------
+
+When storage element emulation is enabled, storage elements can be set in a
+degraded state to emulate defective storage elements (e.g. defective heads of a
+hard disk). This is done by issuing the "degrade_element" command to
+/dev/zloop-control, specifying the ID of the target device and the ID of the
+storage element to degrade::
+
+        $ echo "degrade_element id=X,element_id=Y" > /dev/zloop-control
+
+Depending on the type of the storage element specified, a read or write access
+to sectors of a zone served by a degraded storage element will result in a
+failure. For partial access storage elements, write accesses are always failed
+if the write element serving the zone is degraded. Likewise, read accesses are
+failed if the read element serving the zone is degraded. When storage elements
+are emulated as all access physical elements, both read and write operations are
+failed when the storage element serving the zones being accessed is degraded.
+
+5) Example
 ----------
 
 The following sequence of commands creates a 2GB zoned device with zones of 64
drivers/block/zloop.c +98 -2

184 lines not on this page · raw message

Signature · 1 line
2.56.1
author maintainer
#15 lore raw

PATCH v11 10/13 scsi: scsi_proto: refine the definition of storage element health

In reply to Damien Le Moal

A disk may report the health of a storage element as being within or outside of the manufacturer's specification limits using a range of values for the health field. However, in scsi_proto.h, we only define the start of each range using the SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS and SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS enum value.

Define the full range by renaming SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS to SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MIN and by defining the healthy range end with SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MAX. The same treatment is done for the outside range with the renaming of SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS to SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MIN, and the definition of the range upper value with SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MAX.

---
 include/scsi/scsi_proto.h | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
include/scsi/scsi_proto.h +4 -2
index 9c7d31aa6bae..f290a214e3e0 100644
@@ -408,9 +408,11 @@ enum scsi_phys_element_type {
 /* SCSI physical element health. */
 enum scsi_phys_element_health {
 	SCSI_PHYS_ELEM_HEALTH_NOT_REPORTED		= 0x00,
-	SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS	= 0x01,
+	SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MIN	= 0x01,
+	SCSI_PHYS_ELEM_HEALTH_WITHIN_SPEC_LIMITS_MAX	= 0x63,
 	SCSI_PHYS_ELEM_HEALTH_AT_SPEC_LIMITS		= 0x64,
-	SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS	= 0x65,
+	SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MIN	= 0x65,
+	SCSI_PHYS_ELEM_HEALTH_OUTSIDE_SPEC_LIMITS_MAX	= 0xCF,
 	SCSI_PHYS_ELEM_HEALTH_DEPOP_REVOKE_ERR		= 0xFB,
 	SCSI_PHYS_ELEM_HEALTH_DEPOP_REVOKE_IN_PROGRESS	= 0xFC,
 	SCSI_PHYS_ELEM_HEALTH_DEPOP_ERR			= 0xFD,
Signature · 1 line
2.56.1
author maintainer
#16 lore raw

PATCH v11 11/13 scsi: sd_zbc: always revalidate zones for disks supporting head depopulation

In reply to Damien Le Moal

sd_zbc_revalidate_zones() skips revalidating the zones of a ZBC device if the zone size and total number of zones of the disk has not changed. This is to avoid the sometimes slow report zones for the entire disk that blk_revalidate_disk_zones() will generate.

For ZBC devices that support the storage element depopulation feature, a revalidation after executing a passthrough REMOVE ELEMENT AND TRUNCATE command will thus correctly trigger a zone revalidation as the disk capacity and number of zones changed.

However, for devices supporting the data preserving head depopulation variant of the depopulation feature (REMOVE ELEMENT AND MODIFY ZONES command), a disk capacity and number of zones does not change after a head is depopulated. Only the conditions of the zones served by the head that was depopulated change. These zones are transitioned to either read-only or offline.

While the block layer storage element management functions always call blk_revalidate_disk_zones() to catch these changes, a user scan, libata device rescan after EH or a depopulation or restoration action executed through SG_IO would not. In these cases, not calling blk_revalidate_disk_zones() prevents the block layer from taking appropriate actions on the zone write plugs of the disk.

Avoid any issue with the block layer view of the zone conditions by not skipping the call to blk_revalidate_disk_zones() for disks that support both the GET PHYSICAL ELEMENTS STATUS and REMOVE ELEMENT AND MODIFY ZONES commands. This check is done from sd_zbc_read_zones() using the helper function sd_zbc_check_depop_supported(). The new SCSI disk flags depop_report_supported and depop_modify_zones_supported are defined to remember the result of this check.

This change will slow down a little SMR disk revalidation. However, user forced revalidations are normally rare, and other revalidations triggered by udev only happen when the disk device files is not in use anymore (closed). So this slow down should in practice not be an issue.

---
 drivers/scsi/sd.h     |  2 ++
 drivers/scsi/sd_zbc.c | 47 +++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 47 insertions(+), 2 deletions(-)
drivers/scsi/sd.h +2 -0
index 574af8243016..84c87d0f05d8 100644
@@ -156,6 +156,8 @@ struct scsi_disk {
 	unsigned	ignore_medium_access_errors : 1;
 	unsigned	rscs : 1; /* reduced stream control support */
 	unsigned	use_atomic_write_boundary : 1;
+	unsigned	depop_report_supported : 1;
+	unsigned	depop_modify_zones_supported : 1;
 };
 #define to_scsi_disk(obj) container_of(obj, struct scsi_disk, disk_dev)
 
drivers/scsi/sd_zbc.c +45 -2

81 lines not on this page · raw message

Signature · 1 line
2.56.1
author maintainer
#17 lore raw

PATCH v11 12/13 scsi: sd_zbc: define storage element management operations

In reply to Damien Le Moal

Define the storage element management operations using struct blk_se_ops. These operations are valid only on SMR disks supporting the storage element depopulation featu, with support for the GET PHYSICAL ELEMENTS STATUS command being mandatory. The support for the modify_zones operation is already indicated by the depop_modify_zones_supported SCSI disk flag. sd_zbc_check_depop_supported() is modified to add detection for the command RESTORE ELEMENTS AND REBUILD, which is indicated with the new SCSI disk flag depop_restore_supported.

Both the remove_element and restore_elements operations will block until the operation completes or an error is detected.

---
 drivers/scsi/sd.c     |   3 +
 drivers/scsi/sd.h     |   3 +
 drivers/scsi/sd_zbc.c | 489 +++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 494 insertions(+), 1 deletion(-)
drivers/scsi/sd.c +3 -0
index dc45df819c42..34b44d72b2b7 100644
@@ -3939,6 +3939,9 @@ static const struct block_device_operations sd_fops = {
 	.get_unique_id		= sd_get_unique_id,
 	.free_disk		= scsi_disk_free_disk,
 	.pr_ops			= &sd_pr_ops,
+#ifdef CONFIG_BLK_DEV_ZONED
+	.se_ops			= &sd_zbc_se_ops,
+#endif
 };
 
 /**
drivers/scsi/sd.h +3 -0
index 84c87d0f05d8..8a32e5f771d8 100644
@@ -158,6 +158,7 @@ struct scsi_disk {
 	unsigned	use_atomic_write_boundary : 1;
 	unsigned	depop_report_supported : 1;
 	unsigned	depop_modify_zones_supported : 1;
+	unsigned	depop_restore_supported : 1;
 };
 #define to_scsi_disk(obj) container_of(obj, struct scsi_disk, disk_dev)
 
@@ -244,6 +245,8 @@ unsigned int sd_zbc_complete(struct scsi_cmnd *cmd, unsigned int good_bytes,
 int sd_zbc_report_zones(struct gendisk *disk, sector_t sector,
 		unsigned int nr_zones, struct blk_report_zones_args *args);
 
+extern const struct blk_se_ops sd_zbc_se_ops;
+
 #else /* CONFIG_BLK_DEV_ZONED */
 
 static inline int sd_zbc_read_zones(struct scsi_disk *sdkp,
drivers/scsi/sd_zbc.c +488 -1

525 lines not on this page · raw message

Signature · 1 line
2.56.1
author maintainer
#20 lore raw

PATCH v11 13/13 scsi: sd: fail read commands to offline zones early

In reply to Damien Le Moal

Issuing read commands to offline zones of a ZBC disk will result in failures. So do not issue such command and fail them early in sd_setup_read_write_cmnd(). This change also checks write commands, which is fine but should actually never happen as the block layer zone write plugging code already fails write BIOs to offline zones early in the submission path.

As the code comment explains, since disk_zone_is_offline() uses the block layer cached zone condition, this is inherently racy with zone revalidation updating the zone condition. However, this will never prevent valid read operations as the only case where we could proceed is if the read is received after a restoration of the disk storage element completes but before the zones are revalidated. And in that case, all zones are empty and thus have no data to read.

---
 drivers/scsi/sd.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
drivers/scsi/sd.c +19 -1
index 34b44d72b2b7..8b1c05de541d 100644
@@ -1424,7 +1424,8 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd)
 {
 	struct request *rq = scsi_cmd_to_rq(cmd);
 	struct scsi_device *sdp = cmd->device;
-	struct scsi_disk *sdkp = scsi_disk(rq->q->disk);
+	struct gendisk *disk = rq->q->disk;
+	struct scsi_disk *sdkp = scsi_disk(disk);
 	sector_t lba = sectors_to_logical(sdp, blk_rq_pos(rq));
 	sector_t threshold;
 	unsigned int nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
@@ -1456,6 +1457,23 @@ static blk_status_t sd_setup_read_write_cmnd(struct scsi_cmnd *cmd)
 		goto fail;
 	}
 
+	/*
+	 * The block layer already prevents writes to offline zones of zoned
+	 * block devices. But there are no checks for reads, so check this here.
+	 * Since disk_zone_is_offline() uses the block layer cached zone
+	 * condition, this is inherently racy with zone revalidation updating
+	 * the zone condition. However, this will never prevent valid read
+	 * operations as the only case where we could proceed is if the read is
+	 * received after a restoration of the disk storage element completes
+	 * but before the zones are revalidated. And in that case, all zones are
+	 * empty and thus have no data to read.
+	 * Note that we are quiet about the error (no message) to avoid flooding
+	 * the kernel log.
+	 */
+	if (sdkp->device->type == TYPE_ZBC &&
+	    disk_zone_is_offline(disk, blk_rq_pos(rq)))
+		goto fail;
+
 	/*
 	 * Some SD card readers can't handle accesses which touch the
 	 * last one or two logical blocks. Split accesses as needed.
Signature · 1 line
2.56.1
#14 lore raw

Re: [PATCH v11 09/13] zloop: add degrade_element control command

In reply to Damien Le Moal
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] zloop: zloop_rw() fails to set -EIO when zloop_zone_rw_is_ok() fails --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] zloop: zloop_rw() fails to set -EIO when zloop_zone_rw_is_ok() fails
--

commit c96e6d1fd4da5400dd5befd118994f57ffa77920
Author: Damien Le Moal <[email protected]>

zloop: add degrade_element control command

This commit adds the "degrade_element" control command to mark a zloop storage element as degraded so that reads or writes to zones served by it fail.

Quoted 4 lines
drivers/block/zloop.c +0 -0
index a7806a73d7148..dcfcd100b7baa 100644

[ ... ]

Quoted 18 lines
drivers/block/zloop.c +14 -0
@@ -1173,6 +1177,31 @@ static int zloop_report_elements(struct gendisk *disk,
 	return 0;
 }
 
+static int zloop_degrade_element(struct zloop_device *zlo,
+				 unsigned int element_id)
+{
+	struct blk_se *se;
+	int ret = 0;
+
+	if (zlo->stor_elements == ZLOOP_STOR_ELEMENTS_NONE)
+		return -EOPNOTSUPP;
+
+	if (!element_id || element_id > zlo->nr_elements)
+		return -EINVAL;
+
+	mutex_lock(&zlo->stor_elements_lock);
+

drivers/block/zloop.c static int zloop_report_elements(struct gendisk *disk, · line 1203

Quoted 10 lines
drivers/block/zloop.c +10 -0
+	se = &zlo->elements[element_id - 1];
+	if (READ_ONCE(se->status) <= BLK_SE_STS_DEGRADED)
+		WRITE_ONCE(se->status, BLK_SE_STS_DEGRADED);
+	else
+		ret = -EINVAL;
+
+	mutex_unlock(&zlo->stor_elements_lock);
+
+	return ret;
+}

[Severity: High]
Can a read to a sequential zone on a degraded element complete with cmd->ret == 0 instead of -EIO after a failed write?

When a write to a sequential zone fails (including when rejected by a degraded element), zloop_complete_rq() sets ZLOOP_ZONE_SEQ_ERROR. On the next I/O to that zone, zloop_rw() clears ZLOOP_ZONE_SEQ_ERROR and calls zloop_update_seq_zone(), which succeeds and overwrites ret with 0 before zloop_zone_rw_is_ok() checks the element status:

drivers/block/zloop.c:zloop_rw() {
    ...
	if (test_and_clear_bit(ZLOOP_ZONE_SEQ_ERROR, &zone->flags)) {
		mutex_lock(&zone->lock);
		ret = zloop_update_seq_zone(zlo, zone_no);
		mutex_unlock(&zone->lock);
		if (ret)
			goto out;
	}

	/*
	 * Check if we can accept read or write commands based on the
	 * health state of the storage element serving the target zone.
	 */
	if (!zloop_zone_rw_is_ok(zlo, zone, op_is_write(req_op(rq))))
		goto out;
    ...
out:
	if (ret != -EIOCBQUEUED)
		zloop_rw_complete(&cmd->iocb, ret);
    ...
}

When zloop_zone_rw_is_ok() returns false and jumps to out, ret is still 0. For a read request, zloop_complete_rq() treats cmd->ret == 0 as a short read, zeroes the request buffer via zloop_fill_zero_rq(), and completes the request with BLK_STS_OK instead of failing it.

Setting ret = -EIO before jumping to out when zloop_zone_rw_is_ok() fails (or running zloop_zone_rw_is_ok() before the ZLOOP_ZONE_SEQ_ERROR check) would avoid this.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
#18 lore raw

Re: [PATCH v11 12/13] scsi: sd_zbc: define storage element management operations

In reply to Damien Le Moal
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] scsi: sd_zbc: unused pre-restore report in sd_zbc_restore_storage_elements() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] scsi: sd_zbc: unused pre-restore report in sd_zbc_restore_storage_elements()
--

commit 243eb4ceb1c6dd8c1c98cb3da00f3669c1215225
Author: Damien Le Moal <[email protected]>

scsi: sd_zbc: define storage element management operations

This commit defines storage element management operations for ZBC SCSI disks to support reporting, removing, and restoring storage elements.

Quoted 4 lines
drivers/scsi/sd_zbc.c +0 -0
index bf2e60071d577..13cfcef19c38a 100644

[ ... ]

Quoted 4 lines
drivers/scsi/sd_zbc.c +0 -0
@@ -560,6 +573,480 @@ static void sd_zbc_print_zones(struct scsi_disk *sdkp)
 			  sdkp->zone_info.zone_blocks);
 }
 

[ ... ]

Quoted 75 lines
drivers/scsi/sd_zbc.c +75 -0
+static int sd_zbc_wait_for_se_restore(struct scsi_disk *sdkp,
+				      struct sd_zbc_se_report *rep)
+{
+	unsigned int i, in_progress, in_error;
+	int ret;
+
+	/*
+	 * Poll the storage elements health to see if we have any element still
+	 * being restored or if we got errors. Since restoration is a best
+	 * effort process, we care only about failed restoration for elements
+	 * that are allowed to be restored.
+	 * Since we do not know how long the operation may take, we cannot have
+	 * a timeout on this loop. So check for fatal signals so that the user
+	 * can terminate this retry loop if the device misbehaves.
+	 */
+	do {
+		msleep(1000);
+
+		ret = sd_zbc_report_se(sdkp, rep);
+		if (ret)
+			return ret;
+		if (rep->nr_reported_elements != rep->nr_elements)
+			return -EIO;
+
+		in_progress = 0;
+		in_error = 0;
+		for (i = 0; i < rep->nr_reported_elements; i++) {
+			switch (rep->elements[i].status) {
+			case BLK_SE_STS_RESTORE_IN_PROGRESS:
+				in_progress++;
+				break;
+			case BLK_SE_STS_RESTORE_ERROR:
+				if (rep->elements[i].restore_allowed)
+					in_error++;
+				break;
+			default:
+				break;
+			}
+		}
+
+		if (!in_progress) {
+			if (in_error)
+				return -EIO;
+			return 0;
+		}
+	} while (!fatal_signal_pending(current));
+
+	return -EINTR;
+}
+
+static int sd_zbc_restore_storage_elements(struct gendisk *disk)
+{
+	struct scsi_disk *sdkp = scsi_disk(disk);
+	struct sd_zbc_se_report rep;
+	int ret;
+
+	if (!sdkp->depop_restore_supported)
+		return -EOPNOTSUPP;
+
+	/*
+	 * Get the resources we need to prepare for waiting for the operation to
+	 * complete.
+	 */
+	ret = sd_zbc_alloc_se_report(NULL, disk->nr_storage_elements,
+				     true, &rep);
+	if (ret)
+		return ret;
+
+	/* To detect restoration errors, count the number of such elements. */
+	ret = sd_zbc_report_se(sdkp, &rep);
+	if (ret)
+		goto free;
+	if (rep.nr_reported_elements != rep.nr_elements) {
+		ret = -EIO;
+		goto free;

drivers/scsi/sd_zbc.c static void sd_zbc_print_zones(struct scsi_disk *sdkp) · line 1034

Quoted 10 lines
drivers/scsi/sd_zbc.c +10 -0
+	}
+
+	/*
+	 * Execute the restore command and wait for the elements restoration
+	 * operation to complete.
+	 */
+	ret = sd_zbc_execute_storage_element_cmd(sdkp,
+					SAI_RESTORE_ELEMENTS_AND_REBUILD, 0);
+	if (!ret)
+		ret = sd_zbc_wait_for_se_restore(sdkp, &rep);

[Severity: Low]
Is the sd_zbc_report_se() call in sd_zbc_restore_storage_elements() missing the element count described by the comment, or is it left over? Nothing inspects rep.elements before sd_zbc_wait_for_se_restore() overwrites rep on its first iteration, so this issues a redundant GET PHYSICAL ELEMENT STATUS command on every restore operation.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
#10 lore raw

Re: [PATCH v11 08/13] zloop: add storage element emulation

In reply to Damien Le Moal
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] zloop: unreset ret == 0 in zloop_rw() when zloop_zone_rw_is_ok() fails --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] zloop: unreset ret == 0 in zloop_rw() when zloop_zone_rw_is_ok() fails
--

commit a3e7202e262b13b2a09d14b2d67deaa658ca4af4
Author: Damien Le Moal <[email protected]>

zloop: add storage element emulation

This commit adds storage element depopulation emulation to the zloop driver, controlled by the new stor_elements option and implemented via struct blk_se_ops callbacks.

Quoted 4 lines
drivers/block/zloop.c +0 -0
index ae7a0b0281884..a7806a73d7148 100644

[ ... ]

drivers/block/zloop.c static void zloop_rw(struct zloop_cmd *cmd) · line 827

Quoted 10 lines
drivers/block/zloop.c +6 -0
@@ -671,6 +819,13 @@ static void zloop_rw(struct zloop_cmd *cmd)
 			goto out;
 	}
 
+	/*
+	 * Check if we can accept read or write commands based on the
+	 * health state of the storage element serving the target zone.
+	 */
+	if (!zloop_zone_rw_is_ok(zlo, zone, op_is_write(req_op(rq))))
+		goto out;

[Severity: High]
Can zloop_rw() jump to out with ret == 0 when zloop_zone_rw_is_ok() rejects a command?

Right above this check, if ZLOOP_ZONE_SEQ_ERROR is set (for example, after an earlier failed write to a sequential zone), zloop_rw() calls ret = zloop_update_seq_zone(zlo, zone_no), which overwrites ret = -EIO with 0 on success:

drivers/block/zloop.c:zloop_rw() {
    ...
	if (test_and_clear_bit(ZLOOP_ZONE_SEQ_ERROR, &zone->flags)) {
		mutex_lock(&zone->lock);
		ret = zloop_update_seq_zone(zlo, zone_no);
		mutex_unlock(&zone->lock);
		if (ret)
			goto out;
	}
    ...
}

When zloop_zone_rw_is_ok() then returns false and jumps to out, zloop_rw_complete() sets cmd->ret = 0 instead of -EIO. In zloop_complete_rq(), a rejected REQ_OP_READ with cmd->ret == 0 is treated as a 0-byte short read, zeroing the caller's read buffer via zloop_fill_zero_rq(rq, 0) and completing with BLK_STS_OK instead of BLK_STS_IOERR. A rejected write hits the partial-write check, logs a spurious 0-byte partial write error, and re-sets ZLOOP_ZONE_SEQ_ERROR.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Reviewed-by on v11 6/13
#7 lore raw

Re: [PATCH v11 06/13] zloop: fix ordered zone append handling

In reply to Damien Le Moal
Quoted 24 lines · Damien Le Moal

For oredered zone append commands, the function zloop_set_zone_append_sector() increments a zone write pointer without checking if the zone is already open (implicitly or explicitly). This check is done only when the command is later executed using a work item, and the zone is only then implicitly open if needed. This creates a short time window where a report zone can show an empty zone with a write pointer that is not equal to the zone start sector, which is incorrect.

Calling zloop_do_open_zone() in zloop_set_zone_append_sector() is required but introduces locking complexity as the zone mutex cannot be taken in the context of zloop_queue_rq(). Instead introduce the ordered workqueue append_workqueue to handle zone append commands when ordered zone append is enabled. This ensure that all zone append operations are executed in the order of there submission. This removes the need for setting the command sector in the context of zloop_queue_rq() and we can rely on the reguar command execution done by zloop_rw(). With this change, zloop_seq_write_prep() is simplified and zloop_set_zone_append_sector() is removed entirely.

This change does reduce the command execution concurrency and impact performance with ordered zone append enabled for file systems using zone append. This is however not a critical option and the default unordered zone append handling is unaffected.

Quoted 7 lines
Fixes fcc6eaa3a03a ("zloop: introduce the ordered_zone_append configuration parameter")
---
  drivers/block/zloop.c | 113 +++++++++++++++++-------------------------
  1 file changed, 45 insertions(+), 68 deletions(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#11 lore raw

Re: [PATCH v11 08/13] zloop: add storage element emulation

In reply to Damien Le Moal
Quoted 31 lines · Damien Le Moal

Emulate the storage element depopulation feature of SMR disks in the zloop driver. This emulation is controlled using the new stor_elements option. This option can take several values:
- ZLOOP_STOR_ELEMENTS_NONE (0): no emulation (default)
- ZLOOP_STOR_ELEMENTS_RDWR (1): emulate all access storage elements (e.g. read+write heads)
- ZLOOP_STOR_ELEMENTS_PAIRS (2): emulate fractional access storage elements (e.g. pairs of read and write heads)

If enabled with the value 1 or 2, the number of storage elements, or of pairs of fractional access storage elements, is automatically calculated based on the number of zones of the device so that we have at least 2 and at most 32 storage elements or pairs of fractional access storage elements.

The mapping of zones to storage elements is defined simply as the modulo of a zone number with the number of storage elements for the all access storage elements emulation and with the number of storage element pairs for the fractional access storage element emulation case. E.g, removing a storage element from a set of 4 all access storage elements will offline 1 zone every 4 zones.

The storage element management operations are specified with the structure zloop_se_ops (struct blk_se_ops).

---
  .../admin-guide/blockdev/zoned_loop.rst       |  11 +-
  drivers/block/zloop.c                         | 710 +++++++++++++++++-
  2 files changed, 694 insertions(+), 27 deletions(-)

[ .. ]

Quoted 12 lines
drivers/block/zloop.c +1 -0
index 6bb68022ab30..1731e4ad5c98 100644
@@ -18,6 +18,7 @@
  #include <linux/parser.h>
  #include <linux/seq_file.h>
  #include <linux/xattr.h>
+#include <linux/delay.h>
 
  /*
   * Options for adding (and removing) a device.

[ .. ]

drivers/block/zloop.c static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts, · line 1705

Quoted 8 lines
drivers/block/zloop.c +4 -0
@@ -1149,6 +1699,74 @@ static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts,
  	return ret;
  }
 
+#define ZLOOP_MIN_STOR_ELEMENTS			2
+#define ZLOOP_MAX_STOR_ELEMENTS			32
+#define ZLOOP_MIN_ZONES_PER_STOR_ELEMENTS	32
+

Any particular reason for these limits?
Can't we infer these limits from other settings
(eg number of zones)?

Otherwise looks good.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Reviewed-by on v11 12/13
#19 lore raw

Re: [PATCH v11 12/13] scsi: sd_zbc: define storage element management operations

In reply to Damien Le Moal
Quoted 12 lines · Damien Le Moal

Define the storage element management operations using struct blk_se_ops. These operations are valid only on SMR disks supporting the storage element depopulation featu, with support for the GET PHYSICAL ELEMENTS STATUS command being mandatory. The support for the modify_zones operation is already indicated by the depop_modify_zones_supported SCSI disk flag. sd_zbc_check_depop_supported() is modified to add detection for the command RESTORE ELEMENTS AND REBUILD, which is indicated with the new SCSI disk flag depop_restore_supported.

Both the remove_element and restore_elements operations will block until the operation completes or an error is detected.

Quoted 7 lines
---
  drivers/scsi/sd.c     |   3 +
  drivers/scsi/sd.h     |   3 +
  drivers/scsi/sd_zbc.c | 489 +++++++++++++++++++++++++++++++++++++++++-
  3 files changed, 494 insertions(+), 1 deletion(-)

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
author maintainer
#12 lore raw

Re: [PATCH v11 08/13] zloop: add storage element emulation

In reply to Hannes Reinecke
Quoted 48 lines · Hannes Reinecke
Quoted 31 lines · Damien Le Moal

Emulate the storage element depopulation feature of SMR disks in the zloop driver. This emulation is controlled using the new stor_elements option. This option can take several values:
- ZLOOP_STOR_ELEMENTS_NONE (0): no emulation (default)
- ZLOOP_STOR_ELEMENTS_RDWR (1): emulate all access storage elements (e.g. read+write heads)
- ZLOOP_STOR_ELEMENTS_PAIRS (2): emulate fractional access storage elements (e.g. pairs of read and write heads)

If enabled with the value 1 or 2, the number of storage elements, or of pairs of fractional access storage elements, is automatically calculated based on the number of zones of the device so that we have at least 2 and at most 32 storage elements or pairs of fractional access storage elements.

The mapping of zones to storage elements is defined simply as the modulo of a zone number with the number of storage elements for the all access storage elements emulation and with the number of storage element pairs for the fractional access storage element emulation case. E.g, removing a storage element from a set of 4 all access storage elements will offline 1 zone every 4 zones.

The storage element management operations are specified with the structure zloop_se_ops (struct blk_se_ops).

---
  .../admin-guide/blockdev/zoned_loop.rst       |  11 +-
  drivers/block/zloop.c                         | 710 +++++++++++++++++-
  2 files changed, 694 insertions(+), 27 deletions(-)

[ .. ]

Quoted 12 lines
drivers/block/zloop.c +1 -0
index 6bb68022ab30..1731e4ad5c98 100644
@@ -18,6 +18,7 @@
  #include <linux/parser.h>
  #include <linux/seq_file.h>
  #include <linux/xattr.h>
+#include <linux/delay.h>
 
  /*
   * Options for adding (and removing) a device.

[ .. ]

Quoted 6 lines

drivers/block/zloop.c static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts, · line 1703

Quoted 6 lines
drivers/block/zloop.c +2 -0
@@ -1149,6 +1699,74 @@ static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts,
  	return ret;
  }
 
+#define ZLOOP_MIN_STOR_ELEMENTS			2
+#define ZLOOP_MAX_STOR_ELEMENTS			32

This one is indeed a limit, so that we do not go crazy on the number of storage elements, resulting in an emulation that is representative of a real disk. Note that for a real disk, this would mean that we have 16-disks (1 head per surface, 2 heads per disk) and no disk today on the market have that many. So this is already "large".

Quoted 1 line

drivers/block/zloop.c static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts, · line 1704

Quoted 1 line
drivers/block/zloop.c +1 -0
+#define ZLOOP_MIN_ZONES_PER_STOR_ELEMENTS	32

This one is not a limit, but rather what we aim for when the number of zones is low so that we have a reasonable number of zones per element. Basically, to avoid the rather silly case on one zone per element for small devices.

Quoted 11 lines

drivers/block/zloop.c static int zloop_init_zone(struct zloop_device *zlo, struct zloop_options *opts, · line 1705

Quoted 1 line
drivers/block/zloop.c +1 -0
+

Any particular reason for these limits?
Can't we infer these limits from other settings
(eg number of zones)?

Otherwise looks good.

Cheers,

Hannes

Signature · 2 lines
Damien Le Moal
Western Digital Research