block: fix I/O errors in BLKRRPART
Subsystems (2)
v3 cover block: fix I/O errors in BLKRRPART
Hello,
This patch fixes I/O errors during BLKRRPART ioctl() behavior right
after format operation that changed logical block size of the block device with a same file descriptor opened.
Testcase:
The following testcase is a case of NVMe namespace with the following
conditions:
- Current LBA format is lbaf=0 (512 bytes logical block size)
- LBA Format(lbaf=1) has 4096 bytes logical block size
# Format block device logical block size 512B to 4096B nvme format /dev/nvme0n1 --lbaf=1 --force
This will cause I/O errors because BLKRRPART ioctl() happened right after
the format command with same file descriptor opened in application (e.g., nvme-cli) like:
fd = open("/dev/nvme0n1", O_RDONLY); nvme_format(fd, ...);
if (ioctl(fd, BLKRRPART) < 0)
...
Errors:
We can see the Read command with Number of LBA(NLB) 0xffff(65535) which
was under-flowed because BLKRRPART operation requested request size based on i_blkbits of the block device which is 9 via buffer_head.
[dmesg-snip]
[ 10.771740] blk_update_request: operation not supported error, dev nvme0n1, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 10.780262] Buffer I/O error on dev nvme0n1, logical block 0, async page read[event-snip]
kworker/0:1H-56 [000] .... 913.456922: nvme_setup_cmd: nvme0: disk=nvme0n1, qid=1, cmdid=216, nsid=1, flags=0x0, meta=0x0, cmd=(nvme_cmd_read slba=0, len=65535, ctrl=0x0, dsmgmt=0, reftag=0)
ksoftirqd/0-9 [000] .Ns. 916.566351: nvme_complete_rq: nvme0: disk=nvme0n1, qid=1, cmdid=216, res=0x0, retries=0, flags=0x0, status=0x4002 The patch below fixes the I/O errors by rejecting I/O requests from the
block layer with setting a flag to gendisk until the file descriptor re-opened to be updated by __blkdev_get(). This is based on the previous discussion [1].
Since V2:
- Cover letter with testcase and error logs attached. Removed un-related
changes: empty line. (Chaitanya, [2])
- Put blkdev with blkdev_put_no_open().
Since V1:
- Updated patch to reject I/O rather than updating i_blkbits of the block device's inode directly from driver. (Christoph, [1])
[1] https://lore.kernel.org/linux-nvme/[email protected]/T/#t
[2] https://lore.kernel.org/linux-nvme/[email protected]/T/#t
Thanks,
Minwoo Im (1):
block: reject I/O for same fd if block size changed
block/blk-settings.c | 8 ++++++++ block/partitions/core.c | 11 +++++++++++ fs/block_dev.c | 6 ++++++ include/linux/genhd.h | 1 + 4 files changed, 26 insertions(+)
Signature · 1 line
2.17.1
RFC PATCH v3 1/1 block: reject I/O for same fd if block size changed
This patch fixes I/O errors during BLKRRPART ioctl() behavior right after format operation that changed logical block size of the block device with a same file descriptor opened.
This issue can be easily reproduced with a single format command in case of NVMe (logical block size 512B to 4096B).
nvme format /dev/nvme0n1 --lbaf=1 --force
This is because the application, nvme-cli format subcommand issues an admin command followed by BLKRRPART ioctl to re-read partition information without closing the file descriptor. If file descriptor stays opened, __blkdev_get() will not be invoked at all even logical block size has been changed.
It will cause I/O errors with invalid Read operations during the BLKRRPART ioctl due to i_blkbits mismatch. The invalid operations in BLKRRPART happens with under-flowed Number of LBA(NLB) values 0xffff(65535) because i_blkbits is still set to 9 even the logical block size has been updated to 4096. The BLKRRPART will lead buffer_head to hold 512B data which is less than the logical lock size of the block device.
The root cause, which is because i_blkbits of inode of the block device is not updated, can be solved easily by re-opening file descriptor again from application. But, that's just for application's business and kernel should reject invalid Read operations during the BLKRRPART ioctl.
This patch rejects I/O from the path of add_partitions() to avoid issuing invalid Read operations to device. It also sets a flag to gendisk in blk_queue_logical_block_size to minimize caller-side updates.
--- block/blk-settings.c | 8 ++++++++ block/partitions/core.c | 11 +++++++++++ fs/block_dev.c | 6 ++++++ include/linux/genhd.h | 1 + 4 files changed, 26 insertions(+)
block/blk-settings.c +8 -0
@@ -328,6 +328,14 @@ EXPORT_SYMBOL(blk_queue_max_segment_size);void blk_queue_logical_block_size (struct request_queue *q,unsigned int size) {struct queue_limits *limits = &q->limits; +struct block_device *bdev; + +if (q->backing_dev_info && q->backing_dev_info->owner && + limits->logical_block_size != size) { + bdev =blkdev_get_no_open (q->backing_dev_info->owner->devt); + bdev->bd_disk->flags |=GENHD_FL_BLOCK_SIZE_CHANGED ; +blkdev_put_no_open (bdev); + } limits->logical_block_size = size;
block/partitions/core.c +11 -0
@@ -618,6 +618,17 @@ int blk_add_partitions(struct gendisk *disk, struct block_device *bdev)if (!disk_part_scan_enabled (disk))return 0 ; +/* +* Reject to check partition information if block size has been changed +* in the runtime. If block size of a block device has been changed, +* the file descriptor should be opened agian to update the blkbits. +*/ +if (disk->flags &GENHD_FL_BLOCK_SIZE_CHANGED ) { +pr_warn ("%s: rejecting checking partition. fd should be opened again.\n" , + disk->disk_name); +return -EBADFD ; + } + state =check_partition (disk, bdev);if (!state)return 0 ;
fs/block_dev.c +6 -0
@@ -131,6 +131,12 @@ EXPORT_SYMBOL(truncate_bdev_range);static void set_init_blocksize (struct block_device *bdev) { bdev->bd_inode->i_blkbits =blksize_bits (bdev_logical_block_size (bdev)); + +/* +* Allow I/O commands for this block device. We can say that this +* block device has been set to a proper block size. +*/ + bdev->bd_disk->flags &= ~GENHD_FL_BLOCK_SIZE_CHANGED ; }int set_blocksize (struct block_device *bdev,int size)
include/linux/genhd.h +1 -0
@@ -103,6 +103,7 @@ struct partition_meta_info {#define GENHD_FL_BLOCK_EVENTS_ON_EXCL_WRITE 0x0100#define GENHD_FL_NO_PART_SCAN 0x0200#define GENHD_FL_HIDDEN 0x0400 +#define GENHD_FL_BLOCK_SIZE_CHANGED 0x0800enum {DISK_EVENT_MEDIA_CHANGE =1 <<0 ,/* media changed */
Signature · 1 line
2.17.1
Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed
block/blk-settings.c EXPORT_SYMBOL(blk_queue_max_segment_size); · line 338
Quoted 6 lines · Minwoo Im
block/blk-settings.c +6
-0+if (q->backing_dev_info && q->backing_dev_info->owner && + limits->logical_block_size != size) { + bdev =blkdev_get_no_open (q->backing_dev_info->owner->devt); + bdev->bd_disk->flags |=GENHD_FL_BLOCK_SIZE_CHANGED ; +blkdev_put_no_open (bdev); + }
We really need the backpointer from the queue to the gendisk I've wanted to add for a while. Can we at least restrict this to a live gendisk?
Also I think the size change flag should go into the ->state field and use the atomic bitops helpers to avoid concurrency problems.
Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed
Quoted 10 lines · Christoph Hellwig
block/blk-settings.c EXPORT_SYMBOL(blk_queue_max_segment_size); · line 338
Quoted 6 lines · Minwoo Im
block/blk-settings.c +6
-0+if (q->backing_dev_info && q->backing_dev_info->owner && + limits->logical_block_size != size) { + bdev =blkdev_get_no_open (q->backing_dev_info->owner->devt); + bdev->bd_disk->flags |=GENHD_FL_BLOCK_SIZE_CHANGED ; +blkdev_put_no_open (bdev); + }We really need the backpointer from the queue to the gendisk I've wanted to add for a while. Can we at least restrict this to a live gendisk?
Alternatively we could make this request_queue QUEUE* flag for now.
Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed
Hello Christoph,
Thanks for your review.
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Christoph Hellwig
block/blk-settings.c EXPORT_SYMBOL(blk_queue_max_segment_size); · line 338
Quoted 6 lines · Minwoo Im
block/blk-settings.c +6
-0+if (q->backing_dev_info && q->backing_dev_info->owner && + limits->logical_block_size != size) { + bdev =blkdev_get_no_open (q->backing_dev_info->owner->devt); + bdev->bd_disk->flags |=GENHD_FL_BLOCK_SIZE_CHANGED ; +blkdev_put_no_open (bdev); + }We really need the backpointer from the queue to the gendisk I've wanted to add for a while. Can we at least restrict this to a live gendisk?
It was a point that I really would like to ask by RFC whether we can have backpointer to the gendisk from the request_queue. And I'd like to have it to simplify this routine and for future usages also.
I will restrict this one by checking GENHD_FL_UP flag from the gendisk for the next patch.
Quoted 2 lines
Alternatively we could make this
request_queueQUEUE* flag for now.
As this patch rejects I/O from the block layer partition code, can we have this flag in gendisk rather than request_queue ?
Thanks,
Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed
Quoted 3 lines · Minwoo Im
It was a point that I really would like to ask by RFC whether we can have backpointer to the gendisk from the
request_queue. And I'd like to have it to simplify this routine and for future usages also.
I think it is the right thing to do, at least mid-term, although I don't want to enforce the burden on you right now.
Quoted 8 lines
I will restrict this one by checking
GENHD_FL_UPflag from the gendisk for the next patch.Quoted 2 lines
Alternatively we could make this
request_queueQUEUE* flag for now.As this patch rejects I/O from the block layer partition code, can we have this flag in gendisk rather than
request_queue?
For now we can as the request_queue is required. I have some plans to clean up this area, but just using a request_queue flag for now is probably the simplest, even if it means more work for me later.
Re: [RFC PATCH V3 1/1] block: reject I/O for same fd if block size changed
Hello,
Quoted 12 lines · Christoph Hellwig
Quoted 3 lines · Minwoo Im
It was a point that I really would like to ask by RFC whether we can have backpointer to the gendisk from the
request_queue. And I'd like to have it to simplify this routine and for future usages also.I think it is the right thing to do, at least mid-term, although I don't want to enforce the burden on you right now.
Quoted 4 lines
I will restrict this one by checking
GENHD_FL_UPflag from the gendisk for the next patch.Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Alternatively we could make this
request_queueQUEUE* flag for now.As this patch rejects I/O from the block layer partition code, can we have this flag in gendisk rather than
request_queue?For now we can as the
request_queueis required. I have some plans to clean up this area, but just using arequest_queueflag for now is probably the simplest, even if it means more work for me later.
Please let me prepare the next quick fix for this issue with request_queue flag.
Thanks!