Skip to contentSkip to conversation

block: fix integrity offset/length conversions

PATCH Superseded Caleb Sander Mateos 2 participants 14 messages Apr 17 – May 13

Subsystems (5)
SCSI Block layer NVMe SCSI target +1 more subsystems

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover block: fix integrity offset/length conversions

The block layer's integrity code currently sets the seed (initial reference tag) in units of 512-byte sectors but increments it in units of integrity intervals. Not only do the T10 DIF formats require ref tags to be the lower bits of the logical block address, but mixing the two units means the ref tags used for a particular logical block vary based on its offset within a read/write request. This looks to be a longstanding bug affecting block devices that support integrity with block sizes > 512 bytes; I'm surprised it wasn't noticed before.

Also fix the newly added fs_bio_integrity_verify() to pass bio_integrity_verify() a struct bdev_iter representing the data instead of the integrity. Most of the integrity data is currently being skipped.

v3:
- Drop bi and bip arguments to bip_set_seed() (Christoph)

v2:
- Reorder fixes before refactoring commits
- Use u64, SECTOR_SHIFT (Christoph)
- Don't take sector_t in bip_set_seed() (Christoph)

Caleb Sander Mateos (6):
block: use integrity interval instead of sector as seed
bio-integrity-fs: pass data iter to bio_integrity_verify()
blk-integrity: take u64 in bio_integrity_intervals()
bio-integrity-fs: use integrity interval instead of sector as seed
t10-pi: use bio_integrity_intervals() helper
blk-integrity: avoid sector_t in bip_{get,set}_seed()

 block/bio-integrity-fs.c            |  8 ++++++--
 block/bio-integrity.c               |  4 ++--
 block/t10-pi.c                      |  7 ++++---
 drivers/nvme/target/io-cmd-bdev.c   |  3 +--
 drivers/target/target_core_iblock.c |  3 +--
 include/linux/bio-integrity.h       | 11 -----------
 include/linux/blk-integrity.h       | 27 ++++++++++++++++++++-------
 include/linux/bvec.h                |  1 +
 8 files changed, 35 insertions(+), 29 deletions(-)
Signature · 1 line
2.45.2
#1 lore raw

PATCH v3 1/6 block: use integrity interval instead of sector as seed

bio_integrity_setup_default() and blk_integrity_iterate() set the integrity seed (initial reference tag) to the absolute address in the block device in units of 512-byte sectors. The seed is correctly incremented/decremented in units of integrity intervals in bio_integrity_map_iter(), bio_integrity_advance(), and blk_integrity_interval(). As a result, the ref tag written or read to a particular integrity interval on a block device with integrity interval size > 512 bytes varies with the starting offset of the read/write.

Convert the initial seed to units of integrity intervals so a consistent ref tag is used for each integrity interval.

Fixes 3be91c4a3d09 ("block: Deprecate the use of the term sector in the context of block integrity")
Fixes 63573e359d05 ("bio-integrity: Restore original iterator on verify stage")
---
 block/bio-integrity.c | 3 ++-
 block/t10-pi.c        | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
block/bio-integrity.c +2 -1
index e79eaf047794..3ad6a6799f17 100644
@@ -103,12 +103,13 @@ void bio_integrity_free_buf(struct bio_integrity_payload *bip)
 
 void bio_integrity_setup_default(struct bio *bio)
 {
 	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
 	struct bio_integrity_payload *bip = bio_integrity(bio);
+	u64 seed = bio->bi_iter.bi_sector >> (bi->interval_exp - SECTOR_SHIFT);
 
-	bip_set_seed(bip, bio->bi_iter.bi_sector);
+	bip_set_seed(bip, seed);
 
 	if (bi->csum_type) {
 		bip->bip_flags |= BIP_CHECK_GUARD;
 		if (bi->csum_type == BLK_INTEGRITY_CSUM_IP)
 			bip->bip_flags |= BIP_IP_CHECKSUM;
block/t10-pi.c +2 -1
index a19b4e102a83..e58d5eb6cefb 100644
@@ -308,18 +308,19 @@ static blk_status_t blk_integrity_iterate(struct bio *bio,
 					  struct bvec_iter *data_iter,
 					  bool verify)
 {
 	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
 	struct bio_integrity_payload *bip = bio_integrity(bio);
+	u64 seed = data_iter->bi_sector >> (bi->interval_exp - SECTOR_SHIFT);
 	struct blk_integrity_iter iter = {
 		.bio = bio,
 		.bip = bip,
 		.bi = bi,
 		.data_iter = *data_iter,
 		.prot_iter = bip->bip_iter,
 		.interval_remaining = 1 << bi->interval_exp,
-		.seed = data_iter->bi_sector,
+		.seed = seed,
 		.csum = 0,
 	};
 	blk_status_t ret = BLK_STS_OK;
 
 	while (iter.data_iter.bi_size && ret == BLK_STS_OK) {
Signature · 1 line
2.45.2
#2 lore raw

PATCH v3 2/6 bio-integrity-fs: pass data iter to bio_integrity_verify()

In reply to Caleb Sander Mateos

bio_integrity_verify() expects the passed struct bvec_iter to be an iterator over bio data, not integrity. So construct a separate data bvec_iter without the bio_integrity_bytes() conversion and pass it to bio_integrity_verify() instead of bip_iter.

Fixes 0bde8a12b554 ("block: add fs_bio_integrity helpers")
---
 block/bio-integrity-fs.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
block/bio-integrity-fs.c +5 -1
index acb1e5f270d2..0daa42d9ead7 100644
@@ -53,21 +53,25 @@ EXPORT_SYMBOL_GPL(fs_bio_integrity_generate);
 
 int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
 {
 	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
 	struct bio_integrity_payload *bip = bio_integrity(bio);
+	struct bvec_iter data_iter = {
+		.bi_sector	= sector,
+		.bi_size	= size,
+	};
 
 	/*
 	 * Reinitialize bip->bip_iter.
 	 *
 	 * This is for use in the submitter after the driver is done with the
 	 * bio.  Requires the submitter to remember the sector and the size.
 	 */
 	memset(&bip->bip_iter, 0, sizeof(bip->bip_iter));
 	bip->bip_iter.bi_sector = sector;
 	bip->bip_iter.bi_size = bio_integrity_bytes(bi, size >> SECTOR_SHIFT);
-	return blk_status_to_errno(bio_integrity_verify(bio, &bip->bip_iter));
+	return blk_status_to_errno(bio_integrity_verify(bio, &data_iter));
 }
 
 static int __init fs_bio_integrity_init(void)
 {
 	fs_bio_integrity_cache = kmem_cache_create("fs_bio_integrity",
Signature · 1 line
2.45.2
#3 lore raw

PATCH v3 3/6 blk-integrity: take u64 in bio_integrity_intervals()

In reply to Caleb Sander Mateos

To allow bio_integrity_intervals() to convert an absolute sector to an absolute integrity interval, use u64 for its argument and return types. Also use SECTOR_SHIFT instead of the magic constant 9.

---
 include/linux/blk-integrity.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
include/linux/blk-integrity.h +7 -7
index b1b530613c34..825d777c078b 100644
@@ -64,23 +64,23 @@ queue_max_integrity_segments(const struct request_queue *q)
 {
 	return q->limits.max_integrity_segments;
 }
 
 /**
- * bio_integrity_intervals - Return number of integrity intervals for a bio
+ * bio_integrity_intervals - Convert sectors to integrity intervals
  * @bi:		blk_integrity profile for device
- * @sectors:	Size of the bio in 512-byte sectors
+ * @sectors:	Number of 512-byte sectors
  *
  * Description: The block layer calculates everything in 512 byte
  * sectors but integrity metadata is done in terms of the data integrity
  * interval size of the storage device.  Convert the block layer sectors
  * to the appropriate number of integrity intervals.
  */
-static inline unsigned int bio_integrity_intervals(struct blk_integrity *bi,
-						   unsigned int sectors)
+static inline u64 bio_integrity_intervals(const struct blk_integrity *bi,
+					  u64 sectors)
 {
-	return sectors >> (bi->interval_exp - 9);
+	return sectors >> (bi->interval_exp - SECTOR_SHIFT);
 }
 
 static inline unsigned int bio_integrity_bytes(struct blk_integrity *bi,
 					       unsigned int sectors)
 {
@@ -151,12 +151,12 @@ static inline unsigned short
 queue_max_integrity_segments(const struct request_queue *q)
 {
 	return 0;
 }
 
-static inline unsigned int bio_integrity_intervals(struct blk_integrity *bi,
-						   unsigned int sectors)
+static inline u64 bio_integrity_intervals(const struct blk_integrity *bi,
+					  u64 sectors)
 {
 	return 0;
 }
 
 static inline unsigned int bio_integrity_bytes(struct blk_integrity *bi,
Signature · 1 line
2.45.2
#4 lore raw

PATCH v3 4/6 bio-integrity-fs: use integrity interval instead of sector as seed

In reply to Caleb Sander Mateos

bip_iter.bi_sector is meant to be in units of integrity intervals rather than 512-byte sectors. bio_integrity_verify() doesn't actually use it currently (it uses the passed in struct bvec_iter's bi_sector instead). But let's set it to the expected value for consistency.

---
 block/bio-integrity-fs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/bio-integrity-fs.c +1 -1
index 0daa42d9ead7..0ad1c5ab64e9 100644
@@ -65,11 +65,11 @@ int fs_bio_integrity_verify(struct bio *bio, sector_t sector, unsigned int size)
 	 *
 	 * This is for use in the submitter after the driver is done with the
 	 * bio.  Requires the submitter to remember the sector and the size.
 	 */
 	memset(&bip->bip_iter, 0, sizeof(bip->bip_iter));
-	bip->bip_iter.bi_sector = sector;
+	bip->bip_iter.bi_sector = bio_integrity_intervals(bi, sector);
 	bip->bip_iter.bi_size = bio_integrity_bytes(bi, size >> SECTOR_SHIFT);
 	return blk_status_to_errno(bio_integrity_verify(bio, &data_iter));
 }
 
 static int __init fs_bio_integrity_init(void)
Signature · 1 line
2.45.2
#5 lore raw

PATCH v3 5/6 t10-pi: use bio_integrity_intervals() helper

In reply to Caleb Sander Mateos

Use bio_integrity_intervals() to convert blk_rq_pos(rq) to integrity intervals to reduce code duplication.

---
 block/t10-pi.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/t10-pi.c +1 -1
index e58d5eb6cefb..787950dec50a 100644
@@ -541,11 +541,11 @@ static void __blk_reftag_remap(struct bio *bio, struct blk_integrity *bi,
 
 static void blk_integrity_remap(struct request *rq, unsigned int nr_bytes,
 				bool prep)
 {
 	struct blk_integrity *bi = &rq->q->limits.integrity;
-	u64 ref = blk_rq_pos(rq) >> (bi->interval_exp - SECTOR_SHIFT);
+	u64 ref = bio_integrity_intervals(bi, blk_rq_pos(rq));
 	unsigned intervals = nr_bytes >> bi->interval_exp;
 	struct bio *bio;
 
 	if (!(bi->flags & BLK_INTEGRITY_REF_TAG))
 		return;
Signature · 1 line
2.45.2
#6 lore raw

PATCH v3 6/6 blk-integrity: avoid sector_t in bip_{get,set}_seed()

In reply to Caleb Sander Mateos

bip_set_seed() and big_get_seed() take/return a sector_t value that's actually an integrity interval number. This is confusing, so pass struct bio instead to bip_set_seed() and convert the bio's device address to integrity intervals.

Open-code the access to bip->bip_iter.bi_sector in the one caller of bip_set_seed() that doesn't use the bio device address for the seed. Open-code bip_get_seed() in its one caller.

Add a comment to struct bvec_iter's bi_sector field explaining its alternate use for bip_iter.

---
 block/bio-integrity.c               |  5 ++---
 block/t10-pi.c                      |  2 +-
 drivers/nvme/target/io-cmd-bdev.c   |  3 +--
 drivers/target/target_core_iblock.c |  3 +--
 include/linux/bio-integrity.h       | 11 -----------
 include/linux/blk-integrity.h       | 13 +++++++++++++
 include/linux/bvec.h                |  1 +
 7 files changed, 19 insertions(+), 19 deletions(-)
block/bio-integrity.c +2 -3
index 3ad6a6799f17..8549d9148b0b 100644
@@ -103,13 +103,12 @@ void bio_integrity_free_buf(struct bio_integrity_payload *bip)
 
 void bio_integrity_setup_default(struct bio *bio)
 {
 	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
 	struct bio_integrity_payload *bip = bio_integrity(bio);
-	u64 seed = bio->bi_iter.bi_sector >> (bi->interval_exp - SECTOR_SHIFT);
 
-	bip_set_seed(bip, seed);
+	bip_set_seed(bio);
 
 	if (bi->csum_type) {
 		bip->bip_flags |= BIP_CHECK_GUARD;
 		if (bi->csum_type == BLK_INTEGRITY_CSUM_IP)
 			bip->bip_flags |= BIP_IP_CHECKSUM;
@@ -472,11 +471,11 @@ int bio_integrity_map_iter(struct bio *bio, struct uio_meta *meta)
 
 	it.count = integrity_bytes;
 	ret = bio_integrity_map_user(bio, &it);
 	if (!ret) {
 		bio_uio_meta_to_bip(bio, meta);
-		bip_set_seed(bio_integrity(bio), meta->seed);
+		bio_integrity(bio)->bip_iter.bi_sector = meta->seed;
 		iov_iter_advance(&meta->iter, integrity_bytes);
 		meta->seed += bio_integrity_intervals(bi, bio_sectors(bio));
 	}
 	return ret;
 }
block/t10-pi.c +1 -1
index 787950dec50a..71367fd082bd 100644
@@ -510,11 +510,11 @@ static void blk_reftag_remap_prepare(struct blk_integrity *bi,
 static void __blk_reftag_remap(struct bio *bio, struct blk_integrity *bi,
 			       unsigned *intervals, u64 *ref, bool prep)
 {
 	struct bio_integrity_payload *bip = bio_integrity(bio);
 	struct bvec_iter iter = bip->bip_iter;
-	u64 virt = bip_get_seed(bip);
+	u64 virt = bip->bip_iter.bi_sector;
 	union pi_tuple *ptuple;
 	union pi_tuple tuple;
 
 	if (prep && bip->bip_flags & BIP_MAPPED_INTEGRITY) {
 		*ref += bio->bi_iter.bi_size >> bi->interval_exp;
drivers/nvme/target/io-cmd-bdev.c +1 -2
index f2d9e8901df4..65fce51b024a 100644
@@ -218,12 +218,11 @@ static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio,
 		pr_err("Unable to allocate bio_integrity_payload\n");
 		return PTR_ERR(bip);
 	}
 
 	/* virtual start sector must be in integrity interval units */
-	bip_set_seed(bip, bio->bi_iter.bi_sector >>
-		     (bi->interval_exp - SECTOR_SHIFT));
+	bip_set_seed(bio);
 
 	resid = bio_integrity_bytes(bi, bio_sectors(bio));
 	while (resid > 0 && sg_miter_next(miter)) {
 		len = min_t(size_t, miter->length, resid);
 		rc = bio_integrity_add_page(bio, miter->page, len,
drivers/target/target_core_iblock.c +1 -2
index 1087d1d17c36..1aeb5be529a8 100644
@@ -706,12 +706,11 @@ iblock_alloc_bip(struct se_cmd *cmd, struct bio *bio,
 		pr_err("Unable to allocate bio_integrity_payload\n");
 		return PTR_ERR(bip);
 	}
 
 	/* virtual start sector must be in integrity interval units */
-	bip_set_seed(bip, bio->bi_iter.bi_sector >>
-				  (bi->interval_exp - SECTOR_SHIFT));
+	bip_set_seed(bio);
 
 	pr_debug("IBLOCK BIP Size: %u Sector: %llu\n", bip->bip_iter.bi_size,
 		 (unsigned long long)bip->bip_iter.bi_sector);
 
 	resid = bio_integrity_bytes(bi, bio_sectors(bio));
include/linux/bio-integrity.h +0 -11
index af5178434ec6..edcd0855abba 100644
@@ -56,21 +56,10 @@ static inline bool bio_integrity_flagged(struct bio *bio, enum bip_flags flag)
 		return bip->bip_flags & flag;
 
 	return false;
 }
 
-static inline sector_t bip_get_seed(struct bio_integrity_payload *bip)
-{
-	return bip->bip_iter.bi_sector;
-}
-
-static inline void bip_set_seed(struct bio_integrity_payload *bip,
-				sector_t seed)
-{
-	bip->bip_iter.bi_sector = seed;
-}
-
 void bio_integrity_init(struct bio *bio, struct bio_integrity_payload *bip,
 		struct bio_vec *bvecs, unsigned int nr_vecs);
 struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, gfp_t gfp,
 		unsigned int nr);
 int bio_integrity_add_page(struct bio *bio, struct page *page, unsigned int len,
include/linux/blk-integrity.h +13 -0
index 825d777c078b..c82b2f6fe194 100644
@@ -85,10 +85,23 @@ static inline unsigned int bio_integrity_bytes(struct blk_integrity *bi,
 					       unsigned int sectors)
 {
 	return bio_integrity_intervals(bi, sectors) * bi->metadata_size;
 }
 
+/**
+ * bip_set_seed - Set bip reference tag seed from bio device address
+ * @bio:	struct bio whose integrity payload's ref tag seed to initialize
+ */
+static inline void bip_set_seed(struct bio *bio)
+{
+	struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
+	struct bio_integrity_payload *bip = bio_integrity(bio);
+
+	bip->bip_iter.bi_sector =
+		bio_integrity_intervals(bi, bio->bi_iter.bi_sector);
+}
+
 static inline bool blk_integrity_rq(const struct request *rq)
 {
 	return rq->cmd_flags & REQ_INTEGRITY;
 }
 
include/linux/bvec.h +1 -0
index d36dd476feda..3dc88f5cd367 100644
@@ -76,10 +76,11 @@ static inline void bvec_set_virt(struct bio_vec *bv, void *vaddr,
 
 struct bvec_iter {
 	/*
 	 * Current device address in 512 byte sectors. Only updated by the bio
 	 * iter wrappers and not the bvec iterator helpers themselves.
+	 * For bip_iter, this is overloaded to represent the integrity interval.
 	 */
 	sector_t		bi_sector;
 
 	/*
 	 * Remaining size in bytes.
Signature · 1 line
2.45.2
#7 lore raw
In reply to Caleb Sander Mateos

Hi Caleb!

Quoted 3 lines

The block layer's integrity code currently sets the seed (initial reference tag) in units of 512-byte sectors but increments it in units of integrity intervals

I don't necessarily agree with the premise that the seed needs to be expressed in any particular unit. The seed is a start value, nothing more.

We happen to set it to the block number in the block layer since we need to be able to know what to compare against on completion (for Type 1 + the restrictive Linux implementation of Type 2). But that does not imply that the seed needs to be specified in any particular unit. Submitters set the seed to whichever value makes sense to them (i.e. it could be the offset within a file as opposed to the eventual LBA on the backend device). And then that seed is incremented by 1 for each integrity interval of data in the PI sent to/received from the device. The conversion between the submitter's view of what the first ref tag should be (i.e. seed) and what is required by the hardware (for instance lower 32 bits of device LBA) is the reason we perform remapping. The seed is intentionally different in the submitter's protection envelope compared to the device's protection envelope.

Using the block layer block number as seed was just a convenience since that provided a predictable value for any I/O that had its PI autogenerated. I never intended for the actual LBA to be used as seed value on a 4Kn device. Initially we just used 0 as the seed. Leveraging the block number just added a bit of additional protection.

I confess I haven't tested 4Kn in a while since things sort of converged on 512e. But I used to run nightly tests on a SCSI storage with 4Kn blocks just fine.

Quoted 3 lines

This looks to be a longstanding bug affecting block devices that support integrity with block sizes > 512 bytes; I'm surprised it wasn't noticed before.

Are you seeing this with NVMe or SCSI?

Signature · 1 line
Martin K. Petersen
#8 lore raw
Quoted 11 lines · Martin K. Petersen

Hi Caleb!

Quoted 3 lines

The block layer's integrity code currently sets the seed (initial reference tag) in units of 512-byte sectors but increments it in units of integrity intervals

I don't necessarily agree with the premise that the seed needs to be expressed in any particular unit. The seed is a start value, nothing more.

NVM Command Set specification 1.1 section 5.3.3 requires the reference tag to increment by 1 per logical block, so that seems to determine the increment unit:

Quoted 15 lines

If the Reference Tag Check bit of the PRCHK field is set to ‘1’ and the namespace is formatted for Type 1 or Type 2 protection, then the controller compares the Logical Block Reference Tag to the computed reference tag. The computed reference tag depends on the Protection Type:
▪ If the namespace is formatted for Type 1 protection, the value of the computed reference tag for the first logical block of the command is the value contained in the Initial Logical Block Reference Tag (ILBRT) or Expected Initial Logical Block Reference Tag (EILBRT) field in the command, and the computed reference tag is incremented for each subsequent logical block. The controller shall complete the command with a status of Invalid Protection Information if the ILBRT field or the EILBRT field does not match the value of the least significant bits of the SLBA field sized to the number of bits in the Logical Block Reference Tag (refer to section 5.3.1.4).
Note: Unlike SCSI Protection Information Type 1 protection which implicitly uses the least significant four bytes of the LBA, the controller always uses the ILBRT or

Quoted 8 lines

EILBRT field and requires the host to initialize the ILBRT or EILBRT field to the least significant bits of the LBA sized to the number of bits in the Logical Block Reference Tag when Type 1 protection is used.
▪ If the namespace is formatted for Type 2 protection, the value of the computed reference tag for the first logical block of the command is the value contained in the Initial Logical Block Reference Tag (ILBRT) or Expected Initial Logical Block Reference Tag (EILBRT) field in the command, and the computed reference tag is incremented for each subsequent logical block.

The ref tag used for a particular block needs to be consistent. And since reftag(block N) can be computed as the reftag(M) + N - M if block N is accessed as part of an I/O that begins at block M, the function must be of the form reftag(block N) = N + c for some constant c. Thus, the ref tag seed needs to be computed in units of logical blocks (integrity intervals); no other unit (e.g. 512-byte sectors) works.
To see the issue with the current approach, consider an example accessing LBA 1 on a device with a 4 KB block size. If the block is written as part of a write that begins at LBA 0, its ref tag in the generated PI will be 1 (sector 0 + 1 integrity interval). If it's later read by a read starting at LBA 1, its expected ref tag will be 8 (sector 8 + 0 integrity intervals), and the auto-integrity code will fail the read due to a reftag mismatch. This seems completely unworkable for a block storage device.

Quoted 7 lines

We happen to set it to the block number in the block layer since we need to be able to know what to compare against on completion (for Type 1 + the restrictive Linux implementation of Type 2). But that does not imply that the seed needs to be specified in any particular unit. Submitters set the seed to whichever value makes sense to them (i.e. it could be the offset within a file as opposed to the eventual LBA on the backend

I agree, the seed doesn't need to match the final LBA, but it does need to be in *units* of logical blocks, plus some constant offset.

Quoted 18 lines

device). And then that seed is incremented by 1 for each integrity interval of data in the PI sent to/received from the device. The conversion between the submitter's view of what the first ref tag should be (i.e. seed) and what is required by the hardware (for instance lower 32 bits of device LBA) is the reason we perform remapping. The seed is intentionally different in the submitter's protection envelope compared to the device's protection envelope.

Using the block layer block number as seed was just a convenience since that provided a predictable value for any I/O that had its PI autogenerated. I never intended for the actual LBA to be used as seed value on a 4Kn device. Initially we just used 0 as the seed. Leveraging the block number just added a bit of additional protection.

I confess I haven't tested 4Kn in a while since things sort of converged on 512e. But I used to run nightly tests on a SCSI storage with 4Kn blocks just fine.

Quoted 5 lines
Quoted 3 lines

This looks to be a longstanding bug affecting block devices that support integrity with block sizes > 512 bytes; I'm surprised it wasn't noticed before.

Are you seeing this with NVMe or SCSI?

With a ublk device. It should affect any block device that supports integrity and has a logical block size > 512.

Best,
Caleb

#9 lore raw

Hi Caleb!

Quoted 3 lines

NVM Command Set specification 1.1 section 5.3.3 requires the reference tag to increment by 1 per logical block, so that seems to determine the increment unit:

SCSI allows PI to be interleaved at intervals smaller than the logical block size. This was done for PI compatibility in mixed environments with both 512[en] and 4Kn disks. Interleaving allows 8 bytes of PI per 512 bytes of data on devices using 4 KB logical blocks. That is the reason why we use the term "integrity interval" instead of assuming logical block size.

Quoted 7 lines

The ref tag used for a particular block needs to be consistent. And since reftag(block N) can be computed as the reftag(M) + N - M if block N is accessed as part of an I/O that begins at block M, the function must be of the form reftag(block N) = N + c for some constant c. Thus, the ref tag seed needs to be computed in units of logical blocks (integrity intervals); no other unit (e.g. 512-byte sectors) works.

Whoever attaches the PI decides on the seed value. In the case of the block layer it made sense to use block layer sector number since that value is inevitably going to be the same for a future read.

Note that with MD, DM, and partitioning in the mix, the sector number seen by whoever submits the I/O is going to be different from the LBAs on the target devices which eventually receive the I/O. Nobody says there is a computable constant offset. Think scattered LVM extent allocations. Or RAID stripes placed at mismatched LBA offsets.

Quoted 7 lines

To see the issue with the current approach, consider an example accessing LBA 1 on a device with a 4 KB block size. If the block is written as part of a write that begins at LBA 0, its ref tag in the generated PI will be 1 (sector 0 + 1 integrity interval). If it's later read by a read starting at LBA 1, its expected ref tag will be 8 (sector 8 + 0 integrity intervals), and the auto-integrity code will fail the read due to a reftag mismatch.

Something is broken, then. Because the ref tag in the received PI should have been remapped to start at 8 in that case.

Quoted 2 lines

I agree, the seed doesn't need to match the final LBA, but it does need to be in *units* of logical blocks, plus some constant offset.

Your concept of "unit" still sends the wrong message. The seed is an integer value used to initialize a counter or hardware register. The seed only has meaning to whichever entity submits the I/O. To everything else it is a value used for remapping ref tags from the I/O submitter's point of view to whichever interpretation is mandated by the storage hardware's PI format.

Quoted 2 lines

With a ublk device. It should affect any block device that supports integrity and has a logical block size > 512.

It sounds like the seed value is set incorrectly for reads in your configuration.

Signature · 1 line
Martin K. Petersen
#10 lore raw
Quoted 8 lines · Martin K. Petersen

Hi Caleb!

Quoted 3 lines

NVM Command Set specification 1.1 section 5.3.3 requires the reference tag to increment by 1 per logical block, so that seems to determine the increment unit:

Quoted 6 lines

SCSI allows PI to be interleaved at intervals smaller than the logical block size. This was done for PI compatibility in mixed environments with both 512[en] and 4Kn disks. Interleaving allows 8 bytes of PI per 512 bytes of data on devices using 4 KB logical blocks. That is the reason why we use the term "integrity interval" instead of assuming logical block size.

Thanks for the explanation, I'm not too familiar with SCSI. I meant to refer to integrity intervals in my explanation if they differ from the logical block size.

Quoted 9 lines
Quoted 8 lines

The ref tag used for a particular block needs to be consistent. And since reftag(block N) can be computed as the reftag(M) + N - M if block N is accessed as part of an I/O that begins at block M, the function must be of the form reftag(block N) = N + c for some constant c. Thus, the ref tag seed needs to be computed in units of logical blocks (integrity intervals); no other unit (e.g. 512-byte sectors) works.

Quoted 3 lines

Whoever attaches the PI decides on the seed value. In the case of the block layer it made sense to use block layer sector number since that value is inevitably going to be the same for a future read.

I'm not following "going to be the same for a future read". The block can be read back by an I/O with a different starting offset/sector/seed, as my example illustrates. When the integrity interval size differs from the sector size (512 bytes), mixing the two units results in a different ref tag seed for the block depending on the starting offset of the I/O.

Quoted 6 lines

Note that with MD, DM, and partitioning in the mix, the sector number seen by whoever submits the I/O is going to be different from the LBAs on the target devices which eventually receive the I/O. Nobody says there is a computable constant offset. Think scattered LVM extent allocations. Or RAID stripes placed at mismatched LBA offsets.

The constant offset relationship still needs to hold over any contiguous range of a backing block device that can be accessed by a single I/O. For example, with partitions, it's not possible for a single I/O to cross a partition boundary, so each partition can have a different constant offset between the ref tags and absolute integrity interval numbers. With RAID, each shard can have a different constant offset. etc.

Quoted 11 lines
Quoted 8 lines

To see the issue with the current approach, consider an example accessing LBA 1 on a device with a 4 KB block size. If the block is written as part of a write that begins at LBA 0, its ref tag in the generated PI will be 1 (sector 0 + 1 integrity interval). If it's later read by a read starting at LBA 1, its expected ref tag will be 8 (sector 8 + 0 integrity intervals), and the auto-integrity code will fail the read due to a reftag mismatch.

Something is broken, then. Because the ref tag in the received PI should have been remapped to start at 8 in that case.

Ah, I missed the remapping piece. Thanks for pointing that out. I guess I was testing with a ublk device that doesn't advertise BLK_INTEGRITY_REF_TAG. Since commit 203247c5cb97 ("blk-integrity:
support arbitrary buffer alignment"), the ref tag is unconditionally set in the PI from the (sector) seed, but the remapping is conditional on BLK_INTEGRITY_REF_TAG. That explains why I was seeing ref tags in the PI that didn't match the integrity interval numbers.

So seems like patch 1 ("block: use integrity interval instead of sector as seed") doesn't need a Fixes tag. Still, I'm confused why the auto-integrity code bothers setting the seed to the sector number in the first place if it's going to be remapped later. Why not just leave the seed zeroed?

Best,
Caleb

Quoted 19 lines
Quoted 3 lines

I agree, the seed doesn't need to match the final LBA, but it does need to be in *units* of logical blocks, plus some constant offset.

Your concept of "unit" still sends the wrong message. The seed is an integer value used to initialize a counter or hardware register. The seed only has meaning to whichever entity submits the I/O. To everything else it is a value used for remapping ref tags from the I/O submitter's point of view to whichever interpretation is mandated by the storage hardware's PI format.

Quoted 2 lines

With a ublk device. It should affect any block device that supports integrity and has a logical block size > 512.

It sounds like the seed value is set incorrectly for reads in your configuration.

--
Martin K. Petersen

#11 lore raw
Quoted 77 lines · Caleb Sander Mateos
Quoted 8 lines · Martin K. Petersen

Hi Caleb!

Quoted 3 lines

NVM Command Set specification 1.1 section 5.3.3 requires the reference tag to increment by 1 per logical block, so that seems to determine the increment unit:

Quoted 6 lines

SCSI allows PI to be interleaved at intervals smaller than the logical block size. This was done for PI compatibility in mixed environments with both 512[en] and 4Kn disks. Interleaving allows 8 bytes of PI per 512 bytes of data on devices using 4 KB logical blocks. That is the reason why we use the term "integrity interval" instead of assuming logical block size.

Thanks for the explanation, I'm not too familiar with SCSI. I meant to refer to integrity intervals in my explanation if they differ from the logical block size.

Quoted 9 lines
Quoted 8 lines

The ref tag used for a particular block needs to be consistent. And since reftag(block N) can be computed as the reftag(M) + N - M if block N is accessed as part of an I/O that begins at block M, the function must be of the form reftag(block N) = N + c for some constant c. Thus, the ref tag seed needs to be computed in units of logical blocks (integrity intervals); no other unit (e.g. 512-byte sectors) works.

Quoted 3 lines

Whoever attaches the PI decides on the seed value. In the case of the block layer it made sense to use block layer sector number since that value is inevitably going to be the same for a future read.

I'm not following "going to be the same for a future read". The block can be read back by an I/O with a different starting offset/sector/seed, as my example illustrates. When the integrity interval size differs from the sector size (512 bytes), mixing the two units results in a different ref tag seed for the block depending on the starting offset of the I/O.

Quoted 6 lines

Note that with MD, DM, and partitioning in the mix, the sector number seen by whoever submits the I/O is going to be different from the LBAs on the target devices which eventually receive the I/O. Nobody says there is a computable constant offset. Think scattered LVM extent allocations. Or RAID stripes placed at mismatched LBA offsets.

The constant offset relationship still needs to hold over any contiguous range of a backing block device that can be accessed by a single I/O. For example, with partitions, it's not possible for a single I/O to cross a partition boundary, so each partition can have a different constant offset between the ref tags and absolute integrity interval numbers. With RAID, each shard can have a different constant offset. etc.

Quoted 11 lines
Quoted 8 lines

To see the issue with the current approach, consider an example accessing LBA 1 on a device with a 4 KB block size. If the block is written as part of a write that begins at LBA 0, its ref tag in the generated PI will be 1 (sector 0 + 1 integrity interval). If it's later read by a read starting at LBA 1, its expected ref tag will be 8 (sector 8 + 0 integrity intervals), and the auto-integrity code will fail the read due to a reftag mismatch.

Something is broken, then. Because the ref tag in the received PI should have been remapped to start at 8 in that case.

Ah, I missed the remapping piece. Thanks for pointing that out. I guess I was testing with a ublk device that doesn't advertise BLK_INTEGRITY_REF_TAG. Since commit 203247c5cb97 ("blk-integrity:
support arbitrary buffer alignment"), the ref tag is unconditionally set in the PI from the (sector) seed, but the remapping is conditional on BLK_INTEGRITY_REF_TAG. That explains why I was seeing ref tags in the PI that didn't match the integrity interval numbers.

Quoted 5 lines

So seems like patch 1 ("block: use integrity interval instead of sector as seed") doesn't need a Fixes tag. Still, I'm confused why the auto-integrity code bothers setting the seed to the sector number in the first place if it's going to be remapped later. Why not just leave the seed zeroed?

Martin,
I would appreciate a response here. Would you be okay with patch 1 if the Fixes tags were dropped? Do you think we can get rid of the ref tag seed initialization entirely if the ref tags get remapped later anyways? Even if patch 1 is not required for correctness, patch 2 is a fix for a separate issue introduced in the 7.1 merge window and has reviews from Christoph and Anuj. I would prefer not to hold up that fix over this ref tag seed discussion.

Best,
Caleb

#12 lore raw

Hi Caleb!

Sorry about the delay. Been away for a few weeks...

Quoted 5 lines
Quoted 5 lines

So seems like patch 1 ("block: use integrity interval instead of sector as seed") doesn't need a Fixes tag. Still, I'm confused why the auto-integrity code bothers setting the seed to the sector number in the first place if it's going to be remapped later. Why not just leave the seed zeroed?

It adds a bit of extra protection in the sense that it there is one more parameter that can be validated. The premise of the integrity infrastructure is that things in the two supplied buffers (data + PI) as well as the control path (bip in the block layer case plus the SCSI or NVMe command fields) all need to agree for the I/O to go through.

It is valid to generate the PI starting with 0. But that is indistinguishable from "the seed value was not initialized".

Quoted 2 lines

I would appreciate a response here. Would you be okay with patch 1 if the Fixes tags were dropped?

I am afraid I still don't completely understand why things are broken.

For writes, the meaning of the bip seed is: "This is the value you should expect in the ref tag for the first integrity interval in the PI buffer I prepared". With block layer autoprotect, the seed is set before generating the PI and thus implicitly affects the generation.

When the write operation subsequently reaches the bottom of the stack, we will check that the first ref tag in the PI buffer matches the supplied seed value. And then proceed to remap the ref tags for each protection interval to the target LBA + n since that is what the storage requires (ignoring the odd Type 2 interval mismatch for now).

For reads, the meaning of the bip seed is: "This is what I expect to receive in the ref tag for the first integrity interval in the PI buffer". At the bottom of the stack we will receive PI from the storage and that will contain ref tags matching the lower 32 bits of the LBA since that is what the hardware returns. And we will then remap all those ref tags starting with whichever bip seed value was requested by the caller. It doesn't matter whether the requested seed value was 0, 10, or 42. The ref tags are remapped to whatever the caller wants them to be.

I tend to think of the seed as a register you program with the value you want. And then hardware or software remaps between what the storage device's protection envelope requires and what the application (or in this case the block layer) requested. With SCSI + DIX 1.1, the seed literally controls a remapping register in the HBA ASIC. In NVMe we have ILBRT/EILBRT.

Signature · 1 line
Martin K. Petersen
#13 lore raw
Quoted 18 lines · Martin K. Petersen

Hi Caleb!

Sorry about the delay. Been away for a few weeks...

Quoted 5 lines
Quoted 5 lines

So seems like patch 1 ("block: use integrity interval instead of sector as seed") doesn't need a Fixes tag. Still, I'm confused why the auto-integrity code bothers setting the seed to the sector number in the first place if it's going to be remapped later. Why not just leave the seed zeroed?

It adds a bit of extra protection in the sense that it there is one more parameter that can be validated. The premise of the integrity infrastructure is that things in the two supplied buffers (data + PI) as well as the control path (bip in the block layer case plus the SCSI or NVMe command fields) all need to agree for the I/O to go through.

Quoted 7 lines

It is valid to generate the PI starting with 0. But that is indistinguishable from "the seed value was not initialized".

Quoted 2 lines

I would appreciate a response here. Would you be okay with patch 1 if the Fixes tags were dropped?

I am afraid I still don't completely understand why things are broken.

Nothing is broken, I just mean that the seed value stored in bip_iter.bi_sector is strange in that it's initialized in units of 512-byte sectors but incremented in units of integrity intervals. As you point out, the remapping step makes the initial seed value irrelevant, but I was certainly confused by it when I printed it during some debugging. I can update the commit message to clarify the rationale for the change.

Quoted 22 lines

For writes, the meaning of the bip seed is: "This is the value you should expect in the ref tag for the first integrity interval in the PI buffer I prepared". With block layer autoprotect, the seed is set before generating the PI and thus implicitly affects the generation.

When the write operation subsequently reaches the bottom of the stack, we will check that the first ref tag in the PI buffer matches the supplied seed value. And then proceed to remap the ref tags for each protection interval to the target LBA + n since that is what the storage requires (ignoring the odd Type 2 interval mismatch for now).

For reads, the meaning of the bip seed is: "This is what I expect to receive in the ref tag for the first integrity interval in the PI buffer". At the bottom of the stack we will receive PI from the storage and that will contain ref tags matching the lower 32 bits of the LBA since that is what the hardware returns. And we will then remap all those ref tags starting with whichever bip seed value was requested by the caller. It doesn't matter whether the requested seed value was 0, 10, or 42. The ref tags are remapped to whatever the caller wants them to be.

Quoted 6 lines

I tend to think of the seed as a register you program with the value you want. And then hardware or software remaps between what the storage device's protection envelope requires and what the application (or in this case the block layer) requested. With SCSI + DIX 1.1, the seed literally controls a remapping register in the HBA ASIC. In NVMe we have ILBRT/EILBRT.

What I find confusing is that the seed value stored in bip_iter.bi_sector isn't what's actually passed to the SCSI/NVMe device. It's only used in blk_integrity_iterate() and __blk_reftag_remap() to generate/verify/remap the reftags in the integrity/PI buffer. However, (E)ILBRT field (taking NVMe as an example) comes from the physical block device offset rather than the reftag seed. See t10_pi_ref_tag(), which returns blk_rq_pos() converted to integrity intervals. It looks like this works because the remap step ensures the reftags passed in the integrity buffer match the physical integrity interval numbers, but this means the device is comparing physical integrity interval numbers rather than reftag seeds. My point is that if the remap step undoes the effect of the seed by setting all the reftags in the integrity buffer to their physical integrity interval, I don't see why the block integrity code bothers setting a seed in the first place.

But it sounds like this may be a longer discussion, so I will split out the two fixes for 7.1 into a separate series.

Thanks,
Caleb