status of block-integrity

We have the block integrity code to support DIF/DIX in the the tree for about 5 and a half years, and we still don't have a single consumer of it. By normal kernel rules it should never have been merged, or at least the bitrot long removed.

Given that we'll have a lot of work to do in this area with block multiqueue I think it's time to either kill it off for good or make sure we can actually use and test it.

Is not sd_dif.c a consumer of blk_integrity.c logic..?

Speak of the devil..

I've been working on enabling DIF support in scsi-mq recently, and AFAICT the only part that is required in blk-mq for DIF emulation to function with scsi-debug is the following patch.

commit 1428a390cc16025f93905852777d4afd8aeba05d

Author: Nicholas Bellinger <[email protected]>
Date:   Sun Dec 22 11:58:49 2013 +0000

blk-mq: Add bio_integrity setup to blk_mq_make_request

This patch adds the missing bio_integrity_enabled() + bio_integrity_prep() setup into blk_mq_make_request() in order to use DIF protection with scsi-mq.

index c79126e..a520c39 100644
@@ -916,6 +916,11 @@ static void blk_mq_make_request(struct request_queue *q, struct bio *bio)
 
        blk_queue_bounce(q, &bio);
 
+       if (bio_integrity_enabled(bio) && bio_integrity_prep(bio)) {
+               bio_endio(bio, -EIO);
+               return;
+       }
+
        if (use_plug && blk_attempt_plug_merge(q, bio, &request_count))
                return;
 

Christoph> We have the block integrity code to support DIF/DIX in the Christoph> the tree for about 5 and a half years, and we still don't Christoph> have a single consumer of it.

What do you mean? If you have a DIX-capable HBA (lpfc, qla2xxx, zfcp) then integrity protection is active from the block layer down. The only code that's not currently being exercised are the tag interleaving functions. I was hoping the FS people would use them for back pointers but nobody seemed to bite.

Christoph> Given that we'll have a lot of work to do in this area with Christoph> block multiqueue I think it's time to either kill it off for Christoph> good or make sure we can actually use and test it.

I don't understand why multiqueue would require a lot of work? It's just an extra scatterlist per request.

And obviously, if there's anything that needs to be done in this area I'll be happy to do so...

Martin K. Petersen	Oracle Linux Engineering

With single consumer of it I obviously meant the various symbols for the consumer side as well as the application tag support.

Patch to remove the dead code below:

---
From: Christoph Hellwig <[email protected]>
Subject: [PATCH] block: remove dead on arrival integrity code

index 03cf717..0b14db7 100644
@@ -194,7 +194,6 @@ int blk_integrity_merge_rq(struct request_queue *q, struct request *req,
 
 	return 0;
 }
-EXPORT_SYMBOL(blk_integrity_merge_rq);
 
 int blk_integrity_merge_bio(struct request_queue *q, struct request *req,
 			    struct bio *bio)

@@ -214,7 +213,6 @@ int blk_integrity_merge_bio(struct request_queue *q, struct request *req,
 
 	return 0;
 }
-EXPORT_SYMBOL(blk_integrity_merge_bio);
 
 struct integrity_sysfs_entry {
 	struct attribute attr;

@@ -414,8 +412,6 @@ int blk_integrity_register(struct gendisk *disk, struct blk_integrity *template)
 		bi->generate_fn = template->generate_fn;
 		bi->verify_fn = template->verify_fn;
 		bi->tuple_size = template->tuple_size;
-		bi->set_tag_fn = template->set_tag_fn;
-		bi->get_tag_fn = template->get_tag_fn;
 		bi->tag_size = template->tag_size;
 	} else
 		bi->name = bi_unsupported_name;

index 060ff69..4ff2d82 100644
@@ -576,14 +576,9 @@ struct biovec_slab {
 
 #define bio_integrity(bio) (bio->bi_integrity != NULL)
 
-extern struct bio_integrity_payload *bio_integrity_alloc(struct bio *, gfp_t, unsigned int);
 extern void bio_integrity_free(struct bio *);
-extern int bio_integrity_add_page(struct bio *, struct page *, unsigned int, unsigned int);
 extern int bio_integrity_enabled(struct bio *bio);
-extern int bio_integrity_set_tag(struct bio *, void *, unsigned int);
-extern int bio_integrity_get_tag(struct bio *, void *, unsigned int);
 extern int bio_integrity_prep(struct bio *);
-extern void bio_integrity_endio(struct bio *, int);
 extern void bio_integrity_advance(struct bio *, unsigned int);
 extern void bio_integrity_trim(struct bio *, unsigned int, unsigned int);
 extern void bio_integrity_split(struct bio *, struct bio_pair *, int);

index 1b135d4..d6db54e 100644
@@ -1430,14 +1430,10 @@ struct blk_integrity_exchg {
 
 typedef void (integrity_gen_fn) (struct blk_integrity_exchg *);
 typedef int (integrity_vrfy_fn) (struct blk_integrity_exchg *);
-typedef void (integrity_set_tag_fn) (void *, void *, unsigned int);
-typedef void (integrity_get_tag_fn) (void *, void *, unsigned int);
 
 struct blk_integrity {
 	integrity_gen_fn	*generate_fn;
 	integrity_vrfy_fn	*verify_fn;
-	integrity_set_tag_fn	*set_tag_fn;
-	integrity_get_tag_fn	*get_tag_fn;
 
 	unsigned short		flags;
 	unsigned short		tuple_size;

You can add there are SCSI targets out there employing this feature as well. =;^)

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Martin K. Petersen
Sent: Monday, December 23, 2013 8:35 AM
To: Christoph Hellwig
Cc: Jens Axboe; [email protected]; [email protected]
Subject: Re: status of block-integrity

Christoph> We have the block integrity code to support DIF/DIX in the Christoph> the tree for about 5 and a half years, and we still don't Christoph> have a single consumer of it.

What do you mean? If you have a DIX-capable HBA (lpfc, qla2xxx, zfcp) then integrity protection is active from the block layer down. The only code that's not currently being exercised are the tag interleaving functions. I was hoping the FS people would use them for back pointers but nobody seemed to bite.

Christoph> Given that we'll have a lot of work to do in this area with Christoph> block multiqueue I think it's time to either kill it off for Christoph> good or make sure we can actually use and test it.

I don't understand why multiqueue would require a lot of work? It's just an extra scatterlist per request.

And obviously, if there's anything that needs to be done in this area I'll be happy to do so...

Martin K. Petersen	Oracle Linux Engineering

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

One of the major knocks on linux file systems (except for btrfs) that I hear is the lack of full data path checksums. DIF/DIX + xfs or ext4 done right will give us another answer here. I don't think it will be common, it is a request that comes in for very large storage customers most commonly.

We do have devices that support this and are working to get more vendor testing done, so I would hate to see us throw out the code instead of fixing it up for the end users that see value here.

I think that we can get this working & agree with the call to continue this discussion (here and at LSF :))

Ric

I would indeed like to have a discussion at LSF about the future of DIX. DIF is not an issue, as most HBAs support it already and we actually need it for proper connectivity.

DIX, OTOH, has been left dormant since time immemorial, with the only known (supposed) user being Oracle.
(I actually talked to the DB/2 folks about it, and the response was a polite feigned interest ...)

We need to come up with a concise story here (either integrate with filesystems or have a userland interface), otherwise it's just dead code and indeed should be removed.

Plus so far I've had exactly _one_ request for DIX, and even that came from a company which has its own custom storage array firmware. Making me wonder if DIF/DIX is really that important or more of an tick-mark during procuring ...

Even so, it would warrant a discussion at LSF.

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

I think there's a terminology confusion here; you seem to be using DIX to mean the TCP CRC and DIF to mean T10DIF. I've seen other people use DIX to mean separate SGLs for metadata and DIF to mean interleaved data. Can you confirm which thing you mean here?

Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

No, I think you're confusing algorithms with protocols. DIF and DIX are two names for protection envelopes. DIF verifies integrity from the HBA to the device surface. DIX verifies integrity from an application to the HBA. Both DIF and DIX have pluggable checksum algorithms (and, in theory, as long as the HBA does the conversion, they don't have to use the same one, although the confusion over protection types and algorithms is so dense already that the only way not to go insane is to use the same end to end one). Oracle has the best data sources to explain this, including Martin's slides:

https://oss.oracle.com/projects/data-integrity/documentation/

The specific problem is that there's no defined interface for any application to use DIX easily because it has to supply additional protection information when it reads or writes data and there's no agreed way to extend read/write to do this and, as Martin has said, thinging about trying to do this with mmap leads to a "bonghit bonanza".

So, the question is do we need to bother with DIX at all? No filesystem uses it and there seems to be weak user demand at best. We could just strip DIX, losing the protection envelope from the application to the HBA but keeping DIF, which is the protection envelope from the HBA to the device.

James

James> No, I think you're confusing algorithms with protocols. DIF and James> DIX are two names for protection envelopes. DIF verifies James> integrity from the HBA to the device surface. DIX verifies James> integrity from an application to the HBA.

Actually, DIX is a data integrity-aware HBA programming interface. We have an implementation of that interface in the SCSI layer and in some of the initiator drivers (lpfc, qla2xxx, mptNsas).

There is no single name for stuff above DIX. Other than "block layer data integrity goo", "page cache black magic" and "let's add a few fields to struct iocb".

James> So, the question is do we need to bother with DIX at all? No James> filesystem uses it

...explicitly. Every filesystem uses it implicitly. There are only two reasons for filesystems to want to be explicitly "block layer data integrity goo"-aware:

1. To be able to use the application tag space for back pointers or other metadata without requiring disk format changes.

2. To facilitate passthrough of protection information submitted via the $TBD application programming interface.

I was hoping the extN folks would be interested in (1) but there were no takers. (2) is hard but not forgotten. In any case the status quo is that there is no point in filesystems manually generating protection information when the block layer is going to do it for them when the bio is submitted.

Martin K. Petersen	Oracle Linux Engineering

Which would make an ideal topic for LSF, wouldn't it?

Personally, I doubt it's a good idea to kill it off, but a proper (userland) API for it has been a long time missing.

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Hannes> Personally, I doubt it's a good idea to kill it off, but a Hannes> proper (userland) API for it has been a long time missing.

Before we throw the baby out with the bath water, maybe Darrick can fill us in on the progress of the aio passthrough interface?

Martin K. Petersen	Oracle Linux Engineering

I haven't made much progress on it -- I haven't seen any earnest demand for it.

Last year Chuck Lever said that some NFS working group was looking defining an interface it... has there been any progress? It doesn't sound like there has been.

--D

Of course not. Who should be demanding it? Application developer tend to code to existing interfaces.

Well, the point is that without any defined (userland) interface it's quite hard to pursuade any userland application developer to use it.

Plus (as hch rightly pointed out) as there is no defined userland interface the question is why we bother with all the DIX stuff in the block layer.
DIF support would be perfectly sufficient to cover any connectivity issues.

And one feels _really_ silly trying to convince customers from the benefits of DIX they can't even use.
I've tried that several times, and it doesn't get better over time...

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hannes> Plus (as hch rightly pointed out) as there is no defined Hannes> userland interface the question is why we bother with all the Hannes> DIX stuff in the block layer.

Because it catches problems in the path between block layer and HBA ASIC? FWIW, we find more issues there than we do between initiator and target.

API issues aside, another reason adoption has been slow is that very few applications truly care about this stuff. The current approach in which data is protected when the I/O is submitted by the filesystem is good enough for most things. Saves the filesystem people the trouble of dealing with it too.

In reality there are only a handful of applications that would actually benefit from an explicit userland API. Mostly in the database department. All the potential consumers of an interface I talked to wanted to use aio so that's why we've focused our efforts there.

Both Darrick and I have been busy with other projects the last little while. I'll start looking at this again when I'm done with copy offload...

Martin K. Petersen	Oracle Linux Engineering

But how should it do that exactly?
As there is no user (apart from oracleasm) no-one can attach protection information to any data, so even the most dedicated admin cannot exercise this path, let alone find issues here.

aio is perfectly fine; all I care is to have _any_ way of feeding protection information into the kernel.

Speaking of which, are there any patches?
Doug Gilbert and I are currently discussing LID4 / ROD Token copy for sg3_utils and the block layer, so any patches would be very helpful here.

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hannes,

Hannes> As there is no user (apart from oracleasm) no-one can attach Hannes> protection information to any data, so even the most dedicated Hannes> admin cannot exercise this path, let alone find issues here.

That's not how it works!

If the filesystem has not attached protection information to a bio the block layer will do it for you. The block layer generates protection information for writes and verifies it for reads. That's how it's worked since day one. The code is there, it is used by everyone with a DIX-capable HBA. See Documentation/block/data-integrity.txt.

Normal applications do not want to have to deal with generating protection information, using an async I/O model, keeping completion state around for extended periods of time to figure out whether the I/O actually completed or not and so on. So the kernel-to-platter protection scheme we have in place now is good enough.

That doesn't mean that I'm not interested in augmenting libaio. I am. Very. And I know of several applications that are keen to use it. But getting page cache passthrough and filesystem interaction working is non-trivial. That's what has inhibited progress, not extending the libaio API.

Hannes> Doug Gilbert and I are currently discussing LID4 / ROD Token Hannes> copy for sg3_utils and the block layer, so any patches would be Hannes> very helpful here.

I'm only doing LID1 right now. Any particular reason you are exploring LID4 and ROD?

I resumed my efforts before Christmas but I keep running into issues. I'm guessing I'm a week or two from having something that is suitable for consumption.

Martin K. Petersen	Oracle Linux Engineering

Ah. I stand corrected.
Sorry.

Same here. Actually I _like_ DIX, but the missing libaio / userland API makes it very hard to utilize it.

Yes. LID4/ROD token is far easier to use (conceptually).
With LID1/XCopy you have the ambiguity on where to actually send the command to; the spec is silent in this area. Also for LID1/XCopy you have three steps:
- Query the source disk
- Query the target disk
- Send the command to either source or target
Which is very awkward and one has to think really carefully on how to implement this without all sorts of layering violations. And you have to mix-and-match between all the various xcopy descriptors;
if there's only one it's easy enough, but when several things are getting interesting.

With LID4/ROD Token copy you basically have _two_ steps:
- Get the ROD Token from the source device
- Send the ROD Token to the target device

Which is far easier (conceptually).
Or that's the hope, anyway.
Also, the ROD Token in principle has an independent lifetime, so you could take an arbitrary amount of time between those steps. It might expire, though, but then failure is always an option when working with copy offload.

As said, Doug and me are working on putting this into sg3_utils, then we'll have a better idea on the actual workings.

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hannes> With LID1/XCopy you have the ambiguity on where to actually send Hannes> the command to; the spec is silent in this area.

Yeah, right now it's a coin toss.

However, thanks to VAAI most arrays support LID1. I'm trying to leverage that. Doesn't in any way preclude LID4 being supported as well.

Hannes> As said, Doug and me are working on putting this into sg3_utils, Hannes> then we'll have a better idea on the actual workings.

Cool! That'll save me some headaches when we start seeing LID4 devices.

Martin K. Petersen	Oracle Linux Engineering

You must be thinking of some other Chuck Lever ;-)

What I promised to deliver was plumbing in the NFS protocol to support end-to-end data integrity. That's here:

https://datatracker.ietf.org/doc/draft-cel-nfsv4-end2end-data-protection/

The issue of system call API is as sticky for NFS as it is for other e2e implementations. Without an API, other projects have been allowed to take up the time I would have spent on an NFS prototype.

What's more, some of the fields in the T10 tag are meaningless for byte-stream: an application, for example, will know nothing of block addresses, since those are chosen by the underlying filesystem. NFS (or any byte stream e2e integrity API) will have to define a different protection envelope with perhaps different tag contents.

But we do have a range of potential use cases for NFS: hypervisors that emulate block devices using NFS files, strong cryptographic data checksums, and of course the Oracle database. No real demand, as others have said.

Chuck Lever
chuck[dot]lever[at]oracle[dot]com