RFC: SCSI Generic version 4 interface
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer SG_IO ioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.
Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI command
Feel free to make suggestions.
Doug Gilbert
SCSI Generic version 4 interface structure
==========================================
Version 1.1Goals:
- handle both generalized request/response and data_out/data_in independently in same invocation. Alternatively the request and data_out could be in one invocation and the response and data_in in a following invocation. This allows for the most complicated SCSI commands: tagged variable length cdbs with bidirectional data transfers.
- support multiple protocols. If they are generalized request-response protocols then they can choose either the request/response part of the interface or the data_out/data_in part.
- layered error/condition reporting: (OS) driver, transport and device (logical unit). Method used to present this struct to OS (e.g. ioctl()) may also report error (e.g. EPERM).
- allow for auxiliary information to be passed back for the application client to consider
- same structure can be used for a synchronous (e.g. interruptible ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
- leave device (lu) or target addressing issues to some other mechanism (what SCSI standards call the I_T_L or the I_T nexus respectively) as they are transport dependent. However do include the tag level (the "_Q" part of a I_T_L_Q nexus).
- stay close enough to struct sg_io_hdr (sg version 3 interface) to use with existing SG_IO ioctls, current implementations expect 'S' in 'guard'
Comments:
- use 64 bit integers to convey pointers. Does this help with 32 to 64 bit thunking?
- should there be more (or less) spare fields?
- we could allow the "struct" to be rubbery, only requiring the first three integers and perhaps an additional_len field
- the write() usage in the sg driver's asynchronous interface has caused problems when mistakenly applied to a block device node rather than a sg device node. Using an ioctl(flag_async) followed by a read() for asynchronous work offers similar functionality and is safer. Using ioctl(flag_async_start) and ioctl(flag_async_finish) is another possibility.
- rather than have a separate ATA pass through mechanism, the SAT defined ATA PASS THROUGH SCSI commands could be used with the driver implementation routing the ATA commands to their subsystem. This could be flagged so it didn't preclude a SAT layer in a SCSI transport (e.g. MPT SAS HBA firmware).
- if SAM/SPC does not define an enumeration for lesser used input fields, then use the value 0 for inert/off/don't_care .
-------------------------------------------------------------------
struct sg_io_v4
{
int32_t guard; // [i] 'Q' to differentiate from v3
uint32_t protocol; // [i] 0 -> SCSI , ....
uint32_t subprotocol; // [i] 0 -> SCSI command, 1 -> SCSI task
// management function, ....
uint32_t request_len; // [i] in bytes
uint64_t request; // [i], [*i] {SCSI: cdb}
uint32_t request_attr; // [i] {SCSI: task attribute}
uint32_t request_tag; // [i] {SCSI: task tag (only if flagged)}
uint32_t request_priority; // [i] {SCSI: task priority}
uint32_t max_response_len; // [i] in bytes
uint64_t response; // [i], [*o] {SCSI: (auto)sense data}
// "din_" for data in (from device); "dout_" for data out (to device)
uint32_t dout_iovec_count; // [i] 0 -> "flat" data transfer
// else dout_xfer points to array of iovec
uint32_t dout_xfer_len; // [i] bytes to be transferred to device
uint64_t dout_xfer; // [i], [*i]
uint32_t din_iovec_count; // [i]
uint32_t din_xfer_len; // [i] bytes to be transferred from device
uint64_t din_xfer; // [i], [*o]
uint32_t timeout; // [i] units: seconds or milliseconds?
uint32_t flags; // [i] bit mask
uint64_t usr_ptr; // [i->o] unused internally
uint32_t spare_in; // [i]
uint32_t driver_status; // [o] 0 -> ok
uint32_t transport_status; // [o] 0 -> ok
uint32_t device_status; // [o] {SCSI: command completion status}
uint32_t retry_delay; // [o] {SCSI: status auxiliary information}
uint32_t info; // [o] additional information
uint32_t duration; // [o] time to complete, in milliseconds
uint32_t response_len_wr;// [o] bytes of response actually written
int32_t din_resid; // [o] actual_din_xfer_len - din_xfer_len
uint32_t generated_tag; // [o] {SCSI: task tag that transport chose}
uint32_t spare_out; // [o]};
-------------------------------------------------------------------------
Doug Gilbert
6th November 2006
Quoted 8 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.
Quoted 8 lines
Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI command
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
Jeff
Quoted 18 lines · Jeff Garzik
Quoted 8 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Quoted 8 lines
Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI command
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
Signature · 1 line
Jens Axboe
Quoted 18 lines · Jens Axboe
Quoted 17 lines · Jeff Garzik
Quoted 16 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI command
Quoted 8 lines
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
In the simplest case this is just the next generation version of the sg_io_hdr structure used by the SG_IO ioctl interface. It pitches at about the same level as modern SCSI transports (e.g. FCP, SAS and iSCSI). It leaves session and addressing issues, with the exception (arguably) of tags, to others. The "others" could include the bsg driver or perhaps a transport layer/driver.
Reference:
SAM-4 version 7 section 5.1 "The Execute command
procedure call"
http://www.t10.org/ftp/t10/drafts/sam4/sam4r07.pdfDoug Gilbert
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100
Quoted 19 lines · Jeff Garzik
Quoted 18 lines
Quoted 16 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI command
Quoted 8 lines
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
Quoted 28 lines · FUJITA Tomonori
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 24 lines · Jeff Garzik
Quoted 22 lines
Quoted 8 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Quoted 8 lines
Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandWhy avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
Quoted 7 lines
Quoted 3 lines
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
Signature · 1 line
Jens Axboe
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100
Quoted 33 lines · FUJITA Tomonori
Quoted 32 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 27 lines · Jeff Garzik
Quoted 18 lines
Quoted 8 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Quoted 8 lines
Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandQuoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
Quoted 6 lines
Quoted 3 lines
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.
Quoted 37 lines · FUJITA Tomonori
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100Quoted 33 lines · FUJITA Tomonori
Quoted 32 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 27 lines · Jeff Garzik
Quoted 18 lines
Quoted 8 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Quoted 8 lines
Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandQuoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.
bsg chooses the best way to handle the buffers on its own right now, in my opinion sg has grown too many "features" that aren't very useful. If the buffer and length are properly aligned, bsg does direct io. If not, it does indirect. mmap io may be useful, we could add that if anyone is actually using it.
Signature · 1 line
Jens Axboe
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:30:20 +0100
Quoted 42 lines · FUJITA Tomonori
Quoted 41 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100Quoted 37 lines · FUJITA Tomonori
Quoted 35 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 19 lines · Jeff Garzik
Quoted 18 lines
Quoted 16 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandQuoted 8 lines
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.bsg chooses the best way to handle the buffers on its own right now, in my opinion sg has grown too many "features" that aren't very useful.
I agree. But if bsg replaces sg3 code, it supports the full features, doesn't it? If bsg does not replace sg3 code, we have more duplication.
I think it would be nice for bsg to support only sg4 (does not support some of the old sg3 interfaces) and not add sg4 support to sg.c and scsi_ioctl.c (of course they can share some code).
Oh, bsg's sg3 doesn't work on 32-bit userspace and 64-bit kernel too if applicationos use the iovec interface, does it?
Quoted 2 lines
If the buffer and length are properly aligned, bsg does direct io. If not, it does indirect.
Yeah.
Quoted 2 lines
mmap io may be useful, we could add that if anyone is actually using it.
sg mmap io is tricky. sg_io_hdr doesn't have a mapped address, instead, the sg driver keeps a mapped address when an application calls mmap (that means sg cannot handle multiple mmap io requests simultaneously).
Quoted 54 lines · FUJITA Tomonori
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:30:20 +0100Quoted 42 lines · FUJITA Tomonori
Quoted 41 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100Quoted 37 lines · FUJITA Tomonori
Quoted 35 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 19 lines · Jeff Garzik
Quoted 18 lines
Quoted 16 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandQuoted 8 lines
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.bsg chooses the best way to handle the buffers on its own right now, in my opinion sg has grown too many "features" that aren't very useful.
Quoted 7 lines
I agree. But if bsg replaces sg3 code, it supports the full features, doesn't it? If bsg does not replace sg3 code, we have more duplication.
I think it would be nice for bsg to support only sg4 (does not support some of the old sg3 interfaces) and not add sg4 support to
sg.candscsi_ioctl.c(of course they can share some code).
Yeah, I agree with that (if only to reduce my headache of actually supporting the code!). When the sg v4 command interface is fully fleshed out, I'm fine with moving bsg in that direction and dumping sg v3 support completely.
Quoted 2 lines
Oh, bsg's sg3 doesn't work on 32-bit userspace and 64-bit kernel too if applicationos use the iovec interface, does it?
Probably not, haven't looked or checked :-)
Quoted 7 lines
Quoted 2 lines
mmap io may be useful, we could add that if anyone is actually using it.
sg mmap io is tricky.
sg_io_hdrdoesn't have a mapped address, instead, the sg driver keeps a mapped address when an application calls mmap (that means sg cannot handle multiple mmap io requests simultaneously).
With sg v3 support out of the picture, that's not a worry. I'll probably look into a different request/response architecture as well, the user visible ring buffer is a strong approach in my opinion.
Signature · 1 line
Jens Axboe
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:57:56 +0100
Quoted 59 lines · FUJITA Tomonori
Quoted 58 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:30:20 +0100Quoted 42 lines · FUJITA Tomonori
Quoted 41 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100Quoted 37 lines · FUJITA Tomonori
Quoted 35 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 19 lines · Jeff Garzik
Quoted 18 lines
Quoted 16 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandQuoted 8 lines
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.bsg chooses the best way to handle the buffers on its own right now, in my opinion sg has grown too many "features" that aren't very useful.
I agree. But if bsg replaces sg3 code, it supports the full features, doesn't it? If bsg does not replace sg3 code, we have more duplication.
Quoted 8 lines
Quoted 3 lines
I think it would be nice for bsg to support only sg4 (does not support some of the old sg3 interfaces) and not add sg4 support to
sg.candscsi_ioctl.c(of course they can share some code).Yeah, I agree with that (if only to reduce my headache of actually supporting the code!). When the sg v4 command interface is fully fleshed out, I'm fine with moving bsg in that direction and dumping sg v3 support completely.
Nice.
BTW, do you plan to use bsg only for sg4 (like putting sg4 hdr into bsg_command directly) or provide generic interfaces (I mean that sg4 calls something like bsg_register_protocol) ?
Quoted 4 lines
Quoted 2 lines
Oh, bsg's sg3 doesn't work on 32-bit userspace and 64-bit kernel too if applicationos use the iovec interface, does it?
Probably not, haven't looked or checked :-)
Whatever sg3 interface is used, bsg's sg3 doesn't work in such environments, I think.
Quoted 11 lines
Quoted 7 lines
Quoted 2 lines
mmap io may be useful, we could add that if anyone is actually using it.
sg mmap io is tricky.
sg_io_hdrdoesn't have a mapped address, instead, the sg driver keeps a mapped address when an application calls mmap (that means sg cannot handle multiple mmap io requests simultaneously).With sg v3 support out of the picture, that's not a worry. I'll probably look into a different request/response architecture as well, the user visible ring buffer is a strong approach in my opinion.
The scsi target infrastructure uses something like that. I'd like to use bsg for it in the future.
Quoted 72 lines · FUJITA Tomonori
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:57:56 +0100Quoted 59 lines · FUJITA Tomonori
Quoted 58 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:30:20 +0100Quoted 42 lines · FUJITA Tomonori
Quoted 41 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Thu, 7 Dec 2006 09:06:59 +0100Quoted 37 lines · FUJITA Tomonori
Quoted 35 lines
From: Jens Axboe <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 7 Nov 2006 16:31:48 +0100Quoted 19 lines · Jeff Garzik
Quoted 18 lines
Quoted 16 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This is the same structure that is used by the block layer
SG_IOioctl. A few people have asked whether I had forgotten that I agreed to write the proposal. So here it is. Those who have seen it have made comments, some of which have been incorporated.Some shortcomings of the sg version 3 interface are:
- can't handle commands with bidirectional data (either can the SCSI subsystem at the moment)
- if it was a bit more general it could carry other request/response protocols (e.g. Task Management Functions and SMP in Serial Attached SCSI)
- no way of associating a task attribute or task tag with a SCSI commandQuoted 8 lines
Quoted 4 lines
Why avoid Jens Axboe's bsg?
It seems like that is already a good interface for carrying other req/resp protocols.
I don't think Doug is avoiding that (if you are Doug, please do explain :-), but rather outlining the next generation command format that bsg should support for future use.
sg4 can be implemented on bsg nicely, I think. But why does the current bsg support sg3 partially? Do you plan to add the rest of sg3 (which doesn't work nicely) to bsg?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
What parts of sg v3 are missing? It's been a while since I implemented that stuff, so I forget if I knowingly left out some features.
SG_FLAG_MMAP_IO, indirect IO, probably I miss something more.bsg chooses the best way to handle the buffers on its own right now, in my opinion sg has grown too many "features" that aren't very useful.
I agree. But if bsg replaces sg3 code, it supports the full features, doesn't it? If bsg does not replace sg3 code, we have more duplication.
Quoted 8 lines
Quoted 3 lines
I think it would be nice for bsg to support only sg4 (does not support some of the old sg3 interfaces) and not add sg4 support to
sg.candscsi_ioctl.c(of course they can share some code).Yeah, I agree with that (if only to reduce my headache of actually supporting the code!). When the sg v4 command interface is fully fleshed out, I'm fine with moving bsg in that direction and dumping sg v3 support completely.
Quoted 5 lines
Nice.
BTW, do you plan to use bsg only for sg4 (like putting sg4 hdr into
bsg_commanddirectly) or provide generic interfaces (I mean that sg4 calls something likebsg_register_protocol) ?
I try not to over design such things, so it'll just be sg v4. If another cases arises, we can introduce something like what you suggest.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
mmap io may be useful, we could add that if anyone is actually using it.
sg mmap io is tricky.
sg_io_hdrdoesn't have a mapped address, instead, the sg driver keeps a mapped address when an application calls mmap (that means sg cannot handle multiple mmap io requests simultaneously).
Quoted 6 lines
Quoted 3 lines
With sg v3 support out of the picture, that's not a worry. I'll probably look into a different request/response architecture as well, the user visible ring buffer is a strong approach in my opinion.
The scsi target infrastructure uses something like that. I'd like to use bsg for it in the future.
Neat, I'll take patches for that :-)
My intention was to hangon a little and see if the kevent stuff got merged, then probably utilize that. If not, cook up my own.
Signature · 1 line
Jens Axboe
From: Douglas Gilbert <[email protected]>
Subject: RFC: SCSI Generic version 4 interface
Date: Mon, 06 Nov 2006 16:47:30 -0500
Quoted 5 lines
SCSI Generic version 4 interface structure ========================================== Version 1.1Goals:
(snip)
Quoted 2 lines
- same structure can be used for a synchronous (e.g. interruptible ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
Can you provide details on "asynchronous" part?
The scsi target code needs sg for:
- SAN gateway (like iSCSI to FC)
- SCSI device support in Xen (corresponds to raw device mapping in VMware); enables domU to use SCSI devices. domU sends SCSI commands via the virtual HBA driver to dom0 and then dom0 uses sg to perform them.
The scsi target code uses the single user-space daemon so needs an asynchronous interface, i.e. sends requests and receives the completions asynchronously.
Quoted 8 lines · FUJITA Tomonori
From: Douglas Gilbert <[email protected]>
Subject: RFC: SCSI Generic version 4 interface
Date: Mon, 06 Nov 2006 16:47:30 -0500Quoted 4 lines
SCSI Generic version 4 interface structure ========================================== Version 1.1
Quoted 8 lines
Quoted 1 line
Goals:
(snip)
Quoted 2 lines
- same structure can be used for a synchronous (e.g. interruptible ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
Can you provide details on "asynchronous" part?
I was trying to keep away from the implementation details but various people have pushed for more details ...
Well as you are aware, the sg driver uses a write(), poll_or_signal,read() sequence to do asynchronous IO in version 3 (and roughly the same model in version 2).
That write() has always made me uneasy and it is accident prone. It could easily be replaced by and SG_IO ioctl with a 'flag | async_request'. The question is how far that gets "down the stack" before it returns to the user. Some "tag" is required to be sent back to the caller of an async request to:
- enable the caller to identify the response
- enable the caller to issue task management
functions: abort task or query task
That is the purpose of the 'generated_tag' [o] field. I commented that field as transport layer generated (i.e. by the LLD) but it doesn't have to be, as long as it is mappable to one.
The asynchronous notification that the command is completed can be the current flavour of the month. The sg driver uses file descriptor based techniques. They seem to work in practice but the kernel folks don't seem to like them and they could be dropped or mishandled by the user space application.
Having a binary flag associated with the asynchronous notification for worked/didn't could be useful. Still a driver (e.g. sg) can't clear down its state information for the command in question until it receives some active acknowledgment from the user space program. The sg driver uses read() for this but this could just as easily be the SG_IO ioctl with a
'flag | async_complete'. The version 3 sg driver can filter those completions with its pack_id field (blocking or non-blocking) or take the first available. The sg v4 interface could filter via its request_tag (and a flag setting).
The sg driver has to cope with various non-completion situations:
- associated user space file descriptor closed by the time the async notification received: easy, throw away response and associated state
- rmmod sg
- app dies after receiving async notification: driver gets release() on that fd and throws away pending completion state
- app falls silent: that leads to a stuck command in the sg driver or sg could be told to run its own auto completion timer
The async completion phase (i.e. the read() in sg v3) has very little data to move as the xfer data pointers and sense data pointer have been set up in the async request. That only leaves: scsi_status, transport errors, resid and some accounting (e.g. command duration). Its most important function is being an active acknowledgment that the app received the async command complete notification.
As for data buffer control, I would like to look at tgt's ring buffers.
Quoted 3 lines
The scsi target code needs sg for:
- SAN gateway (like iSCSI to FC)
on the initiator side, I assume.
Quoted 8 lines
- SCSI device support in Xen (corresponds to raw device mapping in VMware); enables domU to use SCSI devices. domU sends SCSI commands via the virtual HBA driver to dom0 and then dom0 uses sg to perform them.
The scsi target code uses the single user-space daemon so needs an asynchronous interface, i.e. sends requests and receives the completions asynchronously.
Doug Gilbert
From: Douglas Gilbert <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 21 Nov 2006 17:06:46 -0500
Quoted 22 lines
Quoted 8 lines · FUJITA Tomonori
From: Douglas Gilbert <[email protected]>
Subject: RFC: SCSI Generic version 4 interface
Date: Mon, 06 Nov 2006 16:47:30 -0500Quoted 4 lines
SCSI Generic version 4 interface structure ========================================== Version 1.1Quoted 8 lines
Quoted 1 line
Goals:
(snip)
Quoted 2 lines
- same structure can be used for a synchronous (e.g. interruptible ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
Can you provide details on "asynchronous" part?
I was trying to keep away from the implementation details but various people have pushed for more details ...
Quoted 6 lines
Well as you are aware, the sg driver uses a
write(), poll_or_signal,read() sequence to do asynchronous IO in version 3 (and roughly the same model in version 2).That
write()has always made me uneasy and it is accident prone.
I think write and read interface works nicely (though it's accident prone as you said). They enable user-space applications to easily send and receive sg_io_v4 structures asynchronously in batches (v3 can't in batches). As you know, the current bsg adopts this interface.
Another possible interface that I prefer is to send and receive sg_io_v4 structures via two ring buffers.
Quoted 5 lines
Quoted 3 lines
The scsi target code needs sg for:
- SAN gateway (like iSCSI to FC)
on the initiator side, I assume.
Just to be sure we're talking about the same thing;
I've been building a linux box forwarding SCSI commands from iSCSI initiators to FC disk (an iSCSI-FC router).
Tgt's iSCSI driver runs in user space, so needs user-space APIs to send SCSI commands (with attribute and tag) and TMFs to SCSI devices. Now the iSCSI driver can forward only SCSI commands by using bsg. It needs sg v4.
Some other tgt drivers run in kernel space so we might try to add some tgt's kernel APIs for this later on.
From: FUJITA Tomonori <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Sun, 26 Nov 2006 01:02:20 +0900
Quoted 33 lines
From: Douglas Gilbert <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Tue, 21 Nov 2006 17:06:46 -0500Quoted 22 lines
Quoted 8 lines · FUJITA Tomonori
From: Douglas Gilbert <[email protected]>
Subject: RFC: SCSI Generic version 4 interface
Date: Mon, 06 Nov 2006 16:47:30 -0500Quoted 4 lines
SCSI Generic version 4 interface structure ========================================== Version 1.1Quoted 8 lines
Quoted 1 line
Goals:
(snip)
Quoted 2 lines
- same structure can be used for a synchronous (e.g. interruptible ioctl) or asynchronous (e.g. ioctl()/read() ) pass through.
Can you provide details on "asynchronous" part?
I was trying to keep away from the implementation details but various people have pushed for more details ...
Quoted 6 lines
Well as you are aware, the sg driver uses a
write(), poll_or_signal,read() sequence to do asynchronous IO in version 3 (and roughly the same model in version 2).That
write()has always made me uneasy and it is accident prone.
Quoted 7 lines
I think write and read interface works nicely (though it's accident prone as you said). They enable user-space applications to easily send and receive
sg_io_v4structures asynchronously in batches (v3 can't in batches). As you know, the current bsg adopts this interface.Another possible interface that I prefer is to send and receive
sg_io_v4structures via two ring buffers.
One problem about using non ioctl interfaces (read/write, ring buffer, whatever) is iovec compat issue. If sg4 still supports iovec for data transfer, it needs compat stuff (like sg_build_iovec). And non ioctl interfaces with iovec need to do more nasty compat stuff (like drivrs/input/evdev.c does).
So I think that it's great for sg v4 not to support the old data-transfer interfaces (iovec, dio, and indirect io).
Quoted 4 lines · Douglas Gilbert
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This
Feel free to make suggestions.
OK... A unified multiple device mmap buffer interface for would probably be useful
as well. This is to get around the problems inherent in the current design that keep more than one device from using a mmap/reserve buffer allocated for another device.
What I would personally like to be able to do with the SG interface is have a
shared memory region where I can kick off a few dozen tagged commands to a device, get notification when they have completed and then send a similar command to a second device, all without involving the CPU in any copy operations, or too many page management operations.
In my dream world this would require some kind of tagging of commands as well
so I don't have to handle a bunch of async completions for commands in a group that need to complete as a group.
Quoted 4 lines · Jeremy Linton
A unified multiple device mmap buffer interface for would probably be useful
as well. This is to get around the problems inherent in the current design that keep more than one device from using a mmap/reserve buffer allocated for another device.
Quoted 8 lines
What I would personally like to be able to do with the SG interface is have a
shared memory region where I can kick off a few dozen tagged commands to a device, get notification when they have completed and then send a similar command to a second device, all without involving the CPU in any copy operations, or too many page management operations.
In my dream world this would require some kind of tagging of commands as well
so I don't have to handle a bunch of async completions for commands in a group that need to complete as a group.
This is all starting to sound/feel a lot like the current proposed design for netchannels: http://lwn.net/Articles/206699/
If we did an AF_SCSI (I've seen it mooted before), then I suppose we could just use netchannels.
The other route is tweaking the current netchannels proposal to be a bit less networking-centric and use kiocbs (or something ...) to hold the data that's being transmitted.
From: Matthew Wilcox <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Wed, 8 Nov 2006 10:22:18 -0700
Quoted 17 lines · Jeremy Linton
Quoted 4 lines
A unified multiple device mmap buffer interface for would probably be useful
as well. This is to get around the problems inherent in the current design that keep more than one device from using a mmap/reserve buffer allocated for another device.Quoted 8 lines
What I would personally like to be able to do with the SG interface is have a
shared memory region where I can kick off a few dozen tagged commands to a device, get notification when they have completed and then send a similar command to a second device, all without involving the CPU in any copy operations, or too many page management operations.
In my dream world this would require some kind of tagging of commands as well
so I don't have to handle a bunch of async completions for commands in a group that need to complete as a group.This is all starting to
sound/feela lot like the current proposed design for netchannels: http://lwn.net/Articles/206699/
Quoted 6 lines
If we did an
AF_SCSI(I've seen it mooted before), then I suppose we could just use netchannels.The other route is tweaking the current netchannels proposal to be a bit less networking-centric and use kiocbs (or something ...) to hold the data that's being transmitted.
netchannels works if we can make it less networking-centric, though ring buffer would be preferable for the target code since netchannels needs to allocate and free skbuffs and do copies again and again.
From: Jeremy Linton <[email protected]>
Subject: Re: RFC: SCSI Generic version 4 interface
Date: Wed, 8 Nov 2006 10:48:01 -0600
Quoted 11 lines · Douglas Gilbert
Quoted 4 lines
I was asked to put together a proposal in May this year for a new SCSI Generic interface structure. This
Feel free to make suggestions.
OK... A unified multiple device mmap buffer interface for would probably be useful
Quoted 11 lines
as well. This is to get around the problems inherent in the current design that keep more than one device from using a mmap/reserve buffer allocated for another device.
What I would personally like to be able to do with the SG interface is have a
shared memory region where I can kick off a few dozen tagged commands to a device, get notification when they have completed and then send a similar command to a second device, all without involving the CPU in any copy operations, or too many page management operations.
In my dream world this would require some kind of tagging of commands as well
so I don't have to handle a bunch of async completions for commands in a group that need to complete as a group.
Sounds like it's quite similar to what the target code (tgt) is doing with its own ring buffer interface.
As discussed months ago, tgt has a lot in common with sg. We would like to have the new sg flexible enough to handle various messages that tgt needs. This will also require lots of changes to the block layer functions to map user-space pages.