FC Transport : Async Events via netlink interface
Subsystems (3)
RFC 1/1 FC Transport : Async Events via netlink interface
[netdev folks copied to review the use of the netlink interface]
This patch adds HBAAPI async event support to the FC transport.
Events are pushed to userspace via Netlink. This is per the previous RFC comments given in :
http://marc.theaimsgroup.com/?l=linux-scsi&m=114062896729418&w=2
This patch contains the following changes:
- Add Netlink support to the FC transport
- Creates a new file "include/scsi/scsi_netlink_fc.h", which contains the user-space visible portion of the FC transport netlink messaging
- Allow user apps to register to receive async events
- Add the fc_host_event_post() interface to post async events
- A couple of misc fixes:
- From the prior event post: small dev_loss_tmo mods, with the main fix to validate the module parameter on load.
- Fix fc_user_scan() so it safely walks the rport list
Using netlink worked out very well, and solves the multiple receiver issue. Also looks very extensible for additional HBAAPI function support.
-- james s
PS: Comments on Kconfig change appreciated. I don't have much experience on
changing the kernel config and build process.
drivers/scsi/Kconfig +1 -1
@@ -221,7 +221,7 @@ config SCSI_SPI_ATTRS config SCSI_FC_ATTRS tristate "FiberChannel Transport Attributes" - depends on SCSI + depends on SCSI && NET && NETFILTER && NETFILTER_NETLINK help If you wish to export transport-specific information about each attached FiberChannel device to sysfs, say Y.
drivers/scsi/scsi_transport_fc.c +399 -13
478 lines not on this page · raw message
include/linux/netlink.h +2 -0
@@ -21,6 +21,8 @@#define NETLINK_DNRTMSG 14/* DECnet routing messages */ #define NETLINK_KOBJECT_UEVENT 15/* Kernel messages to userspace */ #define NETLINK_GENERIC 16 +/* leave room for NETLINK_DM (DM Events) and NETLINK_TGT (SCSI Target) */ +#define NETLINK_FCTRANSPORT 19/* SCSI FC Transport */ #define MAX_LINKS 32
include/scsi/scsi_netlink_fc.h +57 -0
57 lines not on this page · raw message
include/scsi/scsi_transport_fc.h +35 -7
60 lines not on this page · raw message
Looks good, comments below.
Quoted 22 lines · James Smart
drivers/scsi/scsi_transport_fc.c +22
-0+static void +fc_send_event (struct fc_nl_user *nluser,struct fc_nl_event *event) +{ +struct sk_buff *skb; +struct nlmsghdr *nlh; +struct fc_nl_event *evt; +const char *name, *fn; +u32 len =NLMSG_SPACE (sizeof (*event)); +int err; + + skb =alloc_skb (len,GFP_KERNEL ); +if (!skb) { + err = -ENOBUFS ; + fn ="alloc_skb" ; + goto send_fail; + } + + nlh =nlmsg_put (skb, nluser->pid,0 ,FC_TRANSPORT_MSG , + len -sizeof (*nlh),0 ); +if (!nlh) { + err = -ENOBUFS ; + fn ="nlmsg_put" ;
drivers/scsi/scsi_transport_fc.c MODULE_PARM_DESC(dev_loss_tmo, · line 638
Quoted 10 lines
drivers/scsi/scsi_transport_fc.c +10
-0+ goto send_fail; + } + evt =NLMSG_DATA (nlh); +memcpy (evt, event,sizeof (*event)); + + err =nlmsg_unicast (fc_nl_sock, skb, nluser->pid); +if (err <0 ) { + fn ="nlmsg_unicast" ; + goto send_fail; + }
Is there some reason that you are not using nlmsg_multicast. The caller of this function is somewhat simulating the function of multicast.
drivers/scsi/scsi_transport_fc.c MODULE_PARM_DESC(dev_loss_tmo, · line 649
Quoted 11 lines
drivers/scsi/scsi_transport_fc.c +11
-0+ +return ; + +send_fail: + name =get_fc_host_event_code_name (event->event_code); +printk (KERN_WARNING +"%s: Dropped Event to PID %d : %s data 0x%08x : %s : err %d\n" , + __FUNCTION__, nluser->pid, (name) ? name :"<unknown>" , + event->event_data, fn, err); +return ; +}
In the send_fail case it looks like you leak skbs. Do you need to add a call to nlmsg_free or kfree_skb?
-andmike
--
Michael Anderson
[email protected]
Quoted 2 lines · Mike Anderson
Is there some reason that you are not using
nlmsg_multicast. The caller of this function is somewhat simulating the function of multicast.
Only that I haven't looked into using groups yet. It certainly makes sense.
Quoted 2 lines
In the
send_failcase it looks like you leak skbs. Do you need to add a call tonlmsg_freeorkfree_skb?
Yep.
I'll include these comments in the revised post. I'll wait a little longer for any further comments.
-- james
[RFC] Netlink and user-space buffer pointers
In reply to Mike AndersonFolks,
To take netlink to where we want to use it within the SCSI subsystem (as the mechanism of choice to replace ioctls), we're going to need to pass user-space buffer pointers.
What is the best, portable manner to pass a pointer between user and kernel space within a netlink message ? The example I've seen is in the iscsi target code - and it's passed between user-kernel space as a u64, then typecast to a void *, and later within the bio_map_xxx functions, as an unsigned long. I assume we are going to continue with this method ?
-- james s
Re: [RFC] Netlink and user-space buffer pointers
Quoted 9 lines · James Smart
To take netlink to where we want to use it within the SCSI subsystem (as the mechanism of choice to replace ioctls), we're going to need to pass user-space buffer pointers.
What is the best, portable manner to pass a pointer between user and kernel space within a netlink message ? The example I've seen is in the iscsi target code - and it's passed between user-kernel space as a u64, then typecast to a void *, and later within the
bio_map_xxxfunctions, as an unsigned long. I assume we are going to continue with this method ?
This might be problematic, since there is a shared receive-queue in the kernel netlink message might get processed in the context of a different process. I didn't find any spots where ISCSI passes pointers over netlink, can you point me to it?
Besides that, netlink protocols should use fixed size architecture independant types, so u64 would be the best choice for pointers.
Re: [RFC] Netlink and user-space buffer pointers
Quoted 4 lines · Patrick McHardy
This might be problematic, since there is a shared receive-queue in the kernel netlink message might get processed in the context of a different process. I didn't find any spots where ISCSI passes pointers over netlink, can you point me to it?
Please explain... Would the pid be set erroneously as well ? Ignoring the kernel-user space pointer issue, we're going to have a tight pid + request_id relationship being maintained across multiple messages. We'll also be depending on the pid events for clean up if an app dies. So I hope pid is consistent.
-- james s
Re: [RFC] Netlink and user-space buffer pointers
Quoted 10 lines · James Smart
Quoted 4 lines · Patrick McHardy
This might be problematic, since there is a shared receive-queue in the kernel netlink message might get processed in the context of a different process. I didn't find any spots where ISCSI passes pointers over netlink, can you point me to it?
Quoted 5 lines
Please explain... Would the pid be set erroneously as well ? Ignoring the kernel-user space pointer issue, we're going to have a tight pid +
request_idrelationship being maintained across multiple messages. We'll also be depending on the pid events for clean up if an app dies. So I hope pid is consistent.
The PID contained in the netlink message itself is correct, current->pid might not be.
Re: [RFC] Netlink and user-space buffer pointers
In reply to James SmartQuoted 5 lines · On Wed, 19 Apr 2006 08:57:25 -0400 James Smart <[email protected]> wrote:
Folks,
To take netlink to where we want to use it within the SCSI subsystem (as the mechanism of choice to replace ioctls), we're going to need to pass user-space buffer pointers.
This changes the design of netlink. It is desired that netlink can be done remotely over the network as well as queueing. The current design is message based, not RPC based. By including a user-space pointer, you are making the message dependent on the context as it is process.
Please rethink your design.
Quoted 11 lines
What is the best, portable manner to pass a pointer between user and kernel space within a netlink message ? The example I've seen is in the iscsi target code - and it's passed between user-kernel space as a u64, then typecast to a void *, and later within the
bio_map_xxxfunctions, as an unsigned long. I assume we are going to continue with this method ?-- james s
-
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] Netlink and user-space buffer pointers
Quoted 9 lines · Stephen Hemminger
Quoted 5 lines · On Wed, 19 Apr 2006 08:57:25 -0400 James Smart <[email protected]> wrote:
Folks,
To take netlink to where we want to use it within the SCSI subsystem (as the mechanism of choice to replace ioctls), we're going to need to pass user-space buffer pointers.
Quoted 7 lines
This changes the design of netlink. It is desired that netlink can be done remotely over the network as well as queueing. The current design is message based, not RPC based. By including a user-space pointer, you are making the message dependent on the context as it is process.
Please rethink your design.
I assume that the message receiver has some way to determine where the message originated (via the sk_buff), and thus could reject it if it didn't meet the right criteria. True ? You just have to be cognizant that it is usable from a remote entity - which is a very good thing.
Re: [RFC] Netlink and user-space buffer pointers
In reply to James SmartQuoted 6 lines · James Smart
Folks,
To take netlink to where we want to use it within the SCSI subsystem (as the mechanism of choice to replace ioctls), we're going to need to pass user-space buffer pointers.
Quoted 6 lines
What is the best, portable manner to pass a pointer between user and kernel space within a netlink message ? The example I've seen is in the iscsi target code - and it's passed between user-kernel space as a u64, then typecast to a void *, and later within the
bio_map_xxxfunctions, as an unsigned long. I assume we are going to continue with this method ?
I do not know if it is needed. For the target code, we use the bio_map_xxx to avoid having to copy the command data which is needed for decent performance. We have also been trying to figure out ways of getting out of using netlink to send the command info (cdb, tag info, etc) around, because in some of Tomo's tests using mmaped packet sockets he was able to imporove performance by removing that copy from the kernel to userspace. We had problems with that though and other nice interfaces like relayfs only allowed us to pass data from the kernel to userspace so we still need another interface to pass things from userspace to the kernel. Still working on this though. If someone knows a interface please let us know.
For the tasks you want to do for the fc class is performance critical? If not, you could do what the iscsi class (for the netdev people this is drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:
/*
* xmitbuf is a buffer that is large enough for the iscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size)
*/
memset(xmitbuf, 0, sizeof(*ev) + hdr_size + data_size);
xmitlen = sizeof(*ev);
ev = xmitbuf;
ev->type = ISCSI_UEVENT_SEND_PDU;
ev->transport_handle = transport_handle;
ev->u.send_pdu.sid = sid;
ev->u.send_pdu.cid = cid;
ev->u.send_pdu.hdr_size = hdr_size;
ev->u.send_pdu.data_size = data_size;then later we do sendmsg()to send down the xmitbuf to the kernel iscsi driver. I think there may be issues with packing structs or 32 bit userspace and 64 bit kernels and other fun things like this so the iscsi pdu and iscsi event have to be defined correctly and I guess we are back to some of the problems with ioctls :(
Re: [RFC] Netlink and user-space buffer pointers
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 8 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:/* * xmitbuf is a buffer that is large enough for the iscsi_event, * iscsi pdu (hdr_size) and iscsi pdu data (data_size) */
Well, the real difference is that the payload of the "message" is actually the payload of the SCSI command or ELS/CT Request. Thus, the payload may range in size from a few hundred bytes to several kbytes (> 1 page) to Mbyte's in size. Rather than buffer all of this, and push it over the socket, thus the extra copies - it would best to have the LLDD simply DMA the payload like on a typical SCSI command. Additionally, there will be response data that can be several kbytes in length.
Quoted 4 lines
... I think there may be issues with packing structs or 32 bit userspace and 64 bit kernels and other fun things like this so the iscsi pdu and iscsi event have to be defined correctly and I guess we are back to some of the problems with ioctls :(
Agreed. In this use of netlink, there's not a lot of wins for netlink over ioctls. It all comes down to 2 things: a) proper portable message definition; and b) what do you do with that non-portable user space buffer pointer ?
-- james s
Re: [RFC] Netlink and user-space buffer pointers
Quoted 10 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 4 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:
Quoted 7 lines
Quoted 4 lines
/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */Well, the real difference is that the payload of the "message" is actually the payload of the SCSI command or ELS/CT Request. Thus, the payload may
I am not sure I follow. For iscsi, everything after the iscsi_event struct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.
Quoted 7 lines
range in size from a few hundred bytes to several kbytes (> 1 page) to Mbyte's in size. Rather than buffer all of this, and push it over the socket,
thus the extra copies - it would best to have the LLDD simply DMA the payload like on a typical SCSI command. Additionally, there will be response data that can be several kbytes in length.
Once you have got the buffer to the class, the class can create a scatterlist to DMA from for the LLD. I thought. iscsi does not do this just because it is software right now. For qla4xxx we do not need something like what you are talking about (see below for what I was thinking about for the initiators). If you are saying the extra step of the copy is plain dumb, I agree, but this happens (you have to suffer some copy and cannot do dio) for sg io as well in some cases. I think for the sg driver the copy_*_user is the default.
Instead of netlink for scsi commands and transport requests....
For scsi commands could we just use sg io, or is there something special about the command you want to send? If you can use sg io for scsi commands, maybe for transport level requests (in my example iscsi pdu) we could modify something like sg/bsg/block layer scsi_ioctl.c to send down transport requests to the classes and encapsulate them in some new struct transport_requests or use the existing struct request but do that thing people keep taling about using the request/request_queue for message passing.
Re: [RFC] Netlink and user-space buffer pointers
Quoted 28 lines · Mike Christie
Quoted 8 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 4 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:Quoted 6 lines
Quoted 4 lines
/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */Well, the real difference is that the payload of the "message" is actually the payload of the SCSI command or ELS/CT Request. Thus, the payload may
I am not sure I follow. For iscsi, everything after the
iscsi_eventstruct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.Quoted 7 lines
range in size from a few hundred bytes to several kbytes (> 1 page) to Mbyte's in size. Rather than buffer all of this, and push it over the socket,
thus the extra copies - it would best to have the LLDD simply DMA the payload like on a typical SCSI command. Additionally, there will be response data that can be several kbytes in length.
Quoted 3 lines
Once you have got the buffer to the class, the class can create a scatterlist to DMA from for the LLD. I thought. iscsi does not do this just because it is software right now. For qla4xxx we do not need
That should be, we do need.
Re: [RFC] Netlink and user-space buffer pointers
In reply to Mike ChristieQuoted 39 lines · Mike Christie
Quoted 8 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 4 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:Quoted 6 lines
Quoted 4 lines
/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */Well, the real difference is that the payload of the "message" is actually the payload of the SCSI command or ELS/CT Request. Thus, the payload may
I am not sure I follow. For iscsi, everything after the
iscsi_eventstruct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.Quoted 7 lines
range in size from a few hundred bytes to several kbytes (> 1 page) to Mbyte's in size. Rather than buffer all of this, and push it over the socket,
thus the extra copies - it would best to have the LLDD simply DMA the payload like on a typical SCSI command. Additionally, there will be response data that can be several kbytes in length.Once you have got the buffer to the class, the class can create a scatterlist to DMA from for the LLD. I thought. iscsi does not do this just because it is software right now. For qla4xxx we do not need something like what you are talking about (see below for what I was thinking about for the initiators). If you are saying the extra step of the copy is plain dumb, I agree, but this happens (you have to suffer some copy and cannot do dio) for sg io as well in some cases. I think for the sg driver the copy_*_user is the default.
Instead of netlink for scsi commands and transport requests....
Quoted 8 lines
For scsi commands could we just use sg io, or is there something special about the command you want to send? If you can use sg io for scsi commands, maybe for transport level requests (in my example iscsi pdu) we could modify something like sg/bsg/block layer
scsi_ioctl.cto send down transport requests to the classes and encapsulate them in some new structtransport_requestsor use the existing struct request but do that thing people keep taling about using the request/request_queue for message passing.
And just to be complete, the problem with this is that it is tied to the request queue and so you cannot just send a transport level request unless it is tied to the device. But for the target stuff we added a request queue to the host so we could inject requests (the idea was to send down those magic message requests) at a higher level. To be able to use that for sg io though it would require some more code and magic as you know.
Re: [RFC] Netlink and user-space buffer pointers
In reply to Mike ChristieNote: We've transitioned off topic. If what this means is "there isn't a good way except by ioctls (which still isn't easily portable) or system calls", then that's ok. Then at least we know the limits and can look at other implementation alternatives.
Quoted 13 lines · Mike Christie
Quoted 12 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 8 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */
Quoted 6 lines
Quoted 2 lines
Well, the real difference is that the payload of the "message" is actually the payload of the SCSI command or ELS/CT Request. Thus, the payload may
I am not sure I follow. For iscsi, everything after the
iscsi_eventstruct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.
True... For a large read/write - it will eventually total what the i/o request size was, and you did have to push it through the socekt. What this discussion really comes down to is the difference between initiator offload and what a target does.
The initiator offloads the "full" i/o from the users - e.g. send command, get response. In the initiator case, the user isn't aware of each and every IU that makes up the i/o. As it's on an i/o basis, the LLDD doing the offload needs the full buffer sitting and ready. DMA is preferred so the buffer doesn't have to be consuming socket/kernel/driver buffers while it's pending - plus speed.
In the target case, the target controls each IU and it's size, thus it only has to have access to as much buffer space as it wants to push the next IU. The i/o can be "paced" by the target. Unfortunately, this is an entirely different use model than users of a scsi initiator expect, and it won't map well into replacing things like our sg_io ioctls.
Quoted 10 lines
Instead of netlink for scsi commands and transport requests....
For scsi commands could we just use sg io, or is there something special about the command you want to send? If you can use sg io for scsi commands, maybe for transport level requests (in my example iscsi pdu) we could modify something like sg/bsg/block layer
scsi_ioctl.cto send down transport requests to the classes and encapsulate them in some new structtransport_requestsor use the existing struct request but do that thing people keep taling about using the request/request_queue for message passing.
Well - there's 2 parts to this answer:
First : IOCTL's are considered dangerous/bad practice and therefore it would
be nice to find a replacement mechanism that eliminates them. If that mechanism has some of the cool features that netlink does, even better. Using sg io, in the manner you indicate, wouldn't remove the ioctl use.
Note: I have OEMs/users that are very confused about the community's statement about ioctls. They've heard they are bad, should never be allowed, will no be longer supported, but yet they are at the heart of DM and sg io and other subsystems. Other than a "grandfathered" explanation, they don't understand why the rules bend for one piece of code but not for another. To them, all the features are just as critical regardless of whose providing them.
Second: transport level i/o could be done like you suggest, and we've
prototyped some of this as well. However, there's something very wrong about putting "block device" wrappers and settings around something that is not a block device. In general, it's a heck of a lot of overhead and still doesn't solve the real issue - how to portably pass that user buffer in to/out of the kernel.
-- james s
Re: [RFC] Netlink and user-space buffer pointers
Quoted 42 lines · James Smart
Note: We've transitioned off topic. If what this means is "there isn't a good
way except by ioctls (which still isn't easily portable) or system calls", then that's ok. Then at least we know the limits and can look at other implementation alternatives.Quoted 14 lines · Mike Christie
Quoted 13 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 9 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */Quoted 7 lines
Quoted 3 lines
Well, the real difference is that the payload of the "message" is actually
the payload of the SCSI command or ELS/CT Request. Thus, the payload mayI am not sure I follow. For iscsi, everything after the
iscsi_eventstruct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.True... For a large read/write - it will eventually total what the i/o request size was, and you did have to push it through the socekt. What this discussion really comes down to is the difference between initiator
offload and what a target does.The initiator offloads the "full" i/o from the users - e.g. send command, get response. In the initiator case, the user isn't aware of each and every IU that makes up the i/o. As it's on an i/o basis, the LLDD doing the offload needs the full buffer sitting and ready. DMA is preferred so the buffer doesn't have to be consuming socket/kernel/driver buffers while it's pending - plus speed.
Quoted 7 lines
In the target case, the target controls each IU and it's size, thus it only has to have access to as much buffer space as it wants to push the next
IU. The i/o can be "paced" by the target. Unfortunately, this is an entirely
different use model than users of a scsi initiator expect, and it won't map well into replacing things like oursg_ioioctls.
I am not talking about the target here. For the open-iscsi initiator that is in mainline that I referecnced in the example we send pdus from userpsace to the LLD. In the future, initaitors that offload some iscsi processing and will login from userspace or have userspace monitor the transport by doing iscsi pings, we need to be able to send these pdus. And the iscsi pdu cannot be broken up at the iscsi level (they can at the interconect level though). From the iscsi host level they have to go out like a scsi command would in that the LLD cannot decide to send out mutiple pdus for he pdu that userspace sends down.
I do agree with you that targets can break down a scsi command into multiple transport level packets as it sees fit.
Quoted 12 lines
Quoted 11 lines
Instead of netlink for scsi commands and transport requests....
For scsi commands could we just use sg io, or is there something special about the command you want to send? If you can use sg io for scsi commands, maybe for transport level requests (in my example iscsi pdu) we could modify something like sg/bsg/block layer
scsi_ioctl.cto send down transport requests to the classes and encapsulate them in some new structtransport_requestsor use the existing struct request but do that thing people keep taling about using the request/request_queue for message passing.
Quoted 4 lines
Well - there's 2 parts to this answer:
First : IOCTL's are considered dangerous/bad practice and therefore it would
Yeah, i am not trying to kill ioctls. I go where the community goes. What I am trying to dois just reuse the sg io mapping code so that we do not end up with sg, st, target, blk scsi_ioctl.c and bsg all doing similar things.
Quoted 15 lines
be nice to find a replacement mechanism that eliminates them. If that mechanism has some of the cool features that netlink does, even better. Using sg io, in the manner you indicate, wouldn't remove the ioctl use.
Note: I have OEMs/users that are very confused about the community's
statement
about ioctls. They've heard they are bad, should never be allowed,
will no
be longer supported, but yet they are at the heart of DM and sg io and
other
subsystems. Other than a "grandfathered" explanation, they don't
understand
why the rules bend for one piece of code but not for another. To them,
all
the features are just as critical regardless of whose providing them.
Quoted 6 lines
Second: transport level i/o could be done like you suggest, and we've
prototyped some of this as well. However, there's something very wrong about putting "block device" wrappers and settings around something that is not a block device. In general, it's a heck of a lot of overhead and still doesn't solve the real issue - how to portably pass that user
buffer
I am not talking about putting block device wrappers. This the magic part and the message passing comes in. A while back I made the requuest queue a class (only sent the patch to Jens and did not follow up). The original reason was for the io sched swap and some multipath stuff, but since with then the request queue would need a block device to be exposed to userspace through sysfs and would not need a block device to send messages throgh. You just need a way to commniutate between userspace and the kernel but it does not have to be through a block device. I think this path has other benefits in that you could do userspace level scanning as well since you do not need the block device and ULD like we do today.
Quoted 4 lines
in to/out of the kernel.
-- james s
Re: [RFC] Netlink and user-space buffer pointers
Quoted 60 lines · Mike Christie
Quoted 40 lines · James Smart
Note: We've transitioned off topic. If what this means is "there isn't a good
way except by ioctls (which still isn't easily portable) or system calls", then that's ok. Then at least we know the limits and can look at other implementation alternatives.Quoted 14 lines · Mike Christie
Quoted 13 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 9 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */Quoted 6 lines
Quoted 3 lines
Well, the real difference is that the payload of the "message" is actually
the payload of the SCSI command or ELS/CT Request. Thus, the payload mayI am not sure I follow. For iscsi, everything after the
iscsi_eventstruct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.True... For a large read/write - it will eventually total what the i/o request size was, and you did have to push it through the socekt. What this discussion really comes down to is the difference between initiator
offload and what a target does.The initiator offloads the "full" i/o from the users - e.g. send command, get response. In the initiator case, the user isn't aware of each and every IU that makes up the i/o. As it's on an i/o basis, the LLDD doing the offload needs the full buffer sitting and ready. DMA is preferred so the buffer doesn't have to be consuming socket/kernel/driver buffers while it's pending - plus speed.
Quoted 7 lines
In the target case, the target controls each IU and it's size, thus it only has to have access to as much buffer space as it wants to push the next
IU. The i/o can be "paced" by the target. Unfortunately, this is an entirely
different use model than users of a scsi initiator expect, and it won't map well into replacing things like oursg_ioioctls.I am not talking about the target here. For the open-iscsi initiator that is in mainline that I referecnced in the example we send pdus from userpsace to the LLD. In the future, initaitors that offload some iscsi processing and will login from userspace or have userspace monitor the transport by doing iscsi pings, we need to be able to send these pdus. And the iscsi pdu cannot be broken up at the iscsi level (they can at the interconect level though). From the iscsi host level they have to go out like a scsi command would in that the LLD cannot decide to send out mutiple pdus for he pdu that userspace sends down.
Quoted 3 lines
I do agree with you that targets can break down a scsi command into multiple transport level packets as it sees fit.
Oh yeah is
FC IU == iscsi tcp packet
or
FC IU == iscsi pdu
?
Re: [RFC] Netlink and user-space buffer pointers
In reply to Mike ChristieQuoted 8 lines · Mike Christie
Quoted 6 lines
Second: transport level i/o could be done like you suggest, and we've
prototyped some of this as well. However, there's something very wrong about putting "block device" wrappers and settings around something that is not a block device. In general, it's a heck of a lot of overhead and still doesn't solve the real issue - how to portably pass that user
buffer
Quoted 12 lines
I am not talking about putting block device wrappers. This the magic part and the message passing comes in. A while back I made the requuest queue a class (only sent the patch to Jens and did not follow up). The original reason was for the io sched swap and some multipath stuff, but since with then the request queue would need a block device to be exposed to userspace through sysfs and would not need a block device to send messages throgh. You just need a way to commniutate between userspace and the kernel but it does not have to be through a block device. I think this path has other benefits in that you could do userspace level scanning as well since you do not need the block device and ULD like we do today.
Again for a bad example look at the target code. The original idea was to use the request_queue for messages because it did not work so well for them to take target requests (we do not need the io scheduler for example) to send request from the target interrupt handler to the netink code. It was over kill becuase all we really want is a per cpu list, but we used the request queue becuase it had the queue limits that we needed. It was mean as temporary and I think tomo is working on replacing that with a per process list and softirq. But the point is that you can use the request queue to pass around requests and still use netlink for the interface between the kernel and userspace.
Re: [RFC] Netlink and user-space buffer pointers
In reply to James Smartsnipped lkml and netdev. I do not think they care do they.
Quoted 6 lines · James Smart
Note: We've transitioned off topic. If what this means is "there isn't a good
way except by ioctls (which still isn't easily portable) or system calls", then that's ok. Then at least we know the limits and can look at other implementation alternatives.
So we have
1. sysfs: prbably bad becuase to pass down every attr for the command we need a seperate file. This does not seem like it meets the binary sysfs file requirement.
2. configfs: again bad becuase we have to pass down eveyr attr for the command in a speperate file.
3. ioctl: Christoph has not allowed scsi drivers to add ioctls so I doubt he would let some other scsi code let it.
4. syscall:
5. netlink: netdev guys do not want use to pass pointers. So we either have a worst case double copy (copy from userspace to network and network to buffer that is within the LLDs queue limits (do like sg where it as a preallocated buffer)) or if we used the bio code like how the ll_rw_blk.c does just fail if the command is to large. In some cases the io may just be too large and have to fail though.
5A. modify netlink to copy data to a buffer that is within some sort of
limits we can DMA from.
5B. debate with netdev guys about passing poitners, in which case we
could have some of the problem 5 has in that when we map data we do not have much control over the pages so we may have to drop down to a copy to some preallocate buffer or fail model.
6. char device
7. what else...
Re: [RFC] Netlink and user-space buffer pointers
Quoted 11 lines · Mike Christie
snipped lkml and netdev. I do not think they care do they.
Quoted 6 lines · James Smart
Note: We've transitioned off topic. If what this means is "there isn't a good
way except by ioctls (which still isn't easily portable) or system calls", then that's ok. Then at least we know the limits and can look at other implementation alternatives.So we have
Ok for those that emailed me off list. All these interfaces have a similar problems in that the kernel cannot get enough pages to make a scatterlist that is within the HBA's limits. block/scsi_ioctl.c handles this by failing the request, sg.c handles this by preallocating buffers, the target code handles this by calling the target transfer function multiple times as James described, and using netlink as described for a target or initiator and for sg io or target commands or transport commands has similar issues if it just cannot get enough pages to make a proper scatterlist, but currently it has the extra limitation in that we are not supposed to pass pointers.
Sorry for the confusion.
Quoted 34 lines
1. sysfs: prbably bad becuase to pass down every attr for the command we need a seperate file. This does not seem like it meets the binary sysfs file requirement.
2. configfs: again bad becuase we have to pass down eveyr attr for the command in a speperate file.
3. ioctl: Christoph has not allowed scsi drivers to add ioctls so I doubt he would let some other scsi code let it.
4. syscall:
5. netlink: netdev guys do not want use to pass pointers. So we either have a worst case double copy (copy from userspace to network and network to buffer that is within the LLDs queue limits (do like sg where it as a preallocated buffer)) or if we used the bio code like how the
ll_rw_blk.cdoes just fail if the command is to large. In some cases the io may just be too large and have to fail though.5A. modify netlink to copy data to a buffer that is within some sort of
limits we can DMA from.
5B. debate with netdev guys about passing poitners, in which case we
could have some of the problem 5 has in that when we map data we do not have much control over the pages so we may have to drop down to a copy to some preallocate buffer or fail model.6. char device
7. what else...
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] Netlink and user-space buffer pointers
In reply to James SmartQuoted 5 lines · James Smart
Note: We've transitioned off topic. If what this means is "there isn't a good
way except by ioctls (which still isn't easily portable) or system calls", then that's ok. Then at least we know the limits and can look at other implementation alternatives.
this topic has been brought-up many times in the past, most recently:
http://thread.gmane.org/gmane.linux.drivers.openib/19525/focus=19525 http://thread.gmane.org/gmane.linux.kernel/387375/focus=387455
where is was suggested to pathscale folks to use some blend of sysfs, netlink sockets and debugfs:
http://kerneltrap.org/node/4394
Quoted 14 lines
Quoted 11 lines
Quoted 1 line
Mike Christie wrote:
Instead of netlink for scsi commands and transport requests....
For scsi commands could we just use sg io, or is there something special about the command you want to send? If you can use sg io for scsi commands, maybe for transport level requests (in my example iscsi pdu) we could modify something like sg/bsg/block layer
scsi_ioctl.cto send down transport requests to the classes and encapsulate them in some new structtransport_requestsor use the existing struct request but do that thing people keep taling about using the request/request_queue for message passing.Well - there's 2 parts to this answer:
Quoted 13 lines
First : IOCTL's are considered dangerous/bad practice and therefore it would
be nice to find a replacement mechanism that eliminates them. If that mechanism has some of the cool features that netlink does, even better. Using sg io, in the manner you indicate, wouldn't remove the ioctl use.
Note: I have OEMs/users that are very confused about the community's statement
about ioctls. They've heard they are bad, should never be allowed, will no be longer supported, but yet they are at the heart of DM and sg io and other
subsystems. Other than a "grandfathered" explanation, they don't understand
why the rules bend for one piece of code but not for another. To them, all the features are just as critical regardless of whose providing them.
I believe it to be the same for most hardware-vendor's customers...
Quoted 4 lines
Second: transport level i/o could be done like you suggest, and we've
prototyped some of this as well. However, there's something very wrong about putting "block device" wrappers and settings around something that is not a block device.
Eeww... no wrappers. Your netlink prototypes certainly get FC- transport further along, but would also be nice if there could be some subsystem consensus on *the* interface.
I honestly don't know which interface is *best*, but from a HBA vendors perspective managing per-request locally allocated memory is undesirable.
Thanks,
av
Re: [RFC] Netlink and user-space buffer pointers
In reply to Mike ChristieQuoted 36 lines · Mike Christie
Quoted 11 lines · James Smart
Quoted 1 line · Mike Christie
For the tasks you want to do for the fc class is performance critical?
No, it should not be.
Quoted 4 lines
If not, you could do what the iscsi class (for the netdev people this is
drivers/scsi/scsi_transport_iscsi.c) does and just suffer a couple copies. For iscsi we do this in userspace to send down a login pdu:Quoted 7 lines
Quoted 4 lines
/*
* xmitbuf is a buffer that is large enough for theiscsi_event,
* iscsi pdu (hdr_size) and iscsi pdu data (data_size) */Well, the real difference is that the payload of the "message" is actually the payload of the SCSI command or ELS/CT Request. Thus, the payload may
I am not sure I follow. For iscsi, everything after the
iscsi_eventstruct can be the iscsi request that is to be transmitted. The payload will not normally be Mbytes but it is not a couple if bytes.Quoted 7 lines
range in size from a few hundred bytes to several kbytes (> 1 page) to Mbyte's in size. Rather than buffer all of this, and push it over the socket,
thus the extra copies - it would best to have the LLDD simply DMA the payload like on a typical SCSI command. Additionally, there will be response data that can be several kbytes in length.
Quoted 8 lines
Once you have got the buffer to the class, the class can create a scatterlist to DMA from for the LLD. I thought. iscsi does not do this just because it is software right now. For qla4xxx we do not need something like what you are talking about (see below for what I was thinking about for the initiators). If you are saying the extra step of the copy is plain dumb, I agree, but this happens (you have to suffer some copy and cannot do dio) for sg io as well in some cases. I think for the sg driver the copy_*_user is the default.
Mike,
Indirect IO is the default in the sg driver because:
- it has always been thus
- the sg driver is less constrained (e.g. max number of scatg elements is a bigger issue with dio)
- the only alignment to worry about is byte alignment (some folks would like bit alignment but you can't please everybody)
- there is no need for the sg driver to pin user pages in memory (as there is with direct IO and mmaped-IO)
Quoted 1 line
Instead of netlink for scsi commands and transport requests....
With a netlink based pass through one might:
- improve on the SG_IO ioctl and add things like tags that are currently missing
- introduce a proper SCSI task management function pass through (no request queue please)
- make other pass throughs for SAS: SMP and STP
- have an alternative to sysfs for various control functions in a HBA (e.g. in SAS: link and hard reset) and fetching performance data from a HBA
Apart from how to get data efficiently between the HBA and the user space, another major issue is the flexibility of the bind() in s_netlink (storage netlink??).
Quoted 8 lines
For scsi commands could we just use sg io, or is there something special about the command you want to send? If you can use sg io for scsi commands, maybe for transport level requests (in my example iscsi pdu) we could modify something like sg/bsg/block layer
scsi_ioctl.cto send down transport requests to the classes and encapsulate them in some new structtransport_requestsor use the existing struct request but do that thing people keep taling about using the request/request_queue for message passing.
Some SG_IO ioctl users want up to 32 MB in one transaction and others want their data fast. Many pass through users view the kernel as an impediment (not so much as "the way" as "in the way").
Doug Gilbert
Quoted 7 lines · James Smart
PS: Comments on
Kconfigchange appreciated. I don't have much experience on
changing the kernel config and build process.diff -upNr a/drivers/scsi/Kconfig b/drivers/scsi/Kconfig
drivers/scsi/Kconfig config SCSI_SPI_ATTRS · line 227
Quoted 8 lines
drivers/scsi/Kconfig +1
-1@@ -221,7 +221,7 @@ config SCSI_SPI_ATTRS config SCSI_FC_ATTRS tristate "FiberChannel Transport Attributes" - depends on SCSI + depends on SCSI && NET && NETFILTER && NETFILTER_NETLINK help If you wish to export transport-specific information about each attached FiberChannel device to sysfs, say Y.
I would use a select here rather than a depends. So enabling a driver that uses FC_ATTRS will force netlink to be added. However, I think you have the wrong symbols here. NETFILTER_NETLINK is the netlink interface for netfilter -- entirely unrelated. It looks like you only need to enable NET in order to force netlink to be built. So something like this would be enough:
config SCSI_FC_ATTRS
tristate "FiberChannel Transport Attributes" depends on SCSI
+ select NET
help
If you wish to export transport-specific information about each attached FiberChannel device to sysfs, say Y.
drivers/scsi/Kconfig
Quoted 3 lines
drivers/scsi/Kconfig +3
-0+#define get_list_head_entry(pos, head, member) \ + pos = list_entry((head)->next, typeof(*pos), member) +
This one sets alarm bells ringing ...
Quoted 4 lines
static void
__exitfc_transport_exit(void)
{drivers/scsi/Kconfig +2
-0+ struct fc_nl_user *nluser; +
drivers/scsi/Kconfig
Quoted 10 lines
drivers/scsi/Kconfig +7
-0+ sock_release(fc_nl_sock->sk_socket); + netlink_unregister_notifier(&fc_netlink_notifier); + while (!list_empty(&fc_nl_user_list)) { + get_list_head_entry(nluser, &fc_nl_user_list, ulist); + list_del(&nluser->ulist); + kfree(nluser); + } transport_class_unregister(&fc_transport_class); transport_class_unregister(&fc_rport_class); transport_class_unregister(&fc_host_class);
I would write this as:
struct fc_nl_user *nluser, *tmp;
list_for_each_entry_safe(nluser, tmp, &fc_nl_user_list, ulist) {
kfree(nluser);
}drivers/scsi/Kconfig config SCSI_SPI_ATTRS · line 227
Quoted 10 lines · Matthew Wilcox
enable NET in order to force netlink to be built. So something like this would be enough:
config
SCSI_FC_ATTRSdrivers/scsi/Kconfig +1
-0tristate "FiberChannel Transport Attributes" depends on SCSI + select NET help If you wish to export transport-specific information about each attached FiberChannel device to sysfs, say Y.
Thanks....
Quoted 5 lines
drivers/scsi/scsi_transport_fc.c line 46
Quoted 3 lines
drivers/scsi/scsi_transport_fc.c +3
-0+#define get_list_head_entry (pos, head, member) \ + pos = list_entry((head)->next, typeof(*pos), member) +This one sets alarm bells ringing ...
Please explain why. I've always wondered why the list macros never let you look, without dequeuing, the head of the list. It will let you look - as long as you use the functions that make it think you're going to walk the list.
Quoted 6 lines
I would write this as:
struct fc_nl_user *nluser, *tmp; list_for_each_entry_safe(nluser, tmp, &fc_nl_user_list, ulist) { kfree(nluser); }
ok.. Subtle though, as you have to know you are consuming the entire list.
-- james