Proposal for handling async event requests
One of the pieces currently missing from the Linux driver is any attempt to send async event requests. Part of the issue is that the driver isn't really in a position to do anything with the asynchronous event results.
One previously proposed solution is to punt the entire thing to userspace. Have a daemon open /dev/nvme{0,1...n} and submit admin commands through the ioctl mechanism. This should work, but if/when there are async events that the driver does need to react to, the daemon then has to tell the driver what happened.
My proposal is that the driver issues async event requests. It handles any that it understands how to handle and any remaining events can be picked up by userspace calling read() on the /dev/nvmeN character devices.
We need to determine a format for the records returned by read(). My proposal is that it looks suspiciously similar to a completion queue entry. That is, a 16-byte record with Dword 0, Dword 1 and the high 15 bits of Dword 3 copied from the CQE. Dword 2 and the bottom 17 bits of Dword 3 would be available for our own purposes to communicate with the daemon.
Any comments on this proposal before we start implementation? One thing I'm particularly interested in is thoughts on how to handle multiple openers of /dev/nvmeN. Option 1 is to permit only a single open() of the device. That means that once the daemon is up and running, nobody else can use this path to issue ioctls, which is probably not great. Option 2 is to allow any number of openers, and if they step on each others toes when some of them get some completion events and others get other completion events ... oh well, too bad. Option 3 is to duplicate all completion events to all openers. Option 4 is to play games with the open flags; maybe O_RDWR gets to read completion entries while O_WRONLY only gets to send ioctls.
Other options?
I think handling should be localized and associated through an existing, running NVMe driver, i.e. not a separate daemon.
1) User space (or supervisor) issues ioctl to "open" and enable AEN feature to a loaded, running driver in anticipation of receiving async events. Ioctl parameters contain pointer to callback routine and buffer space to receive AEN's.
2) When driver receives ioctl it saves incoming parameters, sets up to receive AEN's (otherwise it's "closed"/disabled). The driver can manage the AEN's within, directly or could launch an AEN handler worker thread that it owns or various other ways.
3) When AEN is received, driver or AEN manager calls back through pointer, deposits AEN info.
4) Provide a corresponding "close" ioctl to disable the AEN chatter when not needed.
5) ??? provide unique tags/id's in ioctl parameters to allow multiple openers. Unique tag generator should be located at a central point ... like the driver? (another ioctl to get a unique tag?)
Regards,
Neal
-----Original Message-----
From: Linux-nvme [mailto:[email protected]] On Behalf Of Matthew Wilcox
Sent: Friday, May 31, 2013 9:36 AM
To: linux-nvme at lists.infradead.org
Subject: Proposal for handling async event requests
One of the pieces currently missing from the Linux driver is any attempt to send async event requests. Part of the issue is that the driver isn't really in a position to do anything with the asynchronous event results.
One previously proposed solution is to punt the entire thing to userspace. Have a daemon open /dev/nvme{0,1...n} and submit admin commands through the ioctl mechanism. This should work, but if/when there are async events that the driver does need to react to, the daemon then has to tell the driver what happened.
My proposal is that the driver issues async event requests. It handles any that it understands how to handle and any remaining events can be picked up by userspace calling read() on the /dev/nvmeN character devices.
We need to determine a format for the records returned by read(). My proposal is that it looks suspiciously similar to a completion queue entry. That is, a 16-byte record with Dword 0, Dword 1 and the high 15 bits of Dword 3 copied from the CQE. Dword 2 and the bottom 17 bits of Dword 3 would be available for our own purposes to communicate with the daemon.
Any comments on this proposal before we start implementation? One thing I'm particularly interested in is thoughts on how to handle multiple openers of /dev/nvmeN. Option 1 is to permit only a single open() of the device. That means that once the daemon is up and running, nobody else can use this path to issue ioctls, which is probably not great. Option 2 is to allow any number of openers, and if they step on each others toes when some of them get some completion events and others get other completion events ... oh well, too bad. Option 3 is to duplicate all completion events to all openers. Option 4 is to play games with the open flags; maybe O_RDWR gets to read completion entries while O_WRONLY only gets to send ioctls.
Other options?
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://merlin.infradead.org/mailman/listinfo/linux-nvme
Quoted 1 line · Neal Galbo (ngalbo)
I think handling should be localized and associated through an existing, running NVMe driver, i.e. not a separate daemon.
OK, so we're fundamentally in agreement, it's just a matter of negotiating the API? :-)
Quoted 5 lines
1) User space (or supervisor) issues ioctl to "open" and enable AEN feature to a loaded, running driver in anticipation of receiving async events. Ioctl parameters contain pointer to callback routine and buffer space to receive AEN's.
2) When driver receives ioctl it saves incoming parameters, sets up to receive AEN's (otherwise it's "closed"/disabled). The driver can manage the AEN's within, directly or could launch an AEN handler worker thread that it owns or various other ways.
3) When AEN is received, driver or AEN manager calls back through pointer, deposits AEN info.
4) Provide a corresponding "close" ioctl to disable the AEN chatter when not needed.
5) ??? provide unique tags/id's in ioctl parameters to allow multiple openers. Unique tag generator should be located at a central point ... like the driver? (another ioctl to get a unique tag?)
Yes, this approach definitely works. Essentially, we're putting a completion queue in userspace. How do you envisage notifying userspace that it has new entries? The advantage of read() on the character device is that it can block. We can also use the poll() mechanism to wake up tasks that are waiting for new entries. I suppose we could also use poll() with your scheme.
Another approach would be zero-copy; userspace could mmap the character device and get a completion queue that way. I'm not expecting this to be sufficiently high performance to warrant such an approach :-)
Yes, we do agree.
Yes, I like the read() with blocking idea rather than the polling, you mentioned that. I didn't think about the blocking, I missed that on the first go.
Regards,
Neal
-----Original Message-----
From: Matthew Wilcox [mailto:[email protected]]
Sent: Friday, May 31, 2013 12:53 PM
To: Neal Galbo (ngalbo)
Cc: linux-nvme at lists.infradead.org
Subject: Re: Proposal for handling async event requests
Quoted 1 line · Neal Galbo (ngalbo)
I think handling should be localized and associated through an existing, running NVMe driver, i.e. not a separate daemon.
OK, so we're fundamentally in agreement, it's just a matter of negotiating the API? :-)
Quoted 5 lines
1) User space (or supervisor) issues ioctl to "open" and enable AEN feature to a loaded, running driver in anticipation of receiving async events. Ioctl parameters contain pointer to callback routine and buffer space to receive AEN's.
2) When driver receives ioctl it saves incoming parameters, sets up to receive AEN's (otherwise it's "closed"/disabled). The driver can manage the AEN's within, directly or could launch an AEN handler worker thread that it owns or various other ways.
3) When AEN is received, driver or AEN manager calls back through pointer, deposits AEN info.
4) Provide a corresponding "close" ioctl to disable the AEN chatter when not needed.
5) ??? provide unique tags/id's in ioctl parameters to allow multiple openers. Unique tag generator should be located at a central point ... like the driver? (another ioctl to get a unique tag?)
Yes, this approach definitely works. Essentially, we're putting a completion queue in userspace. How do you envisage notifying userspace that it has new entries? The advantage of read() on the character device is that it can block. We can also use the poll() mechanism to wake up tasks that are waiting for new entries. I suppose we could also use poll() with your scheme.
Another approach would be zero-copy; userspace could mmap the character device and get a completion queue that way. I'm not expecting this to be sufficiently high performance to warrant such an approach :-)