netlink based mq_notify(SIGEV_THREAD)

mq_notify(SIGEV_THREAD) must be implemented in user space. If an event is triggered, the kernel must send a notification to user space, and then glibc must create the thread with the requested attributes for the notification callback. The current implementation in Andrew's -mm tree uses single shot file descriptor - it works, but it's resource hungry.

Attached is a new proposal:
- split netlink_unicast into separate substeps
- use an AF_NETLINK socket for the message queue notification

What do you think?

--
Manfred

// $Header$
// Kernel Version:

//  VERSION = 2
//  PATCHLEVEL = 6
//  SUBLEVEL = 5
//  EXTRAVERSION =-rc3-mm1

@@ -120,6 +120,13 @@
 extern int netlink_register_notifier(struct notifier_block *nb);
 extern int netlink_unregister_notifier(struct notifier_block *nb);
 
+/* finegrained unicast helpers: */
+struct sock *netlink_getsockbypid(struct sock *ssk, u32 pid);
+struct sock *netlink_getsockbyfilp(struct file *filp);
+int netlink_attachskb(struct sock *sk, struct sk_buff *skb, int nonblock, long timeo);
+void netlink_detachskb(struct sock *sk, struct sk_buff *skb);
+int netlink_sendskb(struct sock *sk, struct sk_buff *skb, int protocol);
+
 /*
  *	skb should fit one page. This choice is good for headerless malloc.
  *

@@ -30,8 +30,24 @@
 	long	__reserved[4];	/* ignored for input, zeroed for output */
 };
 
+/*
+ * SIGEV_THREAD implementation:
+ * SIGEV_THREAD must be implemented in user space. If SIGEV_THREAD is passed
+ * to mq_notify, then
+ * - sigev_signo must be the file descriptor of an AF_NETLINK socket. It's not
+ *   necessary that the socket is bound.
+ * - sigev_value.sival_ptr must point to a cookie that is NOTIFY_COOKIE_LEN
+ *   bytes long.
+ * If the notification is triggered, then the cookie is sent to the netlink
+ * socket. The first byte of the cookie is replaced with the NOTIFY_?? codes:
+ * NOTIFY_WOKENUP if the notification got triggered, NOTIFY_REMOVED if it was
+ * removed, either due to a close() on the message queue fd or due to a
+ * mq_notify() that removed the notification.
+ */
 #define NOTIFY_NONE	0
 #define NOTIFY_WOKENUP	1
 #define NOTIFY_REMOVED	2
 
+#define NOTIFY_COOKIE_LEN	32
+
 #endif

I am ignorant about SIGEV_THREAD but from what i gathered above:

- something (from user space??) attempts to create a thread in the kernel
- the kernel sends notification to user space when said thread is created or done doing something it was asked
- something (in glibc/userspace??) is signalled by the kernel to do something with the result

Is the above correct?

Essentially you attempt to open only a single fd via netlink as opposed to open/close behavior you are alluding to, is that correct? then all events are unicast to this fd. I am assuming you dont need to have more than one listener to these events? example, could one process create such a event which multiple processes may be interested in?

I am trying to frob why you mucked around with AF_NETLINK; maybe your response will shed some light.

cheers,
jamal

No - this part is wrong.

This is correct.

mq_notify is a function from the posix message queue interface:
It allows user space to request that a notification should be sent if a new message is waiting in the message queue. There are two options for the notification: a signal or a callback that should be called in the context of a new thread. Signals are trivial, but calling a function in the context of a new thread is tricky: the kernel can't create new user space threads.
Thus the kernel interface for mq_notify with sigev_notify==SIGEV_THREAD is an asynchroneous interface: the initial syscall just registers something and if a message arrives in the queue, then a notice is sent to user space. glibc must then create a SuS compatible interface on top of that.

The problem is how should I sent the information that a message is waiting to user space?

Yes.

Correct, always only one process interested in the notification.

I'm looking for the simplest interface to send 32 byte cookies from kernel to user space. The final send must be non-blocking, setup can block. Someone recommended that I should look at netlink. netlink_unicast was nearly perfect, except that I had to split setup and sending into two functions.

--
Manfred

The above is a good description of the challenge i think.

[..]

Your split of netlink_unicast seems fine ;
I guess the bigger question is whether this interface could be a speacilized netlink protocol instead? It doesnt seem too nasty as is right now, just tending towards cleanliness.
It seems that user space must first open a netlink socket for this to work but somehow the result skb is passed back to userspace using the mq_notify and not via the socket interface opened? Why should user space even bother doing this? The kernel could on its behalf, no? Are you sure there will always be one and only one message outstanding always?

cheers,
jamal

No, the result is returned via the socket fd. It's just created due to the mq_notify call.

There can be multiple messages outstanding. Each sucessful mq_notify call generates exactly one message, but a process could create multiple message queues and then there can be multiple messages in the notification socket.

--
Manfred