Syslets, Threadlets, generic AIO support, v5
Subsystems (6)
cover Syslets, Threadlets, generic AIO support, v5
this is the v5 release of the syslet/threadlet subsystem:
http://redhat.com/~mingo/syslet-patches/
this release took 4 days to get out, but there were a couple of key changes that needed some time to settle down:
- ported the code from v2.6.20 to current -git (v2.6.20-rc2 should be fine as a base)
- 64-bit support in terms of a x86_64 port. Jens has updated the FIO syslet code to work on 64-bit too. (kernel/async.c was pretty 64-bit clean already, it needed minimal changes for basic x86_64 support.)
- 32-bit user-space on 64-bit kernel compat support. 32-bit syslet and threadlet binaries work fine on 64-bit kernels.
- various cleanups and simplifications
the v4->v5 delta is:
17 files changed, 327 insertions(+), 271 deletions(-)
amongst the plans for v6 are cleanups/simplifications to the syslet engine API, a number of suggestions have been made for that already.
the linecount increase in v5 is mostly due to the x86_64 port. The ABI had to change again - see the async-test userspace code for details.
the x86_64 patch is a bit monolithic at the moment, i'll split it up further in v6.
As always, comments, suggestions, reports are welcome!
Ingo
PATCH 1/12 syslets: add async.h include file, kernel-side API definitions
From: Ingo Molnar <[email protected]>
add include/linux/async.h which contains the kernel-side API declarations.
it also provides NOP stubs for the !CONFIG_ASYNC_SUPPORT case.
--- include/linux/async.h | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+)
include/linux/async.h new file +88 -0
@@ -0,0 +1,88 @@ +#ifndef _LINUX_ASYNC_H +#define _LINUX_ASYNC_H + +#include <linux/completion.h> +#include <linux/compiler.h> +#include <linux/syslet.h> +#include <asm/unistd.h> + +/* +* The syslet subsystem - asynchronous syscall execution support. +* +* Syslet-subsystem internal definitions: +*/ + +/* +* The kernel-side copy of a syslet atom - with arguments expanded: +*/ +struct syslet_atom { +unsigned long flags; +unsigned long nr; +long __user *ret_ptr; +struct syslet_uatom __user *next; +unsigned long args[6 ]; +syscall_fn_t *call_table; +unsigned int nr_syscalls; +}; + +/* +* The 'async head' is the thread which has user-space context (ptregs) +* 'below it' - this is the one that can return to user-space: +*/ +struct async_head { +spinlock_t lock; +struct task_struct *user_task; + +struct list_head ready_async_threads; +struct list_head busy_async_threads; + +struct mutex completion_lock; +long events_left; +wait_queue_head_t wait; + +struct async_head_user __user *ahu; + +unsigned long __user *new_stackp; +unsigned long new_ip; +unsigned long restore_stack; +unsigned long restore_ip; +struct completion start_done; +struct completion exit_done; +}; + +/* +* The 'async thread' is either a newly created async thread or it is +* an 'ex-head' - it cannot return to user-space and only has kernel +* context. +*/ +struct async_thread { +struct task_struct *task; +unsigned long user_stack; +unsigned long user_ip; +struct async_head *ah; + +struct list_head entry; + +unsigned int exit; +}; + +/* +* Generic kernel API definitions: +*/ +#ifdef CONFIG_ASYNC_SUPPORT +extern void async_init (struct task_struct *t); +extern void async_exit (struct task_struct *t); +extern void __async_schedule (struct task_struct *t); +#else /* !CONFIG_ASYNC_SUPPORT */ +static inline void async_init (struct task_struct *t) +{ +} +static inline void async_exit (struct task_struct *t) +{ +} +static inline void __async_schedule (struct task_struct *t) +{ +} +#endif /* !CONFIG_ASYNC_SUPPORT */ + +#endif
PATCH 2/12 syslets: add syslet.h include file, user API/ABI definitions
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
add include/linux/syslet.h which contains the user-space API/ABI declarations. Add the new header to include/linux/Kbuild as well.
--- include/linux/Kbuild | 1 include/linux/syslet.h | 155 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 156 insertions(+)
include/linux/Kbuild +1 -0
@@ -141,6 +141,7 @@ header-y += sockios.h header-y += som.h header-y += sound.h header-y += synclink.h +header-y += syslet.h header-y += telephony.h header-y += termios.h header-y += ticable.h
include/linux/syslet.h new file +155 -0
155 lines not on this page · raw message
Re: [patch 02/12] syslets: add syslet.h include file, user API/ABI definitions
Quoted 4 lines · Ingo Molnar
From: Ingo Molnar <[email protected]>
add
include/linux/syslet.hwhich contains the user-space API/ABI declarations. Add the new header toinclude/linux/Kbuildas well.
Hi Ingo,
I'd like to propose a simpler userspace API for syslets. I believe this revised API is just as capable as yours (anything done purely in kernel space with the existing API can also be done with this one).
An "atom" would look like:
struct syslet_uatom {
u32 nr;
u64 ret_ptr;
u64 next;
u64 arg_nr;
u64 args[6];
};The sys_nr, ret_ptr, and next fields would be unchanged. The args array would directly store the arguments to the system call. To optimize the case where only a few arguments are necessary, an explicit argument count would be set in the arg_nr field.
The above is very similar to what Linus and Davide described as a "single submission" syslet interface - it differs only with the addition of the next parameter. As with your API, a null next field would immediately stop the syslet. So, a user wishing to run a single system call asynchronously could use the above interface with the next field set to null.
Of course, the above lacks the syscall return testing capabilities in your atoms. To obtain that capability, one could add a new syscall:
long sys_syslet_helper(long flags, long *ptr, long inc, u64 new_next)
The above is effectively a combination of sys_umem_add and the "flags" field from the existing syslet_uatom. The system call would only be available from syslets. It would add "inc" to the integer stored in "ptr" and return the result. The "flags" field could optionally contain one of:
SYSLET_BRANCH_ON_NONZERO
SYSLET_BRANCH_ON_ZERO
SYSLET_BRANCH_ON_NEGATIVE
SYSLET_BRANCH_ON_NON_POSITIVE
If the flag were set and the return result of the syscall met the specified condition, then the code would arrange for the calling syslet to branch to "new_next" instead of the normal "next".
I would also change the event ring notification system. Instead of building that support into all syslets, one could introduce an "add to head" syscall specifically for that purpose. If done this way, userspace could arrange for this new sys_addtoring call to always be the last uatom executed in a syslet. This would make the support optional - those userspace applications that prefer to use a futex or signal as an event system could arrange to have those system calls as the last one in the chain instead. With this change, the sys_async_exec would simplify to:
long sys_async_exec(struct syslet_uatom *uatom);
As above, I believe this API has as much power as the existing system. The general idea is to make the system call / atoms simpler and use more atoms when building complex chains.
For example, the open & stat case could be done with a chain like the following:
atom1: &atom3->args[1] = sys_open(...)
atom2: sys_syslet_helper(SYSLET_BRANCH_ON_NON_POSITIVE,
&atom3->args[1], 0, atom4)
atom3: sys_stat([arg1 filled above], ...)
atom4: sys_futex(...) // alert parent of completion
It is also possible to use sys_syslet_helper to push a return value to multiple syslet parameters (for example, propagating an fd from open to multiple reads). For example:
atom1: &atom3->args[1] = sys_open(...)
atom2: &atom4->args[1] = sys_syslet_helper(0, &atom3->args[1], 0, 0)
atom3: sys_read([arg1 filled in atom1], ...)
atom4: sys_read([arg1 filled in atom2], ...)
...
Although this is a bit ugly, I must wonder how many times one would build chains complex enough to require it.
Cheers,
-Kevin
Re: [patch 02/12] syslets: add syslet.h include file, user API/ABI definitions
Quoted 14 lines · Kevin O'Connor
I'd like to propose a simpler userspace API for syslets. I believe this revised API is just as capable as yours (anything done purely in kernel space with the existing API can also be done with this one).
An "atom" would look like:
struct syslet_uatom { u32 nr; u64 ret_ptr; u64 next; u64 arg_nr; u64 args[6]; };
Quoted 4 lines
The
sys_nr,ret_ptr, and next fields would be unchanged. The args array would directly store the arguments to the system call. To optimize the case where only a few arguments are necessary, an explicit argument count would be set in thearg_nrfield.
i dont see the advantage of arg_nr - if the arguments are direct then the best way is to just fetch them all, not to do testing upon arg_nr. Furthermore, regarding the indirect pointers, they are quite essential for some uses, see:
http://lkml.org/lkml/2007/2/28/292
Quoted 4 lines
Of course, the above lacks the syscall return testing capabilities in your atoms. To obtain that capability, one could add a new syscall:
long
sys_syslet_helper(long flags, long *ptr, long inc, u64 new_next)
yes. But a 'flags' field is handy anyway, to signal things like NOCOMPLETE or SYSLET_SYNC/SYSLET_ASYNC (a flags field is always useful in such structures). So the condition testing comes 'for free' in essence.
but ... as you can see it with sys_umem_add(), i like the addition of helper syscalls - i just think that this particular one wouldnt be too helpful.
Quoted 8 lines
I would also change the event ring notification system. Instead of building that support into all syslets, one could introduce an "add to head" syscall specifically for that purpose. If done this way, userspace could arrange for this new
sys_addtoringcall to always be the last uatom executed in a syslet. This would make the support optional - those userspace applications that prefer to use a futex or signal as an event system could arrange to have those system calls as the last one in the chain instead. [...]
the problem is signals: a syslet has to return to user-space upon signals or upon a stop condition. So to notify about the precise place of stoppage, the notification ring is a 'built in' property.
(nevertheless, as i mentioned it in a prior mail, i'll create separate ring syscalls, because they are useful for other stuff too.)
Quoted 6 lines
For example, the open & stat case could be done with a chain like the following:
atom1: &atom3->args[1] =
sys_open(...)
atom2: sys_syslet_helper(SYSLET_BRANCH_ON_NON_POSITIVE,&atom3->args[1], 0, atom4)
i dont see a huge conceptual difference between having the syslet helper in flags versus having it in a separate syscall. Other than yours has twice the number of atoms.
Quoted 8 lines
It is also possible to use
sys_syslet_helperto push a return value to multiple syslet parameters (for example, propagating an fd from open to multiple reads). For example:atom1: &atom3->args[1] =
sys_open(...)
atom2: &atom4->args[1] =sys_syslet_helper(0, &atom3->args[1], 0, 0)
atom3:sys_read([arg1 filled in atom1], ...)
atom4:sys_read([arg1 filled in atom2], ...)
try to do this in FIO. You'd have to create many extra atoms to push the fd into the argument fields - instead of just sharing the variable. Sharing is /good/. These 'simplifications' complicate the whole syslet programming model down to being near unusable.
Quoted 2 lines
Although this is a bit ugly, I must wonder how many times one would build chains complex enough to require it.
take a look at FIO.
Ingo
PATCH 3/12 syslets: generic kernel bits
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
add the kernel generic bits - these are present even if !CONFIG_ASYNC_SUPPORT.
--- fs/exec.c | 4 ++++ include/linux/sched.h | 23 ++++++++++++++++++++++- kernel/capability.c | 3 +++ kernel/exit.c | 7 +++++++ kernel/fork.c | 5 +++++ kernel/sched.c | 9 +++++++++ kernel/sys.c | 36 ++++++++++++++++++++++++++++++++++++ 7 files changed, 86 insertions(+), 1 deletion(-)
fs/exec.c +4 -0
@@ -1444,6 +1444,10 @@ static int coredump_wait(int exit_code) tsk->vfork_done =NULL ;complete (vfork_done); } +/* +* Make sure we exit our async context before waiting: +*/ +async_exit (tsk);if (core_waiters)wait_for_completion (&startup_done);
include/linux/sched.h +22 -1
@@ -83,12 +83,12 @@ struct sched_param {#include <linux/timer.h> #include <linux/hrtimer.h> #include <linux/task_io_accounting.h> +#include <linux/async.h> #include <asm/processor.h> struct exec_domain ;struct futex_pi_state ; -/* * List of flags we want to share for kernel threads, * if only because they are not used by them anyway.
@@ -997,6 +997,12 @@ struct task_struct {/* journalling filesystem info */ void *journal_info; +/* async syscall support: */ +struct async_thread *at, *async_ready; +struct async_head *ah; +struct async_thread __at; +struct async_head __ah; +/* VM state */ struct reclaim_state *reclaim_state;
@@ -1055,6 +1061,21 @@ struct task_struct {#endif }; +/* +* Is an async syscall being executed currently? +*/ +#ifdef CONFIG_ASYNC_SUPPORT +static inline int async_syscall (struct task_struct *t) +{ +return t->async_ready !=NULL ; +} +#else /* !CONFIG_ASYNC_SUPPORT */ +static inline int async_syscall (struct task_struct *t) +{ +return 0 ; +} +#endif /* !CONFIG_ASYNC_SUPPORT */ +static inline pid_t process_group (struct task_struct *tsk) {return tsk->signal->pgrp;
kernel/capability.c +3 -0
@@ -178,6 +178,9 @@ asmlinkage long sys_capset(cap_user_headint ret;pid_t pid; +if (async_syscall (current)) +return -ENOSYS ; +if (get_user (version, &header->version))return -EFAULT ;
kernel/exit.c +7 -0
@@ -26,6 +26,7 @@#include <linux/ptrace.h> #include <linux/profile.h> #include <linux/mount.h> +#include <linux/async.h> #include <linux/proc_fs.h> #include <linux/mempolicy.h> #include <linux/taskstats_kern.h>
@@ -890,6 +891,12 @@ fastcall NORET_TYPE void do_exit(long coschedule (); } +/* +* Note: async threads have to exit their context before the MM +* exit (due to the coredumping wait): +*/ +async_exit (tsk); + tsk->flags |=PF_EXITING ;if (unlikely (in_atomic ()))
kernel/fork.c +5 -0
@@ -22,6 +22,7 @@#include <linux/personality.h> #include <linux/mempolicy.h> #include <linux/sem.h> +#include <linux/async.h> #include <linux/file.h> #include <linux/key.h> #include <linux/binfmts.h>
@@ -1056,6 +1057,7 @@ static struct task_struct *copy_process( p->lock_depth =-1 ;/* -1 = no lock */ do_posix_clock_monotonic_gettime (&p->start_time); +async_init (p); p->security =NULL ; p->io_context =NULL ; p->io_wait =NULL ;
@@ -1623,6 +1625,9 @@ asmlinkage long sys_unshare(unsigned lonstruct uts_namespace *uts, *new_uts =NULL ;struct ipc_namespace *ipc, *new_ipc =NULL ; +if (async_syscall (current)) +return -ENOSYS ; +check_unshare_flags (&unshare_flags);/* Return -EINVAL for all unsupported flags */
kernel/sched.c +9 -0
@@ -38,6 +38,7 @@#include <linux/vmalloc.h> #include <linux/blkdev.h> #include <linux/delay.h> +#include <linux/async.h> #include <linux/smp.h> #include <linux/threads.h> #include <linux/timer.h>
@@ -3455,6 +3456,14 @@ asmlinkage void __sched schedule(void) }profile_hit (SCHED_PROFILING ,__builtin_return_address (0 )); + prev = current; +if (unlikely (prev->async_ready)) { +if (prev->state && !(preempt_count () &PREEMPT_ACTIVE ) && + (!(prev->state &TASK_INTERRUPTIBLE ) || + !signal_pending (prev))) +__async_schedule (prev); + } + need_resched:preempt_disable (); prev = current;
kernel/sys.c +36 -0
@@ -941,6 +941,9 @@ asmlinkage long sys_setregid(gid_t rgid,int new_egid = old_egid;int retval; +if (async_syscall (current)) +return -ENOSYS ; + retval =security_task_setgid (rgid, egid, (gid_t)-1 ,LSM_SETID_RE );if (retval)return retval;
@@ -987,6 +990,9 @@ asmlinkage long sys_setgid(gid_t gid)int old_egid = current->egid;int retval; +if (async_syscall (current)) +return -ENOSYS ; + retval =security_task_setgid (gid, (gid_t)-1 , (gid_t)-1 ,LSM_SETID_ID );if (retval)return retval;
@@ -1057,6 +1063,9 @@ asmlinkage long sys_setreuid(uid_t ruid,int old_ruid, old_euid, old_suid, new_ruid, new_euid;int retval; +if (async_syscall (current)) +return -ENOSYS ; + retval =security_task_setuid (ruid, euid, (uid_t)-1 ,LSM_SETID_RE );if (retval)return retval;
@@ -1120,6 +1129,9 @@ asmlinkage long sys_setuid(uid_t uid)int old_ruid, old_suid, new_suid;int retval; +if (async_syscall (current)) +return -ENOSYS ; + retval =security_task_setuid (uid, (uid_t)-1 , (uid_t)-1 ,LSM_SETID_ID );if (retval)return retval;
@@ -1160,6 +1172,9 @@ asmlinkage long sys_setresuid(uid_t ruidint old_suid = current->suid;int retval; +if (async_syscall (current)) +return -ENOSYS ; + retval =security_task_setuid (ruid, euid, suid,LSM_SETID_RES );if (retval)return retval;
@@ -1214,6 +1229,9 @@ asmlinkage long sys_setresgid(gid_t rgid {int retval; +if (async_syscall (current)) +return -ENOSYS ; + retval =security_task_setgid (rgid, egid, sgid,LSM_SETID_RES );if (retval)return retval;
@@ -1269,6 +1287,9 @@ asmlinkage long sys_setfsuid(uid_t uid) {int old_fsuid; +if (async_syscall (current)) +return -ENOSYS ; + old_fsuid = current->fsuid;if (security_task_setuid (uid, (uid_t)-1 , (uid_t)-1 ,LSM_SETID_FS ))return old_fsuid;
@@ -1298,6 +1319,9 @@ asmlinkage long sys_setfsgid(gid_t gid) {int old_fsgid; +if (async_syscall (current)) +return -ENOSYS ; + old_fsgid = current->fsgid;if (security_task_setgid (gid, (gid_t)-1 , (gid_t)-1 ,LSM_SETID_FS ))return old_fsgid;
@@ -1373,6 +1397,9 @@ asmlinkage long sys_setpgid(pid_t pid, pstruct task_struct *group_leader = current->group_leader;int err = -EINVAL ; +if (async_syscall (current)) +return -ENOSYS ; +if (!pid) pid = group_leader->pid;if (!pgid)
@@ -1496,6 +1523,9 @@ asmlinkage long sys_setsid(void)pid_t session;int err = -EPERM ; +if (async_syscall (current)) +return -ENOSYS ; +write_lock_irq (&tasklist_lock);/* Fail if I am already a session leader */
@@ -1739,6 +1769,9 @@ asmlinkage long sys_setgroups(int gidsetstruct group_info *group_info;int retval; +if (async_syscall (current)) +return -ENOSYS ; +if (!capable (CAP_SETGID ))return -EPERM ;if ((unsigned )gidsetsize >NGROUPS_MAX )
@@ -2080,6 +2113,9 @@ asmlinkage long sys_prctl(int option, un {long error; +if (async_syscall (current)) +return -ENOSYS ; + error =security_task_prctl (option, arg2, arg3, arg4, arg5);if (error)return error;
PATCH 4/12 syslets: core code
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
the core syslet / async system calls infrastructure code.
Is built only if CONFIG_ASYNC_SUPPORT is enabled.
--- kernel/Makefile | 1 kernel/async.c | 989 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 990 insertions(+)
kernel/Makefile +1 -0
@@ -10,6 +10,7 @@ obj-y = sched.o fork.o exec_domain.o kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o latency.o nsproxy.o srcu.o +obj-$(CONFIG_ASYNC_SUPPORT) += async.o obj-$(CONFIG_STACKTRACE) += stacktrace.o obj-y += time/ obj-$(CONFIG_DEBUG_MUTEXES) += mutex-debug.o
kernel/async.c new file +989 -0
989 lines not on this page · raw message
PATCH 5/12 syslets: core, documentation
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
Add Documentation/syslet-design.txt with a high-level description of the syslet concepts.
--- Documentation/syslet-design.txt | 137 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 137 insertions(+)
Documentation/syslet-design.txt new file +137 -0
@@ -0,0 +1,137 @@ +Syslets / asynchronous system calls +=================================== + +started by Ingo Molnar <[email protected]> + +Goal: +----- + +The goal of the syslet subsystem is to allow user-space to execute +arbitrary system calls asynchronously. It does so by allowing user-space +to execute "syslets" which are small scriptlets that the kernel can execute +both securely and asynchronously without having to exit to user-space. + +the core syslet concepts are: + +The Syslet Atom: +---------------- + +The syslet atom is a small, fixed-size (44 bytes on 32-bit) piece of +user-space memory, which is the basic unit of execution within the syslet +framework. A syslet represents a single system-call and its arguments. +In addition it also has condition flags attached to it that allows the +construction of larger programs (syslets) from these atoms. + +Arguments to the system call are implemented via pointers to arguments. +This not only increases the flexibility of syslet atoms (multiple syslets +can share the same variable for example), but is also an optimization: +copy_uatom() will only fetch syscall parameters up until the point it +meets the first NULL pointer. 50% of all syscalls have 2 or less +parameters (and 90% of all syscalls have 4 or less parameters). + + [ Note: since the argument array is at the end of the atom, and the + kernel will not touch any argument beyond the first NULL one, atoms + might be packed more tightly. (the only special case exception to + this rule would be SKIP_TO_NEXT_ON_STOP atoms, where the kernel will + jump a full syslet_uatom number of bytes.) ] + +The Syslet: +----------- + +A syslet is a program, represented by a graph of syslet atoms. The +syslet atoms are chained to each other either via the atom->next pointer, +or via the SYSLET_SKIP_TO_NEXT_ON_STOP flag. + +Running Syslets: +---------------- + +Syslets can be run via the sys_async_exec() system call, which takes +the first atom of the syslet as an argument. The kernel does not need +to be told about the other atoms - it will fetch them on the fly as +execution goes forward. + +A syslet might either be executed 'cached', or it might generate a +'cachemiss'. + +'Cached' syslet execution means that the whole syslet was executed +without blocking. The system-call returns the submitted atom's address +in this case. + +If a syslet blocks while the kernel executes a system-call embedded in +one of its atoms, the kernel will keep working on that syscall in +parallel, but it immediately returns to user-space with a NULL pointer, +so the submitting task can submit other syslets. + +Completion of asynchronous syslets: +----------------------------------- + +Completion of asynchronous syslets is done via the 'completion ring', +which is a ringbuffer of syslet atom pointers in user-space memory, +provided by user-space as an argument to the sys_async_exec() syscall. +The kernel fills in the ringbuffer starting at index 0, and user-space +must clear out these pointers. Once the kernel reaches the end of +the ring it wraps back to index 0. The kernel will not overwrite +non-NULL pointers (but will return an error), and thus user-space has +to make sure it completes all events it asked for. + +Waiting for completions: +------------------------ + +Syslet completions can be waited for via the sys_async_wait() +system call - which takes the number of events it should wait for as +a parameter. This system call will also return if the number of +pending events goes down to zero. + +Sample Hello World syslet code: + +---------------------------> +/* + * Set up a syslet atom: + */ +static void +init_atom(struct syslet_uatom *atom, int nr, + void *arg_ptr0, void *arg_ptr1, void *arg_ptr2, + void *arg_ptr3, void *arg_ptr4, void *arg_ptr5, + void *ret_ptr, unsigned long flags, struct syslet_uatom *next) +{ + atom->nr = nr; + atom->arg_ptr[0] = arg_ptr0; + atom->arg_ptr[1] = arg_ptr1; + atom->arg_ptr[2] = arg_ptr2; + atom->arg_ptr[3] = arg_ptr3; + atom->arg_ptr[4] = arg_ptr4; + atom->arg_ptr[5] = arg_ptr5; + atom->ret_ptr = ret_ptr; + atom->flags = flags; + atom->next = next; +} + +int main(int argc, char *argv[]) +{ + unsigned long int fd_out = 1; /* standard output */ + char *buf = "Hello Syslet World!\n"; + unsigned long size = strlen(buf); + struct syslet_uatom atom, *done; + + async_head_init(); + + /* + * Simple syslet consisting of a single atom: + */ + init_atom(&atom, __NR_sys_write, &fd_out, &buf, &size, + NULL, NULL, NULL, NULL, SYSLET_ASYNC, NULL); + done = sys_async_exec(&atom); + if (!done) { + sys_async_wait(1); + if (completion_ring[curr_ring_idx] == &atom) { + completion_ring[curr_ring_idx] = NULL; + printf("completed an async syslet atom!\n"); + } + } else { + printf("completed an cached syslet atom!\n"); + } + + async_head_exit(); + + return 0; +}
PATCH 6/12 x86: split FPU state from task state
In reply to Ingo MolnarFrom: Arjan van de Ven <[email protected]>
Split the FPU save area from the task struct. This allows easy migration of FPU context, and it's generally cleaner. It also allows the following two (future) optimizations:
1) allocate the right size for the actual cpu rather than 512 bytes always
2) only allocate when the application actually uses FPU, so in the first lazy FPU trap. This could save memory for non-fpu using apps.
--- arch/i386/kernel/i387.c | 96 ++++++++++++++++++++--------------------- arch/i386/kernel/process.c | 56 +++++++++++++++++++++++ arch/i386/kernel/traps.c | 10 ---- include/asm-i386/i387.h | 6 +- include/asm-i386/processor.h | 6 ++ include/asm-i386/thread_info.h | 6 ++ kernel/fork.c | 7 ++ 7 files changed, 123 insertions(+), 64 deletions(-)
arch/i386/kernel/i387.c +48 -48
@@ -31,9 +31,9 @@ void mxcsr_feature_mask_init(void)unsigned long mask =0 ;clts ();if (cpu_has_fxsr) { -memset (¤t->thread.i387.fxsave,0 ,sizeof (struct i387_fxsave_struct )); - asmvolatile ("fxsave %0" : :"m" (current->thread.i387.fxsave)); - mask = current->thread.i387.fxsave.mxcsr_mask; +memset (¤t->thread.i387->fxsave,0 ,sizeof (struct i387_fxsave_struct )); + asmvolatile ("fxsave %0" : :"m" (current->thread.i387->fxsave)); + mask = current->thread.i387->fxsave.mxcsr_mask;if (mask ==0 ) mask =0x0000ffbf ; } mxcsr_feature_mask &= mask;
@@ -49,16 +49,16 @@ void mxcsr_feature_mask_init(void)void init_fpu (struct task_struct *tsk) {if (cpu_has_fxsr) { -memset (&tsk->thread.i387.fxsave,0 ,sizeof (struct i387_fxsave_struct )); - tsk->thread.i387.fxsave.cwd =0x37f ; +memset (&tsk->thread.i387->fxsave,0 ,sizeof (struct i387_fxsave_struct )); + tsk->thread.i387->fxsave.cwd =0x37f ;if (cpu_has_xmm) - tsk->thread.i387.fxsave.mxcsr =0x1f80 ; + tsk->thread.i387->fxsave.mxcsr =0x1f80 ; }else { -memset (&tsk->thread.i387.fsave,0 ,sizeof (struct i387_fsave_struct )); - tsk->thread.i387.fsave.cwd =0xffff037fu ; - tsk->thread.i387.fsave.swd =0xffff0000u ; - tsk->thread.i387.fsave.twd =0xffffffffu ; - tsk->thread.i387.fsave.fos =0xffff0000u ; +memset (&tsk->thread.i387->fsave,0 ,sizeof (struct i387_fsave_struct )); + tsk->thread.i387->fsave.cwd =0xffff037fu ; + tsk->thread.i387->fsave.swd =0xffff0000u ; + tsk->thread.i387->fsave.twd =0xffffffffu ; + tsk->thread.i387->fsave.fos =0xffff0000u ; }/* only the device not available exception or ptrace can call init_fpu */ set_stopped_child_used_math (tsk);
@@ -152,18 +152,18 @@ static inline unsigned long twd_fxsr_to_unsigned short get_fpu_cwd (struct task_struct *tsk ) {if ( cpu_has_fxsr ) { -return tsk->thread.i387.fxsave.cwd; +return tsk->thread.i387->fxsave.cwd; }else { -return (unsigned short )tsk->thread.i387.fsave.cwd; +return (unsigned short )tsk->thread.i387->fsave.cwd; } }unsigned short get_fpu_swd (struct task_struct *tsk ) {if ( cpu_has_fxsr ) { -return tsk->thread.i387.fxsave.swd; +return tsk->thread.i387->fxsave.swd; }else { -return (unsigned short )tsk->thread.i387.fsave.swd; +return (unsigned short )tsk->thread.i387->fsave.swd; } }
@@ -171,9 +171,9 @@ unsigned short get_fpu_swd( struct task_unsigned short get_fpu_twd (struct task_struct *tsk ) {if ( cpu_has_fxsr ) { -return tsk->thread.i387.fxsave.twd; +return tsk->thread.i387->fxsave.twd; }else { -return (unsigned short )tsk->thread.i387.fsave.twd; +return (unsigned short )tsk->thread.i387->fsave.twd; } }#endif /* 0 */
@@ -181,7 +181,7 @@ unsigned short get_fpu_twd( struct task_unsigned short get_fpu_mxcsr (struct task_struct *tsk ) {if ( cpu_has_xmm ) { -return tsk->thread.i387.fxsave.mxcsr; +return tsk->thread.i387->fxsave.mxcsr; }else {return 0x1f80 ; }
@@ -192,27 +192,27 @@ unsigned short get_fpu_mxcsr( struct tasvoid set_fpu_cwd (struct task_struct *tsk,unsigned short cwd ) {if ( cpu_has_fxsr ) { - tsk->thread.i387.fxsave.cwd = cwd; + tsk->thread.i387->fxsave.cwd = cwd; }else { - tsk->thread.i387.fsave.cwd = ((long )cwd |0xffff0000u ); + tsk->thread.i387->fsave.cwd = ((long )cwd |0xffff0000u ); } }void set_fpu_swd (struct task_struct *tsk,unsigned short swd ) {if ( cpu_has_fxsr ) { - tsk->thread.i387.fxsave.swd = swd; + tsk->thread.i387->fxsave.swd = swd; }else { - tsk->thread.i387.fsave.swd = ((long )swd |0xffff0000u ); + tsk->thread.i387->fsave.swd = ((long )swd |0xffff0000u ); } }void set_fpu_twd (struct task_struct *tsk,unsigned short twd ) {if ( cpu_has_fxsr ) { - tsk->thread.i387.fxsave.twd =twd_i387_to_fxsr (twd); + tsk->thread.i387->fxsave.twd =twd_i387_to_fxsr (twd); }else { - tsk->thread.i387.fsave.twd = ((long )twd |0xffff0000u ); + tsk->thread.i387->fsave.twd = ((long )twd |0xffff0000u ); } }
@@ -298,8 +298,8 @@ static inline int save_i387_fsave( strucstruct task_struct *tsk = current;unlazy_fpu ( tsk ); - tsk->thread.i387.fsave.status = tsk->thread.i387.fsave.swd; -if (__copy_to_user ( buf, &tsk->thread.i387.fsave, + tsk->thread.i387->fsave.status = tsk->thread.i387->fsave.swd; +if (__copy_to_user ( buf, &tsk->thread.i387->fsave,sizeof (struct i387_fsave_struct ) ) )return -1 ;return 1 ;
@@ -312,15 +312,15 @@ static int save_i387_fxsave( struct _fpsunlazy_fpu ( tsk ); -if (convert_fxsr_to_user ( buf, &tsk->thread.i387.fxsave ) ) +if (convert_fxsr_to_user ( buf, &tsk->thread.i387->fxsave ) )return -1 ; - err |=__put_user ( tsk->thread.i387.fxsave.swd, &buf->status ); + err |=__put_user ( tsk->thread.i387->fxsave.swd, &buf->status ); err |=__put_user (X86_FXSR_MAGIC , &buf->magic );if ( err )return -1 ; -if (__copy_to_user ( &buf->_fxsr_env[0 ], &tsk->thread.i387.fxsave, +if (__copy_to_user ( &buf->_fxsr_env[0 ], &tsk->thread.i387->fxsave,sizeof (struct i387_fxsave_struct ) ) )return -1 ;return 1 ;
@@ -343,7 +343,7 @@ int save_i387( struct _fpstate __user *breturn save_i387_fsave ( buf ); } }else { -return save_i387_soft ( ¤t->thread.i387.soft, buf ); +return save_i387_soft ( ¤t->thread.i387->soft, buf ); } }
@@ -351,7 +351,7 @@ static inline int restore_i387_fsave( st {struct task_struct *tsk = current;clear_fpu ( tsk ); -return __copy_from_user ( &tsk->thread.i387.fsave, buf, +return __copy_from_user ( &tsk->thread.i387->fsave, buf,sizeof (struct i387_fsave_struct ) ); }
@@ -360,11 +360,11 @@ static int restore_i387_fxsave( struct _int err;struct task_struct *tsk = current;clear_fpu ( tsk ); - err =__copy_from_user ( &tsk->thread.i387.fxsave, &buf->_fxsr_env[0 ], + err =__copy_from_user ( &tsk->thread.i387->fxsave, &buf->_fxsr_env[0 ],sizeof (struct i387_fxsave_struct ) );/* mxcsr reserved bits must be masked to zero for security reasons */ - tsk->thread.i387.fxsave.mxcsr &= mxcsr_feature_mask; -return err ?1 :convert_fxsr_from_user ( &tsk->thread.i387.fxsave, buf ); + tsk->thread.i387->fxsave.mxcsr &= mxcsr_feature_mask; +return err ?1 :convert_fxsr_from_user ( &tsk->thread.i387->fxsave, buf ); }int restore_i387 (struct _fpstate __user *buf )
@@ -378,7 +378,7 @@ int restore_i387( struct _fpstate __user err =restore_i387_fsave ( buf ); } }else { - err =restore_i387_soft ( ¤t->thread.i387.soft, buf ); + err =restore_i387_soft ( ¤t->thread.i387->soft, buf ); }set_used_math ();return err;
@@ -391,7 +391,7 @@ int restore_i387( struct _fpstate __userstatic inline int get_fpregs_fsave (struct user_i387_struct __user *buf,struct task_struct *tsk ) { -return __copy_to_user ( buf, &tsk->thread.i387.fsave, +return __copy_to_user ( buf, &tsk->thread.i387->fsave,sizeof (struct user_i387_struct ) ); }
@@ -399,7 +399,7 @@ static inline int get_fpregs_fxsave( str struct task_struct *tsk ) {return convert_fxsr_to_user ( (struct _fpstate __user *)buf, - &tsk->thread.i387.fxsave ); + &tsk->thread.i387->fxsave ); }int get_fpregs (struct user_i387_struct __user *buf,struct task_struct *tsk )
@@ -411,7 +411,7 @@ int get_fpregs( struct user_i387_structreturn get_fpregs_fsave ( buf, tsk ); } }else { -return save_i387_soft ( &tsk->thread.i387.soft, +return save_i387_soft ( &tsk->thread.i387->soft, (struct _fpstate __user *)buf ); } }
@@ -419,14 +419,14 @@ int get_fpregs( struct user_i387_structstatic inline int set_fpregs_fsave (struct task_struct *tsk,struct user_i387_struct __user *buf ) { -return __copy_from_user ( &tsk->thread.i387.fsave, buf, +return __copy_from_user ( &tsk->thread.i387->fsave, buf,sizeof (struct user_i387_struct ) ); }static inline int set_fpregs_fxsave (struct task_struct *tsk,struct user_i387_struct __user *buf ) { -return convert_fxsr_from_user ( &tsk->thread.i387.fxsave, +return convert_fxsr_from_user ( &tsk->thread.i387->fxsave, (struct _fpstate __user *)buf ); }
@@ -439,7 +439,7 @@ int set_fpregs( struct task_struct *tsk,return set_fpregs_fsave ( tsk, buf ); } }else { -return restore_i387_soft ( &tsk->thread.i387.soft, +return restore_i387_soft ( &tsk->thread.i387->soft, (struct _fpstate __user *)buf ); } }
@@ -447,7 +447,7 @@ int set_fpregs( struct task_struct *tsk,int get_fpxregs (struct user_fxsr_struct __user *buf,struct task_struct *tsk ) {if ( cpu_has_fxsr ) { -if (__copy_to_user ( buf, &tsk->thread.i387.fxsave, +if (__copy_to_user ( buf, &tsk->thread.i387->fxsave,sizeof (struct user_fxsr_struct ) ))return -EFAULT ;return 0 ;
@@ -461,11 +461,11 @@ int set_fpxregs( struct task_struct *tskint ret =0 ;if ( cpu_has_fxsr ) { -if (__copy_from_user ( &tsk->thread.i387.fxsave, buf, +if (__copy_from_user ( &tsk->thread.i387->fxsave, buf,sizeof (struct user_fxsr_struct ) )) ret = -EFAULT ;/* mxcsr reserved bits must be masked to zero for security reasons */ - tsk->thread.i387.fxsave.mxcsr &= mxcsr_feature_mask; + tsk->thread.i387->fxsave.mxcsr &= mxcsr_feature_mask; }else { ret = -EIO ; }
@@ -479,7 +479,7 @@ int set_fpxregs( struct task_struct *tskstatic inline void copy_fpu_fsave (struct task_struct *tsk,struct user_i387_struct *fpu ) { -memcpy ( fpu, &tsk->thread.i387.fsave, +memcpy ( fpu, &tsk->thread.i387->fsave,sizeof (struct user_i387_struct ) ); }
@@ -490,10 +490,10 @@ static inline void copy_fpu_fxsave( struunsigned short *from;int i; -memcpy ( fpu, &tsk->thread.i387.fxsave,7 *sizeof (long ) ); +memcpy ( fpu, &tsk->thread.i387->fxsave,7 *sizeof (long ) ); to = (unsigned short *)&fpu->st_space[0 ]; - from = (unsigned short *)&tsk->thread.i387.fxsave.st_space[0 ]; + from = (unsigned short *)&tsk->thread.i387->fxsave.st_space[0 ];for ( i =0 ; i <8 ; i++, to +=5 , from +=8 ) {memcpy ( to, from,5 *sizeof (unsigned short ) ); }
@@ -540,7 +540,7 @@ int dump_task_extended_fpu(struct task_sif (fpvalid) {if (tsk == current)unlazy_fpu (tsk); -memcpy (fpu, &tsk->thread.i387.fxsave,sizeof (*fpu)); +memcpy (fpu, &tsk->thread.i387->fxsave,sizeof (*fpu)); }return fpvalid; }
arch/i386/kernel/process.c +55 -1
65 lines not on this page · raw message
arch/i386/kernel/traps.c +0 -10
@@ -1157,16 +1157,6 @@ void __init trap_init(void)set_trap_gate (19 ,&simd_coprocessor_error);if (cpu_has_fxsr) { -/* -* Verify that the FXSAVE/FXRSTOR data will be 16-byte aligned. -* Generates a compile-time "error: zero width for bit-field" if -* the alignment is wrong. -*/ -struct fxsrAlignAssert { -int _:!(offsetof(struct task_struct , - thread.i387.fxsave) & 15); - }; -printk (KERN_INFO "Enabling fast FPU save and restore... " );set_in_cr4 (X86_CR4_OSFXSR );printk ("done.\n" );
include/asm-i386/i387.h +3 -3
@@ -34,7 +34,7 @@ extern void init_fpu(struct task_struct "nop ; frstor %1", \ "fxrstor %1", \ X86_FEATURE_FXSR, \ - "m" ((tsk)->thread.i387.fxsave)) + "m" ((tsk)->thread.i387->fxsave))extern void kernel_fpu_begin (void );#define kernel_fpu_end () do { stts(); preempt_enable(); } while(0)
@@ -60,8 +60,8 @@ static inline void __save_init_fpu( stru"fxsave %[fx]\n" "bt $7,%[fsw] ; jnc 1f ; fnclex\n1:" , X86_FEATURE_FXSR, - [fx] "m" (tsk->thread.i387.fxsave), - [fsw] "m" (tsk->thread.i387.fxsave.swd) : "memory"); + [fx] "m" (tsk->thread.i387->fxsave), + [fsw] "m" (tsk->thread.i387->fxsave.swd) : "memory");/* AMD K7/K8 CPUs don't save/restore FDP/FIP/FOP unless an exception is pending. Clear the x87 state here by setting it to fixed values. safe_address is a random variable that should be in L1 */
include/asm-i386/processor.h +5 -1
@@ -407,7 +407,7 @@ struct thread_struct {/* fault info */ unsigned long cr2, trap_no, error_code;/* floating point info */ -union i387_union i387; +union i387_union *i387;/* virtual 86 mode info */ struct vm86_struct __user * vm86_info;unsigned long screen_bitmap;
@@ -420,11 +420,15 @@ struct thread_struct {unsigned long io_bitmap_max; }; + +extern union i387_union init_i387_context; +#define INIT_THREAD { \ .vm86_info = NULL, \ .sysenter_cs = __KERNEL_CS, \ .io_bitmap_ptr = NULL, \ .fs = __KERNEL_PDA, \ + .i387 = &init_i387_context, \ }/*
include/asm-i386/thread_info.h +6 -0
@@ -102,6 +102,12 @@ static inline struct thread_info *curren#define free_thread_info (info) kfree(info) +#define __HAVE_ARCH_TASK_STRUCT_ALLOCATOR +extern struct task_struct *alloc_task_struct (void ); +extern void free_task_struct (struct task_struct *tsk); +extern void memcpy_task_struct (struct task_struct *dst,struct task_struct *src); +extern void task_struct_slab_init (void ); +#else /* !__ASSEMBLY__ */ /* how to get the thread information struct from ASM */
kernel/fork.c +6 -1
@@ -84,6 +84,8 @@ int nr_processes(void)#ifndef __HAVE_ARCH_TASK_STRUCT_ALLOCATOR# define alloc_task_struct () kmem_cache_alloc(task_struct_cachep, GFP_KERNEL)# define free_task_struct (tsk) kmem_cache_free(task_struct_cachep, (tsk)) +# define memcpy_task_struct (dst, src) *dst = *src; +static struct kmem_cache *task_struct_cachep;#endif
@@ -138,6 +140,8 @@ void __init fork_init(unsigned long memp task_struct_cachep =kmem_cache_create ("task_struct" ,sizeof (struct task_struct ),ARCH_MIN_TASKALIGN ,SLAB_PANIC ,NULL ,NULL ); +#else +task_struct_slab_init ();#endif /*
@@ -176,7 +180,8 @@ static struct task_struct *dup_task_strureturn NULL ; } - *tsk = *orig; +memcpy_task_struct (tsk, orig); + tsk->thread_info = ti;setup_thread_stack (tsk, orig);
PATCH 7/12 syslets: x86, add create_async_thread() method
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
add the create_async_thread() way of creating kernel threads:
these threads first execute a kernel function and when they return from it they execute user-space.
An architecture must implement this interface before it can turn CONFIG_ASYNC_SUPPORT on.
--- arch/i386/kernel/entry.S | 25 +++++++++++++++++++++++++ arch/i386/kernel/process.c | 31 +++++++++++++++++++++++++++++++ include/asm-i386/processor.h | 17 +++++++++++++++++ include/asm-i386/unistd.h | 10 ++++++++++ 4 files changed, 83 insertions(+)
arch/i386/kernel/entry.S +25 -0
@@ -1034,6 +1034,31 @@ ENTRY(kernel_thread_helper) CFI_ENDPROC ENDPROC(kernel_thread_helper) +ENTRY(async_thread_helper) + CFI_STARTPROC + /* + * Allocate space on the stack for pt-regs. + * sizeof(struct pt_regs) == 64, and we've got 8 bytes on the + * kernel stack already: + */ + subl $64-8, %esp + CFI_ADJUST_CFA_OFFSET 64-8 + movl %edx,%eax + push %edx + CFI_ADJUST_CFA_OFFSET 4 + call *%ebx + addl $4, %esp + CFI_ADJUST_CFA_OFFSET -4 + + movl %eax, PT_EAX(%esp) + + GET_THREAD_INFO(%ebp) + + jmp syscall_exit + CFI_ENDPROC +ENDPROC(async_thread_helper) + + .section .rodata,"a" #include "syscall_table.S"
arch/i386/kernel/process.c +31 -0
@@ -355,6 +355,37 @@ int kernel_thread(int (*fn)(void *), voiEXPORT_SYMBOL (kernel_thread);/* +* This gets run with %ebx containing the +* function to call, and %edx containing +* the "args". +*/ +extern void async_thread_helper (void ); + +/* +* Create an async thread +*/ +int create_async_thread (long (*fn)(void *),void * arg,unsigned long flags) +{ +struct pt_regs regs; + +memset (®s,0 ,sizeof (regs)); + + regs.ebx = (unsigned long ) fn; + regs.edx = (unsigned long ) arg; + + regs.xds = __USER_DS; + regs.xes = __USER_DS; + regs.xfs = __KERNEL_PDA; + regs.orig_eax =-1 ; + regs.eip = (unsigned long ) async_thread_helper; + regs.xcs = __KERNEL_CS |get_kernel_rpl (); + regs.eflags =X86_EFLAGS_IF |X86_EFLAGS_SF |X86_EFLAGS_PF |0x2 ; + +/* Ok, create the new task.. */ +return do_fork (flags,0 , ®s,0 ,NULL ,NULL ); +} + +/* * Free current thread data structures etc.. */ void exit_thread (void )
include/asm-i386/processor.h +17 -0
@@ -472,6 +472,11 @@ extern void prepare_to_copy(struct task_*/ extern int kernel_thread (int (*fn)(void *),void * arg,unsigned long flags); +/* +* create an async thread: +*/ +extern int create_async_thread (long (*fn)(void *),void * arg,unsigned long flags); +extern unsigned long thread_saved_pc (struct task_struct *tsk);void show_trace (struct task_struct *task,struct pt_regs *regs,unsigned long *stack);
@@ -504,6 +509,18 @@ unsigned long get_wchan(struct task_stru#define KSTK_EIP (task) (task_pt_regs(task)->eip)#define KSTK_ESP (task) (task_pt_regs(task)->esp) +/* +* Register access methods for async syscall support. +* +* Note, task_stack_reg() must not be an lvalue, hence this macro: +*/ +#define task_stack_reg (t) \ + ({ unsigned long __esp = task_pt_regs(t)->esp; __esp; }) +#define set_task_stack_reg (t, new_stack) \ + do { task_pt_regs(t)->esp = (new_stack); } while (0) +#define task_ip_reg (t) task_pt_regs(t)->eip +#define task_ret_reg (t) task_pt_regs(t)->eax +struct microcode_header {unsigned int hdrver;
include/asm-i386/unistd.h +10 -0
@@ -1,6 +1,8 @@#ifndef _ASM_I386_UNISTD_H_#define _ASM_I386_UNISTD_H_ +#include <linux/linkage.h> +/* * This file contains the system call numbers. */
@@ -330,6 +332,14 @@#define NR_syscalls 320 +#ifndef __ASSEMBLY__ + +typedef asmlinkage long (*syscall_fn_t )(long ,long ,long ,long ,long ,long ); + +extern syscall_fn_t sys_call_table[NR_syscalls]; + +#endif +#define __ARCH_WANT_IPC_PARSE_VERSION#define __ARCH_WANT_OLD_READDIR#define __ARCH_WANT_OLD_STAT
PATCH 8/12 syslets: x86, add move_user_context() method
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
add the move_user_context() method to move the user-space context of one kernel thread to another kernel thread. User-space might notice the changed TID, but execution, stack and register contents (general purpose and FPU) are still the same.
An architecture must implement this interface before it can turn CONFIG_ASYNC_SUPPORT on.
--- arch/i386/kernel/process.c | 21 +++++++++++++++++++++ include/asm-i386/system.h | 7 +++++++ 2 files changed, 28 insertions(+)
arch/i386/kernel/process.c +21 -0
@@ -839,6 +839,27 @@ unsigned long get_wchan(struct task_stru }/* +* Move user-space context from one kernel thread to another. +* This includes registers and FPU state. Callers must make +* sure that neither task is running user context at the moment: +*/ +void +move_user_context (struct task_struct *new_task,struct task_struct *old_task) +{ +struct pt_regs *old_regs =task_pt_regs (old_task); +struct pt_regs *new_regs =task_pt_regs (new_task); +union i387_union *tmp; + + *new_regs = *old_regs; +/* +* Flip around the FPU state too: +*/ + tmp = new_task->thread.i387; + new_task->thread.i387 = old_task->thread.i387; + old_task->thread.i387 = tmp; +} + +/* * sys_alloc_thread_area: get a yet unused TLS descriptor index. */ static int get_free_idx (void )
include/asm-i386/system.h +7 -0
@@ -33,6 +33,13 @@ extern struct task_struct * FASTCALL(__s "2" (prev), "d" (next)); \ }while (0 ) +/* +* Move user-space context from one kernel thread to another. +* This includes registers and FPU state for now: +*/ +extern void +move_user_context (struct task_struct *new_task,struct task_struct *old_task); +#define _set_base (addr,base) do { unsigned long __pr; \ __asm__ __volatile__ ("movw %%dx,%1\n\t" \ "rorl $16,%%edx\n\t" \
PATCH 9/12 syslets: x86, mark async unsafe syscalls
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
mark clone() and fork() as not available for async execution. Both need an intact user context beneath them to work.
--- arch/i386/kernel/ioport.c | 6 ++++++ arch/i386/kernel/ldt.c | 3 +++ arch/i386/kernel/process.c | 6 ++++++ arch/i386/kernel/vm86.c | 6 ++++++ 4 files changed, 21 insertions(+)
arch/i386/kernel/ioport.c +6 -0
@@ -62,6 +62,9 @@ asmlinkage long sys_ioperm(unsigned longstruct tss_struct * tss;unsigned long *bitmap; +if (async_syscall (current)) +return -ENOSYS ; +if ((from + num <= from) || (from + num >IO_BITMAP_BITS ))return -EINVAL ;if (turn_on && !capable (CAP_SYS_RAWIO ))
@@ -139,6 +142,9 @@ asmlinkage long sys_iopl(unsigned long uunsigned int old = (regs->eflags >>12 ) &3 ;struct thread_struct *t = ¤t->thread; +if (async_syscall (current)) +return -ENOSYS ; +if (level >3 )return -EINVAL ;/* Trying to gain more privileges? */
arch/i386/kernel/ldt.c +3 -0
@@ -233,6 +233,9 @@ asmlinkage int sys_modify_ldt(int func, {int ret = -ENOSYS ; +if (async_syscall (current)) +return -ENOSYS ; +switch (func) {case 0 : ret =read_ldt (ptr, bytecount);
arch/i386/kernel/process.c +6 -0
@@ -750,6 +750,9 @@ struct task_struct fastcall * __switch_t asmlinkage int sys_fork(struct pt_regs regs) { +if (async_syscall (current)) +return -ENOSYS ; +return do_fork (SIGCHLD , regs.esp, ®s,0 ,NULL ,NULL ); }
@@ -759,6 +762,9 @@ asmlinkage int sys_clone(struct pt_regsunsigned long newsp;int __user *parent_tidptr, *child_tidptr; +if (async_syscall (current)) +return -ENOSYS ; + clone_flags = regs.ebx; newsp = regs.ecx; parent_tidptr = (int __user *)regs.edx;
arch/i386/kernel/vm86.c +6 -0
@@ -209,6 +209,9 @@ asmlinkage int sys_vm86old(struct pt_regstruct task_struct *tsk;int tmp, ret = -EPERM ; +if (async_syscall (current)) +return -ENOSYS ; + tsk = current;if (tsk->thread.saved_esp0) goto out;
@@ -239,6 +242,9 @@ asmlinkage int sys_vm86(struct pt_regs rint tmp, ret;struct vm86plus_struct __user *v86; +if (async_syscall (current)) +return -ENOSYS ; + tsk = current;switch (regs.ebx) {case VM86_REQUEST_IRQ :
PATCH 10/12 syslets: x86: enable ASYNC_SUPPORT
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
enable CONFIG_ASYNC_SUPPORT on x86.
--- arch/i386/Kconfig | 4 ++++ 1 file changed, 4 insertions(+)
arch/i386/Kconfig +4 -0
@@ -55,6 +55,10 @@ config ZONE_DMA bool default y +config ASYNC_SUPPORT + bool + default y + config SBUS bool
PATCH 11/12 syslets: x86, wire up the syslet system calls
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
wire up the new syslet / async system call syscalls and make it thus available to user-space.
--- arch/i386/kernel/syscall_table.S | 6 ++++++ include/asm-i386/unistd.h | 8 +++++++- 2 files changed, 13 insertions(+), 1 deletion(-)
arch/i386/kernel/syscall_table.S +6 -0
@@ -319,3 +319,9 @@ ENTRY(sys_call_table) .long sys_move_pages .long sys_getcpu .long sys_epoll_pwait + .long sys_async_exec /* 320 */ + .long sys_async_wait + .long sys_umem_add + .long sys_async_thread + .long sys_threadlet_on + .long sys_threadlet_off /* 325 */
include/asm-i386/unistd.h +7 -1
@@ -327,10 +327,16 @@#define __NR_move_pages 317#define __NR_getcpu 318#define __NR_epoll_pwait 319 +#define __NR_async_exec 320 +#define __NR_async_wait 321 +#define __NR_umem_add 322 +#define __NR_async_thread 323 +#define __NR_threadlet_on 324 +#define __NR_threadlet_off 325#ifdef __KERNEL__ -#define NR_syscalls 320 +#define NR_syscalls 326#ifndef __ASSEMBLY__
PATCH 12/12 syslets: x86_64: add syslet/threadlet support
In reply to Ingo MolnarFrom: Ingo Molnar <[email protected]>
add the arch specific bits of syslet/threadlet support to x86_64.
--- arch/x86_64/Kconfig | 4 ++ arch/x86_64/ia32/ia32entry.S | 20 ++++++++++- arch/x86_64/kernel/entry.S | 72 ++++++++++++++++++++++++++++++++++++++++- arch/x86_64/kernel/process.c | 11 ++++++ include/asm-x86_64/processor.h | 16 +++++++++ include/asm-x86_64/system.h | 12 ++++++ include/asm-x86_64/unistd.h | 29 +++++++++++++++- 7 files changed, 160 insertions(+), 4 deletions(-)
arch/x86_64/Kconfig +4 -0
@@ -36,6 +36,10 @@ config ZONE_DMA32 bool default y +config ASYNC_SUPPORT + bool + default y + config LOCKDEP_SUPPORT bool default y
arch/x86_64/ia32/ia32entry.S +19 -1
@@ -368,6 +368,14 @@ quiet_ni_syscall: PTREGSCALL stub32_vfork, sys_vfork, %rdi PTREGSCALL stub32_iopl, sys_iopl, %rsi PTREGSCALL stub32_rt_sigsuspend, sys_rt_sigsuspend, %rdx + /* + * sys_async_thread() and sys_async_exec() both take 2 parameters, + * none of which is ptregs - but the syscalls rely on being able to + * modify ptregs. So we put ptregs into the 3rd parameter - so it's + * unused and it also does not mess up the first 2 parameters: + */ + PTREGSCALL stub32_compat_async_exec, compat_sys_async_exec, %rdx + PTREGSCALL stub32_compat_async_thread, sys_async_thread, %rdx ENTRY(ia32_ptregs_common) popq %r11
@@ -394,6 +402,9 @@ END(ia32_ptregs_common) .section .rodata,"a" .align 8 +.globl compat_sys_call_table +compat_sys_call_table: +.globl ia32_sys_call_table ia32_sys_call_table: .quad sys_restart_syscall .quad sys_exit
@@ -714,9 +725,16 @@ ia32_sys_call_table: .quad compat_sys_get_robust_list .quad sys_splice .quad sys_sync_file_range - .quad sys_tee + .quad sys_tee /* 315 */ .quad compat_sys_vmsplice .quad compat_sys_move_pages .quad sys_getcpu .quad sys_epoll_pwait + .quad stub32_compat_async_exec /* 320 */ + .quad sys_async_wait + .quad sys_umem_add + .quad stub32_compat_async_thread + .quad sys_threadlet_on + .quad sys_threadlet_off /* 325 */ +.globl ia32_syscall_end ia32_syscall_end:
arch/x86_64/kernel/entry.S +71 -1
90 lines not on this page · raw message
arch/x86_64/kernel/process.c +11 -0
@@ -418,6 +418,17 @@ void release_thread(struct task_struct * } } +/* +* Move user-space context from one kernel thread to another. +* Callers must make sure that neither task is running user context +* at the moment: +*/ +void +move_user_context (struct task_struct *new_task,struct task_struct *old_task) +{ + *task_pt_regs (new_task) = *task_pt_regs (old_task); +} +static inline void set_32bit_tls (struct task_struct *t,int tls,u32 addr) {struct user_desc ud = {
include/asm-x86_64/processor.h +16 -0
@@ -322,6 +322,11 @@ extern void prepare_to_copy(struct task_extern long kernel_thread (int (*fn)(void *),void * arg,unsigned long flags);/* +* create an async thread: +*/ +extern long create_async_thread (long (*fn)(void *),void * arg,unsigned long flags); + +/* * Return saved PC of a blocked thread. * What is this good for? it will be always the scheduler or ret_from_fork. */
@@ -332,6 +337,17 @@ extern unsigned long get_wchan(struct ta#define KSTK_EIP (tsk) (task_pt_regs(tsk)->rip)#define KSTK_ESP (tsk) -1/* sorry. doesn't work for syscall. */ +/* +* Register access methods for async syscall support. +* +* Note, task_stack_reg() must not be an lvalue, hence this macro: +*/ +#define task_stack_reg (t) \ + ({ unsigned long __rsp = task_pt_regs(t)->rsp; __rsp; }) +#define set_task_stack_reg (t, new_stack) \ + do { task_pt_regs(t)->rsp = (new_stack); } while (0) +#define task_ip_reg (t) task_pt_regs(t)->rip +#define task_ret_reg (t) task_pt_regs(t)->raxstruct microcode_header {unsigned int hdrver;
include/asm-x86_64/system.h +11 -1
@@ -20,6 +20,8 @@#define __EXTRA_CLOBBER \ ,"rcx","rbx","rdx","r8","r9","r10","r11","r12","r13","r14","r15" +struct task_struct ; +/* Save restore flags to clear handle leaking NT */ #define switch_to (prev,next,last) \ asm volatile(SAVE_CONTEXT \
@@ -42,7 +44,15 @@ [thread_info] "i" (offsetof(struct task_struct, thread_info)), \[pda_pcurrent] " i" (offsetof(struct x8664_pda, pcurrent)) \ : " memory", " cc" __EXTRA_CLOBBER) - + + +/* +* Move user-space context from one kernel thread to another. +* This includes registers and FPU state for now: + */ +extern void +move_user_context(struct task_struct *new_task, struct task_struct *old_task); +extern void load_gs_index (unsigned );/*
include/asm-x86_64/unistd.h +28 -1
@@ -619,8 +619,21 @@ __SYSCALL(__NR_sync_file_range, sys_sync __SYSCALL(__NR_vmsplice, sys_vmsplice)#define __NR_move_pages 279 __SYSCALL(__NR_move_pages, sys_move_pages) +#define __NR_async_exec 280 +__SYSCALL(__NR_async_exec, stub_async_exec) +#define __NR_async_wait 281 +__SYSCALL(__NR_async_wait, sys_async_wait) +#define __NR_umem_add 282 +__SYSCALL(__NR_umem_add, sys_umem_add) +#define __NR_async_thread 283 +__SYSCALL(__NR_async_thread, stub_async_thread) +#define __NR_threadlet_on 284 +__SYSCALL(__NR_threadlet_on, sys_threadlet_on) +#define __NR_threadlet_off 285 +__SYSCALL(__NR_threadlet_off, sys_threadlet_off) -#define __NR_syscall_max __NR_move_pages +#define __NR_syscall_max __NR_threadlet_off +#define NR_syscalls __NR_syscall_max#ifndef __NO_STUBS#define __ARCH_WANT_OLD_READDIR
@@ -654,6 +667,20 @@ __SYSCALL(__NR_move_pages, sys_move_page#include <linux/types.h> #include <asm/ptrace.h> +typedef asmlinkage long (*syscall_fn_t )(long ,long ,long ,long ,long ,long ); + +extern syscall_fn_t sys_call_table[NR_syscalls]; + +#ifdef CONFIG_COMPAT + +extern syscall_fn_t compat_sys_call_table[]; +extern syscall_fn_t ia32_syscall_end; +extern syscall_fn_t ia32_sys_call_table; + +#define compat_NR_syscalls (&ia32_syscall_end - compat_sys_call_table) + +#endif +asmlinkage long sys_iopl (unsigned int level,struct pt_regs *regs);asmlinkage long sys_ioperm (unsigned long from,unsigned long num,int turn_on);struct sigaction ;
Quoted 3 lines · On Wed, 28 Feb 2007 22:39:38 +0100 Ingo Molnar <[email protected]> wrote:
- 32-bit user-space on 64-bit kernel compat support. 32-bit syslet and threadlet binaries work fine on 64-bit kernels.
Yay! :-)
--
Cheers,
Stephen Rothwell [email protected]
http://www.canb.auug.org.au/~sfr/
Hi Ingo,
Quoted 3 lines
this is the v5 release of the syslet/threadlet subsystem:
Nice!
I tried to port this to ppc64 but found a few problems:
The 64bit powerpc ABI has the concept of a TOC (r2) which is used for per function data. This means this wont work:
if (ah->restore_stack) {
set_task_stack_reg(new_task, ah->restore_stack);
task_ip_reg(new_task) = ah->restore_ip;
/*
* The return code 0 is needed to tell the
* head user-context that the threadlet went async:
*/
task_ret_reg(new_task) = 0;
}I think we would want to change restore_ip to restore_function, and then create a per arch helper, perhaps:
void set_user_context(struct task_struct *task, unsigned long stack,
unsigned long function, unsigned long retval);
ppc64 could then grab the ip and r2 values from the function descriptor.
The other issue involves the syscall table:
asmlinkage struct syslet_uatom __user * sys_async_exec(struct syslet_uatom __user *uatom,
struct async_head_user __user *ahu)
{
return __sys_async_exec(uatom, ahu, sys_call_table, NR_syscalls);
}This exposes the layout of the syscall table. Unfortunately it wont work on ppc64. In arch/powerpc/kernel/systbl.S:
#define COMPAT_SYS(func) .llong .sys_##func,.compat_sys_##func
Both syscall tables are overlaid.
Anton
Quoted 8 lines · Anton Blanchard
Hi Ingo,
Quoted 3 lines
this is the v5 release of the syslet/threadlet subsystem:
Nice!
I too went and downloaded patches-v5 for review.
First off, one problem I noticed in sys_async_wait:
+ ah->events_left = min_wait_events - (kernel_ring_idx - user_ring_idx);
This completely misses the wraparound case of kernel_ring_idx < user_ring_idx. I wonder if this is causing some of the benchmark problems? (add max_ring_index if kernel < user).
Quoted 4 lines
I tried to port this to ppc64 but found a few problems:
The 64bit powerpc ABI has the concept of a TOC (r2) which is used for per function data. This means this wont work:
[deleted]
Quoted 18 lines
I think we would want to change
restore_iptorestore_function, and then create a per arch helper, perhaps:void set_user_context(struct
task_struct*task, unsigned long stack,unsigned long function, unsigned long retval);ppc64 could then grab the ip and r2 values from the function descriptor.
The other issue involves the syscall table:
asmlinkage struct
syslet_uatom__user* sys_async_exec(structsyslet_uatom__user*uatom,
structasync_head_user__user*ahu)
{
return __sys_async_exec(uatom, ahu,sys_call_table,NR_syscalls);
}
Quoted 8 lines
This exposes the layout of the syscall table. Unfortunately it wont work on ppc64. In
arch/powerpc/kernel/systbl.S:#define
COMPAT_SYS(func).llong .sys_##func,.compat_sys_##funcBoth syscall tables are overlaid.
Anton
In addition, the entries in the table are not function pointers, they
are the actual code targets. So we need a arch helper to invoke the
system call.
Here is another problem with your compat code. Just telling user space that everything is u64 and having the kernel retrieve pointers and ulong doesn't work, you have to actually copy in u64 values and truncate them down. Your current code is broken on all 32bit big endian kernels. Actually, the check needs to be that the upper 32 bits are 0 or return -EINVAL.
In addition, the compat syscall entry points assume that the arguments have been truncated to compat_ulong values by the syscall entry path, and that they only need to do sign extension (and/or pointer conversion on s390 with its 31 bit pointers). So all compat kernels are broken.
The two of these things together makes me think we want two copy functions. At that point we may as well define the struct uatom in terms of ulong and compat_ulong for the compat_uatom. That would lead to two copies of exec_uatom, but the elimination of passing the syscall table as an argument down. The need_resched and signal check could become part of the common next_uatom routine, although it would need to know uatom+1 instead of doing the addition in itself.
Other observations:
All the logic setting at and async_ready is a bit hard to follow. After some analysis, t->at is only ever set to &t->__at and async_ready is only set to the same at or NULL. Both of these should become flags, and at->task should be converted to container_of. Also, the name at is hard to grep / search for.
The stop flags are decoded with a case but are not densely encoded, rather they are one hot. We either need to error on multiple stop bits being set, stop on each possible condition, or encode them densely.
There is no check for flags being set that are not recognized. If we ever add a flag for another stop condition this would lead to incorrect execution by the kernel.
There are some syscalls that can return -EFAULT but later have force_syscall_noerror. We should create a stop on ERROR and clear the force_noerror flag between syscalls. The umem_add syscall should add force_noerror if the put_user succeeds.
In copy_uatom, you call verify_read on the entire uatom. This means that the struct with all user space size has to be within the process limit, which violates your assertion that userspace doesn't need the whole structure. If we add the requirement that the space that would be occupied by the complete atom has to exist, then we can copy the whole struct uatom with copy_from_user and then copy the args with get_user. User space can still pack them more densely, and we can still stop copying on a null arg pointer. Actually, calling access_ok then __get_user can be more expensive on some architectures because they have to verify both start and length on access_ok but can only verify start on get_user because they have unmapped areas between user space and kernel space. This would also mean that we don't check arg_ptr for NULL without verifying that get_user actually worked. The gotos in exec_uatom are just a while loop with a break.
sys_umem_add should be in /lib under lib-y in the Makefile. In fact declaring the function weak does not make it a weak syscall implementation on some architectures.
Weak syscalls aliases to sys_ni_syscall are needed for when async support is not selected in Kconfig.
The Documentation patch does not explain threadlets. In fact, I had to read the old patch announcement to find out what a threadlet was and how the concept was different from a syslet.
Do we need 32 bits for the syscall number? I wonder if we
would be better served by a 16 bit syscall number and 16 bits reserved for future use.
The how does userspace actually find out about the error doing the completion in the syslet case? Its actually executing in an async_thread. The only case a return code is given to userspace is sys_async_thread, and those are mostly invisible to the thread.
I'm not sure if the api for threadlets is ready. Why does user space need scan for the stack being consumed? Can't it infer that by the return code of the sync task?
<partial-thought> It appears you expect restore_ip and restore_stack implement a minimal longjmp back to userspace. Instead of passing these explicitly, why not treat async_on as a setjmp and return to that point. As Anton pointed out, a simple stack pointer and next instruction pointer are not enough for all architectures. </partial-thought>
Why must the sp reg macro not be an lvalue? To enforce your stack clear api?
There seemed to be some copied comments in the x86-64 code (including some space vs tabs).
Is testing t->async_ready what we want to exclude system calls on? I thought that would catch callers who might go async on schedule()
instead of those running async in the background. I think we want
somthing more like t->at or whatever flag that becomes, saying that the thread is a background thread. Or maybe a combination.
milton