Explaination of system call function flow in TT mode

Hi,Blaisorblade:

I have studied the TT mode of UML source code 2.6.7 for some time.But I still can't work out the system call function flow in TT mode.I have read some documents and comments on that but all of them are very rough...

Is there any more detailed document about the system call function flow in UML TT mode?(had better to the function level)

Thanks a lot!

Alex

do_syscall is called, and then this is done:

                               sig = SIGUSR2;
                               tracing = 0;

... after, this saves the new tracing value inside "task", which is a struct task_struct.

                      set_tracing(task, tracing);

Quoted 8 lines · Hi,Blaisorblade. You

do_syscall is called, and then this is done:

                               sig = SIGUSR2;
                               tracing = 0;

... after, this saves the new tracing value inside "task", which is a struct task_struct.

                      set_tracing(task, tracing);

But what's the exact meaning of the task in the tracer function? It is defined as:

task = cpu_tasks[proc_id].task

Is it the task struct of the traced thread?

What's the difference between "task->pid" and
"task->thread.mode.tt.extern_pid"?

I know in TT mode every thread in UML
will has a corresponding thread in host.

But I don't know the exact
relationship between them.

And I didn't found how UML creat the
corresponding thread in host.

Thanks a lot!

Quoted 8 lines · Alex LIU

Hi,Blaisorblade:

I have studied the TT mode of UML source code 2.6.7 for some time.But I still can't work out the system call function flow in TT mode.I have read some documents and comments on that but all of them are very rough...

Is there any more detailed document about the system call function flow in UML TT mode?(had better to the function level)

I don't know if the slides on the main site could be of help for you (I don't think so, but you might try).

However, I've decided to post a thorough description of the flow to the list and to you... To be correct, I've studied the source while writing this mail... I had a rough idea of what happens, I just didn't dig enough to discover all details because I didn't need it yet.

First study man 2 ptrace, especially about PTRACE_SYSCALL.

However, the core mechanism is that tracer() ptrace()s the child:

(around line 235 of
arch/um/kernel/tt/tracer.c, sorry but references are from around 2.6.9, it should not be too difficult).

while(1){ //this is executed for the whole lifetime of the child.
CATCH_EINTR(pid = waitpid(-1, &status, WUNTRACED));
...

                else if(WIFSTOPPED(status)){
...
                        sig = WSTOPSIG(status);
...
                        switch(sig){

and when the tracee executes a syscall, as explained in ptrace docs,
waitpid() will return that the child was stopped by a SIGTRAP, so we get here:

case SIGTRAP: //this has changed in recent kernels to case (SIGTRAP + 0x80):

do_syscall is called, and then this is done:

                                sig = SIGUSR2;
                                tracing = 0;

... after, this saves the new tracing value inside "task", which is a struct task_struct.

                       set_tracing(task, tracing);

afterwards, the set value of sig is used so:

//cont_type is normally set to PTRACE_SYSCALL, but since now tracing == 0, it will be PTRACE_CONT.

                       if(ptrace(cont_type, pid, 0, sig) != 0){

}

and this makes sure with ptraces that the child sees a SIGUSR2 signal when resuming. (it's not done through kill(), see sig = SIGUSR2 and the ptrace() call near the end using it.) Since the signal is sent this way, it will be received and handled by the child thread. Now, we are resuming with PTRACE_CONT, because we are going to execute the UML code which will handle the syscall, so we don't want syscalls to be intercepted.

Then, the SIGUSR2 signal handler is invoked (it's sig_handler_common_tt which calls sig_info[SIGUSR2]-> handler, i.e. usr2_handler). It will call syscall_handler_tt, which will do the syscall execution (with tracing turned off) and saves the actual result (through SC_SET_SYSCALL_RETURN(sc, result), which manipulates the saved registers, specifically the value which will be stored back in EAX).

Finally, to switch back to the user mode, during the return path of sig_handler_common_tt(), set_user_mode(NULL) is called; if it sees that tracing is 0 (what it reads is the value set by set_tracing()) it sends a SIGUSR1 signal:

int set_user_mode(void *t)

{
        struct task_struct *task;

        task = t ? t : current;
        if(task->thread.mode.tt.tracing)
                return(1);
        task->thread.request.op = OP_TRACE_ON;
        os_usr1_process(os_getpid()); /*this is a wrapper for the kill() to
send the signal.*/
        return(0);
}

Now, this signal is handled by tracer(): in fact, the child (who gets the signal) is ptraced, so the ptracer can examine each signal and decide what to do. (Above, for SIGUSR2, we said there was an exception, but it happened because the signal was sent using ptrace()).

Here is the piece of code:

                        switch(sig){
                        case SIGUSR1:
                                sig = 0; // so the child won't see the signal.
                                op = do_proc_op(task, proc_id);
                                switch(op){
                                case OP_TRACE_ON:
                                        arch_leave_kernel(task, pid);
                                        tracing = 1;
                                        break;

As you see, tracing is switched back to 1, so at the end this iteration we will resume the child with PTRACE_SYSCALL in user mode... and he will see the syscall return value. I hope I didn't miss anything.

Quoted 3 lines

Thanks a lot!

Alex

--
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade

-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Hi,

Could you explain what happens in SKAS mode, then? How a UML kenel in SKAS mode handles the UML system calls differently from one in TT mode? i tried to understand from the code, but i had hard time to understand.

In SKAS mode, instead, the UML scheduler switches back and forth between various "processes" (inside UML) which are all running inside userspace, and ptrace the various host threads (actually collapsed into one userspace thread, thanks to SKAS mode) which run only the real userspace code. So the ptraced context always executes userspace code and we never turn ptracing off.
--
Paolo Giarrusso, aka Blaisorblade
Skype user "PaoloGiarrusso"
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade

Hi, Thanks for the reply. Let me have one following question.

As my understand, in SKAS mode, tracer and tracee are different host processes, and they have UML kernel code and application code respectively. (in TT mode, tracee has both kernel and application code)

Suppose the tracee attmpts to invoke a host system call. then, it will be intercepted by the tracer as in TT mode. (In TT mode, the tracer turns off tracing and sends SIGUSR2 to the tracee, and then, the signal handler in the tracee will execute the system call.)

but, in
SKAS mode, because the tracee doesn't have the UML kernel code, only the tracer knows about the system call code. The tracer cannot simply send a signal to make the tracee execute the system call. Then how does it do? Does the tracer execute the system call for the tracee?

Ok, now i'm getting close. Thanks a lot!!!
but i have two more :)

1) in SKAS mode, the tracer (UML kernel) will execute a system call for the tracee (the user process). But what if the system call blocks? then, the tracer, the UML kernel, will be blocked, too, right? it means the whole UML system will be blocked?

2) in TT mode, what if the user process (tracee) installs its own signal handler for SIGUSR2?

Quoted 7 lines · Young Koh

Ok, now i'm getting close. Thanks a lot!!!
but i have two more :)

1) in SKAS mode, the tracer (UML kernel) will execute a system call for the tracee (the user process). But what if the system call blocks? then, the tracer, the UML kernel, will be blocked, too, right? it means the whole UML system will be blocked?

Good question. However the tracee is one of the threads running inside UML, i.e. switch_to_skas switches between the different tracees. However, that's done only on timer interrupts, and possibly those interrupts are blocked by the syscall execution.

Indeed, I remember that when the UBD I/O is run by the UML kernel and not by a separate thread (this has been removed) the whole system becomes *really* slow, so probably the answer to your question is "Yes, but UML tries to avoid this".

Quoted 2 lines

2) in TT mode, what if the user process (tracee) installs its own signal handler for SIGUSR2?

The syscall is intercepted so it can't install signal handlers; the one it installs are handled inside the UML kernel.
--
Paolo Giarrusso, aka Blaisorblade
Skype user "PaoloGiarrusso"
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade

the tracer (UML kernel) saves and restores the tracee's registers using ptrace() in order to get syscall paramaters and save the return value. while doing this, the tracer would need to care about only six registers(eax, ebx, ecx, edx, esi, and edi) used for the syscall parameter passing. I think, the tracer doesn't have to save all the registers of the tracee, because the tracer will execute the syscall for the tracee and the context of tracee will not be affected.

but, in arch/um/kernel/skas/process.c, move_registers() function gets and sets even floating point registers, which i don't think will be affected by the system call execution.
i tested and ran a UML kernel with the second ptrace() (used for floating point registers) commented out, it seems to work. (probably only for SKAS mode)

Quoted 7 lines · Young Koh

the tracer (UML kernel) saves and restores the tracee's registers using ptrace() in order to get syscall paramaters and save the return value. while doing this, the tracer would need to care about only six registers(eax, ebx, ecx, edx, esi, and edi) used for the syscall parameter passing. I think, the tracer doesn't have to save all the registers of the tracee, because the tracer will execute the syscall for the tracee and the context of tracee will not be affected.

This is true, but it's easier to get all of the GP registers than it is to selectively grab the syscall-related ones.

Quoted 6 lines

but, in arch/um/kernel/skas/process.c, move_registers() function gets and sets even floating point registers, which i don't think will be affected by the system call execution.
i tested and ran a UML kernel with the second ptrace() (used for floating point registers) commented out, it seems to work. (probably only for SKAS mode)

Yeah, this is reasonable. You have to be careful that you save and restore any registers that might be used by one of the stubs, but they don't use FP.

Jeff

Quoted 3 lines

Yeah, this is reasonable. You have to be careful that you save and restore any registers that might be used by one of the stubs, but they don't use FP.

I also thought about not saving FP-regs on each kernel entry. But if you do this optimization, you need to save / restore FP-regs on switch_to. Also you need to get the FP-regs when setting up a signal-handler stackframe. And they have to be restored on sys_(rt_)sigreturn from the values found in the stackframe.

As s390 can give me regs and FP-regs in one single ptrace call, I decided to not do this optimization. Maybe for i386 and x86_64 it would be worth the cost.

Bodo

Quoted 2 lines

Jeff

Quoted 9 lines

Quoted 3 lines

Yeah, this is reasonable. You have to be careful that you save and restore any registers that might be used by one of the stubs, but they don't use FP.

I also thought about not saving FP-regs on each kernel entry. But if you do this optimization, you need to save / restore FP-regs on switch_to. Also you need to get the FP-regs when setting up a signal-handler stackframe. And they have to be restored on sys_(rt_)sigreturn from the values found in the stackframe.

Thanks for the replies. let me understand them. In a SKAS mode UML kernel, the pseudo code of a system call invocation would be like the following.

wait4(); // wait until a user process raises a syscall save_registers(); // copy the user process' registers to UML kernel space execute_syscall(); // execute the syscall in UML kernel context restore_registers(); // copy the user registers back to the user process

but, during execute_syscall(), even if its stubs use all the registers and/or it happens to invoke switch_to(), it will happen all in the UML kernel's context. that means the user process's context will be protected by host kernel's context switching mechanism. doesn't it? because the host kernel will automatically save/restore a process' registers when the process is stopped and resumed, all we care about in the above routine should be to get system call parameters from the tracee and save the return value to it. shouldn't it? (again, only for SKAS)

Quoted 19 lines · Young Koh

Quoted 9 lines

Quoted 3 lines

Yeah, this is reasonable. You have to be careful that you save and restore any registers that might be used by one of the stubs, but they don't use FP.

I also thought about not saving FP-regs on each kernel entry. But if you do this optimization, you need to save / restore FP-regs on switch_to. Also you need to get the FP-regs when setting up a signal-handler stackframe. And they have to be restored on sys_(rt_)sigreturn from the values found in the stackframe.

Thanks for the replies. let me understand them. In a SKAS mode UML kernel, the pseudo code of a system call invocation would be like the following.

wait4(); // wait until a user process raises a syscall save_registers(); // copy the user process' registers to UML kernel space execute_syscall(); // execute the syscall in UML kernel context restore_registers(); // copy the user registers back to the user process

Quoted 9 lines

but, during execute_syscall(), even if its stubs use all the registers and/or it happens to invoke switch_to(), it will happen all in the UML kernel's context. that means the user process's context will be protected by host kernel's context switching mechanism. doesn't it? because the host kernel will automatically save/restore a process' registers when the process is stopped and resumed, all we care about in the above routine should be to get system call parameters from the tracee and save the return value to it. shouldn't it? (again, only for SKAS)

Wait a moment, we have two cases:

1) For the syscall execution, everything should be ok (apart the case the syscall code needs to explicitly access the guest's registers).

2) switch_to is used for process context switching, so is a different thing from syscall execution. It's called by schedule() (which is called, mainly, both by syscalls which do explicitly scheduling and by the timer tick; there are also some additional cases when using preemption, but UML does not use them; look for cond_resched() and similar things in kernel sources).

In this case, instead, for SKAS mode, we must switch the registers manually, since we have collapsed everything in one host process. For TT and SKAS0 mode, instead, it's not needed, right Jeff and Bodo?

In this case, instead, for SKAS mode, we must switch the registers manually, since we have collapsed everything in one host process. For TT and SKAS0 mode, instead, it's not needed, right Jeff and Bodo?

Thanks for the replies. let me understand them. In a SKAS mode UML kernel, the pseudo code of a system call invocation would be like the following.

wait4(); // wait until a user process raises a syscall save_registers(); // copy the user process' registers to UML kernel space execute_syscall(); // execute the syscall in UML kernel context restore_registers(); // copy the user registers back to the user process

but, during execute_syscall(), even if its stubs use all the registers and/or it happens to invoke switch_to(), it will happen all in the UML kernel's context. that means the user process's context will be protected by host kernel's context switching mechanism. doesn't it? because the host kernel will automatically save/restore a process' registers when the process is stopped and resumed, all we care about in the above routine should be to get system call parameters from the tracee and save the return value to it. shouldn't it? (again, only for SKAS)

I also thought about not saving FP-regs on each kernel entry. But if you do this optimization, you need to save / restore FP-regs on switch_to. Also you need to get the FP-regs when setting up a signal-handler stackframe. And they
have to be restored on sys_(rt_)sigreturn from the values found in the stackframe.

I hope, I didn't miss some other places that would need adaption.

Quoted 5 lines · Bodo Stroesser

I also thought about not saving FP-regs on each kernel entry. But if you do this optimization, you need to save / restore FP-regs on switch_to. Also you need to get the FP-regs when setting up a signal-handler stackframe. And they have to be restored on sys_(rt_)sigreturn from the values found in the stackframe.

True, but these are much less frequent than kernel entries/exits.

Quoted 3 lines

I hope, I didn't miss some other places that would need adaption.

Offhand, this sounds right.

Jeff

Quoted 8 lines · Jeff Dike

Quoted 6 lines · Bodo Stroesser

I also thought about not saving FP-regs on each kernel entry. But if you do this optimization, you need to save / restore FP-regs on switch_to. Also you need to get the FP-regs when setting up a signal-handler stackframe. And they
have to be restored on sys_(rt_)sigreturn from the values found in the stackframe.

Quoted 1 line

True, but these are much less frequent than kernel entries/exits.

Yes, so it makes sense to optimize this for i386 and x86_64.

On s390, I'm not sure, what to do. Using s390's PTRACE_PEEK/POKEUSR_AREA UML can read or write all regs including FP in a single ptrace call. So, the change would speed up that call a bit, but would that be enough to pay the cost of additional ptrace calls in switch_to or signal handling?

Suppose the tracee attmpts to invoke a host system call. then, it will be intercepted by the tracer as in TT mode. (In TT mode, the tracer turns off tracing and sends SIGUSR2 to the tracee, and then, the signal handler in the tracee will execute the system call.) but, in SKAS mode, because the tracee doesn't have the UML kernel code, only the tracer knows about the system call code. The tracer cannot simply send a signal to make the tracee execute the system call. Then how does it do? Does the tracer execute the system call for the tracee?

Quoted 8 lines · Alex LIU

Hi,Blaisorblade:

I have studied the TT mode of UML source code 2.6.7 for some time.But I still can't work out the system call function flow in TT mode.I have read some documents and comments on that but all of them are very rough...

Is there any more detailed document about the system call function flow in UML TT mode?(had better to the function level)

I don't know if the slides on the main site could be of help for you (I don't think so, but you might try).

However, I've decided to post a thorough description of the flow to the list and to you... To be correct, I've studied the source while writing this mail... I had a rough idea of what happens, I just didn't dig enough to discover all details because I didn't need it yet.

First study man 2 ptrace, especially about PTRACE_SYSCALL.

However, the core mechanism is that tracer() ptrace()s the child:

(around line 235 of
arch/um/kernel/tt/tracer.c, sorry but references are from around 2.6.9, it should not be too difficult).

while(1){ //this is executed for the whole lifetime of the child.
CATCH_EINTR(pid = waitpid(-1, &status, WUNTRACED));
...

                else if(WIFSTOPPED(status)){
...
                        sig = WSTOPSIG(status);
...
                        switch(sig){

and when the tracee executes a syscall, as explained in ptrace docs,
waitpid() will return that the child was stopped by a SIGTRAP, so we get here:

case SIGTRAP: //this has changed in recent kernels to case (SIGTRAP + 0x80):

do_syscall is called, and then this is done:

                                sig = SIGUSR2;
                                tracing = 0;

... after, this saves the new tracing value inside "task", which is a struct task_struct.

                       set_tracing(task, tracing);

afterwards, the set value of sig is used so:

//cont_type is normally set to PTRACE_SYSCALL, but since now tracing == 0, it will be PTRACE_CONT.

                       if(ptrace(cont_type, pid, 0, sig) != 0){

}

and this makes sure with ptraces that the child sees a SIGUSR2 signal when resuming. (it's not done through kill(), see sig = SIGUSR2 and the ptrace() call near the end using it.) Since the signal is sent this way, it will be received and handled by the child thread. Now, we are resuming with PTRACE_CONT, because we are going to execute the UML code which will handle the syscall, so we don't want syscalls to be intercepted.

Then, the SIGUSR2 signal handler is invoked (it's sig_handler_common_tt which calls sig_info[SIGUSR2]-> handler, i.e. usr2_handler). It will call syscall_handler_tt, which will do the syscall execution (with tracing turned off) and saves the actual result (through SC_SET_SYSCALL_RETURN(sc, result), which manipulates the saved registers, specifically the value which will be stored back in EAX).

Finally, to switch back to the user mode, during the return path of sig_handler_common_tt(), set_user_mode(NULL) is called; if it sees that tracing is 0 (what it reads is the value set by set_tracing()) it sends a SIGUSR1 signal:

int set_user_mode(void *t)

{
        struct task_struct *task;

        task = t ? t : current;
        if(task->thread.mode.tt.tracing)
                return(1);
        task->thread.request.op = OP_TRACE_ON;
        os_usr1_process(os_getpid()); /*this is a wrapper for the kill() to
send the signal.*/
        return(0);
}

Now, this signal is handled by tracer(): in fact, the child (who gets the signal) is ptraced, so the ptracer can examine each signal and decide what to do. (Above, for SIGUSR2, we said there was an exception, but it happened because the signal was sent using ptrace()).

Here is the piece of code:

                        switch(sig){
                        case SIGUSR1:
                                sig = 0; // so the child won't see the signal.
                                op = do_proc_op(task, proc_id);
                                switch(op){
                                case OP_TRACE_ON:
                                        arch_leave_kernel(task, pid);
                                        tracing = 1;
                                        break;

As you see, tracing is switched back to 1, so at the end this iteration we will resume the child with PTRACE_SYSCALL in user mode... and he will see the syscall return value. I hope I didn't miss anything.

Quoted 3 lines

Thanks a lot!

Alex

--
Paolo Giarrusso, aka Blaisorblade
Linux registered user n. 292729
http://www.user-mode-linux.org/~blaisorblade

-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting Tool for open source databases. Create drag-&-drop reports. Save time by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc. Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

I have a question about system call implementation in tt mode. it seems that an invoked system call will be executed in tracee's context. (the tracer sends SIGUSR2 to the tracee and tracee executes the system call in usr2_handler) however, 'current' macro, which is used to find the current task structure, would only work in Kernel mode stack (which i assume is the tracer's stack), cause the current task structure is at the bottom of the kernel stack. then, how the system calls can be executed in tracee's context?