fs, gs purpose & multicore prog
Hello,
I've some questions about IA-32e in Linux.
- What is the FS and GS segments role inside the kernel ? (I was thinking about thread local storage)
- When I do a "mov %fs ..." instruction (in a module), it seems that %fs is equal to 0 (idem for %gs). Are these registers not always filled ?
- What is the purpose of MSR_FS_BASE and MSR_GS_BASE ? (I thought they were filled with "gdt[fs_entry].base")
- My last question is about the kernel programation of multi-core (or multiprocessor)
architecture. I don't see a lot of documentation about that on Internet. Do you have some docs/urls about this topic.
Maybe someone can briefly explain how the execution flow are given to the different cores.
Thanks in advance.
Eric
Quoted 7 lines · Eric Lacombe
Hello,
I've some questions about IA-32e in Linux.
- What is the FS and GS segments role inside the kernel ? (I was thinking about thread local storage)
In a 32-bit kernel %fs is the base of the per-cpu data area. In a 64-bit kernel %gs points to the pda (processor data area). The pda is a single structure, whereas per-cpu data is a section that per-cpu variables get put into.
Quoted 3 lines
- When I do a "mov %fs ..." instruction (in a module), it seems that %fs is equal to 0 (idem for %gs). Are these registers not always filled ?
On 32-bit they will always have a value, or you'll get a GPF. On 64-bit the value of the selector doesn't matter because the MSRs are the real content.
Quoted 3 lines
- What is the purpose of
MSR_FS_BASEandMSR_GS_BASE? (I thought they were filled with "gdt[fs_entry].base")
On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it only stores the low 32-bits. When you load a segment register with a selector, it picks up from the gdt. If you want a full 64-bit offset, you need to write it to the msr.
Quoted 7 lines
- My last question is about the kernel programation of multi-core (or multiprocessor)
architecture. I don't see a lot of documentation about that on Internet. Do you have some docs/urls about this topic.
Maybe someone can briefly explain how the execution flow are given to the different cores.
To the kernel they're all just cpus, and it runs tasks on them as usual. There are a few tweaks in the scheduler to pay attention to the shared caches and so on.
J
Hi,
Thanks for your answers. I've some new questions now ;)
Quoted 9 lines · On Thursday 04 September 2008 18:49:42 Jeremy Fitzhardinge wrote:
Quoted 2 lines · Eric Lacombe
- What is the FS and GS segments role inside the kernel ? (I was thinking about thread local storage)
In a 32-bit kernel %fs is the base of the per-cpu data area. In a 64-bit kernel %gs points to the pda (processor data area). The pda is a single structure, whereas per-cpu data is a section that per-cpu variables get put into.
So, %gs is not used in 32-bit kernel and %fs is not used in 64-bit kernel. Is it right ? Why there were different choices of design ?
Quoted 7 lines
Quoted 3 lines
- When I do a "mov %fs ..." instruction (in a module), it seems that %fs is equal to 0 (idem for %gs). Are these registers not always filled ?
On 32-bit they will always have a value, or you'll get a GPF. On 64-bit the value of the selector doesn't matter because the MSRs are the real content.
ok, but what about the limits and access types?
Quoted 8 lines
Quoted 3 lines
- What is the purpose of
MSR_FS_BASEandMSR_GS_BASE? (I thought they were filled with "gdt[fs_entry].base")On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it only stores the low 32-bits. When you load a segment register with a selector, it picks up from the gdt. If you want a full 64-bit offset, you need to write it to the msr.
Ok, I just saw that a 64-bit base in segment descriptor is only available for the system descriptor.
Quoted 11 lines
Quoted 7 lines
- My last question is about the kernel programation of multi-core (or multiprocessor)
architecture. I don't see a lot of documentation about that on Internet. Do you have some docs/urls about this topic.
Maybe someone can briefly explain how the execution flow are given to the different cores.To the kernel they're all just cpus, and it runs tasks on them as usual. There are a few tweaks in the scheduler to pay attention to the shared caches and so on.
Ok, but how does the kernel technically run tasks on different processor (or core)? My question was ambiguous, I was not assuming that I knew how multiprocessor works.
Thanks again.
Eric
Quoted 2 lines
J
Quoted 3 lines · Eric Lacombe
So, %gs is not used in 32-bit kernel and %fs is not used in 64-bit kernel. Is it right ? Why there were different choices of design ?
They both use the opposite from what their respective usermodes use for thread local storage, since there may be a slight performance advantage to doing so (at least on 32-bit). 32-bit's use of %gs for usermode TLS is old and arbitrary, probably predating x86-64 (at least its wide availability). The use of %gs for kernel per-cpu data is architectural on 64-bit, because of the "swapgs" instruction; there's no "swapfs" instruction.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
- When I do a "mov %fs ..." instruction (in a module), it seems that %fs is equal to 0 (idem for %gs). Are these registers not always filled ?
On 32-bit they will always have a value, or you'll get a GPF. On 64-bit the value of the selector doesn't matter because the MSRs are the real content.
ok, but what about the limits and access types?
They don't exist in 64-bit. The GDT contains them, but they're not enforced.
Quoted 12 lines
Quoted 8 lines
Quoted 3 lines
- What is the purpose of
MSR_FS_BASEandMSR_GS_BASE? (I thought they were filled with "gdt[fs_entry].base")On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it only stores the low 32-bits. When you load a segment register with a selector, it picks up from the gdt. If you want a full 64-bit offset, you need to write it to the msr.
Ok, I just saw that a 64-bit base in segment descriptor is only available for the system descriptor.
Yes, the IDT has double-wide entries to fit 64-bit values, but they didn't extend that to the GDT. Or something - I last looked at this a couple of months ago, and it never sticks in my brain for long.
Quoted 4 lines
Ok, but how does the kernel technically run tasks on different processor (or core)? My question was ambiguous, I was not assuming that I knew how multiprocessor works.
That's a very broad question. A good proportion of the core kernel code is dedicated to doing just that. Very roughly, at boot time it brings up all the cpus, and they more or less run independently each looking for work to do in the form of processes waiting to run on the run queue. They collectively run the scheduler algorithms to work out who runs what when; almost everything run in the kernel is a task - both usermode processes and kernel threads. Except for the stuff which isn't.
J
Thanks again ;)
Quoted 1 line · On Vendredi 05 Septembre 2008 16:51:39 Jeremy Fitzhardinge wrote:
Eric Lacombe wrote:
[...]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
- What is the purpose of
MSR_FS_BASEandMSR_GS_BASE? (I thought they were filled with "gdt[fs_entry].base")On 64-bit, the GDT isn't large enough to hold a 64-bit offset, so it only stores the low 32-bits. When you load a segment register with a selector, it picks up from the gdt. If you want a full 64-bit offset, you need to write it to the msr.
Quoted 6 lines
Quoted 2 lines
Ok, I just saw that a 64-bit base in segment descriptor is only available for the system descriptor.
Yes, the IDT has double-wide entries to fit 64-bit values, but they didn't extend that to the GDT. Or something - I last looked at this a couple of months ago, and it never sticks in my brain for long.
It seems, there also are the TSS and LDT.
Quoted 5 lines
Quoted 4 lines
Ok, but how does the kernel technically run tasks on different processor (or core)? My question was ambiguous, I was not assuming that I knew how multiprocessor works.
Quoted 7 lines
That's a very broad question. A good proportion of the core kernel code is dedicated to doing just that. Very roughly, at boot time it brings up all the cpus, and they more or less run independently each looking for work to do in the form of processes waiting to run on the run queue. They collectively run the scheduler algorithms to work out who runs what when; almost everything run in the kernel is a task - both usermode processes and kernel threads. Except for the stuff which isn't.
I know these things ;) but what I wanted to know is the "x86 architectural details". In fact, I saw that during the machine init the BIOS select a cpu on the bus to be the BSP (bootstrap proc). The others are then the APs (Appli proc). Then the kernel runs on the BSP. What I wonder is how the kernel gives execution flow to the APs.
Thanks.
Eric
Quoted 2 lines
J
Quoted 6 lines · Eric Lacombe
I know these things ;) but what I wanted to know is the "x86 architectural details". In fact, I saw that during the machine init the BIOS select a cpu on the bus to be the BSP (bootstrap proc). The others are then the APs (Appli proc). Then the kernel runs on the BSP. What I wonder is how the kernel gives execution flow to the APs.
See arch/x86/kernel/smpboot.c (esp do_boot_cpu()), and chapter 7 of "Intel® 64 and IA-32 Architectures Software Developer’s Manual", volume 3a.
J