Skip to contentSkip to conversation

arch/xen is a bad idea

DISCUSSION Andi Kleen 14 participants 38 messages Dec 14, 2004 – Feb 25, 2005 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andi Kleen

author
#1 lore raw

"Andi Kleen" <[email protected]> writes:

[again this time with subject. sorry for the screwup] [very late answer]

Quoted 2 lines

Stunned silence I guess - merging an architecture is usually much more controversial ;)

In my opinion it's still an extremly bad idea to have arch/xen an own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.

Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.

I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right? Doing that cleanly in the existing architectures shouldn't be that hard.

I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.

And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in arch/xen will be very pretty.

Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works. Assuming there will be other para Hypervisors in the future too will we end up with even more virtual architectures? It would be much better to have at least a somewhat defined "linux virtual interface" first that is actually understood by multiple people outside the Xen group.

I think before merging stuff the hypervisor interfaces need to be discussed on linux-kernel. Splitting the patches and posting them as individual pieces for i386 with good description will be a good first step for that.

-Andi

#2 lore raw
Quoted 48 lines · Andi Kleen

"Andi Kleen" <[email protected]> writes:

[again this time with subject. sorry for the screwup] [very late answer]

Quoted 2 lines

Stunned silence I guess - merging an architecture is usually much more controversial ;)

In my opinion it's still an extremly bad idea to have arch/xen an own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.

Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.

I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right? Doing that cleanly in the existing architectures shouldn't be that hard.

I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.

And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in arch/xen will be very pretty.

Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works. Assuming there will be other para Hypervisors in the future too will we end up with even more virtual architectures? It would be much better to have at least a somewhat defined "linux virtual interface" first that is actually understood by multiple people outside the Xen group.

I think before merging stuff the hypervisor interfaces need to be discussed on linux-kernel. Splitting the patches and posting them as individual pieces for i386 with good description will be a good first step for that.

Review

Quoted 2 lines
(quoted diff) +0 -2
-Andi
-

Andi, there is at least one other hypervisor interface, mac-on-linux features a kernel module that allows booting other kernels inside the running one, and keeps very good speed anyways.

Their code, at least the user-space part, was also very good coded for my eyes.

Driver support is done by exporting a customized open-firmware device tree and then implementing drivers for these devices on the client OSs.

(just goto http://www.maconlinux.org/ and have a look)

Oh, this is obviusly ppc-only ATM :)

Signature · 5 lines
Greetz, Antonio Vargas aka winden of network

http://wind.codepixel.com/

Las cosas no son lo que parecen, excepto cuando parecen lo que si son.
#3 lore raw
In reply to Andi Kleen
Quoted 7 lines
Quoted 2 lines

Stunned silence I guess - merging an architecture is usually much more controversial ;)

In my opinion it's still an extremly bad idea to have arch/xen an own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.

Andi, I totally agree that merging into i386 could be a long term goal. However, its just not feasible right now. The changes required are way too intrusive. We put considerable effort into investigating this approach, but came to the conclusion that with the current structure of arch i386 it was going to be way too messy.

I really think the best approach is to get arch xen into mainstream Linux, and then work toward integrating i386, x86_64 and xen. From our point of view, the first stage of this is to increase the number of files that are shared unmodified between i386 and xen/i386 (i.e. linked from xen into i386). There's already many such files, but with a few relatively simple changes to i386 we could get quite a few more.

Quoted 6 lines

Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.

We're actually very well setup to handle this, having been doing it for some time. Whenever Linus issues a new mainstream patch, we have a script that rewrites the patch to duplicate the hunks that apply to i386 such that they also hit files that we've modified in xen/i386. This way we keep arch xen/i386 in perfect sync with i386. It takes discipline, but we're pretty good at it now.

Quoted 3 lines

I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right?

It's rather more complicated than that if you want a clean interface that gives good performance. We've taken a very benchmark-driven approach to minimise the overhead of virtualization. The aim is to have such a small overhead that people are happy running virtualized the whole time. I think this is a really important aim.

Quoted 2 lines

Doing that cleanly in the existing architectures shouldn't be that hard.

I've appended a list of some of the areas we need to modify. I think you may have underestimated what needs to happen.

Quoted 5 lines

I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.

We have subdirectories for the i386 and x86_64 specific files, along with a common directory for stuff which is shared between the two e.g. virtual interrupt control etc.

Quoted 2 lines

Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works.

We have an interface manual in the Xen bk repo, but I acknowledge that we haven't always been totally prompt in keeping it up to date and fully detailed. Now the Xen2 interface is frozen, that should be fixable. Even so, it hasn't prevented other independent groups porting OSes to Xen e.g. NetBSD, FreeBSD, Plan9.

Ian

Differences between i386 and xen/i386 files:
- irq setup
- pci bus scanning
- gdt/ldt must be in dedicated read-only pages
- gdt/ldt install/updates
- debug register updates
- pte quicklist/cache
- pmd/pgd are read-only pages
- highmem pte mapping
- dma memory allocation
- idle loop
- different fixmap
- *_to_* macros differ (like page_to_phys)
- *_val macros (pte_val, pmd_val)
- inb/outb
- switching cr3
- cr4 updates
- a few cpu flags need to be cleared
- msr access
- wbinvd call
- mtrr access
- io permission handling
- ioremap
- access to hardware memory
- interrupt enabling/disabling
- setup of trap gates
- tlb flush
- fpu stack switches
- user/kernel mode test
- different segment selectors since the kernel runs in ring1
- pagefault handler stack layout is different
- startup is in 32-bit mode
- start of day initialization is different
- start of day memory probing and pagetable setup
- trap/fault handling
- timer is virtualized
- smp addtl. cpu startup
- smp ipi's

author
#4 lore raw
Quoted 3 lines · Ian Pratt

I really think the best approach is to get arch xen into mainstream Linux, and then work toward integrating i386, x86_64 and xen. From our point of view, the first stage of this is to

I think that's definitely the wrong way. Also in Linux the standard strategy is to first clean up and restructure/refactor code and then merge, not the other way round.

Quoted 4 lines

increase the number of files that are shared unmodified between i386 and xen/i386 (i.e. linked from xen into i386). There's already many such files, but with a few relatively simple changes to i386 we could get quite a few more.

That will still have most of the disadvantages I listed.

Quoted 8 lines
Quoted 7 lines

Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.

Quoted 7 lines

We're actually very well setup to handle this, having been doing it for some time. Whenever Linus issues a new mainstream patch, we have a script that rewrites the patch to duplicate the hunks that apply to i386 such that they also hit files that we've modified in xen/i386. This way we keep arch xen/i386 in perfect sync with i386. It takes discipline, but we're pretty good at it now.

That won't anymore at some time. I found this out on x86-64. It works at the beginning, but eventually the code drift gets so much that it's near impossible to apply any hunk and you have to redo everything. One reason for that is that in Linux code often gets restructured, which makes it very difficult for such mechanized merging procedures to work long term.

Also you have to review every change anyways.

Quoted 10 lines
Quoted 3 lines

I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right?

It's rather more complicated than that if you want a clean interface that gives good performance. We've taken a very benchmark-driven approach to minimise the overhead of virtualization. The aim is to have such a small overhead that people are happy running virtualized the whole time. I think this is a really important aim.

Can you please be a bit more precise on that? What exactly do you need?

Quoted 6 lines
Quoted 3 lines

Doing that cleanly in the existing architectures shouldn't be that hard.

I've appended a list of some of the areas we need to modify. I think you may have underestimated what needs to happen.

Ok that's a start.

Quoted 10 lines
Quoted 6 lines

I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.

We have subdirectories for the i386 and x86_64 specific files, along with a common directory for stuff which is shared between the two e.g. virtual interrupt control etc.

Ok, but I think there will be significant differences in the 64bit part and meshing both together will get extremly messy.

What's your strategy for example to merge changes from arch/x86_64 to xen64? I don't think the way you described above will work in any way.

Quoted 11 lines
Quoted 3 lines

Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works.

We have an interface manual in the Xen bk repo, but I acknowledge that we haven't always been totally prompt in keeping it up to date and fully detailed. Now the Xen2 interface is frozen, that should be fixable. Even so, it hasn't prevented other independent groups porting OSes to Xen e.g. NetBSD, FreeBSD, Plan9.

Ian

Comments on some issues:

You are saying you ran benchmarks on each of them and it turns out to be too slow to emulate them in the standard architectural way? At least for some of them it's hard to believe.

Quoted 3 lines

Differences between i386 and xen/i386 files:
- irq setup

Already two different ways (MSI and non MSI). Adding a third is probably feasible.

Quoted 1 line

- pci bus scanning

We already have at least three different ways to do that, adding a fourth wouldn't be very messy I guess.

Quoted 1 line

- gdt/ldt must be in dedicated read-only pages

That can be probably done in the standard kernel
(without the actual read protection bit)

Quoted 1 line

- gdt/ldt install/updates

gdt load could be just trapped, it only happens once at startup.

LDT i can see some effort needed, but it's already only at a single function in a header file.

Quoted 1 line

- debug register updates

This shouldn't be performance critical. Why is it not simply emulated by the hypervisor?

Quoted 1 line

- pte quicklist/cache

i386 had a pte quicklist some time ago. I think there were plans anyways to readd it.

Quoted 1 line

- pmd/pgd are read-only pages

I suspect this just needs another include layer like the existing PAE/non PAE layer.

Quoted 3 lines

- highmem pte mapping
- dma memory allocation
- idle loop

This is already pluggable (e.g. ACPI has its own)

Quoted 3 lines

- different fixmap
- *_to_* macros differ (like page_to_phys)
- *_val macros (pte_val, pmd_val)

Can be also handled like 2/3 levels I bet.

Quoted 1 line

- inb/outb

Why is it not trapped?

Quoted 4 lines

- switching cr3
- cr4 updates
- a few cpu flags need to be cleared
- msr access

Why don't you just simply trap it? MSRs are normally not performance critical (except on x86-64 for the context switch but you didn't seem to have attacked that at all yet ;-)

Quoted 2 lines

- wbinvd call
- mtrr access

Same.

For all of these it would seem much cleaner to me to just provide instruction emulation in the Hypervisor.

Quoted 1 line

- io permission handling

What is different?

Quoted 5 lines

- ioremap
- access to hardware memory
- interrupt enabling/disabling
- setup of trap gates
- tlb flush

This is already well separated, shouldn't be a big issue to make it different.

Quoted 2 lines

- fpu stack switches
- user/kernel mode test

That can be just changed everywhere in i386 without ill effects.

Quoted 3 lines

- different segment selectors since the kernel runs in ring1
- pagefault handler stack layout is different
- startup is in 32-bit mode

That's a good cleanup anyways. The early boot interface between 16 and 32bit should be clearly defined and then a general 32bit booting interface be defined. There are various other people who would like to have this too. I had some plans to do the equivalent on x86-64 with a 64bit boot interface.

Quoted 2 lines

- start of day initialization is different
- start of day memory probing and pagetable setup

Sounds similar to EFI.

Quoted 2 lines

- trap/fault handling
- timer is virtualized

Details?

First impression is that there is much cleanup potential and that there should be probably some discussion on each of these items.

-Andi

#5 lore raw
Quoted 3 lines · Andi Kleen

I think that's definitely the wrong way. Also in Linux the standard strategy is to first clean up and restructure/refactor code and then merge, not the other way round.

Must be two different Linux OS's out there then. I thought it was
- get interfaces right
- get working
- get correct
- merge
- evolve

If we did it as you describe we'd still not have a SATA layer merged for example.

author
#6 lore raw
Quoted 7 lines · Alan Cox
Quoted 3 lines · Andi Kleen

I think that's definitely the wrong way. Also in Linux the standard strategy is to first clean up and restructure/refactor code and then merge, not the other way round.

Must be two different Linux OS's out there then. I thought it was
- get interfaces right

That is exactly the part that is wrong currently imho. The arch/xen interface is a mess and in its current form unlikely to be maintainable.

-Andi

#7 lore raw
Quoted 2 lines · Andi Kleen

That is exactly the part that is wrong currently imho. The arch/xen interface is a mess and in its current form unlikely to be maintainable.

It seems maintainable and well documented to me. I just don't see where your problem is with this. The kernel/hypervisor interface is clear, and the arch/xen code seems quite sane.

author
#8 lore raw
Quoted 7 lines · Alan Cox
Quoted 2 lines · Andi Kleen

That is exactly the part that is wrong currently imho. The arch/xen interface is a mess and in its current form unlikely to be maintainable.

It seems maintainable and well documented to me. I just don't see where your problem is with this. The kernel/hypervisor interface is clear, and the arch/xen code seems quite sane.

The main problem I see is that it is source code copy, and especially when they both support i386 and x86-64 there will be no sane way to keep it all synchronized with the i386 and x86-64 code bases. It's already hard from a single source like I can attest from x86-64, with two sources it will be likely much more difficult longer term. I just can't see it working well in practice. It will be also nasty for people doing changes because they will need to duplicate i386+x86_64 changes four times in the worst case (i386,x86_64,xen32,xen64)

I guess it may be acceptable if we were maintaining obscure Lance drivers this way ;_), but for a important architecture it just doesn't seem like the right approach to me.

Also e.g. for non performance critical
things like changing MTRRs or debug registers it would be IMHO much cleaner to just emulate the instructions (the ISA is very well defined) and not change the kernel here. From a look at Ian's list the majority of the changes needed for Xen actually fall into this category.

I suspect when the kernel is only changed for the truly performance critical interfaces that cannot be efficiently emulated (like idle/timers/page table updates) the required changes for the para virtualization will become much more manageable and can be cleanly integrated into the respective ports.

And as Pavel points out first merging arch/xen and then migrating into i386 and x86_64 like it was proposed sounds extremly hard and is probably not really practical.

-Andi

#9 lore raw
Quoted 6 lines · Andi Kleen

Also e.g. for non performance critical
things like changing MTRRs or debug registers it would be IMHO much cleaner to just emulate the instructions (the ISA is very well defined) and not change the kernel here. From a look at Ian's list the majority of the changes needed for Xen actually fall into this category.

There are so many problems in snooping and decoding instructions it isn't funny. Aside from the mmap pci buffer half way through instruction that will emulate type stuff there are a lot of awkward issues if you want to emulate multiple mtrr sets (you need PAT).

Xen has tried the decode/patch stuff earlier - see the early NPTL handling and it was neither pretty nor reliable.

author
#10 lore raw
Quoted 10 lines · Alan Cox
Quoted 6 lines · Andi Kleen

Also e.g. for non performance critical
things like changing MTRRs or debug registers it would be IMHO much cleaner to just emulate the instructions (the ISA is very well defined) and not change the kernel here. From a look at Ian's list the majority of the changes needed for Xen actually fall into this category.

There are so many problems in snooping and decoding instructions it isn't funny. Aside from the mmap pci buffer half way through instruction

Hmm? From what I remember reading the code of the Xen hypervisor they are already emulating a lot of instructions (e.g. take a look at xen/arch/x86/x86_32/seg_fixup.c) Emulating some more doesn't seem like a big issue to me.

Quoted 2 lines

that will emulate type stuff there are a lot of awkward issues if you want to emulate multiple mtrr sets (you need PAT).

No way different from doing it with a magic interface. MTRR is exposed to kernel subsystems (a lot of things in the service OS will want to play with it) and to user space (with also a lot of users). The magic interface will need to provide a full MTRR interface anyways. The only difference would be basically how you tell the hypervisor to change them.

[Not saying that PAT is actually needed for this, but as a general comment:]
Also using PAT for that would be definitely a good idea, MTRRs just have so many problems that any step away from them is a good thing.

-Andi

#11 lore raw
Quoted 7 lines
Quoted 2 lines

There are so many problems in snooping and decoding instructions it isn't funny. Aside from the mmap pci buffer half way through instruction

Hmm? From what I remember reading the code of the Xen hypervisor they are already emulating a lot of instructions (e.g. take a look at xen/arch/x86/x86_32/seg_fixup.c) Emulating some more doesn't seem like a big issue to me.

There's actually no emulation in Xen at all. seg_fixup is the closest we come, which is a tiny decoder that is able to work out the effective address of an instruction. It's only through grim necessity we need this in order to be able to make NPTL thread local storage accesses work (-ve segment offsets). We'd much prefer that glibc was slightly tweaked such that we didn't have to do this (it would be a lot faster too).

Anyhow, I really don't think emulation is the issue. The handful of cases that you pointed out that could relatively easily be emulated constitute a tiny fraction of the arch xen patch. Even so, you have to be a bit careful from a performance point of
view: loading debug registers can occur quite frequently, and its much quicker to be able to load them as a batch rather than taking individual faults.

Ian

#12 lore raw
In reply to Andi Kleen
Quoted 4 lines · Andi Kleen

And as Pavel points out first merging arch/xen and then migrating
into i386 and x86_64 like it was proposed sounds extremly hard and is probably not really practical.

It will also create more patchwork and will make the the change history more obscure and harder to follow.

It is a bit pervers to be putting something into the tree with the intention of ripping it up and redoing it in the near future. Why not do it up-front and keep the change history cleaner? From Ian's change list and Andi's comments it doesn't look to me that this is all as much work as people seem to be implying?

I guess if we were to go the way which Ian is proposing it would be

a) Add arch/xen

b) Spend N weeks integrating xen into arch/i386, while also separately maintaining arch/xen.

c) Remove arch/xen

So... why not skip a), c) and half of b)?

#13 lore raw
Quoted 10 lines · Andrew Morton

I guess if we were to go the way which Ian is proposing it would be

a) Add arch/xen

b) Spend N weeks integrating xen into arch/i386, while also separately maintaining arch/xen.

c) Remove arch/xen

So... why not skip a), c) and half of b)?

Because until xen is in the kernel it won't get good exposure Because nobody is sure that b) needs to occur or would occur for another year our two (Xen3)

Xen is not "a slightly x86 arch variant" (and a slight ia64 arch variant) Xen is quite different in its design and its performance comes from exactly that - paravirtualisation by being a virtual CPU that isn't a true x86 is very very fast.

b is a long term research project it is not a "clean this up next month" project as I understand it.

#14 lore raw
In reply to Andrew Morton
Quoted 11 lines

I guess if we were to go the way which Ian is proposing it would be

a) Add arch/xen

b) Spend N weeks integrating xen into arch/i386, while also separately maintaining arch/xen.

c) Remove arch/xen

So... why not skip a), c) and half of b)?

That's not quite what I'm proposing.

Once arch/xen is in the tree, we'd start submitting patches that try and unify more of arch xen and i386 to reduce the number of files that we have to modify.

I think we'd then have to make a decision as to whether merging i386 and xen/i386 is feasible. Further, we'd have to make a decision as to whether what is really wanted is a single kernel that's able to boot-time switch between native i386 and xen, rather than just having a single source base. The two options would probably result in rather different implementations.

In short, merging is far from trivial, and its not even clear quite what is wanted.

I'm not convinced that maintaining xen/i386 in its current form is going to be as hard as Andi thinks. We already share many files unmodified from i386. Keeping i386 and xen/i386 in sync is fairly mechanical: we can apply most of the patches to i386 to xen/i386 directly.

Ian

#15 lore raw
Quoted 4 lines · Ian Pratt

I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.

How are they shared? Inclusion, symlinks, makefile references or copy-n-paste?

#16 lore raw
Quoted 7 lines
Quoted 4 lines · Ian Pratt

I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.

How are they shared? Inclusion, symlinks, makefile references or

The makefile creates symlinks.

Ian

#17 lore raw
Quoted 9 lines · Ian Pratt
Quoted 7 lines
Quoted 4 lines · Ian Pratt

I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.

How are they shared? Inclusion, symlinks, makefile references or

The makefile creates symlinks.

Which is really very nice. If you update arch/i386/ then arch/xen/ automatically gets the update too.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
author
#18 lore raw
In reply to Ian Pratt
Quoted 9 lines · Ian Pratt
Quoted 7 lines
Quoted 4 lines · Ian Pratt

I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.

How are they shared? Inclusion, symlinks, makefile references or

The makefile creates symlinks.

That's deprecated because it breaks with separate objdirs (make O=/other/dir) See the x86-64 makefiles on how to do it properly.

-Andi

#19 lore raw
Quoted 7 lines · Ian Pratt
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines · Ian Pratt

I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

How are they shared? Inclusion, symlinks, makefile references or

The makefile creates symlinks.

That's deprecated because it breaks with separate objdirs (make O=/other/dir) See the x86-64 makefiles on how to do it properly.

Ah, I wasn't previously aware of the O= capability to know that we were breaking it.

We can trivially change over to referencing files in arch/i386 directly from the Makefile. The only downside is that it will make navigating the source slightly harder.

Ian

#20 lore raw
In reply to Ian Pratt
Quoted 8 lines · Rumor has it that on Thu, Dec 16, 2004 at 09:00:52PM +0000 Ian Pratt said:

I think we'd then have to make a decision as to whether merging i386 and xen/i386 is feasible. Further, we'd have to make a decision as to whether what is really wanted is a single kernel that's able to boot-time switch between native i386 and xen, rather than just having a single source base. The two options would probably result in rather different implementations.

The boot-time switch seems to be the ideal. This would allow enterprise Linux vendors to support using Xen w/o having to deal with a whole archicture release (including install kernel etc.). One of the (really strong IMO) arguments for full virtualization methods is beinag able to run stock OSes. At least for Linux guests, a Xen merge with the boot-time option would remove that argument.

Of course, the resultant kernel changes would need to have no measurable impact on native i386... and be possible...

Cheers,

Phil

#21 lore raw
Quoted 3 lines · Philip R Auld

The boot-time switch seems to be the ideal. This would allow enterprise Linux vendors to support using Xen w/o having to deal with a whole archicture release (including install kernel

I have no idea how such a boot-time switch would work for 3rd party device drivers, though, so don't count yourself lucky just yet ;)

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#22 lore raw
Quoted 9 lines · Rumor has it that on Thu, Dec 16, 2004 at 06:08:24PM -0500 Rik van Riel said:
Quoted 3 lines · Philip R Auld

The boot-time switch seems to be the ideal. This would allow enterprise Linux vendors to support using Xen w/o having to deal with a whole archicture release (including install kernel

I have no idea how such a boot-time switch would work for 3rd party device drivers, though, so don't count yourself lucky just yet ;)

I actually meant _OS_ vendors like you ;)

Cheers,

Phil

Quoted 5 lines
Signature · 4 lines
-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
author
#23 lore raw
In reply to Ian Pratt
Quoted 10 lines · Ian Pratt
Quoted 10 lines

I guess if we were to go the way which Ian is proposing it would be

a) Add arch/xen

b) Spend N weeks integrating xen into arch/i386, while also separately maintaining arch/xen.

c) Remove arch/xen

Quoted 7 lines
Quoted 1 line

So... why not skip a), c) and half of b)?

That's not quite what I'm proposing.

Once arch/xen is in the tree, we'd start submitting patches that try and unify more of arch xen and i386 to reduce the number of files that we have to modify.

IMHO reducing the number of files to modify should be done first before attempting a mainline merge. That is how a lot of other more intrusive changes have been merged in the past.

There was a intensive review process and eventually the interfaces were cleaned up a lot of and the thing was merged piece by piece after considerable changes.

Quoted 7 lines

I think we'd then have to make a decision as to whether merging i386 and xen/i386 is feasible. Further, we'd have to make a decision as to whether what is really wanted is a single kernel that's able to boot-time switch between native i386 and xen, rather than just having a single source base. The two options would probably result in rather different implementations.

Even without boot time switch but only CONFIG switch a more shared code base would be probably the only saner way long time. I'm not sure a run time switch is really that good an idea, it would essentially require to run most of pgtable.h etc. through function pointers and I don't like this idea very much. But at least doing the reorganization on the source code level would probably clean up the interface considerably.

Quoted 9 lines

In short, merging is far from trivial, and its not even clear quite what is wanted.

I'm not convinced that maintaining xen/i386 in its current form is going to be as hard as Andi thinks. We already share many files unmodified from i386. Keeping i386 and xen/i386 in sync is fairly mechanical: we can apply most of the patches to i386 to xen/i386 directly.

Perhaps for now, but eventually this will not work anymore due to inevitable code drift (I went exactly through the same phase for a year or so on x86-64. These days I have to redo most patches unless i'm lucky enough that the contributor does both)

And with also supporting x86-64 it will not really work this way well because you would need to merge from two sources. I can't see any mechanical scheme handling that. The only sane way to maintain the merging scheme you proposed would be then arch/xen64, multiplying work again.

-Andi

#24 lore raw
In reply to Ian Pratt

Hi!

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Stunned silence I guess - merging an architecture is usually much more controversial ;)

In my opinion it's still an extremly bad idea to have arch/xen an own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.

Quoted 6 lines

Andi, I totally agree that merging into i386 could be a long term goal. However, its just not feasible right now. The changes required are way too intrusive. We put considerable effort into investigating this approach, but came to the conclusion that with the current structure of arch i386 it was going to be way too messy.

Okay, what about this one:

You merge xen hooks in mainline, but keep maintaining arch/xen out-of-tree? You have to maintain it yourself, anyway, and having hooks merged should make it easy.

When xen is merged into i386 (you said that is your long-term goal anyway), you can merge that into mainline...
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#25 lore raw
Quoted 3 lines

You merge xen hooks in mainline, but keep maintaining arch/xen out-of-tree? You have to maintain it yourself, anyway, and having hooks merged should make it easy.

Having the hooks merged would certainly help.

However, I think the main benefit of being in the mainline tree would be that it would make it easy for other developers to see how arch xen works and thus begin to take us into account when considering changes. If there's going to be a big effort to merge i386, x86_64 and xen going forward, it will certainly help if more people have looked through our code. We'll only get that kind of exposure to developers if we're in the mainstream tree.

The other key area is that our top priority is to decrease the number of files we need to modified from standard i386. For this to happen, we need to submit patches into i386 that abstract a few things behind macros/constants. For example, we'd like to abstract the test to see whether the CPU is in the kernel or not (we run the kernel in ring 1 rather than 0). If arch xen is in the tree, this kind of patch will make rather more sense to people.

Quoted 3 lines

BTW if you merge xen as separate architecture, it will be *very* hard to merge it back to i386. That patch would be huge, and would need to go in "atomically".

I don't see it like that. While continuing to track changes in i386/x86_64, we'd restructure the code under arch xen such that it could build (or even boot) time switch between running native and over Xen. At some point the arch directory could then be renamed. This would be a big project, and one that would involve a lot more people than just the Xen team. Because the x86 architecture is so critical to Linux, I just can't see this happening unless the development takes place in the mainstream tree.

Of course, another reason for inclusion in mainstream is that there's a lot of momentum behind Xen in both the user and developer communities. My guess is that virtualization is going to be pretty pervasive in the data centre, and its one of the key technologies that Linux can win on. The paravirtualized approach used by Xen is going to offer better performance than full x86 virtualization, even if there's extra support in the processor.

Cheers,
Ian

#26 lore raw

Hi!

Quoted 10 lines
Quoted 3 lines

BTW if you merge xen as separate architecture, it will be *very* hard to merge it back to i386. That patch would be huge, and would need to go in "atomically".

I don't see it like that. While continuing to track changes in i386/x86_64, we'd restructure the code under arch xen such that it could build (or even boot) time switch between running native and over Xen. At some point the arch directory could then be renamed. This would be a big project, and one that would involve a lot more people than just the Xen team. Because the x86

So you plan to

merge arch/xen

then modify arch/xen to do all arch/i386 can do

then rm -rf arch/i386, mv arch/xen arch/i386? Well, I'd say that's rather ambitious plan....
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
author
#27 lore raw
In reply to Ian Pratt
Quoted 8 lines · Ian Pratt

The other key area is that our top priority is to decrease the number of files we need to modified from standard i386. For this to happen, we need to submit patches into i386 that abstract a few things behind macros/constants. For example, we'd like to abstract the test to see whether the CPU is in the kernel or not (we run the kernel in ring 1 rather than 0). If arch xen is in the tree, this kind of patch will make rather more sense to people.

That would be a good first step, especially if it results in cleanups. Please go for it.

Quoted 5 lines

I don't see it like that. While continuing to track changes in i386/x86_64, we'd restructure the code under arch xen such that it could build (or even boot) time switch between running native and over Xen. At some point the arch directory could then be renamed. This would be a big project, and one that would involve

This sounds like a massive duplication of effort. You would need to do all that work on arch/xen and in parallel on the native port for the slow merge, and in parallel track a changing target and keep the code usable in mainline.

-Andi

#28 lore raw
In reply to Pavel Machek
Quoted 9 lines · Pavel Machek

Okay, what about this one:

You merge xen hooks in mainline, but keep maintaining arch/xen out-of-tree? You have to maintain it yourself, anyway, and having hooks merged should make it easy.

When xen is merged into i386 (you said that is your long-term goal anyway), you can merge that into mainline...
Pavel

Assuming this is practical and would let xen get exposure sooner rather than later, it would be nice if the hooks could be used for other projects. My impression is that the merge part would come after there is some extensive experience with the operation "in the wild" so to speak. Who knows what operating systems might run in this way.

Signature · 3 lines
   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me
#29 lore raw
Quoted 3 lines · Bill Davidsen

Assuming this is practical and would let xen get exposure sooner rather than later, it would be nice if the hooks could be used for other projects.

Who knows what operating systems might run in this way.

NetBSD and Plan 9 already run on Xen 2.0. IIRC FreeBSD and one of the other BSDs are being ported, too...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#31 lore raw

Re: arch/xen clue?

Quoted 4 lines

For those of us who are clueless as to even what arch/xen is :) Where would be a good place to read and become informed?

http://xen.sf.net

There's a bunch of papers and documentation etc.

Ian

#32 lore raw

Re: arch/xen clue?

In reply to Dorn Hetzel
Quoted 3 lines · Dorn Hetzel

For those of us who are clueless as to even what arch/xen is :) Where would be a good place to read and become informed?

http://www.cl.cam.ac.uk/Research/SRG/netos/xen/index.html

Best regards,

Anton

Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
#34 lore raw
In reply to Ian Pratt

Hi!

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

Stunned silence I guess - merging an architecture is usually much more controversial ;)

In my opinion it's still an extremly bad idea to have arch/xen an own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.

Quoted 6 lines

Andi, I totally agree that merging into i386 could be a long term goal. However, its just not feasible right now. The changes required are way too intrusive. We put considerable effort into investigating this approach, but came to the conclusion that with the current structure of arch i386 it was going to be way too messy.

BTW if you merge xen as separate architecture, it will be *very* hard to merge it back to i386. That patch would be huge, and would need to go in "atomically".

Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#35 lore raw
In reply to Andi Kleen
Quoted 8 lines · Andi Kleen

I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy. And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in arch/xen will be very pretty.

I have an inkling that the xen implementors may have plots of additional architecture ports. If it really is x86/x86-64 -only it isn't worth bothering with a separate arch/ dir, but it would be if it were ported to a large number of architectures.

Maybe they're modelling it after UML which has its own arch/ dir but then grabs assorted things from other architectures; in that event I wouldn't consider it so misguided.

Probably best if the implementors chimed in about all this.

-- wli

#36 lore raw
Quoted 10 lines · Andi Kleen
Quoted 8 lines

I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy. And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in arch/xen will be very pretty.

Quoted 4 lines

I have an inkling that the xen implementors may have plots of additional architecture ports. If it really is x86/x86-64 -only it isn't worth bothering with a separate arch/ dir, but it would be if it were ported to a large number of architectures.

For non x86 architectures, the changes required are typically much less, and it makes sense to integrate them into the normal architecture tree. This is what has been done with ia64, and will be done with ppc. i386 and x86_64 are a special case where the extensive changes required justify having an arch xen directory with i386 and x86_64 subtrees beneath it (although many files are just linked from the normal architecture).

Quoted 3 lines

Maybe they're modelling it after UML which has its own arch/ dir but then grabs assorted things from other architectures; in that event I wouldn't consider it so misguided.

Yep, that's the model, for the x86 family.

Ian

#37 lore raw
In reply to Andi Kleen
Quoted 3 lines · Andi Kleen

In my opinion it's still an extremly bad idea to have arch/xen
an own architecture.

Guys, I'd like to kick this a bit further down the road. Things still seem to be somewhat deadlocked.

To summarise my understanding:

The Xen team still believe that it's best to keep arch/xen, arch/xen/i386, arch/xen/x86_64, etc. And I believe that Andi (who is the world expert on maintaining an i386 derivative) thinks that this is will be a long-term maintenance problem.

I tend to agree with Andi, and I'm not sure that the Xen team fully appreciate the downside of haveing an own-architecture in the kernel.org kernel and the upside of having their code integrated with the most-maintained architecture. It could be that the potential problems haven't been sufficiently well communicated.

Christian has mentioned that Xen would need to hook into the i386 code in ~60 places, which is somewhat more than Ian's 37-bullet-point list.

I get the impression that the Xen team are overly reluctant to make changes to the arch/i386 code and to arch-neutral kernel code. Don't do that - new abstractions, refactoring and generally moving things about is generally a safe thing to do, and can often make things better anyway.

So. Has anyone changed position or otherwise converged? How do we get this resolved?

#38 lore raw

kernel 2.6.8-24.11-smp errors

Hello,

I am using SuSE 9.2 Pro with kernel uname -a:

Linux 2.6.8-24.11-smp #1 SMP Fri Jan 14 13:01:26 UTC 2005 x86_64 x86_64 x86_64 GNU/Linux

Intel(R) Xeon(TM) CPU 2.80GHz

2 GB RAM
1 GB SWAP

and these strange kernel messages keep appearing::

it is a high load machine with iptables running. The /proc/net/ip_conntrack number keeps running higher and does not get less. After a while the maximum is reached. th e only way to solve this is stop iptables, unload the ip modules from the kernel, load them again and start iptables again. Then the messages will not come again if I repeat this frequently. Could it be a kernel bug??? Or maybe something misconfigured???

dmesg:

swapper: page allocation failure. order:1, mode:0x20

Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
      <ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
      <ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
      <ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
      <ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
      <ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
      <ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80133c91>{activate_task+129}
      <ffffffff80137a89>{autoremove_wake_function+9}
<ffffffff80132633>{__wake_up_common+67}
      <ffffffff80132c13>{__wake_up+67} <ffffffff802e2344>{sock_def_readable+68}
      <ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
      <ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
      <ffffffff8030592e>{ip_rcv_finish+574} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff80305e34>{ip_rcv+1188}
      <ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
      <ffffffff802e85b4>{net_rx_action+132}
<ffffffff8013dca1>{__do_softirq+113}
      <ffffffff8013dd55>{do_softirq+53} <ffffffff80113e3f>{do_IRQ+335}
      <ffffffff8010f540>{mwait_idle+0} <ffffffff80110cf5>{ret_from_intr+0}
       <EOI> <ffffffff8010f596>{mwait_idle+86} <ffffffff8010f9ea>{cpu_idle+26}
      <ffffffff804e971a>{start_kernel+490} <ffffffff804e91e0>{_sinittext+480}

swapper: page allocation failure. order:1, mode:0x20

Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
      <ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
      <ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
      <ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
      <ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
      <ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
      <ffffffff802f3a43>{nf_hook_slow+195} <ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8013340a>{task_rq_lock+74}
<ffffffff80133f9b>{try_to_wake_up+747}
      <ffffffff8030ad6b>{ip_queue_xmit+1211}
<ffffffff8013340a>{task_rq_lock+74}
      <ffffffff8013340a>{task_rq_lock+74}
<ffffffff80132633>{__wake_up_common+67}
      <ffffffff80132c13>{__wake_up+67}
<ffffffff80312996>{tcp_ack_saw_tstamp+22}
      <ffffffff80314d90>{tcp_ack+1040} <ffffffff80320b79>{tcp_v4_do_rcv+233}
      <ffffffffa017b034>{:ipt_state:match+36}
<ffffffff803214cb>{tcp_v4_rcv+1835}
      <ffffffff80305fb9>{ip_local_deliver_finish+297}
<ffffffff80305e90>{ip_local_deliver_finish+0}
      <ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff80305e90>{ip_local_deliver_finish+0}
      <ffffffff803062fe>{ip_local_deliver+622}
<ffffffff8030592e>{ip_rcv_finish+574}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff802f3a43>{nf_hook_slow+195} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff80305e34>{ip_rcv+1188} <ffffffff802e9749>{netif_receive_skb+729}
      <ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
<ffffffff80133c91>{activate_task+129}
      <ffffffff80133f9b>{try_to_wake_up+747}
<ffffffff802e85b4>{net_rx_action+132}
      <ffffffff8013dca1>{__do_softirq+113} <ffffffff8013dd55>{do_softirq+53}
      <ffffffff80113e3f>{do_IRQ+335} <ffffffff8010f540>{mwait_idle+0}
      <ffffffff80110cf5>{ret_from_intr+0}  <EOI>
<ffffffff8010f596>{mwait_idle+86}
      <ffffffff8010f9ea>{cpu_idle+26} <ffffffff804e971a>{start_kernel+490}
      <ffffffff804e91e0>{_sinittext+480}

swapper: page allocation failure. order:1, mode:0x20

Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
      <ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
      <ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
      <ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
      <ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
      <ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
      <ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
      <ffffffff802f3a43>{nf_hook_slow+195} <ffffffff80308b90>{ip_dst_output+0}
      <ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80133c91>{activate_task+129}
      <ffffffff8013340a>{task_rq_lock+74}
<ffffffff80132633>{__wake_up_common+67}
      <ffffffff80313e1b>{tcp_fastretrans_alert+859}
<ffffffff8031536d>{tcp_ack+2541}
      <ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
      <ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
      <ffffffff8030592e>{ip_rcv_finish+574} <ffffffff803056f0>{ip_rcv_finish+0}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff802f3a43>{nf_hook_slow+195}
      <ffffffff803056f0>{ip_rcv_finish+0} <ffffffff80305e34>{ip_rcv+1188}
      <ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
      <ffffffff80132c13>{__wake_up+67} <ffffffff802e85b4>{net_rx_action+132}
      <ffffffff8013dca1>{__do_softirq+113} <ffffffff8013dd55>{do_softirq+53}
      <ffffffff80113e3f>{do_IRQ+335} <ffffffff8010f540>{mwait_idle+0}
      <ffffffff80110cf5>{ret_from_intr+0}  <EOI>
<ffffffff8010f596>{mwait_idle+86}
      <ffffffff8010f9ea>{cpu_idle+26} <ffffffff804e971a>{start_kernel+490}
      <ffffffff804e91e0>{_sinittext+480}