arch/xen is a bad idea
"Andi Kleen" <[email protected]> writes:
[again this time with subject. sorry for the screwup] [very late answer]
Quoted 2 lines
Stunned silence I guess - merging an architecture is usually much more controversial ;)
In my opinion it's still an extremly bad idea to have arch/xen an own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.
Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.
I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right? Doing that cleanly in the existing architectures shouldn't be that hard.
I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.
And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in arch/xen will be very pretty.
Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works. Assuming there will be other para Hypervisors in the future too will we end up with even more virtual architectures? It would be much better to have at least a somewhat defined "linux virtual interface" first that is actually understood by multiple people outside the Xen group.
I think before merging stuff the hypervisor interfaces need to be discussed on linux-kernel. Splitting the patches and posting them as individual pieces for i386 with good description will be a good first step for that.
-Andi
Quoted 48 lines · Andi Kleen
"Andi Kleen" <[email protected]> writes:
[again this time with subject. sorry for the screwup] [very late answer]
Quoted 2 lines
Stunned silence I guess - merging an architecture is usually much more controversial ;)
In my opinion it's still an extremly bad idea to have
arch/xenan own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.
I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right? Doing that cleanly in the existing architectures shouldn't be that hard.
I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.
And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in
arch/xenwill be very pretty.Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works. Assuming there will be other para Hypervisors in the future too will we end up with even more virtual architectures? It would be much better to have at least a somewhat defined "linux virtual interface" first that is actually understood by multiple people outside the Xen group.
I think before merging stuff the hypervisor interfaces need to be discussed on linux-kernel. Splitting the patches and posting them as individual pieces for i386 with good description will be a good first step for that.
Review
Quoted 2 lines
(quoted diff) +0
-2-Andi -
Andi, there is at least one other hypervisor interface, mac-on-linux features a kernel module that allows booting other kernels inside the running one, and keeps very good speed anyways.
Their code, at least the user-space part, was also very good coded for my eyes.
Driver support is done by exporting a customized open-firmware device tree and then implementing drivers for these devices on the client OSs.
(just goto http://www.maconlinux.org/ and have a look)
Oh, this is obviusly ppc-only ATM :)
Signature · 5 lines
Greetz, Antonio Vargas aka winden of network http://wind.codepixel.com/ Las cosas no son lo que parecen, excepto cuando parecen lo que si son.
Quoted 7 lines
Quoted 2 lines
Stunned silence I guess - merging an architecture is usually much more controversial ;)
In my opinion it's still an extremly bad idea to have
arch/xenan own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.
Andi, I totally agree that merging into i386 could be a long term goal. However, its just not feasible right now. The changes required are way too intrusive. We put considerable effort into investigating this approach, but came to the conclusion that with the current structure of arch i386 it was going to be way too messy.
I really think the best approach is to get arch xen into mainstream Linux, and then work toward integrating i386, x86_64 and xen. From our point of view, the first stage of this is to increase the number of files that are shared unmodified between i386 and xen/i386 (i.e. linked from xen into i386). There's already many such files, but with a few relatively simple changes to i386 we could get quite a few more.
Quoted 6 lines
Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.
We're actually very well setup to handle this, having been doing it for some time. Whenever Linus issues a new mainstream patch, we have a script that rewrites the patch to duplicate the hunks that apply to i386 such that they also hit files that we've modified in xen/i386. This way we keep arch xen/i386 in perfect sync with i386. It takes discipline, but we're pretty good at it now.
Quoted 3 lines
I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right?
It's rather more complicated than that if you want a clean interface that gives good performance. We've taken a very benchmark-driven approach to minimise the overhead of virtualization. The aim is to have such a small overhead that people are happy running virtualized the whole time. I think this is a really important aim.
Quoted 2 lines
Doing that cleanly in the existing architectures shouldn't be that hard.
I've appended a list of some of the areas we need to modify. I think you may have underestimated what needs to happen.
Quoted 5 lines
I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.
We have subdirectories for the i386 and x86_64 specific files, along with a common directory for stuff which is shared between the two e.g. virtual interrupt control etc.
Quoted 2 lines
Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works.
We have an interface manual in the Xen bk repo, but I acknowledge that we haven't always been totally prompt in keeping it up to date and fully detailed. Now the Xen2 interface is frozen, that should be fixable. Even so, it hasn't prevented other independent groups porting OSes to Xen e.g. NetBSD, FreeBSD, Plan9.
Ian
Differences between i386 and xen/i386 files:
- irq setup
- pci bus scanning
- gdt/ldt must be in dedicated read-only pages
- gdt/ldt install/updates
- debug register updates
- pte quicklist/cache
- pmd/pgd are read-only pages
- highmem pte mapping
- dma memory allocation
- idle loop
- different fixmap
- *_to_* macros differ (like page_to_phys)
- *_val macros (pte_val, pmd_val)
- inb/outb
- switching cr3
- cr4 updates
- a few cpu flags need to be cleared
- msr access
- wbinvd call
- mtrr access
- io permission handling
- ioremap
- access to hardware memory
- interrupt enabling/disabling
- setup of trap gates
- tlb flush
- fpu stack switches
- user/kernel mode test
- different segment selectors since the kernel runs in ring1
- pagefault handler stack layout is different
- startup is in 32-bit mode
- start of day initialization is different
- start of day memory probing and pagetable setup
- trap/fault handling
- timer is virtualized
- smp addtl. cpu startup
- smp ipi's
Quoted 3 lines · Ian Pratt
I really think the best approach is to get arch xen into mainstream Linux, and then work toward integrating i386,
x86_64and xen. From our point of view, the first stage of this is to
I think that's definitely the wrong way. Also in Linux the standard strategy is to first clean up and restructure/refactor code and then merge, not the other way round.
Quoted 4 lines
increase the number of files that are shared unmodified between i386 and xen/i386 (i.e. linked from xen into i386). There's already many such files, but with a few relatively simple changes to i386 we could get quite a few more.
That will still have most of the disadvantages I listed.
Quoted 8 lines
Quoted 7 lines
Currently it's already difficult enough to get people to add fixes to both i386 and x86-64, adding fixes to three or rather four (xen32 and xen64) architectures will be quite bad. In practice we'll likely get much worse code drift and missing fixes. Also I still suspect Ian is underestimating how much work it is long term to keep an Linux architecture uptodate.
Quoted 7 lines
We're actually very well setup to handle this, having been doing it for some time. Whenever Linus issues a new mainstream patch, we have a script that rewrites the patch to duplicate the hunks that apply to i386 such that they also hit files that we've modified in xen/i386. This way we keep arch xen/i386 in perfect sync with i386. It takes discipline, but we're pretty good at it now.
That won't anymore at some time. I found this out on x86-64. It works at the beginning, but eventually the code drift gets so much that it's near impossible to apply any hunk and you have to redo everything. One reason for that is that in Linux code often gets restructured, which makes it very difficult for such mechanized merging procedures to work long term.
Also you have to review every change anyways.
Quoted 10 lines
Quoted 3 lines
I cannot imagine the virtualization hooks are intrusive anyways. The only things it needs to hook idle and the page table updates, right?
It's rather more complicated than that if you want a clean interface that gives good performance. We've taken a very benchmark-driven approach to minimise the overhead of virtualization. The aim is to have such a small overhead that people are happy running virtualized the whole time. I think this is a really important aim.
Can you please be a bit more precise on that? What exactly do you need?
Quoted 6 lines
Quoted 3 lines
Doing that cleanly in the existing architectures shouldn't be that hard.
I've appended a list of some of the areas we need to modify. I think you may have underestimated what needs to happen.
Ok that's a start.
Quoted 10 lines
Quoted 6 lines
I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy.
We have subdirectories for the i386 and
x86_64specific files, along with a common directory for stuff which is shared between the two e.g. virtual interrupt control etc.
Ok, but I think there will be significant differences in the 64bit part and meshing both together will get extremly messy.
What's your strategy for example to merge changes from arch/x86_64 to xen64? I don't think the way you described above will work in any way.
Quoted 11 lines
Quoted 3 lines
Also the other thing I'm worried about is that there is no clear specification on how the Xen<->Linux interface works.
We have an interface manual in the Xen bk repo, but I acknowledge that we haven't always been totally prompt in keeping it up to date and fully detailed. Now the Xen2 interface is frozen, that should be fixable. Even so, it hasn't prevented other independent groups porting OSes to Xen e.g. NetBSD, FreeBSD, Plan9.
Ian
Comments on some issues:
You are saying you ran benchmarks on each of them and it turns out to be too slow to emulate them in the standard architectural way? At least for some of them it's hard to believe.
Quoted 3 lines
Differences between i386 and xen/i386 files:
- irq setup
Already two different ways (MSI and non MSI). Adding a third is probably feasible.
Quoted 1 line
- pci bus scanning
We already have at least three different ways to do that, adding a fourth wouldn't be very messy I guess.
Quoted 1 line
- gdt/ldt must be in dedicated read-only pages
That can be probably done in the standard kernel
(without the actual read protection bit)
Quoted 1 line
- gdt/ldt install/updates
gdt load could be just trapped, it only happens once at startup.
LDT i can see some effort needed, but it's already only at a single function in a header file.
Quoted 1 line
- debug register updates
This shouldn't be performance critical. Why is it not simply emulated by the hypervisor?
Quoted 1 line
- pte quicklist/cache
i386 had a pte quicklist some time ago. I think there were plans anyways to readd it.
Quoted 1 line
- pmd/pgd are read-only pages
I suspect this just needs another include layer like the existing PAE/non PAE layer.
Quoted 3 lines
- highmem pte mapping
- dma memory allocation
- idle loop
This is already pluggable (e.g. ACPI has its own)
Quoted 3 lines
- different fixmap
- *_to_* macros differ (likepage_to_phys)
-*_valmacros (pte_val,pmd_val)
Can be also handled like 2/3 levels I bet.
Quoted 1 line
- inb/outb
Why is it not trapped?
Quoted 4 lines
- switching cr3
- cr4 updates
- a few cpu flags need to be cleared
- msr access
Why don't you just simply trap it? MSRs are normally not performance critical (except on x86-64 for the context switch but you didn't seem to have attacked that at all yet ;-)
Quoted 2 lines
- wbinvd call
- mtrr access
Same.
For all of these it would seem much cleaner to me to just provide instruction emulation in the Hypervisor.
Quoted 1 line
- io permission handling
What is different?
Quoted 5 lines
- ioremap
- access to hardware memory
- interrupt enabling/disabling
- setup of trap gates
- tlb flush
This is already well separated, shouldn't be a big issue to make it different.
Quoted 2 lines
- fpu stack switches
- user/kernel mode test
That can be just changed everywhere in i386 without ill effects.
Quoted 3 lines
- different segment selectors since the kernel runs in ring1
- pagefault handler stack layout is different
- startup is in 32-bit mode
That's a good cleanup anyways. The early boot interface between 16 and 32bit should be clearly defined and then a general 32bit booting interface be defined. There are various other people who would like to have this too. I had some plans to do the equivalent on x86-64 with a 64bit boot interface.
Quoted 2 lines
- start of day initialization is different
- start of day memory probing and pagetable setup
Sounds similar to EFI.
Quoted 2 lines
- trap/fault handling
- timer is virtualized
Details?
First impression is that there is much cleanup potential and that there should be probably some discussion on each of these items.
-Andi
Hi!
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Stunned silence I guess - merging an architecture is usually much more controversial ;)
In my opinion it's still an extremly bad idea to have
arch/xenan own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.
Quoted 6 lines
Andi, I totally agree that merging into i386 could be a long term goal. However, its just not feasible right now. The changes required are way too intrusive. We put considerable effort into investigating this approach, but came to the conclusion that with the current structure of arch i386 it was going to be way too messy.
Okay, what about this one:
You merge xen hooks in mainline, but keep maintaining arch/xen out-of-tree? You have to maintain it yourself, anyway, and having hooks merged should make it easy.
When xen is merged into i386 (you said that is your long-term goal anyway), you can merge that into mainline...
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Hi!
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Stunned silence I guess - merging an architecture is usually much more controversial ;)
In my opinion it's still an extremly bad idea to have
arch/xenan own architecture. It will cause a lot of work long term to maintain it, especially when it gets x86-64 support too. It would be much better to just merge it with i386/x86-64.
Quoted 6 lines
Andi, I totally agree that merging into i386 could be a long term goal. However, its just not feasible right now. The changes required are way too intrusive. We put considerable effort into investigating this approach, but came to the conclusion that with the current structure of arch i386 it was going to be way too messy.
BTW if you merge xen as separate architecture, it will be *very* hard to merge it back to i386. That patch would be huge, and would need to go in "atomically".
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 3 lines
You merge xen hooks in mainline, but keep maintaining
arch/xenout-of-tree? You have to maintain it yourself, anyway, and having hooks merged should make it easy.
Having the hooks merged would certainly help.
However, I think the main benefit of being in the mainline tree would be that it would make it easy for other developers to see how arch xen works and thus begin to take us into account when considering changes. If there's going to be a big effort to merge i386, x86_64 and xen going forward, it will certainly help if more people have looked through our code. We'll only get that kind of exposure to developers if we're in the mainstream tree.
The other key area is that our top priority is to decrease the number of files we need to modified from standard i386. For this to happen, we need to submit patches into i386 that abstract a few things behind macros/constants. For example, we'd like to abstract the test to see whether the CPU is in the kernel or not (we run the kernel in ring 1 rather than 0). If arch xen is in the tree, this kind of patch will make rather more sense to people.
Quoted 3 lines
BTW if you merge xen as separate architecture, it will be *very* hard to merge it back to i386. That patch would be huge, and would need to go in "atomically".
I don't see it like that. While continuing to track changes in i386/x86_64, we'd restructure the code under arch xen such that it could build (or even boot) time switch between running native and over Xen. At some point the arch directory could then be renamed. This would be a big project, and one that would involve a lot more people than just the Xen team. Because the x86 architecture is so critical to Linux, I just can't see this happening unless the development takes place in the mainstream tree.
Of course, another reason for inclusion in mainstream is that there's a lot of momentum behind Xen in both the user and developer communities. My guess is that virtualization is going to be pretty pervasive in the data centre, and its one of the key technologies that Linux can win on. The paravirtualized approach used by Xen is going to offer better performance than full x86 virtualization, even if there's extra support in the processor.
Cheers,
Ian
Quoted 3 lines · Andi Kleen
I think that's definitely the wrong way. Also in Linux the standard strategy is to first clean up and restructure/refactor code and then merge, not the other way round.
Must be two different Linux OS's out there then. I thought it was
- get interfaces right
- get working
- get correct
- merge
- evolve
If we did it as you describe we'd still not have a SATA layer merged for example.
Hi!
Quoted 10 lines
Quoted 3 lines
BTW if you merge xen as separate architecture, it will be *very* hard to merge it back to i386. That patch would be huge, and would need to go in "atomically".
I don't see it like that. While continuing to track changes in i386/x86_64, we'd restructure the code under arch xen such that it could build (or even boot) time switch between running native and over Xen. At some point the arch directory could then be renamed. This would be a big project, and one that would involve a lot more people than just the Xen team. Because the x86
So you plan to
merge arch/xen
then modify arch/xen to do all arch/i386 can do
then rm -rf arch/i386, mv arch/xen arch/i386? Well, I'd say that's rather ambitious plan....
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 7 lines · Alan Cox
Quoted 3 lines · Andi Kleen
I think that's definitely the wrong way. Also in Linux the standard strategy is to first clean up and restructure/refactor code and then merge, not the other way round.
Must be two different Linux OS's out there then. I thought it was
- get interfaces right
That is exactly the part that is wrong currently imho. The arch/xen interface is a mess and in its current form unlikely to be maintainable.
-Andi
Quoted 2 lines · Andi Kleen
That is exactly the part that is wrong currently imho. The
arch/xeninterface is a mess and in its current form unlikely to be maintainable.
It seems maintainable and well documented to me. I just don't see where your problem is with this. The kernel/hypervisor interface is clear, and the arch/xen code seems quite sane.
Quoted 7 lines · Alan Cox
Quoted 2 lines · Andi Kleen
That is exactly the part that is wrong currently imho. The
arch/xeninterface is a mess and in its current form unlikely to be maintainable.It seems maintainable and well documented to me. I just don't see where your problem is with this. The kernel/hypervisor interface is clear, and the
arch/xencode seems quite sane.
The main problem I see is that it is source code copy, and especially when they both support i386 and x86-64 there will be no sane way to keep it all synchronized with the i386 and x86-64 code bases. It's already hard from a single source like I can attest from x86-64, with two sources it will be likely much more difficult longer term. I just can't see it working well in practice. It will be also nasty for people doing changes because they will need to duplicate i386+x86_64 changes four times in the worst case (i386,x86_64,xen32,xen64)
I guess it may be acceptable if we were maintaining obscure Lance drivers this way ;_), but for a important architecture it just doesn't seem like the right approach to me.
Also e.g. for non performance critical
things like changing MTRRs or debug registers it would be IMHO much cleaner to just emulate the instructions (the ISA is very well defined) and not change the kernel here. From a look at Ian's list the majority of the changes needed for Xen actually fall into this category.
I suspect when the kernel is only changed for the truly performance critical interfaces that cannot be efficiently emulated (like idle/timers/page table updates) the required changes for the para virtualization will become much more manageable and can be cleanly integrated into the respective ports.
And as Pavel points out first merging arch/xen and then migrating into i386 and x86_64 like it was proposed sounds extremly hard and is probably not really practical.
-Andi
Quoted 6 lines · Andi Kleen
Also e.g. for non performance critical
things like changing MTRRs or debug registers it would be IMHO much cleaner to just emulate the instructions (the ISA is very well defined) and not change the kernel here. From a look at Ian's list the majority of the changes needed for Xen actually fall into this category.
There are so many problems in snooping and decoding instructions it isn't funny. Aside from the mmap pci buffer half way through instruction that will emulate type stuff there are a lot of awkward issues if you want to emulate multiple mtrr sets (you need PAT).
Xen has tried the decode/patch stuff earlier - see the early NPTL handling and it was neither pretty nor reliable.
Quoted 8 lines · Ian Pratt
The other key area is that our top priority is to decrease the number of files we need to modified from standard i386. For this to happen, we need to submit patches into i386 that abstract a few things behind macros/constants. For example, we'd like to abstract the test to see whether the CPU is in the kernel or not (we run the kernel in ring 1 rather than 0). If arch xen is in the tree, this kind of patch will make rather more sense to people.
That would be a good first step, especially if it results in cleanups. Please go for it.
Quoted 5 lines
I don't see it like that. While continuing to track changes in i386/x86_64, we'd restructure the code under arch xen such that it could build (or even boot) time switch between running native and over Xen. At some point the arch directory could then be renamed. This would be a big project, and one that would involve
This sounds like a massive duplication of effort. You would need to do all that work on arch/xen and in parallel on the native port for the slow merge, and in parallel track a changing target and keep the code usable in mainline.
-Andi
Quoted 10 lines · Alan Cox
Quoted 6 lines · Andi Kleen
Also e.g. for non performance critical
things like changing MTRRs or debug registers it would be IMHO much cleaner to just emulate the instructions (the ISA is very well defined) and not change the kernel here. From a look at Ian's list the majority of the changes needed for Xen actually fall into this category.There are so many problems in snooping and decoding instructions it isn't funny. Aside from the mmap pci buffer half way through instruction
Hmm? From what I remember reading the code of the Xen hypervisor they are already emulating a lot of instructions (e.g. take a look at xen/arch/x86/x86_32/seg_fixup.c) Emulating some more doesn't seem like a big issue to me.
Quoted 2 lines
that will emulate type stuff there are a lot of awkward issues if you want to emulate multiple mtrr sets (you need PAT).
No way different from doing it with a magic interface. MTRR is exposed to kernel subsystems (a lot of things in the service OS will want to play with it) and to user space (with also a lot of users). The magic interface will need to provide a full MTRR interface anyways. The only difference would be basically how you tell the hypervisor to change them.
[Not saying that PAT is actually needed for this, but as a general comment:]
Also using PAT for that would be definitely a good idea, MTRRs just have so many problems that any step away from them is a good thing.
-Andi
Quoted 4 lines · Andi Kleen
And as Pavel points out first merging
arch/xenand then migrating
into i386 andx86_64like it was proposed sounds extremly hard and is probably not really practical.
It will also create more patchwork and will make the the change history more obscure and harder to follow.
It is a bit pervers to be putting something into the tree with the intention of ripping it up and redoing it in the near future. Why not do it up-front and keep the change history cleaner? From Ian's change list and Andi's comments it doesn't look to me that this is all as much work as people seem to be implying?
I guess if we were to go the way which Ian is proposing it would be
a) Add arch/xen
b) Spend N weeks integrating xen into arch/i386, while also separately maintaining arch/xen.
c) Remove arch/xen
So... why not skip a), c) and half of b)?
Quoted 10 lines · Andrew Morton
I guess if we were to go the way which Ian is proposing it would be
a) Add
arch/xenb) Spend N weeks integrating xen into
arch/i386, while also separately maintainingarch/xen.c) Remove
arch/xenSo... why not skip a), c) and half of b)?
Because until xen is in the kernel it won't get good exposure Because nobody is sure that b) needs to occur or would occur for another year our two (Xen3)
Xen is not "a slightly x86 arch variant" (and a slight ia64 arch variant) Xen is quite different in its design and its performance comes from exactly that - paravirtualisation by being a virtual CPU that isn't a true x86 is very very fast.
b is a long term research project it is not a "clean this up next month" project as I understand it.
Quoted 7 lines
Quoted 2 lines
There are so many problems in snooping and decoding instructions it isn't funny. Aside from the mmap pci buffer half way through instruction
Hmm? From what I remember reading the code of the Xen hypervisor they are already emulating a lot of instructions (e.g. take a look at
xen/arch/x86/x86_32/seg_fixup.c) Emulating some more doesn't seem like a big issue to me.
There's actually no emulation in Xen at all. seg_fixup is the closest we come, which is a tiny decoder that is able to work out the effective address of an instruction. It's only through grim necessity we need this in order to be able to make NPTL thread local storage accesses work (-ve segment offsets). We'd much prefer that glibc was slightly tweaked such that we didn't have to do this (it would be a lot faster too).
Anyhow, I really don't think emulation is the issue. The handful of cases that you pointed out that could relatively easily be emulated constitute a tiny fraction of the arch xen patch. Even so, you have to be a bit careful from a performance point of
view: loading debug registers can occur quite frequently, and its much quicker to be able to load them as a batch rather than taking individual faults.
Ian
Quoted 11 lines
I guess if we were to go the way which Ian is proposing it would be
a) Add
arch/xenb) Spend N weeks integrating xen into
arch/i386, while also separately maintainingarch/xen.c) Remove
arch/xenSo... why not skip a), c) and half of b)?
That's not quite what I'm proposing.
Once arch/xen is in the tree, we'd start submitting patches that try and unify more of arch xen and i386 to reduce the number of files that we have to modify.
I think we'd then have to make a decision as to whether merging i386 and xen/i386 is feasible. Further, we'd have to make a decision as to whether what is really wanted is a single kernel that's able to boot-time switch between native i386 and xen, rather than just having a single source base. The two options would probably result in rather different implementations.
In short, merging is far from trivial, and its not even clear quite what is wanted.
I'm not convinced that maintaining xen/i386 in its current form is going to be as hard as Andi thinks. We already share many files unmodified from i386. Keeping i386 and xen/i386 in sync is fairly mechanical: we can apply most of the patches to i386 to xen/i386 directly.
Ian
Quoted 4 lines · Ian Pratt
I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.
How are they shared? Inclusion, symlinks, makefile references or copy-n-paste?
Quoted 7 lines
Quoted 4 lines · Ian Pratt
I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.How are they shared? Inclusion, symlinks, makefile references or
The makefile creates symlinks.
Ian
Quoted 9 lines · Ian Pratt
Quoted 7 lines
Quoted 4 lines · Ian Pratt
I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.How are they shared? Inclusion, symlinks, makefile references or
The makefile creates symlinks.
Which is really very nice. If you update arch/i386/ then arch/xen/ automatically gets the update too.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 8 lines · Rumor has it that on Thu, Dec 16, 2004 at 09:00:52PM +0000 Ian Pratt said:
I think we'd then have to make a decision as to whether merging i386 and xen/i386 is feasible. Further, we'd have to make a decision as to whether what is really wanted is a single kernel that's able to boot-time switch between native i386 and xen, rather than just having a single source base. The two options would probably result in rather different implementations.
The boot-time switch seems to be the ideal. This would allow enterprise Linux vendors to support using Xen w/o having to deal with a whole archicture release (including install kernel etc.). One of the (really strong IMO) arguments for full virtualization methods is beinag able to run stock OSes. At least for Linux guests, a Xen merge with the boot-time option would remove that argument.
Of course, the resultant kernel changes would need to have no measurable impact on native i386... and be possible...
Cheers,
Phil
Quoted 9 lines · Pavel Machek
Okay, what about this one:
You merge xen hooks in mainline, but keep maintaining
arch/xenout-of-tree? You have to maintain it yourself, anyway, and having hooks merged should make it easy.When xen is merged into i386 (you said that is your long-term goal anyway), you can merge that into mainline...
Pavel
Assuming this is practical and would let xen get exposure sooner rather than later, it would be nice if the hooks could be used for other projects. My impression is that the merge part would come after there is some extensive experience with the operation "in the wild" so to speak. Who knows what operating systems might run in this way.
Signature · 3 lines
-bill davidsen ([email protected]) "The secret to procrastination is to put things off until the last possible moment - but no longer" -me
Quoted 3 lines · Philip R Auld
The boot-time switch seems to be the ideal. This would allow enterprise Linux vendors to support using Xen w/o having to deal with a whole archicture release (including install kernel
I have no idea how such a boot-time switch would work for 3rd party device drivers, though, so don't count yourself lucky just yet ;)
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 3 lines · Bill Davidsen
Assuming this is practical and would let xen get exposure sooner rather than later, it would be nice if the hooks could be used for other projects.
Who knows what operating systems might run in this way.
NetBSD and Plan 9 already run on Xen 2.0. IIRC FreeBSD and one of the other BSDs are being ported, too...
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 9 lines · Rumor has it that on Thu, Dec 16, 2004 at 06:08:24PM -0500 Rik van Riel said:
Quoted 3 lines · Philip R Auld
The boot-time switch seems to be the ideal. This would allow enterprise Linux vendors to support using Xen w/o having to deal with a whole archicture release (including install kernel
I have no idea how such a boot-time switch would work for 3rd party device drivers, though, so don't count yourself lucky just yet ;)
I actually meant _OS_ vendors like you ;)
Cheers,
Phil
Quoted 5 lines
Signature · 4 lines
-- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 10 lines · Ian Pratt
Quoted 10 lines
I guess if we were to go the way which Ian is proposing it would be
a) Add
arch/xenb) Spend N weeks integrating xen into
arch/i386, while also separately maintainingarch/xen.c) Remove
arch/xen
Quoted 7 lines
Quoted 1 line
So... why not skip a), c) and half of b)?
That's not quite what I'm proposing.
Once
arch/xenis in the tree, we'd start submitting patches that try and unify more of arch xen and i386 to reduce the number of files that we have to modify.
IMHO reducing the number of files to modify should be done first before attempting a mainline merge. That is how a lot of other more intrusive changes have been merged in the past.
There was a intensive review process and eventually the interfaces were cleaned up a lot of and the thing was merged piece by piece after considerable changes.
Quoted 7 lines
I think we'd then have to make a decision as to whether merging i386 and xen/i386 is feasible. Further, we'd have to make a decision as to whether what is really wanted is a single kernel that's able to boot-time switch between native i386 and xen, rather than just having a single source base. The two options would probably result in rather different implementations.
Even without boot time switch but only CONFIG switch a more shared code base would be probably the only saner way long time. I'm not sure a run time switch is really that good an idea, it would essentially require to run most of pgtable.h etc. through function pointers and I don't like this idea very much. But at least doing the reorganization on the source code level would probably clean up the interface considerably.
Quoted 9 lines
In short, merging is far from trivial, and its not even clear quite what is wanted.
I'm not convinced that maintaining xen/i386 in its current form is going to be as hard as Andi thinks. We already share many files unmodified from i386. Keeping i386 and xen/i386 in sync is fairly mechanical: we can apply most of the patches to i386 to xen/i386 directly.
Perhaps for now, but eventually this will not work anymore due to inevitable code drift (I went exactly through the same phase for a year or so on x86-64. These days I have to redo most patches unless i'm lucky enough that the contributor does both)
And with also supporting x86-64 it will not really work this way well because you would need to merge from two sources. I can't see any mechanical scheme handling that. The only sane way to maintain the merging scheme you proposed would be then arch/xen64, multiplying work again.
-Andi
Quoted 9 lines · Ian Pratt
Quoted 7 lines
Quoted 4 lines · Ian Pratt
I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.How are they shared? Inclusion, symlinks, makefile references or
The makefile creates symlinks.
That's deprecated because it breaks with separate objdirs (make O=/other/dir) See the x86-64 makefiles on how to do it properly.
-Andi
Quoted 7 lines · Ian Pratt
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines · Ian Pratt
I'm not convinced that maintaining xen/i386 in its current form
is going to be as hard as Andi thinks. We already share many files unmodified from i386.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
How are they shared? Inclusion, symlinks, makefile references or
The makefile creates symlinks.
That's deprecated because it breaks with separate objdirs (make O=/other/dir) See the x86-64 makefiles on how to do it properly.
Ah, I wasn't previously aware of the O= capability to know that we were breaking it.
We can trivially change over to referencing files in arch/i386 directly from the Makefile. The only downside is that it will make navigating the source slightly harder.
Ian
Quoted 8 lines · Andi Kleen
I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy. And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in
arch/xenwill be very pretty.
I have an inkling that the xen implementors may have plots of additional architecture ports. If it really is x86/x86-64 -only it isn't worth bothering with a separate arch/ dir, but it would be if it were ported to a large number of architectures.
Maybe they're modelling it after UML which has its own arch/ dir but then grabs assorted things from other architectures; in that event I wouldn't consider it so misguided.
Probably best if the implementors chimed in about all this.
-- wli
Quoted 10 lines · Andi Kleen
Quoted 8 lines
I suspect xen64 will be rather different from xen32 anyways because as far as I can see the tricks Xen32 uses to be fast (segment limits) just plain don't work on 64bit because the segments don't extend into 64bit space. So having both in one architecture may also end up messy. And i386 and x86-64 are in many pieces very different anyways, I have my doubts that trying to mesh them together in
arch/xenwill be very pretty.
Quoted 4 lines
I have an inkling that the xen implementors may have plots of additional architecture ports. If it really is x86/x86-64 -only it isn't worth bothering with a separate
arch/dir, but it would be if it were ported to a large number of architectures.
For non x86 architectures, the changes required are typically much less, and it makes sense to integrate them into the normal architecture tree. This is what has been done with ia64, and will be done with ppc. i386 and x86_64 are a special case where the extensive changes required justify having an arch xen directory with i386 and x86_64 subtrees beneath it (although many files are just linked from the normal architecture).
Quoted 3 lines
Maybe they're modelling it after UML which has its own
arch/dir but then grabs assorted things from other architectures; in that event I wouldn't consider it so misguided.
Yep, that's the model, for the x86 family.
Ian
arch/xen clue?
In reply to Bill DavidsenFor those of us who are clueless as to even what arch/xen is :) Where would be a good place to read and become informed?
-Dorn
Re: arch/xen clue?
Quoted 4 lines
For those of us who are clueless as to even what
arch/xenis :) Where would be a good place to read and become informed?
There's a bunch of papers and documentation etc.
Ian
Re: arch/xen clue?
In reply to Dorn HetzelQuoted 3 lines · Dorn Hetzel
For those of us who are clueless as to even what
arch/xenis :) Where would be a good place to read and become informed?
http://www.cl.cam.ac.uk/Research/SRG/netos/xen/index.html
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
Re: arch/xen clue?
In reply to Dorn HetzelQuoted 3 lines · Dorn Hetzel
For those of us who are clueless as to even what
arch/xenis :) Where would be a good place to read and become informed?
Signature · 2 lines
Måns Rullgård [email protected]
Quoted 3 lines · Andi Kleen
In my opinion it's still an extremly bad idea to have
arch/xen
an own architecture.
Guys, I'd like to kick this a bit further down the road. Things still seem to be somewhat deadlocked.
To summarise my understanding:
The Xen team still believe that it's best to keep arch/xen, arch/xen/i386, arch/xen/x86_64, etc. And I believe that Andi (who is the world expert on maintaining an i386 derivative) thinks that this is will be a long-term maintenance problem.
I tend to agree with Andi, and I'm not sure that the Xen team fully appreciate the downside of haveing an own-architecture in the kernel.org kernel and the upside of having their code integrated with the most-maintained architecture. It could be that the potential problems haven't been sufficiently well communicated.
Christian has mentioned that Xen would need to hook into the i386 code in ~60 places, which is somewhat more than Ian's 37-bullet-point list.
I get the impression that the Xen team are overly reluctant to make changes to the arch/i386 code and to arch-neutral kernel code. Don't do that - new abstractions, refactoring and generally moving things about is generally a safe thing to do, and can often make things better anyway.
So. Has anyone changed position or otherwise converged? How do we get this resolved?
kernel 2.6.8-24.11-smp errors
Hello,
I am using SuSE 9.2 Pro with kernel uname -a:
Linux 2.6.8-24.11-smp #1 SMP Fri Jan 14 13:01:26 UTC 2005 x86_64 x86_64 x86_64 GNU/Linux
Intel(R) Xeon(TM) CPU 2.80GHz
2 GB RAM
1 GB SWAP
and these strange kernel messages keep appearing::
it is a high load machine with iptables running. The /proc/net/ip_conntrack number keeps running higher and does not get less. After a while the maximum is reached. th e only way to solve this is stop iptables, unload the ip modules from the kernel, load them again and start iptables again. Then the messages will not come again if I repeat this frequently. Could it be a kernel bug??? Or maybe something misconfigured???
dmesg:
swapper: page allocation failure. order:1, mode:0x20
Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
<ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
<ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
<ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
<ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
<ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
<ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
<ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80308b90>{ip_dst_output+0}
<ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80133c91>{activate_task+129}
<ffffffff80137a89>{autoremove_wake_function+9}
<ffffffff80132633>{__wake_up_common+67}
<ffffffff80132c13>{__wake_up+67} <ffffffff802e2344>{sock_def_readable+68}
<ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
<ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
<ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
<ffffffff8030592e>{ip_rcv_finish+574} <ffffffff803056f0>{ip_rcv_finish+0}
<ffffffff803056f0>{ip_rcv_finish+0} <ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff803056f0>{ip_rcv_finish+0} <ffffffff80305e34>{ip_rcv+1188}
<ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
<ffffffff802e85b4>{net_rx_action+132}
<ffffffff8013dca1>{__do_softirq+113}
<ffffffff8013dd55>{do_softirq+53} <ffffffff80113e3f>{do_IRQ+335}
<ffffffff8010f540>{mwait_idle+0} <ffffffff80110cf5>{ret_from_intr+0}
<EOI> <ffffffff8010f596>{mwait_idle+86} <ffffffff8010f9ea>{cpu_idle+26}
<ffffffff804e971a>{start_kernel+490} <ffffffff804e91e0>{_sinittext+480}swapper: page allocation failure. order:1, mode:0x20
Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
<ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
<ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
<ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
<ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
<ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
<ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
<ffffffff802f3a43>{nf_hook_slow+195} <ffffffff80308b90>{ip_dst_output+0}
<ffffffff8013340a>{task_rq_lock+74}
<ffffffff80133f9b>{try_to_wake_up+747}
<ffffffff8030ad6b>{ip_queue_xmit+1211}
<ffffffff8013340a>{task_rq_lock+74}
<ffffffff8013340a>{task_rq_lock+74}
<ffffffff80132633>{__wake_up_common+67}
<ffffffff80132c13>{__wake_up+67}
<ffffffff80312996>{tcp_ack_saw_tstamp+22}
<ffffffff80314d90>{tcp_ack+1040} <ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
<ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
<ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
<ffffffff8030592e>{ip_rcv_finish+574}
<ffffffff803056f0>{ip_rcv_finish+0} <ffffffff803056f0>{ip_rcv_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195} <ffffffff803056f0>{ip_rcv_finish+0}
<ffffffff80305e34>{ip_rcv+1188} <ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
<ffffffff80133c91>{activate_task+129}
<ffffffff80133f9b>{try_to_wake_up+747}
<ffffffff802e85b4>{net_rx_action+132}
<ffffffff8013dca1>{__do_softirq+113} <ffffffff8013dd55>{do_softirq+53}
<ffffffff80113e3f>{do_IRQ+335} <ffffffff8010f540>{mwait_idle+0}
<ffffffff80110cf5>{ret_from_intr+0} <EOI>
<ffffffff8010f596>{mwait_idle+86}
<ffffffff8010f9ea>{cpu_idle+26} <ffffffff804e971a>{start_kernel+490}
<ffffffff804e91e0>{_sinittext+480}swapper: page allocation failure. order:1, mode:0x20
Call Trace:<IRQ> <ffffffff8015ecef>{__alloc_pages+1135}
<ffffffff8015e830>{__get_free_pages+16}
<ffffffff80162676>{kmem_getpages+38}
<ffffffff8031e3ea>{tcp_v4_route_req+250}
<ffffffff80162a99>{cache_alloc_refill+665}
<ffffffff80162c46>{kmem_cache_alloc+54}
<ffffffff802e2f0a>{sk_alloc+58}
<ffffffff80323d99>{tcp_create_openreq_child+41}
<ffffffff803229e7>{tcp_v4_syn_recv_sock+87}
<ffffffff80323b7e>{tcp_check_req+606}
<ffffffff8030b4dc>{ip_finish_output+508}
<ffffffff80308b90>{ip_dst_output+0}
<ffffffff8030b668>{ip_output+216} <ffffffff80308bc8>{ip_dst_output+56}
<ffffffff802f3a43>{nf_hook_slow+195} <ffffffff80308b90>{ip_dst_output+0}
<ffffffff80132f9b>{recalc_task_prio+635}
<ffffffff80133c91>{activate_task+129}
<ffffffff8013340a>{task_rq_lock+74}
<ffffffff80132633>{__wake_up_common+67}
<ffffffff80313e1b>{tcp_fastretrans_alert+859}
<ffffffff8031536d>{tcp_ack+2541}
<ffffffff80320b79>{tcp_v4_do_rcv+233}
<ffffffffa017b034>{:ipt_state:match+36}
<ffffffff803214cb>{tcp_v4_rcv+1835}
<ffffffff80305fb9>{ip_local_deliver_finish+297}
<ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff80305e90>{ip_local_deliver_finish+0}
<ffffffff803062fe>{ip_local_deliver+622}
<ffffffff8030592e>{ip_rcv_finish+574} <ffffffff803056f0>{ip_rcv_finish+0}
<ffffffff803056f0>{ip_rcv_finish+0} <ffffffff802f3a43>{nf_hook_slow+195}
<ffffffff803056f0>{ip_rcv_finish+0} <ffffffff80305e34>{ip_rcv+1188}
<ffffffff802e9749>{netif_receive_skb+729}
<ffffffffa00e9c9c>{:e1000:e1000_clean+1820}
<ffffffff80132c13>{__wake_up+67} <ffffffff802e85b4>{net_rx_action+132}
<ffffffff8013dca1>{__do_softirq+113} <ffffffff8013dd55>{do_softirq+53}
<ffffffff80113e3f>{do_IRQ+335} <ffffffff8010f540>{mwait_idle+0}
<ffffffff80110cf5>{ret_from_intr+0} <EOI>
<ffffffff8010f596>{mwait_idle+86}
<ffffffff8010f9ea>{cpu_idle+26} <ffffffff804e971a>{start_kernel+490}
<ffffffff804e91e0>{_sinittext+480}