Skip to contentSkip to conversation

8/8 Create MMU 2/3 level accessors in the sub-arch layer (i386)

PATCH Andi Kleen 5 participants 8 messages Aug 6, 2005 – Aug 7, 2005 1 message not in the archive

lore
Share & tools
lore

Andi Kleen

author
#1 lore raw
In reply to a message not in the archive
Quoted 10 lines · Zachary Amsden

i386 Transparent paravirtualization sub-arch patch #8.

Transparent paravirtualization support for MMU operations.

All operations which update live page table entries have been moved to the sub-architecture layer. Unfortunately, this required yet another parallel set of pgtable-Nlevel-ops.h files, but this avoids the ugliness of having to use #ifdef's all of the code.

This is pure code motion. Anything else would be a bug.

I think that patch is really ugly - it makes hacking VM on i386 even more painful than it already is because the convolutes the file structure even more. Hope it is not applied.

-Andi

#2 lore raw
Quoted 11 lines · Andi Kleen
Quoted 9 lines · Zachary Amsden

i386 Transparent paravirtualization sub-arch patch #8.

Transparent paravirtualization support for MMU operations.

All operations which update live page table entries have been moved to the sub-architecture layer. Unfortunately, this required yet another parallel set of pgtable-Nlevel-ops.h files, but this avoids the ugliness of having to use #ifdef's all of the code.

Quoted 5 lines
Quoted 1 line

This is pure code motion. Anything else would be a bug.

I think that patch is really ugly - it makes hacking VM on i386 even more painful than it already is because the convolutes the file structure even more. Hope it is not applied.

Especially as there's been no user shown for it, similar to all the other ugly patches from vmware.

author
#3 lore raw
Quoted 6 lines
Quoted 3 lines

I think that patch is really ugly - it makes hacking VM on i386 even more painful than it already is because the convolutes the file structure even more. Hope it is not applied.

Especially as there's been no user shown for it, similar to all the other ugly patches from vmware.

Well, some of it can be counted as cleanup or even tuning like the excellent switch_to patch. But not that one and some of the more intrusive patches.

-Andi

Quoted 1 line
#4 lore raw
Quoted 9 lines · Andi Kleen
Quoted 6 lines
Quoted 3 lines

I think that patch is really ugly - it makes hacking VM on i386 even more painful than it already is because the convolutes the file structure even more. Hope it is not applied.

Especially as there's been no user shown for it, similar to all the other ugly patches from vmware.

Well, some of it can be counted as cleanup or even tuning like the excellent switch_to patch. But not that one and some of the more intrusive patches.

Yeah, I said ugly ones specificly. There's been some nice previous ones, but most in this series (all the move of stuff to subarches) are rather horrible and lack lots of explanation.

#5 lore raw
Quoted 12 lines · Christoph Hellwig
Quoted 9 lines · Andi Kleen
Quoted 9 lines
Quoted 5 lines

I think that patch is really ugly - it makes hacking VM on i386 even more painful than it already is because the convolutes the file structure even more. Hope it is not applied.

Especially as there's been no user shown for it, similar to all the other ugly patches from vmware.

Quoted 10 lines
Quoted 4 lines

Well, some of it can be counted as cleanup or even tuning like the excellent switch_to patch. But not that one and some of the more intrusive patches.

Yeah, I said ugly ones specificly. There's been some nice previous ones, but most in this series (all the move of stuff to subarches) are rather horrible and lack lots of explanation.

All of my previous patches have been aimed at fixing bugs, improving performance, reliability and maintinability of the i386 architecture. If you found something that didn't fit one of those categories in my previous patches, then it is either not well enough explained or perhaps inadvertently slipped through from one of my more radical trees - or it could be a bug.

There is a simple explanation for all of this series. The goal is to move all privileged instructions, sensitive instructions, and privilege awareness into a layer where it can be overridden by new code without disrupting the default architecture. On x86, there are a lot of instructions - popf, iret, sgdt, and others which behave differently under different privilege levels, but do not trap. These architectural features must be redefined by any architecture which virtualizes the x86, be it Xen, UML, or an alternative approach. Similarly, certain privileged processor data structures (page tables, descriptor tables) must be protected and accessed in a different manner if one is to utilize the principles of paravirtualization to achieve high performance inside of a virtual machine. I believe this series of patches is one almost clean solution to this. There are obvious problems with the MMU patch, and I'm still trying to come up with a way to properly address that.

That said, I am definitely seeking any feedback you have on how to achieve this goal while being as clean and maintainable as possible - if the Linux community is indeed interested in adopting a paravirtualization approach. Looking from the most general view possible, there are a couple of ways to do this in Linux:

1) Create a new architecture. This is the UML approach, and while it has been successful there, it is difficult to maintain closeness to the hardware architecture without introducing a maintenance burden. This closeness is desirable if one is to achieve high performance and take advantage of more processor specific features.

2) Use the sub-architecture strategy of x86. This approach has a relatively small set of code movement to allow a new virtualized sub-architecture to redefine the privileged and sensitive operations of the processor, as well as to implement easily defined architectural hints which employ higher level virtualization strategies.

3) Use #ifdef'd include/asm-i386 header files. While workable, this has flaws - it is ugly, and it causes the hypervisor header files to sneak into include/asm-i386 rather rapidly destroying maintainability for the native code. This has been attempted before, but if someone were to send those patches to LKML, I would expect them to be rapidly tarred and feathered.

4) Clone entire asm-i386 header files and replace them using an include path, potentially in the sub-architecture level. While this avoids any diffs at all to the native asm-i386 headers, it needlessly duplicates a lot of code, and this creates a greater maintenance burden for somebody. Who that body is can be determined later, but this creates a lot of extra work for that unfortunate person that is wasted time that could easily be spent bettering Linux!

5) Use the HAVE_SUBARCH_PTE_ACCESSOR type approach (similar to the way generic optimizable PTE operations are defined in asm-generic). I have not yet investigated the feasibility of this type of approach, but it seems workable. At least for the MMU patch, some combination of this and other techniques might help make things a lot cleaner.

Do you have ideas? I'm open to all suggestions here. The only goal I have is to make high performance virtualization support in Linux the least disruptive event possible for all parties. Although I'm a bit biased towards i386 from a coding point of view, some of these ideas can cross architecture boundaries as well, so I'm open to feedback from all parties.

Thanks,
Zach

#6 lore raw
Quoted 9 lines · Zachary Amsden
Quoted 6 lines

Yeah, I said ugly ones specificly. There's been some nice previous ones, but most in this series (all the move of stuff to subarches) are rather horrible and lack lots of explanation.

All of my previous patches have been aimed at fixing bugs, improving performance, reliability and maintinability of the i386 architecture.

Yup, with one or two semi-exceptions, all the patches up to this series seem to be good general cleanups - certainly it's good to move all those open-coded asm statements into single-site inlines and macros: people keep on screwing them up.

We do need to wake the Xen poeple up, make sure that these changes suit them as well, or at least don't screw them over (hard to see how it could though).

Quoted 5 lines

If you found something that didn't fit one of those categories in my previous patches, then it is either not well enough explained or perhaps inadvertently slipped through from one of my more radical trees - or it could be a bug.

Quoted 14 lines

There is a simple explanation for all of this series. The goal is to move all privileged instructions, sensitive instructions, and privilege awareness into a layer where it can be overridden by new code without disrupting the default architecture. On x86, there are a lot of instructions - popf, iret, sgdt, and others which behave differently under different privilege levels, but do not trap. These architectural features must be redefined by any architecture which virtualizes the x86, be it Xen, UML, or an alternative approach. Similarly, certain privileged processor data structures (page tables, descriptor tables) must be protected and accessed in a different manner if one is to utilize the principles of paravirtualization to achieve high performance inside of a virtual machine. I believe this series of patches is one almost clean solution to this. There are obvious problems with the MMU patch, and I'm still trying to come up with a way to properly address that.

Yes this later series is in a different category and will require more review/coordination/discussion/waking-up from the Xen team.

#7 lore raw
Quoted 4 lines · * Andrew Morton ([email protected]) wrote:

Yup, with one or two semi-exceptions, all the patches up to this series seem to be good general cleanups - certainly it's good to move all those open-coded asm statements into single-site inlines and macros: people keep on screwing them up.

I agree.

Quoted 3 lines

We do need to wake the Xen poeple up, make sure that these changes suit them as well, or at least don't screw them over (hard to see how it could though).

I have a series of similar patches that I've done for Xen that I'll post shortly.

thanks,
-chris

#8 lore raw
In reply to Andrew Morton
Quoted 15 lines · Andrew Morton
Quoted 11 lines · Zachary Amsden
Quoted 7 lines

Yeah, I said ugly ones specificly. There's been some nice previous ones,

but most in this series (all the move of stuff to subarches) are rather horrible and lack lots of explanation.

All of my previous patches have been aimed at fixing bugs, improving performance, reliability and maintinability of the i386 architecture.

Quoted 10 lines

Yup, with one or two semi-exceptions, all the patches up to this series seem to be good general cleanups - certainly it's good to move all those open-coded asm statements into single-site inlines and macros: people keep on screwing them up.

We do need to wake the Xen poeple up, make sure that these changes suit them as well, or at least don't screw them over (hard to see how it could though).

This patch in particular is still quite controversial. I know at least Andi has objections (quite valid) to the way PAE/non-PAE was dissected, and I would definitely like to address these concerns. Although I have no objection to you committing it to the mm tree right now, please be advised that Chris Wright and I will have to converge quite a bit on this patch, and will likely be doing a substantial amount of rework here to work out Xen compatibilty issues as well as general cleanliness. If it is more convenient for you to live without that churn, by all means feel free to, and we can update the patch once everyone is happy.

Zach