Skip to contentSkip to conversation

Kernel support for peer-to-peer protection models...

DISCUSSION Andi Kleen 3 participants 7 messages Mar 28, 2004 – Mar 30, 2004 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Andi Kleen

author
#1 lore raw
Quoted 6 lines · Ivan Godard

We're a processor startup with a new architecture that we will be porting Linux to. The bulk of the port will be straightforward (well, you know what I mean), except for the protection model supported by the hardware. How would you extend/mod the kernel if you had hardware that:

1) had a large number of distinguishable address spaces

Large or unlimited? If not unlimited you may still run into problems when you give each process such an address space. Limiting the number of processes is probably not an option.

Quoted 4 lines

2) any running code had two of these (code and data environment) it could use arbitrarily, but access to addresses in others was arbitrarily protected
3) flat, unified virtual addresses (64 bit) so that pointers, including inter-space pointers, have the same representation in all spaces

You mean Address 0 is only accessible from Address space 0, but not from Space 1 ?

Maybe you can give each process an different address range, but AFAIK the only people who have done this before are users of non MMU architectures. It will probably require som changes in the portable part of the code. Also porting glibc's ld.so to this will be likely no-fun.

Quoted 3 lines

4) no "supervisor mode"
5) inter-space references require grant of access (transitive) by the accessed space; grants can be entire space or any contiguous subspace

Sounds like the only sane way to handle (4) would be to give the kernel an own address space with the necessary grants to access everything. However this will require an address space switch for every system call. But there is no way around it, linux requires a "shared" kernel mapping at least for part of the kernel memory ("lowmem")

Overall it sounds like your architecture is not very well suited to run Linux.

Quoted 2 lines

10) Drivers can have their own individual space(s) distinct from those of the kernel and the apps. Buggy drivers cannot crash the kernel.

At least you would need to use your own drivers (I believe the IBM iSeries and s390/VM port does it kind of). If your CPU has generic PCI slots this will be a lot of work. Without it it will be lots of work too, but at least the number of drivers required is limited.

Quoted 1 line

Is this model so alien to the existing Kernel that the best approach is to

It is definitely alien.

-Andi

#2 lore raw

----- Original Message -----
From: "Andi Kleen" <[email protected]>
To: "Ivan Godard" <[email protected]>
Cc: <[email protected]>
Sent: Friday, March 26, 2004 10:29 PM
Subject: Re: Kernel support for peer-to-peer protection models...

Quoted 3 lines
Quoted 1 line · Ivan Godard

We're a processor startup with a new architecture that we will be

porting

Quoted 1 line
Quoted 1 line

Linux to. The bulk of the port will be straightforward (well, you know

what

Quoted 8 lines
Quoted 4 lines

I mean), except for the protection model supported by the hardware. How would you extend/mod the kernel if you had hardware that:

1) had a large number of distinguishable address spaces

Large or unlimited? If not unlimited you may still run into problems when you give each process such an address space. Limiting the number of processes is probably not an option.

Large but not unlimited - tens of thousands. Think PIDs. The number of threads (sharing common address spaces) is not limited.

Quoted 1 line
Quoted 1 line

2) any running code had two of these (code and data environment) it

could

Quoted 1 line
Quoted 1 line

use arbitrarily, but access to addresses in others was arbitrarily

protected

Quoted 5 lines
Quoted 2 lines

3) flat, unified virtual addresses (64 bit) so that pointers, including inter-space pointers, have the same representation in all spaces

You mean Address 0 is only accessible from Address space 0, but not from Space 1 ?

A field in a (64 bit) pointer gives the address space number. Each space has an "address 0" which is actually the zeroth byte *in that space* - an actual pointer is a space number and an index in that space. You (i.e. a process) run with a native space that determines your priveledges. If you are running as space 0 then you may or may not be able to address space 1 - it depends on the access you inherited or have obtained. Think mmap. You (almost) always can address all your native space, although there might not be anything there.

Quoted 2 lines

Maybe you can give each process an different address range, but AFAIK the only people who have done this before are users of non MMU

architectures.

Quoted 2 lines

It will probably require som changes in the portable part of the code. Also porting glibc's ld.so to this will be likely no-fun.

Each process gets a different range because each process gets a different native space. Within that space processes can use the same offsets, and typically will so as to avoid pointless relocation.

Quoted 9 lines
Quoted 3 lines

4) no "supervisor mode"
5) inter-space references require grant of access (transitive) by the accessed space; grants can be entire space or any contiguous subspace

Sounds like the only sane way to handle (4) would be to give the kernel an own address space with the necessary grants to access everything. However this will require an address space switch for every system call. But there is no way around it, linux requires a "shared" kernel mapping at least for part of the kernel memory ("lowmem")

We could (trivially) emulate a monolithic kernel in a single space. But that loses the reliability improvement available if the kernel subsystens ran in their own spaces with grants of access to those common structures they individually needed. BTW, there's nothing to be gained by minimizing address switches - it's in hardware, and inter-space references and calls run at the same speed as same-space references and calls.

Quoted 2 lines

Overall it sounds like your architecture is not very well suited to run Linux.

We believe we can adopt the Linux protection model (i.e. the 386 protection model) with no more work than any other port to a new architectire (ahem). But the result would also be as prone to bugs and exploits as a 386 too.

Quoted 1 line
Quoted 1 line

10) Drivers can have their own individual space(s) distinct from those

of

Quoted 6 lines
Quoted 1 line

the kernel and the apps. Buggy drivers cannot crash the kernel.

At least you would need to use your own drivers (I believe the IBM iSeries and s390/VM port does it kind of). If your CPU has generic PCI slots this will be a lot of work. Without it it will be lots of work too, but at least the number of drivers required is limited.

So long as 1) a driver has a driver-load-time defined region of working data space; 2) has a defined code region; 3) gets its buffer addresses etc. as arguments; 4) calls defined OS APIs; and 5) never touches anything except its private code and data, its arguments, and syscalls then it can run in one of our protected environments and be none the wiser. That is, if the driver has been coded to look like a well behaved server process then all is well. If it has hard references to shared kernel data structures then it will break, because those shared spaces are not visible and must be accessed through a service call to someone who owns that structure.

Quoted 1 line
Quoted 1 line

Is this model so alien to the existing Kernel that the best approach is

to

Quoted 2 lines

It is definitely alien.

You don't know the half of it! This is just the protection model :-)

Ivan

author
#3 lore raw
Quoted 10 lines · On Sun, 28 Mar 2004 12:21:36 -0800 "Ivan Godard" <[email protected]> wrote:
Quoted 3 lines

Maybe you can give each process an different address range, but AFAIK the only people who have done this before are users of non MMU

architectures.

Quoted 2 lines

It will probably require som changes in the portable part of the code. Also porting glibc's ld.so to this will be likely no-fun.

Each process gets a different range because each process gets a different native space. Within that space processes can use the same offsets, and typically will so as to avoid pointless relocation.

fork() will be hard and/or inefficient this way.

Quoted 5 lines
Quoted 2 lines

Overall it sounds like your architecture is not very well suited to run Linux.

We believe we can adopt the Linux protection model (i.e. the 386 protection model) with no more work than any other port to a new architectire (ahem).

Just FYI - Linux has been ported to several architectures with similar SASOS capabilities in hardware (IA64 or ppc64 on iseries) and they have all opted to use an conventional protection model.

Quoted 2 lines

So long as 1) a driver has a driver-load-time defined region of working data space; 2) has a defined code region; 3) gets its buffer addresses etc. as

Just (1) alone is a illusion - linux drivers generally work on the shared page pool, just like all other subsystems.

-Andi

#4 lore raw

interlinear

----- Original Message -----
From: "Andi Kleen" <[email protected]>
To: "Ivan Godard" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, March 28, 2004 3:14 PM
Subject: Re: Kernel support for peer-to-peer protection models...

Quoted 11 lines
Quoted 8 lines · On Sun, 28 Mar 2004 12:21:36 -0800 "Ivan Godard" <[email protected]> wrote:
Quoted 3 lines

Maybe you can give each process an different address range, but AFAIK the only people who have done this before are users of non MMU

architectures.

Quoted 2 lines

It will probably require som changes in the portable part of the code. Also porting glibc's ld.so to this will be likely no-fun.

Each process gets a different range because each process gets a

different

Quoted 4 lines
Quoted 2 lines

native space. Within that space processes can use the same offsets, and typically will so as to avoid pointless relocation.

fork() will be hard and/or inefficient this way.

Why? The load image for the new process does not require relocation, so all that's necessary to spawn a process is to allocate a spaceID, map the excutable to that ID in the page tables, push one entry into the TLB, set a couple of hardware registers, and insert it into the readyq. When it get's control it will prompty fault in its code pages (if not already present), and execution from there on is normal. This process is essentially identical to what happens on a conventional, except because there is no aliasing of addreses (flat unified 64-bit model) you don't have to scrub the cache or TLB.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Overall it sounds like your architecture is not very well suited to run Linux.

We believe we can adopt the Linux protection model (i.e. the 386

protection

Quoted 1 line
Quoted 1 line

model) with no more work than any other port to a new architectire

(ahem).

Quoted 2 lines

Just FYI - Linux has been ported to several architectures with similar

SASOS

Quoted 1 line

capabilities in hardware (IA64 or ppc64 on iseries) and they have all

opted to use

Quoted 1 line

an conventional protection model.

Do you know why? Can you point me to the people who did these ports so I can ask?

Quoted 1 line
Quoted 1 line

So long as 1) a driver has a driver-load-time defined region of working

data

Quoted 1 line
Quoted 1 line

space; 2) has a defined code region; 3) gets its buffer addresses etc.

as

Quoted 3 lines

Just (1) alone is a illusion - linux drivers generally work on the shared page pool, just like all other subsystems.

In 1) I'm talking about the driver's local state, not the pages it's trying to fill. That local state will be in the driver's space, and protected from interference by everybody else.

The pages (I assume) are arguments to the driver, i.e. "Fill *here*", and the owner of the page will have exposed the page to the driver before or as part of making the call. Or the call was "Fill some page and return it", and the driver calls the physmem manager who allocates the page, exposes it to the driver, and reurns the address to the driver. When the driver is done it will hand off ownership of the page to the client. I fully expect that the present code for this mechanism will have to be mangled, but I suspect that the kernel already implements some concept of "owner" for a physpage and we can hook the ownership transfer into our model.

I hope :-)

Ivan

#5 lore raw
In reply to Andi Kleen

Hi!

Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

Overall it sounds like your architecture is not very well suited to run Linux.

We believe we can adopt the Linux protection model (i.e. the 386 protection model) with no more work than any other port to a new architectire (ahem).

Just FYI - Linux has been ported to several architectures with similar SASOS capabilities in hardware (IA64 or ppc64 on iseries) and they have all opted to use an conventional protection model.

It might be actually plus for Ivan: if ia64 and ppc64 benefit from changes for mill, it makes them more acceptable.

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
author
#6 lore raw
Quoted 4 lines · On Mon, 29 Mar 2004 17:36:06 +0200 Pavel Machek <[email protected]> wrote:
Quoted 1 line

It might be actually plus for Ivan: if ia64 and ppc64 benefit from changes for mill, it makes them more acceptable.

I doubt either of them will completely redesign their port at this point.

-Andi

#7 lore raw
In reply to Pavel Machek

----- Original Message -----
From: "Pavel Machek" <[email protected]>
To: "Andi Kleen" <[email protected]>
Cc: "Ivan Godard" <[email protected]>; <[email protected]>
Sent: Monday, March 29, 2004 7:36 AM
Subject: Re: Kernel support for peer-to-peer protection models...

Quoted 6 lines

Hi!

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Overall it sounds like your architecture is not very well suited to run Linux.

We believe we can adopt the Linux protection model (i.e. the 386

protection

Quoted 1 line
Quoted 1 line
Quoted 1 line

model) with no more work than any other port to a new architectire

(ahem).

Quoted 2 lines
Quoted 2 lines

Just FYI - Linux has been ported to several architectures with similar

SASOS

Quoted 1 line
Quoted 1 line

capabilities in hardware (IA64 or ppc64 on iseries) and they have all

opted to use

Quoted 5 lines
Quoted 2 lines

an conventional protection model.

It might be actually plus for Ivan: if ia64 and ppc64 benefit from changes for mill, it makes them more acceptable.

Interesting point. Although it's not clear how long ia64 will still exist. Remember the Alpha?

Ivan