GSSAPI Proxy initiative

Hello everyone,
you are all CCed because you either expressed interested, are already involved with or may be affected/interested in this initiative.

At the recent MIT Kerberos Conference there has been some renewed interested in a concept that has surfaced in GSSAPI related circles a few times. That of building a GSSAPI Proxy to allow separation of privileges between applications and key handling.

There are various stakeholders that would like to have a solution in this areas for various reasons:

== Privilege separation ==

One of the needs that GSSAPI Proxy is able to address is separation of privileges. Initiating/accepting is delegated to the GSSAPI Proxy so that applications do not have direct access to the key material. This helps to avoid exposing keys when network facing applications are compromised. On some OSs Mandatory Access Control will help keep those keys secure if they are not directly accessible to the applications.

== Kernel Network File Systems ==

There are a number of file systems implemented in kernel and especially NFS that uses gss-krb5 to negotiate security over the network.

Most implementations use a gssd daemon in user space to handle accepting/initiating. In particular in the Linux kernel a few people feel that the current implementation is sub-optimal and would like to improve it. The functionality needed by the kernel is pretty similar to that needed to proxy gssapi for other apps, so it seem like a good idea to unify these implementations, or at least use a common protocol so that the same implementation can be used if OS vendors/Integrators want to do so.

== PAC/PAD verification ==

When using Kerberos applications can receive tickets that contain a MS-PAC or (in future) a PAD*.
In some scenarios we want to use the PAC to synthesize a user on the target system so that it can create and access files and other resources. A PAC/PAD is very useful for file servers for example because it can standardize how identities are retrieved w/o forcing file server developers to mess with protocols like LDAP and other amenities.

These structures are signed with service keys, the same keys apps have currently access to. If key material is left accessible to apps it means a compromised application could create a PAC/PAD to attempt privilege escalation by injecting this data in the system.

Although these structures are also signed with a KDC key, it would be very expensive to require a roundtrip to the KDC to ask for verification. It would also make the server reliant on KDC availability which is normally not required for kerberos based auth after the ticket is acquired.

== SSH credential forwarding ? ==

Currently, when using GSSAPI in conjunction with SSH, credentials are made available to the remote end by simply copying them over and thus giving the remote host direct access to the user creds. Quite a few people would prefer to use a ssh-agent like approach instead where credentials stay always on the local machine and context establishment is proxied over the ssh channel. Although this initiative will *not* address this feature, we'd like to build the protocol in such a way that it will become easy to implement this mechanism in the future.

==== Current status ====

I am trying to push this initiative and I am organizing and contacting people to enter the planning phase. I have received initial availability from some stakeholders in helping to shape this initiative from protocol design to transport/interface (esp. against the kernel) design, implementation, testing, etc..

Because there are many parties involved I would like to try to keep this discussion focused as much as possible to the proxy functionality.

I see this effort involving mainly 3 areas at this stage.

1) design of the protocol

I this area I had some preliminary discussion with some of the people that are interested in the effort. Due to availability in both kernel and user space implementation is seem like an ONC/RPC - XDR based protocol is the best common denominator and easily implementable by all parties.
Nothing is set in stone yet, any idea is welcome.

2) kernel and user space interfaces to the GSSAPI Proxy Daemon

For user-space to user-space communication is seem clear that a unix socket is probably the more natural transport to use.

From the Linux kernel side deciding on the right upcall mechanism seem not as straight forward. One option seem to be the netlink or rather the connector interface that is layered on top of it. More discussion is definitely needed on this front.

3) Changes to libgssapi to implement transparent proxying for user space applications

Some high level ideas have been floating around on where to hook this proxy layer in libgssapi but it is all in the air for now. Ideas here are very welcome.

Additional areas involve the design of the daemon in terms of scalability, access control and various security considerations, portability (?), etc...

Any feedback is welcome esp. if you have valuable ideas on the design issues.
We will probably keep cross-posting a bit so that we can involve all possible stakeholders, going forward we'll find a way to refocus discussions around a few forums depending on what part of the problem people is working on.
If you know other people/lists that should be involved, please feel free to forward and include them in the discussion.

Regards,
Simo.

Hi Simo,

As you know, I completely agree that we need this. We have several implementations, of course (in Linux, in Solaris, probably elsewhere, such as the BSDs, as well as proprietary ones), but all are -to my knowledge- focused on providing services to kernel-mode distributed filesystems (AFS, NFS, Lustre -- all come to mind). But we need to generalize this into a service for user-land consumers too, not just kernel mode consumers.

A few thoughts:

- XDR is quite fine for this.

- We want stateless GSS daemons, or mostly stateless at least, for
several reasons:

   a) to protect them against DoS, intentional and not;

b) to make GSS daemon restart a non-event (OK, this is a much
lesser concern).

This basically requires credential handle export and
partially-established security context export support. It also requires composite name export (to capture name attributes, which won't be captured by normal exported names).

Statelessness is not strictly necessary, just highly desirable, so
if this proves difficult (or in some cases credential export turns out to not possibly be stateless for some mechanisms/credentials), oh well. But the *protocol* should be designed to facilitate statelessness, and certainly not preclude it.

Note: I really don't want to have an extended sub-thread about
exporting partially-established sec contexts. We've had that enough at KITTEN WG, so unless you (whoever wants to bring this up) are familiar with those threads, please don't bring up the subject, and if you must, then please use a separate subject.

- I would like mechglues to support mechanisms being provided by
multiple providers. This is tricky, and for some mechanisms it will not be possible to make this perfect, but it's worthwhile The reason is this: it should be possible to use a proxy for some credentials and not a proxy for others (particularly on the initiator side, where a user may not have direct access to some of their credentials but maybe can always kinit ones that they do have access to).

This too is a nice to have rather than a critical feature, but if
we don't have this then we need to make it possible for all GSS apps to run with just the proxy provider as the one provider (or: the proxy provider *is* the mechglue as far as the application is concerned). Otherwise we can get into situations where a library should use the proxy while another should not, both in the same process, and that could be obnoxious.

- Finding credentials, and authorization, will be a bit of a problem. How to deal with things like kernel keyrings and KRB5CCNAME-type
environment variables?

One possibility is that the proxy client passes everything the
proxy server needs to the proxy server. But that still requires that the proxy server be able to implement authorization correctly.

For authorization it will be best to run a proxy for each user or
{user, session} if there's session isolation, or {user, label} for labeled systems. This approach helps with finding credentials too.

- But the more GSS proxy server daemon processes we have, the more we
need to worry about ensuring timely termination.

- Thus the protocol should look very much like an XDR type for each
GSS function's input arguments and another for each functions output arguments and return value, plus, for functions dealing in exportable objects, exported objects on input and output, plus, perhaps, environmental / credential location information for certain functions.

The environmental / credential location information may not be needed
in some operating systems (because the proxy server will be able to use APIs like door_ucred(3DOOR), on Solaris, to find out what it needs about the caller), and in any case will need to be either a typed hole a an opaque type defined by each OS.

- Preferably we should have a set pattern for mapping {argument
direction, C type} to/from XDR types that can be applied in an automated fashion to the GSS C bindings header file(s), thus making it possible to avoid having to specify the abstract syntax for the protocol. I believe this is entirely feasible.

Of course, for extensibility purposes we need to be careful to use
wide integer XDR types for any C integer types in the API, and in particular we may want to use octet strings for flags arguments, just in case we widen any of these in a possible future major version of the C bindings (yes, the function names would have to change) and so that we don't unnecessarily constrain other language bindings. (Yes, I know that for req_flags, for example, widening the argument has knock on effects on the wire for actual mechanisms. I'm only asking that we be as forward-looking as possible in the GSS proxy protocol; I'm not proposal actual changes to the API.)

- To make it possible to some day consider the "PGSS" or variant
proposals, we should make the representation of the minor_status argument extensible and both, an input and output.

- The protocol probably need not be straight up ONC RPC however,
though where ONC RPC supports suitable IPC interfaces as transports, it'd certainly be worth considering.

- New functions should be possible to add at any time. For ONC RPC
this would mean we'd need to have at least a registry of procedure numbers. For a non-ONC RPC protocol we could use function name strings to name the functions "on the wire", which would then obviate the need for a new registry, simplifying the addition of extensions.

I'll probably have more to add later.

Quoted 3 lines · Nico Williams

 - We want stateless GSS daemons, or mostly stateless at least, for several reasons:

Quoted 1 line

   a) to protect them against DoS, intentional and not;

I get this.

Quoted 2 lines

   b) to make GSS daemon restart a non-event (OK, this is a much lesser concern).

I am not sure I get this one, can you expand.

Also how do you plan to keep conversations going and still be stateless ? I am not sure transferring state back and forth is necessarily a good thing.

Quoted 4 lines

   This basically requires credential handle export and partially-established security context export support.  It also requires composite name export (to capture name attributes, which won't be captured by normal exported names).

Given one of the aims is to perform privilege separation I wonder if exporting partially-established security contexts is going to expose us to some risk of disclosing to the application stuff we should not. Should we sign/seal each partially established context to avoid tampering ?

Isn't this going to be quite expensive ?

Quoted 5 lines

   Statelessness is not strictly necessary, just highly desirable, so if this proves difficult (or in some cases credential export turns out to not possibly be stateless for some mechanisms/credentials), oh well.  But the *protocol* should be designed to facilitate statelessness, and certainly not preclude it.

I do not want to preclude anything if possible, but I also would like to avoid over-complicating it. Being stateless often makes it slower as

state has to be reconstructed instead of simply being directly available. But I guess we will see how it goes during the implementation phase. I hear you on the DoS side, but there are other methods to prevent DoSs, like rate limiting, etc.

Quoted 7 lines

 - I would like mechglues to support mechanisms being provided by multiple providers.  This is tricky, and for some mechanisms it will not be possible to make this perfect, but it's worthwhile  The reason is this: it should be possible to use a proxy for some credentials and not a proxy for others (particularly on the initiator side, where a user may not have direct access to some of their credentials but maybe can always kinit ones that they do have access to).

The initiator side seem more complex indeed, but in general we need to

discuss how proxy vs non-proxy is going to be selected. The method may differ between initiatior and acceptor as they are objectively different cases. And it may differ also based on what app is using libgssapi.

How I am not sure, one of the objectives is to keep this almost completely transparent to current user space applications, but I guess adding new API to add some sort of awareness is a possibility. But we should avoid adding anything unless it really is absolutely needed.

Quoted 7 lines

   This too is a nice to have rather than a critical feature, but if we don't have this then we need to make it possible for all GSS apps to run with just the proxy provider as the one provider (or: the proxy provider *is* the mechglue as far as the application is concerned). Otherwise we can get into situations where a library should use the proxy while another should not, both in the same process, and that could be obnoxious.

Do you have an example in mind, I think it would benefit everyone to wrap their minds around these needs/issues if we provide some concrete examples here and there.

Quoted 7 lines

 - Finding credentials, and authorization, will be a bit of a problem. How to deal with things like kernel keyrings and KRB5CCNAME-type environment variables?

   One possibility is that the proxy client passes everything the proxy server needs to the proxy server.  But that still requires that the proxy server be able to implement authorization correctly.

I think we need to pass everything, and yet the proxy MUST be allowed to

decide what to trust and what not of course. Also we should not preclude

future developments where the proxy becomes capable of applying policy in order to deny some operations. This means we need to make it very clear to the other side when an error means it should try to proceed w/o the proxy and when an error means the proxy decided to deny the operation and so the library should drop everything and return an error.

Quoted 3 lines

   For authorization it will be best to run a proxy for each user or {user, session} if there's session isolation, or {user, label} for labeled systems.  This approach helps with finding credentials too.

I do not think we should dictate how the proxy is run. On embedded

systems there will be a request to keep it small. I am also not sure that forking a process for each user is desirable. It means having a process sitting there doing nothing most of the time. The GSSAPI Proxy should be considered a very trusted service. I was thinking of a multi-threaded design in order to scale when many requests come in at once (I am thinking of servers being hit in the morning when all users log in roughly at the same time), but that can scale back when load spikes are gone.

Quoted 2 lines

 - But the more GSS proxy server daemon processes we have, the more we need to worry about ensuring timely termination.

This is one reason why I do not favor multiple processes per user.

Quoted 5 lines

 - Thus the protocol should look very much like an XDR type for each GSS function's input arguments and another for each functions output arguments and return value, plus, for functions dealing in exportable objects, exported objects on input and output, plus, perhaps, environmental / credential location information for certain functions.

Agreed.

Quoted 5 lines

 The environmental / credential location information may not be needed in some operating systems (because the proxy server will be able to use APIs like door_ucred(3DOOR), on Solaris, to find out what it needs about the caller), and in any case will need to be either a typed hole a an opaque type defined by each OS.

I am not sure I really want to care about OSs that cannot ask the kernel who's on the other side of a pipe. I would rather let whoever cares about those to send patches later, they can be as simple as requiring the creation of multiple sockets with appropriate permissions. After all if the kernel does not help we cannot make choice of how much trust what the user is sending us stuff anyway, either we always trust all or nothing in those cases.

Quoted 5 lines

 - Preferably we should have a set pattern for mapping {argument direction, C type} to/from XDR types that can be applied in an automated fashion to the GSS C bindings header file(s), thus making it possible to avoid having to specify the abstract syntax for the protocol.  I believe this is entirely feasible.

Looks like worth putting some effort into this, would keep stuff simpler and leaner.

Quoted 3 lines

 - To make it possible to some day consider the "PGSS" or variant proposals, we should make the representation of the minor_status argument extensible and both, an input and output.

I am not familiar with 'PGSS' care to elaborate if it is really something we need to care about ?

Quoted 3 lines

 - The protocol probably need not be straight up ONC RPC however, though where ONC RPC supports suitable IPC interfaces as transports, it'd certainly be worth considering.

We definitely MUST NOT require ONC RPC for the protocol, if someone wants to use it for something specific and build it on our work I am quite fine and I do not want to prevent it. But I thikn we do not want to force all that overhead on Kernels and other implementations by default as we do not really need it for a local daemon.

Quoted 6 lines · Simo Sorce

Quoted 4 lines · Nico Williams

- We want stateless GSS daemons, or mostly stateless at least, for
several reasons:

   a) to protect them against DoS, intentional and not;

Quoted 6 lines

I get this.

Quoted 2 lines

b) to make GSS daemon restart a non-event (OK, this is a much
lesser concern).

I am not sure I get this one, can you expand.

If the proxy daemon process dies, how should the proxy client recover?

If the client merely knows about credential handles for credentials
stored in the proxy, then those are invalidated. If the client has [encrypted] exported credential handle tokens then the new daemon process will be able to decrypt and import the client's credential handles. Statelessness is helpful.

Quoted 3 lines

Also how do you plan to keep conversations going and still be stateless ? I am not sure transferring state back and forth is necessarily a good thing.

What I'd like to see is the state cookies passed to the client be structured something like this (note that this would be an implementation detail, not part of the protocol spec): {object reference, verifier cookie, exported object token, [key ID for key that encrypted the exported token object, if any]}. This allows for speed (no need to re-import state all the time) and statelessness (the server can restart, the server can push state out of a fixed-sized cache).

Quoted 8 lines

Quoted 5 lines

Statelessness is not strictly necessary, just highly desirable, so
if this proves difficult (or in some cases credential export turns out to not possibly be stateless for some mechanisms/credentials), oh well. But the *protocol* should be designed to facilitate statelessness, and certainly not preclude it.

I do not want to preclude anything if possible, but I also would like to avoid over-complicating it. Being stateless often makes it slower as

There's no complication. The protocol needs to allow for arbitrary-sized object handles -- that's all.

Quoted 9 lines

Quoted 7 lines

- I would like mechglues to support mechanisms being provided by
multiple providers. This is tricky, and for some mechanisms it will not be possible to make this perfect, but it's worthwhile The reason is this: it should be possible to use a proxy for some credentials and not a proxy for others (particularly on the initiator side, where a user may not have direct access to some of their credentials but maybe can always kinit ones that they do have access to).

The initiator side seem more complex indeed, but in general we need to

Let's agree to be careful with client/server vs. initiator/acceptor. The latter should refer exactly to the GSS meanings of "initiator" and "acceptor". The former should refer to the proxy client and server.

Quoted 3 lines

discuss how proxy vs non-proxy is going to be selected. The method may differ between initiatior and acceptor as they are objectively different cases. And it may differ also based on what app is using libgssapi.

The GSS-API lacks a concept of caller context handl -- an equivalent to krb5_context. This is a problem. We can't speak of who the application is, not in a world where many libraries use the GSS-API, because there can be multiple applications in one process and we can't easily distinguish which application the caller is.

I refer you to past discussions of how to address this. I really want to avoid that subject for now though.

Quoted 3 lines

Do you have an example in mind, I think it would benefit everyone to wrap their minds around these needs/issues if we provide some concrete examples here and there.

Layered software. Particularly pluggable layers like PAM.

Quoted 8 lines

Quoted 7 lines

- Finding credentials, and authorization, will be a bit of a problem. How to deal with things like kernel keyrings and KRB5CCNAME-type
environment variables?

One possibility is that the proxy client passes everything the
proxy server needs to the proxy server. But that still requires that the proxy server be able to implement authorization correctly.

Quoted 1 line

I think we need to pass everything, and yet the proxy MUST be allowed to

I'd much rather not have to pass anything, but we need to support OSes where that's just how it's done. I'd rather that IPC end-points can find out what what they need about their peers. Think door_ucred(3DOOR).

Quoted 1 line

decide what to trust and what not of course. Also we should not preclude

Trust nothing. Authorization is separate.

Indeed, I'm not interested in dictating implementation details. It's going to be very difficult to avoid discussing possible implementation details, so I don't bother trying not to :) But the protocol itself needs to be agnostic regarding these things.

Multi-threaded is fine. On Solaris it'd have to be multi-processed because it's not possible to have different process credentials for different threads of the same process in Solaris -- ias with this in mind that I mentioned the possible [implementation-specific detail of a] design where there's a daemon process per {user session}, {user, label}, ...

I suppose I need to be more careful in separating what I intend to be part of a standard and what I intend to be an implementation detail when I describe these sorts of things. To me it's clear, but perhaps not so much to others.

Quoted 4 lines

Quoted 3 lines

- The protocol probably need not be straight up ONC RPC however,
though where ONC RPC supports suitable IPC interfaces as transports, it'd certainly be worth considering.

Quoted 5 lines

We definitely MUST NOT require ONC RPC for the protocol, if someone wants to use it for something specific and build it on our work I am quite fine and I do not want to prevent it. But I thikn we do not want to force all that overhead on Kernels and other implementations by default as we do not really need it for a local daemon.

ONC RPC has next to no overhead when the transport is something like doors, for example.

Quoted 1 line · Nico Williams

If the proxy daemon process dies, how should the proxy client recover?

Either wait for the proxy to restart or treat it as an unrecoverable error, just like if the network goes away ?

Ok, I can see how this may help.

Quoted 2 lines

There's no complication.  The protocol needs to allow for arbitrary-sized object handles -- that's all.

Ok, I was complaining about making the server more complicated, but that's probably not really true, we will just trade one set of issues with another as the state is kept 'somewhere' anyway, so I retire my concern.

Quoted 4 lines

I'd much rather not have to pass anything, but we need to support OSes where that's just how it's done.  I'd rather that IPC end-points can find out what what they need about their peers.  Think door_ucred(3DOOR).

I don't think I want to use the PID and then try to pull out environment variables through /proc or similar interface, that would be bad.

For the krb5 mech we only care about a handful of environment variables.

Quoted 1 line

Trust nothing.  Authorization is separate.

That depends on the client.
When the client is the kernel we can decide to trust whatever it says.

Quoted 4 lines

Indeed, I'm not interested in dictating implementation details.  It's going to be very difficult to avoid discussing possible implementation details, so I don't bother trying not to :)  But the protocol itself needs to be agnostic regarding these things.

ACK

Quoted 6 lines

Multi-threaded is fine.  On Solaris it'd have to be multi-processed because it's not possible to have different process credentials for different threads of the same process in Solaris -- ias with this in mind that I mentioned the possible [implementation-specific detail of a] design where there's a daemon process per {user session}, {user, label}, ...

I am not sure you need that separation, but as it is an implementation detail I think we should set discussion around this aside for now.

It is ok to agree that any time we mention implementation details of the proxy itself these are not binding, but just a tool to explain why something may be needed.

Quoted 10 lines

Quoted 9 lines

Quoted 3 lines

 - The protocol probably need not be straight up ONC RPC however, though where ONC RPC supports suitable IPC interfaces as transports, it'd certainly be worth considering.

We definitely MUST NOT require ONC RPC for the protocol, if someone wants to use it for something specific and build it on our work I am quite fine and I do not want to prevent it. But I thikn we do not want to force all that overhead on Kernels and other implementations by default as we do not really need it for a local daemon.

Quoted 2 lines

ONC RPC has next to no overhead when the transport is something like doors, for example.

I mean conceptual and development over head, not talking about performance.

Quoted 5 lines · Simo Sorce

Quoted 1 line · Nico Williams

If the proxy daemon process dies, how should the proxy client recover?

Either wait for the proxy to restart or treat it as an unrecoverable error, just like if the network goes away ?

If state is lost the client has to recover. Sure, it's going to somehow (perhaps by returning an error to the user, who then retries).
Point is: a stateless (or stateless + caching, for perf) design would
avoid this.

For the protocol that just means that handles need to be opaque octet strings, NOT just small integers. Whether a given implementation is stateless is another story. This is what I was driving at.

Quoted 1 line

Ok, I can see how this may help.

:)

Quoted 7 lines

Quoted 2 lines

There's no complication. The protocol needs to allow for arbitrary-sized object handles -- that's all.

Ok, I was complaining about making the server more complicated, but that's probably not really true, we will just trade one set of issues with another as the state is kept 'somewhere' anyway, so I retire my concern.

Right.

Quoted 7 lines

Quoted 4 lines

I'd much rather not have to pass anything, but we need to support OSes where that's just how it's done. I'd rather that IPC end-points can find out what what they need about their peers. Think door_ucred(3DOOR).

I don't think I want to use the PID and then try to pull out environment variables through /proc or similar interface, that would be bad.

I would never have suggested that.

What I had in mind was something like PAGs or keyrings. Or, to be much more specific, search for my name and the string "credentials process groups" -- a PAG on steroids.

The idea is that the IPC peer can observe the other's PAG/keyring/CPG/whatever and use that to find the correct credentials (authorization is still required though).

Quoted 4 lines

What I had in mind was something like PAGs or keyrings.  Or, to be much more specific, search for my name and the string "credentials process groups" -- a PAG on steroids.

Quoted 3 lines

The idea is that the IPC peer can observe the other's PAG/keyring/CPG/whatever and use that to find the correct credentials (authorization is still required though).

Linux already has per-user, per-process and per-thread keyrings which offer a high security storage solution for keys. The problem with those is that they are difficult to use in an asynchronous context when the original user's process/thread context is no longer available to us.

Ideally, though, that's what we'd like to see used.

Quoted 4 lines · Trond Myklebust

Quoted 4 lines

What I had in mind was something like PAGs or keyrings. Or, to be much more specific, search for my name and the string "credentials process groups" -- a PAG on steroids.

Quoted 8 lines

Quoted 3 lines

The idea is that the IPC peer can observe the other's PAG/keyring/CPG/whatever and use that to find the correct credentials (authorization is still required though).

Linux already has per-user, per-process and per-thread keyrings which offer a high security storage solution for keys. The problem with those is that they are difficult to use in an asynchronous context when the original user's process/thread context is no longer available to us.

For async IPC methods you'd want something like SCM_CREDENTIALS to give you the keyring/PAG/whatever information you need abou thte peer.
The ancillary data should be complete enough that you can past the
client process/thread being dead, although it's nice to not have to process a request from a dead entity...

For sync IPC you need something like door_ucred(). And for sync IPC you can make sure to get SIGCANCEL or equivalent when the client gets canceled (this is the default in doors).

Quoted 1 line

Ideally, though, that's what we'd like to see used.

Agreed!

Quoted 21 lines · Nico Williams

Quoted 4 lines · Trond Myklebust

Quoted 4 lines

What I had in mind was something like PAGs or keyrings. Or, to be much more specific, search for my name and the string "credentials process groups" -- a PAG on steroids.

Quoted 8 lines

Quoted 3 lines

The idea is that the IPC peer can observe the other's PAG/keyring/CPG/whatever and use that to find the correct credentials (authorization is still required though).

Linux already has per-user, per-process and per-thread keyrings which offer a high security storage solution for keys. The problem with those is that they are difficult to use in an asynchronous context when the original user's process/thread context is no longer available to us.

For async IPC methods you'd want something like SCM_CREDENTIALS to give you the keyring/PAG/whatever information you need abou thte peer.
The ancillary data should be complete enough that you can past the
client process/thread being dead, although it's nice to not have to process a request from a dead entity...

Quoted 7 lines

For sync IPC you need something like door_ucred(). And for sync IPC you can make sure to get SIGCANCEL or equivalent when the client gets canceled (this is the default in doors).

Quoted 1 line

Ideally, though, that's what we'd like to see used.

Agreed!

I have discussed use of the keyring in a private discussion before starting the thread, and it turns out the keyring has a number of limitations that makes it probably unsuitable for this project.

As an upcall mechanism it has some limitations on the size of the payloads, IIRC limited to a page, and that means that you cannot pass blobs carrying big kerberos tickets.

As a storage mechanism for credential caches it also has size limits. Currently the limit is 20k per user which is not even enough to hold a single ticket in some cases. This limit can be increased of course, but then you end keeping around a huge amount of unswappable ram depending on the number of users.

So for long term storage of credentials we will probably not rely on kernel keyrings, nor as an upcall mechanism.

Quoted 38 lines · Simo Sorce

Quoted 21 lines · Nico Williams

Quoted 4 lines · Trond Myklebust

Quoted 4 lines

What I had in mind was something like PAGs or keyrings. Or, to be much more specific, search for my name and the string "credentials process groups" -- a PAG on steroids.

Quoted 8 lines

Quoted 3 lines

The idea is that the IPC peer can observe the other's PAG/keyring/CPG/whatever and use that to find the correct credentials (authorization is still required though).

Linux already has per-user, per-process and per-thread keyrings which offer a high security storage solution for keys. The problem with those is that they are difficult to use in an asynchronous context when the original user's process/thread context is no longer available to us.

For async IPC methods you'd want something like SCM_CREDENTIALS to give you the keyring/PAG/whatever information you need abou thte peer.
The ancillary data should be complete enough that you can past the
client process/thread being dead, although it's nice to not have to process a request from a dead entity...

Quoted 7 lines

For sync IPC you need something like door_ucred(). And for sync IPC you can make sure to get SIGCANCEL or equivalent when the client gets canceled (this is the default in doors).

Quoted 1 line

Ideally, though, that's what we'd like to see used.

Agreed!

I have discussed use of the keyring in a private discussion before starting the thread, and it turns out the keyring has a number of limitations that makes it probably unsuitable for this project.

As an upcall mechanism it has some limitations on the size of the payloads, IIRC limited to a page, and that means that you cannot pass blobs carrying big kerberos tickets.

Quoted 5 lines

As a storage mechanism for credential caches it also has size limits. Currently the limit is 20k per user which is not even enough to hold a single ticket in some cases. This limit can be increased of course, but then you end keeping around a huge amount of unswappable ram depending on the number of users.

Allowing keys to be swapped out is a really really really really really really bad idea when your kernel depends upon them being available as part of the memory reclaim process, as would be the case when you are talking about NFS, CIFS or AFS dirty page writebacks.

Quoted 2 lines

So for long term storage of credentials we will probably not rely on kernel keyrings, nor as an upcall mechanism.

So this would be an argument for scrapping keyrings from the kernel? If they're not good for kerberos tickets, why would we want to keep them at all?

What would you replace them with, given the above requirements concerning swapping?

No, but we still need to be able to do recovery of rpcsec_gss contexts once they are broken, and right now we have a major flaw due to the fact that recovery depends on a lot of small processes and data that is allowed to be swapped out at the moment when we need them the most (i.e. in a memory reclaim situation).

If the server reboots while our client is in the middle of writing back a file (or several files), then the client needs to recover those rpcsec_gss contexts that authenticate the processes which own any dirty pages that remain to be written out.
Key security is an irrelevant concern once your kernel deadlocks in an OOM state.

Quoted 3 lines

Currently credential caches are stored in files, is there a problem with that model ? Do you need access to credential caches from the kernel when under memory pressure ?

Yes, there is a major problem with that model, and yes we do potentially need access to credential caches when in a recovery situation (which is a situation when we are usually under memory pressure).

On Thu, 2011-11-03 at 15:16 -0700, Myklebust, Trond wrote:

[..]

Quoted 12 lines

No, but we still need to be able to do recovery of rpcsec_gss contexts once they are broken, and right now we have a major flaw due to the fact that recovery depends on a lot of small processes and data that is allowed to be swapped out at the moment when we need them the most (i.e. in a memory reclaim situation).

If the server reboots while our client is in the middle of writing back a file (or several files), then the client needs to recover those rpcsec_gss contexts that authenticate the processes which own any dirty pages that remain to be written out.
Key security is an irrelevant concern once your kernel deadlocks in an OOM state.

[..]

Quoted 8 lines

Quoted 3 lines

Currently credential caches are stored in files, is there a problem with that model ? Do you need access to credential caches from the kernel when under memory pressure ?

Yes, there is a major problem with that model, and yes we do potentially need access to credential caches when in a recovery situation (which is a situation when we are usually under memory pressure).

This sounds like a catch 22 situation.
Even if the keys are pinned in kernel memory you still need the GSSAPI Proxy daemon in order to re-establish the security context, and that process could have been swapped off too.

Quoted 3 lines

It is ok to use keyring if that's deemed the right place for session keys, but I think you already have structures where you currently store them so I don't thik you necessarily need to change that part of the kernel implementation.

No, but we still need to be able to do recovery of rpcsec_gss contexts once they are broken, and right now we have a major flaw due to the fact that recovery depends on a lot of small processes and data that is allowed to be swapped out at the moment when we need them the most (i.e. in a memory reclaim situation).

If the server reboots while our client is in the middle of writing back a file (or several files), then the client needs to recover those rpcsec_gss contexts that authenticate the processes which own any dirty pages that remain to be written out. Key security is an irrelevant concern once your kernel deadlocks in an OOM state.

Quoted 3 lines

Currently credential caches are stored in files, is there a problem with that model ? Do you need access to credential caches from the kernel when under memory pressure ?

Yes, there is a major problem with that model, and yes we do potentially need access to credential caches when in a recovery situation (which is a situation when we are usually under memory pressure).

Quoted 8 lines · Trond

Quoted 3 lines

It is ok to use keyring if that's deemed the right place for session keys, but I think you already have structures where you currently store them so I don't thik you necessarily need to change that part of the kernel implementation.

No, but we still need to be able to do recovery of rpcsec_gss contexts once they are broken, and right now we have a major flaw due to the fact that recovery depends on a lot of small processes and data that is allowed to be swapped out at the moment when we need them the most (i.e. in a memory reclaim situation).

If the server reboots while our client is in the middle of writing back a file (or several files), then the client needs to recover those rpcsec_gss contexts that authenticate the processes which own any dirty pages that remain to be written out. Key security is an irrelevant concern once your kernel deadlocks in an OOM state.

Ah, this problem. Hopefully the client has enough resources to thrash a lot in the process but still manage to recover. A better solution (see below) is possible, but will require more protocol/mechanism work.

Quoted 4 lines

Quoted 3 lines

Currently credential caches are stored in files, is there a problem with that model ? Do you need access to credential caches from the kernel when under memory pressure ?

Quoted 1 line

Yes, there is a major problem with that model, and yes we do potentially need access to credential caches when in a recovery situation (which is a situation when we are usually under memory pressure).

Ideally we could store in each RPCSEC_GSS context (not GSS context) enough state on the client side to recover quickly when the server reboots.

How would we do this? Suppose the server gives the client a "ticket", and a key much like the Kerberos ticket session key is agreed upon or sent by the server -- that could be stored in the RPCSEC_GSS context and could be used to recover it quickly for recovery from server reboot. I'm eliding a lot of details here, but I believe this is fundamentally workable.

A similar solution would be to store some GSS "sub-credential" in the RPCSEC_GSS context, but this would work for Kerberos and maybe not so well for other mechanisms -- and even with Kerberos, the service ticket might be expired when it comes time to recover. So I prefer the RPCSEC_GSS-level solution I mentioned above.

If you agree with me on this then this sub-thread will be best moved to the NFSv4 WG, particularly if we agree on a protocol-level solution.

Nico
--
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines · Nico Williams

Ideally we could store in each RPCSEC_GSS context (not GSS context) enough state on the client side to recover quickly when the server reboots.

You mean not to use the user Kerberos credential to re-establish the GSS context with the server?

Quoted 6 lines

How would we do this?  Suppose the server gives the client a "ticket", and a key much like the Kerberos ticket session key is agreed upon or sent by the server -- that could be stored in the RPCSEC_GSS context and could be used to recover it quickly for recovery from server reboot.  I'm eliding a lot of details here, but I believe this is fundamentally workable.

So re-establish the RPCSEC_GSS session lost at the server on server reboot by storing enough additional info on the client?

Quoted 6 lines · Andy

Quoted 3 lines · Nico Williams

Ideally we could store in each RPCSEC_GSS context (not GSS context) enough state on the client side to recover quickly when the server reboots.

You mean not to use the user Kerberos credential to re-establish the GSS context with the server?

Kerberos has tickets. Other GSS mechanisms don't. The GSS-API completely abstracts this, so there's no way to extract a service ticket and store it alongside the context (RPCSEC_GSS, in this case) where you might need it in the future. Storing all of a GSS-API credential (think of a whole ccache) in kernel memory is not an option either (ccaches have unbounded size).

Moreover, if we do this in a light-weight enough fashion we might be able to handle all of the recovery path in kernel-mode, with no dependence on upcalls. But if we didn't by somehow extracting the service ticket and storing it in the RPCSEC_GSS context we'd probably still need to upcall to make use of it.

Quoted 8 lines

Quoted 6 lines

How would we do this? Suppose the server gives the client a "ticket", and a key much like the Kerberos ticket session key is agreed upon or sent by the server -- that could be stored in the RPCSEC_GSS context and could be used to recover it quickly for recovery from server reboot. I'm eliding a lot of details here, but I believe this is fundamentally workable.

So re-establish the RPCSEC_GSS session lost at the server on server reboot by storing enough additional info on the client?

Yes. And not just server reboot. The server is free to lose RPCSEC_GSS contexts any time it wants to.

Basically, we need a fast re-authentication facility that is easy to code entirely in kernel-mode. Thinking of it this way I would not reuse any Kerberos tech for this. The server would return a ticket in RPCSEC_GSS context establishment, but the ticket would consist of {secret key index, encrypted octet string} and the server and client would both compute a "session key" (for proving ticket possession) with GSS_Pseudo_random() (this way we can make this work even when the GSS mech only does MICs and not wrap tokens). To re-authenticate the client would send the ticket and an authenticator just like in Kerberos, but simpler.

I agree this would be a very nice feature for fast reconnects in NFSv4, but it looks more and more out of topic.
Time to move this sub-thread to the NFSv4 WG ?

Quoted 5 lines · Andy

Quoted 3 lines · Nico Williams

Ideally we could store in each RPCSEC_GSS context (not GSS context) enough state on the client side to recover quickly when the server reboots.

Quoted 1 line

You mean not to use the user Kerberos credential to re-establish the GSS context with the server?

Kerberos has tickets. Other GSS mechanisms don't. The GSS-API completely abstracts this, so there's no way to extract a service ticket and store it alongside the context (RPCSEC_GSS, in this case) where you might need it in the future. Storing all of a GSS-API credential (think of a whole ccache) in kernel memory is not an option either (ccaches have unbounded size).

Moreover, if we do this in a light-weight enough fashion we might be able to handle all of the recovery path in kernel-mode, with no dependence on upcalls. But if we didn't by somehow extracting the service ticket and storing it in the RPCSEC_GSS context we'd probably still need to upcall to make use of it.

Quoted 8 lines

Quoted 6 lines

How would we do this? Suppose the server gives the client a "ticket", and a key much like the Kerberos ticket session key is agreed upon or sent by the server -- that could be stored in the RPCSEC_GSS context and could be used to recover it quickly for recovery from server reboot. I'm eliding a lot of details here, but I believe this is fundamentally workable.

So re-establish the RPCSEC_GSS session lost at the server on server reboot by storing enough additional info on the client?

Yes. And not just server reboot. The server is free to lose RPCSEC_GSS contexts any time it wants to.

Basically, we need a fast re-authentication facility that is easy to code entirely in kernel-mode. Thinking of it this way I would not reuse any Kerberos tech for this. The server would return a ticket in RPCSEC_GSS context establishment, but the ticket would consist of {secret key index, encrypted octet string} and the server and client would both compute a "session key" (for proving ticket possession) with GSS_Pseudo_random() (this way we can make this work even when the GSS mech only does MICs and not wrap tokens). To re-authenticate the client would send the ticket and an authenticator just like in Kerberos, but simpler.

Nico
--
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Well, don't all GSS mechanisms have credentials? We use the UID to map between the RPCSEC_GSS context and the credential, so we don't need to store the credential along side of the context.

That said, I agree that a light-weight method of re-establishing a context is very appealing.

Quoted 7 lines · Simo Sorce

I have discussed use of the keyring in a private discussion before starting the thread, and it turns out the keyring has a number of limitations that makes it probably unsuitable for this project.

As an upcall mechanism it has some limitations on the size of the payloads, IIRC limited to a page, and that means that you cannot pass blobs carrying big kerberos tickets.

Quoted 5 lines

As a storage mechanism for credential caches it also has size limits. Currently the limit is 20k per user which is not even enough to hold a single ticket in some cases. This limit can be increased of course, but then you end keeping around a huge amount of unswappable ram depending on the number of users.

Allowing keys to be swapped out is a really really really really really really bad idea when your kernel depends upon them being available as part of the memory reclaim process, as would be the case when you are talking about NFS, CIFS or AFS dirty page writebacks.

Quoted 2 lines

So for long term storage of credentials we will probably not rely on kernel keyrings, nor as an upcall mechanism.

So this would be an argument for scrapping keyrings from the kernel? If they're not good for kerberos tickets, why would we want to keep them at all?
What would you replace them with, given the above requirements concerning swapping?

Linux already has per-user, per-process and per-thread keyrings which offer a high security storage solution for keys. The problem with those is that they are difficult to use in an asynchronous context when the original user's process/thread context is no longer available to us.

Ideally, though, that's what we'd like to see used.

FYI, I also think Love needs to be on the cc list, if not heimdal-discuss.