Skip to contentSkip to conversation

New API's for better exporting of VFS from user-mode daemons

PATCH In review Boaz Harrosh 14 participants 43 messages Apr 8, 2013 – Jun 4, 2013 3 messages not in the archive VFS

lore
Share & tools
lore
#0 lore raw

cover New API's for better exporting of VFS from user-mode daemons

Hi

Steven suggested to discuss a readdir-plus system call. For a better LS but certainly better FS daemons. I have hijacked his topic to expand it to a slew of requests we, the FS daemons people, would like to see extended, so to make our lives better (and faster). It should be up to Steven and the comity if these are grouped together or separated into different talks.

I would like to send a set of emails. Each below topic in it's own mail. Any interested party, please post to the topic of your heart so we can collect all the information,, header files, status of each topic. The LSF talk should be not for introduction of the topics but for actual decision making.

Some of the topics already carry, experimental or more, implementations and suggested APIs. Some of these topics are just a cry for help, we know what we don't want, and maybe how we would like them, and what we need is from the community to give us a green light as to if it is wanted at all.

This is certainly an exhaustive list, we might want to drop some of them, if they don't generate much interest. Also if I forgot something please post (JV)

Here is the list of topics in this set:
- [1/8] readdir-plus system call (By Steven Whitehouse <[email protected]>)
- [2/8] Sane locks (UNPOSIX locks) (frank)
- [3/8] File delegations, Usermode API of Bruce's pending patches.
- [4/8] PNFS ioctls/syscall
- [5/8] syscall_cred() a system call that receives alternate creds (fsuid fsgid thread-groups)
- [6/8] Rich ACLs (continued, drive through this time)
- [7/8] Single call interface to getattr/setattr (Frank S Filz <[email protected]>)
- [8/8] Fix fsnotify short comings (single fd with recursive notifications).

Thanks
Boaz

#1 lore raw

PATCH 1/8 readdir-plus system call

By: Steven Whitehouse <[email protected]>)

I repeat below Steve's original mail. Steve you said you have some experimental code, could you post an header and a git URL so we can have a look?

I have seen in the Corner of my eye a readdir-plus syscall in FreeBSD I will try to find it and post the Interface header as reply to here. Might as well make sure they should match somewhat.

Steve wrote ...

Quoted 6 lines

As part of the work we've been doing in relation to integration between NFS/Samba and GFS2, Abhi has been working on a readdirplus system call in order to investigate the issues involved with creating such a call. It is still early days yet, but by April there should be some interesting results to present.

Quoted 8 lines

Please add Abhi to the attendee list as well as myself.

Also, it has been some time since we had a NFS/Samba meeting to discuss the other issues which are pending, such as locking, ACLs, etc. So we could perhaps also allow some time to do that face to face rather than over the phone as we've been doing up til now,

Steve.

Thanks
Boaz

#2 lore raw

Re: [1/8] readdir-plus system call

Again, copying in Abhi,

Steve.

Quoted 29 lines · Boaz Harrosh

By: Steven Whitehouse <[email protected]>)

I repeat below Steve's original mail. Steve you said you have some experimental code, could you post an header and a git URL so we can have a look?

I have seen in the Corner of my eye a readdir-plus syscall in FreeBSD I will try to find it and post the Interface header as reply to here. Might as well make sure they should match somewhat.

Steve wrote ...

Quoted 6 lines

As part of the work we've been doing in relation to integration between NFS/Samba and GFS2, Abhi has been working on a readdirplus system call in order to investigate the issues involved with creating such a call. It is still early days yet, but by April there should be some interesting results to present.

Quoted 8 lines

Please add Abhi to the attendee list as well as myself.

Also, it has been some time since we had a NFS/Samba meeting to discuss the other issues which are pending, such as locking, ACLs, etc. So we could perhaps also allow some time to do that face to face rather than over the phone as we've been doing up til now,

Steve.

Thanks
Boaz

#3 lore raw

Re: [Nfs-ganesha-devel] [1/8] readdir-plus system call

All (CC Trond),

This seems timely. Has this effort looed at issues distributed file systems have with cookies/offsets, and whether a new interface can offer future relief there?

Thanks,

Matt

Quoted 43 lines · ----- "Steven Whitehouse

Again, copying in Abhi,

Steve.

Quoted 7 lines · Boaz Harrosh

By: Steven Whitehouse <[email protected]>)

I repeat below Steve's original mail. Steve you said you have some experimental code, could you post an header and a git URL so we can have a look?

I have seen in the Corner of my eye a readdir-plus syscall in

FreeBSD

Quoted 1 line

I will try to find it and post the Interface header as reply to

here.

Quoted 4 lines

Might as well make sure they should match somewhat.

Steve wrote ...

Quoted 1 line

As part of the work we've been doing in relation to integration

between

Quoted 1 line
Quoted 1 line

NFS/Samba and GFS2, Abhi has been working on a readdirplus system

call

Quoted 1 line
Quoted 1 line

in order to investigate the issues involved with creating such a

call.

Quoted 6 lines
Quoted 6 lines

It is still early days yet, but by April there should be some interesting results to present.

Please add Abhi to the attendee list as well as myself.

Also, it has been some time since we had a NFS/Samba meeting to

discuss

Quoted 1 line
Quoted 1 line

the other issues which are pending, such as locking, ACLs, etc. So

we

Quoted 1 line
Quoted 1 line

could perhaps also allow some time to do that face to face rather

than

Quoted 7 lines
Quoted 3 lines

over the phone as we've been doing up til now,

Steve.

Thanks
Boaz

Signature · 10 lines
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 
#4 lore raw

Re: [1/8] readdir-plus system call

In reply to Boaz Harrosh

Hi,

from a Lustre user point of view, having a way to readdirplus (getting dirents + related attrs) would really be helpful. Beyond the mere scope of Ganesha, it can have bunches of cool cases of use.

Philippe

Quoted 28 lines · Boaz Harrosh

By: Steven Whitehouse <[email protected]>)

I repeat below Steve's original mail. Steve you said you have some experimental code, could you post an header and a git URL so we can have a look?

I have seen in the Corner of my eye a readdir-plus syscall in FreeBSD I will try to find it and post the Interface header as reply to here. Might as well make sure they should match somewhat.

Steve wrote ...

Quoted 6 lines

As part of the work we've been doing in relation to integration between NFS/Samba and GFS2, Abhi has been working on a readdirplus system call in order to investigate the issues involved with creating such a call. It is still early days yet, but by April there should be some interesting results to present.

Quoted 8 lines

Please add Abhi to the attendee list as well as myself.

Also, it has been some time since we had a NFS/Samba meeting to discuss the other issues which are pending, such as locking, ACLs, etc. So we could perhaps also allow some time to do that face to face rather than over the phone as we've been doing up til now,

Steve.

Thanks
Boaz

#5 lore raw

Re: [1/8] readdir-plus system call

In reply to Boaz Harrosh

Hi Boaz/All,

----- Original Message -----

Quoted 9 lines

From: "Boaz Harrosh" <[email protected]>
To: "Steven Whitehouse" <[email protected]>, "Steve Dickson" <[email protected]>, "Jeff Layton" <[email protected]>, [email protected], "linux-fsdevel" <[email protected]>, "Ganesha NFS List" <[email protected]>, "Frank S Filz" <[email protected]>, "J. Bruce Fields" <[email protected]>, "Jim Lieb" <[email protected]>, "Venkateswararao Jujjuri" <[email protected]>, "DENIEL Philippe" <[email protected]>
Sent: Monday, April 8, 2013 5:22:46 AM
Subject: [1/8] readdir-plus system call

Quoted 5 lines

By: Steven Whitehouse <[email protected]>)

I repeat below Steve's original mail. Steve you said you have some experimental code, could you post an header and a git URL so we can have a look?

The patchset I'm working on is in a local tree, but the latest bits are available in this Red Hat Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=850426#c14

Quoted 1 line

From a GFS2 perspective, the need for such a system call arose from our talks with Samba folks to better support clustered samba over GFS2. The system call simply collects dirents along with stat and extended attributes and copies the info out to the user buffer. This patchset is a first-attempt at tackling this problem from a GFS2 perspective and is mainly a way to get us talking about possible implementations.

As the patches stand right now, the VFS bits are just hooks and all the real work is done in the GFS2 filesystem. However, there are some bits that could be moved into the VFS so other filesystems can utilize them.

For obtaining stat info, I'm making use of VFS bits of the xstat and fxstat system calls that David Howells proposed here : https://lists.samba.org/archive/samba-technical/2012-April/082906.html

There are 4 parts to my readdirplus (xgetdents()) patches:

Patch 1of4 adds the xgetdents() syscall interface, xreaddir() f_op and the linux_xdirent structure that specifies how the collected data is packaged to the user. From the caller's perspective, it behaves very much like the getdents() syscall except for the -EAGAIN return code. This would require the caller to re-issue the syscall with the same parameters.

Patch 2of4 is a gfs2 patch that adds a data structure that is a resizeable buffer backed by a vector of pages. This is used to collect all the intermediate data before writing it out to the user buffer.

Patch 3of4 is a simple port of the sort() function from lib/sort.c called ctx_sort(). Only difference is that it takes an additional (void *) opaque context pointer and passes it to the compare() and swap() functions. I needed this to be able to sort pointers stored in the vector of pages buffer.

Patch 4of4 has GFS2's implementation of the xreaddir() f_op and all its supporting functions. gfs2_xreaddir() tries to collect the requested data efficiently by ordering disk block accesses based on the filesystem's on-disk layout and also by adjusting the resizeable buffer as needed.

In my quick testing with a 50,000 file directory, xgetdents() is at least twice as fast as getdents()+stat()+getxattr() with a cold cache and nearly thrice as fast when the disk blocks have been cached.

Cheers!
--Abhi

#6 lore raw

Re: [1/8] readdir-plus system call

Quoted 1 line · Abhijith Das

For obtaining stat info, I'm making use of VFS bits of the xstat and fxstat system calls that David Howells proposed here : https://lists.samba.org/archive/samba-technical/2012-April/082906.html

The fxstat_at() system calls probably deserve discussion by themselves. I think the last time they were discussed we were pretty close to agreement on what they should (and shouldn't) do.

Cheers, Andreas

#7 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

In reply to Abhijith Das

Hi all,

Hoping to revive the discussion for $SUBJECT since we ran out of time when Boaz brought it up at LSF. Summary of what was discussed:

- readdirplus syscall can be modeled after NFS' internal readdirplus implementation.
- Need for a directory version counter (change count)
- Need for each entry to have an opaque resume key - The linux_dirent.d_off in getdents(2) does this somewhat.
- Header at top of the returned data with bits to signify what's inside.
- What data to return? entries + stat + xattrs/acls?

The fs/kernel guys were opposed to tossing xattrs/acls into the mix - I tend to agree, after having worked on a draft readdirplus syscall on GFS2 that does xattrs in addition to stat.

The potentially large amount of variable length data to handle and the alloc/realloc/dealloc of said data makes the code quite complicated and hence, difficult to maintain. I had to write a new page-backed resizeable buffer to make this worthwhile (performance was actually worse with kmalloc & friends and kmap/kunmap compared to simply doing getdents()+stat()+getxattr()).

For those who are interested, here are the patches (description in previous email below): https://bugzilla.redhat.com/show_bug.cgi?id=850426#c14
There's an interesting seekwatcher graph on there too that compares the two cases. With a cold cache, almost all the speedup obtained by readdirplus is by being able to order all the disk reads. I've seen a 2x speedup (cold cache) with my test directories, but not much more. When the relevant disk blocks are in cache, readdirplus is about 3x faster - I attribute it to the minimal allocing and user/kernel mode switching that goes on.

We might also get decent performance by simply having a system call that takes the directory as argument and goes off and pre-fetches all the relevant blocks required to do subsequent getdents()+stat()+getxattr() efficiently.

Thoughts?

Cheers!
--Abhi

----- Original Message -----

Quoted 85 lines

From: "Abhijith Das" <[email protected]>
To: "Boaz Harrosh" <[email protected]>
Cc: "Steven Whitehouse" <[email protected]>, "Steve Dickson" <[email protected]>, "Jeff Layton" <[email protected]>, [email protected], "linux-fsdevel" <[email protected]>, "Ganesha NFS List" <[email protected]>, "Frank S Filz" <[email protected]>, "J. Bruce Fields" <[email protected]>, "Jim Lieb" <[email protected]>, "Venkateswararao Jujjuri" <[email protected]>, "DENIEL Philippe" <[email protected]>
Sent: Monday, April 8, 2013 2:02:40 PM
Subject: Re: [1/8] readdir-plus system call

Hi Boaz/All,

----- Original Message -----

Quoted 13 lines

From: "Boaz Harrosh" <[email protected]>
To: "Steven Whitehouse" <[email protected]>, "Steve Dickson" <[email protected]>, "Jeff Layton"
<[email protected]>, [email protected], "linux-fsdevel" <[email protected]>, "Ganesha
NFS List" <[email protected]>, "Frank S Filz" <[email protected]>, "J. Bruce Fields"
<[email protected]>, "Jim Lieb" <[email protected]>, "Venkateswararao Jujjuri" <[email protected]>, "DENIEL
Philippe" <[email protected]>
Sent: Monday, April 8, 2013 5:22:46 AM
Subject: [1/8] readdir-plus system call

Quoted 5 lines

By: Steven Whitehouse <[email protected]>)

I repeat below Steve's original mail. Steve you said you have some experimental code, could you post an header and a git URL so we can have a look?

The patchset I'm working on is in a local tree, but the latest bits are available in this Red Hat Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=850426#c14

From a GFS2 perspective, the need for such a system call arose from our talks with Samba folks to better support clustered samba over GFS2. The system call simply collects dirents along with stat and extended attributes and copies the info out to the user buffer. This patchset is a first-attempt at tackling this problem from a GFS2 perspective and is mainly a way to get us talking about possible implementations.

As the patches stand right now, the VFS bits are just hooks and all the real work is done in the GFS2 filesystem. However, there are some bits that could be moved into the VFS so other filesystems can utilize them.

For obtaining stat info, I'm making use of VFS bits of the xstat and fxstat system calls that David Howells proposed here :
https://lists.samba.org/archive/samba-technical/2012-April/082906.html

There are 4 parts to my readdirplus (xgetdents()) patches:

Patch 1of4 adds the xgetdents() syscall interface, xreaddir() f_op and the linux_xdirent structure that specifies how the collected data is packaged to the user. From the caller's perspective, it behaves very much like the getdents() syscall except for the -EAGAIN return code. This would require the caller to re-issue the syscall with the same parameters.

Patch 2of4 is a gfs2 patch that adds a data structure that is a resizeable buffer backed by a vector of pages. This is used to collect all the intermediate data before writing it out to the user buffer.

Patch 3of4 is a simple port of the sort() function from lib/sort.c called ctx_sort(). Only difference is that it takes an additional (void *) opaque context pointer and passes it to the compare() and swap() functions. I needed this to be able to sort pointers stored in the vector of pages buffer.

Patch 4of4 has GFS2's implementation of the xreaddir() f_op and all its supporting functions. gfs2_xreaddir() tries to collect the requested data efficiently by ordering disk block accesses based on the filesystem's on-disk layout and also by adjusting the resizeable buffer as needed.

In my quick testing with a 50,000 file directory, xgetdents() is at least twice as fast as getdents()+stat()+getxattr() with a cold cache and nearly thrice as fast when the disk blocks have been cached.

Cheers!
--Abhi
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

Quoted 7 lines · Abhijith Das

Hoping to revive the discussion for $SUBJECT since we ran out of time when Boaz brought it up at LSF.
[ ... ]
For those who are interested, here are the patches (description in previous email below):
https://bugzilla.redhat.com/show_bug.cgi?id=850426#c14

Some quick things that struck me as I glanced through the patches:

- Please post the patch series, don't make us go digging through bugzilla.

- Don't use variable size types in the ABI or you'll have to add compat_ wrappers to fix it all up on the stack when going between 32bit userspace and 64bit kernelspace. This is going to be especially nasty if this is a giant sequence of variable length blobs.

  +struct linux_xdirent {
  +	unsigned long        xd_ino;
  +	char                 xd_type;
  +	unsigned long        xd_off;
  +	struct xstat         xd_stat;
  +	unsigned long        xd_reclen;
  +	struct xdirent_blob  xd_blob;
  +};

Notice how, in contrast, David was careful to use naturally aligned fixed-width types in his xstat patch.

- z

#9 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

Zack, thanks for taking a peek at the patches.

Quoted 5 lines

Some quick things that struck me as I glanced through the patches:

- Please post the patch series, don't make us go digging through bugzilla.

Duly noted. I wasn't hoping for my patches to be a serious submission, just something I wrote up as a POC. I was mainly looking to (re)start a conversation about readdirplus to see what's the best way to go about doing this. Your point applies nonetheless; I'll post my patchset again, properly.

Quoted 15 lines

- Don't use variable size types in the ABI or you'll have to add compat_ wrappers to fix it all up on the stack when going between 32bit userspace and 64bit kernelspace. This is going to be especially nasty if this is a giant sequence of variable length blobs.

  +struct linux_xdirent {
  +	unsigned long        xd_ino;
  +	char                 xd_type;
  +	unsigned long        xd_off;
  +	struct xstat         xd_stat;
  +	unsigned long        xd_reclen;
  +	struct xdirent_blob  xd_blob;
  +};
Quoted 3 lines

Notice how, in contrast, David was careful to use naturally aligned fixed-width types in his xstat patch.

Yes, you're right. I'll fix this.

Quoted 1 line

- z

Cheers!
--Abhi

#10 lore raw

Re: Re: [1/8] readdir-plus system call - LSF/MM follow up

Quoted 6 lines · On Tuesday, May 28, 2013 10:49:31 Abhijith Das wrote:

Zack, thanks for taking a peek at the patches.

Quoted 4 lines

Some quick things that struck me as I glanced through the patches:

- Please post the patch series, don't make us go digging through

Quoted 7 lines
Quoted 1 line

bugzilla.

Duly noted. I wasn't hoping for my patches to be a serious submission, just something I wrote up as a POC. I was mainly looking to (re)start a conversation about readdirplus to see what's the best way to go about doing this. Your point applies nonetheless; I'll post my patchset again, properly.

Speaking for the nfs-ganesha project, one of the issues that we couldn't sort out at lsf was whether to to include xattrs. As far as ganesha is concerned, the xstat struct is sufficient. At readdir time, we pretty much just want to build our cache entries and get basic stat info. The only case where we'd really need xattrs would be for acls but that is usually later in the protocol op sequence. I'd just like to get the path from readdir+ thru the callbacks to the xdr into the reply as simple as possible. That and get enough in the cache entry to be ready for the next step.

Quoted 24 lines
Quoted 15 lines

- Don't use variable size types in the ABI or you'll have to add compat_

wrappers to fix it all up on the stack when going between 32bit userspace and 64bit kernelspace. This is going to be especially nasty if this is a giant sequence of variable length blobs.

  +struct linux_xdirent {
  +	unsigned long        xd_ino;
  +	char                 xd_type;
  +	unsigned long        xd_off;
  +	struct xstat         xd_stat;
  +	unsigned long        xd_reclen;
  +	struct xdirent_blob  xd_blob;
  +};
Quoted 2 lines

Notice how, in contrast, David was careful to use naturally aligned fixed-width types in his xstat patch.

Yes, you're right. I'll fix this.

Quoted 1 line

- z

Cheers!
--Abhi

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.

A message that is not in the archive (sent off-list or not yet received): OF27E1911F.3FBABA22-ON87257B79.005C087F-88257B79.005C320B@us.ibm.com

#12 lore raw

Re: Re: Re: [1/8] readdir-plus system call - LSF/MM follow up

Quoted 46 lines · On Tuesday, May 28, 2013 09:47:01 Frank S Filz wrote:

Jim Lieb <[email protected]> wrote on 05/28/2013 08:13:02 AM:

Quoted 10 lines · On Tuesday, May 28, 2013 10:49:31 Abhijith Das wrote:
Quoted 9 lines

Zack, thanks for taking a peek at the patches.

Quoted 5 lines

Some quick things that struck me as I glanced through the patches:

- Please post the patch series, don't make us go digging through

bugzilla.

Duly noted. I wasn't hoping for my patches to be a serious submission,

just

Quoted 2 lines
Quoted 2 lines

something I wrote up as a POC. I was mainly looking to (re)start a conversation about readdirplus to see what's the best way to go about

doing

Quoted 4 lines
Quoted 2 lines

this. Your point applies nonetheless; I'll post my patchset again, properly.

Speaking for the nfs-ganesha project, one of the issues that we couldn't

sort

Quoted 1 line

out at lsf was whether to to include xattrs. As far as ganesha is

concerned,

Quoted 1 line

the xstat struct is sufficient. At readdir time, we pretty much just

want to

Quoted 1 line

build our cache entries and get basic stat info. The only case where

we'd

Quoted 3 lines

really need xattrs would be for acls but that is usually later in the protocol
op sequence. I'd just like to get the path from readdir+ thru the

callbacks

Quoted 1 line

to the xdr into the reply as simple as possible. That and get enough in

Quoted 8 lines

the

Quoted 1 line

cache entry to be ready for the next step.

Actually, ACLs are critical for Ganesha. Unless we decide to have separate attr validity bits for "stat" attributes and ACLs, Ganesha will have a difficult time knowing if the ACL attribute is up to date (or even available).

True enough. But one of the pushbacks was the amount of work needed to get to xattrs where acls live. One thing I heard that made not having acls on the readdir+ pass was a status of some kind that indicated "I have acls..." The readdir is a dir op and so 10k+ entries need to be minimal overhead. we already have the acls of the dir from the lookup. we don't need an entry's acls until we do the lookup on it. at that time we can grab the acls. That was the argument as I remember and I'm willing to accept it. IIRC, the client is going to send us a getattrs later. we can do it then. Is this reasonable?

Quoted 35 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

- Don't use variable size types in the ABI or you'll have to add

compat_

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

wrappers to fix it all up on the stack when going between 32bit userspace and 64bit kernelspace. This is going to be especially

nasty

Quoted 25 lines
Quoted 14 lines
Quoted 9 lines

if this is a giant sequence of variable length blobs.

+struct linux_xdirent {

  +   unsigned long        xd_ino;
  +   char                 xd_type;
  +   unsigned long        xd_off;
  +   struct xstat         xd_stat;
  +   unsigned long        xd_reclen;
  +   struct xdirent_blob  xd_blob;
Quoted 4 lines

+};

Notice how, in contrast, David was careful to use naturally aligned fixed-width types in his xstat patch.

Quoted 6 lines

Yes, you're right. I'll fix this.

Quoted 1 line

- z

Cheers!
--Abhi

--
Jim Lieb
Linux Systems Engineer
Panasas Inc.

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.

A message that is not in the archive (sent off-list or not yet received): OF067A3B49.F63109B6-ON87257B7A.00137A60-88257B7A.00140BC7@us.ibm.com

#14 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

Quoted 35 lines · On Tue, 28 May 2013 20:38:57 -0700 Frank S Filz <[email protected]> wrote:

Jim Lieb <[email protected]> wrote on 05/28/2013 05:57:31 PM:

Quoted 1 line
Quoted 1 line

Actually, ACLs are critical for Ganesha. Unless we decide to have

separate

Quoted 5 lines
Quoted 3 lines

attr validity bits for "stat" attributes and ACLs, Ganesha will have a difficult time knowing if the ACL attribute is up to date (or even available).

True enough. But one of the pushbacks was the amount of work neededto

get to

Quoted 1 line

xattrs where acls live. One thing I heard that made not having acls on

the

Quoted 1 line

readdir+ pass was a status of some kind that indicated "I have acls..."

The

Quoted 2 lines

readdir is a dir op and so 10k+ entries need to be minimal overhead. we already have the acls of the dir from the lookup. we don't need an

entry's

Quoted 1 line

acls until we do the lookup on it. at that time we can grab the acls.

That

Quoted 3 lines

was the argument as I remember and I'm willing to accept it. IIRC, the client
is going to send us a getattrs later. we can do it then. Is this

reasonable?

The ACL COULD be required on READDIR, though I would not expect any clients to ask for ACL on READDIR (though it sure would be handy if Ganesha's PROXY client could do so...).

Fortunately we don't enforce ACE4_READ_ATTR, otherwise we WOULD need ACL on any READDIR...

If there are times when we get attrs without getting ACL, then we will need a separate validity bit for ACL, otherwise we won't be able to tell if we have current ACL for an entry or not.

Quoted 7 lines

What would actually be helpful though, and make Ganesha a lot more efficient is if we could actually get all the ACLs for a directory in one fell swoop with some sort of "compression". Given that a large percentage of files actually have the same ACL, we could get a the 1-4 ACLs that apply, and then a bunch of entries, each indicating which of the 4 ACLs they have.

Most NFS clients aren't going to need ACLs during a READDIR operation. I'll go as so far to say that most NFS clients don't care *at all* about ACLs. Those are things that are enforced by the server and the client doesn't really care to know about them.

The exception is when a client gets an explicit request to either view or change the ACL. For Linux clients (and most other POSIX-y ones), that's never done in any sort of batch form. It's always an operation done against a single dentry.

So, I'm not sure I understand the argument for adding ACLs here. It's not likely to be something you're going to end up stuffing into a READDIR reply.

Signature · 1 line
Jeff Layton <[email protected]>
#15 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

Quoted 51 lines · Jeff Layton
Quoted 35 lines · On Tue, 28 May 2013 20:38:57 -0700 Frank S Filz <[email protected]> wrote:

Jim Lieb <[email protected]> wrote on 05/28/2013 05:57:31 PM:

Quoted 1 line
Quoted 1 line

Actually, ACLs are critical for Ganesha. Unless we decide to have

separate

Quoted 5 lines
Quoted 3 lines

attr validity bits for "stat" attributes and ACLs, Ganesha will have a difficult time knowing if the ACL attribute is up to date (or even available).

True enough. But one of the pushbacks was the amount of work neededto

get to

Quoted 1 line

xattrs where acls live. One thing I heard that made not having acls on

the

Quoted 1 line

readdir+ pass was a status of some kind that indicated "I have acls..."

The

Quoted 2 lines

readdir is a dir op and so 10k+ entries need to be minimal overhead. we already have the acls of the dir from the lookup. we don't need an

entry's

Quoted 1 line

acls until we do the lookup on it. at that time we can grab the acls.

That

Quoted 3 lines

was the argument as I remember and I'm willing to accept it. IIRC, the client
is going to send us a getattrs later. we can do it then. Is this

reasonable?

The ACL COULD be required on READDIR, though I would not expect any clients to ask for ACL on READDIR (though it sure would be handy if Ganesha's PROXY client could do so...).

Fortunately we don't enforce ACE4_READ_ATTR, otherwise we WOULD need ACL on any READDIR...

If there are times when we get attrs without getting ACL, then we will need a separate validity bit for ACL, otherwise we won't be able to tell if we have current ACL for an entry or not.

Quoted 7 lines

What would actually be helpful though, and make Ganesha a lot more efficient is if we could actually get all the ACLs for a directory in one fell swoop with some sort of "compression". Given that a large percentage of files actually have the same ACL, we could get a the 1-4 ACLs that apply, and then a bunch of entries, each indicating which of the 4 ACLs they have.

Most NFS clients aren't going to need ACLs during a READDIR operation. I'll go as so far to say that most NFS clients don't care *at all* about ACLs. Those are things that are enforced by the server and the client doesn't really care to know about them.

Quoted 4 lines

The exception is when a client gets an explicit request to either view or change the ACL. For Linux clients (and most other POSIX-y ones), that's never done in any sort of batch form. It's always an operation done against a single dentry.

An odd exception: in the presence of "posix" acls, "ls -l" requests an acl for every entry, so it can decide whether or not to add a "+" after the mode (which indicates the presence of a non-trivial acl.) Judging from http://www.bestbits.at/richacl/example.html, the same is intended (but not yet implemented) for richacls.

Maybe if that case were common, there'd be some advantage to ls being able to do a readdir plus to the nfs client that the nfs client could translate into a single readdir to the server?

But I hope it doesn't come to that.

--b.

Quoted 6 lines

So, I'm not sure I understand the argument for adding ACLs here. It's not likely to be something you're going to end up stuffing into a READDIR reply.

Signature · 1 line
Jeff Layton <[email protected]>
#16 lore raw

Re: [Lsf-pc] [1/8] readdir-plus system call - LSF/MM follow up

Quoted 11 lines · J. Bruce Fields

An odd exception: in the presence of "posix" acls, "ls -l" requests an acl for every entry, so it can decide whether or not to add a "+" after the mode (which indicates the presence of a non-trivial acl.) Judging from http://www.bestbits.at/richacl/example.html, the same is intended (but not yet implemented) for richacls.

Maybe if that case were common, there'd be some advantage to ls being able to do a readdir plus to the nfs client that the nfs client could translate into a single readdir to the server?

But I hope it doesn't come to that.

Having a xstat flag that a filesystem can set meaning there is no additional security information would be way more efficient for the common case.

#17 lore raw

Re: [Lsf-pc] [1/8] readdir-plus system call - LSF/MM follow up

Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · J. Bruce Fields

An odd exception: in the presence of "posix" acls, "ls -l" requests an acl for every entry, so it can decide whether or not to add a "+" after the mode (which indicates the presence of a non-trivial acl.) Judging from http://www.bestbits.at/richacl/example.html, the same is intended (but not yet implemented) for richacls.

Maybe if that case were common, there'd be some advantage to ls being able to do a readdir plus to the nfs client that the nfs client could translate into a single readdir to the server?

Quoted 5 lines
Quoted 1 line

But I hope it doesn't come to that.

Having a xstat flag that a filesystem can set meaning there is no additional security information would be way more efficient for the common case.

I think that might make sense.

(Though I can't claim any evidence of an actual problem here. Just that if people are counting the stat's on "ls -l" then may find out they run into those extra getxattrs too.)

--b.

#18 lore raw

Re: Re: Re: Re: [1/8] readdir-plus system call - LSF/MM follow up

In reply to a message not in the archive
Quoted 34 lines · On Tuesday, May 28, 2013 20:38:57 Frank S Filz wrote:

Jim Lieb <[email protected]> wrote on 05/28/2013 05:57:31 PM:

Quoted 1 line
Quoted 1 line

Actually, ACLs are critical for Ganesha. Unless we decide to have

separate

Quoted 5 lines
Quoted 3 lines

attr validity bits for "stat" attributes and ACLs, Ganesha will have a difficult time knowing if the ACL attribute is up to date (or even available).

True enough. But one of the pushbacks was the amount of work neededto

get to

Quoted 1 line

xattrs where acls live. One thing I heard that made not having acls on

the

Quoted 1 line

readdir+ pass was a status of some kind that indicated "I have acls..."

The

Quoted 2 lines

readdir is a dir op and so 10k+ entries need to be minimal overhead. we already have the acls of the dir from the lookup. we don't need an

entry's

Quoted 1 line

acls until we do the lookup on it. at that time we can grab the acls.

That

Quoted 3 lines

was the argument as I remember and I'm willing to accept it. IIRC, the client
is going to send us a getattrs later. we can do it then. Is this

Quoted 8 lines

reasonable?

The ACL COULD be required on READDIR, though I would not expect any clients to ask for ACL on READDIR (though it sure would be handy if Ganesha's PROXY client could do so...).

Fortunately we don't enforce ACE4_READ_ATTR, otherwise we WOULD need ACL on any READDIR...

That's an acl on the dir itself. If we carry this further, a recursive ls would readdir+ the top level dir, then lookup each dir in its list in order to get a handle for the next level of readdir+. At the time of the lookup is when we need the acl for that dir. If anywhere in the tree we have 5 dirs and 10k "other" files, regulars, symlinks etc., we only really need the acl above for the 5 dirs. We take the getattr (+get acls) hit 5 times at lookup time for that dir. The 10k other stuff is happy with name, ls -l stuff.

Quoted 13 lines

If there are times when we get attrs without getting ACL, then we will need a separate validity bit for ACL, otherwise we won't be able to tell if we have current ACL for an entry or not.

What would actually be helpful though, and make Ganesha a lot more efficient is if we could actually get all the ACLs for a directory in one fell swoop with some sort of "compression". Given that a large percentage of files actually have the same ACL, we could get a the 1-4 ACLs that apply, and then a bunch of entries, each indicating which of the 4 ACLs they have.

Frank

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.
#19 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

In reply to Abhijith Das
Quoted 1 line · Abhijith Das

Zack, thanks for taking a peek at the patches.

It wasn't clear to me which version of the xstat patches you were basing your work on? Are these the latest ones from David, or ones from a mailing list thread? I've CC'd David, since he might have newer (and hopefully minimally contentious?) versions of the patches.

Quoted 6 lines
Quoted 4 lines

Some quick things that struck me as I glanced through the patches:

- Please post the patch series, don't make us go digging through bugzilla.

Duly noted. I wasn't hoping for my patches to be a serious submission, just something I wrote up as a POC. I was mainly looking to (re)start a conversation about readdirplus to see what's the best way to go about doing this. Your point applies nonetheless; I'll post my patchset again, properly.

It definitely makes sense to start with just the xstat patch, since if we can't get that agreed upon and landed, there is no hope for readdirplus to get consensus.

Could you or David submit the latest xstat patch to the list?

Cheers, Andreas

Quoted 20 lines
Quoted 12 lines

- Don't use variable size types in the ABI or you'll have to add compat_ wrappers to fix it all up on the stack when going between 32bit userspace and 64bit kernelspace. This is going to be especially nasty if this is a giant sequence of variable length blobs.

+struct linux_xdirent {

 +	unsigned long        xd_ino;
 +	char                 xd_type;
 +	unsigned long        xd_off;
 +	struct xstat         xd_stat;
 +	unsigned long        xd_reclen;
 +	struct xdirent_blob  xd_blob;
Quoted 5 lines

+};

Notice how, in contrast, David was careful to use naturally aligned fixed-width types in his xstat patch.

Yes, you're right. I'll fix this.

Quoted 8 lines
Quoted 1 line

- z

Cheers!
--Abhi
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Cheers, Andreas

#20 lore raw

Re: [1/8] readdir-plus system call - LSF/MM follow up

----- Original Message -----

Quoted 11 lines

From: "Andreas Dilger" <[email protected]>
To: "Abhijith Das" <[email protected]>
Cc: "Zach Brown" <[email protected]>, "Boaz Harrosh" <[email protected]>, "Steven Whitehouse" <[email protected]>, "Steve Dickson" <[email protected]>, "David Howells" <[email protected]>, "Jeff Layton" <[email protected]>, [email protected], "linux-fsdevel" <[email protected]>, "Ganesha NFS List" <[email protected]>, "Frank S Filz" <[email protected]>, "J. Bruce Fields" <[email protected]>, "Jim Lieb" <[email protected]>, "Venkateswararao Jujjuri" <[email protected]>, "DENIEL Philippe" <[email protected]>, "Dave Chinner" <[email protected]>
Sent: Tuesday, May 28, 2013 3:00:39 PM
Subject: Re: [1/8] readdir-plus system call - LSF/MM follow up

Quoted 7 lines
Quoted 1 line · Abhijith Das

Zack, thanks for taking a peek at the patches.

It wasn't clear to me which version of the xstat patches you were basing your work on? Are these the latest ones from David, or ones from a mailing list thread? I've CC'd David, since he might have newer (and hopefully minimally contentious?) versions of the patches.

I've based my patches on an older mailing-list version of David's xstat patches. I'll defer to David to post/point to a newer version.

Cheers!
--Abhi

#21 lore raw

PATCH 2/8 Sane locks (UNPOSIX locks)

In reply to Boaz Harrosh

In this topic we do not actually know what it will look like but we know what we do not like.

The most troublesome is the POSIX crap. POSIX says that if any fd of a process on an inode is closed, all locks are lost, even if we used another fd with other modes to acquire those locks. (They had good stuff to smoke at when that stuff was defined)

This is real crap, because it completely kills our ability to acquire some other resources on the file, and/or keep correct access modes. Because as soon as we need a lock we need to open the fd in read/write mode, because if in future a clients need a write access, we cannot do a re-open of the file, we will loose the locks. Now if we open in RW, then we will immediately loose our delegations and also in PNFS a write-open means a different thing then read-open.

So what we urgently need is new locks API that is strictly per fd. When we open an fd for read and then acquire a read lock, we can continue to serve delegations. Only the close of that specific fd will loose the locks. Any other parallel activity in the background will not affect anything.

We can craft an API that is very similar to today's API only with the semantic changes. But we should also consider a completely new API that can cover all the kind of locks, including a notification API. Perhaps also unite that API with the delegations API we want in the next topic.

Thanks
Boaz

#22 lore raw

Re: [Lsf-pc] [2/8] Sane locks (UNPOSIX locks)

Quoted 10 lines · On Mon, 8 Apr 2013 13:25:43 +0300 Boaz Harrosh <[email protected]> wrote:

In this topic we do not actually know what it will look like but we know what we do not like.

The most troublesome is the POSIX crap. POSIX says that if any fd of a process on an inode is closed, all locks are lost, even if we used another fd with other modes to acquire those locks. (They had good stuff to smoke at when that stuff was defined)

Jeremy Allison did some detective work on why this is:

http://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html

See the section on "First Implementation Past the Post".

Quoted 15 lines

This is real crap, because it completely kills our ability to acquire some other resources on the file, and/or keep correct access modes. Because as soon as we need a lock we need to open the fd in read/write mode, because if in future a clients need a write access, we cannot do a re-open of the file, we will loose the locks. Now if we open in RW, then we will immediately loose our delegations and also in PNFS a write-open means a different thing then read-open.

So what we urgently need is new locks API that is strictly per fd. When we open an fd for read and then acquire a read lock, we can continue to serve delegations. Only the close of that specific fd will loose the locks. Any other parallel activity in the background will not affect anything.

Quoted 9 lines

We can craft an API that is very similar to today's API only with the semantic changes. But we should also consider a completely new API that can cover all the kind of locks, including a notification API. Perhaps also unite that API with the delegations API we want in the next topic.

Thanks
Boaz

Perhaps we can simply add a new clone() flag (CLONE_SANELOCKS?) that indicates that the tasks in question want locks that are not affected by close() from other tasks?

Signature · 1 line
Jeff Layton <[email protected]>
#23 lore raw
In reply to Boaz Harrosh

Hi,

Quoted 18 lines · Boaz Harrosh

Hi

Steven suggested to discuss a readdir-plus system call. For a better LS but certainly better FS daemons. I have hijacked his topic to expand it to a slew of requests we, the FS daemons people, would like to see extended, so to make our lives better (and faster). It should be up to Steven and the comity if these are grouped together or separated into different talks.

I would like to send a set of emails. Each below topic in it's own mail. Any interested party, please post to the topic of your heart so we can collect all the information,, header files, status of each topic. The LSF talk should be not for introduction of the topics but for actual decision making.

Some of the topics already carry, experimental or more, implementations and suggested APIs. Some of these topics are just a cry for help, we know what we don't want, and maybe how we would like them, and what we need is from the community to give us a green light as to if it is wanted at all.

Quoted 5 lines

This is certainly an exhaustive list, we might want to drop some of them, if they don't generate much interest. Also if I forgot something please post (JV)

Here is the list of topics in this set:
- [1/8] readdir-plus system call (By Steven Whitehouse <[email protected]>)

This is really Abhi's topic - I've cc'd him, but certainly something I'm interesting in seeing a solution to as well.

Quoted 10 lines

- [2/8] Sane locks (UNPOSIX locks) (frank)
- [3/8] File delegations, Usermode API of Bruce's pending patches.
- [4/8] PNFS ioctls/syscall
- [5/8] syscall_cred() a system call that receives alternate creds (fsuid fsgid thread-groups)
- [6/8] Rich ACLs (continued, drive through this time)
- [7/8] Single call interface to getattr/setattr (Frank S Filz <[email protected]>)
- [8/8] Fix fsnotify short comings (single fd with recursive notifications).

Thanks
Boaz

Looks like a useful set of discussion topics,

Steve.

#24 lore raw

PATCH 3/8 File delegations, Usermode API of Bruce's pending patches.

In reply to Boaz Harrosh

Here too we have not considered how this will look like.

Bruce has a pending patchset that will enable delegations in the Kernel NFSD server. He has introduced a new delegation object at the lock-manager, and so on.

We at user mode would like to acquire such delegations and be notified of their revocation, synchronously so we can recall them form clients before new access is granted.

Again perhaps this API can be united with the new locks API we want.

fctl or syscall ?

Thanks
Boaz

#25 lore raw

PATCH 4/8 PNFS ioctls/syscall

In reply to Boaz Harrosh

In this topic I would like to present the API I use for exporting PNFS layouts, and recall layouts from a Kernel FileSystem.

I use this API from the Ganesha server, to enable pnfs export both from the in Kernel exofs FS, as well as from the proprietary PanFS filesystem driver of the Panasas's cluster. Same exact API works well.

The API is currently an IOCTL based, I understand it might want to be delegated to a syscall. With common code.

The API is based on all the same types and structures as the In Kernel (out-of-tree) PNFS project and API's. 80% of the vectors implemented for the in-kernel server are identical to the in-kernel implementation. The remaining 20% can have a common core code at the FS, and two wrappers one for each server. Mainly the big difference is in how the RECALLs are made.

The two filesystems I coded also use a common library I call libpnfs_logic which can or not be used by FSs to implement their internal logic. This is because all this logic is currently inside the KPNFSD server, but it cannot be when the Server is in user mode. The In Kernel Server can also be converted to use libpnfs_logic but I have not yet attempted such a conversion and the code is currently duplicated.

I will post later, the header I use for the IOCTL and a git URL for the hard cores.

But we need not talk details, we should only discus in principle if it is wanted, and how it should fit with the In-Kernel PNFS project.

Thanks
Boaz

#26 lore raw

PATCH 5/8 syscall_cred() a system call that receives alternate CREDs

In reply to Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.
(Note that permission checking is done by Ganesha core, because
We may cache open fd(s) and such not, another topic)

We could maybe with hard work save the last two calls for reverting to root, but this will force us to audit lots of code that we are not prepared to do right now. And will not save us much.

[thread_setgroups()]
thread_setgroups() is what we use at Ganesha and what Samaba guys use for a per-thread setgroups() call. In the Linux Kernel the setgroups is actually always per thread. It is only the POSIX (crap) pthread layer at glibc that intercepts the setgroups() call (and others), Iterates on all threads that belong to a process, and calls the native Kernel setgroups on them. So thread_setgroups() is just the raw syscall bypassing glibc's processing. We will eventually push this API to glibc.
BTW: this is done exactly the same on FreeBSD, with same exact glibc intervention.

[Proposed]
What Jim proposed is a syscall that receives a struct that has the regular syscalls parameters plus the creds structure with fsuid/fsgid and groups array. Kernel will set these in, call the original syscall, and revert. This will be done on only an interested subset of the syscalls that are one - are related to filesystems (setfsXid) and two - are of interest to us Servers.

Jim care to scribble a structure definition?

Thanks
Boaz

#27 lore raw

Re: [5/8] syscall_cred() a system call that receives alternate CREDs

I do agree with Boaz and Jim. Being capable of "mascarading" a syscall with someone's credential would be really useful. In particular, in the Ganesha's scope, it is required to properly managed quota (you need to create inodes and write to files as the user if you want those inodes and blocks to be added to the right user's bill).

Philippe

Quoted 41 lines · Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.
(Note that permission checking is done by Ganesha core, because
We may cache open fd(s) and such not, another topic)

We could maybe with hard work save the last two calls for reverting to root, but this will force us to audit lots of code that we are not prepared to do right now. And will not save us much.

[thread_setgroups()]
thread_setgroups() is what we use at Ganesha and what Samaba guys use for a per-thread setgroups() call. In the Linux Kernel the setgroups is actually always per thread. It is only the POSIX (crap) pthread layer at glibc that intercepts the setgroups() call (and others), Iterates on all threads that belong to a process, and calls the native Kernel setgroups on them. So thread_setgroups() is just the raw syscall bypassing glibc's processing. We will eventually push this API to glibc.
BTW: this is done exactly the same on FreeBSD, with same exact glibc intervention.

[Proposed]
What Jim proposed is a syscall that receives a struct that has the regular syscalls parameters plus the creds structure with fsuid/fsgid and groups array. Kernel will set these in, call the original syscall, and revert. This will be done on only an interested subset of the syscalls that are one - are related to filesystems (setfsXid) and two - are of interest to us Servers.

Jim care to scribble a structure definition?

Thanks
Boaz

#28 lore raw

Re: [5/8] syscall_cred() a system call that receives alternate CREDs

In reply to Boaz Harrosh
Quoted 5 lines · Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

Quoted 7 lines

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.

To make sure I understand, you're saying that:

- the behavior you get out of those 6 syscalls is correct,
- you just want to be able to do exactly the same thing, but with 1 syscall. (For performance?)

Or is there some other issue?

Quoted 2 lines

(Note that permission checking is done by Ganesha core, because
We may cache open fd(s) and such not, another topic)

Is there anything we could do to make it possible for you to depend on the kernel's permissions checking instead?

--b.

Quoted 27 lines

We could maybe with hard work save the last two calls for reverting to root, but this will force us to audit lots of code that we are not prepared to do right now. And will not save us much.

[thread_setgroups()]
thread_setgroups() is what we use at Ganesha and what Samaba guys use for a per-thread setgroups() call. In the Linux Kernel the setgroups is actually always per thread. It is only the POSIX (crap) pthread layer at glibc that intercepts the setgroups() call (and others), Iterates on all threads that belong to a process, and calls the native Kernel setgroups on them. So thread_setgroups() is just the raw syscall bypassing glibc's processing. We will eventually push this API to glibc.
BTW: this is done exactly the same on FreeBSD, with same exact glibc intervention.

[Proposed]
What Jim proposed is a syscall that receives a struct that has the regular syscalls parameters plus the creds structure with fsuid/fsgid and groups array. Kernel will set these in, call the original syscall, and revert. This will be done on only an interested subset of the syscalls that are one - are related to filesystems (setfsXid) and two - are of interest to us Servers.

Jim care to scribble a structure definition?

Thanks
Boaz

#29 lore raw

Re: [5/8] syscall_cred() a system call that receives alternate CREDs

Quoted 14 lines · J. Bruce Fields
Quoted 5 lines · Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

Quoted 7 lines

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.

Quoted 6 lines

To make sure I understand, you're saying that:

- the behavior you get out of those 6 syscalls is correct,
- you just want to be able to do exactly the same thing, but with 1 syscall. (For performance?)

Yes, performance.

Quoted 8 lines

Or is there some other issue?

Quoted 2 lines

(Note that permission checking is done by Ganesha core, because
We may cache open fd(s) and such not, another topic)

Is there anything we could do to make it possible for you to depend on the kernel's permissions checking instead?

That one is a different topic. I thought like you that we should let the FS have the final disposition. But the guys convinced me that it is not possible. Both because of caching as well as because there are places that NFS3/4/4.1 will allow or deny differently then POSIX.

Some of the other guys on the list have more details then me. Frank?

Quoted 2 lines

--b.

Thanks
Boaz

Quoted 27 lines
Quoted 27 lines

We could maybe with hard work save the last two calls for reverting to root, but this will force us to audit lots of code that we are not prepared to do right now. And will not save us much.

[thread_setgroups()]
thread_setgroups() is what we use at Ganesha and what Samaba guys use for a per-thread setgroups() call. In the Linux Kernel the setgroups is actually always per thread. It is only the POSIX (crap) pthread layer at glibc that intercepts the setgroups() call (and others), Iterates on all threads that belong to a process, and calls the native Kernel setgroups on them. So thread_setgroups() is just the raw syscall bypassing glibc's processing. We will eventually push this API to glibc.
BTW: this is done exactly the same on FreeBSD, with same exact glibc intervention.

[Proposed]
What Jim proposed is a syscall that receives a struct that has the regular syscalls parameters plus the creds structure with fsuid/fsgid and groups array. Kernel will set these in, call the original syscall, and revert. This will be done on only an interested subset of the syscalls that are one - are related to filesystems (setfsXid) and two - are of interest to us Servers.

Jim care to scribble a structure definition?

Thanks
Boaz

#30 lore raw

Re: Re: [5/8] syscall_cred() a system call that receives alternate CREDs

In reply to J. Bruce Fields
Quoted 14 lines · On Monday, April 08, 2013 10:42:02 J. Bruce Fields wrote:
Quoted 5 lines · Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

Quoted 7 lines

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.

Quoted 7 lines

To make sure I understand, you're saying that:

- the behavior you get out of those 6 syscalls is correct,
- you just want to be able to do exactly the same thing, but with 1 syscall. (For performance?)

Or is there some other issue?

I have attached the email I sent around on the nfs-ganesha list with a model api so we know the details.

Boaz replied "performance" but there are also race conditions to consider. If we get signals or ??? somewhere in the sequence, what is our state? Yes, the setfsuid call back to root can still be done but masquerading has any signals etc. be in the context of that user/group and there is one syscall to deal with, not a stream.

There may be selinux/apparmor issues to deal with too. If we first masquerade the thread and then apply all these access checks, as far as the kernel is concerned, it is the masqueraded user.

Quoted 8 lines
Quoted 4 lines

(Note that permission checking is done by Ganesha core, because

We may cache open fd(s) and such not, another topic)

Is there anything we could do to make it possible for you to depend on the kernel's permissions checking instead?

I concur with Frank's assessment here. There are more instances where nfs- ganesha is doing a syscall as the server than as the masqueraded user. In the pNFS case, this hardly happens at all. We looked at having the kernel do it but found that we also had to do it and mixing gets seriously messy. For starters, we really do want to share fd's.

Quoted 30 lines

--b.

Quoted 28 lines

We could maybe with hard work save the last two calls for reverting to root, but this will force us to audit lots of code that we are not prepared to do right now. And will not save us much.

[thread_setgroups()]
thread_setgroups() is what we use at Ganesha and what Samaba guys use for a per-thread setgroups() call. In the Linux Kernel the setgroups is actually always per thread. It is only the POSIX (crap) pthread layer at glibc that intercepts the setgroups() call (and others), Iterates on all threads that belong to a process, and calls the native Kernel setgroups
on them. So thread_setgroups() is just the raw syscall bypassing glibc's processing. We will eventually push this API to glibc.
BTW: this is done exactly the same on FreeBSD, with same exact glibc intervention.

[Proposed]
What Jim proposed is a syscall that receives a struct that has the regular syscalls parameters plus the creds structure with fsuid/fsgid and groups array. Kernel will set these in, call the original syscall, and revert. This will be done on only an interested subset of the syscalls that are one - are related to filesystems (setfsXid) and two - are of interest to us Servers.

Jim care to scribble a structure definition?

Thanks
Boaz

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.

In replying to the creds RFC branch, an idea came to me. What we need is a syscall for server syscalls. At first, I thought of doing something like what was done for the *at calls. That got pretty silly with some calls only needing an extra flag and others needing extra args. All of the glibc and abi pain was a mess I'd rather not repeat.

How about this idea:

/**
* @brief Syscall entry point for servers that need to masquerade as others *
* This is a privileged syscall.
*
* @param syscall_number [IN] syscall number from syscall.h
* @param syscall_args IN] the arguments for that syscall in a vector
mimicing the syscall prototype.
* @param creds [IN] credentials to use. See definition in fsal_types.h */

int server_syscall(int syscall_number, void *syscall_args, struct creds *creds);

This syscall would have its own matching vector of the kernels calls it does. Maybe this is a bit in the syscall vector. Point being not all calls would be supported, only a small set.

The syscall args would be packaged and managed like ioctl does it now. This is an extra dereference in the syscall processing to validate the struct and copy the args in/out. The same applies to creds only instead of applying them to the specific syscall's stack frame, they would go into the "effective" uid/gid for the thread.

We save the back and forth across the syscall barrier with slightly more overhead per affected call which is less than the multiple roundtrips for setfsuid/gid. As a priv'd syscall, it becomes outside the set of "posix" compliance so we can also bypass things like posix lock behavior. It is also expandable without breaking the bank on syscalls or moving ABIs.

Further rationale for this is that the *at calls and handle calls do have more general use and therefore fit in the set of general syscalls. This is an enabler for servers that can take over in user space tasks that once were mandated into the kernel because of these user masquerading issues.

Last point, No, I haven't researched what the Samba team has lobbied for but I suspect that if they are asking for variant syscalls like the *at case, this has lower impact.

Jim

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.
#31 lore raw

Re: Re: [5/8] syscall_cred() a system call that receives alternate CREDs

Quoted 23 lines · Jim Lieb
Quoted 14 lines · On Monday, April 08, 2013 10:42:02 J. Bruce Fields wrote:
Quoted 5 lines · Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

Quoted 7 lines

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.

Quoted 7 lines

To make sure I understand, you're saying that:

- the behavior you get out of those 6 syscalls is correct,
- you just want to be able to do exactly the same thing, but with 1 syscall. (For performance?)

Or is there some other issue?

Quoted 8 lines

I have attached the email I sent around on the nfs-ganesha list with a model api so we know the details.

Boaz replied "performance" but there are also race conditions to consider. If we get signals or ??? somewhere in the sequence, what is our state? Yes, the setfsuid call back to root can still be done but masquerading has any signals etc. be in the context of that user/group and there is one syscall to deal with, not a stream.

Sorry, I don't understand what you're saying here. Could you give an example showing a sequence of events with the wrong result?

Quoted 3 lines

There may be selinux/apparmor issues to deal with too. If we first masquerade the thread and then apply all these access checks, as far as the kernel is concerned, it is the masqueraded user.

I don't understand here either.

--b.

#32 lore raw

Re: Re: Re: [5/8] syscall_cred() a system call that receives alternate CREDs

Quoted 27 lines · On Monday, April 08, 2013 14:31:20 J. Bruce Fields wrote:
Quoted 26 lines · Jim Lieb
Quoted 14 lines · On Monday, April 08, 2013 10:42:02 J. Bruce Fields wrote:
Quoted 5 lines · Boaz Harrosh

From: Jim Lieb <[email protected]>

In current NFS Server (Ganesha) lots of operation becomes 6 syscalls (Or is it 7?)

Quoted 7 lines

- setfsuid(), setfsgid(), thread_setgroups()
- The OP
- Revert setfsuid(), setfsgid() to root

This is because if we do all these file operations as root then FS will not account for the quota a user have on create files, data space, and so on.

Quoted 7 lines

To make sure I understand, you're saying that:
- the behavior you get out of those 6 syscalls is correct,
- you just want to be able to do exactly the same thing, but

with 1 syscall. (For performance?)

Or is there some other issue?

I have attached the email I sent around on the nfs-ganesha list with a model api so we know the details.

Quoted 8 lines
Quoted 5 lines

Boaz replied "performance" but there are also race conditions to consider.
If we get signals or ??? somewhere in the sequence, what is our state?
Yes, the setfsuid call back to root can still be done but masquerading has any signals etc. be in the context of that user/group and there is one syscall to deal with, not a stream.

Sorry, I don't understand what you're saying here. Could you give an example showing a sequence of events with the wrong result?

We are setting user, primary group, and alt groups in sequence before we do the actual work (read/write/...). This is a potential TOCTOU race. Granted, there is little/no real atomic guarantee but implied in the syscall model is that creds don't change for the duration of a syscall. We go back to userspace multiple times with creds in intermediate state(s). Signals can happen anytime but are only checked on the way back out of the syscall or we can hold them off at critical times within a single syscall. Which syscall is is the one where the signal occurred? In our case, we minimally use signals (do no i/o etc.) but they are still there. If it is one syscall, we know.

We currently have an RFC implementation of a "creds wrapper" but it is still in flux and the codiing of all these calls to "get it right" is ugly. One call, done right would be much better.

We also have a problem with the setgroups. We escape in Linux because the kernel doesn't do it process wide and glibc fakes it. I don't want to depend on that. In FreeBSD, we can't do it at all since the creds are shared at the proc level. Note that I am constrained to think about portability and it's easier to sell a new syscall than to hack fundamental kernel structures which is why the "do to all" bit is in glibc...

Quoted 6 lines
Quoted 4 lines

There may be selinux/apparmor issues to deal with too. If we first masquerade the thread and then apply all these access checks, as far as the kernel is concerned, it is the masqueraded user.

I don't understand here either.

There is the security context nfs-ganesha would live in but actions on behalf of clients are (or will be in 4.2+) be in the context of the client. This is outside my expertise but I'd like to have a "masquerading" framework in place where it could be added in a known way, or at least we are thinking about it.

Capabilities have also been thrown into the mix. I will be the first to defer to the selinux/apparmor heavies but I'd like to have all that capability constricted down to one syscall that can be controlled, i.e. selinux says only real samba and real nfs-ganesha can do this call.

Quoted 2 lines

--b.

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.
#33 lore raw

Re: [5/8] syscall_cred() a system call that receives alternate CREDs

Quoted 10 lines · Jim Lieb

We are setting user, primary group, and alt groups in sequence before we do the actual work (read/write/...). This is a potential TOCTOU race. Granted, there is little/no real atomic guarantee but implied in the syscall model is that creds don't change for the duration of a syscall. We go back to userspace multiple times with creds in intermediate state(s). Signals can happen anytime but are only checked on the way back out of the syscall or we can hold them off at critical times within a single syscall. Which syscall is is the one where the signal occurred? In our case, we minimally use signals (do no i/o etc.) but they are still there. If it is one syscall, we know.

signals are crap long before our case. I would not really care about signals. The performance argument is good enough for me.

Quoted 7 lines

We currently have an RFC implementation of a "creds wrapper" but it is still in flux and the codiing of all these calls to "get it right" is ugly. One call, done right would be much better.

We also have a problem with the setgroups. We escape in Linux because the kernel doesn't do it process wide and glibc fakes it. I don't want to depend on that. In FreeBSD, we can't do it at all since the creds are shared at the

Sachin found that FBSD might be exactly the same as Linux here. Please talk to him to make sure?

Quoted 4 lines

proc level. Note that I am constrained to think about portability and it's easier to sell a new syscall than to hack fundamental kernel structures which is why the "do to all" bit is in glibc...

But yes a new syscall with new defined semantics is definitely a better way to go. Just for the sake of thread_setgroups such a call is well worth it. Completely bypass POSIX and be done with it. So strongly yes here.

Bruce, got the point? Current code with the undocumented thread_setgroups is a POSIX hack. And will break at any given moment in time. Only a new syscall with defined semantics will ever be correct.

Thanks
Boaz

#34 lore raw

Re: Re: [5/8] syscall_cred() a system call that receives alternate CREDs

Quoted 12 lines · On Tuesday, April 09, 2013 00:33:32 Boaz Harrosh wrote:
Quoted 11 lines · Jim Lieb

We are setting user, primary group, and alt groups in sequence before we do
the actual work (read/write/...). This is a potential TOCTOU race. Granted, there is little/no real atomic guarantee but implied in the syscall model is that creds don't change for the duration of a syscall. We go back to userspace multiple times with creds in intermediate state(s). Signals can happen anytime but are only checked on the way back out of the syscall or we can hold them off at critical times within a single syscall. Which syscall is is the one where the signal occurred?
In our case, we minimally use signals (do no i/o etc.) but they are
still there. If it is one syscall, we know.

Quoted 2 lines

signals are crap long before our case. I would not really care about signals. The performance argument is good enough for me.

In our case, true. But this is a syscall and we will be stuck with it forever. The reason for this syscall request, along with the locking request it that they were too narrow in scope. This is, or can be, and issue for non- threaded apps that do use pollling, signaling, and all the rest.

Quoted 11 lines
Quoted 9 lines

We currently have an RFC implementation of a "creds wrapper" but it is still in flux and the codiing of all these calls to "get it right" is ugly. One call, done right would be much better.

We also have a problem with the setgroups. We escape in Linux because the kernel doesn't do it process wide and glibc fakes it. I don't want to depend on that. In FreeBSD, we can't do it at all since the creds are shared at the

Sachin found that FBSD might be exactly the same as Linux here. Please talk to him to make sure?

I checked with Herb et al and they have a hack for CIFS (which scares me a bit...) but they confirmed that stock FreeBSD has the creds as a property of the proc, not a thread. Their structure sounds similar to Mach, aka OSF/1, aka Digital UNIX which had thread structs only contain the thread specific state and left things like creds and open fds in the proc struct. To handle their CIFS need, they added a creds_like (emphasis on the "like") struct that gets allocated on a CIFS related event (not sure on details) but this is specific to their kernel. Stock FreeBSD doesn't have it and it doesn't sound like we can use it.

Quoted 6 lines
Quoted 5 lines

proc level. Note that I am constrained to think about portability and it's
easier to sell a new syscall than to hack fundamental kernel structures which is why the "do to all" bit is in glibc...

Quoted 7 lines

But yes a new syscall with new defined semantics is definitely a better way to go. Just for the sake of thread_setgroups such a call is well worth it. Completely bypass POSIX and be done with it. So strongly yes here.

Bruce, got the point? Current code with the undocumented thread_setgroups is a POSIX hack. And will break at any given moment in time. Only a new syscall with defined semantics will ever be correct.

And my comments about selinux are to explore/define/get_alarmed about that dimension as well given that selinux and friends are another layer/alternate universe of access control. Hmm. Might need a flags arg in the api...

Quoted 3 lines

Thanks
Boaz

Signature · 3 lines
Jim Lieb
Linux Systems Engineer
Panasas Inc.
#35 lore raw

PATCH 6/8 Rich ACLs (continued, drive through this time)

In reply to Boaz Harrosh
Quoted 1 line

From the top of my head please correct me where I'm wrong.

Anish Kumar from IBM has posted a patchset to support rich ACLs at Filesystems of interest Specially also by the NFS client FS, and through the KNFSD by the exported FSs of interest.

His approach was to have both the POSIX-ACLs and the RICH-ACLs, co exist at the VFS and FS layer, and to interact nicely together. But a supporting FS needs to support two APIs and do its own code refactoring at the bottom.

What the VFS community wanted to see is that all FS APIs change to the bigger RICH-ACLs API, and the translation from rich to POSIX is done at the generic VFS layer, thoughts doing the refactoring once at the VFS.

Anish pressed between his IBM obligations and the huge amount of coding it will take to convert all FSs, has put the project on a back burner, and we have not seen any farther changes from him.

JV Please find and post a git tree with the latest code. And if there are some documentation post them here. Please also try to find the original ML thread that is captured above.

[Proposition]

There are more interested parties then Anish in this matter. I know for a fact that IBM, Panasas RedHat, and any NFS and CIFS community member would like to see this work done. I propose that we get a status update, on what's there now. Talk about a strategy for an incremental but complete Kernel transformation, and finally submission. And I think All parties above should invest in time and resources together to drive this through. I would like to see all this done under Anish's guide if possible.

[JV what is Anish's email ?]

[user-mode API]
And we will need a good local API for get/set of rich ACLs, currently they are interfaced through either a Windows CIFS client or through some NFS4 test applications. There should be an API that can be used both at the local FS as well as via the NFS-client. (And the new readdir-plus should give us a flag if there are ACLs present at an entry)

Thanks
Boaz

#36 lore raw

Re: [6/8] Rich ACLs (continued, drive through this time)

Hi Boaz,

Quoted 5 lines · Boaz Harrosh

From the top of my head please correct me where I'm wrong.

Anish Kumar from IBM has posted a patchset to support rich ACLs at Filesystems of interest Specially also by the NFS client FS, and through the KNFSD by the exported FSs of interest.

Quoted 8 lines

His approach was to have both the POSIX-ACLs and the RICH-ACLs, co exist at the VFS and FS layer, and to interact nicely together. But a supporting FS needs to support two APIs and do its own code refactoring at the bottom.

What the VFS community wanted to see is that all FS APIs change to the bigger RICH-ACLs API, and the translation from rich to POSIX is done at the generic VFS layer, thoughts doing the refactoring once at the VFS.

Yes, RICH-ACLs is important topic. HFS+ uses NFSv4 ACLs model but with own peculiarities. I published not so recently path set with implementation of ACLs support in HFS+ driver. I used as a basis mapping code (NFSv4 <-> POSIX ACLs) that it was implemented in nfsd. And J. Bruce Fields makes remark about necessity to generalize mapping code with purpose of sharing it between drivers. So, I am working on generalization mapping code (NFSv4 <-> POSIX ACLs) for using it in HFS+ driver. I hope that I'll finish this work soon. But, anyway, the RICH-ACLs scheme is more natural for HFS+.

With the best regards,
Vyacheslav Dubeyko.

Quoted 32 lines

Anish pressed between his IBM obligations and the huge amount of coding it will take to convert all FSs, has put the project on a back burner, and we have not seen any farther changes from him.

JV Please find and post a git tree with the latest code. And if there are some documentation post them here. Please also try to find the original ML thread that is captured above.

[Proposition]

There are more interested parties then Anish in this matter. I know for a fact that IBM, Panasas RedHat, and any NFS and CIFS community member would like to see this work done. I propose that we get a status update, on what's there now. Talk about a strategy for an incremental but complete Kernel transformation, and finally submission. And I think All parties above should invest in time and resources together to drive this through. I would like to see all this done under Anish's guide if possible.

[JV what is Anish's email ?]

[user-mode API]
And we will need a good local API for get/set of rich ACLs, currently they are interfaced through either a Windows CIFS client or through some NFS4 test applications. There should be an API that can be used both at the local FS as well as via the NFS-client. (And the new readdir-plus should give us a flag if there are ACLs present at an entry)

Thanks
Boaz

--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#37 lore raw

Re: [6/8] Rich ACLs (continued, drive through this time)

In reply to Boaz Harrosh
Quoted 23 lines · Boaz Harrosh

From the top of my head please correct me where I'm wrong.

Anish Kumar from IBM has posted a patchset to support rich ACLs at Filesystems of interest Specially also by the NFS client FS, and through the KNFSD by the exported FSs of interest.

His approach was to have both the POSIX-ACLs and the RICH-ACLs, co exist at the VFS and FS layer, and to interact nicely together. But a supporting FS needs to support two APIs and do its own code refactoring at the bottom.

What the VFS community wanted to see is that all FS APIs change to the bigger RICH-ACLs API, and the translation from rich to POSIX is done at the generic VFS layer, thoughts doing the refactoring once at the VFS.

Anish pressed between his IBM obligations and the huge amount of coding it will take to convert all FSs, has put the project on a back burner, and we have not seen any farther changes from him.

JV Please find and post a git tree with the latest code. And if there are some documentation post them here. Please also try to find the original ML thread that is captured above.

[Proposition]

Quoted 8 lines

There are more interested parties then Anish in this matter. I know for a fact that IBM, Panasas RedHat, and any NFS and CIFS community member would like to see this work done. I propose that we get a status update, on what's there now. Talk about a strategy for an incremental but complete Kernel transformation, and finally submission. And I think All parties above should invest in time and resources together to drive this through. I would like to see all this done under Anish's guide if possible.

[JV what is Anish's email ?]

Included Aneesh in the discussion.

Quoted 10 lines

[user-mode API]
And we will need a good local API for get/set of rich ACLs, currently they are interfaced through either a Windows CIFS client or through some NFS4 test applications. There should be an API that can be used both at the local FS as well as via the NFS-client. (And the new readdir-plus should give us a flag if there are ACLs present at an entry)

Thanks
Boaz

A message that is not in the archive (sent off-list or not yet received): OF4A1A78E0.CB4DED3E-ON87257B47.00549E35-88257B47.005520A8@us.ibm.com

#40 lore raw

Re: [7/8] Single call interface to getattr/setattr

Quoted 18 lines · Frank S Filz

Boaz Harrosh <[email protected]> wrote on 04/08/2013 03:43:23 AM:

Quoted 6 lines

[7/8] Single call interface to getattr/setattr

From: Frank S Filz <[email protected]>

Frank this is your call, I'm not at all familiar with this subject. Shoot ...

This is mostly on the setattr side, being able to do the following in a single call (with each item optional):

set mode
set acl
set owner
set owner_group
set atime
set mtime
set size

Quoted 5 lines

on the getattr side, the same, which would be a stat+getacl merged into one call.

Would be nice to be able to do an create and open(O_CREAT) with all those attr also (again, each attr being optional).

Just adding a comment. If we are going into new territory here then these should be RICH-ACLs and not POSIX ACLs, kill two birds with one stone.

Quoted 2 lines

Frank

Thanks Frank
Boaz

#41 lore raw

PATCH 8/8 Fix fsnotify short comings (single fd with recursive notifications).

In reply to Boaz Harrosh

From: DENIEL Philippe <[email protected]>

DENIEL Has reported that watching directories through the new fsnotify API, might miss some events like deletes and that a directory watch is not recursive, which means that we need to open two fd(s) for each directory in the cache. (Which halves our fd cache size).

Again from the top of my head, and I know nothing of this subject.

DENIEL please add any information here, so we can talk about it at LSF.

Thanks
Boaz

#42 lore raw

Re: [8/8] Fix fsnotify short comings (single fd with recursive notifications).

Quoted 11 lines · Boaz Harrosh

From: DENIEL Philippe <[email protected]>

DENIEL Has reported that watching directories through the new fsnotify API, might miss some events like deletes and that a directory watch is not recursive, which means that we need to open two fd(s) for each directory in the cache. (Which halves our fd cache size).

Again from the top of my head, and I know nothing of this subject.

DENIEL please add any information here, so we can talk about it at LSF.

What I have seen was this:
- if dnotify() is used, it gets every events. If used on a
directory, it will get every creation and deletion. But it is not recursive, so we need to use dnotify on every inode we manage.
- if fanotify is used, it is recursive and by using it on the root
of a filesystem, then you have events from the whole underlying tree. The trouble is that deletion seem not to be caught.

I do have tests written in C for that.

regards

Philippe

maintainer
#43 lore raw

Re: [8/8] Fix fsnotify short comings (single fd with recursive notifications).

Quoted 7 lines · DENIEL Philippe

What I have seen was this:
- if dnotify() is used, it gets every events. If used on a
directory, it will get every creation and deletion. But it is not recursive, so we need to use dnotify on every inode we manage.
- if fanotify is used, it is recursive and by using it on the
root of a filesystem, then you have events from the whole underlying tree. The trouble is that deletion seem not to be caught.

All *notify APIs are broken by design, film at 11... See also: "Doctor, it hurts when I do it"

#44 lore raw

Re: [8/8] Fix fsnotify short comings (single fd with recursive notifications).

In reply to DENIEL Philippe
Quoted 20 lines · DENIEL Philippe
Quoted 11 lines · Boaz Harrosh

From: DENIEL Philippe <[email protected]>

DENIEL Has reported that watching directories through the new fsnotify API, might miss some events like deletes and that a directory watch is not recursive, which means that we need to open two fd(s) for each directory in the cache. (Which halves our fd cache size).

Again from the top of my head, and I know nothing of this subject.

DENIEL please add any information here, so we can talk about it at LSF.

What I have seen was this:
- if dnotify() is used, it gets every events. If used on a
directory, it will get every creation and deletion. But it is not recursive, so we need to use dnotify on every inode we manage.
- if fanotify is used, it is recursive and by using it on the
root of a filesystem, then you have events from the whole underlying tree. The trouble is that deletion seem not to be caught.

Quoted 1 line

I do have tests written in C for that.

What are you actually using notifications for? If you had adequate leases/delegations, for example, would you still need them?

--b.

#45 lore raw
In reply to Boaz Harrosh
Quoted 21 lines · Boaz Harrosh

Hi

Steven suggested to discuss a readdir-plus system call. For a better LS but certainly better FS daemons. I have hijacked his topic to expand it to a slew of requests we, the FS daemons people, would like to see extended, so to make our lives better (and faster). It should be up to Steven and the comity if these are grouped together or separated into different talks.

I would like to send a set of emails. Each below topic in it's own mail. Any interested party, please post to the topic of your heart so we can collect all the information,, header files, status of each topic. The LSF talk should be not for introduction of the topics but for actual decision making.

Some of the topics already carry, experimental or more, implementations and suggested APIs. Some of these topics are just a cry for help, we know what we don't want, and maybe how we would like them, and what we need is from the community to give us a green light as to if it is wanted at all.

This is certainly an exhaustive list, we might want to drop some of them, if they don't generate much interest. Also if I forgot something please post (JV)

Quoted 9 lines

Here is the list of topics in this set:
- [1/8] readdir-plus system call (By Steven Whitehouse <[email protected]>)
- [2/8] Sane locks (UNPOSIX locks) (frank)
- [3/8] File delegations, Usermode API of Bruce's pending patches.
- [4/8] PNFS ioctls/syscall
- [5/8] syscall_cred() a system call that receives alternate creds (fsuid fsgid thread-groups)
- [6/8] Rich ACLs (continued, drive through this time)
- [7/8] Single call interface to getattr/setattr (Frank S Filz <[email protected]>)
- [8/8] Fix fsnotify short comings (single fd with recursive notifications).

WCC Provision. User level API need to :

getattr()
perform operation write/read/something
getattr()

Breaking this into 3 calls gives all kinds of race possibilities and makes WCC useless. Having one single interface to get all this info atomically will be a great help.

Quoted 4 lines

Thanks
Boaz