nfs4_getfacl "Failed getxattr operation" when too many ACL entries exist

Recently we've been prototyping serving Solaris ZFS exports via NFSv4 to some Linux hosts. These will some day be exposed to general users, so I've been testing things to see if I can break them. Anyway, it seems that nfs4_getfacl is only able to read ACLs with up to 208 entries. nfs4_setfacl is able to insert a 209th entry, but any attempts to view or edit the ACLs after that fail with:

Failed getxattr operation
: Input/output error

There are two ways to make the ACLs readable again:
1) Have someone log in to the Solaris box and remove some of the entries
2) Reset the ACLs using nfs4_setfacl -s some spec

Has anyone run into this issue before? Is it fixable? I didn't reach the same problem locally on the Solaris box, nor on another Solaris box with the same NFS mount, so it looks like it's a problem specific to Linux. Here's the versions of relevant packages on the test box running Gentoo (did I miss any?):
Kernel: 2.6.23-gentoo-r8
nfs-utils-1.1.0-r1
attr-2.4.39
nfs4-acl-tools-0.3.2

Quoted 14 lines · Brian De Wolf

Recently we've been prototyping serving Solaris ZFS exports via NFSv4 to some Linux hosts. These will some day be exposed to general users, so I've been testing things to see if I can break them. Anyway, it seems that nfs4_getfacl is only able to read ACLs with up to 208 entries. nfs4_setfacl is able to insert a 209th entry, but any attempts to view or edit the ACLs after that fail with:

Failed getxattr operation
: Input/output error

There are two ways to make the ACLs readable again:
1) Have someone log in to the Solaris box and remove some of the entries
2) Reset the ACLs using nfs4_setfacl -s some spec

Quoted 9 lines

Has anyone run into this issue before? Is it fixable? I didn't reach the same problem locally on the Solaris box, nor on another Solaris box with the same NFS mount, so it looks like it's a problem specific to Linux. Here's the versions of relevant packages on the test box running Gentoo (did I miss any?):
Kernel: 2.6.23-gentoo-r8
nfs-utils-1.1.0-r1
attr-2.4.39
nfs4-acl-tools-0.3.2

honestly, this probably stems from some naive, unrevisited <ahem>
assumptions still lingering nfs4-acl-tools code that need fixing. at the -very- least, nfs4_setfacl could save the original ACL and attempt to restore it if the setxattr() call fails.

it's possible this case involves the server, but i suspect the
tools. i'll look at this tomorrow and get back to you.

	d
	.

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 26 lines · david m. richter

Quoted 14 lines · Brian De Wolf

Recently we've been prototyping serving Solaris ZFS exports via NFSv4 to some Linux hosts. These will some day be exposed to general users, so I've been testing things to see if I can break them. Anyway, it seems that nfs4_getfacl is only able to read ACLs with up to 208 entries. nfs4_setfacl is able to insert a 209th entry, but any attempts to view or edit the ACLs after that fail with:

Failed getxattr operation
: Input/output error

There are two ways to make the ACLs readable again:
1) Have someone log in to the Solaris box and remove some of the entries
2) Reset the ACLs using nfs4_setfacl -s some spec

Quoted 9 lines

Has anyone run into this issue before? Is it fixable? I didn't reach the same problem locally on the Solaris box, nor on another Solaris box with the same NFS mount, so it looks like it's a problem specific to Linux. Here's the versions of relevant packages on the test box running Gentoo (did I miss any?):
Kernel: 2.6.23-gentoo-r8
nfs-utils-1.1.0-r1
attr-2.4.39
nfs4-acl-tools-0.3.2

Quoted 4 lines

honestly, this probably stems from some naive, unrevisited <ahem>
assumptions still lingering nfs4-acl-tools code that need fixing. at the -very- least, nfs4_setfacl could save the original ACL and attempt to restore it if the setxattr() call fails.

sorry, misread part of your letter the first time around -- it'd
be very bizarre if nfs4_getfacl influenced the ACL in any way, so i suspect that something's going awry with nfs4_setfacl. seeing such an arbitrary limit of 208 or 209 ACEs looks like the tools being dumb.

Quoted 31 lines · david m. richter

Quoted 29 lines · david m. richter

Quoted 14 lines · Brian De Wolf

Recently we've been prototyping serving Solaris ZFS exports via NFSv4 to some Linux hosts. These will some day be exposed to general users, so I've been testing things to see if I can break them. Anyway, it seems that nfs4_getfacl is only able to read ACLs with up to 208 entries. nfs4_setfacl is able to insert a 209th entry, but any attempts to view or edit the ACLs after that fail with:

Failed getxattr operation
: Input/output error

There are two ways to make the ACLs readable again:
1) Have someone log in to the Solaris box and remove some of the entries
2) Reset the ACLs using nfs4_setfacl -s some spec

Quoted 9 lines

Has anyone run into this issue before? Is it fixable? I didn't reach the same problem locally on the Solaris box, nor on another Solaris box with the same NFS mount, so it looks like it's a problem specific to Linux. Here's the versions of relevant packages on the test box running Gentoo (did I miss any?):
Kernel: 2.6.23-gentoo-r8
nfs-utils-1.1.0-r1
attr-2.4.39
nfs4-acl-tools-0.3.2

honestly, this probably stems from some naive, unrevisited <ahem>
assumptions still lingering nfs4-acl-tools code that need fixing. at the -very- least, nfs4_setfacl could save the original ACL and attempt to restore it if the setxattr() call fails.

Quoted 2 lines

sorry, misread part of your letter the first time around -- it'd
be very bizarre if nfs4_getfacl influenced the ACL in any way, so i suspect that something's going awry with nfs4_setfacl. seeing such an arbitrary limit of 208 or 209 ACEs looks like the tools being dumb.

I haven't looked at this code in a while. From a quick look.... It appears the kernel limits ACLs to 64K (xdr-encoded). One ACE has length

16 + (length of user/group name rounded up to multiple of 4)

But to be hitting that limit with 208 entries I think you'd have to have user/group names (including domain) of about 300 characters.

Anyway, strace'ing nfs4_getfacl/nfs4_setfacl would verify whether the error was coming from the kernel or the tools.

I have to ask: how many acl entries do you need?

Quoted 47 lines · J. Bruce Fields

Quoted 37 lines · david m. richter

Quoted 35 lines · david m. richter

Quoted 33 lines · Brian De Wolf

Recently we've been prototyping serving Solaris ZFS exports via NFSv4 to some
Linux hosts. These will some day be exposed to general users, so I've been
testing things to see if I can break them. Anyway, it seems that nfs4_getfacl
is only able to read ACLs with up to 208 entries. nfs4_setfacl is able to
insert a 209th entry, but any attempts to view or edit the ACLs after that
fail with:

Failed getxattr operation
: Input/output error

There are two ways to make the ACLs readable again:
1) Have someone log in to the Solaris box and remove some of the entries
2) Reset the ACLs using nfs4_setfacl -s some spec

Has anyone run into this issue before? Is it fixable? I didn't reach the
same problem locally on the Solaris box, nor on another Solaris box with the
same NFS mount, so it looks like it's a problem specific to Linux. Here's the
versions of relevant packages on the test box running Gentoo (did I miss
any?):
Kernel: 2.6.23-gentoo-r8
nfs-utils-1.1.0-r1
attr-2.4.39
nfs4-acl-tools-0.3.2

Quoted 8 lines

Quoted 4 lines

honestly, this probably stems from some naive, unrevisited <ahem>
assumptions still lingering nfs4-acl-tools code that need fixing. at the -very- least, nfs4_setfacl could save the original ACL and attempt to restore it if the setxattr() call fails.

sorry, misread part of your letter the first time around -- it'd be
very bizarre if nfs4_getfacl influenced the ACL in any way, so i suspect that something's going awry with nfs4_setfacl. seeing such an arbitrary limit of 208 or 209 ACEs looks like the tools being dumb.

Quoted 4 lines

I haven't looked at this code in a while. From a quick look.... It appears the kernel limits ACLs to 64K (xdr-encoded). One ACE has length

16 + (length of user/group name rounded up to multiple of 4)

More or less, yes. An strace of the ruleset "A::OWNER@:" yields a getxattr buffer size of 28 bytes.

Quoted 2 lines

But to be hitting that limit with 208 entries I think you'd have to have user/group names (including domain) of about 300 characters.

Unfortunately not. With 209 lines of "A::OWNER@:", it breaks. 208 lines of this makes a getxattr buffer of size 4996. If I use "A::EVERYONE@:", it ends up breaking at 180 lines. At 179 lines, this requires a buffer of 4988 bytes. It looks like there might be a ceiling at 5000 bytes?

Quoted 2 lines

Anyway, strace'ing nfs4_getfacl/nfs4_setfacl would verify whether the error was coming from the kernel or the tools.

This is when the attributes list is too long:

getxattr("hello", "system.nfs4_acl"..., 0x0, 0) = -1 EIO (Input/output error)

I couldn't find a mention of EIO in the man pages for getxattr(2) or stat(2).

Quoted 1 line

I have to ask: how many acl entries do you need?

We don't plan on using huge ACLs, but it's nice to know they'll work if someone tries to use them. If I could limit the maximum number of ACL entries to something smaller, I would have done that instead, but it's not configurable.

Quoted 54 lines · Brian De Wolf

Quoted 42 lines · J. Bruce Fields

Quoted 41 lines · david m. richter

Quoted 39 lines · david m. richter

Quoted 20 lines · Brian De Wolf

Recently we've been prototyping serving Solaris ZFS exports via NFSv4 to some
Linux hosts. These will some day be exposed to general users, so I've been
testing things to see if I can break them. Anyway, it seems that nfs4_getfacl
is only able to read ACLs with up to 208 entries. nfs4_setfacl is able to
insert a 209th entry, but any attempts to view or edit the ACLs after that
fail with:

Failed getxattr operation
: Input/output error

There are two ways to make the ACLs readable again:
1) Have someone log in to the Solaris box and remove some of the entries
2) Reset the ACLs using nfs4_setfacl -s some spec

Quoted 13 lines

Has anyone run into this issue before? Is it fixable? I didn't reach the
same problem locally on the Solaris box, nor on another Solaris box with the
same NFS mount, so it looks like it's a problem specific to Linux. Here's the
versions of relevant packages on the test box running Gentoo (did I miss
any?):
Kernel: 2.6.23-gentoo-r8
nfs-utils-1.1.0-r1
attr-2.4.39
nfs4-acl-tools-0.3.2

honestly, this probably stems from some naive, unrevisited <ahem>
assumptions still lingering nfs4-acl-tools code that need fixing. at the -very- least, nfs4_setfacl could save the original ACL and attempt to restore it if the setxattr() call fails.

Quoted 8 lines

Quoted 4 lines

sorry, misread part of your letter the first time around -- it'd be
very bizarre if nfs4_getfacl influenced the ACL in any way, so i suspect that something's going awry with nfs4_setfacl. seeing such an arbitrary limit of 208 or 209 ACEs looks like the tools being dumb.

I haven't looked at this code in a while. From a quick look.... It appears the kernel limits ACLs to 64K (xdr-encoded). One ACE has length

16 + (length of user/group name rounded up to multiple of 4)

More or less, yes. An strace of the ruleset "A::OWNER@:" yields a getxattr buffer size of 28 bytes.

Quoted 6 lines

Quoted 2 lines

But to be hitting that limit with 208 entries I think you'd have to have user/group names (including domain) of about 300 characters.

Unfortunately not. With 209 lines of "A::OWNER@:", it breaks. 208 lines of this makes a getxattr buffer of size 4996. If I use "A::EVERYONE@:", it ends up breaking at 180 lines. At 179 lines, this requires a buffer of 4988 bytes. It looks like there might be a ceiling at 5000 bytes?

oh good, i was going to ask this very thing :)

Quoted 6 lines

Quoted 2 lines

Anyway, strace'ing nfs4_getfacl/nfs4_setfacl would verify whether the error was coming from the kernel or the tools.

This is when the attributes list is too long:

getxattr("hello", "system.nfs4_acl"..., 0x0, 0) = -1 EIO (Input/output error)

Quoted 1 line

I couldn't find a mention of EIO in the man pages for getxattr(2) or stat(2).

yup, i had the same thought; the manpages don't have the whole
story. glancing through, it looks like there are some ways that NFS could end up returning an EIO that'd percolate back through sys_getxattr() ..

if you would, could you get a tcpdump of both the
nfs4_setfacl-setting-a-too-long-ACL and the subsequent nfs4_getfacl-barfs-up-EIO problems? please use a snaplen of 0 just to make sure the payloads come through. you can email it to me, if you like.

Quoted 5 lines

Quoted 1 line

I have to ask: how many acl entries do you need?

We don't plan on using huge ACLs, but it's nice to know they'll work if someone tries to use them. If I could limit the maximum number of ACL entries to something smaller, I would have done that instead, but it's not configurable.

<nod> the 64K size limit (linux/limits.h) goes for all individual
xattrs, so there's some ceiling, but that ceiling should be a long way off. if you can gin up that tcpdump capture, that'd be great.

	thanks,

	d
	.