Skip to contentSkip to conversation

Memory corruption in nfs3_xdr_setaclargs()

PATCH Sachin S. Prabhu 3 participants 8 messages Jan 20, 2009 – Mar 6, 2009 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

A mistake in calculating the space left in the header in nfs3_xdr_setaclargs() can cause memory corruption when setting a large number of acls.

Reproducer:

On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in /etc/exports
/test *(rw)

On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.

A change in call_header changes the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly in call_header 334ccfd545bba9690515f2c5c167d5adb161989b

The iov_len is set to the size of the header in call_header(). req->rq_slen = xdr_adjust_iovec(&req->rq_svec[0], p);

nfs3_xdr_setaclargs() depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header.

        /* put as much of the acls into head as possible. */
        len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
        len -= len_in_head;
        req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));

Since at this stage, iov_len < base, len_in_head will always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.

The following patch which set len_in_head to 0 was tested with the reproducer and was found to fix the problem.

nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req
 	*p++ = htonl(args->mask);
 	base = (char *)p - (char *)buf->head->iov_base;
 	/* put as much of the acls into head as possible. */
-	len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	if ( buf->head->iov_len > base )
+		len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	else
+		len_in_head = 0;
 	len -= len_in_head;
 	req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
 

Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.

Sachin Prabhu

#1 lore raw
Quoted 37 lines · Sachin S. Prabhu

A mistake in calculating the space left in the header in nfs3_xdr_setaclargs() can cause memory corruption when setting a large number of acls.

Reproducer:

On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in /etc/exports
/test *(rw)

On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.

A change in call_header changes the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly in call_header 334ccfd545bba9690515f2c5c167d5adb161989b

The iov_len is set to the size of the header in call_header(). req->rq_slen = xdr_adjust_iovec(&req->rq_svec[0], p);

nfs3_xdr_setaclargs() depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header.

        /* put as much of the acls into head as possible. */
        len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
        len -= len_in_head;
        req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
Quoted 7 lines

Since at this stage, iov_len < base, len_in_head will always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.

The following patch which set len_in_head to 0 was tested with the reproducer and was found to fix the problem.

Looks alright. Could you please add a s-o-b line?

Cheers
Trond

Quoted 19 lines
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req
 	*p++ = htonl(args->mask);
 	base = (char *)p - (char *)buf->head->iov_base;
 	/* put as much of the acls into head as possible. */
-	len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	if ( buf->head->iov_len > base )
+		len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	else
+		len_in_head = 0;
 	len -= len_in_head;
 	req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
 

Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.

Sachin Prabhu

A message that is not in the archive (sent off-list or not yet received): 1232475301.7055.14.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#3 lore raw
Quoted 42 lines · Trond Myklebust
Quoted 41 lines · Sachin S. Prabhu

A mistake in calculating the space left in the header in nfs3_xdr_setaclargs() can cause memory corruption when setting a large number of acls.

Reproducer:

On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in /etc/exports
/test *(rw)

On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.

A change in call_header changes the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly in call_header 334ccfd545bba9690515f2c5c167d5adb161989b

The iov_len is set to the size of the header in call_header(). req->rq_slen = xdr_adjust_iovec(&req->rq_svec[0], p);

nfs3_xdr_setaclargs() depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header.

        /* put as much of the acls into head as possible. */
        len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
        len -= len_in_head;
        req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));

Since at this stage, iov_len < base, len_in_head will always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.

Quoted 9 lines
Quoted 3 lines

The following patch which set len_in_head to 0 was tested with the reproducer and was found to fix the problem.

Looks alright. Could you please add a s-o-b line?

Cheers
Trond

nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -712,7 +712,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req
 	*p++ = htonl(args->mask);
 	base = (char *)p - (char *)buf->head->iov_base;
 	/* put as much of the acls into head as possible. */
-	len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	if ( buf->head->iov_len > base )
+		len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	else
+		len_in_head = 0;
 	len -= len_in_head;
 	req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
 
#4 lore raw
In reply to a message not in the archive
Quoted 71 lines · Trond Myklebust

Date: Tue, 20 Jan 2009 13:15:01 -0500
From: Trond Myklebust <[email protected]>
To: Sachin S. Prabhu <[email protected]>
Cc: [email protected]
Subject: Re: Memory corruption in nfs3_xdr_setaclargs()

Quoted 37 lines · Sachin S. Prabhu

A mistake in calculating the space left in the header in nfs3_xdr_setaclargs() can cause memory corruption when setting a large number of acls.

Reproducer:

On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in /etc/exports
/test *(rw)

On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.

A change in call_header changes the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly in call_header 334ccfd545bba9690515f2c5c167d5adb161989b

The iov_len is set to the size of the header in call_header(). req->rq_slen = xdr_adjust_iovec(&req->rq_svec[0], p);

nfs3_xdr_setaclargs() depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header.

        /* put as much of the acls into head as possible. */
        len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
        len -= len_in_head;
        req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
Quoted 7 lines

Since at this stage, iov_len < base, len_in_head will always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.

The following patch which set len_in_head to 0 was tested with the reproducer and was found to fix the problem.

Looks alright. Could you please add a s-o-b line?

Cheers
Trond

Quoted 14 lines
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req
 	*p++ = htonl(args->mask);
 	base = (char *)p - (char *)buf->head->iov_base;
 	/* put as much of the acls into head as possible. */
-	len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	if ( buf->head->iov_len > base )
+		len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	else
+		len_in_head = 0;
 	len -= len_in_head;
 	req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
 
Quoted 5 lines
Quoted 5 lines

Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.

Sachin Prabhu

Trond,

Do you know what happened to this patch? Given the nasty corruption aspect to the bug, we were expecting to see a fix for this merged into the mainline by now.

Thanks

-Kevin

--
Kevin W. Rudd
IBM Global Services - Linux L3
1-800-426-7378, 1-503-578-4161, T/L 775-4161

A message that is not in the archive (sent off-list or not yet received): alpine.LFD.1.10.0903050829420.3949-mupu0Q0mUPfkOmf+N4b0O9FgqiXiwxn+0E9HWUfgJXw@public.gmane.org

#6 lore raw
Quoted 76 lines · Kevin W. Rudd
Quoted 74 lines · Trond Myklebust

Date: Tue, 20 Jan 2009 13:15:01 -0500
From: Trond Myklebust <[email protected]>
To: Sachin S. Prabhu <[email protected]>
Cc: [email protected]
Subject: Re: Memory corruption in nfs3_xdr_setaclargs()

Quoted 37 lines · Sachin S. Prabhu

A mistake in calculating the space left in the header in nfs3_xdr_setaclargs() can cause memory corruption when setting a large number of acls.

Reproducer:

On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in /etc/exports
/test *(rw)

On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.

A change in call_header changes the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly in call_header 334ccfd545bba9690515f2c5c167d5adb161989b

The iov_len is set to the size of the header in call_header(). req->rq_slen = xdr_adjust_iovec(&req->rq_svec[0], p);

nfs3_xdr_setaclargs() depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header.

        /* put as much of the acls into head as possible. */
        len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
        len -= len_in_head;
        req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
Quoted 7 lines

Since at this stage, iov_len < base, len_in_head will always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.

The following patch which set len_in_head to 0 was tested with the reproducer and was found to fix the problem.

Looks alright. Could you please add a s-o-b line?

Cheers
Trond

Quoted 17 lines
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req
 	*p++ = htonl(args->mask);
 	base = (char *)p - (char *)buf->head->iov_base;
 	/* put as much of the acls into head as possible. */
-	len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	if ( buf->head->iov_len > base )
+		len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
+	else
+		len_in_head = 0;
 	len -= len_in_head;
 	req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
 

Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Sachin Prabhu

Trond,

Do you know what happened to this patch? Given the nasty corruption aspect to the bug, we were expecting to see a fix for this merged into the mainline by now.

Going over that particular procedure reveals a number of bugs that all should be fixed.

Trying to estimate how many bytes are free in the header is _WRONG_. That memory is allocated with certain assumptions, which are set in the definition of ACL3_setaclargs_sz. In this case, the assumptions appear to be that we will fit a maximum of 5 ACEs in the header.
IOW: the real fix for your corruption problem is simply to set len_in_head to 2*(2+5*3). Currently, the patch supplied by Sachin will always set it to zero.

Secondly, doing page allocation in the XDR routine is _WRONG_. For one thing, look at the major memory leak which happens if the client needs to resend the request (which can happen every now and again due to a dropped tcp connection, or all the time due to UDP retransmits). Those pages should have been allocated in nfs3_proc_getacl(), and indeed that is where the wretched things are freed.

Trond

A message that is not in the archive (sent off-list or not yet received): 1236279188.13361.30.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#8 lore raw
Quoted 2 lines · Trond Myklebust

Going over that particular procedure reveals a number of bugs that all should be fixed.

So true ;-)

Quoted 7 lines

Trying to estimate how many bytes are free in the header is _WRONG_. That memory is allocated with certain assumptions, which are set in the definition of ACL3_setaclargs_sz. In this case, the assumptions appear to be that we will fit a maximum of 5 ACEs in the header.
IOW: the real fix for your corruption problem is simply to set len_in_head to 2*(2+5*3). Currently, the patch supplied by Sachin will always set it to zero.

Does setting len_in_head to 2*(2+5*3) take into account the header space already consumed? The initial patch was just a quick-n-dirty way of making sure we didn't overwrite the initial buffer, and a mechanism for generating some discussion around the other deficiencies in this procedure. I'll take the blame for it being a quick hack to simply take the heat off for a critical customer situation :)

Quoted 6 lines

Secondly, doing page allocation in the XDR routine is _WRONG_. For one thing, look at the major memory leak which happens if the client needs to resend the request (which can happen every now and again due to a dropped tcp connection, or all the time due to UDP retransmits). Those pages should have been allocated in nfs3_proc_getacl(), and indeed that is where the wretched things are freed.

That's an issue I hadn't noticed, but this isn't really my area of expertice. Thanks for taking a closer look at this area of code.

-Kevin

A message that is not in the archive (sent off-list or not yet received): alpine.LFD.1.10.0903051135040.3949-mupu0Q0mUPfkOmf+N4b0O9FgqiXiwxn+0E9HWUfgJXw@public.gmane.org

#10 lore raw
Quoted 30 lines · Kevin W. Rudd
Quoted 2 lines · Trond Myklebust

Going over that particular procedure reveals a number of bugs that all should be fixed.

So true ;-)

Quoted 7 lines

Trying to estimate how many bytes are free in the header is _WRONG_. That memory is allocated with certain assumptions, which are set in the definition of ACL3_setaclargs_sz. In this case, the assumptions appear to be that we will fit a maximum of 5 ACEs in the header.
IOW: the real fix for your corruption problem is simply to set len_in_head to 2*(2+5*3). Currently, the patch supplied by Sachin will always set it to zero.

Does setting len_in_head to 2*(2+5*3) take into account the header space already consumed? The initial patch was just a quick-n-dirty way of making sure we didn't overwrite the initial buffer, and a mechanism for generating some discussion around the other deficiencies in this procedure. I'll take the blame for it being a quick hack to simply take the heat off for a critical customer situation :)

Quoted 6 lines

Secondly, doing page allocation in the XDR routine is _WRONG_. For one thing, look at the major memory leak which happens if the client needs to resend the request (which can happen every now and again due to a dropped tcp connection, or all the time due to UDP retransmits). Those pages should have been allocated in nfs3_proc_getacl(), and indeed that is where the wretched things are freed.

Quoted 2 lines

That's an issue I hadn't noticed, but this isn't really my area of expertice. Thanks for taking a closer look at this area of code.

Does the attached patch help?

Trond

A message that is not in the archive (sent off-list or not yet received): 1236370937.7244.52.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#12 lore raw
Quoted 5 lines · Trond Myklebust

Does the attached patch help?

Trond

Yes. No corruption or issues on my large-ACL-list test system. Thanks.

-Kevin