Memory corruption in nfs3_xdr_setaclargs()
Subsystems (2)
A mistake in calculating the space left in the header in nfs3_xdr_setaclargs() can cause memory corruption when setting a large number of acls.
Reproducer:
On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in /etc/exports
/test *(rw)
On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.
A change in call_header changes the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly in call_header 334ccfd545bba9690515f2c5c167d5adb161989b
The iov_len is set to the size of the header in call_header(). req->rq_slen = xdr_adjust_iovec(&req->rq_svec[0], p);
nfs3_xdr_setaclargs() depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header.
/* put as much of the acls into head as possible. */
len_in_head = min_t(unsigned int, buf->head->iov_len - base, len);
len -= len_in_head;
req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));Since at this stage, iov_len < base, len_in_head will always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.
The following patch which set len_in_head to 0 was tested with the reproducer and was found to fix the problem.
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req *p++ =htonl (args->mask); base = (char *)p - (char *)buf->head->iov_base;/* put as much of the acls into head as possible. */ - len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +if ( buf->head->iov_len > base ) + len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +else + len_in_head =0 ; len -= len_in_head; req->rq_slen =xdr_adjust_iovec (req->rq_svec, p + (len_in_head >>2 ));
Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.
Sachin Prabhu
Quoted 37 lines · Sachin S. Prabhu
A mistake in calculating the space left in the header in
nfs3_xdr_setaclargs()can cause memory corruption when setting a large number of acls.Reproducer:
On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in/etc/exports
/test *(rw)On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.A change in
call_headerchanges the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly incall_header334ccfd545bba9690515f2c5c167d5adb161989bThe
iov_lenis set to the size of the header incall_header().req->rq_slen=xdr_adjust_iovec(&req->rq_svec[0], p);
nfs3_xdr_setaclargs()depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header./* put as much of the acls into head as possible. */ len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); len -= len_in_head; req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));
Quoted 7 lines
Since at this stage,
iov_len< base,len_in_headwill always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.The following patch which set
len_in_headto 0 was tested with the reproducer and was found to fix the problem.
Looks alright. Could you please add a s-o-b line?
Cheers
Trond
Quoted 19 lines
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4
-1@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req *p++ =htonl (args->mask); base = (char *)p - (char *)buf->head->iov_base;/* put as much of the acls into head as possible. */ - len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +if ( buf->head->iov_len > base ) + len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +else + len_in_head =0 ; len -= len_in_head; req->rq_slen =xdr_adjust_iovec (req->rq_svec, p + (len_in_head >>2 ));Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.
Sachin Prabhu
A message that is not in the archive (sent off-list or not yet received): 1232475301.7055.14.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 42 lines · Trond Myklebust
Quoted 41 lines · Sachin S. Prabhu
A mistake in calculating the space left in the header in
nfs3_xdr_setaclargs()can cause memory corruption when setting a large number of acls.Reproducer:
On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in/etc/exports
/test *(rw)On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.A change in
call_headerchanges the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly incall_header334ccfd545bba9690515f2c5c167d5adb161989bThe
iov_lenis set to the size of the header incall_header().req->rq_slen=xdr_adjust_iovec(&req->rq_svec[0], p);
nfs3_xdr_setaclargs()depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header./* put as much of the acls into head as possible. */ len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); len -= len_in_head; req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));Since at this stage,
iov_len< base,len_in_headwill always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.
Quoted 9 lines
Quoted 3 lines
The following patch which set
len_in_headto 0 was tested with the reproducer and was found to fix the problem.Looks alright. Could you please add a s-o-b line?
Cheers
Trond
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4 -1
@@ -712,7 +712,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req *p++ =htonl (args->mask); base = (char *)p - (char *)buf->head->iov_base;/* put as much of the acls into head as possible. */ - len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +if ( buf->head->iov_len > base ) + len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +else + len_in_head =0 ; len -= len_in_head; req->rq_slen =xdr_adjust_iovec (req->rq_svec, p + (len_in_head >>2 ));
Quoted 71 lines · Trond Myklebust
Date: Tue, 20 Jan 2009 13:15:01 -0500
From: Trond Myklebust <[email protected]>
To: Sachin S. Prabhu <[email protected]>
Cc: [email protected]
Subject: Re: Memory corruption innfs3_xdr_setaclargs()Quoted 37 lines · Sachin S. Prabhu
A mistake in calculating the space left in the header in
nfs3_xdr_setaclargs()can cause memory corruption when setting a large number of acls.Reproducer:
On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in/etc/exports
/test *(rw)On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.A change in
call_headerchanges the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly incall_header334ccfd545bba9690515f2c5c167d5adb161989bThe
iov_lenis set to the size of the header incall_header().req->rq_slen=xdr_adjust_iovec(&req->rq_svec[0], p);
nfs3_xdr_setaclargs()depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header./* put as much of the acls into head as possible. */ len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); len -= len_in_head; req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));Quoted 7 lines
Since at this stage,
iov_len< base,len_in_headwill always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.The following patch which set
len_in_headto 0 was tested with the reproducer and was found to fix the problem.Looks alright. Could you please add a s-o-b line?
Cheers
TrondQuoted 14 lines
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4
-1@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req *p++ =htonl (args->mask); base = (char *)p - (char *)buf->head->iov_base;/* put as much of the acls into head as possible. */ - len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +if ( buf->head->iov_len > base ) + len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +else + len_in_head =0 ; len -= len_in_head; req->rq_slen =xdr_adjust_iovec (req->rq_svec, p + (len_in_head >>2 ));
Quoted 5 lines
Quoted 5 lines
Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.
Sachin Prabhu
Trond,
Do you know what happened to this patch? Given the nasty corruption aspect to the bug, we were expecting to see a fix for this merged into the mainline by now.
Thanks
-Kevin
--
Kevin W. Rudd
IBM Global Services - Linux L3
1-800-426-7378, 1-503-578-4161, T/L 775-4161
A message that is not in the archive (sent off-list or not yet received): alpine.LFD.1.10.0903050829420.3949-mupu0Q0mUPfkOmf+N4b0O9FgqiXiwxn+0E9HWUfgJXw@public.gmane.org
Quoted 76 lines · Kevin W. Rudd
Quoted 74 lines · Trond Myklebust
Date: Tue, 20 Jan 2009 13:15:01 -0500
From: Trond Myklebust <[email protected]>
To: Sachin S. Prabhu <[email protected]>
Cc: [email protected]
Subject: Re: Memory corruption innfs3_xdr_setaclargs()Quoted 37 lines · Sachin S. Prabhu
A mistake in calculating the space left in the header in
nfs3_xdr_setaclargs()can cause memory corruption when setting a large number of acls.Reproducer:
On Server:
1) Create directory /test and set mode 777.
mkdir /test; chmod 777 /test
2) Add 200 users and set default acl for user on /test
for i in {1..200}; do echo $i; useradd user$i; setfacl -m d:u:user$i:rwx /test;done
3) Add export /test in/etc/exports
/test *(rw)On client
1) Mount server:/test
mount server:/test /mnt
2) Create large number of directories on the the share. cd/mnt; for i in {1..1000}; do mkdir $i; done At this point, the client should crash.A change in
call_headerchanges the value req->rq_snd_buf->head[0]->iov_len to reflect the exact size of the header.
[PATCH] RPC: Ensure XDR iovec length is initialized correctly incall_header334ccfd545bba9690515f2c5c167d5adb161989bThe
iov_lenis set to the size of the header incall_header().req->rq_slen=xdr_adjust_iovec(&req->rq_svec[0], p);
nfs3_xdr_setaclargs()depends on the older behavior and uses this value when calculating the number of ACLs it can fit into the header./* put as much of the acls into head as possible. */ len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); len -= len_in_head; req->rq_slen = xdr_adjust_iovec(req->rq_svec, p + (len_in_head >> 2));Quoted 7 lines
Since at this stage,
iov_len< base,len_in_headwill always be set to len. For a large number of ACLs, this will end up over-writing other parts of memory on the nfs client.The following patch which set
len_in_headto 0 was tested with the reproducer and was found to fix the problem.Looks alright. Could you please add a s-o-b line?
Cheers
TrondQuoted 17 lines
nfs/nfs3xdr.c renamed from nfs/nfs3xdr.c.orig +4
-1@@ -691,7 +691,10 @@ nfs3_xdr_setaclargs(struct rpc_rqst *req *p++ =htonl (args->mask); base = (char *)p - (char *)buf->head->iov_base;/* put as much of the acls into head as possible. */ - len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +if ( buf->head->iov_len > base ) + len_in_head = min_t(unsigned int, buf->head->iov_len - base, len); +else + len_in_head =0 ; len -= len_in_head; req->rq_slen =xdr_adjust_iovec (req->rq_svec, p + (len_in_head >>2 ));Thanks to Kevin Rudd who did the major legwork here to figure the problem and create the patch.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Sachin Prabhu
Trond,
Do you know what happened to this patch? Given the nasty corruption aspect to the bug, we were expecting to see a fix for this merged into the mainline by now.
Going over that particular procedure reveals a number of bugs that all should be fixed.
Trying to estimate how many bytes are free in the header is _WRONG_. That memory is allocated with certain assumptions, which are set in the definition of ACL3_setaclargs_sz. In this case, the assumptions appear to be that we will fit a maximum of 5 ACEs in the header.
IOW: the real fix for your corruption problem is simply to set len_in_head to 2*(2+5*3). Currently, the patch supplied by Sachin will always set it to zero.
Secondly, doing page allocation in the XDR routine is _WRONG_. For one thing, look at the major memory leak which happens if the client needs to resend the request (which can happen every now and again due to a dropped tcp connection, or all the time due to UDP retransmits). Those pages should have been allocated in nfs3_proc_getacl(), and indeed that is where the wretched things are freed.
Trond
A message that is not in the archive (sent off-list or not yet received): 1236279188.13361.30.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 2 lines · Trond Myklebust
Going over that particular procedure reveals a number of bugs that all should be fixed.
So true ;-)
Quoted 7 lines
Trying to estimate how many bytes are free in the header is _WRONG_. That memory is allocated with certain assumptions, which are set in the definition of
ACL3_setaclargs_sz. In this case, the assumptions appear to be that we will fit a maximum of 5 ACEs in the header.
IOW: the real fix for your corruption problem is simply to setlen_in_headto 2*(2+5*3). Currently, the patch supplied by Sachin will always set it to zero.
Does setting len_in_head to 2*(2+5*3) take into account the header space already consumed? The initial patch was just a quick-n-dirty way of making sure we didn't overwrite the initial buffer, and a mechanism for generating some discussion around the other deficiencies in this procedure. I'll take the blame for it being a quick hack to simply take the heat off for a critical customer situation :)
Quoted 6 lines
Secondly, doing page allocation in the XDR routine is _WRONG_. For one thing, look at the major memory leak which happens if the client needs to resend the request (which can happen every now and again due to a dropped tcp connection, or all the time due to UDP retransmits). Those pages should have been allocated in
nfs3_proc_getacl(), and indeed that is where the wretched things are freed.
That's an issue I hadn't noticed, but this isn't really my area of expertice. Thanks for taking a closer look at this area of code.
-Kevin
A message that is not in the archive (sent off-list or not yet received): alpine.LFD.1.10.0903051135040.3949-mupu0Q0mUPfkOmf+N4b0O9FgqiXiwxn+0E9HWUfgJXw@public.gmane.org
Quoted 30 lines · Kevin W. Rudd
Quoted 2 lines · Trond Myklebust
Going over that particular procedure reveals a number of bugs that all should be fixed.
So true ;-)
Quoted 7 lines
Trying to estimate how many bytes are free in the header is _WRONG_. That memory is allocated with certain assumptions, which are set in the definition of
ACL3_setaclargs_sz. In this case, the assumptions appear to be that we will fit a maximum of 5 ACEs in the header.
IOW: the real fix for your corruption problem is simply to setlen_in_headto 2*(2+5*3). Currently, the patch supplied by Sachin will always set it to zero.Does setting
len_in_headto 2*(2+5*3) take into account the header space already consumed? The initial patch was just a quick-n-dirty way of making sure we didn't overwrite the initial buffer, and a mechanism for generating some discussion around the other deficiencies in this procedure. I'll take the blame for it being a quick hack to simply take the heat off for a critical customer situation :)Quoted 6 lines
Secondly, doing page allocation in the XDR routine is _WRONG_. For one thing, look at the major memory leak which happens if the client needs to resend the request (which can happen every now and again due to a dropped tcp connection, or all the time due to UDP retransmits). Those pages should have been allocated in
nfs3_proc_getacl(), and indeed that is where the wretched things are freed.
Quoted 2 lines
That's an issue I hadn't noticed, but this isn't really my area of expertice. Thanks for taking a closer look at this area of code.
Does the attached patch help?
Trond
A message that is not in the archive (sent off-list or not yet received): 1236370937.7244.52.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 5 lines · Trond Myklebust
Does the attached patch help?
Trond
Yes. No corruption or issues on my large-ACL-list test system. Thanks.
-Kevin