Skip to contentSkip to conversation

Linux 2.4.33-rc2

DISCUSSION Marcelo Tosatti 3 participants 8 messages Jun 21, 2006 – Jul 6, 2006

lore
Share & tools
lore
#0 lore raw

A few problems appeared on -rc1... More networking security updates.

Summary of changes from v2.4.33-rc1 to v2.4.33-rc2
============================================

Marcelo Tosatti:

      Change VERSION to v2.4.33-rc2

Mikael Pettersson:

      [PATCH 2.4.33-rc1] repair __ide_dma_no_op breakage

Solar Designer:
[NETFILTER]: Fix do_add_counters race, possible oops or info leak (CVE-2006-0039)

Vlad Yasevich:
[SCTP]: Validate the parameter length in HB-ACK chunk. (CVE-2006-1857) [SCTP]: Respect the real chunk length when walking parameters. (CVE-2006-1858)

Willy Tarreau:
Fix vfs_unlink/NFS NULL pointer dereference range checking for sleep states sent to /proc/acpi/sleep

#1 lore raw
Quoted 16 lines · Marcelo Tosatti

A few problems appeared on -rc1... More networking security updates.

Summary of changes from v2.4.33-rc1 to v2.4.33-rc2
============================================

Marcelo Tosatti:

     Change VERSION to v2.4.33-rc2

Mikael Pettersson:

     [PATCH 2.4.33-rc1] repair __ide_dma_no_op breakage

Solar Designer:
[NETFILTER]: Fix do_add_counters race, possible oops or info leak (CVE-2006-0039)

Quoted 7 lines

Vlad Yasevich:
[SCTP]: Validate the parameter length in HB-ACK chunk. (CVE-2006-1857) [SCTP]: Respect the real chunk length when walking parameters. (CVE-2006-1858)

Willy Tarreau:
Fix vfs_unlink/NFS NULL pointer dereference range checking for sleep states sent to /proc/acpi/sleep

Things are looking up ;) <http://bugsplatter.mine.nu/test/linux-2.4/>

+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
| kernel version |deltree|hal |niner |peetoo |pooh |sempro |silly |tosh |
+ - - - - - - - - + - - - + - - - + - - - + - - - + - - - + - - - + - - - + - - - +

| 2.4.33-rc2      |   Y   |   Y   |   Y   |   Y   |       |   Y   |   Y   |   Y   |
| 2.4.33-rc1      |   -   |   -   |   -   |   -   |       |   X   |   -   |   X   |

Grant.

#2 lore raw
In reply to Marcelo Tosatti
Quoted 2 lines · Marcelo Tosatti

Willy Tarreau:
Fix vfs_unlink/NFS NULL pointer dereference

Marcelo, I'm not sure this one is perfect yet. Today, while packaging a lot of files for our distro at work, I came up with a problem where deleting a file on NFS, and later simply accessing (read/write/create) a file on the NFS file system did block. However, I could kill all the offending processes. This was after a full day of mkdir/create/open/ unlink... (tens of thoudands of those), so it is not much reproduceable.

I could not unmount the NFS anymore, while other users had no problem. Rebooting the client solved the problem. I caught an RPC trace (attached), not sure if it can help. I must say that I'm also running Trond's NFS patches which I suspected first, but with which I never encountered a single problem for years.

The fact that the problem appeared during an rm -rf made me think about the vfs_unlink() patch. I went to read it again an I'm wondering if we have not inserted a new problem (please forgive my ignorance here) :

in 2.4.32, we had the following sequence :

        down(&dir->i_zombie);
        if (may_delete(dir, dentry, 0) != 0) return;
        lock_kernel();
        error = dir->i_op->unlink(dir, dentry);
        unlock_kernel();
        if (!error)
              d_delete(dentry);
        up(&dir->i_zombie);
        if (!error)
                inode_dir_notify(dir, DN_DELETE);

int 2.4.33-rc2, we have :

        if (may_delete(dir, dentry, 0) != 0) return;
        inode = dentry->d_inode;

        atomic_inc(&inode->i_count);
        double_down(&dir->i_zombie, &inode->i_zombie);
 
        lock_kernel();
        error = dir->i_op->unlink(dir, dentry);
        unlock_kernel();

        double_up(&dir->i_zombie, &inode->i_zombie);
        iput(inode);

        if (!error) {
                d_delete(dentry);
                inode_dir_notify(dir, DN_DELETE);
        }

What I notice is that in 2.4.32, d_delete(dentry) was performed between down(&dir->i_zombie) and up(&dir->i_zombie), while now it's completely outside. I wonder if this can cause race conditions or not, but at least, I'm sure that we have changed the locking sequence, which might have some impact.

Do you think I'm searching in the wrong direction ? I worry a bit, because getting a deadlock after only one day, it's a bit early :-/

Thanks,
Willy

--- dmesg after writing to /proc/sys/sunrpc/* ---

nfs: flush(a/100663641)
nfs: write(utm-gateway/truc(100663641), 5@0)
nfs: flush(a/100663641)
RPC: 43724 new task procpid 12145
RPC: 43724 rpc_execute flgs 1
RPC: 43724 deleting timer
RPC: 43724 call_start nfs3 proc 7 (async)
RPC: 43724 deleting timer
RPC: 43724 call_reserve
RPC: 43724 reserved req 925b01cc xid 8c207ac6
RPC: 43724 deleting timer
RPC: 43724 call_reserveresult (status 0)
RPC: 43724 deleting timer
RPC: 43724 call_allocate (status 0)
RPC:      allocated buffer 7faa7800
RPC: 43724 deleting timer
RPC: 43724 call_encode (status 0)
RPC: 43724 deleting timer
RPC: 43724 call_bind xprt 925b0000 is connected
RPC: 43724 deleting timer
RPC: 43724 call_transmit (status 0)
RPC: 43724 xprt_transmit(8c207ac6)
RPC: 43724 xprt_cwnd_limited cong = 0 cwnd = 4021
RPC:      xprt_sendmsg(0) = 188
RPC: 43724 xmit complete
RPC: 43724 sleep_on(queue "xprt_pending" time 7214624)
RPC: 43724 added to queue 925b0058 "xprt_pending"
RPC: 43724 setting alarm for 104 ms
RPC:      wake_up_next(925b004c "xprt_resend")
RPC:      wake_up_next(925b0040 "xprt_sending")
RPC:      udp_data_ready...
RPC:      udp_data_ready client 925b0000
RPC: 43724 received reply
RPC:      cong 256, cwnd was 4021, now 4021
RPC:      wake_up_next(925b004c "xprt_resend")
RPC:      wake_up_next(925b0040 "xprt_sending")
RPC: 43724 has input (136 bytes)
RPC: 43724 __rpc_wake_up_task (now 7214625 inh 0)
RPC: 43724 disabling timer
RPC: 43724 removed from queue 925b0058 "xprt_pending"
RPC: 43724 added to queue 9c46448c "schedq"
RPC:      __rpc_wake_up_task done
#3 lore raw
Quoted 60 lines · Willy Tarreau
Quoted 2 lines · Marcelo Tosatti

Willy Tarreau:
Fix vfs_unlink/NFS NULL pointer dereference

Marcelo, I'm not sure this one is perfect yet. Today, while packaging a lot of files for our distro at work, I came up with a problem where deleting a file on NFS, and later simply accessing (read/write/create) a file on the NFS file system did block. However, I could kill all the offending processes. This was after a full day of mkdir/create/open/ unlink... (tens of thoudands of those), so it is not much reproduceable.

I could not unmount the NFS anymore, while other users had no problem. Rebooting the client solved the problem. I caught an RPC trace (attached), not sure if it can help. I must say that I'm also running Trond's NFS patches which I suspected first, but with which I never encountered a single problem for years.

The fact that the problem appeared during an rm -rf made me think about the vfs_unlink() patch. I went to read it again an I'm wondering if we have not inserted a new problem (please forgive my ignorance here) :

in 2.4.32, we had the following sequence :

       down(&dir->i_zombie);
       if (may_delete(dir, dentry, 0) != 0) return;
       lock_kernel();
       error = dir->i_op->unlink(dir, dentry);
       unlock_kernel();
       if (!error)
             d_delete(dentry);
       up(&dir->i_zombie);
       if (!error)
               inode_dir_notify(dir, DN_DELETE);

int 2.4.33-rc2, we have :

       if (may_delete(dir, dentry, 0) != 0) return;
       inode = dentry->d_inode;

       atomic_inc(&inode->i_count);
       double_down(&dir->i_zombie, &inode->i_zombie);

       lock_kernel();
       error = dir->i_op->unlink(dir, dentry);
       unlock_kernel();

       double_up(&dir->i_zombie, &inode->i_zombie);
       iput(inode);

       if (!error) {
               d_delete(dentry);
               inode_dir_notify(dir, DN_DELETE);
       }

What I notice is that in 2.4.32, d_delete(dentry) was performed between down(&dir->i_zombie) and up(&dir->i_zombie), while now it's completely outside. I wonder if this can cause race conditions or not, but at least, I'm sure that we have changed the locking sequence, which might have some impact.

Quoted 4 lines

Do you think I'm searching in the wrong direction ? I worry a bit, because getting a deadlock after only one day, it's a bit early :-/

Assuming you mean something like the patch below? Doesn't cause any problems (yet, still testing) like eat files or segfault here as reported for -rc1 +/- various patches ;)

Cheers,
Grant.

fs/namei.c +2 -1
@@ -1497,13 +1497,14 @@
 			lock_kernel();
 			error = dir->i_op->unlink(dir, dentry);
 			unlock_kernel();
+			if (!error)
+				d_delete(dentry);
 		}
 	}
 	double_up(&dir->i_zombie, &inode->i_zombie);
 	iput(inode);
 
 	if (!error) {
-		d_delete(dentry);
 		inode_dir_notify(dir, DN_DELETE);
 	}
 	return error;
#4 lore raw

Hi Grant,

Quoted 1 line · Grant Coady

On Tue, 4 Jul 2006 00:07:36 +0200, Willy Tarreau <[email protected]> wrote:

(...)

Quoted 6 lines
Quoted 6 lines

What I notice is that in 2.4.32, d_delete(dentry) was performed between down(&dir->i_zombie) and up(&dir->i_zombie), while now it's completely outside. I wonder if this can cause race conditions or not, but at least, I'm sure that we have changed the locking sequence, which might have some impact.

Quoted 7 lines
Quoted 4 lines

Do you think I'm searching in the wrong direction ? I worry a bit, because getting a deadlock after only one day, it's a bit early :-/

Assuming you mean something like the patch below? Doesn't cause any problems (yet, still testing) like eat files or segfault here as reported for -rc1 +/- various patches ;)

yes, exactly this. I don't know if it's correct and/or needed. In 2.6, the d_delete() is performed outside the lock. I'd like someone's advise on this one. Also, I'll look for an NFS client stress test to try to reproduce the problem, because I don't like it when problems like this only appear once a day. And playing with the VFS does not make me happy at all.

Quoted 2 lines

Cheers,
Grant.

Cheers,
Willy

Quoted 18 lines
fs/namei.c +2 -1
@@ -1497,13 +1497,14 @@
 			lock_kernel();
 			error = dir->i_op->unlink(dir, dentry);
 			unlock_kernel();
+			if (!error)
+				d_delete(dentry);
 		}
 	}
 	double_up(&dir->i_zombie, &inode->i_zombie);
 	iput(inode);
 
 	if (!error) {
-		d_delete(dentry);
 		inode_dir_notify(dir, DN_DELETE);
 	}
 	return error;
#5 lore raw
In reply to Grant Coady

Hi,

Quoted 80 lines · Grant Coady
Quoted 60 lines · Willy Tarreau
Quoted 2 lines · Marcelo Tosatti

Willy Tarreau:
Fix vfs_unlink/NFS NULL pointer dereference

Marcelo, I'm not sure this one is perfect yet. Today, while packaging a lot of files for our distro at work, I came up with a problem where deleting a file on NFS, and later simply accessing (read/write/create) a file on the NFS file system did block. However, I could kill all the offending processes. This was after a full day of mkdir/create/open/ unlink... (tens of thoudands of those), so it is not much reproduceable.

I could not unmount the NFS anymore, while other users had no problem. Rebooting the client solved the problem. I caught an RPC trace (attached), not sure if it can help. I must say that I'm also running Trond's NFS patches which I suspected first, but with which I never encountered a single problem for years.

The fact that the problem appeared during an rm -rf made me think about the vfs_unlink() patch. I went to read it again an I'm wondering if we have not inserted a new problem (please forgive my ignorance here) :

in 2.4.32, we had the following sequence :

       down(&dir->i_zombie);
       if (may_delete(dir, dentry, 0) != 0) return;
       lock_kernel();
       error = dir->i_op->unlink(dir, dentry);
       unlock_kernel();
       if (!error)
             d_delete(dentry);
       up(&dir->i_zombie);
       if (!error)
               inode_dir_notify(dir, DN_DELETE);

int 2.4.33-rc2, we have :

       if (may_delete(dir, dentry, 0) != 0) return;
       inode = dentry->d_inode;

       atomic_inc(&inode->i_count);
       double_down(&dir->i_zombie, &inode->i_zombie);

       lock_kernel();
       error = dir->i_op->unlink(dir, dentry);
       unlock_kernel();

       double_up(&dir->i_zombie, &inode->i_zombie);
       iput(inode);

       if (!error) {
               d_delete(dentry);
               inode_dir_notify(dir, DN_DELETE);
       }

What I notice is that in 2.4.32, d_delete(dentry) was performed between down(&dir->i_zombie) and up(&dir->i_zombie), while now it's completely outside. I wonder if this can cause race conditions or not, but at least, I'm sure that we have changed the locking sequence, which might have some impact.

Quoted 4 lines

Do you think I'm searching in the wrong direction ? I worry a bit, because getting a deadlock after only one day, it's a bit early :-/

Assuming you mean something like the patch below? Doesn't cause any problems (yet, still testing) like eat files or segfault here as reported for -rc1 +/- various patches ;)

Cheers,
Grant.

fs/namei.c +2 -0
@@ -1497,13 +1497,14 @@
 			lock_kernel();
 			error = dir->i_op->unlink(dir, dentry);
 			unlock_kernel();
+			if (!error)
+				d_delete(dentry);

fs/namei.c line 1510

Quoted 10 lines
fs/namei.c +0 -1
 		}
 	}
 	double_up(&dir->i_zombie, &inode->i_zombie);
 	iput(inode);
 
 	if (!error) {
-		d_delete(dentry);
 		inode_dir_notify(dir, DN_DELETE);
 	}
 	return error;

after a full day of stress-test of multiple parallel tar xUf, and ffsb at full CPU load, I could not reproduce the problem on the exact same kernel I first saw it on. So I think I had bad luck and the problem is not related to the vfs_unlink() patch, so unless anyone else reports a problem or tells us why it is right or wrong, it would seem reasonable to keep it as it is in -rc2.

Regards,
Willy

#6 lore raw
Quoted 88 lines · Willy Tarreau

Hi,

Quoted 85 lines · Grant Coady
Quoted 60 lines · Willy Tarreau
Quoted 2 lines · Marcelo Tosatti

Willy Tarreau:
Fix vfs_unlink/NFS NULL pointer dereference

Marcelo, I'm not sure this one is perfect yet. Today, while packaging a lot of files for our distro at work, I came up with a problem where deleting a file on NFS, and later simply accessing (read/write/create) a file on the NFS file system did block. However, I could kill all the offending processes. This was after a full day of mkdir/create/open/ unlink... (tens of thoudands of those), so it is not much reproduceable.

I could not unmount the NFS anymore, while other users had no problem. Rebooting the client solved the problem. I caught an RPC trace (attached), not sure if it can help. I must say that I'm also running Trond's NFS patches which I suspected first, but with which I never encountered a single problem for years.

The fact that the problem appeared during an rm -rf made me think about the vfs_unlink() patch. I went to read it again an I'm wondering if we have not inserted a new problem (please forgive my ignorance here) :

in 2.4.32, we had the following sequence :

       down(&dir->i_zombie);
       if (may_delete(dir, dentry, 0) != 0) return;
       lock_kernel();
       error = dir->i_op->unlink(dir, dentry);
       unlock_kernel();
       if (!error)
             d_delete(dentry);
       up(&dir->i_zombie);
       if (!error)
               inode_dir_notify(dir, DN_DELETE);

int 2.4.33-rc2, we have :

       if (may_delete(dir, dentry, 0) != 0) return;
       inode = dentry->d_inode;

       atomic_inc(&inode->i_count);
       double_down(&dir->i_zombie, &inode->i_zombie);

       lock_kernel();
       error = dir->i_op->unlink(dir, dentry);
       unlock_kernel();

       double_up(&dir->i_zombie, &inode->i_zombie);
       iput(inode);

       if (!error) {
               d_delete(dentry);
               inode_dir_notify(dir, DN_DELETE);
       }

What I notice is that in 2.4.32, d_delete(dentry) was performed between down(&dir->i_zombie) and up(&dir->i_zombie), while now it's completely outside. I wonder if this can cause race conditions or not, but at least, I'm sure that we have changed the locking sequence, which might have some impact.

Quoted 4 lines

Do you think I'm searching in the wrong direction ? I worry a bit, because getting a deadlock after only one day, it's a bit early :-/

Assuming you mean something like the patch below? Doesn't cause any problems (yet, still testing) like eat files or segfault here as reported for -rc1 +/- various patches ;)

Cheers,
Grant.

fs/namei.c +2 -0
@@ -1497,13 +1497,14 @@
 			lock_kernel();
 			error = dir->i_op->unlink(dir, dentry);
 			unlock_kernel();
+			if (!error)
+				d_delete(dentry);
 		}
 	}
 	double_up(&dir->i_zombie, &inode->i_zombie);
 	iput(inode);
 
Quoted 12 lines

fs/namei.c line 1510

Quoted 5 lines
fs/namei.c +0 -1
 	if (!error) {
-		d_delete(dentry);
 		inode_dir_notify(dir, DN_DELETE);
 	}
 	return error;

after a full day of stress-test of multiple parallel tar xUf, and ffsb at full CPU load, I could not reproduce the problem on the exact same kernel I first saw it on. So I think I had bad luck and the problem is not related to the vfs_unlink() patch, so unless anyone else reports a problem or tells us why it is right or wrong, it would seem reasonable to keep it as it is in -rc2.

Hi Willy,

Got this with unpatched -rc2, tosh is NFS server, niner is client:

grant@niner:/home/nfstest$ ls -l
total 228474
drwxr-xr-x  19 grant wheel       680 2006-03-20 16:53 linux-2.6.16/
-rw-r--r--   1 grant wheel 233953280 2006-07-05 18:27 linux-2.6.16.tar
drwxr-xr-x  19 grant wheel       680 2006-03-20 16:53 linux-2.6.16b/
grant@niner:/home/nfstest$ x=0; while [ ! $(diff -rq linux-2.6.16 linux-2.6.16b) ]; do ((x++)); echo "trial $x"; rm -rf linux-2.6.16b; mv linux-2.6.16 linux-2.6.16b; tar xf linux-2.6.16.tar; done
trial 1
...
trial 29
rm: cannot remove directory `linux-2.6.16b/drivers/cdrom': Directory not empty
-bash: [: too many arguments
grant@niner:/home/nfstest$ ls -l
total 228474
drwxr-xr-x  19 grant wheel       680 2006-03-20 16:53 linux-2.6.16/
-rw-r--r--   1 grant wheel 233953280 2006-07-05 18:27 linux-2.6.16.tar
drwxr-xr-x   4 grant wheel       104 2006-07-06 11:01 linux-2.6.16b/
grant@niner:/home/nfstest$ rm -rf linux-2.6.16b/

The 'rm -rf linux-2.6.16b' completed okay, a mystery?

This is with two slow (500MHz) boxen with -rc2.
Only idea I get from logs is during the test:

Jul  5 19:01:19 niner kernel: nfs: server tosh not responding, still trying
Jul  5 19:01:19 niner kernel: nfs: server tosh OK

... about one pair each 2 to 5 mins

Jul  6 11:16:08 niner kernel: nfs: server tosh not responding, still trying
Jul  6 11:16:08 niner kernel: nfs: server tosh OK
Jul  6 11:26:57 niner -- MARK --
Jul  6 11:46:57 niner -- MARK --

Other pair of boxen with patched -rc2 completed 146 trials overnight along with compiling 2.4 kernel over NFS as well since morning, 64 completed. No 'server not responding messages' logged.

I'll change the two running boxen to straight -rc2 and see if catch anything.

Grant.

#7 lore raw

Hi Grant,

Quoted 1 line · Grant Coady

On Wed, 5 Jul 2006 22:51:37 +0200, Willy Tarreau <[email protected]> wrote:

(...)

Quoted 29 lines
Quoted 6 lines

after a full day of stress-test of multiple parallel tar xUf, and ffsb at full CPU load, I could not reproduce the problem on the exact same kernel I first saw it on. So I think I had bad luck and the problem is not related to the vfs_unlink() patch, so unless anyone else reports a problem or tells us why it is right or wrong, it would seem reasonable to keep it as it is in -rc2.

Hi Willy,

Got this with unpatched -rc2, tosh is NFS server, niner is client:

grant@niner:/home/nfstest$ ls -l
total 228474
drwxr-xr-x  19 grant wheel       680 2006-03-20 16:53 linux-2.6.16/
-rw-r--r--   1 grant wheel 233953280 2006-07-05 18:27 linux-2.6.16.tar
drwxr-xr-x  19 grant wheel       680 2006-03-20 16:53 linux-2.6.16b/
grant@niner:/home/nfstest$ x=0; while [ ! $(diff -rq linux-2.6.16 linux-2.6.16b) ]; do ((x++)); echo "trial $x"; rm -rf linux-2.6.16b; mv linux-2.6.16 linux-2.6.16b; tar xf linux-2.6.16.tar; done
trial 1
...
trial 29
rm: cannot remove directory `linux-2.6.16b/drivers/cdrom': Directory not empty
-bash: [: too many arguments
grant@niner:/home/nfstest$ ls -l
total 228474
drwxr-xr-x  19 grant wheel       680 2006-03-20 16:53 linux-2.6.16/
-rw-r--r--   1 grant wheel 233953280 2006-07-05 18:27 linux-2.6.16.tar
drwxr-xr-x   4 grant wheel       104 2006-07-06 11:01 linux-2.6.16b/
grant@niner:/home/nfstest$ rm -rf linux-2.6.16b/
Quoted 1 line

The 'rm -rf linux-2.6.16b' completed okay, a mystery?

you might have had a '.nfs0000*' file inthe directory which prevented rmmod from working, but it was finally removed by the rm -rf.

Quoted 6 lines

This is with two slow (500MHz) boxen with -rc2.
Only idea I get from logs is during the test:

Jul  5 19:01:19 niner kernel: nfs: server tosh not responding, still trying
Jul  5 19:01:19 niner kernel: nfs: server tosh OK
Quoted 6 lines

... about one pair each 2 to 5 mins

Jul  6 11:16:08 niner kernel: nfs: server tosh not responding, still trying
Jul  6 11:16:08 niner kernel: nfs: server tosh OK
Jul  6 11:26:57 niner -- MARK --
Jul  6 11:46:57 niner -- MARK --

I get this if the server spends too much time writing data back to the disks. Doing this on the server fixed the problem for me :

# echo 50 25000 0 0 100 100 60 45 0 >/proc/sys/vm/bdflush
Quoted 3 lines

Other pair of boxen with patched -rc2 completed 146 trials overnight along with compiling 2.4 kernel over NFS as well since morning, 64 completed. No 'server not responding messages' logged.

Was it on the same server and while other clients saw the server disappear ?

Quoted 2 lines

I'll change the two running boxen to straight -rc2 and see if catch anything.

OK, similarly, it might be interesting to apply your patch to niner to see if the rmmod error happens again.

Quoted 1 line

Grant.

Thanks for your tests,
Willy