Skip to contentSkip to conversation

Deadlock on the mm->mmap_sem

PATCH New Manfred Spraul 7 participants 43 messages Sep 17, 2001 – Sep 20, 2001

Subsystems (21)
lore
Share & tools
lore
#0 lore raw
Quoted 4 lines

What happens is that proc_pid_read_maps grabs the mmap_sem as a reader, and *while it holds the lock*, does a copy_to_user. This can of course page-fault, and the handler will also grab the mmap_sem (if it is the same task).

Ok, that's a bug.
You must not call copy_to_user with the mmap semaphore acquired - linux semaphores are not recursive.

Quoted 2 lines

Any ideas how to fix this? Should proc_pid_read_maps just drop the lock before copy_to_user?

Yes, and preferable switch to multiline copies - a full page temporary buffer is allocated, transfering data on a line-by-line base is way too much overhead (and the current volatile_task is an ugly hack).

--
Manfred

#1 lore raw

[ David, Andrea - can you check this out? ]

In article <001701c13fc2$cda19a90$010411ac@local>,

Quoted 8 lines · Manfred Spraul
Quoted 4 lines

What happens is that proc_pid_read_maps grabs the mmap_sem as a reader, and *while it holds the lock*, does a copy_to_user. This can of course page-fault, and the handler will also grab the mmap_sem (if it is the same task).

Ok, that's a bug.
You must not call copy_to_user with the mmap semaphore acquired - linux semaphores are not recursive.

No, that's not the bug.

The mmap semaphore is a read-write semaphore, and it _is_ permissible to call "copy_to_user()" and friends while holding the read lock.

The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block new readers, exactly because of this situation.

The situation wrt read-write spinlocks is exactly the same, btw, except there we have "readers can have interrupts enabled even if interrupts also take read locks" instead of having user-level faults.

Why do we want to explicitly allow this behaviour wrt mmap_sem? Because some things are inherently racy without it (ie threaded processes that read or write the address space - coredumping, ptrace etc).

Linus

#2 lore raw
In reply to Manfred Spraul
Quoted 9 lines · Linus Torvalds

[ David, Andrea - can you check this out? ]

In article <001701c13fc2$cda19a90$010411ac@local>,

Quoted 5 lines · Manfred Spraul
Quoted 4 lines

What happens is that proc_pid_read_maps grabs the mmap_sem as a reader, and *while it holds the lock*, does a copy_to_user. This can of course page-fault, and the handler will also grab the mmap_sem (if it is the same task).

Quoted 5 lines
Quoted 3 lines

Ok, that's a bug.
You must not call copy_to_user with the mmap semaphore acquired - linux semaphores are not recursive.

No, that's not the bug.

agreed.

Quoted 6 lines

The mmap semaphore is a read-write semaphore, and it _is_ permissible to call "copy_to_user()" and friends while holding the read lock.

The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block new readers, exactly because of this situation.

Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to read_lock without clearing irqs). Thanks so much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first down_write (for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock). The fix is a few liner for my implementation, here it is:

lib/rwsem_spinlock.c renamed from lib/rwsem_spinlock.c.~1~ +3 -1
@@ -73,11 +73,13 @@
 
 void down_read(struct rw_semaphore *sem)
 {
+	int count;
 	CHECK_MAGIC(sem->__magic);
 
 	spin_lock(&sem->lock);
+	count = sem->count;
 	sem->count += RWSEM_READ_BIAS;
-	if (__builtin_expect(sem->count, 0) < 0)
+	if (__builtin_expect(count < 0 && !(count & RWSEM_READ_MASK), 0))
 		rwsem_down_failed(sem, RWSEM_READ_BLOCKING_BIAS);
 	spin_unlock(&sem->lock);
 }

it will be applied to next -aa. For the mainline semaphores I assume David will take care of that.

For the record, I'm using spinlock based rwsemphores. Last time I checked my asm semaphores I found a small race in up_write, I didn't checked if the mainlines semaphores were affected too but I just preferred to stay safe with the spinlock in the meantime (in the microbenchmark the spinlock based rwsems weren't that much slower [and my optimized version is much faster than the mainline spinlock based rwsem] so using asm it's not a noticeable improvement in the macro real life benchmarks and the robustness of the spinlock is quite unvaluable, even more now that allowed me to do a bugfix without panicing in doing those changes). I think I will return to the asm rwsem only after proofing my implementation with math or after writing an automated simulation that checks their correctness in all possible race combinations (assuming they're mutex and with a variable number of threads).

Andrea

#3 lore raw
In reply to Manfred Spraul

From: "Andrea Arcangeli" <[email protected]>

Quoted 1 line
Quoted 1 line

The mmap semaphore is a read-write semaphore, and it _is_

permissible to

Quoted 4 lines
Quoted 4 lines

call "copy_to_user()" and friends while holding the read lock.

The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block

new

Quoted 4 lines
Quoted 1 line

readers, exactly because of this situation.

Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to read_lock without clearing irqs). Thanks

so

Quoted 4 lines

much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first down_write (for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock).

i386 has a fair rwsemaphore, too - probably other archs must be modified as well.

Quoted 3 lines

The fix is a few liner for my
implementation, here it is:

Obivously your patch fixes the race, but we could starve down_write() if there are many page faults.
Which multithreaded apps rely on mmap for file io? innd, perhaps samba if mmap is enabled (I'm not sure what's the default and if samba is multithreaded).

If you compile a kernel for 80386, then i386 uses the generic semaphores.
Could someone with innd compile his kernel for i386, apply Andrea's patch and check that the performance doesn't break down?

IMHO modifying proc_pid_read_maps() is far simpler - I'm not aware of another recursive mmap_sem user.
--
Manfred

#4 lore raw
In reply to Manfred Spraul
Quoted 12 lines · Manfred Spraul

From: "Andrea Arcangeli" <[email protected]>

Quoted 1 line
Quoted 1 line

The mmap semaphore is a read-write semaphore, and it _is_

permissible to

Quoted 4 lines
Quoted 4 lines

call "copy_to_user()" and friends while holding the read lock.

The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block

new

Quoted 4 lines
Quoted 1 line

readers, exactly because of this situation.

Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to read_lock without clearing irqs). Thanks

Quoted 8 lines

so

Quoted 4 lines

much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first down_write (for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock).

i386 has a fair rwsemaphore, too - probably other archs must be modified as well.

yes, actually my patch was against the rwsem patch in -aa, and in -aa I'm using the generic semaphores for all archs in the tree so it fixes the race for all them. The mainline semaphores are slightly different.

Quoted 6 lines
Quoted 3 lines

The fix is a few liner for my
implementation, here it is:

Obivously your patch fixes the race, but we could starve down_write() if there are many page faults.

Yes.

Quoted 2 lines

IMHO modifying proc_pid_read_maps() is far simpler - I'm not aware of another recursive mmap_sem user.

if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks. I think another option would be to have reacursion allowed on the default read locks and then make a down_read_fair that will block at if there's a down_write under us. we can very cleanly implement this, the same can be done cleanly also for the spinlocks: read_lock_fair. One can even mix the read_lock/read_lock_fair or the down_read/down_read_fair together. For example assuming we use the recursive semaphore fix in proc_pid_read_maps the down_read over there could be converted to a down_read_fair (but that's just an exercise, if the page fault isn't fair it doesn't worth to have proc_pid_read_maps fair either).

Andrea

#5 lore raw
Quoted 7 lines
Quoted 3 lines

i386 has a fair rwsemaphore, too - probably other archs must be modified as well.

yes, actually my patch was against the rwsem patch in -aa, and in -aa I'm using the generic semaphores for all archs in the tree so it fixes the race for all them. The mainline semaphores are slightly different.

Wasn't there a problem with unfair rw-semaphores? I can't remember exactly now...

Quoted 15 lines
Quoted 2 lines

IMHO modifying proc_pid_read_maps() is far simpler - I'm not aware of another recursive mmap_sem user.

if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks. I think another option would be to have reacursion allowed on the default read locks and then make a down_read_fair that will block at if there's a down_write under us. we can very cleanly implement this, the same can be done cleanly also for the spinlocks: read_lock_fair. One can even mix the read_lock/read_lock_fair or the down_read/down_read_fair together. For example assuming we use the recursive semaphore fix in proc_pid_read_maps the down_read over there could be converted to a down_read_fair (but that's just an exercise, if the page fault isn't fair it doesn't worth to have proc_pid_read_maps fair either).

If this were to be done, I'd prefer to keep down_read() as being fair and add a down_read_unfair(). This'd have the least impact on the current behaviour, and I suspect we actually want fairness most of the time.

Of course, I'd personally prefer to avoid recursive semaphore situations where possible too... it sounds far too much like trouble waiting to happen, but we can't have everything.

David

#6 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Linus Torvalds

The mmap semaphore is a read-write semaphore, and it _is_ permissible to call "copy_to_user()" and friends while holding the read lock.

The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block new readers, exactly because of this situation.

Quoted 7 lines

The situation wrt read-write spinlocks is exactly the same, btw, except there we have "readers can have interrupts enabled even if interrupts also take read locks" instead of having user-level faults.

Why do we want to explicitly allow this behaviour wrt mmap_sem? Because some things are inherently racy without it (ie threaded processes that read or write the address space - coredumping, ptrace etc).

Hmmm... I don't think this is possible with XADD based semaphores as they stand (my version or Andrea's).

With the current XADD based stuff, you can't distinguish between one writer running and a queue of sleeping locks and one reader running and a queue of sleeping locks without counting the sleepers

	Sem(sleepers)	Proc 1	Proc 2	Proc 3	Proc 4	Proc 5
	========	======	======	======	======	======
	00000000(0)
			-->down_read()
			<--down_read()
	00000001(0)
				-->down_write()
				-->down_write_failed()
				[schedule]
	FFFF0001(1)
					-->down_write()
					-->down_write_failed()
					[schedule]
	FFFE0001(2)
						-->down_write()
						-->down_write_failed()
						[schedule]
	FFFD0001(3)
							-->down_read_unfair()
	FFFC0002(3)
							is the active proc
							R or W?

	Sem		Proc 1	Proc 2	Proc 3	Proc 4
	========	======	======	======	======
	00000000(0)
			-->down_write()
			<--down_write()
	FFFF0001(0)
				-->down_write()
				-->down_write_failed()
				[schedule]
	FFFE0001(1)
					-->down_write()
					-->down_write_failed()
					[schedule]
	FFFD0001(2)
						-->down_read_unfair()
	FFFC0002(2)
						is the active proc R or W?

In fact, it's worse than that: you can't tell the difference between two active readers and a queue of sleepers and one active writer, one failed read or write attempt as yet unprocessed, and a queue of sleepers

	Sem(sleepers)	Proc 1	Proc 2	Proc 3	Proc 4	Proc 5
	==============	======	======	======	======	======
	00000000(0)
			-->down_read()
			<--down_read()
	00000001(0)
				-->down_read()
				<--down_read()
	00000002(0)
					-->down_write()
	FFFF0003(0)
					-->down_write_failed()
					[schedule]
	FFFF0002(1)
						-->down_write()
	FFFE0003(1)
							-->down_read_unfair()
	FFFE0004(1)
							since the LSW>2 does
							this mean there are 2+
							readers active?

	Sem(sleepers)	Proc 1	Proc 2	Proc 3	Proc 4	Proc 5
	==============	======	======	======	======	======
	00000000(0)
			-->down_write()
			<--down_write()
	FFFF0001(0)
				-->down_write()
	FFFE0002(0)
				-->down_write_failed()
				[schedule]
	FFFE0001(1)
					-->down_read()
	FFFE0002(1)
						-->down_read()
	FFFE0003(1)
							-->down_read_unfair()
	FFFE0004(1)
							since the LSW>2 does
							this mean there are 2+
							readers active?

I think it might well be too hard to do unfair reads with the XADD based stuff. The problem is that you can't compensate for the effect on the counter of failed attempts to get read or write locks, even when you've got the semaphore spinlock (the queue length is of no help).

I think that this problem can only be solved by going to the spinlock version, and maintaining a flag to say what sort of lock is currently active.

David

#7 lore raw
In reply to Manfred Spraul
Quoted 1 line
Quoted 1 line

IMHO modifying proc_pid_read_maps() is far simpler - I'm not aware

of

Quoted 5 lines
Quoted 1 line

another recursive mmap_sem user.

if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks.

But shouldn't that change wait until 2.5? Especially since another huge mm change was just merged?
proc_pid_read_maps contains further bugs - afaics it it could skip lines if PAGE_SIZE > 4096 and a file path is nearly 4096 bytes long. I'll post a patch to proc_pid_read_maps - modifying the rw semaphore behaviour just asks for trouble.

--
Manfred

#8 lore raw
In reply to Manfred Spraul
Quoted 27 lines · Andrea Arcangeli
Quoted 12 lines · Manfred Spraul

From: "Andrea Arcangeli" <[email protected]>

Quoted 1 line
Quoted 1 line

The mmap semaphore is a read-write semaphore, and it _is_

permissible to

Quoted 4 lines
Quoted 4 lines

call "copy_to_user()" and friends while holding the read lock.

The bug appears to be in the implementation of the write semaphore - down_write() doesn't undestand that blocked writes must not block

new

Quoted 4 lines
Quoted 1 line

readers, exactly because of this situation.

Exactly, same reason for which we need the same property from the rw spinlocks (to be allowed to read_lock without clearing irqs). Thanks

Quoted 8 lines

so

Quoted 4 lines

much for reminding me about this! Unfortunately my rwsemaphores are blocking readers at the first down_write (for the better fairness property issuse, but I obviously forgotten that doing so I would introduce such a deadlock).

i386 has a fair rwsemaphore, too - probably other archs must be modified as well.

yes, actually my patch was against the rwsem patch in -aa, and in -aa I'm using the generic semaphores for all archs in the tree so it fixes the race for all them. The mainline semaphores are slightly different.

Quoted 12 lines

if that's the very only place that could be a viable option but OTOH I like to be allowed to use recursion on the read locks as with the spinlocks. I think another option would be to have reacursion allowed on the default read locks and then make a down_read_fair that will block at if there's a down_write under us. we can very cleanly implement this, the same can be done cleanly also for the spinlocks: read_lock_fair. One can even mix the read_lock/read_lock_fair or the down_read/down_read_fair together. For example assuming we use the recursive semaphore fix in proc_pid_read_maps the down_read over there could be converted to a down_read_fair (but that's just an exercise, if the page fault isn't fair it doesn't worth to have proc_pid_read_maps fair either).

Be careful; If another user can grab your semaphore for read for a short time (eg for "top" or similar usage), he can construct several threads that
do this in a busy loop; the end result is that this evil user is capable of blocking out writers FOREVER if semaphores are unfair; nice DoS....

#9 lore raw
In reply to Manfred Spraul

Attached is a rewritten proc_pid_read_maps. Patch against 2.4.9-ac9, but should apply against 2.4.9 as well.

As a side-effect, it's more efficient since it tries to return multiple lines in each call, without the volatile_task hack.

Code that's older than 2.4.3 can't cause recursions - that was before the rw-mmap_sem change.

* ptrace doesn't cause any recursions, it always copies into temporary kernel buffers.

* the multithreaded coredump patch added down_read() around binfmt->core_dump(). I'll think about the simplest way to fix that.

--
Manfred

fs/proc/array.c +105 -99
@@ -537,136 +537,142 @@
 #define MAPS_LINE_FORMAT8	  "%016lx-%016lx %s %016lx %s %lu"
 #define MAPS_LINE_MAX8	73 /* sum of 16  1  16  1 4 1 16 1 5 1 10 1 */
 
-#define MAPS_LINE_MAX	MAPS_LINE_MAX8
+#define MAPS_LINE_FORMAT	(sizeof(void*) == 4 ? MAPS_LINE_FORMAT4 : MAPS_LINE_FORMAT8)
+#define MAPS_LINE_MAX	(sizeof(void*) == 4 ?  MAPS_LINE_MAX4 :  MAPS_LINE_MAX8)
 
+int proc_pid_maps_get_line (char *buf, struct vm_area_struct *map)
+{
+	/* produce the next line */
+	char *line;
+	char str[5];
+	int flags;
+	kdev_t dev;
+	unsigned long ino;
+	int len;
+
+	flags = map->vm_flags;
+
+	str[0] = flags & VM_READ ? 'r' : '-';
+	str[1] = flags & VM_WRITE ? 'w' : '-';
+	str[2] = flags & VM_EXEC ? 'x' : '-';
+	str[3] = flags & VM_MAYSHARE ? 's' : 'p';
+	str[4] = 0;
+
+	dev = 0;
+	ino = 0;
+	if (map->vm_file != NULL) {
+		dev = map->vm_file->f_dentry->d_inode->i_dev;
+		ino = map->vm_file->f_dentry->d_inode->i_ino;
+		line = d_path(map->vm_file->f_dentry,
+			      map->vm_file->f_vfsmnt,
+			      buf, PAGE_SIZE);
+		buf[PAGE_SIZE-1] = '\n';
+		line -= MAPS_LINE_MAX;
+		if(line < buf)
+			line = buf;
+	} else
+		line = buf;
+
+	len = sprintf(line,
+		      MAPS_LINE_FORMAT,
+		      map->vm_start, map->vm_end, str, map->vm_pgoff << PAGE_SHIFT,
+		      kdevname(dev), ino);
+
+	if(map->vm_file) {
+		int i;
+		for(i = len; i < MAPS_LINE_MAX; i++)
+			line[i] = ' ';
+		len = buf + PAGE_SIZE - line;
+		memmove(buf, line, len);
+	} else
+		line[len++] = '\n';
+	return len;
+}
 
 ssize_t proc_pid_read_maps (struct task_struct *task, struct file * file, char * buf,
 			  size_t count, loff_t *ppos)
 {
 	struct mm_struct *mm;
 	struct vm_area_struct * map, * next;
-	char * destptr = buf, * buffer;
-	loff_t lineno;
-	ssize_t column, i;
-	int volatile_task;
+	char *tmp, *kbuf;
 	long retval;
+	int off, lineno, loff;
 
+	/* reject calls with out of range parameters immediately */
+	retval = 0;
+	if (*ppos > LONG_MAX)
+		goto out;
+	if (count == 0)
+		goto out;
+	off = (long)*ppos;
 	/*
 	 * We might sleep getting the page, so get it first.
 	 */
 	retval = -ENOMEM;
-	buffer = (char*)__get_free_page(GFP_KERNEL);
-	if (!buffer)
+	kbuf = (char*)__get_free_page(GFP_KERNEL);
+	if (!kbuf)
 		goto out;
 
-	if (count == 0)
-		goto getlen_out;
+	tmp = (char*)__get_free_page(GFP_KERNEL);
+	if (!tmp)
+		goto out_free1;
+
 	task_lock(task);
 	mm = task->mm;
 	if (mm)
 		atomic_inc(&mm->mm_users);
 	task_unlock(task);
+	retval = 0;
 	if (!mm)
-		goto getlen_out;
-
-	/* Check whether the mmaps could change if we sleep */
-	volatile_task = (task != current || atomic_read(&mm->mm_users) > 2);
-
-	/* decode f_pos */
-	lineno = *ppos >> MAPS_LINE_SHIFT;
-	column = *ppos & (MAPS_LINE_LENGTH-1);
+		goto out_free2;
 
-	/* quickly go to line lineno */
 	down_read(&mm->mmap_sem);
-	for (map = mm->mmap, i = 0; map && (i < lineno); map = map->vm_next, i++)
-		continue;
-
-	for ( ; map ; map = next ) {
-		/* produce the next line */
-		char *line;
-		char str[5], *cp = str;
-		int flags;
-		kdev_t dev;
-		unsigned long ino;
-		int maxlen = (sizeof(void*) == 4) ?
-			MAPS_LINE_MAX4 :  MAPS_LINE_MAX8;
+	map = mm->mmap;
+	lineno = 0;
+	loff = 0;
+	if (count > PAGE_SIZE)
+		count = PAGE_SIZE;
+	while (map) {
 		int len;
-
-		/*
-		 * Get the next vma now (but it won't be used if we sleep).
-		 */
-		next = map->vm_next;
-		flags = map->vm_flags;
-
-		*cp++ = flags & VM_READ ? 'r' : '-';
-		*cp++ = flags & VM_WRITE ? 'w' : '-';
-		*cp++ = flags & VM_EXEC ? 'x' : '-';
-		*cp++ = flags & VM_MAYSHARE ? 's' : 'p';
-		*cp++ = 0;
-
-		dev = 0;
-		ino = 0;
-		if (map->vm_file != NULL) {
-			dev = map->vm_file->f_dentry->d_inode->i_dev;
-			ino = map->vm_file->f_dentry->d_inode->i_ino;
-			line = d_path(map->vm_file->f_dentry,
-				      map->vm_file->f_vfsmnt,
-				      buffer, PAGE_SIZE);
-			buffer[PAGE_SIZE-1] = '\n';
-			line -= maxlen;
-			if(line < buffer)
-				line = buffer;
-		} else
-			line = buffer;
-
-		len = sprintf(line,
-			      sizeof(void*) == 4 ? MAPS_LINE_FORMAT4 : MAPS_LINE_FORMAT8,
-			      map->vm_start, map->vm_end, str, map->vm_pgoff << PAGE_SHIFT,
-			      kdevname(dev), ino);
-
-		if(map->vm_file) {
-			for(i = len; i < maxlen; i++)
-				line[i] = ' ';
-			len = buffer + PAGE_SIZE - line;
-		} else
-			line[len++] = '\n';
-		if (column >= len) {
-			column = 0; /* continue with next line at column 0 */
-			lineno++;
-			continue; /* we haven't slept */
+		if (off > MAPS_LINE_LENGTH) {
+			off -= MAPS_LINE_LENGTH;
+			goto next;
 		}
-
-		i = len-column;
-		if (i > count)
-			i = count;
-		copy_to_user(destptr, line+column, i); /* may have slept */
-		destptr += i;
-		count   -= i;
-		column  += i;
-		if (column >= len) {
-			column = 0; /* next time: next line at column 0 */
-			lineno++;
+		len = proc_pid_maps_get_line(tmp, map);
+		len -= off;
+		if (len > 0) {
+			if (retval+len > count) {
+				/* only partial line transfer possible */
+				len = count - retval;
+				/* save the offset where the next read
+				 * must start */
+				loff = len+off;
+			}
+			memcpy(kbuf+retval, tmp+off, len);
+			retval += len;
 		}
-
-		/* done? */
-		if (count == 0)
-			break;
-
-		/* By writing to user space, we might have slept.
-		 * Stop the loop, to avoid a race condition.
-		 */
-		if (volatile_task)
+		off = 0;
+next:
+		if (!loff)
+			lineno++;
+		if (retval >= count)
 			break;
+		if (loff) BUG();
+		map = map->vm_next;
+	}
+	if (retval > count) BUG();
+	if (copy_to_user(buf, kbuf, retval)) {
+		retval = -EFAULT;
+	} else {
+		*ppos = (lineno << MAPS_LINE_SHIFT) + loff;
 	}
 	up_read(&mm->mmap_sem);
-
-	/* encode f_pos */
-	*ppos = (lineno << MAPS_LINE_SHIFT) + column;
 	mmput(mm);
 
-getlen_out:
-	retval = destptr - buf;
-	free_page((unsigned long)buffer);
+out_free2:
+	free_page((unsigned long)tmp);
+out_free1:
+	free_page((unsigned long)kbuf);
 out:
 	return retval;
 }
#10 lore raw

Okay preliminary as-yet-untested patch to cure coredumping of the need to hold the mm semaphore:

- kernel/fork.c: function to partially copy an mm_struct and attach it to the task_struct in place of the old.

- include/linux/mm.h: declaration for above function

- fs/exec.c: have do_coredump() call this function and not get the read lock around the binfmt coredumper

David

fs/exec.c +8 -2
diff -uNr linux-2.4.10-pre11/fs/exec.c linux-rwsem/fs/exec.c
@@ -947,6 +947,14 @@
 	if (current->rlim[RLIMIT_CORE].rlim_cur < binfmt->min_coredump)
 		goto fail;
 
+	/* make sure the attached VM has a single ref (this process) to make
+	 * sure only do_exit() will change the VMA list, so we don't have to
+	 * lock the mm->sem around the binfmt coredumper
+	 */
+	retval = copy_mm_for_coredump(current);
+	if (retval<0)
+		goto fail;
+
 	memcpy(corename,"core.", 5);
 	corename[4] = '\0';
  	if (core_uses_pid || atomic_read(&current->mm->mm_users) != 1)
@@ -969,9 +977,7 @@
 	if (do_truncate(file->f_dentry, 0) != 0)
 		goto close_fail;
 
-	down_read(&current->mm->mmap_sem);
 	retval = binfmt->core_dump(signr, regs, file);
-	up_read(&current->mm->mmap_sem);
 
 close_fail:
 	filp_close(file, NULL);
include/linux/mm.h +1 -0
diff -uNr linux-2.4.10-pre11/include/linux/mm.h linux-rwsem/include/linux/mm.h
@@ -615,6 +615,7 @@
 }
 
 extern struct vm_area_struct *find_extend_vma(struct mm_struct *mm, unsigned long addr);
+extern struct mm_struct *copy_mm_for_coredump(struct task_struct *tsk);
 
 #endif /* __KERNEL__ */
 
kernel/fork.c +49 -0

55 lines not on this page · raw message

#11 lore raw
Quoted 2 lines

Okay preliminary as-yet-untested patch to cure coredumping of the need> to hold the mm semaphore:

If you want codd for this its in the older -ac tree. Linus decided it wasnt justified so it went out

#14 lore raw
In reply to David Howells

Okay tested patch to cure coredumping of the need to hold the mm semaphore:

- kernel/fork.c: function to partially copy an mm_struct and attach it to the task_struct in place of the old.

- include/linux/mm.h: declaration for above function

- fs/exec.c: have do_coredump() call this function and not get the read lock around the binfmt coredumper

It works, and you can core dump without oopsing.

David

fs/exec.c +8 -2
diff -uNr -x TAGS linux-2.4.10-pre11/fs/exec.c linux-rwsem/fs/exec.c
@@ -947,6 +947,14 @@
 	if (current->rlim[RLIMIT_CORE].rlim_cur < binfmt->min_coredump)
 		goto fail;
 
+	/* make sure the attached VM has a single ref (this process) to make
+	 * sure only do_exit() will change the VMA list, so we don't have to
+	 * lock the mm->sem around the binfmt coredumper
+	 */
+	retval = copy_mm_for_coredump(current);
+	if (retval<0)
+		goto fail;
+
 	memcpy(corename,"core.", 5);
 	corename[4] = '\0';
  	if (core_uses_pid || atomic_read(&current->mm->mm_users) != 1)
@@ -969,9 +977,7 @@
 	if (do_truncate(file->f_dentry, 0) != 0)
 		goto close_fail;
 
-	down_read(&current->mm->mmap_sem);
 	retval = binfmt->core_dump(signr, regs, file);
-	up_read(&current->mm->mmap_sem);
 
 close_fail:
 	filp_close(file, NULL);
include/linux/mm.h +1 -0
diff -uNr -x TAGS linux-2.4.10-pre11/include/linux/mm.h linux-rwsem/include/linux/mm.h
@@ -615,6 +615,7 @@
 }
 
 extern struct vm_area_struct *find_extend_vma(struct mm_struct *mm, unsigned long addr);
+extern int copy_mm_for_coredump(struct task_struct *tsk);
 
 #endif /* __KERNEL__ */
 
kernel/fork.c +49 -0

55 lines not on this page · raw message

#12 lore raw
In reply to Alan Cox
Quoted 6 lines
Quoted 3 lines

Okay preliminary as-yet-untested patch to cure coredumping of the need> to hold the mm semaphore:

If you want codd for this its in the older -ac tree. Linus decided it wasnt justified so it went out

Arjan said there was such a beast in the -ac stuff, but I guess this explains why I couldn't find it... Do you have any idea which -ac's?

Oh well, I'd already done my patch anyway. Does it look okay to you.

David

#13 lore raw
Quoted 5 lines
Quoted 2 lines

If you want codd for this its in the older -ac tree. Linus decided it wasnt justified so it went out

Arjan said there was such a beast in the -ac stuff, but I guess this explains why I couldn't find it... Do you have any idea which -ac's?

It went in with the threaded core dump patch

#15 lore raw
In reply to David Howells
Quoted 6 lines · David Howells

Okay preliminary as-yet-untested patch to cure coredumping of the need to hold the mm semaphore:

- kernel/fork.c: function to partially copy an mm_struct and attach it to the task_struct in place of the old.

Oh, please no.

If the choice is between a hack to do strange and incomprehensible things for a special case, and just making the semaphores do the same thing rw-spinlocks do and make the problem go away naturally, Ill take #2 any day. The patches already exist, after all.

Linus

#16 lore raw

Looking through the do_page_fault(), I noticed there's a race in expand stack because expand_stack() expects the caller to have the mm-sem write-locked.

I've attached a patch that might fix it appropriately. Alternatively, it may be worth applying Andrea's 00_silent-stack-overflow-10 patch which fixes this and something else too.

David

include/linux/mm.h +5 -2
diff -uNr linux-2.4.10-pre12/include/linux/mm.h linux-rwsem/include/linux/mm.h
@@ -586,11 +586,11 @@
 	 * before relocating the vma range ourself.
 	 */
 	address &= PAGE_MASK;
+	spin_lock(&vma->vm_mm->page_table_lock);
 	grow = (vma->vm_start - address) >> PAGE_SHIFT;
 	if (vma->vm_end - address > current->rlim[RLIMIT_STACK].rlim_cur ||
 	    ((vma->vm_mm->total_vm + grow) << PAGE_SHIFT) > current->rlim[RLIMIT_AS].rlim_cur)
-		return -ENOMEM;
-	spin_lock(&vma->vm_mm->page_table_lock);
+		goto nomem;
 	vma->vm_start = address;
 	vma->vm_pgoff -= grow;
 	vma->vm_mm->total_vm += grow;
@@ -598,6 +598,9 @@
 		vma->vm_mm->locked_vm += grow;
 	spin_unlock(&vma->vm_mm->page_table_lock);
 	return 0;
+ nomem:
+	spin_unlock(&vma->vm_mm->page_table_lock);
+	return -ENOMEM;
 }
 
 /* Look up the first VMA which satisfies  addr < vm_end,  NULL if none. */
#17 lore raw
Quoted 7 lines · David Howells

Looking through the do_page_fault(), I noticed there's a race in expand stack because expand_stack() expects the caller to have the mm-sem write-locked.

I've attached a patch that might fix it appropriately. Alternatively, it may be worth applying Andrea's 00_silent-stack-overflow-10 patch which fixes this and something else too.

Yep, it's here:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre11aa1/00_silent-stack-overflow-10

I also added the documentation on the locking on top of expand_stack.

My patch also enforced a gap of one page (sysctl configurable in with page granularity) between a growsdown vma and its previous vma, so that we can more easily trap stack overflows on the heap. (such part isn't related to the race fix but it was controversial but since it's quite useful too I didn't splitted it out :)

Andrea

#18 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · David Howells

Okay preliminary as-yet-untested patch to cure coredumping of the need to hold the mm semaphore:

- kernel/fork.c: function to partially copy an mm_struct and attach it to the task_struct in place of the old.

Quoted 6 lines

Oh, please no.

If the choice is between a hack to do strange and incomprehensible things for a special case, and just making the semaphores do the same thing rw-spinlocks do and make the problem go away naturally, Ill take #2 any day. The patches already exist, after all.

But surely giving rw-semaphores this behaviour is even worse... It introduces the possibility of livelock, and so DoS attacks, and it affects more than just access to the mm_struct.

Also comparing them to rw-spinlocks isn't really fair IMHO since they have different restrictions. Things inside spinlocks aren't allowed to sleep, and mustn't incur pagefaults.

I also don't think the hack is that bad. All it's doing is taking a copy of the process's VM decription so that it knows that nobody is going to modify it whilst a coredump is in progress. Furthermore, it _only_ affects the coredump path, and the coredump path is just about the last thing on the agenda for a dying process.

However, if you don't like that, how about just changing the lock on mm_struct to a special mm_struct-only type lock that has a recursive lock operation for use by the pagefault handler (and _only_ the pagefault handler)? I've attached a patch to do just that. This introduces five operations:

- mm_lock_shared() - get shared lock fairly
- mm_lock_shared_recursive() - get shared lock unfairly
- mm_unlock_shared() - release shared lock
- mm_lock_exclusive() - get exclusive lock
- mm_unlock_exclusive() - release exclusive lock

David

arch/alpha/kernel/osf_sys.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/alpha/kernel/osf_sys.c linux-mmsem/arch/alpha/kernel/osf_sys.c
@@ -241,9 +241,9 @@
 			goto out;
 	}
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	ret = do_mmap(file, addr, len, prot, flags, off);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if (file)
 		fput(file);
 out:
arch/alpha/mm/fault.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/alpha/mm/fault.c linux-mmsem/arch/alpha/mm/fault.c
@@ -113,7 +113,7 @@
 		goto vmalloc_fault;
 #endif
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -147,7 +147,7 @@
 	 * the fault.
 	 */
 	fault = handle_mm_fault(mm, vma, address, cause > 0);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	if (fault < 0)
 		goto out_of_memory;
@@ -161,7 +161,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	if (user_mode(regs)) {
 		force_sig(SIGSEGV, current);
@@ -198,7 +198,7 @@
 	if (current->pid == 1) {
 		current->policy |= SCHED_YIELD;
 		schedule();
-		down_read(&mm->mmap_sem);
+		mm_lock_shared_recursive(mm);
 		goto survive;
 	}
 	printk(KERN_ALERT "VM: killing process %s(%d)\n",
arch/arm/kernel/sys_arm.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/arm/kernel/sys_arm.c linux-mmsem/arch/arm/kernel/sys_arm.c
@@ -74,9 +74,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
@@ -125,9 +125,9 @@
 	    vectors_base() == 0)
 		goto out;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	ret = do_mremap(addr, old_len, new_len, flags, new_addr);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 out:
 	return ret;
arch/arm/mm/fault-common.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/arm/mm/fault-common.c linux-mmsem/arch/arm/mm/fault-common.c
@@ -251,9 +251,9 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	fault = __do_page_fault(mm, addr, error_code, tsk);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Handle the "normal" case first
arch/cris/kernel/sys_cris.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/cris/kernel/sys_cris.c linux-mmsem/arch/cris/kernel/sys_cris.c
@@ -59,9 +59,9 @@
                         goto out;
         }
 
-        down_write(&current->mm->mmap_sem);
+        mm_lock_exclusive(current->mm);
         error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-        up_write(&current->mm->mmap_sem);
+        mm_unlock_exclusive(current->mm);
 
         if (file)
                 fput(file);
arch/cris/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/cris/mm/fault.c linux-mmsem/arch/cris/mm/fault.c
@@ -266,7 +266,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -324,7 +324,7 @@
                 goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 	
 	/*
@@ -334,7 +334,7 @@
 
  bad_area:
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
  bad_area_nosemaphore:
 	DPG(show_registers(regs));
@@ -397,14 +397,14 @@
 	 */
 
  out_of_memory:
-        up_read(&mm->mmap_sem);
+        mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if(user_mode(regs))
 		do_exit(SIGKILL);
 	goto no_context;
 
  do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
          * Send a sigbus, regardless of whether we were in kernel
arch/i386/kernel/ldt.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/i386/kernel/ldt.c linux-mmsem/arch/i386/kernel/ldt.c
@@ -73,7 +73,7 @@
 	 * the GDT index of the LDT is allocated dynamically, and is
 	 * limited by MAX_LDT_DESCRIPTORS.
 	 */
-	down_write(&mm->mmap_sem);
+	mm_lock_exclusive(mm);
 	if (!mm->context.segments) {
 		void * segments = vmalloc(LDT_ENTRIES*LDT_ENTRY_SIZE);
 		error = -ENOMEM;
@@ -124,7 +124,7 @@
 	error = 0;
 
 out_unlock:
-	up_write(&mm->mmap_sem);
+	mm_unlock_exclusive(mm);
 out:
 	return error;
 }
arch/i386/kernel/sys_i386.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/i386/kernel/sys_i386.c linux-mmsem/arch/i386/kernel/sys_i386.c
@@ -55,9 +55,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/i386/mm/fault.c +6 -6
diff -uNr linux-2.4.10-pre12/arch/i386/mm/fault.c linux-mmsem/arch/i386/mm/fault.c
@@ -191,7 +191,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 
 	vma = find_vma(mm, address);
 	if (!vma)
@@ -265,7 +265,7 @@
 		if (bit < 32)
 			tsk->thread.screen_bitmap |= 1 << bit;
 	}
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
 /*
@@ -273,7 +273,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/* User mode accesses just cause a SIGSEGV */
 	if (error_code & 4) {
@@ -341,11 +341,11 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	if (tsk->pid == 1) {
 		tsk->policy |= SCHED_YIELD;
 		schedule();
-		down_read(&mm->mmap_sem);
+		mm_lock_shared_recursive(mm);
 		goto survive;
 	}
 	printk("VM: killing process %s\n", tsk->comm);
@@ -354,7 +354,7 @@
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/ia64/ia32/sys_ia32.c +8 -8
diff -uNr linux-2.4.10-pre12/arch/ia64/ia32/sys_ia32.c linux-mmsem/arch/ia64/ia32/sys_ia32.c
@@ -245,9 +245,9 @@
 		}
 		__copy_user(back, (char *)addr + len, PAGE_SIZE - ((addr + len) & ~PAGE_MASK));
 	}
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	r = do_mmap(0, baddr, len + (addr - baddr), prot, flags | MAP_ANONYMOUS, 0);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if (r < 0)
 		return(r);
 	if (addr == 0)
@@ -291,9 +291,9 @@
 		poff = offset & PAGE_MASK;
 		len += offset - poff;
 
-		down_write(&current->mm->mmap_sem);
+		mm_lock_exclusive(current->mm);
 		error = do_mmap_pgoff(file, addr, len, prot, flags, poff >> PAGE_SHIFT);
-		up_write(&current->mm->mmap_sem);
+		mm_unlock_exclusive(current->mm);
 
 		if (!IS_ERR((void *) error))
 			error += offset - poff;
@@ -338,9 +338,9 @@
 	if ((a.offset & ~PAGE_MASK) != 0)
 		return -EINVAL;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	retval = do_mmap_pgoff(file, a.addr, a.len, a.prot, a.flags, a.offset >> PAGE_SHIFT);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 #else
 	retval = ia32_do_mmap(file, a.addr, a.len, a.prot, a.flags, a.fd, a.offset);
 #endif
@@ -2605,11 +2605,11 @@
 		return(-EFAULT);
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	addr = do_mmap_pgoff(file, IA32_IOBASE,
 			     IOLEN, PROT_READ|PROT_WRITE, MAP_SHARED,
 			     (ia64_iobase & ~PAGE_OFFSET) >> PAGE_SHIFT);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (addr >= 0) {
 		ia64_set_kr(IA64_KR_IO_BASE, addr);
arch/ia64/kernel/sys_ia64.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/ia64/kernel/sys_ia64.c linux-mmsem/arch/ia64/kernel/sys_ia64.c
@@ -106,7 +106,7 @@
 	 * check and the clearing of r8.  However, we can't call sys_brk() because we need
 	 * to acquire the mmap_sem before we can do the test...
 	 */
-	down_write(&mm->mmap_sem);
+	mm_lock_exclusive(mm);
 
 	if (brk < mm->end_code)
 		goto out;
@@ -146,7 +146,7 @@
 	mm->brk = brk;
 out:
 	retval = mm->brk;
-	up_write(&mm->mmap_sem);
+	mm_unlock_exclusive(mm);
 	regs->r8 = 0;		/* ensure large retval isn't mistaken as error code */
 	return retval;
 }
@@ -205,9 +205,9 @@
 	if (rgn_index(addr) != rgn_index(addr + len))
 		return -EINVAL;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	addr = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/ia64/mm/fault.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/ia64/mm/fault.c linux-mmsem/arch/ia64/mm/fault.c
@@ -60,7 +60,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 
 	vma = find_vma_prev(mm, address, &prev_vma);
 	if (!vma)
@@ -112,7 +112,7 @@
 	      default:
 		goto out_of_memory;
 	}
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
   check_expansion:
@@ -135,7 +135,7 @@
 	goto good_area;
 
   bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	if (isr & IA64_ISR_SP) {
 		/*
 		 * This fault was due to a speculative load set the "ed" bit in the psr to
@@ -184,7 +184,7 @@
 	return;
 
   out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", current->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
arch/m68k/kernel/sys_m68k.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/m68k/kernel/sys_m68k.c linux-mmsem/arch/m68k/kernel/sys_m68k.c
@@ -59,9 +59,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
@@ -146,9 +146,9 @@
 	}
 	a.flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, a.addr, a.len, a.prot, a.flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if (file)
 		fput(file);
 out:
arch/m68k/mm/fault.c +3 -3
diff -uNr linux-2.4.10-pre12/arch/m68k/mm/fault.c linux-mmsem/arch/m68k/mm/fault.c
@@ -101,7 +101,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 
 	vma = find_vma(mm, address);
 	if (!vma)
@@ -168,7 +168,7 @@
 	#warning should be obsolete now...
 	if (CPU_IS_040_OR_060)
 		flush_tlb_page(vma, address);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return 0;
 
 /*
@@ -203,6 +203,6 @@
 	current->thread.faddr = address;
 
 send_sig:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return send_fault_sig(regs);
 }
arch/mips/kernel/irixelf.c +10 -10
diff -uNr linux-2.4.10-pre12/arch/mips/kernel/irixelf.c linux-mmsem/arch/mips/kernel/irixelf.c
@@ -314,12 +314,12 @@
 		   (unsigned long) elf_prot, (unsigned long) elf_type,
 		   (unsigned long) (eppnt->p_offset & 0xfffff000));
 #endif
-	    down_write(&current->mm->mmap_sem);
+	    mm_lock_exclusive(current->mm);
 	    error = do_mmap(interpreter, vaddr,
 			    eppnt->p_filesz + (eppnt->p_vaddr & 0xfff),
 			    elf_prot, elf_type,
 			    eppnt->p_offset & 0xfffff000);
-	    up_write(&current->mm->mmap_sem);
+	    mm_unlock_exclusive(current->mm);
 
 	    if(error < 0 && error > -1024) {
 		    printk("Aieee IRIX interp mmap error=%d\n", error);
@@ -498,12 +498,12 @@
 		prot  = (epp->p_flags & PF_R) ? PROT_READ : 0;
 		prot |= (epp->p_flags & PF_W) ? PROT_WRITE : 0;
 		prot |= (epp->p_flags & PF_X) ? PROT_EXEC : 0;
-	        down_write(&current->mm->mmap_sem);
+	        mm_lock_exclusive(current->mm);
 		(void) do_mmap(fp, (epp->p_vaddr & 0xfffff000),
 			       (epp->p_filesz + (epp->p_vaddr & 0xfff)),
 			       prot, EXEC_MAP_FLAGS,
 			       (epp->p_offset & 0xfffff000));
-	        up_write(&current->mm->mmap_sem);
+	        mm_unlock_exclusive(current->mm);
 
 		/* Fixup location tracking vars. */
 		if((epp->p_vaddr & 0xfffff000) < *estack)
@@ -762,10 +762,10 @@
 	 * Since we do not have the power to recompile these, we
 	 * emulate the SVr4 behavior.  Sigh.
 	 */
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	(void) do_mmap(NULL, 0, 4096, PROT_READ | PROT_EXEC,
 		       MAP_FIXED | MAP_PRIVATE, 0);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 #endif
 
 	start_thread(regs, elf_entry, bprm->p);
@@ -837,14 +837,14 @@
 	while(elf_phdata->p_type != PT_LOAD) elf_phdata++;
 	
 	/* Now use mmap to map the library into memory. */
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap(file,
 			elf_phdata->p_vaddr & 0xfffff000,
 			elf_phdata->p_filesz + (elf_phdata->p_vaddr & 0xfff),
 			PROT_READ | PROT_WRITE | PROT_EXEC,
 			MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE,
 			elf_phdata->p_offset & 0xfffff000);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	k = elf_phdata->p_vaddr + elf_phdata->p_filesz;
 	if (k > elf_bss) elf_bss = k;
@@ -916,12 +916,12 @@
 		prot  = (hp->p_flags & PF_R) ? PROT_READ : 0;
 		prot |= (hp->p_flags & PF_W) ? PROT_WRITE : 0;
 		prot |= (hp->p_flags & PF_X) ? PROT_EXEC : 0;
-		down_write(&current->mm->mmap_sem);
+		mm_lock_exclusive(current->mm);
 		retval = do_mmap(filp, (hp->p_vaddr & 0xfffff000),
 				 (hp->p_filesz + (hp->p_vaddr & 0xfff)),
 				 prot, (MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE),
 				 (hp->p_offset & 0xfffff000));
-		up_write(&current->mm->mmap_sem);
+		mm_unlock_exclusive(current->mm);
 
 		if(retval != (hp->p_vaddr & 0xfffff000)) {
 			printk("irix_mapelf: do_mmap fails with %d!\n", retval);
arch/mips/kernel/syscall.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/mips/kernel/syscall.c linux-mmsem/arch/mips/kernel/syscall.c
@@ -69,9 +69,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/mips/kernel/sysirix.c +8 -8
diff -uNr linux-2.4.10-pre12/arch/mips/kernel/sysirix.c linux-mmsem/arch/mips/kernel/sysirix.c
@@ -471,7 +471,7 @@
 		if (retval)
 			return retval;
 
-		down_read(&mm->mmap_sem);
+		mm_lock_shared(mm);
 		pgdp = pgd_offset(mm, addr);
 		pmdp = pmd_offset(pgdp, addr);
 		ptep = pte_offset(pmdp, addr);
@@ -484,7 +484,7 @@
 				                   PAGE_SHIFT, pageno);
 			}
 		}
-		up_read(&mm->mmap_sem);
+		mm_unlock_shared(mm);
 		break;
 	}
 
@@ -534,7 +534,7 @@
 	struct mm_struct *mm = current->mm;
 	int ret;
 
-	down_write(&mm->mmap_sem);
+	mm_lock_exclusive(mm);
 	if (brk < mm->end_code) {
 		ret = -ENOMEM;
 		goto out;
@@ -592,7 +592,7 @@
 	ret = 0;
 
 out:
-	up_write(&mm->mmap_sem);
+	mm_unlock_exclusive(mm);
 	return ret;
 }
 
@@ -1082,9 +1082,9 @@
 
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	retval = do_mmap(file, addr, len, prot, flags, offset);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if (file)
 		fput(file);
 
@@ -1642,9 +1642,9 @@
 
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/mips/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/mips/mm/fault.c linux-mmsem/arch/mips/mm/fault.c
@@ -72,7 +72,7 @@
 	printk("[%s:%d:%08lx:%ld:%08lx]\n", current->comm, current->pid,
 	       address, write, regs->cp0_epc);
 #endif
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -115,7 +115,7 @@
 		goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
 /*
@@ -123,7 +123,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 bad_area_nosemaphore:
 	/* User mode accesses just cause a SIGSEGV */
@@ -177,14 +177,14 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/mips64/kernel/linux32.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/kernel/linux32.c linux-mmsem/arch/mips64/kernel/linux32.c
@@ -443,10 +443,10 @@
 	 *  `execve' frees all current memory we only have to do an
 	 *  `munmap' if the `execve' failes.
 	 */
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	av = (char **) do_mmap_pgoff(0, 0, len, PROT_READ | PROT_WRITE,
 				     MAP_PRIVATE | MAP_ANONYMOUS, 0);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (IS_ERR(av))
 		return (long) av;
arch/mips64/kernel/syscall.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/kernel/syscall.c linux-mmsem/arch/mips64/kernel/syscall.c
@@ -65,9 +65,9 @@
 	}
         flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
         error = do_mmap(file, addr, len, prot, flags, offset);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
         if (file)
                 fput(file);
 out:
arch/mips64/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/mips64/mm/fault.c linux-mmsem/arch/mips64/mm/fault.c
@@ -124,7 +124,7 @@
 	printk("Cpu%d[%s:%d:%08lx:%ld:%08lx]\n", smp_processor_id(), current->comm,
 		current->pid, address, write, regs->cp0_epc);
 #endif
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -167,7 +167,7 @@
 		goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
 /*
@@ -175,7 +175,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 bad_area_nosemaphore:
 	if (user_mode(regs)) {
@@ -233,14 +233,14 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/parisc/kernel/sys_parisc.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/parisc/kernel/sys_parisc.c linux-mmsem/arch/parisc/kernel/sys_parisc.c
@@ -51,7 +51,7 @@
 	struct file * file = NULL;
 	int error;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	lock_kernel();
 	if (!(flags & MAP_ANONYMOUS)) {
 		error = -EBADF;
@@ -65,7 +65,7 @@
 		fput(file);
 out:
 	unlock_kernel();
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return error;
 }
 
arch/parisc/mm/fault.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/parisc/mm/fault.c linux-mmsem/arch/parisc/mm/fault.c
@@ -175,7 +175,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	vma = pa_find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -218,14 +218,14 @@
 	      default:
 		goto out_of_memory;
 	}
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
 /*
  * Something tried to access memory that isn't in our memory map..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	if (user_mode(regs)) {
 		struct siginfo si;
@@ -275,7 +275,7 @@
 	parisc_terminate("Bad Address (null pointer deref?)",regs,code,address);
 
   out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", current->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
arch/ppc/kernel/syscalls.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/kernel/syscalls.c linux-mmsem/arch/ppc/kernel/syscalls.c
@@ -202,9 +202,9 @@
 			goto out;
 	}
 	
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	ret = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if (file)
 		fput(file);
 out:
arch/ppc/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/ppc/mm/fault.c linux-mmsem/arch/ppc/mm/fault.c
@@ -103,7 +103,7 @@
 		bad_page_fault(regs, address, SIGSEGV);
 		return;
 	}
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -163,7 +163,7 @@
                 goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	/*
 	 * keep track of tlb+htab misses that are good addrs but
 	 * just need pte's created via handle_mm_fault()
@@ -173,7 +173,7 @@
 	return;
 
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	pte_errors++;	
 
 	/* User mode accesses cause a SIGSEGV */
@@ -194,7 +194,7 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", current->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
@@ -202,7 +202,7 @@
 	return;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	info.si_signo = SIGBUS;
 	info.si_errno = 0;
 	info.si_code = BUS_ADRERR;
arch/s390/kernel/sys_s390.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/s390/kernel/sys_s390.c linux-mmsem/arch/s390/kernel/sys_s390.c
@@ -61,9 +61,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/s390/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/s390/mm/fault.c linux-mmsem/arch/s390/mm/fault.c
@@ -113,7 +113,7 @@
 	 * task's user address space, so we search the VMAs
 	 */
 
-        down_read(&mm->mmap_sem);
+        mm_lock_shared_recursive(mm);
 
         vma = find_vma(mm, address);
         if (!vma)
@@ -164,7 +164,7 @@
 		goto out_of_memory;
 	}
 
-        up_read(&mm->mmap_sem);
+        mm_unlock_shared(mm);
         return;
 
 /*
@@ -172,7 +172,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-        up_read(&mm->mmap_sem);
+        mm_unlock_shared(mm);
 
         /* User mode accesses just cause a SIGSEGV */
         if (regs->psw.mask & PSW_PROBLEM_STATE) {
@@ -231,14 +231,14 @@
  * us unable to handle the page fault gracefully.
 */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if (regs->psw.mask & PSW_PROBLEM_STATE)
 		do_exit(SIGKILL);
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/s390x/kernel/binfmt_elf32.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/s390x/kernel/binfmt_elf32.c linux-mmsem/arch/s390x/kernel/binfmt_elf32.c
@@ -194,11 +194,11 @@
 	if(!addr)
 		addr = 0x40000000;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	map_addr = do_mmap(filep, ELF_PAGESTART(addr),
 			   eppnt->p_filesz + ELF_PAGEOFFSET(eppnt->p_vaddr), prot, type,
 			   eppnt->p_offset - ELF_PAGEOFFSET(eppnt->p_vaddr));
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return(map_addr);
 }
 
arch/s390x/kernel/exec32.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/s390x/kernel/exec32.c linux-mmsem/arch/s390x/kernel/exec32.c
@@ -54,7 +54,7 @@
 	if (!mpnt) 
 		return -ENOMEM; 
 	
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	{
 		mpnt->vm_mm = current->mm;
 		mpnt->vm_start = PAGE_MASK & (unsigned long) bprm->p;
@@ -77,7 +77,7 @@
 		}
 		stack_base += PAGE_SIZE;
 	}
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	
 	return 0;
 }
arch/s390x/kernel/linux32.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/s390x/kernel/linux32.c linux-mmsem/arch/s390x/kernel/linux32.c
@@ -4186,14 +4186,14 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
 	if (!IS_ERR((void *) error) && error + len >= 0x80000000ULL) {
 		/* Result is out of bounds.  */
 		do_munmap(current->mm, addr, len);
 		error = -ENOMEM;
 	}
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/s390x/kernel/sys_s390.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/s390x/kernel/sys_s390.c linux-mmsem/arch/s390x/kernel/sys_s390.c
@@ -61,9 +61,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/s390x/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/s390x/mm/fault.c linux-mmsem/arch/s390x/mm/fault.c
@@ -141,7 +141,7 @@
 	 * task's user address space, so we search the VMAs
 	 */
 
-        down_read(&mm->mmap_sem);
+        mm_lock_shared_recursive(mm);
 
         vma = find_vma(mm, address);
         if (!vma) {
@@ -195,7 +195,7 @@
 		goto out_of_memory;
 	}
 
-        up_read(&mm->mmap_sem);
+        mm_unlock_shared(mm);
         return;
 
 /*
@@ -203,7 +203,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-        up_read(&mm->mmap_sem);
+        mm_unlock_shared(mm);
 
         /* User mode accesses just cause a SIGSEGV */
         if (regs->psw.mask & PSW_PROBLEM_STATE) {
@@ -262,14 +262,14 @@
  * us unable to handle the page fault gracefully.
 */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if (regs->psw.mask & PSW_PROBLEM_STATE)
 		do_exit(SIGKILL);
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/sh/kernel/sys_sh.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/sh/kernel/sys_sh.c linux-mmsem/arch/sh/kernel/sys_sh.c
@@ -96,9 +96,9 @@
 			goto out;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	if (file)
 		fput(file);
arch/sh/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/sh/mm/fault.c linux-mmsem/arch/sh/mm/fault.c
@@ -105,7 +105,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared_recursive(mm);
 
 	vma = find_vma(mm, address);
 	if (!vma)
@@ -147,7 +147,7 @@
 		goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
 /*
@@ -155,7 +155,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	if (user_mode(regs)) {
 		tsk->thread.address = address;
@@ -204,14 +204,14 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/sparc/kernel/sys_sparc.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/sparc/kernel/sys_sparc.c linux-mmsem/arch/sparc/kernel/sys_sparc.c
@@ -242,9 +242,9 @@
 
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	retval = do_mmap_pgoff(file, addr, len, prot, flags, pgoff);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 out_putf:
 	if (file)
@@ -288,7 +288,7 @@
 	if (old_len > TASK_SIZE - PAGE_SIZE ||
 	    new_len > TASK_SIZE - PAGE_SIZE)
 		goto out;
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	if (flags & MREMAP_FIXED) {
 		if (ARCH_SUN4C_SUN4 &&
 		    new_addr < 0xe0000000 &&
@@ -323,7 +323,7 @@
 	}
 	ret = do_mremap(addr, old_len, new_len, flags, new_addr);
 out_sem:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 out:
 	return ret;       
 }
arch/sparc/kernel/sys_sunos.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/sparc/kernel/sys_sunos.c linux-mmsem/arch/sparc/kernel/sys_sunos.c
@@ -116,9 +116,9 @@
 	}
 
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	retval = do_mmap(file, addr, len, prot, flags, off);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if(!ret_type)
 		retval = ((retval < PAGE_OFFSET) ? 0 : retval);
 
@@ -145,7 +145,7 @@
 	unsigned long rlim;
 	unsigned long newbrk, oldbrk;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	if(ARCH_SUN4C_SUN4) {
 		if(brk >= 0x20000000 && brk < 0xe0000000) {
 			goto out;
@@ -208,7 +208,7 @@
 	do_brk(oldbrk, newbrk-oldbrk);
 	retval = 0;
 out:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return retval;
 }
 
arch/sparc/mm/fault.c +9 -9
diff -uNr linux-2.4.10-pre12/arch/sparc/mm/fault.c linux-mmsem/arch/sparc/mm/fault.c
@@ -222,7 +222,7 @@
         if (in_interrupt() || !mm)
                 goto no_context;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 
 	/*
 	 * The kernel referencing a bad kernel pointer can lock up
@@ -272,7 +272,7 @@
 	default:
 		goto out_of_memory;
 	}
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 
 	/*
@@ -280,7 +280,7 @@
 	 * Fix it, but check if it's kernel or user first..
 	 */
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 bad_area_nosemaphore:
 	/* User mode accesses just cause a SIGSEGV */
@@ -336,14 +336,14 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", tsk->comm);
 	if (from_user)
 		do_exit(SIGKILL);
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	info.si_signo = SIGBUS;
 	info.si_errno = 0;
 	info.si_code = BUS_ADRERR;
@@ -477,7 +477,7 @@
 	printk("wf<pid=%d,wr=%d,addr=%08lx>\n",
 	       tsk->pid, write, address);
 #endif
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 	vma = find_vma(mm, address);
 	if(!vma)
 		goto bad_area;
@@ -498,10 +498,10 @@
 	}
 	if (!handle_mm_fault(mm, vma, address, write))
 		goto do_sigbus;
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return;
 bad_area:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 #if 0
 	printk("Window whee %s [%d]: segfaults at %08lx\n",
 	       tsk->comm, tsk->pid, address);
@@ -516,7 +516,7 @@
 	return;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	info.si_signo = SIGBUS;
 	info.si_errno = 0;
 	info.si_code = BUS_ADRERR;
arch/sparc64/kernel/binfmt_aout32.c +6 -6
diff -uNr linux-2.4.10-pre12/arch/sparc64/kernel/binfmt_aout32.c linux-mmsem/arch/sparc64/kernel/binfmt_aout32.c
@@ -277,24 +277,24 @@
 			goto beyond_if;
 		}
 
-	        down_write(&current->mm->mmap_sem);
+	        mm_lock_exclusive(current->mm);
 		error = do_mmap(bprm->file, N_TXTADDR(ex), ex.a_text,
 			PROT_READ | PROT_EXEC,
 			MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE | MAP_EXECUTABLE,
 			fd_offset);
-	        up_write(&current->mm->mmap_sem);
+	        mm_unlock_exclusive(current->mm);
 
 		if (error != N_TXTADDR(ex)) {
 			send_sig(SIGKILL, current, 0);
 			return error;
 		}
 
-	        down_write(&current->mm->mmap_sem);
+	        mm_lock_exclusive(current->mm);
  		error = do_mmap(bprm->file, N_DATADDR(ex), ex.a_data,
 				PROT_READ | PROT_WRITE | PROT_EXEC,
 				MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE | MAP_EXECUTABLE,
 				fd_offset + ex.a_text);
-	        up_write(&current->mm->mmap_sem);
+	        mm_unlock_exclusive(current->mm);
 		if (error != N_DATADDR(ex)) {
 			send_sig(SIGKILL, current, 0);
 			return error;
@@ -369,12 +369,12 @@
 	start_addr =  ex.a_entry & 0xfffff000;
 
 	/* Now use mmap to map the library into memory. */
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap(file, start_addr, ex.a_text + ex.a_data,
 			PROT_READ | PROT_WRITE | PROT_EXEC,
 			MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE,
 			N_TXTOFF(ex));
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	retval = error;
 	if (error != start_addr)
 		goto out;
arch/sparc64/kernel/sys_sparc.c +6 -6
diff -uNr linux-2.4.10-pre12/arch/sparc64/kernel/sys_sparc.c linux-mmsem/arch/sparc64/kernel/sys_sparc.c
@@ -292,9 +292,9 @@
 			goto out_putf;
 	}
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	retval = do_mmap(file, addr, len, prot, flags, off);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 out_putf:
 	if (file)
@@ -310,9 +310,9 @@
 	if (len > -PAGE_OFFSET ||
 	    (addr < PAGE_OFFSET && addr + len > -PAGE_OFFSET))
 		return -EINVAL;
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	ret = do_munmap(current->mm, addr, len);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return ret;
 }
 
@@ -332,7 +332,7 @@
 		goto out;
 	if (addr < PAGE_OFFSET && addr + old_len > -PAGE_OFFSET)
 		goto out;
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	if (flags & MREMAP_FIXED) {
 		if (new_addr < PAGE_OFFSET &&
 		    new_addr + new_len > -PAGE_OFFSET)
@@ -363,7 +363,7 @@
 	}
 	ret = do_mremap(addr, old_len, new_len, flags, new_addr);
 out_sem:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 out:
 	return ret;       
 }
arch/sparc64/kernel/sys_sparc32.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/sparc64/kernel/sys_sparc32.c linux-mmsem/arch/sparc64/kernel/sys_sparc32.c
@@ -4141,7 +4141,7 @@
 		goto out;
 	if (addr > 0xf0000000UL - old_len)
 		goto out;
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	if (flags & MREMAP_FIXED) {
 		if (new_addr > 0xf0000000UL - new_len)
 			goto out_sem;
@@ -4171,7 +4171,7 @@
 	}
 	ret = do_mremap(addr, old_len, new_len, flags, new_addr);
 out_sem:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 out:
 	return ret;       
 }
arch/sparc64/kernel/sys_sunos32.c +4 -4
diff -uNr linux-2.4.10-pre12/arch/sparc64/kernel/sys_sunos32.c linux-mmsem/arch/sparc64/kernel/sys_sunos32.c
@@ -100,12 +100,12 @@
 	flags &= ~_MAP_NEW;
 
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	retval = do_mmap(file,
 			 (unsigned long) addr, (unsigned long) len,
 			 (unsigned long) prot, (unsigned long) flags,
 			 (unsigned long) off);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if(!ret_type)
 		retval = ((retval < 0xf0000000) ? 0 : retval);
 out_putf:
@@ -126,7 +126,7 @@
 	unsigned long rlim;
 	unsigned long newbrk, oldbrk, brk = (unsigned long) baddr;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	if (brk < current->mm->end_code)
 		goto out;
 	newbrk = PAGE_ALIGN(brk);
@@ -170,7 +170,7 @@
 	do_brk(oldbrk, newbrk-oldbrk);
 	retval = 0;
 out:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return retval;
 }
 
arch/sparc64/mm/fault.c +5 -5
diff -uNr linux-2.4.10-pre12/arch/sparc64/mm/fault.c linux-mmsem/arch/sparc64/mm/fault.c
@@ -306,7 +306,7 @@
 		address &= 0xffffffff;
 	}
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -378,7 +378,7 @@
 		goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	goto fault_done;
 
 	/*
@@ -387,7 +387,7 @@
 	 */
 bad_area:
 	insn = get_fault_insn(regs, insn);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 handle_kernel_fault:
 	do_kernel_fault(regs, si_code, fault_code, insn, address);
@@ -400,7 +400,7 @@
  */
 out_of_memory:
 	insn = get_fault_insn(regs, insn);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	printk("VM: killing process %s\n", current->comm);
 	if (!(regs->tstate & TSTATE_PRIV))
 		do_exit(SIGKILL);
@@ -412,7 +412,7 @@
 
 do_sigbus:
 	insn = get_fault_insn(regs, insn);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/sparc64/solaris/misc.c +2 -2
diff -uNr linux-2.4.10-pre12/arch/sparc64/solaris/misc.c linux-mmsem/arch/sparc64/solaris/misc.c
@@ -92,12 +92,12 @@
 	ret_type = flags & _MAP_NEW;
 	flags &= ~_MAP_NEW;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
 	retval = do_mmap(file,
 			 (unsigned long) addr, (unsigned long) len,
 			 (unsigned long) prot, (unsigned long) flags, off);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if(!ret_type)
 		retval = ((retval < 0xf0000000) ? 0 : retval);
 	                        
drivers/char/mem.c +3 -3
diff -uNr linux-2.4.10-pre12/drivers/char/mem.c linux-mmsem/drivers/char/mem.c
@@ -350,7 +350,7 @@
 
 	mm = current->mm;
 	/* Oops, this was forgotten before. -ben */
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 
 	/* For private mappings, just map in zero pages. */
 	for (vma = find_vma(mm, addr); vma; vma = vma->vm_next) {
@@ -374,7 +374,7 @@
 			goto out_up;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	
 	/* The shared case is hard. Let's do the conventional zeroing. */ 
 	do {
@@ -389,7 +389,7 @@
 
 	return size;
 out_up:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return size;
 }
 
drivers/sgi/char/graphics.c +2 -2
diff -uNr linux-2.4.10-pre12/drivers/sgi/char/graphics.c linux-mmsem/drivers/sgi/char/graphics.c
@@ -152,11 +152,11 @@
 		 * sgi_graphics_mmap
 		 */
 		disable_gconsole ();
-		down_write(&current->mm->mmap_sem);
+		mm_lock_exclusive(current->mm);
 		r = do_mmap (file, (unsigned long)vaddr,
 			     cards[board].g_regs_size, PROT_READ|PROT_WRITE,
 			     MAP_FIXED|MAP_PRIVATE, 0);
-		up_write(&current->mm->mmap_sem);
+		mm_unlock_exclusive(current->mm);
 		if (r)
 			return r;
 	}
drivers/sgi/char/shmiq.c +2 -2
diff -uNr linux-2.4.10-pre12/drivers/sgi/char/shmiq.c linux-mmsem/drivers/sgi/char/shmiq.c
@@ -285,11 +285,11 @@
 			s = req.arg * sizeof (struct shmqevent) +
 			    sizeof (struct sharedMemoryInputQueue);
 			v = sys_munmap (vaddr, s);
-			down_write(&current->mm->mmap_sem);
+			mm_lock_exclusive(current->mm);
 			do_munmap(current->mm, vaddr, s);
 			do_mmap(filp, vaddr, s, PROT_READ | PROT_WRITE,
 			        MAP_PRIVATE|MAP_FIXED, 0);
-			up_write(&current->mm->mmap_sem);
+			mm_unlock_exclusive(current->mm);
 			shmiqs[minor].events = req.arg;
 			shmiqs[minor].mapped = 1;
 
fs/binfmt_aout.c +6 -6
diff -uNr linux-2.4.10-pre12/fs/binfmt_aout.c linux-mmsem/fs/binfmt_aout.c
@@ -377,24 +377,24 @@
 			goto beyond_if;
 		}
 
-		down_write(&current->mm->mmap_sem);
+		mm_lock_exclusive(current->mm);
 		error = do_mmap(bprm->file, N_TXTADDR(ex), ex.a_text,
 			PROT_READ | PROT_EXEC,
 			MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE | MAP_EXECUTABLE,
 			fd_offset);
-		up_write(&current->mm->mmap_sem);
+		mm_unlock_exclusive(current->mm);
 
 		if (error != N_TXTADDR(ex)) {
 			send_sig(SIGKILL, current, 0);
 			return error;
 		}
 
-		down_write(&current->mm->mmap_sem);
+		mm_lock_exclusive(current->mm);
  		error = do_mmap(bprm->file, N_DATADDR(ex), ex.a_data,
 				PROT_READ | PROT_WRITE | PROT_EXEC,
 				MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE | MAP_EXECUTABLE,
 				fd_offset + ex.a_text);
-		up_write(&current->mm->mmap_sem);
+		mm_unlock_exclusive(current->mm);
 		if (error != N_DATADDR(ex)) {
 			send_sig(SIGKILL, current, 0);
 			return error;
@@ -476,12 +476,12 @@
 		goto out;
 	}
 	/* Now use mmap to map the library into memory. */
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap(file, start_addr, ex.a_text + ex.a_data,
 			PROT_READ | PROT_WRITE | PROT_EXEC,
 			MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE,
 			N_TXTOFF(ex));
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	retval = error;
 	if (error != start_addr)
 		goto out;
fs/binfmt_elf.c +6 -6
diff -uNr linux-2.4.10-pre12/fs/binfmt_elf.c linux-mmsem/fs/binfmt_elf.c
@@ -224,11 +224,11 @@
 {
 	unsigned long map_addr;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	map_addr = do_mmap(filep, ELF_PAGESTART(addr),
 			   eppnt->p_filesz + ELF_PAGEOFFSET(eppnt->p_vaddr), prot, type,
 			   eppnt->p_offset - ELF_PAGEOFFSET(eppnt->p_vaddr));
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return(map_addr);
 }
 
@@ -743,10 +743,10 @@
 		   Since we do not have the power to recompile these, we
 		   emulate the SVr4 behavior.  Sigh.  */
 		/* N.B. Shouldn't the size here be PAGE_SIZE?? */
-		down_write(&current->mm->mmap_sem);
+		mm_lock_exclusive(current->mm);
 		error = do_mmap(NULL, 0, 4096, PROT_READ | PROT_EXEC,
 				MAP_FIXED | MAP_PRIVATE, 0);
-		up_write(&current->mm->mmap_sem);
+		mm_unlock_exclusive(current->mm);
 	}
 
 #ifdef ELF_PLAT_INIT
@@ -827,7 +827,7 @@
 	while (elf_phdata->p_type != PT_LOAD) elf_phdata++;
 
 	/* Now use mmap to map the library into memory. */
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	error = do_mmap(file,
 			ELF_PAGESTART(elf_phdata->p_vaddr),
 			(elf_phdata->p_filesz +
@@ -836,7 +836,7 @@
 			MAP_FIXED | MAP_PRIVATE | MAP_DENYWRITE,
 			(elf_phdata->p_offset -
 			 ELF_PAGEOFFSET(elf_phdata->p_vaddr)));
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	if (error != ELF_PAGESTART(elf_phdata->p_vaddr))
 		goto out_free_ph;
 
fs/exec.c +4 -4
diff -uNr linux-2.4.10-pre12/fs/exec.c linux-mmsem/fs/exec.c
@@ -307,7 +307,7 @@
 	if (!mpnt) 
 		return -ENOMEM; 
 	
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	{
 		mpnt->vm_mm = current->mm;
 		mpnt->vm_start = PAGE_MASK & (unsigned long) bprm->p;
@@ -330,7 +330,7 @@
 		}
 		stack_base += PAGE_SIZE;
 	}
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	
 	return 0;
 }
@@ -969,9 +969,9 @@
 	if (do_truncate(file->f_dentry, 0) != 0)
 		goto close_fail;
 
-	down_read(&current->mm->mmap_sem);
+	mm_lock_shared(current->mm);
 	retval = binfmt->core_dump(signr, regs, file);
-	up_read(&current->mm->mmap_sem);
+	mm_unlock_shared(current->mm);
 
 close_fail:
 	filp_close(file, NULL);
fs/proc/array.c +8 -8
diff -uNr linux-2.4.10-pre12/fs/proc/array.c linux-mmsem/fs/proc/array.c
@@ -181,7 +181,7 @@
 	unsigned long data = 0, stack = 0;
 	unsigned long exec = 0, lib = 0;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 	for (vma = mm->mmap; vma; vma = vma->vm_next) {
 		unsigned long len = (vma->vm_end - vma->vm_start) >> 10;
 		if (!vma->vm_file) {
@@ -212,7 +212,7 @@
 		mm->rss << (PAGE_SHIFT-10),
 		data - stack, stack,
 		exec - lib, lib);
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	return buffer;
 }
 
@@ -317,7 +317,7 @@
 	task_unlock(task);
 	if (mm) {
 		struct vm_area_struct *vma;
-		down_read(&mm->mmap_sem);
+		mm_lock_shared(mm);
 		vma = mm->mmap;
 		while (vma) {
 			vsize += vma->vm_end - vma->vm_start;
@@ -325,7 +325,7 @@
 		}
 		eip = KSTK_EIP(task);
 		esp = KSTK_ESP(task);
-		up_read(&mm->mmap_sem);
+		mm_unlock_shared(mm);
 	}
 
 	wchan = get_wchan(task);
@@ -479,7 +479,7 @@
 	task_unlock(task);
 	if (mm) {
 		struct vm_area_struct * vma;
-		down_read(&mm->mmap_sem);
+		mm_lock_shared(mm);
 		vma = mm->mmap;
 		while (vma) {
 			pgd_t *pgd = pgd_offset(mm, vma->vm_start);
@@ -500,7 +500,7 @@
 				drs += pages;
 			vma = vma->vm_next;
 		}
-		up_read(&mm->mmap_sem);
+		mm_unlock_shared(mm);
 		mmput(mm);
 	}
 	return sprintf(buffer,"%d %d %d %d %d %d %d\n",
@@ -577,7 +577,7 @@
 	column = *ppos & (MAPS_LINE_LENGTH-1);
 
 	/* quickly go to line lineno */
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 	for (map = mm->mmap, i = 0; map && (i < lineno); map = map->vm_next, i++)
 		continue;
 
@@ -658,7 +658,7 @@
 		if (volatile_task)
 			break;
 	}
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 
 	/* encode f_pos */
 	*ppos = (lineno << MAPS_LINE_SHIFT) + column;
fs/proc/base.c +2 -2
diff -uNr linux-2.4.10-pre12/fs/proc/base.c linux-mmsem/fs/proc/base.c
@@ -64,7 +64,7 @@
 	task_unlock(task);
 	if (!mm)
 		goto out;
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 	vma = mm->mmap;
 	while (vma) {
 		if ((vma->vm_flags & VM_EXECUTABLE) && 
@@ -76,7 +76,7 @@
 		}
 		vma = vma->vm_next;
 	}
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	mmput(mm);
 out:
 	return result;
include/linux/sched.h +67 -10

89 lines not on this page · raw message

ipc/shm.c +4 -4
diff -uNr linux-2.4.10-pre12/ipc/shm.c linux-mmsem/ipc/shm.c
@@ -619,9 +619,9 @@
 	shp->shm_nattch++;
 	shm_unlock(shmid);
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	user_addr = (void *) do_mmap (file, addr, file->f_dentry->d_inode->i_size, prot, flags, 0);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 
 	down (&shm_ids.sem);
 	if(!(shp = shm_lock(shmid)))
@@ -650,14 +650,14 @@
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *shmd, *shmdnext;
 
-	down_write(&mm->mmap_sem);
+	mm_lock_exclusive(mm);
 	for (shmd = mm->mmap; shmd; shmd = shmdnext) {
 		shmdnext = shmd->vm_next;
 		if (shmd->vm_ops == &shm_vm_ops
 		    && shmd->vm_start - (shmd->vm_pgoff << PAGE_SHIFT) == (ulong) shmaddr)
 			do_munmap(mm, shmd->vm_start, shmd->vm_end - shmd->vm_start);
 	}
-	up_write(&mm->mmap_sem);
+	mm_unlock_exclusive(mm);
 	return 0;
 }
 
kernel/acct.c +2 -2
diff -uNr linux-2.4.10-pre12/kernel/acct.c linux-mmsem/kernel/acct.c
@@ -315,13 +315,13 @@
 	vsize = 0;
 	if (current->mm) {
 		struct vm_area_struct *vma;
-		down_read(&current->mm->mmap_sem);
+		mm_lock_shared(current->mm);
 		vma = current->mm->mmap;
 		while (vma) {
 			vsize += vma->vm_end - vma->vm_start;
 			vma = vma->vm_next;
 		}
-		up_read(&current->mm->mmap_sem);
+		mm_unlock_shared(current->mm);
 	}
 	vsize = vsize / 1024;
 	ac.ac_mem = encode_comp_t(vsize);
kernel/fork.c +5 -3
diff -uNr linux-2.4.10-pre12/kernel/fork.c linux-mmsem/kernel/fork.c
@@ -216,7 +216,9 @@
 {
 	atomic_set(&mm->mm_users, 1);
 	atomic_set(&mm->mm_count, 1);
-	init_rwsem(&mm->mmap_sem);
+	spin_lock_init(&mm->mmsem_lock);
+	mm->mmsem_activity = 0;
+	INIT_LIST_HEAD(&mm->mmsem_waiters);
 	mm->page_table_lock = SPIN_LOCK_UNLOCKED;
 	mm->pgd = pgd_alloc(mm);
 	if (mm->pgd)
@@ -333,9 +335,9 @@
 	if (!mm_init(mm))
 		goto fail_nomem;
 
-	down_write(&oldmm->mmap_sem);
+	mm_lock_exclusive(oldmm);
 	retval = dup_mmap(mm);
-	up_write(&oldmm->mmap_sem);
+	mm_unlock_exclusive(oldmm);
 
 	if (retval)
 		goto free_pt;
kernel/ksyms.c +2 -0
diff -uNr linux-2.4.10-pre12/kernel/ksyms.c linux-mmsem/kernel/ksyms.c
@@ -87,6 +87,8 @@
 EXPORT_SYMBOL(exit_files);
 EXPORT_SYMBOL(exit_fs);
 EXPORT_SYMBOL(exit_sighand);
+EXPORT_SYMBOL(__mm_lock_wait);
+EXPORT_SYMBOL(__mm_lock_wake);
 
 /* internal kernel memory management */
 EXPORT_SYMBOL(_alloc_pages);
kernel/ptrace.c +2 -2
diff -uNr linux-2.4.10-pre12/kernel/ptrace.c linux-mmsem/kernel/ptrace.c
@@ -208,13 +208,13 @@
 	if (!mm)
 		return 0;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 	vma = find_extend_vma(mm, addr);
 	copied = 0;
 	if (vma)
 		copied = access_mm(mm, vma, addr, buf, len, write);
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	mmput(mm);
 	return copied;
 }
mm/filemap.c +6 -6
diff -uNr linux-2.4.10-pre12/mm/filemap.c linux-mmsem/mm/filemap.c
@@ -1949,7 +1949,7 @@
 	struct vm_area_struct * vma;
 	int unmapped_error, error = -EINVAL;
 
-	down_read(&current->mm->mmap_sem);
+	mm_lock_shared(current->mm);
 	if (start & ~PAGE_MASK)
 		goto out;
 	len = (len + ~PAGE_MASK) & PAGE_MASK;
@@ -1995,7 +1995,7 @@
 		vma = vma->vm_next;
 	}
 out:
-	up_read(&current->mm->mmap_sem);
+	mm_unlock_shared(current->mm);
 	return error;
 }
 
@@ -2298,7 +2298,7 @@
 	int unmapped_error = 0;
 	int error = -EINVAL;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 
 	if (start & ~PAGE_MASK)
 		goto out;
@@ -2349,7 +2349,7 @@
 	}
 
 out:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return error;
 }
 
@@ -2451,7 +2451,7 @@
 	int unmapped_error = 0;
 	long error = -EINVAL;
 
-	down_read(&current->mm->mmap_sem);
+	mm_lock_shared(current->mm);
 
 	if (start & ~PAGE_CACHE_MASK)
 		goto out;
@@ -2503,7 +2503,7 @@
 	}
 
 out:
-	up_read(&current->mm->mmap_sem);
+	mm_unlock_shared(current->mm);
 	return error;
 }
 
mm/memory.c +3 -3
diff -uNr linux-2.4.10-pre12/mm/memory.c linux-mmsem/mm/memory.c
@@ -464,7 +464,7 @@
 	if (err)
 		return err;
 
-	down_read(&mm->mmap_sem);
+	mm_lock_shared(mm);
 
 	err = -EFAULT;
 	iobuf->locked = 0;
@@ -522,12 +522,12 @@
 		ptr += PAGE_SIZE;
 	}
 
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	dprintk ("map_user_kiobuf: end OK\n");
 	return 0;
 
  out_unlock:
-	up_read(&mm->mmap_sem);
+	mm_unlock_shared(mm);
 	unmap_kiobuf(iobuf);
 	dprintk ("map_user_kiobuf: end %d\n", err);
 	return err;
mm/mlock.c +8 -8
diff -uNr linux-2.4.10-pre12/mm/mlock.c linux-mmsem/mm/mlock.c
@@ -198,7 +198,7 @@
 	unsigned long lock_limit;
 	int error = -ENOMEM;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	len = PAGE_ALIGN(len + (start & ~PAGE_MASK));
 	start &= PAGE_MASK;
 
@@ -219,7 +219,7 @@
 
 	error = do_mlock(start, len, 1);
 out:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return error;
 }
 
@@ -227,11 +227,11 @@
 {
 	int ret;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	len = PAGE_ALIGN(len + (start & ~PAGE_MASK));
 	start &= PAGE_MASK;
 	ret = do_mlock(start, len, 0);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return ret;
 }
 
@@ -268,7 +268,7 @@
 	unsigned long lock_limit;
 	int ret = -EINVAL;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	if (!flags || (flags & ~(MCL_CURRENT | MCL_FUTURE)))
 		goto out;
 
@@ -286,7 +286,7 @@
 
 	ret = do_mlockall(flags);
 out:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return ret;
 }
 
@@ -294,8 +294,8 @@
 {
 	int ret;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	ret = do_mlockall(0);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return ret;
 }
mm/mmap.c +86 -4

113 lines not on this page · raw message

mm/mprotect.c +2 -2
diff -uNr linux-2.4.10-pre12/mm/mprotect.c linux-mmsem/mm/mprotect.c
@@ -281,7 +281,7 @@
 	if (end == start)
 		return 0;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 
 	vma = find_vma_prev(current->mm, start, &prev);
 	error = -EFAULT;
@@ -332,6 +332,6 @@
 		prev->vm_mm->map_count--;
 	}
 out:
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return error;
 }
mm/mremap.c +2 -2
diff -uNr linux-2.4.10-pre12/mm/mremap.c linux-mmsem/mm/mremap.c
@@ -346,8 +346,8 @@
 {
 	unsigned long ret;
 
-	down_write(&current->mm->mmap_sem);
+	mm_lock_exclusive(current->mm);
 	ret = do_mremap(addr, old_len, new_len, flags, new_addr);
-	up_write(&current->mm->mmap_sem);
+	mm_unlock_exclusive(current->mm);
 	return ret;
 }
#20 lore raw
In reply to Linus Torvalds

Here's a patch to make rwsems unfair.

David

arch/alpha/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/alpha/config.in linux-rwsem/arch/alpha/config.in
@@ -5,8 +5,6 @@
 
 define_bool CONFIG_ALPHA y
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 
 mainmenu_name "Kernel configuration of Linux for Alpha machines"
 
arch/arm/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/config.in linux-rwsem/arch/arm/config.in
@@ -9,8 +9,6 @@
 define_bool CONFIG_SBUS n
 define_bool CONFIG_MCA n
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 
 mainmenu_option next_comment
arch/arm/def-configs/anakin +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/anakin linux-rwsem/arch/arm/def-configs/anakin
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/assabet +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/assabet linux-rwsem/arch/arm/def-configs/assabet
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/bitsy +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/bitsy linux-rwsem/arch/arm/def-configs/bitsy
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/ebsa110 +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/ebsa110 linux-rwsem/arch/arm/def-configs/ebsa110
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/flexanet +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/flexanet linux-rwsem/arch/arm/def-configs/flexanet
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/integrator +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/integrator linux-rwsem/arch/arm/def-configs/integrator
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/lart +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/lart linux-rwsem/arch/arm/def-configs/lart
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/neponset +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/neponset linux-rwsem/arch/arm/def-configs/neponset
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/pleb +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/pleb linux-rwsem/arch/arm/def-configs/pleb
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/rpc +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/rpc linux-rwsem/arch/arm/def-configs/rpc
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/arm/def-configs/shark +0 -2
diff -uNr linux-2.4.10-pre12/arch/arm/def-configs/shark linux-rwsem/arch/arm/def-configs/shark
@@ -6,8 +6,6 @@
 # CONFIG_SBUS is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/cris/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/cris/config.in linux-rwsem/arch/cris/config.in
@@ -5,8 +5,6 @@
 mainmenu_name "Linux/CRIS Kernel Configuration"
 
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/cris/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/cris/defconfig linux-rwsem/arch/cris/defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 
 #
 # Code maturity level options
arch/i386/config.in +0 -4
diff -uNr linux-2.4.10-pre12/arch/i386/config.in linux-rwsem/arch/i386/config.in
@@ -50,8 +50,6 @@
    define_bool CONFIG_X86_CMPXCHG n
    define_bool CONFIG_X86_XADD n
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
-   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 else
    define_bool CONFIG_X86_WP_WORKS_OK y
    define_bool CONFIG_X86_INVLPG y
@@ -59,8 +57,6 @@
    define_bool CONFIG_X86_XADD y
    define_bool CONFIG_X86_BSWAP y
    define_bool CONFIG_X86_POPAD_OK y
-   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 fi
 if [ "$CONFIG_M486" = "y" ]; then
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
arch/i386/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/i386/defconfig linux-rwsem/arch/i386/defconfig
@@ -42,8 +42,6 @@
 CONFIG_X86_XADD=y
 CONFIG_X86_BSWAP=y
 CONFIG_X86_POPAD_OK=y
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 CONFIG_X86_L1_CACHE_SHIFT=5
 CONFIG_X86_TSC=y
 CONFIG_X86_GOOD_APIC=y
arch/ia64/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/ia64/config.in linux-rwsem/arch/ia64/config.in
@@ -23,8 +23,6 @@
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
 define_bool CONFIG_SBUS n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 if [ "$CONFIG_IA64_HP_SIM" = "n" ]; then
   define_bool CONFIG_ACPI y
arch/m68k/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/m68k/config.in linux-rwsem/arch/m68k/config.in
@@ -4,8 +4,6 @@
 #
 
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux/68k Kernel Configuration"
 
arch/mips/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/config.in linux-rwsem/arch/mips/config.in
@@ -68,8 +68,6 @@
    fi
 bool 'Support for Alchemy Semi PB1000 board' CONFIG_MIPS_PB1000
 
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 #
 # Select some configuration options automatically for certain systems.
arch/mips/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig linux-rwsem/arch/mips/defconfig
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_ARC32=y
arch/mips/defconfig-atlas +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-atlas linux-rwsem/arch/mips/defconfig-atlas
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_PCI=y
arch/mips/defconfig-ddb5476 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-ddb5476 linux-rwsem/arch/mips/defconfig-ddb5476
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_ISA=y
arch/mips/defconfig-ddb5477 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-ddb5477 linux-rwsem/arch/mips/defconfig-ddb5477
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_CPU_LITTLE_ENDIAN=y
arch/mips/defconfig-decstation +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-decstation linux-rwsem/arch/mips/defconfig-decstation
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 # CONFIG_ISA is not set
arch/mips/defconfig-ip22 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-ip22 linux-rwsem/arch/mips/defconfig-ip22
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_ARC32=y
arch/mips/defconfig-it8172 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-it8172 linux-rwsem/arch/mips/defconfig-it8172
@@ -37,8 +37,6 @@
 # CONFIG_IT8172_SCR1 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_PCI=y
arch/mips/defconfig-malta +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-malta linux-rwsem/arch/mips/defconfig-malta
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_I8259=y
arch/mips/defconfig-nino +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-nino linux-rwsem/arch/mips/defconfig-nino
@@ -35,8 +35,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_PC_KEYB=y
arch/mips/defconfig-ocelot +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-ocelot linux-rwsem/arch/mips/defconfig-ocelot
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_PCI=y
arch/mips/defconfig-pb1000 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-pb1000 linux-rwsem/arch/mips/defconfig-pb1000
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 CONFIG_MIPS_PB1000=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_MIPS_AU1000=y
arch/mips/defconfig-rm200 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips/defconfig-rm200 linux-rwsem/arch/mips/defconfig-rm200
@@ -32,8 +32,6 @@
 # CONFIG_MIPS_ITE8172 is not set
 # CONFIG_MIPS_IVR is not set
 # CONFIG_MIPS_PB1000 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_MCA is not set
 # CONFIG_SBUS is not set
 CONFIG_ARC32=y
arch/mips64/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/config.in linux-rwsem/arch/mips64/config.in
@@ -27,8 +27,6 @@
 fi
 endmenu
 
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 #
 # Select some configuration options automatically based on user selections
arch/mips64/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/defconfig linux-rwsem/arch/mips64/defconfig
@@ -19,8 +19,6 @@
 # CONFIG_REPLICATE_KTEXT is not set
 # CONFIG_REPLICATE_EXHANDLERS is not set
 CONFIG_SMP=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 CONFIG_BOOT_ELF64=y
 CONFIG_ARC64=y
 CONFIG_COHERENT_IO=y
arch/mips64/defconfig-ip22 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/defconfig-ip22 linux-rwsem/arch/mips64/defconfig-ip22
@@ -12,8 +12,6 @@
 #
 CONFIG_SGI_IP22=y
 # CONFIG_SGI_IP27 is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 CONFIG_BOOT_ELF32=y
 CONFIG_ARC32=y
 CONFIG_BOARD_SCACHE=y
arch/mips64/defconfig-ip27 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/defconfig-ip27 linux-rwsem/arch/mips64/defconfig-ip27
@@ -19,8 +19,6 @@
 # CONFIG_REPLICATE_KTEXT is not set
 # CONFIG_REPLICATE_EXHANDLERS is not set
 CONFIG_SMP=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 CONFIG_BOOT_ELF64=y
 CONFIG_ARC64=y
 CONFIG_COHERENT_IO=y
arch/mips64/defconfig-ip32 +0 -2
diff -uNr linux-2.4.10-pre12/arch/mips64/defconfig-ip32 linux-rwsem/arch/mips64/defconfig-ip32
@@ -13,8 +13,6 @@
 # CONFIG_SGI_IP22 is not set
 # CONFIG_SGI_IP27 is not set
 CONFIG_SGI_IP32=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 CONFIG_BOOT_ELF32=y
 CONFIG_ARC32=y
 CONFIG_PC_KEYB=y
arch/parisc/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/parisc/config.in linux-rwsem/arch/parisc/config.in
@@ -7,8 +7,6 @@
 
 define_bool CONFIG_PARISC y
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/ppc/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/config.in linux-rwsem/arch/ppc/config.in
@@ -4,8 +4,6 @@
 # see Documentation/kbuild/config-language.txt.
 #
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 
 mainmenu_name "Linux/PowerPC Kernel Configuration"
 
arch/ppc/configs/IVMS8_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/IVMS8_defconfig linux-rwsem/arch/ppc/configs/IVMS8_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/SM850_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/SM850_defconfig linux-rwsem/arch/ppc/configs/SM850_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/SPD823TS_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/SPD823TS_defconfig linux-rwsem/arch/ppc/configs/SPD823TS_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/TQM823L_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/TQM823L_defconfig linux-rwsem/arch/ppc/configs/TQM823L_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/TQM850L_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/TQM850L_defconfig linux-rwsem/arch/ppc/configs/TQM850L_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/TQM860L_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/TQM860L_defconfig linux-rwsem/arch/ppc/configs/TQM860L_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/apus_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/apus_defconfig linux-rwsem/arch/ppc/configs/apus_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/bseip_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/bseip_defconfig linux-rwsem/arch/ppc/configs/bseip_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/common_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/common_defconfig linux-rwsem/arch/ppc/configs/common_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/est8260_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/est8260_defconfig linux-rwsem/arch/ppc/configs/est8260_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/gemini_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/gemini_defconfig linux-rwsem/arch/ppc/configs/gemini_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/ibmchrp_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/ibmchrp_defconfig linux-rwsem/arch/ppc/configs/ibmchrp_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/mbx_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/mbx_defconfig linux-rwsem/arch/ppc/configs/mbx_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/oak_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/oak_defconfig linux-rwsem/arch/ppc/configs/oak_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/power3_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/power3_defconfig linux-rwsem/arch/ppc/configs/power3_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/rpxcllf_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/rpxcllf_defconfig linux-rwsem/arch/ppc/configs/rpxcllf_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/rpxlite_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/rpxlite_defconfig linux-rwsem/arch/ppc/configs/rpxlite_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/configs/walnut_defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/configs/walnut_defconfig linux-rwsem/arch/ppc/configs/walnut_defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/ppc/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/ppc/defconfig linux-rwsem/arch/ppc/defconfig
@@ -2,8 +2,6 @@
 # Automatically generated make config: don't edit
 #
 # CONFIG_UID16 is not set
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 
 #
 # Code maturity level options
arch/s390/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/s390/config.in linux-rwsem/arch/s390/config.in
@@ -7,8 +7,6 @@
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux Kernel Configuration"
 define_bool CONFIG_ARCH_S390 y
arch/s390/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/s390/defconfig linux-rwsem/arch/s390/defconfig
@@ -5,8 +5,6 @@
 # CONFIG_EISA is not set
 # CONFIG_MCA is not set
 CONFIG_UID16=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 CONFIG_ARCH_S390=y
 
 #
arch/s390x/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/s390x/config.in linux-rwsem/arch/s390x/config.in
@@ -6,8 +6,6 @@
 define_bool CONFIG_ISA n
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux Kernel Configuration"
 define_bool CONFIG_ARCH_S390 y
arch/s390x/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/s390x/defconfig linux-rwsem/arch/s390x/defconfig
@@ -4,8 +4,6 @@
 # CONFIG_ISA is not set
 # CONFIG_EISA is not set
 # CONFIG_MCA is not set
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 CONFIG_ARCH_S390=y
 CONFIG_ARCH_S390X=y
 
arch/sh/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/sh/config.in linux-rwsem/arch/sh/config.in
@@ -7,8 +7,6 @@
 define_bool CONFIG_SUPERH y
 
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/sparc/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/sparc/config.in linux-rwsem/arch/sparc/config.in
@@ -48,8 +48,6 @@
 define_bool CONFIG_SUN_CONSOLE y
 define_bool CONFIG_SUN_AUXIO y
 define_bool CONFIG_SUN_IO y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 bool 'Support for SUN4 machines (disables SUN4[CDM] support)' CONFIG_SUN4
 if [ "$CONFIG_SUN4" != "y" ]; then
arch/sparc/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/sparc/defconfig linux-rwsem/arch/sparc/defconfig
@@ -38,8 +38,6 @@
 CONFIG_SUN_CONSOLE=y
 CONFIG_SUN_AUXIO=y
 CONFIG_SUN_IO=y
-CONFIG_RWSEM_GENERIC_SPINLOCK=y
-# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
 # CONFIG_SUN4 is not set
 # CONFIG_PCI is not set
 CONFIG_SUN_OPENPROMFS=m
arch/sparc64/config.in +0 -2
diff -uNr linux-2.4.10-pre12/arch/sparc64/config.in linux-rwsem/arch/sparc64/config.in
@@ -33,8 +33,6 @@
 
 # Global things across all Sun machines.
 define_bool CONFIG_HAVE_DEC_LOCK y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 define_bool CONFIG_ISA n
 define_bool CONFIG_ISAPNP n
 define_bool CONFIG_EISA n
arch/sparc64/defconfig +0 -2
diff -uNr linux-2.4.10-pre12/arch/sparc64/defconfig linux-rwsem/arch/sparc64/defconfig
@@ -23,8 +23,6 @@
 CONFIG_SMP=y
 CONFIG_SPARC64=y
 CONFIG_HAVE_DEC_LOCK=y
-# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
-CONFIG_RWSEM_XCHGADD_ALGORITHM=y
 # CONFIG_ISA is not set
 # CONFIG_ISAPNP is not set
 # CONFIG_EISA is not set
drivers/char/sysrq.c +3 -3
diff -uNr linux-2.4.10-pre12/drivers/char/sysrq.c linux-rwsem/drivers/char/sysrq.c
@@ -32,7 +32,7 @@
 
 #include <asm/ptrace.h>
 
-extern void wakeup_bdflush(int);
+extern void wakeup_bdflush();
 extern void reset_vc(unsigned int);
 extern struct list_head super_blocks;
 
@@ -221,7 +221,7 @@
 static void sysrq_handle_sync(int key, struct pt_regs *pt_regs,
 		struct kbd_struct *kbd, struct tty_struct *tty) {
 	emergency_sync_scheduled = EMERG_SYNC;
-	wakeup_bdflush(0);
+	wakeup_bdflush();
 }
 static struct sysrq_key_op sysrq_sync_op = {
 	handler:	sysrq_handle_sync,
@@ -232,7 +232,7 @@
 static void sysrq_handle_mountro(int key, struct pt_regs *pt_regs,
 		struct kbd_struct *kbd, struct tty_struct *tty) {
 	emergency_sync_scheduled = EMERG_REMOUNT;
-	wakeup_bdflush(0);
+	wakeup_bdflush();
 }
 static struct sysrq_key_op sysrq_mountro_op = {
 	handler:	sysrq_handle_mountro,
include/asm-alpha/rwsem.h +0 -208

208 lines not on this page · raw message

include/asm-i386/rwsem.h +0 -226

226 lines not on this page · raw message

include/asm-ppc/rwsem.h +0 -137

137 lines not on this page · raw message

include/asm-sparc64/rwsem.h +0 -233

233 lines not on this page · raw message

include/linux/rwsem-spinlock.h +0 -62

62 lines not on this page · raw message

include/linux/rwsem.h +55 -29

122 lines not on this page · raw message

lib/Makefile +2 -5
diff -uNr linux-2.4.10-pre12/lib/Makefile linux-rwsem/lib/Makefile
@@ -8,12 +8,9 @@
 
 L_TARGET := lib.a
 
-export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o
+export-objs := cmdline.o dec_and_lock.o rwsem.o
 
-obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o
-
-obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o
-obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
+obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o rwsem.o
 
 ifneq ($(CONFIG_HAVE_DEC_LOCK),y) 
   obj-y += dec_and_lock.o
lib/rwsem-spinlock.c +0 -239

239 lines not on this page · raw message

lib/rwsem.c +38 -155

241 lines not on this page · raw message

#21 lore raw
In reply to David Howells
Quoted 2 lines · David Howells

Here's a patch to make rwsems unfair.

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even allows more than 2^31 concurrent readers on 64 bit archs ;).

just apply this patch on top of pre12:

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre11aa1/00_rwsem-19

and then I'll send another incremental patch to kill the XCHGADD garbage. (but this patch will be enough to provide efficient and unfair rwsem to all architectures, so fixing the deadlock)

I inlined the patch below (may need some conversion to likely/unlikely as well).

arch/alpha/config.in +1 -2
diff -urN rwsem-ref/arch/alpha/config.in rwsem/arch/alpha/config.in
@@ -5,8 +5,7 @@
 
 define_bool CONFIG_ALPHA y
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
+define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
 
 mainmenu_name "Kernel configuration of Linux for Alpha machines"
 
arch/arm/config.in +0 -2
diff -urN rwsem-ref/arch/arm/config.in rwsem/arch/arm/config.in
@@ -10,8 +10,6 @@
 define_bool CONFIG_MCA n
 define_bool CONFIG_UID16 y
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
-
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/cris/config.in +0 -1
diff -urN rwsem-ref/arch/cris/config.in rwsem/arch/cris/config.in
@@ -6,7 +6,6 @@
 
 define_bool CONFIG_UID16 y
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/i386/config.in +1 -3
diff -urN rwsem-ref/arch/i386/config.in rwsem/arch/i386/config.in
@@ -51,7 +51,6 @@
    define_bool CONFIG_X86_XADD n
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
    define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 else
    define_bool CONFIG_X86_WP_WORKS_OK y
    define_bool CONFIG_X86_INVLPG y
@@ -59,8 +58,7 @@
    define_bool CONFIG_X86_XADD y
    define_bool CONFIG_X86_BSWAP y
    define_bool CONFIG_X86_POPAD_OK y
-   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
+   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
 fi
 if [ "$CONFIG_M486" = "y" ]; then
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
arch/ia64/config.in +0 -1
diff -urN rwsem-ref/arch/ia64/config.in rwsem/arch/ia64/config.in
@@ -24,7 +24,6 @@
 define_bool CONFIG_MCA n
 define_bool CONFIG_SBUS n
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 if [ "$CONFIG_IA64_HP_SIM" = "n" ]; then
   define_bool CONFIG_ACPI y
arch/m68k/config.in +0 -1
diff -urN rwsem-ref/arch/m68k/config.in rwsem/arch/m68k/config.in
@@ -5,7 +5,6 @@
 
 define_bool CONFIG_UID16 y
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux/68k Kernel Configuration"
 
arch/mips/config.in +0 -1
diff -urN rwsem-ref/arch/mips/config.in rwsem/arch/mips/config.in
@@ -69,7 +69,6 @@
 bool 'Support for Alchemy Semi PB1000 board' CONFIG_MIPS_PB1000
 
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 #
 # Select some configuration options automatically for certain systems.
arch/mips64/config.in +0 -1
diff -urN rwsem-ref/arch/mips64/config.in rwsem/arch/mips64/config.in
@@ -28,7 +28,6 @@
 endmenu
 
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 #
 # Select some configuration options automatically based on user selections
arch/parisc/config.in +0 -1
diff -urN rwsem-ref/arch/parisc/config.in rwsem/arch/parisc/config.in
@@ -8,7 +8,6 @@
 define_bool CONFIG_PARISC y
 define_bool CONFIG_UID16 n
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/ppc/config.in +1 -2
diff -urN rwsem-ref/arch/ppc/config.in rwsem/arch/ppc/config.in
@@ -4,8 +4,7 @@
 # see Documentation/kbuild/config-language.txt.
 #
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
+define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
 
 mainmenu_name "Linux/PowerPC Kernel Configuration"
 
arch/s390/config.in +0 -1
diff -urN rwsem-ref/arch/s390/config.in rwsem/arch/s390/config.in
@@ -8,7 +8,6 @@
 define_bool CONFIG_MCA n
 define_bool CONFIG_UID16 y
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux Kernel Configuration"
 define_bool CONFIG_ARCH_S390 y
arch/s390x/config.in +0 -1
diff -urN rwsem-ref/arch/s390x/config.in rwsem/arch/s390x/config.in
@@ -7,7 +7,6 @@
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux Kernel Configuration"
 define_bool CONFIG_ARCH_S390 y
arch/sh/config.in +0 -1
diff -urN rwsem-ref/arch/sh/config.in rwsem/arch/sh/config.in
@@ -8,7 +8,6 @@
 
 define_bool CONFIG_UID16 y
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/sparc/config.in +0 -1
diff -urN rwsem-ref/arch/sparc/config.in rwsem/arch/sparc/config.in
@@ -49,7 +49,6 @@
 define_bool CONFIG_SUN_AUXIO y
 define_bool CONFIG_SUN_IO y
 define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 bool 'Support for SUN4 machines (disables SUN4[CDM] support)' CONFIG_SUN4
 if [ "$CONFIG_SUN4" != "y" ]; then
arch/sparc64/config.in +2 -2
diff -urN rwsem-ref/arch/sparc64/config.in rwsem/arch/sparc64/config.in
@@ -33,8 +33,8 @@
 
 # Global things across all Sun machines.
 define_bool CONFIG_HAVE_DEC_LOCK y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
+# sorry I broke it again
+define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
 define_bool CONFIG_ISA n
 define_bool CONFIG_ISAPNP n
 define_bool CONFIG_EISA n
include/asm-alpha/rwsem_xchgadd.h +27 -0
diff -urN rwsem-ref/include/asm-alpha/rwsem_xchgadd.h rwsem/include/asm-alpha/rwsem_xchgadd.h
@@ -0,0 +1,27 @@
+#ifndef _ALPHA_RWSEM_XCHGADD_H
+#define _ALPHA_RWSEM_XCHGADD_H
+
+/* WRITEME */
+
+static inline void __down_read(struct rw_semaphore *sem)
+{
+}
+
+static inline void __down_write(struct rw_semaphore *sem)
+{
+}
+
+static inline void __up_read(struct rw_semaphore *sem)
+{
+}
+
+static inline void __up_write(struct rw_semaphore *sem)
+{
+}
+
+static inline long rwsem_xchgadd(long value, long * count)
+{
+	return value;
+}
+
+#endif
include/asm-i386/rwsem.h +0 -226

226 lines not on this page · raw message

include/asm-i386/rwsem_xchgadd.h +93 -0

93 lines not on this page · raw message

include/linux/rwsem-spinlock.h +0 -62

62 lines not on this page · raw message

include/linux/rwsem.h +5 -66

85 lines not on this page · raw message

include/linux/rwsem_spinlock.h +62 -0

62 lines not on this page · raw message

include/linux/rwsem_xchgadd.h +104 -0

104 lines not on this page · raw message

include/linux/sched.h +1 -1
diff -urN rwsem-ref/include/linux/sched.h rwsem/include/linux/sched.h
@@ -239,7 +239,7 @@
 	pgd:		swapper_pg_dir, 		\
 	mm_users:	ATOMIC_INIT(2), 		\
 	mm_count:	ATOMIC_INIT(1), 		\
-	mmap_sem:	__RWSEM_INITIALIZER(name.mmap_sem), \
+	mmap_sem:	RWSEM_INITIALIZER(name.mmap_sem), \
 	page_table_lock: SPIN_LOCK_UNLOCKED, 		\
 	mmlist:		LIST_HEAD_INIT(name.mmlist),	\
 }
lib/Makefile +3 -3
diff -urN rwsem-ref/lib/Makefile rwsem/lib/Makefile
@@ -8,12 +8,12 @@
 
 L_TARGET := lib.a
 
-export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o
+export-objs := cmdline.o dec_and_lock.o rwsem_spinlock.o rwsem_xchgadd.o
 
 obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o
 
-obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o
-obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
+obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem_spinlock.o
+obj-$(CONFIG_RWSEM_XCHGADD) += rwsem_xchgadd.o
 
 ifneq ($(CONFIG_HAVE_DEC_LOCK),y) 
   obj-y += dec_and_lock.o
lib/rwsem.c +0 -210

210 lines not on this page · raw message

lib/rwsem_spinlock.c +126 -0

126 lines not on this page · raw message

lib/rwsem_xchgadd.c +92 -0

92 lines not on this page · raw message

Andrea

#27 lore raw
In reply to David Howells
Quoted 2 lines · Andrea Arcangeli

I inlined the patch below (may need some conversion to likely/unlikely as well).

I don't know about you, but I'm mildly concerned that copyright attributions vanished.

-ben

#22 lore raw
In reply to Andrea Arcangeli
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one

Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.

Quoted 1 line

(it even allows more than 2^31 concurrent readers on 64 bit archs ;).

Easy enough to fix. Just apply this as well:

include/linux/rwsem.h Wed Sep 19 19:23:47 2001 renamed from include/linux/rwsem.h Wed Sep 19 19:23:44 2001 +1 -1
@@ -26,7 +26,7 @@
  * - if wait_list is not empty, then there are processes waiting for the semaphore
  */
 struct rw_semaphore {
-       int                     activity;
+       long                    activity;
        spinlock_t              lock;
        struct list_head        wait_list;
 };

David

#30 lore raw
In reply to David Howells
Quoted 2 lines · Benjamin LaHaise

I don't know about you, but I'm mildly concerned that copyright attributions vanished.

can you be more precise? which copyright attribution? my rwsem are written totally from scratch, they doesn't share anything with the previous code, it is just a code replacement. As far I can tell deleted some file, and with it also its headers but that seems not to infinge any copyright, I didn't removed only the copyright attribution. If I removed only the copyright attribution please let me know of course, that would been a silly mistake.

To be pedantic the only idea I shared with the old code (but that's just the idea, not the implementation, so AFIK only a patent on such idea could protect it from its free usage usage) is to return the rwsem again from rwsem_wake and friends to avoid saving it in the asm slow path, and I written that:

/*
* We return the semaphore itself from the C functions so we can pass it
* in %eax via regparm and we don't need to declare %eax clobbered by C.
* This is mostly for x86 but maybe other archs can make a use of it
* too.
* Idea is from David Howells <[email protected]>.
*/

And the xadd version is scheduled for removal anyways soon btw (David just dropped it in its implementation).

Andrea

#23 lore raw
In reply to David Howells
Quoted 4 lines · David Howells
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one

Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.

then why your microbenchmarks says my version is much faster?

Andrea

#29 lore raw
In reply to David Howells
Quoted 5 lines · David Howells
Quoted 3 lines

I don't know about you, but I'm mildly concerned that copyright attributions vanished.

I concur with that.

can you be a little more accurate on which copyright attributions vanished by mistake?

Andrea

#31 lore raw
In reply to David Howells
Quoted 5 lines · Andrea Arcangeli

To be pedantic the only idea I shared with the old code (but that's just the idea, not the implementation, so AFIK only a patent on such idea could protect it from its free usage usage) is to return the rwsem again from rwsem_wake and friends to avoid saving it in the asm slow path, and I written that:

Your patch moved a bunch of code into asm-i386/rwsem_xchgadd.h. That code was derived from the spinlock code by me into the first rwsems, then David reworked bits of it, as wel as you. But there is no copyright on that file indicating this heritage. If you look at how strict commercial copyright control can be, even copying a single line of code mentally by retyping it can still mandate the copyright legacy. I'm sure it's just an oversight, but it's probably one we *all* need to be reminded of every now and again.

-ben

#32 lore raw
In reply to David Howells
Quoted 7 lines · Benjamin LaHaise
Quoted 5 lines · Andrea Arcangeli

To be pedantic the only idea I shared with the old code (but that's just the idea, not the implementation, so AFIK only a patent on such idea could protect it from its free usage usage) is to return the rwsem again from rwsem_wake and friends to avoid saving it in the asm slow path, and I written that:

Quoted 8 lines

Your patch moved a bunch of code into asm-i386/rwsem_xchgadd.h. That code was derived from the spinlock code by me into the first rwsems, then David reworked bits of it, as wel as you. But there is no copyright on that file indicating this heritage. If you look at how strict commercial copyright control can be, even copying a single line of code mentally by retyping it can still mandate the copyright legacy. I'm sure it's just an oversight, but it's probably one we *all* need to be reminded of every now and again.

I recall I wrote such code without copying anything, I certainly copied the way of doing things (pushl dx,cx and save ax via returning it in the slow path) as I wrote in the comment, but not the code itself. Infact at first i was probably also pushing eax, and even now that I use the same logic the two versions should be slightly different. and I think mine has a race this is why I'm not using such code and with the unfair thing such code is scheduled for removal since it's unfixable. really it sounds like too me too I cut and pasted the static inline void __down_read(struct rw_semaphore *sem) declarations, since I tend to left a space between "*" and the variable name but not always.

Andrea

#33 lore raw

PATCH 1/1 attempt #2 (Re: Deadlock on the mm->mmap_sem)

In reply to David Howells

Here's a revised patch to make rwsems unfair. This makes the activity counter a long (so it can handle up to 2^63 readers on a 64-bit arch) and fixes a small bug in the lib/rwsem.c file. This now passes all the rwsem tests that I've got.

David

#24 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Andrea Arcangeli
Quoted 4 lines · David Howells
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one

Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.

then why your microbenchmarks says my version is much faster?

They don't:

[my rwsem-unfair-2.diff patch]

#rds #wrs time :  rd granted [spread]  wr granted [spread]
==== ==== =====   ===================  ===================
   1    0   10s:    25384639 [0.008%]           0 [0.000%]
   2    0   10s:     8007388 [0.013%]           0 [0.000%]
   4    0   10s:     8004833 [0.007%]           0 [0.000%]
   0    1   10s:           0 [0.000%]    25717288 [0.009%]
   0    4   10s:           0 [0.000%]     2817293 [2.323%]
   4    2   10s:    12742523 [0.008%]         114 [5.394%]
  30    1   10s:    12739163 [0.002%]          92 [1.739%]
  30   15   50s:    63733591 [0.003%]         286 [2.711%]

[Andrea's 00_rwsem-19 patch]

#rds #wrs time :  rd granted [spread]  wr granted [spread]
==== ==== =====   ===================  ===================
   1    0   10s:    24577938 [0.003%]           0 [0.000%]
   2    0   10s:     5631224 [0.016%]           0 [0.000%]
   4    0   10s:     5633280 [0.009%]           0 [0.000%]
   0    1   10s:           0 [0.000%]    24572785 [0.007%]
   0    4   10s:           0 [0.000%]     3444786 [0.091%]
   4    2   10s:     8036291 [0.008%]         122 [4.918%]
  30    1   10s:     8036490 [0.006%]          89 [1.253%]
  30   15   50s:    40200472 [0.001%]         265 [2.717%]

[Unpatched kernel]

#rds #wrs time :  rd granted [spread]  wr granted [spread]
==== ==== =====   ===================  ===================
   1    0   10s:    30118903 [0.008%]           0 [0.000%]
   2    0   10s:    12543737 [0.007%]           0 [0.000%]
   4    0   10s:    12543731 [0.006%]           0 [0.000%]
   0    1   10s:           0 [0.000%]    28408147 [0.008%]
   0    4   10s:           0 [0.000%]     1554495 [0.172%]
   4    2   10s:     1119656 [0.076%]      560020 [0.077%]
  30    1   10s:     5361923 [0.024%]       55787 [0.012%]
  30   15   50s:     5322473 [0.049%]     2661621 [0.050%]

David

#25 lore raw
Quoted 11 lines · David Howells
Quoted 7 lines · Andrea Arcangeli
Quoted 4 lines · David Howells
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one

Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.

then why your microbenchmarks says my version is much faster?

They don't:

ok, so I drop my objections, you must have changed something radical since the last time I benchmarked your generic rwsemaphores, I must have missed you did those changes, sorry, so I'll just drop my rwsem patch and I'll apply yours.

Andrea

#26 lore raw
In reply to David Howells
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · David Howells
Quoted 4 lines · Andrea Arcangeli
Quoted 3 lines · David Howells
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Not so:-) Your patch, Andrea, grabs the spinlock far more than is necessary.

then why your microbenchmarks says my version is much faster?

They don't:

ok, so I drop my objections, you must have changed something radical

hey no! It was a very unfair comparison! my rwsem are non inlined, yours are.

Andrea

#34 lore raw
In reply to David Howells
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even

allows

Quoted 2 lines

more than 2^31 concurrent readers on 64 bit archs ;).

Andrea,

implementing recursive semaphores is trivial, but do you have any idea how to fix the latency problem?
Multithreaded, iobound apps that use mmap are unusable with unfair semaphores.

My test app hangs for minutes within mprotect().

--
Manfred

#19 lore raw
In reply to David Howells
Quoted 4 lines

I also don't think the hack is that bad. All it's doing is taking a copy of the process's VM decription so that it knows that nobody is going to modify it whilst a coredump is in progress.

You break the locking scheme of the mm structure.
Right now the rules are

1 get a mm_struct pointer by whatever means (walk the process list and

    read task->mm, walk the mm_list)

2 increase mm_users
3 release the spinlock you acquired for 1
4 you can do with the result what you want.

With your patch applied, we would have to restrict rule 4 - at least modifying the vma list is not possible anymore, probably further changes.
AFAIK right now no external mm_struct user modifies the vma list, but it could be a problem in the future.

Quoted 6 lines

However, if you don't like that, how about just changing the lock on mm_struct to a special mm_struct-only type lock that has a recursive lock operation for use by the pagefault handler (and _only_ the pagefault handler)? I've attached a patch to do just that. This introduces five operations:

Does that solve the latency problem? That problem is pagefaults vs. another operation.

--
Manfred

#42 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

If the choice is between a hack to do strange and incomprehensible things for a special case, and just making the semaphores do the same thing rw-spinlocks do and make the problem go away naturally, Ill take #2 any day. The patches already exist, after all.

I've attached a recursive semaphore patch against 2.4.10-pre11 - but it makes mmap io unusable:

Testcase:
* file, mmaped, 300 MB, 128 MB Ram
* 2 threads: touch random pages
* third thread: calls mprotect(0xFFFF00000,0x1000, PAGE_READ)

Result:
mprotect hangs forever (minutes) with recursive semaphores.

With fair semaphores, mprotect returns after ~80 milliseconds with 5 worker threads, after ~380 milliseconds with 20 worker threads (slow IDE disk).

One alternative to David's patch would be moving the locking into the coredump handlers - would you prefer that?

--
Manfred

lib/rwsem-spinlock.c +1 -1
@@ -115,7 +115,7 @@
 
 	spin_lock(&sem->wait_lock);
 
-	if (sem->activity>=0 && list_empty(&sem->wait_list)) {
+	if (sem->activity>=0) {
 		/* granted */
 		sem->activity++;
 		spin_unlock(&sem->wait_lock);
arch/i386/config.in +2 -2
@@ -59,8 +59,8 @@
    define_bool CONFIG_X86_XADD y
    define_bool CONFIG_X86_BSWAP y
    define_bool CONFIG_X86_POPAD_OK y
-   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
+   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
+   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 fi
 if [ "$CONFIG_M486" = "y" ]; then
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
#35 lore raw
In reply to David Howells
Quoted 9 lines · Manfred Spraul
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even

allows

Quoted 2 lines

more than 2^31 concurrent readers on 64 bit archs ;).

Andrea,

implementing recursive semaphores is trivial, but do you have any idea how to fix the latency problem?

yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a mm_down_read macro. such way it will be recursive _and_ fair.

Andrea

#40 lore raw
In reply to David Howells
Quoted 8 lines · Andrea Arcangeli
Quoted 7 lines · Manfred Spraul
Quoted 2 lines

if we go generic then I strongly recommend my version of the generic semaphores is _much_ faster (and cleaner) than this one (it even

allows

Quoted 2 lines

more than 2^31 concurrent readers on 64 bit archs ;).

Andrea,

Quoted 7 lines
Quoted 2 lines

implementing recursive semaphores is trivial, but do you have any idea how to fix the latency problem?

yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a mm_down_read macro. such way it will be recursive _and_ fair.

ok, here it is attached a rwsem patch (now with the fast path inlined :) of my version of the rwsem-spinlock semaphores (as only option all over the ports). It's against pre12 (it is not inlined in the email since it's not readable anyways)

and below you find (this time inlined) an incremental patch to be applied on top of the attachment that implements the read recursive and at the same time fair rw semaphores. This should close all the problems.

But keep in mind the rwsem by default are _fair_, so you cannot do read recursion unless you use the recursive version and you pass a rw_sem_recursor to it.

arch/alpha/mm/fault.c +4 -4
diff -urN rwsem/arch/alpha/mm/fault.c rwsem-recurisve/arch/alpha/mm/fault.c
@@ -113,7 +113,7 @@
 		goto vmalloc_fault;
 #endif
 
-	down_read(&mm->mmap_sem);
+	down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -147,7 +147,7 @@
 	 * the fault.
 	 */
 	fault = handle_mm_fault(mm, vma, address, cause > 0);
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 
 	if (fault < 0)
 		goto out_of_memory;
@@ -161,7 +161,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 
 	if (user_mode(regs)) {
 		force_sig(SIGSEGV, current);
@@ -198,7 +198,7 @@
 	if (current->pid == 1) {
 		current->policy |= SCHED_YIELD;
 		schedule();
-		down_read(&mm->mmap_sem);
+		down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 		goto survive;
 	}
 	printk(KERN_ALERT "VM: killing process %s(%d)\n",
arch/i386/mm/fault.c +6 -6
diff -urN rwsem/arch/i386/mm/fault.c rwsem-recurisve/arch/i386/mm/fault.c
@@ -191,7 +191,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 
 	vma = find_vma(mm, address);
 	if (!vma)
@@ -265,7 +265,7 @@
 		if (bit < 32)
 			tsk->thread.screen_bitmap |= 1 << bit;
 	}
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	return;
 
 /*
@@ -273,7 +273,7 @@
  * Fix it, but check if it's kernel or user first..
  */
 bad_area:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 
 	/* User mode accesses just cause a SIGSEGV */
 	if (error_code & 4) {
@@ -341,11 +341,11 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	if (tsk->pid == 1) {
 		tsk->policy |= SCHED_YIELD;
 		schedule();
-		down_read(&mm->mmap_sem);
+		down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 		goto survive;
 	}
 	printk("VM: killing process %s\n", tsk->comm);
@@ -354,7 +354,7 @@
 	goto no_context;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 
 	/*
 	 * Send a sigbus, regardless of whether we were in kernel
arch/ia64/mm/fault.c +4 -4
diff -urN rwsem/arch/ia64/mm/fault.c rwsem-recurisve/arch/ia64/mm/fault.c
@@ -60,7 +60,7 @@
 	if (in_interrupt() || !mm)
 		goto no_context;
 
-	down_read(&mm->mmap_sem);
+	down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 
 	vma = find_vma_prev(mm, address, &prev_vma);
 	if (!vma)
@@ -112,7 +112,7 @@
 	      default:
 		goto out_of_memory;
 	}
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	return;
 
   check_expansion:
@@ -135,7 +135,7 @@
 	goto good_area;
 
   bad_area:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	if (isr & IA64_ISR_SP) {
 		/*
 		 * This fault was due to a speculative load set the "ed" bit in the psr to
@@ -184,7 +184,7 @@
 	return;
 
   out_of_memory:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	printk("VM: killing process %s\n", current->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
arch/ppc/mm/fault.c +5 -5
diff -urN rwsem/arch/ppc/mm/fault.c rwsem-recurisve/arch/ppc/mm/fault.c
@@ -103,7 +103,7 @@
 		bad_page_fault(regs, address, SIGSEGV);
 		return;
 	}
-	down_read(&mm->mmap_sem);
+	down_read(&mm->mmap_sem, &current->mm_recursor);
 	vma = find_vma(mm, address);
 	if (!vma)
 		goto bad_area;
@@ -163,7 +163,7 @@
                 goto out_of_memory;
 	}
 
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	/*
 	 * keep track of tlb+htab misses that are good addrs but
 	 * just need pte's created via handle_mm_fault()
@@ -173,7 +173,7 @@
 	return;
 
 bad_area:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	pte_errors++;	
 
 	/* User mode accesses cause a SIGSEGV */
@@ -194,7 +194,7 @@
  * us unable to handle the page fault gracefully.
  */
 out_of_memory:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	printk("VM: killing process %s\n", current->comm);
 	if (user_mode(regs))
 		do_exit(SIGKILL);
@@ -202,7 +202,7 @@
 	return;
 
 do_sigbus:
-	up_read(&mm->mmap_sem);
+	up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
 	info.si_signo = SIGBUS;
 	info.si_errno = 0;
 	info.si_code = BUS_ADRERR;
fs/exec.c +2 -2
diff -urN rwsem/fs/exec.c rwsem-recurisve/fs/exec.c
@@ -969,9 +969,9 @@
 	if (do_truncate(file->f_dentry, 0) != 0)
 		goto close_fail;
 
-	down_read(&current->mm->mmap_sem);
+	down_read_recursive(&current->mm->mmap_sem, &current->mm_recursor);
 	retval = binfmt->core_dump(signr, regs, file);
-	up_read(&current->mm->mmap_sem);
+	up_read_recursive(&current->mm->mmap_sem, &current->mm_recursor);
 
 close_fail:
 	filp_close(file, NULL);
fs/proc/array.c +8 -2
diff -urN rwsem/fs/proc/array.c rwsem-recurisve/fs/proc/array.c
@@ -577,7 +577,10 @@
 	column = *ppos & (MAPS_LINE_LENGTH-1);
 
 	/* quickly go to line lineno */
-	down_read(&mm->mmap_sem);
+	if (task == current)
+		down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
+	else
+		down_read(&mm->mmap_sem);
 	for (map = mm->mmap, i = 0; map && (i < lineno); map = map->vm_next, i++)
 		continue;
 
@@ -658,7 +661,10 @@
 		if (volatile_task)
 			break;
 	}
-	up_read(&mm->mmap_sem);
+	if (task == current)
+		up_read_recursive(&mm->mmap_sem, &current->mm_recursor);
+	else
+		up_read(&mm->mmap_sem);
 
 	/* encode f_pos */
 	*ppos = (lineno << MAPS_LINE_SHIFT) + column;
include/linux/rwsem.h +34 -0
diff -urN rwsem/include/linux/rwsem.h rwsem-recurisve/include/linux/rwsem.h
@@ -18,6 +18,11 @@
 #endif
 };
 
+struct rw_sem_recursor
+{
+	int counter;
+};
+
 #if RWSEM_DEBUG
 #define __SEM_DEBUG_INIT(name) \
 	, (long)&(name).__magic
@@ -42,6 +47,7 @@
 	__SEM_DEBUG_INIT(name)			\
 }
 #define RWSEM_INITIALIZER(name) __RWSEM_INITIALIZER(name, 0)
+#define RWSEM_RECURSOR_INITIALIZER ((struct rw_sem_recursor) { 0, })
 
 #define __DECLARE_RWSEM(name, count) \
 	struct rw_semaphore name = __RWSEM_INITIALIZER(name, count)
@@ -112,6 +118,34 @@
 	spin_lock(&sem->lock);
 	sem->count -= RWSEM_WRITE_BIAS;
 	if (unlikely(sem->count))
+		rwsem_wake(sem);
+	spin_unlock(&sem->lock);
+}
+
+static inline void down_read_recursive(struct rw_semaphore *sem,
+				       struct rw_sem_recursor * recursor)
+{
+	int count, counter;
+	CHECK_MAGIC(sem->__magic);
+
+	spin_lock(&sem->lock);
+	count = sem->count;
+	sem->count += RWSEM_READ_BIAS;
+	counter = recursor->counter++;
+	if (unlikely(count < 0 && !counter && !(count & RWSEM_READ_MASK)))
+		rwsem_down_failed(sem, RWSEM_READ_BLOCKING_BIAS);
+	spin_unlock(&sem->lock);
+}
+
+static inline void up_read_recursive(struct rw_semaphore *sem,
+				     struct rw_sem_recursor * recursor)
+{
+	CHECK_MAGIC(sem->__magic);
+
+	spin_lock(&sem->lock);
+	sem->count -= RWSEM_READ_BIAS;
+	recursor->counter--;
+	if (unlikely(sem->count < 0 && !(sem->count & RWSEM_READ_MASK)))
 		rwsem_wake(sem);
 	spin_unlock(&sem->lock);
 }
include/linux/sched.h +2 -0
diff -urN rwsem/include/linux/sched.h rwsem-recurisve/include/linux/sched.h
@@ -315,6 +315,7 @@
 
 	struct task_struct *next_task, *prev_task;
 	struct mm_struct *active_mm;
+	struct rw_sem_recursor mm_recursor;
 	struct list_head local_pages;
 	unsigned int allocation_order, nr_local_pages;
 
@@ -460,6 +461,7 @@
     policy:		SCHED_OTHER,					\
     mm:			NULL,						\
     active_mm:		&init_mm,					\
+    mm_recursor:	RWSEM_RECURSOR_INITIALIZER,			\
     cpus_allowed:	-1,						\
     run_list:		LIST_HEAD_INIT(tsk.run_list),			\
     next_task:		&tsk,						\

Andrea

arch/alpha/config.in +0 -2
diff -urN 2.4.10pre12/arch/alpha/config.in rwsem/arch/alpha/config.in
@@ -5,8 +5,6 @@
 
 define_bool CONFIG_ALPHA y
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 
 mainmenu_name "Kernel configuration of Linux for Alpha machines"
 
arch/arm/config.in +0 -2
diff -urN 2.4.10pre12/arch/arm/config.in rwsem/arch/arm/config.in
@@ -9,8 +9,6 @@
 define_bool CONFIG_SBUS n
 define_bool CONFIG_MCA n
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 
 mainmenu_option next_comment
arch/cris/config.in +0 -2
diff -urN 2.4.10pre12/arch/cris/config.in rwsem/arch/cris/config.in
@@ -5,8 +5,6 @@
 mainmenu_name "Linux/CRIS Kernel Configuration"
 
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/i386/config.in +0 -4
diff -urN 2.4.10pre12/arch/i386/config.in rwsem/arch/i386/config.in
@@ -50,8 +50,6 @@
    define_bool CONFIG_X86_CMPXCHG n
    define_bool CONFIG_X86_XADD n
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
-   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 else
    define_bool CONFIG_X86_WP_WORKS_OK y
    define_bool CONFIG_X86_INVLPG y
@@ -59,8 +57,6 @@
    define_bool CONFIG_X86_XADD y
    define_bool CONFIG_X86_BSWAP y
    define_bool CONFIG_X86_POPAD_OK y
-   define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-   define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 fi
 if [ "$CONFIG_M486" = "y" ]; then
    define_int  CONFIG_X86_L1_CACHE_SHIFT 4
arch/ia64/config.in +0 -2
diff -urN 2.4.10pre12/arch/ia64/config.in rwsem/arch/ia64/config.in
@@ -23,8 +23,6 @@
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
 define_bool CONFIG_SBUS n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 if [ "$CONFIG_IA64_HP_SIM" = "n" ]; then
   define_bool CONFIG_ACPI y
arch/m68k/config.in +0 -2
diff -urN 2.4.10pre12/arch/m68k/config.in rwsem/arch/m68k/config.in
@@ -4,8 +4,6 @@
 #
 
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux/68k Kernel Configuration"
 
arch/mips/config.in +0 -2
diff -urN 2.4.10pre12/arch/mips/config.in rwsem/arch/mips/config.in
@@ -68,8 +68,6 @@
    fi
 bool 'Support for Alchemy Semi PB1000 board' CONFIG_MIPS_PB1000
 
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 #
 # Select some configuration options automatically for certain systems.
arch/mips64/config.in +0 -3
diff -urN 2.4.10pre12/arch/mips64/config.in rwsem/arch/mips64/config.in
@@ -27,9 +27,6 @@
 fi
 endmenu
 
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
-
 #
 # Select some configuration options automatically based on user selections
 #
arch/parisc/config.in +0 -2
diff -urN 2.4.10pre12/arch/parisc/config.in rwsem/arch/parisc/config.in
@@ -7,8 +7,6 @@
 
 define_bool CONFIG_PARISC y
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/ppc/config.in +0 -2
diff -urN 2.4.10pre12/arch/ppc/config.in rwsem/arch/ppc/config.in
@@ -4,8 +4,6 @@
 # see Documentation/kbuild/config-language.txt.
 #
 define_bool CONFIG_UID16 n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 
 mainmenu_name "Linux/PowerPC Kernel Configuration"
 
arch/s390/config.in +0 -2
diff -urN 2.4.10pre12/arch/s390/config.in rwsem/arch/s390/config.in
@@ -7,8 +7,6 @@
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux Kernel Configuration"
 define_bool CONFIG_ARCH_S390 y
arch/s390x/config.in +0 -2
diff -urN 2.4.10pre12/arch/s390x/config.in rwsem/arch/s390x/config.in
@@ -6,8 +6,6 @@
 define_bool CONFIG_ISA n
 define_bool CONFIG_EISA n
 define_bool CONFIG_MCA n
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_name "Linux Kernel Configuration"
 define_bool CONFIG_ARCH_S390 y
arch/sh/config.in +0 -2
diff -urN 2.4.10pre12/arch/sh/config.in rwsem/arch/sh/config.in
@@ -7,8 +7,6 @@
 define_bool CONFIG_SUPERH y
 
 define_bool CONFIG_UID16 y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 mainmenu_option next_comment
 comment 'Code maturity level options'
arch/sparc/config.in +0 -2
diff -urN 2.4.10pre12/arch/sparc/config.in rwsem/arch/sparc/config.in
@@ -48,8 +48,6 @@
 define_bool CONFIG_SUN_CONSOLE y
 define_bool CONFIG_SUN_AUXIO y
 define_bool CONFIG_SUN_IO y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK y
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM n
 
 bool 'Support for SUN4 machines (disables SUN4[CDM] support)' CONFIG_SUN4
 if [ "$CONFIG_SUN4" != "y" ]; then
arch/sparc64/config.in +0 -2
diff -urN 2.4.10pre12/arch/sparc64/config.in rwsem/arch/sparc64/config.in
@@ -33,8 +33,6 @@
 
 # Global things across all Sun machines.
 define_bool CONFIG_HAVE_DEC_LOCK y
-define_bool CONFIG_RWSEM_GENERIC_SPINLOCK n
-define_bool CONFIG_RWSEM_XCHGADD_ALGORITHM y
 define_bool CONFIG_ISA n
 define_bool CONFIG_ISAPNP n
 define_bool CONFIG_EISA n
include/asm-alpha/rwsem.h +0 -208

208 lines not on this page · raw message

include/asm-i386/rwsem.h +0 -226

226 lines not on this page · raw message

include/linux/rwsem-spinlock.h +0 -62

62 lines not on this page · raw message

include/linux/rwsem.h +88 -48

168 lines not on this page · raw message

include/linux/sched.h +1 -1
diff -urN 2.4.10pre12/include/linux/sched.h rwsem/include/linux/sched.h
@@ -239,7 +239,7 @@
 	pgd:		swapper_pg_dir, 		\
 	mm_users:	ATOMIC_INIT(2), 		\
 	mm_count:	ATOMIC_INIT(1), 		\
-	mmap_sem:	__RWSEM_INITIALIZER(name.mmap_sem), \
+	mmap_sem:	RWSEM_INITIALIZER(name.mmap_sem), \
 	page_table_lock: SPIN_LOCK_UNLOCKED, 		\
 	mmlist:		LIST_HEAD_INIT(name.mmlist),	\
 }
lib/Makefile +2 -5
diff -urN 2.4.10pre12/lib/Makefile rwsem/lib/Makefile
@@ -8,12 +8,9 @@
 
 L_TARGET := lib.a
 
-export-objs := cmdline.o dec_and_lock.o rwsem-spinlock.o rwsem.o
+export-objs := cmdline.o dec_and_lock.o rwsem.o
 
-obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o
-
-obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o
-obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
+obj-y := errno.o ctype.o string.o vsprintf.o brlock.o cmdline.o bust_spinlocks.o rbtree.o rwsem.o
 
 ifneq ($(CONFIG_HAVE_DEC_LOCK),y) 
   obj-y += dec_and_lock.o
lib/rwsem-spinlock.c +0 -239

239 lines not on this page · raw message

lib/rwsem.c +43 -190

253 lines not on this page · raw message

#36 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · Andrea Arcangeli

yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a mm_down_read macro. such way it will be recursive _and_ fair.

You'd need a counter per-process per-mm_struct. Otherwise you couldn't do a recursive read lock simultaneously in two or more different processes, and also allow any one process to lock multiple mm_structs.

David

#37 lore raw
Quoted 10 lines · David Howells
Quoted 4 lines · Andrea Arcangeli

yes, one solution to the latency problem without writing the ugly code would be simply to add a per-process counter to pass to a modified rwsem api, then to hide the trickery in a mm_down_read macro. such way it will be recursive _and_ fair.

You'd need a counter per-process per-mm_struct. Otherwise you couldn't do a recursive read lock simultaneously in two or more different processes, and also allow any one process to lock multiple mm_structs.

the process doesn't need to lock multiple mm_structs at the same time.

I mean, we just need to allow a single task to go through, doesn't matter if the other tasks/threads are stuck, they will wait the write to finish. that's exactly where the fairness cames from.

The only thing that matters is that if a certain task passes the first read lock of its mmstruct semaphore, it will also pass any other further recursive lock again of its own _same_ mmstruct. So a per-process recursor is all what we need.

Must not be per-mm, per-mm would work but it would simply introduce the unfairness again.

Andrea

#38 lore raw
Quoted 1 line · Andrea Arcangeli

the process doesn't need to lock multiple mm_structs at the same time.

fork, ptrace, /proc/pid/mem, /proc/pid/maps

All have to be able to lock two process's mm_structs simultaneously, even if it's indirectly through copy_to_user() or copy_from_user().

David

#39 lore raw
Quoted 8 lines · David Howells
Quoted 1 line · Andrea Arcangeli

the process doesn't need to lock multiple mm_structs at the same time.

fork, ptrace, /proc/pid/mem, /proc/pid/maps

All have to be able to lock two process's mm_structs simultaneously, even if it's indirectly through copy_to_user() or copy_from_user().

ptrace doesn't use down_read_recursive, nor /proc/<>/mem, nor fork.

for /proc/<pid>/maps this check takes care of it of course (or it could get unfair again: only when we're faulting on our vm we're allowed to go through):

	if (task == current)
		down_read_recursive(&mm->mmap_sem, &current->mm_recursor);
	else
		down_read(&mm->mmap_sem);

Andrea