Skip to contentSkip to conversation

aio: remove aio-max-nr and instead use the memlock rlimit to limit the number of pages pinned for the aio completion ring

PATCH In review Jeff Moyer 9 participants 49 messages Mar 9, 2009 – Mar 18, 2009

Subsystems (7)
lore
Share & tools
lore

Jeff Moyer

author
#0 lore raw

PATCH 1/1 aio: remove aio-max-nr and instead use the memlock rlimit to limit the number of pages pinned for the aio completion ring

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

It's worth noting that, by default, aio-max-nr was set to 65536 requests. By default, the memlock rlimit is set to 64kb per process. With this patch in place, a single process can specify 2045 for the nr_events parameter of io_setup. Or, for the worst case scenario, a process can only create 16 io contexts, each with a single event (since the minimum ring size is a page).

I created a simple program that sets the process' memlock rlimit and then drops the CAP_IPC_LOCK capability (the memlock rlimit is a command line parameter, so you can see how the nr_events allowable changes with different limits in place). Then, it calls io_setup/io_destroy in a loop, increasing nr_events until it fails. Finally, it calls io_setup in a loop with a single event to see how many iterations it can perform before receiving -EAGAIN. I ran the test on a patched kernel and the results are in line with expectations.

I also ran the aio-dio-regress regression tests, and all passed but one. The one that failed only failed due to an expected return code of -ENOMEM, and instead got -EAGAIN. Part of the patch below returns a proper error code from aio_setup_ring. So, I'm calling this a test issue, but we can debate this nuance if it is deemed important.

Further, as a result of this exercise, I noticed that there are numerous places in the kernel that test to see if locking memory will exceed the maximum amount of allowed locked memory. I've factored out that bit of code in a follow-on patch that I will post.

Finally, I updated the aio man pages to reflect this change (and fix references to aio_context_t that should be io_context_t).

You can find a copy of the test program I used at:
http://people.redhat.com/jmoyer/aio/ioctx_alloc.c
I'll apologize in advance for the crudity of the test code.

For those reviewing the below patch, it may be worth looking at:

commit d55b5fdaf40846221d543937b786956e27837fda

Author: Zach Brown <[email protected]>
Date:   Mon Nov 7 00:59:31 2005 -0800

[PATCH] aio: remove aio_max_nr accounting race

The below patch basically reverts the above commit. There is no accounting race now, because we are charging per-process rlimits instead of a system-wide maximum number of aio requests. This has the added benefit of reducing some code complexity.

Comments and suggestions are encouraged. I'll follow up with the other two patches I mentioned shortly.

Thanks!
Jeff

fs/aio.c +26 -34
index 8fa77e2..3bbda9d 100644
@@ -43,9 +43,8 @@
 #endif
 
 /*------ sysctl variables----*/
-static DEFINE_SPINLOCK(aio_nr_lock);
-unsigned long aio_nr;		/* current system wide number of aio requests */
-unsigned long aio_max_nr = 0x10000; /* system wide maximum number of aio requests */
+/* current system wide number of aio requests */
+atomic_t aio_nr = ATOMIC_INIT(0);
 /*----end sysctl variables---*/
 
 static struct kmem_cache	*kiocb_cachep;
@@ -90,6 +89,7 @@ static void aio_free_ring(struct kioctx *ctx)
 	if (info->mmap_size) {
 		down_write(&ctx->mm->mmap_sem);
 		do_munmap(ctx->mm, info->mmap_base, info->mmap_size);
+		ctx->mm->locked_vm -= info->nr_pages;
 		up_write(&ctx->mm->mmap_sem);
 	}
 
@@ -106,6 +106,8 @@ static int aio_setup_ring(struct kioctx *ctx)
 	unsigned nr_events = ctx->max_reqs;
 	unsigned long size;
 	int nr_pages;
+	unsigned long locked;
+	unsigned long lock_limit;
 
 	/* Compensate for the ring buffer's head/tail overlap entry */
 	nr_events += 2;	/* 1 is required, 2 for good luck */
@@ -130,6 +132,20 @@ static int aio_setup_ring(struct kioctx *ctx)
 	info->mmap_size = nr_pages * PAGE_SIZE;
 	dprintk("attempting mmap of %lu bytes\n", info->mmap_size);
 	down_write(&ctx->mm->mmap_sem);
+	/*
+	 * Check that the memory reserved for the completion ring does
+	 * not exceed the memlock memory limit.
+	 */
+	locked = nr_pages + ctx->mm->locked_vm;
+	lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
+	lock_limit >>= PAGE_SHIFT;
+	if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) {
+		up_write(&ctx->mm->mmap_sem);
+		info->mmap_size = 0;
+		aio_free_ring(ctx);
+		return -EAGAIN;
+	}
+
 	info->mmap_base = do_mmap(NULL, 0, info->mmap_size, 
 				  PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_PRIVATE,
 				  0);
@@ -144,6 +160,7 @@ static int aio_setup_ring(struct kioctx *ctx)
 	info->nr_pages = get_user_pages(current, ctx->mm,
 					info->mmap_base, nr_pages, 
 					1, 0, info->ring_pages, NULL);
+	ctx->mm->locked_vm += info->nr_pages;
 	up_write(&ctx->mm->mmap_sem);
 
 	if (unlikely(info->nr_pages != nr_pages)) {
@@ -194,16 +211,8 @@ static int aio_setup_ring(struct kioctx *ctx)
 static void ctx_rcu_free(struct rcu_head *head)
 {
 	struct kioctx *ctx = container_of(head, struct kioctx, rcu_head);
-	unsigned nr_events = ctx->max_reqs;
 
 	kmem_cache_free(kioctx_cachep, ctx);
-
-	if (nr_events) {
-		spin_lock(&aio_nr_lock);
-		BUG_ON(aio_nr - nr_events > aio_nr);
-		aio_nr -= nr_events;
-		spin_unlock(&aio_nr_lock);
-	}
 }
 
 /* __put_ioctx
@@ -217,6 +226,7 @@ static void __put_ioctx(struct kioctx *ctx)
 	cancel_delayed_work(&ctx->wq);
 	cancel_work_sync(&ctx->wq.work);
 	aio_free_ring(ctx);
+	atomic_sub(ctx->max_reqs, &aio_nr);
 	mmdrop(ctx->mm);
 	ctx->mm = NULL;
 	pr_debug("__put_ioctx: freeing %p\n", ctx);
@@ -240,7 +250,7 @@ static struct kioctx *ioctx_alloc(unsigned nr_events)
 {
 	struct mm_struct *mm;
 	struct kioctx *ctx;
-	int did_sync = 0;
+	int ret;
 
 	/* Prevent overflows */
 	if ((nr_events > (0x10000000U / sizeof(struct io_event))) ||
@@ -249,9 +259,6 @@ static struct kioctx *ioctx_alloc(unsigned nr_events)
 		return ERR_PTR(-EINVAL);
 	}
 
-	if ((unsigned long)nr_events > aio_max_nr)
-		return ERR_PTR(-EAGAIN);
-
 	ctx = kmem_cache_zalloc(kioctx_cachep, GFP_KERNEL);
 	if (!ctx)
 		return ERR_PTR(-ENOMEM);
@@ -269,26 +276,11 @@ static struct kioctx *ioctx_alloc(unsigned nr_events)
 	INIT_LIST_HEAD(&ctx->run_list);
 	INIT_DELAYED_WORK(&ctx->wq, aio_kick_handler);
 
-	if (aio_setup_ring(ctx) < 0)
+	if ((ret = aio_setup_ring(ctx)) < 0)
 		goto out_freectx;
 
-	/* limit the number of system wide aios */
-	do {
-		spin_lock_bh(&aio_nr_lock);
-		if (aio_nr + nr_events > aio_max_nr ||
-		    aio_nr + nr_events < aio_nr)
-			ctx->max_reqs = 0;
-		else
-			aio_nr += ctx->max_reqs;
-		spin_unlock_bh(&aio_nr_lock);
-		if (ctx->max_reqs || did_sync)
-			break;
-
-		/* wait for rcu callbacks to have completed before giving up */
-		synchronize_rcu();
-		did_sync = 1;
-		ctx->max_reqs = nr_events;
-	} while (1);
+	/* update the number of system wide aios */
+	atomic_add(ctx->max_reqs, &aio_nr);  /* undone by __put_ioctx */
 
 	if (ctx->max_reqs == 0)
 		goto out_cleanup;
@@ -309,7 +301,7 @@ out_cleanup:
 out_freectx:
 	mmdrop(mm);
 	kmem_cache_free(kioctx_cachep, ctx);
-	ctx = ERR_PTR(-ENOMEM);
+	ctx = ERR_PTR(ret);
 
 	dprintk("aio: error allocating ioctx %p\n", ctx);
 	return ctx;
include/linux/aio.h +1 -2
index b16a957..09ec393 100644
@@ -233,7 +233,6 @@ static inline struct kiocb *list_kiocb(struct list_head *h)
 }
 
 /* for sysctl: */
-extern unsigned long aio_nr;
-extern unsigned long aio_max_nr;
+extern atomic_t aio_nr;
 
 #endif /* __LINUX__AIO_H */
kernel/sysctl.c +1 -8
index c5ef44f..32ced38 100644
@@ -1385,14 +1385,7 @@ static struct ctl_table fs_table[] = {
 		.data		= &aio_nr,
 		.maxlen		= sizeof(aio_nr),
 		.mode		= 0444,
-		.proc_handler	= &proc_doulongvec_minmax,
-	},
-	{
-		.procname	= "aio-max-nr",
-		.data		= &aio_max_nr,
-		.maxlen		= sizeof(aio_max_nr),
-		.mode		= 0644,
-		.proc_handler	= &proc_doulongvec_minmax,
+		.proc_handler	= &proc_dointvec,
 	},
 #endif /* CONFIG_AIO */
 #ifdef CONFIG_INOTIFY_USER
author
#1 lore raw

PATCH 1/1 factor out checks against the memlock rlimit

Hi,

There are several places in the kernel where the memlock rlimit is checked, all duplicating code. I added another in a recent patch and it made me feel dirty, so this patch factors all of that into a single function, can_mlock_pages.

The infiniband implementation of the rlimit check was actually broken. Weeding through changelogs showed that the initial implementation was wrong, code was #if 0'd out because it didn't work, and instead of fixing things, they just removed the code all together. If my assessment of that is wrong, please let me know! ;-)

Comments, as always, are encouraged.

Cheers,
Jeff

drivers/infiniband/core/umem.c +2 -5
index 6f7c096..c4d4f1b 100644
@@ -135,14 +135,11 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr,
 
 	down_write(&current->mm->mmap_sem);
 
-	locked     = npages + current->mm->locked_vm;
-	lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur >> PAGE_SHIFT;
-
-	if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) {
+	if (!can_mlock_pages(npages)) {
 		ret = -ENOMEM;
 		goto out;
 	}
-
+	locked = npages + current->mm->locked_vm;
 	cur_base = addr & PAGE_MASK;
 
 	ret = 0;
drivers/infiniband/hw/ipath/ipath_user_pages.c +1 -4
index 0190edc..9ab17e3 100644
@@ -58,10 +58,7 @@ static int __get_user_pages(unsigned long start_page, size_t num_pages,
 	size_t got;
 	int ret;
 
-	lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur >>
-		PAGE_SHIFT;
-
-	if (num_pages > lock_limit) {
+	if (!can_mlock_pages(num_pages)) {
 		ret = -ENOMEM;
 		goto bail;
 	}
fs/aio.c +1 -4
index 3bbda9d..0ca64eb 100644
@@ -136,10 +136,7 @@ static int aio_setup_ring(struct kioctx *ctx)
 	 * Check that the memory reserved for the completion ring does
 	 * not exceed the memlock memory limit.
 	 */
-	locked = nr_pages + ctx->mm->locked_vm;
-	lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
-	lock_limit >>= PAGE_SHIFT;
-	if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) {
+	if (!can_mlock_pages(nr_pages)) {
 		up_write(&ctx->mm->mmap_sem);
 		info->mmap_size = 0;
 		aio_free_ring(ctx);
include/linux/mm.h +1 -0
index 065cdf8..79c1127 100644
@@ -737,6 +737,7 @@ extern unsigned long shmem_get_unmapped_area(struct file *file,
 #endif
 
 extern int can_do_mlock(void);
+extern int can_mlock_pages(unsigned long);
 extern int user_shm_lock(size_t, struct user_struct *);
 extern void user_shm_unlock(size_t, struct user_struct *);
 
mm/mlock.c +28 -7
index cbe9e05..4ec96c2 100644
@@ -31,6 +31,33 @@ int can_do_mlock(void)
 }
 EXPORT_SYMBOL(can_do_mlock);
 
+/**
+ * can_mlock_pages() - tell whether mlocking nr_pages is possible
+ * @nr_pages:	Number of pages to be locked in memory
+ *
+ * Boolean function which tells whether the MEMLOCK rlimit will prevent
+ * locking nr_pages pages.
+ */
+int can_mlock_pages(unsigned long nr_pages)
+{
+	unsigned long locked, lock_limit;
+
+	if (capable(CAP_IPC_LOCK))
+		return 1;
+
+	locked = nr_pages;
+	locked += current->mm->locked_vm;
+
+	lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
+	lock_limit >>= PAGE_SHIFT;
+
+	/* check against resource limits */
+	if (locked <= lock_limit)
+		return 1;
+	return 0;
+}
+EXPORT_SYMBOL(can_mlock_pages);
+
 #ifdef CONFIG_UNEVICTABLE_LRU
 /*
  * Mlocked pages are marked with PageMlocked() flag for efficient testing
@@ -505,14 +532,8 @@ SYSCALL_DEFINE2(mlock, unsigned long, start, size_t, len)
 	len = PAGE_ALIGN(len + (start & ~PAGE_MASK));
 	start &= PAGE_MASK;
 
-	locked = len >> PAGE_SHIFT;
-	locked += current->mm->locked_vm;
-
-	lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
-	lock_limit >>= PAGE_SHIFT;
-
 	/* check against resource limits */
-	if ((locked <= lock_limit) || capable(CAP_IPC_LOCK))
+	if (can_mlock_pages(len >> PAGE_SHIFT))
 		error = do_mlock(start, len, 1);
 	up_write(&current->mm->mmap_sem);
 	return error;
mm/mmap.c +2 -12
index 00ced3e..52d1c32 100644
@@ -975,12 +975,7 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned long addr,
 
 	/* mlock MCL_FUTURE? */
 	if (vm_flags & VM_LOCKED) {
-		unsigned long locked, lock_limit;
-		locked = len >> PAGE_SHIFT;
-		locked += mm->locked_vm;
-		lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
-		lock_limit >>= PAGE_SHIFT;
-		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+		if (!can_mlock_pages(len >> PAGE_SHIFT))
 			return -EAGAIN;
 	}
 
@@ -2006,12 +2001,7 @@ unsigned long do_brk(unsigned long addr, unsigned long len)
 	 * mlock MCL_FUTURE?
 	 */
 	if (mm->def_flags & VM_LOCKED) {
-		unsigned long locked, lock_limit;
-		locked = len >> PAGE_SHIFT;
-		locked += mm->locked_vm;
-		lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
-		lock_limit >>= PAGE_SHIFT;
-		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+		if (!can_mlock_pages(len>>PAGE_SHIFT))
 			return -EAGAIN;
 	}
 
mm/mremap.c +1 -5
index a39b7b9..6fb50e2 100644
@@ -342,12 +342,8 @@ unsigned long do_mremap(unsigned long addr,
 			goto out;
 	}
 	if (vma->vm_flags & VM_LOCKED) {
-		unsigned long locked, lock_limit;
-		locked = mm->locked_vm << PAGE_SHIFT;
-		lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur;
-		locked += new_len - old_len;
 		ret = -EAGAIN;
-		if (locked > lock_limit && !capable(CAP_IPC_LOCK))
+		if (!can_mlock_pages((new_len - old_len)>>PAGE_SHIFT))
 			goto out;
 	}
 	if (!may_expand_vm(mm, (new_len - old_len) >> PAGE_SHIFT)) {
author
#2 lore raw

PATCH 1/1 man-pages: add documentation about the memlock implications of io_setup

In reply to Jeff Moyer

Hi,

This patch, against man-pages-3.19, adds verbiage surrounding the newly proposed accounting of pinned aio completion ring pages against the memlock rlimit. It also fixes up a few references to aio_context_t (which should be io_context_t). Sorry for conflating the two, I can break it apart if needed.

Cheers,
Jeff

man2/io_cancel.2 renamed from man2/io_cancel.2.orig +1 -1
diff -up man-pages-3.19/man2/io_cancel.2.orig man-pages-3.19/man2/io_cancel.2
@@ -32,7 +32,7 @@ io_cancel \- cancel an outstanding async
 .\"#include <linux/aio.h>
 .sp
 .\" .HP 16
-.BI "int io_cancel(aio_context_t " ctx_id ", struct iocb *" iocb ,
+.BI "int io_cancel(io_context_t " ctx_id ", struct iocb *" iocb ,
 .BI "              struct io_event *" result );
 .\" .ad
 .\" .hy
man2/io_destroy.2 renamed from man2/io_destroy.2.orig +1 -1
diff -up man-pages-3.19/man2/io_destroy.2.orig man-pages-3.19/man2/io_destroy.2
@@ -31,7 +31,7 @@ io_destroy \- destroy an asynchronous I/
 .\" #include <linux/aio.h>
 .sp
 .\" .HP 17
-.BI "int io_destroy(aio_context_t " ctx );
+.BI "int io_destroy(io_context_t " ctx );
 .\" .ad
 .\" .hy
 .sp
man2/io_setup.2 renamed from man2/io_setup.2.orig +6 -3
diff -up man-pages-3.19/man2/io_setup.2.orig man-pages-3.19/man2/io_setup.2
@@ -31,7 +31,7 @@ io_setup \- create an asynchronous I/O c
 .\" #include <linux/aio.h>
 .sp
 .\" .HP 15
-.BI "int io_setup(unsigned " nr_events ", aio_context_t *" ctxp );
+.BI "int io_setup(unsigned " nr_events ", io_context_t *" ctxp );
 .\" .ad
 .\" .hy
 .sp
@@ -45,7 +45,9 @@ at least \fInr_events\fP.
 \fIctxp\fP must not point to an AIO context that already exists, and must
 be initialized to 0 prior to the call.
 On successful creation of the AIO context, \fI*ctxp\fP is filled in
-with the resulting handle.
+with the resulting handle.  Memory is reserved by the kernel for the completion
+ring and, because this memory is pinned, it will be charged against the
+process's memlock rlimit.
 .SH "RETURN VALUE"
 On success,
 .BR io_setup ()
@@ -54,7 +56,8 @@ For the failure return, see NOTES.
 .SH "ERRORS"
 .TP
 .B EAGAIN
-The specified \fInr_events\fP exceeds the user's limit of available events.
+The specified \fInr_events\fP exceeds the process's limit of available events.
+Try increasing the memlock rlimit for the process.
 .TP
 .B EFAULT
 An invalid pointer is passed for \fIctxp\fP.
man2/io_submit.2 renamed from man2/io_submit.2.orig +2 -2
diff -up man-pages-3.19/man2/io_submit.2.orig man-pages-3.19/man2/io_submit.2
@@ -31,7 +31,7 @@ io_submit \- submit asynchronous I/O blo
 .\" #include <linux/aio.h>
 .sp
 .\" .HP 16
-.BI "int io_submit(aio_context_t " ctx_id ", long " nr \
+.BI "int io_submit(io_context_t " ctx_id ", long " nr \
 ", struct iocb **" iocbpp );
 .\" .ad
 .\" .hy
@@ -63,7 +63,7 @@ The file descriptor specified in the fir
 One of the data structures points to invalid data.
 .TP
 .B EINVAL
-The \fIaio_context\fP specified by \fIctx_id\fP is invalid.
+The \fIio_context\fP specified by \fIctx_id\fP is invalid.
 \fInr\fP is less than 0.
 The \fIiocb\fP at *iocbpp[0] is not properly initialized,
 or the operation specified is invalid for the file descriptor
man7/capabilities.7 renamed from man7/capabilities.7.orig +2 -1
diff -up man-pages-3.19/man7/capabilities.7.orig man-pages-3.19/man7/capabilities.7
@@ -129,7 +129,8 @@ Lock memory
 .RB ( mlock (2),
 .BR mlockall (2),
 .BR mmap (2),
-.BR shmctl (2)).
+.BR shmctl (2),
+.BR io_setup (2)).
 .TP
 .B CAP_IPC_OWNER
 Bypass permission checks for operations on System V IPC objects.
#3 lore raw

Re: [patch] man-pages: add documentation about the memlock implications of io_setup

[CC+=Vegard Nossum]

Hi Jeff,

Note that my address for man-pages is [email protected].

Quoted 7 lines · Jeff Moyer

Hi,

This patch, against man-pages-3.19, adds verbiage surrounding the newly proposed accounting of pinned aio completion ring pages against the memlock rlimit. It also fixes up a few references to aio_context_t (which should be io_context_t). Sorry for conflating the two, I can break it apart if needed.

Breaking apart really would be better. Could you resubmit (to mtk.manpages@, and CC linux-man@vger) please?

When you resubmit, could you please clarify the following:

1. For the "proposed accounting of pinned aio completion ring pages against the memlock rlimit", can you tell me which mainline kernel version this change (will) appear(s) in. (You say "proposed", so that it's not clear when/if the change will be merged into mainline. A git commit reference could be useful to me here.)

2. Regarding aio_context_t vs io_context_t, Vegard Nossum also raised this issue a while back, and I replied that the aio_context_t type is used because:

[[
These pages attempt to document the bare syscall API. Take a look at the kernel sources: there the type *is* aio_context_t.
]]

Perhaps I am/was being muddle-headed here, since, after all, the pages do say "Link with -laio." (Once upon a time the pages actually *did* use "io_context_t", but I changed that during a big clean-up of the io_*.2 pages a year or two back.) Your thoughts? (Why do we even have aio_context_t in the kernel versus io_context_t in libaio?)

Cheers,

Michael

Quoted 92 lines

diff -up man-pages-3.19/man2/io_cancel.2.orig man-pages-3.19/man2/io_cancel.2

man2/io_cancel.2     2009-03-08 11:58:01.000000000 -0400 renamed from man2/io_cancel.2.orig        2009-03-08 11:57:50.000000000 -0400 +0 -0
@@ -32,7 +32,7 @@ io_cancel \- cancel an outstanding async

 .\"#include <linux/aio.h>
 .sp
 .\" .HP 16
-.BI "int io_cancel(aio_context_t " ctx_id ", struct iocb *" iocb , +.BI "int io_cancel(io_context_t " ctx_id ", struct iocb *" iocb , .BI "              struct io_event *" result );
 .\" .ad
 .\" .hy
diff -up man-pages-3.19/man2/io_destroy.2.orig man-pages-3.19/man2/io_destroy.2

man2/io_destroy.2    2009-03-08 11:58:16.000000000 -0400 renamed from man2/io_destroy.2.orig       2009-03-08 11:58:08.000000000 -0400 +0 -0
@@ -31,7 +31,7 @@ io_destroy \- destroy an asynchronous I/

 .\" #include <linux/aio.h>
 .sp
 .\" .HP 17
-.BI "int io_destroy(aio_context_t " ctx );
+.BI "int io_destroy(io_context_t " ctx );
 .\" .ad
 .\" .hy
 .sp
diff -up man-pages-3.19/man2/io_setup.2.orig man-pages-3.19/man2/io_setup.2

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +0 -0
@@ -31,7 +31,7 @@ io_setup \- create an asynchronous I/O c

 .\" #include <linux/aio.h>
 .sp
 .\" .HP 15
-.BI "int io_setup(unsigned " nr_events ", aio_context_t *" ctxp ); +.BI "int io_setup(unsigned " nr_events ", io_context_t *" ctxp ); .\" .ad
 .\" .hy
 .sp

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +0 -0
@@ -45,7 +45,9 @@ at least \fInr_events\fP.

 \fIctxp\fP must not point to an AIO context that already exists, and must be initialized to 0 prior to the call.
 On successful creation of the AIO context, \fI*ctxp\fP is filled in

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +3 -1
-with the resulting handle.
+with the resulting handle.  Memory is reserved by the kernel for the completion
+ring and, because this memory is pinned, it will be charged against the
+process's memlock rlimit.

 .SH "RETURN VALUE"
 On success,
 .BR io_setup ()

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +0 -0
@@ -54,7 +56,8 @@ For the failure return, see NOTES.

 .SH "ERRORS"
 .TP
 .B EAGAIN

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +2 -1
-The specified \fInr_events\fP exceeds the user's limit of available events.
+The specified \fInr_events\fP exceeds the process's limit of available events.
+Try increasing the memlock rlimit for the process.

 .TP
 .B EFAULT
 An invalid pointer is passed for \fIctxp\fP.
diff -up man-pages-3.19/man2/io_submit.2.orig man-pages-3.19/man2/io_submit.2

man2/io_submit.2     2009-03-08 11:58:34.000000000 -0400 renamed from man2/io_submit.2.orig        2009-03-08 11:58:26.000000000 -0400 +0 -0
@@ -31,7 +31,7 @@ io_submit \- submit asynchronous I/O blo

 .\" #include <linux/aio.h>
 .sp
 .\" .HP 16
-.BI "int io_submit(aio_context_t " ctx_id ", long " nr \
+.BI "int io_submit(io_context_t " ctx_id ", long " nr \
 ", struct iocb **" iocbpp );
 .\" .ad
 .\" .hy

man2/io_submit.2     2009-03-08 11:58:34.000000000 -0400 renamed from man2/io_submit.2.orig        2009-03-08 11:58:26.000000000 -0400 +0 -0
@@ -63,7 +63,7 @@ The file descriptor specified in the fir

 One of the data structures points to invalid data. .TP
 .B EINVAL

man2/io_submit.2     2009-03-08 11:58:34.000000000 -0400 renamed from man2/io_submit.2.orig        2009-03-08 11:58:26.000000000 -0400 +1 -1
-The \fIaio_context\fP specified by \fIctx_id\fP is invalid.
+The \fIio_context\fP specified by \fIctx_id\fP is invalid.

 \fInr\fP is less than 0.
 The \fIiocb\fP at *iocbpp[0] is not properly initialized,
 or the operation specified is invalid for the file descriptor
diff -up man-pages-3.19/man7/capabilities.7.orig man-pages-3.19/man7/capabilities.7

man7/capabilities.7  2009-03-08 11:53:14.000000000 -0400 renamed from man7/capabilities.7.orig     2009-03-08 11:52:56.000000000 -0400 +2 -1
@@ -129,7 +129,8 @@ Lock memory
  .RB ( mlock (2),
  .BR mlockall (2),
  .BR mmap (2),
-.BR shmctl (2)).
+.BR shmctl (2),
+.BR io_setup (2)).

 .TP
 .B CAP_IPC_OWNER
 Bypass permission checks for operations on System V IPC objects.

#4 lore raw

Re: [patch] man-pages: add documentation about the memlock implications of io_setup

In reply to Jeff Moyer

[RESEND: Really CC+=Vegard Nossum, this time, and += mtk.manpages@, as well]

Hi Jeff,

Note that my address for man-pages is [email protected].

Quoted 7 lines · Jeff Moyer

Hi,

This patch, against man-pages-3.19, adds verbiage surrounding the newly proposed accounting of pinned aio completion ring pages against the memlock rlimit. It also fixes up a few references to aio_context_t (which should be io_context_t). Sorry for conflating the two, I can break it apart if needed.

Breaking apart really would be better. Could you resubmit (to mtk.manpages@, and CC linux-man@vger) please?

When you resubmit, could you please clarify the following:

1. For the "proposed accounting of pinned aio completion ring pages against the memlock rlimit", can you tell me which mainline kernel version this change (will) appear(s) in. (You say "proposed", so that it's not clear when/if the change will be merged into mainline. A git commit reference could be useful to me here.)

2. Regarding aio_context_t vs io_context_t, Vegard Nossum also raised this issue a while back, and I replied that the aio_context_t type is used because:

[[
These pages attempt to document the bare syscall API. Take a look at the kernel sources: there the type *is* aio_context_t.
]]

Perhaps I am/was being muddle-headed here, since, after all, the pages do say "Link with -laio." (Once upon a time the pages actually *did* use "io_context_t", but I changed that during a big clean-up of the io_*.2 pages a year or two back.) Your thoughts? (Why do we even have aio_context_t in the kernel versus io_context_t in libaio?)

Cheers,

Michael

Quoted 92 lines

diff -up man-pages-3.19/man2/io_cancel.2.orig man-pages-3.19/man2/io_cancel.2

man2/io_cancel.2     2009-03-08 11:58:01.000000000 -0400 renamed from man2/io_cancel.2.orig        2009-03-08 11:57:50.000000000 -0400 +0 -0
@@ -32,7 +32,7 @@ io_cancel \- cancel an outstanding async

 .\"#include <linux/aio.h>
 .sp
 .\" .HP 16
-.BI "int io_cancel(aio_context_t " ctx_id ", struct iocb *" iocb , +.BI "int io_cancel(io_context_t " ctx_id ", struct iocb *" iocb , .BI "              struct io_event *" result );
 .\" .ad
 .\" .hy
diff -up man-pages-3.19/man2/io_destroy.2.orig man-pages-3.19/man2/io_destroy.2

man2/io_destroy.2    2009-03-08 11:58:16.000000000 -0400 renamed from man2/io_destroy.2.orig       2009-03-08 11:58:08.000000000 -0400 +0 -0
@@ -31,7 +31,7 @@ io_destroy \- destroy an asynchronous I/

 .\" #include <linux/aio.h>
 .sp
 .\" .HP 17
-.BI "int io_destroy(aio_context_t " ctx );
+.BI "int io_destroy(io_context_t " ctx );
 .\" .ad
 .\" .hy
 .sp
diff -up man-pages-3.19/man2/io_setup.2.orig man-pages-3.19/man2/io_setup.2

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +0 -0
@@ -31,7 +31,7 @@ io_setup \- create an asynchronous I/O c

 .\" #include <linux/aio.h>
 .sp
 .\" .HP 15
-.BI "int io_setup(unsigned " nr_events ", aio_context_t *" ctxp ); +.BI "int io_setup(unsigned " nr_events ", io_context_t *" ctxp ); .\" .ad
 .\" .hy
 .sp

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +0 -0
@@ -45,7 +45,9 @@ at least \fInr_events\fP.

 \fIctxp\fP must not point to an AIO context that already exists, and must be initialized to 0 prior to the call.
 On successful creation of the AIO context, \fI*ctxp\fP is filled in

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +3 -1
-with the resulting handle.
+with the resulting handle.  Memory is reserved by the kernel for the completion
+ring and, because this memory is pinned, it will be charged against the
+process's memlock rlimit.

 .SH "RETURN VALUE"
 On success,
 .BR io_setup ()

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +0 -0
@@ -54,7 +56,8 @@ For the failure return, see NOTES.

 .SH "ERRORS"
 .TP
 .B EAGAIN

man2/io_setup.2      2009-03-08 11:57:09.000000000 -0400 renamed from man2/io_setup.2.orig 2009-03-08 11:53:39.000000000 -0400 +2 -1
-The specified \fInr_events\fP exceeds the user's limit of available events.
+The specified \fInr_events\fP exceeds the process's limit of available events.
+Try increasing the memlock rlimit for the process.

 .TP
 .B EFAULT
 An invalid pointer is passed for \fIctxp\fP.
diff -up man-pages-3.19/man2/io_submit.2.orig man-pages-3.19/man2/io_submit.2

man2/io_submit.2     2009-03-08 11:58:34.000000000 -0400 renamed from man2/io_submit.2.orig        2009-03-08 11:58:26.000000000 -0400 +0 -0
@@ -31,7 +31,7 @@ io_submit \- submit asynchronous I/O blo

 .\" #include <linux/aio.h>
 .sp
 .\" .HP 16
-.BI "int io_submit(aio_context_t " ctx_id ", long " nr \
+.BI "int io_submit(io_context_t " ctx_id ", long " nr \
 ", struct iocb **" iocbpp );
 .\" .ad
 .\" .hy

man2/io_submit.2     2009-03-08 11:58:34.000000000 -0400 renamed from man2/io_submit.2.orig        2009-03-08 11:58:26.000000000 -0400 +0 -0
@@ -63,7 +63,7 @@ The file descriptor specified in the fir

 One of the data structures points to invalid data. .TP
 .B EINVAL

man2/io_submit.2     2009-03-08 11:58:34.000000000 -0400 renamed from man2/io_submit.2.orig        2009-03-08 11:58:26.000000000 -0400 +1 -1
-The \fIaio_context\fP specified by \fIctx_id\fP is invalid.
+The \fIio_context\fP specified by \fIctx_id\fP is invalid.

 \fInr\fP is less than 0.
 The \fIiocb\fP at *iocbpp[0] is not properly initialized,
 or the operation specified is invalid for the file descriptor
diff -up man-pages-3.19/man7/capabilities.7.orig man-pages-3.19/man7/capabilities.7

man7/capabilities.7  2009-03-08 11:53:14.000000000 -0400 renamed from man7/capabilities.7.orig     2009-03-08 11:52:56.000000000 -0400 +2 -1
@@ -129,7 +129,8 @@ Lock memory
  .RB ( mlock (2),
  .BR mlockall (2),
  .BR mmap (2),
-.BR shmctl (2)).
+.BR shmctl (2),
+.BR io_setup (2)).

 .TP
 .B CAP_IPC_OWNER
 Bypass permission checks for operations on System V IPC objects.

author
#5 lore raw

Re: [patch] man-pages: add documentation about the memlock implications of io_setup

Quoted 5 lines · Michael Kerrisk

[RESEND: Really CC+=Vegard Nossum, this time, and += mtk.manpages@, as well]

Hi Jeff,

Note that my address for man-pages is [email protected].

Sorry about that!

Quoted 11 lines · Jeff Moyer
Quoted 7 lines

Hi,

This patch, against man-pages-3.19, adds verbiage surrounding the newly proposed accounting of pinned aio completion ring pages against the memlock rlimit. It also fixes up a few references to aio_context_t (which should be io_context_t). Sorry for conflating the two, I can break it apart if needed.

Breaking apart really would be better. Could you resubmit (to mtk.manpages@, and CC linux-man@vger) please?

Sure thing. I'll address your question when I resubmit.

Thanks!

Jeff

#6 lore raw
In reply to Jeff Moyer
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

Signature · 1 line
error compiling committee.c: too many arguments to function
author
#7 lore raw
Quoted 15 lines · Avi Kivity
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Quoted 3 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

I'm happy to look into such a scheme, should anyone show me data that points to this NUMA issue as an actual performance problem today. In the absence of such data, I simply can't justify the work at the moment.

Thanks for taking a look!

-Jeff

#8 lore raw
Quoted 11 lines · Jeff Moyer
Quoted 4 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

Or you could queue the completions somewhere, and only copy them to user memory when io_getevents() is called. I think the plan was once to allow events to be consumed opportunistically even without io_getevents(), though.

Quoted 4 lines

I'm happy to look into such a scheme, should anyone show me data that points to this NUMA issue as an actual performance problem today. In the absence of such data, I simply can't justify the work at the moment.

Right now page migration is a dead duck. Outside HPC, there is now support for triggering it or for getting the scheduler to prefer a process's memory node. Only a minority of hosts are NUMA.

I think that will/should change in the near future. Nehalem-based servers mean that NUMA will be commonplace. The larger core counts will mean that hosts will run several unrelated applications (often through virtualization); such partitioning can easily benefit from page migration.

Quoted 2 lines

Thanks for taking a look!

Sorry, I didn't actually take a look at the patches. I only reacted to the description - I am allergic to pinned memory.

Signature · 1 line
Do not meddle in the internals of kernels, for they are subtle and quick to panic.
#9 lore raw
Quoted 4 lines · Avi Kivity

Or you could queue the completions somewhere, and only copy them to user memory when io_getevents() is called. I think the plan was once to allow events to be consumed opportunistically even without io_getevents(), though.

Isn't that integrated (or to be integrated) with the vring buffer thingy used by virtio?

If not, should it be?

Quoted 2 lines

Sorry, I didn't actually take a look at the patches. I only reacted to the description - I am allergic to pinned memory.

While we're at it I'm allergic to fixed size ring buffers :-)

-- Jamie

#10 lore raw
Quoted 8 lines · Jamie Lokier
Quoted 5 lines · Avi Kivity

Or you could queue the completions somewhere, and only copy them to user memory when io_getevents() is called. I think the plan was once to allow events to be consumed opportunistically even without io_getevents(), though.

Quoted 6 lines

Isn't that integrated (or to be integrated) with the vring buffer thingy used by virtio?

If not, should it be?

vringfd has been abandoned, AFAICT. I think it's too similar to aio anyway.

Signature · 1 line
error compiling committee.c: too many arguments to function
#11 lore raw
In reply to Jeff Moyer
Quoted 20 lines · Jeff Moyer
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Quoted 3 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

Quoted 5 lines

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

This aio_completion from interrupt handlers keep us from using SLAB_DESTROY_BY_RCU instead of call_rcu() for "struct file" freeing.

http://lkml.org/lkml/2008/12/17/364

I would love it we could get rid of this mess...

#12 lore raw
Quoted 26 lines · Eric Dumazet
Quoted 16 lines · Jeff Moyer
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Quoted 8 lines
Quoted 3 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

Quoted 6 lines

This aio_completion from interrupt handlers keep us from using SLAB_DESTROY_BY_RCU instead of call_rcu() for "struct file" freeing.

http://lkml.org/lkml/2008/12/17/364

I would love if we could get rid of this mess...

Speaking of that, I tried to take a look at this aio stuff and have one question.

Assuming that __fput() cannot be called from interrupt context. -> fput() should not be called from interrupt context as well.

How comes we call fput(req->ki_eventfd) from really_put_req() from interrupt context ?

If user program closes eventfd, then inflight AIO requests can trigger a bug.

#13 lore raw
Quoted 35 lines · Eric Dumazet
Quoted 25 lines · Eric Dumazet
Quoted 16 lines · Jeff Moyer
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Quoted 8 lines
Quoted 3 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

Quoted 6 lines

This aio_completion from interrupt handlers keep us from using SLAB_DESTROY_BY_RCU instead of call_rcu() for "struct file" freeing.

http://lkml.org/lkml/2008/12/17/364

I would love if we could get rid of this mess...

Speaking of that, I tried to take a look at this aio stuff and have one question.

Quoted 9 lines

Assuming that __fput() cannot be called from interrupt context. -> fput() should not be called from interrupt context as well.

How comes we call fput(req->ki_eventfd) from really_put_req() from interrupt context ?

If user program closes eventfd, then inflight AIO requests can trigger a bug.

Path could be :

1) fput() changes so that calling it from interrupt context is possible (Using a working queue to make sure __fput() is called from process context)

2) Changes aio to use fput() as is (and zap its internal work_queue and aio_fput_routine() stuff)

3) Once atomic_long_dec_and_test(&filp->f_count) only performed in fput(), SLAB_DESTROY_BY_RCU for "struct file" get back :)

#14 lore raw

PATCH 1/1 fs: fput() can be called from interrupt context

Quoted 47 lines · Eric Dumazet
Quoted 34 lines · Eric Dumazet
Quoted 25 lines · Eric Dumazet
Quoted 16 lines · Jeff Moyer
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Quoted 8 lines
Quoted 3 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

Quoted 6 lines

This aio_completion from interrupt handlers keep us from using SLAB_DESTROY_BY_RCU instead of call_rcu() for "struct file" freeing.

http://lkml.org/lkml/2008/12/17/364

I would love if we could get rid of this mess...

Speaking of that, I tried to take a look at this aio stuff and have one question.

Quoted 9 lines

Assuming that __fput() cannot be called from interrupt context. -> fput() should not be called from interrupt context as well.

How comes we call fput(req->ki_eventfd) from really_put_req() from interrupt context ?

If user program closes eventfd, then inflight AIO requests can trigger a bug.

Path could be :

Quoted 8 lines

1) fput() changes so that calling it from interrupt context is possible (Using a working queue to make sure __fput() is called from process context)

2) Changes aio to use fput() as is (and zap its internal work_queue and aio_fput_routine() stuff)

3) Once atomic_long_dec_and_test(&filp->f_count) only performed in fput(), SLAB_DESTROY_BY_RCU for "struct file" get back :)

Please find first patch against linux-2.6

Next patch (2) can cleanup aio code, but it probably can wait linux-2.6.30

Thank you

[PATCH] fs: fput() can be called from interrupt context

Current aio/eventfd code can call fput() from interrupt context, which is not allowed.

In order to fix the problem and prepare SLAB_DESTROY_BY_RCU use for "struct file" allocation/freeing in 2.6.30, we might extend existing workqueue infrastructure and allow fput() to be called from interrupt context.

This unfortunalty adds a pointer to 'struct file'.

---
 fs/file.c               |   55 ++++++++++++++++++++++++++------------
 fs/file_table.c         |   10 +++++-
 include/linux/fdtable.h |    1
 include/linux/fs.h      |    1
 4 files changed, 49 insertions(+), 18 deletions(-)
fs/file.c +39 -16
index f313314..8c819a8 100644
@@ -24,6 +24,7 @@ struct fdtable_defer {
 	spinlock_t lock;
 	struct work_struct wq;
 	struct fdtable *next;
+	struct file *fhead;
 };
 
 int sysctl_nr_open __read_mostly = 1024*1024;
@@ -67,24 +68,53 @@ static void free_fdtable_work(struct work_struct *work)
 	struct fdtable_defer *f =
 		container_of(work, struct fdtable_defer, wq);
 	struct fdtable *fdt;
+	struct file *fhead;
 
-	spin_lock_bh(&f->lock);
-	fdt = f->next;
-	f->next = NULL;
-	spin_unlock_bh(&f->lock);
-	while(fdt) {
+	spin_lock_irq(&f->lock);
+	fdt      = f->next;
+	fhead    = f->fhead;
+	f->next  = NULL;
+	f->fhead = NULL;
+	spin_unlock_irq(&f->lock);
+
+	while (fdt) {
 		struct fdtable *next = fdt->next;
+
 		vfree(fdt->fd);
 		free_fdset(fdt);
 		kfree(fdt);
 		fdt = next;
 	}
+
+	while (fhead) {
+		struct file *next = fhead->f_next;
+
+		__fput(fhead);
+		fhead = next;
+	}
+}
+
+void fd_defer_queue(struct fdtable *fdt, struct file *file)
+{
+	struct fdtable_defer *fddef = &get_cpu_var(fdtable_defer_list);
+
+	spin_lock_irq(&fddef->lock);
+	if (fdt) {
+		fdt->next = fddef->next;
+		fddef->next = fdt;
+	}
+	if (file) {
+		file->f_next = fddef->fhead;
+		fddef->fhead = file;
+	}
+	spin_unlock_irq(&fddef->lock);
+	schedule_work(&fddef->wq);
+	put_cpu_var(fdtable_defer_list);
 }
 
 void free_fdtable_rcu(struct rcu_head *rcu)
 {
 	struct fdtable *fdt = container_of(rcu, struct fdtable, rcu);
-	struct fdtable_defer *fddef;
 
 	BUG_ON(!fdt);
 
@@ -101,16 +131,8 @@ void free_fdtable_rcu(struct rcu_head *rcu)
 		kfree(fdt->fd);
 		kfree(fdt->open_fds);
 		kfree(fdt);
-	} else {
-		fddef = &get_cpu_var(fdtable_defer_list);
-		spin_lock(&fddef->lock);
-		fdt->next = fddef->next;
-		fddef->next = fdt;
-		/* vmallocs are handled from the workqueue context */
-		schedule_work(&fddef->wq);
-		spin_unlock(&fddef->lock);
-		put_cpu_var(fdtable_defer_list);
-	}
+	} else
+		fd_defer_queue(fdt, NULL);
 }
 
 /*
@@ -410,6 +432,7 @@ static void __devinit fdtable_defer_list_init(int cpu)
 	spin_lock_init(&fddef->lock);
 	INIT_WORK(&fddef->wq, free_fdtable_work);
 	fddef->next = NULL;
+	fddef->fhead = NULL;
 }
 
 void __init files_defer_init(void)
fs/file_table.c +8 -2
index bbeeac6..77f42d8 100644
@@ -21,6 +21,7 @@
 #include <linux/fsnotify.h>
 #include <linux/sysctl.h>
 #include <linux/percpu_counter.h>
+#include <linux/interrupt.h>
 
 #include <asm/atomic.h>
 
@@ -220,10 +221,15 @@ int init_file(struct file *file, struct vfsmount *mnt, struct dentry *dentry,
 }
 EXPORT_SYMBOL(init_file);
 
+
 void fput(struct file *file)
 {
-	if (atomic_long_dec_and_test(&file->f_count))
-		__fput(file);
+	if (atomic_long_dec_and_test(&file->f_count)) {
+		if (unlikely(!in_interrupt()))
+			fd_defer_queue(NULL, file);
+		else
+			__fput(file);
+	}
 }
 
 EXPORT_SYMBOL(fput);
include/linux/fdtable.h +1 -0
index 09d6c5b..42e5e8e 100644
@@ -35,6 +35,7 @@ struct fdtable {
 	struct fdtable *next;
 };
 
+extern void fd_defer_queue(struct fdtable *fdt, struct file *file);
 /*
  * Open file table structure
  */
include/linux/fs.h +1 -0
index 92734c0..94beb0e 100644
@@ -872,6 +872,7 @@ struct file {
 #ifdef CONFIG_DEBUG_WRITECOUNT
 	unsigned long f_mnt_write_state;
 #endif
+	struct file		*f_next;
 };
 extern spinlock_t files_lock;
 #define file_list_lock() spin_lock(&files_lock);
#15 lore raw

PATCH 1/1 aio: fput() can be called from interrupt context

Quoted 10 lines · Eric Dumazet
Quoted 10 lines

Path could be :

1) fput() changes so that calling it from interrupt context is possible (Using a working queue to make sure __fput() is called from process context)

2) Changes aio to use fput() as is (and zap its internal work_queue and aio_fput_routine() stuff)

3) Once atomic_long_dec_and_test(&filp->f_count) only performed in fput(), SLAB_DESTROY_BY_RCU for "struct file" get back :)

Here is the second patch

Thank you

[PATCH] aio: cleanup, since fput() is IRQ safe

Once fput() is IRQ safe, we can cleanup aio code and delete its work_queue.

---
 fs/aio.c |   52 ++--------------------------------------------------
 1 files changed, 2 insertions(+), 50 deletions(-)
fs/aio.c +2 -50
index 8fa77e2..b0351a1 100644
@@ -53,13 +53,6 @@ static struct kmem_cache	*kioctx_cachep;
 
 static struct workqueue_struct *aio_wq;
 
-/* Used for rare fput completion. */
-static void aio_fput_routine(struct work_struct *);
-static DECLARE_WORK(fput_work, aio_fput_routine);
-
-static DEFINE_SPINLOCK(fput_lock);
-static LIST_HEAD(fput_head);
-
 static void aio_kick_handler(struct work_struct *);
 static void aio_queue_work(struct kioctx *);
 
@@ -469,15 +462,7 @@ static struct kiocb *__aio_get_req(struct kioctx *ctx)
 static inline struct kiocb *aio_get_req(struct kioctx *ctx)
 {
 	struct kiocb *req;
-	/* Handle a potential starvation case -- should be exceedingly rare as 
-	 * requests will be stuck on fput_head only if the aio_fput_routine is 
-	 * delayed and the requests were the last user of the struct file.
-	 */
 	req = __aio_get_req(ctx);
-	if (unlikely(NULL == req)) {
-		aio_fput_routine(NULL);
-		req = __aio_get_req(ctx);
-	}
 	return req;
 }
 
@@ -498,30 +483,6 @@ static inline void really_put_req(struct kioctx *ctx, struct kiocb *req)
 		wake_up(&ctx->wait);
 }
 
-static void aio_fput_routine(struct work_struct *data)
-{
-	spin_lock_irq(&fput_lock);
-	while (likely(!list_empty(&fput_head))) {
-		struct kiocb *req = list_kiocb(fput_head.next);
-		struct kioctx *ctx = req->ki_ctx;
-
-		list_del(&req->ki_list);
-		spin_unlock_irq(&fput_lock);
-
-		/* Complete the fput */
-		__fput(req->ki_filp);
-
-		/* Link the iocb into the context's free list */
-		spin_lock_irq(&ctx->ctx_lock);
-		really_put_req(ctx, req);
-		spin_unlock_irq(&ctx->ctx_lock);
-
-		put_ioctx(ctx);
-		spin_lock_irq(&fput_lock);
-	}
-	spin_unlock_irq(&fput_lock);
-}
-
 /* __aio_put_req
  *	Returns true if this put was the last user of the request.
  */
@@ -540,17 +501,8 @@ static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
-	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
-		get_ioctx(ctx);
-		spin_lock(&fput_lock);
-		list_add(&req->ki_list, &fput_head);
-		spin_unlock(&fput_lock);
-		queue_work(aio_wq, &fput_work);
-	} else
-		really_put_req(ctx, req);
+	fput(req->ki_filp);
+	really_put_req(ctx, req);
 	return 1;
 }
 
maintainer
#16 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

In reply to Eric Dumazet
Quoted 59 lines · On Thu, 12 Mar 2009 06:18:26 +0100 Eric Dumazet <[email protected]> wrote:

Eric Dumazet a __crit :

Quoted 47 lines

Eric Dumazet a __crit :

Quoted 34 lines

Eric Dumazet a __crit :

Quoted 25 lines

Jeff Moyer a __crit :

Quoted 16 lines
Quoted 14 lines · Avi Kivity
Quoted 13 lines · Jeff Moyer

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

Quoted 8 lines
Quoted 3 lines

Is it not possible to get rid of the pinning entirely? Pinning interferes with page migration which is important for NUMA, among other issues.

aio_complete is called from interrupt handlers, so can't block faulting in a page. Zach mentions there is a possibility of handing completions off to a kernel thread, with all of the performance worries and extra bookkeeping that go along with such a scheme (to help frame my concerns, I often get lambasted over .5% performance regressions).

Quoted 6 lines

This aio_completion from interrupt handlers keep us from using SLAB_DESTROY_BY_RCU instead of call_rcu() for "struct file" freeing.

http://lkml.org/lkml/2008/12/17/364

I would love if we could get rid of this mess...

Speaking of that, I tried to take a look at this aio stuff and have one question.

Quoted 9 lines

Assuming that __fput() cannot be called from interrupt context. -> fput() should not be called from interrupt context as well.

How comes we call fput(req->ki_eventfd) from really_put_req() from interrupt context ?

If user program closes eventfd, then inflight AIO requests can trigger a bug.

Path could be :

Quoted 8 lines

1) fput() changes so that calling it from interrupt context is possible (Using a working queue to make sure __fput() is called from process context)

2) Changes aio to use fput() as is (and zap its internal work_queue and aio_fput_routine() stuff)

3) Once atomic_long_dec_and_test(&filp->f_count) only performed in fput(), SLAB_DESTROY_BY_RCU for "struct file" get back :)

Please find first patch against linux-2.6

Quoted 8 lines

Next patch (2) can cleanup aio code, but it probably can wait linux-2.6.30

Thank you

[PATCH] fs: fput() can be called from interrupt context

Current aio/eventfd code can call fput() from interrupt context, which is not allowed.

The changelog forgot to tell us where this happens, and under what circumstances.

See, there might be other ways of fixing the bug,

Quoted 6 lines

In order to fix the problem and prepare SLAB_DESTROY_BY_RCU use for "struct file" allocation/freeing in 2.6.30, we might extend existing workqueue infrastructure and allow fput() to be called from interrupt context.

This unfortunalty adds a pointer to 'struct file'.

Quoted 7 lines
---
 fs/file.c               |   55 ++++++++++++++++++++++++++------------
 fs/file_table.c         |   10 +++++-
 include/linux/fdtable.h |    1
 include/linux/fs.h      |    1
 4 files changed, 49 insertions(+), 18 deletions(-)

which might not have some or all of the above problems.

I assume you're referring to really_put_req(), and commit 9c3060bedd84144653a2ad7bea32389f65598d40.

Quoted 1 line

From the above email straggle I extract "If user program closes

eventfd, then inflight AIO requests can trigger a bug" and I don't immediately see anything in there which would prevent this.

Did you reproduce the bug, and confirm that the patch fixes it?

Are there simpler ways of fixing it? Maybe sneak a call to wait_for_all_aios() into the right place? I doubt if it's performance critical, as nobody seems to have ever hit the bug.

Bear in mind that if the bug _is_ real then it's now out there, and we would like a fix which is usable by 2.6.<two-years-worth>.

etcetera..

#17 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 21 lines · Andrew Morton
Quoted 19 lines · On Thu, 12 Mar 2009 06:18:26 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 10 lines · Eric Dumazet

Path could be :

1) fput() changes so that calling it from interrupt context is possible (Using a working queue to make sure __fput() is called from process context)

2) Changes aio to use fput() as is (and zap its internal work_queue and aio_fput_routine() stuff)

3) Once atomic_long_dec_and_test(&filp->f_count) only performed in fput(), SLAB_DESTROY_BY_RCU for "struct file" get back :)

Please find first patch against linux-2.6

Next patch (2) can cleanup aio code, but it probably can wait linux-2.6.30

Thank you

[PATCH] fs: fput() can be called from interrupt context

Quoted 7 lines
Quoted 2 lines

Current aio/eventfd code can call fput() from interrupt context, which is not allowed.

The changelog forgot to tell us where this happens, and under what circumstances.

See, there might be other ways of fixing the bug,

Sure

Quoted 18 lines
Quoted 7 lines

In order to fix the problem and prepare SLAB_DESTROY_BY_RCU use for "struct file" allocation/freeing in 2.6.30, we might extend existing workqueue infrastructure and allow fput() to be called from interrupt context.

This unfortunalty adds a pointer to 'struct file'.

Quoted 7 lines
---
 fs/file.c               |   55 ++++++++++++++++++++++++++------------
 fs/file_table.c         |   10 +++++-
 include/linux/fdtable.h |    1
 include/linux/fs.h      |    1
 4 files changed, 49 insertions(+), 18 deletions(-)

which might not have some or all of the above problems.

Quoted 8 lines

I assume you're referring to really_put_req(), and commit 9c3060bedd84144653a2ad7bea32389f65598d40.

Quoted 1 line

From the above email straggle I extract "If user program closes

eventfd, then inflight AIO requests can trigger a bug" and I don't immediately see anything in there which would prevent this.

Did you reproduce the bug, and confirm that the patch fixes it?

take Davide program : http://www.xmailserver.org/eventfd-aio-test.c

and add at line 318 :
close(afd);

It should produce the kernel bug...

Quoted 4 lines

Are there simpler ways of fixing it? Maybe sneak a call to wait_for_all_aios() into the right place? I doubt if it's performance critical, as nobody seems to have ever hit the bug.

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

If you want to add another kludge to properly fput(req->ki_eventfd), be my guest :-(

Quoted 5 lines

Bear in mind that if the bug _is_ real then it's now out there, and we would like a fix which is usable by 2.6.<two-years-worth>.

etcetera..

maintainer
#18 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 9 lines · On Thu, 12 Mar 2009 07:10:38 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 2 lines

Did you reproduce the bug, and confirm that the patch fixes it?

take Davide program : http://www.xmailserver.org/eventfd-aio-test.c

and add at line 318 :
close(afd);

It should produce the kernel bug...

"should"?

Quoted 7 lines
Quoted 4 lines

Are there simpler ways of fixing it? Maybe sneak a call to wait_for_all_aios() into the right place? I doubt if it's performance critical, as nobody seems to have ever hit the bug.

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Quoted 6 lines

If you want to add another kludge to properly fput(req->ki_eventfd), be my guest :-(

Quoted 3 lines

Bear in mind that if the bug _is_ real then it's now out there, and we would like a fix which is usable by 2.6.<two-years-worth>.

The patches are large and scary and it would be a real problem to merge them into 2.6.29 at this stage, let alone 2.6.25, etc.

Especially as the code which you sent out appears to be untested:

fs/file_table.c int init_file(struct file *file, struct vfsmount *mnt, struct dentry *dentry, · line 228

Quoted 6 lines

void fput(struct file *file)
{

fs/file_table.c +2 -2
-	if (atomic_long_dec_and_test(&file->f_count))
-		__fput(file);
+	if (atomic_long_dec_and_test(&file->f_count)) {
+		if (unlikely(!in_interrupt()))
                             ^
Quoted 5 lines
fs/file_table.c +4 -0
+			fd_defer_queue(NULL, file);
+		else
+			__fput(file);
+	}
 }
#19 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 10 lines · Andrew Morton
Quoted 2 lines

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Would it be a huge problem, performance-wise, to stop making ->f_count tricks in __aio_put_req, and always offload to fput_work the task of releasing the requests?
If that's a huge problem, IMO the lower impact fix would be to use aio_fput_routine to loop into a second list, releasing the eventual eventfd file*. There's no need, IMO, to turn the whole fput() into IRQ-callable just for this case, when we can contain it into the particular KAIO+eventfd usage.

- Davide

#20 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 13 lines · Davide Libenzi
Quoted 10 lines · Andrew Morton
Quoted 2 lines

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Quoted 8 lines

Would it be a huge problem, performance-wise, to stop making ->f_count tricks in __aio_put_req, and always offload to fput_work the task of releasing the requests?
If that's a huge problem, IMO the lower impact fix would be to use aio_fput_routine to loop into a second list, releasing the eventual eventfd file*. There's no need, IMO, to turn the whole fput() into IRQ-callable just for this case, when we can contain it into the particular KAIO+eventfd usage.

Eric, are you working on this or should I? I missed where the conversation between you and Andrew is, at this point, WRT this issue.

- Davide

#21 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 22 lines · Davide Libenzi
Quoted 11 lines · Davide Libenzi
Quoted 9 lines · Andrew Morton
Quoted 2 lines

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Quoted 8 lines

Would it be a huge problem, performance-wise, to stop making ->f_count tricks in __aio_put_req, and always offload to fput_work the task of releasing the requests?
If that's a huge problem, IMO the lower impact fix would be to use aio_fput_routine to loop into a second list, releasing the eventual eventfd file*. There's no need, IMO, to turn the whole fput() into IRQ-callable just for this case, when we can contain it into the particular KAIO+eventfd usage.

Quoted 2 lines

Eric, are you working on this or should I? I missed where the conversation between you and Andrew is, at this point, WRT this issue.

I wish I could, but I currently have too much stress from my day job.

BTW, I could not produce a crash, so this problem is hypothetic for the moment :)

#22 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

In reply to Davide Libenzi
Quoted 13 lines · Davide Libenzi
Quoted 10 lines · Andrew Morton
Quoted 2 lines

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Quoted 9 lines

Would it be a huge problem, performance-wise, to stop making ->f_count tricks in __aio_put_req, and always offload to fput_work the task of releasing the requests?
If that's a huge problem, IMO the lower impact fix would be to use aio_fput_routine to loop into a second list, releasing the eventual eventfd file*. There's no need, IMO, to turn the whole fput() into IRQ-callable just for this case, when we can contain it into the particular KAIO+eventfd usage.

Do you really want to see eventd doing umounts and remote flock() calls? This really needs to be run in a thread that can cope with __long__ waits, unavailable servers, etc...

Trond

#23 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 24 lines · Trond Myklebust
Quoted 13 lines · Davide Libenzi
Quoted 10 lines · Andrew Morton
Quoted 2 lines

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Quoted 9 lines

Would it be a huge problem, performance-wise, to stop making ->f_count tricks in __aio_put_req, and always offload to fput_work the task of releasing the requests?
If that's a huge problem, IMO the lower impact fix would be to use aio_fput_routine to loop into a second list, releasing the eventual eventfd file*. There's no need, IMO, to turn the whole fput() into IRQ-callable just for this case, when we can contain it into the particular KAIO+eventfd usage.

Quoted 3 lines

Do you really want to see eventd doing umounts and remote flock() calls? This really needs to be run in a thread that can cope with __long__ waits, unavailable servers, etc...

Did I miss something? This wouldn't be (eventually) on keventd shoulders, but on aio work queue (aio_wq).
I guess we could do the same optimization we're already doing for ki_filp, for ki_eventfd ...

- Davide

---
 fs/aio.c |   32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
fs/aio.c +24 -8
Index: linux-2.6.mod/fs/aio.c
===================================================================
@@ -485,8 +485,6 @@ static inline void really_put_req(struct
 {
 	assert_spin_locked(&ctx->ctx_lock);
 
-	if (!IS_ERR(req->ki_eventfd))
-		fput(req->ki_eventfd);
 	if (req->ki_dtor)
 		req->ki_dtor(req);
 	if (req->ki_iovec != &req->ki_inline_vec)
@@ -508,8 +506,11 @@ static void aio_fput_routine(struct work
 		list_del(&req->ki_list);
 		spin_unlock_irq(&fput_lock);
 
-		/* Complete the fput */
-		__fput(req->ki_filp);
+		/* Complete the fput(s) */
+		if (req->ki_filp != NULL)
+			__fput(req->ki_filp);
+		if (!IS_ERR(req->ki_eventfd))
+			__fput(req->ki_eventfd);
 
 		/* Link the iocb into the context's free list */
 		spin_lock_irq(&ctx->ctx_lock);
@@ -527,12 +528,14 @@ static void aio_fput_routine(struct work
  */
 static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 {
+	int schedule_putreq = 0;
+
 	dprintk(KERN_DEBUG "aio_put(%p): f_count=%ld\n",
 		req, atomic_long_read(&req->ki_filp->f_count));
 
 	assert_spin_locked(&ctx->ctx_lock);
 
-	req->ki_users --;
+	req->ki_users--;
 	BUG_ON(req->ki_users < 0);
 	if (likely(req->ki_users))
 		return 0;
@@ -540,10 +543,23 @@ static int __aio_put_req(struct kioctx *
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
+	/*
+	 * Try to optimize the aio and eventfd file* puts, by avoiding to
+	 * schedule work in case it is not __fput() time. In normal cases,
+	 * we wouldn not be holding the last reference to the file*, so
+	 * this function will be executed w/out any aio kthread wakeup.
 	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
+	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count)))
+		schedule_putreq++;
+	else
+		req->ki_filp = NULL;
+	if (unlikely(!IS_ERR(req->ki_eventfd))) {
+		if (unlikely(atomic_long_dec_and_test(&req->ki_eventfd->f_count)))
+			schedule_putreq++;
+		else
+			req->ki_eventfd = ERR_PTR(-EINVAL);
+	}
+	if (unlikely(schedule_putreq)) {
 		get_ioctx(ctx);
 		spin_lock(&fput_lock);
 		list_add(&req->ki_list, &fput_head);
#24 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 26 lines · Davide Libenzi
Quoted 24 lines · Trond Myklebust
Quoted 22 lines · Davide Libenzi
Quoted 10 lines · Andrew Morton
Quoted 2 lines

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Would it be a huge problem, performance-wise, to stop making ->f_count tricks in __aio_put_req, and always offload to fput_work the task of releasing the requests?
If that's a huge problem, IMO the lower impact fix would be to use aio_fput_routine to loop into a second list, releasing the eventual eventfd file*. There's no need, IMO, to turn the whole fput() into IRQ-callable just for this case, when we can contain it into the particular KAIO+eventfd usage.

Quoted 8 lines
Quoted 3 lines

Do you really want to see eventd doing umounts and remote flock() calls? This really needs to be run in a thread that can cope with __long__ waits, unavailable servers, etc...

Did I miss something? This wouldn't be (eventually) on keventd shoulders, but on aio work queue (aio_wq).
I guess we could do the same optimization we're already doing for ki_filp, for ki_eventfd ...

Last I checked, a call to schedule_work(&fddef->wq) would still run fd_defer_queue() under keventd.

That said, even if you were to run it under aio_wq, the argument remains the same: do you really want to add potentially long lasting sleeps onto a work queue that was designed to service fast i/o requests?

Trond

#25 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 7 lines · Trond Myklebust
Quoted 4 lines

Did I miss something? This wouldn't be (eventually) on keventd shoulders, but on aio work queue (aio_wq).
I guess we could do the same optimization we're already doing for ki_filp, for ki_eventfd ...

Last I checked, a call to schedule_work(&fddef->wq) would still run fd_defer_queue() under keventd.

Oh, you were commenting about Eric's patch-set. I thought you were talking about the current status (that my reply to Andrew was hinting to), and was wondering for a while from where fd_defer_queue() came from.

Quoted 3 lines

That said, even if you were to run it under aio_wq, the argument remains the same: do you really want to add potentially long lasting sleeps onto a work queue that was designed to service fast i/o requests?

No, and if you roll back, you couldn't miss to notice that mine was a question about the f_count direct manipulation/optimization. We can do the same optimization for eventfd, although both will fall in the aio kthread in any case, if we're holding the last insteance of the file* (that I guess we can consider an unlikely case).
Dunno how worth it could be adding a new NCPU*thread(s) just to handle delayed fputs.

- Davide

#26 lore raw

PATCH 1/1 eventfd - remove fput() call from possible IRQ context

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but conceptually the bug is there.
This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

- Davide

---
 fs/aio.c |   32 ++++++++++++++++++++++++--------
 1 file changed, 24 insertions(+), 8 deletions(-)
fs/aio.c +24 -8
Index: linux-2.6.mod/fs/aio.c
===================================================================
@@ -485,8 +485,6 @@ static inline void really_put_req(struct
 {
 	assert_spin_locked(&ctx->ctx_lock);
 
-	if (!IS_ERR(req->ki_eventfd))
-		fput(req->ki_eventfd);
 	if (req->ki_dtor)
 		req->ki_dtor(req);
 	if (req->ki_iovec != &req->ki_inline_vec)
@@ -508,8 +506,11 @@ static void aio_fput_routine(struct work
 		list_del(&req->ki_list);
 		spin_unlock_irq(&fput_lock);
 
-		/* Complete the fput */
-		__fput(req->ki_filp);
+		/* Complete the fput(s) */
+		if (req->ki_filp != NULL)
+			__fput(req->ki_filp);
+		if (!IS_ERR(req->ki_eventfd))
+			__fput(req->ki_eventfd);
 
 		/* Link the iocb into the context's free list */
 		spin_lock_irq(&ctx->ctx_lock);
@@ -527,12 +528,14 @@ static void aio_fput_routine(struct work
  */
 static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 {
+	int schedule_putreq = 0;
+
 	dprintk(KERN_DEBUG "aio_put(%p): f_count=%ld\n",
 		req, atomic_long_read(&req->ki_filp->f_count));
 
 	assert_spin_locked(&ctx->ctx_lock);
 
-	req->ki_users --;
+	req->ki_users--;
 	BUG_ON(req->ki_users < 0);
 	if (likely(req->ki_users))
 		return 0;
@@ -540,10 +543,23 @@ static int __aio_put_req(struct kioctx *
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
+	/*
+	 * Try to optimize the aio and eventfd file* puts, by avoiding to
+	 * schedule work in case it is not __fput() time. In normal cases,
+	 * we wouldn not be holding the last reference to the file*, so
+	 * this function will be executed w/out any aio kthread wakeup.
 	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
+	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count)))
+		schedule_putreq++;
+	else
+		req->ki_filp = NULL;
+	if (unlikely(!IS_ERR(req->ki_eventfd))) {
+		if (unlikely(atomic_long_dec_and_test(&req->ki_eventfd->f_count)))
+			schedule_putreq++;
+		else
+			req->ki_eventfd = ERR_PTR(-EINVAL);
+	}
+	if (unlikely(schedule_putreq)) {
 		get_ioctx(ctx);
 		spin_lock(&fput_lock);
 		list_add(&req->ki_list, &fput_head);
maintainer
#27 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context

Quoted 7 lines · Davide Libenzi

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but conceptually the bug is there.
This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

This looks reasonably sane, the only concern I have with it is that I think it logically makes more sense to use the same convention for fi_filp and ki_eventfd, as the different in IS_ERR vs checking for NULL is a bit confusing. Aside from that, it looks like it should fix the problem correctly.

-ben

#28 lore raw

PATCH 1/1 eventfd - remove fput() call from possible IRQ context (2nd rev)

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but conceptually the bug is there.
This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

Quoted 5 lines · Benjamin LaHaise

This looks reasonably sane, the only concern I have with it is that I think it logically makes more sense to use the same convention for fi_filp and ki_eventfd, as the different in IS_ERR vs checking for NULL is a bit confusing. Aside from that, it looks like it should fix the problem correctly.

Makes sense.

- Davide

---
 fs/aio.c |   37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)
fs/aio.c +27 -10
Index: linux-2.6.mod/fs/aio.c
===================================================================
@@ -443,7 +443,7 @@ static struct kiocb *__aio_get_req(struc
 	req->private = NULL;
 	req->ki_iovec = NULL;
 	INIT_LIST_HEAD(&req->ki_run_list);
-	req->ki_eventfd = ERR_PTR(-EINVAL);
+	req->ki_eventfd = NULL;
 
 	/* Check if the completion queue has enough free space to
 	 * accept an event from this io.
@@ -485,8 +485,6 @@ static inline void really_put_req(struct
 {
 	assert_spin_locked(&ctx->ctx_lock);
 
-	if (!IS_ERR(req->ki_eventfd))
-		fput(req->ki_eventfd);
 	if (req->ki_dtor)
 		req->ki_dtor(req);
 	if (req->ki_iovec != &req->ki_inline_vec)
@@ -508,8 +506,11 @@ static void aio_fput_routine(struct work
 		list_del(&req->ki_list);
 		spin_unlock_irq(&fput_lock);
 
-		/* Complete the fput */
-		__fput(req->ki_filp);
+		/* Complete the fput(s) */
+		if (req->ki_filp != NULL)
+			__fput(req->ki_filp);
+		if (req->ki_eventfd != NULL)
+			__fput(req->ki_eventfd);
 
 		/* Link the iocb into the context's free list */
 		spin_lock_irq(&ctx->ctx_lock);
@@ -527,12 +528,14 @@ static void aio_fput_routine(struct work
  */
 static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 {
+	int schedule_putreq = 0;
+
 	dprintk(KERN_DEBUG "aio_put(%p): f_count=%ld\n",
 		req, atomic_long_read(&req->ki_filp->f_count));
 
 	assert_spin_locked(&ctx->ctx_lock);
 
-	req->ki_users --;
+	req->ki_users--;
 	BUG_ON(req->ki_users < 0);
 	if (likely(req->ki_users))
 		return 0;
@@ -540,10 +543,23 @@ static int __aio_put_req(struct kioctx *
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
+	/*
+	 * Try to optimize the aio and eventfd file* puts, by avoiding to
+	 * schedule work in case it is not __fput() time. In normal cases,
+	 * we wouldn not be holding the last reference to the file*, so
+	 * this function will be executed w/out any aio kthread wakeup.
 	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
+	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count)))
+		schedule_putreq++;
+	else
+		req->ki_filp = NULL;
+	if (unlikely(req->ki_eventfd != NULL)) {
+		if (unlikely(atomic_long_dec_and_test(&req->ki_eventfd->f_count)))
+			schedule_putreq++;
+		else
+			req->ki_eventfd = NULL;
+	}
+	if (unlikely(schedule_putreq)) {
 		get_ioctx(ctx);
 		spin_lock(&fput_lock);
 		list_add(&req->ki_list, &fput_head);
@@ -1009,7 +1025,7 @@ int aio_complete(struct kiocb *iocb, lon
 	 * eventfd. The eventfd_signal() function is safe to be called
 	 * from IRQ context.
 	 */
-	if (!IS_ERR(iocb->ki_eventfd))
+	if (iocb->ki_eventfd != NULL)
 		eventfd_signal(iocb->ki_eventfd, 1);
 
 put_rq:
@@ -1608,6 +1624,7 @@ static int io_submit_one(struct kioctx *
 		req->ki_eventfd = eventfd_fget((int) iocb->aio_resfd);
 		if (IS_ERR(req->ki_eventfd)) {
 			ret = PTR_ERR(req->ki_eventfd);
+			req->ki_eventfd = NULL;
 			goto out_put_req;
 		}
 	}
#29 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

fs/aio.c static int __aio_put_req(struct kioctx * · line 556

Quoted 1 line · Davide Libenzi
fs/aio.c +1 -0
+	if (unlikely(req->ki_eventfd != NULL)) {

Is this really unlikely? Having an eventfd seems to be the modern, cool, recommended way to do AIO.

The other unlikelies look ok.

-- Jamie

#30 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

Quoted 5 lines · Jamie Lokier

fs/aio.c static int __aio_put_req(struct kioctx * · line 556

Quoted 1 line · Davide Libenzi
fs/aio.c +1 -0
+	if (unlikely(req->ki_eventfd != NULL)) {

Is this really unlikely? Having an eventfd seems to be the modern, cool, recommended way to do AIO.

I don't have statistics at hand, but I guess that nowadays AIO+eventfd is still not very popular. It doesn't make much of a difference either ways I guess, so we could drop the unlikely().

- Davide

author
#31 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

In reply to Davide Libenzi
Quoted 3 lines · Davide Libenzi

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but conceptually the bug is there.

I've attached a test program which can reproduce the fput call in interrupt context. It's a modified version of the eventfd test that Rusty wrote for the libaio test harness. I verified that fput was in fact being called in interrupt context by using systemtap to print out the "thead_indent" of fput calls, and observing a "swapper(0)" in the output. After applying your patch, I confirmed that __fput is no longer called from interrupt context. Strangely enough, I never did get any output from the might_sleep in __fput. I can't explain that.

I have some minor comments inlined below.

Quoted 29 lines

This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

Quoted 5 lines · Benjamin LaHaise

This looks reasonably sane, the only concern I have with it is that I think it logically makes more sense to use the same convention for fi_filp and ki_eventfd, as the different in IS_ERR vs checking for NULL is a bit confusing. Aside from that, it looks like it should fix the problem correctly.

Makes sense.

- Davide

---
 fs/aio.c |   37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)

Index: linux-2.6.mod/fs/aio.c
===================================================================

fs/aio.c +0 -0

...

Quoted 17 lines
fs/aio.c +3 -1
@@ -527,12 +528,14 @@ static void aio_fput_routine(struct work
  */
 static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 {
+	int schedule_putreq = 0;
+
 	dprintk(KERN_DEBUG "aio_put(%p): f_count=%ld\n",
 		req, atomic_long_read(&req->ki_filp->f_count));
 
 	assert_spin_locked(&ctx->ctx_lock);
 
-	req->ki_users --;
+	req->ki_users--;
 	BUG_ON(req->ki_users < 0);
 	if (likely(req->ki_users))
 		return 0;

fs/aio.c static int __aio_put_req(struct kioctx * · line 549

Quoted 9 lines
fs/aio.c +4 -2
@@ -540,10 +543,23 @@ static int __aio_put_req(struct kioctx *
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
+	/*
+	 * Try to optimize the aio and eventfd file* puts, by avoiding to
+	 * schedule work in case it is not __fput() time. In normal cases,
+	 * we wouldn not be holding the last reference to the file*, so
              ^^^^^^^^^^

tyop

fs/aio.c static int __aio_put_req(struct kioctx * · line 561

Quoted 13 lines
fs/aio.c +11 -1
+	 * this function will be executed w/out any aio kthread wakeup.
 	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
+	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count)))
+		schedule_putreq++;
+	else
+		req->ki_filp = NULL;
+	if (unlikely(req->ki_eventfd != NULL)) {
+		if (unlikely(atomic_long_dec_and_test(&req->ki_eventfd->f_count)))
+			schedule_putreq++;
+		else
+			req->ki_eventfd = NULL;
+	}

I agree with Jamie that you should get rid of the unlikely.

Thanks for taking care of this, Davide.

Cheers,
Jeff

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <sys/types.h>
#include <errno.h>
#include <assert.h>
#include <sys/eventfd.h>
#include <libaio.h>

int
main(int argc, char **argv)
{

#define SIZE	(256*1024*1024)
	char *buf;
	struct io_event io_event;
	struct iocb iocb;
	struct iocb *iocbs[] = { &iocb };
	int rwfd, efd;
	int res;
	io_context_t	io_ctx;

	efd = eventfd(0, 0);
	if (efd < 0) {
		perror("eventfd");
		exit(1);
	}

	rwfd = open("rwfile", O_RDWR|O_DIRECT);		assert(rwfd != -1);
	if (posix_memalign((void **)&buf, getpagesize(), SIZE) < 0) {
		perror("posix_memalign");
		exit(1);
	}
	memset(buf, 0x42, SIZE);

	/* Write test. */
	res = io_queue_init(1024, &io_ctx);		assert(res == 0);
	io_prep_pwrite(&iocb, rwfd, buf, SIZE, 0);
	io_set_eventfd(&iocb, efd);
	res = io_submit(io_ctx, 1, iocbs);		assert(res == 1);

	/* Now close the eventfd so that AIO has the last reference */
	close(efd);

	/* Keep this process around so that the aio subsystem does not hold
	 * the last reference on the rwfd, otherwise the really_put_req will
	 * be called from process context */
	res = io_getevents(io_ctx, 1, 1, &io_event, NULL);
	if (res != 1) {
		if (res < 0) {
			errno = -res;
			perror("io_getevents");
		} else
			printf("io_getevents did not return 1 event after "
			       "closing eventfd\n");
		exit(1);
	}
	assert(io_event.res == SIZE);
	printf("eventfd write test [SUCCESS]\n");

	return 0;
}

/*
* Local variables:
* c-basic-offset: 8
* compile-command: "gcc -o eventfd-in-irq eventfd-in-irq.c -laio -g3"
* End:
*/

#32 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

Quoted 1 line · Jeff Moyer

I agree with Jamie that you should get rid of the unlikely.

Ok, will repost today with typo and unlikely fixed.

- Davide

#33 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

In reply to Jeff Moyer
Quoted 132 lines · Jeff Moyer
Quoted 3 lines · Davide Libenzi

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but conceptually the bug is there.

I've attached a test program which can reproduce the fput call in interrupt context. It's a modified version of the eventfd test that Rusty wrote for the libaio test harness. I verified that fput was in fact being called in interrupt context by using systemtap to print out the "thead_indent" of fput calls, and observing a "swapper(0)" in the output. After applying your patch, I confirmed that __fput is no longer called from interrupt context. Strangely enough, I never did get any output from the might_sleep in __fput. I can't explain that.

I have some minor comments inlined below.

Quoted 28 lines

This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

Quoted 5 lines · Benjamin LaHaise

This looks reasonably sane, the only concern I have with it is that I think it logically makes more sense to use the same convention for fi_filp and ki_eventfd, as the different in IS_ERR vs checking for NULL is a bit confusing. Aside from that, it looks like it should fix the problem correctly.

Makes sense.

- Davide

---
 fs/aio.c |   37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)

Index: linux-2.6.mod/fs/aio.c
===================================================================

fs/aio.c +0 -0

...

Quoted 17 lines
fs/aio.c +3 -1
@@ -527,12 +528,14 @@ static void aio_fput_routine(struct work
  */
 static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 {
+	int schedule_putreq = 0;
+
 	dprintk(KERN_DEBUG "aio_put(%p): f_count=%ld\n",
 		req, atomic_long_read(&req->ki_filp->f_count));
 
 	assert_spin_locked(&ctx->ctx_lock);
 
-	req->ki_users --;
+	req->ki_users--;
 	BUG_ON(req->ki_users < 0);
 	if (likely(req->ki_users))
 		return 0;

fs/aio.c static int __aio_put_req(struct kioctx * · line 549

Quoted 9 lines
fs/aio.c +4 -2
@@ -540,10 +543,23 @@ static int __aio_put_req(struct kioctx *
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
+	/*
+	 * Try to optimize the aio and eventfd file* puts, by avoiding to
+	 * schedule work in case it is not __fput() time. In normal cases,
+	 * we wouldn not be holding the last reference to the file*, so
              ^^^^^^^^^^

tyop

fs/aio.c static int __aio_put_req(struct kioctx * · line 561

Quoted 13 lines
fs/aio.c +11 -1
+	 * this function will be executed w/out any aio kthread wakeup.
 	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
+	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count)))
+		schedule_putreq++;
+	else
+		req->ki_filp = NULL;
+	if (unlikely(req->ki_eventfd != NULL)) {
+		if (unlikely(atomic_long_dec_and_test(&req->ki_eventfd->f_count)))
+			schedule_putreq++;
+		else
+			req->ki_eventfd = NULL;
+	}

I agree with Jamie that you should get rid of the unlikely.

Thanks for taking care of this, Davide.

Cheers,
Jeff

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <sys/types.h>
#include <errno.h>
#include <assert.h>
#include <sys/eventfd.h>
#include <libaio.h>

int
main(int argc, char **argv)
{

#define SIZE	(256*1024*1024)
	char *buf;
	struct io_event io_event;
	struct iocb iocb;
	struct iocb *iocbs[] = { &iocb };
	int rwfd, efd;
	int res;
	io_context_t	io_ctx;

	efd = eventfd(0, 0);
	if (efd < 0) {
		perror("eventfd");
		exit(1);
	}

	rwfd = open("rwfile", O_RDWR|O_DIRECT);		assert(rwfd != -1);
	if (posix_memalign((void **)&buf, getpagesize(), SIZE) < 0) {
		perror("posix_memalign");
		exit(1);
Quoted 8 lines
	}
	memset(buf, 0x42, SIZE);

	/* Write test. */
	res = io_queue_init(1024, &io_ctx);		assert(res == 0);
	io_prep_pwrite(&iocb, rwfd, buf, SIZE, 0);
	io_set_eventfd(&iocb, efd);
	res = io_submit(io_ctx, 1, iocbs);		assert(res == 1);

yes but io_submit() is blocking. so your close(efd) will come after the release in fs/aio.c

I suggest you start a thread just before io_submit() and give it this work :

void *thread_work(void *arg)

{
	usleep(10000);
	close(efd);
	return (void *)0;
}
Quoted 30 lines
	/* Now close the eventfd so that AIO has the last reference */
	close(efd);

	/* Keep this process around so that the aio subsystem does not hold
	 * the last reference on the rwfd, otherwise the really_put_req will
	 * be called from process context */
	res = io_getevents(io_ctx, 1, 1, &io_event, NULL);
	if (res != 1) {
		if (res < 0) {
			errno = -res;
			perror("io_getevents");
		} else
			printf("io_getevents did not return 1 event after "
			       "closing eventfd\n");
		exit(1);
	}
	assert(io_event.res == SIZE);
	printf("eventfd write test [SUCCESS]\n");

	return 0;
}

/*
* Local variables:
* c-basic-offset: 8
* compile-command: "gcc -o eventfd-in-irq eventfd-in-irq.c -laio -g3"
* End:
*/

author
#34 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

Quoted 7 lines · Eric Dumazet
Quoted 7 lines
	rwfd = open("rwfile", O_RDWR|O_DIRECT);		assert(rwfd != -1);
	if (posix_memalign((void **)&buf, getpagesize(), SIZE) < 0) {
		perror("posix_memalign");
		exit(1);
	}
	memset(buf, 0x42, SIZE);
Quoted 7 lines
Quoted 5 lines
	/* Write test. */
	res = io_queue_init(1024, &io_ctx);		assert(res == 0);
	io_prep_pwrite(&iocb, rwfd, buf, SIZE, 0);
	io_set_eventfd(&iocb, efd);
	res = io_submit(io_ctx, 1, iocbs);		assert(res == 1);

yes but io_submit() is blocking. so your close(efd) will come after the release in fs/aio.c

I'm not sure why you think io_submit is blocking. In my setup, I preallocated the file, and the test code opens it with O_DIRECT. So, io_submit should return after the dio is issued, and the I/O size is large enough that it should still be outstanding when io_submit returns.

Cheers,
Jeff

#35 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

Quoted 15 lines · Jeff Moyer
Quoted 12 lines · Eric Dumazet
Quoted 12 lines
	rwfd = open("rwfile", O_RDWR|O_DIRECT);		assert(rwfd != -1);
	if (posix_memalign((void **)&buf, getpagesize(), SIZE) < 0) {
		perror("posix_memalign");
		exit(1);
	}
	memset(buf, 0x42, SIZE);

	/* Write test. */
	res = io_queue_init(1024, &io_ctx);		assert(res == 0);
	io_prep_pwrite(&iocb, rwfd, buf, SIZE, 0);
	io_set_eventfd(&iocb, efd);
	res = io_submit(io_ctx, 1, iocbs);		assert(res == 1);
Quoted 6 lines
Quoted 1 line

yes but io_submit() is blocking. so your close(efd) will come after the release in fs/aio.c

I'm not sure why you think io_submit is blocking. In my setup, I preallocated the file, and the test code opens it with O_DIRECT. So, io_submit should return after the dio is issued, and the I/O size is large enough that it should still be outstanding when io_submit returns.

Hmm.. io_submit() is a blocking syscall, this is how I understood fs/aio.c

Then, using strace -tt -T on your program, I can confirm it is quite a long syscall (3.5 seconds, about time needed to write a 256 MB file on my disk ;) )

16:33:15.861301 eventfd(0) = 3 <0.000010>
16:33:15.861344 open("rwfile", O_RDWR|O_DIRECT) = 4 <0.000010>
16:33:15.861444 mmap(NULL, 268443648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f08eec55000 <0.000008> 16:33:16.100040 io_setup(1024, {139676618563584}) = 0 <0.000047>
16:33:16.100149 io_submit(139676618563584, 1, {...}) = 1 <3.558085>
16:33:19.658292 io_getevents(139676618563584, 1, 1, {...}NULL) = 1 <0.258104>
16:33:19.916486 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(136, 0), ...}) = 0 <0.000007> 16:33:19.916559 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f08ff3b9000 <0.000009> 16:33:19.916607 write(1, "eventfd write test [SUCCESS]\n"..., 29eventfd write test [SUCCESS]
) = 29 <0.000008>
16:33:19.916653 exit_group(0) = ?

author
#36 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (2nd rev)

Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Jeff Moyer
Quoted 12 lines · Eric Dumazet
Quoted 12 lines
	rwfd = open("rwfile", O_RDWR|O_DIRECT);		assert(rwfd != -1);
	if (posix_memalign((void **)&buf, getpagesize(), SIZE) < 0) {
		perror("posix_memalign");
		exit(1);
	}
	memset(buf, 0x42, SIZE);

	/* Write test. */
	res = io_queue_init(1024, &io_ctx);		assert(res == 0);
	io_prep_pwrite(&iocb, rwfd, buf, SIZE, 0);
	io_set_eventfd(&iocb, efd);
	res = io_submit(io_ctx, 1, iocbs);		assert(res == 1);
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

yes but io_submit() is blocking. so your close(efd) will come after the release in fs/aio.c

I'm not sure why you think io_submit is blocking. In my setup, I preallocated the file, and the test code opens it with O_DIRECT. So, io_submit should return after the dio is issued, and the I/O size is large enough that it should still be outstanding when io_submit returns.

Hmm.. io_submit() is a blocking syscall, this is how I understood fs/aio.c

Hi, Eric,

The whole point of io_submit is to allow you to submit I/O without waiting for it. There are known cases where io_submit will block, of course, such as when we run out of request descriptors. See the io_submit.stp script for some examples.[1]

Now, I admit I was testing using an SSD, so I didn't actually notice the time it took for the 256MB write (!!!). I tried the reproducer I posted on my F9 box, and here is the output I get:

BUG: sleeping function called from invalid context at

fs/file_table.c:262
in_atomic():1, irqs_disabled():1
Pid: 0, comm: swapper Not tainted 2.6.27.15-78.2.23.fc9.x86_64 #1

Call Trace:
 <IRQ>  [<ffffffff8103892e>] __might_sleep+0xe7/0xec
 [<ffffffff810bfa86>] __fput+0x35/0x16d
 [<ffffffff810bfbd3>] fput+0x15/0x17
 [<ffffffff810d71bb>] really_put_req+0x34/0x9c
 [<ffffffff810d72f0>] __aio_put_req+0xcd/0xda
 [<ffffffff810d7f77>] aio_complete+0x15d/0x19f
 [<ffffffff810e7016>] dio_bio_end_aio+0x8e/0xa0
 [<ffffffff810e32ab>] bio_endio+0x2a/0x2c
 [<ffffffff8113beae>] req_bio_endio+0x9d/0xba
 [<ffffffff8113c073>] __end_that_request_first+0x1a8/0x2b5
 [<ffffffff8113cb89>] blk_end_io+0x2f/0xa9
 [<ffffffff8113cc2f>] blk_end_request+0xe/0x10
 [<ffffffffa005d30b>] scsi_end_request+0x30/0x90 [scsi_mod]
 [<ffffffffa005d9e9>] scsi_io_completion+0x1aa/0x3b3 [scsi_mod]
 [<ffffffffa0057658>] scsi_finish_command+0xde/0xe7 [scsi_mod]
 [<ffffffffa005de68>] scsi_softirq_done+0xe4/0xed [scsi_mod]
 [<ffffffff8113baa8>] blk_done_softirq+0x7e/0x8e
 [<ffffffff81045146>] __do_softirq+0x7e/0x10c
 [<ffffffff81011bfc>] call_softirq+0x1c/0x28
 [<ffffffff81012e06>] do_softirq+0x4d/0xb0
 [<ffffffff81044d1b>] irq_exit+0x4e/0x9d
 [<ffffffff81013122>] do_IRQ+0x147/0x169
 [<ffffffff81010963>] ret_from_intr+0x0/0x2e
 <EOI>  [<ffffffff810173a9>] ? mwait_idle+0x3e/0x4f
 [<ffffffff810173a0>] ? mwait_idle+0x35/0x4f
 [<ffffffff8100f2a7>] ? cpu_idle+0xb2/0x10b
 [<ffffffff812af35d>] ? rest_init+0x61/0x63

So, I think it is a valid reproducer as it stands.

Quoted 2 lines

Then, using strace -tt -T on your program, I can confirm it is quite a long syscall (3.5 seconds, about time needed to write a 256 MB file on my disk ;) )

Did you preallocate the file?

Cheers,
Jeff

[1] http://sourceware.org/systemtap/wiki/ScriptsTools?action=AttachFile&do=view&target=io_submit.stp

#37 lore raw

PATCH 1/1 eventfd - remove fput() call from possible IRQ context (3rd rev)

In reply to Jeff Moyer

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but Jeff said he was able to, with the attached test program. Independently from this, the bug is conceptually there, so we might be better off fixing it.
This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

Quoted 5 lines · Benjamin LaHaise

This looks reasonably sane, the only concern I have with it is that I think it logically makes more sense to use the same convention for fi_filp and ki_eventfd, as the different in IS_ERR vs checking for NULL is a bit confusing. Aside from that, it looks like it should fix the problem correctly.

Makes sense.

- Davide

---
 fs/aio.c |   37 +++++++++++++++++++++++++++----------
 1 file changed, 27 insertions(+), 10 deletions(-)
fs/aio.c +27 -10
Index: linux-2.6.mod/fs/aio.c
===================================================================
@@ -443,7 +443,7 @@ static struct kiocb *__aio_get_req(struc
 	req->private = NULL;
 	req->ki_iovec = NULL;
 	INIT_LIST_HEAD(&req->ki_run_list);
-	req->ki_eventfd = ERR_PTR(-EINVAL);
+	req->ki_eventfd = NULL;
 
 	/* Check if the completion queue has enough free space to
 	 * accept an event from this io.
@@ -485,8 +485,6 @@ static inline void really_put_req(struct
 {
 	assert_spin_locked(&ctx->ctx_lock);
 
-	if (!IS_ERR(req->ki_eventfd))
-		fput(req->ki_eventfd);
 	if (req->ki_dtor)
 		req->ki_dtor(req);
 	if (req->ki_iovec != &req->ki_inline_vec)
@@ -508,8 +506,11 @@ static void aio_fput_routine(struct work
 		list_del(&req->ki_list);
 		spin_unlock_irq(&fput_lock);
 
-		/* Complete the fput */
-		__fput(req->ki_filp);
+		/* Complete the fput(s) */
+		if (req->ki_filp != NULL)
+			__fput(req->ki_filp);
+		if (req->ki_eventfd != NULL)
+			__fput(req->ki_eventfd);
 
 		/* Link the iocb into the context's free list */
 		spin_lock_irq(&ctx->ctx_lock);
@@ -527,12 +528,14 @@ static void aio_fput_routine(struct work
  */
 static int __aio_put_req(struct kioctx *ctx, struct kiocb *req)
 {
+	int schedule_putreq = 0;
+
 	dprintk(KERN_DEBUG "aio_put(%p): f_count=%ld\n",
 		req, atomic_long_read(&req->ki_filp->f_count));
 
 	assert_spin_locked(&ctx->ctx_lock);
 
-	req->ki_users --;
+	req->ki_users--;
 	BUG_ON(req->ki_users < 0);
 	if (likely(req->ki_users))
 		return 0;
@@ -540,10 +543,23 @@ static int __aio_put_req(struct kioctx *
 	req->ki_cancel = NULL;
 	req->ki_retry = NULL;
 
-	/* Must be done under the lock to serialise against cancellation.
-	 * Call this aio_fput as it duplicates fput via the fput_work.
+	/*
+	 * Try to optimize the aio and eventfd file* puts, by avoiding to
+	 * schedule work in case it is not __fput() time. In normal cases,
+	 * we would not be holding the last reference to the file*, so
+	 * this function will be executed w/out any aio kthread wakeup.
 	 */
-	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
+	if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count)))
+		schedule_putreq++;
+	else
+		req->ki_filp = NULL;
+	if (req->ki_eventfd != NULL) {
+		if (unlikely(atomic_long_dec_and_test(&req->ki_eventfd->f_count)))
+			schedule_putreq++;
+		else
+			req->ki_eventfd = NULL;
+	}
+	if (unlikely(schedule_putreq)) {
 		get_ioctx(ctx);
 		spin_lock(&fput_lock);
 		list_add(&req->ki_list, &fput_head);
@@ -1009,7 +1025,7 @@ int aio_complete(struct kiocb *iocb, lon
 	 * eventfd. The eventfd_signal() function is safe to be called
 	 * from IRQ context.
 	 */
-	if (!IS_ERR(iocb->ki_eventfd))
+	if (iocb->ki_eventfd != NULL)
 		eventfd_signal(iocb->ki_eventfd, 1);
 
 put_rq:
@@ -1608,6 +1624,7 @@ static int io_submit_one(struct kioctx *
 		req->ki_eventfd = eventfd_fget((int) iocb->aio_resfd);
 		if (IS_ERR(req->ki_eventfd)) {
 			ret = PTR_ERR(req->ki_eventfd);
+			req->ki_eventfd = NULL;
 			goto out_put_req;
 		}
 	}

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <sys/types.h>
#include <errno.h>
#include <assert.h>
#include <sys/eventfd.h>
#include <libaio.h>

int
main(int argc, char **argv)
{

#define SIZE	(256*1024*1024)
	char *buf;
	struct io_event io_event;
	struct iocb iocb;
	struct iocb *iocbs[] = { &iocb };
	int rwfd, efd;
	int res;
	io_context_t	io_ctx;

	efd = eventfd(0, 0);
	if (efd < 0) {
		perror("eventfd");
		exit(1);
	}

	rwfd = open("rwfile", O_RDWR|O_DIRECT);		assert(rwfd != -1);
	if (posix_memalign((void **)&buf, getpagesize(), SIZE) < 0) {
		perror("posix_memalign");
		exit(1);
	}
	memset(buf, 0x42, SIZE);

	/* Write test. */
	res = io_queue_init(1024, &io_ctx);		assert(res == 0);
	io_prep_pwrite(&iocb, rwfd, buf, SIZE, 0);
	io_set_eventfd(&iocb, efd);
	res = io_submit(io_ctx, 1, iocbs);		assert(res == 1);

	/* Now close the eventfd so that AIO has the last reference */
	close(efd);

	/* Keep this process around so that the aio subsystem does not hold
	 * the last reference on the rwfd, otherwise the really_put_req will
	 * be called from process context */
	res = io_getevents(io_ctx, 1, 1, &io_event, NULL);
	if (res != 1) {
		if (res < 0) {
			errno = -res;
			perror("io_getevents");
		} else
			printf("io_getevents did not return 1 event after "
			       "closing eventfd\n");
		exit(1);
	}
	assert(io_event.res == SIZE);
	printf("eventfd write test [SUCCESS]\n");

	return 0;
}
author
#38 lore raw

Re: [patch] eventfd - remove fput() call from possible IRQ context (3rd rev)

Quoted 19 lines · Davide Libenzi

The following patch remove a possible source of fput() call from inside IRQ context. Myself, like Eric, wasn't able to reproduce an fput() call from IRQ context, but Jeff said he was able to, with the attached test program. Independently from this, the bug is conceptually there, so we might be better off fixing it.
This patch adds an optimization similar to the one we already do on ->ki_filp, on ->ki_eventfd. Playing with ->f_count directly is not pretty in general, but the alternative here would be to add a brand new delayed fput() infrastructure, that I'm not sure is worth it.

Quoted 5 lines · Benjamin LaHaise

This looks reasonably sane, the only concern I have with it is that I think it logically makes more sense to use the same convention for fi_filp and ki_eventfd, as the different in IS_ERR vs checking for NULL is a bit confusing. Aside from that, it looks like it should fix the problem correctly.

Quoted 5 lines

Makes sense.

#39 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

In reply to Andrew Morton
Quoted 11 lines · Andrew Morton
Quoted 7 lines · On Thu, 12 Mar 2009 07:10:38 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 1 line

Did you reproduce the bug, and confirm that the patch fixes it?

take Davide program : http://www.xmailserver.org/eventfd-aio-test.c

and add at line 318 :
close(afd);

It should produce the kernel bug...

"should"?

Sorry I had to run this morning, so I was a litle bit lazy...

Maybe some kind of O_DIRECT / NFS / blockdev setup or something calling aio_complete() from interrupt context ? I am not familiar of this part, but considering spin_lock_irq()/spin_unlock_irq() calls in fs/aio.c, there must be a reason for this ?

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Are there simpler ways of fixing it? Maybe sneak a call to wait_for_all_aios() into the right place? I doubt if it's performance critical, as nobody seems to have ever hit the bug.

Take the time to check how fs/aio.c handle the fput(req->ki_filp) case (or read my 2nd patch, it should spot the thing)

Quoted 8 lines

Well yes, a kludge like that seems a bit safer.

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Hum... Do you mean "if (in_interrupt())" is not the right test to perform ?

Quoted 4 lines
Quoted 4 lines

If you want to add another kludge to properly fput(req->ki_eventfd), be my guest :-(

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Bear in mind that if the bug _is_ real then it's now out there, and we would like a fix which is usable by 2.6.<two-years-worth>.

The patches are large and scary and it would be a real problem to merge them into 2.6.29 at this stage, let alone 2.6.25, etc.

Especially as the code which you sent out appears to be untested:

I actually tested it and got a working kernel before sending patch, please trust me...

Oh yes, my dev machine always called the slow path then, this is why I didnt noticed.

Thank you for spotting this inverted test.

Quoted 15 lines

fs/file_table.c int init_file(struct file *file, struct vfsmount *mnt, struct dentry *dentry, · line 228

Quoted 6 lines

void fput(struct file *file)
{

fs/file_table.c +2 -2
-	if (atomic_long_dec_and_test(&file->f_count))
-		__fput(file);
+	if (atomic_long_dec_and_test(&file->f_count)) {
+		if (unlikely(!in_interrupt()))
                             ^
Quoted 5 lines
fs/file_table.c +4 -0
+			fd_defer_queue(NULL, file);
+		else
+			__fput(file);
+	}
 }
maintainer
#40 lore raw

Re: [PATCH] fs: fput() can be called from interrupt context

Quoted 9 lines · On Thu, 12 Mar 2009 20:22:06 +0100 Eric Dumazet <[email protected]> wrote:
Quoted 6 lines

It's somewhat encouraging that we're apparently already doing fput() from within keventd (although how frequently?). There might be problems with file locking, security code, etc from doing fput() from an unexpected thread. And then there are all the usual weird problem with using the keventd queues which take a long time to get discovered.

Hum... Do you mean "if (in_interrupt())" is not the right test to perform ?

No, in_interrupt() seems appropriate.

It's a bit weak, because it means that the proposed code can't be used for calling fput() inside spinlock. A better interface would be to add a new fput_atomic() and let the caller decide which functions should be called, depending upon what context the caller is running in.

maintainer
#41 lore raw
In reply to Eric Dumazet
Quoted 2 lines · Eric Dumazet

Assuming that __fput() cannot be called from interrupt context. -> fput() should not be called from interrupt context as well.

It doesn't get called in irq context. The atomic dec is done in irq context, and if that is the last user of the file descriptor, it's placed on a work queue and then released in process context.

-ben

#42 lore raw
Quoted 8 lines · Benjamin LaHaise
Quoted 2 lines · Eric Dumazet

Assuming that __fput() cannot be called from interrupt context. -> fput() should not be called from interrupt context as well.

It doesn't get called in irq context. The atomic dec is done in irq context, and if that is the last user of the file descriptor, it's placed on a work queue and then released in process context.

really_put_req() itself can be called from interrupt context, unless I am wrong. (its 4:24 am here ;) )

        if (unlikely(atomic_long_dec_and_test(&req->ki_filp->f_count))) {
                get_ioctx(ctx);
                spin_lock(&fput_lock);
                list_add(&req->ki_list, &fput_head);
                spin_unlock(&fput_lock);
                queue_work(aio_wq, &fput_work);
        } else
                really_put_req(ctx, req); /* from interrupt context */

static inline void really_put_req(struct kioctx *ctx, struct kiocb *req)

{
        assert_spin_locked(&ctx->ctx_lock);

        if (!IS_ERR(req->ki_eventfd))
                fput(req->ki_eventfd);   /* BANG : can be called from interrupt context */
        if (req->ki_dtor)
                req->ki_dtor(req);
        if (req->ki_iovec != &req->ki_inline_vec)
                kfree(req->ki_iovec);
        kmem_cache_free(kiocb_cachep, req);
        ctx->reqs_active--;

        if (unlikely(!ctx->reqs_active && ctx->dead))
                wake_up(&ctx->wait);
}

Thank you

maintainer
#43 lore raw
Quoted 2 lines · Eric Dumazet
        if (!IS_ERR(req->ki_eventfd))
                fput(req->ki_eventfd);   /* BANG : can be called from interrupt context */

...

Quoted 1 line

Thank you

That's a bug in the eventfd code, not aio. Davide: please fix.

-ben

#44 lore raw
Quoted 8 lines · Benjamin LaHaise
Quoted 2 lines · Eric Dumazet
        if (!IS_ERR(req->ki_eventfd))
                fput(req->ki_eventfd);   /* BANG : can be called from interrupt context */

...

Quoted 1 line

Thank you

That's a bug in the eventfd code, not aio. Davide: please fix.

Hmm... what about fget_light() ... is it Davide fault too ?

aio breaks the fget_light() concept too, if process is mono threaded.

/*
* Lightweight file lookup - no refcnt increment if fd table isn't shared.
* You can use this only if it is guranteed that the current task already
* holds a refcnt to that file. That check has to be done at fget() only
* and a flag is returned to be passed to the corresponding fput_light().
* There must not be a cloning between an fget_light/fput_light pair. */
struct file *fget_light(unsigned int fd, int *fput_needed)

{
        struct file *file;
        struct files_struct *files = current->files;

        *fput_needed = 0;
        if (likely((atomic_read(&files->count) == 1))) {
                file = fcheck_files(files, fd);
        } else {
                rcu_read_lock();
                file = fcheck_files(files, fd);
                if (file) {
                        if (atomic_long_inc_not_zero(&file->f_count))
                                *fput_needed = 1;
                        else
                                /* Didn't get the reference, someone's freed */
                                file = NULL;
                }
                rcu_read_unlock();
        }

        return file;
}
maintainer
#45 lore raw
Quoted 1 line · Eric Dumazet

aio breaks the fget_light() concept too, if process is mono threaded.

AIO requests cannot use fget_light(). The user space program could very well use the close() syscall to destroy the file descriptor while an io request is in flight, so you cannot avoid the reference counting overhead.

-ben

#46 lore raw
Quoted 6 lines · Benjamin LaHaise
Quoted 1 line · Eric Dumazet

aio breaks the fget_light() concept too, if process is mono threaded.

AIO requests cannot use fget_light(). The user space program could very well use the close() syscall to destroy the file descriptor while an io request is in flight, so you cannot avoid the reference counting overhead.

Yes, you are right, AIO doesnt break fget_light()

maintainer
#47 lore raw
In reply to Jeff Moyer
Quoted 53 lines · On Mon, 09 Mar 2009 11:49:57 -0400 Jeff Moyer <[email protected]> wrote:

Hi,

Believe it or not, I get numerous questions from customers about the suggested tuning value of aio-max-nr. aio-max-nr limits the total number of io events that can be reserved, system wide, for aio completions. Each time io_setup is called, a ring buffer is allocated that can hold nr_events I/O completions. That ring buffer is then mapped into the process' address space, and the pages are pinned in memory. So, the reason for this upper limit (I believe) is to keep a malicious user from pinning all of kernel memory. Now, this sounds like a much better job for the memlock rlimit to me, hence the following patch.

It's worth noting that, by default, aio-max-nr was set to 65536 requests. By default, the memlock rlimit is set to 64kb per process. With this patch in place, a single process can specify 2045 for the nr_events parameter of io_setup. Or, for the worst case scenario, a process can only create 16 io contexts, each with a single event (since the minimum ring size is a page).

I created a simple program that sets the process' memlock rlimit and then drops the CAP_IPC_LOCK capability (the memlock rlimit is a command line parameter, so you can see how the nr_events allowable changes with different limits in place). Then, it calls io_setup/io_destroy in a loop, increasing nr_events until it fails. Finally, it calls io_setup in a loop with a single event to see how many iterations it can perform before receiving -EAGAIN. I ran the test on a patched kernel and the results are in line with expectations.

I also ran the aio-dio-regress regression tests, and all passed but one. The one that failed only failed due to an expected return code of -ENOMEM, and instead got -EAGAIN. Part of the patch below returns a proper error code from aio_setup_ring. So, I'm calling this a test issue, but we can debate this nuance if it is deemed important.

Further, as a result of this exercise, I noticed that there are numerous places in the kernel that test to see if locking memory will exceed the maximum amount of allowed locked memory. I've factored out that bit of code in a follow-on patch that I will post.

Finally, I updated the aio man pages to reflect this change (and fix references to aio_context_t that should be io_context_t).

You can find a copy of the test program I used at:
http://people.redhat.com/jmoyer/aio/ioctx_alloc.c
I'll apologize in advance for the crudity of the test code.

For those reviewing the below patch, it may be worth looking at:

commit d55b5fdaf40846221d543937b786956e27837fda

Author: Zach Brown <[email protected]>
Date:   Mon Nov 7 00:59:31 2005 -0800
Quoted 6 lines

[PATCH] aio: remove aio_max_nr accounting race

The below patch basically reverts the above commit. There is no accounting race now, because we are charging per-process rlimits instead of a system-wide maximum number of aio requests. This has the added benefit of reducing some code complexity.

It's risky to simply remove an existing tunable. What if someone's mission-critical startup script which is provided by a super-slow or even no-longer-in-business vendor does

	if (write(aoi-max-nr, something) == error)
		crash_and_burn()

?

It would be prudent to have a more cautious update scheme. Leave the existing tunable in place. Keep it working if possible. If someone uses it, blurt out a loud printk telling them that they're using a deprecated interface and informing them how to update.

Then at some later time we can remove the old interface.

fs/aio.c line 47

Quoted 6 lines

/*------ sysctl variables----*/

fs/aio.c +2 -3
-static DEFINE_SPINLOCK(aio_nr_lock);
-unsigned long aio_nr;		/* current system wide number of aio requests */
-unsigned long aio_max_nr = 0x10000; /* system wide maximum number of aio requests */
+/* current system wide number of aio requests */
+atomic_t aio_nr = ATOMIC_INIT(0);

Was it a conscious decision to downgrade this from a long' type to an int' type? It _could_ have used atomic_long_t.

author
#48 lore raw
Quoted 7 lines · Andrew Morton

It's risky to simply remove an existing tunable. What if someone's mission-critical startup script which is provided by a super-slow or even no-longer-in-business vendor does

	if (write(aoi-max-nr, something) == error)
		crash_and_burn()
Quoted 8 lines

?

It would be prudent to have a more cautious update scheme. Leave the existing tunable in place. Keep it working if possible. If someone uses it, blurt out a loud printk telling them that they're using a deprecated interface and informing them how to update.

Then at some later time we can remove the old interface.

You are absolutely right. The more I think about this, the less enthused I am about it. I still believe the change is the right way to go, but there are years worth of deployments using the current scheme, and there are existing documents detailing how to work within that scheme. I hereby rescind this patch.

I will follow-up with the memlock cleanup, though; let me know if you have comments on that patch. Zach mentioned that he hates my new function name (can_mlock_pages), so I guess at least that will change. ;)

Cheers,
Jeff