Skip to contentSkip to conversation

rearrange kioctx and aio_ring_info

PATCH In review Chen, Kenneth W 4 participants 5 messages Oct 12, 2006 – Oct 13, 2006 Aio

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rearrange kioctx and aio_ring_info

The following patch is result of our kernel data latency analysis on a large database setup (128 logical CPU, 1 TB memory, a couple thousand disks). The analysis is done at an individual cache line level through hardware performance monitor unit. What we did essentially is to collect data access latency on every kernel memory address and look for top 50 longest average access latency. The hardware performance unit also keeps track of instruction address for each collected data access. This allows us to further associate each of the top 50 cache lines to instruction sequences that caused long latency.

One structure stood out in the woods is the AIO kioctx structure. In particular, kioctx->users and kioctx->ctx_lock are the two hottest variables within one cache line (there are multiple of these cache lines since db app uses multiple kioctx). What also appears to be a more serious cacheline conflict between io_submit_one and aio_complete path: aio_complete() finishes process an iocb, unlocks ctx->ctx_lock (which implies it just lost ownership of that cache line), turns around and access wait->task_list and also does an atomic update to ctx->users. I think that is causing cache lines to be bounced around when another process took the ctx_lock in between.

So here we go, I'd like to take this opportunity to rearrange kioctx, put variables that used together in close proximity. The rearrangement doesn't change number of cache line touched in io_submit_one or io_getevents, however, it reduces one line in aio_complete path. Also I think we ought to watch out on updating variable that is in the same cache line of ctx_lock.

Thoughts? Comments? Flames?

include/linux/aio.h +13 -16
diff -Nurp linux-2.6.18/include/linux/aio.h linux-2.6.18.ken/include/linux/aio.h
@@ -156,41 +156,38 @@ struct aio_ring {
 
 #define AIO_RING_PAGES	8
 struct aio_ring_info {
-	unsigned long		mmap_base;
-	unsigned long		mmap_size;
-
-	struct page		**ring_pages;
 	spinlock_t		ring_lock;
-	long			nr_pages;
-
+	int			nr_pages;
 	unsigned		nr, tail;
 
+	struct page		**ring_pages;
 	struct page		*internal_pages[AIO_RING_PAGES];
+
+	unsigned long		mmap_base;
+	unsigned long		mmap_size;
 };
 
 struct kioctx {
 	atomic_t		users;
 	int			dead;
-	struct mm_struct	*mm;
-
-	/* This needs improving */
-	unsigned long		user_id;
-	struct kioctx		*next;
-
 	wait_queue_head_t	wait;
 
 	spinlock_t		ctx_lock;
-
 	int			reqs_active;
 	struct list_head	active_reqs;	/* used for cancellation */
 	struct list_head	run_list;	/* used for kicked reqs */
 
-	/* sys_io_setup currently limits this to an unsigned int */
-	unsigned		max_reqs;
-
 	struct aio_ring_info	ring_info;
 
+	/* This needs improving */
+	unsigned long		user_id;
+	struct kioctx		*next;
+	struct mm_struct	*mm;
+
 	struct work_struct	wq;
+
+	/* sys_io_setup currently limits this to an unsigned int */
+	unsigned		max_reqs;
 };
 
 /* prototypes */
#1 lore raw
Quoted 2 lines

Thoughts? Comments? Flames?

Well, to start there should be a comment in the code warning future readers about the specific rational behind the resulting ordering.

I'll try to think hardware about it next week.

- z

#2 lore raw
Quoted 2 lines

I'll try to think hardware about it next week.

("harder", "hardware", who's to say?!)

#3 lore raw
In reply to Chen, Kenneth W
Quoted 9 lines · Kenneth W

The following patch is result of our kernel data latency analysis on a large database setup (128 logical CPU, 1 TB memory, a couple thousand disks). The analysis is done at an individual cache line level through hardware performance monitor unit. What we did essentially is to collect data access latency on every kernel memory address and look for top 50 longest average access latency. The hardware performance unit also keeps track of instruction address for each collected data access. This allows us to further associate each of the top 50 cache lines to instruction sequences that caused long latency.

Do you have performance numbers of before/after this change ? How much is it worth?

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#4 lore raw

Dave Jones wrote on Thursday, October 12, 2006 5:04 PM

Quoted 11 lines · Kenneth W
Quoted 9 lines

The following patch is result of our kernel data latency analysis on a large database setup (128 logical CPU, 1 TB memory, a couple thousand disks). The analysis is done at an individual cache line level through hardware performance monitor unit. What we did essentially is to collect data access latency on every kernel memory address and look for top 50 longest average access latency. The hardware performance unit also keeps track of instruction address for each collected data access. This allows us to further associate each of the top 50 cache lines to instruction sequences that caused long latency.

Quoted 1 line

Do you have performance numbers of before/after this change ?

Not yet, I just queued up experiment on our internal benchmark setup. I posted the patch here to get some early discussion on how to attack the problem.