rearrange kioctx and aio_ring_info
PATCH 1/1 rearrange kioctx and aio_ring_info
The following patch is result of our kernel data latency analysis on a large database setup (128 logical CPU, 1 TB memory, a couple thousand disks). The analysis is done at an individual cache line level through hardware performance monitor unit. What we did essentially is to collect data access latency on every kernel memory address and look for top 50 longest average access latency. The hardware performance unit also keeps track of instruction address for each collected data access. This allows us to further associate each of the top 50 cache lines to instruction sequences that caused long latency.
One structure stood out in the woods is the AIO kioctx structure. In particular, kioctx->users and kioctx->ctx_lock are the two hottest variables within one cache line (there are multiple of these cache lines since db app uses multiple kioctx). What also appears to be a more serious cacheline conflict between io_submit_one and aio_complete path: aio_complete() finishes process an iocb, unlocks ctx->ctx_lock (which implies it just lost ownership of that cache line), turns around and access wait->task_list and also does an atomic update to ctx->users. I think that is causing cache lines to be bounced around when another process took the ctx_lock in between.
So here we go, I'd like to take this opportunity to rearrange kioctx, put variables that used together in close proximity. The rearrangement doesn't change number of cache line touched in io_submit_one or io_getevents, however, it reduces one line in aio_complete path. Also I think we ought to watch out on updating variable that is in the same cache line of ctx_lock.
Thoughts? Comments? Flames?
include/linux/aio.h +13 -16
@@ -156,41 +156,38 @@ struct aio_ring {#define AIO_RING_PAGES 8struct aio_ring_info { -unsigned long mmap_base; -unsigned long mmap_size; - -struct page **ring_pages;spinlock_t ring_lock; -long nr_pages; - +int nr_pages;unsigned nr, tail; +struct page **ring_pages;struct page *internal_pages[AIO_RING_PAGES ]; + +unsigned long mmap_base; +unsigned long mmap_size; };struct kioctx {atomic_t users;int dead; -struct mm_struct *mm; - -/* This needs improving */ -unsigned long user_id; -struct kioctx *next; -wait_queue_head_t wait;spinlock_t ctx_lock; -int reqs_active;struct list_head active_reqs;/* used for cancellation */ struct list_head run_list;/* used for kicked reqs */ -/* sys_io_setup currently limits this to an unsigned int */ -unsigned max_reqs; -struct aio_ring_info ring_info; +/* This needs improving */ +unsigned long user_id; +struct kioctx *next; +struct mm_struct *mm; +struct work_struct wq; + +/* sys_io_setup currently limits this to an unsigned int */ +unsigned max_reqs; };/* prototypes */
Quoted 2 lines
Thoughts? Comments? Flames?
Well, to start there should be a comment in the code warning future readers about the specific rational behind the resulting ordering.
I'll try to think hardware about it next week.
- z
Quoted 2 lines
I'll try to think hardware about it next week.
("harder", "hardware", who's to say?!)
Quoted 9 lines · Kenneth W
The following patch is result of our kernel data latency analysis on a large database setup (128 logical CPU, 1 TB memory, a couple thousand disks). The analysis is done at an individual cache line level through hardware performance monitor unit. What we did essentially is to collect data access latency on every kernel memory address and look for top 50 longest average access latency. The hardware performance unit also keeps track of instruction address for each collected data access. This allows us to further associate each of the top 50 cache lines to instruction sequences that caused long latency.
Do you have performance numbers of before/after this change ? How much is it worth?
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Dave Jones wrote on Thursday, October 12, 2006 5:04 PM
Quoted 11 lines · Kenneth W
Quoted 9 lines
The following patch is result of our kernel data latency analysis on a large database setup (128 logical CPU, 1 TB memory, a couple thousand disks). The analysis is done at an individual cache line level through hardware performance monitor unit. What we did essentially is to collect data access latency on every kernel memory address and look for top 50 longest average access latency. The hardware performance unit also keeps track of instruction address for each collected data access. This allows us to further associate each of the top 50 cache lines to instruction sequences that caused long latency.
Quoted 1 line
Do you have performance numbers of before/after this change ?
Not yet, I just queued up experiment on our internal benchmark setup. I posted the patch here to get some early discussion on how to attack the problem.