Skip to contentSkip to conversation

fio sync read 4k block size 35% regression

DISCUSSION Zhang, Yanmin 2 participants 7 messages Jul 1, 2009 – Jul 3, 2009 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

#1 lore raw
Quoted 25 lines · Yanmin

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();
Quoted 5 lines

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

Yanmin, thanks for the tests!

Maybe the patch posted here can restore the performance:

http://lkml.org/lkml/2009/5/21/319
Subject Re: [PATCH] readahead:add blk_run_backing_dev

Thanks,
Fengguang

A message that is not in the archive (sent off-list or not yet received): 20090701041010.GB22364@localhost

#3 lore raw
Quoted 29 lines · Wu Fengguang
Quoted 28 lines · Yanmin

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Quoted 8 lines
Quoted 2 lines

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

Yanmin, thanks for the tests!

Maybe the patch posted here can restore the performance:

http://lkml.org/lkml/2009/5/21/319

I tried it and it doesn't help.

Yanmin

#4 lore raw
Quoted 33 lines · Yanmin
Quoted 32 lines · Wu Fengguang
Quoted 30 lines · Yanmin

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

Quoted 6 lines
Quoted 5 lines

Yanmin, thanks for the tests!

Maybe the patch posted here can restore the performance:

http://lkml.org/lkml/2009/5/21/319

I tried it and it doesn't help.

Then let's check what's happening behind the scene :)

Please apply the attached patch and run

echo 1 > /debug/readahead/trace_enable
# do benchmark
echo 0 > /debug/readahead/trace_enable

and send the dmesg which will contain lots of lines like

        [   54.738105] readahead-initial0(pid=3290(zsh), dev=00:10(0:10), ino=105910(dmesg), req=0+1, ra=0+4-3, async=0) = 2
        [   54.751801] readahead-subsequent(pid=3290(dmesg), dev=00:10(0:10), ino=105910(dmesg), req=1+60, ra=4+8-8, async=1, miss=0) = 0

Thank you very much,
Fengguang

mm/readahead.c +32 -7
@@ -16,6 +16,7 @@
 #include <linux/task_io_accounting_ops.h>
 #include <linux/pagevec.h>
 #include <linux/pagemap.h>
+#include <linux/marker.h>
 
 /*
  * Initialise a struct file's readahead state.  Assumes that the caller has
@@ -225,6 +226,10 @@ int force_page_cache_readahead(struct ad
 		offset += this_chunk;
 		nr_to_read -= this_chunk;
 	}
+
+	trace_mark(readahead_fadvise, "%p %p %lu %lu %d",
+			mapping, filp, offset, nr_to_read, ret);
+
 	return ret;
 }
 
@@ -242,13 +247,20 @@ unsigned long max_sane_readahead(unsigne
  * Submit IO for the read-ahead request in file_ra_state.
  */
 unsigned long ra_submit(struct file_ra_state *ra,
-		       struct address_space *mapping, struct file *filp)
+			struct address_space *mapping,
+			struct file *filp,
+			pgoff_t offset,
+			unsigned long req_size,
+			int async)
 {
 	int actual;
 
 	actual = __do_page_cache_readahead(mapping, filp,
 					ra->start, ra->size, ra->async_size);
 
+	trace_mark(readahead_generic, "%p %p %lu %lu %p %d %d",
+			mapping, filp, offset, req_size, ra, actual, async);
+
 	return actual;
 }
 
@@ -375,6 +387,7 @@ static int try_context_readahead(struct 
 	if (size >= offset)
 		size *= 2;
 
+	ra_set_pattern(ra, RA_PATTERN_CONTEXT);
 	ra->start = offset;
 	ra->size = get_init_ra_size(size + req_size, max);
 	ra->async_size = ra->size;
@@ -396,8 +409,10 @@ ondemand_readahead(struct address_space 
 	/*
 	 * start of file
 	 */
-	if (!offset)
+	if (!offset) {
+		ra_set_pattern(ra, RA_PATTERN_INITIAL0);
 		goto initial_readahead;
+	}
 
 	/*
 	 * It's the expected callback offset, assume sequential access.
@@ -405,6 +420,7 @@ ondemand_readahead(struct address_space 
 	 */
 	if ((offset == (ra->start + ra->size - ra->async_size) ||
 	     offset == (ra->start + ra->size))) {
+		ra_set_pattern(ra, RA_PATTERN_SUBSEQUENT);
 		ra->start += ra->size;
 		ra->size = get_next_ra_size(ra, max);
 		ra->async_size = ra->size;
@@ -427,6 +443,7 @@ ondemand_readahead(struct address_space 
 		if (!start || start - offset > max)
 			return 0;
 
+		ra_set_pattern(ra, RA_PATTERN_HIT_MARKER);
 		ra->start = start;
 		ra->size = start - offset;	/* old async_size */
 		ra->size += req_size;
@@ -438,14 +455,18 @@ ondemand_readahead(struct address_space 
 	/*
 	 * oversize read
 	 */
-	if (req_size > max)
+	if (req_size > max) {
+		ra_set_pattern(ra, RA_PATTERN_OVERSIZE);
 		goto initial_readahead;
+	}
 
 	/*
 	 * sequential cache miss
 	 */
-	if (offset - (ra->prev_pos >> PAGE_CACHE_SHIFT) <= 1UL)
+	if (offset - (ra->prev_pos >> PAGE_CACHE_SHIFT) <= 1UL) {
+		ra_set_pattern(ra, RA_PATTERN_INITIAL);
 		goto initial_readahead;
+	}
 
 	/*
 	 * Query the page cache and look for the traces(cached history pages)
@@ -456,9 +477,12 @@ ondemand_readahead(struct address_space 
 
 	/*
 	 * standalone, small random read
-	 * Read as is, and do not pollute the readahead state.
 	 */
-	return __do_page_cache_readahead(mapping, filp, offset, req_size, 0);
+	ra_set_pattern(ra, RA_PATTERN_RANDOM);
+	ra->start = offset;
+	ra->size = req_size;
+	ra->async_size = 0;
+	goto readit;
 
 initial_readahead:
 	ra->start = offset;
@@ -476,7 +500,8 @@ readit:
 		ra->size += ra->async_size;
 	}
 
-	return ra_submit(ra, mapping, filp);
+	return ra_submit(ra, mapping, filp, offset, req_size,
+			 hit_readahead_marker);
 }
 
 /**
include/linux/fs.h +46 -0

57 lines not on this page · raw message

include/linux/mm.h +4 -1
@@ -1195,7 +1195,10 @@ void page_cache_async_readahead(struct a
 unsigned long max_sane_readahead(unsigned long nr);
 unsigned long ra_submit(struct file_ra_state *ra,
 			struct address_space *mapping,
-			struct file *filp);
+			struct file *filp,
+			pgoff_t offset,
+			unsigned long req_size,
+			int async);
 
 /* Do stack extension */
 extern int expand_stack(struct vm_area_struct *vma, unsigned long address);
mm/filemap.c +8 -3
@@ -1020,7 +1020,7 @@ static void shrink_readahead_size_eio(st
  * This is really ugly. But the goto's actually try to clarify some
  * of the logic when it comes to error handling etc.
  */
-static void do_generic_file_read(struct file *filp, loff_t *ppos,
+void do_generic_file_read(struct file *filp, loff_t *ppos,
 		read_descriptor_t *desc, read_actor_t actor)
 {
 	struct address_space *mapping = filp->f_mapping;
@@ -1471,6 +1471,7 @@ static void do_sync_mmap_readahead(struc
 
 	if (VM_SequentialReadHint(vma) ||
 			offset - 1 == (ra->prev_pos >> PAGE_CACHE_SHIFT)) {
+		ra->flags |= RA_FLAG_MMAP;
 		page_cache_sync_readahead(mapping, ra, file, offset,
 					  ra->ra_pages);
 		return;
@@ -1491,10 +1492,12 @@ static void do_sync_mmap_readahead(struc
 	 */
 	ra_pages = max_sane_readahead(ra->ra_pages);
 	if (ra_pages) {
+		ra->flags |= RA_FLAG_MMAP;
+		ra_set_pattern(ra, RA_PATTERN_MMAP_AROUND);
 		ra->start = max_t(long, 0, offset - ra_pages/2);
 		ra->size = ra_pages;
 		ra->async_size = 0;
-		ra_submit(ra, mapping, file);
+		ra_submit(ra, mapping, file, offset, 1, 0);
 	}
 }
 
@@ -1515,9 +1518,11 @@ static void do_async_mmap_readahead(stru
 		return;
 	if (ra->mmap_miss > 0)
 		ra->mmap_miss--;
-	if (PageReadahead(page))
+	if (PageReadahead(page)) {
+		ra->flags |= RA_FLAG_MMAP;
 		page_cache_async_readahead(mapping, ra, file,
 					   page, offset, ra->ra_pages);
+	}
 }
 
 /**
mm/readahead_trace.c new file +514 -0

514 lines not on this page · raw message

mm/Kconfig +19 -0
@@ -260,3 +260,22 @@ config NOMMU_INITIAL_TRIM_EXCESS
 	  of 1 says that all excess pages should be trimmed.
 
 	  See Documentation/nommu-mmap.txt for more information.
+
+config READAHEAD_TRACE
+	tristate "Readahead tracing"
+	select MARKERS
+	select KPROBES
+	select DEBUGFS
+	default y
+	help
+	 This module injects code to show detailed readahead traces and do
+	 readahead events accounting.
+
+	 To actually get the data:
+
+	 # mount -t debugfs none /sys/kernel/debug
+
+	 After that you can do the following:
+
+	 # cat /sys/kernel/debug/readahead/stats     # check counters
+	 # echo > /sys/kernel/debug/readahead/stats  # reset counters
mm/Makefile +1 -2
@@ -40,5 +40,4 @@ obj-$(CONFIG_SMP) += allocpercpu.o
 endif
 obj-$(CONFIG_QUICKLIST) += quicklist.o
 obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o page_cgroup.o
-obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
-obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
+obj-$(CONFIG_READAHEAD_TRACE) += readahead_trace.o
#5 lore raw
Quoted 45 lines · Wu Fengguang
Quoted 33 lines · Yanmin
Quoted 32 lines · Wu Fengguang
Quoted 30 lines · Yanmin

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

Quoted 6 lines
Quoted 5 lines

Yanmin, thanks for the tests!

Maybe the patch posted here can restore the performance:

http://lkml.org/lkml/2009/5/21/319

I tried it and it doesn't help.

Then let's check what's happening behind the scene :)

Please apply the attached patch and run

Quoted 8 lines

echo 1 > /debug/readahead/trace_enable
# do benchmark
echo 0 > /debug/readahead/trace_enable

and send the dmesg which will contain lots of lines like

        [   54.738105] readahead-initial0(pid=3290(zsh), dev=00:10(0:10), ino=105910(dmesg), req=0+1, ra=0+4-3, async=0) = 2
        [   54.751801] readahead-subsequent(pid=3290(dmesg), dev=00:10(0:10), ino=105910(dmesg), req=1+60, ra=4+8-8, async=1, miss=0) = 0

I enlarged sys log buffer to 2MB and captured below data.

In addition, I added new test cases to use mmap to read the files sequentionally. On this machine, there is about 40% regression. reverting your patch fixes it.

On another machine with another JBOD (7 SAS disks), fio_mmap_sync_read_4k (64k/128k) has about 30% regression. But it's not caused by your patch. I am bisecting it on the 2nd machine now.

Yanmin

#6 lore raw
Quoted 50 lines · Yanmin
Quoted 49 lines · Wu Fengguang
Quoted 33 lines · Yanmin
Quoted 32 lines · Wu Fengguang
Quoted 30 lines · Yanmin

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

Quoted 6 lines
Quoted 5 lines

Yanmin, thanks for the tests!

Maybe the patch posted here can restore the performance:

http://lkml.org/lkml/2009/5/21/319

I tried it and it doesn't help.

Then let's check what's happening behind the scene :)

Please apply the attached patch and run

echo 1 > /debug/readahead/trace_enable
# do benchmark
echo 0 > /debug/readahead/trace_enable

Quoted 5 lines
Quoted 4 lines

and send the dmesg which will contain lots of lines like

        [   54.738105] readahead-initial0(pid=3290(zsh), dev=00:10(0:10), ino=105910(dmesg), req=0+1, ra=0+4-3, async=0) = 2
        [   54.751801] readahead-subsequent(pid=3290(dmesg), dev=00:10(0:10), ino=105910(dmesg), req=1+60, ra=4+8-8, async=1, miss=0) = 0

I enlarged sys log buffer to 2MB and captured below data.

Thank you! Judging from these readahead requests, it's doing perfect 64-page IOs. So the readahead size for sdm1 is 256KB? As long as the real IO size is 256KB, it should not create that much performance regression.

However there are some CPU overheads caused by readahead cache hits (the lines start with ==>). In this case, 4 processes are sequentially reading the same file, whoever goes first and foremost will trigger the readahead IO. Or sometimes two processes hit the PG_readahead page almost at the same time, and submit duplicate readahead requests for the same page(s). The duplicate efforts will be detected at radix tree insertion time, so won't lead to duplicate IO.

That is a known issue, and should only cost a little CPU time.

Since I cannot find out the (real) problem by reasoning, can you provide more data please?
- make sure the regression(s) disappear with 2.6.31-rc1+revert-51daa88ebd8e0
- collect some readahead traces with 2.6.31-rc1+revert-51daa88ebd8e0
- collect some blktrace data on sdm1 with 2.6.31-rc1

        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143693+1, ra=143757+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143757+1, ra=143821+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143821+1, ra=143885+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143885+1, ra=143949+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143949+1, ra=144013+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144013+1, ra=144077+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144077+1, ra=144141+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144141+1, ra=144205+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144205+1, ra=144269+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144269+1, ra=144333+64-64, async=1) = 64
        readahead-marker(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=144333+1, ra=144397+64-64, async=1) = 64
        readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=144397+1, ra=144461+64-64, async=1) = 64
        readahead-marker(pid=7102(fio), dev=08:c1(sdm1), ino=49156(f4), req=144461+1, ra=144525+64-64, async=1) = 64
        readahead-marker(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144525+1, ra=144589+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144589+1, ra=144653+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144653+1, ra=144717+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144717+1, ra=144781+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144781+1, ra=144845+64-64, async=1) = 64
==>     readahead-marker(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=144845+1, ra=144909+64-64, async=1) = 64
==>     readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144909+1, ra=144909+128-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144973+1, ra=145037+64-64, async=1) = 64
        readahead-marker(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145037+1, ra=145101+64-64, async=1) = 64
==>     readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145101+1, ra=145165+64-64, async=1) = 64
==>     readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=145101+1, ra=145101+128-64, async=1) = 64
==>     readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145165+1, ra=145229+64-64, async=1) = 64
==>     readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=145165+1, ra=145229+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=145229+1, ra=145293+64-64, async=1) = 64
        readahead-marker(pid=7154(fio), dev=08:c1(sdm1), ino=49156(f4), req=145357+1, ra=145421+64-64, async=1) = 64
        readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145293+1, ra=145293+128-64, async=1) = 64
Quoted 2 lines

In addition, I added new test cases to use mmap to read the files sequentionally. On this machine, there is about 40% regression. reverting your patch fixes it.

Does that also involves multiple processes concurrently reading the same file?

Quoted 3 lines

On another machine with another JBOD (7 SAS disks), fio_mmap_sync_read_4k (64k/128k) has about 30% regression. But it's not caused by your patch. I am bisecting it on the 2nd machine now.

That may well related to my readahead patches.

Thanks,
Fengguang

#7 lore raw
Quoted 51 lines · Wu Fengguang
Quoted 50 lines · Yanmin
Quoted 49 lines · Wu Fengguang
Quoted 33 lines · Yanmin
Quoted 32 lines · Wu Fengguang
Quoted 30 lines · Yanmin

Comapraing with 2.6.30, fio sync read (block size 4k) has about 35% regression with kernel 2.6.31-rc1 on my stoakley machine with a JBOD (13 SCSI disks).

Every disk has 1 partition and 4 1-GB files. Start 10 processes per disk to do sync read sequentinally.

Bisected down to below patch.

51daa88ebd8e0d437289f589af29d4b39379ea76 is first bad commit commit 51daa88ebd8e0d437289f589af29d4b39379ea76

Author: Wu Fengguang <[email protected]>
Date:   Tue Jun 16 15:31:24 2009 -0700
    readahead: remove sync/async readahead call dependency

    The readahead call scheme is error-prone in that it expects the call sites
    to check for async readahead after doing a sync one.  I.e.

                        if (!page)
                                page_cache_sync_readahead();
                        page = find_get_page();
                        if (page && PageReadahead(page))
                                page_cache_async_readahead();

I also test block size 64k and 128k, but they don't have regression. Perhaps the default read_ahead_kb is equal to 128?

Other 2 machines have no such regression. The JBODS of the 2 machines consists of 12 and 7 SATA/SAS disks while every disk has 2 partitions.

Quoted 6 lines
Quoted 5 lines

Yanmin, thanks for the tests!

Maybe the patch posted here can restore the performance:

http://lkml.org/lkml/2009/5/21/319

I tried it and it doesn't help.

Then let's check what's happening behind the scene :)

Please apply the attached patch and run

echo 1 > /debug/readahead/trace_enable
# do benchmark
echo 0 > /debug/readahead/trace_enable

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

and send the dmesg which will contain lots of lines like

        [   54.738105] readahead-initial0(pid=3290(zsh), dev=00:10(0:10), ino=105910(dmesg), req=0+1, ra=0+4-3, async=0) = 2
        [   54.751801] readahead-subsequent(pid=3290(dmesg), dev=00:10(0:10), ino=105910(dmesg), req=1+60, ra=4+8-8, async=1, miss=0) = 0

I enlarged sys log buffer to 2MB and captured below data.

Thank you! Judging from these readahead requests, it's doing perfect 64-page IOs. So the readahead size for sdm1 is 256KB?

[ymzhang@lkp-st02-x8664 work]$ cat /sys/block/sdm/queue/read_ahead_kb 128

We use the default readahead configuration.

Quoted 12 lines

As long as the
real IO size is 256KB, it should not create that much performance regression.

However there are some CPU overheads caused by readahead cache hits (the lines start with ==>). In this case, 4 processes are sequentially reading the same file, whoever goes first and foremost will trigger the readahead IO. Or sometimes two processes hit the PG_readahead page almost at the same time, and submit duplicate readahead requests for the same page(s). The duplicate efforts will be detected at radix tree insertion time, so won't lead to duplicate IO.

Quoted 5 lines

That is a known issue, and should only cost a little CPU time.

Since I cannot find out the (real) problem by reasoning, can you provide more data please?
- make sure the regression(s) disappear with 2.6.31-rc1+revert-51daa88ebd8e0

I make sure.

Quoted 2 lines

- collect some readahead traces with 2.6.31-rc1+revert-51daa88ebd8e0
- collect some blktrace data on sdm1 with 2.6.31-rc1

I'm busy and will collect data next week.

Quoted 34 lines
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143693+1, ra=143757+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143757+1, ra=143821+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143821+1, ra=143885+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143885+1, ra=143949+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=143949+1, ra=144013+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144013+1, ra=144077+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144077+1, ra=144141+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144141+1, ra=144205+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144205+1, ra=144269+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144269+1, ra=144333+64-64, async=1) = 64
        readahead-marker(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=144333+1, ra=144397+64-64, async=1) = 64
        readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=144397+1, ra=144461+64-64, async=1) = 64
        readahead-marker(pid=7102(fio), dev=08:c1(sdm1), ino=49156(f4), req=144461+1, ra=144525+64-64, async=1) = 64
        readahead-marker(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144525+1, ra=144589+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144589+1, ra=144653+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144653+1, ra=144717+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144717+1, ra=144781+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144781+1, ra=144845+64-64, async=1) = 64
==>     readahead-marker(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=144845+1, ra=144909+64-64, async=1) = 64
==>     readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144909+1, ra=144909+128-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=144973+1, ra=145037+64-64, async=1) = 64
        readahead-marker(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145037+1, ra=145101+64-64, async=1) = 64
==>     readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145101+1, ra=145165+64-64, async=1) = 64
==>     readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=145101+1, ra=145101+128-64, async=1) = 64
==>     readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145165+1, ra=145229+64-64, async=1) = 64
==>     readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=145165+1, ra=145229+64-64, async=1) = 64
        readahead-subsequent(pid=7115(fio), dev=08:c1(sdm1), ino=49156(f4), req=145229+1, ra=145293+64-64, async=1) = 64
        readahead-marker(pid=7154(fio), dev=08:c1(sdm1), ino=49156(f4), req=145357+1, ra=145421+64-64, async=1) = 64
        readahead-subsequent(pid=7167(fio), dev=08:c1(sdm1), ino=49156(f4), req=145293+1, ra=145293+128-64, async=1) = 64
Quoted 2 lines

In addition, I added new test cases to use mmap to read the files sequentionally. On this machine, there is about 40% regression. reverting your patch fixes it.

Quoted 7 lines

Does that also involves multiple processes concurrently reading the same file?

Quoted 3 lines

On another machine with another JBOD (7 SAS disks), fio_mmap_sync_read_4k (64k/128k) has about 30% regression. But it's not caused by your patch. I am bisecting it on the 2nd machine now.

That may well related to my readahead patches.

Not sure. But the bisect is stable to report Linus' patch.