Skip to contentSkip to conversation

Accelerate dbench

RFC In review Daniel Phillips 3 participants 6 messages Aug 5, 2001 – Aug 6, 2001

Subsystems (2)
lore
Share & tools
lore
#1 lore raw
In reply to Daniel Phillips

mm/filemap.c line 312

Quoted 8 lines · Daniel Phillips
mm/filemap.c renamed from 2.4.7.clean/mm/filemap.c +2 -2
 
-	/* Mark the page referenced, kswapd will find it later. */
 	SetPageReferenced(page);
-
+	if (!PageActive(page))
+		activate_page(page);

I think this is wrong.

By doing this the page will end up at the far end of the active list with the referenced bit already set.

This doesn't allow us to distinguish between pages which get accessed again after we put them on the active list and pages which aren't.

Also, it effectively gives a double boost for this one access...

regards,

Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

#0 lore raw

RFC 1/1 Accelerate dbench

This simple patch to 2.4.8-pre4 accelerates dbench quite spectacularly for me. It also slows down my more realistic make+grep load, so it's not generally a good thing, but... if we could detect the conditions where this activation strategy is desirable, it could turn into a good thing.

This doesn't address any burning issue at the moment, it's just for comment. Note that it effectively turns check_used_once into a no-op.

The effect of the patch is to always activate pages when they are successfully looked up, not when first created. This allows unused readahead pages to be quickly reclaimed. (Buffer pages could easily be treated the same way but I didn't try it.) Together with -pre4's shorter IO queue (also probably not a good thing, as Linus pointed out) this seems to be very friendly to dbench, for reasons I still don't understand.

With this patch, I saw very large flucuations in the timings obtained from dbench 12 on this 64 MB Vaio, but the times ranged from good to excellent, varying by about 30%. This is with identical running conditions, starting from a clean reboot each time, and mkfs'ing the test partition. Go figure. As always, the best dbench times were associated with very unfair scheduling, with frequent bursts where a single process makes rapid progress and the others appear to be pushed aside.

mm/filemap.c renamed from 2.4.7.clean/mm/filemap.c +2 -2
@@ -307,9 +307,9 @@
 		if (page->index == offset)
 			break;
 	}
-	/* Mark the page referenced, kswapd will find it later. */
 	SetPageReferenced(page);
-
+	if (!PageActive(page))
+		activate_page(page);
 not_found:
 	return page;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

#2 lore raw
In reply to Rik van Riel
Quoted 16 lines · Rik van Riel

mm/filemap.c line 312

Quoted 8 lines · Daniel Phillips
mm/filemap.c renamed from 2.4.7.clean/mm/filemap.c +2 -2
 
-	/* Mark the page referenced, kswapd will find it later. */
 	SetPageReferenced(page);
-
+	if (!PageActive(page))
+		activate_page(page);

I think this is wrong.

By doing this the page will end up at the far end of the active list with the referenced bit already set.

Quoted 6 lines

This doesn't allow us to distinguish between pages which get accessed again after we put them on the active list and pages which aren't.

Also, it effectively gives a double boost for this one access...

How wrong could it be when it's turning in results like this:

dbench 12, 2.4.8-pre4 vanilla
12.76user 76.49system 6:20.56elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (426major+405minor)pagefaults 0swaps

dbench 12, 2.4.8-pre4 with immediate activate in find_page 16.81user 58.05system 3:13.92elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (1112major+632minor)pagefaults 0swaps

Almost twice as fast with the patch. Mind you, it doesn't turn in this spectacular result all the time, sometimes it's more like:

dbench 12, 2.4.8-pre4 with immediate activate in find_page 15.81user 69.09system 4:40.60elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (991major+766minor)pagefaults 0swaps

which is still very good compared to the stock kernel and its predecessors. (Note the huge difference between timings take under identical conditions, from clean reboots.)

But you're right, this fits our aging model better:

- SetPageReferenced(page);
+ if (PageActive(page))
+ SetPageReferenced(page);
+ else
+ activate_page(page);

So I'll try it...<time passes>...OK, it doesn't make a lot of difference, results still range from "pretty good" to "really great". Not really suprising, I have this growing gut feeling that we're not doing that well on the active page aging anyway, and that random selection of candidates for trial on the inactive queue would perform almost as well - which might be worth testing. Anyway, I'm putting this on the back burner for now. Interesting as it is, it's hardly a burning issue.

--
Daniel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

#3 lore raw
Quoted 10 lines · Daniel Phillips
(quoted diff) +4 -1
-  	SetPageReferenced(page);
+	if (PageActive(page))
+	  	SetPageReferenced(page);
+	else
+		activate_page(page);

So I'll try it...<time passes>...OK, it doesn't make a lot of difference, results still range from "pretty good" to "really great". Not really suprising, I have this growing gut feeling that we're not doing that well on the active page aging anyway,

Yes, I have the feeling that exponential down aging wasn't such a good idea in combination with the fact that most of the access bits are "hidden" in page tables ...

Quoted 4 lines

and that random selection of candidates for trial on the inactive queue would perform almost as well - which might be worth testing. Anyway, I'm putting this on the back burner for now. Interesting as it is, it's hardly a burning issue.

Well, we found that doing instant activation gives a huge performance increase. That's one important point already ;)

regards,

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/ http://distro.conectiva.com/

Send all your spam to [email protected] (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

#5 lore raw
In reply to Daniel Phillips
Quoted 6 lines · Daniel Phillips

How wrong could it be when it's turning in results like this:

dbench 12, 2.4.8-pre4 vanilla
12.76user 76.49system 6:20.56elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (426major+405minor)pagefaults 0swaps

Oi! Didn't Andrew say not to optimise for dbench?

We've had some interesting times with ext3 and dbench lately. It all boils down to the fact that dbench deletes its own working files inside the kupdate writeback interval.

This means that:

a) If dbench is running fast enough to delete its file within the writeback interval, it'll run even faster because it does less IO! Non-linear behaviour there.

b) If a butterfly flaps its wing, and something triggers bdflush then your dbench throughput is demolished, because data which ordinarily is deleted before ever getting written out ends up hitting disk.

It was discovered that with one particular workload ext2
was not triggering bdflush but ext3 was. Twiddling the
bdflush nfract and nfract_sync parameters prevented this and our throughput went from something unmentionable up to 85% of ext2. (actually, it was a teeny bit faster with 80 clients - dunno why).

All this was with ext3 in data writeback mode, of course - the other journalling modes write data out within 5 seconds anyway, which is another reason why ext3 dbench numbers are unrepresentatively lower than ext2 - we do about four times as much I/O!

So... This artifact makes *gross* throughput differences, and if your VM changes are somehow causing changed flush behaviour then perhaps you won't see what you're looking for.

And note the positive feedback cycle: a slower dbench run will result in more IO, which will result in a slower dbench run, which will....

For VM/fs tuning efforts I'd recommend that you consider hacking dbench to not delete its files - just rename them or something.

(Not blaming dbench here - it is merely emulating netbench dopiness).

-
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/

#4 lore raw
In reply to Rik van Riel
Quoted 7 lines · Rik van Riel
Quoted 6 lines · Daniel Phillips
(quoted diff) +4 -1
-  	SetPageReferenced(page);
+	if (PageActive(page))
+	  	SetPageReferenced(page);
+	else
+		activate_page(page);
Quoted 8 lines
Quoted 4 lines

So I'll try it...<time passes>...OK, it doesn't make a lot of difference, results still range from "pretty good" to "really great". Not really suprising, I have this growing gut feeling that we're not doing that well on the active page aging anyway,

Yes, I have the feeling that exponential down aging wasn't such a good idea in combination with the fact that most of the access bits are "hidden" in page tables ...

Ignoring the hardware access bits in page_launder has to hurt. I'm playing with an idea now to get access to the hardware referenced bit for swap pages on the inactive queue, needs more research.

Quoted 7 lines
Quoted 4 lines

and that random selection of candidates for trial on the inactive queue would perform almost as well - which might be worth testing. Anyway, I'm putting this on the back burner for now. Interesting as it is, it's hardly a burning issue.

Well, we found that doing instant activation gives a huge performance increase. That's one important point already ;)

*Note*: only for dbench, a bizarre load I don't fully understand. For my more realistic make+grep load, this change slows it down as I would expect. But yes, it indicates that there may be more big gains to get from replacement policy once we understand what the dbench load really is and how to detect that kind of load automatically. Or maybe if it just results in a useful adjustment available from userlond it would be worth the effort.

There is one disturbing possibility I haven't looked into: what if dbench has a bug? What if its not really doing all the IO its supposed to on those really fast runs?

--
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/