sys_time() speedup
PATCH 1/1 sys_time() speedup
Subject: [patch] sys_time() speedup
From: Ingo Molnar <[email protected]>
improve performance of sys_time(). sys_time() returns time in seconds, but it does so by calling do_gettimeofday() and then returning the tv_sec portion of the GTOD time. But the data structure "xtime", which is updated by every timer/scheduler tick, already offers HZ granularity time.
the patch improves the sysbench OLTP macrobenchmark significantly:
2.6.22-rc6:
#threads
1: transactions: 3733 (373.21 per sec.) 2: transactions: 6676 (667.46 per sec.) 3: transactions: 6957 (695.50 per sec.) 4: transactions: 7055 (705.48 per sec.) 5: transactions: 6596 (659.33 per sec.)
2.6.22-rc6 + sys_time.patch:
1: transactions: 4005 (400.47 per sec.) 2: transactions: 7379 (737.77 per sec.) 3: transactions: 7347 (734.49 per sec.) 4: transactions: 7468 (746.65 per sec.) 5: transactions: 7428 (742.47 per sec.)
mixed API uses of gettimeofday() and time() are guaranteed to be coherent via the use of a at-most-once-per-second slowpath that updates xtime.
--- kernel/time.c | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-)
kernel/time.c +22 -5
@@ -57,14 +57,17 @@ EXPORT_SYMBOL(sys_tz);*/ asmlinkage long sys_time (time_t __user * tloc) { -time_t i; -struct timeval tv; +/* +* We read xtime.tv_sec atomically - it's updated +* atomically by update_wall_time(), so no need to +* even read-lock the xtime seqlock: +*/ +time_t i = xtime.tv_sec; -do_gettimeofday (&tv); - i = tv.tv_sec; +smp_rmb ();/* sys_time() results are coherent */ if (tloc) { -if (put_user (i,tloc)) +if (put_user (i, tloc)) i = -EFAULT ; }return i;
@@ -373,6 +376,20 @@ void do_gettimeofday (struct timeval *tv tv->tv_sec = sec; tv->tv_usec = usec; + +/* +* Make sure xtime.tv_sec [returned by sys_time()] always +* follows the gettimeofday() result precisely. This +* condition is extremely unlikely, it can hit at most +* once per second: +*/ +if (unlikely (xtime.tv_sec != tv->tv_sec)) { +unsigned long flags; + +write_seqlock_irqsave (&xtime_lock); +update_wall_time (); +write_seqlock_irqrestore (&xtime_lock); + } }EXPORT_SYMBOL (do_gettimeofday);
Hi,
Quoted 1 line · Ingo Molnar
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
Quoted 11 lines
kernel/time.c +7
-0@@ -373,6 +376,20 @@ void do_gettimeofday (struct timeval *tv tv->tv_sec = sec; tv->tv_usec = usec; + +/* +* Make sure xtime.tv_sec [returned by sys_time()] always +* follows the gettimeofday() result precisely. This +* condition is extremely unlikely, it can hit at most +* once per second: +*/
kernel/time.c void do_gettimeofday (struct timeval *tv · line 395
Quoted 10 lines
kernel/time.c +7
-0+if (unlikely (xtime.tv_sec != tv->tv_sec)) { +unsigned long flags; + +write_seqlock_irqsave (&xtime_lock); +update_wall_time (); +write_seqlock_irqrestore (&xtime_lock); + } }EXPORT_SYMBOL (do_gettimeofday);
Is this the do_gettimeofday() inside CONFIG_TIME_INTERPOLATION? What did you test?
There can be many ways to read the clock, do you want to put this hook everywhere? Wouldn't it be better to improve the clock performance?
bye, Roman
Quoted 8 lines · Roman Zippel
Hi,
Quoted 1 line · Ingo Molnar
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
It seems to me that Ingo's patch offers slightly improved performance for any program using the time() system call, with no real drawbacks, so why wouldn't we want to use it?
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Hi,
Quoted 9 lines · Jesper Juhl
Quoted 6 lines · Ingo Molnar
Quoted 2 lines
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
It seems to me that Ingo's patch offers slightly improved performance for any program using the
time()system call, with no real drawbacks, so why wouldn't we want to use it?
How do you come to the conclusion it has no real drawbacks? Ingo provided no information about his test setup and his patch was a little strange, so I can't say that yet.
bye, Roman
Quoted 8 lines · Roman Zippel
Hi,
Quoted 4 lines · Jesper Juhl
Quoted 4 lines · Ingo Molnar
Quoted 2 lines
the patch improves the sysbench OLTP macrobenchmark significantly:
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Has that any real practical relevance?
It seems to me that Ingo's patch offers slightly improved performance for any program using the
time()system call, with no real drawbacks, so why wouldn't we want to use it?How do you come to the conclusion it has no real drawbacks?
His change to do_gettimeofday() will of course slow that path down a tiny bit since he's adding an extra 'if', but since it's wrapped in unlikely() and should hit at most one time pr second I would guess the performance impact of that to be negligible.
The change to sys_time() does away with some local variables and replaces the call to do_gettimeofday() with a memory barrier and a simple read of xtime.tv_se. I find it hard to believe (although I have not tested it) that that wouldn't be faster than the original.
So that's how I came to that conclusion; just reading the patch, going over what it does in my head and thinking about it a bit. Not the most scientific of things I admit.
Even if it is not faster, what would make it slower? Have you spotted something I have not?
Quoted 3 lines
Ingo provided no information about his test setup and his patch was a little strange, so I can't say that yet.
What did you find strange about it? I'm currious.
Sure it needs testing and of course it would be nice with some more details on Ingo's test setup.
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Hi,
Quoted 2 lines · Jesper Juhl
Even if it is not faster, what would make it slower? Have you spotted something I have not?
There are other ways to read the clock and would require similiar synchronization hooks.
Some archs can implement sys_time() in userspace, so there this change would be useless.
I don't know what clock was used in the test, so maybe it can be replaced with a faster clock.
AFAICT OLTP is not really a common application for most users, so there may be other ways to optimize this special case. Just reading the patch isn't enough here, you have to look at the whole picture and some pieces are still missing...
bye, Roman
Quoted 6 lines · Jesper Juhl
Quoted 2 lines
Ingo provided no information about his test setup and his patch was a little strange, so I can't say that yet.
What did you find strange about it? I'm currious. Sure it needs testing and of course it would be nice with some more details on Ingo's test setup.
My patch improves MySQL wall-clock performance by ~10% on a dual-core box [see the numbers i cited in my initial mail, they were ran on a T60 with a 1.83 GHz Core2Duo] and by 7% on an 8-way box:
2.6.22-rc6:
#threads
9: transactions: 8440 (843.29 per sec.) 9: transactions: 8423 (841.18 per sec.) 9: transactions: 8511 (849.98 per sec.) 9: transactions: 8473 (846.23 per sec.)
2.6.22-rc6 + sys_time.patch:
#threads
9: transactions: 9043 (903.36 per sec.) 9: transactions: 9020 (900.78 per sec.) 9: transactions: 8974 (896.61 per sec.) 9: transactions: 9007 (899.97 per sec.)
[ to reproduce it, run sysbench 0.4.8 with --test=oltp --num-threads=9.
The other tests show similar speedup, so this is in no way limited to OLTP. ]
in other words, if you are using MySQL in a serious way then this patch provides you a real-world speedup equivalent to upgrading a 1.66 GHz Core2Duo to a 1.83 GHz Core2Duo. I'd call that anything but "slightly improved performance" ;-)
if you are curious why Roman's reaction to this patch was so negative:
i'm extremely curious myself too! ;-) That man, with his eternal negativism (i dare anyone to point me to a _single_ lkml posting of Roman where he gives any positive feedback to anyone) is a pure walking mystery to me ;)
( whether there is any correlation between a decade long fundamental
suckage and stagnation of the Linux time and NTP subsystem and Roman's decade long negative feedback presence in that area of code is left up to the reader. :)
This current ... interesting piece of Roman about a _single_ trivial unlikely() branch in do_gettimeofday() borders on the ridiculous. My patch might be wrong for various reasons, but that single 'if (unlikely())' statement is not one of those reasons =B-)
Ingo
Hi,
Quoted 1 line · Ingo Molnar
if you are curious why Roman's reaction to this patch was so negative:
Instead of answering all the open questions, pretty much the second thing you do is to discredit me personally. :-(
BTW there is a difference between critical and negative...
bye, Roman
Quoted 3 lines · Ingo Molnar
My patch improves MySQL wall-clock performance by ~10% on a dual-core box [see the numbers i cited in my initial mail, they were ran on a T60 with a 1.83 GHz Core2Duo] and by 7% on an 8-way box:
mysql isn't froznen in stone like some commercial db, you can look into its source, the testsuite isn't either I guess. Did you ask yourself at least once why it's calling time() so many times per second? The timestamp sql type should always trigger gettimeofday calls AFIK.
If there's a good reason to call time so frequently in an important app like mysql, then your patch sure is a good idea, but at first glance it looks fishy that time is so performance critical.
I'm not objecting the patch itself, if there's a legitimate reason to call time so frequently that's sure a fine optimization despite the branch in gettimeofday, but I'm asking why time is called so many times on this specific workload, because I'm fairly certain that in average (desktop and server) gettimeofday is called much more frequently and if it was up to me to tell, I would expect time microoptimizations to result in irrelevant performance differences. Infact I seem to recall that even other commercial dbs calls tends to call floods of gettimeofday while I can't recall any time in the strace output. So unless there is a legitimate reason to call time() dozen thousand times per second, if it was my choice, I would prefer to have gettimeofday as fast as it can be. Not that a branch will make any measurable difference, but still that would be my choice.
But perhaps you already know why time is called so frequently, I certainly don't, nor I would expect it.
Thanks.
Quoted 5 lines · Andrea Arcangeli
Quoted 3 lines · Ingo Molnar
My patch improves MySQL wall-clock performance by ~10% on a dual-core box [see the numbers i cited in my initial mail, they were ran on a T60 with a 1.83 GHz Core2Duo] and by 7% on an 8-way box:
Quoted 8 lines
mysql isn't froznen in stone like some commercial db, you can look into its source, the testsuite isn't either I guess. Did you ask yourself at least once why it's calling
time()so many times per second? The timestamp sql type should always trigger gettimeofday calls AFIK.If there's a good reason to call time so frequently in an important app like mysql, then your patch sure is a good idea, but at first glance it looks fishy that time is so performance critical.
It's not an unreasonable expectation (from userspace's point of view) that the simpler sys_time() system call would run faster than gettimeofday. I suspect that this may be the case on other platforms.
However, your entire argument seems to be "It helps userspace run faster? Weird. It shouldn't. They must be doing something wrong, therefore the patch is stupid."
If that is your argument, then it's bogus. The bottom line is that code, deployed today, significantly and impressively benefits from the patch. And it's not a microbenchmark, that ignores the effect on the rest of the system. Ingo said that both MySQL and X are improved by this, so between the two of them that covers most every desktop system and many servers.
Quoted 6 lines
I'm not objecting the patch itself, if there's a legitimate reason to call time so frequently that's sure a fine optimization despite the branch in gettimeofday, but I'm asking why time is called so many times on this specific workload, because I'm fairly certain that in average (desktop and server) gettimeofday is called much more frequently
You're absolutely correct, gettimeofday *is* called much more frequently. But not enough to outweigh the gains from sys_time improvements.
Quoted 6 lines
and if it was up to me to tell, I would expect time microoptimizations to result in irrelevant performance differences. Infact I seem to recall that even other commercial dbs calls tends to call floods of gettimeofday while I can't recall any time in the strace output. So unless there is a legitimate reason to call
time()dozen thousand times per second,
(Why does this argument always rear its head on topics in timekeeping?)
Userspace does, whether you or I think it is legitimate or not, and that's the reality we need to deal with. Further, I'd argue that it *does* make sense, as sys_time() should be as cheap or cheaper than gettimeofday(), and bottom line, userspace code needs to know the time for lots of reasonable reasons.
Quoted 4 lines
if it was my choice, I
would prefer to have gettimeofday as fast as it can be. Not that a branch will make any measurable difference, but still that would be my choice.
Then you'd be penny-wise and pound-foolish, as the saying goes. Said un-idiomatically, you're optimizing from a microbenchmark point of view, which is not guaranteed to always be a win.
Ray
Quoted 2 lines · Ray Lee
faster? Weird. It shouldn't. They must be doing something wrong, therefore the patch is stupid."
Just in case it's not obvious the above are Ray Lee words, mine not.
-----------
#!/usr/bin/env stap
# edited top.stp from systemtap
global syscalls
function print_top () {
printf ("SYSCALL\t\t\t\tCOUNT\n")
foreach ([name] in syscalls- limit 20)
printf("%-20s\t\t%5d\n",name, syscalls[name])
printf("--------------------------------------\n")
}probe syscall.time {
syscalls[probefunc()]++
}
probe syscall.gettimeofday {
syscalls[probefunc()]++
}# print top syscalls every 5 seconds
probe timer.ms(5000) {
print_top ()
}
-----------The above while running various huge sql operations with real life postgresql app running sql in loop for a minute or so (sorry no mysql setup but the world isn't mysql, I'd rather want to see oracle if something):
SYSCALL COUNTsys_gettimeofday 4998sys_time 120
--------------------------------------
SYSCALL COUNTsys_gettimeofday 9989sys_time 185
--------------------------------------
SYSCALL COUNTsys_gettimeofday 15219sys_time 335
--------------------------------------
SYSCALL COUNTsys_gettimeofday 21215sys_time 428
--------------------------------------
SYSCALL COUNTsys_gettimeofday 26194sys_time 629
--------------------------------------
SYSCALL COUNTsys_gettimeofday 30752sys_time 734
--------------------------------------
SYSCALL COUNTsys_gettimeofday 37379sys_time 976
--------------------------------------
SYSCALL COUNTsys_gettimeofday 42381sys_time 1125
--------------------------------------
SYSCALL COUNTsys_gettimeofday 47722sys_time 1391
--------------------------------------
SYSCALL COUNTsys_gettimeofday 53138sys_time 1520
--------------------------------------
SYSCALL COUNTsys_gettimeofday 57499sys_time 1651
--------------------------------------
SYSCALL COUNTsys_gettimeofday 62314sys_time 1712
--------------------------------------
SYSCALL COUNTsys_gettimeofday 66874sys_time 1827
--------------------------------------
SYSCALL COUNTsys_gettimeofday 71757sys_time 2007
--------------------------------------
SYSCALL COUNTsys_gettimeofday 76335sys_time 2240
SYSCALL COUNTsys_gettimeofday 80469sys_time 2354
--------------------------------------
SYSCALL COUNTsys_gettimeofday 85420sys_time 2519
--------------------------------------
SYSCALL COUNTsys_gettimeofday 90662sys_time 2648
--------------------------------------
SYSCALL COUNTsys_gettimeofday 95513sys_time 2909
--------------------------------------
SYSCALL COUNTsys_gettimeofday 100767sys_time 3111
--------------------------------------
SYSCALL COUNTsys_gettimeofday 106553sys_time 3427
--------------------------------------
SYSCALL COUNTsys_gettimeofday 112300sys_time 3673
--------------------------------------
SYSCALL COUNTsys_gettimeofday 115706sys_time 3793
SYSCALL COUNTsys_gettimeofday 119842sys_time 3893
--------------------------------------
SYSCALL COUNTsys_gettimeofday 123054sys_time 4113
--------------------------------------
SYSCALL COUNTsys_gettimeofday 126286sys_time 4250
--------------------------------------
SYSCALL COUNTsys_gettimeofday 129077sys_time 4396
--------------------------------------
SYSCALL COUNTsys_gettimeofday 132002sys_time 4506
--------------------------------------
SYSCALL COUNTsys_gettimeofday 138518sys_time 4800
--------------------------------------
SYSCALL COUNTsys_gettimeofday 143572sys_time 4901
--------------------------------------
SYSCALL COUNTsys_gettimeofday 148621sys_time 5069
Now if I play some music with projectm in the background:
SYSCALL COUNTsys_gettimeofday 6337sys_time 128
--------------------------------------
SYSCALL COUNTsys_gettimeofday 11462sys_time 249
--------------------------------------
SYSCALL COUNTsys_gettimeofday 21905sys_time 332
--------------------------------------
SYSCALL COUNTsys_gettimeofday 36205sys_time 494
--------------------------------------
SYSCALL COUNTsys_gettimeofday 53792sys_time 569
--------------------------------------
SYSCALL COUNTsys_gettimeofday 69699sys_time 709
--------------------------------------
SYSCALL COUNTsys_gettimeofday 85663sys_time 791
--------------------------------------
SYSCALL COUNTsys_gettimeofday 101427sys_time 908
--------------------------------------
SYSCALL COUNTsys_gettimeofday 117199sys_time 985
--------------------------------------
SYSCALL COUNTsys_gettimeofday 132963sys_time 1100
--------------------------------------
SYSCALL COUNTsys_gettimeofday 148974sys_time 1162
--------------------------------------
SYSCALL COUNTsys_gettimeofday 164677sys_time 1288
--------------------------------------
SYSCALL COUNTsys_gettimeofday 180697sys_time 1356
--------------------------------------
SYSCALL COUNTsys_gettimeofday 196517sys_time 1479
--------------------------------------
SYSCALL COUNTsys_gettimeofday 209618sys_time 1546
--------------------------------------
SYSCALL COUNTsys_gettimeofday 214914sys_time 1726
--------------------------------------
SYSCALL COUNTsys_gettimeofday 221257sys_time 1798
If I listen some music on youtube.com:
SYSCALL COUNTsys_gettimeofday 5151sys32_gettimeofday 3951sys_time 202compat_sys_time 1
--------------------------------------
SYSCALL COUNT sys32_gettimeofday 16427 sys_gettimeofday 10247 sys_time 306 compat_sys_time 1 -------------------------------------- SYSCALL COUNT sys32_gettimeofday 42436 sys_gettimeofday 15633 sys_time 438 compat_sys_time 1 -------------------------------------- SYSCALL COUNT sys32_gettimeofday 64931 sys_gettimeofday 21121 sys_time 509 compat_sys_time 1 -------------------------------------- SYSCALL COUNT sys32_gettimeofday 94366 sys_gettimeofday 26399 sys_time 638 compat_sys_time 1 -------------------------------------- SYSCALL COUNT sys32_gettimeofday 117573 sys_gettimeofday 33100 sys_time 722 compat_sys_time 1 -------------------------------------- SYSCALL COUNT sys32_gettimeofday 137415 sys_gettimeofday 38357 sys_time 863 compat_sys_time 1 -------------------------------------- SYSCALL COUNT sys32_gettimeofday 154831 sys_gettimeofday 43459
sys_time 933compat_sys_time 1
Those aren't gettimeofday heavy users at all, they're the normal "light" kernel users of the most common workloads. The sql wasn't gettimeofday intensive either. The very heavy users will run hundred thousands gettimeofday or more per second so they're not even interesting to measure because the sys_time system time will be close to zero compared to the gettimeofday one.
BTW, to run the above I had to disable vsyscall64 sysctl, my x2 would never waste any time running sys_time or sys_gettimeofday in the first place ;).
This patch may be a good tradeoff anyway, but IMHO it's not possible to judje the idea of adding a branch to sys_gettimeofday to make sys_time faster without knowing why mysql calls time() so frequently. Even if you care only about the present and you don't care about the future, mysql isn't the only db in the marketplace and like I wrote in the other email the others I've seen were running floods of gettimeofday (I repeat, some even went as far as giving an option to call rdtsc directly on the few servers with tsc synchronized in smp).
Hi,
On Tue, 26 Jun 2007, Ingo Molnar wrote:
Another BTW before someone takes this seriously:
Quoted 4 lines
( whether there is any correlation between a decade long fundamental
suckage and stagnation of the Linux time and NTP subsystem and Roman's decade long negative feedback presence in that area of code is left up to the reader. :)
That's complete bullshit.
Quoted 4 lines
This current ... interesting piece of Roman about a _single_ trivial
unlikely()branch indo_gettimeofday()borders on the ridiculous. My patch might be wrong for various reasons, but that single 'if (unlikely())' statement is not one of those reasons =B-)
That's even more nonsense, that wasn't what my mail was about and Andrew understood me correctly, so you could have too.
Ingo, I at least know that I'm difficult to deal with and try to take this into account, which is hard for me, but you don't even seem to know what kind of ass you are towards people who don't suck up to you (I guess you simply get away with it far too often).
bye, Roman
Quoted 7 lines · On Tue, 26 Jun 2007 19:08:27 +0200 (CEST) Roman Zippel <[email protected]>…
Quoted 4 lines
This current ... interesting piece of Roman about a _single_ trivial
unlikely()branch indo_gettimeofday()borders on the ridiculous. My patch might be wrong for various reasons, but that single 'if (unlikely())' statement is not one of those reasons =B-)That's even more nonsense, that wasn't what my mail was about and Andrew understood me correctly, so you could have too.
umm, yeah. Ingo went a bit over the top there, IMO.
It boils down to: is sys_time() called at more or less than 1/2000th the frequency of gettimeofday(), across the expected lifetime of 2.6.23 and later? Ingo has a couple of (surprising) examples where the sys_time() call frequency _is_ high, but whether that will remain true across 2.6.23 and later is an open question.
How does mysql call sys_time() at all, if time(2) uses the vsyscall page??
Will contemporary-to-2.6.23-and-later mysqls do this?
All this isn't super-trivial silliness, either. gettimeofday() is, for many workloads, the kernel's most time-critical codepath bar none, I believe.
Quoted 7 lines · On Mon, 25 Jun 2007 23:09:46 +0200 Roman Zippel <[email protected]> wrote:
Hi,
Quoted 1 line · Ingo Molnar
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
Interesting question. The patch adds a new test-n-branch to gettimeofday() so if gettimeofday() is used much more frequently than time(), we lose.
Quoted 20 lines
Quoted 20 lines
kernel/time.c +14
-0@@ -373,6 +376,20 @@ void do_gettimeofday (struct timeval *tv tv->tv_sec = sec; tv->tv_usec = usec; + +/* +* Make sure xtime.tv_sec [returned by sys_time()] always +* follows the gettimeofday() result precisely. This +* condition is extremely unlikely, it can hit at most +* once per second: +*/ +if (unlikely (xtime.tv_sec != tv->tv_sec)) { +unsigned long flags; + +write_seqlock_irqsave (&xtime_lock); +update_wall_time (); +write_seqlock_irqrestore (&xtime_lock); + } }
Quoted 3 lines
kernel/time.c void do_gettimeofday (struct timeval *tv · line 395
Quoted 1 line
kernel/time.c +0
-0EXPORT_SYMBOL (do_gettimeofday);Is this the
do_gettimeofday()insideCONFIG_TIME_INTERPOLATION?
Yes.
Quoted 3 lines
What did you test?
There can be many ways to read the clock, do you want to put this hook everywhere?
Yeah, it isn't immediately obvious (to this little black duck) why similar fixups weren't needed in timekeeping.c.
Quoted 4 lines · On Mon, 25 Jun 2007 15:15:08 -0700 Andrew Morton <[email protected]>…
On Mon, 25 Jun 2007 23:09:46 +0200
Roman Zippel <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines · Ingo Molnar
Quoted 2 lines
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
Interesting question. The patch adds a new test-n-branch to
gettimeofday()so ifgettimeofday()is used much more frequently thantime(), we lose.
Isn't gettimeofday() called *lots* by the X server and programs - one of the reasons we bother putting it in the VDSO.
Signature · 3 lines
Cheers, Stephen Rothwell [email protected] http://www.canb.auug.org.au/~sfr/
Quoted 5 lines · Andrew Morton
Quoted 3 lines
Quoted 1 line
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
Interesting question. [...]
i'm missing the <sarcastic> tag i guess ;-)
<sarcastic> Oh my, does database macro-performance have any relevance to Linux bread and butter markets in general. Boggle, it is a really difficult question i suspect. </sarcastic>
If we ignore those few million database and web server Linux boxes on the market and concentrate purely on the few m68k boxes that are still in existance, _then_ we might be doubtful about this question ;-)
Quoted 2 lines
[...] The patch adds a new test-n-branch to
gettimeofday()so ifgettimeofday()is used much more frequently thantime(), we lose.
given that the cost to sys_gettimeofday() is less than a cycle (we test a value already in a register, with an unlikely hint), and the benefit to sys_time() is around 6000 cycles (or more), sys_gettimeofday() would have to be used thousands of times more frequently than sys_time() - which it clearly isnt. As a test i just triggered a really X-intense workload and for that gettimeofday-dominated landscape there was still 1 sys_time() call for every 50 gettimeofday calls - so it's a small win even for this X workload.
Ingo
Quoted 11 lines · On Tue, 26 Jun 2007 17:26:29 +0200 Ingo Molnar <[email protected]> wrote:
Quoted 5 lines · Andrew Morton
Quoted 3 lines
Quoted 1 line
the patch improves the sysbench OLTP macrobenchmark significantly:
Has that any real practical relevance?
Interesting question. [...]
i'm missing the <sarcastic> tag i guess ;-)
Quoted 7 lines
<sarcastic> Oh my, does database macro-performance have any relevance to Linux bread and butter markets in general. Boggle, it is a really difficult question i suspect. </sarcastic>
If we ignore those few million database and web server Linux boxes on the market and concentrate purely on the few m68k boxes that are still in existance, _then_ we might be doubtful about this question ;-)
On my machine, time(2) doesn't do any syscall at all - it uses the vsyscall page. I'd be surprised if a database uses sys_time() either.
Quoted 11 lines
Quoted 2 lines
[...] The patch adds a new test-n-branch to
gettimeofday()so ifgettimeofday()is used much more frequently thantime(), we lose.given that the cost to
sys_gettimeofday()is less than a cycle (we test a value already in a register, with an unlikely hint), and the benefit tosys_time()is around 6000 cycles (or more),sys_gettimeofday()would have to be used thousands of times more frequently thansys_time()- which it clearly isnt. As a test i just triggered a really X-intense workload and for that gettimeofday-dominated landscape there was still 1sys_time()call for every 50 gettimeofday calls - so it's a small win even for this X workload.
So something in X is somehow calling sys_time()? How come, and is that an outlier? How generalisable is this observation?
Quoted 2 lines · Andrew Morton
On my machine, time(2) doesn't do any syscall at all - it uses the vsyscall page. I'd be surprised if a database uses
sys_time()either.
Large boxes unfortunately can't always use vsyscalls... that's a real pity. I also had to disable the vsyscalls64 to generate some number.
I think there shall be a perfectly accurate but not monotone mode for gettimeofday so we can enable rdtscp (via sysctl or/and prctl). Aware apps can enable the prctl, aware or brave admins can turn on the sysctl. Vojtech and others should have proper patches to merge for this.
Quoted 2 lines · Andrew Morton
Interesting question. The patch adds a new test-n-branch to
gettimeofday()so ifgettimeofday()is used much more frequently thantime(), we lose.
I think gettimeofday is generally used much more frequently than time. Real db calls gettimeofday not time, infact some real db related app even go as far as calling rdtsc directly (on hardware where the tsc is synchronized). What's the point of calling time so many times per second when it'll always return the same value anyway?
I think this is a case of the simulator not simulating the real workload and hence that should be fixed instead of optimizing for the erratic simulator.
Just place a systemtap for time and gettimeofday, run a real db or a videogame and then show the number of time vs gettimeofday calls.
Quoted 6 lines · On Mon, 25 Jun 2007 15:15:08 -0700 Andrew Morton <[email protected]>…
Quoted 3 lines
What did you test?
There can be many ways to read the clock, do you want to put this hook everywhere?Yeah, it isn't immediately obvious (to this little black duck) why similar fixups weren't needed in
timekeeping.c.
did this get addressed?
Quoted 59 lines · Ingo Molnar
Subject: [patch]
sys_time()speedup
From: Ingo Molnar <[email protected]>improve performance of
sys_time().sys_time()returns time in seconds, but it does so by callingdo_gettimeofday()and then returning thetv_secportion of the GTOD time. But the data structure "xtime", which is updated by every timer/scheduler tick, already offers HZ granularity time.the patch improves the sysbench OLTP macrobenchmark significantly:
2.6.22-rc6:
#threads
1: transactions: 3733 (373.21 per sec.) 2: transactions: 6676 (667.46 per sec.) 3: transactions: 6957 (695.50 per sec.) 4: transactions: 7055 (705.48 per sec.) 5: transactions: 6596 (659.33 per sec.)2.6.22-rc6 + sys_time.patch:
1: transactions: 4005 (400.47 per sec.) 2: transactions: 7379 (737.77 per sec.) 3: transactions: 7347 (734.49 per sec.) 4: transactions: 7468 (746.65 per sec.) 5: transactions: 7428 (742.47 per sec.)mixed API uses of
gettimeofday()andtime()are guaranteed to be coherent via the use of a at-most-once-per-second slowpath that updates xtime.Signed-off-by Ingo Molnar <[email protected]>--- kernel/time.c | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-)Index:
linux/kernel/time.c
===================================================================kernel/time.c +8
-5@@ -57,14 +57,17 @@ EXPORT_SYMBOL(sys_tz);*/ asmlinkage long sys_time (time_t __user * tloc) { -time_t i; -struct timeval tv; +/* +* We read xtime.tv_sec atomically - it's updated +* atomically by update_wall_time(), so no need to +* even read-lock the xtime seqlock: +*/ +time_t i = xtime.tv_sec; -do_gettimeofday (&tv); - i = tv.tv_sec; +smp_rmb ();/* sys_time() results are coherent */ if (tloc) { -if (put_user (i,tloc)) +if (put_user (i, tloc)) i = -EFAULT ; }return i;
kernel/time.c void do_gettimeofday (struct timeval *tv · line 385
Quoted 10 lines
kernel/time.c +7
-0@@ -373,6 +376,20 @@ void do_gettimeofday (struct timeval *tv tv->tv_sec = sec; tv->tv_usec = usec; + +/* +* Make sure xtime.tv_sec [returned by sys_time()] always +* follows the gettimeofday() result precisely. This +* condition is extremely unlikely, it can hit at most +* once per second: +*/
Unfortunatly, some arches (x86_64) can call both sys_time() and vgettimeofday().
And vgettimeofday() cannot update xtime (its mapped readonly in vsyscall page), so the coherency wont be guaranted.
Also, I thought glibc time(0) was calling gettimeofday() on x86_64, so I wonder on which machine you got your bench results.
Are you still using a 32 bits platform ? :)
Quoted 4 lines · Ingo Molnar
Subject: [patch]
sys_time()speedup
From: Ingo Molnar <[email protected]>improve performance of
sys_time().sys_time()returns time in seconds, but it does so by callingdo_gettimeofday()and then returning thetv_secportion of the GTOD time. But the data structure "xtime", which is updated by every timer/scheduler tick, already offers HZ granularity time.
How well synchronized is xtime with real-time ?
Programs invoking sys_time() do expect it to be as accurate as gettimeofday(), even if only at 1-second boundaries.
Cheers
Quoted 11 lines · Mark Lord
Quoted 8 lines · Ingo Molnar
Subject: [patch]
sys_time()speedup
From: Ingo Molnar <[email protected]>improve performance of
sys_time().sys_time()returns time in seconds, but it does so by callingdo_gettimeofday()and then returning thetv_secportion of the GTOD time. But the data structure "xtime", which is updated by every timer/scheduler tick, already offers HZ granularity time.How well synchronized is xtime with real-time ?
it's updated by every jiffy.
Quoted 2 lines
Programs invoking
sys_time()do expect it to be as accurate asgettimeofday(), even if only at 1-second boundaries.
yes.
Ingo
Quoted 12 lines · Ingo Molnar
Quoted 10 lines · Mark Lord
Quoted 8 lines · Ingo Molnar
Subject: [patch]
sys_time()speedup
From: Ingo Molnar <[email protected]>improve performance of
sys_time().sys_time()returns time in seconds, but it does so by callingdo_gettimeofday()and then returning thetv_secportion of the GTOD time. But the data structure "xtime", which is updated by every timer/scheduler tick, already offers HZ granularity time.
Quoted 3 lines
Quoted 1 line
How well synchronized is xtime with real-time ?
it's updated by every jiffy.
Well, NTP_INTERVAL_LENGTH to be specific which is every jiffie with one
exception: With dynticks this is a bit more complicated, and xtime is updated only twice a second.
thanks
-john