DTrace-like analysis possible with future Linux kernels?
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."
Quoted 3 lines · Miles Lane
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."
We've been pushing for the inclusion of the Linux Trace Toolkit in the kernel for the past 5 years. As of late, it seems that the pending argument against its inclusion is: How is this useful to end users? In answer to that, I had already posted the same pointer as above:
http://marc.theaimsgroup.com/?l=linux-kernel&m=108938594031379&w=2
Since then, I've had the chance to discuss this matter at the Kernel Summit, and again I was told that this was a sales problem (i.e. it must be demonstrated that this is actually useful to users.) So, as the developers of the Linux Trace Toolkit, it would help us a lot if you could explain to this list why the sort of functionality provided by DTrace is something you would personally find useful.
Thanks,
Karim
Signature · 3 lines
Author, Speaker, Developer, Consultant Pushing Embedded and Real-Time Linux Systems Beyond the Limits http://www.opersys.com || [email protected] || 1-866-677-4546
Quoted 3 lines · Miles Lane
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."
That article is way too hypey.
It sounds like one of those strange american commercials you see sometimes at night, where two overenthusiastic persons are telling you how much that strange fruit juice machine has changed their lives, with making them loose 200 pounds in 6 days and improving their performance at beach volleyball a lot due to subneutronic antigravity manipulation. You usually can't watch those commercials for longer than 5 minutes.
The same applies to that article, I couldn't even read it completely, it was just too much.
And is it just me or did that article really take that long to mentioning what dtrace actually IS?
Come on, it's profiling. As presented by that article, it is even more micro optimization than one would think. What with tweaking the disk I/O improvements and all... If my harddisk accesses were a microsecond more immediate or my filesystem giving a quantum more transfer rate, it would be nice, but I certainly wouldn't get enthusiastic and I bet nobody would even notice.
Maybe, without that article, I would recognize it as a fine thing (and by "fine" I don't mean "the best thing since sliced bread"), but that piece of text was just too ridiculous to take anything serious.
I sure hope that article is meant sarcastically. By the way, did I miss something or is profiling suddenly a new thing again?
Regards,
Julien
Quoted 6 lines · Julien Oster
Come on, it's profiling. As presented by that article, it is even more micro optimization than one would think. What with tweaking the disk I/O improvements and all... If my harddisk accesses were a microsecond more immediate or my filesystem giving a quantum more transfer rate, it would be nice, but I certainly wouldn't get enthusiastic and I bet nobody would even notice.
Yes and no. LTT is just profiling too. Both of them are making people notice because they allow that system profiling work to be done by two-banana grade operations staff not by the company wizard.
Neither are perfect - users want a "why is it going slow button" with a "make it work" and "beat the crap out of the user who caused it" option set.
"Profiling for the people" as it were.. (as opposed to the current fad of 'profiling the people')
Alan
Quoted 5 lines · Alan Cox
Quoted 1 line · Julien Oster
Come on, it's profiling. As presented by that article, it is even more
<snip>
"Profiling for the people" as it were.. (as opposed to the current fad of 'profiling the people')
Well profiling for user space developers. Certainly for embedded "soft realtime" work I've found LTT really useful in understanding where the contentions where in my user-space code. And also why the old pthread mutex didn't work well with SCHED_RT priorities :-(
If it was my choice I'd like to see LTT merged, but of course its not all about me much as I wish it was ;-)
Signature · 5 lines
Alex, Kernel Hacker: http://www.bennee.com/~alex/ Assembly language experience is [important] for the maturity and understanding of how computers work that it provides. -- D. Gries
Quoted 7 lines · Alex Bennee
Well profiling for user space developers. Certainly for embedded "soft realtime" work I've found LTT really useful in understanding where the contentions where in my user-space code. And also why the old pthread mutex didn't work well with
SCHED_RTpriorities :-(If it was my choice I'd like to see LTT merged, but of course its not all about me much as I wish it was ;-)
Same here - LTT turned out to be a useful tool for debugging performance issues in several USB gadget drivers. Sure, it would have been possible to instrument the boxes with the scope and trace things in hardware, but with LTT it was much easier to handle.
Robert
Signature · 5 lines
Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
Pengutronix - Linux Solutions for Science and Industry
Handelsregister: Amtsgericht Hildesheim, HRA 2686
Hornemannstraße 12, 31137 Hildesheim, Germany
Phone: +49-5121-28619-0 | Fax: +49-5121-28619-4* Julien Oster:
Quoted 7 lines
Quoted 3 lines · Miles Lane
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."That article is way too hypey.
Maybe, but DTrace seems to solve one really pressing problem: tracking disk I/O to the processes causing it. Unexplained high I/O utilization is a *very* common problem, and there aren't any tools to diagnose it.
Most other system resources can be tracked quite easily: disk space, CPU time, committed address space, even network I/O (with tcpdump and netstat -p). But there's no such thing for disk I/O.
Quoted 10 lines · Florian Weimer
* Julien Oster:
Quoted 7 lines
Quoted 3 lines · Miles Lane
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."That article is way too hypey.
Quoted 8 lines
Maybe, but DTrace seems to solve one really pressing problem: tracking disk I/O to the processes causing it. Unexplained high I/O utilization is a *very* common problem, and there aren't any tools to diagnose it.
Most other system resources can be tracked quite easily: disk space, CPU time, committed address space, even network I/O (with tcpdump and netstat -p). But there's no such thing for disk I/O.
Why can't this be done be looking at the major faults a process causes?
Not very slick indeed, but it can be done.
I wrote a small silly /proc/pid/stat parser at
http://serkiaden.mine.nu/procextract.c
One could quite easily hack up a tool to monitor I/O per process or does it need to be very more precise?
* Alexander Nyberg:
Quoted 5 lines
Quoted 3 lines
Most other system resources can be tracked quite easily: disk space, CPU time, committed address space, even network I/O (with tcpdump and netstat -p). But there's no such thing for disk I/O.
Why can't this be done be looking at the major faults a process causes?
Because only paging results in major faults, normal I/O with read()/write() (or the p*() variants) does not.
Quoted 2 lines
One could quite easily hack up a tool to monitor I/O per process or does it need to be very more precise?
It would be nice to obtain file names, too.
Quoted 4 lines · On Fri, 20 Aug 2004 15:46:33 +0200 Florian Weimer <[email protected]> wrote:
Quoted 2 lines
One could quite easily hack up a tool to monitor I/O per process or does it need to be very more precise?
It would be nice to obtain file names, too.
I bet you could even implement this quite simply as a new special oprofile event.
Quoted 8 lines · Julien Oster
Quoted 3 lines · Miles Lane
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."That article is way too hypey.
Quoted 7 lines
It sounds like one of those strange american commercials you see sometimes at night, where two overenthusiastic persons are telling you how much that strange fruit juice machine has changed their lives, with making them loose 200 pounds in 6 days and improving their performance at beach volleyball a lot due to subneutronic antigravity manipulation. You usually can't watch those commercials for longer than 5 minutes.
Probably you did try use DTrace even less than 5 minutes :->
Quoted 7 lines
The same applies to that article, I couldn't even read it completely, it was just too much.
And is it just me or did that article really take that long to mentioning what dtrace actually IS?
Come on, it's profiling.
Bullsh* :-|
DTrace is development tool for kernel and *application* develepers
because it have some tracing functionalities .. but *ALSO* .. not mainly.
It can be used *ALSO* as profiling tool but this functionality is small piece of this and most exciting part of DTrace is outhere both above areas. *ALSO* DTrace can bring classic profiling/tracing tasks to new much more effective level (look below on [1]).
For many cases where I try use DTrace and which I heard where it was uses by my friends it was used on normal *administration* tasks (look for example on SysAdmin DTrace article where on beging this article was used example how DTrace can be used for finding source of some system misconfiguration) where before was used tools like variouse {k,vm,prt,net,io,nfs,lock,...}stat tools and in mamy cases it allow answer for the same administrators question where this tools can't be used in simple way or where answering on some question can take so much time where many people will say "huh .. _maybe_ I will try this later".
Again: DTrace is *ALSO* admininstration tool and this is why I can't understand why in IBM KProbe/DProbe patch it is marked as "depends on DEBUG_KERNEL" which is IMO bigest mistake on thinking level about this :>
In Solaris DTrace is enabled in _normal production_ kernel and you can hang any probe or probes set without restarting system or any runed application which was compiled withoud debug info.
[1] Remember: if you want profile some part of code you mast _first_
(re)compile them with profiling enabled. If you wand debug some code first you must (re)compile code with enabled generate debug info. All this takes time .. and in all this cases you must restart system or traced application (also takes time).
In many cases (even not trivial) using DTrace you can perform tracing/profiling/measuring _without recompiling and also without_
_restating_ runed code (which is *very valueable*) .. all this in few
seconds.
Some enterprise systems have limited summary time to few hours per year and restart cycle can take houre or more (checking and initialize hardware components). If you will try say for admin this kind system "restat your
system using this kernel image which have enabled some additional printk() and show me syslog output" probaly you will never see this admin again. Also mamy bugs can be observed _only_ on highly loaded systems where
adding ptrace() based tracing can even kill system (trace using DTrace
takes less power from system than ptrace()).
For bring few levels up kernel *development* speed on finding some bugs source and measure some consequences adding/modifing some part of code it will be good have two very well prepared on Solaris things:
- crash dumps,
- DTrace.
On Solaris also you can combine above (you can generate crash dump using signal from DTrace program and you can review DTrace data from system crash dump image).
When you see some strange behavior without system destabization current/cassic Linux kernel development look now like:
1) if you have good kernel knowledge you can imagine which part of them
is source of same observed strange behavior,
2) after looking on kernel source you can cut of area to part where bug
exist,
3) after few recompilations you can say in which area bug exist and after
few other iteration stage 3) you can say what maust be fixed.
This classic way.
New way using DTrace can look like:
1) by using few times modified DTrace *programs* (in many cases prepared in one line command line parameters) you can locate which part of runng code (for example by what is on stack) is source of observed behavior,
2) after locate bad area code you can find associated to this source code,
3) and now from this limited area you can start thinging about "what I must know about kernel" for finding source of problem.
New way is kind of anal development. In much more cases it will allow find source of bug and have source even prepare better or worse fix by _not only_ highly expirinced kernel developers.
Also stage 1) and 2) can be performed by *not real developer* (?!).
In classic way if you are not skilled you can't even try pass 1) stage :> Passing classic variant stage 3) requires installed development enviroment.
kloczek
PS. Very interesting commens about this thread is on Bryan Cantrill (DTrace developer) blog:
http://blogs.sun.com/roller/page/bmc/20040820#dtrace_on_lkml
Bryan blog is also yet another Dtrace knowledge source ..
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 2 lines · On Sat, 21 Aug 2004 08:03:10 +0200 (CEST) Tomasz K³oczko…
[1] Remember: if you want profile some part of code you mast _first_ (re)compile them with profiling enabled.
If you use oprofile or valgrind, no you don't.
Quoted 7 lines · David S. Miller
Quoted 2 lines · On Sat, 21 Aug 2004 08:03:10 +0200 (CEST) Tomasz Kłoczko…
[1] Remember: if you want profile some part of code you mast _first_ (re)compile them with profiling enabled.
If you use oprofile or valgrind, no you don't.
Of course .. but oprofile can't be used on areas where DTrace can be :)
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Tomasz Kłoczko <[email protected]> writes:
Hello Tomasz,
Quoted 1 line
Probably you did try use DTrace even less than 5 minutes :->
No, I didn't. I clearly referred to that article, not to dtrace itself.
Quoted 4 lines
PS. Very interesting commens about this thread is on Bryan Cantrill (DTrace developer) blog:
http://blogs.sun.com/roller/page/bmc/20040820#dtrace_on_lkml
Bryan blog is also yet another Dtrace knowledge source ..
Oh, yeah, great. A whole blog entry dedicated to me. Now I am a moron, absolutely clueless and I am "looking to confirm preconceived notions rather than understand new technology".
Sorry, but that goes a little too far. No, I didn't try out dtrace and, right after reading the article (and that's the important thing!) I didn't seek for further information about it, I'm not a Solaris System Administrator right now (I was, some years ago). And all I was saying is that this *article* was just ridiculous.
Please read this paragraph of my response to it again:
| Maybe, without that article, I would recognize it as a fine thing | (and by "fine" I don't mean "the best thing since sliced bread"), | but that piece of text was just too ridiculous to take anything | serious.
That should make it obvious, shouldn't it? I would have written the same thing If I read a similar article about, for example, vmware, UML or valgrind - and I really think those are really great inventions.
But in that article, I was just missing the objectiveness. A quick note about the fact that Sun's been introducing dtrace for Solaris 10 and what it is, what it does, would have been much better instead of talking about a "Cantrill explosion", how "DTrace has completely changed the way I do business" (actual quotes).
Florian and Alan told me in a quick and objective manner why dtrace is a good thing, and I am glad for that information. I never stated that DTrace was a bad thing. I repeat it again - if I had any use for it, and I maybe have in future - it looks like I would consider DTrace a very nice thing to have. From the (non-insulting) replys I got, I understood that DTrace actually is one.
Bryan Cantrill, I can understand that you have to defend DTrace. But please, PLEASE stop saying that I am a clueless moron if I wasn't even ranting about you, ranting about DTrace, but just about *that single article* and it's presentation of DTrace to me.
And then all those comments about Linux users and developers being very defensive about DTrace... heck, can't I even critisize the quality of an ARTICLE without being accused of being a Linux maniac which fights against Solaris? I am using Solaris myself. Some years ago, I was a System Administrator for - guess what - Solaris machines. The last thing I want to step into is a religious war between Solaris and Linux. Does that mean I am not allowed to express my opinion about the public press anymore?
I read about dtrace now, I think it's a good invention. If at any place at any time we two would meet I maybe would say to you "Bryan, I like DTrace very much". If I was to administer Solaris systems right now, I would probably even say: "Bryan, DTrace has helped me a lot". But I would STILL say that that article was CRAP, because that is just what that article is from my point of view.
Now, I really don't know how to make this any more clear.
Julien
On Sat, 21 Aug 2004, Julien Oster wrote:
[..]
Quoted 9 lines
Quoted 4 lines
PS. Very interesting commens about this thread is on Bryan Cantrill (DTrace developer) blog:
http://blogs.sun.com/roller/page/bmc/20040820#dtrace_on_lkml
Bryan blog is also yet another Dtrace knowledge source ..Oh, yeah, great. A whole blog entry dedicated to me. Now I am a moron, absolutely clueless and I am "looking to confirm preconceived notions rather than understand new technology".
Quoted 5 lines
Sorry, but that goes a little too far. No, I didn't try out dtrace and, right after reading the article (and that's the important thing!) I didn't seek for further information about it, I'm not a Solaris System Administrator right now (I was, some years ago). And all I was saying is that this *article* was just ridiculous.
s/DTrace/<something_other>/ .. and yes in any other cases also if you are
not never using this <something_other> and try say publicaly what is it maybe you can't be moron but your camment ~100% will be _like_ moron
comment :_)
Why in this case you are comment like moron ? Because DTrace is consequense spending may hundrets hours by many many people (probablty not only from Sun and not only developers) .. it is probaly bigest innovation on operating system word in last few years.
It is very hard to describe in short article what DTrace is and what is not ..
and I can undestand why peple like you after reading some short text will see in this *only* tracing tool or *only* profiling tool (*olny* tools which they know) .. simple because tool like DTrace partialy creates
new class of tools.
You can know debuger, profiler, any other (statical) tracing tool and maybe some tools for measuring some interesting parameters but DTrace isn't simple combination above because it have programing abilities. For example you can add expression when and what from some bigger set parameters/points must be traced or not .. all depending on current program/kernel state.
DTrace power isn't in hooking atomic probes abilities but in combine this with very small but smart/powerfull programable VM and on collecting some data in few usefull forms (tables [1], hashes ..) and reporting all this in readable form. This why current Linux KProbe/DProbe isn't so usefull as current Solaris DTrace.
[1] in simple tables or tables indexed using probe results or eveven current stack path or other vector some variables set.
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 10 lines
Quoted 5 lines
PS. Very interesting commens about this thread is on Bryan Cantrill (DTrace developer) blog:
http://blogs.sun.com/roller/page/bmc/20040820#dtrace_on_lkml
Bryan blog is also yet another Dtrace knowledge source ..Oh, yeah, great. A whole blog entry dedicated to me. Now I am a moron, absolutely clueless and I am "looking to confirm preconceived notions rather than understand new technology".
Quoted 5 lines
Sorry, but that goes a little too far. No, I didn't try out dtrace and, right after reading the article (and that's the important thing!) I didn't seek for further information about it, I'm not a Solaris System Administrator right now (I was, some years ago). And all I was saying is that this *article* was just ridiculous.
As a reminder, you _didn't_ read the entire article, by your own admission:
That article is way too hypey... I couldn't even read it completely, it was just too much.
And that was actually my point: you spent more time denigrating the article for lack of supporting detail than it would have taken you to _finish_ the article and discover those supporting details.
Quoted 5 lines
But in that article, I was just missing the objectiveness. A quick note about the fact that Sun's been introducing dtrace for Solaris 10 and what it is, what it does, would have been much better instead of talking about a "Cantrill explosion", how "DTrace has completely changed the way I do business" (actual quotes).
You don't like customer quotes? It seems to me that quotes like that one (or like the other customer quotes that appear in the article) give weight to the claims. Don't you like to hear from people who have actually _used_ a technology? I know that I do -- those who have used a technology are likely to have a much more balanced view on its strengths and weaknesses than those who have just read about it. (Indeed, this is true of pretty much anything -- experience matters.)
Quoted 3 lines
Florian and Alan told me in a quick and objective manner why dtrace is a good thing, and I am glad for that information. I never stated that DTrace was a bad thing.
Oh really? You wrote:
Come on, it's profiling. As presented by that article, it is even more micro optimization than one would think. What with tweaking the disk I/O improvements and all... If my harddisk accesses were a microsecond more immediate or my filesystem giving a quantum more transfer rate, it would be nice, but I certainly wouldn't get enthusiastic and I bet nobody would even notice.
So come on, yourself: it's _not_ just "profiling" and DTrace finds problems that are _much_ larger than "micro-optimizations" (indeed, that's the whole damn point), and finishing the article would have told you that.
Quoted 4 lines
Bryan Cantrill, I can understand that you have to defend DTrace. But please, PLEASE stop saying that I am a clueless moron if I wasn't even ranting about you, ranting about DTrace, but just about *that single article* and it's presentation of DTrace to me.
You were attacking more than just the article; you ended with:
I sure hope that article is meant sarcastically. By the way, did I miss something or is profiling suddenly a new thing again?
You asked if you were missing something, and I replied that you were missing plenty. Presumably you now feel informed (if a little embarrassed), and I think that those that you misinformed also now realize that what you provided them was misinformation. So as far as I'm concerned, that's the end of that.
- Bryan
--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
Quoted 5 lines · Bryan Cantrill
Quoted 5 lines
Sorry, but that goes a little too far. No, I didn't try out dtrace and, right after reading the article (and that's the important thing!) I didn't seek for further information about it, I'm not a Solaris System Administrator right now (I was, some years ago). And all I was saying is that this *article* was just ridiculous.
I believe that the main problem in this dispute is just a missunderstanding based on cultural differences between Europe and America and the way articles are written and perceived.
Quoted 5 lines
Quoted 5 lines
But in that article, I was just missing the objectiveness. A quick note about the fact that Sun's been introducing dtrace for Solaris 10 and what it is, what it does, would have been much better instead of talking about a "Cantrill explosion", how "DTrace has completely changed the way I do business" (actual quotes).
True, to europeans this sounds far too overenthusiastic - almost like a commercial - and will most certainly lead to the impression, that the article is not very serious.
Europeans try to write serious articles VERY neutral - any personal opinion(s) will only be a short statement at the very end of the article.
It's just what we are used to!
Quoted 7 lines
You don't like customer quotes? It seems to me that quotes like that one (or like the other customer quotes that appear in the article) give weight to the claims. Don't you like to hear from people who have actually _used_ a technology? I know that I do -- those who have used a technology are likely to have a much more balanced view on its strengths and weaknesses than those who have just read about it. (Indeed, this is true of pretty much anything -- experience matters.)
Nobody likes them here, really. A product should stand out and prove quality by its own, we don't like to be told about it by people we don't personally know.(even if they may have a good reputation)
Another example to what europeans will regard as "unserious".
When we watch american commercials, which are sometimes broadcasted in Tv, we very often think about this as very odd. Simply very different to how this is done in Europe.
"Customer Quotes" have a very, VERY bad reputation, it smells like manipulation (whether it is or not).
Quoted 4 lines
Quoted 4 lines
Bryan Cantrill, I can understand that you have to defend DTrace. But please, PLEASE stop saying that I am a clueless moron if I wasn't even ranting about you, ranting about DTrace, but just about *that single article* and it's presentation of DTrace to me.
Personal insults will not lead to anything, and IMHO should not be made at all. A response sticking to the facts might have been more useful.
Quoted 4 lines
You were attacking more than just the article; you ended with:
I sure hope that article is meant sarcastically. By the way, did I miss something or is profiling suddenly a new thing again?
Maybe a lapse made by Julien.
But the main topic still seems to be the style the article was written in.
Quoted 5 lines
You asked if you were missing something, and I replied that you were missing plenty. Presumably you now feel informed (if a little embarrassed), and I think that those that you misinformed also now realize that what you provided them was misinformation. So as far as I'm concerned, that's the end of that.
Embarrassed?
Yes he most certainly will be, but what information did he provide to others? I dont't see much information about DTrace itself.
I hope nobody feels offended by my statements.
Christoph Halder
Quoted 3 lines · Tomasz Kłoczko
Again: DTrace is *ALSO* admininstration tool and this is why I can't understand why in IBM KProbe/DProbe patch it is marked as "depends on
DEBUG_KERNEL" which is IMO bigest mistake on thinking level about this :>
Because its a debugging feature
Quoted 3 lines
In Solaris DTrace is enabled in
_normalproduction_kernel and you can hang any probe or probes set without restarting system or any runed application which was compiled withoud debug info.
Solaris only runs on large computers. You don't want kprobes randomly on your phone, pda, wireless router. Solaris deals with an extremely narrow market segment of "big computers for people with lots of money".
Quoted 2 lines
[1] Remember: if you want profile some part of code you mast _first_ (re)compile them with profiling enabled. If you wand debug some code
OProfile doesn't require this.
Quoted 3 lines
Some enterprise systems have limited summary time to few hours per year and restart cycle can take houre or more (checking and initialize hardware components). If you will try say for admin this kind system "restat your
Enterprise users generally get kernels from vendors who have done the analysis of needs for them (and hopefully got it right). They generally don't ftp 2.6.8.1 type make config and try it out.
Quoted 5 lines
For bring few levels up kernel *development* speed on finding some bugs source and measure some consequences adding/modifing some part of code it will be good have two very well prepared on Solaris things:
- crash dumps,
- DTrace.
We have crash dumps - at least all the enterprise vendors do. Linus doesn't seem to like that stuff so much.
Quoted 8 lines
When you see some strange behavior without system destabization current/cassic Linux kernel development look now like:
1) if you have good kernel knowledge you can imagine which part of them is source of same observed strange behavior,
2) after looking on kernel source you can cut of area to part where bug exist,
3) after few recompilations you can say in which area bug exist and after few other iteration stage 3) you can say what maust be fixed.
Actually I generally
- Glance across the load meters
- Ask ps where everything is waiting
- Potentially turn on oprofile
- Potentially use netfilter to see who is causing all my traffic
- Maybe strace a few apps to see what is up
- Educate the user concerned (if needed)
I've already got the symbols (and they are in the debuginfo package from the rpm build too). I could insmod kgdb but that level of debugging is generally inappropriate.
DTrace value is ease of use value.
Quoted 2 lines
http://blogs.sun.com/roller/page/bmc/20040820#dtrace_on_lkml
Bryan blog is also yet another Dtrace knowledge source ..
Coo I thought only the Sun CEO spent his life making inappropriate comments 8)
Quoted 6 lines · Alan Cox
Quoted 3 lines · Tomasz K?oczko
Again: DTrace is *ALSO* admininstration tool and this is why I can't understand why in IBM KProbe/DProbe patch it is marked as "depends on
DEBUG_KERNEL" which is IMO bigest mistake on thinking level about this :>Because its a debugging feature
KProbe on ground/idea is very similar to DTrace -> DTrace isn't only debuging tools -> KProbe cen be used not only for debuging.
Yes it *is* debugging feature but it is much more and can be used *also* for mamy other things. So marking them as DEBUG_KERNEL dependens is wrong.
Quoted 7 lines
Quoted 3 lines
In Solaris DTrace is enabled in
_normalproduction_kernel and you can hang any probe or probes set without restarting system or any runed application which was compiled withoud debug info.Solaris only runs on large computers. You don't want kprobes randomly on your phone, pda, wireless router. Solaris deals with an extremely narrow market segment of "big computers for people with lots of money".
No Anal. You are wrong. DTrace isn't only for big computers .. it is not even for computers but for *finding souce some bugs or other behaviors* (not only bugs or bad behaviors :) without specify system size and/or price and/or importance and/or is it runed on developmer computer on not.
Ones more: DTrace isn't classic tracing/debuging/measuring tool. It is
much much more and some additional DTrace abilities aren't used only by
developers but also by administrators for finding source veriouse
problems which in spe *aren't bugs* on application or system code level.
And this why DTrace is enabled in distribution/production kernel.
You can have perfect system and perfect application but because system
and applications coexist in one enviroment this will create not empty set
bad cases.
Using yor thing path: KProbe/Dtrace is for development and yes it must depend on DEBUG_KERNEL.ptrace() is also for tracing and ver offen used by developers but it is enabled by default and it is not only for developers. So .. ptrace() must also depend on DEBUG_KERNEL. This is of corse wrong .. why ? Because strace just like DTrace/KProbe isn't only development tool and/or for developers.
IIRC all Dtrace probes can be divided in to two classes: zero effect and near zero effect. First mean: if you don't use probe this do not degrade system speed (it uses online self modified binary code without reservation memory by nop instructions for insert call entry on compilation stage). In Solaris kernel exist few thousands avalaible probes and IIRC only very small subset is "near zero effect" (uses nop instructions). All other *do not degrade system speed* and this *why* this is enabled in production kernel because price this is ~nothing !!
Solaris have very well archived binary compatibilities even on kernel level. This offen will mean: you can use some binary modules from older kernels and use them with good effect in latest Solaris. Latest Solaris (SX) have DTrace. 1 + 1 = you can trace some old code in some limited area (not the same as in code prepared for kernel with DTrace) using DTrace using "zero effect" probes if you know some entry points in this code. The same for usser space applications.
Quoted 4 lines
Quoted 2 lines
[1] Remember: if you want profile some part of code you mast _first_ (re)compile them with profiling enabled. If you wand debug some code
OProfile doesn't require this.
As same as KProbe/DTrace. Can you use OProfile for something other tnan profiling ? Probably yes and this answer opens: probably it will be good prepare some common code for KProbe and Oprofile.
Quoted 7 lines
Quoted 3 lines
Some enterprise systems have limited summary time to few hours per year and restart cycle can take houre or more (checking and initialize hardware components). If you will try say for admin this kind system "restat your
Enterprise users generally get kernels from vendors who have done the analysis of needs for them (and hopefully got it right). They generally don't ftp 2.6.8.1 type make config and try it out.
But using thins like KProbe (which is similar to DTrace) for many Linux
developers will ollow better find bugs in 2.6.8.1 :)
Are they enterprise users ? Is it realy subject _only_ chained to "enterprise users" or "vendors" ? IMO no. And again: DTrace *isn't only for kernel* (current KProbe too) and it is
much more than tracinfg tool so it is importand for developers and not develpers too.
So it will be good stop disscuss about "yes or no ?" and start about
"how and when in Linux ?" ..
Quoted 8 lines
Quoted 5 lines
For bring few levels up kernel *development* speed on finding some bugs source and measure some consequences adding/modifing some part of code it will be good have two very well prepared on Solaris things:
- crash dumps,
- DTrace.We have crash dumps - at least all the enterprise vendors do. Linus doesn't seem to like that stuff so much.
Linux CD to Solaris CD have very long distance ..
Yes it work but compare to Solaris state and as says my fiend "ca~ it only work". It need some more advanced additional tools for analize and report data from CD.
To Linus: things like CD or KProbe/DProbe allow catch problem by not skilled person and analize them in something other in diffret location in much more easier way than now. If now in source tree is integrated OProfile it will be good integrate ASAP also things like KProbes and CD. It is not only extenging entropy kernel tree. IMO KProbe can bring some
functionalities wich can be common also for OProfile and probably in future IMO OProfile can be droped.
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 5 lines · Tomasz Kłoczko
Using yor thing path: KProbe/Dtrace is for development and yes it must depend on
DEBUG_KERNEL.ptrace()is also for tracing and ver offen used by developers but it is enabled by default and it is not only for developers. So ..ptrace()must also depend onDEBUG_KERNEL.
ptrace is for debugging user space, as for example is oprofile. kprobes is for debugging including kernel internal goings on
Quoted 3 lines
compilation stage). In Solaris kernel exist few thousands avalaible probes and IIRC only very small subset is "near zero effect" (uses nop instructions).
Sounds like a kprobes clone 8).
Quoted 5 lines
Quoted 1 line
OProfile doesn't require this.
As same as KProbe/DTrace. Can you use OProfile for something other tnan profiling ? Probably yes and this answer opens: probably it will be good prepare some common code for KProbe and Oprofile.
Oprofile lets you work on stuff like cache affinity, tuning array walks and prefetches. Short of running the app under cachegrind its one of the most detailed ways of getting all the profile register data from the x86 processors.
Quoted 2 lines
So it will be good stop disscuss about "yes or no ?" and start about "how and when in Linux ?" ..
When you send patches ?
Quoted 4 lines
Quoted 2 lines
We have crash dumps - at least all the enterprise vendors do. Linus doesn't seem to like that stuff so much.
It need some more advanced additional tools for analize and report data from CD.
Standard debugging tools. The system dumps across the network to a server and then you can analyse it offline
Quoted 4 lines
OProfile it will be good integrate ASAP also things like KProbes and CD. It is not only extenging entropy kernel tree. IMO KProbe can bring some functionalities wich can be common also for OProfile and probably in future IMO OProfile can be droped.
You clearly haven't understood what Oprofile does. Its a parallel technology that is more in common with say Intel's vtune.
Quoted 9 lines · Alan Cox
Quoted 5 lines · Tomasz K?oczko
Using yor thing path: KProbe/Dtrace is for development and yes it must depend on
DEBUG_KERNEL.ptrace()is also for tracing and ver offen used by developers but it is enabled by default and it is not only for developers. So ..ptrace()must also depend onDEBUG_KERNEL.ptrace is for debugging user space, as for example is oprofile. kprobes is for debugging including kernel internal goings on
Yes. It *is* for debuging/tracing in kernel space but like DTrace is *also* in user space.
Quoted 5 lines
Quoted 3 lines
compilation stage). In Solaris kernel exist few thousands avalaible probes and IIRC only very small subset is "near zero effect" (uses nop instructions).
Sounds like a kprobes clone 8).
Look on some time stamps both projects.
Seems KProbes is clone DTrace ..
Quoted 10 lines
Quoted 5 lines
Quoted 1 line
OProfile doesn't require this.
As same as KProbe/DTrace. Can you use OProfile for something other tnan profiling ? Probably yes and this answer opens: probably it will be good prepare some common code for KProbe and Oprofile.
Oprofile lets you work on stuff like cache affinity, tuning array walks and prefetches. Short of running the app under cachegrind its one of the most detailed ways of getting all the profile register data from the x86 processors.
The same is KProbes but you can combined with small programs associated with called probes. Again: DTrace/KProbes is much more than traditional profiling/tracing/measuring tools.
Quoted 4 lines
Quoted 2 lines
So it will be good stop disscuss about "yes or no ?" and start about "how and when in Linux ?" ..
When you send patches ?
KProbes patches was annouced on lkml few times.
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 3 lines · Tomasz K?oczko
As same as KProbe/DTrace. Can you use OProfile for something other tnan profiling ? Probably yes and this answer opens: probably it will be good prepare some common code for KProbe and Oprofile.
I don't see an overlap here, except maybe the possibility of delivering sample events into the kprobes framework
Quoted 3 lines
It is not only extenging entropy kernel tree. IMO KProbe can bring some functionalities wich can be common also for OProfile and probably in future IMO OProfile can be droped.
This seems extremely unlikely to say the least, compare the methods used.
regards
john
Quoted 8 lines · Alan Cox
Quoted 3 lines · Tomasz Kłoczko
In Solaris DTrace is enabled in
_normalproduction_kernel and you can hang any probe or probes set without restarting system or any runed application which was compiled withoud debug info.Solaris only runs on large computers. You don't want kprobes randomly on your phone, pda, wireless router. Solaris deals with an extremely narrow market segment of "big computers for people with lots of money".
Overstatement.
Solaris runs on x86 platform, and runs quite well. And guess what - DTrace runs on x86 like a charm.
Quoted 4 lines
Quoted 2 lines
[1] Remember: if you want profile some part of code you mast _first_ (re)compile them with profiling enabled. If you wand debug some code
OProfile doesn't require this.
I must admit I don't know OProfile.
But can you profile already running application without interuption (not to mention stopping it) to it? Sure, DTrace introduces some overhead but if it's not acceptable just stop DTrace and application again runs with its original speed. What about getting structure contents, function arguments and returns, etc... all on the fly. And then D languages which saves a lot of time with its aggregations, thread local variables, speculations. Sure you can use Perl, AWK, and so on - but it takes time - a lot of time.
Quoted 7 lines
Quoted 3 lines
Some enterprise systems have limited summary time to few hours per year and restart cycle can take houre or more (checking and initialize hardware components). If you will try say for admin this kind system "restat your
Enterprise users generally get kernels from vendors who have done the analysis of needs for them (and hopefully got it right). They generally don't ftp 2.6.8.1 type make config and try it out.
I think you missed the point.
DTrace is not only about kernel, and it's definitly not a tool for ONLY kernel developers. It's a great tool for user land developers and for sys admins. And it's really easy (well...) to corelate data from kernel and application. All in production and it just works.
I think Tomasz was writing not only about kernel/system uptimes but also about application uptimes. And DTrace can help in both cases.
Coming back to kernel - if you have for example some kind of memory leak in kernel then support guys can provide you with DTrace script, see what's going on and get problem solved without unnecessary system restarts. Then provide you with new kernel (patch). So even if enterprise user do not want recompile its own kernel, if there's a kernel problem DTrace can save him some downtime.
Quoted 8 lines
Actually I generally
(quoted diff) +0
-6- Glance across the load meters - Ask ps where everything is waiting - Potentially turn on oprofile - Potentially use netfilter to see who is causing all my traffic - Maybe strace a few apps to see what is up - Educate the user concerned (if needed)
Quoted 5 lines
I've already got the symbols (and they are in the debuginfo package from the rpm build too). I could insmod kgdb but that level of debugging is generally inappropriate.
DTrace value is ease of use value.
Sure it's one of its values.
I would add safety of running DTrace in a production. In fact it sould be number one feature. DTrace does great job in preventing you from crashing system or applications. I would add that there is easy (at least a lot easier then without DTrace)
way to understand what's going on in system and which appliacation and why is cousing it. For example Bryan example with xcalls. And all of that in production systems.
Sure, you can make your own module on Linux, load it and trace whatever you want. But:
1. it's not easy
2. requires quite knowledge about kernel
3. could easly crash your kernel
4. takes time
5. only kernel level - what about applications and correlation between apps and kernel?
...
...
My point is DTrace is really awesome tool.
Sure you can do many things without DTrace but it will take much more time. And there are a lot of things you can do with DTrace which are really hard to do in a production using it.
It just saves your time and gives you answers to questions you would not even ask before DTrace 'coz of inacceptable time it would take to answer them. It's really 'fun' to see what's going on in system and/or appliactions with DTrace.
My post sounds almost like marketing crap - but it is really what I find in DTrace. I must admit that I did not really appreciate this tool until I've started using it.
Signature · 2 lines
Robert Milkowski [email protected]
Quoted 8 lines · On Mon, 23 Aug 2004 21:48:57 +0200 (CEST) Robert Milkowski…
Quoted 4 lines
Quoted 2 lines
[1] Remember: if you want profile some part of code you mast _first_ (re)compile them with profiling enabled. If you wand debug some code
OProfile doesn't require this.
I must admit I don't know OProfile.
But can you profile already running application without interuption (not to mention stopping it) to it?
Yes, this is exactly what oprofile allows you to do. Same with things like valgrind.
Quoted 2 lines · Robert Milkowski
Solaris runs on x86 platform, and runs quite well. And guess what - DTrace runs on x86 like a charm.
Larger x86 boxes. I can't seem to find PDA's with Solaris or phones with Solaris or $70 wireless routers with Solaris.
Quoted 2 lines
I must admit I don't know OProfile.
But can you profile already running application without interuption
Yes
Quoted 2 lines
What about getting structure contents, function arguments and returns, etc... all on the fly.
ptrace. Actually there are folks who want to take ptrace a bit further for some things - at least one vendor posted some proposals which when recast into ptrace extensions look good.
Quoted 1 line
I think you missed the point.
Nope
Quoted 2 lines
Sure, you can make your own module on Linux, load it and trace whatever you want. But:
Why do that, why not use the existing functionality that the kernel provides built on the stuff Intel AMD and friends stuck in the CPU. I'm not claiming our debugging tools are as good as dtrace but most of it (especially with kprobes patches installed) is essentially a UI design issue.
Alan
Quoted 6 lines · Alan Cox
Quoted 2 lines · Robert Milkowski
Solaris runs on x86 platform, and runs quite well. And guess what - DTrace runs on x86 like a charm.
Larger x86 boxes. I can't seem to find PDA's with Solaris or phones with Solaris or $70 wireless routers with Solaris.
I don't thing naming anything larger than PDA or phone as "large computer" is correct/acceptable :o) If fact Solaris works quite well on usual desktop size computer.
Probalby after full porting Solaris to x86 using system on embeded system definitely will not be only potential solution. Even if Solaris after this still can'd be useable on embeded-like systems
this can't matter on DTrace subject :)
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 1 line · On Sun, 29 Aug 2004 02:14:03 +0200 (CEST) Tomasz K³oczko…
If fact Solaris works quite well on usual desktop size computer.
Check out the Solaris driver selection on x86 these days, it still stinks. It is unlikely they'll ever have the coverage Linux does any time soon.
Frankly, if the only specific technical feature Sun has to brag about in Solaris 10 is DTrace, that's pretty sad. Even more so, most of the bugs I see being fixed in Solaris kernel patches are performance regressions against Linux. This, given how things were 6 or 7 years ago and the things the Solaris folks used to flame us for, I find particularly amusing.
<disclaimer>
I'm not try to advocate on Solaris or on Linux. I'm interested to incorporate DTrace like solution in Liunux. </disclaimer>
Quoted 9 lines · David S. Miller
Quoted 1 line · On Sun, 29 Aug 2004 02:14:03 +0200 (CEST) Tomasz Kłoczko…
If fact Solaris works quite well on usual desktop size computer.
Check out the Solaris driver selection on x86 these days, it still stinks. It is unlikely they'll ever have the coverage Linux does any time soon.
Quoted 6 lines
Frankly, if the only specific technical feature Sun has to brag about in Solaris 10 is DTrace, that's pretty sad. Even more so, most of the bugs I see being fixed in Solaris kernel patches are performance regressions against Linux. This, given how things were 6 or 7 years ago and the things the Solaris folks used to flame us for, I find particularly amusing.
What about zoning (which seems is much more than simple jailing) ? What about zfs which will be probably will next comparable to DTrace step forward ? (probably will come with next express build). On big computers Solaris also have much more better scaleability than Linux (I'm offen smile when I'm see questions like "Is Linux enterprise ready ?" ;). On small servers Linux is good alternative or in many cases is comparable (on choosing OS can decide another not stricte technical things) or is slightly better solution (for exemple now much more easied find well skilled adminis on Linux than on Solaris) but on medium or large computers (workgroup and higher solutions) stll in mamny cases is worse or much more worse solution for example in hardware utilization and needed funcitionalities (I'm talking about Linux vs. Solaris on sparc and also on x86 platform). Even on two way systems Solaris (10) still *much more* *better* handles threads ..
For me isn't so importand how many hardware correctly handles this or another OS. If choosen OS handles correctly my hardware I'm quite happy (in my case Linux still can't handle SunSwift T1000 NIC on my E250 ;o)
kloczek
Signature · 4 lines
----------------------------------------------------------- *Ludzie nie mają problemów, tylko sobie sami je stwarzają* ----------------------------------------------------------- Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: [email protected]*
Quoted 2 lines · On Sun, 29 Aug 2004 12:45:12 +0200 (CEST) Tomasz K³oczko…
Even on two way systems Solaris (10) still *much more* *better* handles threads ..
Back this up with facts.
Quoted 8 lines · David S. Miller
Quoted 1 line · On Sun, 29 Aug 2004 02:14:03 +0200 (CEST) Tomasz Kłoczko…
If fact Solaris works quite well on usual desktop size computer.
Check out the Solaris driver selection on x86 these days, it still stinks. It is unlikely they'll ever have the coverage Linux does any time soon.
You are right with that. However it's getting better.
On the other hand Solaris works on a really wide spectrum of x86 servers (Dell, HP, IBM). I've installed it on x86 servers from 1-way to 8-way Compaqs. I've installed it on several desktop systems too. But you are right - Linux has more drivers on x86 for some 'exotic' hardware and home use. And has much more drivers for PCI RAID cards.
Quoted 6 lines
Frankly, if the only specific technical feature Sun has to brag about in Solaris 10 is DTrace, that's pretty sad. Even more so, most of the bugs I see being fixed in Solaris kernel patches are performance regressions against Linux. This, given how things were 6 or 7 years ago and the things the Solaris folks used to flame us for, I find particularly amusing.
1. Why do you think that DTrace is the onle 'cool' feature in Solaris 10? Please stop FUD.
Frankly, if the only specific technical feature Linux has to brag about in Linux 2.6 is KProbes, that's pretty sad.
:P
2. "most of the bugs I see being fixed in Solaris kernel patches"
1. there are no patches to Solaris 10 kernel so far, so you can't see them
2. you are probably talking about some patches for Solaris 9 kernel coming from project ATLAS (performance improvements on x86 - to be as fast or faster then other OSes on the same hardware)
3. and definitely you overstated saying these are most of the patches in fact I would be suprised if there are more then 10-20 such patches (and hundreds others). Maybe you see this 'coz you are looking for word 'Linux' in patches? :)))
3. "Solaris folks used to flame us for,"
You know - there're trolls in every community.
And this thread was about DTrace and Linux tracing technologies (and not trolls, PDAs, other features)... and I know, Linux can run on PDAs. Ok, so when it comes to profiling and debugging:
1. Solaris has DTrace, ptools ant others
2. Linux runs on PDAs
:)))))))
ps. sorry for that... on the other hand a little bit of humour is ok :)
Signature · 2 lines
Robert Milkowski [email protected]
Quoted 6 lines · Alan Cox
Quoted 2 lines · Robert Milkowski
Solaris runs on x86 platform, and runs quite well. And guess what - DTrace runs on x86 like a charm.
Larger x86 boxes. I can't seem to find PDA's with Solaris or phones with Solaris or $70 wireless routers with Solaris.
Yeah, I can agree with you that tools like DTrace aren't very usefull on PDA or on phone. :)
Quoted 16 lines
Quoted 2 lines
I must admit I don't know OProfile.
But can you profile already running application without interuptionYes
Quoted 2 lines
What about getting structure contents, function arguments and returns, etc... all on the fly.
ptrace. Actually there are folks who want to take ptrace a bit further for some things - at least one vendor posted some proposals which when recast into ptrace extensions look good.
Quoted 1 line
I think you missed the point.
Nope
Quoted 8 lines
Quoted 2 lines
Sure, you can make your own module on Linux, load it and trace whatever you want. But:
Why do that, why not use the existing functionality that the kernel provides built on the stuff Intel AMD and friends stuck in the CPU. I'm not claiming our debugging tools are as good as dtrace but most of it (especially with kprobes patches installed) is essentially a UI design issue.
ok, so maybe real example.
Let's say you want aggregate by user stack if given thread number of specified process is taken off cpu by scheduler and is in SLEEP state and is off cpu for more then one second. You want output evey 10s. With DTrace it's relly simple. Ususally if I want quick answer I'm gonna write like this:
bash-2.05b# dtrace -n sched:::off-cpu'/curlwpsinfo->pr_state == SSLEEP &&
pid == 18819 && tid == 3/{self->t1 = timestamp;}'
-n sched:::on-cpu'/self->t1 && (timestamp - self->t1)
> 1000000000/{@[ustack()]=count();self->t1=0;}'
-n tick-10s'{printa(@);}'But this is not so readable so let put it in a more readable form (script)
#!/usr/sbin/dtace -s
sched:::off-cpu
/curlwpsinfo->pr_state == SSLEEP && pid == 18819 && tid == 3/ {
{self->t1 = timestamp;
}
sched:::on-cpu
/self->t1 && (timestamp - self->t1) > 1000000000/
{
@[ustack()]=count();
self->t1 = 0;
}
tick-10s
{
printa(@);
}
Here is what you get:
25 34812 :tick-10s
libc.so.1`__pollsys+0x4
libc.so.1`poll+0x88
wpserver`wpio_loop_pool+0x9c
libc.so.1`_lwp_start
26Which means you get 26 times the same stack.
Or maybe another example which shows how one can easly learn something about apps behaviour and about system.
Let's say you see a lot of fspgin's in vmstat and want to know which applications are cousing it. And to complicate things you have some applications running from inted like daemon (so fork() + exec() every request) and of course you want aggregate as a whole for such application.
So, with DTrace:
bash-2.05b# dtrace -n vminfo:::fspgin'{@[execname]=sum(arg0);}'
dtrace: description 'vminfo:::fspgin' matched 1 probe ^C
Application-A 282 Application-B 304 Application-C 335
zsched 1200
bash-2.05b#
Well, now yo can go further and see Application-C in more detail. But instead let's say you don't know why zsched is cousing fspgins. So lets' learn why.
bash-2.05b# dtrace -n vminfo:::fspgin'/execname == "zsched"/{@[stack()]=count();}'
dtrace: description 'vminfo:::fspgin' matched 1 probe
^C
genunix`pageio_setup+0x1f8
nfs`nfs3_readahead+0xc0
nfs`nfs_async_start+0x2c8
unix`thread_start+0x4
405bash-2.05b#
Well, it's doing nfs3 read aheads.
You can disable read aheads for nfs3 and see if it disappears - and it does. And all in a production without stopping anything.
Simple, easy and safe.
Now how much work and knowledge would it be needed to get the same results with KProbes, oprofile, ... and how much time will it take?
And how easy would it be with KProbes to panick kernel? All examples above I've just did on a production server.
DTrace lets you correlate data from kernel level and application level which is really usuefull. It gives sys admins the power to answer to see what is really happening in system and why.
And these are simple cases which do not show all DTrace features. Real fun starts with more complicated examples :)
Of course you can do SOME things with DProbes or Oprofile that you could with DTrace but usually it will take MUCH more time with them then with DTrace. And there are things you can do with DTrace which you can't with DProbes and others in a reasonable time period.
btw: about UI - it's really important. DTrace agreagations for example
save a lot of time in analyzing data. Especially when usually you get huge amount of data to analyze from a production systems. With
'D' language you get most of data aggregation done during collectioning data. I know there's Perl, Awk, etc. but it takes time.
btw2: you mention ptrace, AFAIK it would have more performance impact then
DTrace/KProbes technologies. Second, I'm not sure if it's still the case, but there were (are?) some problems using ptrace on threaded applications. And still all you see is application level - no correlation between kernel and app (for example you want to see if given code path in application is cousing fspgins or xcalls or something else...)
btw3: and Oprofile, Kprobes, ptrace are separate tools and you have
then corelate data which will often be not possible.
Signature · 2 lines
Robert Milkowski [email protected]
Quoted 30 lines · Julien Oster
Quoted 2 lines · Miles Lane
http://www.theregister.co.uk/2004/07/08/dtrace_user_take/:
"Sun sees DTrace as a big advantage for Solaris over other versions of Unix and Linux."That article is way too hypey.
It sounds like one of those strange american commercials you see sometimes at night, where two overenthusiastic persons are telling you how much that strange fruit juice machine has changed their lives, with making them loose 200 pounds in 6 days and improving their performance at beach volleyball a lot due to subneutronic antigravity manipulation. You usually can't watch those commercials for longer than 5 minutes.
The same applies to that article, I couldn't even read it completely, it was just too much.
And is it just me or did that article really take that long to mentioning what dtrace actually IS?
Come on, it's profiling. As presented by that article, it is even more micro optimization than one would think. What with tweaking the disk I/O improvements and all... If my harddisk accesses were a microsecond more immediate or my filesystem giving a quantum more transfer rate, it would be nice, but I certainly wouldn't get enthusiastic and I bet nobody would even notice.
Quoted 7 lines
Maybe, without that article, I would recognize it as a fine thing (and by "fine" I don't mean "the best thing since sliced bread"), but that piece of text was just too ridiculous to take anything serious.
I sure hope that article is meant sarcastically. By the way, did I miss something or is profiling suddenly a new thing again?
[I have 4000 emails from lkml to read, so please forgive me if this discussion is dead.]
DTrace was exactly what we needed here to figure out what was making our E450 server perform so badly. We managed to find and eliminate all sorts of bottlenecks, and now, all of our NFS activity is CPU bound on the server.
Perhaps Linux never suffers from these sorts of problems that require tuning things such as inode cache sizes, etc???