add performance tracing facility

PATCH Superseded Karsten Blees 3 participants 23 messages Jun 15, 2016 Kbuild

Share & tools

Karsten Blees

author

2016-06-15 23:01 UTC #0 lore raw

v5 cover add performance tracing facility

Here's v5 of the performance tracing patch series, now including a bunch of cleanups and adding timestamp, file and line to all trace output.

I'm particularly interested in feedback for the output format. As file names have different lengths, printing file:line as prefix results in unaligned output:

Quoted 1 line

GIT_TRACE=1 git stash list

00:12:10.544266 git.c:512 trace: exec: 'git-stash' 'list' 00:12:10.544266 run-command.c:337 trace: run_command: 'git-stash' 'list' 00:12:10.649779 git.c:312 trace: built-in: git 'rev-parse' '--git-dir'

We could add separators to make it easier to parse, e.g.:

Quoted 1 line

GIT_TRACE=1 git stash list

[00:12:10.544266 git.c:512] trace: exec: 'git-stash' 'list' [00:12:10.544266 run-command.c:337] trace: run_command: 'git-stash' 'list' [00:12:10.649779 git.c:312] trace: built-in: git 'rev-parse' '--git-dir'

Or print file:line at the end (but what about multi-line messages, such as packet-trace?):

Quoted 1 line

GIT_TRACE=1 git stash list

00:12:10.544266 trace: exec: 'git-stash' 'list' (git.c:512) 00:12:10.544266 trace: run_command: 'git-stash' 'list' (run-command.c:337) 00:12:10.649779 trace: built-in: git 'rev-parse' '--git-dir' (git.c:312)

Karsten Blees (11):
trace: move trace declarations from cache.h to new trace.h
trace: consistently name the format parameter
trace: remove redundant printf format attribute
trace: factor out printing to the trace file
trace: add infrastructure to augment trace output with additional info
trace: add current timestamp to all trace output
trace: move code around, in preparation to file:line output
trace: add 'file:line' to all trace output
trace: add high resolution timer function to debug performance issues
trace: add trace_performance facility to debug performance issues
git: add performance tracing for git's main() function to debug
scripts

 Makefile          |   7 ++
 cache.h           |  13 +--
 config.mak.uname  |   1 +
 git-compat-util.h |   4 +
 git.c             |   2 +
 trace.c           | 304 +++++++++++++++++++++++++++++++++++++++++++++++-------
 trace.h           |  96 +++++++++++++++++
 7 files changed, 379 insertions(+), 48 deletions(-)
 create mode 100644 trace.h

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #1 lore raw

PATCH v5 1/11 trace: move trace declarations from cache.h to new trace.h

Also include direct dependencies (strbuf.h and git-compat-util.h for __attribute__) so that trace.h can be used independently of cache.h, e.g. in test programs.

Signed-off-by Karsten Blees <[email protected]>

---
 cache.h | 13 ++-----------
 trace.h | 17 +++++++++++++++++
 2 files changed, 19 insertions(+), 11 deletions(-)
 create mode 100644 trace.h

cache.h +2 ~~-11~~

index 1e4b4f0..79ac940 100644
@@ -7,6 +7,7 @@
 #include "advice.h"
 #include "gettext.h"
 #include "convert.h"
+#include "trace.h"
 
 #include SHA1_HEADER
 #ifndef git_SHA_CTX

@@ -1378,17 +1379,7 @@ extern void *alloc_tag_node(void);
 extern void *alloc_object_node(void);
 extern void alloc_report(void);
 
-/* trace.c */
-__attribute__((format (printf, 1, 2)))
-extern void trace_printf(const char *format, ...);
-__attribute__((format (printf, 2, 3)))
-extern void trace_argv_printf(const char **argv, const char *format, ...);
-extern void trace_repo_setup(const char *prefix);
-extern int trace_want(const char *key);
-__attribute__((format (printf, 2, 3)))
-extern void trace_printf_key(const char *key, const char *fmt, ...);
-extern void trace_strbuf(const char *key, const struct strbuf *buf);
-
+/* pkt-line.c */
 void packet_trace_identity(const char *prog);
 
 /* add */

trace.h new file +17 -0

new file mode 100644
index 0000000..6cc4541
@@ -0,0 +1,17 @@
+#ifndef TRACE_H
+#define TRACE_H
+
+#include "git-compat-util.h"
+#include "strbuf.h"
+
+__attribute__((format (printf, 1, 2)))
+extern void trace_printf(const char *format, ...);
+__attribute__((format (printf, 2, 3)))
+extern void trace_argv_printf(const char **argv, const char *format, ...);
+extern void trace_repo_setup(const char *prefix);
+extern int trace_want(const char *key);
+__attribute__((format (printf, 2, 3)))
+extern void trace_printf_key(const char *key, const char *fmt, ...);
+extern void trace_strbuf(const char *key, const struct strbuf *buf);
+
+#endif /* TRACE_H */

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #2 lore raw

PATCH v5 2/11 trace: consistently name the format parameter

In reply to Karsten Blees

The format parameter to trace_printf functions is sometimes abbreviated 'fmt'. Rename to 'format' everywhere (consistent with POSIX' printf specification).

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 22 +++++++++++-----------
 trace.h |  2 +-
 2 files changed, 12 insertions(+), 12 deletions(-)

trace.c +11 ~~-11~~

index 08180a9..37a7fa9 100644
@@ -62,7 +62,7 @@ static int get_trace_fd(const char *key, int *need_close)
 static const char err_msg[] = "Could not trace into fd given by "
 	"GIT_TRACE environment variable";
 
-static void trace_vprintf(const char *key, const char *fmt, va_list ap)
+static void trace_vprintf(const char *key, const char *format, va_list ap)
 {
 	struct strbuf buf = STRBUF_INIT;

@@ -70,25 +70,25 @@ static void trace_vprintf(const char *key, const char *fmt, va_list ap)
 		return;
 
 	set_try_to_free_routine(NULL);	/* is never reset */
-	strbuf_vaddf(&buf, fmt, ap);
+	strbuf_vaddf(&buf, format, ap);
 	trace_strbuf(key, &buf);
 	strbuf_release(&buf);
 }
 
 __attribute__((format (printf, 2, 3)))
-void trace_printf_key(const char *key, const char *fmt, ...)
+void trace_printf_key(const char *key, const char *format, ...)
 {
 	va_list ap;
-	va_start(ap, fmt);
-	trace_vprintf(key, fmt, ap);
+	va_start(ap, format);
+	trace_vprintf(key, format, ap);
 	va_end(ap);
 }
 
-void trace_printf(const char *fmt, ...)
+void trace_printf(const char *format, ...)
 {
 	va_list ap;
-	va_start(ap, fmt);
-	trace_vprintf("GIT_TRACE", fmt, ap);
+	va_start(ap, format);
+	trace_vprintf("GIT_TRACE", format, ap);
 	va_end(ap);
 }

@@ -106,7 +106,7 @@ void trace_strbuf(const char *key, const struct strbuf *buf)
 		close(fd);
 }
 
-void trace_argv_printf(const char **argv, const char *fmt, ...)
+void trace_argv_printf(const char **argv, const char *format, ...)
 {
 	struct strbuf buf = STRBUF_INIT;
 	va_list ap;

@@ -117,8 +117,8 @@ void trace_argv_printf(const char **argv, const char *fmt, ...)
 		return;
 
 	set_try_to_free_routine(NULL);	/* is never reset */
-	va_start(ap, fmt);
-	strbuf_vaddf(&buf, fmt, ap);
+	va_start(ap, format);
+	strbuf_vaddf(&buf, format, ap);
 	va_end(ap);
 
 	sq_quote_argv(&buf, argv, 0);

trace.h +1 -1

index 6cc4541..8fea50b 100644
@@ -11,7 +11,7 @@ extern void trace_argv_printf(const char **argv, const char *format, ...);
 extern void trace_repo_setup(const char *prefix);
 extern int trace_want(const char *key);
 __attribute__((format (printf, 2, 3)))
-extern void trace_printf_key(const char *key, const char *fmt, ...);
+extern void trace_printf_key(const char *key, const char *format, ...);
 extern void trace_strbuf(const char *key, const struct strbuf *buf);
 
 #endif /* TRACE_H */

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #3 lore raw

PATCH v5 3/11 trace: remove redundant printf format attribute

In reply to Karsten Blees

trace_printf_key() is the only non-static function that duplicates the printf format attribute in the .c file, remove it for consistency.

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 1 -
 1 file changed, 1 deletion(-)

trace.c +0 -1

index 37a7fa9..3e31558 100644
@@ -75,7 +75,6 @@ static void trace_vprintf(const char *key, const char *format, va_list ap)
 	strbuf_release(&buf);
 }
 
-__attribute__((format (printf, 2, 3)))
 void trace_printf_key(const char *key, const char *format, ...)
 {
 	va_list ap;

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #4 lore raw

PATCH v5 4/11 trace: factor out printing to the trace file

In reply to Karsten Blees

Opening and writing to the trace file is currently duplicated in trace_strbuf() and trace_argv_printf(). Factor out this logic to prepare for adding timestamp and file:line to trace output.

In case of trace_argv_printf(), this adds an additional trace_want() check to prevent unnecessary string formatting.

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 37 +++++++++++++++++++------------------
 1 file changed, 19 insertions(+), 18 deletions(-)

trace.c +19 ~~-18~~

index 3e31558..b7ca51b 100644
@@ -62,6 +62,21 @@ static int get_trace_fd(const char *key, int *need_close)
 static const char err_msg[] = "Could not trace into fd given by "
 	"GIT_TRACE environment variable";
 
+/* Print 'buf' verbatim to trace file designated by env var 'key' */
+static void do_trace_print(const char *key, const struct strbuf *buf)
+{
+	int fd, need_close = 0;
+
+	fd = get_trace_fd(key, &need_close);
+	if (!fd)
+		return;
+
+	write_or_whine_pipe(fd, buf->buf, buf->len, err_msg);
+
+	if (need_close)
+		close(fd);
+}
+
 static void trace_vprintf(const char *key, const char *format, va_list ap)
 {
 	struct strbuf buf = STRBUF_INIT;

@@ -71,7 +86,7 @@ static void trace_vprintf(const char *key, const char *format, va_list ap)
 
 	set_try_to_free_routine(NULL);	/* is never reset */
 	strbuf_vaddf(&buf, format, ap);
-	trace_strbuf(key, &buf);
+	do_trace_print(key, &buf);
 	strbuf_release(&buf);
 }

@@ -93,26 +108,15 @@ void trace_printf(const char *format, ...)
 
 void trace_strbuf(const char *key, const struct strbuf *buf)
 {
-	int fd, need_close = 0;
-
-	fd = get_trace_fd(key, &need_close);
-	if (!fd)
-		return;
-
-	write_or_whine_pipe(fd, buf->buf, buf->len, err_msg);
-
-	if (need_close)
-		close(fd);
+	do_trace_print(key, buf);
 }
 
 void trace_argv_printf(const char **argv, const char *format, ...)
 {
 	struct strbuf buf = STRBUF_INIT;
 	va_list ap;
-	int fd, need_close = 0;
 
-	fd = get_trace_fd("GIT_TRACE", &need_close);
-	if (!fd)
+	if (!trace_want("GIT_TRACE"))
 		return;
 
 	set_try_to_free_routine(NULL);	/* is never reset */

@@ -122,11 +126,8 @@ void trace_argv_printf(const char **argv, const char *format, ...)
 
 	sq_quote_argv(&buf, argv, 0);
 	strbuf_addch(&buf, '\n');
-	write_or_whine_pipe(fd, buf.buf, buf.len, err_msg);
+	do_trace_print("GIT_TRACE", &buf);
 	strbuf_release(&buf);
-
-	if (need_close)
-		close(fd);
 }
 
 static const char *quote_crnl(const char *path)

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #5 lore raw

PATCH v5 5/11 trace: add infrastructure to augment trace output with additional info

In reply to Karsten Blees

To be able to add a common prefix or suffix to all trace output (e.g. a timestamp or file:line of the caller), factor out common setup and cleanup tasks of the trace* functions.

When adding a common prefix, it makes sense that the output of each trace call starts on a new line. Add '\n' in case the caller forgot.

Note that this explicitly limits trace output to line-by-line, it is no longer possible to trace-print just part of a line. Until now, this was just an implicit assumption (trace-printing part of a line worked, but messed up the trace file if multiple threads or processes were involved).

Thread-safety / inter-process-safety is also the reason why we need to do the prefixing and suffixing in memory rather than issuing multiple write() calls. Write_or_whine_pipe() / xwrite() is atomic unless the size exceeds MAX_IO_SIZE (8MB, see wrapper.c). In case of trace_strbuf, this costs an additional string copy (which should be irrelevant for performance in light of actual file IO).

While we're at it, rename trace_strbuf's 'buf' argument, which suggests that the function is modifying the buffer. Trace_strbuf() currently is the only trace API that can print arbitrary binary data (without barfing on '%' or stopping at '\0'), so 'data' seems more appropriate.

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 45 ++++++++++++++++++++++++++++++++++-----------
 trace.h |  2 +-
 2 files changed, 35 insertions(+), 12 deletions(-)

trace.c +34 ~~-11~~

index b7ca51b..c920429 100644
@@ -77,17 +77,37 @@ static void do_trace_print(const char *key, const struct strbuf *buf)
 		close(fd);
 }
 
+static int prepare_trace_line(const char *key, struct strbuf *buf)
+{
+	if (!trace_want(key))
+		return 0;
+
+	set_try_to_free_routine(NULL);	/* is never reset */
+
+	/* add line prefix here */
+
+	return 1;
+}
+
+static void print_trace_line(const char *key, struct strbuf *buf)
+{
+	/* append newline if missing */
+	if (buf->len && buf->buf[buf->len - 1] != '\n')
+		strbuf_addch(buf, '\n');
+
+	do_trace_print(key, buf);
+	strbuf_release(buf);
+}
+
 static void trace_vprintf(const char *key, const char *format, va_list ap)
 {
 	struct strbuf buf = STRBUF_INIT;
 
-	if (!trace_want(key))
+	if (!prepare_trace_line(key, &buf))
 		return;
 
-	set_try_to_free_routine(NULL);	/* is never reset */
 	strbuf_vaddf(&buf, format, ap);
-	do_trace_print(key, &buf);
-	strbuf_release(&buf);
+	print_trace_line(key, &buf);
 }
 
 void trace_printf_key(const char *key, const char *format, ...)

@@ -106,9 +126,15 @@ void trace_printf(const char *format, ...)
 	va_end(ap);
 }
 
-void trace_strbuf(const char *key, const struct strbuf *buf)
+void trace_strbuf(const char *key, const struct strbuf *data)
 {
-	do_trace_print(key, buf);
+	struct strbuf buf = STRBUF_INIT;
+
+	if (!prepare_trace_line(key, &buf))
+		return;
+
+	strbuf_addbuf(&buf, data);
+	print_trace_line(key, &buf);
 }
 
 void trace_argv_printf(const char **argv, const char *format, ...)

@@ -116,18 +142,15 @@ void trace_argv_printf(const char **argv, const char *format, ...)
 	struct strbuf buf = STRBUF_INIT;
 	va_list ap;
 
-	if (!trace_want("GIT_TRACE"))
+	if (!prepare_trace_line("GIT_TRACE", &buf))
 		return;
 
-	set_try_to_free_routine(NULL);	/* is never reset */
 	va_start(ap, format);
 	strbuf_vaddf(&buf, format, ap);
 	va_end(ap);
 
 	sq_quote_argv(&buf, argv, 0);
-	strbuf_addch(&buf, '\n');
-	do_trace_print("GIT_TRACE", &buf);
-	strbuf_release(&buf);
+	print_trace_line("GIT_TRACE", &buf);
 }
 
 static const char *quote_crnl(const char *path)

trace.h +1 -1

index 8fea50b..e03db2f 100644
@@ -12,6 +12,6 @@ extern void trace_repo_setup(const char *prefix);
 extern int trace_want(const char *key);
 __attribute__((format (printf, 2, 3)))
 extern void trace_printf_key(const char *key, const char *format, ...);
-extern void trace_strbuf(const char *key, const struct strbuf *buf);
+extern void trace_strbuf(const char *key, const struct strbuf *data);
 
 #endif /* TRACE_H */

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #6 lore raw

PATCH v5 6/11 trace: add current timestamp to all trace output

In reply to Karsten Blees

This is useful to tell apart trace output of separate test runs.

It can also be used for basic, coarse-grained performance analysis. Note that the accuracy is tainted by writing to the trace file, and you have to calculate the deltas yourself (which is next to impossible if multiple threads or processes are involved).

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

trace.c +10 -1

index c920429..5a3393a 100644
@@ -79,12 +79,21 @@ static void do_trace_print(const char *key, const struct strbuf *buf)
 
 static int prepare_trace_line(const char *key, struct strbuf *buf)
 {
+	struct timeval tv;
+	struct tm tm;
+	time_t secs;
+
 	if (!trace_want(key))
 		return 0;
 
 	set_try_to_free_routine(NULL);	/* is never reset */
 
-	/* add line prefix here */
+	/* print current timestamp */
+	gettimeofday(&tv, NULL);
+	secs = tv.tv_sec;
+	localtime_r(&secs, &tm);
+	strbuf_addf(buf, "%02d:%02d:%02d.%06ld ", tm.tm_hour, tm.tm_min,
+		    tm.tm_sec, tv.tv_usec);
 
 	return 1;
 }

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #7 lore raw

PATCH v5 7/11 trace: move code around, in preparation to file:line output

In reply to Karsten Blees

No functional changes, just move stuff around so that the next patch isn't that ugly...

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 36 ++++++++++++++++++------------------
 trace.h | 12 ++++++++----
 2 files changed, 26 insertions(+), 22 deletions(-)

trace.c +18 ~~-18~~

index 5a3393a..c86d33c 100644
@@ -119,20 +119,20 @@ static void trace_vprintf(const char *key, const char *format, va_list ap)
 	print_trace_line(key, &buf);
 }
 
-void trace_printf_key(const char *key, const char *format, ...)
+void trace_argv_printf(const char **argv, const char *format, ...)
 {
+	struct strbuf buf = STRBUF_INIT;
 	va_list ap;
-	va_start(ap, format);
-	trace_vprintf(key, format, ap);
-	va_end(ap);
-}
 
-void trace_printf(const char *format, ...)
-{
-	va_list ap;
+	if (!prepare_trace_line("GIT_TRACE", &buf))
+		return;
+
 	va_start(ap, format);
-	trace_vprintf("GIT_TRACE", format, ap);
+	strbuf_vaddf(&buf, format, ap);
 	va_end(ap);
+
+	sq_quote_argv(&buf, argv, 0);
+	print_trace_line("GIT_TRACE", &buf);
 }
 
 void trace_strbuf(const char *key, const struct strbuf *data)

@@ -146,20 +146,20 @@ void trace_strbuf(const char *key, const struct strbuf *data)
 	print_trace_line(key, &buf);
 }
 
-void trace_argv_printf(const char **argv, const char *format, ...)
+void trace_printf(const char *format, ...)
 {
-	struct strbuf buf = STRBUF_INIT;
 	va_list ap;
-
-	if (!prepare_trace_line("GIT_TRACE", &buf))
-		return;
-
 	va_start(ap, format);
-	strbuf_vaddf(&buf, format, ap);
+	trace_vprintf("GIT_TRACE", format, ap);
 	va_end(ap);
+}
 
-	sq_quote_argv(&buf, argv, 0);
-	print_trace_line("GIT_TRACE", &buf);
+void trace_printf_key(const char *key, const char *format, ...)
+{
+	va_list ap;
+	va_start(ap, format);
+	trace_vprintf(key, format, ap);
+	va_end(ap);
 }
 
 static const char *quote_crnl(const char *path)

trace.h +8 -4

index e03db2f..5c7f2dc 100644
@@ -4,14 +4,18 @@
 #include "git-compat-util.h"
 #include "strbuf.h"
 
-__attribute__((format (printf, 1, 2)))
-extern void trace_printf(const char *format, ...);
-__attribute__((format (printf, 2, 3)))
-extern void trace_argv_printf(const char **argv, const char *format, ...);
 extern void trace_repo_setup(const char *prefix);
 extern int trace_want(const char *key);
+
+__attribute__((format (printf, 1, 2)))
+extern void trace_printf(const char *format, ...);
+
 __attribute__((format (printf, 2, 3)))
 extern void trace_printf_key(const char *key, const char *format, ...);
+
+__attribute__((format (printf, 2, 3)))
+extern void trace_argv_printf(const char **argv, const char *format, ...);
+
 extern void trace_strbuf(const char *key, const struct strbuf *data);
 
 #endif /* TRACE_H */

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #8 lore raw

PATCH v5 8/11 trace: add 'file:line' to all trace output

In reply to Karsten Blees

This is useful to see where trace output came from.

Add 'const char *file, int line' parameters to the printing functions and rename them to *_fl.

Add trace_printf* and trace_strbuf macros resolving to the *_fl functions and let the preprocessor fill in __FILE__ and __LINE__.

As the trace_printf* functions take a variable number of arguments, this requires variadic macros (i.e. '#define foo(...) foo_impl(__VA_ARGS__)'. Though part of C99, it is unclear whether older compilers support this. Thus keep the old functions and only enable variadic macros for GNUC and MSVC 2005+ (_MSC_VER 1400). This has the nice side effect that the old C-style declarations serve as documentation how the macros are to be used.

Signed-off-by Karsten Blees <[email protected]>

---
 git-compat-util.h |  4 ++++
 trace.c           | 69 +++++++++++++++++++++++++++++++++++++++++++++----------
 trace.h           | 49 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+), 12 deletions(-)

git-compat-util.h +4 -0

index 7849d31..e5a773b 100644
@@ -700,6 +700,10 @@ void git_qsort(void *base, size_t nmemb, size_t size,
 #endif
 #endif
 
+#if defined(__GNUC__) || (_MSC_VER >= 1400)
+#define HAVE_VARIADIC_MACROS 1
+#endif
+
 /*
  * Preserves errno, prints a message, but gives no warning for ENOENT.
  * Always returns the return value of unlink(2).

trace.c +57 ~~-12~~

129 lines not on this page · raw message

trace.h +49 -0

59 lines not on this page · raw message

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #9 lore raw

PATCH v5 9/11 trace: add high resolution timer function to debug performance issues

In reply to Karsten Blees

Add a getnanotime() function that returns nanoseconds since 01/01/1970 as unsigned 64-bit integer (i.e. overflows in july 2554). This is easier to work with than e.g. struct timeval or struct timespec. Basing the timer on the epoch allows using the results (div 10e9) with other time-related APIs.

To simplify adaption to different platforms, split the implementation into a common getnanotime() and a platform-specific highres_nanos() function.

The common getnanotime() function handles errors, falling back to gettimeofday() if highres_nanos() isn't implemented or doesn't work.

getnanotime() is also responsible for normalizing to the epoch. The offset to the system clock is calculated only once on initialization, i.e. manually setting the system clock has no impact on the timer (except if the fallback gettimeofday() is in use). Git processes are typically short lived, so we don't need to handle clock drift.

The highres_nanos() function returns monotonically increasing nanoseconds relative to some arbitrary point in time (e.g. system boot), or 0 on failure. Providing platform-specific implementations should be relatively easy, e.g. adapting to clock_gettime() as defined by the POSIX realtime extensions is seven lines of code.

This version includes highres_nanos() implementations for:
* Linux: using clock_gettime(CLOCK_MONOTONIC)
* Windows: using QueryPerformanceCounter()

Todo:
* enable clock_gettime() on more platforms
* add Mac OSX version, e.g. using mach_absolute_time + mach_timebase_info

Signed-off-by Karsten Blees <[email protected]>

---
 Makefile         |  7 +++++
 config.mak.uname |  1 +
 trace.c          | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 trace.h          |  1 +
 4 files changed, 91 insertions(+)

Makefile +7 -0

index 07ea105..80f4390 100644
@@ -340,6 +340,8 @@ all::
 #
 # Define GMTIME_UNRELIABLE_ERRORS if your gmtime() function does not
 # return NULL when it receives a bogus time_t.
+#
+# Define HAVE_CLOCK_GETTIME if your platform has clock_gettime in librt.
 
 GIT-VERSION-FILE: FORCE
 	@$(SHELL_PATH) ./GIT-VERSION-GEN

@@ -1497,6 +1499,11 @@ ifdef GMTIME_UNRELIABLE_ERRORS
 	BASIC_CFLAGS += -DGMTIME_UNRELIABLE_ERRORS
 endif
 
+ifdef HAVE_CLOCK_GETTIME
+	BASIC_CFLAGS += -DHAVE_CLOCK_GETTIME
+	EXTLIBS += -lrt
+endif
+
 ifeq ($(TCLTK_PATH),)
 NO_TCLTK = NoThanks
 endif

config.mak.uname +1 -0

index 1ae675b..dad2618 100644
@@ -34,6 +34,7 @@ ifeq ($(uname_S),Linux)
 	HAVE_PATHS_H = YesPlease
 	LIBC_CONTAINS_LIBINTL = YesPlease
 	HAVE_DEV_TTY = YesPlease
+	HAVE_CLOCK_GETTIME = YesPlease
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
 	HAVE_ALLOCA_H = YesPlease

trace.c +82 -0

85 lines not on this page · raw message

trace.h +1 -0

index 916dc2b..c4964aa 100644
@@ -6,6 +6,7 @@
 
 extern void trace_repo_setup(const char *prefix);
 extern int trace_want(const char *key);
+extern uint64_t getnanotime(void);
 
 #ifndef HAVE_VARIADIC_MACROS

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #10 lore raw

Re: [PATCH v5 09/11] trace: add high resolution timer function to debug performance issues

Am 11.06.2014 10:01, schrieb Karsten Blees:

Quoted 1 line

the epoch allows using the results (div 10e9) with other time-related APIs.

s/10e9/1e9/

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Junio C Hamano

2016-06-15 23:01 UTC #11 lore raw

Re: [PATCH v5 09/11] trace: add high resolution timer function to debug performance issues

Quoted 4 lines · Karsten Blees

Am 11.06.2014 10:01, schrieb Karsten Blees:
Quoted 1 line
the epoch allows using the results (div 10e9) with other time-related APIs.
s/10e9/1e9/

That replacement is fine but the "(div 1e9)" still wants to be clarified. What did you exactly mean by that? If the result is divided by 10^9 then it yields the number of seconds?

Thanks.

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #12 lore raw

Re: [PATCH v5 09/11] trace: add high resolution timer function to debug performance issues

Am 17.06.2014 18:44, schrieb Junio C Hamano:

Quoted 5 lines

Quoted 3 lines · Karsten Blees
Am 11.06.2014 10:01, schrieb Karsten Blees:
Quoted 1 line
the epoch allows using the results (div 10e9) with other time-related APIs.

Quoted 8 lines

Quoted 1 line
s/10e9/1e9/
That replacement is fine but the "(div 1e9)" still wants to be clarified. What did you exactly mean by that? If the result is divided by 10^9 then it yields the number of seconds?
Thanks.

Div 10^9 yields a time_t. But as not all time-related APIs use time_t, perhaps its best to drop the "(div 1e9)" altogether. That the function returns nanoseconds should be clear enough. Will drop in the next round.

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #13 lore raw

PATCH v5 10/11 trace: add trace_performance facility to debug performance issues

In reply to Karsten Blees

Add trace_performance and trace_performance_since macros that print a duration and an optional printf-formatted text to the file specified in environment variable GIT_TRACE_PERFORMANCE.

These macros, in conjunction with getnanotime(), are intended to simplify performance measurements from within the application (i.e. profiling via manual instrumentation, rather than using an external profiling tool).

Unless enabled via GIT_TRACE_PERFORMANCE, these macros have no noticeable impact on performance, so that test code for well known time killers may be shipped in release builds. Alternatively, a developer could provide an additional performance patch (not meant for master) that allows reviewers to reproduce performance tests more easily, e.g. on other platforms or using their own repositories.

Usage examples:

Simple use case (measure one code section):

  uint64_t start = getnanotime();
  /* code section to measure */
  trace_performance_since(start, "foobar");

Medium use case (measure consecutive code sections):

  uint64_t start = getnanotime();
  /* first code section to measure */
  start = trace_performance_since(start, "first foobar");
  /* second code section to measure */
  trace_performance_since(start, "second foobar");

Complex use case (measure repetitive code sections):

  uint64_t t = 0;
  for (;;) {
    /* ignore */
    t -= getnanotime();
    /* code section to measure */
    t += getnanotime();
    /* ignore */
  }
  trace_performance(t, "frotz");

Signed-off-by Karsten Blees <[email protected]>

---
 trace.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 trace.h | 24 ++++++++++++++++++++++++
 2 files changed, 73 insertions(+)

trace.c +49 -0

index 4bd52f2..0551509 100644
@@ -153,6 +153,27 @@ void trace_strbuf_fl(const char *file, int line, const char *key,
 	print_trace_line(key, &buf);
 }
 
+static const char *GIT_TRACE_PERFORMANCE = "GIT_TRACE_PERFORMANCE";
+
+static void trace_performance_vfl(const char *file, int line,
+				      uint64_t nanos, const char *format,
+				      va_list ap)
+{
+	struct strbuf buf = STRBUF_INIT;
+
+	if (!prepare_trace_line(file, line, GIT_TRACE_PERFORMANCE, &buf))
+		return;
+
+	strbuf_addf(&buf, "performance: %.9f s", (double) nanos / 1000000000);
+
+	if (format && *format) {
+		strbuf_addstr(&buf, ": ");
+		strbuf_vaddf(&buf, format, ap);
+	}
+
+	print_trace_line(GIT_TRACE_PERFORMANCE, &buf);
+}
+
 #ifndef HAVE_VARIADIC_MACROS
 
 void trace_printf(const char *format, ...)

@@ -184,6 +205,24 @@ void trace_strbuf(const char *key, const struct strbuf *data)
 	trace_strbuf_fl(NULL, 0, key, data);
 }
 
+uint64_t trace_performance(uint64_t nanos, const char *format, ...)
+{
+	va_list ap;
+	va_start(ap, format);
+	trace_performance_vfl(NULL, 0, nanos, format, ap);
+	va_end(ap);
+	return getnanotime();
+}
+
+uint64_t trace_performance_since(uint64_t start, const char *format, ...)
+{
+	va_list ap;
+	va_start(ap, format);
+	trace_performance_vfl(NULL, 0, getnanotime() - start, format, ap);
+	va_end(ap);
+	return getnanotime();
+}
+
 #else
 
 void trace_printf_key_fl(const char *file, int line, const char *key,

@@ -204,6 +243,16 @@ void trace_argv_printf_fl(const char *file, int line, const char **argv,
 	va_end(ap);
 }
 
+uint64_t trace_performance_fl(const char *file, int line, uint64_t nanos,
+			      const char *format, ...)
+{
+	va_list ap;
+	va_start(ap, format);
+	trace_performance_vfl(file, line, nanos, format, ap);
+	va_end(ap);
+	return getnanotime();
+}
+
 #endif /* HAVE_VARIADIC_MACROS */

trace.h +24 -0

index c4964aa..9687563 100644
@@ -21,6 +21,20 @@ extern void trace_argv_printf(const char **argv, const char *format, ...);
 
 extern void trace_strbuf(const char *key, const struct strbuf *data);
 
+/*
+ * Prints specified time (in nanoseconds) if GIT_TRACE_PERFORMANCE is enabled.
+ * Returns current time in nanoseconds.
+ */
+__attribute__((format (printf, 2, 3)))
+extern uint64_t trace_performance(uint64_t nanos, const char *format, ...);
+
+/*
+ * Prints time since 'start' if GIT_TRACE_PERFORMANCE is enabled.
+ * Returns current time in nanoseconds.
+ */
+__attribute__((format (printf, 2, 3)))
+extern uint64_t trace_performance_since(uint64_t start, const char *format, ...);
+
 #else
 
 /*

@@ -56,6 +70,13 @@ extern void trace_strbuf(const char *key, const struct strbuf *data);
 #define trace_strbuf(key, data) \
 	trace_strbuf_fl(__FILE__, __LINE__, key, data)
 
+#define trace_performance(nanos, ...) \
+	trace_performance_fl(__FILE__, __LINE__, nanos, __VA_ARGS__)
+
+#define trace_performance_since(start, ...) \
+	trace_performance_fl(__FILE__, __LINE__, getnanotime() - (start), \
+			     __VA_ARGS__)
+
 /* backend functions, use non-*fl macros instead */
 __attribute__((format (printf, 4, 5)))
 extern void trace_printf_key_fl(const char *file, int line, const char *key,

@@ -65,6 +86,9 @@ extern void trace_argv_printf_fl(const char *file, int line, const char **argv,
 				 const char *format, ...);
 extern void trace_strbuf_fl(const char *file, int line, const char *key,
 			    const struct strbuf *data);
+__attribute__((format (printf, 4, 5)))
+extern uint64_t trace_performance_fl(const char *file, int line,
+				     uint64_t nanos, const char *fmt, ...);
 
 #endif /* HAVE_VARIADIC_MACROS */

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Junio C Hamano

2016-06-15 23:01 UTC #14 lore raw

Re: [PATCH v5 10/11] trace: add trace_performance facility to debug performance issues

Quoted 16 lines · Karsten Blees

Simple use case (measure one code section):
  uint64_t start = getnanotime();
  /* code section to measure */
  trace_performance_since(start, "foobar");
Medium use case (measure consecutive code sections):
  uint64_t start = getnanotime();
  /* first code section to measure */
  start = trace_performance_since(start, "first foobar");
  /* second code section to measure */
  trace_performance_since(start, "second foobar");
Complex use case (measure repetitive code sections):

Quoted 9 lines

  uint64_t t = 0;
  for (;;) {
    /* ignore */
    t -= getnanotime();
    /* code section to measure */
    t += getnanotime();
    /* ignore */
  }
  trace_performance(t, "frotz");

Hmph. Even though trace_performance() makes an extra call to getnanotime() in order to return, examples do not use the returned value? The second example is a good illustration why it makes sense for trace_performance_since(), though.

trace.c void trace_strbuf_fl(const char *file, int line, const char *key, · line 161

Quoted 4 lines

trace.c +4 -0

+static void trace_performance_vfl(const char *file, int line,
+				      uint64_t nanos, const char *format,
+				      va_list ap)
+{

Just being curious, but what does "v" stand for?

The "_fl" suffix you gave to printf family made a lot of sense to me, and I would have expected that this to be called with a "_flt" suffix, meaning "we have file, line and then time as extra parameters" or something.

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #15 lore raw

Re: [PATCH v5 10/11] trace: add trace_performance facility to debug performance issues

Am 17.06.2014 19:11, schrieb Junio C Hamano:

Quoted 28 lines

Quoted 16 lines · Karsten Blees

Simple use case (measure one code section):
  uint64_t start = getnanotime();
  /* code section to measure */
  trace_performance_since(start, "foobar");
Medium use case (measure consecutive code sections):
  uint64_t start = getnanotime();
  /* first code section to measure */
  start = trace_performance_since(start, "first foobar");
  /* second code section to measure */
  trace_performance_since(start, "second foobar");
Complex use case (measure repetitive code sections):

Quoted 9 lines

  uint64_t t = 0;
  for (;;) {
    /* ignore */
    t -= getnanotime();
    /* code section to measure */
    t += getnanotime();
    /* ignore */
  }
  trace_performance(t, "frotz");

Quoted 5 lines

Hmph. Even though trace_performance() makes an extra call to getnanotime() in order to return, examples do not use the returned value? The second example is a good illustration why it makes sense for trace_performance_since(), though.

Right, it makes no sense for trace_performance(), and for trace_performance_since() only if followed by another 'measured' code section. In that special case, I think it wouldn't hurt if you had to write:

  uint64_t start = getnanotime();
  /* first code section to measure */
  trace_performance_since(start, "first foobar");

  start = getnanotime();
  /* second code section to measure */
  trace_performance_since(start, "second foobar");

So I guess I'll drop the return value (and the second example, which is then redundant to the first).

Quoted 7 lines

trace.c void trace_strbuf_fl(const char *file, int line, const char *key, · line 161
Quoted 4 lines
trace.c +4 -0
+static void trace_performance_vfl(const char *file, int line,
+				      uint64_t nanos, const char *format,
+				      va_list ap)
+{
Just being curious, but what does "v" stand for?

trace_performance_vfl(, va_list)
vs.
trace_performance_fl(, ...)

Will change to trace_performance_vprintf_fl()

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Junio C Hamano

2016-06-15 23:01 UTC #16 lore raw

Re: [PATCH v5 10/11] trace: add trace_performance facility to debug performance issues

Quoted 13 lines · Karsten Blees

Right, it makes no sense for trace_performance(), and for trace_performance_since() only if followed by another 'measured' code section. In that special case, I think it wouldn't hurt if you had to write:
  uint64_t start = getnanotime();
  /* first code section to measure */
  trace_performance_since(start, "first foobar");

  start = getnanotime();
  /* second code section to measure */
  trace_performance_since(start, "second foobar");

Quoted 2 lines

So I guess I'll drop the return value (and the second example, which is then redundant to the first).

That also sounds OK to me.

Quoted 8 lines

Quoted 7 lines
trace.c void trace_strbuf_fl(const char *file, int line, const char *key, · line 161
Quoted 4 lines
trace.c +4 -0
+static void trace_performance_vfl(const char *file, int line,
+				      uint64_t nanos, const char *format,
+				      va_list ap)
+{
Just being curious, but what does "v" stand for?

Quoted 5 lines

trace_performance_vfl(, va_list)
vs.
trace_performance_fl(, ...)
Will change to trace_performance_vprintf_fl()

Ah, OK. The name with 'vprintf' in it does sound better.

Thanks.

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #17 lore raw

PATCH v5 11/11 git: add performance tracing for git's main() function to debug scripts

In reply to Karsten Blees

Use trace_performance to measure and print execution time and command line arguments of the entire main() function. In constrast to the shell's 'time' utility, which measures total time of the parent process, this logs all involved git commands recursively. This is particularly useful to debug performance issues of scripted commands (i.e. which git commands were called with which parameters, and how long did they execute).

Due to git's deliberate use of exit(), the implementation uses an atexit routine rather than just adding trace_performance_since() at the end of main().

Usage example: > GIT_TRACE_PERFORMANCE=~/git-trace.log git stash list

Creates a log file like this:
23:57:38.638765 trace.c:405 performance: 0.000310107 s: git command: 'git' 'rev-parse' '--git-dir' 23:57:38.644387 trace.c:405 performance: 0.000261759 s: git command: 'git' 'rev-parse' '--show-toplevel' 23:57:38.646207 trace.c:405 performance: 0.000304468 s: git command: 'git' 'config' '--get-colorbool' 'color.interactive' 23:57:38.648491 trace.c:405 performance: 0.000241667 s: git command: 'git' 'config' '--get-color' 'color.interactive.help' 'red bold' 23:57:38.650465 trace.c:405 performance: 0.000243063 s: git command: 'git' 'config' '--get-color' '' 'reset' 23:57:38.654850 trace.c:405 performance: 0.025126313 s: git command: 'git' 'stash' 'list'

Signed-off-by Karsten Blees <[email protected]>

---
 git.c   |  2 ++
 trace.c | 22 ++++++++++++++++++++++
 trace.h |  1 +
 3 files changed, 25 insertions(+)

git.c +2 -0

index 7780572..d4daeb8 100644
@@ -568,6 +568,8 @@ int main(int argc, char **av)
 
 	git_setup_gettext();
 
+	trace_command_performance(argv);
+
 	/*
 	 * "git-xxxx" is the same as "git xxxx", but we obviously:
 	 *

trace.c +22 -0

index 0551509..9fc3921 100644
@@ -395,3 +395,25 @@ inline uint64_t getnanotime(void)
 		return now;
 	}
 }
+
+static uint64_t command_start_time;
+static struct strbuf command_line = STRBUF_INIT;
+
+static void print_command_performance_atexit(void)
+{
+	trace_performance_since(command_start_time, "git command:%s",
+				command_line.buf);
+}
+
+void trace_command_performance(const char **argv)
+{
+	if (!trace_want(GIT_TRACE_PERFORMANCE))
+		return;
+
+	if (!command_start_time)
+		atexit(print_command_performance_atexit);
+
+	strbuf_reset(&command_line);
+	sq_quote_argv(&command_line, argv, 0);
+	command_start_time = getnanotime();
+}

trace.h +1 -0

index 9687563..a51e731 100644
@@ -7,6 +7,7 @@
 extern void trace_repo_setup(const char *prefix);
 extern int trace_want(const char *key);
 extern uint64_t getnanotime(void);
+extern void trace_command_performance(const char **argv);
 
 #ifndef HAVE_VARIADIC_MACROS

Signature · 21 lines

1.9.2.msysgit.0.501.gaeecf09

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Junio C Hamano

2016-06-15 23:01 UTC #18 lore raw

In reply to Karsten Blees

Quoted 9 lines · Karsten Blees

Here's v5 of the performance tracing patch series, now including a bunch of cleanups and adding timestamp, file and line to all trace output.
I'm particularly interested in feedback for the output format. As file names have different lengths, printing file:line as prefix results in unaligned output:
Quoted 1 line
GIT_TRACE=1 git stash list
00:12:10.544266 git.c:512 trace: exec: 'git-stash' 'list' 00:12:10.544266 run-command.c:337 trace: run_command: 'git-stash' 'list' 00:12:10.649779 git.c:312 trace: built-in: git 'rev-parse' '--git-dir'

Quoted 6 lines

We could add separators to make it easier to parse, e.g.:
Quoted 1 line
GIT_TRACE=1 git stash list
[00:12:10.544266 git.c:512] trace: exec: 'git-stash' 'list' [00:12:10.544266 run-command.c:337] trace: run_command: 'git-stash' 'list' [00:12:10.649779 git.c:312] trace: built-in: git 'rev-parse' '--git-dir'

This is easier to parse if " " and ":" are found in the names of _our_ source files and "]" isn't, but is that really the case?

Quoted 4 lines

Or print file:line at the end (but what about multi-line messages, such as packet-trace?):
Quoted 1 line
GIT_TRACE=1 git stash list
00:12:10.544266 trace: exec: 'git-stash' 'list' (git.c:512)

Somehow I find this a lot harder to read than the other two.

I am not commenting on the usefulness of the patch series (yet), though.

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #19 lore raw

Am 12.06.2014 20:30, schrieb Junio C Hamano:

Quoted 13 lines

Quoted 11 lines · Karsten Blees
Here's v5 of the performance tracing patch series, now including a bunch of cleanups and adding timestamp, file and line to all trace output.
I'm particularly interested in feedback for the output format. As file names have different lengths, printing file:line as prefix results in unaligned output:
Quoted 1 line
GIT_TRACE=1 git stash list
00:12:10.544266 git.c:512 trace: exec: 'git-stash' 'list' 00:12:10.544266 run-command.c:337 trace: run_command: 'git-stash' 'list' 00:12:10.649779 git.c:312 trace: built-in: git 'rev-parse' '--git-dir'
We could add separators to make it easier to parse, e.g.:

Quoted 8 lines

Quoted 4 lines
Quoted 1 line
GIT_TRACE=1 git stash list
[00:12:10.544266 git.c:512] trace: exec: 'git-stash' 'list' [00:12:10.544266 run-command.c:337] trace: run_command: 'git-stash' 'list' [00:12:10.649779 git.c:312] trace: built-in: git 'rev-parse' '--git-dir'
This is easier to parse if " " and ":" are found in the names of _our_ source files and "]" isn't, but is that really the case?

By "parsing" I actually meant the HumanEye (tm) parser, not lex/yacc and friends ("[]" just make nice recognizable separators).

However, I think it shouldn't be too complicated to properly align the output, at least for the majority of 'short' file names in the git code base. E.g.:

00:12:10.544266 git.c:512                trace: exec: 'git-stash' 'list'
00:12:10.544266 run-command.c:337        trace: run_command: 'git-stash' 'list'
00:12:10.649779 git.c:312                trace: built-in: git 'rev-parse' '--git-dir'

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Duy Nguyen

2016-06-15 23:01 UTC #20 lore raw

In reply to Karsten Blees

Quoted 8 lines · Karsten Blees

Here's v5 of the performance tracing patch series, now including a bunch of cleanups and adding timestamp, file and line to all trace output.
I'm particularly interested in feedback for the output format. As file names have different lengths, printing file:line as prefix results in unaligned output:
Quoted 1 line
GIT_TRACE=1 git stash list
00:12:10.544266 git.c:512 trace: exec: 'git-stash' 'list' 00:12:10.544266 run-command.c:337 trace: run_command: 'git-stash' 'list' 00:12:10.649779 git.c:312 trace: built-in: git 'rev-parse' '--git-dir'

Can I have an (build-time) option to show <function>:<line> instead of <file>:<line>? I know it's not supported by all compilers, which may make support a bit cumbersome..

Signature · 21 lines

Duy

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Karsten Blees

author

2016-06-15 23:01 UTC #21 lore raw

Am 25.06.2014 16:28, schrieb Duy Nguyen:

Quoted 5 lines · Karsten Blees

Quoted 4 lines
Here's v5 of the performance tracing patch series, now including a bunch of cleanups and adding timestamp, file and line to all trace output.
I'm particularly interested in feedback for the output format. As file names have different lengths, printing file:line as prefix results in unaligned output:

Quoted 9 lines

Quoted 4 lines
Quoted 1 line
GIT_TRACE=1 git stash list
00:12:10.544266 git.c:512 trace: exec: 'git-stash' 'list' 00:12:10.544266 run-command.c:337 trace: run_command: 'git-stash' 'list' 00:12:10.649779 git.c:312 trace: built-in: git 'rev-parse' '--git-dir'
Can I have an (build-time) option to show <function>:<line> instead of <file>:<line>? I know it's not supported by all compilers, which may make support a bit cumbersome..

Is this really useful? <file>:<line> is unique, but <function>:<line> is not. E.g. in case of "hash_name:47" you'd have to guess if its the one in attr.c or name-hash.c...

Signature · 18 lines

-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Duy Nguyen

2016-06-15 23:01 UTC #22 lore raw

Quoted 11 lines · Karsten Blees

Am 25.06.2014 16:28, schrieb Duy Nguyen:
Quoted 10 lines · Karsten Blees
Quoted 8 lines
Here's v5 of the performance tracing patch series, now including a bunch of cleanups and adding timestamp, file and line to all trace output.
I'm particularly interested in feedback for the output format. As file names have different lengths, printing file:line as prefix results in unaligned output:
Quoted 1 line
GIT_TRACE=1 git stash list
00:12:10.544266 git.c:512 trace: exec: 'git-stash' 'list' 00:12:10.544266 run-command.c:337 trace: run_command: 'git-stash' 'list' 00:12:10.649779 git.c:312 trace: built-in: git 'rev-parse' '--git-dir'

Quoted 6 lines

Quoted 4 lines
Can I have an (build-time) option to show <function>:<line> instead of <file>:<line>? I know it's not supported by all compilers, which may make support a bit cumbersome..
Is this really useful? <file>:<line> is unique, but <function>:<line> is not. E.g. in case of "hash_name:47" you'd have to guess if its the one in attr.c or name-hash.c...

It depends on your view. If I'm tracing a certain operation, function names let me know roughly what's going on without looking at the code because all (or many of) the names and their purposes are already in my mind. But for publishing the traces, then I agree <file>:<line> is better.

Signature · 21 lines

Duy

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.