xen: debugtrace cleanup and per-cpu buffer support
v7 cover xen: debugtrace cleanup and per-cpu buffer support
Another debugtrace enhancement I needed for core scheduling debugging, plus some cleanup.
Changes in V7:
- some comments by Jan addressed
Changes in V6:
- some comments by Jan addressed
- added patch 5
Changes in V5:
- several comments by Jan addressed (code: patches 1 and 4, commit message of patch 3)
Changes in V4:
- replaced patch 1 (original one was committed, new one requested by Jan Beulich)
- several comments by Jan addressed
Changes in V3:
- rebase to current staging
Changes in V2:
- added new patch 1 (preparing the move of debugtrace coding)
- patch 4 (v1 patch 3): avoid leaking buffer
Juergen Gross (5):
xen: fix debugtrace clearing
xen: move debugtrace coding to common/debugtrace.c
xen: refactor debugtrace data
xen: add per-cpu buffer option to debugtrace
xen: add debugtrace entry when entry count is wrapping
docs/misc/xen-command-line.pandoc | 7 +- xen/common/Makefile | 1 + xen/common/debugtrace.c | 303 ++++++++++++++++++++++++++++++++++++++ xen/drivers/char/console.c | 178 +--------------------- 4 files changed, 309 insertions(+), 180 deletions(-) create mode 100644 xen/common/debugtrace.c
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 1/5 xen: fix debugtrace clearing
After dumping the debugtrace buffer it is cleared. This results in some entries not being printed in case the buffer is dumped again before having wrapped.
While at it remove the trailing zero byte in the buffer as it is no longer needed. Commit b5e6e1ee8da59f introduced passing the number of chars to be printed in the related interfaces, so the trailing 0 byte is no longer required.
---
V5:
- invalidate last_buf instead of last_prd after printing the buffer (Jan Beulich)
V6:
- instead of moving last_buf to file scope use a boolean for buffer invalidation (Jan Beulich)
V7:
- init debugtrace_buf_empty as true (Jan Beulich)
--- xen/drivers/char/console.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
xen/drivers/char/console.c +7 -7
@@ -1173,6 +1173,7 @@ static char *debugtrace_buf; /* Debug-trace buffer */static unsigned int debugtrace_prd;/* Producer index */ static unsigned int debugtrace_kilobytes =128 , debugtrace_bytes;static unsigned int debugtrace_used; +static bool debugtrace_buf_empty = true;static DEFINE_SPINLOCK (debugtrace_lock);integer_param ("debugtrace" , debugtrace_kilobytes);
@@ -1184,16 +1185,17 @@ static void debugtrace_dump_worker(void)printk ("debugtrace_dump() starting\n" );/* Print oldest portion of the ring. */ -ASSERT (debugtrace_buf[debugtrace_bytes -1 ] ==0 );if ( debugtrace_buf[debugtrace_prd] !='\0' )console_serial_puts (&debugtrace_buf[debugtrace_prd], - debugtrace_bytes - debugtrace_prd -1 ); + debugtrace_bytes - debugtrace_prd);/* Print youngest portion of the ring. */ debugtrace_buf[debugtrace_prd] ='\0' ;console_serial_puts (&debugtrace_buf[0 ], debugtrace_prd);memset (debugtrace_buf,'\0' , debugtrace_bytes); + debugtrace_prd =0 ; + debugtrace_buf_empty = true;printk ("debugtrace_dump() finished\n" ); }
@@ -1241,8 +1243,7 @@ static void debugtrace_add_to_buf(char *buf)for ( p = buf; *p !='\0' ; p++ ) { debugtrace_buf[debugtrace_prd++] = *p; -/* Always leave a nul byte at the end of the buffer. */ -if ( debugtrace_prd == (debugtrace_bytes -1 ) ) +if ( debugtrace_prd == debugtrace_bytes ) debugtrace_prd =0 ; } }
@@ -1264,8 +1265,6 @@ void debugtrace_printk(const char *fmt, ...)spin_lock_irqsave (&debugtrace_lock, flags); -ASSERT (debugtrace_buf[debugtrace_bytes -1 ] ==0 ); -va_start (args, fmt); nr =vscnprintf (buf,sizeof (buf), fmt, args);va_end (args);
@@ -1279,8 +1278,9 @@ void debugtrace_printk(const char *fmt, ...) }else { -if (strcmp (buf, last_buf) ) +if ( debugtrace_buf_empty ||strcmp (buf, last_buf) ) { + debugtrace_buf_empty = false; last_prd = debugtrace_prd; last_count = ++count;safe_strcpy (last_buf, buf);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 2/5 xen: move debugtrace coding to common/debugtrace.c
In reply to Juergen GrossInstead of living in drivers/char/console.c move the debugtrace related coding to a new file common/debugtrace.c
No functional change, code movement only.
--- xen/common/Makefile | 1 + xen/common/debugtrace.c | 180 +++++++++++++++++++++++++++++++++++++++++++++ xen/drivers/char/console.c | 178 +------------------------------------------- 3 files changed, 182 insertions(+), 177 deletions(-) create mode 100644 xen/common/debugtrace.c
xen/common/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-y += bsearch.o obj-$(CONFIG_CORE_PARKING) += core_parking.o obj-y += cpu.o obj-y += cpupool.o +obj-$(CONFIG_DEBUG_TRACE) += debugtrace.o obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o obj-y += domctl.o obj-y += domain.o
xen/common/debugtrace.c new file +180 -0
180 lines not on this page · raw message
xen/drivers/char/console.c +1 -177
184 lines not on this page · raw message
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 3/5 xen: refactor debugtrace data
In reply to Juergen GrossAs a preparation for per-cpu buffers do a little refactoring of the debugtrace data: put the needed buffer admin data into the buffer as it will be needed for each buffer. In order not to limit buffer size switch the related fields from unsigned int to unsigned long, as on huge machines with RAM in the TB range it might be interesting to support buffers >4GB.
While at it switch debugtrace_send_to_console and debugtrace_used to bool and delete an empty line.
---
V4:
- renamed struct debugtrace_data_s (Jan Beulich)
- renamed debtr_data (Jan Beulich)
- remove unneeded condition (Jan Beulich)
- recalc debugtrace_bytes (Jan Beulich)
V6:
- undo putting bytes into struct debugtrace_data (Jan Beulich)
--- xen/common/debugtrace.c | 70 ++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 30 deletions(-)
xen/common/debugtrace.c +40 -30
@@ -15,34 +15,40 @@#include <xen/watchdog.h> /* Send output direct to console, or buffer it? */ -static volatile int debugtrace_send_to_console; +static volatile bool debugtrace_send_to_console; -static char *debugtrace_buf;/* Debug-trace buffer */ -static unsigned int debugtrace_prd;/* Producer index */ -static unsigned int debugtrace_kilobytes =128 , debugtrace_bytes; -static unsigned int debugtrace_used; +struct debugtrace_data { +unsigned long prd;/* Producer index. */ +char buf[]; +}; + +static struct debugtrace_data *dt_data; + +static unsigned int debugtrace_kilobytes =128 ; +static unsigned long debugtrace_bytes;static bool debugtrace_buf_empty = true; +static bool debugtrace_used;static DEFINE_SPINLOCK (debugtrace_lock);integer_param ("debugtrace" , debugtrace_kilobytes);static void debugtrace_dump_worker (void ) { -if ( (debugtrace_bytes ==0 ) || !debugtrace_used ) +if ( !debugtrace_used )return ;printk ("debugtrace_dump() starting\n" );/* Print oldest portion of the ring. */ -if ( debugtrace_buf[debugtrace_prd] !='\0' ) -console_serial_puts (&debugtrace_buf[debugtrace_prd], - debugtrace_bytes - debugtrace_prd); +if ( dt_data->buf[dt_data->prd] !='\0' ) +console_serial_puts (&dt_data->buf[dt_data->prd], + debugtrace_bytes - dt_data->prd);/* Print youngest portion of the ring. */ - debugtrace_buf[debugtrace_prd] ='\0' ; -console_serial_puts (&debugtrace_buf[0 ], debugtrace_prd); + dt_data->buf[dt_data->prd] ='\0' ; +console_serial_puts (&dt_data->buf[0 ], dt_data->prd); -memset (debugtrace_buf,'\0' , debugtrace_bytes); - debugtrace_prd =0 ; +memset (dt_data->buf,'\0' , debugtrace_bytes); + dt_data->prd =0 ; debugtrace_buf_empty = true;printk ("debugtrace_dump() finished\n" );
@@ -68,7 +74,6 @@ static void debugtrace_toggle(void)spin_unlock_irqrestore (&debugtrace_lock, flags);watchdog_enable (); - }void debugtrace_dump (void )
@@ -90,26 +95,27 @@ static void debugtrace_add_to_buf(char *buf)for ( p = buf; *p !='\0' ; p++ ) { - debugtrace_buf[debugtrace_prd++] = *p; -if ( debugtrace_prd == debugtrace_bytes ) - debugtrace_prd =0 ; + dt_data->buf[dt_data->prd++] = *p; +if ( dt_data->prd == debugtrace_bytes ) + dt_data->prd =0 ; } }void debugtrace_printk (const char *fmt, ...) {static char buf[1024 ], last_buf[1024 ]; -static unsigned int count, last_count, last_prd; +static unsigned int count, last_count; +static unsigned long last_prd;char cntbuf[24 ];va_list args;unsigned long flags;unsigned int nr; -if ( debugtrace_bytes ==0 ) +if ( !dt_data )return ; - debugtrace_used =1 ; + debugtrace_used = true;spin_lock_irqsave (&debugtrace_lock, flags);
@@ -129,14 +135,14 @@ void debugtrace_printk(const char *fmt, ...)if ( debugtrace_buf_empty ||strcmp (buf, last_buf) ) { debugtrace_buf_empty = false; - last_prd = debugtrace_prd; + last_prd = dt_data->prd; last_count = ++count;safe_strcpy (last_buf, buf);snprintf (cntbuf,sizeof (cntbuf),"%u " , count); }else { - debugtrace_prd = last_prd; + dt_data->prd = last_prd;snprintf (cntbuf,sizeof (cntbuf),"%u-%u " , last_count, ++count); }debugtrace_add_to_buf (cntbuf);
@@ -154,23 +160,27 @@ static void debugtrace_key(unsigned char key)static int __init debugtrace_init (void ) {int order; -unsigned int kbytes, bytes; +unsigned long kbytes; +struct debugtrace_data *data;/* Round size down to next power of two. */ while ( (kbytes = (debugtrace_kilobytes & (debugtrace_kilobytes-1 ))) !=0 ) debugtrace_kilobytes = kbytes; - bytes = debugtrace_kilobytes <<10 ; -if ( bytes ==0 ) + debugtrace_bytes = debugtrace_kilobytes <<10 ; +if ( debugtrace_bytes ==0 )return 0 ; - order =get_order_from_bytes (bytes); - debugtrace_buf =alloc_xenheap_pages (order,0 ); -ASSERT (debugtrace_buf !=NULL ); + order =get_order_from_bytes (debugtrace_bytes); + data =alloc_xenheap_pages (order,0 ); +if ( !data ) +return -ENOMEM ; -memset (debugtrace_buf,'\0' , bytes); + debugtrace_bytes =PAGE_SIZE << order; +memset (data,'\0' , debugtrace_bytes); + debugtrace_bytes -=sizeof (*data); - debugtrace_bytes = bytes; + dt_data = data;register_keyhandler ('T' , debugtrace_key,"toggle debugtrace to console/buffer" ,0 );
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 4/5 xen: add per-cpu buffer option to debugtrace
In reply to Juergen Grossdebugtrace is normally writing trace entries into a single trace buffer. There are cases where this is not optimal, e.g. when hunting a bug which requires writing lots of trace entries and one cpu is stuck. This will result in other cpus filling the trace buffer and finally overwriting the interesting trace entries of the hanging cpu.
In order to be able to debug such situations add the capability to use per-cpu trace buffers. This can be selected by specifying the debugtrace boot parameter with the modifier "cpu:", like:
debugtrace=cpu:16
At the same time switch the parsing function to accept size modifiers (e.g. 4M or 1G).
Printing out the trace entries is done for each buffer in order to minimize the effort needed during printing. As each entry is prefixed with its sequence number sorting the entries can easily be done when analyzing them.
---
V2:
- only allocate buffer if not already done so
V4:
- unsigned int -> unsigned long (Jan Beulich)
- replace check for bytes < PAGE_SIZE by !bytes (Jan Beulich)
- print info which buffer allocation failed (Jan Beulich)
- replace switch by if in cpu notifier handler (Jan Beulich) V5:
- don't silently ignore trailing characters when parsing buffer size (Jan Beulich)
- limit scope of some variables (Jan Beulich)
- adjust error message format (Jan Beulich)
V6:
- move calculation of final debugtrace_bytes value to parameter parsing
--- docs/misc/xen-command-line.pandoc | 7 +- xen/common/debugtrace.c | 160 +++++++++++++++++++++++++++++--------- 2 files changed, 128 insertions(+), 39 deletions(-)
docs/misc/xen-command-line.pandoc +4 -3
@@ -644,12 +644,13 @@ over the PCI busses sequentially) or by PCI device (must be on segment 0). Limits the number lines printed in Xen stack traces. ### debugtrace -> `= <integer>` +> `= [cpu:]<size>` > Default: `128` -Specify the size of the console debug trace buffer in KiB. The debug -trace feature is only enabled in debugging builds of Xen. +Specify the size of the console debug trace buffer. By specifying `cpu:` +additionally a trace buffer of the specified size is allocated per cpu. +The debug trace feature is only enabled in debugging builds of Xen. ### dma_bits > `= <integer>`
xen/common/debugtrace.c +124 -36
244 lines not on this page · raw message
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v7 4/5] xen: add per-cpu buffer option to debugtrace
Quoted 15 lines · Juergen Gross
debugtrace is normally writing trace entries into a single trace buffer. There are cases where this is not optimal, e.g. when hunting a bug which requires writing lots of trace entries and one cpu is stuck. This will result in other cpus filling the trace buffer and finally overwriting the interesting trace entries of the hanging cpu.
In order to be able to debug such situations add the capability to use per-cpu trace buffers. This can be selected by specifying the debugtrace boot parameter with the modifier "cpu:", like:
debugtrace=cpu:16
At the same time switch the parsing function to accept size modifiers (e.g. 4M or 1G).
Quoted 6 lines
Printing out the trace entries is done for each buffer in order to minimize the effort needed during printing. As each entry is prefixed with its sequence number sorting the entries can easily be done when analyzing them.
Signed-off-by Juergen Gross <[email protected]>
albeit preferably with the __read_mostly added as indicated in reply to v6 (can still be done while committing).
Quoted 7 lines
---
V2:
- only allocate buffer if not already done so
V4:
- unsigned int -> unsigned long (Jan Beulich)
- replace check for bytes <PAGE_SIZEby !bytes (Jan Beulich)
- print info which buffer allocation failed (Jan Beulich)
Quoted 8 lines
- replace switch by if in cpu notifier handler (Jan Beulich) V5:
- don't silently ignore trailing characters when parsing buffer size (Jan Beulich)
- limit scope of some variables (Jan Beulich)
- adjust error message format (Jan Beulich)
V6:
- move calculation of finaldebugtrace_bytesvalue to parameter parsing
V7: ...
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v7 5/5 xen: add debugtrace entry when entry count is wrapping
In reply to Juergen GrossThe debugtrace entry count is a 32 bit variable, so it can wrap when lots of trace entries are being produced. Making it wider would result in a waste of buffer space as the printed count value would consume more bytes when not wrapping.
So instead of letting the count value grow to huge values let it wrap and add a wrap counter printed in this situation. This will keep the needed buffer space at today's value while avoiding to loose a way to sort all entries in case multiple trace buffers are involved.
Note that the wrap message will be printed before the first trace entry in case output is switched to console early. This is on purpose in order to enable a future support of debugtrace to console without any allocated buffer.
--- xen/common/debugtrace.c | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-)
xen/common/debugtrace.c +29 -4
@@ -15,11 +15,14 @@#include <xen/spinlock.h> #include <xen/watchdog.h> +#define DEBUGTRACE_COUNT_WRAP 99999999 +/* Send output direct to console, or buffer it? */ static volatile bool debugtrace_send_to_console;struct debugtrace_data {unsigned long prd;/* Producer index. */ +unsigned long wrap_cnt;char buf[]; };
@@ -72,6 +75,7 @@ static void debugtrace_dump_buffer(struct debugtrace_data *data,/* Print youngest portion of the ring. */ data->buf[data->prd] ='\0' ;console_serial_puts (&data->buf[0 ], data->prd); +printk ("wrap: %lu\n" , data->wrap_cnt);memset (data->buf,'\0' , debugtrace_bytes); data->prd =0 ;
@@ -153,9 +157,9 @@ void debugtrace_printk(const char *fmt, ...) {static char buf[1024 ], last_buf[1024 ];static unsigned int count, last_count, last_cpu; -static unsigned long last_prd; +static unsigned long last_prd, wrap_cnt; -char cntbuf[24 ]; +char cntbuf[50 ];va_list args;unsigned long flags;unsigned int nr;
@@ -173,10 +177,23 @@ void debugtrace_printk(const char *fmt, ...) nr =vsnprintf (buf,sizeof (buf), fmt, args);va_end (args); +if ( count ==DEBUGTRACE_COUNT_WRAP ) + { + count =0 ; + wrap_cnt++; + } +if ( debugtrace_send_to_console ) { -unsigned int n =scnprintf (cntbuf,sizeof (cntbuf),"%u " , ++count); +unsigned int n; + +if ( count ==0 ) + { + n =scnprintf (cntbuf,sizeof (cntbuf),"wrap: %lu\n" , wrap_cnt); +console_serial_puts (cntbuf, n); + } + n =scnprintf (cntbuf,sizeof (cntbuf),"%u " , ++count);console_serial_puts (cntbuf, n);console_serial_puts (buf, nr); }
@@ -184,8 +201,16 @@ void debugtrace_printk(const char *fmt, ...) {unsigned int cpu = debugtrace_per_cpu ?smp_processor_id () :0 ; -if ( debugtrace_buf_empty || cpu != last_cpu ||strcmp (buf, last_buf) ) +if ( debugtrace_buf_empty || cpu != last_cpu || + wrap_cnt != data->wrap_cnt ||strcmp (buf, last_buf) ) { +if ( wrap_cnt != data->wrap_cnt ) + { +snprintf (cntbuf,sizeof (cntbuf),"wrap: %lu->%lu\n" , + data->wrap_cnt, wrap_cnt); +debugtrace_add_to_buf (cntbuf); + data->wrap_cnt = wrap_cnt; + } debugtrace_buf_empty = false; last_prd = data->prd; last_count = ++count;
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v7 5/5] xen: add debugtrace entry when entry count is wrapping
Quoted 10 lines · Juergen Gross
The debugtrace entry count is a 32 bit variable, so it can wrap when lots of trace entries are being produced. Making it wider would result in a waste of buffer space as the printed count value would consume more bytes when not wrapping.
So instead of letting the count value grow to huge values let it wrap and add a wrap counter printed in this situation. This will keep the needed buffer space at today's value while avoiding to loose a way to sort all entries in case multiple trace buffers are involved.
Quoted 6 lines
Note that the wrap message will be printed before the first trace entry in case output is switched to console early. This is on purpose in order to enable a future support of debugtrace to console without any allocated buffer.
Signed-off-by Juergen Gross <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel