Trace events to pstore
Subsystems (5)
RFC PATCH 1/7 tracing: Add support to allocate pages from persistent memory
In reply to Nachammai KaruppiahAdd support in ring buffer to allocate pages from persistent RAM buffer. This feature supports switching to persistent memory and vice-versa. A new option 'persist' has been added and once this is enabled, the pages in ring buffer are freed up and new pages are allocated from persistent memory.
--- kernel/trace/Kconfig | 10 ++ kernel/trace/ring_buffer.c | 257 ++++++++++++++++++++++++++++++++++++++++++++- kernel/trace/trace.c | 12 ++- kernel/trace/trace.h | 3 +- 4 files changed, 279 insertions(+), 3 deletions(-)
kernel/trace/Kconfig +10 -0
@@ -739,6 +739,16 @@ config GCOV_PROFILE_FTRACE Note that on a kernel compiled with this config, ftrace will run significantly slower. +config TRACE_EVENTS_TO_PSTORE + bool "Enable users to store trace records in persistent storage" + default n + help + This option enables users to store trace records in a + persistent RAM buffer so that they can be retrieved after + system reboot. + + If unsure, say N. + config FTRACE_SELFTEST bool
kernel/trace/ring_buffer.c +256 -1
293 lines not on this page · raw message
kernel/trace/trace.c +11 -1
@@ -48,6 +48,7 @@#include <linux/fsnotify.h> #include <linux/irq_work.h> #include <linux/workqueue.h> +#include <linux/ramtrace.h> #include "trace.h" #include "trace_output.h"
@@ -265,7 +266,8 @@ unsigned long long ns2usecs(u64 nsec)/* trace_flags that are default zero for instances */ #define ZEROED_TRACE_FLAGS \ - (TRACE_ITER_EVENT_FORK | TRACE_ITER_FUNC_FORK) + (TRACE_ITER_EVENT_FORK | TRACE_ITER_FUNC_FORK | \ + TRACE_ITER_PERSIST)/* * The global_trace is the descriptor that holds the top-level tracing
@@ -4851,6 +4853,14 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)trace_printk_control (enabled); } +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (mask ==TRACE_ITER_PERSIST ) { +ring_buffer_switch_memory (tr->array_buffer.buffer, + tr->current_trace->name, + tr->clock_id, enabled); + } +#endif +return 0 ; }
kernel/trace/trace.h +2 -1
@@ -1336,7 +1336,8 @@ extern int trace_get_user(struct trace_parser *parser, const char __user *ubuf,FUNCTION_FLAGS \ FGRAPH_FLAGS \STACK_FLAGS \ -BRANCH_FLAGS + BRANCH_FLAGS \ + C(PERSIST, "persist"),/* * By defining C, we can make TRACE_FLAGS a list of bit names
Signature · 1 line
2.7.4
cover Trace events to pstore
Hi,
This patch series adds support to store trace events in pstore.
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
This requires a new backend - ramtrace that allocates pages from persistent storage for the tracing utility. This feature can be enabled using TRACE_EVENTS_TO_PSTORE.
In this feature, the new backend is used only as a page allocator and once the users chooses to use pstore to record trace entries, the ring buffer pages are freed and allocated in pstore. Once this switch is done, ring_buffer continues to operate just as before without much overhead. Since the ring buffer uses the persistent RAM buffer directly to record trace entries, all tracers would also persist across reboot.
To test this feature, I used a simple module that would call panic during a write operation to file in tracefs directory. Before writing to the file, the ring buffer is moved to persistent RAM buffer through command line as shown below,
$echo 1 > /sys/kernel/tracing/options/persist
Writing to the file,
$echo 1 > /sys/kernel/tracing/crash/panic_on_write
The above write operation results in system crash. After reboot, once the pstore is mounted, the trace entries from previous boot are available in file, /sys/fs/pstore/trace-ramtrace-0
Looking through this file, gives us the stack trace that led to the crash.
<...>-1 [001] .... 49.083909: __vfs_write <-vfs_write
<...>-1 [001] .... 49.083933: panic <-panic_on_write
<...>-1 [001] d... 49.084195: printk <-panic
<...>-1 [001] d... 49.084201: vprintk_func <-printk
<...>-1 [001] d... 49.084207: vprintk_default <-printk
<...>-1 [001] d... 49.084211: vprintk_emit <-printk
<...>-1 [001] d... 49.084216: __printk_safe_enter <-vprintk_emit
<...>-1 [001] d... 49.084219: _raw_spin_lock <-vprintk_emit
<...>-1 [001] d... 49.084223: vprintk_store <-vprintk_emit Patchwise oneline description is given below:
Patch 1 adds support to allocate ring buffer pages from persistent RAM buffer.
Patch 2 introduces a new backend, ramtrace.
Patch 3 adds methods to read previous boot pages from pstore.
Patch 4 adds the functionality to allocate page-sized memory from pstore.
Patch 5 adds the seq_operation methods to iterate through trace entries.
Patch 6 modifies ring_buffer to allocate from ramtrace when pstore is used.
Patch 7 adds ramtrace DT node as child-node of /reserved-memory.
Nachammai Karuppiah (7):
tracing: Add support to allocate pages from persistent memory
pstore: Support a new backend, ramtrace
pstore: Read and iterate through trace entries in PSTORE
pstore: Allocate and free page-sized memory in persistent RAM buffer
tracing: Add support to iterate through pages retrieved from pstore
tracing: Use ramtrace alloc and free methods while using persistent
RAM
dt-bindings: ramtrace: Add ramtrace DT node
.../bindings/reserved-memory/ramtrace.txt | 13 + drivers/of/platform.c | 1 + fs/pstore/Makefile | 2 + fs/pstore/inode.c | 46 +- fs/pstore/platform.c | 1 + fs/pstore/ramtrace.c | 821 +++++++++++++++++++++ include/linux/pstore.h | 3 + include/linux/ramtrace.h | 28 + include/linux/ring_buffer.h | 19 + include/linux/trace.h | 13 + kernel/trace/Kconfig | 10 + kernel/trace/ring_buffer.c | 663 ++++++++++++++++- kernel/trace/trace.c | 312 +++++++- kernel/trace/trace.h | 5 +- 14 files changed, 1924 insertions(+), 13 deletions(-) create mode 100644 Documentation/devicetree/bindings/reserved-memory/ramtrace.txt create mode 100644 fs/pstore/ramtrace.c create mode 100644 include/linux/ramtrace.h
Signature · 1 line
2.7.4
RFC PATCH 5/7 tracing: Add support to iterate through pages retrieved from pstore
Add a new trace_array, pstore_trace. This descriptor holds the top-level buffers used for managing the pages retrieved from persistent RAM. Since pstore_trace uses the pages that pertain to previous boot, there is no write that happens to these buffers. The reads are non-consuming and hence we do not have to serialize the readers.
The buffers in pstore_trace are disabled once the user switches live tracing to use persistent RAM buffer.
During the first seq_start method call to read the previous boot trace entries, the top-level buffers of pstore_trace are set up. The pages retrieved from pstore are used to construct cpu_buffer->pages for pstore_trace.
--- include/linux/ring_buffer.h | 19 +++ include/linux/trace.h | 13 ++ kernel/trace/ring_buffer.c | 284 +++++++++++++++++++++++++++++++++++++++++ kernel/trace/trace.c | 300 +++++++++++++++++++++++++++++++++++++++++++- kernel/trace/trace.h | 2 + 5 files changed, 616 insertions(+), 2 deletions(-)
include/linux/ring_buffer.h +19 -0
@@ -18,6 +18,13 @@ struct ring_buffer_event {u32 array[]; }; +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +struct data_page { +struct list_head list; +struct buffer_data_page *page; +}; +#endif +/** * enum ring_buffer_type - internal ring buffer types *
@@ -210,4 +217,16 @@ int trace_rb_cpu_prepare(unsigned int cpu, struct hlist_node *node);#define trace_rb_cpu_prepare NULL#endif +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +struct trace_buffer *reconstruct_ring_buffer (void ); + +void ring_buffer_order_pages (struct list_head *pages); +int ring_buffer_switch_memory (struct trace_buffer *buffer, +const char *tracer_name,int clock_id, +bool persist); +void ring_buffer_set_tracer_name (struct trace_buffer *buffer, +const char *tracer_name); +void ring_buffer_free_pstore_trace (struct trace_buffer *buffer); +#endif +#endif /* _LINUX_RING_BUFFER_H */
include/linux/trace.h +13 -0
@@ -32,6 +32,19 @@ int trace_array_printk(struct trace_array *tr, unsigned long ip,void trace_array_put (struct trace_array *tr);struct trace_array *trace_array_get_by_name (const char *name);int trace_array_destroy (struct trace_array *tr); + + +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +struct trace_iterator ; + +void *pstore_trace_start (struct seq_file *m,loff_t *pos); +void *pstore_trace_next (struct seq_file *m,void *v,loff_t *pos); +int pstore_trace_show (struct seq_file *m,void *v); +void pstore_trace_stop (struct seq_file *m,void *v); +int pstore_tracing_release (struct trace_iterator *iter); +void pstore_tracing_erase (void ); +#endif +#endif /* CONFIG_TRACING */ #endif /* _LINUX_TRACE_H */
kernel/trace/ring_buffer.c +284 -0
314 lines not on this page · raw message
kernel/trace/trace.c +298 -2
338 lines not on this page · raw message
kernel/trace/trace.h +2 -0
@@ -2078,4 +2078,6 @@ static __always_inline void trace_iterator_reset(struct trace_iterator *iter) iter->pos = -1; } +void pstore_tracing_off (void ); +#endif /* _LINUX_KERNEL_TRACE_H */
Signature · 1 line
2.7.4
RFC PATCH 2/7 pstore: Support a new backend, ramtrace
In reply to Nachammai Karuppiahramtrace provides persistent RAM storage for trace, so they can be recovered after a reboot. This is a child-node of "/reserved-memory", and is named "ramtrace" after the backend.
ramtrace supports allocation and deallocation of page-sized memory
This functionality is used by ring buffer in trace when the user switches to persistent storage. The ring buffer writes directly to the persistent RAM and doesn't use the write API provided by pstore. For these reasons, a new backend is needed.
Required properties:
- compatible: must be "ramtrace"
- reg: region of memory that is preserved between reboots
Also, can be enabled on kernel command line as, ramtrace.mem_address=0x100000000 ramtrace.mem_size=0x7ffffff
--- drivers/of/platform.c | 1 + fs/pstore/Makefile | 2 + fs/pstore/ramtrace.c | 346 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/pstore.h | 3 + include/linux/ramtrace.h | 15 ++ 5 files changed, 367 insertions(+) create mode 100644 fs/pstore/ramtrace.c create mode 100644 include/linux/ramtrace.h
drivers/of/platform.c +1 -0
@@ -511,6 +511,7 @@ static const struct of_device_id reserved_mem_matches[] = { { .compatible = "qcom,rmtfs-mem" }, { .compatible = "qcom,cmd-db" }, { .compatible = "ramoops" }, + { .compatile = "ramtrace" }, {} };
fs/pstore/Makefile +2 -0
@@ -18,3 +18,5 @@ obj-$(CONFIG_PSTORE_ZONE) += pstore_zone.o pstore_blk-objs += blk.o obj-$(CONFIG_PSTORE_BLK) += pstore_blk.o + +obj-$(CONFIG_TRACE_EVENTS_TO_PSTORE) += ramtrace.o
fs/pstore/ramtrace.c new file +346 -0
346 lines not on this page · raw message
include/linux/pstore.h +3 -0
@@ -39,6 +39,8 @@ enum pstore_type_id {PSTORE_TYPE_PMSG =7 ,PSTORE_TYPE_PPC_OPAL =8 , +PSTORE_TYPE_TRACE =9 , +/* End of the list */ PSTORE_TYPE_MAX };
@@ -202,6 +204,7 @@ struct pstore_info {#define PSTORE_FLAGS_CONSOLE BIT(1)#define PSTORE_FLAGS_FTRACE BIT(2)#define PSTORE_FLAGS_PMSG BIT(3) +#define PSTORE_FLAGS_TRACE BIT(4)extern int pstore_register (struct pstore_info *);extern void pstore_unregister (struct pstore_info *);
include/linux/ramtrace.h new file +15 -0
@@ -0,0 +1,15 @@ +#include <linux/list.h> +#include <linux/trace.h> +#include <linux/trace_events.h> +#include <linux/pstore.h> + +/* +* Ramoops platform data +* @mem_size memory size for ramtrace +* @mem_address physical memory address to contain ramtrace +*/ + +struct ramtrace_platform_data { +unsigned long mem_size; +phys_addr_t mem_address; +};
Signature · 1 line
2.7.4
RFC PATCH 7/7 dt-bindings: ramtrace: Add ramtrace DT node
In reply to Nachammai KaruppiahAdd ramtrace as child node of /reserved-memory.
--- .../devicetree/bindings/reserved-memory/ramtrace.txt | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 Documentation/devicetree/bindings/reserved-memory/ramtrace.txt
Documentation/devicetree/bindings/reserved-memory/ramtrace.txt new file +13 -0
@@ -0,0 +1,13 @@ +Ramtrace trace events logger +============================ + +ramtrace provides persistent RAM storage for ring buffer in which the trace +events are recorded, so they can be recovered after a reboot. This is a +child-node of "/reserved-memory", and is named "ramtrace" after the backend, +rather than "pstore" which is the subsystem. + +Required properties: + +- compatible: must be "ramtrace" + +- reg: region of memory that is preserved between reboots
Signature · 1 line
2.7.4
RFC PATCH 4/7 pstore: Allocate and free page-sized memory in persistent RAM buffer
In reply to Nachammai Karuppiahramtrace backend acts as a page allocator and manages the persistent RAM buffer.
ramtrace supports allocation and deallocation of page-sized memory through methods, ramtrace_alloc_page and ramtrace_free_page. This functionality is required by ring buffer in trace when the user switches to persistent storage.
Just prior to allocating pages for recording trace entries, ramtrace backend frees up the list used to maintain pages pertaining to previous boot. After this, reading previous boot trace entries from /sys/fs/pstore is disabled.
--- fs/pstore/ramtrace.c | 336 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/ramtrace.h | 13 ++ 2 files changed, 349 insertions(+)
fs/pstore/ramtrace.c +336 -0
@@ -19,6 +19,17 @@ module_param(mem_size, ulong, 0400);MODULE_PARM_DESC (mem_size,"size of reserved RAM used to store trace data" ); +struct ramtrace_pagelist { +struct list_head list; +void *page; +}; + +struct tr_persistent_info { +char *tracer_name; +int trace_clock; +unsigned int nr_cpus; +struct list_head **data_pages; +};struct ramtrace_context {phys_addr_t phys_addr;/* Physical address of the persistent memory */
@@ -37,6 +48,50 @@ struct ramtrace_context {int read_buffer_status; }; +/* +* The first page in the ramtrace area is the metadata page, followed by +* bitmap pages and then the buffer_data_page allocated by trace. +* Each bitmap page can represent upto page_size * 8 number of pages. +* The number of bitmaps needed per cpu is determined by the size of the +* pstore memory. Each CPU is allocated sufficient bitmap pages to represent +* the entire memory region. +* The figure below illustrates how the ramtrace memory area is organized. +* +* +------------------------------------------+ +* | metadata | +* +------------------------------------------+ +* | CPU 1 Bitmap 1 to buffer pages | +* +------------------------------------------+ +* | CPU 1 Bitmap 2 to buffer pages | +* +------------------------------------------+ +* | . . . | +* +------------------------------------------+ +* | CPU 1 Bitmap K to buffer pages | +* +------------------------------------------+ +* | CPU 2 Bitmap 1 to buffer pages | +* +------------------------------------------+ +* | CPU 2 Bitmap 2 to buffer pages | +* +------------------------------------------+ +* | . . . | +* +------------------------------------------+ +* | CPU 2 Bitmap K to buffer pages | +* +------------------------------------------+ +* | . . . . . . | +* +------------------------------------------+ +* | CPU N Bitmap K to buffer pages | +* +------------------------------------------+ +* | buffer_data_page 1 belonging to any CPU | +* +------------------------------------------+ +* | buffer_data_page 2 belonging to any CPU | +* +------------------------------------------+ +* | . . . | +* +------------------------------------------+ +* | buffer_data_page (K x 4096) | +* | belonging to any CPU | +* +------------------------------------------+ +*/ + +static void ramtrace_read_pages (void );static int ramtrace_pstore_open (struct pstore_info *psi);static ssize_t ramtrace_pstore_read (struct pstore_record *record);static int ramtrace_pstore_erase (struct pstore_record *record);
@@ -122,6 +177,227 @@ static int ramtrace_pstore_erase(struct pstore_record *record)static struct platform_device *dummy; +bool is_ramtrace_available (void ) +{ +return (trace_ctx.size >0 ) ?1 :0 ; +} + +int ramtrace_available_mem (void ) +{ +return trace_ctx.pages_available; +} + +/** +* ramtrace_init_bitmap: Initialize bitmap pages. +* +* This method allocates and initializes bitmap pages. +*/ +static void ramtrace_init_bitmap (unsigned int npages) +{ +int i; +unsigned long flags; +struct ramtrace_pagelist *freelist = trace_ctx.freelist; + + trace_ctx.bitmap_pages =kmalloc_array (npages,sizeof (void *), +GFP_KERNEL ); +spin_lock_irqsave (&trace_ctx.lock, flags); +for (i =0 ; i < npages; i++) { +struct ramtrace_pagelist *freelist_node; +void *page; + + freelist_node =list_next_entry (freelist, list); + page = freelist_node->page; +memset (page,0 ,PAGE_SIZE ); + trace_ctx.bitmap_pages[i] = page; +list_del (&freelist_node->list); +kfree (freelist_node); + } +spin_unlock_irqrestore (&trace_ctx.lock, flags); + trace_ctx.base_address = trace_ctx.bitmap_pages[npages -1 ] +PAGE_SIZE ; +} + + +static void ramtrace_write_int (int **buffer,int n) +{ + **buffer = n; + (*buffer)++; +} + +void ramtrace_set_clock_id (int clock_id) +{ + *(trace_ctx.clock_id) = clock_id; +} + +void ramtrace_set_tracer_name (const char *tracer_name) +{ +sprintf (trace_ctx.tracer_name,"%s" , tracer_name); +} + +/* +* init_ramtrace_pages: Initialize metadata page, bitmap and trace context. +* +* Below is the layout of the metadata page. +* +------------------------------------------+ +* | Kernel Version | +* +------------------------------------------+ +* | tracer_name | +* +------------------------------------------+ +* | Number of CPU’s Buffers = N | +* +------------------------------------------+ +* | trace_clock_name | +* +------------------------------------------+ +* | pages per cpu | +* +------------------------------------------+ +*/ + +int +init_ramtrace_pages (int cpu,unsigned long npages,const char *tracer_name, +int clock_id) +{ +const char kernel_version[] =UTS_RELEASE ; +struct ramtrace_pagelist *freelist_node; +void *metapage; +unsigned long flags; +int n_bitmap =0 ; +int ramtrace_pages; + + + ramtrace_pages = (trace_ctx.size >>PAGE_SHIFT ) -1 ; + +/* Calculate number of bitmap pages required for npages */ + n_bitmap = ramtrace_pages / ((PAGE_SIZE <<3 ) + cpu); + +if (ramtrace_pages % (PAGE_SIZE <<3 ) > cpu) + n_bitmap++; +if (ramtrace_pages - n_bitmap < npages) +return 1 ; + +spin_lock_irqsave (&trace_ctx.lock, flags); + freelist_node =list_next_entry (trace_ctx.freelist, list); + metapage = freelist_node->page; +list_del (&freelist_node->list); +spin_unlock_irqrestore (&trace_ctx.lock, flags); + +pstore_tracing_erase (); +free_persist_info (); + +/* Initialize metadata page */ +ramtrace_write_int ((int **)&metapage, cpu); + trace_ctx.clock_id = (int *)metapage; +ramtrace_write_int ((int **)&metapage, clock_id); +ramtrace_write_int ((int **)&metapage, n_bitmap); +sprintf (metapage,"%s" , kernel_version); + metapage +=strlen (kernel_version) +1 ; + trace_ctx.tracer_name = (char *)metapage; +sprintf (metapage,"%s" , tracer_name); + +kfree (freelist_node); + trace_ctx.cpu = cpu; + trace_ctx.num_bitmap_per_cpu = n_bitmap; + trace_ctx.pages_available = ramtrace_pages - n_bitmap; +ramtrace_init_bitmap (cpu * n_bitmap); +return 0 ; +} + +static void ramtrace_set_bit (char *bitmap,int index) +{ + bitmap[index >>3 ] |= (1 << index %8 ); +} + +static bool ramtrace_is_allocated (char *bitmap,int index) +{ +return bitmap[index >>3 ] & (1 << index %8 ); +} + +static void ramtrace_reset_bit (char *bitmap,int index) +{ + bitmap[index >>3 ] &= ~(1 << index %8 ); +} + + +void *ramtrace_alloc_page (int cpu) +{ +void *address =NULL ; +struct ramtrace_pagelist *freelist = trace_ctx.freelist; + +if (!list_empty (&freelist->list)) { +struct ramtrace_pagelist *freelist_node; +char *bitmap_page; +unsigned long page_num; +unsigned long flags; +int index, bitmap_page_index; + +/* Acquire lock and obtain a page from freelist */ +spin_lock_irqsave (&trace_ctx.lock, flags); + freelist_node =list_next_entry (freelist, list); +list_del (&freelist_node->list); + trace_ctx.pages_available--; +spin_unlock_irqrestore (&trace_ctx.lock, flags); + + address = freelist_node->page; +memset (address,0 ,PAGE_SIZE ); + +/* Determine the bitmap index for the allocated page */ + page_num = (address - trace_ctx.base_address) >>PAGE_SHIFT ; + +/* Every bitmap page represents PAGE_SIZE * 8 or +* 1 << (PAGE_SHIFT + 3) pages. Determine the nth bitmap for +* this cpu assosciated with the allocated page address. +*/ + bitmap_page_index = page_num >> (PAGE_SHIFT +3 ); + bitmap_page = trace_ctx.bitmap_pages[trace_ctx.num_bitmap_per_cpu * cpu + bitmap_page_index]; +/* Determine the index */ + index = page_num - (bitmap_page_index << (PAGE_SHIFT +3 )); + +ramtrace_set_bit (bitmap_page, index); + + } +return address; + +} + +void ramtrace_free_page (void *page_address,int cpu) +{ +void *bitmap; +int index; + +/* +* Determine the page number by calculating the offset from the base +* address and divide it by page size. +* Each bitmap can hold page_size * 8 indices. In case we have more +* than one bitmap per cpu, divide page_num by (page size * 8). +*/ +unsigned long page_num = (page_address - trace_ctx.base_address) >>PAGE_SHIFT ; +int bitmap_page_index = page_num >> (PAGE_SHIFT +3 ); + +if (page_address ==NULL ) +return ; + bitmap = (char *)(trace_ctx.bitmap_pages[trace_ctx.num_bitmap_per_cpu * cpu + bitmap_page_index]); +/* +* When a single bitmap per cpu is used, page_num gives the index +* in the bitmap. In case of multiple bitmaps per cpu, +* page_num - bitmap_page_index * page_size * 8 gives the index. +* Note: When page_num is less than (page_size * 8), bitmap_page_index +* is zero. +* */ + index = page_num - (bitmap_page_index << (PAGE_SHIFT +3 )); +if (ramtrace_is_allocated (bitmap, index)) { +struct ramtrace_pagelist *freelist_node = +kmalloc (sizeof (struct ramtrace_pagelist ),GFP_KERNEL ); +unsigned long flags; + + freelist_node->page = page_address; +spin_lock_irqsave (&trace_ctx.lock, flags); +list_add_tail (&freelist_node->list, &(trace_ctx.freelist->list)); + trace_ctx.pages_available++; +spin_unlock_irqrestore (&trace_ctx.lock, flags); + +ramtrace_reset_bit (bitmap, index); + } + +} + +static int ramtrace_parse_dt (struct platform_device *pdev,struct ramtrace_platform_data *pdata) {
@@ -232,6 +508,32 @@ ramtrace_read_bitmap(int n_cpu, int n_bitmap, struct list_head **per_cpu) } +struct list_head *ramtrace_get_read_buffer (int n_cpu) +{ +if (n_cpu >= (trace_ctx.persist_info)->nr_cpus) +return NULL ; + +return (trace_ctx.persist_info)->data_pages[n_cpu]; +} + +int ramtrace_get_prev_boot_nr_cpus (void ) +{ +return (trace_ctx.persist_info)->nr_cpus; +} + +int ramtrace_get_prev_boot_clock_id (void ) +{ +return (trace_ctx.persist_info)->trace_clock; +} + +char *ramtrace_get_prev_boot_tracer_name (void ) +{ +return (trace_ctx.persist_info)->tracer_name; +} + + + +static void ramtrace_read_pages (void ) {void *metapage = trace_ctx.vaddr;
@@ -270,6 +572,40 @@ static void ramtrace_read_pages(void) out: trace_ctx.persist_info = persist; } + +/** +* free_persist_info - free the list pertaining to previous boot. +* +* Free the list and array that was allocated to manage previous boot data. +* Note: There is no need to free the ramtrace pages memory area. +*/ +static void free_persist_info (void ) +{ +struct tr_persistent_info *persist; +int i; + + persist = trace_ctx.persist_info; + +if (persist) { +for (i =0 ; i < persist->nr_cpus; i++) { +struct ramtrace_pagelist *node, *tmp; +struct list_head *page_list = persist->data_pages[i]; + +if (page_list ==NULL ) +continue ; +list_for_each_entry_safe (node, tmp, page_list, list) { +list_del (&node->list); +kfree (node); + } +kfree (page_list); + } +kfree (persist->data_pages); +kfree (persist->tracer_name); +kfree (persist); + } + trace_ctx.persist_info =NULL ; +} +static int ramtrace_init_mem (struct ramtrace_context *ctx) {
include/linux/ramtrace.h +13 -0
@@ -13,3 +13,16 @@ struct ramtrace_platform_data {unsigned long mem_size;phys_addr_t mem_address; }; + +void *ramtrace_alloc_page (int cpu); +void ramtrace_free_page (void *address,int cpu); +void ramtrace_dump (void ); +int init_ramtrace_pages (int cpu,unsigned long npages, +const char *tracer_name,int clock_id); +bool is_ramtrace_available (void ); +struct list_head *ramtrace_get_read_buffer (int cpu); +char *ramtrace_get_prev_boot_tracer_name (void ); +int ramtrace_get_prev_boot_clock_id (void ); +int ramtrace_get_prev_boot_nr_cpus (void ); +int ramtrace_available_mem (void ); +void ramtrace_set_tracer_name (const char *tracer_name);
Signature · 1 line
2.7.4
RFC PATCH 3/7 pstore: Read and iterate through trace entries in PSTORE
In reply to Nachammai KaruppiahIncase of trace records that are being read from pstore, the numerical *pos is not enough to hold state from session to session. A trace iterator is better suited for this. So for PSTORE_TYPE_TRACE record types, the seq_file->private field holds pointer to trace_iterator and not pstore_private. For this reason, PSTORE_TYPE_TRACE requires a different file_operations implementation, pstore_trace_file_operations.
The method, ramtrace_read_pages initiates the retrieval of pages from pstore.
The first page in the pstore is the metadata page. Below is the layout of the metadata page.
+------------------------------------------+ | Kernel Version | +------------------------------------------+ | tracer_name | +------------------------------------------+ | Number of CPU’s Buffers = N | +------------------------------------------+ | trace_clock_name | +------------------------------------------+ | Number of bitmap pages per cpu | +------------------------------------------+
The metadata page is then followed by bitmap pages which is a bitmap of pages that are allocated per CPU. From the bitmaps, the list of pages per cpu is computed and ordered by timestamp.
--- fs/pstore/inode.c | 46 +++++++++++++++-- fs/pstore/platform.c | 1 + fs/pstore/ramtrace.c | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 181 insertions(+), 5 deletions(-)
fs/pstore/inode.c +41 -5
@@ -147,13 +147,20 @@ static int pstore_file_open(struct inode *inode, struct file *file)if (ps->record->type ==PSTORE_TYPE_FTRACE ) sops = &pstore_ftrace_seq_ops; +if (ps->record->type ==PSTORE_TYPE_TRACE ) + sops = ps->record->psi->data; + err =seq_open (file, sops);if (err <0 )return err; - - sf = file->private_data; - sf->private = ps; - +/* Incase of PSTORE_TYPE_TRACE, the private field in seq_file +* would be initialized later during seq_read and freed in +* seq_release. +*/ +if (ps->record->type !=PSTORE_TYPE_TRACE ) { + sf = file->private_data; + sf->private = ps; + }return 0 ; }
@@ -173,6 +180,30 @@ static const struct file_operations pstore_file_operations = { .release = seq_release, }; +static ssize_t pstore_trace_file_read (struct file *file,char __user *userbuf, +size_t count,loff_t *ppos) +{ +return seq_read (file, userbuf, count, ppos); +} + +extern int pstore_tracing_release (void *v); + +static int pstore_trace_file_release (struct inode *inode,struct file *file) +{ +struct seq_file *s = file->private_data; +void *v = s->private; + +pstore_tracing_release (v); +return seq_release (inode, file); +} + +static const struct file_operations pstore_trace_file_operations = { + .open = pstore_file_open, + .read = pstore_trace_file_read, + .llseek = pstore_file_llseek, + .release = pstore_trace_file_release, +}; +/* * When a file is unlinked from our file system we call the * platform driver to erase the record from persistent store.
@@ -369,7 +400,6 @@ int pstore_mkfile(struct dentry *root, struct pstore_record *record)if (!inode) goto fail; inode->i_mode =S_IFREG |0444 ; - inode->i_fop = &pstore_file_operations; scnprintf(name, sizeof(name), "%s-%s-%llu%s", pstore_type_to_name(record->type), record->psi->name, record->id,
@@ -386,6 +416,12 @@ int pstore_mkfile(struct dentry *root, struct pstore_record *record) private->dentry = dentry; private->record = record; inode->i_size = private->total_size = size; + +if (record->type ==PSTORE_TYPE_TRACE ) + inode->i_fop = &pstore_trace_file_operations; +else + inode->i_fop = &pstore_file_operations; + inode->i_private = private;if (record->time.tv_sec)
fs/pstore/platform.c +1 -0
@@ -58,6 +58,7 @@ static const char * const pstore_type_names[] = {"powerpc-common" ,"pmsg" ,"powerpc-opal" , + "trace", };static int pstore_new_entry;
fs/pstore/ramtrace.c +139 -0
151 lines not on this page · raw message
Signature · 1 line
2.7.4
RFC PATCH 6/7 tracing: Use ramtrace alloc and free methods while using persistent RAM
In reply to Nachammai KaruppiahIf persistent RAM is being used to record trace entries, allocate and free pages using ramtrace_alloc_page and ramtrace_free_page.
--- kernel/trace/ring_buffer.c | 122 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 119 insertions(+), 3 deletions(-)
kernel/trace/ring_buffer.c +119 -3
@@ -353,6 +353,18 @@ static void free_buffer_page(struct buffer_page *bpage)kfree (bpage); } +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +static void +free_buffer_page_cpu (struct buffer_page *bpage,int cpu,bool use_pstore) +{ +if (use_pstore) { +ramtrace_free_page (bpage->page, cpu); +kfree (bpage); + }else +free_buffer_page (bpage); +} +#endif +/* * We need to fit the time_stamp delta into 27 bits. */
@@ -1200,7 +1212,12 @@ static int rb_check_pages(struct ring_buffer_per_cpu *cpu_buffer)return 0 ; } +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +static int __rb_allocate_pages (long nr_pages,struct list_head *pages,int cpu, + bool use_pstore) +#else static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu) +#endif {struct buffer_page *bpage, *tmp;bool user_thread = current->mm !=NULL ;
@@ -1214,6 +1231,11 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)* to prevent doing any allocation when it is obvious that it is * not going to succeed. */ +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (use_pstore) + i =ramtrace_available_mem (); + else +#endif i =si_mem_available ();if (i < nr_pages)return -ENOMEM ;
@@ -1246,10 +1268,22 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)list_add (&bpage->list, pages); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (use_pstore) { +void *address =ramtrace_alloc_page (cpu); + +if (!address) + goto free_pages; + bpage->page = address; + }else { +#endif page =alloc_pages_node (cpu_to_node (cpu), mflags,0 );if (!page) goto free_pages; bpage->page =page_address (page); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE + } +#endif rb_init_page (bpage->page);if (user_thread &&fatal_signal_pending (current))
@@ -1263,7 +1297,11 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu) free_pages:list_for_each_entry_safe (bpage, tmp, pages, list) {list_del_init (&bpage->list); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (bpage, cpu, use_pstore); +#else free_buffer_page (bpage); +#endif }if (user_thread)clear_current_oom_origin ();
@@ -1278,7 +1316,12 @@ static int rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,WARN_ON (!nr_pages); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (__rb_allocate_pages (nr_pages, &pages, cpu_buffer->cpu, + cpu_buffer->use_pstore)) +#else if (__rb_allocate_pages(nr_pages, &pages, cpu_buffer->cpu)) +#endif return -ENOMEM ;/*
@@ -1414,10 +1457,23 @@ rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)rb_check_bpage (cpu_buffer, bpage); cpu_buffer->reader_page = bpage; + +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (cpu_buffer->use_pstore) { +void *address =ramtrace_alloc_page (cpu); + +if (!address) + goto fail_free_reader; + bpage->page = address; + }else { +#endif page =alloc_pages_node (cpu_to_node (cpu),GFP_KERNEL ,0 );if (!page) goto fail_free_reader; bpage->page =page_address (page); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE + } +#endif rb_init_page (bpage->page);INIT_LIST_HEAD (&cpu_buffer->reader_page->list);
@@ -1436,7 +1492,12 @@ rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)return cpu_buffer; fail_free_reader: +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (cpu_buffer->reader_page, cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (cpu_buffer->reader_page); +#endif fail_free_buffer:kfree (cpu_buffer);
@@ -1447,18 +1508,32 @@ static void rb_free_cpu_buffer(struct ring_buffer_per_cpu *cpu_buffer) {struct list_head *head = cpu_buffer->pages;struct buffer_page *bpage, *tmp; - +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (cpu_buffer->reader_page, cpu_buffer->cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (cpu_buffer->reader_page); +#endif rb_head_page_deactivate (cpu_buffer);if (head) {list_for_each_entry_safe (bpage, tmp, head, list) {list_del_init (&bpage->list); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (bpage, cpu_buffer->cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (bpage); +#endif } bpage = list_entry(head, struct buffer_page, list); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (bpage, cpu_buffer->cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (bpage); +#endif }kfree (cpu_buffer);
@@ -1832,7 +1907,12 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)* We have already removed references to this list item, just * free up the buffer_page and its page */ +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (to_remove_page, cpu_buffer->cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (to_remove_page); +#endif nr_removed--; }while (to_remove_page != last_page);
@@ -1913,7 +1993,12 @@ rb_insert_pages(struct ring_buffer_per_cpu *cpu_buffer)list_for_each_entry_safe (bpage, tmp, &cpu_buffer->new_pages, list) {list_del_init (&bpage->list); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (bpage, cpu_buffer->cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (bpage); +#endif } }return success;
@@ -2252,8 +2337,14 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size,* allocated without receiving ENOMEM */ INIT_LIST_HEAD (&cpu_buffer->new_pages); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (__rb_allocate_pages (cpu_buffer->nr_pages_to_update, + &cpu_buffer->new_pages, cpu, + cpu_buffer->use_pstore)) { +#else if (__rb_allocate_pages (cpu_buffer->nr_pages_to_update, &cpu_buffer->new_pages, cpu)) { +#endif /* not enough memory for new pages */ err = -ENOMEM ; goto out_err;
@@ -2319,7 +2410,12 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size,INIT_LIST_HEAD (&cpu_buffer->new_pages);if (cpu_buffer->nr_pages_to_update >0 &&__rb_allocate_pages (cpu_buffer->nr_pages_to_update, +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE + &cpu_buffer->new_pages, cpu_id, + cpu_buffer->use_pstore)) { +#else &cpu_buffer->new_pages, cpu_id)) { +#endif err = -ENOMEM ; goto out_err; }
@@ -2379,7 +2475,12 @@ int ring_buffer_resize(struct trace_buffer *buffer, unsigned long size,list_for_each_entry_safe (bpage, tmp, &cpu_buffer->new_pages, list) {list_del_init (&bpage->list); +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +free_buffer_page_cpu (bpage, cpu, + cpu_buffer->use_pstore); +#else free_buffer_page (bpage); +#endif } } out_err_unlock:
@@ -5184,13 +5285,22 @@ void *ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)if (bpage) goto out; +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (cpu_buffer->use_pstore) { + bpage = (struct buffer_data_page *)ramtrace_alloc_page (cpu); +if (!bpage) +return ERR_PTR (-ENOMEM ); + }else { +#endif page =alloc_pages_node (cpu_to_node (cpu),GFP_KERNEL | __GFP_NORETRY,0 );if (!page)return ERR_PTR (-ENOMEM ); bpage =page_address (page); - +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE + } +#endif out:rb_init_page (bpage);down_read (&trace_read_sem);
@@ -5229,7 +5339,13 @@ void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu, void *dataarch_spin_unlock (&cpu_buffer->lock);local_irq_restore (flags); - out: +out: +#ifdef CONFIG_TRACE_EVENTS_TO_PSTORE +if (cpu_buffer->use_pstore) { +ramtrace_free_page (bpage, cpu); +return ; + } +#endif free_page ((unsigned long )bpage);up_read (&trace_read_sem); }
Signature · 1 line
2.7.4
Quoted 11 lines · Nachammai Karuppiah
Hi,
This patch series adds support to store trace events in pstore.
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
Just to add, Nachammai was my intern in the recent outreachy program and we designed together a way for trace events to be written to pstore backed memory directory instead of regular memory. The basic idea is to allocate frace's ring buffer on pstore memory and have it right there. Then recover it on reboot. Nachammai wrote the code with some guidance :) . I talked to Steve as well in the past about the basic of idea of this. Steve is on vacation this week though.
This is similar to what +Sai Prakash Ranjan was trying to do sometime
ago: https://lkml.org/lkml/2018/9/8/221 . But that approach involved higher overhead due to synchronization of writing to the otherwise lockless ring buffer.
+Brian Norris has also expressed interest for this feature.
thanks,
- Joel
Quoted 85 lines
This requires a new backend - ramtrace that allocates pages from persistent storage for the tracing utility. This feature can be enabled using
TRACE_EVENTS_TO_PSTORE.
In this feature, the new backend is used only as a page allocator and once the users chooses to use pstore to record trace entries, the ring buffer pages are freed and allocated in pstore. Once this switch is done,ring_buffercontinues to operate just as before without much overhead. Since the ring buffer uses the persistent RAM buffer directly to record trace entries, all tracers would also persist across reboot.To test this feature, I used a simple module that would call panic during a write operation to file in tracefs directory. Before writing to the file, the ring buffer is moved to persistent RAM buffer through command line as shown below,
$echo 1 >
/sys/kernel/tracing/options/persistWriting to the file,
$echo 1 >/sys/kernel/tracing/crash/panic_on_writeThe above write operation results in system crash. After reboot, once the pstore is mounted, the trace entries from previous boot are available in file,
/sys/fs/pstore/trace-ramtrace-0Looking through this file, gives us the stack trace that led to the crash.
<...>-1 [001] .... 49.083909: __vfs_write <-vfs_write <...>-1 [001] .... 49.083933: panic <-panic_on_write <...>-1 [001] d... 49.084195: printk <-panic <...>-1 [001] d... 49.084201: vprintk_func <-printk <...>-1 [001] d... 49.084207: vprintk_default <-printk <...>-1 [001] d... 49.084211: vprintk_emit <-printk <...>-1 [001] d... 49.084216: __printk_safe_enter <-vprintk_emit <...>-1 [001] d... 49.084219: _raw_spin_lock <-vprintk_emit <...>-1 [001] d... 49.084223: vprintk_store <-vprintk_emitPatchwise oneline description is given below:
Patch 1 adds support to allocate ring buffer pages from persistent RAM buffer.
Patch 2 introduces a new backend, ramtrace.
Patch 3 adds methods to read previous boot pages from pstore.
Patch 4 adds the functionality to allocate page-sized memory from pstore.
Patch 5 adds the
seq_operationmethods to iterate through trace entries.Patch 6 modifies
ring_bufferto allocate from ramtrace when pstore is used.Patch 7 adds ramtrace DT node as child-node of /reserved-memory.
Nachammai Karuppiah (7):
tracing: Add support to allocate pages from persistent memory
pstore: Support a new backend, ramtrace
pstore: Read and iterate through trace entries in PSTORE
pstore: Allocate and free page-sized memory in persistent RAM buffer
tracing: Add support to iterate through pages retrieved from pstore
tracing: Use ramtrace alloc and free methods while using persistent
RAM
dt-bindings: ramtrace: Add ramtrace DT node.../bindings/reserved-memory/ramtrace.txt | 13 + drivers/of/platform.c | 1 + fs/pstore/Makefile | 2 + fs/pstore/inode.c | 46 +- fs/pstore/platform.c | 1 + fs/pstore/ramtrace.c | 821 +++++++++++++++++++++ include/linux/pstore.h | 3 + include/linux/ramtrace.h | 28 + include/linux/ring_buffer.h | 19 + include/linux/trace.h | 13 + kernel/trace/Kconfig | 10 + kernel/trace/ring_buffer.c | 663 ++++++++++++++++- kernel/trace/trace.c | 312 +++++++- kernel/trace/trace.h | 5 +- 14 files changed, 1924 insertions(+), 13 deletions(-) create mode 100644 Documentation/devicetree/bindings/reserved-memory/ramtrace.txt create mode 100644 fs/pstore/ramtrace.c create mode 100644 include/linux/ramtrace.h--
2.7.4
Quoted 8 lines · Joel Fernandes
Quoted 5 lines · Nachammai Karuppiah
Hi,
This patch series adds support to store trace events in pstore.
Been a long day...
Quoted 10 lines
Quoted 6 lines
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
Just to add, Nachammai was my intern in the recent outreachy program and we designed together a way for trace events to be written to pstore backed memory directory instead of regular memory. The basic
s/directory/directly/
Quoted 2 lines
idea is to allocate frace's ring buffer on pstore memory and have it right there. Then recover it on reboot. Nachammai wrote the code with
s/right/write/
- Joel
Quoted 99 lines
some guidance :) . I talked to Steve as well in the past about the basic of idea of this. Steve is on vacation this week though.
This is similar to what +Sai Prakash Ranjan was trying to do sometime
ago: https://lkml.org/lkml/2018/9/8/221 . But that approach involved higher overhead due to synchronization of writing to the otherwise lockless ring buffer.+Brian Norris has also expressed interest for this feature.
thanks,
- Joel
Quoted 85 lines
This requires a new backend - ramtrace that allocates pages from persistent storage for the tracing utility. This feature can be enabled using
TRACE_EVENTS_TO_PSTORE.
In this feature, the new backend is used only as a page allocator and once the users chooses to use pstore to record trace entries, the ring buffer pages are freed and allocated in pstore. Once this switch is done,ring_buffercontinues to operate just as before without much overhead. Since the ring buffer uses the persistent RAM buffer directly to record trace entries, all tracers would also persist across reboot.To test this feature, I used a simple module that would call panic during a write operation to file in tracefs directory. Before writing to the file, the ring buffer is moved to persistent RAM buffer through command line as shown below,
$echo 1 >
/sys/kernel/tracing/options/persistWriting to the file,
$echo 1 >/sys/kernel/tracing/crash/panic_on_writeThe above write operation results in system crash. After reboot, once the pstore is mounted, the trace entries from previous boot are available in file,
/sys/fs/pstore/trace-ramtrace-0Looking through this file, gives us the stack trace that led to the crash.
<...>-1 [001] .... 49.083909: __vfs_write <-vfs_write <...>-1 [001] .... 49.083933: panic <-panic_on_write <...>-1 [001] d... 49.084195: printk <-panic <...>-1 [001] d... 49.084201: vprintk_func <-printk <...>-1 [001] d... 49.084207: vprintk_default <-printk <...>-1 [001] d... 49.084211: vprintk_emit <-printk <...>-1 [001] d... 49.084216: __printk_safe_enter <-vprintk_emit <...>-1 [001] d... 49.084219: _raw_spin_lock <-vprintk_emit <...>-1 [001] d... 49.084223: vprintk_store <-vprintk_emitPatchwise oneline description is given below:
Patch 1 adds support to allocate ring buffer pages from persistent RAM buffer.
Patch 2 introduces a new backend, ramtrace.
Patch 3 adds methods to read previous boot pages from pstore.
Patch 4 adds the functionality to allocate page-sized memory from pstore.
Patch 5 adds the
seq_operationmethods to iterate through trace entries.Patch 6 modifies
ring_bufferto allocate from ramtrace when pstore is used.Patch 7 adds ramtrace DT node as child-node of /reserved-memory.
Nachammai Karuppiah (7):
tracing: Add support to allocate pages from persistent memory
pstore: Support a new backend, ramtrace
pstore: Read and iterate through trace entries in PSTORE
pstore: Allocate and free page-sized memory in persistent RAM buffer
tracing: Add support to iterate through pages retrieved from pstore
tracing: Use ramtrace alloc and free methods while using persistent
RAM
dt-bindings: ramtrace: Add ramtrace DT node.../bindings/reserved-memory/ramtrace.txt | 13 + drivers/of/platform.c | 1 + fs/pstore/Makefile | 2 + fs/pstore/inode.c | 46 +- fs/pstore/platform.c | 1 + fs/pstore/ramtrace.c | 821 +++++++++++++++++++++ include/linux/pstore.h | 3 + include/linux/ramtrace.h | 28 + include/linux/ring_buffer.h | 19 + include/linux/trace.h | 13 + kernel/trace/Kconfig | 10 + kernel/trace/ring_buffer.c | 663 ++++++++++++++++- kernel/trace/trace.c | 312 +++++++- kernel/trace/trace.h | 5 +- 14 files changed, 1924 insertions(+), 13 deletions(-) create mode 100644 Documentation/devicetree/bindings/reserved-memory/ramtrace.txt create mode 100644 fs/pstore/ramtrace.c create mode 100644 include/linux/ramtrace.h--
2.7.4
Quoted 22 lines · Joel Fernandes
Quoted 11 lines · Nachammai Karuppiah
Hi,
This patch series adds support to store trace events in pstore.
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
Just to add, Nachammai was my intern in the recent outreachy program and we designed together a way for trace events to be written to pstore backed memory directory instead of regular memory. The basic idea is to allocate frace's ring buffer on pstore memory and have it right there. Then recover it on reboot. Nachammai wrote the code with some guidance :) . I talked to Steve as well in the past about the basic of idea of this. Steve is on vacation this week though.
Quoted 7 lines
This is similar to what +Sai Prakash Ranjan was trying to do sometime
ago: https://lkml.org/lkml/2018/9/8/221 . But that approach involved higher overhead due to synchronization of writing to the otherwise lockless ring buffer.+Brian Norris has also expressed interest for this feature.
Great work Nachammai and Joel, I have few boards with warm reboot support and will test this series in coming days.
Thanks,
Sai
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 15 lines · Joel Fernandes
Quoted 11 lines · Nachammai Karuppiah
Hi,
This patch series adds support to store trace events in pstore.
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
Quoted 7 lines
Just to add, Nachammai was my intern in the recent outreachy program and we designed together a way for trace events to be written to pstore backed memory directory instead of regular memory. The basic idea is to allocate frace's ring buffer on pstore memory and have it right there. Then recover it on reboot. Nachammai wrote the code with some guidance :) . I talked to Steve as well in the past about the basic of idea of this. Steve is on vacation this week though.
ramoops is already the RAM backend for pstore and ramoops already has an ftrace region defined. What am I missing?
From a DT standpoint, we already have a reserved persistent RAM binding too. There's already too much kernel specifics on how it is used, we don't need more of that in DT. We're not going to add another separate region (actually, you can have as many regions defined as you want. They will just all be 'ramoops' compatible).
Rob
Hi Rob,
(Back from holidays, digging through the email pile). Reply below:
Quoted 25 lines · Rob Herring
Quoted 15 lines · Joel Fernandes
Quoted 11 lines · Nachammai Karuppiah
Hi,
This patch series adds support to store trace events in pstore.
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
Quoted 7 lines
Just to add, Nachammai was my intern in the recent outreachy program and we designed together a way for trace events to be written to pstore backed memory directory instead of regular memory. The basic idea is to allocate frace's ring buffer on pstore memory and have it right there. Then recover it on reboot. Nachammai wrote the code with some guidance :) . I talked to Steve as well in the past about the basic of idea of this. Steve is on vacation this week though.
Quoted 2 lines
ramoops is already the RAM backend for pstore and ramoops already has an ftrace region defined. What am I missing?
ramoops is too slow for tracing. Honestly, the ftrace functionality in ramoops should be removed in favor of Nachammai's patches (she did it for events but function tracing could be trivially added). No one uses the current ftrace in pstore because it is darned slow. ramoops sits in between the writing of the ftrace record and the memory being written to adding more overhead in the process, while also writing ftrace records in a non-ftrace format. So ramoop's API and infrastructure fundamentally does not meet the requirements of high speed persistent tracing. The idea of this work is to keep the trace events enabled for a long period time (possibly even in production) and low overhead until the problem like machine crashing happens.
Quoted 5 lines
From a DT standpoint, we already have a reserved persistent RAM binding too. There's already too much kernel specifics on how it is used, we don't need more of that in DT. We're not going to add another separate region (actually, you can have as many regions defined as you want. They will just all be 'ramoops' compatible).
I agree with the sentiment here on DT. Maybe the DT can be generalized to provide a ram region to which either ramoops or ramtrace can attach.
- Joel
Steven Rostedt
maintainerQuoted 2 lines · On Thu, 10 Sep 2020 21:25:11 -0400 Joel Fernandes <[email protected]>…
Hi Rob,
(Back from holidays, digging through the email pile). Reply below:
What ever happen to this?
Sorry, I was expecting more replies, and when there was nothing, it got lost in my inbox.
Quoted 48 lines · Rob Herring
Quoted 25 lines · Joel Fernandes
Quoted 15 lines
Quoted 11 lines · Nachammai Karuppiah
Hi,
This patch series adds support to store trace events in pstore.
Storing trace entries in persistent RAM would help in understanding what happened just before the system went down. The trace events that led to the crash can be retrieved from the pstore after a warm reboot. This will help debug what happened before machine’s last breath. This has to be done in a scalable way so that tracing a live system does not impact the performance of the system.
Quoted 7 lines
Just to add, Nachammai was my intern in the recent outreachy program and we designed together a way for trace events to be written to pstore backed memory directory instead of regular memory. The basic idea is to allocate frace's ring buffer on pstore memory and have it right there. Then recover it on reboot. Nachammai wrote the code with some guidance :) . I talked to Steve as well in the past about the basic of idea of this. Steve is on vacation this week though.
Quoted 2 lines
ramoops is already the RAM backend for pstore and ramoops already has an ftrace region defined. What am I missing?
ramoops is too slow for tracing. Honestly, the ftrace functionality in ramoops should be removed in favor of Nachammai's patches (she did it for events but function tracing could be trivially added). No one uses the current ftrace in pstore because it is darned slow. ramoops sits in between the writing of the ftrace record and the memory being written to adding more overhead in the process, while also writing ftrace records in a non-ftrace format. So ramoop's API and infrastructure fundamentally does not meet the requirements of high speed persistent tracing. The idea of this work is to keep the trace events enabled for a long period time (possibly even in production) and low overhead until the problem like machine crashing happens.
Quoted 5 lines
From a DT standpoint, we already have a reserved persistent RAM binding too. There's already too much kernel specifics on how it is used, we don't need more of that in DT. We're not going to add another separate region (actually, you can have as many regions defined as you want. They will just all be 'ramoops' compatible).
Quoted 3 lines
I agree with the sentiment here on DT. Maybe the DT can be generalized to provide a ram region to which either ramoops or ramtrace can attach.
Right,
Perhaps just remove patch 7, but still have the ramoops work move forward?
-- Steve
Quoted 12 lines · Steven Rostedt
Quoted 2 lines · On Thu, 10 Sep 2020 21:25:11 -0400 Joel Fernandes <[email protected]>…
Hi Rob,
(Back from holidays, digging through the email pile). Reply below:What ever happen to this?
Sorry, I was expecting more replies, and when there was nothing, it got lost in my inbox.
[...]
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
From a DT standpoint, we already have a reserved persistent RAM binding too. There's already too much kernel specifics on how it is used, we don't need more of that in DT. We're not going to add another separate region (actually, you can have as many regions defined as you want. They will just all be 'ramoops' compatible).
Quoted 7 lines
Quoted 3 lines
I agree with the sentiment here on DT. Maybe the DT can be generalized to provide a ram region to which either ramoops or ramtrace can attach.
Right,
Perhaps just remove patch 7, but still have the ramoops work move forward?
This was an internship project submission which stalled after the internship ended, I imagine Nachammai has moved on to doing other things since.
I am curious how this came on your radar after 2 years, did someone tell you to prioritize improving performance of ftrace on pstore? I could probably make time to work on it more if someone has a usecase for this or something.
Thanks,
- Joel
Steven Rostedt
maintainerQuoted 4 lines · On Fri, 1 Jul 2022 12:37:35 -0400 Joel Fernandes <[email protected]> wrote:
I am curious how this came on your radar after 2 years, did someone tell you to prioritize improving performance of ftrace on pstore? I could probably make time to work on it more if someone has a usecase for this or something.
I'm looking into ways to extract the ftrace ring buffer from crashes, and it was brought up that pstore was used before.
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 4 lines · On Fri, 1 Jul 2022 12:37:35 -0400 Joel Fernandes <[email protected]> wrote:
I am curious how this came on your radar after 2 years, did someone tell you to prioritize improving performance of ftrace on pstore? I could probably make time to work on it more if someone has a usecase for this or something.
I'm looking into ways to extract the ftrace ring buffer from crashes, and it was brought up that pstore was used before.
Interesting. In the case of pstore, you know exactly where the pages are for ftrace. How would you know that for the buddy system where pages are in the wild wild west? I guess you would need to track where ftrace pages where allocated, within the crash dump/report.
- Joel
Steven Rostedt
maintainerQuoted 4 lines · On Fri, 1 Jul 2022 12:53:17 -0400 Joel Fernandes <[email protected]> wrote:
Interesting. In the case of pstore, you know exactly where the pages are for ftrace. How would you know that for the buddy system where pages are in the wild wild west? I guess you would need to track where ftrace pages where allocated, within the crash dump/report.
kexec/kdump already does that (of course it requires the DWARF symbols of the kernel to be accessible by the kdump kernel).
But if we write the raw ftrace data into persistent memory that can survive a reboot, then we can extract that raw data and convert it back to text offline.
Thus, I would like to remove the converting to text and compressing into pstore, and possibly look at a solution that simply writes the raw data into pstore.
-- Steve