Register read/write tracing with dynamic debug and pstore
Subsystems (6)
cover Register read/write tracing with dynamic debug and pstore
Hi,
This patch series adds a new tracing facility for register reads and writes called Register Trace Buffer(RTB).
We also add pstore support through which we can save all register read/write logs into a persistent ram buffer that can be dumped after reboot.
It can be used to determine from where register was read/written before unclocked access or some kind of bus hang or an unexpected reset caused by some buggy driver which happens a lot during initial development stages.
In addition to this, we provide dynamic debug support to filter out unwanted logs and limit trace to only specific files or directories since there can be aweful lot of register events and we will be interested only in specific drivers or subsystems which we will be working on. Last few RTB entries will give us the hint for debugging. With dynamic debug, we are also reducing the overhead of tracing considerably.
Also as a bonus, this tracing can be extended to include IRQ, printk, context switch and lot other things with proper hooks. It can be very helpful for real case debug scenarios.
Below is a simple example of identifying cause for bus hang in qcom mdp tested on db410c. This hang was intentionally introduced just to show the usecase of RTB.
The module used can be found here: https://github.com/saiprakash-ranjan/Bus-Hang which does an unclocked access and will reset db410c and later logs can be viewed through pstore.
Note: I just copied bus_hang.c to drivers/soc/qcom and built it.
1) Set bootargs with dyndbg parameter as below:
# dyndbg="file drivers/soc/qcom/* +p"
2) Bus hang by reading below debugfs entry with bus_hang module.
# cat /sys/kernel/debug/hang/bus_hang
3) After restart, we can find the cause in last entry i.e. (bus_hang_mdp+0x98/0xb0)
# cat /sys/fs/pstore/rtb-ramoops-0 [LOGK_WRITEL ] ts:1373101930 data:ffff00000cd065a4 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668 [LOGK_WRITEL ] ts:1373311878 data:ffff00000cd06608 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668 [LOGK_READL ] ts:18177142294 data:ffff00000ab85040 <ffff00000867cdc8> bus_hang_mdp+0x98/0xb0
4) Offending register access found as below:
# (gdb)
# (gdb) list *(bus_hang_mdp+0x98)
# 0xffff00000867cdc8 is in bus_hang_mdp (drivers/soc/qcom/bus_hang.c:10).
# 5 static int bus_hang_mdp(void *data, u64 *val)
# 6 {
# 7 void *p = ioremap(0x01a01000, SZ_4K);
# 8 unsigned int a;
# 9
# 10 a = __raw_readl((void *)((unsigned long)p + 0x40)); <----
# 11
# 12 *val = a;
# 13
# 14 return 0;
# (gdb)
There will be a lot more real usecases where RTB can be used. Maybe we can test on other boards as well.
Patchwise one line description is given below:
This trace module is based on RTB driver in CAF.
Patch 1 provides the api called uncached_logk which can be used to log register accesses.
Patch 2 adds the pstore support for viewing the logs.
Patch 3 adds dynamic debug support to filter the register readl/writel access.
Sai Prakash Ranjan (3):
tracing: Add support for logging data to uncached buffer
pstore: Add register readl/writel tracing support
dynamic_debug: Add support for dynamic register trace
.../bindings/reserved-memory/ramoops.txt | 7 +- arch/arm64/include/asm/io.h | 93 ++++++++++ fs/pstore/Kconfig | 12 ++ fs/pstore/Makefile | 1 + fs/pstore/inode.c | 71 +++++++- fs/pstore/internal.h | 8 + fs/pstore/platform.c | 4 + fs/pstore/ram.c | 42 ++++- fs/pstore/rtb.c | 45 +++++ include/linux/dynamic_debug.h | 10 ++ include/linux/pstore.h | 2 + include/linux/pstore_ram.h | 1 + include/linux/rtb.h | 31 ++++ kernel/trace/Kconfig | 8 + kernel/trace/Makefile | 2 + kernel/trace/trace_rtb.c | 163 ++++++++++++++++++ 16 files changed, 495 insertions(+), 5 deletions(-) create mode 100644 fs/pstore/rtb.c create mode 100644 include/linux/rtb.h create mode 100644 kernel/trace/trace_rtb.c
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
RFC PATCH 1/3 tracing: Add support for logging data to uncached buffer
Add RTB trace support to write data to a small uncached buffer. When a system reset occurs, valuable data may still be remaining in the cache (e.g. last printks) and this data will probably be lost, giving an incomplete picture of what the system was last doing. By logging useful information to this uncached region (e.g. readl/writel, last printk), a better picture of what the system was last doing can be obtained.
Dummy platform device is created to use dma api to allocate uncached memory.
We add an additional property called rtb-size in ramoops device tree node for pstore RTB buffer size and use the same in the trace module. DT documentation has been modified to include this.
Information logged in this buffer include log type(readl/writel), timestamp, extra data from the caller, caller ip and name. This can be extented as needed to include more options(e.g. cpu) for better debugging.
Also RTB panic notifier with high priority is used to make sure that RTB is disabled right after a kernel panic so that log buffer could be prevented from being flooded with some I/O operations.
This is based on RTB driver in CAF. Link below:
* https://source.codeaurora.org/quic/la/kernel/msm-4.9
Modified to support pstore for viewing traces.
--- .../bindings/reserved-memory/ramoops.txt | 7 +- include/linux/rtb.h | 24 +++ kernel/trace/Kconfig | 7 + kernel/trace/Makefile | 2 + kernel/trace/trace_rtb.c | 160 ++++++++++++++++++ 5 files changed, 198 insertions(+), 2 deletions(-) create mode 100644 include/linux/rtb.h create mode 100644 kernel/trace/trace_rtb.c
Documentation/devicetree/bindings/reserved-memory/ramoops.txt +5 -2
@@ -14,8 +14,8 @@ Any remaining space will be used for a circular buffer of oops and panic records. These records have a configurable size, with a size of 0 indicating that they should be disabled. -At least one of "record-size", "console-size", "ftrace-size", or "pmsg-size" -must be set non-zero, but are otherwise optional as listed below. +At least one of "record-size", "console-size", "ftrace-size", "pmsg-size" or +"rtb-size" must be set non-zero, but are otherwise optional as listed below. Required properties:
@@ -42,6 +42,9 @@ Optional properties: - pmsg-size: size in bytes of log buffer reserved for userspace messages (defaults to 0: disabled) +- rtb-size: size in bytes of log buffer reserved for RTB buffer traces. + (defaults to 0: disabled) + - unbuffered: if present, use unbuffered mappings to map the reserved region (defaults to buffered mappings)
include/linux/rtb.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _RTB_H +#define _RTB_H + +struct rtb_layout { +const char *log_type; +u32 idx; +u64 caller; +u64 data; +u64 timestamp; +} __attribute__ ((__packed__ )); + +#if defined(CONFIG_RTB ) +void uncached_logk (const char *log_type,void *data); +int rtb_init (void ); +void rtb_exit (void ); +#else +static inline void uncached_logk (const char *log_type, +void *data) { } +static inline int rtb_init (void ) {return 0 ; } +static inline void rtb_exit (void ) { } +#endif + +#endif /* _RTB_H */
kernel/trace/Kconfig +7 -0
@@ -722,6 +722,13 @@ config TRACING_EVENTS_GPIO help Enable tracing events for gpio subsystem +config RTB + bool "Register Trace Buffer" + help + Add support for logging different events to a small uncached + region. This is designed to aid in debugging reset cases where the + caches may not be flushed before the target resets. + endif # FTRACE endif # TRACING_SUPPORT
kernel/trace/Makefile +2 -0
@@ -72,4 +72,6 @@ obj-$(CONFIG_UPROBE_EVENTS) += trace_uprobe.o obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o +obj-$(CONFIG_RTB) += trace_rtb.o + libftrace-y := ftrace.o
kernel/trace/trace_rtb.c new file +160 -0
160 lines not on this page · raw message
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Steven Rostedt
maintainerRe: [RFC PATCH 1/3] tracing: Add support for logging data to uncached buffer
Sorry for the late reply, I actually wrote this email over a week ago, but never hit send. And the email was pushed back behind other windows. :-/
Quoted 24 lines · On Fri, 3 Aug 2018 19:58:42 +0530 Sai Prakash Ranjan…
kernel/trace/trace_rtb.c new file +18
-0@@ -0,0 +1,160 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2018 The Linux Foundation. All rights reserved. +*/ +#include <linux/atomic.h> +#include <linux/dma-mapping.h> +#include <linux/export.h> +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/of_device.h> +#include <linux/platform_device.h> +#include <linux/rtb.h> +#include <linux/sched/clock.h> + +static struct platform_device *rtb_dev; +static atomic_t rtb_idx; + +struct rtb_state {
kernel/trace/trace_rtb.c line 28
Quoted 10 lines
kernel/trace/trace_rtb.c new file +10
-0+struct rtb_layout *rtb; +phys_addr_t phys; +unsigned int nentries; +unsigned int size; +int enabled; +}; + +static struct rtb_state rtb = { + .enabled =0 , +};
No need for the initialization, you could just have:
static struct rtb_state rtb;
And it will be initialized to all zeros. Or did you do that to document that it is not enabled at boot?
Quoted 23 lines
kernel/trace/trace_rtb.c new file +23
-0+ +static int rtb_panic_notifier (struct notifier_block *this, +unsigned long event,void *ptr) +{ + rtb.enabled =0 ; +return NOTIFY_DONE ; +} + +static struct notifier_block rtb_panic_blk = { + .notifier_call = rtb_panic_notifier, + .priority =INT_MAX , +}; + +static void rtb_write_type (const char *log_type, +struct rtb_layout *start) +{ + start->log_type = log_type; +} + +static void rtb_write_caller (u64 caller,struct rtb_layout *start) +{ + start->caller = caller; +}
kernel/trace/trace_rtb.c line 61
Quoted 10 lines
kernel/trace/trace_rtb.c new file +10
-0+ +static void rtb_write_data (u64 data,struct rtb_layout *start) +{ + start->data = data; +} + +static void rtb_write_timestamp (struct rtb_layout *start) +{ + start->timestamp =sched_clock (); +}
Why have the above static functions? They are not very helpful, and appear to be actually confusing. They are used once.
kernel/trace/trace_rtb.c line 73
Quoted 12 lines
kernel/trace/trace_rtb.c new file +12
-0+ +static void uncached_logk_pc_idx (const char *log_type,u64 caller, +u64 data,int idx) +{ +struct rtb_layout *start; + + start = &rtb.rtb[idx & (rtb.nentries -1 )]; + +rtb_write_type (log_type, start); +rtb_write_caller (caller, start); +rtb_write_data (data, start); +rtb_write_timestamp (start);
How is the above better than:
start->log_type = log_type; start->caller = caller; start->data = data;
start->timestamp = sched_clock();
??
Quoted 13 lines
kernel/trace/trace_rtb.c new file +13
-0+/* Make sure data is written */ +mb (); +} + +static int rtb_get_idx (void ) +{ +int i, offset; + + i =atomic_inc_return (&rtb_idx); + i--; + +/* Check if index has wrapped around */ + offset = (i & (rtb.nentries - 1)) -
kernel/trace/trace_rtb.c line 96
Quoted 10 lines
kernel/trace/trace_rtb.c new file +10
-0+ ((i -1 ) & (rtb.nentries -1 )); +if (offset <0 ) { + i =atomic_inc_return (&rtb_idx); + i--; + } + +return i; +} + +noinline void notrace uncached_logk(const char *log_type, void *data)
BTW, all files in this directory have their functions notrace by default.
-- Steve
Quoted 64 lines
kernel/trace/trace_rtb.c new file +64
-0+{ +int i; + +if (!rtb.enabled) +return ; + + i =rtb_get_idx (); +uncached_logk_pc_idx (log_type, (u64)(__builtin_return_address (0 )), + (u64)(data), i); +} +EXPORT_SYMBOL (uncached_logk); + +int rtb_init (void ) +{ +struct device_node *np; +u32 size; +int ret; + + np =of_find_node_by_name (NULL ,"ramoops" ); +if (!np) +return -ENODEV ; + + ret =of_property_read_u32 (np,"rtb-size" , &size); +if (ret) { +of_node_put (np); +return ret; + } + + rtb.size = size; + +/* Create a dummy platform device to use dma api */ + rtb_dev =platform_device_register_simple ("rtb" ,-1 ,NULL ,0 ); +if (IS_ERR (rtb_dev)) +return PTR_ERR (rtb_dev); + +/* +* The device is a dummy, so arch_setup_dma_ops +* is not called, thus leaving the device with dummy DMA ops +* which returns null in case of arm64. +*/ +of_dma_configure (&rtb_dev->dev,NULL , true); + rtb.rtb =dma_alloc_coherent (&rtb_dev->dev, rtb.size, + &rtb.phys,GFP_KERNEL ); +if (!rtb.rtb) +return -ENOMEM ; + + rtb.nentries = rtb.size /sizeof (struct rtb_layout ); +/* Round this down to a power of 2 */ + rtb.nentries =__rounddown_pow_of_two (rtb.nentries); + +memset (rtb.rtb,0 , rtb.size); +atomic_set (&rtb_idx,0 ); + +atomic_notifier_chain_register (&panic_notifier_list, + &rtb_panic_blk); + rtb.enabled =1 ; +return 0 ; +} + +void rtb_exit (void ) +{ +dma_free_coherent (&rtb_dev->dev, rtb.size, rtb.rtb, rtb.phys); +platform_device_unregister (rtb_dev); +}
Re: [RFC PATCH 1/3] tracing: Add support for logging data to uncached buffer
Quoted 6 lines · Steven Rostedt
Sorry for the late reply, I actually wrote this email over a week ago, but never hit send. And the email was pushed back behind other windows. :-/
Thanks for the review Steven.
And no problem on late reply, I was working on Will's comment about instrumentation in arch code and was about to respin a v2 patch. I have replied inline, let me know if any more corrections or improvements can be done. I would also like if Kees or someone from pstore could comment on patch 2.
Quoted 40 lines
Quoted 24 lines · On Fri, 3 Aug 2018 19:58:42 +0530 Sai Prakash Ranjan…
kernel/trace/trace_rtb.c new file +18
-0@@ -0,0 +1,160 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (C) 2018 The Linux Foundation. All rights reserved. +*/ +#include <linux/atomic.h> +#include <linux/dma-mapping.h> +#include <linux/export.h> +#include <linux/kernel.h> +#include <linux/module.h> +#include <linux/of_device.h> +#include <linux/platform_device.h> +#include <linux/rtb.h> +#include <linux/sched/clock.h> + +static struct platform_device *rtb_dev; +static atomic_t rtb_idx; + +struct rtb_state {kernel/trace/trace_rtb.c line 28
Quoted 10 lines
kernel/trace/trace_rtb.c new file +10
-0+struct rtb_layout *rtb; +phys_addr_t phys; +unsigned int nentries; +unsigned int size; +int enabled; +}; + +static struct rtb_state rtb = { + .enabled =0 , +};No need for the initialization, you could just have:
Quoted 5 lines
static struct
rtb_statertb;And it will be initialized to all zeros. Or did you do that to document that it is not enabled at boot?
I will correct it in v2. RTB will not be enabled until pstore is registered since we use pstore for logs. I will add a comment above the static declaration saying so.
Quoted 28 lines
Quoted 28 lines
kernel/trace/trace_rtb.c new file +28
-0+ +static int rtb_panic_notifier (struct notifier_block *this, +unsigned long event,void *ptr) +{ + rtb.enabled =0 ; +return NOTIFY_DONE ; +} + +static struct notifier_block rtb_panic_blk = { + .notifier_call = rtb_panic_notifier, + .priority =INT_MAX , +}; + +static void rtb_write_type (const char *log_type, +struct rtb_layout *start) +{ + start->log_type = log_type; +} + +static void rtb_write_caller (u64 caller,struct rtb_layout *start) +{ + start->caller = caller; +} + +static void rtb_write_data (u64 data,struct rtb_layout *start) +{ + start->data = data; +}
Quoted 9 lines
kernel/trace/trace_rtb.c line 61
Quoted 5 lines
kernel/trace/trace_rtb.c new file +5
-0+ +static void rtb_write_timestamp (struct rtb_layout *start) +{ + start->timestamp =sched_clock (); +}Why have the above static functions? They are not very helpful, and appear to be actually confusing. They are used once.
Yes you are right, will remove those.
Quoted 15 lines
kernel/trace/trace_rtb.c line 73
Quoted 12 lines
kernel/trace/trace_rtb.c new file +12
-0+ +static void uncached_logk_pc_idx (const char *log_type,u64 caller, +u64 data,int idx) +{ +struct rtb_layout *start; + + start = &rtb.rtb[idx & (rtb.nentries -1 )]; + +rtb_write_type (log_type, start); +rtb_write_caller (caller, start); +rtb_write_data (data, start); +rtb_write_timestamp (start);How is the above better than:
Quoted 7 lines
start->log_type = log_type; start->caller = caller; start->data = data;
start->timestamp=sched_clock();??
Sure, will change it to above and post v2.
Quoted 18 lines
Quoted 18 lines
kernel/trace/trace_rtb.c new file +18
-0+/* Make sure data is written */ +mb (); +} + +static int rtb_get_idx (void ) +{ +int i, offset; + + i =atomic_inc_return (&rtb_idx); + i--; + +/* Check if index has wrapped around */ + offset = (i & (rtb.nentries -1 )) - + ((i -1 ) & (rtb.nentries -1 )); +if (offset <0 ) { + i =atomic_inc_return (&rtb_idx); + i--; + }
Quoted 9 lines
kernel/trace/trace_rtb.c line 96
Quoted 5 lines
kernel/trace/trace_rtb.c new file +5
-0+ +return i; +} + +noinline void notrace uncached_logk(const char *log_type, void *data)BTW, all files in this directory have their functions notrace by default.
Oh I missed it. Will remove notrace.
- Sai Prakash
RFC PATCH 2/3 pstore: Add register readl/writel tracing support
In reply to Sai Prakash Ranjanreadl/writel are typically used for reading from memory mapped registers, which can cause hangs if accessed unclocked. Tracing these events can help in debugging various issues faced during initial development.
We log this trace information in persistent ram buffer which can be viewed after reset.
We use pstore_rtb_call() to write the RTB log to pstore. RTB buffer size is taken from ramoops dt node with additional property called rtb-size.
For reading the trace after mounting pstore, rtb-ramoops entry can be seen in /sys/fs/pstore/ as in below sample output.
Sample output of tracing register reads/writes in drivers:
# mount -t pstore pstore /sys/fs/pstore # tail /sys/fs/pstore/rtb-ramoops-0 [LOGK_READL ] ts:36468476204 data:ffff00000800d0fc <ffff0000084e9ee0> gic_check_gicv2+0x58/0x60 [LOGK_WRITEL ] ts:36468477715 data:ffff00000800d000 <ffff0000084e9fac> gic_cpu_if_up+0xc4/0x110 [LOGK_READL ] ts:36468478548 data:ffff00000800d000 <ffff0000084e9fd8> gic_cpu_if_up+0xf0/0x110 [LOGK_WRITEL ] ts:36468480319 data:ffff00000800d000 <ffff0000084e9fac> gic_cpu_if_up+0xc4/0x110 [LOGK_READL ] ts:36468481048 data:ffff00000800d00c <ffff000008081a34> gic_handle_irq+0xac/0x128 [LOGK_WRITEL ] ts:36468482923 data:ffff00000800d010 <ffff000008081aac> gic_handle_irq+0x124/0x128 [LOGK_READL ] ts:36468483184 data:ffff00000800d00c <ffff000008081a34> gic_handle_irq+0xac/0x128 [LOGK_WRITEL ] ts:36468485215 data:ffff00000800d010 <ffff000008081aac> gic_handle_irq+0x124/0x128 [LOGK_READL ] ts:36468486309 data:ffff00000800d00c <ffff000008081a34> gic_handle_irq+0xac/0x128 [LOGK_WRITEL ] ts:36468488236 data:ffff00000800d010 <ffff000008081aac> gic_handle_irq+0x124/0x128
Output has below 5 fields:
* Log type, Timestamp, Data from caller which is the address of read/write, Caller ip and Caller name.
--- fs/pstore/Kconfig | 12 +++++++ fs/pstore/Makefile | 1 + fs/pstore/inode.c | 71 +++++++++++++++++++++++++++++++++++++- fs/pstore/internal.h | 8 +++++ fs/pstore/platform.c | 4 +++ fs/pstore/ram.c | 42 ++++++++++++++++++++-- fs/pstore/rtb.c | 45 ++++++++++++++++++++++++ include/linux/pstore.h | 2 ++ include/linux/pstore_ram.h | 1 + include/linux/rtb.h | 7 ++++ kernel/trace/trace_rtb.c | 3 ++ 11 files changed, 193 insertions(+), 3 deletions(-) create mode 100644 fs/pstore/rtb.c
fs/pstore/Kconfig +12 -0
@@ -113,6 +113,18 @@ config PSTORE_PMSG If unsure, say N. +config PSTORE_RTB + bool "Log register operations like read/write" + depends on PSTORE && PSTORE!=m + depends on RTB + help + When this option is enabled, rtb driver will log all register + reads/writes into a persistent ram buffer that can be decoded + and dumped after reboot through pstore filesystem. It can be used + to debug readl/writel access. + + If unsure, say N. + config PSTORE_FTRACE bool "Persistent function tracer" depends on PSTORE
fs/pstore/Makefile +1 -0
@@ -9,6 +9,7 @@ pstore-objs += inode.o platform.o pstore-$(CONFIG_PSTORE_FTRACE) += ftrace.o pstore-$(CONFIG_PSTORE_PMSG) += pmsg.o +pstore-$(CONFIG_PSTORE_RTB) += rtb.o ramoops-objs += ram.o ram_core.o obj-$(CONFIG_PSTORE_RAM) += ramoops.o
fs/pstore/inode.c +70 -1
107 lines not on this page · raw message
fs/pstore/internal.h +8 -0
@@ -25,6 +25,14 @@ static inline void pstore_register_pmsg(void) {}static inline void pstore_unregister_pmsg (void ) {}#endif +#ifdef CONFIG_PSTORE_RTB +extern void pstore_register_rtb (void ); +extern void pstore_unregister_rtb (void ); +#else +static inline void pstore_register_rtb (void ) {} +static inline void pstore_unregister_rtb (void ) {} +#endif +extern struct pstore_info *psinfo;extern void pstore_set_kmsg_bytes (int );
fs/pstore/platform.c +4 -0
@@ -573,6 +573,8 @@ int pstore_register(struct pstore_info *psi)pstore_register_ftrace ();if (psi->flags &PSTORE_FLAGS_PMSG )pstore_register_pmsg (); +if (psi->flags &PSTORE_FLAGS_RTB ) +pstore_register_rtb ();/* Start watching for new records, if desired. */ if (pstore_update_ms >=0 ) {
@@ -602,6 +604,8 @@ void pstore_unregister(struct pstore_info *psi)del_timer_sync (&pstore_timer);flush_work (&pstore_work); +if (psi->flags &PSTORE_FLAGS_RTB ) +pstore_unregister_rtb ();if (psi->flags &PSTORE_FLAGS_PMSG )pstore_unregister_pmsg ();if (psi->flags &PSTORE_FLAGS_FTRACE )
fs/pstore/ram.c +40 -2
156 lines not on this page · raw message
fs/pstore/rtb.c new file +45 -0
45 lines not on this page · raw message
include/linux/pstore.h +2 -0
@@ -44,6 +44,7 @@ enum pstore_type_id {PSTORE_TYPE_PPC_COMMON =6 ,PSTORE_TYPE_PMSG =7 ,PSTORE_TYPE_PPC_OPAL =8 , +PSTORE_TYPE_RTB =9 , PSTORE_TYPE_UNKNOWN = 255 };
@@ -193,6 +194,7 @@ struct pstore_info {#define PSTORE_FLAGS_CONSOLE (1 << 1)#define PSTORE_FLAGS_FTRACE (1 << 2)#define PSTORE_FLAGS_PMSG (1 << 3) +#define PSTORE_FLAGS_RTB (1 << 4)extern int pstore_register (struct pstore_info *);extern void pstore_unregister (struct pstore_info *);
include/linux/pstore_ram.h +1 -0
@@ -97,6 +97,7 @@ struct ramoops_platform_data {unsigned long console_size;unsigned long ftrace_size;unsigned long pmsg_size; +unsigned long rtb_size;int dump_oops;u32 flags;struct persistent_ram_ecc_info ecc_info;
include/linux/rtb.h +7 -0
@@ -21,4 +21,11 @@ static inline int rtb_init(void) { return 0; }static inline void rtb_exit (void ) { }#endif +#if defined(CONFIG_PSTORE_RTB ) +extern void pstore_rtb_call (struct rtb_layout *start); +#else +static inline void pstore_rtb_call (struct rtb_layout *start) +{ } +#endif +#endif /* _RTB_H */
kernel/trace/trace_rtb.c +3 -0
@@ -73,6 +73,9 @@ static void uncached_logk_pc_idx(const char *log_type, u64 caller,rtb_write_timestamp (start);/* Make sure data is written */ mb (); +#if defined(CONFIG_PSTORE_RTB ) +pstore_rtb_call (start); +#endif }static int rtb_get_idx (void )
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
RFC PATCH 3/3 dynamic_debug: Add support for dynamic register trace
In reply to Sai Prakash RanjanIntroduce dynamic debug filtering mechanism to register tracing as dynamic_rtb() which will reduce a lot of overhead otherwise of tracing all the register reads/writes in all files.
Now we can just specify the file name or any wildcard pattern as any other dynamic debug facility in bootargs and dynamic rtb will just trace them and the output can be seen in pstore.
TODO: Now we use same 'p' flag but will add a separate flag for register trace later.
Example for tracing all register reads/writes in drivers/soc/qcom/* below:
# dyndbg="file drivers/soc/qcom/* +p" in bootargs
# reboot -f
# mount -t pstore pstore /sys/fs/pstore
# cat /sys/fs/pstore/rtb-ramoops-0
[LOGK_WRITEL ] ts:1373030419 data:ffff00000d5065a4 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668
[LOGK_WRITEL ] ts:1373360576 data:ffff00000d506608 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668Also we add uncached_logk api to readl/writel definitions for arm64 as of now. This can be extended to arm as well later for tracing.
--- arch/arm64/include/asm/io.h | 93 +++++++++++++++++++++++++++++++++++ include/linux/dynamic_debug.h | 10 ++++ kernel/trace/Kconfig | 1 + 3 files changed, 104 insertions(+)
arch/arm64/include/asm/io.h +93 -0
@@ -22,6 +22,7 @@#ifdef __KERNEL__#include <linux/types.h> +#include <linux/rtb.h> #include <asm/byteorder.h> #include <asm/barrier.h>
@@ -36,6 +37,7 @@/* * Generic IO read/write. These perform native-endian accesses. */ +#if !defined(CONFIG_RTB )#define __raw_writeb __raw_writebstatic inline void __raw_writeb (u8 val,volatile void __iomem *addr) {
@@ -104,6 +106,97 @@ static inline u64 __raw_readq(const volatile void __iomem *addr) : "=r" (val) : "r" (addr));return val; } +#else +static inline void __raw_writeb_log (u8 val,volatile void __iomem *addr) +{ + asmvolatile ("strb %w0, [%1]" : :"rZ" (val),"r" (addr)); +} + +static inline void __raw_writew_log (u16 val,volatile void __iomem *addr) +{ + asmvolatile ("strh %w0, [%1]" : :"rZ" (val),"r" (addr)); +} + +static inline void __raw_writel_log (u32 val,volatile void __iomem *addr) +{ + asmvolatile ("str %w0, [%1]" : :"rZ" (val),"r" (addr)); +} + +static inline void __raw_writeq_log (u64 val,volatile void __iomem *addr) +{ + asmvolatile ("str %x0, [%1]" : :"rZ" (val),"r" (addr)); +} + +static inline u8 __raw_readb_log (const volatile void __iomem *addr) +{ +u8 val; + + asm volatile(ALTERNATIVE("ldrb %w0, [%1]", + "ldarb %w0, [%1]", + ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) + : "=r" (val) : "r" (addr)); +return val; +} + +static inline u16 __raw_readw_log (const volatile void __iomem *addr) +{ +u16 val; + + asm volatile(ALTERNATIVE("ldrh %w0, [%1]", + "ldarh %w0, [%1]", + ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) + : "=r" (val) : "r" (addr)); +return val; +} + +static inline u32 __raw_readl_log (const volatile void __iomem *addr) +{ +u32 val; + + asm volatile(ALTERNATIVE("ldr %w0, [%1]", + "ldar %w0, [%1]", + ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) + : "=r" (val) : "r" (addr)); +return val; +} + +static inline u64 __raw_readq_log (const volatile void __iomem *addr) +{ +u64 val; + + asm volatile(ALTERNATIVE("ldr %0, [%1]", + "ldar %0, [%1]", + ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE) + : "=r" (val) : "r" (addr)); +return val; +} + +#define __raw_write_logged (v, a, _t) ({ \ + volatile void __iomem *_a = (a); \ + void *_addr = (void __force *)(_a); \ + dynamic_rtb("LOGK_WRITEL", _addr); \ + __raw_write##_t##_log((v), _a); \ + }) + +#define __raw_writeb (v, a) __raw_write_logged((v), a, b) +#define __raw_writew (v, a) __raw_write_logged((v), a, w) +#define __raw_writel (v, a) __raw_write_logged((v), a, l) +#define __raw_writeq (v, a) __raw_write_logged((v), a, q) + +#define __raw_read_logged (a, _l, _t) ({ \ + _t __a; \ + const volatile void __iomem *_a = (const volatile void __iomem *)(a);\ + void *_addr = (void __force *)(_a); \ + dynamic_rtb("LOGK_READL", _addr); \ + __a = __raw_read##_l##_log(_a); \ + __a; \ + }) + +#define __raw_readb (a) __raw_read_logged((a), b, u8) +#define __raw_readw (a) __raw_read_logged((a), w, u16) +#define __raw_readl (a) __raw_read_logged((a), l, u32) +#define __raw_readq (a) __raw_read_logged((a), q, u64) +#endif /* IO barriers */ #define __iormb () rmb()
include/linux/dynamic_debug.h +10 -0
@@ -155,6 +155,16 @@ do { \ buf, len, ascii); \ } while (0) +#if defined(CONFIG_RTB ) +#define dynamic_rtb (log_type, data) \ +do { \ + DEFINE_DYNAMIC_DEBUG_METADATA(descriptor, \ + __builtin_constant_p(log_type) ? log_type : "rtb");\ + if (DYNAMIC_DEBUG_BRANCH(descriptor)) \ + uncached_logk(log_type, data); \ +} while (0) +#endif +#else #include <linux/string.h>
kernel/trace/Kconfig +1 -0
@@ -724,6 +724,7 @@ config TRACING_EVENTS_GPIO config RTB bool "Register Trace Buffer" + depends on DYNAMIC_DEBUG help Add support for logging different events to a small uncached region. This is designed to aid in debugging reset cases where the
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Re: [RFC PATCH 3/3] dynamic_debug: Add support for dynamic register trace
Quoted 21 lines · Sai Prakash Ranjan
Introduce dynamic debug filtering mechanism to register tracing as
dynamic_rtb()which will reduce a lot of overhead otherwise of tracing all the register reads/writes in all files.Now we can just specify the file name or any wildcard pattern as any other dynamic debug facility in bootargs and dynamic rtb will just trace them and the output can be seen in pstore.
TODO: Now we use same 'p' flag but will add a separate flag for register trace later.
Example for tracing all register reads/writes in
drivers/soc/qcom/*below:# dyndbg="file
drivers/soc/qcom/*+p" in bootargs# reboot -f # mount -t pstore pstore /sys/fs/pstore # cat /sys/fs/pstore/rtb-ramoops-0 [LOGK_WRITEL ] ts:1373030419 data:ffff00000d5065a4 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668 [LOGK_WRITEL ] ts:1373360576 data:ffff00000d506608 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668
Quoted 6 lines
Also we add
uncached_logkapi to readl/writel definitions for arm64 as of now. This can be extended to arm as well later for tracing.Signed-off-by Sai Prakash Ranjan <[email protected]>--- arch/arm64/include/asm/io.h | 93 +++++++++++++++++++++++++++++++++++
Putting all of this in the arch code, which basically duplicates everything, feels very wrong to me. Perhaps take a look at the ongoing work for instrumenting the atomics and take some inspiration from there?
Ideally, the architecture just needs to provide the low-level primivites (which it already does) and the core can generate instruments versions if required.
Will
Re: [RFC PATCH 3/3] dynamic_debug: Add support for dynamic register trace
Quoted 29 lines · Will Deacon
Quoted 21 lines · Sai Prakash Ranjan
Introduce dynamic debug filtering mechanism to register tracing as
dynamic_rtb()which will reduce a lot of overhead otherwise of tracing all the register reads/writes in all files.Now we can just specify the file name or any wildcard pattern as any other dynamic debug facility in bootargs and dynamic rtb will just trace them and the output can be seen in pstore.
TODO: Now we use same 'p' flag but will add a separate flag for register trace later.
Example for tracing all register reads/writes in
drivers/soc/qcom/*below:# dyndbg="file
drivers/soc/qcom/*+p" in bootargs# reboot -f # mount -t pstore pstore /sys/fs/pstore # cat /sys/fs/pstore/rtb-ramoops-0 [LOGK_WRITEL ] ts:1373030419 data:ffff00000d5065a4 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668 [LOGK_WRITEL ] ts:1373360576 data:ffff00000d506608 <ffff00000867cb44> qcom_smsm_probe+0x51c/0x668Quoted 6 lines
Also we add
uncached_logkapi to readl/writel definitions for arm64 as of now. This can be extended to arm as well later for tracing.Signed-off-by Sai Prakash Ranjan <[email protected]>--- arch/arm64/include/asm/io.h | 93 +++++++++++++++++++++++++++++++++++
Quoted 8 lines
Putting all of this in the arch code, which basically duplicates everything, feels very wrong to me. Perhaps take a look at the ongoing work for instrumenting the atomics and take some inspiration from there?
Ideally, the architecture just needs to provide the low-level primivites (which it already does) and the core can generate instruments versions if required.
Hi Will,
Thanks for the review. Will look at instrumented atomics implementation and get back. Let me know if anything else can be improved.
- Sai Prakash