Skip to contentSkip to conversation

Add gdb python scripts as kernel debugging helpers

PATCH Superseded Jan Kiszka 4 participants 33 messages Jan 29, 2013 – Feb 14, 2013

Subsystems (3)

v5 of 13 Superseded Latest is v13 →

lore
Share & tools
lore

Jan Kiszka

author maintainer
#0 lore raw

v5 cover Add gdb python scripts as kernel debugging helpers

Version 5 comes with the following changes:
- moved tutorial into Documentation/gdb-kernel-debugging.txt
- improved caching of gdb.Type objects, ensure they are in sync with currently loaded symbols
- added new functions and commands
- lx_module -- Find module by name and return the module variable
- lx_modvar -- Return global variable of a module
- lx-lsmod -- List currently loaded modules

See http://lkml.indiana.edu/hypermail/linux/kernel/1210.0/01598.html for the original description and

git://git.kiszka.org/linux.git queues/gdb-scripts

for the latest version.

Jan

Jan Kiszka (20):
scripts/gdb: Add infrastructure
scripts/gdb: Add cache for type objects
scripts/gdb: Add container_of helper and convenience function scripts/gdb: Add module iteration helper
scripts/gdb: Add lx-symbols command
scripts/gdb: Add internal helper and convenience function to look up
a module
scripts/gdb: Add lx_modvar convenience function scripts/gdb: Add get_target_endianness helper
scripts/gdb: Add read_u16/32/64 helpers
scripts/gdb: Add lx-dmesg command
scripts/gdb: Add task iteration helper
scripts/gdb: Add helper and convenience function to look up tasks scripts/gdb: Add is_target_arch helper
scripts/gdb: Add internal helper and convenience function to retrieve
thread_info
scripts/gdb: Add get_gdbserver_type helper
scripts/gdb: Add internal helper and convenience function for per-cpu
lookup
scripts/gdb: Add lx_current convenience function scripts/gdb: Add helper to iterate over CPU masks scripts/gdb: Add lx-lsmod command
scripts/gdb: Add basic documentation

 Documentation/gdb-kernel-debugging.txt |  155 +++++++++++++++++++++++++++++++
 Makefile                               |    5 +-
 scripts/Makefile                       |    3 +-
 scripts/gdb/Makefile                   |    9 ++
 scripts/gdb/dmesg.py                   |   63 +++++++++++++
 scripts/gdb/module.py                  |  149 ++++++++++++++++++++++++++++++
 scripts/gdb/percpu.py                  |  110 ++++++++++++++++++++++
 scripts/gdb/symbols.py                 |  147 +++++++++++++++++++++++++++++
 scripts/gdb/task.py                    |   99 ++++++++++++++++++++
 scripts/gdb/utils.py                   |  158 ++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py             |   29 ++++++
 11 files changed, 925 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt
 create mode 100644 scripts/gdb/Makefile
 create mode 100644 scripts/gdb/dmesg.py
 create mode 100644 scripts/gdb/module.py
 create mode 100644 scripts/gdb/percpu.py
 create mode 100644 scripts/gdb/symbols.py
 create mode 100644 scripts/gdb/task.py
 create mode 100644 scripts/gdb/utils.py
 create mode 100644 scripts/gdb/vmlinux-gdb.py
Signature · 1 line
1.7.3.4
author maintainer
#1 lore raw

PATCH v5 1/20 scripts/gdb: Add infrastructure

This provides the basic infrastructure to load kernel-specific python helper scripts when debugging the kernel in gdb.

The loading mechanism is based on gdb loading for <objfile>-gdb.py when opening <objfile>. Therefore, this places a corresponding link to the main helper script into the output directory that contains vmlinux.

The main scripts will pull in submodules containing Linux specific gdb commands and functions. To avoid polluting the source directory with compiled python modules, we link to them from the object directory.

Due to gdb.parse_and_eval, we depend on gdb >= 7.1. We need to pre-process the version string returned by gdb as some distros tend to prefix it with their name.

This feature depends on CONFIG_DEBUG_INFO.

---
 Makefile                   |    5 ++++-
 scripts/Makefile           |    3 ++-
 scripts/gdb/Makefile       |    9 +++++++++
 scripts/gdb/utils.py       |   17 +++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |   22 ++++++++++++++++++++++
 5 files changed, 54 insertions(+), 2 deletions(-)
 create mode 100644 scripts/gdb/Makefile
 create mode 100644 scripts/gdb/utils.py
 create mode 100644 scripts/gdb/vmlinux-gdb.py
Makefile +4 -1
index 2d3c92c..f0d2b2d 100644
@@ -774,6 +774,9 @@ endif
 ifdef CONFIG_BUILD_DOCSRC
 	$(Q)$(MAKE) $(build)=Documentation
 endif
+ifdef CONFIG_DEBUG_INFO
+	$(Q)ln -fsn $(srctree)/scripts/gdb/vmlinux-gdb.py
+endif
 	+$(call if_changed,link-vmlinux)
 
 # The actual objects are generated when descending, 
@@ -1019,7 +1022,7 @@ MRPROPER_FILES += .config .config.old .version .old_version $(version_h) \
 		  Module.symvers tags TAGS cscope* GPATH GTAGS GRTAGS GSYMS \
 		  signing_key.priv signing_key.x509 x509.genkey		\
 		  extra_certificates signing_key.x509.keyid		\
-		  signing_key.x509.signer
+		  signing_key.x509.signer vmlinux-gdb.py
 
 # clean - Delete most, but leave enough to build external modules
 #
scripts/Makefile +2 -1
index 01e7adb..3204b91 100644
@@ -37,6 +37,7 @@ subdir-$(CONFIG_MODVERSIONS) += genksyms
 subdir-y                     += mod
 subdir-$(CONFIG_SECURITY_SELINUX) += selinux
 subdir-$(CONFIG_DTC)         += dtc
+subdir-$(CONFIG_DEBUG_INFO)  += gdb
 
 # Let clean descend into subdirs
-subdir-	+= basic kconfig package selinux
+subdir-	+= basic kconfig package selinux gdb
scripts/gdb/Makefile new file +9 -0
new file mode 100644
index 0000000..34ccd06
@@ -0,0 +1,9 @@
+always := gdb-scripts
+
+$(obj)/gdb-scripts:
+ifneq ($(KBUILD_SRC),)
+	$(Q)ln -fsn $(srctree)/$(obj)/*.py $(objtree)/$(obj)
+endif
+	@:
+
+clean-files := *.pyc $(if $(KBUILD_SRC),*.py)
scripts/gdb/utils.py new file +17 -0
new file mode 100644
index 0000000..f75eae6
@@ -0,0 +1,17 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  common utilities
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+import re
+
+gdb_version = re.sub("^[^0-9]*", "", gdb.VERSION)
scripts/gdb/vmlinux-gdb.py new file +22 -0
new file mode 100644
index 0000000..bcb45cc
@@ -0,0 +1,22 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  loader module
+#
+# Copyright (c) Siemens AG, 2012
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import os
+
+sys.path = [ os.path.dirname(__file__) + "/scripts/gdb" ] + sys.path
+
+from utils import gdb_version
+
+if gdb_version < "7.1":
+	print "NOTE: gdb 7.1 or later required for Linux helper scripts " \
+	      "to work."
Signature · 1 line
1.7.3.4
#2 lore raw

Re: [PATCH v5 01/20] scripts/gdb: Add infrastructure

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jan" == Jan Kiszka <[email protected]> writes:

Jan> +if gdb_version < "7.1":
Jan> + print "NOTE: gdb 7.1 or later required for Linux helper scripts " \
Jan> + "to work."

FWIW you can just directly feature-test for the things you know you need.
Like: if not hasattr(gdb, 'parse_and_eval'): ...
In this case, 7.1 is old enough that it probably doesn't matter. For newer things, though, it is better to feature-test because distros sometimes forward-port patches, so the version number isn't a reliable indicator.

Tom

author maintainer
#3 lore raw

PATCH v5 2/20 scripts/gdb: Add cache for type objects

In reply to Jan Kiszka

Type lookups are very slow in gdb-python which is often noticeable when iterating over a number of objects. Introduce the helper class CachedType that keeps a reference to a gdb.Type object but also refreshes it after an object file has been loaded.

---
 scripts/gdb/utils.py |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)
scripts/gdb/utils.py +21 -0
index f75eae6..fdff85e 100644
@@ -15,3 +15,24 @@ import gdb
 import re
 
 gdb_version = re.sub("^[^0-9]*", "", gdb.VERSION)
+
+
+class CachedType:
+	_type = None
+
+	def _new_objfile_handler(self, event):
+		self._type = None
+		gdb.events.new_objfile.disconnect(self._new_objfile_handler)
+
+	def __init__(self, name):
+		self._name = name
+
+	def get_type(self):
+		if self._type == None:
+			self._type = gdb.lookup_type(self._name)
+			if self._type == None:
+				raise gdb.GdbError("cannot resolve " \
+						   "type '%s'" % self._name)
+			gdb.events.new_objfile.connect(
+				self._new_objfile_handler)
+		return self._type
Signature · 1 line
1.7.3.4
author maintainer
#4 lore raw

PATCH v5 3/20 scripts/gdb: Add container_of helper and convenience function

In reply to Jan Kiszka

Provide an internal helper with container_of semantics. As type lookups are very slow in gdb-python and we need a type "long" for this, cache the reference to this type object. Then export the helper also as a convenience function form use at the gdb command line.

---
 scripts/gdb/utils.py       |   34 ++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    2 ++
 2 files changed, 36 insertions(+), 0 deletions(-)
scripts/gdb/utils.py +34 -0
index fdff85e..d1f6f12 100644
@@ -36,3 +36,37 @@ class CachedType:
 			gdb.events.new_objfile.connect(
 				self._new_objfile_handler)
 		return self._type
+
+
+long_type = CachedType("long")
+
+def get_long_type():
+	global long_type
+	return long_type.get_type()
+
+
+def offset_of(typeobj, field):
+	element = gdb.Value(0).cast(typeobj)
+	return int(str(element[field].address).split()[0], 16)
+
+def container_of(ptr, typeobj, member):
+	return (ptr.cast(get_long_type()) -
+		offset_of(typeobj, member)).cast(typeobj)
+
+
+class ContainerOf(gdb.Function):
+	__doc__ = "Return pointer to containing data structure.\n" \
+		  "\n" \
+		  "$container_of(PTR, \"TYPE\", \"ELEMENT\"): Given PTR, return a pointer to the\n" \
+		  "data structure of the type TYPE in which PTR is the address of ELEMENT.\n" \
+		  "Note that TYPE and ELEMENT have to be quoted as strings."
+
+	def __init__(self):
+		super(ContainerOf, self).__init__("container_of")
+
+	def invoke(self, ptr, typename, elementname):
+		return container_of(ptr,
+				    gdb.lookup_type(typename.string()).pointer(),
+				    elementname.string())
+
+ContainerOf()
scripts/gdb/vmlinux-gdb.py +2 -0
index bcb45cc..8ef76c9 100644
@@ -20,3 +20,5 @@ from utils import gdb_version
 if gdb_version < "7.1":
 	print "NOTE: gdb 7.1 or later required for Linux helper scripts " \
 	      "to work."
+else:
+	import utils
Signature · 1 line
1.7.3.4
author maintainer
#5 lore raw

PATCH v5 4/20 scripts/gdb: Add module iteration helper

In reply to Jan Kiszka

Will soon be used for loading symbols, printing global variables or listing modules.

---
 scripts/gdb/module.py |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/module.py
scripts/gdb/module.py new file +28 -0
new file mode 100644
index 0000000..01b23df
@@ -0,0 +1,28 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  module tools
+#
+# Copyright (c) Siemens AG, 2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+
+module_type = CachedType("struct module")
+
+def for_each_module(func, arg = None):
+	global module_type
+	module_ptr_type = module_type.get_type().pointer()
+	modules = gdb.parse_and_eval("modules")
+	entry = modules['next']
+	while entry != modules.address:
+		module = container_of(entry, module_ptr_type, "list")
+		func(module, arg)
+		entry = entry['next']
Signature · 1 line
1.7.3.4
author maintainer
#6 lore raw

PATCH v5 5/20 scripts/gdb: Add lx-symbols command

In reply to Jan Kiszka

This is probably the most useful helper when debugging kernel modules:
lx-symbols will first reload vmlinux. Then it searches recursively for *.ko files in the specified paths and the current directory. Finally it walks the kernel's module list, issuing the necessary add-symbol-file command for each loaded module so that gdb know which module symbol correspond to which address.

In addition, the command installs a silent breakpoint in the load_module function at the point where the module was loaded but not yet initialized. The breakpoint handler will try to load symbols from the module files found during lx-symbols execution. This way, breakpoints can be set to module initialization functions, and there is usually no need to explicitly call lx-symbols after (re-)loading a module.

---
 scripts/gdb/symbols.py     |  147 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 148 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/symbols.py
scripts/gdb/symbols.py new file +147 -0
new file mode 100644
index 0000000..8b672e2
@@ -0,0 +1,147 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  load kernel and module symbols
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+import os
+import re
+import string
+
+from module import for_each_module
+from utils import *
+
+class LinuxSymbols(gdb.Command):
+	__doc__ = "(Re-)load symbols of Linux kernel and currently loaded modules.\n" \
+		  "\n" \
+		  "The kernel (vmlinux) is taken from the current working directly. Modules (.ko)\n" \
+		  "are scanned recursively, starting in the same directory. Optionally, the module\n" \
+		  "search path can be extended by a space separated list of paths passed to the\n" \
+		  "lx-symbols command."
+
+	module_files = []
+	breakpoint = None
+
+	def __init__(self):
+		super(LinuxSymbols, self).__init__("lx-symbols",
+						   gdb.COMMAND_FILES,
+						   gdb.COMPLETE_FILENAME)
+
+	def _load_module_symbols(self, module, arg = None):
+		module_name = module['name'].string()
+		module_addr = str(module['module_core']).split()[0]
+		module_pattern = ".*/" + \
+			string.replace(module_name, "_", r"[_\-]") + r"\.ko$"
+		module_path = ""
+		for name in self.module_files:
+			if re.match(module_pattern, name):
+				module_path = name
+				break
+
+		if module_path:
+			print "loading @" + module_addr + ": " + module_path
+			if gdb.VERSION >= "7.2":
+				gdb.execute("add-symbol-file " + \
+					    module_path + " " + module_addr,
+					    to_string = True)
+			else:
+				gdb.execute("add-symbol-file " + \
+					    module_path + " " + module_addr)
+		else:
+			print "no module object found for '" + \
+			      module_name + "'"
+
+	if gdb_version >= "7.3":
+		class _LoadModuleBreakpoint(gdb.Breakpoint):
+			def __init__(self, spec, gdb_command):
+				super(LinuxSymbols._LoadModuleBreakpoint,
+				      self).__init__(spec, internal = True)
+				self.silent = True
+				self.gdb_command = gdb_command
+
+			def stop(self):
+				module = gdb.parse_and_eval("mod")
+				self.gdb_command._load_module_symbols(module)
+				return False
+
+		def _find_breakpoint_location(self):
+			breakpoint_match = "^[0-9]*[\t]*err = parse_args\(.*"
+
+			src = gdb.execute("list kernel/module.c:load_module",
+					  to_string = True)
+			done = False
+			lineno = None
+			while not done:
+				src = gdb.execute("list", to_string = True)
+				for line in src.splitlines():
+					if re.match(breakpoint_match, line):
+						done = True
+						lineno = line.split()[0]
+						break
+					elif re.match("^[0-9]*\t}$", line):
+						done = True
+						break
+			return lineno
+
+	def _load_all_module_symbols(self, arg):
+		if gdb.parse_and_eval("modules.next == &modules"):
+			print "no modules found"
+			return
+
+		self.module_files = []
+		paths = arg.split()
+		paths.append(os.getcwd())
+		for path in paths:
+			print "scanning for modules in " + path
+			for root, dirs, files in os.walk(path):
+				for name in files:
+					if re.match(r".*\.ko$", name):
+						self.module_files.append(
+							root + "/" + name)
+
+		for_each_module(self._load_module_symbols)
+
+	def invoke(self, arg, from_tty):
+		print "loading vmlinux"
+
+		saved_states = []
+		if (gdb.breakpoints() != None):
+			for breakpoint in gdb.breakpoints():
+				saved_states.append({
+					'breakpoint': breakpoint,
+					'enabled': breakpoint.enabled })
+
+		# drop all current symbols and reload vmlinux
+		gdb.execute("symbol-file", to_string = True)
+		gdb.execute("symbol-file vmlinux")
+
+		self._load_all_module_symbols(arg)
+
+		for saved_state in saved_states:
+			saved_state['breakpoint'].enabled = \
+				saved_state['enabled']
+
+		if gdb_version < "7.3":
+			print "Note: symbol update on module loading not " \
+			      "supported with this gdb version"
+		else:
+			if self.breakpoint != None:
+				self.breakpoint.delete()
+				self.breakpoint = None
+			lineno = self._find_breakpoint_location()
+			if lineno != None:
+				self.breakpoint = self._LoadModuleBreakpoint(
+					lineno, self)
+			else:
+				print "unable to detect breakpoint location " \
+				      "of load module completion"
+
+LinuxSymbols()
scripts/gdb/vmlinux-gdb.py +1 -0
index 8ef76c9..3553b7d 100644
@@ -22,3 +22,4 @@ if gdb_version < "7.1":
 	      "to work."
 else:
 	import utils
+	import symbols
Signature · 1 line
1.7.3.4
#7 lore raw

Re: [PATCH v5 05/20] scripts/gdb: Add lx-symbols command

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jan" == Jan Kiszka <[email protected]> writes:

Jan> In addition, the command installs a silent breakpoint in the load_module Jan> function at the point where the module was loaded but not yet Jan> initialized. The breakpoint handler will try to load symbols from the Jan> module files found during lx-symbols execution. This way, breakpoints Jan> can be set to module initialization functions, and there is usually no Jan> need to explicitly call lx-symbols after (re-)loading a module.

It is a nice approach, but I wonder whether the kernel's gdb stub could use the existing qXfer:libraries remote protocol packet. And, if not, could we extend gdb to make it work.

Jan> + def _find_breakpoint_location(self):
Jan> + breakpoint_match = "^[0-9]*[\t]*err = parse_args\(.*"
Jan> +
Jan> + src = gdb.execute("list kernel/module.c:load_module",
Jan> + to_string = True)

Whatever works -- but I think there are better ways.

The simplest is introducing a function that is called at the right spot with the right arguments. It doesn't need to do anything, just be a name where you can put a breakpoint.

Jan> + import symbols

I think it's better to put everything into its own package, e.g. 'import linux.symbols', to try to avoid conflicts with other python modules that may get loaded.

Tom

author maintainer
#8 lore raw

Re: [PATCH v5 05/20] scripts/gdb: Add lx-symbols command

Quoted 9 lines · Tom Tromey
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jan" == Jan Kiszka <[email protected]> writes:

Jan> In addition, the command installs a silent breakpoint in the load_module Jan> function at the point where the module was loaded but not yet Jan> initialized. The breakpoint handler will try to load symbols from the Jan> module files found during lx-symbols execution. This way, breakpoints Jan> can be set to module initialization functions, and there is usually no Jan> need to explicitly call lx-symbols after (re-)loading a module.

Quoted 3 lines

It is a nice approach, but I wonder whether the kernel's gdb stub could use the existing qXfer:libraries remote protocol packet. And, if not, could we extend gdb to make it work.

That would help kgdb, but not QEMU or hardware debuggers. They have no "Linux awareness" built in.

Quoted 7 lines

Jan> + def _find_breakpoint_location(self):
Jan> + breakpoint_match = "^[0-9]*[\t]*err = parse_args\(.*"
Jan> +
Jan> + src = gdb.execute("list kernel/module.c:load_module",
Jan> + to_string = True)

Quoted 5 lines

Whatever works -- but I think there are better ways.

The simplest is introducing a function that is called at the right spot with the right arguments. It doesn't need to do anything, just be a name where you can put a breakpoint.

Yes, I have this on my todo list. I already have a local config here where the current heuristic broke - once again. We likely just need to enforce un-inlining of do_init_module.

Quoted 6 lines

Jan> + import symbols

I think it's better to put everything into its own package, e.g. 'import linux.symbols', to try to avoid conflicts with other python modules that may get loaded.

OK.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
author maintainer
#9 lore raw

PATCH v5 6/20 scripts/gdb: Add internal helper and convenience function to look up a module

In reply to Jan Kiszka

Add the internal helper get_module_by_name to obtain the module structure corresponding to the given name. Also export this service as a convenience function.

---
 scripts/gdb/module.py      |   30 ++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 31 insertions(+), 0 deletions(-)
scripts/gdb/module.py +30 -0
index 01b23df..e309c0a 100644
@@ -26,3 +26,33 @@ def for_each_module(func, arg = None):
 		module = container_of(entry, module_ptr_type, "list")
 		func(module, arg)
 		entry = entry['next']
+
+def find_module_by_name(name):
+	def match_name(module, arg):
+		if module['name'].string() == arg['name']:
+			arg['module'] = module
+
+	arg = { 'name': name, 'module': None }
+	for_each_module(match_name, arg)
+
+	return arg['module']
+
+
+class LxModule(gdb.Function):
+	__doc__ = "Find module by name and return the module variable.\n" \
+		  "\n" \
+		  "$lx_module(MODULE): Given the name MODULE, iterate over all loaded modules of\n" \
+		  "the target and return that module variable which MODULE matches."
+
+	def __init__(self):
+		super(LxModule, self).__init__("lx_module")
+
+	def invoke(self, mod_name):
+		mod_name = mod_name.string()
+		module = find_module_by_name(mod_name)
+		if module:
+			return module.dereference()
+		else:
+			raise gdb.GdbError("Unable to find MODULE " + mod_name)
+
+LxModule()
scripts/gdb/vmlinux-gdb.py +1 -0
index 3553b7d..2a466a0 100644
@@ -23,3 +23,4 @@ if gdb_version < "7.1":
 else:
 	import utils
 	import symbols
+	import module
Signature · 1 line
1.7.3.4
author maintainer
#10 lore raw

PATCH v5 7/20 scripts/gdb: Add lx_modvar convenience function

In reply to Jan Kiszka

This function looks up and returns global module variables. Gdb is only aware of their types and section offsets, not their absolute addresses. The function either searches the symbol table of a specified module or it derives the module name from the object file of the current frame.

---
 scripts/gdb/module.py |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/utils.py  |    7 +++++++
 2 files changed, 55 insertions(+), 0 deletions(-)
scripts/gdb/module.py +48 -0
index e309c0a..333729a 100644
@@ -12,6 +12,8 @@
 #
 
 import gdb
+import os
+import string
 
 from utils import *
 
@@ -56,3 +58,49 @@ class LxModule(gdb.Function):
 			raise gdb.GdbError("Unable to find MODULE " + mod_name)
 
 LxModule()
+
+
+class LxModvar(gdb.Function):
+	__doc__ = "Return global variable of a module.\n" \
+		  "\n" \
+		  "$lx_modvar(\"VAR\"[, \"MODULE\"]): Return the global variable called VAR that is\n" \
+		  "defined by given MODULE. If MODULE is omitted, the current frame is used to\n" \
+		  "try finding the corresponding module name."
+
+	def __init__(self):
+		super(LxModvar, self).__init__("lx_modvar")
+
+	def _lookup_mod_symbol(self, module, var_name):
+		char_ptr_type = get_char_type().pointer()
+		for i in range(0, int(module['num_symtab'])):
+			symtab_entry = module['symtab'][i]
+			idx = int(symtab_entry['st_name'])
+			synname_addr = module['strtab'][idx].address
+			symname = synname_addr.cast(char_ptr_type)
+			if symname.string() == var_name:
+				return symtab_entry['st_value']
+		return None
+
+	def invoke(self, var_name, mod_name = None):
+		if (mod_name == None):
+			obj = gdb.selected_frame().function().symtab.objfile
+			mod_name = string.replace(
+				os.path.basename(obj.filename), ".ko", "")
+			module = find_module_by_name(mod_name)
+			if module == None:
+				raise gdb.GdbError("Current frame does not " \
+						   "belong to a module")
+		else:
+			mod_name = mod_name.string()
+			module = find_module_by_name(mod_name)
+			if module == None:
+				raise gdb.GdbError("Unable to find MODULE " +
+						   mod_name)
+		var_name = var_name.string()
+		var_addr = self._lookup_mod_symbol(module, var_name)
+		if var_addr == None:
+			raise gdb.GdbError("Unable to find VAR " + var_name)
+		var = gdb.parse_and_eval(var_name)
+		return var_addr.cast(var.type.pointer()).dereference()
+
+LxModvar()
scripts/gdb/utils.py +7 -0
index d1f6f12..13f903a 100644
@@ -45,6 +45,13 @@ def get_long_type():
 	return long_type.get_type()
 
 
+char_type = CachedType("char")
+
+def get_char_type():
+	global char_type
+	return char_type.get_type()
+
+
 def offset_of(typeobj, field):
 	element = gdb.Value(0).cast(typeobj)
 	return int(str(element[field].address).split()[0], 16)
Signature · 1 line
1.7.3.4
author maintainer
#11 lore raw

PATCH v5 8/20 scripts/gdb: Add get_target_endianness helper

In reply to Jan Kiszka

Parse the target endianness from the output of "show endian" and cache the result to return it via the new helper get_target_endiannes. We will need it for reading integers from buffers that contain target memory.

---
 scripts/gdb/utils.py |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)
scripts/gdb/utils.py +17 -0
index 13f903a..1c7cb42 100644
@@ -77,3 +77,20 @@ class ContainerOf(gdb.Function):
 				    elementname.string())
 
 ContainerOf()
+
+
+BIG_ENDIAN = 0
+LITTLE_ENDIAN = 1
+target_endianness = None
+
+def get_target_endianness():
+	global target_endianness
+	if target_endianness == None:
+		endian = gdb.execute("show endian", False, True)
+		if endian.find("little endian") >= 0:
+			target_endianness = LITTLE_ENDIAN
+		elif endian.find("big endian") >= 0:
+			target_endianness = BIG_ENDIAN
+		else:
+			raise gdb.GdgError("unknown endianness '%s'" % endian)
+	return target_endianness
Signature · 1 line
1.7.3.4
author maintainer
#12 lore raw

PATCH v5 9/20 scripts/gdb: Add read_u16/32/64 helpers

In reply to Jan Kiszka

Add helpers for reading integers from target memory buffers. Required when caching the memory access is more efficient than reading individual values via gdb.

---
 scripts/gdb/utils.py |   18 ++++++++++++++++++
 1 files changed, 18 insertions(+), 0 deletions(-)
scripts/gdb/utils.py +18 -0
index 1c7cb42..9b23052 100644
@@ -94,3 +94,21 @@ def get_target_endianness():
 		else:
 			raise gdb.GdgError("unknown endianness '%s'" % endian)
 	return target_endianness
+
+def read_u16(buffer):
+	if get_target_endianness() == LITTLE_ENDIAN:
+		return ord(buffer[0]) + (ord(buffer[1]) << 8)
+	else:
+		return ord(buffer[1]) + (ord(buffer[0]) << 8)
+
+def read_u32(buffer):
+	if get_target_endianness() == LITTLE_ENDIAN:
+		return read_u16(buffer[0:2]) + (read_u16(buffer[2:4]) << 16)
+	else:
+		return read_u16(buffer[2:4]) + (read_u16(buffer[0:2]) << 16)
+
+def read_u64(buffer):
+	if get_target_endianness() == LITTLE_ENDIAN:
+		return read_u32(buffer[0:4]) + (read_u32(buffer[4:8]) << 32)
+	else:
+		return read_u32(buffer[4:8]) + (read_u32(buffer[0:4]) << 32)
Signature · 1 line
1.7.3.4
author maintainer
#13 lore raw

PATCH v5 10/20 scripts/gdb: Add lx-dmesg command

In reply to Jan Kiszka

This pokes into the log buffer of the debugged kernel, dumping it to the gdb console. Helping in case the target should or can no longer execute dmesg itself.

---
 scripts/gdb/dmesg.py       |   63 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 64 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/dmesg.py
scripts/gdb/dmesg.py new file +63 -0
new file mode 100644
index 0000000..1bc3368
@@ -0,0 +1,63 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  kernel log buffer dump
+#
+# Copyright (c) Siemens AG, 2011, 2012
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+import string
+
+from utils import *
+
+class LinuxDmesg(gdb.Command):
+	__doc__ = "Print Linux kernel log buffer."
+
+	def __init__(self):
+		super(LinuxDmesg, self).__init__("lx-dmesg", gdb.COMMAND_DATA)
+
+	def invoke(self, arg, from_tty):
+		log_buf_addr = \
+			int(str(gdb.parse_and_eval("log_buf")).split()[0], 16)
+		log_first_idx = int(gdb.parse_and_eval("log_first_idx"))
+		log_next_idx = int(gdb.parse_and_eval("log_next_idx"))
+		log_buf_len = int(gdb.parse_and_eval("log_buf_len"))
+
+		inf = gdb.inferiors()[0]
+		start = log_buf_addr + log_first_idx
+		if log_first_idx < log_next_idx:
+			log_buf_2nd_half = -1
+			length = log_next_idx - log_first_idx
+			log_buf = inf.read_memory(start, length)
+		else:
+			log_buf_2nd_half = log_buf_len - log_first_idx
+			log_buf = inf.read_memory(start, log_buf_2nd_half) + \
+				  inf.read_memory(log_buf_addr, log_next_idx)
+
+		pos = 0
+		while pos < log_buf.__len__():
+			length = read_u16(log_buf[pos + 8 : pos + 10])
+			if length == 0:
+				if log_buf_2nd_half == -1:
+					print "Corrupted log buffer!"
+					break
+				pos = log_buf_2nd_half
+				continue
+
+			text_len = read_u16(log_buf[pos + 10 : pos + 12])
+			time_stamp = read_u64(log_buf[pos : pos + 8])
+
+			for line in log_buf[pos + 16 :
+					    pos + 16 + text_len].splitlines():
+				print "[%13.6f] " % \
+				      (time_stamp / 1000000000.0) + line
+
+			pos += length
+
+LinuxDmesg()
scripts/gdb/vmlinux-gdb.py +1 -0
index 2a466a0..4a5685b 100644
@@ -24,3 +24,4 @@ else:
 	import utils
 	import symbols
 	import module
+	import dmesg
Signature · 1 line
1.7.3.4
author maintainer
#14 lore raw

PATCH v5 11/20 scripts/gdb: Add task iteration helper

In reply to Jan Kiszka

The internal helper for_each_task iterates over all tasks of the target, calling the provided function on each.

---
 scripts/gdb/task.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/task.py
scripts/gdb/task.py new file +35 -0
new file mode 100644
index 0000000..6b73dbf
@@ -0,0 +1,35 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  task & thread tools
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+
+task_type = CachedType("struct task_struct")
+
+def for_each_task(func, arg = None):
+	global task_type
+	task_ptr_type = task_type.get_type().pointer()
+	init_task = gdb.parse_and_eval("init_task")
+	g = init_task.address
+	while True:
+		g =  container_of(g['tasks']['next'], task_ptr_type, "tasks")
+		if g == init_task.address:
+			break;
+		t = g
+		while True:
+			func(t, arg)
+			t = container_of(t['thread_group']['next'],
+					 task_ptr_type, "thread_group")
+			if t == g:
+				break
Signature · 1 line
1.7.3.4
author maintainer
#15 lore raw

PATCH v5 12/20 scripts/gdb: Add helper and convenience function to look up tasks

In reply to Jan Kiszka

Add the helper task_by_pid that can look up a task by its PID. Also export it as a convenience function.

---
 scripts/gdb/task.py        |   29 +++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 30 insertions(+), 0 deletions(-)
scripts/gdb/task.py +29 -0
index 6b73dbf..fe42eb0 100644
@@ -33,3 +33,32 @@ def for_each_task(func, arg = None):
 					 task_ptr_type, "thread_group")
 			if t == g:
 				break
+
+def get_task_by_pid(pid):
+	def match_pid(t, arg):
+		if int(t['pid']) == arg['pid']:
+			arg['task'] = t
+
+	arg = { 'pid': pid, 'task': None }
+	for_each_task(match_pid, arg)
+
+	return arg['task']
+
+
+class LxTaskByPidFunc(gdb.Function):
+	__doc__ = "Find Linux task by PID and return the task_struct variable.\n" \
+		  "\n" \
+		  "$lx_task_by_pid(PID): Given PID, iterate over all tasks of the target and\n" \
+		  "return that task_struct variable which PID matches."
+
+	def __init__(self):
+		super(LxTaskByPidFunc, self).__init__("lx_task_by_pid")
+
+	def invoke(self, pid):
+		task = get_task_by_pid(pid)
+		if task:
+			return task.dereference()
+		else:
+			raise gdb.GdbError("No task of PID " + str(pid))
+
+LxTaskByPidFunc()
scripts/gdb/vmlinux-gdb.py +1 -0
index 4a5685b..e1c5dbc 100644
@@ -25,3 +25,4 @@ else:
 	import symbols
 	import module
 	import dmesg
+	import task
Signature · 1 line
1.7.3.4
author maintainer
#16 lore raw

PATCH v5 13/20 scripts/gdb: Add is_target_arch helper

In reply to Jan Kiszka

This helper caches to result of "show architecture" and matches the provided arch (sub-)string against that output.

---
 scripts/gdb/utils.py |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
scripts/gdb/utils.py +9 -0
index 9b23052..bcf4278 100644
@@ -112,3 +112,12 @@ def read_u64(buffer):
 		return read_u32(buffer[0:4]) + (read_u32(buffer[4:8]) << 32)
 	else:
 		return read_u32(buffer[4:8]) + (read_u32(buffer[0:4]) << 32)
+
+
+target_arch = None
+
+def is_target_arch(arch):
+	global target_arch
+	if target_arch == None:
+		target_arch = gdb.execute("show architecture", False, True)
+	return target_arch.find(arch) >= 0
Signature · 1 line
1.7.3.4
#17 lore raw

Re: [PATCH v5 13/20] scripts/gdb: Add is_target_arch helper

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jan" == Jan Kiszka <[email protected]> writes:

Jan> +def is_target_arch(arch):
Jan> + global target_arch
Jan> + if target_arch == None:
Jan> + target_arch = gdb.execute("show architecture", False, True)
Jan> + return target_arch.find(arch) >= 0

FYI - the next gdb will have a Frame.architecture method for getting the architecture of a frame. You might consider using it when it is available. "show architecture" is mildly unfortunate to cache given its "auto" behavior.

Tom

author maintainer
#18 lore raw

PATCH v5 14/20 scripts/gdb: Add internal helper and convenience function to retrieve thread_info

In reply to Jan Kiszka

Add the internal helper get_thread_info that calculated the thread_info from a given task variable. Also export this service as a convenience function.

Note: ia64 version is untested.

---
 scripts/gdb/task.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)
scripts/gdb/task.py +35 -0
index fe42eb0..8e96238 100644
@@ -62,3 +62,38 @@ class LxTaskByPidFunc(gdb.Function):
 			raise gdb.GdbError("No task of PID " + str(pid))
 
 LxTaskByPidFunc()
+
+
+thread_info_type = CachedType("struct thread_info")
+
+ia64_task_size = None
+
+def get_thread_info(task):
+	global thread_info_type
+	thread_info_ptr_type = thread_info_type.get_type().pointer()
+	if is_target_arch("ia64"):
+		global ia64_task_size
+		if ia64_task_size = None:
+			ia64_task_size = gdb.parse_and_eval(
+						"sizeof(struct task_struct)")
+		thread_info_addr = task.address + ia64_task_size
+		thread_info = thread_info_addr.cast(thread_info_ptr_type)
+	else:
+		thread_info = task['stack'].cast(thread_info_ptr_type)
+	return thread_info.dereference()
+
+
+class LxThreadInfoFunc (gdb.Function):
+	# Calculate Linux thread_info from task variable.
+	__doc__ = "Calculate Linux thread_info from task variable.\n" \
+		  "\n" \
+		  "$lx_thread_info(TASK): Given TASK, return the corresponding thread_info\n" \
+		  "variable."
+
+	def __init__(self):
+		super(LxThreadInfoFunc, self).__init__("lx_thread_info")
+
+	def invoke(self, task):
+		return get_thread_info(task)
+
+LxThreadInfoFunc()
Signature · 1 line
1.7.3.4
#19 lore raw

Re: [PATCH v5 14/20] scripts/gdb: Add internal helper and convenience function to retrieve thread_info

Quoted 11 lines · Jan Kiszka

Add the internal helper get_thread_info that calculated the thread_info from a given task variable. Also export this service as a convenience function.

Note: ia64 version is untested.

---

[ … ]

Quoted 4 lines
scripts/gdb/task.py +4 -0
+class LxThreadInfoFunc (gdb.Function):
+	# Calculate Linux thread_info from task variable.
+	__doc__ = "Calculate Linux thread_info from task variable.\n" \
+		  "\n" \

scripts/gdb/task.py class LxTaskByPidFunc(gdb.Function): · line 99

Quoted 10 lines
scripts/gdb/task.py +10 -0
+		  "$lx_thread_info(TASK): Given TASK, return the corresponding thread_info\n" \
+		  "variable."
+
+	def __init__(self):
+		super(LxThreadInfoFunc, self).__init__("lx_thread_info")
+
+	def invoke(self, task):
+		return get_thread_info(task)
+
+LxThreadInfoFunc()

Nice, you can plug commands into one-another:

(gdb) p $lx_thread_info($lx_current())
$12 = {task = 0xffffffff81a14440, exec_domain = 0xffffffff81a21e00, flags = 0, status = 0, cpu = 0,

  preempt_count = 1, addr_limit = {seg = 18446744073709551615}, restart_block = {
    fn = 0xffffffff81054cd0 <do_no_restart_syscall>, {futex = {uaddr = 0x0, val = 0, flags = 0, bitset = 0, 
        time = 0, uaddr2 = 0x0}, nanosleep = {clockid = 0, rmtp = 0x0, compat_rmtp = 0x0, expires = 0}, 
      poll = {ufds = 0x0, nfds = 0, has_timeout = 0, tv_sec = 0, tv_nsec = 0}}}, sysenter_return = 0x0, 

sig_on_uaccess_error = 0, uaccess_err = 0}

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#20 lore raw

PATCH v5 15/20 scripts/gdb: Add get_gdbserver_type helper

In reply to Jan Kiszka

This helper probes the type of the gdb server. Supported are QEMU and KGDB so far. Knowledge about the gdb server is required e.g. to retrieve the current CPU or current task.

---
 scripts/gdb/utils.py |   35 +++++++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)
scripts/gdb/utils.py +35 -0
index bcf4278..d381f63 100644
@@ -121,3 +121,38 @@ def is_target_arch(arch):
 	if target_arch == None:
 		target_arch = gdb.execute("show architecture", False, True)
 	return target_arch.find(arch) >= 0
+
+
+GDBSERVER_QEMU = 0
+GDBSERVER_KGDB = 1
+gdbserver_type = None
+
+def get_gdbserver_type():
+	def exit_handler(event):
+		global gdbserver_type
+		gdbserver_type = None
+		gdb.events.exited.disconnect(exit_handler)
+
+	def probe_qemu():
+		try:
+			return gdb.execute("monitor info version", False,
+					   True) != ""
+		except:
+			return False
+
+	def probe_kgdb():
+		try:
+			thread_info = gdb.execute("info thread 2", False, True)
+			return thread_info.find("shadowCPU0") >= 0
+		except:
+			return False
+
+	global gdbserver_type
+	if gdbserver_type == None:
+		if probe_qemu():
+			gdbserver_type = GDBSERVER_QEMU
+		elif probe_kgdb():
+			gdbserver_type = GDBSERVER_KGDB
+		if gdbserver_type != None:
+			gdb.events.exited.connect(exit_handler)
+	return gdbserver_type
Signature · 1 line
1.7.3.4
author maintainer
#21 lore raw

PATCH v5 16/20 scripts/gdb: Add internal helper and convenience function for per-cpu lookup

In reply to Jan Kiszka

This function allows to obtain a per-cpu variable, either of the current or an explicitly specified CPU.

Note: sparc64 version is untested.

---
 scripts/gdb/percpu.py      |   61 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 62 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/percpu.py
scripts/gdb/percpu.py new file +61 -0
new file mode 100644
index 0000000..864962c
@@ -0,0 +1,61 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  per-cpu tools
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+from task import *
+
+MAX_CPUS = 4096
+
+def get_current_cpu():
+	if get_gdbserver_type() = GDBSERVER_QEMU:
+		return gdb.selected_thread().num - 1
+	elif get_gdbserver_type() = GDBSERVER_KGDB:
+		tid = gdb.selected_thread().ptid[2]
+		if tid > (0x100000000 - MAX_CPUS - 2):
+			return 0x100000000 - tid - 2
+		else:
+			return get_thread_info(get_task_by_pid(tid))['cpu']
+	else:
+		raise gdb.GdbError("Sorry, obtaining the current CPU is "
+				   "not yet supported with this gdb server.")
+
+def per_cpu(var_ptr, cpu):
+	if cpu = -1:
+		cpu = get_current_cpu()
+	if is_target_arch("sparc:v9"):
+		offset = gdb.parse_and_eval("trap_block[" + str(cpu) +
+					    "].__per_cpu_base")
+	else:
+		offset = gdb.parse_and_eval("__per_cpu_offset[" + str(cpu) +
+					    "]")
+	pointer = var_ptr.cast(get_long_type()) + offset
+	return pointer.cast(var_ptr.type).dereference()
+
+
+class PerCpu(gdb.Function):
+	__doc__ = "Return per-cpu variable.\n" \
+		  "\n" \
+		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
+		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
+		  "Note that VAR has to be quoted as string."
+
+	def __init__(self):
+		super(PerCpu, self).__init__("lx_per_cpu")
+
+	def invoke(self, var_name, cpu = -1):
+		var_ptr = gdb.parse_and_eval("&" + var_name.string())
+		return per_cpu(var_ptr, cpu)
+
+PerCpu()
scripts/gdb/vmlinux-gdb.py +1 -0
index e1c5dbc..b143808 100644
@@ -26,3 +26,4 @@ else:
 	import module
 	import dmesg
 	import task
+	import percpu
Signature · 1 line
1.7.3.4
#22 lore raw

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

Quoted 20 lines · Jan Kiszka

This function allows to obtain a per-cpu variable, either of the current or an explicitly specified CPU.

Note: sparc64 version is untested.

---
 scripts/gdb/percpu.py      |   61 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 62 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/percpu.py
scripts/gdb/percpu.py new file +0 -0
new file mode 100644
index 0000000..864962c

scripts/gdb/percpu.py line 4

Quoted 4 lines
scripts/gdb/percpu.py new file +4 -0
@@ -0,0 +1,61 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  per-cpu tools

This is actually a very cool thing for CPU guys: it can show what kind of hybrid CPUs they've been glueing together, like the following, for example:

...
(gdb) p $lx_per_cpu("cpu_info").x86_virt_bits
$5 = 48 '0'
(gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
$6 = "GenuineIntel\000\000\000"
(gdb) p $lx_per_cpu("cpu_info").x86_model_id
$7 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times> (gdb)

Since when does Intel produce CPUs called "AMD Phenom(tm) 9550 Quad-Core Processor"? .. hahaha..

Quoted 38 lines
scripts/gdb/percpu.py new file +38 -0
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+from task import *
+
+MAX_CPUS = 4096
+
+def get_current_cpu():
+	if get_gdbserver_type() = GDBSERVER_QEMU:
+		return gdb.selected_thread().num - 1
+	elif get_gdbserver_type() = GDBSERVER_KGDB:
+		tid = gdb.selected_thread().ptid[2]
+		if tid > (0x100000000 - MAX_CPUS - 2):
+			return 0x100000000 - tid - 2
+		else:
+			return get_thread_info(get_task_by_pid(tid))['cpu']
+	else:
+		raise gdb.GdbError("Sorry, obtaining the current CPU is "
+				   "not yet supported with this gdb server.")
+
+def per_cpu(var_ptr, cpu):
+	if cpu = -1:
+		cpu = get_current_cpu()
+	if is_target_arch("sparc:v9"):
+		offset = gdb.parse_and_eval("trap_block[" + str(cpu) +
+					    "].__per_cpu_base")
+	else:
+		offset = gdb.parse_and_eval("__per_cpu_offset[" + str(cpu) +
+					    "]")

scripts/gdb/percpu.py line 52

Quoted 10 lines
scripts/gdb/percpu.py new file +10 -0
+	pointer = var_ptr.cast(get_long_type()) + offset
+	return pointer.cast(var_ptr.type).dereference()
+
+
+class PerCpu(gdb.Function):
+	__doc__ = "Return per-cpu variable.\n" \
+		  "\n" \
+		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
+		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
+		  "Note that VAR has to be quoted as string."

Ok, seriously now:

apropos shows the "Return per-cpu... " line above. Have you found out which gdb command shows the rest? help and info both say "Undefined command".

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#23 lore raw

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

Quoted 39 lines · Borislav Petkov
Quoted 20 lines · Jan Kiszka

This function allows to obtain a per-cpu variable, either of the current or an explicitly specified CPU.

Note: sparc64 version is untested.

---
 scripts/gdb/percpu.py      |   61 ++++++++++++++++++++++++++++++++++++++++++++
 scripts/gdb/vmlinux-gdb.py |    1 +
 2 files changed, 62 insertions(+), 0 deletions(-)
 create mode 100644 scripts/gdb/percpu.py
scripts/gdb/percpu.py new file +0 -0
new file mode 100644
index 0000000..864962c

scripts/gdb/percpu.py line 4

Quoted 4 lines
scripts/gdb/percpu.py new file +4 -0
@@ -0,0 +1,61 @@
+#
+# gdb helper commands and functions for Linux kernel debugging
+#
+#  per-cpu tools

This is actually a very cool thing for CPU guys: it can show what kind of hybrid CPUs they've been glueing together, like the following, for example:

...
(gdb) p $lx_per_cpu("cpu_info").x86_virt_bits
$5 = 48 '0'
(gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
$6 = "GenuineIntel\000\000\000"
(gdb) p $lx_per_cpu("cpu_info").x86_model_id
$7 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times> (gdb)

Quoted 2 lines

Since when does Intel produce CPUs called "AMD Phenom(tm) 9550 Quad-Core Processor"? .. hahaha..

Let me guess: You are dumping a weird QEMU/KVM CPU, right?

Quoted 53 lines
Quoted 39 lines
scripts/gdb/percpu.py new file +38 -0
+#
+# Copyright (c) Siemens AG, 2011-2013
+#
+# Authors:
+#  Jan Kiszka <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL version 2.
+#
+
+import gdb
+
+from utils import *
+from task import *
+
+MAX_CPUS = 4096
+
+def get_current_cpu():
+	if get_gdbserver_type() = GDBSERVER_QEMU:
+		return gdb.selected_thread().num - 1
+	elif get_gdbserver_type() = GDBSERVER_KGDB:
+		tid = gdb.selected_thread().ptid[2]
+		if tid > (0x100000000 - MAX_CPUS - 2):
+			return 0x100000000 - tid - 2
+		else:
+			return get_thread_info(get_task_by_pid(tid))['cpu']
+	else:
+		raise gdb.GdbError("Sorry, obtaining the current CPU is "
+				   "not yet supported with this gdb server.")
+
+def per_cpu(var_ptr, cpu):
+	if cpu = -1:
+		cpu = get_current_cpu()
+	if is_target_arch("sparc:v9"):
+		offset = gdb.parse_and_eval("trap_block[" + str(cpu) +
+					    "].__per_cpu_base")
+	else:
+		offset = gdb.parse_and_eval("__per_cpu_offset[" + str(cpu) +
+					    "]")

scripts/gdb/percpu.py line 52

Quoted 10 lines
scripts/gdb/percpu.py new file +10 -0
+	pointer = var_ptr.cast(get_long_type()) + offset
+	return pointer.cast(var_ptr.type).dereference()
+
+
+class PerCpu(gdb.Function):
+	__doc__ = "Return per-cpu variable.\n" \
+		  "\n" \
+		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
+		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
+		  "Note that VAR has to be quoted as string."

Ok, seriously now:

Quoted 3 lines

apropos shows the "Return per-cpu... " line above. Have you found out which gdb command shows the rest? help and info both say "Undefined command".

help function lx_...

It took me a while to find this, too.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
#24 lore raw

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

Quoted 1 line · Jan Kiszka

Let me guess: You are dumping a weird QEMU/KVM CPU, right?

Nah, I actually have the silicon! :-)

Joking, of course. I wish. I'm booting the guest with -cpu phenom (it has been like that since forever in my boot-kernel-in-kvm script) but the host is Intel.

If I do this on an AMD host, all is ok:

(gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
$1 = "AuthenticAMD\000\000\000"
(gdb) p $lx_per_cpu("cpu_info").x86_model_id
$2 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times> (gdb)

[ … ]

Quoted 10 lines
Quoted 10 lines

scripts/gdb/percpu.py line 52

Quoted 6 lines
scripts/gdb/percpu.py new file +6 -0
+class PerCpu(gdb.Function):
+	__doc__ = "Return per-cpu variable.\n" \
+		  "\n" \
+		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
+		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
+		  "Note that VAR has to be quoted as string."

Ok, seriously now:

Quoted 7 lines
Quoted 3 lines

apropos shows the "Return per-cpu... " line above. Have you found out which gdb command shows the rest? help and info both say "Undefined command".

help function lx_...

It took me a while to find this, too.

Maybe worth documenting it then at the end of gdb-kernel-debugging.txt?

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author maintainer
#25 lore raw

Re: [PATCH v5 16/20] scripts/gdb: Add internal helper and convenience function for per-cpu lookup

Quoted 9 lines · Borislav Petkov
Quoted 1 line · Jan Kiszka

Let me guess: You are dumping a weird QEMU/KVM CPU, right?

Nah, I actually have the silicon! :-)

Joking, of course. I wish. I'm booting the guest with -cpu phenom (it has been like that since forever in my boot-kernel-in-kvm script) but the host is Intel.

Quoted 7 lines

If I do this on an AMD host, all is ok:

(gdb) p $lx_per_cpu("cpu_info").x86_vendor_id
$1 = "AuthenticAMD\000\000\000"
(gdb) p $lx_per_cpu("cpu_info").x86_model_id
$2 = "AMD Phenom(tm) 9550 Quad-Core Processor", '\000' <repeats 24 times> (gdb)

Remains a bug of QEMU, though possibly a minor one.

Quoted 17 lines

[ … ]

Quoted 14 lines
Quoted 13 lines

scripts/gdb/percpu.py line 52

Quoted 6 lines
scripts/gdb/percpu.py new file +6 -0
+class PerCpu(gdb.Function):
+	__doc__ = "Return per-cpu variable.\n" \
+		  "\n" \
+		  "$lx_per_cpu(\"VAR\"[, CPU]): Return the per-cpu variable called VAR for the\n" \
+		  "given CPU number. If CPU is omitted, the CPU of the current context is used.\n" \
+		  "Note that VAR has to be quoted as string."

Ok, seriously now:

apropos shows the "Return per-cpu... " line above. Have you found out which gdb command shows the rest? help and info both say "Undefined command".

Quoted 6 lines
Quoted 3 lines

help function lx_...

It took me a while to find this, too.

Maybe worth documenting it then at the end of gdb-kernel-debugging.txt?

Done. Will come with v6 (if needed), otherwise as an add-on patch.

Thanks,
Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE
Corporate Competence Center Embedded Linux
author maintainer
#26 lore raw

PATCH v5 17/20 scripts/gdb: Add lx_current convenience function

In reply to Jan Kiszka

This is a shorthand for *$lx_per_cpu("current_task"), i.e. a convenience function to retrieve the currently running task of the active context.

---
 scripts/gdb/percpu.py |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)
scripts/gdb/percpu.py +16 -0
index 864962c..4dab8d3 100644
@@ -59,3 +59,19 @@ class PerCpu(gdb.Function):
 		return per_cpu(var_ptr, cpu)
 
 PerCpu()
+
+
+class LxCurrentFunc(gdb.Function):
+	__doc__ = "Return current task.\n" \
+		  "\n" \
+		  "$lx_current([CPU]): Return the per-cpu task variable for the given CPU\n" \
+		  "number. If CPU is omitted, the CPU of the current context is used."
+
+	def __init__(self):
+		super(LxCurrentFunc, self).__init__("lx_current")
+
+	def invoke(self, cpu = -1):
+		var_ptr = gdb.parse_and_eval("&current_task")
+		return per_cpu(var_ptr, cpu).dereference()
+
+LxCurrentFunc()
Signature · 1 line
1.7.3.4
author maintainer
#27 lore raw

PATCH v5 18/20 scripts/gdb: Add helper to iterate over CPU masks

In reply to Jan Kiszka

Will be used first to count module references. It is optimized to read the mask only once per stop and to minimize the loop lengths.

---
 scripts/gdb/percpu.py |   33 +++++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)
scripts/gdb/percpu.py +33 -0
index 4dab8d3..bc04188 100644
@@ -43,6 +43,39 @@ def per_cpu(var_ptr, cpu):
 	pointer = var_ptr.cast(get_long_type()) + offset
 	return pointer.cast(var_ptr.type).dereference()
 
+cpu_mask = { }
+
+def for_each_cpu(mask_name, func, arg = None):
+	def invalidate_handler(event):
+		global cpu_online_mask
+		cpu_mask = { }
+		gdb.events.stop.disconnect(invalidate_handler)
+		gdb.events.new_objfile.disconnect(invalidate_handler)
+
+	global cpu_mask
+	mask = None
+	if mask_name in cpu_mask:
+		mask = cpu_mask[mask_name]
+	if mask == None:
+		mask = gdb.parse_and_eval(mask_name + ".bits")
+		cpu_mask[mask_name] = mask
+		gdb.events.stop.connect(invalidate_handler)
+		gdb.events.new_objfile.connect(invalidate_handler)
+
+	max_cpu_id = mask.type.sizeof * 8
+	bits_per_entry = mask[0].type.sizeof * 8
+	for entry in range(max_cpu_id / bits_per_entry):
+		bits = mask[entry]
+		if bits == 0:
+			continue
+		for bit in range(bits_per_entry):
+			if bits & 1:
+				cpu = entry * bits_per_entry + bit
+				func(cpu, arg)
+			bits >>= 1
+			if bits == 0:
+				break
+
 
 class PerCpu(gdb.Function):
 	__doc__ = "Return per-cpu variable.\n" \
Signature · 1 line
1.7.3.4
author maintainer
#28 lore raw

PATCH v5 19/20 scripts/gdb: Add lx-lsmod command

In reply to Jan Kiszka

This adds a lsmod-like command to list all currently loaded modules of the target.

---
 scripts/gdb/module.py |   43 +++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 43 insertions(+), 0 deletions(-)
scripts/gdb/module.py +43 -0
index 333729a..101d2e0 100644
@@ -15,6 +15,7 @@ import gdb
 import os
 import string
 
+from percpu import *
 from utils import *
 
 module_type = CachedType("struct module")
@@ -104,3 +105,45 @@ class LxModvar(gdb.Function):
 		return var_addr.cast(var.type.pointer()).dereference()
 
 LxModvar()
+
+
+class LinuxLsmod(gdb.Command):
+	__doc__ = "List currently loaded modules."
+
+	_module_use_type = CachedType("struct module_use")
+
+	def __init__(self):
+		super(LinuxLsmod, self).__init__("lx-lsmod", gdb.COMMAND_DATA)
+
+	def invoke(self, arg, from_tty):
+		def print_module(module, arg):
+			def collect_ref(cpu, arg):
+				refptr = per_cpu(arg['refptr'], cpu)
+				arg['ref'] += refptr['incs']
+				arg['ref'] -= refptr['decs']
+
+			arg = { 'refptr': module['refptr'], 'ref': 0 }
+			for_each_cpu("cpu_possible_mask", collect_ref, arg)
+
+			print "%s" % module['module_core'] + \
+			      " %-19s" % module['name'].string() + \
+			      " %8s" % module['core_size'] + \
+			      "  %d" % arg['ref'],
+
+			source_list = module['source_list']
+			t = self._module_use_type.get_type().pointer()
+			entry = source_list['next']
+			first = True
+			while entry != source_list.address:
+				use = container_of(entry, t, "source_list")
+				gdb.write((" " if first else ",") + \
+					  use['source']['name'].string())
+				first = False
+				entry = entry['next']
+			print
+
+		print "Address%s    Module                  Size  Used by" % \
+		      "        " if get_long_type().sizeof == 8 else ""
+		for_each_module(print_module)
+
+LinuxLsmod()
Signature · 1 line
1.7.3.4
author maintainer
#29 lore raw

PATCH v5 20/20 scripts/gdb: Add basic documentation

In reply to Jan Kiszka
---
 Documentation/gdb-kernel-debugging.txt |  155 ++++++++++++++++++++++++++++++++
 1 files changed, 155 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt
Documentation/gdb-kernel-debugging.txt new file +155 -0
new file mode 100644
index 0000000..0ea46e1
@@ -0,0 +1,155 @@
+Debugging kernel and modules via gdb
+====================================
+
+The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
+interfaces allow to debug the Linux kernel and its modules during runtime
+using gdb. Gdb comes with a powerful scripting interface for python. The
+kernel provides a collection of helper scripts that can simplify typical
+kernel debugging steps. This is a short tutorial about how to enable and use
+them. It focuses on QEMU/KVM virtual machines as target, but the examples can
+be transferred to the other gdb stubs as well.
+
+
+Requirements
+------------
+
+ o gdb 7.1+ (recommended: 7.3+) with python support enabled (typically true
+   for distributions)
+
+
+Setup
+-----
+
+ o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
+   www.qemu.org for more details)
+
+ o Build the kernel with CONFIG_DEBUG_INFO enabled, but leave
+   CONFIG_DEBUG_INFO_REDUCED off
+
+ o Install that kernel on the guest.
+
+   Alternatively, QEMU allows to boot the kernel directly using -kernel,
+   -append, -initrd command line switches. This is generally only useful if
+   you do not depend on modules. See QEMU documentation for more details on
+   this mode.
+
+ o Enable the gdb stub of QEMU/KVM, either
+    - at VM startup time by appending "-s" to the QEMU command line
+   or
+    - during runtime by issuing "gdbserver" from the QEMU monitor
+      console
+
+ o cd /path/to/linux-build
+
+ o Start gdb: gdb vmlinux
+
+   Note: Some distros may restrict auto-loading of gdb scripts to known safe
+   directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
+
+    add-add-auto-load-safe-path /path/to/linux-build
+
+   to ~/.gdbinit. See gdb help for more details.
+
+ o Attach to the booted guest:
+    (gdb) target remote :1234
+
+
+Examples of using the Linux-provided gdb helpers
+------------------------------------------------
+
+ o Load module (and main kernel) symbols:
+    (gdb) lx-symbols
+    loading vmlinux
+    scanning for modules in /home/user/linux/build
+    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
+    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
+    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
+    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
+    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
+    ...
+    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
+
+ o Set a breakpoint on some not yet loaded module function, e.g.:
+    (gdb) b btrfs_init_sysfs
+    Function "btrfs_init_sysfs" not defined.
+    Make breakpoint pending on future shared library load? (y or [n]) y
+    Breakpoint 1 (btrfs_init_sysfs) pending.
+
+ o Continue the target
+    (gdb) c
+
+ o Load the module on the target and watch the symbols being loaded as well as
+   the breakpoint hit:
+    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
+    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
+    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
+    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
+
+    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
+    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
+
+ o Dump the log buffer of the target kernel:
+    (gdb) lx-dmesg
+    [     0.000000] Initializing cgroup subsys cpuset
+    [     0.000000] Initializing cgroup subsys cpu
+    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
+    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
+    [     0.000000] e820: BIOS-provided physical RAM map:
+    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
+    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
+    ....
+
+ o Examine fields of the current task struct:
+    (gdb) p $lx_current().pid
+    $1 = 4998
+    (gdb) p $lx_current().comm
+    $2 = "modprobe\000\000\000\000\000\000\000"
+
+ o Make use of the per-cpu helper for the current or a specified CPU:
+    (gdb) p $lx_per_cpu("runqueues").nr_running
+    $3 = 1
+    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
+    $4 = 0
+
+ o Dig into hrtimers using the container_of helper:
+    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
+    (gdb) p *$container_of($next, "struct hrtimer", "node")
+    $5 = {
+      node = {
+        node = {
+          __rb_parent_color = 18446612133355256072,
+          rb_right = 0x0 <irq_stack_union>,
+          rb_left = 0x0 <irq_stack_union>
+        },
+        expires = {
+          tv64 = 1835268000000
+        }
+      },
+      _softexpires = {
+        tv64 = 1835268000000
+      },
+      function = 0xffffffff81078232 <tick_sched_timer>,
+      base = 0xffff88003fd0d6f0,
+      state = 1,
+      start_pid = 0,
+      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
+      start_comm = "swapper/2\000\000\000\000\000\000"
+    }
+
+
+List of commands and helper
+---------------------------
+
+The number of commands and convenience helpers may evolve over the time, this
+is just a snapshot of the initial version:
+
+ (gdb) apropos lx
+ function lx_current -- Return current task
+ function lx_module -- Find module by name and return the module variable
+ function lx_modvar -- Return global variable of a module
+ function lx_per_cpu -- Return per-cpu variable
+ function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
+ function lx_thread_info -- Calculate Linux thread_info from task variable
+ lx-dmesg -- Print Linux kernel log buffer
+ lx-lsmod -- List currently loaded modules
+ lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
Signature · 1 line
1.7.3.4
author maintainer
#30 lore raw

PATCH v6 20/20 scripts/gdb: Add basic documentation

---

Just to close the request Boris had regarding help on the commands.

Changes in v6:
- explain how to obtain help on commands and functions
- minor wording fixes

 Documentation/gdb-kernel-debugging.txt |  158 ++++++++++++++++++++++++++++++++
 1 files changed, 158 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt
Documentation/gdb-kernel-debugging.txt new file +158 -0
new file mode 100644
index 0000000..1b5b3da
@@ -0,0 +1,158 @@
+Debugging kernel and modules via gdb
+====================================
+
+The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
+interfaces allow to debug the Linux kernel and its modules during runtime
+using gdb. Gdb comes with a powerful scripting interface for python. The
+kernel provides a collection of helper scripts that can simplify typical
+kernel debugging steps. This is a short tutorial about how to enable and use
+them. It focuses on QEMU/KVM virtual machines as target, but the examples can
+be transferred to the other gdb stubs as well.
+
+
+Requirements
+------------
+
+ o gdb 7.1+ (recommended: 7.3+) with python support enabled (typically true
+   for distributions)
+
+
+Setup
+-----
+
+ o Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
+   www.qemu.org for more details)
+
+ o Build the kernel with CONFIG_DEBUG_INFO enabled, but leave
+   CONFIG_DEBUG_INFO_REDUCED off
+
+ o Install that kernel on the guest.
+
+   Alternatively, QEMU allows to boot the kernel directly using -kernel,
+   -append, -initrd command line switches. This is generally only useful if
+   you do not depend on modules. See QEMU documentation for more details on
+   this mode.
+
+ o Enable the gdb stub of QEMU/KVM, either
+    - at VM startup time by appending "-s" to the QEMU command line
+   or
+    - during runtime by issuing "gdbserver" from the QEMU monitor
+      console
+
+ o cd /path/to/linux-build
+
+ o Start gdb: gdb vmlinux
+
+   Note: Some distros may restrict auto-loading of gdb scripts to known safe
+   directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
+
+    add-add-auto-load-safe-path /path/to/linux-build
+
+   to ~/.gdbinit. See gdb help for more details.
+
+ o Attach to the booted guest:
+    (gdb) target remote :1234
+
+
+Examples of using the Linux-provided gdb helpers
+------------------------------------------------
+
+ o Load module (and main kernel) symbols:
+    (gdb) lx-symbols
+    loading vmlinux
+    scanning for modules in /home/user/linux/build
+    loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
+    loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
+    loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
+    loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
+    loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
+    ...
+    loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
+
+ o Set a breakpoint on some not yet loaded module function, e.g.:
+    (gdb) b btrfs_init_sysfs
+    Function "btrfs_init_sysfs" not defined.
+    Make breakpoint pending on future shared library load? (y or [n]) y
+    Breakpoint 1 (btrfs_init_sysfs) pending.
+
+ o Continue the target
+    (gdb) c
+
+ o Load the module on the target and watch the symbols being loaded as well as
+   the breakpoint hit:
+    loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
+    loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
+    loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
+    loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
+
+    Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
+    36              btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
+
+ o Dump the log buffer of the target kernel:
+    (gdb) lx-dmesg
+    [     0.000000] Initializing cgroup subsys cpuset
+    [     0.000000] Initializing cgroup subsys cpu
+    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
+    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
+    [     0.000000] e820: BIOS-provided physical RAM map:
+    [     0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
+    [     0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
+    ....
+
+ o Examine fields of the current task struct:
+    (gdb) p $lx_current().pid
+    $1 = 4998
+    (gdb) p $lx_current().comm
+    $2 = "modprobe\000\000\000\000\000\000\000"
+
+ o Make use of the per-cpu function for the current or a specified CPU:
+    (gdb) p $lx_per_cpu("runqueues").nr_running
+    $3 = 1
+    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
+    $4 = 0
+
+ o Dig into hrtimers using the container_of helper:
+    (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
+    (gdb) p *$container_of($next, "struct hrtimer", "node")
+    $5 = {
+      node = {
+        node = {
+          __rb_parent_color = 18446612133355256072,
+          rb_right = 0x0 <irq_stack_union>,
+          rb_left = 0x0 <irq_stack_union>
+        },
+        expires = {
+          tv64 = 1835268000000
+        }
+      },
+      _softexpires = {
+        tv64 = 1835268000000
+      },
+      function = 0xffffffff81078232 <tick_sched_timer>,
+      base = 0xffff88003fd0d6f0,
+      state = 1,
+      start_pid = 0,
+      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
+      start_comm = "swapper/2\000\000\000\000\000\000"
+    }
+
+
+List of commands and functions
+------------------------------
+
+The number of commands and convenience functions may evolve over the time,
+this is just a snapshot of the initial version:
+
+ (gdb) apropos lx
+ function lx_current -- Return current task
+ function lx_module -- Find module by name and return the module variable
+ function lx_modvar -- Return global variable of a module
+ function lx_per_cpu -- Return per-cpu variable
+ function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
+ function lx_thread_info -- Calculate Linux thread_info from task variable
+ lx-dmesg -- Print Linux kernel log buffer
+ lx-lsmod -- List currently loaded modules
+ lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
+
+Detailed help can be obtained via "help <command-name>" for commands and "help
+function <function-name>" for convenience functions.
Signature · 1 line
1.7.3.4
#31 lore raw

Re: [PATCH v5 20/20] scripts/gdb: Add basic documentation

In reply to Jan Kiszka
Quoted 19 lines · Jan Kiszka
---
 Documentation/gdb-kernel-debugging.txt |  155  
++++++++++++++++++++++++++++++++
 1 files changed, 155 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/gdb-kernel-debugging.txt
Documentation/gdb-kernel-debugging.txt new file +4 -0
new file mode 100644
index 0000000..0ea46e1
@@ -0,0 +1,155 @@
+Debugging kernel and modules via gdb
+====================================
+
+The kernel debugger kgdb, hypervisors like QEMU or JTAG-based  
Quoted 4 lines

hardware
+interfaces allow to debug the Linux kernel and its modules during runtime
+using gdb.

This could use some clarification.

Technically they're attaching gdb to the hardware using the gdbserver protocol, instead of attaching to a specific process like the gdbserver client program does. (I think of it as "pid 0": your register state isn't a "register profile" but the actual contents of the hardware registers, and your memory map is physical.)

Two of these three methods (jtag and emulator) are independent of linux, and the fact the hardware you're attached to may or may not be running linux (or u-boot, or a bare metal hello world) is irrelevant. Only kgdb requires linux to be running to produce/consume gdbserver procol with information about current hardware state.

When Linux _is_ running, if you want symbol data you can point gdb at an unstripped vmlinux. (Which is an ELF image, the build usually feeds that through objdump and glues a wrapper on the front to make it bootable on bare metal, but this is the one with symbols still in it.) But even with symbols, trying to dig through Linux's various abstractions and data structures to find anything belonging to an actual process from outside the kernel is a flaming pain, so you've made some gdb plugins to help out there. Cool.

Quoted 22 lines

Gdb comes with a powerful scripting interface for python. The +kernel provides a collection of helper scripts that can simplify typical
+kernel debugging steps. This is a short tutorial about how to enable and use
+them. It focuses on QEMU/KVM virtual machines as target, but the examples can

Documentation/gdb-kernel-debugging.txt new file +7 -0
+be transferred to the other gdb stubs as well.
+
+
+Requirements
+------------
+
+ o gdb 7.1+ (recommended: 7.3+) with python support enabled  

(typically true

Documentation/gdb-kernel-debugging.txt new file +7 -0
+   for distributions)
+
+
+Setup
+-----
+
+ o Create a virtual Linux machine for QEMU/KVM (see  
Quoted 2 lines

www.linux-kvm.org and
+ www.qemu.org for more details)

If it helps, I've got a dozen different targets as "system-image-*.tar.bz2" at
http://landley.net/aboriginal/bin and the "run-emulator.sh" script inside each one gives the qemu command line to boot it to a shell prompt. At that shell prompt you can grab the relevant kernel config with "zcat /proc/config.gz" to build your own kernel, and the toolchains I did so with are in the same directory. (The README in there describes other categories of file in the directory.)

That at least gives you a "known working" version to compare your own attempts against.

Quoted 8 lines

+ o Build the kernel with CONFIG_DEBUG_INFO enabled, but leave

Documentation/gdb-kernel-debugging.txt new file +6 -1
+   CONFIG_DEBUG_INFO_REDUCED off
+
+ o Install that kernel on the guest.
+
+   Alternatively, QEMU allows to boot the kernel directly using  
-kernel,
+   -append, -initrd command line switches. This is generally only  
Quoted 4 lines

useful if
+ you do not depend on modules. See QEMU documentation for more
details on
+ this mode.

It's qemu's built-in bootloader, no different than any other bootloader. If your hardware needs a module to access the block device you want to keep the rest of your modules on, A) you're making a peverse architectural decision, B) stick it in the initramfs.

(I could do up an example of that sort of setup if you like. The images I linked to above keep a squashfs on a virtual hard drive, I could do an initramfs that loads the IDE module and then switch_root to it. I just haven't bothered because the easy way works.)

Documentation/gdb-kernel-debugging.txt line 40

Quoted 5 lines

+ o Enable the gdb stub of QEMU/KVM, either

Documentation/gdb-kernel-debugging.txt new file +4 -0
+    - at VM startup time by appending "-s" to the QEMU command line
+   or
+    - during runtime by issuing "gdbserver" from the QEMU monitor
+      console

Initiallly I thought this document would be more about kgdb, but this is about how to use a direct hardware attached debugger (or kgdb emulating one). Hence the clarification up top.

Quoted 8 lines

+ o cd /path/to/linux-build

Documentation/gdb-kernel-debugging.txt new file +4 -0
+
+ o Start gdb: gdb vmlinux
+
+   Note: Some distros may restrict auto-loading of gdb scripts to  

known safe
+ directories. In case gdb reports to refuse loading
vmlinux-gdb.py, add

Documentation/gdb-kernel-debugging.txt line 54

Quoted 7 lines
Documentation/gdb-kernel-debugging.txt new file +7 -0
+
+    add-add-auto-load-safe-path /path/to/linux-build
+
+   to ~/.gdbinit. See gdb help for more details.
+
+ o Attach to the booted guest:
+    (gdb) target remote :1234

I've found the "target remote |" option useful, where you pipe the I/O to another command that forwards data to the target board.

(I once had to debug a program on a board that only had a serial console, and that was always on and attached to another machine. So I piped to ssh which ran a term program which launched netcat on the board to connect the remote end because the FSF didn't bother to make a stdio option for gdbserver so you _have_ to use loopback. Although that was a few versions back, maybe they fixed this...)

*shrug* Possibly off topic...

The rest is about what the plugins do, looks fine to me. :)

Rob

Quoted 128 lines
Documentation/gdb-kernel-debugging.txt new file +9 -0
+
+Examples of using the Linux-provided gdb helpers
+------------------------------------------------
+
+ o Load module (and main kernel) symbols:
+    (gdb) lx-symbols
+    loading vmlinux
+    scanning for modules in /home/user/linux/build
+    loading @0xffffffffa0020000:  

/home/user/linux/build/net/netfilter/xt_tcpudp.ko
+ loading @0xffffffffa0016000:
/home/user/linux/build/net/netfilter/xt_pkttype.ko
+ loading @0xffffffffa0002000:
/home/user/linux/build/net/netfilter/xt_limit.ko
+ loading @0xffffffffa00ca000:
/home/user/linux/build/net/packet/af_packet.ko
+ loading @0xffffffffa003c000:
/home/user/linux/build/fs/fuse/fuse.ko

Documentation/gdb-kernel-debugging.txt new file +2 -0
+    ...
+    loading @0xffffffffa0000000:  

/home/user/linux/build/drivers/ata/ata_generic.ko

Documentation/gdb-kernel-debugging.txt new file +5 -0
+
+ o Set a breakpoint on some not yet loaded module function, e.g.:
+    (gdb) b btrfs_init_sysfs
+    Function "btrfs_init_sysfs" not defined.
+    Make breakpoint pending on future shared library load? (y or  

[n]) y

Documentation/gdb-kernel-debugging.txt new file +6 -0
+    Breakpoint 1 (btrfs_init_sysfs) pending.
+
+ o Continue the target
+    (gdb) c
+
+ o Load the module on the target and watch the symbols being loaded  

as well as

Documentation/gdb-kernel-debugging.txt new file +2 -0
+   the breakpoint hit:
+    loading @0xffffffffa0034000:  

/home/user/linux/build/lib/libcrc32c.ko
+ loading @0xffffffffa0050000:
/home/user/linux/build/lib/lzo/lzo_compress.ko
+ loading @0xffffffffa006e000:
/home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
+ loading @0xffffffffa01b1000:
/home/user/linux/build/fs/btrfs/btrfs.ko

Documentation/gdb-kernel-debugging.txt new file +2 -0
+
+    Breakpoint 1, btrfs_init_sysfs () at  

/home/user/linux/fs/btrfs/sysfs.c:36
+ 36 btrfs_kset = kset_create_and_add("btrfs", NULL,
fs_kobj);

Documentation/gdb-kernel-debugging.txt new file +7 -0
+
+ o Dump the log buffer of the target kernel:
+    (gdb) lx-dmesg
+    [     0.000000] Initializing cgroup subsys cpuset
+    [     0.000000] Initializing cgroup subsys cpu
+    [     0.000000] Linux version 3.8.0-rc4-dbg+ (...
+    [     0.000000] Command line: root=/dev/sda2 resume=/dev/sda1  

vga=0x314

Documentation/gdb-kernel-debugging.txt new file +2 -0
+    [     0.000000] e820: BIOS-provided physical RAM map:
+    [     0.000000] BIOS-e820: [mem  

0x0000000000000000-0x000000000009fbff] usable
+ [ 0.000000] BIOS-e820: [mem
0x000000000009fc00-0x000000000009ffff] reserved

Documentation/gdb-kernel-debugging.txt new file +16 -0
+    ....
+
+ o Examine fields of the current task struct:
+    (gdb) p $lx_current().pid
+    $1 = 4998
+    (gdb) p $lx_current().comm
+    $2 = "modprobe\000\000\000\000\000\000\000"
+
+ o Make use of the per-cpu helper for the current or a specified CPU:
+    (gdb) p $lx_per_cpu("runqueues").nr_running
+    $3 = 1
+    (gdb) p $lx_per_cpu("runqueues", 2).nr_running
+    $4 = 0
+
+ o Dig into hrtimers using the container_of helper:
+    (gdb) set $next =  

$lx_per_cpu("hrtimer_bases").clock_base[0].active.next

Documentation/gdb-kernel-debugging.txt new file +28 -0
+    (gdb) p *$container_of($next, "struct hrtimer", "node")
+    $5 = {
+      node = {
+        node = {
+          __rb_parent_color = 18446612133355256072,
+          rb_right = 0x0 <irq_stack_union>,
+          rb_left = 0x0 <irq_stack_union>
+        },
+        expires = {
+          tv64 = 1835268000000
+        }
+      },
+      _softexpires = {
+        tv64 = 1835268000000
+      },
+      function = 0xffffffff81078232 <tick_sched_timer>,
+      base = 0xffff88003fd0d6f0,
+      state = 1,
+      start_pid = 0,
+      start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
+      start_comm = "swapper/2\000\000\000\000\000\000"
+    }
+
+
+List of commands and helper
+---------------------------
+
+The number of commands and convenience helpers may evolve over the  

time, this

Documentation/gdb-kernel-debugging.txt new file +5 -0
+is just a snapshot of the initial version:
+
+ (gdb) apropos lx
+ function lx_current -- Return current task
+ function lx_module -- Find module by name and return the module  

variable
+ function lx_modvar -- Return global variable of a module
+ function lx_per_cpu -- Return per-cpu variable
+ function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
+ function lx_thread_info -- Calculate Linux thread_info from task variable
+ lx-dmesg -- Print Linux kernel log buffer
+ lx-lsmod -- List currently loaded modules
+ lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
--
1.7.3.4

#32 lore raw
In reply to Jan Kiszka
Quoted 9 lines · Jan Kiszka

Version 5 comes with the following changes:
- moved tutorial into Documentation/gdb-kernel-debugging.txt
- improved caching of gdb.Type objects, ensure they are in sync with currently loaded symbols
- added new functions and commands
- lx_module -- Find module by name and return the module variable
- lx_modvar -- Return global variable of a module
- lx-lsmod -- List currently loaded modules

Quoted 6 lines

See http://lkml.indiana.edu/hypermail/linux/kernel/1210.0/01598.html for the original description and

git://git.kiszka.org/linux.git queues/gdb-scripts

for the latest version.

Ok, I've seldomly shown any feelings when acking a patch{,set} besides maybe "hm, ok, fine" but this patchset is very very cool and it is an enormous fun playing with it.

So maybe this is the most enthusiasm I've shown sofar:

Thanks Jan, very good work!

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--