usb: early: add support for early printk through USB3 debug port

xHCI debug capability (DbC) is an optional but standalone functionality provided by an xHCI host controller. With DbC hardware initialized, the system will present a debug device through the USB3 debug port (normally the first USB3 port). The debug device is fully compliant with the USB framework and provides the equivalent of a very high performance (USB3) full-duplex serial link between the debug host and target. The DbC functionality is independent of xHCI host. There isn't any precondition from xHCI host side for DbC to work.

This patch set adds support for early printk functionality through a USB3 debug port by 1) initializing and enabling the DbC hardware during early boot; 2) registering a boot console to the system so that early printk messages can go through the USB3 debug port. It also includes some lines of changes in usb_debug driver so that it can be bound when a USB3 debug device is enumerated.

This code is designed to be used only for kernel debugging when machine crashes very early before the console code is initialized. It makes the life of kernel debugging easier when people work with a modern machine without any legacy serial ports.

---
Change log:
v3->v4:
- Rename the document with .dst suffix.
- Add the list of hardware that has been succesfuly tested on in the document.

v2->v3:
- Removed spinlock usage.
- Removed work queue usage.
- Refined the user guide document.

v1->v2:
- Refactor the duplicate code in xdbc_early_start() and xdbc_handle_external_reset().
- Free resources when hardware not used any more.
- Refine the user guide document.

Lu Baolu (4):
usb: dbc: early driver for xhci debug capability
x86: add support for earlyprintk via USB3 debug port
usb: serial: usb_debug: add support for dbc debug device
usb: doc: add document for USB3 debug port usage

 Documentation/kernel-parameters.txt   |    1 +
 Documentation/usb/usb3-debug-port.rst |   95 +++
 arch/x86/Kconfig.debug                |   14 +
 arch/x86/kernel/early_printk.c        |    5 +
 arch/x86/kernel/setup.c               |    7 +
 drivers/usb/Kconfig                   |    3 +
 drivers/usb/Makefile                  |    2 +-
 drivers/usb/early/Makefile            |    1 +
 drivers/usb/early/xhci-dbc.c          | 1050 +++++++++++++++++++++++++++++++++
 drivers/usb/early/xhci-dbc.h          |  202 +++++++
 drivers/usb/serial/usb_debug.c        |   28 +-
 include/linux/usb/xhci-dbgp.h         |   22 +
 12 files changed, 1426 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/usb/usb3-debug-port.rst
 create mode 100644 drivers/usb/early/xhci-dbc.c
 create mode 100644 drivers/usb/early/xhci-dbc.h
 create mode 100644 include/linux/usb/xhci-dbgp.h

2.1.4

xHCI debug capability (DbC) is an optional but standalone functionality provided by an xHCI host controller. Software learns this capability by walking through the extended capability list of the host. xHCI specification describes DbC in section 7.6.

This patch introduces the code to probe and initialize the debug capability hardware during early boot. With hardware initialized, the debug target (system on which this code is running) will present a debug device through the debug port (normally the first USB3 port). The debug device is fully compliant with the USB framework and provides the equivalent of a very high performance (USB3) full-duplex serial link between the debug host and target. The DbC functionality is independent of xHCI host. There isn't any precondition from xHCI host side for DbC to work.

This patch also includes bulk out and bulk in interfaces. These interfaces could be used to implement early printk bootconsole or hook to various system debuggers.

This code is designed to be only used for kernel debugging when machine crashes very early before the console code is initialized. For normal operation it is not recommended.

---
 arch/x86/Kconfig.debug        |   14 +
 drivers/usb/Kconfig           |    3 +
 drivers/usb/Makefile          |    2 +-
 drivers/usb/early/Makefile    |    1 +
 drivers/usb/early/xhci-dbc.c  | 1050 +++++++++++++++++++++++++++++++++++++++++
 drivers/usb/early/xhci-dbc.h  |  202 ++++++++
 include/linux/usb/xhci-dbgp.h |   22 +
 7 files changed, 1293 insertions(+), 1 deletion(-)
 create mode 100644 drivers/usb/early/xhci-dbc.c
 create mode 100644 drivers/usb/early/xhci-dbc.h
 create mode 100644 include/linux/usb/xhci-dbgp.h

index 67eec55..13e85b7 100644
@@ -29,6 +29,7 @@ config EARLY_PRINTK
 config EARLY_PRINTK_DBGP
 	bool "Early printk via EHCI debug port"
 	depends on EARLY_PRINTK && PCI
+	select USB_EARLY_PRINTK
 	---help---
 	  Write kernel log output directly into the EHCI debug port.
 

@@ -48,6 +49,19 @@ config EARLY_PRINTK_EFI
 	  This is useful for kernel debugging when your machine crashes very
 	  early before the console code is initialized.
 
+config EARLY_PRINTK_XDBC
+	bool "Early printk via xHCI debug port"
+	depends on EARLY_PRINTK && PCI
+	select USB_EARLY_PRINTK
+	---help---
+	  Write kernel log output directly into the xHCI debug port.
+
+	  This is useful for kernel debugging when your machine crashes very
+	  early before the console code is initialized. For normal operation
+	  it is not recommended because it looks ugly and doesn't cooperate
+	  with klogd/syslogd or the X server. You should normally N here,
+	  unless you want to debug such a crash.
+
 config X86_PTDUMP_CORE
 	def_bool n
 

index fbe493d..9313fff 100644
@@ -19,6 +19,9 @@ config USB_EHCI_BIG_ENDIAN_MMIO
 config USB_EHCI_BIG_ENDIAN_DESC
 	bool
 
+config USB_EARLY_PRINTK
+	bool
+
 menuconfig USB_SUPPORT
 	bool "USB support"
 	depends on HAS_IOMEM

index 7791af6..0c37838 100644
@@ -49,7 +49,7 @@ obj-$(CONFIG_USB_MICROTEK)	+= image/
 obj-$(CONFIG_USB_SERIAL)	+= serial/
 
 obj-$(CONFIG_USB)		+= misc/
-obj-$(CONFIG_EARLY_PRINTK_DBGP)	+= early/
+obj-$(CONFIG_USB_EARLY_PRINTK)	+= early/
 
 obj-$(CONFIG_USB_ATM)		+= atm/
 obj-$(CONFIG_USB_SPEEDTOUCH)	+= atm/

index 24bbe51..2db5906 100644
@@ -3,3 +3,4 @@
 #
 
 obj-$(CONFIG_EARLY_PRINTK_DBGP) += ehci-dbgp.o
+obj-$(CONFIG_EARLY_PRINTK_XDBC) += xhci-dbc.o

new file mode 100644
index 0000000..aa9441f
@@ -0,0 +1,22 @@
+/*
+ * Standalone xHCI debug capability driver
+ *
+ * Copyright (C) 2016 Intel Corporation
+ *
+ * Author: Lu Baolu <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __LINUX_XHCI_DBGP_H
+#define __LINUX_XHCI_DBGP_H
+
+#ifdef CONFIG_EARLY_PRINTK_XDBC
+int __init early_xdbc_parse_parameter(char *s);
+int __init early_xdbc_setup_hardware(void);
+void __init early_xdbc_register_console(void);
+#endif /* CONFIG_EARLY_PRINTK_XDBC */
+
+#endif /* __LINUX_XHCI_DBGP_H */

2.1.4

...

	oldbase = xdbc.xhci_base;
	base = ioremap(....);
	xdbc.xhci_base = base;
	early_iounmap(oldbase);

Hi,

If there isn't a good solution, I will remove the ioremap code and let it use the early mapped one.

Best regards,
Lu Baolu

Hi,

I can use a spinlock_irq to protect reentrancy of interrupt on the same cpu. But I have no idea about the nmi one. This seems to be a common issue for all early printk drivers.

Peter, any idea?

Best regards,
Lu Baolu

spinlock wont work due to NMIs.

No. The other early printk drivers like serial do not have that problem as they simply do:

   while (*buf) {
      while (inb(UART) & TX_BUSY)
   	 cpu_relax();
      outb(*buf++, UART);
   }

The wait for the UART to become ready is independent of the context as it solely depends on the hardware.

As a result you can see the output from irq/nmi intermingled with the one from thread context, but that's the only problem they have.

The only thing you can do to make this work is to prevent printing in NMI context:

write()

{
	if (in_nmi())
		return;
	
	raw_spinlock_irqsave(&lock, flags);
	....

That fully serializes the writes and just ignores NMI context printks. Not optimal, but I fear that's all you can do.

Thanks,

tglx

Right, which is why actual UARTs rule. If only laptops still had pinouts for them life would be sooooo much better.

Ideally the USB debug port would be a virtual UART and its interface as simple and robust.

I would also suggest telling the hardware people they have designed something near the brink of useless. If you cannot do random exception context debugging (#DB, #NMI, #MCE etc..) then there's a whole host of problems that simply cannot be debugged.

Also note that kdb runs from NMI context, so you'll not be able to support that either.

Hi Peter,

Things become complicated when it comes to USB debug port. But it's still addressable.

At this time, we can do it like this.

write()

{
	if (in_nmi() && raw_spin_is_locked(&lock))
		return;

	raw_spinlock_irqsave(&lock, flags);
	....

This will filter some messages from NMI handler in case that another thread is holding the spinlock. I have no idea about how much chance could a debug user faces this. But it might further be fixed with below enhancement.

write()

{
	if (in_nmi() && raw_spin_is_locked(&lock)) {
		produce_a_pending_item_in_ring();
		return;
	}

	raw_spinlock_irqsave(&lock, flags);

	while (!pending_item_ring_is_empty)
		consume_a_pending_item_in_ring();

	....

We can design the pending item ring in a producer-consumer model. It's easy to avoid race between the producer and consumer.

Best regards,
Lu Baolu

Please use raw_spin_trlock_irqsave() instead, spin_is_locked() is fairly icky.

Also, there's a bunch of exception contexts that do not set in_nmi(). That is in_nmi() is really only set for #NM. #MC and #DB and others do not set this.

Problem is that the consumer might never happen, those are the fun most bugs.

Not being able to deal with random nested exception context really reduces the utility of this thing.

Again, a UART rules. Make a virtual UART in hardware, that'd be totally awesome. This thing, I'm not convinced its worth having.

Hi Peter,

Sure.

That's worth another fix patch. Let me look into it later.

This is the initial work. It helps at least in cases where people need to dump kernel messages but lacking of a console. It's also a cheap way, people don't need to buy any third-party devices.

With more and more people trying and enhancing it, it will become more robust and helpful.

Best regards,
Lu Baolu

Well, not really. The hardware does not allow you what an UART allows you to do and never will, no matter how many people are trying to enhance the software part of it.

Thanks,

tglx

Hi,

Yes, of course.

Yes, you are right.

Yes. But I want to add a bit more.

write()

{
	if (in_nmi() && raw_spin_is_locked(&lock)) {
		trace("... ...");
		return;
	}

	raw_spinlock_irqsave(&lock, flags);
	....

Best regards,
Lu Baolu

Hi,

Or...?

write()

{
	if (in_nmi() && raw_spin_is_locked(&lock)) {
		save_nmi_message_in_local_buf();
		set_nmi_message_pending_flag();
		return;
	}

	if (nmi_message_pending_flag_is_set()) {
		write_nmi_message();
		clear_nmi_message_pending_flag();
	}

	raw_spinlock_irqsave(&lock, flags);
	....

Best regards,
Lu Baolu

Add support for early printk by writing debug messages to the
USB3 debug port. Users can use this type of early printk by
specifying kernel parameter of "earlyprintk=xdbc". This gives users a chance of providing debug output.

The hardware for USB3 debug port requires DMA memory blocks. This requires to delay setting up debugging hardware and registering boot console until the memblocks are filled.

---
 Documentation/kernel-parameters.txt | 1 +
 arch/x86/kernel/early_printk.c      | 5 +++++
 arch/x86/kernel/setup.c             | 7 +++++++
 3 files changed, 13 insertions(+)

index 37babf9..99b64b3 100644
@@ -1178,6 +1178,7 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 			earlyprintk=ttySn[,baudrate]
 			earlyprintk=dbgp[debugController#]
 			earlyprintk=pciserial,bus:device.function[,baudrate]
+			earlyprintk=xdbc[xhciController#]
 
 			earlyprintk is useful when the kernel crashes before
 			the normal console is initialized. It is not enabled by

index 8a12199..c4031b9 100644
@@ -17,6 +17,7 @@
 #include <asm/intel-mid.h>
 #include <asm/pgtable.h>
 #include <linux/usb/ehci_def.h>
+#include <linux/usb/xhci-dbgp.h>
 #include <linux/efi.h>
 #include <asm/efi.h>
 #include <asm/pci_x86.h>

@@ -381,6 +382,10 @@ static int __init setup_early_printk(char *buf)
 		if (!strncmp(buf, "efi", 3))
 			early_console_register(&early_efi_console, keep);
 #endif
+#ifdef CONFIG_EARLY_PRINTK_XDBC
+		if (!strncmp(buf, "xdbc", 4))
+			early_xdbc_parse_parameter(buf + 4);
+#endif
 
 		buf++;
 	}

index 9c337b0..09d4a56 100644
@@ -70,6 +70,8 @@
 #include <linux/tboot.h>
 #include <linux/jiffies.h>
 
+#include <linux/usb/xhci-dbgp.h>
+
 #include <video/edid.h>
 
 #include <asm/mtrr.h>

@@ -1096,6 +1098,11 @@ void __init setup_arch(char **cmdline_p)
 	memblock_set_current_limit(ISA_END_ADDRESS);
 	memblock_x86_fill();
 
+#ifdef CONFIG_EARLY_PRINTK_XDBC
+	if (!early_xdbc_setup_hardware())
+		early_xdbc_register_console();
+#endif
+
 	reserve_bios_regions();
 
 	if (efi_enabled(EFI_MEMMAP)) {

2.1.4

This patch add dbc debug device support in usb_debug driver.

---
 drivers/usb/serial/usb_debug.c | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

index ca2fa5b..92f7e5c 100644
@@ -32,7 +32,18 @@ static const struct usb_device_id id_table[] = {
 	{ USB_DEVICE(0x0525, 0x127a) },
 	{ },
 };
-MODULE_DEVICE_TABLE(usb, id_table);
+
+static const struct usb_device_id dbc_id_table[] = {
+	{ USB_DEVICE(0x1d6b, 0x0004) },
+	{ },
+};
+
+static const struct usb_device_id id_table_combined[] = {
+	{ USB_DEVICE(0x0525, 0x127a) },
+	{ USB_DEVICE(0x1d6b, 0x0004) },
+	{ },
+};
+MODULE_DEVICE_TABLE(usb, id_table_combined);
 
 /* This HW really does not support a serial break, so one will be
  * emulated when ever the break state is set to true.

@@ -71,9 +82,20 @@ static struct usb_serial_driver debug_device = {
 	.process_read_urb =	usb_debug_process_read_urb,
 };
 
+static struct usb_serial_driver dbc_device = {
+	.driver = {
+		.owner =	THIS_MODULE,
+		.name =		"xhci_dbc",
+	},
+	.id_table =		dbc_id_table,
+	.num_ports =		1,
+	.break_ctl =		usb_debug_break_ctl,
+	.process_read_urb =	usb_debug_process_read_urb,
+};
+
 static struct usb_serial_driver * const serial_drivers[] = {
-	&debug_device, NULL
+	&debug_device, &dbc_device, NULL
 };
 
-module_usb_serial_driver(serial_drivers, id_table);
+module_usb_serial_driver(serial_drivers, id_table_combined);
 MODULE_LICENSE("GPL");

2.1.4

Add Documentation/usb/usb3-debug-port.rst. This document includes the user guide for USB3 debug port.

---
 Documentation/usb/usb3-debug-port.rst | 95 +++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
 create mode 100644 Documentation/usb/usb3-debug-port.rst

new file mode 100644
index 0000000..70eabe4
@@ -0,0 +1,95 @@
+===============
+USB3 debug port
+===============
+
+:Author: Lu Baolu <[email protected]>
+:Date: October 2016
+
+GENERAL
+=======
+
+This is a HOWTO for using USB3 debug port on x86 systems.
+
+Before using any kernel debugging functionalities based on USB3
+debug port, you need to check 1) whether debug port is supported
+by the xHCI host, 2) which port is used for debugging purpose
+(normally the first USB3 root port). You must have a USB 3.0
+super-speed A-to-A debugging cable to connect the debug target
+with a debug host. In this document, a debug target stands for
+the system under debugging; while, a debug host stands for a
+stand-alone system that is able to talk to the debugging target
+through the USB3 debug port.
+
+EARLY PRINTK
+============
+
+On debug target system, you need to customize a debugging kernel
+with CONFIG_EARLY_PRINTK_XDBC enabled. And add below kernel boot
+parameter::
+
+	"earlyprintk=xdbc"
+
+If there are multiple xHCI controllers in the system, you can
+append a host contoller index to this kernel parameter. This
+index is started from 0.
+
+If you are going to leverage the keep option defined by the
+early printk framework to keep the boot console alive after
+early boot, you'd better add below kernel boot parameter::
+
+	"usbcore.autosuspend=-1"
+
+On debug host side, you don't need to customize the kernel, but
+you need to disable usb subsystem runtime power management by
+adding below kernel boot parameter::
+
+	"usbcore.autosuspend=-1"
+
+Before starting the debug target, you should connect the debug
+port on debug target with a root port or port of any external hub
+on the debug host. The cable used to connect these two ports
+should be a USB 3.0 super-speed A-to-A debugging cable.
+
+During early boot of debug target, DbC (the debug engine for USB3
+debug port) hardware gets initialized. Debug host should be able
+to enumerate the debug target as a debug device. Debug host will
+then bind the debug device with the usb_debug driver module and
+create the /dev/ttyUSB0 device.
+
+If device enumeration goes smoothly, you should be able to see
+below kernel messages on debug host::
+
+	# tail -f /var/log/kern.log
+	[ 1815.983374] usb 4-3: new SuperSpeed USB device number 4 using xhci_hcd
+	[ 1815.999595] usb 4-3: LPM exit latency is zeroed, disabling LPM.
+	[ 1815.999899] usb 4-3: New USB device found, idVendor=1d6b, idProduct=0004
+	[ 1815.999902] usb 4-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
+	[ 1815.999903] usb 4-3: Product: Remote GDB
+	[ 1815.999904] usb 4-3: Manufacturer: Linux
+	[ 1815.999905] usb 4-3: SerialNumber: 0001
+	[ 1816.000240] usb_debug 4-3:1.0: xhci_dbc converter detected
+	[ 1816.000360] usb 4-3: xhci_dbc converter now attached to ttyUSB0
+
+You can run below bash scripts on debug host to read the kernel
+log sent from debug target.
+
+.. code-block:: sh
+
+	===== start of bash scripts =============
+	#!/bin/bash
+
+	while true ; do
+		while [ ! -d /sys/class/tty/ttyUSB0 ] ; do
+			:
+	done
+	cat /dev/ttyUSB0 >> xdbc.log
+	done
+	===== end of bash scripts ===============
+
+You should be able to see the early boot message in xdbc.log.
+
+If it doesn't work, please ask it on the <[email protected]>
+mailing list. Below USB hosts have been verified to work::
+
+	Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller
+	Intel Corporation Wildcat Point-LP USB xHCI Controller

2.1.4