usb: notify hcd when USB device suspend or resume

This patch adds two new entries in hc_driver. With these new entries, USB core can notify host driver when a USB device is about to suspend or just resumed.

The xHCI spec is designed to allow an xHC implementation to cache the endpoint state. Caching endpoint state allows an xHC to reduce latency when handling ERDYs and other USB asynchronous events. However holding this state in xHC consumes resources and power. The xHCI spec designs some methods through which host controller driver can hint xHC about how to optimize its operation, e.g. to determine when it holds state internally or pushes it out to memory, when to power down logic, etc.

When a USB device is going to suspend, states of all endpoints cached in the xHC should be pushed out to memory to save power and resources. Vice versa, when a USB device resumes, those states should be brought back to the cache. USB core suspends or resumes a USB device by sending set/clear port feature requests to the parent hub where the USB device is connected. Unfortunately, these operations are transparent to xHCI driver unless the USB device is plugged in a root port. This patch utilizes the callback entries to notify xHCI driver whenever a USB device suspends or resumes.

It is harmless if a USB devices under USB 3.0 host controller suspends or resumes without a notification to hcd driver. However there may be less opportunities for power savings and there may be increased latency for restarting an endpoint. The precise impact will be different for each xHC implementation. It all depends on what an implementation does with the hints.

Signed-off-by Lu Baolu <[email protected]>

drivers/usb/core/hub.c +4 -0

+	hcd_suspend_notify(udev, msg);
+
 	/* see 7.1.7.6 */
 	if (hub_is_superspeed(hub->hdev))
 		status = hub_set_port_link_state(hub, port1, USB_SS_PORT_LS_U3);

@@ -3169,6 +3177,8 @@ int usb_port_suspend(struct usb_device *udev, pm_message_t msg)
 	if (status) {
 		dev_dbg(&port_dev->dev, "can't suspend, status %d\n", status);
 
+		hcd_resume_notify(udev, msg);
+
 		/* Try to enable USB3 LPM and LTM again */
 		usb_unlocked_enable_lpm(udev);
  err_lpm3:

drivers/usb/core/hub.c +1 -0

+	hcd_resume_notify(udev, msg);
 	if (status == 0) {
 		udev->port_is_suspended = 0;
 		if (hub_is_superspeed(hub->hdev)) {

include/linux/usb/hcd.h +0 -0

index 68b1e83..621d9b7 100644
@@ -383,7 +383,13 @@ struct hc_driver {
 	int	(*find_raw_port_number)(struct usb_hcd *, int);
 	/* Call for power on/off the port if necessary */

Quoted 37 lines · Lu Baolu

This patch adds two new entries in hc_driver. With these new entries, USB core can notify host driver when a USB device is about to suspend or just resumed.

The xHCI spec is designed to allow an xHC implementation to cache the endpoint state. Caching endpoint state allows an xHC to reduce latency when handling ERDYs and other USB asynchronous events. However holding this state in xHC consumes resources and power. The xHCI spec designs some methods through which host controller driver can hint xHC about how to optimize its operation, e.g. to determine when it holds state internally or pushes it out to memory, when to power down logic, etc.

When a USB device is going to suspend, states of all endpoints cached in the xHC should be pushed out to memory to save power and resources. Vice versa, when a USB device resumes, those states should be brought back to the cache. USB core suspends or resumes a USB device by sending set/clear port feature requests to the parent hub where the USB device is connected. Unfortunately, these operations are transparent to xHCI driver unless the USB device is plugged in a root port. This patch utilizes the callback entries to notify xHCI driver whenever a USB device suspends or resumes.

It is harmless if a USB devices under USB 3.0 host controller suspends or resumes without a notification to hcd driver. However there may be less opportunities for power savings and there may be increased latency for restarting an endpoint. The precise impact will be different for each xHC implementation. It all depends on what an implementation does with the hints.

Signed-off-by Lu Baolu <[email protected]>

drivers/usb/core/hub.c +0 -0

@@ -3144,6 +3144,14 @@ int usb_port_suspend(struct usb_device *udev, pm_message_t msg)
  			goto err_lpm3;
  	}
 

drivers/usb/core/hub.c int usb_port_suspend(struct usb_device *udev, pm_message_t msg) · line 3152

Quoted 6 lines

drivers/usb/core/hub.c +6 -0

+	/*
+	 * Call back to hcd if it expects. xHCI compatible host controller
+	 * driver expects to be notified prior to selectively suspending a
+	 * device. xHCI hcd could optimize the endpoint cache for power
+	 * saving purpose. Refer to 4.15.1.1 of xHCI 1.1 for more information.
+	 */

Doesn't this comment belong in the xhci-hcd source code rather than the hub driver source code?

Quoted 16 lines

drivers/usb/core/hub.c +4 -0

+	hcd_suspend_notify(udev, msg);
+
  	/* see 7.1.7.6 */
  	if (hub_is_superspeed(hub->hdev))
  		status = hub_set_port_link_state(hub, port1, USB_SS_PORT_LS_U3);

@@ -3169,6 +3177,8 @@ int usb_port_suspend(struct usb_device *udev, pm_message_t msg)
  	if (status) {
  		dev_dbg(&port_dev->dev, "can't suspend, status %d\n", status);
 
+		hcd_resume_notify(udev, msg);
+
  		/* Try to enable USB3 LPM and LTM again */
  		usb_unlocked_enable_lpm(udev);
   err_lpm3:

drivers/usb/core/hub.c int usb_port_resume(struct usb_device *udev, pm_message_t msg) · line 3439

Quoted 8 lines

drivers/usb/core/hub.c +5 -0

@@ -3422,6 +3432,12 @@ int usb_port_resume(struct usb_device *udev, pm_message_t msg)
  	}
 
   SuspendCleared:
+	/* Call back to hcd if it expects. xHCI compatible host controller
+	 * driver expects to be notified after a device is resumed. xHCI
+	 * hcd could optimize the endpoint cache for latency reducing
+	 * purpose. Refer to 4.15.1.1 of xHCI 1.1 for more information.
+	 */

Same for this comment.

Quoted 15 lines

drivers/usb/core/hub.c +1 -0

+	hcd_resume_notify(udev, msg);
  	if (status == 0) {
  		udev->port_is_suspended = 0;
  		if (hub_is_superspeed(hub->hdev)) {

include/linux/usb/hcd.h +2 -1

index 68b1e83..621d9b7 100644
@@ -383,7 +383,13 @@ struct hc_driver {
  	int	(*find_raw_port_number)(struct usb_hcd *, int);
  	/* Call for power on/off the port if necessary */
  	int	(*port_power)(struct usb_hcd *hcd, int portnum, bool enable);
-
+	/* Call back to hcd when a USB device is going to suspend or just
+	 * resumed.

include/linux/usb/hcd.h struct hc_driver { · line 393

Quoted 6 lines

include/linux/usb/hcd.h +5 -0

+	 */
+	void	(*device_suspend)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
+	void	(*device_resume)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
  };

Your callbacks don't use the msg argument. What makes you think it is needed?

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 7 lines

include/linux/usb/hcd.h struct hc_driver { · line 393

Quoted 5 lines

include/linux/usb/hcd.h +4 -0

+	void	(*device_suspend)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
+	void	(*device_resume)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
  };

Your callbacks don't use the msg argument. What makes you think it is needed?

This msg argument is valuable. XHCI spec defines a capability named FSC (Force Save context Capability). When this capability is implemented, the Save State operation (do during host suspend) shall save any cached Slot, Endpoint, Stream or other Context information to memory. xHCI hcd could use this "msg" to determine whether it needs to issue stop endpoint with SP (suspend) bit set.

Quoted 7 lines · Baolu

Quoted 7 lines

include/linux/usb/hcd.h struct hc_driver { · line 393

Quoted 5 lines

include/linux/usb/hcd.h +4 -0

+	void	(*device_suspend)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
+	void	(*device_resume)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
   };

Your callbacks don't use the msg argument. What makes you think it is needed?

Quoted 6 lines

This msg argument is valuable. XHCI spec defines a capability named FSC (Force Save context Capability). When this capability is implemented, the Save State operation (do during host suspend) shall save any cached Slot, Endpoint, Stream or other Context information to memory. xHCI hcd could use this "msg" to determine whether it needs to issue stop endpoint with SP (suspend) bit set.

I don't understand. What is the advantage of using FSC?

How would xhci-hcd use "msg" to determine this? And why doesn't your 2/3 patch already do it?

Alan Stern

Quoted 9 lines · Alan Stern

Quoted 7 lines · Baolu

Quoted 7 lines

include/linux/usb/hcd.h struct hc_driver { · line 393

Quoted 5 lines

include/linux/usb/hcd.h +4 -0

+	void	(*device_suspend)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
+	void	(*device_resume)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
   };

Your callbacks don't use the msg argument. What makes you think it is needed?

Quoted 7 lines

Quoted 6 lines

This msg argument is valuable. XHCI spec defines a capability named FSC (Force Save context Capability). When this capability is implemented, the Save State operation (do during host suspend) shall save any cached Slot, Endpoint, Stream or other Context information to memory. xHCI hcd could use this "msg" to determine whether it needs to issue stop endpoint with SP (suspend) bit set.

I don't understand. What is the advantage of using FSC?

I'm sorry, I didn't make it clear.

As part of host suspend, controller save state will be issued to save host internal state in xhci_suspend():

If FSC is supported, the cached Slot, Endpoint, Stream, or other Context information are also saved.

Hence, when FSC is supported, software does not have to issue Stop Endpoint Command to push public and private endpoint state into memory as part of system suspend process.

The logic in xhci_device_suspend() will look like:

if xhci_device_suspend() callback was called due to system suspend, (mesg.event & PM_EVENT_SUSPEND is true) and FSC is supported by the xHC implementation, xhci_device_suspend() could ignore stop endpoint command, since CSS will be done in xhc_suspend() later and where all the endpoint caches will be pushed to memory.

Quoted 21 lines · Baolu

Quoted 9 lines · Alan Stern

Quoted 7 lines · Baolu

Quoted 7 lines

include/linux/usb/hcd.h struct hc_driver { · line 393

Quoted 5 lines

include/linux/usb/hcd.h +4 -0

+	void	(*device_suspend)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
+	void	(*device_resume)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
    };

Your callbacks don't use the msg argument. What makes you think it is needed?

Quoted 7 lines

Quoted 6 lines

This msg argument is valuable. XHCI spec defines a capability named FSC (Force Save context Capability). When this capability is implemented, the Save State operation (do during host suspend) shall save any cached Slot, Endpoint, Stream or other Context information to memory. xHCI hcd could use this "msg" to determine whether it needs to issue stop endpoint with SP (suspend) bit set.

I don't understand. What is the advantage of using FSC?

I'm sorry, I didn't make it clear.

As part of host suspend, controller save state will be issued to save host internal state in xhci_suspend():

...

Quoted 3 lines

If FSC is supported, the cached Slot, Endpoint, Stream, or other Context information are also saved.

Quoted 3 lines

Hence, when FSC is supported, software does not have to issue Stop Endpoint Command to push public and private endpoint state into memory as part of system suspend process.

Why do you have to push this state into memory at all? Does the controller hardware lose the cached state information when it is in low power?

Quoted 8 lines

The logic in xhci_device_suspend() will look like:

if xhci_device_suspend() callback was called due to system suspend, (mesg.event & PM_EVENT_SUSPEND is true) and FSC is supported by the xHC implementation, xhci_device_suspend() could ignore stop endpoint command, since CSS will be done in xhc_suspend() later and where all the endpoint caches will be pushed to memory.

I still don't understand this. You said earlier that according to section 4.15.1.1 of the xHCI spec, the endpoint rings should _always_ be stopped with SP set when a device is suspended. Now you're

saying that they don't need to be stopped during a system suspend if the controller supports FSC. (Or maybe you're saying they need to be stopped but SP doesn't need to be set -- it's hard to tell.)

So which is it? Do you need to stop the endpoint rings? Is it okay not to set SP?

Alan Stern

Quoted 9 lines

Quoted 6 lines

If FSC is supported, the cached Slot, Endpoint, Stream, or other Context information are also saved.

Hence, when FSC is supported, software does not have to issue Stop Endpoint Command to push public and private endpoint state into memory as part of system suspend process.

Why do you have to push this state into memory at all? Does the controller hardware lose the cached state information when it is in low power?

I don't think controller hardware will lose the cached state information when it is in low power. But since cache in controller consumes power and resources, by pushing state into memory, hardware can power off the cache logic during suspend. Hence more power saving gains.

Quoted 11 lines

Quoted 8 lines

The logic in xhci_device_suspend() will look like:

if xhci_device_suspend() callback was called due to system suspend, (mesg.event & PM_EVENT_SUSPEND is true) and FSC is supported by the xHC implementation, xhci_device_suspend() could ignore stop endpoint command, since CSS will be done in xhc_suspend() later and where all the endpoint caches will be pushed to memory.

I still don't understand this. You said earlier that according to section 4.15.1.1 of the xHCI spec, the endpoint rings should _always_ be stopped with SP set when a device is suspended. Now you're

The intention of stop endpoint with SP set is to tell hardware that "a device is going to suspend, hardware don't need to contain the endpoint state in internal cache anymore". Hardware _could_ use this hint to push endpoint state into memory to reduce power consumption.

Quoted 3 lines

saying that they don't need to be stopped during a system suspend if the controller supports FSC. (Or maybe you're saying they need to be stopped but SP doesn't need to be set -- it's hard to tell.)

Even FSC is supported, controller hardware still need to push cached endpoint state into memory when a USB device is suspended. The difference is when FSC is enforced, CSS command will push any cached endpoint state into memory unconditionally.

So, when xhci_device_suspend() knows that CSS command will be executed later and CSS command will push cached endpoint state into memory (a.k.a. FSC is enforced), it could skip issuing stop endpoint command with SP bit set to avoid duplication and reduce the suspend time.

This is the case for system suspend since CSS command is part of xhci_suspend() and xhci_suspend() will be executed after all USB devices have been suspended. But it's not case for run-time suspend (auto-pm) since USB device suspend and host controller suspend are independent for run-time case.

That's the reason why I wanted to keep 'msg' parameter. But just as Greg said, we don't need to keep a parameter when it's not used and can add it later when it is required.

Quoted 3 lines

So which is it? Do you need to stop the endpoint rings? Is it okay not to set SP?

"stop endpoint" and "stop endpoint with SP set" serve different purposes in Linux xhci driver as my understanding. "stop endpoint" command is used to stop a active ring when upper layer want to cancel a URB. "stop endpoint with SP set" is used to hint hardware that a USB is going to suspend. Hence "stop endpoint with SP set" must be executed in case that the transfer ring is empty.

Quoted 33 lines · Baolu

Quoted 9 lines

Quoted 6 lines

If FSC is supported, the cached Slot, Endpoint, Stream, or other Context information are also saved.

Hence, when FSC is supported, software does not have to issue Stop Endpoint Command to push public and private endpoint state into memory as part of system suspend process.

Why do you have to push this state into memory at all? Does the controller hardware lose the cached state information when it is in low power?

I don't think controller hardware will lose the cached state information when it is in low power. But since cache in controller consumes power and resources, by pushing state into memory, hardware can power off the cache logic during suspend. Hence more power saving gains.

Quoted 10 lines

Quoted 7 lines

The logic in xhci_device_suspend() will look like:

if xhci_device_suspend() callback was called due to system suspend, (mesg.event & PM_EVENT_SUSPEND is true) and FSC is supported by the xHC implementation, xhci_device_suspend() could ignore stop endpoint command, since CSS will be done in xhc_suspend() later and where all the endpoint caches will be pushed to memory.

I still don't understand this. You said earlier that according to section 4.15.1.1 of the xHCI spec, the endpoint rings should _always_ be stopped with SP set when a device is suspended. Now you're

The intention of stop endpoint with SP set is to tell hardware that "a device is going to suspend, hardware don't need to contain the endpoint state in internal cache anymore". Hardware _could_ use this hint to push endpoint state into memory to reduce power consumption.

Quoted 3 lines

saying that they don't need to be stopped during a system suspend if the controller supports FSC. (Or maybe you're saying they need to be stopped but SP doesn't need to be set -- it's hard to tell.)

Quoted 4 lines

Even FSC is supported, controller hardware still need to push cached endpoint state into memory when a USB device is suspended. The difference is when FSC is enforced, CSS command will push any cached endpoint state into memory unconditionally.

You said above that the hardware _could_ push endpoint state into memory. Now you're saying it _needs_ to do this! Make up your mind.

Quoted 18 lines

So, when xhci_device_suspend() knows that CSS command will be executed later and CSS command will push cached endpoint state into memory (a.k.a. FSC is enforced), it could skip issuing stop endpoint command with SP bit set to avoid duplication and reduce the suspend time.

This is the case for system suspend since CSS command is part of xhci_suspend() and xhci_suspend() will be executed after all USB devices have been suspended. But it's not case for run-time suspend (auto-pm) since USB device suspend and host controller suspend are independent for run-time case.

That's the reason why I wanted to keep 'msg' parameter. But just as Greg said, we don't need to keep a parameter when it's not used and can add it later when it is required.

Quoted 8 lines

Quoted 2 lines

So which is it? Do you need to stop the endpoint rings? Is it okay not to set SP?

"stop endpoint" and "stop endpoint with SP set" serve different purposes in Linux xhci driver as my understanding. "stop endpoint" command is used to stop a active ring when upper layer want to cancel a URB. "stop endpoint with SP set" is used to hint hardware that a USB is going to suspend. Hence "stop endpoint with SP set" must be executed in case that the transfer ring is empty.

(How does the contents of the transfer ring affect anything? Besides, there are never any active URBs when a device gets suspended, so the transfer ring will _always_ be empty at such times.)

This is still extremely confusing. You're not doing a good job of explaining the situation clearly and logically.

Let's see if I understand it correctly:

When the controller goes into suspend, you want the cache to be pushed into memory so that the cache can be powered down, thereby saving additional energy.

If the hardware supports FSC, this will happen automatically.

If the hardware doesn't support FSC, the cached data won't get pushed to memory unless the driver tells the controller to do so at the time the device is suspended. But this will slow things down, so the driver should avoid doing it when it's not needed.

During system suspend you know in advance that the controller will be suspended. Therefore the driver should push the cache to memory if the hardware doesn't support FSC. During runtime suspend you don't know in advance whether the controller will be suspended, so the driver should not push the cache to memory.

But what happens in the case where all the devices have gone into runtime suspend, so the controller also goes into runtime suspend, and the hardware doesn't support FSC? It seems that in this case the cache would have to remain powered on.

Also, it's still not clear what the driver needs to do differently in order to push out the cached data. You have managed to imply both:

Issuing Stop Endpoint with SP set is mandatory whenever a device is suspended, and

The SP bit is what tells the controller to push the endpoint state to memory.

This doesn't make sense.

One last question: How much extra time does it take to push the cached data to memory? Bear in mind that suspending a device is not a rapid operation; if pushing the data takes no more than a few microseconds, I think it would be worthwhile to do it always, even when it's not necessary.

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I'm sorry that I confused you.

FSC is a different thing from what this patch series does.

Quoted 5 lines

Let's see if I understand it correctly:

When the controller goes into suspend, you want the cache to be pushed into memory so that the cache can be powered down, thereby saving additional energy.

Not the controller goes into suspend, but USB devices.

In order to talking to USB devices, xHCI has an endpoint state for each endpoint of a device. When a USB device goes into suspend, xHC driver could ask the hardware to push the state from cache to the memory. Thereby saving additional energy. This is the intention of this patch series.

Quoted 15 lines

If the hardware supports FSC, this will happen automatically.

If the hardware doesn't support FSC, the cached data won't get pushed to memory unless the driver tells the controller to do so at the time the device is suspended. But this will slow things down, so the driver should avoid doing it when it's not needed.

During system suspend you know in advance that the controller will be suspended. Therefore the driver should push the cache to memory if the hardware doesn't support FSC. During runtime suspend you don't know in advance whether the controller will be suspended, so the driver should not push the cache to memory.

Quoted 4 lines

But what happens in the case where all the devices have gone into runtime suspend, so the controller also goes into runtime suspend, and the hardware doesn't support FSC? It seems that in this case the cache would have to remain powered on.

FSC is not part of this patch series. It was introduced when I tried to explain the reason why I kept 'msg' parameter in the callback.

If the hardware support FSC, another operation named "save context operation" will push everything in hardware cache into memory. So when a USB device is going to suspend and xhci_device_suspend() callback is being called, software can do an optimization. That is, if "save context operation" will push everything in hardware cache to memory later, xhci_device_suspend() could skip asking hardware for cache operation and let "save context operation" do it later.

One example of this situation is system suspend. During system suspend, below process will be done for a USB3 fabric.

1) all USB devices suspend
2) root hub suspend
3) host controller suspend

In 1), xhci_device_suspend() call back will be called for each device suspend.
In 3) "save context operation" will be executed.

In this case, if FSC is supported, xhci_device_suspend() could skip asking host hardware for cache operation.

Quoted 10 lines

Also, it's still not clear what the driver needs to do differently in order to push out the cached data. You have managed to imply both:

Issuing Stop Endpoint with SP set is mandatory whenever a device is suspended, and

The SP bit is what tells the controller to push the endpoint state to memory.

This doesn't make sense.

Software doesn't handle the cache directly. XHC driver just tell hardware that a device is going to suspend. XHC driver does this through stop endpoint
command by setting the SP (suspend) bit in the command trb.

During xHC handles stop endpoint command, it will check SP bit, if it's set, xHC knows that the endpoint state could be pushed out to memory now, since device is going to suspend. The operation of cache is very hardware implementation dependent.

Quoted 5 lines

One last question: How much extra time does it take to push the cached data to memory? Bear in mind that suspending a device is not a rapid operation; if pushing the data takes no more than a few microseconds, I think it would be worthwhile to do it always, even when it's not necessary.

It depends on how much time software spends in xhci_device_suspend(). I don't have data in hand, but I agree with you that if this time is no more than a few microseconds, we can let do it always.

Quoted 3 lines · Baolu

I'm sorry that I confused you.

FSC is a different thing from what this patch series does.

I know that. The patch series, in its current form, is fine. Now I'm trying to understand what you originally wanted to do.

Quoted 6 lines

Quoted 5 lines

Let's see if I understand it correctly:

When the controller goes into suspend, you want the cache to be pushed into memory so that the cache can be powered down, thereby saving additional energy.

Not the controller goes into suspend, but USB devices.

Suppose a USB device goes into runtime suspend, and suppose the hardware pushes the cached information for the endpoints on that device into memory. The cache will still contain information for other, non-suspended devices. Consequently the cache _can't_ be powered down at this time, right? You seem to be saying:

When a USB device goes into suspend, you want the cache to be pushed into memory so that the cache can be powered down, thereby saving additional energy.

But that is just wrong! The only time you know the hardware can safely power-down the cache is when the _controller_ goes into suspend. At that time you _know_ all the attached devices are suspended, so the cache isn't needed.

Therefore, when the controller goes into suspend, you want to make sure that the cache has been pushed into memory. The actual push operation can occur earlier, during xhci_device_suspend, but the time when the information's location matters is during xhci_suspend.

Quoted 5 lines

In order to talking to USB devices, xHCI has an endpoint state for each endpoint of a device. When a USB device goes into suspend, xHC driver could ask the hardware to push the state from cache to the memory. Thereby saving additional energy. This is the intention of this patch series.

You don't save any energy when the _device_ goes into suspend. You save energy only when the _controller_ goes into suspend, because that's the only time when the cache can be powered down. Right?

Quoted 23 lines

Quoted 16 lines

If the hardware supports FSC, this will happen automatically.

If the hardware doesn't support FSC, the cached data won't get pushed to memory unless the driver tells the controller to do so at the time the device is suspended. But this will slow things down, so the driver should avoid doing it when it's not needed.

During system suspend you know in advance that the controller will be suspended. Therefore the driver should push the cache to memory if the hardware doesn't support FSC. During runtime suspend you don't know in advance whether the controller will be suspended, so the driver should not push the cache to memory.

Quoted 4 lines

But what happens in the case where all the devices have gone into runtime suspend, so the controller also goes into runtime suspend, and the hardware doesn't support FSC? It seems that in this case the cache would have to remain powered on.

FSC is not part of this patch series. It was introduced when I tried to explain the reason why I kept 'msg' parameter in the callback.

Quoted 7 lines

If the hardware support FSC, another operation named "save context operation" will push everything in hardware cache into memory. So when a USB device is going to suspend and xhci_device_suspend() callback is being called, software can do an optimization. That is, if "save context operation" will push everything in hardware cache to memory later, xhci_device_suspend() could skip asking hardware for cache operation and let "save context operation" do it later.

That's more or less the same as what I wrote above, except that I said it would happen automatically if the hardware supports FSC. It isn't automatic; it requires the driver to issue a "save context operation" command.

Quoted 12 lines

One example of this situation is system suspend. During system suspend, below process will be done for a USB3 fabric.

1) all USB devices suspend
2) root hub suspend
3) host controller suspend

In 1), xhci_device_suspend() call back will be called for each device suspend.
In 3) "save context operation" will be executed.

Quoted 2 lines

In this case, if FSC is supported, xhci_device_suspend() could skip asking host hardware for cache operation.

That's basically what I said.

But now why should "msg" matter? It seems that xhci_device_suspend() should skip asking for the cache operation whenever FSC is supported, regardless of whether you're talking about runtime suspend or system suspend.

Quoted 16 lines

Quoted 11 lines

Also, it's still not clear what the driver needs to do differently in order to push out the cached data. You have managed to imply both:

Issuing Stop Endpoint with SP set is mandatory whenever a device is suspended, and

The SP bit is what tells the controller to push the endpoint state to memory.

This doesn't make sense.

Software doesn't handle the cache directly. XHC driver just tell hardware that a device is going to suspend. XHC driver does this through stop endpoint
command by setting the SP (suspend) bit in the command trb.

Quoted 4 lines

During xHC handles stop endpoint command, it will check SP bit, if it's set, xHC knows that the endpoint state could be pushed out to memory now, since device is going to suspend. The operation of cache is very hardware implementation dependent.

Does that mean the driver has no control over whether "stop endpoint with SP set" will push the cached information into memory? In that case, how can xhci_device_suspend skip asking the hardware for the cache operation?

Quoted 6 lines

Quoted 6 lines

One last question: How much extra time does it take to push the cached data to memory? Bear in mind that suspending a device is not a rapid operation; if pushing the data takes no more than a few microseconds, I think it would be worthwhile to do it always, even when it's not necessary.

Quoted 3 lines

It depends on how much time software spends in xhci_device_suspend(). I don't have data in hand, but I agree with you that if this time is no more than a few microseconds, we can let do it always.

Above you seem to be saying that there's no way to prevent it! Your exact words are: "The operation of cache is very hardware implementation dependent."

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

From software point of view, the extra time it takes to ask for cache
operation can be measured in xhci_device_suspend(). I can measure the data later when I complete my tasks in hand. If that data is ignorable comparing to the suspend time, we can simply do it always.

Quoted 11 lines

include/linux/usb/hcd.h +0 -0

Quoted 10 lines

index 68b1e83..621d9b7 100644

include/linux/usb/hcd.h +2 -1

@@ -383,7 +383,13 @@ struct hc_driver {
 	int	(*find_raw_port_number)(struct usb_hcd *, int);
 	/* Call for power on/off the port if necessary */
 	int	(*port_power)(struct usb_hcd *hcd, int portnum, bool enable);
-
+	/* Call back to hcd when a USB device is going to suspend or just
+	 * resumed.

Quoted 6 lines

include/linux/usb/hcd.h struct hc_driver { · line 393

Quoted 6 lines

include/linux/usb/hcd.h +5 -0

+	 */
+	void	(*device_suspend)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
+	void	(*device_resume)(struct usb_hcd *, struct usb_device *udev,
+			pm_message_t msg);
 };

Your callbacks don't use the msg argument. What makes you think it is needed?

Alan Stern

This patch implements device_suspend/device_resume entries for xHC driver. device_suspend will be called when a USB device is about to suspend. It will issue a stop endpoint command for each endpoint in this device. The Suspend(SP) bit in the command TRB will set which will give xHC a hint about the suspend. device_resume will be called when a USB device is just resumed. It will ring doorbells of all endpoint unconditionally. XHC may use these suspend/resume hints to optimize its operation.

Signed-off-by Lu Baolu <[email protected]>

#else /* CONFIG_PM */

int xhci_set_usb2_hardware_lpm(struct usb_hcd *hcd,

drivers/usb/host/xhci.c +0 -0

@@ -4704,6 +4728,16 @@ int xhci_disable_usb3_lpm_timeout(struct usb_hcd *hcd,
 {
 	return 0;
 }

Quoted 9 lines · Lu Baolu

This patch implements device_suspend/device_resume entries for xHC driver. device_suspend will be called when a USB device is about to suspend. It will issue a stop endpoint command for each endpoint in this device. The Suspend(SP) bit in the command TRB will set which will give xHC a hint about the suspend. device_resume will be called when a USB device is just resumed. It will ring doorbells of all endpoint unconditionally. XHC may use these suspend/resume hints to optimize its operation.

Signed-off-by Lu Baolu <[email protected]>

...

Quoted 7 lines

  #else /* CONFIG_PM */

int xhci_set_usb2_hardware_lpm(struct usb_hcd *hcd,

drivers/usb/host/xhci.c +0 -0

@@ -4704,6 +4728,16 @@ int xhci_disable_usb3_lpm_timeout(struct usb_hcd *hcd,
  {
  	return 0;
  }

drivers/usb/host/xhci.c int xhci_disable_usb3_lpm_timeout(struct usb_hcd *hcd, · line 4740

Quoted 10 lines

drivers/usb/host/xhci.c +10 -0

+
+void xhci_device_suspend(struct usb_hcd *hcd,
+			struct usb_device *udev, pm_message_t msg)
+{
+}
+
+void xhci_device_resume(struct usb_hcd *hcd,
+			struct usb_device *udev, pm_message_t msg)
+{
+}

You don't need to have empty functions. Just do this:

#define xhci_device_suspend NULL
#define xhci_device_resume NULL

in the appropriate place, when CONFIG_PM is not enabled.

Alan Stern