NVMe: Runtime suspend
RFC PATCH 1/1 NVMe: Runtime suspend
Hi, I am new to the mailing list in case my comment mention here was discussed in the past. Please let me know the past discussion thread if it exists. Here is my comment to the patch:
* How is the current NVMe power states defined by a storage device exposed/used in the kernel? I looked at the driver/block/nvme-scsi.c, and only can see that they were mapped to supporting scsi START STOP UNIT cmd inside nvme_trans_power_state(). So is it accurate to say that no (power management) code in linux kernel take advantage of these NVMe power states?
* I think your patch is a good idea, but you should consider to extend it such that it uses runtime power management to create a dynamic power reduction model based on the idleness of IO activity. For example, if the storage drive supports 3 NVMe power states (0, 1, 2), we can let nvme_runtime_suspend() to determine which NVMe power states to use among 0, 1 or 2 based on the IO activity. This way creates a more scalable solution that gives better balance between power saving and performance (i.e. exit latency) in the middle of D0 and D3.
* As for your observation of the potential issue, I think if a device is in suspended mode, driver should allowed to post a command to IO queue as long as the device can be awaken. Of course, the service of the command will be queued and delay serviced only after runtime resume is completed.
/Winson
RFC PATCH 1/1 NVMe: Runtime suspend
Quoted 5 lines · Winson Yung (wyung)
* How is the current NVMe power states defined by a storage device exposed/used in the kernel? I looked at the
driver/block/nvme-scsi.c, and only can see that they were mapped to supporting scsi START STOP UNIT cmd insidenvme_trans_power_state(). So is it accurate to say that no (power management) code in linux kernel take advantage of these NVMe power states?
Yes, that's true that we're not making use of power states in the core driver. You can only test this out on your device with IOCTLs, either passthrough or the SG_IO you mentioned.
Quoted 6 lines
* I think your patch is a good idea, but you should consider to extend it such that it uses runtime power management to create a dynamic power reduction model based on the idleness of IO activity. For example, if the storage drive supports 3 NVMe power states (0, 1, 2), we can let
nvme_runtime_suspend()to determine which NVMe power states to use among 0, 1 or 2 based on the IO activity.
That's kind of the idea, but I was only letting the user choose a policy for idle time until runtime suspending, and which power state to go to. It would take more driver work to allow a policy with more than 2 power states, but maybe we should support more like how the autonomous power transitions feature allows.
Quoted 2 lines
This way creates a more scalable solution that gives better balance between power saving and performance (i.e. exit latency) in the middle of D0 and D3.
I'm not sure we want to take let the device go to D3 at runtime suspend. D3 would require we reinitialize the device and add more overhead in going from idle to resume. Maybe it's acceptable, but I was assuming not.
Quoted 4 lines
* As for your observation of the potential issue, I think if a device is in suspended mode, driver should allowed to post a command to IO queue as long as the device can be awaken. Of course, the service of the command will be queued and delay serviced only after runtime resume is completed.
I actually did get some clarification on this. The spec defines that a device in a non-operational power automatically transistions to the last operational power state that it was in when an IO submission occurs. This was in the autonomous power state section, but I'm told this applies to the device even when not using autonomous power states.
RFC PATCH 1/1 NVMe: Runtime suspend
See my comment inline below
Quoted 19 lines · Winson Yung (wyung)
Quoted 6 lines
* How is the current NVMe power states defined by a storage device exposed/used in the kernel? I looked at the
driver/block/nvme-scsi.c, and only can see that they were mapped to supporting scsi START STOP UNIT cmd insidenvme_trans_power_state(). So is it accurate to say that no (power management) code in linux kernel take advantage of these NVMe power states?Yes, that's true that we're not making use of power states in the core driver. You can only test this out on your device with IOCTLs, either passthrough or the
SG_IOyou mentioned.Quoted 7 lines
* I think your patch is a good idea, but you should consider to extend it such that it uses runtime power management to create a dynamic power reduction model based on the idleness of IO activity. For example, if the storage drive supports 3 NVMe power states (0, 1, 2), we can let
nvme_runtime_suspend()to determine which NVMe power states to use among 0, 1
or 2 based on the IO activity.
Quoted 6 lines
That's kind of the idea, but I was only letting the user choose a policy for idle time until runtime suspending, and which power state to go to. It would take more driver work to allow a policy with more than 2 power states, but maybe we should support more like how the autonomous power transitions feature allows.
Agree, I think beyond 2 power states, the current implementation will be difficult to use, that is why I was proposing to have a dynamic power reduction model based on the idleness of IO activity, and let runtime PM to request different power transition automagically. Each of these NVMe power states definition are very vendor specific, so for sure it will work best with vendor's own driver.
I read from NVMe spec 1.1a, it says D1 and D2 is not recommended to be implemented. D0 is the active states, and device only enter D3 state when system going to S3/4. So it seems making sense to have NVMe PM states for either host OS (to initiate) or device (to enter autonomously) with a shallower sleep state while system is still in S0.
What will happen if host try to initiate entering one of these NVMe PM states while device firmware is also trying to do the same? Is autonomous power transition defined by NVMe implies device firmware can transition device (if capable) to a different NVMe PM state by itself? Should the host OS use set feature cmd to disable autonomous power transition feature in this case?
Quoted 23 lines
Quoted 4 lines
This way creates a more scalable solution that gives better balance between
power saving and performance (i.e. exit latency) in the middle of D0 and D3.I'm not sure we want to take let the device go to D3 at runtime suspend. D3 would require we reinitialize the device and add more overhead in going from idle to resume. Maybe it's acceptable, but I was assuming not.
Quoted 7 lines
* As for your observation of the potential issue, I think if a device is in
suspended mode, driver should allowed to post a command to IO queue as long
as the device can be awaken. Of course, the service of the command will be
queued and delay serviced only after runtime resume is completed.I actually did get some clarification on this. The spec defines that a device in a non-operational power automatically transistions to the last operational power state that it was in when an IO submission occurs. This was in the autonomous power state section, but I'm told this applies to the device even when not using autonomous power states.
RFC PATCH 1/1 NVMe: Runtime suspend
Quoted 6 lines · Winson Yung (wyung)
Agree, I think beyond 2 power states, the current implementation will be difficult to use, that is why I was proposing to have a dynamic power reduction model based on the idleness of IO activity, and let runtime PM to request different power transition automagically. Each of these NVMe power states definition are very vendor specific, so for sure it will work best with vendor's own driver.
Hi Winson,
I just wanted to note that "vendor's own driver" is not really an option. Linux distributions are very unwilling to ship different drivers for the same hardware, and OEMs and ISVs do not want to certify against multiple drivers. This was why NVM Express was formed; to give everybody a common hardware interface and have one driver for many hardware implementations.
We've already seen a couple of hardware vendors try to ship their own Linux driver and been told "No" by their customers and the ISVs. If there are changes you would like to the Linux driver to make your device perform better, let's discuss how we can make that happen. While I work for Intel, I don't work for the part of Intel that makes SSDs.
Quoted 5 lines
I read from NVMe spec 1.1a, it says D1 and D2 is not recommended to be implemented. D0 is the active states, and device only enter D3 state when system going to S3/4. So it seems making sense to have NVMe PM states for either host OS (to initiate) or device (to enter autonomously) with a shallower sleep state while system is still in S0.
D1/D2 are inactive states; reading & writing device registers don't work, so the only point in implementing D1/D2 is for reduced latency to return to D0.
RFC PATCH 1/1 NVMe: Runtime suspend
Hi Matthew, so how you see the NVMe linux driver should support system (i.e. ultrabook and tablet) to achieve better battery life and thermal over temperature protection in S0? Correct me if I am wrong, I don't see current driver has such mechanism put in place beyond the conventional suspend/resume support, even with this patch.
/Winson
-----Original Message-----
From: Matthew Wilcox [mailto:[email protected]]
Sent: Monday, May 19, 2014 2:35 AM
To: Winson Yung (wyung)
Cc: Keith Busch; linux-nvme at lists.infradead.org
Subject: Re: [RFC PATCH] NVMe: Runtime suspend
Quoted 6 lines · Winson Yung (wyung)
Agree, I think beyond 2 power states, the current implementation will be difficult to use, that is why I was proposing to have a dynamic power reduction model based on the idleness of IO activity, and let runtime PM to request different power transition automagically. Each of these NVMe power states definition are very vendor specific, so for sure it will work best with vendor's own driver.
Hi Winson,
I just wanted to note that "vendor's own driver" is not really an option.
Linux distributions are very unwilling to ship different drivers for the same hardware, and OEMs and ISVs do not want to certify against multiple drivers. This was why NVM Express was formed; to give everybody a common hardware interface and have one driver for many hardware implementations.
We've already seen a couple of hardware vendors try to ship their own Linux driver and been told "No" by their customers and the ISVs.
If there are changes you would like to the Linux driver to make your device perform better, let's discuss how we can make that happen. While I work for Intel, I don't work for the part of Intel that makes SSDs.
Quoted 5 lines
I read from NVMe spec 1.1a, it says D1 and D2 is not recommended to be implemented. D0 is the active states, and device only enter D3 state when system going to S3/4. So it seems making sense to have NVMe PM states for either host OS (to initiate) or device (to enter autonomously) with a shallower sleep state while system is still in S0.
D1/D2 are inactive states; reading & writing device registers don't work, so the only point in implementing D1/D2 is for reduced latency to return to D0.
RFC PATCH 1/1 NVMe: Runtime suspend
Quoted 6 lines · Winson Yung (wyung)
Hi Matthew,
so how you see the NVMe linux driver should support system (i.e. ultrabook and tablet) to achieve better battery life and thermal over temperature protection in S0? Correct me if I am wrong, I don't see current driver has such mechanism put in place beyond the conventional suspend/resume support, even with this patch.
This is what the RFC patch in question was proposing to address. The only thing the patch added was supporting NVM-e low power states with a user-defined policy, so I'm not sure what you're missing if you don't see that mechanism in the patch. If all you are asking is to support more than 2 power states, then we can talk about that.
Having the policy be user-defined should make it work exactly how you'd want for a vendor's unique power states without needing a vendor specific driver.
I don't think we want these going into lower D-states as the latency to get out of there is probably too high for a runtime setting: we have to do a device shutdown and full reinitilazation for D-state transitions.
I highly suspect that devices targeted to tablets and ultrabooks will support the autonomous power state transitions feature, but lacking that, I wanted to at least get some feedback if a driver initiated power state transition is even desirable.
Quoted 39 lines
/Winson
-----Original Message-----
From: Matthew Wilcox [mailto:willy at linux.intel.com]
Sent: Monday, May 19, 2014 2:35 AM
To: Winson Yung (wyung)
Cc: Keith Busch; linux-nvme at lists.infradead.org
Subject: Re: [RFC PATCH] NVMe: Runtime suspendQuoted 6 lines · Winson Yung (wyung)
Agree, I think beyond 2 power states, the current implementation will be difficult to use, that is why I was proposing to have a dynamic power reduction model based on the idleness of IO activity, and let runtime PM to request different power transition automagically. Each of these NVMe power states definition are very vendor specific, so for sure it will work best with vendor's own driver.
Hi Winson,
I just wanted to note that "vendor's own driver" is not really an option. Linux distributions are very unwilling to ship different drivers for the same hardware, and OEMs and ISVs do not want to certify against multiple drivers. This was why NVM Express was formed; to give everybody a common hardware interface and have one driver for many hardware implementations.
We've already seen a couple of hardware vendors try to ship their own Linux driver and been told "No" by their customers and the ISVs. If there are changes you would like to the Linux driver to make your device perform better, let's discuss how we can make that happen. While I work for Intel, I don't work for the part of Intel that makes SSDs.
Quoted 5 lines
I read from NVMe spec 1.1a, it says D1 and D2 is not recommended to be implemented. D0 is the active states, and device only enter D3 state when system going to S3/4. So it seems making sense to have NVMe PM states for either host OS (to initiate) or device (to enter autonomously) with a shallower sleep state while system is still in S0.
D1/D2 are inactive states; reading & writing device registers don't work, so the only point in implementing D1/D2 is for reduced latency to return to D0.