PCI: device link quirk for NVIDIA GPU
cover PCI: device link quirk for NVIDIA GPU
NVIDIA Turing GPU [1] has hardware support for USB Type-C and VirtualLink [2]. The Turing GPU is a multi-function PCI device which has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)
Currently NVIDIA and Nouveau GPU drivers only manage function 0. Rest of the functions are managed by other drivers. These functions internally in the hardware are tightly coupled. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by the other PCI functions, which leads to functional failure. In the mainline kernel, the device link is present between function 0 and function 1. This patch series deals with creating a similar kind of device link between function 0 and functions 2 and 3.
[1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
[2] https://en.wikipedia.org/wiki/VirtualLink
Abhishek Sahu (2):
PCI: Code reorganization for VGA device link
PCI: Create device link for NVIDIA GPU
drivers/pci/quirks.c | 67 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 15 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/2 PCI: Code reorganization for VGA device link
This patch does minor code reorganization. It introduces a helper function which creates device link from the non-VGA controller (consumer) to the VGA (supplier) and uses this helper function for creating device link from integrated HDA controller to VGA. It will help in subsequent patches which require a similar kind of device link from USB/Type-C USCI controller to VGA.
--- drivers/pci/quirks.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-)
drivers/pci/quirks.c +29 -15
@@ -4916,36 +4916,50 @@ static void quirk_fsl_no_msi(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_FREESCALE ,PCI_ANY_ID , quirk_fsl_no_msi);/* -* GPUs with integrated HDA controller for streaming audio to attached displays -* need a device link from the HDA controller (consumer) to the GPU (supplier) -* so that the GPU is powered up whenever the HDA controller is accessed. -* The GPU and HDA controller are functions 0 and 1 of the same PCI device. -* The device link stays in place until shutdown (or removal of the PCI device -* if it's hotplugged). Runtime PM is allowed by default on the HDA controller -* to prevent it from permanently keeping the GPU awake. +* GPUs can be multi-function PCI device which can contain controllers other +* than VGA (like Audio, USB, etc.). Internally in the hardware, these non-VGA +* controllers are tightly coupled with VGA controller. Whenever these +* controllers are runtime active, the VGA controller should also be in active +* state. Normally, in these GPUs, the VGA controller is present at function 0. +* +* This is a helper function which creates device link from the non-VGA +* controller (consumer) to the VGA (supplier). The device link stays in place +* until shutdown (or removal of the PCI device if it's hotplugged). +* Runtime PM is allowed by default on these non-VGA controllers to prevent +* it from permanently keeping the GPU awake. */ -static void quirk_gpu_hda (struct pci_dev *hda) +static void +pci_create_device_link_with_vga (struct pci_dev *pdev,unsigned int devfn) {struct pci_dev *gpu; -if (PCI_FUNC (hda->devfn) !=1 ) +if (PCI_FUNC (pdev->devfn) != devfn)return ; - gpu =pci_get_domain_bus_and_slot (pci_domain_nr (hda->bus), - hda->bus->number, -PCI_DEVFN (PCI_SLOT (hda->devfn),0 )); + gpu =pci_get_domain_bus_and_slot (pci_domain_nr (pdev->bus), + pdev->bus->number, +PCI_DEVFN (PCI_SLOT (pdev->devfn),0 ));if (!gpu || (gpu->class >>16 ) !=PCI_BASE_CLASS_DISPLAY ) {pci_dev_put (gpu);return ; } -if (!device_link_add (&hda->dev, &gpu->dev, +if (!device_link_add (&pdev->dev, &gpu->dev,DL_FLAG_STATELESS |DL_FLAG_PM_RUNTIME )) -pci_err (hda,"cannot link HDA to GPU %s\n" ,pci_name (gpu)); +pci_err (pdev,"cannot link with VGA %s\n" ,pci_name (gpu)); -pm_runtime_allow (&hda->dev); +pm_runtime_allow (&pdev->dev);pci_dev_put (gpu); } + +/* +* Create device link for GPUs with integrated HDA controller for streaming +* audio to attached displays. +*/ +static void quirk_gpu_hda (struct pci_dev *hda) +{ +pci_create_device_link_with_vga (hda,1 ); +}DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_ATI ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda); DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,
Signature · 1 line
2.17.1
Re: [PATCH 1/2] PCI: Code reorganization for VGA device link
[+cc Lukas]
Quoted 32 lines · Abhishek Sahu
This patch does minor code reorganization. It introduces a helper function which creates device link from the non-VGA controller (consumer) to the VGA (supplier) and uses this helper function for creating device link from integrated HDA controller to VGA. It will help in subsequent patches which require a similar kind of device link from USB/Type-C USCI controller to VGA.
Signed-off-by Abhishek Sahu <[email protected]>--- drivers/pci/quirks.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-)drivers/pci/quirks.c +5
-7@@ -4916,36 +4916,50 @@ static void quirk_fsl_no_msi(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_FREESCALE ,PCI_ANY_ID , quirk_fsl_no_msi);/* -* GPUs with integrated HDA controller for streaming audio to attached displays -* need a device link from the HDA controller (consumer) to the GPU (supplier) -* so that the GPU is powered up whenever the HDA controller is accessed. -* The GPU and HDA controller are functions 0 and 1 of the same PCI device. -* The device link stays in place until shutdown (or removal of the PCI device -* if it's hotplugged). Runtime PM is allowed by default on the HDA controller -* to prevent it from permanently keeping the GPU awake. +* GPUs can be multi-function PCI device which can contain controllers other +* than VGA (like Audio, USB, etc.). Internally in the hardware, these non-VGA +* controllers are tightly coupled with VGA controller. Whenever these +* controllers are runtime active, the VGA controller should also be in active +* state. Normally, in these GPUs, the VGA controller is present at function 0.
drivers/pci/quirks.c static void quirk_fsl_no_msi(struct pci_dev *pdev) · line 4932
Quoted 10 lines
drivers/pci/quirks.c +8
-1+* +* This is a helper function which creates device link from the non-VGA +* controller (consumer) to the VGA (supplier). The device link stays in place +* until shutdown (or removal of the PCI device if it's hotplugged). +* Runtime PM is allowed by default on these non-VGA controllers to prevent +* it from permanently keeping the GPU awake. */ -static void quirk_gpu_hda (struct pci_dev *hda) +static void +pci_create_device_link_with_vga (struct pci_dev *pdev,unsigned int devfn)
There's nothing in this functionality that depends on VGA, so let's remove "GPU, "VGA", etc from the description, the function name, the local variable name, and the log message. Maybe you need to allow the caller to supply the class type (PCI_BASE_CLASS_DISPLAY for current users, but Lukas mentioned a NIC that might be able to use this too).
Follow the prevailing indentation style, with return type and function name on the same line, i.e.,
static void pci_create_device_link(...)
Quoted 13 lines
drivers/pci/quirks.c +4
-4{struct pci_dev *gpu; -if (PCI_FUNC (hda->devfn) !=1 ) +if (PCI_FUNC (pdev->devfn) != devfn)return ; - gpu =pci_get_domain_bus_and_slot (pci_domain_nr (hda->bus), - hda->bus->number, -PCI_DEVFN (PCI_SLOT (hda->devfn),0 )); + gpu =pci_get_domain_bus_and_slot (pci_domain_nr (pdev->bus), + pdev->bus->number, +PCI_DEVFN (PCI_SLOT (pdev->devfn),0 ));
drivers/pci/quirks.c static void quirk_fsl_no_msi(struct pci_dev *pdev) · line 4949
Quoted 10 lines
drivers/pci/quirks.c +2
-2if (!gpu || (gpu->class >>16 ) !=PCI_BASE_CLASS_DISPLAY ) {pci_dev_put (gpu);return ; } -if (!device_link_add (&hda->dev, &gpu->dev, +if (!device_link_add (&pdev->dev, &gpu->dev,DL_FLAG_STATELESS |DL_FLAG_PM_RUNTIME )) -pci_err (hda,"cannot link HDA to GPU %s\n" ,pci_name (gpu)); +pci_err (pdev,"cannot link with VGA %s\n" ,pci_name (gpu));
I think we should emit a message in the success case, too. There is one in device_link_add(), but it's a dev_dbg() so we can't count on it being in the log. I'd like a pci_info() that we can count on.
Quoted 19 lines
drivers/pci/quirks.c +10
-1-pm_runtime_allow (&hda->dev); +pm_runtime_allow (&pdev->dev);pci_dev_put (gpu); } + +/* +* Create device link for GPUs with integrated HDA controller for streaming +* audio to attached displays. +*/ +static void quirk_gpu_hda (struct pci_dev *hda) +{ +pci_create_device_link_with_vga (hda,1 ); +}DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_ATI ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda); DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,Signature · 1 line
2.17.1
Re: [PATCH 1/2] PCI: Code reorganization for VGA device link
Quoted 41 lines · Bjorn Helgaas
[+cc Lukas]
Quoted 38 lines · Abhishek Sahu
This patch does minor code reorganization. It introduces a helper function which creates device link from the non-VGA controller (consumer) to the VGA (supplier) and uses this helper function for creating device link from integrated HDA controller to VGA. It will help in subsequent patches which require a similar kind of device link from USB/Type-C USCI controller to VGA.
Signed-off-by Abhishek Sahu <[email protected]>--- drivers/pci/quirks.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-)drivers/pci/quirks.c +11
-7@@ -4916,36 +4916,50 @@ static void quirk_fsl_no_msi(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_FREESCALE ,PCI_ANY_ID , quirk_fsl_no_msi);/* -* GPUs with integrated HDA controller for streaming audio to attached displays -* need a device link from the HDA controller (consumer) to the GPU (supplier) -* so that the GPU is powered up whenever the HDA controller is accessed. -* The GPU and HDA controller are functions 0 and 1 of the same PCI device. -* The device link stays in place until shutdown (or removal of the PCI device -* if it's hotplugged). Runtime PM is allowed by default on the HDA controller -* to prevent it from permanently keeping the GPU awake. +* GPUs can be multi-function PCI device which can contain controllers other +* than VGA (like Audio, USB, etc.). Internally in the hardware, these non-VGA +* controllers are tightly coupled with VGA controller. Whenever these +* controllers are runtime active, the VGA controller should also be in active +* state. Normally, in these GPUs, the VGA controller is present at function 0. +* +* This is a helper function which creates device link from the non-VGA +* controller (consumer) to the VGA (supplier). The device link stays in place +* until shutdown (or removal of the PCI device if it's hotplugged). +* Runtime PM is allowed by default on these non-VGA controllers to prevent +* it from permanently keeping the GPU awake.
Quoted 8 lines
drivers/pci/quirks.c static void quirk_fsl_no_msi(struct pci_dev *pdev) · line 4932
Quoted 4 lines
drivers/pci/quirks.c +2
-1*/ -static void quirk_gpu_hda (struct pci_dev *hda) +static void +pci_create_device_link_with_vga (struct pci_dev *pdev,unsigned int devfn)There's nothing in this functionality that depends on VGA, so let's remove "GPU, "VGA", etc from the description, the function name, the local variable name, and the log message. Maybe you need to allow the
Thanks. Then we can make this function generic where we can pass device link supplier and supplier pci class mask. It will help in creating device link from one function (other than 0 also) to any another function. Later on, same can be used by non GPUs devices also, if required.
Quoted 6 lines
caller to supply the class type (
PCI_BASE_CLASS_DISPLAYfor current users, but Lukas mentioned a NIC that might be able to use this too).Follow the prevailing indentation style, with return type and function name on the same line, i.e.,
I will fix in v2.
Quoted 20 lines
static void
pci_create_device_link(...)Quoted 18 lines
drivers/pci/quirks.c +4
-4{struct pci_dev *gpu; -if (PCI_FUNC (hda->devfn) !=1 ) +if (PCI_FUNC (pdev->devfn) != devfn)return ; - gpu =pci_get_domain_bus_and_slot (pci_domain_nr (hda->bus), - hda->bus->number, -PCI_DEVFN (PCI_SLOT (hda->devfn),0 )); + gpu =pci_get_domain_bus_and_slot (pci_domain_nr (pdev->bus), + pdev->bus->number, +PCI_DEVFN (PCI_SLOT (pdev->devfn),0 ));if (!gpu || (gpu->class >>16 ) !=PCI_BASE_CLASS_DISPLAY ) {pci_dev_put (gpu);return ; }
Quoted 10 lines
drivers/pci/quirks.c static void quirk_fsl_no_msi(struct pci_dev *pdev) · line 4949
Quoted 5 lines
drivers/pci/quirks.c +2
-2-if (!device_link_add (&hda->dev, &gpu->dev, +if (!device_link_add (&pdev->dev, &gpu->dev,DL_FLAG_STATELESS |DL_FLAG_PM_RUNTIME )) -pci_err (hda,"cannot link HDA to GPU %s\n" ,pci_name (gpu)); +pci_err (pdev,"cannot link with VGA %s\n" ,pci_name (gpu));I think we should emit a message in the success case, too. There is one in
device_link_add(), but it's adev_dbg()so we can't count on it being in the log. I'd like apci_info()that we can count on.
I will add this in v2.
Regards,
Abhishek
Quoted 19 lines
Quoted 19 lines
drivers/pci/quirks.c +10
-1-pm_runtime_allow (&hda->dev); +pm_runtime_allow (&pdev->dev);pci_dev_put (gpu); } + +/* +* Create device link for GPUs with integrated HDA controller for streaming +* audio to attached displays. +*/ +static void quirk_gpu_hda (struct pci_dev *hda) +{ +pci_create_device_link_with_vga (hda,1 ); +}DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_ATI ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda); DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,Signature · 1 line
2.17.1
PATCH 2/2 PCI: Create device link for NVIDIA GPU
In reply to Abhishek SahuNVIDIA Turing GPUs include hardware support for USB Type-C and VirtualLink. It helps in delivering the power, display, and data required to power VR headsets through a single USB Type-C connector. The Turing GPU is a multi-function PCI device has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)
The function 0 is tightly coupled with other functions in the hardware. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by other functions which leads to functional failure. So if any of these functions (1/2/3) are active, then function 0 should also be in active state. 'commit 07f4f97d7b4b ("vga_switcheroo: Use device link for HDA controller")' creates the device link from function 1 to function 0. A similar kind of device link needs to be created between function 0 and functions 2 and 3 for NVIDIA Turing GPU.
This patch does the same and create the required device links. It will make function 0 to be runtime PM active if other functions are still active.
--- drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
drivers/pci/quirks.c +23 -0
@@ -4967,6 +4967,29 @@ DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda); +/* Create device link for NVIDIA GPU with integrated USB controller to VGA. */ +static void quirk_gpu_usb (struct pci_dev *usb) +{ +pci_create_device_link_with_vga (usb,2 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_USB ,8 , quirk_gpu_usb); + +/* +* Create device link for NVIDIA GPU with integrated Type-C UCSI controller +* to VGA. Currently there is no class code defined for UCSI device over PCI +* so using UNKNOWN class for now and it will be updated when UCSI +* over PCI gets a class code. +*/ +#define PCI_CLASS_SERIAL_UNKNOWN 0x0c80 +static void quirk_gpu_usb_typec_ucsi (struct pci_dev *ucsi) +{ +pci_create_device_link_with_vga (ucsi,3 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_UNKNOWN ,8 , + quirk_gpu_usb_typec_ucsi); +/* * Some IDT switches incorrectly flag an ACS Source Validation error on * completions for config read requests even though PCIe r4.0, sec
Signature · 1 line
2.17.1
Re: [PATCH 2/2] PCI: Create device link for NVIDIA GPU
[+cc Lukas, author of 07f4f97d7b4b ("vga_switcheroo: Use device link for HDA controller")]
Quoted 6 lines · Abhishek Sahu
NVIDIA Turing GPUs include hardware support for USB Type-C and VirtualLink. It helps in delivering the power, display, and data required to power VR headsets through a single USB Type-C connector. The Turing GPU is a multi-function PCI device has the following four functions:
Quoted 7 lines
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)The function 0 is tightly coupled with other functions in the hardware. When function 0 goes in runtime suspended state,
"Runtime suspended" is a Linux concept, not a PCI concept. Please replace this with the appropriate PCI term, e.g., "D3hot" or whatever it is.
Quoted 4 lines
then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by other functions which leads to functional failure. So if any of these functions (1/2/3) are active, then function 0 should also be in active state.
Instead of "active" and "active state", please use the specific states required in terms of PCI.
Quoted 4 lines
'commit
07f4f97d7b4b("vga_switcheroo: Use device link for HDA controller")' creates the device link from function 1 to function 0. A similar kind of device link needs to be created between function 0 and functions 2 and 3 for NVIDIA Turing GPU.
I can't point to language that addresses this, but this sounds like a case of the GPU not conforming to the PCI spec. The general assumption is that the OS should be able to discover everything it needs to do power management directly from the architected PCI config space.
It is definitely not ideal to have to add quirks like this for devices designed this way. Such quirks force us to do otherwise unnecessary OS updates as new devices are released.
If all the devices in a multi-function device were connected intimately enough that they all had to be managed by the same driver, I could imagine putting these non-discoverable dependencies in the driver. But these devices don't seem to be related in that way.
If there *is* spec language that allows dependencies like this, please include the citation in your commit log.
Quoted 20 lines
This patch does the same and create the required device links. It will make function 0 to be runtime PM active if other functions are still active.
Signed-off-by Abhishek Sahu <[email protected]>--- drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)drivers/pci/quirks.c +3
-0@@ -4967,6 +4967,29 @@ DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda); +/* Create device link for NVIDIA GPU with integrated USB controller to VGA. */ +static void quirk_gpu_usb (struct pci_dev *usb) +{
drivers/pci/quirks.c DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID, · line 4982
Quoted 10 lines
drivers/pci/quirks.c +10
-0+pci_create_device_link_with_vga (usb,2 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_USB ,8 , quirk_gpu_usb); + +/* +* Create device link for NVIDIA GPU with integrated Type-C UCSI controller +* to VGA. Currently there is no class code defined for UCSI device over PCI +* so using UNKNOWN class for now and it will be updated when UCSI +* over PCI gets a class code.
Ugh. Here's a good example of having to do yet another OS update.
Quoted 16 lines
drivers/pci/quirks.c +10
-0+*/ +#define PCI_CLASS_SERIAL_UNKNOWN 0x0c80 +static void quirk_gpu_usb_typec_ucsi (struct pci_dev *ucsi) +{ +pci_create_device_link_with_vga (ucsi,3 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_UNKNOWN ,8 , + quirk_gpu_usb_typec_ucsi); +/* * Some IDT switches incorrectly flag an ACS Source Validation error on * completions for config read requests even though PCIe r4.0, sec Signature · 1 line
2.17.1
Re: [PATCH 2/2] PCI: Create device link for NVIDIA GPU
Thanks Bjorn for your review.
Quoted 15 lines · Bjorn Helgaas
[+cc Lukas, author of
07f4f97d7b4b("vga_switcheroo: Use device link for HDA controller")]Quoted 11 lines · Abhishek Sahu
NVIDIA Turing GPUs include hardware support for USB Type-C and VirtualLink. It helps in delivering the power, display, and data required to power VR headsets through a single USB Type-C connector. The Turing GPU is a multi-function PCI device has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)
Quoted 6 lines
Quoted 2 lines
The function 0 is tightly coupled with other functions in the hardware. When function 0 goes in runtime suspended state,
"Runtime suspended" is a Linux concept, not a PCI concept. Please replace this with the appropriate PCI term, e.g., "D3hot" or whatever it is.
Sure. I will change this.
Quoted 8 lines
Quoted 5 lines
then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by other functions which leads to functional failure. So if any of these functions (1/2/3) are active, then function 0 should also be in active state.
Instead of "active" and "active state", please use the specific states required in terms of PCI.
Sure. I will use specific states name.
Quoted 11 lines
Quoted 5 lines
'commit
07f4f97d7b4b("vga_switcheroo: Use device link for HDA controller")' creates the device link from function 1 to function 0. A similar kind of device link needs to be created between function 0 and functions 2 and 3 for NVIDIA Turing GPU.I can't point to language that addresses this, but this sounds like a case of the GPU not conforming to the PCI spec. The general assumption is that the OS should be able to discover everything it needs to do power management directly from the architected PCI config space.
The GPU is following PCIe spec but following is the implementation from HW side
Normal GPU has VGA and Audio controller in which Audio is dependent upon VGA. For these GPU, the VGA is managed by GPU driver and Audio is managed by sound driver. Now the VGA driver can go to D3hot while Audio still in D0 if there was no device link (added in commit 07f4f97d7b4b). It would lead to Audio functionality failure. The device link is making sure that GPU is in D0 while Audio is D0.
Now the NVIDIA Turing GPU has one USB Type-C port mainly to support virtual reality headset. With default mode, this USB port will act as normal USB port and any USB device can be connected over it. It will be managed with PCI xHCI USB controller driver. Now, to support VR headset, This USB Type-C alternate mode is going to be used. This alternate mode setting will be managed with [1] and [2] which is part of 5.2-rc1. More detail for this is available in [3] and virtual link open industry standard [4]. These VR frame-buffers are internally going to be rendered by GPU only and managed by function 0 (VGA) driver. Now, from HW side, we need to make sure VGA is in D0 while any of other functions are in D0.
[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/usb/typec/altmodes/nvidia.c
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/i2c/busses/i2c-nvidia-gpu.c
[3]
https://fullcirclemagazine.org/2019/04/30/nvidia-creates-free-virtual-link-driver-for-linux/
[4] https://sites.google.com/view/virtuallink-consortium/home
Quoted 4 lines
It is definitely not ideal to have to add quirks like this for devices designed this way. Such quirks force us to do otherwise unnecessary OS updates as new devices are released.
I can understand but this is the HW requirement. To support, VR headset in GPU, the HW has provided USB Type-C. Now, from SW side, we are working on supporting this with runtime PM. Currently runtime PM is not possible without adding the dependencies between different functions.
Quoted 8 lines
If all the devices in a multi-function device were connected intimately enough that they all had to be managed by the same driver, I could imagine putting these non-discoverable dependencies in the driver. But these devices don't seem to be related in that way.
Quoted 1 line
If there *is* spec language that allows dependencies like this, please
include the citation in your commit log.
The PCIe specification treats each function separately but GPU case is different. So, it won't be part of PCIe spec. in GPU, the different kind of devices are internally coupled in HW but still needs to be managed by different driver.
Quoted 17 lines
Quoted 17 lines
This patch does the same and create the required device links. It will make function 0 to be runtime PM active if other functions are still active.
Signed-off-by Abhishek Sahu <[email protected]>--- drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)drivers/pci/quirks.c +0
-0@@ -4967,6 +4967,29 @@ DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda);
Quoted 16 lines
drivers/pci/quirks.c DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID, · line 4982
Quoted 13 lines
drivers/pci/quirks.c +13
-0+/* Create device link for NVIDIA GPU with integrated USB controller to VGA. */ +static void quirk_gpu_usb (struct pci_dev *usb) +{ +pci_create_device_link_with_vga (usb,2 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_USB ,8 , quirk_gpu_usb); + +/* +* Create device link for NVIDIA GPU with integrated Type-C UCSI controller +* to VGA. Currently there is no class code defined for UCSI device over PCI +* so using UNKNOWN class for now and it will be updated when UCSI +* over PCI gets a class code. Ugh. Here's a good example of having to do yet another OS update.
Correct. But currently we don't have any other way. Same thing we did for
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/i2c/busses/i2c-nvidia-gpu.c#n248
Once it gets a class code then we can replace at all the places.
Regards,
Abhishek
Quoted 16 lines
Quoted 16 lines
drivers/pci/quirks.c +10
-0+*/ +#define PCI_CLASS_SERIAL_UNKNOWN 0x0c80 +static void quirk_gpu_usb_typec_ucsi (struct pci_dev *ucsi) +{ +pci_create_device_link_with_vga (ucsi,3 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_UNKNOWN ,8 , + quirk_gpu_usb_typec_ucsi); +/* * Some IDT switches incorrectly flag an ACS Source Validation error on * completions for config read requests even though PCIe r4.0, sec Signature · 1 line
2.17.1
Re: [PATCH 2/2] PCI: Create device link for NVIDIA GPU
Quoted 8 lines · Abhishek Sahu
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line
If there *is* spec language that allows dependencies like this, please
include the citation in your commit log.
The PCIe specification treats each function separately but GPU case is different. So, it won't be part of PCIe spec. in GPU, the different kind of devices are internally coupled in HW but still needs to be managed by different driver.
GPUs aren't the only devices which have these implicit dependencies.
E.g. Broadcom BCM57765 combines a Gigabit Ethernet chip in function 0 (14e4:16b4) with an SD card reader in function 1 (14e4:16bc) and AFAIK for the latter to work, the former needs to be in D0. I think we're incorrectly not enforcing this dependency in the kernel yet. That chip was built into many MacBooks.
Indeed the inability to specify such inter-function power dependencies in config space seems to be a gaping hole in the PCI spec. Maybe we should always assume that function 0 needs to be in D0 for any other function to work? Not sure if that would be safe.
Thanks,
Lukas
Re: [PATCH 2/2] PCI: Create device link for NVIDIA GPU
In reply to Abhishek Sahu[+cc Rafael, just FYI]
Quoted 25 lines · Abhishek Sahu
Quoted 24 lines · Bjorn Helgaas
Quoted 22 lines · Abhishek Sahu
NVIDIA Turing GPUs include hardware support for USB Type-C and VirtualLink. It helps in delivering the power, display, and data required to power VR headsets through a single USB Type-C connector. The Turing GPU is a multi-function PCI device has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)The function 0 is tightly coupled with other functions in the hardware. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by other functions which leads to functional failure. So if any of these functions (1/2/3) are active, then function 0 should also be in active state.
'commit
07f4f97d7b4b("vga_switcheroo: Use device link for HDA controller")' creates the device link from function 1 to function 0. A similar kind of device link needs to be created between function 0 and functions 2 and 3 for NVIDIA Turing GPU.
Quoted 8 lines
Quoted 5 lines
I can't point to language that addresses this, but this sounds like a case of the GPU not conforming to the PCI spec. The general assumption is that the OS should be able to discover everything it needs to do power management directly from the architected PCI config space.
The GPU is following PCIe spec but following is the implementation from HW side
Unless you can find spec language that talks about D-state dependencies between functions, I claim this is not following the PCIe spec. For example, PCIe r5.0, sec 1.4, says "the PCI/PCIe hardware/software model includes architectural constructs necessary to discover, configure, and use a Function, without needing Function- specific knowledge." Sec 5.1 says "D states are associated with a particular Function" and "PM provides ... a mechanism to identify power management capabilities of a given Function [and] the ability to transition a Function into a certain power management state."
If there *is* something about dependencies between functions in the spec, we should improve the generic PCI core to pay attention to that, and then we wouldn't need this quirk.
If the spec doesn't provide a way to discover them, these dependencies are exceptions from the spec, and we have to handle them as hardware defects, using quirks like this. That's fine, but let's not pretend that this is a conforming device and that adding quirks is the expected process. Just call a spade a spade and say we're working around a defect in this particular device.
I think the best path forward would be to add this quirk for the existing device, and then pursue a spec change to add something like a new PCIe capability to describe the dependencies. Then we could enhance the PCI core once and power management for future devices would "Just Work" without having to add quirks.
Bjorn
Re: [PATCH 2/2] PCI: Create device link for NVIDIA GPU
Quoted 37 lines · Bjorn Helgaas
[+cc Rafael, just FYI]
Quoted 25 lines · Abhishek Sahu
Quoted 24 lines · Bjorn Helgaas
Quoted 22 lines · Abhishek Sahu
NVIDIA Turing GPUs include hardware support for USB Type-C and VirtualLink. It helps in delivering the power, display, and data required to power VR headsets through a single USB Type-C connector. The Turing GPU is a multi-function PCI device has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)The function 0 is tightly coupled with other functions in the hardware. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by other functions which leads to functional failure. So if any of these functions (1/2/3) are active, then function 0 should also be in active state.
'commit
07f4f97d7b4b("vga_switcheroo: Use device link for HDA controller")' creates the device link from function 1 to function 0. A similar kind of device link needs to be created between function 0 and functions 2 and 3 for NVIDIA Turing GPU.Quoted 8 lines
Quoted 5 lines
I can't point to language that addresses this, but this sounds like a case of the GPU not conforming to the PCI spec. The general assumption is that the OS should be able to discover everything it needs to do power management directly from the architected PCI config space.
The GPU is following PCIe spec but following is the implementation from HW side
Quoted 10 lines
Unless you can find spec language that talks about D-state dependencies between functions, I claim this is not following the PCIe spec. For example, PCIe r5.0, sec 1.4, says "the PCI/PCIe hardware/software model includes architectural constructs necessary to discover, configure, and use a Function, without needing Function- specific knowledge." Sec 5.1 says "D states are associated with a particular Function" and "PM provides ... a mechanism to identify power management capabilities of a given Function [and] the ability to transition a Function into a certain power management state."
Thanks Bjorn. Here in case of GPU's these functions are not completely independent so it is not following PCIe spec in that aspect.
Quoted 11 lines
If there *is* something about dependencies between functions in the spec, we should improve the generic PCI core to pay attention to that, and then we wouldn't need this quirk.
If the spec doesn't provide a way to discover them, these dependencies are exceptions from the spec, and we have to handle them as hardware defects, using quirks like this. That's fine, but let's not pretend that this is a conforming device and that adding quirks is the expected process. Just call a spade a spade and say we're working around a defect in this particular device.
Yes. I am agree with that we need to be very careful in adding quirks like this. I will communicate the same to HW team so they can explore other options to handle this in HW design side for future chips.
Quoted 6 lines
I think the best path forward would be to add this quirk for the existing device, and then pursue a spec change to add something like a new PCIe capability to describe the dependencies. Then we could enhance the PCI core once and power management for future devices would "Just Work" without having to add quirks.
Yes. It will be long term process. If other HW has similar requirement then it would be good to have this.
Regards,
Abhishek