PCI: device link quirk for NVIDIA GPU
v2 cover PCI: device link quirk for NVIDIA GPU
* v2:
1. Make the pci device link helper function generic which can be used for other multi-function PCI devices also.
2. Minor changes in comments and commit logs.
* v1:
NVIDIA Turing GPU [1] has hardware support for USB Type-C and VirtualLink [2]. The Turing GPU is a multi-function PCI device which has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)
Currently NVIDIA and Nouveau GPU drivers only manage function 0. Rest of the functions are managed by other drivers. These functions internally in the hardware are tightly coupled. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by the other PCI functions, which leads to functional failure. In the mainline kernel, the device link is present between function 0 and function 1. This patch series deals with creating a similar kind of device link between function 0 and functions 2 and 3.
[1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
[2] https://en.wikipedia.org/wiki/VirtualLink
Abhishek Sahu (2):
PCI: Code reorganization for creating device link
PCI: Create device link for NVIDIA GPU
drivers/pci/quirks.c | 79 +++++++++++++++++++++++++++++++++----------- 1 file changed, 59 insertions(+), 20 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/2 PCI: Code reorganization for creating device link
In Multi-function PCI device, one function (consumer) can have hardware functional dependencies on another function (supplier). Whenever the consumer is active and in D0 state, the supplier should also be in D0 state. Currently, the device link is being created from HDA function to VGA function for GPU's. This patch does minor code reorganization. It introduces a helper function which creates device link from consumer pci device to supplier pci device and uses this helper function for creating device link from HDA to VGA. This helper function can be used in future for creating device link from one function to another function.
---
* Changes from v1:
1. Make the helper function generic which takes supplier class, class shift and function number also.
2. Minor changes in commit log
drivers/pci/quirks.c | 53 +++++++++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 20 deletions(-)
drivers/pci/quirks.c +33 -20
@@ -4916,35 +4916,48 @@ static void quirk_fsl_no_msi(struct pci_dev *pdev)DECLARE_PCI_FIXUP_FINAL (PCI_VENDOR_ID_FREESCALE ,PCI_ANY_ID , quirk_fsl_no_msi);/* -* GPUs with integrated HDA controller for streaming audio to attached displays -* need a device link from the HDA controller (consumer) to the GPU (supplier) -* so that the GPU is powered up whenever the HDA controller is accessed. -* The GPU and HDA controller are functions 0 and 1 of the same PCI device. -* The device link stays in place until shutdown (or removal of the PCI device -* if it's hotplugged). Runtime PM is allowed by default on the HDA controller -* to prevent it from permanently keeping the GPU awake. +* Multi-function PCI devices can have hardware functional dependencies from +* one function (consumer) to another function (supplier). Whenever the +* consumer is in D0 state, the supplier should also be in D0 state. This is +* a helper function which creates device link from the consumer to the +* supplier. The device link stays in place until shutdown (or removal of +* the PCI device if it's hotplugged). Runtime PM is allowed by default on +* consumers to prevent it from permanently keeping the supplier awake. */ -static void quirk_gpu_hda (struct pci_dev *hda) +static void pci_create_device_link (struct pci_dev *pdev,unsigned int consumer, +unsigned int supplier,unsigned int class, +unsigned int class_shift) { -struct pci_dev *gpu; +struct pci_dev *supplier_pdev; -if (PCI_FUNC (hda->devfn) !=1 ) +if (PCI_FUNC (pdev->devfn) != consumer)return ; - gpu =pci_get_domain_bus_and_slot (pci_domain_nr (hda->bus), - hda->bus->number, -PCI_DEVFN (PCI_SLOT (hda->devfn),0 )); -if (!gpu || (gpu->class >>16 ) !=PCI_BASE_CLASS_DISPLAY ) { -pci_dev_put (gpu); + supplier_pdev =pci_get_domain_bus_and_slot (pci_domain_nr (pdev->bus), + pdev->bus->number, +PCI_DEVFN (PCI_SLOT (pdev->devfn), supplier)); +if (!supplier_pdev || (supplier_pdev->class >> class_shift) != class) { +pci_dev_put (supplier_pdev);return ; } -if (!device_link_add (&hda->dev, &gpu->dev, -DL_FLAG_STATELESS |DL_FLAG_PM_RUNTIME )) -pci_err (hda,"cannot link HDA to GPU %s\n" ,pci_name (gpu)); +if (device_link_add (&pdev->dev, &supplier_pdev->dev, +DL_FLAG_STATELESS |DL_FLAG_PM_RUNTIME )) +pci_info (pdev,"Linked with %s\n" ,pci_name (supplier_pdev)); +else +pci_err (pdev,"Cannot link with %s\n" ,pci_name (supplier_pdev)); + +pm_runtime_allow (&pdev->dev); +pci_dev_put (supplier_pdev); +} -pm_runtime_allow (&hda->dev); -pci_dev_put (gpu); +/* +* Create device link for GPUs with integrated HDA controller for streaming +* audio to attached displays. +*/ +static void quirk_gpu_hda (struct pci_dev *hda) +{ +pci_create_device_link (hda,1 ,0 ,PCI_BASE_CLASS_DISPLAY ,16 ); }DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_ATI ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda);
Signature · 1 line
2.17.1
PATCH v2 2/2 PCI: Create device link for NVIDIA GPU
In reply to Abhishek SahuNVIDIA Turing GPUs include hardware support for USB Type-C and VirtualLink. It helps in delivering the power, display, and data required to power VR headsets through a single USB Type-C connector. The Turing GPU is a multi-function PCI device. It has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)
The function 0 is tightly coupled with other functions in the hardware. When function 0 goes in D3 state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are being used by other functions which leads to functional failure. So if any of these functions (1/2/3) are in D0 state, then function 0 should also be in D0 state.
'commit 07f4f97d7b4b ("vga_switcheroo: Use device link for HDA controller")' creates the device link from function 1 to function 0. A similar kind of device link needs to be created between function 0 and functions 2 and 3 for NVIDIA Turing GPU.
This patch does the same and creates the required device links. It will make function 0 to be D0 state if any other function is in D0 state.
---
* Changes from v1:
1. Minor changes in commit log
2. used pci_create_device_link() helper function
drivers/pci/quirks.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
drivers/pci/quirks.c +26 -0
@@ -4966,6 +4966,32 @@ DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID ,PCI_CLASS_MULTIMEDIA_HD_AUDIO ,8 , quirk_gpu_hda); +/* +* Create device link for NVIDIA GPU with integrated USB xHCI Host +* controller to VGA. +*/ +static void quirk_gpu_usb (struct pci_dev *usb) +{ +pci_create_device_link (usb,2 ,0 ,PCI_BASE_CLASS_DISPLAY ,16 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_USB ,8 , quirk_gpu_usb); + +/* +* Create device link for NVIDIA GPU with integrated Type-C UCSI controller +* to VGA. Currently there is no class code defined for UCSI device over PCI +* so using UNKNOWN class for now and it will be updated when UCSI +* over PCI gets a class code. +*/ +#define PCI_CLASS_SERIAL_UNKNOWN 0x0c80 +static void quirk_gpu_usb_typec_ucsi (struct pci_dev *ucsi) +{ +pci_create_device_link (ucsi,3 ,0 ,PCI_BASE_CLASS_DISPLAY ,16 ); +} +DECLARE_PCI_FIXUP_CLASS_FINAL (PCI_VENDOR_ID_NVIDIA ,PCI_ANY_ID , +PCI_CLASS_SERIAL_UNKNOWN ,8 , + quirk_gpu_usb_typec_ucsi); +/* * Some IDT switches incorrectly flag an ACS Source Validation error on * completions for config read requests even though PCIe r4.0, sec
Signature · 1 line
2.17.1
Bjorn Helgaas
AppliedQuoted 28 lines · Abhishek Sahu
* v2:
1. Make the pci device link helper function generic which can be used for other multi-function PCI devices also.
2. Minor changes in comments and commit logs.* v1:
NVIDIA Turing GPU [1] has hardware support for USB Type-C and VirtualLink [2]. The Turing GPU is a multi-function PCI device which has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)Currently NVIDIA and Nouveau GPU drivers only manage function 0. Rest of the functions are managed by other drivers. These functions internally in the hardware are tightly coupled. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by the other PCI functions, which leads to functional failure. In the mainline kernel, the device link is present between function 0 and function 1. This patch series deals with creating a similar kind of device link between function 0 and functions 2 and 3.
Quoted 6 lines
[1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
[2] https://en.wikipedia.org/wiki/VirtualLinkAbhishek Sahu (2):
PCI: Code reorganization for creating device link
PCI: Create device link for NVIDIA GPU
Applied to pci/misc for v5.3, thanks!
Quoted 29 lines · Bjorn Helgaas
Quoted 28 lines · Abhishek Sahu
* v2:
1. Make the pci device link helper function generic which can be used for other multi-function PCI devices also.
2. Minor changes in comments and commit logs.* v1:
NVIDIA Turing GPU [1] has hardware support for USB Type-C and VirtualLink [2]. The Turing GPU is a multi-function PCI device which has the following four functions:
- VGA display controller (Function 0)
- Audio controller (Function 1)
- USB xHCI Host controller (Function 2)
- USB Type-C USCI controller (Function 3)Currently NVIDIA and Nouveau GPU drivers only manage function 0. Rest of the functions are managed by other drivers. These functions internally in the hardware are tightly coupled. When function 0 goes in runtime suspended state, then it will do power gating for most of the hardware blocks. Some of these hardware blocks are used by the other PCI functions, which leads to functional failure. In the mainline kernel, the device link is present between function 0 and function 1. This patch series deals with creating a similar kind of device link between function 0 and functions 2 and 3.
Quoted 8 lines
Quoted 6 lines
[1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
[2] https://en.wikipedia.org/wiki/VirtualLinkAbhishek Sahu (2):
PCI: Code reorganization for creating device link
PCI: Create device link for NVIDIA GPUApplied to pci/misc for v5.3, thanks!
Thanks Bjorn for your review and support!
The runtime PM changes in USB Type-C USCI driver is also applied for v5.3
https://marc.info/?l=linux-usb&m=155994544705901&w=2
It will help in achieving run-time PM for Turing GPUs in v5.3.
Regards,
Abhishek