Skip to contentSkip to conversation

RFC about how to obtain PCIE TPH steer-tag on ARM64 platform

DISCUSSION fengchengwen 3 participants 11 messages Jan 12 – Jan 19 PCI

lore
Share & tools
lore
#0 lore raw

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:
1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".
2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:
Option-1: convert logic core ID to ACPI process ID: use get_acpi_id_for_cpu()
to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.
Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to get MPIDR
in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

Option-1 complies with _DSM ECN, but requires BIOS/ACPI to maintain a mapping table from acpi_process_id to MPIDR.
Option 2 does not comply with _DSM ECN (if extension is required). But it is easy to implement and can be extended to the DT system (ACPI is not supported) I think.

Hope for your feedback.

With best regards,
Chengwen

[1] According to _DSM ECN, the input is defined as: "If the target
is a processor, then this field represents the ACPI Processor UID of the processor as specified in the MADT. If the target is a processor container, then this field represents the ACPI Processor UID of the processor container as specified in the PPTT"
[2] git diff about /drivers/pci/tph.c
@@ -289,6 +289,9 @@ int pcie_tph_get_cpu_st(struct pci_dev *pdev, enum tph_mem_type mem_type,

        rp_acpi_handle = ACPI_HANDLE(rp->bus->bridge);

+#ifdef CONFIG_ARM64
+ cpu_uid = get_acpi_id_for_cpu(cpu_uid);
+#endif

        if (tph_invoke_dsm(rp_acpi_handle, cpu_uid, &info) != AE_OK) {
                *tag = 0;
                return -EINVAL;
#1 lore raw
Quoted 9 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

Quoted 14 lines

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:

Option-1: convert logic core ID to ACPI process ID: use
get_acpi_id_for_cpu() to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.

Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to
get MPIDR in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

Quoted 6 lines

Option-1 complies with _DSM ECN, but requires BIOS/ACPI to maintain a mapping table from acpi_process_id to MPIDR.

Option 2 does not comply with _DSM ECN (if extension is required). But it is easy to implement and can be extended to the DT system (ACPI is not supported) I think.

Sounds like this would be of interest to any OS, not just Linux.

Possibly a topic for the PCI-SIG firmware working group
(https://members.pcisig.com/wg/Firmware/dashboard) or the ACPI spec working group (https://uefi.org/workinggroups)?

Quoted 20 lines

[1] According to _DSM ECN, the input is defined as: "If the target is a processor, then this field represents the ACPI Processor UID of the processor as specified in the MADT. If the target is a processor container, then this field represents the ACPI Processor UID of the processor container as specified in the PPTT"

[2] git diff about /drivers/pci/tph.c

(quoted diff) +3 -0
@@ -289,6 +289,9 @@ int pcie_tph_get_cpu_st(struct pci_dev *pdev, enum tph_mem_type mem_type,
 
        rp_acpi_handle = ACPI_HANDLE(rp->bus->bridge);
 
+#ifdef CONFIG_ARM64
+       cpu_uid = get_acpi_id_for_cpu(cpu_uid);
+#endif
        if (tph_invoke_dsm(rp_acpi_handle, cpu_uid, &info) != AE_OK) {
                *tag = 0;
                return -EINVAL;
#2 lore raw
Quoted 10 lines · Bjorn Helgaas
Quoted 8 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 17 lines
Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

Quoted 9 lines

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:

Option-1: convert logic core ID to ACPI process ID: use
get_acpi_id_for_cpu() to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.

In my opinion, if this is achievable in your BIOS/ACPI, it is clean vs. Option-2 and preferred.

Quoted 4 lines
Quoted 4 lines

Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to
get MPIDR in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

This solution requires you to change the ECN and ratify it (as suggested by Bjorn below). The implementation can also be complicated.

Quoted 7 lines
Quoted 7 lines

Option-1 complies with _DSM ECN, but requires BIOS/ACPI to maintain a mapping table from acpi_process_id to MPIDR.

Option 2 does not comply with _DSM ECN (if extension is required). But it is easy to implement and can be extended to the DT system (ACPI is not supported) I think.

If you plan to revamp it, one (wild) idea is that ST retrieval can be extended to support:
1. ACPI _DSM
2. DT
3. Vendor specific

After that, your MPIDR solution can be plugged-in under (3).

Quoted 27 lines

Sounds like this would be of interest to any OS, not just Linux.

Possibly a topic for the PCI-SIG firmware working group
(https://members.pcisig.com/wg/Firmware/dashboard) or the ACPI spec working group (https://uefi.org/workinggroups)?

Quoted 20 lines

[1] According to _DSM ECN, the input is defined as: "If the target is a processor, then this field represents the ACPI Processor UID of the processor as specified in the MADT. If the target is a processor container, then this field represents the ACPI Processor UID of the processor container as specified in the PPTT"

[2] git diff about /drivers/pci/tph.c

(quoted diff) +3 -0
@@ -289,6 +289,9 @@ int pcie_tph_get_cpu_st(struct pci_dev *pdev, enum tph_mem_type mem_type,
 
        rp_acpi_handle = ACPI_HANDLE(rp->bus->bridge);
 
+#ifdef CONFIG_ARM64
+       cpu_uid = get_acpi_id_for_cpu(cpu_uid);
+#endif
        if (tph_invoke_dsm(rp_acpi_handle, cpu_uid, &info) != AE_OK) {
                *tag = 0;
                return -EINVAL;
#3 lore raw
Quoted 23 lines · Wei Huang
Quoted 22 lines · Bjorn Helgaas
Quoted 9 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

Quoted 4 lines

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Option-1: convert logic core ID to ACPI process ID: use
get_acpi_id_for_cpu() to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.

In my opinion, if this is achievable in your BIOS/ACPI, it is clean vs. Option-2 and preferred.

Yes, it's achievable for out BIOS/ACPI, and we will adopt this option.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to
get MPIDR in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

This solution requires you to change the ECN and ratify it (as suggested by Bjorn below). The implementation can also be complicated.

Indeed, and it may take a long time.

@Wei, I have another question:
The cache is hierarchical, for example, L2 or L3. Does the DSM mode support the specified cache level? I checked the ECN doc and there are "Cache Reference" fields, but the kernel code don't enable them.

Quoted 44 lines
Quoted 8 lines
Quoted 8 lines

Option-1 complies with _DSM ECN, but requires BIOS/ACPI to maintain a mapping table from acpi_process_id to MPIDR.

Option 2 does not comply with _DSM ECN (if extension is required). But it is easy to implement and can be extended to the DT system (ACPI is not supported) I think.

If you plan to revamp it, one (wild) idea is that ST retrieval can be extended to support:
1. ACPI _DSM
2. DT
3. Vendor specific

After that, your MPIDR solution can be plugged-in under (3).

Quoted 27 lines

Sounds like this would be of interest to any OS, not just Linux.

Possibly a topic for the PCI-SIG firmware working group
(https://members.pcisig.com/wg/Firmware/dashboard) or the ACPI spec working group (https://uefi.org/workinggroups)?

Quoted 20 lines

[1] According to _DSM ECN, the input is defined as: "If the target is a processor, then this field represents the ACPI Processor UID of the processor as specified in the MADT. If the target is a processor container, then this field represents the ACPI Processor UID of the processor container as specified in the PPTT"

[2] git diff about /drivers/pci/tph.c

(quoted diff) +3 -0
@@ -289,6 +289,9 @@ int pcie_tph_get_cpu_st(struct pci_dev *pdev, enum tph_mem_type mem_type,
 
        rp_acpi_handle = ACPI_HANDLE(rp->bus->bridge);
 
+#ifdef CONFIG_ARM64
+       cpu_uid = get_acpi_id_for_cpu(cpu_uid);
+#endif
        if (tph_invoke_dsm(rp_acpi_handle, cpu_uid, &info) != AE_OK) {
                *tag = 0;
                return -EINVAL;
#4 lore raw
Quoted 40 lines · fengchengwen
Quoted 23 lines · Wei Huang
Quoted 22 lines · Bjorn Helgaas
Quoted 9 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

Quoted 4 lines

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Option-1: convert logic core ID to ACPI process ID: use
get_acpi_id_for_cpu() to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.

In my opinion, if this is achievable in your BIOS/ACPI, it is clean vs. Option-2 and preferred.

Yes, it's achievable for out BIOS/ACPI, and we will adopt this option.

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to
get MPIDR in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

Quoted 7 lines
Quoted 1 line

This solution requires you to change the ECN and ratify it (as suggested by Bjorn below). The implementation can also be complicated.

Indeed, and it may take a long time.

@Wei, I have another question:
The cache is hierarchical, for example, L2 or L3. Does the DSM mode support the specified cache level? I checked the ECN doc and there are "Cache Reference" fields, but the kernel code don't enable them.

ECN does provide the "Cache Reference" field as part of argument2. Our platforms don't use it and don't plan to use it soon. You can explore the use cases if your platforms need it.

I want to point out that PCIe TLP header doesn't have a field for cache level. So no matter what, TLP header will stuck with 8- or 16-bit STs.

Quoted 1 line
#5 lore raw
Quoted 45 lines · Wei Huang
Quoted 44 lines · fengchengwen
Quoted 23 lines · Wei Huang
Quoted 22 lines · Bjorn Helgaas
Quoted 9 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

Quoted 4 lines

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Option-1: convert logic core ID to ACPI process ID: use
get_acpi_id_for_cpu() to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.

In my opinion, if this is achievable in your BIOS/ACPI, it is clean vs. Option-2 and preferred.

Yes, it's achievable for out BIOS/ACPI, and we will adopt this option.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to
get MPIDR in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

This solution requires you to change the ECN and ratify it (as suggested by Bjorn below). The implementation can also be complicated.

Indeed, and it may take a long time.

Quoted 7 lines
Quoted 3 lines

@Wei, I have another question:
The cache is hierarchical, for example, L2 or L3. Does the DSM mode support the specified cache level? I checked the ECN doc and there are "Cache Reference" fields, but the kernel code don't enable them.

ECN does provide the "Cache Reference" field as part of argument2. Our platforms don't use it and don't plan to use it soon. You can explore the use cases if your platforms need it.

I want to point out that PCIe TLP header doesn't have a field for cache level. So no matter what, TLP header will stuck with 8- or 16-bit STs.

Okay, I will try to understand how to use "Cache Reference" as cache level parameter. I think this would give the user a more refined ability.

Quoted 4 lines
Quoted 2 lines
#6 lore raw
In reply to Bjorn Helgaas
Quoted 10 lines · Bjorn Helgaas
Quoted 8 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 7 lines
Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

partial good

Quoted 22 lines
Quoted 15 lines

Because the ARM64 platform generates steer-tag based on the MPIDR information (at least for the Kunpeng platform). Therefore, we have two option:

Option-1: convert logic core ID to ACPI process ID: use
get_acpi_id_for_cpu() to get ACPI process ID in pcie_tph_get_cpu_st() before invoke dsm [2], and BIOS/ACPI use process ID to get corresponding MPIDR, and then generate steer-tag from MPIDR.

Option-2: convert logic core ID to MPIDR: use cpu_logical_map() to
get MPIDR in pcie_tph_get_cpu_st() before invoke dsm, and BIOS/ACPI use it to generate steer-tag directly.

Quoted 6 lines

Option-1 complies with _DSM ECN, but requires BIOS/ACPI to maintain a mapping table from acpi_process_id to MPIDR.

Option 2 does not comply with _DSM ECN (if extension is required). But it is easy to implement and can be extended to the DT system (ACPI is not supported) I think.

Quoted 5 lines

Sounds like this would be of interest to any OS, not just Linux.

Possibly a topic for the PCI-SIG firmware working group
(https://members.pcisig.com/wg/Firmware/dashboard) or the ACPI spec working group (https://uefi.org/workinggroups)?

Thanks
We finally decided to choose option-1.

Quoted 22 lines
Quoted 21 lines

[1] According to _DSM ECN, the input is defined as: "If the target is a processor, then this field represents the ACPI Processor UID of the processor as specified in the MADT. If the target is a processor container, then this field represents the ACPI Processor UID of the processor container as specified in the PPTT"

[2] git diff about /drivers/pci/tph.c

(quoted diff) +3 -0
@@ -289,6 +289,9 @@ int pcie_tph_get_cpu_st(struct pci_dev *pdev, enum tph_mem_type mem_type,
 
        rp_acpi_handle = ACPI_HANDLE(rp->bus->bridge);
 
+#ifdef CONFIG_ARM64
+       cpu_uid = get_acpi_id_for_cpu(cpu_uid);
+#endif
        if (tph_invoke_dsm(rp_acpi_handle, cpu_uid, &info) != AE_OK) {
                *tag = 0;
                return -EINVAL;
#7 lore raw
Quoted 17 lines · fengchengwen
Quoted 16 lines · Bjorn Helgaas
Quoted 14 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

Quoted 3 lines
Quoted 1 line

PG?

partial good

I still don't know what "partial good" means :) Is that something from a spec?

#8 lore raw
Quoted 18 lines · Bjorn Helgaas
Quoted 17 lines · fengchengwen
Quoted 16 lines · Bjorn Helgaas
Quoted 14 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

PG?

partial good

I still don't know what "partial good" means :) Is that something from a spec?

Because of some issues (like manufacturing variances), certain circuits (e.g., some cores) might not work normally. For these cases, we can use the part at a reduced specification, which is the PG mentioned here.

Quoted 1 line
#9 lore raw
Quoted 21 lines · fengchengwen
Quoted 20 lines · Bjorn Helgaas
Quoted 19 lines · fengchengwen
Quoted 17 lines · Bjorn Helgaas
Quoted 9 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

Quoted 5 lines

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

PG?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

partial good

I still don't know what "partial good" means :) Is that something from a spec?

Because of some issues (like manufacturing variances), certain circuits (e.g., some cores) might not work normally. For these cases, we can use the part at a reduced specification, which is the PG mentioned here.

Ah, got it :) If you go this direction, maybe you can expand that part of the commit log a bit, since I don't think "PG" or "partial good" is a widely-known term.

#10 lore raw
Quoted 27 lines · Bjorn Helgaas
Quoted 26 lines · fengchengwen
Quoted 24 lines · Bjorn Helgaas
Quoted 17 lines · fengchengwen
Quoted 16 lines · Bjorn Helgaas
Quoted 14 lines · fengchengwen

Hi all,

We want to enable PCIE TPH feature on ARM64 platform, but we encounter the following problem:

1. The pcie_tph_get_cpu_st() function invokes the ACPI DSM method to obtain the steer-tag of the CPU. According to the definition of the DSM method [1], the cpu_uid should be "ACPI processor uid".

2. In the current implementation, the ACPI DSM method is invoked directly using the logical core number, which works on the x86 platform but does not work on the ARM64 platform because the logical core ID is not the same as the ACPI processor ID when the PG exists.

Quoted 3 lines
Quoted 1 line

PG?

partial good

I still don't know what "partial good" means :) Is that something from a spec?

Quoted 7 lines
Quoted 3 lines

Because of some issues (like manufacturing variances), certain circuits (e.g., some cores) might not work normally. For these cases, we can use the part at a reduced specification, which is the PG mentioned here.

Ah, got it :) If you go this direction, maybe you can expand that part of the commit log a bit, since I don't think "PG" or "partial good" is a widely-known term.

Alright, thank you for the reminder.

Quoted 1 line