Skip to contentSkip to conversation

New: Uncorrected errors reported for AMD GPU

Bug 216373 Bjorn Helgaas 8 participants 18 messages Aug 18, 2022 – Aug 26, 2022 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

#1 lore raw

[Adding amdgpu folks]

Quoted 7 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216373

Bug ID: 216373
Summary: Uncorrected errors reported for AMD GPU
Kernel Version: v6.0-rc1
Regression: No
...

I marked this as a regression in bugzilla.

Quoted 8 lines

Hardware:
CPU: Intel i7-12700K (Alder Lake)
GPU: AMD RX 6700 XT [1002:73df]
Motherboard: ASUS Prime Z690-A

Problem:
After upgrading to v6.0-rc1 the kernel is now reporting uncorrected PCI errors for my GPU.

Thank you very much for the report and for taking the trouble to bisect it and test Kai-Heng's patch!

I suspect that booting with "pci=noaer" should be a temporary workaround for this issue. If it, can you add that to the bugzilla for anybody else who trips over this?

Quoted 3 lines

I have bisected this issue to: [8795e182b02dc87e343c79e73af6b8b7f9c5e635] PCI/portdrv: Don't disable AER reporting in get_port_device_capability() Reverting that commit causes the errors to cease.

I suspect the errors still occur, but we just don't notice and log them.

Quoted 8 lines

I have also tried Kai-Heng Feng's patch[1] which seems to resolve a similar problem, but it did not fix my issue.

[1]
https://lore.kernel.org/linux-pci/[email protected]/

dmesg snippet:

Quoted 7 lines

pcieport 0000:00:01.0: AER: Multiple Uncorrected (Non-Fatal) error received:
0000:03:00.0
amdgpu 0000:03:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
amdgpu 0000:03:00.0: device [1002:73df] error status/mask=00100000/00000000
amdgpu 0000:03:00.0: [20] UnsupReq (First)
amdgpu 0000:03:00.0: AER: TLP Header: 40000001 0000000f 95e7f000 00000000

I think the TLP header decodes to:

0x40000001 = 0100 0000 ... 0000 0001 binary
0x0000000f = 0000 0000 ... 0000 1111 binary

  Fmt           010b                 3 DW header with data
  Type          0000b  010 0 0000    MWr Memory Write Request
  Length        00 0000 0001b        1 DW
  Requester ID  0x0000               00:00.0
  Tag           0x00
  Last DW BE    0000b                must be zero for 1 DW write
  First DW BE   1111b                all 4 bytes in DW enabled
  Address       0x95e7f000
  Data          0x00000000

So I think this is a 32-bit write of zero to PCI bus address 0x95e7f000.

Your dmesg log says:

pci 0000:02:00.0: PCI bridge to [bus 03]
pci 0000:02:00.0: bridge window [mem 0x95e00000-0x95ffffff]
pci 0000:03:00.0: reg 0x24: [mem 0x95e00000-0x95efffff] [drm] register mmio base: 0x95E00000

So this looks like a write to the device's BAR 5. I don't see a PCI reason why this should fail. Maybe there's some amdgpu reason?

Bjorn

maintainer
#2 lore raw

Hi Bjorn,

Am 18.08.22 um 22:38 schrieb Bjorn Helgaas:

Quoted 68 lines

[Adding amdgpu folks]

Quoted 7 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D216373&data=05%7C01%7Cchristian.koenig%40amd.com%7C62cca3872daa46ee7a0a08da8159950a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637964519011973266%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TFF9LWIXBbdrU27%2FbjDfP8FTUhW874X8%2FA0kIrGrjJs%3D&reserved=0

Bug ID: 216373
Summary: Uncorrected errors reported for AMD GPU
Kernel Version: v6.0-rc1
Regression: No
...

I marked this as a regression in bugzilla.

Quoted 8 lines

Hardware:
CPU: Intel i7-12700K (Alder Lake)
GPU: AMD RX 6700 XT [1002:73df]
Motherboard: ASUS Prime Z690-A

Problem:
After upgrading to v6.0-rc1 the kernel is now reporting uncorrected PCI errors for my GPU.

Thank you very much for the report and for taking the trouble to bisect it and test Kai-Heng's patch!

I suspect that booting with "pci=noaer" should be a temporary workaround for this issue. If it, can you add that to the bugzilla for anybody else who trips over this?

Quoted 3 lines

I have bisected this issue to: [8795e182b02dc87e343c79e73af6b8b7f9c5e635] PCI/portdrv: Don't disable AER reporting in get_port_device_capability() Reverting that commit causes the errors to cease.

I suspect the errors still occur, but we just don't notice and log them.

Quoted 8 lines

I have also tried Kai-Heng Feng's patch[1] which seems to resolve a similar problem, but it did not fix my issue.

[1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Flinux-pci%2F20220706123244.18056-1-kai.heng.feng%40canonical.com%2F&data=05%7C01%7Cchristian.koenig%40amd.com%7C62cca3872daa46ee7a0a08da8159950a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637964519011973266%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Y0ofsDYgNGXoQn2e%2BbCM4NHaMOUnEJPqL8lqs1YJzrQ%3D&reserved=0

dmesg snippet:

Quoted 7 lines

pcieport 0000:00:01.0: AER: Multiple Uncorrected (Non-Fatal) error received:
0000:03:00.0
amdgpu 0000:03:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
amdgpu 0000:03:00.0: device [1002:73df] error status/mask=00100000/00000000
amdgpu 0000:03:00.0: [20] UnsupReq (First)
amdgpu 0000:03:00.0: AER: TLP Header: 40000001 0000000f 95e7f000 00000000

I think the TLP header decodes to:

   0x40000001 = 0100 0000 ... 0000 0001 binary
   0x0000000f = 0000 0000 ... 0000 1111 binary

   Fmt           010b                 3 DW header with data
   Type          0000b  010 0 0000    MWr Memory Write Request
   Length        00 0000 0001b        1 DW
   Requester ID  0x0000               00:00.0
   Tag           0x00
   Last DW BE    0000b                must be zero for 1 DW write
   First DW BE   1111b                all 4 bytes in DW enabled
   Address       0x95e7f000
   Data          0x00000000

So I think this is a 32-bit write of zero to PCI bus address 0x95e7f000.

Your dmesg log says:

Quoted 7 lines
   pci 0000:02:00.0: PCI bridge to [bus 03]
   pci 0000:02:00.0:   bridge window [mem 0x95e00000-0x95ffffff]
   pci 0000:03:00.0: reg 0x24: [mem 0x95e00000-0x95efffff]
   [drm] register mmio base: 0x95E00000

So this looks like a write to the device's BAR 5. I don't see a PCI reason why this should fail. Maybe there's some amdgpu reason?

Well I have seen a couple of boards where stuff like that happened, but from my experience this always has some hardware problem as background.

From my understanding what essentially happens is that a write doesn't make it to the device (e.g. transmission errors can't be corrected).

It's quite likely that the write is then either dropped and doesn't matter that much (just clearing the framebuffer for example) or repeated and because of this everything still seems to work fine.

Either way I suggest to try this with some other hartdware configuration. E.g. put the GPU in another system and see if it still gives the same issues or put another GPU into this system.

Regards,
Christian.

Quoted 2 lines

Bjorn

#3 lore raw
Quoted 81 lines · Christian König

Hi Bjorn,

Am 18.08.22 um 22:38 schrieb Bjorn Helgaas:

Quoted 70 lines

[Adding amdgpu folks]

Quoted 8 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D216373&data=05%7C01%7Clijo.lazar%40amd.com%7C59322ae65b814f132a7e08da81b14a95%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637964895716218989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tSdOYv7x%2BO6Rm01OFSDV0j3gevlhTF9lOq9pY2AixRM%3D&reserved=0

             Bug ID: 216373
            Summary: Uncorrected errors reported for AMD GPU Kernel Version: v6.0-rc1
         Regression: No
...

I marked this as a regression in bugzilla.

Quoted 8 lines

Hardware:
CPU: Intel i7-12700K (Alder Lake)
GPU: AMD RX 6700 XT [1002:73df]
Motherboard: ASUS Prime Z690-A

Problem:
After upgrading to v6.0-rc1 the kernel is now reporting uncorrected PCI errors for my GPU.

Thank you very much for the report and for taking the trouble to bisect it and test Kai-Heng's patch!

I suspect that booting with "pci=noaer" should be a temporary workaround for this issue.  If it, can you add that to the bugzilla for anybody else who trips over this?

Quoted 3 lines

I have bisected this issue to: [8795e182b02dc87e343c79e73af6b8b7f9c5e635] PCI/portdrv: Don't disable AER reporting in get_port_device_capability() Reverting that commit causes the errors to cease.

I suspect the errors still occur, but we just don't notice and log them.

Quoted 9 lines

I have also tried Kai-Heng Feng's patch[1] which seems to resolve a similar problem, but it did not fix my issue.

[1]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Flinux-pci%2F20220706123244.18056-1-kai.heng.feng%40canonical.com%2F&data=05%7C01%7Clijo.lazar%40amd.com%7C59322ae65b814f132a7e08da81b14a95%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637964895716218989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7U52%2BsKIHHn1%2B%2F40dbPS38IGBrBYgBxCXAoFKcrTVGU%3D&reserved=0

dmesg snippet:

Quoted 7 lines

pcieport 0000:00:01.0: AER: Multiple Uncorrected (Non-Fatal) error received:
0000:03:00.0
amdgpu 0000:03:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
amdgpu 0000:03:00.0:   device [1002:73df] error status/mask=00100000/00000000 amdgpu 0000:03:00.0:    [20] UnsupReq               (First) amdgpu 0000:03:00.0: AER:   TLP Header: 40000001 0000000f 95e7f000 00000000

I think the TLP header decodes to:

   0x40000001 = 0100 0000 ... 0000 0001 binary
   0x0000000f = 0000 0000 ... 0000 1111 binary

   Fmt           010b                 3 DW header with data Type          0000b  010 0 0000    MWr Memory Write Request Length        00 0000 0001b        1 DW
   Requester ID  0x0000               00:00.0
   Tag           0x00
   Last DW BE    0000b                must be zero for 1 DW write First DW BE   1111b                all 4 bytes in DW enabled Address       0x95e7f000
   Data          0x00000000

So I think this is a 32-bit write of zero to PCI bus address 0x95e7f000.

Your dmesg log says:

Quoted 7 lines

   pci 0000:02:00.0: PCI bridge to [bus 03]
   pci 0000:02:00.0:   bridge window [mem 0x95e00000-0x95ffffff] pci 0000:03:00.0: reg 0x24: [mem 0x95e00000-0x95efffff] [drm] register mmio base: 0x95E00000

So this looks like a write to the device's BAR 5.  I don't see a PCI reason why this should fail.  Maybe there's some amdgpu reason?

Quoted 9 lines

Well I have seen a couple of boards where stuff like that happened, but from my experience this always has some hardware problem as background.

From my understanding what essentially happens is that a write doesn't
make it to the device (e.g. transmission errors can't be corrected).

It's quite likely that the write is then either dropped and doesn't matter that much (just clearing the framebuffer for example) or repeated and because of this everything still seems to work fine.

Either way I suggest to try this with some other hartdware configuration. E.g. put the GPU in another system and see if it still gives the same issues or put another GPU into this system.

Or, it could be amdgpu or some other software component -

register mmio base: 0x95E00000
Address 0x95e7f000

0x95e7f000 indicates access from CPU to a register offset 0x7FE000. This doesn't look like a valid register offset for this chip (device [1002:73df]). Any other clues in dmesg?

Thanks,
Lijo

Quoted 7 lines

Regards,
Christian.

Quoted 2 lines

Bjorn

#4 lore raw
Quoted 8 lines · Lijo

Or, it could be amdgpu or some other software component -

register mmio base: 0x95E00000
Address 0x95e7f000

0x95e7f000 indicates access from CPU to a register offset 0x7FE000. This doesn't look like a valid register offset for this chip (device [1002:73df]). Any other clues in dmesg?

The complete dmesg is at
https://bugzilla.kernel.org/attachment.cgi?id=301596

#5 lore raw
In reply to Bjorn Helgaas
Quoted 9 lines · Bjorn Helgaas

[Adding amdgpu folks]

Quoted 6 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216373

Bug ID: 216373
Summary: Uncorrected errors reported for AMD GPU
Kernel Version: v6.0-rc1
Regression: No

Tom, thanks for trying out "pci=noaer". Hopefully we won't need the workaround for long.

Could I trouble you to try the debug patch below and see if we get any stack trace clues in dmesg when the error happens? I'm sure the experts would have a better approach, but I'm amdgpu-illiterate, so this is all I can do :)

Bjorn

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +22 -2
index c4a6fe3070b6..fc34c66776bc 100644
@@ -130,6 +130,14 @@ const char *amdgpu_asic_name[] = {
 	"LAST",
 };
 
+void check_write(uint32_t v, void __iomem *base, uint32_t offset)
+{
+	if (offset == 0x7f000) {
+		pr_err("** writing %#010x to %px\n", v, base + offset);
+		dump_stack();
+	}
+}
+
 /**
  * DOC: pcie_replay_count
  *
@@ -512,9 +520,10 @@ void amdgpu_mm_wreg8(struct amdgpu_device *adev, uint32_t offset, uint8_t value)
 	if (amdgpu_device_skip_hw_access(adev))
 		return;
 
-	if (offset < adev->rmmio_size)
+	if (offset < adev->rmmio_size) {
+		check_write(value, adev->rmmio, offset);
 		writeb(value, adev->rmmio + offset);
-	else
+	} else
 		BUG();
 }
 
@@ -542,6 +551,7 @@ void amdgpu_device_wreg(struct amdgpu_device *adev,
 			amdgpu_kiq_wreg(adev, reg, v);
 			up_read(&adev->reset_domain->sem);
 		} else {
+			check_write(v, adev->rmmio, reg * 4);
 			writel(v, ((void __iomem *)adev->rmmio) + (reg * 4));
 		}
 	} else {
@@ -574,6 +584,7 @@ void amdgpu_mm_wreg_mmio_rlc(struct amdgpu_device *adev,
 	} else if ((reg * 4) >= adev->rmmio_size) {
 		adev->pcie_wreg(adev, reg * 4, v);
 	} else {
+		check_write(v, adev->rmmio, reg * 4);
 		writel(v, ((void __iomem *)adev->rmmio) + (reg * 4));
 	}
 }
@@ -689,6 +700,7 @@ u32 amdgpu_device_indirect_rreg(struct amdgpu_device *adev,
 	pcie_index_offset = (void __iomem *)adev->rmmio + pcie_index * 4;
 	pcie_data_offset = (void __iomem *)adev->rmmio + pcie_data * 4;
 
+	check_write(reg_addr, adev->rmmio, pcie_index * 4);
 	writel(reg_addr, pcie_index_offset);
 	readl(pcie_index_offset);
 	r = readl(pcie_data_offset);
@@ -721,10 +733,12 @@ u64 amdgpu_device_indirect_rreg64(struct amdgpu_device *adev,
 	pcie_data_offset = (void __iomem *)adev->rmmio + pcie_data * 4;
 
 	/* read low 32 bits */
+	check_write(reg_addr, adev->rmmio, pcie_index * 4);
 	writel(reg_addr, pcie_index_offset);
 	readl(pcie_index_offset);
 	r = readl(pcie_data_offset);
 	/* read high 32 bits */
+	check_write(reg_addr + 4, adev->rmmio, pcie_index * 4);
 	writel(reg_addr + 4, pcie_index_offset);
 	readl(pcie_index_offset);
 	r |= ((u64)readl(pcie_data_offset) << 32);
@@ -755,8 +769,10 @@ void amdgpu_device_indirect_wreg(struct amdgpu_device *adev,
 	pcie_index_offset = (void __iomem *)adev->rmmio + pcie_index * 4;
 	pcie_data_offset = (void __iomem *)adev->rmmio + pcie_data * 4;
 
+	check_write(reg_addr, adev->rmmio, pcie_index * 4);
 	writel(reg_addr, pcie_index_offset);
 	readl(pcie_index_offset);
+	check_write(reg_data, adev->rmmio, pcie_data * 4);
 	writel(reg_data, pcie_data_offset);
 	readl(pcie_data_offset);
 	spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
@@ -785,13 +801,17 @@ void amdgpu_device_indirect_wreg64(struct amdgpu_device *adev,
 	pcie_data_offset = (void __iomem *)adev->rmmio + pcie_data * 4;
 
 	/* write low 32 bits */
+	check_write(reg_addr, adev->rmmio, pcie_index * 4);
 	writel(reg_addr, pcie_index_offset);
 	readl(pcie_index_offset);
+	check_write((u32)(reg_data & 0xffffffffULL), adev->rmmio, pcie_data * 4);
 	writel((u32)(reg_data & 0xffffffffULL), pcie_data_offset);
 	readl(pcie_data_offset);
 	/* write high 32 bits */
+	check_write(reg_addr + 4, adev->rmmio, pcie_index * 4);
 	writel(reg_addr + 4, pcie_index_offset);
 	readl(pcie_index_offset);
+	check_write((u32)(reg_data >> 32), adev->rmmio, pcie_data * 4);
 	writel((u32)(reg_data >> 32), pcie_data_offset);
 	readl(pcie_data_offset);
 	spin_unlock_irqrestore(&adev->pcie_idx_lock, flags);
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +11 -1
index 9be57389301b..b552d7c27ec0 100644
@@ -36,6 +36,8 @@
 #include "soc15.h"
 #include "nv.h"
 
+extern void check_write(uint32_t v, void __iomem *base, uint32_t offset);
+
 #define POPULATE_UCODE_INFO(vf2pf_info, ucode, ver) \
 	do { \
 		vf2pf_info->ucode_info[ucode].id = ucode; \
@@ -900,11 +902,15 @@ static u32 amdgpu_virt_rlcg_reg_rw(struct amdgpu_device *adev, u32 offset, u32 v
 
 	if (offset == reg_access_ctrl->grbm_cntl) {
 		/* if the target reg offset is grbm_cntl, write to scratch_reg2 */
+		check_write(v, adev->rmmio, 4 * reg_access_ctrl->scratch_reg2);
 		writel(v, scratch_reg2);
+		check_write(v, adev->rmmio, offset * 4);
 		writel(v, ((void __iomem *)adev->rmmio) + (offset * 4));
 	} else if (offset == reg_access_ctrl->grbm_idx) {
 		/* if the target reg offset is grbm_idx, write to scratch_reg3 */
+		check_write(v, adev->rmmio, 4 * reg_access_ctrl->scratch_reg3);
 		writel(v, scratch_reg3);
+		check_write(v, adev->rmmio, offset * 4);
 		writel(v, ((void __iomem *)adev->rmmio) + (offset * 4));
 	} else {
 		/*
@@ -913,10 +919,14 @@ static u32 amdgpu_virt_rlcg_reg_rw(struct amdgpu_device *adev, u32 offset, u32 v
 		 * SCRATCH_REG1[19:0]	= address in dword
 		 * SCRATCH_REG1[26:24]	= Error reporting
 		 */
+		check_write(v, adev->rmmio, 4 * reg_access_ctrl->scratch_reg0);
 		writel(v, scratch_reg0);
+		check_write(offset | flag, adev->rmmio, 4 * reg_access_ctrl->scratch_reg1);
 		writel((offset | flag), scratch_reg1);
-		if (reg_access_ctrl->spare_int)
+		if (reg_access_ctrl->spare_int) {
+			check_write(1, adev->rmmio, 4 * reg_access_ctrl->spare_int);
 			writel(1, spare_int);
+		}
 
 		for (i = 0; i < timeout; i++) {
 			tmp = readl(scratch_reg1);
#6 lore raw
Quoted 11 lines · Bjorn Helgaas
Quoted 9 lines · Bjorn Helgaas

[Adding amdgpu folks]

Quoted 6 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216373

Bug ID: 216373
Summary: Uncorrected errors reported for AMD GPU
Kernel Version: v6.0-rc1
Regression: No

Quoted 7 lines

Tom, thanks for trying out "pci=noaer". Hopefully we won't need the workaround for long.

Could I trouble you to try the debug patch below and see if we get any stack trace clues in dmesg when the error happens? I'm sure the experts would have a better approach, but I'm amdgpu-illiterate, so this is all I can do :)

Thanks for doing this, Tom! For everybody else, Tom attached a dmesg log to the bugzilla: https://bugzilla.kernel.org/attachment.cgi?id=301606

Lots of traces of the form:

  amdgpu_device_wreg.part.0.cold+0xb/0x17 [amdgpu]
  amdgpu_gart_invalidate_tlb+0x22/0x60 [amdgpu]
  gmc_v10_0_hw_init+0x44/0x180 [amdgpu]

  amdgpu_device_wreg.part.0.cold+0xb/0x17 [amdgpu]
  gmc_v10_0_hw_init+0xa8/0x180 [amdgpu]

  amdgpu_device_wreg.part.0.cold+0xb/0x17 [amdgpu]
  gmc_v10_0_flush_gpu_tlb+0x35/0x280 [amdgpu]
  amdgpu_gart_invalidate_tlb+0x46/0x60 [amdgpu]
  gmc_v10_0_hw_init+0x44/0x180 [amdgpu]

I tried connecting the dots but I gave up chasing all the function pointers.

#7 lore raw
Quoted 37 lines · Bjorn Helgaas
Quoted 11 lines · Bjorn Helgaas
Quoted 9 lines · Bjorn Helgaas

[Adding amdgpu folks]

Quoted 6 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D216373&amp;data=05%7C01%7Clijo.lazar%40amd.com%7C958674b7d05040ddd93f08da8216107a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637965328525221166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=otI1jfdKInPamMGrLyhFsVoGnK%2BIgL%2BcyMtBVPsV7bE%3D&amp;reserved=0

Bug ID: 216373
Summary: Uncorrected errors reported for AMD GPU
Kernel Version: v6.0-rc1
Regression: No

Quoted 7 lines

Tom, thanks for trying out "pci=noaer". Hopefully we won't need the workaround for long.

Could I trouble you to try the debug patch below and see if we get any stack trace clues in dmesg when the error happens? I'm sure the experts would have a better approach, but I'm amdgpu-illiterate, so this is all I can do :)

Thanks for doing this, Tom! For everybody else, Tom attached a dmesg
log to the bugzilla: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fattachment.cgi%3Fid%3D301606&amp;data=05%7C01%7Clijo.lazar%40amd.com%7C958674b7d05040ddd93f08da8216107a%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637965328525221166%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=pzmtR3daPOimRkX6fohHQTDiEkp8B9aCJX5Z%2BKUcq2c%3D&amp;reserved=0

Lots of traces of the form:

   amdgpu_device_wreg.part.0.cold+0xb/0x17 [amdgpu]
   amdgpu_gart_invalidate_tlb+0x22/0x60 [amdgpu]
   gmc_v10_0_hw_init+0x44/0x180 [amdgpu]

   amdgpu_device_wreg.part.0.cold+0xb/0x17 [amdgpu]
   gmc_v10_0_hw_init+0xa8/0x180 [amdgpu]

   amdgpu_device_wreg.part.0.cold+0xb/0x17 [amdgpu]
   gmc_v10_0_flush_gpu_tlb+0x35/0x280 [amdgpu]
   amdgpu_gart_invalidate_tlb+0x46/0x60 [amdgpu]
   gmc_v10_0_hw_init+0x44/0x180 [amdgpu]
Quoted 3 lines

I tried connecting the dots but I gave up chasing all the function pointers.

Missed the remap part, the offset is here -

https://elixir.bootlin.com/linux/v6.0-rc1/source/drivers/gpu/drm/amd/amdgpu/nv.c#L680

The trace is coming from *_flush_hdp.

You may also check if *_remap_hdp_registers() is getting called. It is done in nbio_vx_y files, most likely this one for your device -
https://elixir.bootlin.com/linux/v6.0-rc1/source/drivers/gpu/drm/amd/amdgpu/nbio_v2_3.c#L68

Thanks,
Lijo

#8 lore raw
Quoted 6 lines · Lijo

Missed the remap part, the offset is here -

https://elixir.bootlin.com/linux/v6.0-rc1/source/drivers/gpu/drm/amd/amdgpu/nv.c#L680

Quoted 8 lines

The trace is coming from *_flush_hdp.

You may also check if *_remap_hdp_registers() is getting called. It is done in nbio_vx_y files, most likely this one for your device -
https://elixir.bootlin.com/linux/v6.0-rc1/source/drivers/gpu/drm/amd/amdgpu/nbio_v2_3.c#L68

Thanks,
Lijo

Hi Lijo,

I would be happy to test any patches that you think would shed some light on this.

#9 lore raw
Quoted 13 lines · Tom Seewald
Quoted 12 lines · Lijo

Missed the remap part, the offset is here -

https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Fv6.0-rc1%2Fsource%2Fdrivers%2Fgpu%2Fdrm%2Famd%2Famdgpu%2Fnv.c%23L680&amp;data=05%7C01%7Clijo.lazar%40amd.com%7Cac6bd5bb5d4143ff9e5808da852982d8%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637968710652869475%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=U5AkO7jfPGP2veXp%2FQkoLY92%2BHNOdMkwTwQCb0tRJPk%3D&amp;reserved=0

The trace is coming from *_flush_hdp.

You may also check if *_remap_hdp_registers() is getting called. It is done in nbio_vx_y files, most likely this one for your device -
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Fv6.0-rc1%2Fsource%2Fdrivers%2Fgpu%2Fdrm%2Famd%2Famdgpu%2Fnbio_v2_3.c%23L68&amp;data=05%7C01%7Clijo.lazar%40amd.com%7Cac6bd5bb5d4143ff9e5808da852982d8%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637968710652869475%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=N4ZbLJuRgddTqMdMo2vD5iJGMMmUJ1MPUVJwVIKThSU%3D&amp;reserved=0

Quoted 8 lines
Quoted 2 lines

Thanks,
Lijo

Hi Lijo,

I would be happy to test any patches that you think would shed some light on this.

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Thanks,
Lijo

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -0
index d7eb23b8d692..743a3ac909ad 100644
@@ -2376,6 +2376,15 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 				DRM_ERROR("amdgpu_vram_scratch_init failed %d\n", r);
 				goto init_failed;
 			}
+
+			/* remap HDP registers to a hole in mmio space,
+			 * for the purpose of expose those registers
+			 * to process space. This is needed for any early HDP
+			 * flush operation
+			 */
+			if (adev->nbio.funcs->remap_hdp_registers && !amdgpu_sriov_vf(adev))
+				adev->nbio.funcs->remap_hdp_registers(adev);
+
 			r = adev->ip_blocks[i].version->funcs->hw_init((void *)adev);
 			if (r) {
 				DRM_ERROR("hw_init %d failed %d\n", i, r);
drivers/gpu/drm/amd/amdgpu/nv.c +0 -6
index b3fba8dea63c..3ac7fef74277 100644
@@ -1032,12 +1032,6 @@ static int nv_common_hw_init(void *handle)
 	nv_program_aspm(adev);
 	/* setup nbio registers */
 	adev->nbio.funcs->init_registers(adev);
-	/* remap HDP registers to a hole in mmio space,
-	 * for the purpose of expose those registers
-	 * to process space
-	 */
-	if (adev->nbio.funcs->remap_hdp_registers && !amdgpu_sriov_vf(adev))
-		adev->nbio.funcs->remap_hdp_registers(adev);
 	/* enable the doorbell aperture */
 	nv_enable_doorbell_aperture(adev, true);
 
drivers/gpu/drm/amd/amdgpu/soc15.c +0 -6
index fde6154f2009..a0481e37d7cf 100644
@@ -1240,12 +1240,6 @@ static int soc15_common_hw_init(void *handle)
 	soc15_program_aspm(adev);
 	/* setup nbio registers */
 	adev->nbio.funcs->init_registers(adev);
-	/* remap HDP registers to a hole in mmio space,
-	 * for the purpose of expose those registers
-	 * to process space
-	 */
-	if (adev->nbio.funcs->remap_hdp_registers && !amdgpu_sriov_vf(adev))
-		adev->nbio.funcs->remap_hdp_registers(adev);
 
 	/* enable the doorbell aperture */
 	soc15_enable_doorbell_aperture(adev, true);
drivers/gpu/drm/amd/amdgpu/soc21.c +0 -6
index 1ff7fc7bb340..60a1cf03fddc 100644
@@ -643,12 +643,6 @@ static int soc21_common_hw_init(void *handle)
 	soc21_program_aspm(adev);
 	/* setup nbio registers */
 	adev->nbio.funcs->init_registers(adev);
-	/* remap HDP registers to a hole in mmio space,
-	 * for the purpose of expose those registers
-	 * to process space
-	 */
-	if (adev->nbio.funcs->remap_hdp_registers)
-		adev->nbio.funcs->remap_hdp_registers(adev);
 	/* enable the doorbell aperture */
 	soc21_enable_doorbell_aperture(adev, true);
 
#10 lore raw
Quoted 5 lines · Lijo

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Thanks,
Lijo

Thank you for the patch. It applied cleanly to v6.0-rc2 and after booting that kernel I no longer see any messages about PCI errors. I have uploaded a dmesg log to the bug report:
https://bugzilla.kernel.org/attachment.cgi?id=301642

#11 lore raw
Quoted 11 lines · Tom Seewald
Quoted 5 lines · Lijo

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Thanks,
Lijo

Thank you for the patch. It applied cleanly to v6.0-rc2 and after booting that kernel I no longer see any messages about PCI errors. I have uploaded a dmesg log to the bug report:
https://bugzilla.kernel.org/attachment.cgi?id=301642

I did not follow this thread in depth, but FWICT the bug is solved now with this patch. So is it correct, that the now fully enabled AER support in the PCI subsystem in v6.0 helped detecting a bug in the AMD GPU driver?

Thanks,
Stefan

maintainer
#12 lore raw

Am 25.08.22 um 08:40 schrieb Stefan Roese:

Quoted 14 lines · Tom Seewald
Quoted 12 lines · Lijo
Quoted 5 lines

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Thanks,
Lijo

Thank you for the patch. It applied cleanly to v6.0-rc2 and after booting that kernel I no longer see any messages about PCI errors. I have uploaded a dmesg log to the bug report:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fattachment.cgi%3Fid%3D301642&amp;data=05%7C01%7Cchristian.koenig%40amd.com%7Cd55a659245b24864bd2d08da8664ae2d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637970065087671063%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&amp;sdata=vbhJ9OB0jIYr%2FRkDIbQHhRRqhyklnnHOT9Xi8z17MYY%3D&amp;reserved=0

Quoted 4 lines

I did not follow this thread in depth, but FWICT the bug is solved now with this patch. So is it correct, that the now fully enabled AER support in the PCI subsystem in v6.0 helped detecting a bug in the AMD GPU driver?

It looks like it, but I'm not 100% sure about the rational behind it.

Lijo can you explain more on this?

Thanks,
Christian.

Quoted 3 lines

Thanks,
Stefan

#13 lore raw
Quoted 15 lines · Christian König

Am 25.08.22 um 08:40 schrieb Stefan Roese:

Quoted 14 lines · Tom Seewald
Quoted 12 lines · Lijo
Quoted 5 lines

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Thanks,
Lijo

Thank you for the patch. It applied cleanly to v6.0-rc2 and after booting that kernel I no longer see any messages about PCI errors. I have uploaded a dmesg log to the bug report:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fattachment.cgi%3Fid%3D301642&amp;data=05%7C01%7Cchristian.koenig%40amd.com%7Cd55a659245b24864bd2d08da8664ae2d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637970065087671063%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&amp;sdata=vbhJ9OB0jIYr%2FRkDIbQHhRRqhyklnnHOT9Xi8z17MYY%3D&amp;reserved=0

Quoted 9 lines
Quoted 4 lines

I did not follow this thread in depth, but FWICT the bug is solved now with this patch. So is it correct, that the now fully enabled AER support in the PCI subsystem in v6.0 helped detecting a bug in the AMD GPU driver?

It looks like it, but I'm not 100% sure about the rational behind it.

Lijo can you explain more on this?

From the trace, during gmc hw_init it takes this route -

gart_enable -> amdgpu_gtt_mgr_recover -> amdgpu_gart_invalidate_tlb -> amdgpu_device_flush_hdp -> amdgpu_asic_flush_hdp (non-ring based HDP flush)

HDP flush is done using remapped offset which is MMIO_REG_HOLE_OFFSET (0x80000 - PAGE_SIZE)

WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);

However, the remapping is not yet done at this point. It's done at a later point during common block initialization. Access to the unmapped offset '(0x80000 - PAGE_SIZE)' seems to come back as unsupported request and reported through AER.

In the patch, I just moved the remapping before gmc block initialization.

Thanks,
Lijo

Quoted 7 lines

Thanks,
Christian.

Quoted 3 lines

Thanks,
Stefan

maintainer
#14 lore raw

Am 25.08.22 um 09:54 schrieb Lazar, Lijo:

Quoted 30 lines · Christian König
Quoted 15 lines

Am 25.08.22 um 08:40 schrieb Stefan Roese:

Quoted 14 lines · Tom Seewald
Quoted 12 lines · Lijo
Quoted 5 lines

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Thanks,
Lijo

Thank you for the patch. It applied cleanly to v6.0-rc2 and after booting that kernel I no longer see any messages about PCI errors. I have uploaded a dmesg log to the bug report:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fattachment.cgi%3Fid%3D301642&amp;data=05%7C01%7Cchristian.koenig%40amd.com%7Cd55a659245b24864bd2d08da8664ae2d%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637970065087671063%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&amp;sdata=vbhJ9OB0jIYr%2FRkDIbQHhRRqhyklnnHOT9Xi8z17MYY%3D&amp;reserved=0

Quoted 9 lines
Quoted 4 lines

I did not follow this thread in depth, but FWICT the bug is solved now with this patch. So is it correct, that the now fully enabled AER support in the PCI subsystem in v6.0 helped detecting a bug in the AMD GPU driver?

It looks like it, but I'm not 100% sure about the rational behind it.

Lijo can you explain more on this?

From the trace, during gmc hw_init it takes this route -

Quoted 7 lines

gart_enable -> amdgpu_gtt_mgr_recover -> amdgpu_gart_invalidate_tlb -> amdgpu_device_flush_hdp -> amdgpu_asic_flush_hdp (non-ring based HDP flush)

HDP flush is done using remapped offset which is MMIO_REG_HOLE_OFFSET (0x80000 - PAGE_SIZE)

WREG32_NO_KIQ((adev->rmmio_remap.reg_offset + KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);

However, the remapping is not yet done at this point. It's done at a later point during common block initialization. Access to the unmapped offset '(0x80000 - PAGE_SIZE)' seems to come back as unsupported request and reported through AER.

That's interesting behavior. So far AER always indicated some kind of transmission error.

When that happens as well on unmapped areas of the MMIO BAR then we need to keep that in mind.

Thanks,
Christian.

Quoted 13 lines

In the patch, I just moved the remapping before gmc block initialization.

Thanks,
Lijo

Quoted 7 lines

Thanks,
Christian.

Quoted 3 lines

Thanks,
Stefan

#15 lore raw
Quoted 36 lines · Christian König

Am 25.08.22 um 09:54 schrieb Lazar, Lijo:

Quoted 27 lines · Christian König
Quoted 8 lines

Am 25.08.22 um 08:40 schrieb Stefan Roese:

Quoted 7 lines · Tom Seewald
Quoted 5 lines
Quoted 2 lines · Lijo

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

...

Quoted 8 lines
Quoted 4 lines

I did not follow this thread in depth, but FWICT the bug is solved now with this patch. So is it correct, that the now fully enabled AER support in the PCI subsystem in v6.0 helped detecting a bug in the AMD GPU driver?

It looks like it, but I'm not 100% sure about the rational behind it.

Lijo can you explain more on this?

From the trace, during gmc hw_init it takes this route -

gart_enable -> amdgpu_gtt_mgr_recover -> amdgpu_gart_invalidate_tlb -> amdgpu_device_flush_hdp -> amdgpu_asic_flush_hdp (non-ring based HDP flush)

HDP flush is done using remapped offset which is MMIO_REG_HOLE_OFFSET (0x80000 - PAGE_SIZE)

Quoted 7 lines

WREG32_NO_KIQ((adev->rmmio_remap.reg_offset +
KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);

However, the remapping is not yet done at this point. It's done at a later point during common block initialization. Access to the unmapped offset '(0x80000 - PAGE_SIZE)' seems to come back as unsupported request and reported through AER.

Quoted 5 lines

That's interesting behavior. So far AER always indicated some kind of transmission error.

When that happens as well on unmapped areas of the MMIO BAR then we need to keep that in mind.

AER can log many different kinds of errors, some related to hardware issues and some related to software.

PCI writes are normally posted and get no response, so AER is the main way to find out about writes to unimplemented addresses.

Reads do get a response, of course, and reads to unimplemented addresses cause errors that most hardware turns into a ~0 data return (in addition to reporting via AER if enabled).

Bjorn

#16 lore raw

Am 25.08.22 um 19:48 schrieb Bjorn Helgaas:

Quoted 41 lines · Christian König
Quoted 32 lines

Am 25.08.22 um 09:54 schrieb Lazar, Lijo:

Quoted 24 lines · Christian König
Quoted 6 lines

Am 25.08.22 um 08:40 schrieb Stefan Roese:

Quoted 5 lines · Tom Seewald
Quoted 4 lines
Quoted 2 lines · Lijo

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

...

I did not follow this thread in depth, but FWICT the bug is solved now with this patch. So is it correct, that the now fully enabled AER support in the PCI subsystem in v6.0 helped detecting a bug in the AMD GPU driver?

It looks like it, but I'm not 100% sure about the rational behind it.

Lijo can you explain more on this?

From the trace, during gmc hw_init it takes this route -

gart_enable -> amdgpu_gtt_mgr_recover -> amdgpu_gart_invalidate_tlb -> amdgpu_device_flush_hdp -> amdgpu_asic_flush_hdp (non-ring based HDP flush)

HDP flush is done using remapped offset which is MMIO_REG_HOLE_OFFSET (0x80000 - PAGE_SIZE)

Quoted 7 lines

WREG32_NO_KIQ((adev->rmmio_remap.reg_offset +
KFD_MMIO_REMAP_HDP_MEM_FLUSH_CNTL) >> 2, 0);

However, the remapping is not yet done at this point. It's done at a later point during common block initialization. Access to the unmapped offset '(0x80000 - PAGE_SIZE)' seems to come back as unsupported request and reported through AER.

Quoted 5 lines

That's interesting behavior. So far AER always indicated some kind of transmission error.

When that happens as well on unmapped areas of the MMIO BAR then we need to keep that in mind.

AER can log many different kinds of errors, some related to hardware issues and some related to software.

Quoted 6 lines

PCI writes are normally posted and get no response, so AER is the main way to find out about writes to unimplemented addresses.

Reads do get a response, of course, and reads to unimplemented addresses cause errors that most hardware turns into a ~0 data return (in addition to reporting via AER if enabled).

The issue is that previous hardware generations reported this through a device specific interrupt.

It's nice to see that this is finally standardized. I'm just wondering if we could retire our hardware specific interrupt handler for this as well.

Christian.

Quoted 2 lines

Bjorn

#17 lore raw
In reply to Lazar, Lijo

Am 2022-08-24 um 01:10 schrieb Lazar, Lijo:

Quoted 21 lines · Tom Seewald
Quoted 18 lines · Lijo
Quoted 16 lines

Missed the remap part, the offset is here -

https://elixir.bootlin.com/linux/v6.0-rc1/source/drivers/gpu/drm/amd/amdgpu/nv.c#L680

The trace is coming from *_flush_hdp.

You may also check if *_remap_hdp_registers() is getting called. It is done in nbio_vx_y files, most likely this one for your device -
https://elixir.bootlin.com/linux/v6.0-rc1/source/drivers/gpu/drm/amd/amdgpu/nbio_v2_3.c#L68

Thanks,
Lijo

Quoted 6 lines
Quoted 5 lines

Hi Lijo,

I would be happy to test any patches that you think would shed some light on this.

Unfortunately, I don't have any NV platforms to test. Attached is an 'untested-patch' based on your trace logs.

Hi Lijo,

I like that the patch also removes some code duplication. Can you check that this doesn't break GFXv8 GPUs? You may need to add a NULL-check for adev->nbio.funcs to the if-condition.

Regards,
  Felix

Quoted 3 lines

Thanks,
Lijo

#18 lore raw

Re: [Bug 216373] New: Uncorrected errors reported for AMD GPU #forregzbot

In reply to Bjorn Helgaas

TWIMC: this mail is primarily send for documentation purposes and for regzbot, my Linux kernel regression tracking bot. These mails usually contain '#forregzbot' in the subject, to make them easy to spot and filter.

[TLDR: I'm adding this regression report to the list of tracked regressions; all text from me you find below is based on a few templates paragraphs you might have encountered already already in similar form.]

Hi, this is your Linux kernel regression tracker.

Quoted 72 lines · Bjorn Helgaas

[Adding amdgpu folks]

Quoted 7 lines · On Wed, Aug 17, 2022 at 11:45:15PM +0000, [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=216373

Bug ID: 216373
Summary: Uncorrected errors reported for AMD GPU
Kernel Version: v6.0-rc1
Regression: No
...

I marked this as a regression in bugzilla.

Quoted 8 lines

Hardware:
CPU: Intel i7-12700K (Alder Lake)
GPU: AMD RX 6700 XT [1002:73df]
Motherboard: ASUS Prime Z690-A

Problem:
After upgrading to v6.0-rc1 the kernel is now reporting uncorrected PCI errors for my GPU.

Thank you very much for the report and for taking the trouble to bisect it and test Kai-Heng's patch!

I suspect that booting with "pci=noaer" should be a temporary workaround for this issue. If it, can you add that to the bugzilla for anybody else who trips over this?

Quoted 3 lines

I have bisected this issue to: [8795e182b02dc87e343c79e73af6b8b7f9c5e635] PCI/portdrv: Don't disable AER reporting in get_port_device_capability() Reverting that commit causes the errors to cease.

I suspect the errors still occur, but we just don't notice and log them.

Quoted 8 lines

I have also tried Kai-Heng Feng's patch[1] which seems to resolve a similar problem, but it did not fix my issue.

[1]
https://lore.kernel.org/linux-pci/[email protected]/

dmesg snippet:

Quoted 7 lines

pcieport 0000:00:01.0: AER: Multiple Uncorrected (Non-Fatal) error received:
0000:03:00.0
amdgpu 0000:03:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
amdgpu 0000:03:00.0: device [1002:73df] error status/mask=00100000/00000000
amdgpu 0000:03:00.0: [20] UnsupReq (First)
amdgpu 0000:03:00.0: AER: TLP Header: 40000001 0000000f 95e7f000 00000000

I think the TLP header decodes to:

0x40000001 = 0100 0000 ... 0000 0001 binary
0x0000000f = 0000 0000 ... 0000 1111 binary

  Fmt           010b                 3 DW header with data
  Type          0000b  010 0 0000    MWr Memory Write Request
  Length        00 0000 0001b        1 DW
  Requester ID  0x0000               00:00.0
  Tag           0x00
  Last DW BE    0000b                must be zero for 1 DW write
  First DW BE   1111b                all 4 bytes in DW enabled
  Address       0x95e7f000
  Data          0x00000000

So I think this is a 32-bit write of zero to PCI bus address 0x95e7f000.

Your dmesg log says:

Quoted 7 lines

pci 0000:02:00.0: PCI bridge to [bus 03]
pci 0000:02:00.0: bridge window [mem 0x95e00000-0x95ffffff]
pci 0000:03:00.0: reg 0x24: [mem 0x95e00000-0x95efffff] [drm] register mmio base: 0x95E00000

So this looks like a write to the device's BAR 5. I don't see a PCI reason why this should fail. Maybe there's some amdgpu reason?

I'd like to add to the tracking to ensure it's not forgotten.

#regzbot introduced: v5.19..v6.0-rc1 ^
https://bugzilla.kernel.org/show_bug.cgi?id=216373
#regzbot title: pci or amdgpu: Uncorrected errors reported for AMD GPU

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.