Skip to contentSkip to conversation

3.6.11 AMD-Vi: Completion-Wait loop timed out

DISCUSSION Udo van den Heuvel 7 participants 38 messages Jan 20, 2013 – Apr 21, 2013 IOMMU

lore
Share & tools
lore
#0 lore raw

Hello,

See below for a part of the logging on this F2A85X-UP4 with AMD a10-5800k. Box was raid checking I guess.

Jan 20 03:42:08 s3 rsyslogd: [origin software="rsyslogd"
swVersion="5.8.10" x-pid="3031" x-info="http://www.rsyslog.com"]
rsyslogd was HUPed
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:18 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:18 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:18 s3 kernel: ------------[ cut here ]------------
Jan 20 04:11:18 s3 kernel: WARNING: at drivers/iommu/amd_iommu.c:1104
__domain_flush_pages+0x1ad/0x1b0()
Jan 20 04:11:18 s3 kernel: Hardware name: To be filled by O.E.M.
Jan 20 04:11:18 s3 kernel: Modules linked in: vfat fat usb_storage pwc

udf crc_itu_t nfsv3 nfs bnep bluetooth fuse cpufreq_userspace nf_conntrack_netbios_ns eeprom nf_conntrack_broadcast ipt_REJECT ip6t_REJECT it87 iptable_filter hwmon_vid xt_tcpudp ipt_MASQUERADE nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_defrag_ipv4 nf_conntrack ip6table_filter ip_tables ip6_tables x_tables dm_mirror dm_region_hash dm_log ext2 snd_usb_audio snd_usbmidi_lib snd_hwdep snd_rawmidi snd_hda_codec_realtek videobuf2_vmalloc videobuf2_memops videobuf2_core cdc_ether hid_generic videodev binfmt_misc radeon cfbfillrect snd_hda_intel cfbimgblt snd_hda_codec fbcon bitblit cfbcopyarea snd_seq softcursor i2c_algo_bit snd_seq_device font backlight powernow_k8 mperf drm_kms_helper kvm_amd ttm snd_pcm kvm drm fb snd_page_alloc snd_timer snd fbdev k10temp microcode evdev i2c_piix4 xhci_hcd button nfsd exportfs auth_rpcgss nfs_acl lockd sunrpc autofs4 usbhid ehci_hcd ohci_hcd sr_mod cdrom [last unloaded

Jan 20 04:11:18 s3 kernel: : pwc]
Jan 20 04:11:18 s3 kernel: Pid: 506, comm: irq/43-ahci Not tainted
3.6.11 #19
Jan 20 04:11:18 s3 kernel: Call Trace:
Jan 20 04:11:18 s3 kernel: [<ffffffff8103c679>] ?
warn_slowpath_common+0x79/0xc0
Jan 20 04:11:18 s3 kernel: [<ffffffff8134598d>] ?
__domain_flush_pages+0x1ad/0x1b0
Jan 20 04:11:18 s3 kernel: [<ffffffff81345c2b>] ?
__unmap_single.isra.24+0xdb/0x110
Jan 20 04:11:18 s3 kernel: [<ffffffff81346615>] ? unmap_sg+0x55/0xb0
Jan 20 04:11:18 s3 kernel: [<ffffffff812a9a61>] ? ata_sg_clean+0x61/0xd0
Jan 20 04:11:18 s3 kernel: [<ffffffff812b039d>] ?
ata_scsi_qc_complete+0x5d/0x420
Jan 20 04:11:18 s3 kernel: [<ffffffff812a9cc0>] ?
__ata_qc_complete+0x40/0x130
Jan 20 04:11:18 s3 kernel: [<ffffffff812aa05a>] ?
ata_qc_complete_multiple+0x7a/0xc0
Jan 20 04:11:30 s3 kernel: [<ffffffff812c1d2f>] ? ahci_interrupt+0xaf/0x710
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c8e0>] ? irq_thread_fn+0x40/0x40
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c903>] ?
irq_forced_thread_fn+0x23/0x50
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c67b>] ? irq_thread+0x11b/0x180
Jan 20 04:11:30 s3 kernel: [<ffffffff81060d8c>] ? __wake_up_common+0x4c/0x80
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c7e0>] ?
irq_finalize_oneshot+0x100/0x100
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c560>] ?
wake_threads_waitq+0x50/0x50
Jan 20 04:11:30 s3 kernel: [<ffffffff81058d75>] ? kthread+0x85/0x90
Jan 20 04:11:30 s3 kernel: [<ffffffff8141ec34>] ?
kernel_thread_helper+0x4/0x10
Jan 20 04:11:30 s3 kernel: [<ffffffff81058cf0>] ?
kthread_freezable_should_stop+0x50/0x50
Jan 20 04:11:30 s3 kernel: [<ffffffff8141ec30>] ? gs_change+0xb/0xb
Jan 20 04:11:30 s3 kernel: ---[ end trace 73ac82546fadadb1 ]---
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: ------------[ cut here ]------------

And many more of the WARNINGs.

What went wrong? How to fix?

Kind regards,
Udo

#1 lore raw

I know just the guy, CCed. :-)

Quoted 100 lines · Udo van den Heuvel

Hello,

See below for a part of the logging on this F2A85X-UP4 with AMD a10-5800k. Box was raid checking I guess.

Jan 20 03:42:08 s3 rsyslogd: [origin software="rsyslogd"
swVersion="5.8.10" x-pid="3031" x-info="http://www.rsyslog.com"]
rsyslogd was HUPed
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:17 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:18 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:18 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:18 s3 kernel: ------------[ cut here ]------------
Jan 20 04:11:18 s3 kernel: WARNING: at drivers/iommu/amd_iommu.c:1104
__domain_flush_pages+0x1ad/0x1b0()
Jan 20 04:11:18 s3 kernel: Hardware name: To be filled by O.E.M.
Jan 20 04:11:18 s3 kernel: Modules linked in: vfat fat usb_storage pwc

udf crc_itu_t nfsv3 nfs bnep bluetooth fuse cpufreq_userspace nf_conntrack_netbios_ns eeprom nf_conntrack_broadcast ipt_REJECT ip6t_REJECT it87 iptable_filter hwmon_vid xt_tcpudp ipt_MASQUERADE nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_defrag_ipv4 nf_conntrack ip6table_filter ip_tables ip6_tables x_tables dm_mirror dm_region_hash dm_log ext2 snd_usb_audio snd_usbmidi_lib snd_hwdep snd_rawmidi snd_hda_codec_realtek videobuf2_vmalloc videobuf2_memops videobuf2_core cdc_ether hid_generic videodev binfmt_misc radeon cfbfillrect snd_hda_intel cfbimgblt snd_hda_codec fbcon bitblit cfbcopyarea snd_seq softcursor i2c_algo_bit snd_seq_device font backlight powernow_k8 mperf drm_kms_helper kvm_amd ttm snd_pcm kvm drm fb snd_page_alloc snd_timer snd fbdev k10temp microcode evdev i2c_piix4 xhci_hcd button nfsd exportfs auth_rpcgss nfs_acl lockd sunrpc autofs4 usbhid ehci_hcd ohci_hcd sr_mod cdrom [last unloaded

Jan 20 04:11:18 s3 kernel: : pwc]
Jan 20 04:11:18 s3 kernel: Pid: 506, comm: irq/43-ahci Not tainted
3.6.11 #19
Jan 20 04:11:18 s3 kernel: Call Trace:
Jan 20 04:11:18 s3 kernel: [<ffffffff8103c679>] ?
warn_slowpath_common+0x79/0xc0
Jan 20 04:11:18 s3 kernel: [<ffffffff8134598d>] ?
__domain_flush_pages+0x1ad/0x1b0
Jan 20 04:11:18 s3 kernel: [<ffffffff81345c2b>] ?
__unmap_single.isra.24+0xdb/0x110
Jan 20 04:11:18 s3 kernel: [<ffffffff81346615>] ? unmap_sg+0x55/0xb0
Jan 20 04:11:18 s3 kernel: [<ffffffff812a9a61>] ? ata_sg_clean+0x61/0xd0
Jan 20 04:11:18 s3 kernel: [<ffffffff812b039d>] ?
ata_scsi_qc_complete+0x5d/0x420
Jan 20 04:11:18 s3 kernel: [<ffffffff812a9cc0>] ?
__ata_qc_complete+0x40/0x130
Jan 20 04:11:18 s3 kernel: [<ffffffff812aa05a>] ?
ata_qc_complete_multiple+0x7a/0xc0
Jan 20 04:11:30 s3 kernel: [<ffffffff812c1d2f>] ? ahci_interrupt+0xaf/0x710
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c8e0>] ? irq_thread_fn+0x40/0x40
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c903>] ?
irq_forced_thread_fn+0x23/0x50
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c67b>] ? irq_thread+0x11b/0x180
Jan 20 04:11:30 s3 kernel: [<ffffffff81060d8c>] ? __wake_up_common+0x4c/0x80
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c7e0>] ?
irq_finalize_oneshot+0x100/0x100
Jan 20 04:11:30 s3 kernel: [<ffffffff8109c560>] ?
wake_threads_waitq+0x50/0x50
Jan 20 04:11:30 s3 kernel: [<ffffffff81058d75>] ? kthread+0x85/0x90
Jan 20 04:11:30 s3 kernel: [<ffffffff8141ec34>] ?
kernel_thread_helper+0x4/0x10
Jan 20 04:11:30 s3 kernel: [<ffffffff81058cf0>] ?
kthread_freezable_should_stop+0x50/0x50
Jan 20 04:11:30 s3 kernel: [<ffffffff8141ec30>] ? gs_change+0xb/0xb
Jan 20 04:11:30 s3 kernel: ---[ end trace 73ac82546fadadb1 ]---
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: AMD-Vi: Completion-Wait loop timed out
Jan 20 04:11:30 s3 kernel: ------------[ cut here ]------------

And many more of the WARNINGs.

What went wrong? How to fix?

Kind regards,
Udo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
maintainer
#3 lore raw
Quoted 8 lines · Udo van den Heuvel

Hello,

Quoted 1 line · Borislav Petkov

I know just the guy, CCed. :-)

Thanks for the quick response!
I found this similar case:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1073384

Yes, this is a Hardware issue for which the BIOS does not apply the workaround. The only solution for now is to disable the IOMMU on the Trinity based chips. Unfortunatly I don't have access to the hardware any longer to write a workaround in the AMD IOMMU driver.

The question is what to do now, I tend to disable the IOMMU if a Trinity chip is detected. This is not the first report of this problem I encountered.

Regards,

Joerg

#4 lore raw

Hello Jörg,

Quoted 4 lines · Jörg Rödel
Quoted 3 lines · Udo van den Heuvel

Hello,

On 2013-01-20 11:36, Borislav Petkov wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I know just the guy, CCed. :-)

Thanks for the quick response!
I found this similar case:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1073384

Yes, this is a Hardware issue for which the BIOS does not apply the workaround.

Hardware issue? What is wrong c.q. happening?

I have this:

# dmesg|grep IOMMU

[    0.000000] ACPI: IVRS 000000009dd12420 00070 (v02  AMD   AMDIOMMU
00000001 AMD  00000000)
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    1.125636] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

So kernel says I have no IOMMU but still one is found? (!?)

Quoted 2 lines

The only solution for now is to disable the IOMMU on the Trinity based chips.

In PC-BIOS I assume?
I did not yet find an option, but this is the first occurrence. Can the BIOS vendor fix this? If so: please explain so I cna contact Gigabyte (motherboard manufacturer)

Quoted 3 lines

The question is what to do now, I tend to disable the IOMMU if a Trinity chip is detected. This is not the first report of this problem I encountered.

I know, see the URL I posted.
What is the impact of disabling the IOMMU?

Udo

maintainer
#5 lore raw
Quoted 3 lines · Udo van den Heuvel

Hello Jörg,

Hardware issue? What is wrong c.q. happening?

I think it falls under Erratum 455 (which does not mention IOMMU specifically). Point is, there is a hardware workaround for this to make the IOMMU work, but your BIOS does not enable it.

Quoted 10 lines

I have this:

# dmesg|grep IOMMU

[    0.000000] ACPI: IVRS 000000009dd12420 00070 (v02  AMD   AMDIOMMU
00000001 AMD  00000000)
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    1.125636] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

So kernel says I have no IOMMU but still one is found? (!?)

The "Please enable IOMMU ..." line tells you about the GART, not the AMD IOMMU. This is a frequent source of confusion, we probably should remove that line. Trinity has no GART, so there is nothing to find :)

Quoted 4 lines

In PC-BIOS I assume?
I did not yet find an option, but this is the first occurrence. Can the BIOS vendor fix this? If so: please explain so I cna contact Gigabyte (motherboard manufacturer)

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Quoted 1 line

What is the impact of disabling the IOMMU?

Well, it has some security impact and if you have more than 4GB of RAM maybe also some slight performance impact due to DMA bounce buffering. But thats still better as a system that stops working after some time.

Joerg

#6 lore raw
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

Guys, the error description is at
http://marc.info/?l=linux-kernel&m=135867802432660

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#7 lore raw
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

Btw, you're running the latest BIOS from them, I assume?

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#8 lore raw
Quoted 3 lines · Borislav Petkov
Quoted 1 line

Right, Udo, you can try Gigabyte first.

Btw, you're running the latest BIOS from them, I assume?

Nope. But I am beyond their first released BIOS, I am running one of their beta BIOSes. I am two beta updates behind current with F3g. They list as description for BIOS F3k:

1.    Beta BIOS
2.    Modify option of APU and memory voltage
3.    Modify option of CPU PWM switch rate
4.    Modify memory compatibility
5.    Modify ET6 compatibility

Udo

#9 lore raw
Quoted 9 lines · Udo van den Heuvel
Quoted 3 lines · Borislav Petkov
Quoted 1 line

Right, Udo, you can try Gigabyte first.

Btw, you're running the latest BIOS from them, I assume?

Nope. But I am beyond their first released BIOS, I am running one of their beta BIOSes. I am two beta updates behind current with F3g. They list as description for BIOS F3k:

Quoted 5 lines
1.    Beta BIOS
2.    Modify option of APU and memory voltage
3.    Modify option of CPU PWM switch rate
4.    Modify memory compatibility
5.    Modify ET6 compatibility

Yeah, fixes lists are not always exhaustive, especially with BIOS. You could try the latest if you can downgrade it easily if something breaks with F3k.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#10 lore raw
In reply to Borislav Petkov
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

I just did so and referred to the kernel.org bugzilla.

Quoted 2 lines

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

That would be most helpful!
Of course I can help testing but the issue happened only 1 time so far.

The person from
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1073384 had more bad luck in experiencing the issue.

Thanks,
Udo

maintainer
#11 lore raw
In reply to Borislav Petkov
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

BorisO is no longer with AMD afaik. I wrote an email to Sherry and Suravee and asked them to either send me hardware to write the fix on my own or to send a fix for the issue. Let's see what happens...

Joerg

#12 lore raw
Quoted 1 line · Jörg Rödel

BorisO is no longer with AMD afaik.

Why am I not surprised...

Quoted 3 lines

I wrote an email to Sherry and Suravee and asked them to either send me hardware to write the fix on my own or to send a fix for the issue. Let's see what happens...

Btw, while we're at it, here's some more h0rkage from my PD box:

[    0.220022] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.220078] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.220132] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)
Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#13 lore raw
Quoted 8 lines · Borislav Petkov
Quoted 1 line

Let's see what happens...

Btw, while we're at it, here's some more h0rkage from my PD box:

[    0.220022] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.220078] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.220132] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

I have:

# dmesg|grep -i amd-vi

[    1.125636] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.125701] AMD-Vi:  Extended features:  PreF PPR GT IA
[    1.131725] AMD-Vi: Lazy IO/TLB flushing enabled

Is that 'OK'?

Udo

#14 lore raw
Quoted 10 lines · Udo van den Heuvel
Quoted 8 lines · Borislav Petkov
Quoted 1 line

Let's see what happens...

Btw, while we're at it, here's some more h0rkage from my PD box:

[    0.220022] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.220078] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.220132] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)
Quoted 8 lines

I have:

# dmesg|grep -i amd-vi

[    1.125636] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.125701] AMD-Vi:  Extended features:  PreF PPR GT IA
[    1.131725] AMD-Vi: Lazy IO/TLB flushing enabled

Is that 'OK'?

That's simply dumping the IOMMU extended features and yes, it is ok.

Mine happen when enabling CONFIG_IRQ_REMAP and they're somewhat related. Anyways, I decided to show them to Joerg so that he's aware.

Btw, you could try enabling that on your machine and see whether IRQ remapping works there.

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
maintainer
#15 lore raw
In reply to Borislav Petkov
Quoted 4 lines · Borislav Petkov
[    0.220022] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.220078] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.220132] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

Yes, that are BIOS bugs too that prevent interrupt remapping to function reliably. But the good thing is that these bugs can be detected easily to enable a workaround :-)

Joerg

#16 lore raw
Quoted 9 lines · Jörg Rödel
Quoted 4 lines · Borislav Petkov
[    0.220022] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.220078] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.220132] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

Yes, that are BIOS bugs too that prevent interrupt remapping to function reliably. But the good thing is that these bugs can be detected easily to enable a workaround :-)

Well, I'm all ready to test stuff since this is the latest ASUS BIOS and I'm not even going to ask them to fix it there based on previous experience with them :-).

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#17 lore raw
Quoted 9 lines · Borislav Petkov
Quoted 5 lines
Quoted 1 line
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

Yes, that are BIOS bugs too that prevent interrupt remapping to function reliably. But the good thing is that these bugs can be detected easily to enable a workaround :-)

Well, I'm all ready to test stuff since this is the latest ASUS BIOS and I'm not even going to ask them to fix it there based on previous experience with them :-).

I too am ready to test but the Gigabyte case is still open and unanswered.

Udo

#18 lore raw
In reply to Borislav Petkov
Quoted 10 lines · Borislav Petkov <bp <at> alien8.de> writes:
Quoted 1 line · Jörg Rödel

BorisO is no longer with AMD afaik.

Why am I not surprised...

Quoted 3 lines

I wrote an email to Sherry and Suravee and asked them to either send me hardware to write the fix on my own or to send a fix for the issue. Let's see what happens...

Quoted 5 lines

Btw, while we're at it, here's some more h0rkage from my PD box:

[    0.220022] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.220078] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.220132] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS

table

Quoted 2 lines
[    0.220187] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

hello,

I've never posted to this type of message board before so I hope I'm not out of order. I found my way here while trying to solve a shutdown problem in a new linux install (Arch). I've noticed for some time that I have the same lines as above in my dmesg as well as:

ACPI BIOS Bug: Warning: Optional FADT field Pm2CintrolBlock has no address or length: 0x0000000000000000/0x1 (20121018/tbfadt-589)

I have no idea whether this is related to my shutdown problem or not and what I was hoping is if someone would advise me how to find help for my problem. I have tried all the normal routes for my distro (forum and irc) repeatedly, and googled my weeping eyes out - but to no avail.

Any advice would be greatly appreciated.

Thanks
Jake

#19 lore raw
Quoted 1 line · Jake

ACPI BIOS Bug: Warning: Optional FADT field Pm2CintrolBlock has no address or length: 0x0000000000000000/0x1 (20121018/tbfadt-589)

I have the same one:

[    0.000000] ACPI BIOS Bug: Warning: Optional FADT field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20130117/tbfadt-599)

We probably have the same ASUS crap for a board.

Quoted 1 line

I have no idea whether this is related to my shutdown problem or not and

I don't think so as I can suspend/resume/shutdown my box just fine. :)

Quoted 4 lines

what I was hoping is if someone would advise me how to find help for my problem. I have tried all the normal routes for my distro (forum and irc) repeatedly, and googled my weeping eyes out - but to no avail.

Have you tried the upstream kernel yet? I hear 3.9-rc8 will be out tomorrow :-)

HTH.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#20 lore raw
In reply to Jörg Rödel
Quoted 9 lines · Jörg Rödel
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

Are you talking about erratum 746?

-boris

#21 lore raw
Quoted 5 lines · Boris Ostrovsky
Quoted 2 lines
Quoted 2 lines

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

Are you talking about erratum 746?

Link please?
If we have a link I could add that to the Gigabyte case.

Udo

maintainer
#22 lore raw
In reply to Boris Ostrovsky

Hi Boris,

Quoted 1 line · Boris Ostrovsky

Are you talking about erratum 746?

The problems seen here are not about PPR failures, so it is not particularily this erratum, but the workaround looks similar.

Joerg

#23 lore raw
In reply to Borislav Petkov
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

Hi, yes we will try and reproduce the NB clock gating issue on our end and submit a patch ASAP.

And Boris P., I think your IOAPIC not in IVRS issue we've also seen something similar recently (on Xen), so we'll atempt to tackle that one too afterwards.

-Jacob

Quoted 13 lines

Guys, the error description is at
http://marc.info/?l=linux-kernel&m=135867802432660

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#24 lore raw

Udo,

I am trying to debug the issue but need to check one thing on your system. Would you please try the following and check the output value on your system?

# setpci -s 00:00.02 F0.w=90
# setpci -s 00:00.02 F4.w

Thank you,

Suravee

Quoted 32 lines · Jacob Shin
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Jörg Rödel

Yes, the BIOS vendor can fix this issue. They need to disable NB clock gating for the IOMMU.

Right, Udo, you can try Gigabyte first.

Btw, can't we add a quirk to disable NB clock gating? Maybe Boris and Jacob could help. CCed.

Hi, yes we will try and reproduce the NB clock gating issue on our end and submit a patch ASAP.

And Boris P., I think your IOAPIC not in IVRS issue we've also seen something similar recently (on Xen), so we'll atempt to tackle that one too afterwards.

-Jacob

Quoted 13 lines

Guys, the error description is at
http://marc.info/?l=linux-kernel&m=135867802432660

Thanks.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
#25 lore raw
Quoted 5 lines · Suravee Suthikulpanit

Would you please try the following and check the output value on your system?

# setpci -s 00:00.02 F0.w=90
# setpci -s 00:00.02 F4.w
# setpci -s 00:00.02 F0.w=90
# setpci -s 00:00.02 F4.w
0050
#

Udo

#27 lore raw
Quoted 2 lines · Udo van den Heuvel

Gigabyte demonstrate that using ESX 5i IOMMU works fine. (with pictures attached).

There are no attachments to your message.

I am not sure that 5i supports IOMMU (but I may well be wrong).

Quoted 2 lines

What can we bring against that?

How reproducible is the problem that you are seeing?

-boris

maintainer
#28 lore raw
Quoted 9 lines · Boris Ostrovsky
Quoted 2 lines · Udo van den Heuvel

Gigabyte demonstrate that using ESX 5i IOMMU works fine. (with pictures attached).

There are no attachments to your message.

I am not sure that 5i supports IOMMU (but I may well be wrong).

Virtualization use-cases don't change the page-tables for the IOMMU very often. So there is less need to flush the IO-TLB and IOMMU command processing is utilized only from time to time.

In Linux however the page-tables change all the time and there is a much higher load on the IOMMU command buffer which makes it much more likely to trigger the hardware problem.

Joerg

#29 lore raw
In reply to Boris Ostrovsky
Quoted 4 lines · Boris Ostrovsky
Quoted 2 lines

Gigabyte demonstrate that using ESX 5i IOMMU works fine. (with pictures attached).

There are no attachments to your message.

Correct, gigabyte did send them via their support web-interface. Do yo uneed to see them? They just show IOMMU enabled or similar.

Quoted 3 lines
Quoted 1 line

What can we bring against that?

How reproducible is the problem that you are seeing?

Seen once over here. Correlated with raid-check.

Udo

#30 lore raw
Quoted 8 lines · Udo van den Heuvel
Quoted 4 lines · Boris Ostrovsky
Quoted 2 lines

Gigabyte demonstrate that using ESX 5i IOMMU works fine. (with pictures attached).

There are no attachments to your message.

Correct, gigabyte did send them via their support web-interface. Do yo uneed to see them? They just show IOMMU enabled or similar.

No, I thought you ran this yourself.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

What can we bring against that?

How reproducible is the problem that you are seeing?

Seen once over here. Correlated with raid-check.

Then the answer from Gigabyte doesn't prove anything. You can also boot Linux without seeing this problem in most cases.

Your BIOS does not have the required erratum workaround. We will provide a patch to close that hole but since the problem is not easily reproducible (and the erratum is also not easy to trigger) it may be difficult to say whether it really helped with your problem.

-boris

#31 lore raw
Quoted 4 lines · Boris Ostrovsky
Quoted 1 line

Seen once over here. Correlated with raid-check.

Then the answer from Gigabyte doesn't prove anything. You can also boot Linux without seeing this problem in most cases.

That was my situation until the first time it hit.

Quoted 4 lines

Your BIOS does not have the required erratum workaround. We will provide a patch to close that hole but since the problem is not easily reproducible (and the erratum is also not easy to trigger) it may be difficult to say whether it really helped with your problem.

Can we think of certain loads/actions/etc that could help trigger the issue? Then if reproducing is easier we can better say if stuff is actually fixed after the workaround.

Udo

#32 lore raw
Quoted 5 lines · Udo van den Heuvel
Quoted 4 lines · Boris Ostrovsky

Your BIOS does not have the required erratum workaround. We will provide a patch to close that hole but since the problem is not easily reproducible (and the erratum is also not easy to trigger) it may be difficult to say whether it really helped with your problem.

Udo,

I sent out a patch (http://marc.info/?l=linux-kernel&m=135889686523524&w=2) which should implement the workaround for AMD processor family15h model 10-1Fh erratum 746 in the IOMMU driver. In your case, the output from "setpci -s 00:00.02 F4.w" is "0050" which tells me that BIOS doesn't implement the work around. After patching, you should see the following message in "dmesg".

"AMD-Vi: Applying erratum 746 for IOMMU at 0000:00:00.2"

Quoted 5 lines

Can we think of certain loads/actions/etc that could help trigger the issue? Then if reproducing is easier we can better say if stuff is actually fixed after the workaround.

Udo

Looking at the original kernel message, it seems that the the kernel timed out while waiting for the IOMMU
to finish executing the "COMPLETION_WAIT" command. In this particular case, it is issued as part of
"__domain_flush_pages()" while trying to send the "INVALIDATE_IOMMU_PAGE" command to the IOMMU but the command buffer is getting full and the kernel needed to wait for the command buffer to free up. However, the kernel message did not exactly telling us what caused IOMMU to locked up in the first place.

According to my observation, high disk traffic workload should trigger large amount of "INVALIDATE_IOMMU_PAGE". However, this doesn't automatically issuing "COMPLETION_WAIT" command. The following patch slightly modify the code to always issue "COMPLETION_WAIT" after every command. This should help increasing the chance of reproducing the issue.

drivers/iommu/amd_iommu.c +35 -0
index c1c74e0..d05b1f9 100644
@@ -1016,6 +1016,7 @@ static int iommu_queue_command_sync(struct amd_iommu *iommu,
                                    struct iommu_cmd *cmd,
                                    bool sync)
 {
+#if 0
        u32 left, tail, head, next_tail;
        unsigned long flags;
 
@@ -1052,6 +1053,40 @@ again:
 
        spin_unlock_irqrestore(&iommu->lock, flags);
 
+#else
+       u32 tail;
+       unsigned long flags;
+
+       WARN_ON(iommu->cmd_buf_size & CMD_BUFFER_UNINITIALIZED);
+       printk (KERN_DEBUG "AMD-Vi: iommu_queue_command_sync: iommu_queue_command_sync"
+               " data[0]:%#x data[1]:%#x data[2]:%#x data[3]:%#x\n",
+               cmd->data[0], cmd->data[1], cmd->data[2], cmd->data[3] );
+
+       spin_lock_irqsave(&iommu->lock, flags);
+
+       tail = readl(iommu->mmio_base + MMIO_CMD_TAIL_OFFSET);
+       copy_cmd_to_buffer(iommu, cmd, tail);
+
+       spin_unlock_irqrestore(&iommu->lock, flags);
+
+       // Sending completion_wait command
+       {
+               struct iommu_cmd sync_cmd;
+               volatile u64 sem = 0;
+               int ret;
+
+               spin_lock_irqsave(&iommu->lock, flags);
+
+               tail = readl(iommu->mmio_base + MMIO_CMD_TAIL_OFFSET);
+               build_completion_wait(&sync_cmd, (u64)&sem);
+               copy_cmd_to_buffer(iommu, &sync_cmd, tail);
+
+               spin_unlock_irqrestore(&iommu->lock, flags);
+
+               if ((ret = wait_on_sem(&sem)) != 0)
+                       return ret;
+       }
+#endif
        return 0;
 }
 
#33 lore raw
Quoted 11 lines · Suravee Suthikulanit

I sent out a patch
(http://marc.info/?l=linux-kernel&m=135889686523524&w=2) which should implement
the workaround for AMD processor family15h model 10-1Fh erratum 746 in the IOMMU driver.
In your case, the output from "setpci -s 00:00.02 F4.w" is "0050" which tells me that BIOS doesn't
implement the work around. After patching, you should see the following message in "dmesg".

"AMD-Vi: Applying erratum 746 for IOMMU at 0000:00:00.2"

Thanks!
I'll check for that after these messages.

Quoted 4 lines

The following patch slightly modify
the code to always issue "COMPLETION_WAIT" after every command. This should help increasing the chance of reproducing
the issue.

Should I test with these two patches together?
Or should I apply the first one first and then see what the second can help?

Kind regards,
Udo

#34 lore raw
Quoted 15 lines · Udo van den Heuvel
Quoted 11 lines · Suravee Suthikulanit

I sent out a patch
(http://marc.info/?l=linux-kernel&m=135889686523524&w=2) which should implement
the workaround for AMD processor family15h model 10-1Fh erratum 746 in the IOMMU driver.
In your case, the output from "setpci -s 00:00.02 F4.w" is "0050" which tells me that BIOS doesn't
implement the work around. After patching, you should see the following message in "dmesg".

"AMD-Vi: Applying erratum 746 for IOMMU at 0000:00:00.2"

Thanks!
I'll check for that after these messages.

Quoted 6 lines
Quoted 4 lines

The following patch slightly modify
the code to always issue "COMPLETION_WAIT" after every command. This should help increasing the chance of reproducing
the issue.

Should I test with these two patches together?
Or should I apply the first one first and then see what the second can help?

Please try the first one first. If the issue doesn't reproduce, you can use the second patch to try to trigger it.

Thank you,

Suravee

Quoted 5 lines

Kind regards,
Udo

#35 lore raw
In reply to Suravee Suthikulanit
Quoted 3 lines · Suravee Suthikulanit

message in "dmesg".

"AMD-Vi: Applying erratum 746 for IOMMU at 0000:00:00.2"

[    1.091733] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

I assume that is correct.

Kind regards,
Udo

#36 lore raw
Quoted 4 lines · Udo van den Heuvel
Quoted 3 lines · Suravee Suthikulanit

message in "dmesg".

"AMD-Vi: Applying erratum 746 for IOMMU at 0000:00:00.2"

This is expected.

Regards,

Suravee

Quoted 8 lines
[    1.091733] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

I assume that is correct.

Kind regards,
Udo

#37 lore raw
In reply to Boris Ostrovsky
Quoted 4 lines · Boris Ostrovsky
Quoted 1 line · Udo van den Heuvel

Gigabyte demonstrate that using ESX 5i IOMMU works fine.

I forwarded the malinglist links with the patch(es) to Gigabyte support and they forwarded the info to the BIOS_team.

(to be continued)

Udo