Skip to contentSkip to conversation

2.6.25 DMA: Out of SW-IOMMU space - Asus M2N32 AMD 8GB memory

DISCUSSION Jari Aalto 4 participants 8 messages Aug 23, 2008 – Aug 25, 2008

lore
Share & tools
lore

Jari Aalto

author
#0 lore raw

Message from /etc/syslog:

        [1] Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:00:0d.0

My AMD freezes with Kernel 2.6.25 has experienced regular freezing so that only power button can take the system down. This is alarming, because the system can stay up only a few days.

I've spent countless of hours reading related "Out of SW-IOMMU space" (Google) documents. For some people they have worked, for some they haven't and there has not been any clear explanation what options whould/should be used in what chipsets/MBs and why.

I've gone through various combinations of kernel boot options, but nothing seems to completely solve the problem:

iommu=soft swiotlb=65536

Freezing continued, but the disk corruption did not happen any more. Increasing the swiotlb value has not had helped.

iommu=soft,memaper=3 swiotlb=65536

Adding memaper did not help. "Out of SW-IOMMU space" messages [see 1] creept in and I'm preparing to see another freeze eventally.

iommu=noaperture

Same as above. Not progress.

iommu=noagp,noaperture swiotlb=512M

Current options that I use. They were giving hope for 2 days, but then a single "Out of SW-IOMMU space" message appeared. I'm afraid the freeze is about to come.

Should I try following options next? or just "iommu=off"?

        iommu=noagp,noaperture,off swiotlb=512M
                               ===

I don't understand enough what are the effects related to the MCP55 SATA Controller which seems to be the target [See 1; based on device id "00:0d.0"] of these IOMMU messages. Only the plain SATA connectors, not the onboard RAID SATA connectors, are in use for the harddisk.

To best of my knowledge going through this motherboard:

- Asus award bios does not have setting related to IOMMU. I'm using the latest bios 2001 from www.asus.com
- has no aperture setting in bios.
- has no AGP, only PCI and CPIe slots.

My arsenal of knowledge is exhausting, so please, if you have any insight what could be examined further or what could be done to solve the IOMMU problem, let me know.

Jari

Some of the links and threads I've read
---------------------------------------

"Appendix L. Known Issues" > The X86-64 platform (AMD64/EM64T) and 2.6 kernels ftp://download.nvidia.com/XFree86/Linux-x86/1.0-8174/README/32bit_html/appendix-l.html

"What is AGP Aperture size?"
http://www.techpowerup.com/articles/overclocking/vidcard/43

"PCI-DMA: high address but no IOMMU"
http://article.gmane.org/gmane.linux.kernel/342411

"Out of IOMMU space"
http://www.x86-64.org/pipermail/discuss/2005-September/006490.html

"Your BIOS doesn't leave a aperture memory hole"
http://www.linuxquestions.org/questions/linux-hardware-18/your-bios-doesnt-leave-a-aperture-memory-hole-624088/

Hardware details
----------------
OS

        $ cat /etc/debian_version
        lenny/sid               (pinning: that's 90% testing + 10% unstable packages)

Kernel

        $ uname -a
        2.6.25-2-amd64 #1 SMP Mon Jul 14 11:05:23 UTC 2008 x86_64 GNU/Linux

CPU

        $ cat /proc/cpuinfo
        model name      : AMD Athlon(tm) X2 Dual Core Processor BE-2400
        stepping        : 2
        cpu MHz         : 2310.518
        cache size      : 512 KB
        ...
        $ cat /proc/meminfo
        MemTotal:      8266632 kB
        MemFree:        110212 kB
        Buffers:        237132 kB
        Cached:        3803660 kB
        SwapCached:          0 kB
        ...

HD

        $ hdparm -I /dev/sda

ATA device, with non-removable media
Model Number: ST31000340AS
Serial Number: 5QJ01MS4
Firmware Revision: SD01

http://www.seagate.com/ww/v/index.jsp?vgnextoid=0732f141e7f43110VgnVCM100000f5ee0a0aRCRD

MB

        Asus M2N32-SLI Deluxe/Wireless Edition
        - nvidia nForce 590 SLI chipset MCP
        - 2 x PCIe (SLI x16), 1 x PCI (x4), 1 x PCI (x1), 2 x PCI 2.2
        - Socket AM2

        http://www.asus.com/products.aspx?l1=3&l2=101&l3=300&model=1163&modelmenu=1

        $ lspci -nn
        00:0d.0 IDE interface [0101]: nVidia Corporation MCP55 SATA Controller [10de:037f] (rev a2)
        01:00.0 VGA compatible controller [0300]: nVidia Corporation G70 [GeForce 7600 GS] [10de:0392] (rev a1)
        02:0b.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link) [104c:8023]
        03:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
        ...

lspci -vv
----------------------------

00:16.0 PCI bridge: nVidia Corporation MCP55 PCI Express bridge (rev a2)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 00009000-00009fff
Memory behind bridge: fde00000-fdefffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: nVidia Corporation Device 0000
Capabilities: [48] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable+
Address: 00000000fee0300c Data: 4151
Capabilities: [60] HyperTransport: MSI Mapping Enable+ Fixed-
Mapping Address Base: 00000000fee00000
Capabilities: [80] Express (v1) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <4us
ClockPM- Suprise- LLActRep+ BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surpise-
Slot # 0, PowerLimit 0.000000; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd On, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
Capabilities: [100] Virtual Channel <?>
Kernel driver in use: pcieport-driver
Kernel modules: shpchp

[1] Full message from syslog
-----------------------------

Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:00:0d.0
Aug 21 11:01:19 jondo kernel: [174628.279020] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 21 11:01:19 jondo kernel: [174628.279020] ata3.00: cmd 35/00:00:9f:b9:fd/00:04:71:00:00/e0 tag 0 dma 524288 out
Aug 21 11:01:19 jondo kernel: [174628.279020]          res 50/00:00:96:b9:fd/00:00:71:00:00/e0 Emask 0x40 (internal error)
Aug 21 11:01:19 jondo kernel: [174628.279020] ata3.00: status: { DRDY }
Aug 21 11:01:19 jondo kernel: [174628.322932] ata3.00: configured for UDMA/133
Aug 21 11:01:19 jondo kernel: [174628.322932] ata3: EH complete
Aug 21 11:01:19 jondo kernel: [174628.330761] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors (1000205 MB)
Aug 21 11:01:19 jondo kernel: [174628.340876] sd 2:0:0:0: [sda] Write Protect is off
Aug 21 11:01:19 jondo kernel: [174628.340876] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
Aug 21 11:01:19 jondo kernel: [174628.351250] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

dmesg
-------------------------------

[    0.914265] Linux agpgart interface v0.103
...
[    3.687719] ata1: SATA link down (SStatus 0 SControl 0)
[    5.770299] ata2: SATA link down (SStatus 0 SControl 0)
[    5.582800] ACPI: PCI Interrupt Link [APC3] enabled at IRQ 18
[    5.582811] ACPI: PCI Interrupt 0000:02:08.1[A] -> Link [APC3] -> GSI 18 (level, low) -> IRQ 18
[    5.584163] NFORCE-MCP55: 0000:00:0c.0 (rev a1) UDMA133 controller
[    5.584167] NFORCE-MCP55: IDE controller (0x10de:0x036e rev 0xa1) at  PCI slot 0000:00:0c.0
[    5.584187] NFORCE-MCP55: not 100% native mode: will probe irqs later
[    5.584194] NFORCE-MCP55: IDE port disabled
[    5.584198]     ide0: BM-DMA at 0xf400-0xf407, BIOS settings: hda:DMA, hdb:DMA
[    5.584208] Probing IDE interface ide0...
[    5.661667] firewire_ohci: Added fw-ohci device 0000:02:08.1, OHCI version 1.10
[    5.661706] ACPI: PCI Interrupt Link [APC1] enabled at IRQ 16
[    5.661706] ACPI: PCI Interrupt 0000:02:0b.0[A] -> Link [APC1] -> GSI 16 (level, low) -> IRQ 16
[    5.732701] firewire_ohci: Added fw-ohci device 0000:02:0b.0, OHCI version 1.10
[    6.345280] ACPI: PCI Interrupt Link [APCL] enabled at IRQ 20
[    6.345280] ACPI: PCI Interrupt 0000:00:0a.1[B] -> Link [APCL] -> GSI 20 (level, low) -> IRQ 20
[    6.345280] PCI: Setting latency timer of device 0000:00:0a.1 to 64
[    6.345280] ehci_hcd 0000:00:0a.1: EHCI Host Controller
[    6.345280] ehci_hcd 0000:00:0a.1: new USB bus registered, assigned bus number 2
[    6.345280] ehci_hcd 0000:00:0a.1: debug port 1
[    6.345280] PCI: cache line size of 64 is not supported by device 0000:00:0a.1
[    6.345280] ehci_hcd 0000:00:0a.1: irq 20, io mem 0xfe02e000
#1 lore raw
Quoted 3 lines · Jari Aalto
        [1] Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:00:0d.0
        model name      : AMD Athlon(tm) X2 Dual Core Processor
        BE-2400

grep GART_IOMMU .config

Signature · 1 line
Krzysztof Halasa
#2 lore raw
Quoted 7 lines · On Saturday 23 August 2008 13:31:03 Krzysztof Halasa wrote:
Quoted 4 lines · Jari Aalto
        [1] Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of
SW-IOMMU space for 65536 bytes at device 0000:00:0d.0 model name      :
AMD Athlon(tm) X2 Dual Core Processor
        BE-2400

grep GART_IOMMU .config

Agreed, you shouldn't be using the SW-IOMMU on this processor.

That said, do you use the r8169 driver with jumbo frames enabled? Francois Romieu just fixed a leak in it that affected Intel platforms (because some have no hardware IOMMU).

Signature · 2 lines
Cheers,
Alistair.
author
#3 lore raw
Quoted 11 lines · Alistair John Strachan
Quoted 7 lines · On Saturday 23 August 2008 13:31:03 Krzysztof Halasa wrote:
Quoted 4 lines · Jari Aalto
        [1] Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of
SW-IOMMU space for 65536 bytes at device 0000:00:0d.0 model name      :
AMD Athlon(tm) X2 Dual Core Processor
        BE-2400

grep GART_IOMMU .config

Agreed, you shouldn't be using the SW-IOMMU on this processor.

Are these the correct boot options, without swiotlb?

iommu=noagp,noaperture,off

Or just:

iommu=off

Quoted 3 lines

That said, do you use the r8169 driver with jumbo frames enabled? Francois Romieu just fixed a leak in it that affected Intel platforms (because some have no hardware IOMMU).

The onboard WiFi is in use:

        $ lsmod | grep rtl
	rtl8187   39424  0
        $ dmesg
	[    2.758861] usb 1-9: new full speed USB device using ohci_hcd and address 4
	[    2.978258] usb 1-9: configuration #1 chosen from 1 choice
	[    2.985369] usb 1-9: New USB device found, idVendor=0bda, idProduct=8187
	[    2.985372] usb 1-9: New USB device strings: Mfr=1, Product=2, SerialNumber=3
	[    2.985374] usb 1-9: Product: RTL8187_Wireless
	[    2.985376] usb 1-9: Manufacturer: Manufacturer_Realtek_RTL8187_
	[    2.985378] usb 1-9: SerialNumber: 0015AF0B59A6

Jari

#4 lore raw
Quoted 12 lines · Jari Aalto
Quoted 1 line

From Debian stock kernel:

$ grep -Ei 'iommu|agp' /boot/config-2.6.25-2-amd64
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=m
CONFIG_AGP_SIS=m
CONFIG_AGP_VIA=m

Should be fine. SWIOTLB is there for a backup, most (?) Intel machines don't have IOMMU (even the newest desktop boards).

Quoted 7 lines

Are these the correct boot options, without swiotlb?

iommu=noagp,noaperture,off

Or just:

iommu=off

Do you have problems without "iommu=XXX"?

Signature · 1 line
Krzysztof Halasa
author
#5 lore raw
Quoted 20 lines · Krzysztof Halasa
Quoted 12 lines · Jari Aalto
Quoted 1 line

From Debian stock kernel:

$ grep -Ei 'iommu|agp' /boot/config-2.6.25-2-amd64
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=m
CONFIG_AGP_SIS=m
CONFIG_AGP_VIA=m

Should be fine. SWIOTLB is there for a backup, most (?) Intel machines don't have IOMMU (even the newest desktop boards).

Quoted 2 lines

Are these the correct boot options, without swiotlb?

Quoted 7 lines
Quoted 5 lines

iommu=noagp,noaperture,off

Or just:

iommu=off

Do you have problems without "iommu=XXX"?

Initially when the PC was installed (and had no IOMMU) options, that cause the whole harddisk to corrupt and I had to reinstall everything. It might have been a little older kernel (2.6.23?), but I don't recall it exactly.

According to messages in syslog I tried to track down similar incidents and found the iommu articles. But the information was like trial and error.

Would you suggest that "immu=off" would be best option?

Thanks,
Jari

#6 lore raw
In reply to Alistair John Strachan
Quoted 9 lines · Alistair John Strachan
Quoted 5 lines · On Saturday 23 August 2008 13:31:03 Krzysztof Halasa wrote:
Quoted 4 lines · Jari Aalto
        [1] Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of
SW-IOMMU space for 65536 bytes at device 0000:00:0d.0 model name      :
AMD Athlon(tm) X2 Dual Core Processor
        BE-2400

Agreed, you shouldn't be using the SW-IOMMU on this processor.

So the correct boot paramer is?

imoou=off,noagp,noaperture

no swiotlb setting at all?

Quoted 3 lines

That said, do you use the r8169 driver with jumbo frames enabled? Francois Romieu just fixed a leak in it that affected Intel platforms (because some have no hardware IOMMU).

I'm using the onboard WiFi:

$ lsmod | grep rtl
rtl8187                39424  0
$ dmesg ...
[    2.758861] usb 1-9: new full speed USB device using ohci_hcd and address 4
[    2.978258] usb 1-9: configuration #1 chosen from 1 choice
[    2.985369] usb 1-9: New USB device found, idVendor=0bda, idProduct=8187
[    2.985372] usb 1-9: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.985374] usb 1-9: Product: RTL8187_Wireless
[    2.985376] usb 1-9: Manufacturer: Manufacturer_Realtek_RTL8187_
[    2.985378] usb 1-9: SerialNumber: 0015AF0B59A6

I'm not sure how it maps to pci -nn:

00:04.0 PCI bridge [0604]: nVidia Corporation C51 PCI Express Bridge [10de:02fb] (rev a1) 00:09.0 ISA bridge [0601]: nVidia Corporation MCP55 LPC Bridge [10de:0360] (rev a2) 00:09.1 SMBus [0c05]: nVidia Corporation MCP55 SMBus [10de:0368] (rev a2) 00:0a.0 USB Controller [0c03]: nVidia Corporation MCP55 USB Controller [10de:036c] (rev a1) 00:0a.1 USB Controller [0c03]: nVidia Corporation MCP55 USB Controller [10de:036d] (rev a2) 00:0e.0 PCI bridge [0604]: nVidia Corporation MCP55 PCI bridge [10de:0370] (rev a2) 00:10.0 Bridge [0680]: nVidia Corporation MCP55 Ethernet [10de:0373] (rev a2) 00:11.0 Bridge [0680]: nVidia Corporation MCP55 Ethernet [10de:0373] (rev a2) 00:16.0 PCI bridge [0604]: nVidia Corporation MCP55 PCI Express bridge [10de:0375] (rev a2)

Thank you,
Jari

author
#7 lore raw
In reply to Krzysztof Halasa
Quoted 9 lines · Krzysztof Halasa
Quoted 3 lines · Jari Aalto
        [1] Aug 21 11:01:19 jondo kernel: [174628.275859] DMA: Out of SW-IOMMU space for 65536 bytes at device 0000:00:0d.0
        model name      : AMD Athlon(tm) X2 Dual Core Processor
        BE-2400

grep GART_IOMMU .config

From Debian stock kernel:

$ grep -Ei 'iommu|agp' /boot/config-2.6.25-2-amd64
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=m
CONFIG_AGP_SIS=m
CONFIG_AGP_VIA=m

Jari