New: PCI-DMA: Out of IOMMU space

Bugme-new Andrew Morton 5 participants 16 messages Apr 4, 2009 – Apr 7, 2009 2 messages not in the archive libata

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author

2009-04-04 18:49 UTC #1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 18 lines · On Fri, 3 Apr 2009 09:30:19 GMT [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=13001

           Summary: PCI-DMA: Out of IOMMU space
           Product: Platform Specific/Hardware
           Version: 2.5
    Kernel Version: 2.6.29-gentoo
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: x86-64
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: Yes

Quoted 6 lines

Created an attachment (id=20789)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20789)
hwreport generated info
After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. 2.6.28-gentoo-r4 work ok, so it is regression.

It is indeed a regression.

Quoted 13 lines

Dmesg fragments:

Apr  3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for
4096 bytes
Apr  3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for
4096 bytes
Apr  3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1
Apr  3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0
last_issue_tag 0xfafbfcfd
Apr  3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0
Apr  3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0
Apr  3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve

Quoted 8 lines

Apr  3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action
0x6
Apr  3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag
0 ncq 4096 in
Apr  3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40
(internal error)
Apr  3 13:38:46 rngmhpamd ata1.00: status: { DRDY }
Apr  3 13:38:46 rngmhpamd ata1: hard resetting link

Are these scary-looking messages also present in 2.6.28?

If so, perhaps the ata code is leaking DMA memory on the error-handling path?

Quoted 50 lines

Apr  3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl
300)
Apr  3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100
Apr  3 13:38:47 rngmhpamd ata1: EH complete
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Apr  3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:

enabled, doesn't support DPO or FUA

And

Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 4608 bytes
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space

Quoted 8 lines

for 11776 bytes

Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07
driverbyte=0x00
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed

Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?

Grant Grundler

2009-04-07 16:14 UTC #3 lore raw

2009/4/5 Данила Жукоцкий <[email protected]>:

Quoted 39 lines

2009/4/4 Andrew Morton <[email protected]>:
Quoted 38 lines
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 18 lines · On Fri, 3 Apr 2009 09:30:19 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13001
Summary: PCI-DMA: Out of IOMMU space
Product: Platform Specific/Hardware
Version: 2.5
Kernel Version: 2.6.29-gentoo
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: x86-64
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: Yes
Quoted 6 lines
Created an attachment (id=20789)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20789)
hwreport generated info
After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. 2.6.28-gentoo-r4 work ok, so it is regression.
It is indeed a regression.
Quoted 5 lines
Dmesg fragments:
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes

The bug report has a "dmesg" attachment but I wasn't able to find the "Out of IOMMU space" message in the dmesg. Can that be corrected? I was looking for IDE/SATA errors *before* the IOMMU errors.

But I was surprised to find these bits:
...
Kernel command line: mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc iommu=allowdac,merge,memaper=3 3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet Initializing CPU#0
...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM
...

I'm not familiar with iommu= parameter nor the warning about the BIOS. Any comments on that?

thanks,
grant

Quoted 100 lines

Quoted 80 lines
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
Apr 3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1 Apr 3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0 last_issue_tag 0xfafbfcfd
Apr 3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0
Apr 3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0
Apr 3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6
Apr 3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 4096 in
Apr 3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40 (internal error)
Apr 3 13:38:46 rngmhpamd ata1.00: status: { DRDY }
Apr 3 13:38:46 rngmhpamd ata1: hard resetting link
Are these scary-looking messages also present in 2.6.28?
If so, perhaps the ata code is leaking DMA memory on the error-handling path?
Quoted 50 lines
Apr 3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl
300)
Apr 3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100
Apr 3 13:38:47 rngmhpamd ata1: EH complete
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
And
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 4608 bytes
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
Quoted 8 lines
for 11776 bytes
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07
driverbyte=0x00
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Quoted 4 lines
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Upgrade to 2.6.29-gentoo-r1 (2.6.29.1), problem is still here, can easyly trigger it. I boot with default apperture, 64mb, and while write to usb-hdd get this:
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:29:27 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd and address 6
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Данила Жукоцкий

2009-04-07 16:52 UTC #5 lore raw

Forgot reply to all, sorry

---------- Forwarded message ----------
From: Данила Жукоцкий <[email protected]>
Date: 2009/4/7
Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space
To: Grant Grundler <[email protected]>

Yes, in attachment clear dmesg, warnings in bugreport body

Quoted 2 lines

Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes

I got "PCI-DMA: Out of IOMMU space" while trying write data to usb or sata hdd before all other error messages. After that usb and sata drives lost. All other noise is attempts to communicate with died devices.

Quoted 2 lines

Kernel command line:
mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc

I'm boot from 3ware raid with evmc

Quoted 1 line

iommu=allowdac,merge,memaper=3

This is from Documentation/x86/x86_64/boot-options.txt iommu=allowdac Im try to avoid DMA bug. May be that not need. allowdac Allow double-address cycle (DAC) mode, i.e. DMA >4GB. DAC is used with 32-bit PCI to push a 64-bit address in two cycles. When off all DMA over >4GB is forced through an IOMMU or software bounce buffering. merge Do scatter-gather (SG) merging. memaper[=<order>] Allocate an own aperture over RAM with size 32MB<<order. (default: order=1, i.e. 64MB)
With default apperture, 64mb, DMA leak very fast, now i have memaper=5, 1 gb, becouse i must do my job and can't rollback to 2.6.28 due strange mysterious problem with forcedeth nics that i can't explain and solve. If solution for DMA leak will not be found, i'm try to fill bugreport about problem with nics.

Quoted 1 line

3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet

I prefer use msi on that hardware.

Quoted 4 lines

...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM

xw9400 BIOS do not have IOMMU option in the BIOS setup. Now this costs me 1gb of ram

Anyway, i can stable reproduce bug without all this whistlers

2009/4/7 Grant Grundler <[email protected]>:

Quoted 57 lines

2009/4/5 Данила Жукоцкий <[email protected]>:
Quoted 39 lines
2009/4/4 Andrew Morton <[email protected]>:
Quoted 38 lines
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 18 lines · On Fri, 3 Apr 2009 09:30:19 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13001
Summary: PCI-DMA: Out of IOMMU space
Product: Platform Specific/Hardware
Version: 2.5
Kernel Version: 2.6.29-gentoo
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: x86-64
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: Yes
Quoted 6 lines
Created an attachment (id=20789)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20789)
hwreport generated info
After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. 2.6.28-gentoo-r4 work ok, so it is regression.
It is indeed a regression.
Quoted 5 lines
Dmesg fragments:
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
The bug report has a "dmesg" attachment but I wasn't able to find the "Out of IOMMU space" message in the dmesg. Can that be corrected? I was looking for IDE/SATA errors *before* the IOMMU errors.
But I was surprised to find these bits:
...
Kernel command line: mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc iommu=allowdac,merge,memaper=3 3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet Initializing CPU#0
...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM
...

Quoted 107 lines

I'm not familiar with iommu= parameter nor the warning about the BIOS. Any comments on that?
thanks,
grant
Quoted 100 lines
Quoted 80 lines
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
Apr 3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1 Apr 3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0 last_issue_tag 0xfafbfcfd
Apr 3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0
Apr 3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0
Apr 3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6
Apr 3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 4096 in
Apr 3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40 (internal error)
Apr 3 13:38:46 rngmhpamd ata1.00: status: { DRDY }
Apr 3 13:38:46 rngmhpamd ata1: hard resetting link
Are these scary-looking messages also present in 2.6.28?
If so, perhaps the ata code is leaking DMA memory on the error-handling path?
Quoted 50 lines
Apr 3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl
300)
Apr 3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100
Apr 3 13:38:47 rngmhpamd ata1: EH complete
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
And
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 4608 bytes
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
Quoted 8 lines
for 11776 bytes
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07
driverbyte=0x00
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Quoted 4 lines
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Upgrade to 2.6.29-gentoo-r1 (2.6.29.1), problem is still here, can easyly trigger it. I boot with default apperture, 64mb, and while write to usb-hdd get this:
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:29:27 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd and address 6
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Signature · 1 line

С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Данила Жукоцкий

2009-04-07 20:23 UTC #6 lore raw

Thank You, Grant, for Your simple questions! Without "allowdac" after couple of hours testing i cannot reproduce bug! So it is my stupid mistake, i don't understand Why i add this absolutely unusual parameter to boot string. I'm apologize to All off You for this stupid mindfuck. You may close the bug, because it is bug in my damn head.

2009/4/7 Данила Жукоцкий <[email protected]>:

Quoted 226 lines

Forgot reply to all, sorry
---------- Forwarded message ----------
From: Данила Жукоцкий <[email protected]>
Date: 2009/4/7
Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space
To: Grant Grundler <[email protected]>
Yes, in attachment clear dmesg, warnings in bugreport body
Quoted 2 lines
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
I got "PCI-DMA: Out of IOMMU space" while trying write data to usb or sata hdd before all other error messages. After that usb and sata drives lost. All other noise is attempts to communicate with died devices.
Quoted 2 lines
Kernel command line:
mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc
I'm boot from 3ware raid with evmc
Quoted 1 line
iommu=allowdac,merge,memaper=3
This is from Documentation/x86/x86_64/boot-options.txt iommu=allowdac Im try to avoid DMA bug. May be that not need. allowdac Allow double-address cycle (DAC) mode, i.e. DMA >4GB. DAC is used with 32-bit PCI to push a 64-bit address in two cycles. When off all DMA over >4GB is forced through an IOMMU or software bounce buffering. merge Do scatter-gather (SG) merging. memaper[=<order>] Allocate an own aperture over RAM with size 32MB<<order. (default: order=1, i.e. 64MB)
With default apperture, 64mb, DMA leak very fast, now i have memaper=5, 1 gb, becouse i must do my job and can't rollback to 2.6.28 due strange mysterious problem with forcedeth nics that i can't explain and solve. If solution for DMA leak will not be found, i'm try to fill bugreport about problem with nics.
Quoted 1 line
3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet
I prefer use msi on that hardware.
Quoted 4 lines
...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM
xw9400 BIOS do not have IOMMU option in the BIOS setup. Now this costs me 1gb of ram
Anyway, i can stable reproduce bug without all this whistlers
2009/4/7 Grant Grundler <[email protected]>:
Quoted 57 lines
2009/4/5 Данила Жукоцкий <[email protected]>:
Quoted 39 lines
2009/4/4 Andrew Morton <[email protected]>:
Quoted 38 lines
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 18 lines · On Fri, 3 Apr 2009 09:30:19 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13001
Summary: PCI-DMA: Out of IOMMU space
Product: Platform Specific/Hardware
Version: 2.5
Kernel Version: 2.6.29-gentoo
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: x86-64
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: Yes
Quoted 6 lines
Created an attachment (id=20789)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20789)
hwreport generated info
After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. 2.6.28-gentoo-r4 work ok, so it is regression.
It is indeed a regression.
Quoted 5 lines
Dmesg fragments:
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
The bug report has a "dmesg" attachment but I wasn't able to find the "Out of IOMMU space" message in the dmesg. Can that be corrected? I was looking for IDE/SATA errors *before* the IOMMU errors.
But I was surprised to find these bits:
...
Kernel command line: mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc iommu=allowdac,merge,memaper=3 3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet Initializing CPU#0
...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM
...
Quoted 107 lines
I'm not familiar with iommu= parameter nor the warning about the BIOS. Any comments on that?
thanks,
grant
Quoted 100 lines
Quoted 80 lines
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
Apr 3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1 Apr 3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0 last_issue_tag 0xfafbfcfd
Apr 3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0
Apr 3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0
Apr 3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6
Apr 3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 4096 in
Apr 3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40 (internal error)
Apr 3 13:38:46 rngmhpamd ata1.00: status: { DRDY }
Apr 3 13:38:46 rngmhpamd ata1: hard resetting link
Are these scary-looking messages also present in 2.6.28?
If so, perhaps the ata code is leaking DMA memory on the error-handling path?
Quoted 50 lines
Apr 3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl
300)
Apr 3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100
Apr 3 13:38:47 rngmhpamd ata1: EH complete
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
And
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 4608 bytes
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
Quoted 8 lines
for 11776 bytes
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07
driverbyte=0x00
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Quoted 4 lines
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Upgrade to 2.6.29-gentoo-r1 (2.6.29.1), problem is still here, can easyly trigger it. I boot with default apperture, 64mb, and while write to usb-hdd get this:
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:29:27 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd and address 6
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Signature · 1 line

С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Grant Grundler

2009-04-07 20:59 UTC #7 lore raw

2009/4/7 Данила Жукоцкий <[email protected]>

Quoted 2 lines

Thank You, Grant, for Your simple questions!

Welcome!
I was just trying to point out some additional information to folks who might understand the code. I don't in this case but was looking for more clues.

Quoted 5 lines

Without "allowdac" after
couple of hours testing i cannot reproduce bug! So it is my stupid mistake, i don't understand Why i add this absolutely unusual parameter to boot string.

I thought I understood IOMMUs having written code for 4 different implementations.
I don't understand why "allowdac" parameter exists. dma_mask stuff should be handling this already.
Can someone explain *why* (Данила already posted the docs) this parameter exists?
(ie use case that dma_mask APIs don't work.)

Quoted 2 lines

I'm apologize to All off You for this stupid
mindfuck. You may close the bug, because it is bug in my damn head.

My first impression: the bug is either allocdma exists instead of using DMA API (See Documentation/DMA-API.txt) OR the documentation for allocdma is missing warnings about "this could break your system" and clearly specify when it should be used.

hth,
grant

Quoted 237 lines

2009/4/7 Данила Жукоцкий <[email protected]>:
Quoted 218 lines
Forgot reply to all, sorry
---------- Forwarded message ----------
From: Данила Жукоцкий <[email protected]>
Date: 2009/4/7
Subject: Re: [Bugme-new] [Bug 13001] New: PCI-DMA: Out of IOMMU space
To: Grant Grundler <[email protected]>
Yes, in attachment clear dmesg, warnings in bugreport body
Quoted 2 lines
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
I got "PCI-DMA: Out of IOMMU space" while trying write data to usb or sata hdd before all other error messages. After that usb and sata drives lost. All other noise is attempts to communicate with died devices.
Quoted 2 lines
Kernel command line:
mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc
I'm boot from 3ware raid with evmc
Quoted 1 line
iommu=allowdac,merge,memaper=3
This is from Documentation/x86/x86_64/boot-options.txt iommu=allowdac Im try to avoid DMA bug. May be that not need. allowdac Allow double-address cycle (DAC) mode, i.e. DMA >4GB. DAC is used with 32-bit PCI to push a 64-bit address in two cycles. When off all DMA over >4GB is forced through an IOMMU or software bounce buffering. merge Do scatter-gather (SG) merging. memaper[=<order>] Allocate an own aperture over RAM with size 32MB<<order. (default: order=1, i.e. 64MB)
With default apperture, 64mb, DMA leak very fast, now i have memaper=5, 1 gb, becouse i must do my job and can't rollback to 2.6.28 due strange mysterious problem with forcedeth nics that i can't explain and solve. If solution for DMA leak will not be found, i'm try to fill bugreport about problem with nics.
Quoted 1 line
3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet
I prefer use msi on that hardware.
Quoted 4 lines
...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM
xw9400 BIOS do not have IOMMU option in the BIOS setup. Now this costs me 1gb of ram
Anyway, i can stable reproduce bug without all this whistlers
2009/4/7 Grant Grundler <[email protected]>:
Quoted 57 lines
2009/4/5 Данила Жукоцкий <[email protected]>:
Quoted 39 lines
2009/4/4 Andrew Morton <[email protected]>:
Quoted 38 lines
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 18 lines · On Fri, 3 Apr 2009 09:30:19 GMT [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=13001
Summary: PCI-DMA: Out of IOMMU space
Product: Platform Specific/Hardware
Version: 2.5
Kernel Version: 2.6.29-gentoo
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: x86-64
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: Yes
Quoted 6 lines
Created an attachment (id=20789)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20789)
hwreport generated info
After some IO activity the "PCI-DMA: Out of IOMMU space" message appear. 2.6.28-gentoo-r4 work ok, so it is regression.
It is indeed a regression.
Quoted 5 lines
Dmesg fragments:
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
The bug report has a "dmesg" attachment but I wasn't able to find the "Out of IOMMU space" message in the dmesg. Can that be corrected? I was looking for IDE/SATA errors *before* the IOMMU errors.
But I was surprised to find these bits:
...
Kernel command line: mce=bootlog root=/dev/ram0 real_root=/dev/evms/root init=/linuxrc iommu=allowdac,merge,memaper=3 3w_9xxx.use_msi=1 snd-hda-intel.enable_msi=1 doevms quiet Initializing CPU#0
...
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 256 MB of RAM
...
Quoted 107 lines
I'm not familiar with iommu= parameter nor the warning about the BIOS. Any comments on that?
thanks,
grant
Quoted 100 lines
Quoted 80 lines
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 4096 bytes
Apr 3 13:38:46 rngmhpamd ata1: EH in SWNCQ mode,QC:qc_active 0x1 sactive 0x1 Apr 3 13:38:46 rngmhpamd ata1: SWNCQ:qc_active 0x0 defer_bits 0x0 last_issue_tag 0xfafbfcfd
Apr 3 13:38:46 rngmhpamd dhfis 0x0 dmafis 0x0 sdbfis 0x0
Apr 3 13:38:46 rngmhpamd ata1: ATA_REG 0x50 ERR_REG 0x0
Apr 3 13:38:46 rngmhpamd ata1: tag : dhfis dmafis sdbfis sacitve
Quoted 8 lines
Apr 3 13:38:46 rngmhpamd ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6
Apr 3 13:38:46 rngmhpamd ata1.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 4096 in
Apr 3 13:38:46 rngmhpamd res 50/00:00:00:00:00/00:45:00:00:00/a0 Emask 0x40 (internal error)
Apr 3 13:38:46 rngmhpamd ata1.00: status: { DRDY }
Apr 3 13:38:46 rngmhpamd ata1: hard resetting link
Are these scary-looking messages also present in 2.6.28?
If so, perhaps the ata code is leaking DMA memory on the error-handling path?
Quoted 50 lines
Apr 3 13:38:47 rngmhpamd ata1: SATA link up 3.0 Gbps (SStatus 123 SControl
300)
Apr 3 13:38:47 rngmhpamd ata1.00: configured for UDMA/100
Apr 3 13:38:47 rngmhpamd ata1: EH complete
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] 488397168 512-byte hardware
sectors: (250 GB/232 GiB)
Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write Protect is off Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Apr 3 13:38:47 rngmhpamd sd 1:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
And
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 4608 bytes
Mar 31 20:56:18 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:56:48 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:56:48 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:19 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:19 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:57:50 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:57:50 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:21 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 11776 bytes
Mar 31 20:58:21 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:58:52 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd
and address 8
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
Quoted 8 lines
for 11776 bytes
Mar 31 20:58:52 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space
for 69632 bytes
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Unhandled error code
Mar 31 20:59:01 rngmhpamd sd 8:0:0:0: [sdc] Result: hostbyte=0x07
driverbyte=0x00
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Quoted 4 lines
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Upgrade to 2.6.29-gentoo-r1 (2.6.29.1), problem is still here, can easyly trigger it. I boot with default apperture, 64mb, and while write to usb-hdd get this:
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:28:56 rngmhpamd ehci_hcd 0000:00:02.1: PCI-DMA: Out of IOMMU space for 65536 bytes
Apr 5 14:29:27 rngmhpamd usb 1-4: reset high speed USB device using ehci_hcd and address 6
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш" --
To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Grant Grundler

2009-04-07 21:14 UTC #8 lore raw

Forgot one other possibility...

2009/4/7 Grant Grundler <[email protected]>:
....

Quoted 4 lines

My first impression: the bug is either allocdma exists instead of using DMA API (See Documentation/DMA-API.txt) OR the documentation for allocdma is missing warnings about "this could break your system" and clearly specify when it should be used.

or it's a bug in allowdac implementation.

BTW, this whole thread probably needs to be re-posted to either linux-pci or linux kernel. I don't think linux-ide is the right place to resolve this.

grant

Alan Cox

2009-04-06 22:49 UTC #9 lore raw

In reply to Andrew Morton

Quoted 5 lines

Quoted 2 lines
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?

Turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG for a test run.

FUJITA Tomonori

2009-04-07 01:51 UTC #10 lore raw

Quoted 7 lines · On Mon, 6 Apr 2009 23:48:16 +0100 Alan Cox <[email protected]> wrote:

Quoted 5 lines
Quoted 2 lines
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG for a test run.

These options don't tell who leaks dma mappings so possibly it isn't very helpful.

There is no interesting changes in GART IOMMU between 2.6.28 and 2.6.29 so probably this is a driver bug. It's necessary to find out which driver leaks dma mappings.

Данила Жукоцкий

2009-04-07 05:05 UTC #11 lore raw

I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options? Bug triggered when i try mount sata drive on sata_nv and usb hdd on echi_hcd. In all other cases i have no problem.

Also i have mysterious problem similar to
http://lkml.org/lkml/2009/1/11/302 . It prevent me return to normally working before 2.6.28 kernel. Once boot 2.6.29 and found DMA bug i'm reboot to old working 2.6.28 and seen that my eth0 and eth1 forcedetch died with "no link during initialization". Restart interfaces, remove insert module don't help.
I'm not change configs, i'm not rebuild old kernel. I'm not touch it at all. Reboot to 2.6.29 and network work. Cold restart won't help, and i not won't reset BIOS. I looked to dmesg 2.6.28 and seen that irqs for forcedetch look different from other 2.6.28 kernel dmesgs. So 2.6.29 do strange things with my machine, and that things live cold restart. I don't know what happens, i can't understand and can't open case and hardreset bios all time when i want return to old working kernel. Can you explain me what wrong?

2009/4/7 FUJITA Tomonori <[email protected]>:

Quoted 18 lines

Quoted 7 lines · On Mon, 6 Apr 2009 23:48:16 +0100 Alan Cox <[email protected]> wrote:
Quoted 5 lines
Quoted 2 lines
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG for a test run.
These options don't tell who leaks dma mappings so possibly it isn't very helpful.
There is no interesting changes in GART IOMMU between 2.6.28 and 2.6.29 so probably this is a driver bug. It's necessary to find out which driver leaks dma mappings.

Данила Жукоцкий

2009-04-07 08:17 UTC #12 lore raw

Tryed 2.6.29-git14. Can't find CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG. Something strange happens with modules, "Invalid module format" for all, so unusable at all, usb keyboard not work.

2009/4/7 Данила Жукоцкий <[email protected]>:

Quoted 40 lines

I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options? Bug triggered when i try mount sata drive on sata_nv and usb hdd on echi_hcd. In all other cases i have no problem.
Also i have mysterious problem similar to
http://lkml.org/lkml/2009/1/11/302 . It prevent me return to normally working before 2.6.28 kernel. Once boot 2.6.29 and found DMA bug i'm reboot to old working 2.6.28 and seen that my eth0 and eth1 forcedetch died with "no link during initialization". Restart interfaces, remove insert module don't help.
I'm not change configs, i'm not rebuild old kernel. I'm not touch it at all. Reboot to 2.6.29 and network work. Cold restart won't help, and i not won't reset BIOS. I looked to dmesg 2.6.28 and seen that irqs for forcedetch look different from other 2.6.28 kernel dmesgs. So 2.6.29 do strange things with my machine, and that things live cold restart. I don't know what happens, i can't understand and can't open case and hardreset bios all time when i want return to old working kernel. Can you explain me what wrong?
2009/4/7 FUJITA Tomonori <[email protected]>:
Quoted 18 lines
Quoted 7 lines · On Mon, 6 Apr 2009 23:48:16 +0100 Alan Cox <[email protected]> wrote:
Quoted 5 lines
Quoted 2 lines
Mar 31 20:59:01 rngmhpamd end_request: I/O error, dev sdc, sector 1137
Mar 31 20:59:01 rngmhpamd __ratelimit: 246 callbacks suppressed
Do we have any debugging option for dumping the current PCI DMA allocations, find out where it has all gone?
Turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG for a test run.
These options don't tell who leaks dma mappings so possibly it isn't very helpful.
There is no interesting changes in GART IOMMU between 2.6.28 and 2.6.29 so probably this is a driver bug. It's necessary to find out which driver leaks dma mappings.

Signature · 1 line

С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

FUJITA Tomonori

2009-04-07 08:40 UTC #13 lore raw

In reply to Данила Жукоцкий

Quoted 3 lines

I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options?

2.6.26.1 has these options. Probably you don't enable DEBUG_KERNEL. But the feature is not much useful, as I said.

Quoted 2 lines

Bug triggered when i try mount sata drive on sata_nv and usb hdd on echi_hcd. In all other cases i have no problem.

One of them might cause you problem. As I wrote in the previous mail, we need to know which driver is bad.

1. build a kernel without usb storage or any nics. Then do lots of I/Os for minutes to see if sata_nv is fine.

2. if sata_ntv seems to be fine, build a kernel without any nics (I guess your box can't boot without sata_nv). Then do lots of I/Os for usb storage.

3. If the both is fine, you could have other bad driver (might be nic).

FUJITA Tomonori

2009-04-07 08:45 UTC #14 lore raw

Quoted 6 lines · On Tue, 7 Apr 2009 17:38:30 +0900 FUJITA Tomonori…

Quoted 3 lines
I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options?
2.6.26.1 has these options. Probably you don't enable DEBUG_KERNEL. But the feature is not much useful, as I said.

Wait, you use 2.6.29, right?

Данила Жукоцкий

2009-04-07 08:52 UTC #15 lore raw

My bad, just a typo, i use 2.6.29.1 of course. My boot device 3ware raid, so i try isolate bug now as you say.

2009/4/7 FUJITA Tomonori <[email protected]>:

Quoted 12 lines

Quoted 6 lines · On Tue, 7 Apr 2009 17:38:30 +0900 FUJITA Tomonori…
Quoted 3 lines
I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options?
2.6.26.1 has these options. Probably you don't enable DEBUG_KERNEL. But the feature is not much useful, as I said.
Wait, you use 2.6.29, right?

Signature · 1 line

С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Данила Жукоцкий

2009-04-07 11:08 UTC #16 lore raw

Ok, what's done.
I build kernel with CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG, forcedeth, all usb and sata stuff as modules, reboot with default apperture, 64mb, rmmod ehci_usb, forcedeth. From sata_nv hdd /dev/sdb1 i can read 80gb data without problems, dmesg flooded with "dma_map_sg overflow" messages. When i try write data to /dev/sdb1 computer hung completely after one or two seconds. No oopses, clear logs, just hung and i hardreset it.

2009/4/7 Данила Жукоцкий <[email protected]>:

Quoted 22 lines

My bad, just a typo, i use 2.6.29.1 of course. My boot device 3ware raid, so i try isolate bug now as you say.
2009/4/7 FUJITA Tomonori <[email protected]>:
Quoted 12 lines
Quoted 6 lines · On Tue, 7 Apr 2009 17:38:30 +0900 FUJITA Tomonori…
Quoted 3 lines
I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options?
2.6.26.1 has these options. Probably you don't enable DEBUG_KERNEL. But the feature is not much useful, as I said.
Wait, you use 2.6.29, right?
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Данила Жукоцкий

2009-04-07 14:21 UTC #17 lore raw

I repeat test in bare console, single-user mode with all modules removed from kernel, only libata, ahci, sata_nv and pata_amd inserted. After read and write ~30gb data dmesg show first "dma_map_sg overflow" message on write operation. May be this info help.

2009/4/7 Данила Жукоцкий <[email protected]>

Quoted 34 lines

Ok, what's done.
I build kernel with CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG, forcedeth, all usb and sata stuff as modules, reboot with default apperture, 64mb, rmmod ehci_usb, forcedeth. From sata_nv hdd /dev/sdb1 i can read 80gb data without problems, dmesg flooded with "dma_map_sg overflow" messages. When i try write data to /dev/sdb1 computer hung completely after one or two seconds. No oopses, clear logs, just hung and i hardreset it.
2009/4/7 Данила Жукоцкий <[email protected]>:
Quoted 22 lines
My bad, just a typo, i use 2.6.29.1 of course. My boot device 3ware raid, so i try isolate bug now as you say.
2009/4/7 FUJITA Tomonori <[email protected]>:
Quoted 12 lines
Quoted 6 lines · On Tue, 7 Apr 2009 17:38:30 +0900 FUJITA Tomonori…
Quoted 3 lines
I can't turn on CONFIG_IOMMU_LEAK and CONFIG_IOMMU_DEBUG because 2.6.26.1 not have such options in .config. Maybe git kernel have this options?
2.6.26.1 has these options. Probably you don't enable DEBUG_KERNEL. But the feature is not much useful, as I said.
Wait, you use 2.6.29, right?
--
С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"

Signature · 1 line

С уважением Данила Жукоцкий, системный администратор ЗАО "Роснефтегазмаш"