Skip to contentSkip to conversation

Disabling DMA with ICH10?

DISCUSSION Lennart Baruschka 5 participants 11 messages Sep 28, 2009 – Sep 30, 2009

lore
Share & tools
lore
#0 lore raw

Hello,

I'm new to this list, so please forgive me if this is the wrong place to ask. I'm trying to build a 2.6.30.5 kernel causing as little DMA traffic as possible in order to reduce latencies on a real-time system (using RTAI).

1. Is there a way to completely disable DMA usage system-wide? If not:
2. My system uses an ICH10 chipset, the harddrive is connected to the Intel SATA controller. Is there a way to disable DMA and switch back to PIO? I tried compiling with libata support and without SCSI support, but the kernel is unable to mount root (no matter if /dev/sda1 or /dev/hda1), then.
3. Another source seems to be the ATL1E driver for ethernet. Is it possible to either disable DMA for this?

If disabling DMA is impossible for some reason for one or both drivers, is it at least possible to reduce the burst sizes so the bus gets free for a short time during which I could access it from my real time kernel module?

Please point me in the right direction; any suggestion will be greatly appreciated!

Cheers,
Lennart

#1 lore raw
Quoted 7 lines · El Mon, 28 Sep 2009 23:31:21 +0200 Lennart Baruschka <[email protected]>…

Hello,

I'm new to this list, so please forgive me if this is the wrong place to ask. I'm trying to build a 2.6.30.5 kernel causing as little DMA traffic as possible in order to reduce latencies on a real-time system (using RTAI).

Quoted 7 lines

1. Is there a way to completely disable DMA usage system-wide? If not:
2. My system uses an ICH10 chipset, the harddrive is connected to the Intel SATA controller. Is there a way to disable DMA and switch back to PIO? I tried compiling with libata support and without SCSI support, but the kernel is unable to mount root (no matter if /dev/sda1 or /dev/hda1), then.

AFAIK libata needs scsi support. To disable dma in libata I get from Documentation/kernel-parameters.txt --> libata.dma = 0

Quoted 2 lines

3. Another source seems to be the ATL1E driver for ethernet. Is it possible to either disable DMA for this?

That I dunno

Quoted 11 lines

If disabling DMA is impossible for some reason for one or both drivers, is it at least possible to reduce the burst sizes so the bus gets free for a short time during which I could access it from my real time kernel module?

Please point me in the right direction; any suggestion will be greatly appreciated!

Cheers,
Lennart

Saludos,
Alejandro

#2 lore raw
In reply to Lennart Baruschka
Quoted 1 line

1. Is there a way to completely disable DMA usage system-wide?

No

Quoted 5 lines

2. My system uses an ICH10 chipset, the harddrive is connected to the Intel SATA controller. Is there a way to disable DMA and switch back to PIO? I tried compiling with libata support and without SCSI support, but the kernel is unable to mount root (no matter if /dev/sda1 or /dev/hda1), then.

If you disable DMA you will make your performance and latency worse not better as the PIO transfers will stall the bus and thus the processor.

Quoted 4 lines

If disabling DMA is impossible for some reason for one or both drivers, is it at least possible to reduce the burst sizes so the bus gets free for a short time during which I could access it from my real time kernel module?

You want the disk controller in AHCI mode to minimise CPU stalls and then any burst size configuration and the like is down to the PCI configuration. On some systems where memory or bus bandwidth is an issue then forcing the disk to run at a lower speed can help, at least for PATA. What effect it will have on SATA I am less sure.

If you really are that latency sensitive then the more normal approach would be to lock one core for real time use, load the critical code into that core CPU cache and run from cache. If you are utterly pushing the limit you might even do crazy stuff like use on thread on the core to execute RT stuff and the other to issue any I/O accesses that might stall.

And then on many systems things like BIOS SMI glue will still clobber you 8(

Alan

#3 lore raw

Hi,

Quoted 8 lines · Alan Cox
Quoted 5 lines

2. My system uses an ICH10 chipset, the harddrive is connected to the Intel SATA controller. Is there a way to disable DMA and switch back to PIO? I tried compiling with libata support and without SCSI support, but the kernel is unable to mount root (no matter if /dev/sda1 or /dev/hda1), then.

If you disable DMA you will make your performance and latency worse not better as the PIO transfers will stall the bus and thus the processor.

I thought that PIO transfers (which I understand to be write32()/read32()'s) unlike DMA transfers could be interrupted by an high-priority interrupt. Is that wrong?

Quoted 5 lines

If you really are that latency sensitive then the more normal approach would be to lock one core for real time use, load the critical code into that core CPU cache and run from cache. If you are utterly pushing the limit you might even do crazy stuff like use on thread on the core to execute RT stuff and the other to issue any I/O accesses that might stall.

Actually, that's what I do - except for locking the page, yet. I do need to access the PCI bus in real time, though. So I wonder what happens when the RT CPU is getting data from the PCI device, doing some calculations on it and then writing back some data to the device, __while at the same time__ another (non-RT) CPU starts a DMA transfer. I figured the DMA would block the PCI bus, having my interrupt wait for it to finish. That's why I'm trying to avoid DMA.

Please let me know at what point I am mistaken.

Cheers
Lennart

Signature · 2 lines
Jetzt kostenlos herunterladen: Internet Explorer 8 und Mozilla Firefox 3.5 -
sicherer, schneller und einfacher! http://portal.gmx.net/de/go/atbrowser
#4 lore raw
Quoted 9 lines · On Tue, 29 Sep 2009 09:54:16 +0200 "Lennart Baruschka" <[email protected]> wrote:

Hi,

Quoted 6 lines · Alan Cox
Quoted 5 lines

2. My system uses an ICH10 chipset, the harddrive is connected to the Intel SATA controller. Is there a way to disable DMA and switch back to PIO? I tried compiling with libata support and without SCSI support, but the kernel is unable to mount root (no matter if /dev/sda1 or /dev/hda1), then.

Quoted 7 lines
Quoted 3 lines

If you disable DMA you will make your performance and latency worse not better as the PIO transfers will stall the bus and thus the processor.

I thought that PIO transfers (which I understand to be write32()/read32()'s) unlike DMA transfers could be interrupted by an high-priority interrupt. Is that wrong?

a single PIO instruction will NOT be interrupted by an interrupt. And can easily take several microseconds (remember: 8Mhz bus emulation)

while DMA is going on, the CPU on the other hand is just happy executing instructions, fetching stuff from memory etc etc.

Quoted 7 lines

Actually, that's what I do - except for locking the page, yet. I do need to access the PCI bus in real time, though. So I wonder what happens when the RT CPU is getting data from the PCI device, doing some calculations on it and then writing back some data to the device, __while at the same time__ another (non-RT) CPU starts a DMA transfer. I figured the DMA would block the PCI bus, having my interrupt wait for it to finish. That's why I'm trying to avoid DMA.

the PCI bus is time sliced, with typical transfer sizes being like 256 byte bursts...

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#5 lore raw
Quoted 3 lines · Arjan van de Ven

a single PIO instruction will NOT be interrupted by an interrupt. And can easily take several microseconds (remember: 8Mhz bus emulation)

Not that long, there is no emulation there (except for port #80 (hex?) and similar). Normal 33 MHz access. But it's still slow, of course - every access takes at least 4(?) bus cycles (IIRC: address, initiator ready, device selected, target ready). No bursts (at least on typical x86 hw).

Of course ICH10 AHCI isn't connected with PCI 33/32 bus.

Anyway: PIO = bad idea.

Signature · 1 line
Krzysztof Halasa
#6 lore raw
Quoted 8 lines · On Tue, 29 Sep 2009 14:29:27 +0200 Krzysztof Halasa <[email protected]> wrote:
Quoted 3 lines · Arjan van de Ven

a single PIO instruction will NOT be interrupted by an interrupt. And can easily take several microseconds (remember: 8Mhz bus emulation)

Not that long, there is no emulation there (except for port #80 (hex?) and similar). Normal 33 MHz access.

ATA accesses go across the cable and while some of them are snooped, cached and other magic is done there are cases it turns into a transaction back and forth with the drive - that *is* slow. PIO 0 in fact is ISA speed

#7 lore raw
Quoted 7 lines · Alan Cox
Quoted 2 lines

Not that long, there is no emulation there (except for port #80 (hex?) and similar). Normal 33 MHz access.

ATA accesses go across the cable and while some of them are snooped, cached and other magic is done there are cases it turns into a transaction back and forth with the drive - that *is* slow. PIO 0 in fact is ISA speed

Sure, PATA inserts wait states, the 4 bus cycles per access (or something like that) is the minimum for PIO on PCI. But it's still normal 33 MHz access (with as many wait states as the PATA mode needs), not ISA 8 MHz emulation (fastest PIO would be faster than ISA).

Signature · 1 line
Krzysztof Halasa
#8 lore raw
Quoted 10 lines · On Tue, 29 Sep 2009 22:35:54 +0200 Krzysztof Halasa <[email protected]> wrote:
Quoted 7 lines · Alan Cox
Quoted 2 lines

Not that long, there is no emulation there (except for port #80 (hex?) and similar). Normal 33 MHz access.

ATA accesses go across the cable and while some of them are snooped, cached and other magic is done there are cases it turns into a transaction back and forth with the drive - that *is* slow. PIO 0 in fact is ISA speed

Quoted 4 lines

Sure, PATA inserts wait states, the 4 bus cycles per access (or something like that) is the minimum for PIO on PCI. But it's still normal 33 MHz access (with as many wait states as the PATA mode needs), not ISA 8 MHz emulation (fastest PIO would be faster than ISA)

You are missing the point. The sequence is

INB foo

	CPU -> PCI device		read this register
	PCI device -> Disk
			trundle whirr whirrr
			clunk clunk thud
	Disk -> PCI device
	PCI device -> CPU... "5"

INB completes

The INB is not interruptible mid instruction and stalls the CPU for the full period of the message passing back and forth across the bus. So PIO 0 on PCI stalls the bus for the equivalent of an ISA access, and PIO4 while a good deal faster is still a very long stall in hard real time terms.

#9 lore raw
Quoted 9 lines · Alan Cox

INB foo

	CPU -> PCI device		read this register
	PCI device -> Disk
			trundle whirr whirrr
			clunk clunk thud
	Disk -> PCI device
	PCI device -> CPU... "5"
Quoted 7 lines

INB completes

The INB is not interruptible mid instruction and stalls the CPU for the full period of the message passing back and forth across the bus. So PIO 0 on PCI stalls the bus for the equivalent of an ISA access, and PIO4 while a good deal faster is still a very long stall in hard real time terms.

Sure :-)

That's why I wrote it's a bad idea. I just wanted to correct the statement about ISA speed - it's not (necessarily) ISA speed, though it's slow and uses 100% CPU (core) time. (And PIO on PCI* in general is not much faster, though usually faster than ISA).

Signature · 1 line
Krzysztof Halasa
#10 lore raw
In reply to Lennart Baruschka
Quoted 3 lines

I thought that PIO transfers (which I understand to be write32()/read32()'s) unlike DMA transfers could be interrupted by an high-priority interrupt. Is that wrong?

Yes. An I/O cycle will stall that CPU thread (and sometimes more) until it completes. Not only that but the only way to do PIO only at the hardware level is to take the controller out of AHCI mode into legacy mode which means doing a lot of PIO accesses routed to the drive.

Quoted 2 lines

Actually, that's what I do - except for locking the page, yet. I do need to access the PCI bus in real time, though. So I wonder what happens

I didnt think any system with an ICH10 even had a legacy PCI bus.

Quoted 4 lines

when the RT CPU is getting data from the PCI device, doing some calculations on it and then writing back some data to the device, __while at the same time__ another (non-RT) CPU starts a DMA transfer. I figured the DMA would block the PCI bus,

It depends upon how you have configured the bus, devices and chipset what priority rules are active and also what size bursts are used for the transfers. In particular read up on the PCI latency setting for legacy PCI bus devices.

Alan