Skip to contentSkip to conversation

Memory mapped IO vs Port IO

DISCUSSION Andi Kleen 6 participants 13 messages Sep 11, 2003 – Sep 12, 2003 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20030911160116.GI21596@parcelfarce.linux.theplanet.co.uk.suse.lists.linux.kernel

Andi Kleen

author
#1 lore raw
Quoted 2 lines · Matthew Wilcox

There's a lot of drivers in the tree that allow you to access the device either via IO port space or IO mem space. Here's some examples:

My gut feeling is to just fix the drivers to make this runtime switchable and get rid of the compile time options.

This would help distributions (who normally want to build conservative by default, but still allow the users easy tuning without recompilation) For that it would be nice if a standard module parameter or maybe sysfs option existed.

The overhead of checking for PIO vs mmio at runtime in the drivers should be completely in the noise on any non ancient CPU (both MMIO and PIO typically take hundreds or thousands of CPU cycles for the bus access, having an dynamic function call or an if before that is makes no difference at all)

-Andi

#2 lore raw
Quoted 5 lines · Andi Kleen

The overhead of checking for PIO vs mmio at runtime in the drivers should be completely in the noise on any non ancient CPU (both MMIO and PIO typically take hundreds or thousands of CPU cycles for the bus access, having an dynamic function call or an if before that is makes no difference at all)

That's not true for MMIO writes which are posted. They should take no longer than a memory write. For MMIO reads and PIO reads & writes, you are, of course, correct.

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
author
#3 lore raw
Quoted 10 lines · On Thu, 11 Sep 2003 17:25:04 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 5 lines · Andi Kleen

The overhead of checking for PIO vs mmio at runtime in the drivers should be completely in the noise on any non ancient CPU (both MMIO and PIO typically take hundreds or thousands of CPU cycles for the bus access, having an dynamic function call or an if before that is makes no difference at all)

That's not true for MMIO writes which are posted. They should take no longer than a memory write. For MMIO reads and PIO reads & writes, you are, of course, correct.

Even a memory write is tens to hundres of cycles.

-Andi

#4 lore raw
Quoted 7 lines · Andi Kleen
Quoted 3 lines · On Thu, 11 Sep 2003 17:25:04 +0100 Matthew Wilcox <[email protected]> wrote:

That's not true for MMIO writes which are posted. They should take no longer than a memory write. For MMIO reads and PIO reads & writes, you are, of course, correct.

Even a memory write is tens to hundres of cycles.

So are mispredicted branches.

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
#5 lore raw
In reply to Andi Kleen
Quoted 1 line · Andi Kleen

Even a memory write is tens to hundres of cycles.

Not from the CPU's perspective. It is done in parallel with other instructions.

-- Jamie

author
#6 lore raw
Quoted 5 lines · On Thu, 11 Sep 2003 18:12:05 +0100 Jamie Lokier <[email protected]> wrote:
Quoted 1 line · Andi Kleen

Even a memory write is tens to hundres of cycles.

Not from the CPU's perspective. It is done in parallel with other instructions.

Only when there are more instructions to execute. But device driver code often does a following read e.g. to check if it can submit another request to the hardware.

My claim is basically:

Change everybody who currently does

#ifdef CONFIG_MMIO
	writel(... )
	readl(...)
#else
	outl( ... ) 
	inl ( ...) 

#endif

to

	if (dev->mmio) { 
		writel(); 
		real();
	} else { 
		outl();
		inl();
	} 

and you will have a hard time to benchmark the difference on any non ancient system in actual driver operation.

-Andi

#7 lore raw
Quoted 34 lines · Andi Kleen
Quoted 5 lines · On Thu, 11 Sep 2003 18:12:05 +0100 Jamie Lokier <[email protected]> wrote:
Quoted 1 line · Andi Kleen

Even a memory write is tens to hundres of cycles.

Not from the CPU's perspective. It is done in parallel with other instructions.

Only when there are more instructions to execute. But device driver code often does a following read e.g. to check if it can submit another request to the hardware.

My claim is basically:

Change everybody who currently does

#ifdef CONFIG_MMIO
	writel(... )
	readl(...)
#else
	outl( ... ) 
	inl ( ...) 

#endif

to

	if (dev->mmio) { 
		writel(); 
		real();
	} else { 
		outl();
		inl();
	} 
Quoted 2 lines

and you will have a hard time to benchmark the difference on any non ancient system in actual driver operation.

Shouldn't that be

dev->dev_ops->writel(...);
dev->dev_ops->readl(...);

with no conditionals?

Signature · 5 lines
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		[email protected]

		Remember Cernan and Schmitt
#8 lore raw
In reply to Andi Kleen
Quoted 34 lines · Andi Kleen
Quoted 5 lines · On Thu, 11 Sep 2003 18:12:05 +0100 Jamie Lokier <[email protected]> wrote:
Quoted 1 line · Andi Kleen

Even a memory write is tens to hundres of cycles.

Not from the CPU's perspective. It is done in parallel with other instructions.

Only when there are more instructions to execute. But device driver code often does a following read e.g. to check if it can submit another request to the hardware.

My claim is basically:

Change everybody who currently does

#ifdef CONFIG_MMIO
        writel(... )
        readl(...)
#else
        outl( ... ) 
        inl ( ...) 

#endif

to

        if (dev->mmio) { 
                writel(); 
                real();
        } else { 
                outl();
                inl();
        } 
Quoted 6 lines

and you will have a hard time to benchmark the difference on any non ancient system
in actual driver operation.

-Andi

Hello,
Wouldn't it be better if we set the IN and OUT function pointers to the right functions during driver init. based on the setting of dev->mmio. And throughout the driver, we just call the IN and OUT functions by their pointers. Then we don't have to do if (dev->mmio) every time. It's similar to the concept of virtual member function in C++.

ADT

#9 lore raw
Quoted 5 lines · Anthony Dominic Truong

Wouldn't it be better if we set the IN and OUT function pointers to the right functions during driver init. based on the setting of dev->mmio. And throughout the driver, we just call the IN and OUT functions by their pointers. Then we don't have to do if (dev->mmio) every time. It's similar to the concept of virtual member function in C++.

I think it would be faster to hide the "if (mmio)" part into an inline function which calls one or the other, given a resource cookie.

Branch prediction will remove most of the cost of a conditional test, which is always the same in these drivers after all, but I don't think it's quite so good at predicting indirect function calls.

-- Jamie

#10 lore raw
In reply to Anthony Dominic Truong
Quoted 22 lines · Anthony Dominic Truong
Quoted 21 lines · Andi Kleen
#ifdef CONFIG_MMIO
        writel(... )
        readl(...)
#else
        outl( ... ) 
        inl ( ...) 

#endif

to

        if (dev->mmio) { 
                writel(); 
                real();
        } else { 
                outl();
                inl();
        } 

and you will have a hard time to benchmark the difference on any non ancient system
in actual driver operation.

Quoted 8 lines
Quoted 2 lines

-Andi

Hello,
Wouldn't it be better if we set the IN and OUT function pointers to the right functions during driver init. based on the setting of dev->mmio. And throughout the driver, we just call the IN and OUT functions by their pointers. Then we don't have to do if (dev->mmio) every time. It's similar to the concept of virtual member function in C++.

I'd rather not see any more pointer dereferences or even branches than absolutely necessary. Right now, readX/writeX and inX/outX are usually inlines, and outX and writeX can be very fast. So I'd prefer either a global CONFIG_MMIO option or consistent driver specific options that explain what the difference between MMIO and port I/O actually is.

Jesse

#11 lore raw

Ok, Andi asked for benchmarks, so I ran some. Let this should be a lesson on why you shouldn't use port I/O :) I ran these on an SGI Altix w/900 MHz McKinley processors.

Just straight calls to the routines (all of these are based on the average of 100 iterations):
writeq(val, reg) time: 64 cycles
outl(val, reg) time: 2126 cycles

A simple branch:
if (use_mmio)
writeq(val, reg) time: 132 cycles
else
outl(val, reg) time: 1990 cycles

Using a wrapper to point to one of the routines:
wrapper->write(val, reg) time: 215 cycles
wrapper->out(val, reg) time: 3931 cycles

Thanks,
Jesse

#12 lore raw
In reply to Andi Kleen
Quoted 7 lines · Andi Kleen

My gut feeling is to just fix the drivers to make this runtime switchable and get rid of the compile time options.

This would help distributions (who normally want to build conservative by default, but still allow the users easy tuning without recompilation) For that it would be nice if a standard module parameter or maybe sysfs option existed.

Another way to help distributions is to compile those drivers twice, once for each access type. There aren't all that many drivers that need it.

Quoted 5 lines

The overhead of checking for PIO vs mmio at runtime in the drivers should be completely in the noise on any non ancient CPU (both MMIO and PIO typically take hundreds or thousands of CPU cycles for the bus access, having an dynamic function call or an if before that is makes no difference at all)

Ah, but what about the ancient CPUs?

-- Jamie

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#14 lore raw
Quoted 4 lines · [email protected] (Jesse Barnes) writes:

I like this idea, esp. as a global option. It would help e.g. visws which can't reliably do MMIO. Most platforms could default MMIO to y also, to make things easier.

The problem AFAIK is usually that there are some buggy older chipset versions that do not work properly with MMIO, but others do. That's especially an issue on widely cloned cards like Tulip.

It's not a platform specific thing, more a device specific setting. Of course some platforms want to disable it completely too, but that's rare.

-Andi