map_user_kiobuf problem in 2.4.0-test3
Hi,
Here's the scenario:
2.4.0-test3 SMP build running on a single 800MHz PIII (Dell GX-300) After obtaining a mapping to a high memory region (i.e. either PCI memory or physical memory reserved by passing mem=XXX to the kernel at boot), I am trying to write a raw device with data in the mapped region.
This fails, with map_user_kiobuf spitting out "Mapped page missing" The raw write works, of course, if the mapping is to a kmalloc'ed buffer.
I have tried the above with 2.2.14 SMP, and it works, so something in 2.4 is broken.
On another interesting note: The raw devices I'm writing to are Fibre Channel drives controlled by a Qlogic 2200 adapter (in 2.2.14 I'm using the Qlogic driver). When writing large sequential blocks to a single drive, I reached 8MB/s when the memory was mapped to the high reserved region, while CPU utilization was down to about 5%. When the mapping was to PCI space, I was able to write at only 4MB/s, and CPU utilization was up to 60%! This is very strange, since if the transfer rate was for some unknown reason lower in the case of PCI (vs. high physical memory), then one would expect the CPU utilization to be even lower, since the adapter performs DMA. But instead, the CPU is sweating... So, it appears that there's a problem in 2.2.14 as well, when the mapping is to PCI space... Additionally, the max transfer rate of 8MB/s seems rather slow - don't know why yet...
-Mark
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 7 lines
On another interesting note: The raw devices I'm writing to are Fibre Channel drives controlled by a Qlogic 2200 adapter (in 2.2.14 I'm using the Qlogic driver). When writing large sequential blocks to a single drive, I reached 8MB/s when the memory was mapped to the high reserved region, while CPU utilization was down to about 5%. When the mapping was to PCI space, I was able to write at only 4MB/s, and CPU utilization was up to 60%!
The data is copied from a buffer to the pci device. DMA transfers going directly to pci space is impossible without (small) changes to 2.2. 2.4 has the theoretic possibility to do it and checks already for that, but how it should be done practically I'd like to know too.
bye, Roman
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi again,
Quoted 8 lines · Roman Zippel
Quoted 7 lines
On another interesting note: The raw devices I'm writing to are Fibre Channel drives controlled by a Qlogic 2200 adapter (in 2.2.14 I'm using the Qlogic driver). When writing large sequential blocks to a single drive, I reached 8MB/s when the memory was mapped to the high reserved region, while CPU utilization was down to about 5%. When the mapping was to PCI space, I was able to write at only 4MB/s, and CPU utilization was up to 60%!
Quoted 4 lines
The data is copied from a buffer to the pci device. DMA transfers going directly to pci space is impossible without (small) changes to 2.2. 2.4 has the theoretic possibility to do it and checks already for that, but how it should be done practically I'd like to know too.
Okay - if you're right, then this explains the poor performance I'm seeing for PCI-PCI raw I/O on 2.2.14... Raw I/O is senseless if bounce buffers are used. Still, raw I/O (the kiobuf stuff) seems to be quite broken on 2.4.0t3...
-Mark
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
Hi,
Quoted 10 lines · Mark Mokryn
Here's the scenario:
2.4.0-test3 SMP build running on a single 800MHz PIII (Dell GX-300) After obtaining a mapping to a high memory region (i.e. either PCI memory or physical memory reserved by passing mem=XXX to the kernel at boot), I am trying to write a raw device with data in the mapped region.
This fails, withmap_user_kiobufspitting out "Mapped page missing" The raw write works, of course, if the mapping is to a kmalloc'ed buffer.
Quoted 2 lines
I have tried the above with 2.2.14 SMP, and it works, so something in 2.4 is broken.
No. 2.4 is merely "changed" --- that's different!
The 2.2 mechanism was fine for all memory under 4GB --- ie. memory you can address with an unsigned 32 bit integer. 2.4 includes large memory support for 64GB memory on Intel, and we looked at ways of making kiobufs work with this. The only reasonable solution was to use "struct page *" pointers in the kiobuf in 2.4.
That is a design decision, and is deliberate. It means that we don't have the ability to address IO aperture pages directly using the normal struct page *'s from the mem_map array.
Eventually we want to be able to allow new struct page arrays to be allocated at run time to describe arbitrary high-memory ranges, and that will provide natural support for kiobufs on IO apertures. For now, however, it just won't work, and we're aware of the missing functionality.
Quoted 10 lines
On another interesting note: The raw devices I'm writing to are Fibre Channel drives controlled by a Qlogic 2200 adapter (in 2.2.14 I'm using the Qlogic driver). When writing large sequential blocks to a single drive, I reached 8MB/s when the memory was mapped to the high reserved region, while CPU utilization was down to about 5%. When the mapping was to PCI space, I was able to write at only 4MB/s, and CPU utilization was up to 60%! This is very strange, since if the transfer rate was for some unknown reason lower in the case of PCI (vs. high physical memory), then one would expect the CPU utilization to be even lower, since the adapter performs DMA. But instead, the CPU is sweating...
PCI reads are very, very slow. If the CPU is contending with a large DMA transfer for access to the bus, then you might find that the high CPU utilisation is just coming about due to the CPU's IO or memory accesses experiencing a massive slowdown due to bus contention. PCI is a _lot_ faster for write than for read to PCI-mapped memory.
Quoted 2 lines
So, it appears that
there's a problem in 2.2.14 as well, when the mapping is to PCI space...
I'd need to see a bus analysis first --- this could well be hardware...
Quoted 2 lines
Additionally, the max transfer rate of 8MB/s seems rather slow - don't know why yet...
I have no trouble getting much more than that from raw I/O, but the qlogics may have bandwidth problems if the IO size is small (current raw IO for 2.2 limits the IO chunk size to 64kB).
Cheers,
Stephen
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/