ia64: race flushing icache in do_no_page path
Could you please confirm that I understand correctly what is in the:
Dual-Core Update to the Intel Itanium 2 Processor Reference Manual...
"2.3.3.2 L2 Caches
...
Any coherence request to identify whether a cache line is in the processor will invalidate that line from the L2I cache."
This makes sure that the DMAs invalidate the L2L cache.
"2.7.4 Instruction Cache Coherence Optimization
Coherence requests of the L1I and L2I caches will invalidate the line if it is in the cache. Montecito allows instruction requests on the system interface to be filtered such that they will not initiate coherence requests of the L1I and L2I caches. This will allow instructions to be cached at the L1I and L2I levels across multiple processors in a coherent domain. This optimization is enabled by default, but may be disabled by PAL_SET_PROC_FEATURES bit 5 of the Montecito feature_set (18)."
Machines star up whit bit 5 = 0, reading instruction pages via NFS has to flush them from L2I.
I was wondering if instead of modifying do_no_page() and Co., should not we make nfs_readpage() be DMA-like?
(No possible regression for most of the page I/O-s.)
I.e. it should be the responsibility of a file system to make sure it supports instruction pages correctly. The base kernel should provide such file systems with an architecture dependent macro...
Thanks,
Zoltan Menyhart
Quoted 3 lines · On Wed, 04 Jul 2007 16:24:38 +0200 Zoltan Menyhart <[email protected]>…
Machines star up whit bit 5 = 0, reading instruction pages via NFS has to flush them from L2I.
In our test, we confirmed that this can be fixed by flushing L2I just before SetPageUptodate() in NFS.
Quoted 8 lines
I was wondering if instead of modifying
do_no_page()and Co., should not we makenfs_readpage()be DMA-like?
(No possible regression for most of the page I/O-s.)
I.e. it should be the responsibility of a file system to make sure it supports instruction pages correctly. The base kernel should provide such file systems with an architecture dependent macro...
IMHO, for example, race in cooy-on-write (was fixed by Tony Luck) has to be fixed by MemoryManagement layer.
And only a race in do_no_page() seems to be able to be fixed by FS layer.
BTW, can we know whether a page is filled by DMA or not ?
Thanks,
-Kame
Quoted 9 lines · KAMEZAWA Hiroyuki
Quoted 3 lines · On Wed, 04 Jul 2007 16:24:38 +0200 Zoltan Menyhart <[email protected]>…
Machines star up whit bit 5 = 0, reading instruction pages via NFS has to flush them from L2I.
In our test, we confirmed that this can be fixed by flushing L2I just before
SetPageUptodate()in NFS.
I can agree.
We can be more permissive: it can be done anywhere after the new data is put in place and before nfs_readpage() or nfs_readpages() returns.
I saw your patch http://marc.info/?l=linux-mm&m8352909826277&w=2
that modifies e.g. mm/memory.c and not the NFS layer.
Have you proposed a patch against the NFS layer?
Quoted 10 lines
Quoted 7 lines
I was wondering if instead of modifying
do_no_page()and Co., should not we makenfs_readpage()be DMA-like?
(No possible regression for most of the page I/O-s.)
I.e. it should be the responsibility of a file system to make sure it supports instruction pages correctly. The base kernel should provide such file systems with an architecture dependent macro...IMHO, for example, race in cooy-on-write (was fixed by Tony Luck) has to be fixed by MemoryManagement layer.
I can agree.
Note that it has not got much performance impact from our point of view because there are not too many COW paves with executable code...
Quoted 1 line
And only a race in
do_no_page()seems to be able to be fixed by FS layer.
If it were do_no_page() that would fix the problem, then it should know where the page comes from in order not to flush the I-cache in vain. do_no_page() just calls vma->vm_ops->nopage() that goes down to fs_op->readpage() / fs_op->readpages().
On the other hand, these latter routines do not know why they fetch the page(s). Note that a page can be mapped more than one times, with different permission bits.
As a consequence, these routines will flush the I-cache in vain in most of the cases.
I prefer a performance impact to some non DMA based file systems and adding nothing to the path for the majority of the cases.
Quoted 1 line
BTW, can we know whether a page is filled by DMA or not ?
Well, the file systems based on block devices use DMA transfer (with the exception of using bounce buffers, in that case it is the responsibility of the bounce buffering layer to flush the I-cache).
Network based file systems require revision and update...
Note that only the processors with separate L2I require attention, the L1I is guaranteed to be flushed when you change the corresponding TBL entry.
The base kernel should provide a macro / service (that cares for the processor model) for the file systems.
Thanks,
Zoltan Menyhart
Quoted 9 lines · Zoltan Menyhart
Quoted 2 lines · KAMEZAWA Hiroyuki
In our test, we confirmed that this can be fixed by flushing L2I just before
SetPageUptodate()in NFS.I can agree.
We can be more permissive: it can be done anywhere after the new data is put in place and beforenfs_readpage()ornfs_readpages()returns.
Quoted 4 lines
I saw your patch http://marc.info/?l=linux-mm&m8352909826277&w=2
that modifies e.g.mm/memory.cand not the NFS layer.Have you proposed a patch against the NFS layer?
This really doesn't look like a job for the file system layer.
That would require all sorts of file system readpage routines to be modified to handle memory management details that are already handled by the memory.c functions. The do_no_page code is already dealing with the necessary icache flushing operations. It just happens to be doing it with a bad race condition for ia64.
Signature · 1 line
Mike Stroyan <[email protected]>