iov_iter: optimize page_copy_sane()
Subsystems (2)
PATCH 1/1 iov_iter: optimize page_copy_sane()
Avoid cache line miss dereferencing struct page if we can.
page_copy_sane() mostly deals with order-0 pages.
Signed-off-by Eric Dumazet <[email protected]>
Cc Al Viro <[email protected]>
--- lib/iov_iter.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
lib/iov_iter.c +12 -2
@@ -854,8 +854,18 @@ EXPORT_SYMBOL(_copy_from_iter_full_nocache);static inline bool page_copy_sane (struct page *page,size_t offset,size_t n) { -struct page *head =compound_head (page); -size_t v = n + offset +page_address (page) -page_address (head); +struct page *head; +size_t v = n + offset; + +/* +* Fast path if this is an order-0 page, +* or if the copied area fits the first page of a compound one. +*/ +if (likely (n <= v && v <=PAGE_SIZE )) +return true; + + head =compound_head (page); + v += (page - head) <<PAGE_SHIFT ;if (likely (n <= v && v <= (PAGE_SIZE <<compound_order (head))))return true;
Signature · 1 line
2.20.0.rc2.403.gdbc3b29805-goog
Quoted 26 lines · Eric Dumazet
Avoid cache line miss dereferencing struct page if we can.
page_copy_sane()mostly deals with order-0 pages.Signed-off-by Eric Dumazet <[email protected]>Cc Al Viro <[email protected]>--- lib/iov_iter.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)lib/iov_iter.c +5
-2@@ -854,8 +854,18 @@ EXPORT_SYMBOL(_copy_from_iter_full_nocache);static inline bool page_copy_sane (struct page *page,size_t offset,size_t n) { -struct page *head =compound_head (page); -size_t v = n + offset +page_address (page) -page_address (head); +struct page *head; +size_t v = n + offset; + +/* +* Fast path if this is an order-0 page,
lib/iov_iter.c EXPORT_SYMBOL(_copy_from_iter_full_nocache); · line 871
Quoted 10 lines
lib/iov_iter.c +7
-0+* or if the copied area fits the first page of a compound one. +*/ +if (likely (n <= v && v <=PAGE_SIZE )) +return true; + + head =compound_head (page); + v += (page - head) <<PAGE_SHIFT ;if (likely (n <= v && v <= (PAGE_SIZE <<compound_order (head))))return true;
Hi Alexander !
Any news on this patch ?
Thanks !