fs/aio: Replace kmap{,_atomic}() with kmap_local_page()
PATCH RESEND 1/1 fs/aio: Replace kmap{,_atomic}() with kmap_local_page()
The use of kmap() and kmap_atomic() are being deprecated in favor of kmap_local_page().
There are two main problems with kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.
With kmap_local_page() the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.
Since its use in fs/aio.c is safe everywhere, it should be preferred.
Therefore, replace kmap() and kmap_atomic() with kmap_local_page() in fs/aio.c.
Tested with xfstests on a QEMU/KVM x86_32 VM, 6GB RAM, booting a kernel with HIGHMEM64GB enabled.
---
I've tested with "./check -g aio". The tests in this group fail 3/26 times, with and without my patch. Therefore, these changes don't introduce further errors. I'm not aware of any further tests I may run, so that any suggestions would be precious and much appreciated :-)
I'm resending this patch because some recipients were missing in the previous submissions. In the meantime I'm also adding some more information in the commit message. There are no changes in the code.
fs/aio.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
fs/aio.c +16 -16
@@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ctx->user_id = ctx->mmap_base; ctx->nr_events = nr_events;/* trusted copy */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->nr = nr_events;/* user copy */ ring->id = ~0U ; ring->head = ring->tail =0 ;
@@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events) ring->compat_features =AIO_RING_COMPAT_FEATURES ; ring->incompat_features =AIO_RING_INCOMPAT_FEATURES ; ring->header_length =sizeof (struct aio_ring ); -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]);return 0 ;
@@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm)* we are protected from page migration * changes ring_pages by ->ring_lock. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->id = ctx->id; -kunmap_atomic (ring); +kunmap_local (ring);return 0 ; }
@@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx *ctx)* against ctx->completed_events below will make sure we do the * safe/right thing. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; -kunmap_atomic (ring); +kunmap_local (ring);refill_reqs_available (ctx, head, ctx->tail); }
@@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)if (++tail >= ctx->nr_events) tail =0 ; - ev_page =kmap_atomic (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); + ev_page =kmap_local_page (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); event = ev_page + pos %AIO_EVENTS_PER_PAGE ; *event = iocb->ki_res; -kunmap_atomic (ev_page); +kunmap_local (ev_page);flush_dcache_page (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,
@@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb) ctx->tail = tail; - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; ring->tail = tail; -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]); ctx->completed_events++;
@@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx *ctx,mutex_lock (&ctx->ring_lock);/* Access to ->ring_pages here is protected by ctx->ring_lock. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; tail = ring->tail; -kunmap_atomic (ring); +kunmap_local (ring);/* * Ensure that once we've read the current tail pointer, that
@@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx *ctx, avail =min (avail, nr - ret); avail =min_t (long, avail,AIO_EVENTS_PER_PAGE - pos); - ev =kmap (page); + ev =kmap_local_page (page); copy_ret =copy_to_user (event + ret, ev + pos,sizeof (*ev) * avail); -kunmap (page); +kunmap_local (ev);if (unlikely (copy_ret)) { ret = -EFAULT ;
@@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx, head %= ctx->nr_events; } - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->head = head; -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]);pr_debug ("%li h%u t%u\n" , ret, head, tail);
Signature · 1 line
2.36.1
Quoted 9 lines · Fabio M. De Francesco
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.
Quoted 7 lines
With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.Since its use in
fs/aio.cis safe everywhere, it should be preferred.
That sentence is very ambiguous. I don't know what "its" refers to, and I'm not sure what "safe" means in this context.
The patch looks okay to me.
Quoted 17 lines · CEST Jeff Moyer
Quoted 15 lines · Fabio M. De Francesco
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.
Quoted 4 lines
Quoted 1 line
Since its use in
fs/aio.cis safe everywhere, it should be preferred.That sentence is very ambiguous. I don't know what "its" refers to, and I'm not sure what "safe" means in this context.
I'm sorry for not being clearer.
"its use" means "the use of kmap_local_page()". Few lines above you may also see "It is faster", meaning "kmap_local_page() is faster".
The "safety" is a very concise way to assert that I've checked, by code inspection and by testing (as it is better detailed some lines below) that these conversions (1) don't break any of the rules of use of local mapping when converting kmap() (please read highmem.rst about these) and (2) the call sites of kmap_atomic() didn't rely on its side effects (pagefaults disable and potential preemption disables).
Therefore, you may read it as it was: "The use of kmap_local_page() in fs/ aio.c has been carefully checked to assure that the conversions won't break the code, therefore the newer API is preferred".
I hope it makes my argument clearer.
Quoted 5 lines
The patch looks okay to me.
Reviewed-by Jeff Moyer <[email protected]>
Thank you so much for the "Reviewed-by" tag.
Regards,
Fabio
Quoted 20 lines · Fabio M. De Francesco
Quoted 19 lines · CEST Jeff Moyer
Quoted 16 lines · Fabio M. De Francesco
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.Since its use in
fs/aio.cis safe everywhere, it should be preferred.
Quoted 7 lines
Quoted 2 lines
That sentence is very ambiguous. I don't know what "its" refers to, and I'm not sure what "safe" means in this context.
I'm sorry for not being clearer.
"its use" means "the use of
kmap_local_page()". Few lines above you may also see "It is faster", meaning "kmap_local_page()is faster".
Got it, thanks.
Quoted 6 lines
The "safety" is a very concise way to assert that I've checked, by code inspection and by testing (as it is better detailed some lines below) that these conversions (1) don't break any of the rules of use of local mapping when converting
kmap()(please readhighmem.rstabout these) and (2) the call sites ofkmap_atomic()didn't rely on its side effects (pagefaults disable and potential preemption disables).
OK, good. I agree that the aio code wasn't relying on the side effects of kmap_atomic.
Quoted 5 lines
Therefore, you may read it as it was: "The use of
kmap_local_page()infs/aio.chas been carefully checked to assure that the conversions won't break the code, therefore the newer API is preferred".I hope it makes my argument clearer.
Yes, thank you for explaining!
-Jeff
Quoted 12 lines
Quoted 6 lines
The patch looks okay to me.
Reviewed-by Jeff Moyer <[email protected]>Thank you so much for the "Reviewed-by" tag.
Regards,
Fabio
fs/aio.c static int aio_setup_ring(struct kioctx *ctx, unsigned
Quoted 46 lines · On domenica 16 ottobre 2022 17:06:56 CET Fabio M. De Francesco wrote:
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.Since its use in
fs/aio.cis safe everywhere, it should be preferred.Therefore, replace
kmap()andkmap_atomic()withkmap_local_page()infs/aio.c.Tested with xfstests on a QEMU/KVM
x86_32VM, 6GB RAM, booting a kernel with HIGHMEM64GB enabled.Cc "Venkataramanan, Anirudh" <[email protected]>Suggested-by Ira Weiny <[email protected]>Reviewed-by Ira Weiny <[email protected]>Signed-off-by Fabio M. De Francesco <[email protected]>---
I've tested with "./check -g aio". The tests in this group fail 3/26 times, with and without my patch. Therefore, these changes don't introduce further errors. I'm not aware of any further tests I may run, so that any suggestions would be precious and much appreciated :-)
I'm resending this patch because some recipients were missing in the previous submissions. In the meantime I'm also adding some more information in the commit message. There are no changes in the code.
fs/aio.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)fs/aio.c +0
-0@@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned
int
fs/aio.c static int aio_setup_ring(struct kioctx *ctx, unsigned
Quoted 9 lines
fs/aio.c +1
-1nr_events) ctx->user_id = ctx->mmap_base; ctx->nr_events = nr_events;/* trusted copy */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->nr = nr_events;/* user copy */ ring->id = ~0U ; ring->head = ring->tail =0 ;@@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned
int
Quoted 36 lines
fs/aio.c +1
-1nr_events) ring->compat_features = AIO_RING_COMPAT_FEATURES; ring->incompat_features =AIO_RING_INCOMPAT_FEATURES ; ring->header_length =sizeof (struct aio_ring ); -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]);return 0 ;@@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct
mm_struct*mm) * we are protected from page migration
* changesring_pagesby -ring_lock.fs/aio.c +0
-1*/ - ring = kmap_atomic(ctx-ring_pages[0]);
fs/aio.c +1
-0+ ring = kmap_local_page(ctx-ring_pages[0]);
fs/aio.c +3
-3ring->id = ctx->id; -kunmap_atomic (ring); +kunmap_local (ring);return 0 ; }@@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx*ctx) * against ctx->completed_events below will make sure we do the * safe/right thing. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; -kunmap_atomic (ring); +kunmap_local (ring);refill_reqs_available (ctx, head, ctx->tail); }
fs/aio.c static void aio_complete(struct aio_kiocb *iocb) · line 1135
Quoted 5 lines
fs/aio.c +1
-1@@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)if (++tail >= ctx->nr_events) tail =0 ; - ev_page =kmap_atomic (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); + ev_page = kmap_local_page(ctx->ring_pages[pos /
AIO_EVENTS_PER_PAGE]);
Quoted 61 lines
fs/aio.c +9
-10event = ev_page + pos %AIO_EVENTS_PER_PAGE ; *event = iocb->ki_res; -kunmap_atomic (ev_page); +kunmap_local (ev_page);flush_dcache_page (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,@@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb) ctx->tail = tail; - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; ring->tail = tail; -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]); ctx->completed_events++;@@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx *ctx,mutex_lock (&ctx->ring_lock);/* Access to ->ring_pages here is protected by ctx->ring_lock. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; tail = ring->tail; -kunmap_atomic (ring); +kunmap_local (ring);/* * Ensure that once we've read the current tail pointer, that @@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx *ctx, avail =min (avail, nr - ret); avail =min_t (long, avail,AIO_EVENTS_PER_PAGE - pos); - ev =kmap (page); + ev =kmap_local_page (page); copy_ret =copy_to_user (event + ret, ev + pos,sizeof (*ev) * avail); -kunmap (page); +kunmap_local (ev);if (unlikely (copy_ret)) { ret = -EFAULT ;@@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx, head %= ctx->nr_events; } - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->head = head; -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]);pr_debug ("%li h%u t%u\n" , ret, head, tail); --
Quoted 1 line
2.36.1
Al, Benjamin,
I'm sending a gentle ping for this old patch too (and thanking Al again for pointing me out how fs/ufs and fs/sysv conversions must be reworked and mistakes fixed).
About this I've had Ira's and Jeff's "Reviewed-by:" tags. I also responded in this thread to a couple of objections from Jeff which were regarding some ambiguities in the commit message.
Please let me know if here too there are mistakes which must be fixed and code to be reworked.
I'm currently just a little more than a pure hobbyist, therefore please be patient for the time it takes because until mid February 2023 I'll only be able to work few hours per weeks using my spare time.
I'm looking forward to hearing from you.
Regards,
Fabio
Quoted 20 lines · On domenica 16 ottobre 2022 17:06:56 CET Fabio M. De Francesco wrote:
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.Since its use in
fs/aio.cis safe everywhere, it should be preferred.Therefore, replace
kmap()andkmap_atomic()withkmap_local_page()infs/aio.c.
Quoted 6 lines
Tested with xfstests on a QEMU/KVM
x86_32VM, 6GB RAM, booting a kernel with HIGHMEM64GB enabled.Cc "Venkataramanan, Anirudh" <[email protected]>Suggested-by Ira Weiny <[email protected]>Reviewed-by Ira Weiny <[email protected]>
Quoted 2 lines
Signed-off-by Fabio M. De Francesco <[email protected]>---
I'm sorry to resend again. Last time I forgot to forward the "Reviewed-by:" tag from Jeff (thanks!).
fs/aio.c static int aio_setup_ring(struct kioctx *ctx, unsigned
Quoted 18 lines
I've tested with "./check -g aio". The tests in this group fail 3/26 times, with and without my patch. Therefore, these changes don't introduce further errors. I'm not aware of any further tests I may run, so that any suggestions would be precious and much appreciated :-)
I'm resending this patch because some recipients were missing in the previous submissions. In the meantime I'm also adding some more information in the commit message. There are no changes in the code.
fs/aio.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)fs/aio.c +0
-0@@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned
int
fs/aio.c static int aio_setup_ring(struct kioctx *ctx, unsigned
Quoted 9 lines
fs/aio.c +1
-1nr_events) ctx->user_id = ctx->mmap_base; ctx->nr_events = nr_events;/* trusted copy */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->nr = nr_events;/* user copy */ ring->id = ~0U ; ring->head = ring->tail =0 ;@@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned
int
Quoted 36 lines
fs/aio.c +1
-1nr_events) ring->compat_features = AIO_RING_COMPAT_FEATURES; ring->incompat_features =AIO_RING_INCOMPAT_FEATURES ; ring->header_length =sizeof (struct aio_ring ); -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]);return 0 ;@@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct
mm_struct*mm) * we are protected from page migration
* changesring_pagesby -ring_lock.fs/aio.c +0
-1*/ - ring = kmap_atomic(ctx-ring_pages[0]);
fs/aio.c +1
-0+ ring = kmap_local_page(ctx-ring_pages[0]);
fs/aio.c +3
-3ring->id = ctx->id; -kunmap_atomic (ring); +kunmap_local (ring);return 0 ; }@@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx*ctx) * against ctx->completed_events below will make sure we do the * safe/right thing. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; -kunmap_atomic (ring); +kunmap_local (ring);refill_reqs_available (ctx, head, ctx->tail); }
fs/aio.c static void aio_complete(struct aio_kiocb *iocb) · line 1135
Quoted 5 lines
fs/aio.c +1
-1@@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)if (++tail >= ctx->nr_events) tail =0 ; - ev_page =kmap_atomic (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); + ev_page = kmap_local_page(ctx->ring_pages[pos /
AIO_EVENTS_PER_PAGE]);
Quoted 62 lines
fs/aio.c +9
-10event = ev_page + pos %AIO_EVENTS_PER_PAGE ; *event = iocb->ki_res; -kunmap_atomic (ev_page); +kunmap_local (ev_page);flush_dcache_page (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,@@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb) ctx->tail = tail; - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; ring->tail = tail; -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]); ctx->completed_events++;@@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx *ctx,mutex_lock (&ctx->ring_lock);/* Access to ->ring_pages here is protected by ctx->ring_lock. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; tail = ring->tail; -kunmap_atomic (ring); +kunmap_local (ring);/* * Ensure that once we've read the current tail pointer, that @@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx *ctx, avail =min (avail, nr - ret); avail =min_t (long, avail,AIO_EVENTS_PER_PAGE - pos); - ev =kmap (page); + ev =kmap_local_page (page); copy_ret =copy_to_user (event + ret, ev + pos,sizeof (*ev) * avail); -kunmap (page); +kunmap_local (ev);if (unlikely (copy_ret)) { ret = -EFAULT ;@@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx, head %= ctx->nr_events; } - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->head = head; -kunmap_atomic (ring); +kunmap_local (ring);flush_dcache_page (ctx->ring_pages[0 ]);pr_debug ("%li h%u t%u\n" , ret, head, tail); --2.36.1
Quoted 28 lines · On giovedì 1 dicembre 2022 15:29:17 CET Fabio M. De Francesco wrote:
Quoted 20 lines · On domenica 16 ottobre 2022 17:06:56 CET Fabio M. De Francesco wrote:
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.Since its use in
fs/aio.cis safe everywhere, it should be preferred.Therefore, replace
kmap()andkmap_atomic()withkmap_local_page()infs/aio.c.Quoted 6 lines
Tested with xfstests on a QEMU/KVM
x86_32VM, 6GB RAM, booting a kernel with HIGHMEM64GB enabled.Cc "Venkataramanan, Anirudh" <[email protected]>Suggested-by Ira Weiny <[email protected]>Reviewed-by Ira Weiny <[email protected]>
Quoted 15 lines
Reviewed-by Jeff Moyer <[email protected]>Quoted 2 lines
Signed-off-by Fabio M. De Francesco <[email protected]>---
I'm sorry to resend again. Last time I forgot to forward the "Reviewed-by:" tag from Jeff (thanks!).
Quoted 7 lines
I've tested with "./check -g aio". The tests in this group fail 3/26 times, with and without my patch. Therefore, these changes don't introduce further errors. I'm not aware of any further tests I may run, so that any suggestions would be precious and much appreciated :-)
I'm resending this patch because some recipients were missing in the previous submissions. In the meantime I'm also adding some more
information
Quoted 116 lines
fs/aio.c static int aio_setup_ring(struct kioctx *ctx, unsigned
Quoted 10 lines
in the commit message. There are no changes in the code.
fs/aio.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)fs/aio.c +0
-0@@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsignedint
fs/aio.c static int aio_setup_ring(struct kioctx *ctx, unsigned
Quoted 12 lines
fs/aio.c +1
-1nr_events) ctx->user_id = ctx->mmap_base; ctx->nr_events = nr_events;/* trusted copy */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->nr = nr_events;/* user copy */ fs/aio.c +0
-0ring->id = ~0U ; ring->head = ring->tail =0 ;@@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsignedint
Quoted 57 lines
fs/aio.c +1
-1nr_events) ring->compat_features = AIO_RING_COMPAT_FEATURES; ring->incompat_features =AIO_RING_INCOMPAT_FEATURES ; ring->header_length =sizeof (struct aio_ring ); -kunmap_atomic (ring); +kunmap_local (ring);fs/aio.c +0
-0flush_dcache_page (ctx->ring_pages[0 ]);return 0 ;@@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct
mm_struct*mm) * we are protected from page migration* changes
ring_pagesby -
ring_lock.fs/aio.c +0
-1*/ - ring = kmap_atomic(ctx-ring_pages[0]);
fs/aio.c +1
-0+ ring = kmap_local_page(ctx-ring_pages[0]);
fs/aio.c +3
-3ring->id = ctx->id; -kunmap_atomic (ring); +kunmap_local (ring);return 0 ; }@@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx*ctx) * against ctx->completed_events below will make sure we do the * safe/right thing. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; -kunmap_atomic (ring); +kunmap_local (ring);refill_reqs_available (ctx, head, ctx->tail); }fs/aio.c static void aio_complete(struct aio_kiocb *iocb) · line 1137
Quoted 7 lines
fs/aio.c +1
-1@@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)if (++tail >= ctx->nr_events) tail =0 ; - ev_page =kmap_atomic (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); + ev_page = kmap_local_page(ctx->ring_pages[pos /
AIO_EVENTS_PER_PAGE]);Quoted 21 lines
fs/aio.c +1
-1event = ev_page + pos %AIO_EVENTS_PER_PAGE ; *event = iocb->ki_res; -kunmap_atomic (ev_page); +kunmap_local (ev_page);fs/aio.c +1
-1flush_dcache_page (ctx->ring_pages[pos /AIO_EVENTS_PER_PAGE ]); pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,@@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb) ctx->tail = tail; - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; ring->tail = tail;
Quoted 8 lines
fs/aio.c static long aio_read_events_ring(struct kioctx
Quoted 8 lines
fs/aio.c +1
-1-kunmap_atomic (ring); +kunmap_local (ring);fs/aio.c +0
-0flush_dcache_page (ctx->ring_pages[0 ]); ctx->completed_events++;@@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx
*ctx,
Quoted 11 lines
Quoted 11 lines
fs/aio.c +1
-1mutex_lock (&ctx->ring_lock);/* Access to ->ring_pages here is protected by ctx->ring_lock. */ - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); head = ring->head; tail = ring->tail;
Quoted 8 lines
fs/aio.c static long aio_read_events_ring(struct kioctx
Quoted 8 lines
fs/aio.c +1
-1-kunmap_atomic (ring); +kunmap_local (ring);fs/aio.c +0
-0/* * Ensure that once we've read the current tail pointer, that @@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx
*ctx,
Quoted 32 lines
Quoted 32 lines
fs/aio.c +2
-2avail =min (avail, nr - ret); avail =min_t (long, avail,AIO_EVENTS_PER_PAGE - pos); - ev =kmap (page); + ev =kmap_local_page (page); copy_ret =copy_to_user (event + ret, ev + pos,sizeof (*ev) * avail); -kunmap (page); +kunmap_local (ev);fs/aio.c +2
-2if (unlikely (copy_ret)) { ret = -EFAULT ;@@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx, head %= ctx->nr_events; } - ring =kmap_atomic (ctx->ring_pages[0 ]); + ring =kmap_local_page (ctx->ring_pages[0 ]); ring->head = head; -kunmap_atomic (ring); +kunmap_local (ring);
Quoted 6 lines
Quoted 6 lines
fs/aio.c +0
-1flush_dcache_page (ctx->ring_pages[0 ]);pr_debug ("%li h%u t%u\n" , ret, head, tail); --2.36.1
Please disregard this patch because I just sent a v2 with some additional information in the commit message and added Jeff's "Reviewed-by" tag.
Thanks,
Fabio
Quoted 15 lines · Fabio M. De Francesco
The use of
kmap()andkmap_atomic()are being deprecated in favor ofkmap_local_page().There are two main problems with
kmap(): (1) It comes with an overhead as the mapping space is restricted and protected by a global lock for synchronization and (2) it also requires global TLB invalidation when the kmap’s pool wraps and it might block when the mapping space is fully utilized until a slot becomes available.With
kmap_local_page()the mappings are per thread, CPU local, can take page faults, and can be called from any context (including interrupts). It is faster thankmap()in kernels with HIGHMEM enabled. Furthermore, the tasks can be preempted and, when they are scheduled to run again, the kernel virtual addresses are restored and still valid.
Quoted 7 lines
Since its use in
fs/aio.cis safe everywhere, it should be preferred.Therefore, replace
kmap()andkmap_atomic()withkmap_local_page()infs/aio.c.Tested with xfstests on a QEMU/KVM
x86_32VM, 6GB RAM, booting a kernel with HIGHMEM64GB enabled.
I was just looking over this code and made the same kmap -> kmap_local change, but you've done a more complete version - nice.
For context, I was the one who added the kmap() call, because copy_to_user() can sleep - anything else at the time would've been more awkward, and highmem machines were already on the way out. But kmap_local is perfect here :)