fs/buffer.c: disable per-CPU buffer_head cache for isolated CPUs
PATCH 1/1 fs/buffer.c: disable per-CPU buffer_head cache for isolated CPUs
For certain types of applications (for example PLC software or RAN processing), upon occurrence of an event, it is necessary to complete a certain task in a maximum amount of time (deadline).
One way to express this requirement is with a pair of numbers, deadline time and execution time, where:
* deadline time: length of time between event and deadline.
* execution time: length of time it takes for processing of event to occur on a particular hardware platform (uninterrupted).
The particular values depend on use-case. For the case where the realtime application executes in a virtualized guest, an IPI which must be serviced in the host will cause the following sequence of events:
1) VM-exit
2) execution of IPI (and function call)
3) VM-entry
Which causes an excess of 50us latency as observed by cyclictest (this violates the latency requirement of vRAN application with 1ms TTI, for example).
invalidate_bh_lrus calls an IPI on each CPU that has non empty per-CPU cache:
on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1);
The performance when using the per-CPU LRU cache is as follows:
42 ns per __find_get_block
68 ns per __find_get_block_slow
Given that the main use cases for latency sensitive applications do not involve block I/O (data necessary for program operation is locked in RAM), disable per-CPU buffer_head caches for isolated CPUs.
fs/buffer.c +6 -1
@@ -49,6 +49,7 @@#include <trace/events/block.h> #include <linux/fscrypt.h> #include <linux/fsverity.h> +#include <linux/sched/isolation.h> #include "internal.h"
@@ -1289,7 +1290,7 @@ static void bh_lru_install(struct buffer_head *bh)* failing page migration. * Skip putting upcoming bh into bh_lru until migration is done. */ -if (lru_cache_disabled ()) { +if (lru_cache_disabled () ||cpu_is_isolated (smp_processor_id ())) {bh_lru_unlock ();return ; }
@@ -1319,6 +1320,10 @@ lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size)check_irqs_on ();bh_lru_lock (); +if (cpu_is_isolated (smp_processor_id ())) { +bh_lru_unlock (); +return NULL ; + }for (i =0 ; i <BH_LRU_SIZE ; i++) {struct buffer_head *bh =__this_cpu_read (bh_lrus.bhs[i]);
Ping, apparently there is no objection to this patch...
Christian, what is the preferred tree for integration?
Quoted 74 lines · Marcelo Tosatti
For certain types of applications (for example PLC software or RAN processing), upon occurrence of an event, it is necessary to complete a certain task in a maximum amount of time (deadline).
One way to express this requirement is with a pair of numbers, deadline time and execution time, where:
* deadline time: length of time between event and deadline.
* execution time: length of time it takes for processing of event to occur on a particular hardware platform (uninterrupted).The particular values depend on use-case. For the case where the realtime application executes in a virtualized guest, an IPI which must be serviced in the host will cause the following sequence of events:
1) VM-exit
2) execution of IPI (and function call)
3) VM-entryWhich causes an excess of 50us latency as observed by cyclictest (this violates the latency requirement of vRAN application with 1ms TTI, for example).
invalidate_bh_lruscalls an IPI on each CPU that has non empty per-CPU cache:on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1);The performance when using the per-CPU LRU cache is as follows:
42 ns per
__find_get_block
68 ns per__find_get_block_slowGiven that the main use cases for latency sensitive applications do not involve block I/O (data necessary for program operation is locked in RAM), disable per-CPU
buffer_headcaches for isolated CPUs.Signed-off-by Marcelo Tosatti <[email protected]>fs/buffer.c +6
-1@@ -49,6 +49,7 @@#include <trace/events/block.h> #include <linux/fscrypt.h> #include <linux/fsverity.h> +#include <linux/sched/isolation.h> #include "internal.h" @@ -1289,7 +1290,7 @@ static void bh_lru_install(struct buffer_head *bh)* failing page migration. * Skip putting upcoming bh into bh_lru until migration is done. */ -if (lru_cache_disabled ()) { +if (lru_cache_disabled () ||cpu_is_isolated (smp_processor_id ())) {bh_lru_unlock ();return ; }@@ -1319,6 +1320,10 @@ lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size)check_irqs_on ();bh_lru_lock (); +if (cpu_is_isolated (smp_processor_id ())) { +bh_lru_unlock (); +return NULL ; + }for (i =0 ; i <BH_LRU_SIZE ; i++) {struct buffer_head *bh =__this_cpu_read (bh_lrus.bhs[i]);
Christian Brauner
maintainerQuoted 4 lines · Marcelo Tosatti
Ping, apparently there is no objection to this patch...
Christian, what is the preferred tree for integration?
It'd be good if we could get an Ack from someone familiar with isolated cpus for this; or just in general from someone who can ack this.
Quoted 8 lines · Christian Brauner
Quoted 4 lines · Marcelo Tosatti
Ping, apparently there is no objection to this patch...
Christian, what is the preferred tree for integration?
It'd be good if we could get an Ack from someone familiar with isolated cpus for this; or just in general from someone who can ack this.
Frederic?
Quoted 31 lines · Marcelo Tosatti
For certain types of applications (for example PLC software or RAN processing), upon occurrence of an event, it is necessary to complete a certain task in a maximum amount of time (deadline).
One way to express this requirement is with a pair of numbers, deadline time and execution time, where:
* deadline time: length of time between event and deadline.
* execution time: length of time it takes for processing of event to occur on a particular hardware platform (uninterrupted).The particular values depend on use-case. For the case where the realtime application executes in a virtualized guest, an IPI which must be serviced in the host will cause the following sequence of events:
1) VM-exit
2) execution of IPI (and function call)
3) VM-entryWhich causes an excess of 50us latency as observed by cyclictest (this violates the latency requirement of vRAN application with 1ms TTI, for example).
invalidate_bh_lruscalls an IPI on each CPU that has non empty per-CPU cache:on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1);
Quoted 8 lines
The performance when using the per-CPU LRU cache is as follows:
42 ns per
__find_get_block
68 ns per__find_get_block_slowGiven that the main use cases for latency sensitive applications do not involve block I/O (data necessary for program operation is locked in RAM), disable per-CPU
buffer_headcaches for isolated CPUs.
So what happens if they ever do I/O then? Like if they need to do some prep work before entering an isolated critical section?
Thanks.
Quoted 36 lines
Signed-off-by Marcelo Tosatti <[email protected]>fs/buffer.c +6
-1@@ -49,6 +49,7 @@#include <trace/events/block.h> #include <linux/fscrypt.h> #include <linux/fsverity.h> +#include <linux/sched/isolation.h> #include "internal.h" @@ -1289,7 +1290,7 @@ static void bh_lru_install(struct buffer_head *bh)* failing page migration. * Skip putting upcoming bh into bh_lru until migration is done. */ -if (lru_cache_disabled ()) { +if (lru_cache_disabled () ||cpu_is_isolated (smp_processor_id ())) {bh_lru_unlock ();return ; }@@ -1319,6 +1320,10 @@ lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size)check_irqs_on ();bh_lru_lock (); +if (cpu_is_isolated (smp_processor_id ())) { +bh_lru_unlock (); +return NULL ; + }for (i =0 ; i <BH_LRU_SIZE ; i++) {struct buffer_head *bh =__this_cpu_read (bh_lrus.bhs[i]);
Quoted 32 lines · Frederic Weisbecker
Quoted 31 lines · Marcelo Tosatti
For certain types of applications (for example PLC software or RAN processing), upon occurrence of an event, it is necessary to complete a certain task in a maximum amount of time (deadline).
One way to express this requirement is with a pair of numbers, deadline time and execution time, where:
* deadline time: length of time between event and deadline.
* execution time: length of time it takes for processing of event to occur on a particular hardware platform (uninterrupted).The particular values depend on use-case. For the case where the realtime application executes in a virtualized guest, an IPI which must be serviced in the host will cause the following sequence of events:
1) VM-exit
2) execution of IPI (and function call)
3) VM-entryWhich causes an excess of 50us latency as observed by cyclictest (this violates the latency requirement of vRAN application with 1ms TTI, for example).
invalidate_bh_lruscalls an IPI on each CPU that has non empty per-CPU cache:on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1);
Quoted 8 lines
Quoted 8 lines
The performance when using the per-CPU LRU cache is as follows:
42 ns per
__find_get_block
68 ns per__find_get_block_slowGiven that the main use cases for latency sensitive applications do not involve block I/O (data necessary for program operation is locked in RAM), disable per-CPU
buffer_headcaches for isolated CPUs.
Hi Frederic,
Quoted 2 lines
So what happens if they ever do I/O then? Like if they need to do some prep work before entering an isolated critical section?
Then instead of going through the per-CPU LRU buffer_head cache (__find_get_block), isolated CPUs will work as if their per-CPU cache is always empty, going through the slowpath (__find_get_block_slow). The algorithm is:
/*
* Perform a pagecache lookup for the matching buffer. If it's there, refresh
* it in the LRU and mark it as accessed. If it is not present then return
* NULL
*/
struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block, unsigned size)
{
struct buffer_head *bh = lookup_bh_lru(bdev, block, size);
if (bh == NULL) {
/* __find_get_block_slow will mark the page accessed */
bh = __find_get_block_slow(bdev, block);
if (bh)
bh_lru_install(bh);
} else
touch_buffer(bh);
return bh;
}EXPORT_SYMBOL(__find_get_block);
I think the performance difference between the per-CPU LRU cache VS __find_get_block_slow was much more significant when the cache was introduced. Nowadays its only 26ns (moreover modern filesystems do not use buffer_head's).
Quoted 1 line
Thanks.
Thank you for the review.
Quoted 38 lines
Quoted 36 lines
Signed-off-by Marcelo Tosatti <[email protected]>fs/buffer.c +6
-1@@ -49,6 +49,7 @@#include <trace/events/block.h> #include <linux/fscrypt.h> #include <linux/fsverity.h> +#include <linux/sched/isolation.h> #include "internal.h" @@ -1289,7 +1290,7 @@ static void bh_lru_install(struct buffer_head *bh)* failing page migration. * Skip putting upcoming bh into bh_lru until migration is done. */ -if (lru_cache_disabled ()) { +if (lru_cache_disabled () ||cpu_is_isolated (smp_processor_id ())) {bh_lru_unlock ();return ; }@@ -1319,6 +1320,10 @@ lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size)check_irqs_on ();bh_lru_lock (); +if (cpu_is_isolated (smp_processor_id ())) { +bh_lru_unlock (); +return NULL ; + }for (i =0 ; i <BH_LRU_SIZE ; i++) {struct buffer_head *bh =__this_cpu_read (bh_lrus.bhs[i]);
Frederic Weisbecker
Acked-byQuoted 28 lines · Marcelo Tosatti
Quoted 2 lines
So what happens if they ever do I/O then? Like if they need to do some prep work before entering an isolated critical section?
Then instead of going through the per-CPU LRU
buffer_headcache (__find_get_block), isolated CPUs will work as if their per-CPU cache is always empty, going through the slowpath (__find_get_block_slow). The algorithm is:/*
* Perform a pagecache lookup for the matching buffer. If it's there, refresh
* it in the LRU and mark it as accessed. If it is not present then return
* NULL
*/
structbuffer_head*__find_get_block(struct block_device *bdev, sector_t block, unsigned size){ struct buffer_head *bh = lookup_bh_lru(bdev, block, size); if (bh == NULL) { /* __find_get_block_slow will mark the page accessed */ bh = __find_get_block_slow(bdev, block); if (bh) bh_lru_install(bh); } else touch_buffer(bh); return bh; }
Quoted 6 lines
EXPORT_SYMBOL(__find_get_block);I think the performance difference between the per-CPU LRU cache VS
__find_get_block_slowwas much more significant when the cache was introduced. Nowadays its only 26ns (moreover modern filesystems do not use buffer_head's).
Sounds good then!
Thanks!
Quoted 8 lines · Marcelo Tosatti
For certain types of applications (for example PLC software or RAN processing), upon occurrence of an event, it is necessary to complete a certain task in a maximum amount of time (deadline).
One way to express this requirement is with a pair of numbers, deadline time and execution time, where:
[...]
Applied to the vfs.misc branch of the vfs/vfs.git tree. Patches in the vfs.misc branch should appear in linux-next soon.
Please report any outstanding bugs that were missed during review in a new review to the original patch series allowing us to drop it.
It's encouraged to provide Acked-bys and Reviewed-bys even though the patch has now been applied. If possible patch trailers will be updated.
Note that commit hashes shown below are subject to change due to rebase, trailer updates or similar. If in doubt, please check the listed branch.
tree: https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git branch: vfs.misc
[1/1] fs/buffer.c: disable per-CPU buffer_head cache for isolated CPUs
https://git.kernel.org/vfs/vfs/c/9ed7cfdf38b8