Extract bg queue logic out fuse_conn::lock
Subsystems (2)
cover Extract bg queue logic out fuse_conn::lock
Hi, Miklos,
the patchset makes background queue and background-related data be protected with new bg_lock. This allows async reads not take fc->lock in fuse_request_send_background(), and this gives performace-visible increase on out async read-write test (1.7-5 % better). See [6/6] for the test results.
The new lock protects max_background, congestion_threshold, num_background, active_background, bg_queue and blocked fields. See [5/6] for the details.
Patches [1-3/6] are preparations. I've already sent you two of them, and you said at least one is already applied, but since they are yet no in linux-next, I add them to make kbuild test robot happy.
[4/6] is a fix of existing issue. I don't know whether stable needs this fix, so I don't send it in separate. Please, say if it's needed resend it in this way.
The test results is for our proprietary vstorage fuse driver. Feel free to ask me, in case of you want the patchset to be tested on some other fuse driver and with some other test.
Thanks,
Kirill
---
Kirill Tkhai (6):
fuse: Use list_first_entry() in flush_bg_queue()
fuse: Move clear_bit() up in request_end()
fuse: Underline congestion_threshold and max_background may be read w/o fc->lock
fuse: Lock fc->lock during changing num_background and congestion_threshold
fs: Introduce fuse_conn::bg_lock
fuse: Do not take fuse_conn::lock on fuse_request_send_background()
fs/fuse/control.c | 34 ++++++++++++++++++++++++++-------- fs/fuse/dev.c | 42 +++++++++++++++++++++++++++--------------- fs/fuse/file.c | 2 +- fs/fuse/fuse_i.h | 11 +++++++++-- fs/fuse/inode.c | 4 ++++ 5 files changed, 67 insertions(+), 26 deletions(-)
--
PATCH 1/6 fuse: Use list_first_entry() in flush_bg_queue()
This cleanup patch makes the function to use the primitive instead of direct dereferencing.
Also, move fiq dereferencing out of cycle, since it's always constant.
--- fs/fuse/dev.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/fuse/dev.c +3 -2
@@ -353,12 +353,13 @@ void fuse_queue_forget(struct fuse_conn *fc, struct fuse_forget_link *forget,static void flush_bg_queue (struct fuse_conn *fc) { +struct fuse_iqueue *fiq = &fc->iq; +while (fc->active_background < fc->max_background && !list_empty (&fc->bg_queue)) {struct fuse_req *req; -struct fuse_iqueue *fiq = &fc->iq; - req = list_entry(fc->bg_queue.next, struct fuse_req, list); + req = list_first_entry(&fc->bg_queue, struct fuse_req, list);list_del (&req->list); fc->active_background++;spin_lock (&fiq->waitq.lock);
PATCH 2/6 fuse: Move clear_bit() up in request_end()
In reply to Kirill TkhaiNobody can clear FR_BACKGROUND bit on processing
request in parallel, so it's possible to do it
out of fc->lock.
--- fs/fuse/dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fuse/dev.c +1 -1
@@ -390,8 +390,8 @@ static void request_end(struct fuse_conn *fc, struct fuse_req *req)WARN_ON (test_bit (FR_PENDING , &req->flags));WARN_ON (test_bit (FR_SENT , &req->flags));if (test_bit (FR_BACKGROUND , &req->flags)) { -spin_lock (&fc->lock);clear_bit (FR_BACKGROUND , &req->flags); +spin_lock (&fc->lock);if (fc->num_background == fc->max_background) fc->blocked =0 ;
PATCH 3/6 fuse: Underline congestion_threshold and max_background may be read w/o fc->lock
In reply to Kirill TkhaiSince they are of unsigned int type, it's allowed to read them unlocked during reporting to userspace. Let's underline this fact with READ_ONCE() macroses.
--- fs/fuse/control.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/fuse/control.c +2 -2
@@ -107,7 +107,7 @@ static ssize_t fuse_conn_max_background_read(struct file *file,if (!fc)return 0 ; - val = fc->max_background; + val =READ_ONCE (fc->max_background);fuse_conn_put (fc);return fuse_conn_limit_read (file, buf, len, ppos, val);
@@ -144,7 +144,7 @@ static ssize_t fuse_conn_congestion_threshold_read(struct file *file,if (!fc)return 0 ; - val = fc->congestion_threshold; + val =READ_ONCE (fc->congestion_threshold);fuse_conn_put (fc);return fuse_conn_limit_read (file, buf, len, ppos, val);
PATCH 4/6 fuse: Lock fc->lock during changing num_background and congestion_threshold
In reply to Kirill TkhaiFunctions sequences like request_end()->flush_bg_queue() require max_background and congestion_threshold are constant during their execution. Otherwise, checks like
if (fc->num_background == fc->max_background)
made in different time may behave not like expected.
--- fs/fuse/control.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-)
fs/fuse/control.c +24 -6
@@ -125,7 +125,12 @@ static ssize_t fuse_conn_max_background_write(struct file *file,if (ret >0 ) {struct fuse_conn *fc =fuse_ctl_file_conn_get (file);if (fc) { +spin_lock (&fc->lock); fc->max_background = val; + fc->blocked = (fc->num_background >= fc->max_background); +if (!fc->blocked &&waitqueue_active (&fc->blocked_waitq)) +wake_up (&fc->blocked_waitq); +spin_unlock (&fc->lock);fuse_conn_put (fc); } }
@@ -155,18 +160,31 @@ static ssize_t fuse_conn_congestion_threshold_write(struct file *file, size_t count, loff_t *ppos) { unsigned uninitialized_var(val); +struct fuse_conn *fc;ssize_t ret; ret =fuse_conn_limit_write (file, buf, count, ppos, &val, max_user_congthresh); -if (ret >0 ) { -struct fuse_conn *fc =fuse_ctl_file_conn_get (file); -if (fc) { - fc->congestion_threshold = val; -fuse_conn_put (fc); +if (ret <=0 ) + goto out; + fc =fuse_ctl_file_conn_get (file); +if (!fc) + goto out; + +spin_lock (&fc->lock); + fc->congestion_threshold = val; +if (fc->sb) { +if (fc->num_background < fc->congestion_threshold) { +clear_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); +clear_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); + }else { +set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); +set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); } } - +spin_unlock (&fc->lock); +fuse_conn_put (fc); +out:return ret; }
PATCH 5/6 fs: Introduce fuse_conn::bg_lock
In reply to Kirill TkhaiTo reduce contention of fuse_conn::lock, this patch introduces bg_lock for protection of fields related to background queue. These are: max_background, congestion_threshold, num_background, active_background, bg_queue and blocked.
This allows next patch to make async reads not requiring fuse_conn::lock, so async reads and writes will have better performance executed in parallel.
--- fs/fuse/control.c | 8 ++++---- fs/fuse/dev.c | 20 ++++++++++++-------- fs/fuse/file.c | 2 +- fs/fuse/fuse_i.h | 8 ++++++-- fs/fuse/inode.c | 3 +++ 5 files changed, 26 insertions(+), 15 deletions(-)
fs/fuse/control.c +4 -4
@@ -125,12 +125,12 @@ static ssize_t fuse_conn_max_background_write(struct file *file,if (ret >0 ) {struct fuse_conn *fc =fuse_ctl_file_conn_get (file);if (fc) { -spin_lock (&fc->lock); +spin_lock (&fc->bg_lock); fc->max_background = val; fc->blocked = (fc->num_background >= fc->max_background);if (!fc->blocked &&waitqueue_active (&fc->blocked_waitq))wake_up (&fc->blocked_waitq); -spin_unlock (&fc->lock); +spin_unlock (&fc->bg_lock);fuse_conn_put (fc); } }
@@ -171,7 +171,7 @@ static ssize_t fuse_conn_congestion_threshold_write(struct file *file,if (!fc) goto out; -spin_lock (&fc->lock); +spin_lock (&fc->bg_lock); fc->congestion_threshold = val;if (fc->sb) {if (fc->num_background < fc->congestion_threshold) {
@@ -182,7 +182,7 @@ static ssize_t fuse_conn_congestion_threshold_write(struct file *file,set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); } } -spin_unlock (&fc->lock); +spin_unlock (&fc->bg_lock);fuse_conn_put (fc); out:return ret;
fs/fuse/dev.c +12 -8
@@ -287,10 +287,10 @@ void fuse_put_request(struct fuse_conn *fc, struct fuse_req *req)* We get here in the unlikely case that a background * request was allocated but not sent */ -spin_lock (&fc->lock); +spin_lock (&fc->bg_lock);if (!fc->blocked)wake_up (&fc->blocked_waitq); -spin_unlock (&fc->lock); +spin_unlock (&fc->bg_lock); }if (test_bit (FR_WAITING , &req->flags)) {
@@ -391,7 +391,7 @@ static void request_end(struct fuse_conn *fc, struct fuse_req *req)WARN_ON (test_bit (FR_SENT , &req->flags));if (test_bit (FR_BACKGROUND , &req->flags)) {clear_bit (FR_BACKGROUND , &req->flags); -spin_lock (&fc->lock); +spin_lock (&fc->bg_lock);if (fc->num_background == fc->max_background) fc->blocked =0 ;
@@ -406,7 +406,7 @@ static void request_end(struct fuse_conn *fc, struct fuse_req *req) fc->num_background--; fc->active_background--;flush_bg_queue (fc); -spin_unlock (&fc->lock); +spin_unlock (&fc->bg_lock); }wake_up (&req->waitq);if (req->end)
@@ -579,8 +579,8 @@ ssize_t fuse_simple_request(struct fuse_conn *fc, struct fuse_args *args)* * fc->connected must have been checked previously */ -void fuse_request_send_background_locked (struct fuse_conn *fc, -struct fuse_req *req) +void fuse_request_send_background_nocheck (struct fuse_conn *fc, +struct fuse_req *req) {BUG_ON (!test_bit (FR_BACKGROUND , &req->flags));if (!test_bit (FR_WAITING , &req->flags)) {
@@ -588,6 +588,7 @@ void fuse_request_send_background_locked(struct fuse_conn *fc,atomic_inc (&fc->num_waiting); }__set_bit (FR_ISREPLY , &req->flags); +spin_lock (&fc->bg_lock); fc->num_background++;if (fc->num_background == fc->max_background) fc->blocked =1 ;
@@ -597,6 +598,7 @@ void fuse_request_send_background_locked(struct fuse_conn *fc, }list_add_tail (&req->list, &fc->bg_queue);flush_bg_queue (fc); +spin_unlock (&fc->bg_lock); }void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req)
@@ -604,7 +606,7 @@ void fuse_request_send_background(struct fuse_conn *fc, struct fuse_req *req)BUG_ON (!req->end);spin_lock (&fc->lock);if (fc->connected) { -fuse_request_send_background_locked (fc, req); +fuse_request_send_background_nocheck (fc, req);spin_unlock (&fc->lock); }else {spin_unlock (&fc->lock);
@@ -2105,7 +2107,6 @@ void fuse_abort_conn(struct fuse_conn *fc, bool is_abort)LIST_HEAD (to_end); fc->connected =0 ; - fc->blocked =0 ; fc->aborted = is_abort;fuse_set_initialized (fc);list_for_each_entry (fud, &fc->devices, entry) {
@@ -2127,8 +2128,11 @@ void fuse_abort_conn(struct fuse_conn *fc, bool is_abort)list_splice_tail_init (&fpq->processing, &to_end);spin_unlock (&fpq->lock); } +spin_lock (&fc->bg_lock); + fc->blocked =0 ; fc->max_background =UINT_MAX ;flush_bg_queue (fc); +spin_unlock (&fc->bg_lock);spin_lock (&fiq->waitq.lock); fiq->connected =0 ;
fs/fuse/file.c +1 -1
@@ -1502,7 +1502,7 @@ __acquires(fc->lock) req->in.args[1 ].size = inarg->size; fi->writectr++; -fuse_request_send_background_locked (fc, req); +fuse_request_send_background_nocheck (fc, req);return ; out_free:
fs/fuse/fuse_i.h +6 -2
@@ -500,6 +500,10 @@ struct fuse_conn {/** The list of background requests set aside for later queuing */ struct list_head bg_queue; +/** Protects: max_background, congestion_threshold, num_background, +* active_background, bg_queue, blocked */ +spinlock_t bg_lock; +/** Flag indicating that INIT reply has been received. Allocating * any fuse request will be suspended until the flag is set */ int initialized;
@@ -857,8 +861,8 @@ ssize_t fuse_simple_request(struct fuse_conn *fc, struct fuse_args *args);*/ void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req); -void fuse_request_send_background_locked (struct fuse_conn *fc, -struct fuse_req *req); +void fuse_request_send_background_nocheck (struct fuse_conn *fc, +struct fuse_req *req);/* Abort all requests */ void fuse_abort_conn (struct fuse_conn *fc,bool is_abort);
fs/fuse/inode.c +3 -0
@@ -606,6 +606,7 @@ void fuse_conn_init(struct fuse_conn *fc, struct user_namespace *user_ns) {memset (fc,0 ,sizeof (*fc));spin_lock_init (&fc->lock); +spin_lock_init (&fc->bg_lock);init_rwsem (&fc->killsb);refcount_set (&fc->count,1 );atomic_set (&fc->dev_count,1 );
@@ -853,6 +854,7 @@ static void process_init_limits(struct fuse_conn *fc, struct fuse_init_out *arg)sanitize_global_limit (&max_user_bgreq);sanitize_global_limit (&max_user_congthresh); +spin_lock (&fc->bg_lock);if (arg->max_background) { fc->max_background = arg->max_background;
@@ -866,6 +868,7 @@ static void process_init_limits(struct fuse_conn *fc, struct fuse_init_out *arg) fc->congestion_threshold > max_user_congthresh) fc->congestion_threshold = max_user_congthresh; } +spin_unlock (&fc->bg_lock); }static void process_init_reply (struct fuse_conn *fc,struct fuse_req *req)
PATCH 6/6 fuse: Do not take fuse_conn::lock on fuse_request_send_background()
In reply to Kirill TkhaiCurrently, we take fc->lock there only to check for fc->connected. But this flag is changed only on connection abort, which is very rare operation. Good thing looks to make fuse_request_send_background() faster, while fuse_abort_conn() slowler.
So, we make fuse_request_send_background() lockless and mark (fc->connected == 1) region as RCU-protected. Abort function just uses synchronize_sched() to wait till all pending background requests is being queued, and then makes ordinary abort.
Note, that synchronize_sched() is used instead of synchronize_rcu(), since we want to check for fc->connected without rcu_dereference() in fuse_request_send_background() (i.e., not to add memory barriers to this hot path).
Also, note fuse_conn::aborting field is introduced. It's aimed to synchronize two sequential aborts executing in parallel:
we do not want the secondly called abort finishes before the first one is done, after it sees fc->connected has already became 0.
Our storage test shows performance increase on parallel read and write with the patchset (7 test runs, 300 seconds of execution of each, aio):
./io_iops --read --write --iops -u 48g -s 4k -p 24 -n 24 --aio -q 128 -t 300 -f /mnt/vstorage/file
Before (iops):
25721.58203 (worst)
26179.43359
26092.58594
25789.96484
26238.63477 (best)
25985.88867
25987.66406
After (iops):
26852.27539
27547.60742
26322.29688
27643.33398 (best)
26753.10547
26157.91016 (worst)
26694.63477
All *after* runs are better than any *before* run, except of one. The best *after* run is 5% better than the best *before* run. The worst *after* run is 1.7% better then the worst *before* run.
--- fs/fuse/dev.c | 17 ++++++++++++----- fs/fuse/fuse_i.h | 3 +++ fs/fuse/inode.c | 1 + 3 files changed, 16 insertions(+), 5 deletions(-)
fs/fuse/dev.c +12 -5
@@ -575,8 +575,6 @@ ssize_t fuse_simple_request(struct fuse_conn *fc, struct fuse_args *args) }/* -* Called under fc->lock -* * fc->connected must have been checked previously */ void fuse_request_send_background_nocheck(struct fuse_conn *fc,
@@ -604,12 +602,12 @@ void fuse_request_send_background_nocheck(struct fuse_conn *fc,void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req) {BUG_ON (!req->end); -spin_lock (&fc->lock); +rcu_read_lock_sched ();if (fc->connected) {fuse_request_send_background_nocheck (fc, req); -spin_unlock (&fc->lock); +rcu_read_unlock_sched (); }else { -spin_unlock (&fc->lock); +rcu_read_unlock_sched (); req->out.h.error = -ENOTCONN ; req->end (fc, req);fuse_put_request (fc, req);
@@ -2107,6 +2105,13 @@ void fuse_abort_conn(struct fuse_conn *fc, bool is_abort)LIST_HEAD (to_end); fc->connected =0 ; + fc->aborting = true; +spin_unlock (&fc->lock); + +/* Propagate fc->connected */ +synchronize_sched (); + +spin_lock (&fc->lock); fc->aborted = is_abort;fuse_set_initialized (fc);list_for_each_entry (fud, &fc->devices, entry) {
@@ -2145,12 +2150,14 @@ void fuse_abort_conn(struct fuse_conn *fc, bool is_abort)spin_unlock (&fiq->waitq.lock);kill_fasync (&fiq->fasync,SIGIO ,POLL_IN );end_polls (fc); + fc->aborting = false;wake_up_all (&fc->blocked_waitq);spin_unlock (&fc->lock);end_requests (fc, &to_end); }else {spin_unlock (&fc->lock); +wait_event (fc->blocked_waitq, !fc->aborting); } }EXPORT_SYMBOL_GPL (fuse_abort_conn);
fs/fuse/fuse_i.h +3 -0
@@ -523,6 +523,9 @@ struct fuse_conn {abort and device release */ unsigned connected; +/** Connection is now aborting */ +bool aborting; +/** Connection aborted via sysfs */ bool aborted;
fs/fuse/inode.c +1 -0
@@ -624,6 +624,7 @@ void fuse_conn_init(struct fuse_conn *fc, struct user_namespace *user_ns) fc->blocked =0 ; fc->initialized =0 ; fc->connected =1 ; + fc->aborting = false; fc->attr_version =1 ;get_random_bytes (&fc->scramble_key,sizeof (fc->scramble_key)); fc->pid_ns =get_pid_ns (task_active_pid_ns (current));
Miklos Szeredi
maintainerRe: [PATCH 6/6] fuse: Do not take fuse_conn::lock on fuse_request_send_background()
Quoted 10 lines · Kirill Tkhai
Currently, we take
fc->lockthere only to check forfc->connected. But this flag is changed only on connection abort, which is very rare operation. Good thing looks to makefuse_request_send_background()faster, whilefuse_abort_conn()slowler.So, we make
fuse_request_send_background()lockless and mark (fc->connected== 1) region as RCU-protected. Abort function just usessynchronize_sched()to wait till all pending background requests is being queued, and then makes ordinary abort.
Quoted 4 lines
Note, that
synchronize_sched()is used instead ofsynchronize_rcu(), since we want to check forfc->connectedwithoutrcu_dereference()infuse_request_send_background()(i.e., not to add memory barriers to this hot path).
Apart from the inaccuracies in the above (_sched variant is for scheduling and NMI taking code; _sched variant requires rcu_dereference() as well; rcu_dereference() does not add barriers; rcu_dereference() is only for pointers, so we can't use it for an integer), wouldn't it be simpler to just use bg_lock for checking ->connected, and lock bg_lock (as well as fc->lock) when setting ->connected?
Updated patch below (untested).
Thanks,
Miklos
---
Subject: fuse: do not take fc->lock in fuse_request_send_background()
From: Kirill Tkhai <[email protected]>
Date: Mon, 27 Aug 2018 18:29:56 +0300
Currently, we take fc->lock there only to check for fc->connected. But this flag is changed only on connection abort, which is very rare operation.
--- fs/fuse/dev.c | 46 +++++++++++++++++++++++----------------------- fs/fuse/file.c | 4 +++- fs/fuse/fuse_i.h | 4 +--- 3 files changed, 27 insertions(+), 27 deletions(-)
fs/fuse/dev.c +23 -23
@@ -574,42 +574,38 @@ ssize_t fuse_simple_request(struct fuse_return ret; } -/* -* Called under fc->lock -* -* fc->connected must have been checked previously -*/ -void fuse_request_send_background_nocheck (struct fuse_conn *fc, -struct fuse_req *req) +bool fuse_request_queue_background (struct fuse_conn *fc,struct fuse_req *req) { -BUG_ON (!test_bit (FR_BACKGROUND , &req->flags)); +bool queued = false; + +WARN_ON (!test_bit (FR_BACKGROUND , &req->flags));if (!test_bit (FR_WAITING , &req->flags)) {__set_bit (FR_WAITING , &req->flags);atomic_inc (&fc->num_waiting); }__set_bit (FR_ISREPLY , &req->flags);spin_lock (&fc->bg_lock); - fc->num_background++; -if (fc->num_background == fc->max_background) - fc->blocked =1 ; -if (fc->num_background == fc->congestion_threshold && fc->sb) { -set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); -set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); +if (likely (fc->connected)) { + fc->num_background++; +if (fc->num_background == fc->max_background) + fc->blocked =1 ; +if (fc->num_background == fc->congestion_threshold && fc->sb) { +set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); +set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); + } +list_add_tail (&req->list, &fc->bg_queue); +flush_bg_queue (fc); + queued = true; } -list_add_tail (&req->list, &fc->bg_queue); -flush_bg_queue (fc);spin_unlock (&fc->bg_lock); + +return queued; }void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req) { -BUG_ON (!req->end); -spin_lock (&fc->lock); -if (fc->connected) { -fuse_request_send_background_nocheck (fc, req); -spin_unlock (&fc->lock); - }else { -spin_unlock (&fc->lock); +WARN_ON (!req->end); +if (!fuse_request_queue_background (fc, req)) { req->out.h.error = -ENOTCONN ; req->end (fc, req);fuse_put_request (fc, req);
@@ -2112,7 +2108,11 @@ void fuse_abort_conn(struct fuse_conn *fstruct fuse_req *req, *next;LIST_HEAD (to_end); +/* Background queuing checks fc->connected under bg_lock */ +spin_lock (&fc->bg_lock); fc->connected =0 ; +spin_unlock (&fc->bg_lock); + fc->aborted = is_abort;fuse_set_initialized (fc);list_for_each_entry (fud, &fc->devices, entry) {
fs/fuse/fuse_i.h +1 -3
@@ -863,9 +863,7 @@ ssize_t fuse_simple_request(struct fuse_* Send a request in the background */ void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req); - -void fuse_request_send_background_nocheck (struct fuse_conn *fc, -struct fuse_req *req); +bool fuse_request_queue_background (struct fuse_conn *fc,struct fuse_req *req);/* Abort all requests */ void fuse_abort_conn (struct fuse_conn *fc,bool is_abort);
fs/fuse/file.c +3 -1
@@ -1487,6 +1487,7 @@ __acquires(fc->lock)struct fuse_inode *fi =get_fuse_inode (req->inode);struct fuse_write_in *inarg = &req->misc.write.in;__u64 data_size = req->num_pages *PAGE_SIZE ; +bool queued;if (!fc->connected) goto out_free;
@@ -1502,7 +1503,8 @@ __acquires(fc->lock) req->in.args[1 ].size = inarg->size; fi->writectr++; -fuse_request_send_background_nocheck (fc, req); + queued =fuse_request_queue_background (fc, req); +WARN_ON (!queued);return ; out_free:
Re: [PATCH 6/6] fuse: Do not take fuse_conn::lock on fuse_request_send_background()
Quoted 16 lines · Miklos Szeredi
Quoted 10 lines · Kirill Tkhai
Currently, we take
fc->lockthere only to check forfc->connected. But this flag is changed only on connection abort, which is very rare operation. Good thing looks to makefuse_request_send_background()faster, whilefuse_abort_conn()slowler.So, we make
fuse_request_send_background()lockless and mark (fc->connected== 1) region as RCU-protected. Abort function just usessynchronize_sched()to wait till all pending background requests is being queued, and then makes ordinary abort.Quoted 4 lines
Note, that
synchronize_sched()is used instead ofsynchronize_rcu(), since we want to check forfc->connectedwithoutrcu_dereference()infuse_request_send_background()(i.e., not to add memory barriers to this hot path).
Quoted 4 lines
Apart from the inaccuracies in the above (
_schedvariant is for scheduling and NMI taking code;_schedvariant requiresrcu_dereference()as well;rcu_dereference()does not add barriers;rcu_dereference()is only for pointers, so we can't use it for an integer),
Writing this I was inspired by expand_fdtable(). Yes, the description confuses, and we don't need rcu_dereference() since we do not touch memory pointed by __rcu pointer, we have no pointer at all. synchronize_sched() guarantees:
On systems with more than one CPU, when synchronize_sched() returns, each CPU is guaranteed to have executed a full memory barrier since the end of its last RCU-sched read-side critical section whose beginning preceded the call to synchronize_sched().
(and rcu_dereference() unfolds in smp_read_barrier_depends(), which I mean as
added barriers)
But it does not so matter. I'm OK with the patch you updated.
Quoted 5 lines
wouldn't it be simpler to just use
bg_lock
for checking ->connected, and lockbg_lock(as well asfc->lock) when setting ->connected?Updated patch below (untested).
Tested it. Works for me.
Thanks,
Kirill
Quoted 128 lines
---
Subject: fuse: do not takefc->lockinfuse_request_send_background()
From: Kirill Tkhai <[email protected]>
Date: Mon, 27 Aug 2018 18:29:56 +0300Currently, we take
fc->lockthere only to check forfc->connected. But this flag is changed only on connection abort, which is very rare operation.Signed-off-by Kirill Tkhai <[email protected]>Signed-off-by Miklos Szeredi <[email protected]>--- fs/fuse/dev.c | 46 +++++++++++++++++++++++----------------------- fs/fuse/file.c | 4 +++- fs/fuse/fuse_i.h | 4 +--- 3 files changed, 27 insertions(+), 27 deletions(-)fs/fuse/dev.c +23
-23@@ -574,42 +574,38 @@ ssize_t fuse_simple_request(struct fuse_return ret; } -/* -* Called under fc->lock -* -* fc->connected must have been checked previously -*/ -void fuse_request_send_background_nocheck (struct fuse_conn *fc, -struct fuse_req *req) +bool fuse_request_queue_background (struct fuse_conn *fc,struct fuse_req *req) { -BUG_ON (!test_bit (FR_BACKGROUND , &req->flags)); +bool queued = false; + +WARN_ON (!test_bit (FR_BACKGROUND , &req->flags));if (!test_bit (FR_WAITING , &req->flags)) {__set_bit (FR_WAITING , &req->flags);atomic_inc (&fc->num_waiting); }__set_bit (FR_ISREPLY , &req->flags);spin_lock (&fc->bg_lock); - fc->num_background++; -if (fc->num_background == fc->max_background) - fc->blocked =1 ; -if (fc->num_background == fc->congestion_threshold && fc->sb) { -set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); -set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); +if (likely (fc->connected)) { + fc->num_background++; +if (fc->num_background == fc->max_background) + fc->blocked =1 ; +if (fc->num_background == fc->congestion_threshold && fc->sb) { +set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); +set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); + } +list_add_tail (&req->list, &fc->bg_queue); +flush_bg_queue (fc); + queued = true; } -list_add_tail (&req->list, &fc->bg_queue); -flush_bg_queue (fc);spin_unlock (&fc->bg_lock); + +return queued; }void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req) { -BUG_ON (!req->end); -spin_lock (&fc->lock); -if (fc->connected) { -fuse_request_send_background_nocheck (fc, req); -spin_unlock (&fc->lock); - }else { -spin_unlock (&fc->lock); +WARN_ON (!req->end); +if (!fuse_request_queue_background (fc, req)) { req->out.h.error = -ENOTCONN ; req->end (fc, req);fuse_put_request (fc, req);@@ -2112,7 +2108,11 @@ void fuse_abort_conn(struct fuse_conn *fstruct fuse_req *req, *next;LIST_HEAD (to_end); +/* Background queuing checks fc->connected under bg_lock */ +spin_lock (&fc->bg_lock); fc->connected =0 ; +spin_unlock (&fc->bg_lock); + fc->aborted = is_abort;fuse_set_initialized (fc);list_for_each_entry (fud, &fc->devices, entry) {fs/fuse/fuse_i.h +1
-3@@ -863,9 +863,7 @@ ssize_t fuse_simple_request(struct fuse_* Send a request in the background */ void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req); - -void fuse_request_send_background_nocheck (struct fuse_conn *fc, -struct fuse_req *req); +bool fuse_request_queue_background (struct fuse_conn *fc,struct fuse_req *req);/* Abort all requests */ void fuse_abort_conn (struct fuse_conn *fc,bool is_abort);fs/fuse/file.c +3
-1@@ -1487,6 +1487,7 @@ __acquires(fc->lock)struct fuse_inode *fi =get_fuse_inode (req->inode);struct fuse_write_in *inarg = &req->misc.write.in;__u64 data_size = req->num_pages *PAGE_SIZE ; +bool queued;if (!fc->connected) goto out_free;@@ -1502,7 +1503,8 @@ __acquires(fc->lock) req->in.args[1 ].size = inarg->size; fi->writectr++; -fuse_request_send_background_nocheck (fc, req); + queued =fuse_request_queue_background (fc, req); +WARN_ON (!queued);return ; out_free:
Re: [PATCH 6/6] fuse: Do not take fuse_conn::lock on fuse_request_send_background()
Hi, Miklos,
should I resend the series with the patch you changed, or you are already taken it since there is your SoB?
Thanks,
Kirill
Quoted 175 lines · Kirill Tkhai
Quoted 16 lines · Miklos Szeredi
Quoted 10 lines · Kirill Tkhai
Currently, we take
fc->lockthere only to check forfc->connected. But this flag is changed only on connection abort, which is very rare operation. Good thing looks to makefuse_request_send_background()faster, whilefuse_abort_conn()slowler.So, we make
fuse_request_send_background()lockless and mark (fc->connected== 1) region as RCU-protected. Abort function just usessynchronize_sched()to wait till all pending background requests is being queued, and then makes ordinary abort.Quoted 4 lines
Note, that
synchronize_sched()is used instead ofsynchronize_rcu(), since we want to check forfc->connectedwithoutrcu_dereference()infuse_request_send_background()(i.e., not to add memory barriers to this hot path).Quoted 4 lines
Apart from the inaccuracies in the above (
_schedvariant is for scheduling and NMI taking code;_schedvariant requiresrcu_dereference()as well;rcu_dereference()does not add barriers;rcu_dereference()is only for pointers, so we can't use it for an integer),Writing this I was inspired by
expand_fdtable(). Yes, the description confuses, and we don't needrcu_dereference()since we do not touch memory pointed by__rcupointer, we have no pointer at all.synchronize_sched()guarantees:On systems with more than one CPU, when
synchronize_sched()returns, each CPU is guaranteed to have executed a full memory barrier since the end of its last RCU-sched read-side critical section whose beginning preceded the call tosynchronize_sched().(and
rcu_dereference()unfolds insmp_read_barrier_depends(), which I mean as
added barriers)But it does not so matter. I'm OK with the patch you updated.
Quoted 5 lines
wouldn't it be simpler to just use
bg_lock
for checking ->connected, and lockbg_lock(as well asfc->lock) when setting ->connected?Updated patch below (untested).
Tested it. Works for me.
Thanks,
KirillQuoted 128 lines
---
Subject: fuse: do not takefc->lockinfuse_request_send_background()
From: Kirill Tkhai <[email protected]>
Date: Mon, 27 Aug 2018 18:29:56 +0300Currently, we take
fc->lockthere only to check forfc->connected. But this flag is changed only on connection abort, which is very rare operation.Signed-off-by Kirill Tkhai <[email protected]>Signed-off-by Miklos Szeredi <[email protected]>--- fs/fuse/dev.c | 46 +++++++++++++++++++++++----------------------- fs/fuse/file.c | 4 +++- fs/fuse/fuse_i.h | 4 +--- 3 files changed, 27 insertions(+), 27 deletions(-)fs/fuse/dev.c +23
-23@@ -574,42 +574,38 @@ ssize_t fuse_simple_request(struct fuse_return ret; } -/* -* Called under fc->lock -* -* fc->connected must have been checked previously -*/ -void fuse_request_send_background_nocheck (struct fuse_conn *fc, -struct fuse_req *req) +bool fuse_request_queue_background (struct fuse_conn *fc,struct fuse_req *req) { -BUG_ON (!test_bit (FR_BACKGROUND , &req->flags)); +bool queued = false; + +WARN_ON (!test_bit (FR_BACKGROUND , &req->flags));if (!test_bit (FR_WAITING , &req->flags)) {__set_bit (FR_WAITING , &req->flags);atomic_inc (&fc->num_waiting); }__set_bit (FR_ISREPLY , &req->flags);spin_lock (&fc->bg_lock); - fc->num_background++; -if (fc->num_background == fc->max_background) - fc->blocked =1 ; -if (fc->num_background == fc->congestion_threshold && fc->sb) { -set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); -set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); +if (likely (fc->connected)) { + fc->num_background++; +if (fc->num_background == fc->max_background) + fc->blocked =1 ; +if (fc->num_background == fc->congestion_threshold && fc->sb) { +set_bdi_congested (fc->sb->s_bdi,BLK_RW_SYNC ); +set_bdi_congested (fc->sb->s_bdi,BLK_RW_ASYNC ); + } +list_add_tail (&req->list, &fc->bg_queue); +flush_bg_queue (fc); + queued = true; } -list_add_tail (&req->list, &fc->bg_queue); -flush_bg_queue (fc);spin_unlock (&fc->bg_lock); + +return queued; }void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req) { -BUG_ON (!req->end); -spin_lock (&fc->lock); -if (fc->connected) { -fuse_request_send_background_nocheck (fc, req); -spin_unlock (&fc->lock); - }else { -spin_unlock (&fc->lock); +WARN_ON (!req->end); +if (!fuse_request_queue_background (fc, req)) { req->out.h.error = -ENOTCONN ; req->end (fc, req);fuse_put_request (fc, req);@@ -2112,7 +2108,11 @@ void fuse_abort_conn(struct fuse_conn *fstruct fuse_req *req, *next;LIST_HEAD (to_end); +/* Background queuing checks fc->connected under bg_lock */ +spin_lock (&fc->bg_lock); fc->connected =0 ; +spin_unlock (&fc->bg_lock); + fc->aborted = is_abort;fuse_set_initialized (fc);list_for_each_entry (fud, &fc->devices, entry) {fs/fuse/fuse_i.h +1
-3@@ -863,9 +863,7 @@ ssize_t fuse_simple_request(struct fuse_* Send a request in the background */ void fuse_request_send_background (struct fuse_conn *fc,struct fuse_req *req); - -void fuse_request_send_background_nocheck (struct fuse_conn *fc, -struct fuse_req *req); +bool fuse_request_queue_background (struct fuse_conn *fc,struct fuse_req *req);/* Abort all requests */ void fuse_abort_conn (struct fuse_conn *fc,bool is_abort);fs/fuse/file.c +3
-1@@ -1487,6 +1487,7 @@ __acquires(fc->lock)struct fuse_inode *fi =get_fuse_inode (req->inode);struct fuse_write_in *inarg = &req->misc.write.in;__u64 data_size = req->num_pages *PAGE_SIZE ; +bool queued;if (!fc->connected) goto out_free;@@ -1502,7 +1503,8 @@ __acquires(fc->lock) req->in.args[1 ].size = inarg->size; fi->writectr++; -fuse_request_send_background_nocheck (fc, req); + queued =fuse_request_queue_background (fc, req); +WARN_ON (!queued);return ; out_free:
Miklos Szeredi
maintainerRe: [PATCH 6/6] fuse: Do not take fuse_conn::lock on fuse_request_send_background()
Quoted 4 lines · Kirill Tkhai
Hi, Miklos,
should I resend the series with the patch you changed, or you are already taken it since there is your SoB?
I already took that series.
Thanks,
Miklos
Hi, Miklos,
Have you seen this? Some reaction on this?
Thanks,
Kirill
Quoted 48 lines · Kirill Tkhai
Hi, Miklos,
the patchset makes background queue and background-related data be protected with new
bg_lock. This allows async reads not takefc->lockinfuse_request_send_background(), and this gives performace-visible increase on out async read-write test (1.7-5 % better). See [6/6] for the test results.The new lock protects
max_background,congestion_threshold,num_background,active_background,bg_queueand blocked fields. See [5/6] for the details.Patches [1-3/6] are preparations. I've already sent you two of them, and you said at least one is already applied, but since they are yet no in linux-next, I add them to make kbuild test robot happy.
[4/6] is a fix of existing issue. I don't know whether stable needs this fix, so I don't send it in separate. Please, say if it's needed resend it in this way.
The test results is for our proprietary vstorage fuse driver. Feel free to ask me, in case of you want the patchset to be tested on some other fuse driver and with some other test.
Thanks,
Kirill
---Kirill Tkhai (6):
fuse: Uselist_first_entry()inflush_bg_queue()
fuse: Moveclear_bit()up inrequest_end()
fuse: Underlinecongestion_thresholdandmax_backgroundmay be read w/ofc->lock
fuse: Lockfc->lockduring changingnum_backgroundandcongestion_threshold
fs: Introducefuse_conn::bg_lock
fuse: Do not takefuse_conn::lockonfuse_request_send_background()fs/fuse/control.c | 34 ++++++++++++++++++++++++++-------- fs/fuse/dev.c | 42 +++++++++++++++++++++++++++--------------- fs/fuse/file.c | 2 +- fs/fuse/fuse_i.h | 11 +++++++++-- fs/fuse/inode.c | 4 ++++ 5 files changed, 67 insertions(+), 26 deletions(-)--
Signed-off-by Kirill Tkhai <[email protected]>
Miklos Szeredi
maintainerQuoted 3 lines · Kirill Tkhai
Hi, Miklos,
Have you seen this? Some reaction on this?
Applied and pushed to
git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git#for-next
Thanks,
Miklos