fuse: add kernel-enforced request timeout option
Subsystems (3)
v8 cover fuse: add kernel-enforced request timeout option
There are situations where fuse servers can become unresponsive or stuck, for example if the server is in a deadlock. Currently, there's no good way to detect if a server is stuck and needs to be killed manually.
This patchset adds a timeout option where if the server does not reply to a request by the time the timeout elapses, the connection will be aborted. This patchset also adds two dynamically configurable fuse sysctls "default_request_timeout" and "max_request_timeout" for controlling/enforcing timeout behavior system-wide.
Existing systems running fuse servers will not be affected unless they explicitly opt into the timeout.
v7:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v7 -> v8:
* Use existing lists for checking expirations (Miklos)
v6:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v6 -> v7:
- Make timer per-connection instead of per-request (Miklos)
- Make default granularity of time minutes instead of seconds
- Removed the reviewed-bys since the interface of this has changed (now minutes, instead of seconds)
v5:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v5 -> v6:
- Gate sysctl.o behind CONFIG_SYSCTL in makefile (kernel test robot)
- Reword/clarify last sentence in cover letter (Miklos)
v4:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v4 -> v5:
- Change timeout behavior from aborting request to aborting connection (Miklos)
- Clarify wording for sysctl documentation (Jingbo)
v3:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v3 -> v4:
- Fix wording on some comments to make it more clear
- Use simpler logic for timer (eg remove extra if checks, use mod timer API) (Josef)
- Sanity-check should be on FR_FINISHING not FR_FINISHED (Jingbo)
- Fix comment for "processing queue", add req->fpq = NULL safeguard (Bernd)
v2:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v2 -> v3:
- Disarm / rearm timer in dev_do_read to handle race conditions (Bernrd)
- Disarm timer in error handling for fatal interrupt (Yafang)
- Clean up do_fuse_request_end (Jingbo)
- Add timer for notify retrieve requests
- Fix kernel test robot errors for #define no-op functions
v1:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v1 -> v2:
- Add timeout for background requests
- Handle resend race condition
- Add sysctls
Joanne Koong (3):
fs_parser: add fsparam_u16 helper
fuse: add optional kernel-enforced timeout for requests
fuse: add default_request_timeout and max_request_timeout sysctls
Documentation/admin-guide/sysctl/fs.rst | 27 +++++++++ fs/fs_parser.c | 14 +++++ fs/fuse/dev.c | 80 +++++++++++++++++++++++++ fs/fuse/fuse_i.h | 31 ++++++++++ fs/fuse/inode.c | 33 ++++++++++ fs/fuse/sysctl.c | 20 +++++++ include/linux/fs_parser.h | 9 ++- 7 files changed, 211 insertions(+), 3 deletions(-)
Signature · 1 line
2.43.5
PATCH v8 1/3 fs_parser: add fsparam_u16 helper
Add a fsparam helper for unsigned 16 bit values.
--- fs/fs_parser.c | 14 ++++++++++++++ include/linux/fs_parser.h | 9 ++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)
fs/fs_parser.c +14 -0
@@ -210,6 +210,20 @@ int fs_param_is_bool(struct p_log *log, const struct fs_parameter_spec *p, }EXPORT_SYMBOL (fs_param_is_bool); +int fs_param_is_u16 (struct p_log *log,const struct fs_parameter_spec *p, +struct fs_parameter *param,struct fs_parse_result *result) +{ +int base = (unsigned long )p->data; +if (param->type != fs_value_is_string) +return fs_param_bad_value (log, param); +if (!*param->string && (p->flags & fs_param_can_be_empty)) +return 0 ; +if (kstrtou16 (param->string, base, &result->uint_16) <0 ) +return fs_param_bad_value (log, param); +return 0 ; +} +EXPORT_SYMBOL (fs_param_is_u16); +int fs_param_is_u32 (struct p_log *log,const struct fs_parameter_spec *p,struct fs_parameter *param,struct fs_parse_result *result) {
include/linux/fs_parser.h +6 -3
@@ -26,9 +26,10 @@ typedef int fs_param_type(struct p_log *,/* * The type of parameter expected. */ -fs_param_type fs_param_is_bool, fs_param_is_u32, fs_param_is_s32, fs_param_is_u64, - fs_param_is_enum, fs_param_is_string, fs_param_is_blob, fs_param_is_blockdev, - fs_param_is_path, fs_param_is_fd, fs_param_is_uid, fs_param_is_gid; +fs_param_type fs_param_is_bool, fs_param_is_u16, fs_param_is_u32, fs_param_is_s32, + fs_param_is_u64, fs_param_is_enum, fs_param_is_string, fs_param_is_blob, + fs_param_is_blockdev, fs_param_is_path, fs_param_is_fd, fs_param_is_uid, + fs_param_is_gid;/* * Specification of the type of value a parameter wants.
@@ -55,6 +56,7 @@ struct fs_parse_result {union {bool boolean;/* For spec_bool */ int int_32;/* For spec_s32/spec_enum */ +u16 uint_16;/* For spec_u16 */ unsigned int uint_32;/* For spec_u32{,_octal,_hex}/spec_enum */ u64 uint_64;/* For spec_u64 */ kuid_t uid;
@@ -119,6 +121,7 @@ static inline bool fs_validate_description(const char *name,#define fsparam_flag_no (NAME ,OPT ) \ __fsparam(NULL, NAME, OPT, fs_param_neg_with_no, NULL)#define fsparam_bool (NAME ,OPT ) __fsparam(fs_param_is_bool, NAME, OPT, 0, NULL) +#define fsparam_u16 (NAME ,OPT ) __fsparam(fs_param_is_u16, NAME, OPT, 0, NULL)#define fsparam_u32 (NAME ,OPT ) __fsparam(fs_param_is_u32, NAME, OPT, 0, NULL)#define fsparam_u32oct (NAME ,OPT ) \ __fsparam(fs_param_is_u32, NAME, OPT, 0, (void *)8)
Signature · 1 line
2.43.5
Re: [PATCH v8 1/3] fs_parser: add fsparam_u16 helper
Quoted 52 lines · Joanne Koong
Add a fsparam helper for unsigned 16 bit values.
Signed-off-by Joanne Koong <[email protected]>--- fs/fs_parser.c | 14 ++++++++++++++ include/linux/fs_parser.h | 9 ++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)fs/fs_parser.c +14
-0@@ -210,6 +210,20 @@ int fs_param_is_bool(struct p_log *log, const struct fs_parameter_spec *p, }EXPORT_SYMBOL (fs_param_is_bool); +int fs_param_is_u16 (struct p_log *log,const struct fs_parameter_spec *p, +struct fs_parameter *param,struct fs_parse_result *result) +{ +int base = (unsigned long )p->data; +if (param->type != fs_value_is_string) +return fs_param_bad_value (log, param); +if (!*param->string && (p->flags & fs_param_can_be_empty)) +return 0 ; +if (kstrtou16 (param->string, base, &result->uint_16) <0 ) +return fs_param_bad_value (log, param); +return 0 ; +} +EXPORT_SYMBOL (fs_param_is_u16); +int fs_param_is_u32 (struct p_log *log,const struct fs_parameter_spec *p,struct fs_parameter *param,struct fs_parse_result *result) {include/linux/fs_parser.h +4
-3@@ -26,9 +26,10 @@ typedef int fs_param_type(struct p_log *,/* * The type of parameter expected. */ -fs_param_type fs_param_is_bool, fs_param_is_u32, fs_param_is_s32, fs_param_is_u64, - fs_param_is_enum, fs_param_is_string, fs_param_is_blob, fs_param_is_blockdev, - fs_param_is_path, fs_param_is_fd, fs_param_is_uid, fs_param_is_gid; +fs_param_type fs_param_is_bool, fs_param_is_u16, fs_param_is_u32, fs_param_is_s32, + fs_param_is_u64, fs_param_is_enum, fs_param_is_string, fs_param_is_blob, + fs_param_is_blockdev, fs_param_is_path, fs_param_is_fd, fs_param_is_uid, + fs_param_is_gid;/* * Specification of the type of value a parameter wants.
include/linux/fs_parser.h struct fs_parse_result { · line 60
Quoted 5 lines
include/linux/fs_parser.h +1
-0@@ -55,6 +56,7 @@ struct fs_parse_result {union {bool boolean;/* For spec_bool */ int int_32;/* For spec_s32/spec_enum */ +u16 uint_16;/* For spec_u16 * unsigned int uint_32; /* For spec_u32{,_octal,_hex}/spec_enum */
Given fs_param_is_u16() has "int base = (unsigned long)p->data;", shouldn't the comment also mention ",_octal,_hex}/spec_enum"? Or should the function get slightly modified to always use a base of 10? Or 0 with auto detection as for fs_param_is_u64()?
include/linux/fs_parser.h static inline bool fs_validate_description(const char *name, · line 127
Quoted 10 lines
include/linux/fs_parser.h +1
-0u64 uint_64;/* For spec_u64 */ kuid_t uid;@@ -119,6 +121,7 @@ static inline bool fs_validate_description(const char *name,#define fsparam_flag_no (NAME ,OPT ) \ __fsparam(NULL, NAME, OPT, fs_param_neg_with_no, NULL)#define fsparam_bool (NAME ,OPT ) __fsparam(fs_param_is_bool, NAME, OPT, 0, NULL) +#define fsparam_u16 (NAME ,OPT ) __fsparam(fs_param_is_u16, NAME, OPT, 0, NULL)#define fsparam_u32 (NAME ,OPT ) __fsparam(fs_param_is_u32, NAME, OPT, 0, NULL)#define fsparam_u32oct (NAME ,OPT ) \ __fsparam(fs_param_is_u32, NAME, OPT, 0, (void *)8)
Re: [PATCH v8 1/3] fs_parser: add fsparam_u16 helper
Quoted 53 lines · Bernd Schubert
Quoted 49 lines · Joanne Koong
Add a fsparam helper for unsigned 16 bit values.
Signed-off-by Joanne Koong <[email protected]>--- fs/fs_parser.c | 14 ++++++++++++++ include/linux/fs_parser.h | 9 ++++++--- 2 files changed, 20 insertions(+), 3 deletions(-)fs/fs_parser.c +14
-0@@ -210,6 +210,20 @@ int fs_param_is_bool(struct p_log *log, const struct fs_parameter_spec *p, }EXPORT_SYMBOL (fs_param_is_bool); +int fs_param_is_u16 (struct p_log *log,const struct fs_parameter_spec *p, +struct fs_parameter *param,struct fs_parse_result *result) +{ +int base = (unsigned long )p->data; +if (param->type != fs_value_is_string) +return fs_param_bad_value (log, param); +if (!*param->string && (p->flags & fs_param_can_be_empty)) +return 0 ; +if (kstrtou16 (param->string, base, &result->uint_16) <0 ) +return fs_param_bad_value (log, param); +return 0 ; +} +EXPORT_SYMBOL (fs_param_is_u16); +int fs_param_is_u32 (struct p_log *log,const struct fs_parameter_spec *p,struct fs_parameter *param,struct fs_parse_result *result) {include/linux/fs_parser.h +4
-3@@ -26,9 +26,10 @@ typedef int fs_param_type(struct p_log *,/* * The type of parameter expected. */ -fs_param_type fs_param_is_bool, fs_param_is_u32, fs_param_is_s32, fs_param_is_u64, - fs_param_is_enum, fs_param_is_string, fs_param_is_blob, fs_param_is_blockdev, - fs_param_is_path, fs_param_is_fd, fs_param_is_uid, fs_param_is_gid; +fs_param_type fs_param_is_bool, fs_param_is_u16, fs_param_is_u32, fs_param_is_s32, + fs_param_is_u64, fs_param_is_enum, fs_param_is_string, fs_param_is_blob, + fs_param_is_blockdev, fs_param_is_path, fs_param_is_fd, fs_param_is_uid, + fs_param_is_gid;
Quoted 13 lines
include/linux/fs_parser.h struct fs_parse_result { · line 60
Quoted 8 lines
include/linux/fs_parser.h +1
-0/* * Specification of the type of value a parameter wants. @@ -55,6 +56,7 @@ struct fs_parse_result {union {bool boolean;/* For spec_bool */ int int_32;/* For spec_s32/spec_enum */ +u16 uint_16;/* For spec_u16 * unsigned int uint_32; /* For spec_u32{,_octal,_hex}/spec_enum */ Given
fs_param_is_u16()has "int base = (unsigned long)p->data;", shouldn't the comment also mention ",_octal,_hex}/spec_enum"? Or should the function get slightly modified to always use a base of 10? Or 0 with auto detection as forfs_param_is_u64()?
Thanks for taking a look at this patchset, Bernd!
I'm not sure what the correct answer here is either with whether this should follow fs_param_is_u32() or fs_param_is_u64(). I leaned towards fs_param_is_u32() because that seemed more permissive in supporting octal/hex.
I see in the commit history (commit 328de5287b10a) that Al Viro added both of these. After the fuse parts of this patchset gets feedback from Miklos, I'll cc Al on the next iteration and hopefully that should give us some more clarity.
Thanks,
Joanne
Quoted 14 lines
include/linux/fs_parser.h static inline bool fs_validate_description(const char *name, · line 127
Quoted 11 lines
include/linux/fs_parser.h +1
-0u64 uint_64;/* For spec_u64 */ kuid_t uid;@@ -119,6 +121,7 @@ static inline bool fs_validate_description(const char *name,#define fsparam_flag_no (NAME ,OPT ) \ __fsparam(NULL, NAME, OPT, fs_param_neg_with_no, NULL)#define fsparam_bool (NAME ,OPT ) __fsparam(fs_param_is_bool, NAME, OPT, 0, NULL) +#define fsparam_u16 (NAME ,OPT ) __fsparam(fs_param_is_u16, NAME, OPT, 0, NULL)#define fsparam_u32 (NAME ,OPT ) __fsparam(fs_param_is_u32, NAME, OPT, 0, NULL)#define fsparam_u32oct (NAME ,OPT ) \ __fsparam(fs_param_is_u32, NAME, OPT, 0, (void *)8)Reviewed-by Bernd Schubert <[email protected]>
PATCH v8 2/3 fuse: add optional kernel-enforced timeout for requests
In reply to Joanne KoongThere are situations where fuse servers can become unresponsive or stuck, for example if the server is deadlocked. Currently, there's no good way to detect if a server is stuck and needs to be killed manually.
This commit adds an option for enforcing a timeout (in minutes) for requests where if the timeout elapses without the server responding to the request, the connection will be automatically aborted.
Please note that these timeouts are not 100% precise. The request may take an extra FUSE_TIMEOUT_TIMER_FREQ seconds beyond the requested max timeout due to how it's internally implemented.
--- fs/fuse/dev.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 21 +++++++++++++ fs/fuse/inode.c | 21 +++++++++++++ 3 files changed, 122 insertions(+)
fs/fuse/dev.c +80 -0
@@ -45,6 +45,82 @@ static struct fuse_dev *fuse_get_dev(struct file *file)return READ_ONCE (file->private_data); } +static bool request_expired (struct fuse_conn *fc,struct fuse_req *req) +{ +return jiffies > req->create_time + fc->timeout.req_timeout; +} + +/* +* Check if any requests aren't being completed by the specified request +* timeout. To do so, we: +* - check the fiq pending list +* - check the bg queue +* - check the fpq io and processing lists +* +* To make this fast, we only check against the head request on each list since +* these are generally queued in order of creation time (eg newer requests get +* queued to the tail). We might miss a few edge cases (eg requests transitioning +* between lists, re-sent requests at the head of the pending list having a +* later creation time than other requests on that list, etc.) but that is fine +* since if the request never gets fulfilled, it will eventually be caught. +*/ +void fuse_check_timeout (struct timer_list *timer) +{ + struct fuse_conn *fc = container_of(timer, struct fuse_conn, timeout.timer); +struct fuse_iqueue *fiq = &fc->iq; +struct fuse_req *req; +struct fuse_dev *fud; +struct fuse_pqueue *fpq; +bool expired = false; +int i; + +spin_lock (&fiq->lock); + req = list_first_entry_or_null(&fiq->pending, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fiq->lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->bg_lock); + req = list_first_entry_or_null(&fc->bg_queue, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fc->bg_lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->lock); +if (!fc->connected) { +spin_unlock (&fc->lock); +return ; + } +list_for_each_entry (fud, &fc->devices, entry) { + fpq = &fud->pq; +spin_lock (&fpq->lock); + req = list_first_entry_or_null(&fpq->io, struct fuse_req, list); +if (req &&request_expired (fc, req)) + goto fpq_abort; + +for (i =0 ; i <FUSE_PQ_HASH_SIZE ; i++) { + req = list_first_entry_or_null(&fpq->processing[i], struct fuse_req, list); +if (req &&request_expired (fc, req)) + goto fpq_abort; + } +spin_unlock (&fpq->lock); + } +spin_unlock (&fc->lock); + +mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ ); +return ; + +fpq_abort: +spin_unlock (&fpq->lock); +spin_unlock (&fc->lock); +abort_conn: +fuse_abort_conn (fc); +} +static void fuse_request_init (struct fuse_mount *fm,struct fuse_req *req) {INIT_LIST_HEAD (&req->list);
@@ -53,6 +129,7 @@ static void fuse_request_init(struct fuse_mount *fm, struct fuse_req *req)refcount_set (&req->count,1 );__set_bit (FR_PENDING , &req->flags); req->fm = fm; + req->create_time = jiffies; }static struct fuse_req *fuse_request_alloc (struct fuse_mount *fm,gfp_t flags)
@@ -2296,6 +2373,9 @@ void fuse_abort_conn(struct fuse_conn *fc)spin_unlock (&fc->lock);end_requests (&to_end); + +if (fc->timeout.req_timeout) +timer_delete (&fc->timeout.timer); }else {spin_unlock (&fc->lock); }
fs/fuse/fuse_i.h +21 -0
@@ -435,6 +435,9 @@ struct fuse_req {/** fuse_mount this request belongs to */ struct fuse_mount *fm; + +/** When (in jiffies) the request was created */ +unsigned long create_time; };struct fuse_iqueue ;
@@ -525,6 +528,16 @@ struct fuse_pqueue {struct list_head io; }; +/* Frequency (in seconds) of request timeout checks, if opted into */ +#define FUSE_TIMEOUT_TIMER_FREQ 60 * HZ + +struct fuse_timeout { +struct timer_list timer; + +/* Request timeout (in jiffies). 0 = no timeout */ +unsigned long req_timeout; +}; +/** * Fuse device instance */
@@ -571,6 +584,8 @@ struct fuse_fs_context {enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize; +/* Request timeout (in minutes). 0 = no timeout (infinite wait) */ +unsigned int req_timeout;const char *subtype;/* DAX device, may be NULL */
@@ -914,6 +929,9 @@ struct fuse_conn {/** IDR for backing files ids */ struct idr backing_files_map;#endif + +/** Only used if the connection enforces request timeouts */ +struct fuse_timeout timeout; };/*
@@ -1175,6 +1193,9 @@ void fuse_request_end(struct fuse_req *req);void fuse_abort_conn (struct fuse_conn *fc);void fuse_wait_aborted (struct fuse_conn *fc); +/* Check if any requests timed out */ +void fuse_check_timeout (struct timer_list *timer); +/** * Invalidate inode attributes */
fs/fuse/inode.c +21 -0
@@ -735,6 +735,7 @@ enum {OPT_ALLOW_OTHER ,OPT_MAX_READ ,OPT_BLKSIZE , +OPT_REQUEST_TIMEOUT , OPT_ERR };
@@ -749,6 +750,7 @@ static const struct fs_parameter_spec fuse_fs_parameters[] = {fsparam_u32 ("max_read" ,OPT_MAX_READ ),fsparam_u32 ("blksize" ,OPT_BLKSIZE ),fsparam_string ("subtype" ,OPT_SUBTYPE ), + fsparam_u16 ("request_timeout", OPT_REQUEST_TIMEOUT), {} };
@@ -844,6 +846,10 @@ static int fuse_parse_param(struct fs_context *fsc, struct fs_parameter *param) ctx->blksize = result.uint_32;break ; +case OPT_REQUEST_TIMEOUT : + ctx->req_timeout = result.uint_16; +break ; +default :return -EINVAL ; }
@@ -973,6 +979,8 @@ void fuse_conn_put(struct fuse_conn *fc)if (IS_ENABLED (CONFIG_FUSE_DAX ))fuse_dax_conn_free (fc); +if (fc->timeout.req_timeout) +timer_shutdown_sync (&fc->timeout.timer);if (fiq->ops->release) fiq->ops->release (fiq);put_pid_ns (fc->pid_ns);
@@ -1691,6 +1699,18 @@ int fuse_init_fs_context_submount(struct fs_context *fsc) }EXPORT_SYMBOL_GPL (fuse_init_fs_context_submount); +static void fuse_init_fc_timeout (struct fuse_conn *fc,struct fuse_fs_context *ctx) +{ +if (ctx->req_timeout) { +if (check_mul_overflow (ctx->req_timeout *60 ,HZ , &fc->timeout.req_timeout)) + fc->timeout.req_timeout =U32_MAX ; +timer_setup (&fc->timeout.timer, fuse_check_timeout,0 ); +mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ ); + }else { + fc->timeout.req_timeout =0 ; + } +} +int fuse_fill_super_common (struct super_block *sb,struct fuse_fs_context *ctx) {struct fuse_dev *fud =NULL ;
@@ -1753,6 +1773,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) fc->destroy = ctx->destroy; fc->no_control = ctx->no_control; fc->no_force_umount = ctx->no_force_umount; +fuse_init_fc_timeout (fc, ctx); err = -ENOMEM ; root =fuse_get_root_inode (sb, ctx->rootmode);
Signature · 1 line
2.43.5
Re: [PATCH v8 2/3] fuse: add optional kernel-enforced timeout for requests
Quoted 82 lines · Joanne Koong
There are situations where fuse servers can become unresponsive or stuck, for example if the server is deadlocked. Currently, there's no good way to detect if a server is stuck and needs to be killed manually.
This commit adds an option for enforcing a timeout (in minutes) for requests where if the timeout elapses without the server responding to the request, the connection will be automatically aborted.
Please note that these timeouts are not 100% precise. The request may take an extra
FUSE_TIMEOUT_TIMER_FREQseconds beyond the requested max timeout due to how it's internally implemented.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 21 +++++++++++++ fs/fuse/inode.c | 21 +++++++++++++ 3 files changed, 122 insertions(+)fs/fuse/dev.c +55
-0@@ -45,6 +45,82 @@ static struct fuse_dev *fuse_get_dev(struct file *file)return READ_ONCE (file->private_data); } +static bool request_expired (struct fuse_conn *fc,struct fuse_req *req) +{ +return jiffies > req->create_time + fc->timeout.req_timeout; +} + +/* +* Check if any requests aren't being completed by the specified request +* timeout. To do so, we: +* - check the fiq pending list +* - check the bg queue +* - check the fpq io and processing lists +* +* To make this fast, we only check against the head request on each list since +* these are generally queued in order of creation time (eg newer requests get +* queued to the tail). We might miss a few edge cases (eg requests transitioning +* between lists, re-sent requests at the head of the pending list having a +* later creation time than other requests on that list, etc.) but that is fine +* since if the request never gets fulfilled, it will eventually be caught. +*/ +void fuse_check_timeout (struct timer_list *timer) +{ + struct fuse_conn *fc = container_of(timer, struct fuse_conn, timeout.timer); +struct fuse_iqueue *fiq = &fc->iq; +struct fuse_req *req; +struct fuse_dev *fud; +struct fuse_pqueue *fpq; +bool expired = false; +int i; + +spin_lock (&fiq->lock); + req = list_first_entry_or_null(&fiq->pending, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fiq->lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->bg_lock); + req = list_first_entry_or_null(&fc->bg_queue, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fc->bg_lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->lock); +if (!fc->connected) { +spin_unlock (&fc->lock); +return ; + } +list_for_each_entry (fud, &fc->devices, entry) { + fpq = &fud->pq; +spin_lock (&fpq->lock); + req = list_first_entry_or_null(&fpq->io, struct fuse_req, list); +if (req &&request_expired (fc, req))
fs/fuse/dev.c static struct fuse_dev *fuse_get_dev(struct file *file) · line 112
Quoted 10 lines
fs/fuse/dev.c +10
-0+ goto fpq_abort; + +for (i =0 ; i <FUSE_PQ_HASH_SIZE ; i++) { + req = list_first_entry_or_null(&fpq->processing[i], struct fuse_req, list); +if (req &&request_expired (fc, req)) + goto fpq_abort; + } +spin_unlock (&fpq->lock); + } +spin_unlock (&fc->lock);
I really don't have a strong opinion on that - I wonder if it wouldn't be better for this part to have an extra timeout list per fud or pq as previously. That would slightly increases memory usage and overhead per request as a second list is needed, but would reduce these 1/min cpu spikes as only one list per fud would need to be checked. But then, it would be easy to change that later, if timeout checks turn out to be a problem.
Quoted 133 lines
fs/fuse/dev.c +15
-0+ +mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ ); +return ; + +fpq_abort: +spin_unlock (&fpq->lock); +spin_unlock (&fc->lock); +abort_conn: +fuse_abort_conn (fc); +} +static void fuse_request_init (struct fuse_mount *fm,struct fuse_req *req) {INIT_LIST_HEAD (&req->list);@@ -53,6 +129,7 @@ static void fuse_request_init(struct fuse_mount *fm, struct fuse_req *req)refcount_set (&req->count,1 );__set_bit (FR_PENDING , &req->flags); req->fm = fm; + req->create_time = jiffies; }static struct fuse_req *fuse_request_alloc (struct fuse_mount *fm,gfp_t flags)@@ -2296,6 +2373,9 @@ void fuse_abort_conn(struct fuse_conn *fc)spin_unlock (&fc->lock);end_requests (&to_end); + +if (fc->timeout.req_timeout) +timer_delete (&fc->timeout.timer); }else {spin_unlock (&fc->lock); }fs/fuse/fuse_i.h +21
-0@@ -435,6 +435,9 @@ struct fuse_req {/** fuse_mount this request belongs to */ struct fuse_mount *fm; + +/** When (in jiffies) the request was created */ +unsigned long create_time; };struct fuse_iqueue ;@@ -525,6 +528,16 @@ struct fuse_pqueue {struct list_head io; }; +/* Frequency (in seconds) of request timeout checks, if opted into */ +#define FUSE_TIMEOUT_TIMER_FREQ 60 * HZ + +struct fuse_timeout { +struct timer_list timer; + +/* Request timeout (in jiffies). 0 = no timeout */ +unsigned long req_timeout; +}; +/** * Fuse device instance */ @@ -571,6 +584,8 @@ struct fuse_fs_context {enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize; +/* Request timeout (in minutes). 0 = no timeout (infinite wait) */ +unsigned int req_timeout;const char *subtype;/* DAX device, may be NULL */ @@ -914,6 +929,9 @@ struct fuse_conn {/** IDR for backing files ids */ struct idr backing_files_map;#endif + +/** Only used if the connection enforces request timeouts */ +struct fuse_timeout timeout; };/* @@ -1175,6 +1193,9 @@ void fuse_request_end(struct fuse_req *req);void fuse_abort_conn (struct fuse_conn *fc);void fuse_wait_aborted (struct fuse_conn *fc); +/* Check if any requests timed out */ +void fuse_check_timeout (struct timer_list *timer); +/** * Invalidate inode attributes */ fs/fuse/inode.c +8
-0@@ -735,6 +735,7 @@ enum {OPT_ALLOW_OTHER ,OPT_MAX_READ ,OPT_BLKSIZE , +OPT_REQUEST_TIMEOUT , OPT_ERR };@@ -749,6 +750,7 @@ static const struct fs_parameter_spec fuse_fs_parameters[] = {fsparam_u32 ("max_read" ,OPT_MAX_READ ),fsparam_u32 ("blksize" ,OPT_BLKSIZE ),fsparam_string ("subtype" ,OPT_SUBTYPE ), + fsparam_u16 ("request_timeout", OPT_REQUEST_TIMEOUT), {} };@@ -844,6 +846,10 @@ static int fuse_parse_param(struct fs_context *fsc, struct fs_parameter *param) ctx->blksize = result.uint_32;break ; +case OPT_REQUEST_TIMEOUT : + ctx->req_timeout = result.uint_16; +break ; +default :return -EINVAL ; }@@ -973,6 +979,8 @@ void fuse_conn_put(struct fuse_conn *fc)if (IS_ENABLED (CONFIG_FUSE_DAX ))fuse_dax_conn_free (fc); +if (fc->timeout.req_timeout) +timer_shutdown_sync (&fc->timeout.timer);if (fiq->ops->release) fiq->ops->release (fiq);put_pid_ns (fc->pid_ns);
fs/fuse/inode.c int fuse_init_fs_context_submount(struct fs_context *fsc) · line 1706
Quoted 8 lines
fs/fuse/inode.c +5
-0@@ -1691,6 +1699,18 @@ int fuse_init_fs_context_submount(struct fs_context *fsc) }EXPORT_SYMBOL_GPL (fuse_init_fs_context_submount); +static void fuse_init_fc_timeout (struct fuse_conn *fc,struct fuse_fs_context *ctx) +{ +if (ctx->req_timeout) { +if (check_mul_overflow (ctx->req_timeout *60 ,HZ , &fc->timeout.req_timeout)) + fc->timeout.req_timeout =U32_MAX ;
ULONG_MAX?
Quoted 11 lines
fs/fuse/inode.c +7
-0+timer_setup (&fc->timeout.timer, fuse_check_timeout,0 ); +mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ ); + }else { + fc->timeout.req_timeout =0 ; + } +} +int fuse_fill_super_common (struct super_block *sb,struct fuse_fs_context *ctx) {struct fuse_dev *fud =NULL ;
fs/fuse/inode.c int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) · line 1779
Quoted 7 lines
fs/fuse/inode.c +1
-0@@ -1753,6 +1773,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) fc->destroy = ctx->destroy; fc->no_control = ctx->no_control; fc->no_force_umount = ctx->no_force_umount; +fuse_init_fc_timeout (fc, ctx); err = -ENOMEM ; root =fuse_get_root_inode (sb, ctx->rootmode);
Re: [PATCH v8 2/3] fuse: add optional kernel-enforced timeout for requests
Quoted 97 lines · Bernd Schubert
Quoted 82 lines · Joanne Koong
There are situations where fuse servers can become unresponsive or stuck, for example if the server is deadlocked. Currently, there's no good way to detect if a server is stuck and needs to be killed manually.
This commit adds an option for enforcing a timeout (in minutes) for requests where if the timeout elapses without the server responding to the request, the connection will be automatically aborted.
Please note that these timeouts are not 100% precise. The request may take an extra
FUSE_TIMEOUT_TIMER_FREQseconds beyond the requested max timeout due to how it's internally implemented.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 21 +++++++++++++ fs/fuse/inode.c | 21 +++++++++++++ 3 files changed, 122 insertions(+)fs/fuse/dev.c +55
-0@@ -45,6 +45,82 @@ static struct fuse_dev *fuse_get_dev(struct file *file)return READ_ONCE (file->private_data); } +static bool request_expired (struct fuse_conn *fc,struct fuse_req *req) +{ +return jiffies > req->create_time + fc->timeout.req_timeout; +} + +/* +* Check if any requests aren't being completed by the specified request +* timeout. To do so, we: +* - check the fiq pending list +* - check the bg queue +* - check the fpq io and processing lists +* +* To make this fast, we only check against the head request on each list since +* these are generally queued in order of creation time (eg newer requests get +* queued to the tail). We might miss a few edge cases (eg requests transitioning +* between lists, re-sent requests at the head of the pending list having a +* later creation time than other requests on that list, etc.) but that is fine +* since if the request never gets fulfilled, it will eventually be caught. +*/ +void fuse_check_timeout (struct timer_list *timer) +{ + struct fuse_conn *fc = container_of(timer, struct fuse_conn, timeout.timer); +struct fuse_iqueue *fiq = &fc->iq; +struct fuse_req *req; +struct fuse_dev *fud; +struct fuse_pqueue *fpq; +bool expired = false; +int i; + +spin_lock (&fiq->lock); + req = list_first_entry_or_null(&fiq->pending, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fiq->lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->bg_lock); + req = list_first_entry_or_null(&fc->bg_queue, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fc->bg_lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->lock); +if (!fc->connected) { +spin_unlock (&fc->lock); +return ; + } +list_for_each_entry (fud, &fc->devices, entry) { + fpq = &fud->pq; +spin_lock (&fpq->lock); + req = list_first_entry_or_null(&fpq->io, struct fuse_req, list); +if (req &&request_expired (fc, req))fs/fuse/dev.c static struct fuse_dev *fuse_get_dev(struct file *file) · line 112
Quoted 10 lines
fs/fuse/dev.c +10
-0+ goto fpq_abort; + +for (i =0 ; i <FUSE_PQ_HASH_SIZE ; i++) { + req = list_first_entry_or_null(&fpq->processing[i], struct fuse_req, list); +if (req &&request_expired (fc, req)) + goto fpq_abort; + } +spin_unlock (&fpq->lock); + } +spin_unlock (&fc->lock);
Quoted 8 lines
I really don't have a strong opinion on that - I wonder if it wouldn't be better for this part to have an extra timeout list per fud or pq as previously. That would slightly increases memory usage and overhead per request as a second list is needed, but would reduce these 1/min cpu spikes as only one list per fud would need to be checked. But then, it would be easy to change that later, if timeout checks turn out to be a problem.
Thanks for the review.
On v7 [1] which used an extra timeout list, the feedback was
"One thing I worry about is adding more roadblocks on the way to making request queuing more scalable.
Currently there's fc->num_waiting that's touched on all requests and bg_queue/bg_lock that are touched on background requests. We should be trying to fix these bottlenecks instead of adding more.
Can't we use the existing lists to scan requests?
It's more complex, obviously, but at least it doesn't introduce yet another per-fc list to worry about."
Quoted 137 lines
Quoted 137 lines
fs/fuse/dev.c +15
-0+ +mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ ); +return ; + +fpq_abort: +spin_unlock (&fpq->lock); +spin_unlock (&fc->lock); +abort_conn: +fuse_abort_conn (fc); +} +static void fuse_request_init (struct fuse_mount *fm,struct fuse_req *req) {INIT_LIST_HEAD (&req->list);@@ -53,6 +129,7 @@ static void fuse_request_init(struct fuse_mount *fm, struct fuse_req *req)refcount_set (&req->count,1 );__set_bit (FR_PENDING , &req->flags); req->fm = fm; + req->create_time = jiffies; }static struct fuse_req *fuse_request_alloc (struct fuse_mount *fm,gfp_t flags)@@ -2296,6 +2373,9 @@ void fuse_abort_conn(struct fuse_conn *fc)spin_unlock (&fc->lock);end_requests (&to_end); + +if (fc->timeout.req_timeout) +timer_delete (&fc->timeout.timer); }else {spin_unlock (&fc->lock); }fs/fuse/fuse_i.h +21
-0@@ -435,6 +435,9 @@ struct fuse_req {/** fuse_mount this request belongs to */ struct fuse_mount *fm; + +/** When (in jiffies) the request was created */ +unsigned long create_time; };struct fuse_iqueue ;@@ -525,6 +528,16 @@ struct fuse_pqueue {struct list_head io; }; +/* Frequency (in seconds) of request timeout checks, if opted into */ +#define FUSE_TIMEOUT_TIMER_FREQ 60 * HZ + +struct fuse_timeout { +struct timer_list timer; + +/* Request timeout (in jiffies). 0 = no timeout */ +unsigned long req_timeout; +}; +/** * Fuse device instance */ @@ -571,6 +584,8 @@ struct fuse_fs_context {enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize; +/* Request timeout (in minutes). 0 = no timeout (infinite wait) */ +unsigned int req_timeout;const char *subtype;/* DAX device, may be NULL */ @@ -914,6 +929,9 @@ struct fuse_conn {/** IDR for backing files ids */ struct idr backing_files_map;#endif + +/** Only used if the connection enforces request timeouts */ +struct fuse_timeout timeout; };/* @@ -1175,6 +1193,9 @@ void fuse_request_end(struct fuse_req *req);void fuse_abort_conn (struct fuse_conn *fc);void fuse_wait_aborted (struct fuse_conn *fc); +/* Check if any requests timed out */ +void fuse_check_timeout (struct timer_list *timer); +/** * Invalidate inode attributes */ fs/fuse/inode.c +8
-0@@ -735,6 +735,7 @@ enum {OPT_ALLOW_OTHER ,OPT_MAX_READ ,OPT_BLKSIZE , +OPT_REQUEST_TIMEOUT , OPT_ERR };@@ -749,6 +750,7 @@ static const struct fs_parameter_spec fuse_fs_parameters[] = {fsparam_u32 ("max_read" ,OPT_MAX_READ ),fsparam_u32 ("blksize" ,OPT_BLKSIZE ),fsparam_string ("subtype" ,OPT_SUBTYPE ), + fsparam_u16 ("request_timeout", OPT_REQUEST_TIMEOUT), {} };@@ -844,6 +846,10 @@ static int fuse_parse_param(struct fs_context *fsc, struct fs_parameter *param) ctx->blksize = result.uint_32;break ; +case OPT_REQUEST_TIMEOUT : + ctx->req_timeout = result.uint_16; +break ; +default :return -EINVAL ; }@@ -973,6 +979,8 @@ void fuse_conn_put(struct fuse_conn *fc)if (IS_ENABLED (CONFIG_FUSE_DAX ))fuse_dax_conn_free (fc); +if (fc->timeout.req_timeout) +timer_shutdown_sync (&fc->timeout.timer);if (fiq->ops->release) fiq->ops->release (fiq);put_pid_ns (fc->pid_ns);@@ -1691,6 +1699,18 @@ int fuse_init_fs_context_submount(struct fs_context *fsc) }EXPORT_SYMBOL_GPL (fuse_init_fs_context_submount);
Quoted 8 lines
fs/fuse/inode.c int fuse_init_fs_context_submount(struct fs_context *fsc) · line 1706
Quoted 5 lines
fs/fuse/inode.c +5
-0+static void fuse_init_fc_timeout (struct fuse_conn *fc,struct fuse_fs_context *ctx) +{ +if (ctx->req_timeout) { +if (check_mul_overflow (ctx->req_timeout *60 ,HZ , &fc->timeout.req_timeout)) + fc->timeout.req_timeout =U32_MAX ;
ULONG_MAX?
Nice, I'll change this to ULONG_MAX. We only run into this overflow on 32-bit systems (and only if the kernel has configured HZ to greater than 1092) so U32_MAX is the same as ULONG_MAX, but ULONG_MAX looks nicer since "fc->timeout.req_timeout" is an unsigned long.
Thanks,
Joanne
Quoted 22 lines
Quoted 12 lines
fs/fuse/inode.c +7
-0+timer_setup (&fc->timeout.timer, fuse_check_timeout,0 ); +mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ ); + }else { + fc->timeout.req_timeout =0 ; + } +} +int fuse_fill_super_common (struct super_block *sb,struct fuse_fs_context *ctx) {struct fuse_dev *fud =NULL ;fs/fuse/inode.c int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) · line 1779
Quoted 7 lines
fs/fuse/inode.c +1
-0@@ -1753,6 +1773,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) fc->destroy = ctx->destroy; fc->no_control = ctx->no_control; fc->no_force_umount = ctx->no_force_umount; +fuse_init_fc_timeout (fc, ctx); err = -ENOMEM ; root =fuse_get_root_inode (sb, ctx->rootmode);Reviewed-by Bernd Schubert <[email protected]>
Re: [PATCH v8 2/3] fuse: add optional kernel-enforced timeout for requests
Quoted 119 lines · Joanne Koong
Quoted 97 lines · Bernd Schubert
Quoted 82 lines · Joanne Koong
There are situations where fuse servers can become unresponsive or stuck, for example if the server is deadlocked. Currently, there's no good way to detect if a server is stuck and needs to be killed manually.
This commit adds an option for enforcing a timeout (in minutes) for requests where if the timeout elapses without the server responding to the request, the connection will be automatically aborted.
Please note that these timeouts are not 100% precise. The request may take an extra
FUSE_TIMEOUT_TIMER_FREQseconds beyond the requested max timeout due to how it's internally implemented.Signed-off-by Joanne Koong <[email protected]>--- fs/fuse/dev.c | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 21 +++++++++++++ fs/fuse/inode.c | 21 +++++++++++++ 3 files changed, 122 insertions(+)fs/fuse/dev.c +55
-0@@ -45,6 +45,82 @@ static struct fuse_dev *fuse_get_dev(struct file *file)return READ_ONCE (file->private_data); } +static bool request_expired (struct fuse_conn *fc,struct fuse_req *req) +{ +return jiffies > req->create_time + fc->timeout.req_timeout; +} + +/* +* Check if any requests aren't being completed by the specified request +* timeout. To do so, we: +* - check the fiq pending list +* - check the bg queue +* - check the fpq io and processing lists +* +* To make this fast, we only check against the head request on each list since +* these are generally queued in order of creation time (eg newer requests get +* queued to the tail). We might miss a few edge cases (eg requests transitioning +* between lists, re-sent requests at the head of the pending list having a +* later creation time than other requests on that list, etc.) but that is fine +* since if the request never gets fulfilled, it will eventually be caught. +*/ +void fuse_check_timeout (struct timer_list *timer) +{ + struct fuse_conn *fc = container_of(timer, struct fuse_conn, timeout.timer); +struct fuse_iqueue *fiq = &fc->iq; +struct fuse_req *req; +struct fuse_dev *fud; +struct fuse_pqueue *fpq; +bool expired = false; +int i; + +spin_lock (&fiq->lock); + req = list_first_entry_or_null(&fiq->pending, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fiq->lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->bg_lock); + req = list_first_entry_or_null(&fc->bg_queue, struct fuse_req, list); +if (req) + expired =request_expired (fc, req); +spin_unlock (&fc->bg_lock); +if (expired) + goto abort_conn; + +spin_lock (&fc->lock); +if (!fc->connected) { +spin_unlock (&fc->lock); +return ; + } +list_for_each_entry (fud, &fc->devices, entry) { + fpq = &fud->pq; +spin_lock (&fpq->lock); + req = list_first_entry_or_null(&fpq->io, struct fuse_req, list); +if (req &&request_expired (fc, req))fs/fuse/dev.c static struct fuse_dev *fuse_get_dev(struct file *file) · line 112
Quoted 10 lines
fs/fuse/dev.c +10
-0+ goto fpq_abort; + +for (i =0 ; i <FUSE_PQ_HASH_SIZE ; i++) { + req = list_first_entry_or_null(&fpq->processing[i], struct fuse_req, list); +if (req &&request_expired (fc, req)) + goto fpq_abort; + } +spin_unlock (&fpq->lock); + } +spin_unlock (&fc->lock);Quoted 8 lines
I really don't have a strong opinion on that - I wonder if it wouldn't be better for this part to have an extra timeout list per fud or pq as previously. That would slightly increases memory usage and overhead per request as a second list is needed, but would reduce these 1/min cpu spikes as only one list per fud would need to be checked. But then, it would be easy to change that later, if timeout checks turn out to be a problem.
Thanks for the review.
On v7 [1] which used an extra timeout list, the feedback was
"One thing I worry about is adding more roadblocks on the way to making request queuing more scalable.
Currently there's
fc->num_waitingthat's touched on all requests and bg_queue/bg_lock that are touched on background requests. We should be trying to fix these bottlenecks instead of adding more.
Quoted 7 lines
Can't we use the existing lists to scan requests?
It's more complex, obviously, but at least it doesn't introduce yet another per-fc list to worry about."
Yeah I know, but I'm just a bit scared about the cpu spikes this gives on a recent systems (like 96 cores AMDs we have in the lab) and when device clone is activated (and Miklos had also asked to use the same hash lists to find requests with fuse-uring).
The previous discussion was mainly about avoiding a new global lock, i.e. I wonder if can't use a mix - avoid the new list and its lock, but still avoid scanning through [FUSE_PQ_HASH_SIZE] lists per cloned-device/core.
I'm also not opposed against the current version and optimize it later.
Thanks,
Bernd
PATCH v8 3/3 fuse: add default_request_timeout and max_request_timeout sysctls
In reply to Joanne KoongIntroduce two new sysctls, "default_request_timeout" and "max_request_timeout". These control how long (in minutes) a server can take to reply to a request. If the server does not reply by the timeout, then the connection will be aborted.
"default_request_timeout" sets the default timeout if no timeout is specified by the fuse server on mount. 0 (default) indicates no default timeout should be enforced. If the server did specify a timeout, then default_request_timeout will be ignored.
"max_request_timeout" sets the max amount of time the server may take to reply to a request. 0 (default) indicates no maximum timeout. If max_request_timeout is set and the fuse server attempts to set a timeout greater than max_request_timeout, the system will use max_request_timeout as the timeout. Similarly, if default_request_timeout is greater than max_request_timeout, the system will use max_request_timeout as the timeout. If the server does not request a timeout and default_request_timeout is set to 0 but max_request_timeout is set, then the timeout will be max_request_timeout.
Please note that these timeouts are not 100% precise. The request may take an extra FUSE_TIMEOUT_TIMER_FREQ seconds beyond the set max timeout due to how it's internally implemented.
$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 0
$ echo 65536 | sudo tee /proc/sys/fs/fuse/default_request_timeout tee: /proc/sys/fs/fuse/default_request_timeout: Invalid argument
$ echo 65535 | sudo tee /proc/sys/fs/fuse/default_request_timeout 65535
$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 65535
$ echo 0 | sudo tee /proc/sys/fs/fuse/default_request_timeout 0
$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 0
--- Documentation/admin-guide/sysctl/fs.rst | 27 +++++++++++++++++++++++++ fs/fuse/fuse_i.h | 10 +++++++++ fs/fuse/inode.c | 16 +++++++++++++-- fs/fuse/sysctl.c | 20 ++++++++++++++++++ 4 files changed, 71 insertions(+), 2 deletions(-)
Documentation/admin-guide/sysctl/fs.rst +27 -0
@@ -342,3 +342,30 @@ filesystems: ``/proc/sys/fs/fuse/max_pages_limit`` is a read/write file for setting/getting the maximum number of pages that can be used for servicing requests in FUSE. + +``/proc/sys/fs/fuse/default_request_timeout`` is a read/write file for +setting/getting the default timeout (in minutes) for a fuse server to +reply to a kernel-issued request in the event where the server did not +specify a timeout at mount. If the server set a timeout, +then default_request_timeout will be ignored. The default +"default_request_timeout" is set to 0. 0 indicates a no-op (eg +requests will not have a default request timeout set if no timeout was +specified by the server). + +``/proc/sys/fs/fuse/max_request_timeout`` is a read/write file for +setting/getting the maximum timeout (in minutes) for a fuse server to +reply to a kernel-issued request. A value greater than 0 automatically opts +the server into a timeout that will be at most "max_request_timeout", even if +the server did not specify a timeout and default_request_timeout is set to 0. +If max_request_timeout is greater than 0 and the server set a timeout greater +than max_request_timeout or default_request_timeout is set to a value greater +than max_request_timeout, the system will use max_request_timeout as the +timeout. 0 indicates a no-op (eg requests will not have an upper bound on the +timeout and if the server did not request a timeout and default_request_timeout +was not set, there will be no timeout). + +Please note that for the timeout options, if the server does not respond to +the request by the time the timeout elapses, then the connection to the fuse +server will be aborted. Please also note that the timeouts are not 100% +precise (eg you may set 10 minutes but the timeout may kick in after 11 +minutes).
fs/fuse/fuse_i.h +10 -0
@@ -46,6 +46,16 @@/** Maximum of max_pages received in init_out */ extern unsigned int fuse_max_pages_limit; +/* +* Default timeout (in minutes) for the server to reply to a request +* before the connection is aborted, if no timeout was specified on mount. +*/ +extern unsigned int fuse_default_req_timeout; +/* +* Max timeout (in minutes) for the server to reply to a request before +* the connection is aborted. +*/ +extern unsigned int fuse_max_req_timeout;/** List of active connections */ extern struct list_head fuse_conn_list;
fs/fuse/inode.c +14 -2
@@ -36,6 +36,9 @@ DEFINE_MUTEX(fuse_mutex);static int set_global_limit (const char *val,const struct kernel_param *kp);unsigned int fuse_max_pages_limit =256 ; +/* default is no timeout */ +unsigned int fuse_default_req_timeout =0 ; +unsigned int fuse_max_req_timeout =0 ;unsigned max_user_bgreq; module_param_call(max_user_bgreq, set_global_limit, param_get_uint,
@@ -1701,8 +1704,17 @@ EXPORT_SYMBOL_GPL(fuse_init_fs_context_submount);static void fuse_init_fc_timeout (struct fuse_conn *fc,struct fuse_fs_context *ctx) { -if (ctx->req_timeout) { -if (check_mul_overflow (ctx->req_timeout *60 ,HZ , &fc->timeout.req_timeout)) +unsigned int timeout = ctx->req_timeout ?: fuse_default_req_timeout; + +if (fuse_max_req_timeout) { +if (!timeout) + timeout = fuse_max_req_timeout; +else + timeout =min (timeout, fuse_max_req_timeout); + } + +if (timeout) { +if (check_mul_overflow (timeout *60 ,HZ , &fc->timeout.req_timeout)) fc->timeout.req_timeout =U32_MAX ;timer_setup (&fc->timeout.timer, fuse_check_timeout,0 );mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ );
fs/fuse/sysctl.c +20 -0
@@ -13,6 +13,8 @@ static struct ctl_table_header *fuse_table_header;/* Bound by fuse_init_out max_pages, which is a u16 */ static unsigned int sysctl_fuse_max_pages_limit =65535 ; +static unsigned int sysctl_fuse_max_req_timeout_limit =U16_MAX ; +static struct ctl_table fuse_sysctl_table[] = { { .procname ="max_pages_limit" ,
@@ -23,6 +25,24 @@ static struct ctl_table fuse_sysctl_table[] = { .extra1 = SYSCTL_ONE, .extra2 = &sysctl_fuse_max_pages_limit, }, + { + .procname = "default_request_timeout", + .data = &fuse_default_req_timeout, + .maxlen = sizeof(fuse_default_req_timeout), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &sysctl_fuse_max_req_timeout_limit, + }, + { + .procname = "max_request_timeout", + .data = &fuse_max_req_timeout, + .maxlen = sizeof(fuse_max_req_timeout), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = &sysctl_fuse_max_req_timeout_limit, + }, };int fuse_sysctl_register (void )
Signature · 1 line
2.43.5
Re: [PATCH v8 3/3] fuse: add default_request_timeout and max_request_timeout sysctls
Quoted 121 lines · Joanne Koong
Introduce two new sysctls, "
default_request_timeout" and "max_request_timeout". These control how long (in minutes) a server can take to reply to a request. If the server does not reply by the timeout, then the connection will be aborted."
default_request_timeout" sets the default timeout if no timeout is specified by the fuse server on mount. 0 (default) indicates no default timeout should be enforced. If the server did specify a timeout, thendefault_request_timeoutwill be ignored."
max_request_timeout" sets the max amount of time the server may take to reply to a request. 0 (default) indicates no maximum timeout. Ifmax_request_timeoutis set and the fuse server attempts to set a timeout greater thanmax_request_timeout, the system will usemax_request_timeoutas the timeout. Similarly, ifdefault_request_timeoutis greater thanmax_request_timeout, the system will usemax_request_timeoutas the timeout. If the server does not request a timeout anddefault_request_timeoutis set to 0 butmax_request_timeoutis set, then the timeout will bemax_request_timeout.Please note that these timeouts are not 100% precise. The request may take an extra
FUSE_TIMEOUT_TIMER_FREQseconds beyond the set max timeout due to how it's internally implemented.$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 0$ echo 65536 | sudo tee /proc/sys/fs/fuse/default_request_timeout tee: /proc/sys/fs/fuse/default_request_timeout: Invalid argument$ echo 65535 | sudo tee /proc/sys/fs/fuse/default_request_timeout 65535$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 65535$ echo 0 | sudo tee /proc/sys/fs/fuse/default_request_timeout 0$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 0Signed-off-by Joanne Koong <[email protected]>--- Documentation/admin-guide/sysctl/fs.rst | 27 +++++++++++++++++++++++++ fs/fuse/fuse_i.h | 10 +++++++++ fs/fuse/inode.c | 16 +++++++++++++-- fs/fuse/sysctl.c | 20 ++++++++++++++++++ 4 files changed, 71 insertions(+), 2 deletions(-)Documentation/admin-guide/sysctl/fs.rst +27
-0@@ -342,3 +342,30 @@ filesystems: ``/proc/sys/fs/fuse/max_pages_limit`` is a read/write file for setting/getting the maximum number of pages that can be used for servicing requests in FUSE. + +``/proc/sys/fs/fuse/default_request_timeout`` is a read/write file for +setting/getting the default timeout (in minutes) for a fuse server to +reply to a kernel-issued request in the event where the server did not +specify a timeout at mount. If the server set a timeout, +then default_request_timeout will be ignored. The default +"default_request_timeout" is set to 0. 0 indicates a no-op (eg +requests will not have a default request timeout set if no timeout was +specified by the server). + +``/proc/sys/fs/fuse/max_request_timeout`` is a read/write file for +setting/getting the maximum timeout (in minutes) for a fuse server to +reply to a kernel-issued request. A value greater than 0 automatically opts +the server into a timeout that will be at most "max_request_timeout", even if +the server did not specify a timeout and default_request_timeout is set to 0. +If max_request_timeout is greater than 0 and the server set a timeout greater +than max_request_timeout or default_request_timeout is set to a value greater +than max_request_timeout, the system will use max_request_timeout as the +timeout. 0 indicates a no-op (eg requests will not have an upper bound on the +timeout and if the server did not request a timeout and default_request_timeout +was not set, there will be no timeout). + +Please note that for the timeout options, if the server does not respond to +the request by the time the timeout elapses, then the connection to the fuse +server will be aborted. Please also note that the timeouts are not 100% +precise (eg you may set 10 minutes but the timeout may kick in after 11 +minutes).fs/fuse/fuse_i.h +10
-0@@ -46,6 +46,16 @@/** Maximum of max_pages received in init_out */ extern unsigned int fuse_max_pages_limit; +/* +* Default timeout (in minutes) for the server to reply to a request +* before the connection is aborted, if no timeout was specified on mount. +*/ +extern unsigned int fuse_default_req_timeout; +/* +* Max timeout (in minutes) for the server to reply to a request before +* the connection is aborted. +*/ +extern unsigned int fuse_max_req_timeout;/** List of active connections */ extern struct list_head fuse_conn_list;fs/fuse/inode.c +3
-0@@ -36,6 +36,9 @@ DEFINE_MUTEX(fuse_mutex);static int set_global_limit (const char *val,const struct kernel_param *kp);unsigned int fuse_max_pages_limit =256 ; +/* default is no timeout */ +unsigned int fuse_default_req_timeout =0 ; +unsigned int fuse_max_req_timeout =0 ;unsigned max_user_bgreq; module_param_call(max_user_bgreq, set_global_limit, param_get_uint,
fs/fuse/inode.c EXPORT_SYMBOL_GPL(fuse_init_fs_context_submount); · line 1711
Quoted 10 lines
fs/fuse/inode.c +5
-2@@ -1701,8 +1704,17 @@ EXPORT_SYMBOL_GPL(fuse_init_fs_context_submount);static void fuse_init_fc_timeout (struct fuse_conn *fc,struct fuse_fs_context *ctx) { -if (ctx->req_timeout) { -if (check_mul_overflow (ctx->req_timeout *60 ,HZ , &fc->timeout.req_timeout)) +unsigned int timeout = ctx->req_timeout ?: fuse_default_req_timeout; + +if (fuse_max_req_timeout) { +if (!timeout) + timeout = fuse_max_req_timeout;
Hmm, so 'max' might be used as 'min' as well. Isn't that a bit confusing?
Quoted 22 lines
fs/fuse/inode.c +5
-0+else + timeout = min(timeout, fuse_max_req_timeout)> + } + + if (timeout) { + if (check_mul_overflow(timeout * 60, HZ, &fc->timeout.req_timeout)) fc->timeout.req_timeout = U32_MAX;timer_setup (&fc->timeout.timer, fuse_check_timeout,0 );mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ );fs/fuse/sysctl.c +2
-0@@ -13,6 +13,8 @@ static struct ctl_table_header *fuse_table_header;/* Bound by fuse_init_out max_pages, which is a u16 */ static unsigned int sysctl_fuse_max_pages_limit =65535 ; +static unsigned int sysctl_fuse_max_req_timeout_limit =U16_MAX ; +static struct ctl_table fuse_sysctl_table[] = { { .procname ="max_pages_limit" ,
fs/fuse/sysctl.c static struct ctl_table fuse_sysctl_table[] = { · line 34
Quoted 10 lines
fs/fuse/sysctl.c +7
-0@@ -23,6 +25,24 @@ static struct ctl_table fuse_sysctl_table[] = { .extra1 = SYSCTL_ONE, .extra2 = &sysctl_fuse_max_pages_limit, }, + { + .procname = "default_request_timeout", + .data = &fuse_default_req_timeout, + .maxlen = sizeof(fuse_default_req_timeout), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO,
There is slight whitespace issue here - spaces instead of tabs.
fs/fuse/sysctl.c static struct ctl_table fuse_sysctl_table[] = { · line 43
Quoted 9 lines
fs/fuse/sysctl.c +9
-0+ .extra2 = &sysctl_fuse_max_req_timeout_limit, + }, + { + .procname = "max_request_timeout", + .data = &fuse_max_req_timeout, + .maxlen = sizeof(fuse_max_req_timeout), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO,
And here.
fs/fuse/sysctl.c static struct ctl_table fuse_sysctl_table[] = { · line 48
Quoted 5 lines
fs/fuse/sysctl.c +2
-0+ .extra2 = &sysctl_fuse_max_req_timeout_limit, + }, };int fuse_sysctl_register (void )
Re: [PATCH v8 3/3] fuse: add default_request_timeout and max_request_timeout sysctls
Quoted 124 lines · Bernd Schubert
Quoted 122 lines · Joanne Koong
Introduce two new sysctls, "
default_request_timeout" and "max_request_timeout". These control how long (in minutes) a server can take to reply to a request. If the server does not reply by the timeout, then the connection will be aborted."
default_request_timeout" sets the default timeout if no timeout is specified by the fuse server on mount. 0 (default) indicates no default timeout should be enforced. If the server did specify a timeout, thendefault_request_timeoutwill be ignored."
max_request_timeout" sets the max amount of time the server may take to reply to a request. 0 (default) indicates no maximum timeout. Ifmax_request_timeoutis set and the fuse server attempts to set a timeout greater thanmax_request_timeout, the system will usemax_request_timeoutas the timeout. Similarly, ifdefault_request_timeoutis greater thanmax_request_timeout, the system will usemax_request_timeoutas the timeout. If the server does not request a timeout anddefault_request_timeoutis set to 0 butmax_request_timeoutis set, then the timeout will bemax_request_timeout.Please note that these timeouts are not 100% precise. The request may take an extra
FUSE_TIMEOUT_TIMER_FREQseconds beyond the set max timeout due to how it's internally implemented.$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 0$ echo 65536 | sudo tee /proc/sys/fs/fuse/default_request_timeout tee: /proc/sys/fs/fuse/default_request_timeout: Invalid argument$ echo 65535 | sudo tee /proc/sys/fs/fuse/default_request_timeout 65535$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 65535$ echo 0 | sudo tee /proc/sys/fs/fuse/default_request_timeout 0$ sysctl -a | grep fuse.default_request_timeout fs.fuse.default_request_timeout = 0Signed-off-by Joanne Koong <[email protected]>--- Documentation/admin-guide/sysctl/fs.rst | 27 +++++++++++++++++++++++++ fs/fuse/fuse_i.h | 10 +++++++++ fs/fuse/inode.c | 16 +++++++++++++-- fs/fuse/sysctl.c | 20 ++++++++++++++++++ 4 files changed, 71 insertions(+), 2 deletions(-)Documentation/admin-guide/sysctl/fs.rst +27
-0@@ -342,3 +342,30 @@ filesystems: ``/proc/sys/fs/fuse/max_pages_limit`` is a read/write file for setting/getting the maximum number of pages that can be used for servicing requests in FUSE. + +``/proc/sys/fs/fuse/default_request_timeout`` is a read/write file for +setting/getting the default timeout (in minutes) for a fuse server to +reply to a kernel-issued request in the event where the server did not +specify a timeout at mount. If the server set a timeout, +then default_request_timeout will be ignored. The default +"default_request_timeout" is set to 0. 0 indicates a no-op (eg +requests will not have a default request timeout set if no timeout was +specified by the server). + +``/proc/sys/fs/fuse/max_request_timeout`` is a read/write file for +setting/getting the maximum timeout (in minutes) for a fuse server to +reply to a kernel-issued request. A value greater than 0 automatically opts +the server into a timeout that will be at most "max_request_timeout", even if +the server did not specify a timeout and default_request_timeout is set to 0. +If max_request_timeout is greater than 0 and the server set a timeout greater +than max_request_timeout or default_request_timeout is set to a value greater +than max_request_timeout, the system will use max_request_timeout as the +timeout. 0 indicates a no-op (eg requests will not have an upper bound on the +timeout and if the server did not request a timeout and default_request_timeout +was not set, there will be no timeout). + +Please note that for the timeout options, if the server does not respond to +the request by the time the timeout elapses, then the connection to the fuse +server will be aborted. Please also note that the timeouts are not 100% +precise (eg you may set 10 minutes but the timeout may kick in after 11 +minutes).fs/fuse/fuse_i.h +10
-0@@ -46,6 +46,16 @@/** Maximum of max_pages received in init_out */ extern unsigned int fuse_max_pages_limit; +/* +* Default timeout (in minutes) for the server to reply to a request +* before the connection is aborted, if no timeout was specified on mount. +*/ +extern unsigned int fuse_default_req_timeout; +/* +* Max timeout (in minutes) for the server to reply to a request before +* the connection is aborted. +*/ +extern unsigned int fuse_max_req_timeout;/** List of active connections */ extern struct list_head fuse_conn_list;fs/fuse/inode.c +3
-0@@ -36,6 +36,9 @@ DEFINE_MUTEX(fuse_mutex);static int set_global_limit (const char *val,const struct kernel_param *kp);unsigned int fuse_max_pages_limit =256 ; +/* default is no timeout */ +unsigned int fuse_default_req_timeout =0 ; +unsigned int fuse_max_req_timeout =0 ;unsigned max_user_bgreq; module_param_call(max_user_bgreq, set_global_limit, param_get_uint,@@ -1701,8 +1704,17 @@ EXPORT_SYMBOL_GPL(fuse_init_fs_context_submount);
Quoted 11 lines
fs/fuse/inode.c EXPORT_SYMBOL_GPL(fuse_init_fs_context_submount); · line 1711
Quoted 9 lines
fs/fuse/inode.c +5
-2static void fuse_init_fc_timeout (struct fuse_conn *fc,struct fuse_fs_context *ctx) { -if (ctx->req_timeout) { -if (check_mul_overflow (ctx->req_timeout *60 ,HZ , &fc->timeout.req_timeout)) +unsigned int timeout = ctx->req_timeout ?: fuse_default_req_timeout; + +if (fuse_max_req_timeout) { +if (!timeout) + timeout = fuse_max_req_timeout;Hmm, so 'max' might be used as 'min' as well. Isn't that a bit confusing?
Interesting. Yeah, i don't think there's a good name that encapsulates the behavior of this (eg sets a min timeout if no timeout is set but sets the max timeout if a timeout is set). in my head, I think of it more as "max timeout that is automatically enforced if there's no timeout".
Thanks,
Joanne
Quoted 28 lines
Quoted 28 lines
fs/fuse/inode.c +5
-0+else + timeout = min(timeout, fuse_max_req_timeout)> + } + + if (timeout) { + if (check_mul_overflow(timeout * 60, HZ, &fc->timeout.req_timeout)) fc->timeout.req_timeout = U32_MAX;timer_setup (&fc->timeout.timer, fuse_check_timeout,0 );mod_timer (&fc->timeout.timer, jiffies +FUSE_TIMEOUT_TIMER_FREQ );fs/fuse/sysctl.c +3
-0@@ -13,6 +13,8 @@ static struct ctl_table_header *fuse_table_header;/* Bound by fuse_init_out max_pages, which is a u16 */ static unsigned int sysctl_fuse_max_pages_limit =65535 ; +static unsigned int sysctl_fuse_max_req_timeout_limit =U16_MAX ; +static struct ctl_table fuse_sysctl_table[] = { { .procname ="max_pages_limit" ,@@ -23,6 +25,24 @@ static struct ctl_table fuse_sysctl_table[] = { .extra1 = SYSCTL_ONE, .extra2 = &sysctl_fuse_max_pages_limit, }, + {
Quoted 8 lines
fs/fuse/sysctl.c static struct ctl_table fuse_sysctl_table[] = { · line 34
Quoted 6 lines
fs/fuse/sysctl.c +6
-0+ .procname = "default_request_timeout", + .data = &fuse_default_req_timeout, + .maxlen = sizeof(fuse_default_req_timeout), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO,There is slight whitespace issue here - spaces instead of tabs.
Thanks for noting - I'll fix this and the one below.
Quoted 21 lines
fs/fuse/sysctl.c static struct ctl_table fuse_sysctl_table[] = { · line 43
Quoted 11 lines
fs/fuse/sysctl.c +9
-0+ .extra2 = &sysctl_fuse_max_req_timeout_limit, + }, + { + .procname = "max_request_timeout", + .data = &fuse_max_req_timeout, + .maxlen = sizeof(fuse_max_req_timeout), + .mode = 0644, + .proc_handler = proc_douintvec_minmax, + .extra1 = SYSCTL_ZERO,And here.
fs/fuse/sysctl.c static struct ctl_table fuse_sysctl_table[] = { · line 48
Quoted 5 lines
fs/fuse/sysctl.c +2
-0+ .extra2 = &sysctl_fuse_max_req_timeout_limit, + }, };int fuse_sysctl_register (void )Reviewed-by Bernd Schubert <[email protected]>
Quoted 71 lines · Joanne Koong
There are situations where fuse servers can become unresponsive or stuck, for example if the server is in a deadlock. Currently, there's no good way to detect if a server is stuck and needs to be killed manually.
This patchset adds a timeout option where if the server does not reply to a request by the time the timeout elapses, the connection will be aborted. This patchset also adds two dynamically configurable fuse sysctls "
default_request_timeout" and "max_request_timeout" for controlling/enforcing timeout behavior system-wide.Existing systems running fuse servers will not be affected unless they explicitly opt into the timeout.
v7:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v7 -> v8:
* Use existing lists for checking expirations (Miklos)v6:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v6 -> v7:
- Make timer per-connection instead of per-request (Miklos)
- Make default granularity of time minutes instead of seconds
- Removed the reviewed-bys since the interface of this has changed (now minutes, instead of seconds)v5:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v5 -> v6:
- Gate sysctl.o behindCONFIG_SYSCTLin makefile (kernel test robot)
- Reword/clarify last sentence in cover letter (Miklos)v4:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v4 -> v5:
- Change timeout behavior from aborting request to aborting connection (Miklos)
- Clarify wording for sysctl documentation (Jingbo)v3:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v3 -> v4:
- Fix wording on some comments to make it more clear
- Use simpler logic for timer (eg remove extra if checks, use mod timer API) (Josef)
- Sanity-check should be onFR_FINISHINGnotFR_FINISHED(Jingbo)
- Fix comment for "processing queue", addreq->fpq= NULL safeguard (Bernd)v2:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v2 -> v3:
- Disarm / rearm timer indev_do_readto handle race conditions (Bernrd)
- Disarm timer in error handling for fatal interrupt (Yafang)
- Clean updo_fuse_request_end(Jingbo)
- Add timer for notify retrieve requests
- Fix kernel test robot errors for #define no-op functionsv1:
https://lore.kernel.org/linux-fsdevel/[email protected]/
Changes from v1 -> v2:
- Add timeout for background requests
- Handle resend race condition
- Add sysctlsJoanne Koong (3):
fs_parser: addfsparam_u16helper
fuse: add optional kernel-enforced timeout for requests
fuse: adddefault_request_timeoutandmax_request_timeoutsysctls
Quoted 9 lines
Documentation/admin-guide/sysctl/fs.rst | 27 +++++++++ fs/fs_parser.c | 14 +++++ fs/fuse/dev.c | 80 +++++++++++++++++++++++++ fs/fuse/fuse_i.h | 31 ++++++++++ fs/fuse/inode.c | 33 ++++++++++ fs/fuse/sysctl.c | 20 +++++++ include/linux/fs_parser.h | 9 ++- 7 files changed, 211 insertions(+), 3 deletions(-)
Just checking in on this patchset - any comments or thoughts?
Thanks,
Joanne
Quoted 3 lines
--
2.43.5