Skip to contentSkip to conversation

fanotify: Make wait for permission event response interruptible

PATCH Superseded Jan Kara 3 participants 15 messages Jan 8, 2019 – Feb 13, 2019

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Jan Kara

author maintainer
#0 lore raw

cover fanotify: Make wait for permission event response interruptible

Hello,

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation) - reported e.g. in [1]. Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event (as reported e.g. here [2]).

This patch set modifies fanotify so that it will use interruptible wait when waiting for fanotify permission event response. Patches are based on current Linus' tree for the ease of testing (I plan to rebase them on top of Amir's pending changes later). I have also create LTP test which stresses handling of permission events while sending processes signals to test the new code - I'll send that separately later. Review, comments, and testing are welcome.

[1] https://lore.kernel.org/lkml/153474898224.6806.12518115530793064797.stgit@buzz/
[2] https://lore.kernel.org/lkml/[email protected]/

Honza

author maintainer
#1 lore raw

PATCH 1/4 fanotify: Fold dequeue_event() into process_access_response()

Fold dequeue_event() into process_access_response(). This will make changes to use of ->response field easier.

---
 fs/notify/fanotify/fanotify_user.c | 41 ++++++++++++--------------------------
 1 file changed, 13 insertions(+), 28 deletions(-)
fs/notify/fanotify/fanotify_user.c +13 -28
index 9c870b0d2b56..908ebc421d15 100644
@@ -144,28 +144,6 @@ static int fill_event_metadata(struct fsnotify_group *group,
 	return ret;
 }
 
-static struct fanotify_perm_event_info *dequeue_event(
-				struct fsnotify_group *group, int fd)
-{
-	struct fanotify_perm_event_info *event, *return_e = NULL;
-
-	spin_lock(&group->notification_lock);
-	list_for_each_entry(event, &group->fanotify_data.access_list,
-			    fae.fse.list) {
-		if (event->fd != fd)
-			continue;
-
-		list_del_init(&event->fae.fse.list);
-		return_e = event;
-		break;
-	}
-	spin_unlock(&group->notification_lock);
-
-	pr_debug("%s: found return_re=%p\n", __func__, return_e);
-
-	return return_e;
-}
-
 static int process_access_response(struct fsnotify_group *group,
 				   struct fanotify_response *response_struct)
 {
@@ -194,14 +172,21 @@ static int process_access_response(struct fsnotify_group *group,
 	if ((response & FAN_AUDIT) && !FAN_GROUP_FLAG(group, FAN_ENABLE_AUDIT))
 		return -EINVAL;
 
-	event = dequeue_event(group, fd);
-	if (!event)
-		return -ENOENT;
+	spin_lock(&group->notification_lock);
+	list_for_each_entry(event, &group->fanotify_data.access_list,
+			    fae.fse.list) {
+		if (event->fd != fd)
+			continue;
 
-	event->response = response;
-	wake_up(&group->fanotify_data.access_waitq);
+		list_del_init(&event->fae.fse.list);
+		event->response = response;
+		spin_unlock(&group->notification_lock);
+		wake_up(&group->fanotify_data.access_waitq);
+		return 0;
+	}
+	spin_unlock(&group->notification_lock);
 
-	return 0;
+	return -ENOENT;
 }
 
 static ssize_t copy_event_to_user(struct fsnotify_group *group,
Signature · 1 line
2.16.4
#2 lore raw

Re: [PATCH 1/4] fanotify: Fold dequeue_event() into process_access_response()

Quoted 6 lines · Jan Kara

Fold dequeue_event() into process_access_response(). This will make changes to use of ->response field easier.

---

Looks ok

Thanks,
Amir.

author maintainer
#3 lore raw

PATCH 2/4 fanotify: Move locking inside get_one_event()

In reply to Jan Kara

get_one_event() has a single caller and that just locks notification_lock around the call. Move locking inside get_one_event() as that will make using ->response field for permission event state easier.

---
 fs/notify/fanotify/fanotify_user.c | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)
fs/notify/fanotify/fanotify_user.c +11 -15
index 908ebc421d15..2b2c8b8a17bd 100644
@@ -51,25 +51,24 @@ struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
  * Get an fsnotify notification event if one exists and is small
  * enough to fit in "count". Return an error pointer if the count
  * is not large enough.
- *
- * Called with the group->notification_lock held.
  */
 static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
 					    size_t count)
 {
-	assert_spin_locked(&group->notification_lock);
-
-	pr_debug("%s: group=%p count=%zd\n", __func__, group, count);
+	struct fsnotify_event *event = NULL;
 
+	spin_lock(&group->notification_lock);
 	if (fsnotify_notify_queue_is_empty(group))
-		return NULL;
+		goto out;
 
-	if (FAN_EVENT_METADATA_LEN > count)
-		return ERR_PTR(-EINVAL);
-
-	/* held the notification_lock the whole time, so this is the
-	 * same event we peeked above */
-	return fsnotify_remove_first_event(group);
+	if (FAN_EVENT_METADATA_LEN > count) {
+		event = ERR_PTR(-EINVAL);
+		goto out;
+	}
+	event = fsnotify_remove_first_event(group);
+out:
+	spin_unlock(&group->notification_lock);
+	return event;
 }
 
 static int create_fd(struct fsnotify_group *group,
@@ -261,10 +260,7 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
 
 	add_wait_queue(&group->notification_waitq, &wait);
 	while (1) {
-		spin_lock(&group->notification_lock);
 		kevent = get_one_event(group, count);
-		spin_unlock(&group->notification_lock);
-
 		if (IS_ERR(kevent)) {
 			ret = PTR_ERR(kevent);
 			break;
Signature · 1 line
2.16.4
#4 lore raw

Re: [PATCH 2/4] fanotify: Move locking inside get_one_event()

Quoted 18 lines · Jan Kara

get_one_event() has a single caller and that just locks notification_lock around the call. Move locking inside get_one_event() as that will make using ->response field for permission event state easier.

---
 fs/notify/fanotify/fanotify_user.c | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)
fs/notify/fanotify/fanotify_user.c +0 -0
index 908ebc421d15..2b2c8b8a17bd 100644
@@ -51,25 +51,24 @@ struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
  * Get an fsnotify notification event if one exists and is small
  * enough to fit in "count". Return an error pointer if the count

fs/notify/fanotify/fanotify_user.c struct kmem_cache *fanotify_perm_event_cachep __read_mostly; · line 62

Quoted 10 lines
fs/notify/fanotify/fanotify_user.c +0 -5
  * is not large enough.
- *
- * Called with the group->notification_lock held.
  */
 static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
                                            size_t count)
 {
-       assert_spin_locked(&group->notification_lock);
-
-       pr_debug("%s: group=%p count=%zd\n", __func__, group, count);

I see you are slowly cleaning up pr_debug calls. Any particular reason? Out of all the spam pr_debug calls in the code, this one looks rather useful.

Otherwise, looks ok.

Thanks,
Amir.

author maintainer
#5 lore raw

Re: [PATCH 2/4] fanotify: Move locking inside get_one_event()

Quoted 24 lines · Amir Goldstein
Quoted 23 lines · Jan Kara

get_one_event() has a single caller and that just locks notification_lock around the call. Move locking inside get_one_event() as that will make using ->response field for permission event state easier.

---
 fs/notify/fanotify/fanotify_user.c | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)
fs/notify/fanotify/fanotify_user.c +0 -2
index 908ebc421d15..2b2c8b8a17bd 100644
@@ -51,25 +51,24 @@ struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
  * Get an fsnotify notification event if one exists and is small
  * enough to fit in "count". Return an error pointer if the count
  * is not large enough.
- *
- * Called with the group->notification_lock held.
  */
 static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
Quoted 8 lines

fs/notify/fanotify/fanotify_user.c struct kmem_cache *fanotify_perm_event_cachep __read_mostly; · line 62

Quoted 5 lines
fs/notify/fanotify/fanotify_user.c +0 -3
                                            size_t count)
 {
-       assert_spin_locked(&group->notification_lock);
-
-       pr_debug("%s: group=%p count=%zd\n", __func__, group, count);

I see you are slowly cleaning up pr_debug calls. Any particular reason? Out of all the spam pr_debug calls in the code, this one looks rather useful.

I didn't find it useful but if you think I it, I'll leave it in.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author maintainer
#6 lore raw

PATCH 3/4 fanotify: Track permission event state

In reply to Jan Kara

Track whether permission event got already reported to userspace and whether userspace already answered to the permission event in high bits of its ->response field. Also protect stores to ->response field by group->notification_lock. This will allow aborting wait for reply to permission event from userspace.

---
 fs/notify/fanotify/fanotify.c      | 16 ++++++++++------
 fs/notify/fanotify/fanotify.h      | 10 +++++++++-
 fs/notify/fanotify/fanotify_user.c | 17 ++++++++++++-----
 3 files changed, 31 insertions(+), 12 deletions(-)
fs/notify/fanotify/fanotify.c +10 -6
index 3723f3d18d20..cca13adc3a4c 100644
@@ -62,13 +62,19 @@ static int fanotify_get_response(struct fsnotify_group *group,
 				 struct fsnotify_iter_info *iter_info)
 {
 	int ret;
+	unsigned int response;
 
+	BUILD_BUG_ON(FAN_EVENT_STATE_MASK & (FAN_AUDIT | FAN_ALLOW | FAN_DENY));
 	pr_debug("%s: group=%p event=%p\n", __func__, group, event);
 
-	wait_event(group->fanotify_data.access_waitq, event->response);
+	wait_event(group->fanotify_data.access_waitq,
+		   (event->response & FAN_EVENT_STATE_MASK) ==
+							FAN_EVENT_ANSWERED);
+
+	response = event->response & ~FAN_EVENT_STATE_MASK;
 
 	/* userspace responded, convert to something usable */
-	switch (event->response & ~FAN_AUDIT) {
+	switch (response & ~FAN_AUDIT) {
 	case FAN_ALLOW:
 		ret = 0;
 		break;
@@ -78,10 +84,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
 	}
 
 	/* Check if the response should be audited */
-	if (event->response & FAN_AUDIT)
-		audit_fanotify(event->response & ~FAN_AUDIT);
-
-	event->response = 0;
+	if (response & FAN_AUDIT)
+		audit_fanotify(response & ~FAN_AUDIT);
 
 	pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
 		 group, event, ret);
fs/notify/fanotify/fanotify.h +9 -1
index ea05b8a401e7..954d997745c3 100644
@@ -22,6 +22,12 @@ struct fanotify_event_info {
 	struct pid *pid;
 };
 
+/* State of permission event we store inside response field */
+#define FAN_EVENT_STATE_MASK 0xc0000000
+
+#define FAN_EVENT_REPORTED 0x40000000	/* Event reported to userspace */
+#define FAN_EVENT_ANSWERED 0x80000000	/* Event answered by userspace */
+
 /*
  * Structure for permission fanotify events. It gets allocated and freed in
  * fanotify_handle_event() since we wait there for user response. When the
@@ -31,7 +37,9 @@ struct fanotify_event_info {
  */
 struct fanotify_perm_event_info {
 	struct fanotify_event_info fae;
-	int response;	/* userspace answer to question */
+	unsigned int response;	/* userspace answer to the event. We also use
+				 * high bits of this field for recording state
+				 * of the event. */
 	int fd;		/* fd we passed to userspace for this event */
 };
 
fs/notify/fanotify/fanotify_user.c +12 -5
index 2b2c8b8a17bd..611c2ff50d64 100644
@@ -50,7 +50,8 @@ struct kmem_cache *fanotify_perm_event_cachep __read_mostly;
 /*
  * Get an fsnotify notification event if one exists and is small
  * enough to fit in "count". Return an error pointer if the count
- * is not large enough.
+ * is not large enough. When permission event is dequeued, its state is
+ * updated accordingly.
  */
 static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
 					    size_t count)
@@ -66,6 +67,8 @@ static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
 		goto out;
 	}
 	event = fsnotify_remove_first_event(group);
+	if (fanotify_is_perm_event(event->mask))
+		FANOTIFY_PE(event)->response = FAN_EVENT_REPORTED;
 out:
 	spin_unlock(&group->notification_lock);
 	return event;
@@ -178,7 +181,7 @@ static int process_access_response(struct fsnotify_group *group,
 			continue;
 
 		list_del_init(&event->fae.fse.list);
-		event->response = response;
+		event->response = response | FAN_EVENT_ANSWERED;
 		spin_unlock(&group->notification_lock);
 		wake_up(&group->fanotify_data.access_waitq);
 		return 0;
@@ -301,7 +304,10 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
 			fsnotify_destroy_event(group, kevent);
 		} else {
 			if (ret <= 0) {
-				FANOTIFY_PE(kevent)->response = FAN_DENY;
+				spin_lock(&group->notification_lock);
+				FANOTIFY_PE(kevent)->response =
+						FAN_DENY | FAN_EVENT_ANSWERED;
+				spin_unlock(&group->notification_lock);
 				wake_up(&group->fanotify_data.access_waitq);
 			} else {
 				spin_lock(&group->notification_lock);
@@ -372,7 +378,7 @@ static int fanotify_release(struct inode *ignored, struct file *file)
 			 event);
 
 		list_del_init(&event->fae.fse.list);
-		event->response = FAN_ALLOW;
+		event->response = FAN_ALLOW | FAN_EVENT_ANSWERED;
 	}
 
 	/*
@@ -387,7 +393,8 @@ static int fanotify_release(struct inode *ignored, struct file *file)
 			fsnotify_destroy_event(group, fsn_event);
 			spin_lock(&group->notification_lock);
 		} else {
-			FANOTIFY_PE(fsn_event)->response = FAN_ALLOW;
+			FANOTIFY_PE(fsn_event)->response =
+					FAN_ALLOW | FAN_EVENT_ANSWERED;
 		}
 	}
 	spin_unlock(&group->notification_lock);
Signature · 1 line
2.16.4
#7 lore raw

Re: [PATCH 3/4] fanotify: Track permission event state

Quoted 71 lines · Jan Kara

Track whether permission event got already reported to userspace and whether userspace already answered to the permission event in high bits of its ->response field. Also protect stores to ->response field by group->notification_lock. This will allow aborting wait for reply to permission event from userspace.

---
 fs/notify/fanotify/fanotify.c      | 16 ++++++++++------
 fs/notify/fanotify/fanotify.h      | 10 +++++++++-
 fs/notify/fanotify/fanotify_user.c | 17 ++++++++++++-----
 3 files changed, 31 insertions(+), 12 deletions(-)
fs/notify/fanotify/fanotify.c +10 -6
index 3723f3d18d20..cca13adc3a4c 100644
@@ -62,13 +62,19 @@ static int fanotify_get_response(struct fsnotify_group *group,
                                 struct fsnotify_iter_info *iter_info)
 {
        int ret;
+       unsigned int response;
 
+       BUILD_BUG_ON(FAN_EVENT_STATE_MASK & (FAN_AUDIT | FAN_ALLOW | FAN_DENY));
        pr_debug("%s: group=%p event=%p\n", __func__, group, event);
 
-       wait_event(group->fanotify_data.access_waitq, event->response);
+       wait_event(group->fanotify_data.access_waitq,
+                  (event->response & FAN_EVENT_STATE_MASK) ==
+                                                       FAN_EVENT_ANSWERED);
+
+       response = event->response & ~FAN_EVENT_STATE_MASK;
 
        /* userspace responded, convert to something usable */
-       switch (event->response & ~FAN_AUDIT) {
+       switch (response & ~FAN_AUDIT) {
        case FAN_ALLOW:
                ret = 0;
                break;
@@ -78,10 +84,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
        }
 
        /* Check if the response should be audited */
-       if (event->response & FAN_AUDIT)
-               audit_fanotify(event->response & ~FAN_AUDIT);
-
-       event->response = 0;
+       if (response & FAN_AUDIT)
+               audit_fanotify(response & ~FAN_AUDIT);
 
        pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
                 group, event, ret);
fs/notify/fanotify/fanotify.h +6 -0
index ea05b8a401e7..954d997745c3 100644
@@ -22,6 +22,12 @@ struct fanotify_event_info {
        struct pid *pid;
 };
 
+/* State of permission event we store inside response field */
+#define FAN_EVENT_STATE_MASK 0xc0000000
+
+#define FAN_EVENT_REPORTED 0x40000000  /* Event reported to userspace */
+#define FAN_EVENT_ANSWERED 0x80000000  /* Event answered by userspace */
+
 /*
  * Structure for permission fanotify events. It gets allocated and freed in
  * fanotify_handle_event() since we wait there for user response. When the

fs/notify/fanotify/fanotify.h struct fanotify_event_info { · line 42

Quoted 7 lines
fs/notify/fanotify/fanotify.h +3 -1
@@ -31,7 +37,9 @@ struct fanotify_event_info {
  */
 struct fanotify_perm_event_info {
        struct fanotify_event_info fae;
-       int response;   /* userspace answer to question */
+       unsigned int response;  /* userspace answer to the event. We also use
+                                * high bits of this field for recording state
+                                * of the event. */

What's wrong with:

short response; /* userspace answer to question */
short state; /* the state of this event */

Will make for a much simpler code/patch IMO.

Thanks,
Amir.

author maintainer
#8 lore raw

Re: [PATCH 3/4] fanotify: Track permission event state

Quoted 11 lines · Amir Goldstein

fs/notify/fanotify/fanotify.h struct fanotify_event_info { · line 42

Quoted 8 lines
fs/notify/fanotify/fanotify.h +3 -1
@@ -31,7 +37,9 @@ struct fanotify_event_info {
  */
 struct fanotify_perm_event_info {
        struct fanotify_event_info fae;
-       int response;   /* userspace answer to question */
+       unsigned int response;  /* userspace answer to the event. We also use
+                                * high bits of this field for recording state
+                                * of the event. */

What's wrong with:

Quoted 4 lines

short response; /* userspace answer to question */
short state; /* the state of this event */

Will make for a much simpler code/patch IMO.

I guess you're right. I'm somewhat cautious about types shorter than int as I do read 'state' locklessly and for shorter-than-int types compiler can do fancy things like read-modify-write cycle of a full word when storing some value in either response or state. But I think this particular case should be fine.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author maintainer
#9 lore raw

PATCH 4/4 fanotify: Use interruptible wait when waiting for permission events

In reply to Jan Kara

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation). Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event.

Fix these problems by using interruptible sleep for waiting for response to fanotify event. This is slightly tricky though - we have to detect when the event got already reported to userspace as in that case we must not free the event. Instead we push the responsibility for freeing the event to the process that will write response to the event.

---
 fs/notify/fanotify/fanotify.c      | 36 ++++++++++++++++++++++++++----
 fs/notify/fanotify/fanotify.h      |  1 +
 fs/notify/fanotify/fanotify_user.c | 45 ++++++++++++++++++++++++--------------
 fs/notify/notification.c           | 20 +++++++++++------
 include/linux/fsnotify_backend.h   |  3 +++
 5 files changed, 78 insertions(+), 27 deletions(-)
fs/notify/fanotify/fanotify.c +32 -4
index cca13adc3a4c..cb7f7c5484fc 100644
@@ -57,6 +57,13 @@ static int fanotify_merge(struct list_head *list, struct fsnotify_event *event)
 	return 0;
 }
 
+/*
+ * Wait for response to permission event. The function also takes care of
+ * freeing the permission event (or offloads that in case the wait is canceled
+ * by a signal). The function returns 0 in case access got allowed by userspace,
+ * -EPERM in case userspace disallowed the access, and -ERESTARTSYS in case
+ * the wait got interrupted by a signal.
+ */
 static int fanotify_get_response(struct fsnotify_group *group,
 				 struct fanotify_perm_event_info *event,
 				 struct fsnotify_iter_info *iter_info)
@@ -67,9 +74,31 @@ static int fanotify_get_response(struct fsnotify_group *group,
 	BUILD_BUG_ON(FAN_EVENT_STATE_MASK & (FAN_AUDIT | FAN_ALLOW | FAN_DENY));
 	pr_debug("%s: group=%p event=%p\n", __func__, group, event);
 
-	wait_event(group->fanotify_data.access_waitq,
+	ret = wait_event_interruptible(group->fanotify_data.access_waitq,
 		   (event->response & FAN_EVENT_STATE_MASK) ==
 							FAN_EVENT_ANSWERED);
+	/* Signal pending? */
+	if (ret < 0) {
+		spin_lock(&group->notification_lock);
+		/* Event reported to userspace and no answer yet? */
+		if ((event->response & FAN_EVENT_STATE_MASK) ==
+		    FAN_EVENT_REPORTED) {
+			/* Event will get freed once userspace answers to it */
+			event->response = FAN_EVENT_CANCELED;
+			spin_unlock(&group->notification_lock);
+			return ret;
+		}
+		/* Event not yet reported? Just remove it. */
+		if ((event->response & FAN_EVENT_STATE_MASK) == 0)
+			fsnotify_remove_queued_event(group, &event->fae.fse);
+		/*
+		 * Event may be also answered in case signal delivery raced
+		 * with wakeup. In that case we have nothing to do besides
+		 * freeing the event and reporting error.
+		 */
+		spin_unlock(&group->notification_lock);
+		goto out;
+	}
 
 	response = event->response & ~FAN_EVENT_STATE_MASK;
 
@@ -87,8 +116,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
 	if (response & FAN_AUDIT)
 		audit_fanotify(response & ~FAN_AUDIT);
 
-	pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
-		 group, event, ret);
+out:
+	fsnotify_destroy_event(group, &event->fae.fse);
 	
 	return ret;
 }
@@ -259,7 +288,6 @@ static int fanotify_handle_event(struct fsnotify_group *group,
 	} else if (fanotify_is_perm_event(mask)) {
 		ret = fanotify_get_response(group, FANOTIFY_PE(fsn_event),
 					    iter_info);
-		fsnotify_destroy_event(group, fsn_event);
 	}
 finish:
 	if (fanotify_is_perm_event(mask))
fs/notify/fanotify/fanotify.h +1 -0
index 954d997745c3..57286518f420 100644
@@ -27,6 +27,7 @@ struct fanotify_event_info {
 
 #define FAN_EVENT_REPORTED 0x40000000	/* Event reported to userspace */
 #define FAN_EVENT_ANSWERED 0x80000000	/* Event answered by userspace */
+#define FAN_EVENT_CANCELED 0xc0000000	/* Event got canceled by a signal */
 
 /*
  * Structure for permission fanotify events. It gets allocated and freed in
fs/notify/fanotify/fanotify_user.c +29 -16

87 lines not on this page · raw message

fs/notify/notification.c +13 -7
index 3c3e36745f59..2195a5cf745a 100644
@@ -141,6 +141,18 @@ int fsnotify_add_event(struct fsnotify_group *group,
 	return ret;
 }
 
+void fsnotify_remove_queued_event(struct fsnotify_group *group,
+				  struct fsnotify_event *event)
+{
+	assert_spin_locked(&group->notification_lock);
+	/*
+	 * We need to init list head for the case of overflow event so that
+	 * check in fsnotify_add_event() works
+	 */
+	list_del_init(&event->list);
+	group->q_len--;
+}
+
 /*
  * Remove and return the first event from the notification list.  It is the
  * responsibility of the caller to destroy the obtained event
@@ -155,13 +167,7 @@ struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group *group)
 
 	event = list_first_entry(&group->notification_list,
 				 struct fsnotify_event, list);
-	/*
-	 * We need to init list head for the case of overflow event so that
-	 * check in fsnotify_add_event() works
-	 */
-	list_del_init(&event->list);
-	group->q_len--;
-
+	fsnotify_remove_queued_event(group, event);
 	return event;
 }
 
include/linux/fsnotify_backend.h +3 -0
index 7639774e7475..cddf839bac96 100644
@@ -416,6 +416,9 @@ extern bool fsnotify_notify_queue_is_empty(struct fsnotify_group *group);
 extern struct fsnotify_event *fsnotify_peek_first_event(struct fsnotify_group *group);
 /* return AND dequeue the first event on the notification queue */
 extern struct fsnotify_event *fsnotify_remove_first_event(struct fsnotify_group *group);
+/* Remove event queued in the notification list */
+extern void fsnotify_remove_queued_event(struct fsnotify_group *group,
+					 struct fsnotify_event *event);
 
 /* functions used to manipulate the marks attached to inodes */
 
Signature · 1 line
2.16.4
#10 lore raw

Re: [PATCH 4/4] fanotify: Use interruptible wait when waiting for permission events

Quoted 9 lines · Jan Kara

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation). Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event.

Quoted 8 lines

Fix these problems by using interruptible sleep for waiting for response to fanotify event. This is slightly tricky though - we have to detect when the event got already reported to userspace as in that case we must not free the event.

Instead we push the responsibility for
freeing the event to the process that will write response to the event.

It a bit hard to follow this patch. Can we make the patch that shifts responsibility to free the event a separate patch? fsnotify_remove_queued_event() helper can tag along with it or separate patch as you wish.

Quoted 3 lines

---

I think it would be good to add reported-by tand the links you provided in cover letter in this patch as well.

...

fs/notify/fanotify/fanotify.c static int fanotify_get_response(struct fsnotify_group *group, · line 91

Quoted 7 lines
fs/notify/fanotify/fanotify.c +0 -2
@@ -87,8 +116,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
        if (response & FAN_AUDIT)
                audit_fanotify(response & ~FAN_AUDIT);
 
-       pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
-                group, event, ret);

Looks useful. Why remove?

...

Quoted 6 lines
fs/notify/fanotify/fanotify_user.c +5 -0
+static void set_event_response(struct fsnotify_group *group,
+                              struct fanotify_perm_event_info *event,
+                              unsigned int response)
+{
+       spin_lock(&group->notification_lock);

fs/notify/fanotify/fanotify_user.c static int fill_event_metadata(struct fsnotify_group *group, · line 163

Quoted 10 lines
fs/notify/fanotify/fanotify_user.c +10 -0
+       /* Waiter got aborted by a signal? Free the event. */
+       if (unlikely(event->response == FAN_EVENT_CANCELED)) {
+               spin_unlock(&group->notification_lock);
+               fsnotify_destroy_event(group, &event->fae.fse);
+               return;
+       }
+       event->response = response | FAN_EVENT_ANSWERED;
+       spin_unlock(&group->notification_lock);
+}
+

I don't understand. AFAICS, all callers of set_event_response() call immediately after spin_unlock(&group->notification_lock), without any user wait involved.
I think it makes more sense for set_event_response() to assert the lock than it is to take it.

Am I missing anything?

Thanks,
Amir.

author maintainer
#11 lore raw

Re: [PATCH 4/4] fanotify: Use interruptible wait when waiting for permission events

Quoted 15 lines · Amir Goldstein
Quoted 13 lines · Jan Kara

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation). Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event.

Fix these problems by using interruptible sleep for waiting for response to fanotify event. This is slightly tricky though - we have to detect when the event got already reported to userspace as in that case we must not free the event.

Quoted 8 lines
Quoted 3 lines

Instead we push the responsibility for
freeing the event to the process that will write response to the event.

It a bit hard to follow this patch. Can we make the patch that shifts responsibility to free the event a separate patch? fsnotify_remove_queued_event() helper can tag along with it or separate patch as you wish.

I'll have a look how to best split this. It should be possible.

Quoted 5 lines
Quoted 2 lines

---

I think it would be good to add reported-by tand the links you provided in cover letter in this patch as well.

Good point. Will do.

Quoted 9 lines

fs/notify/fanotify/fanotify.c static int fanotify_get_response(struct fsnotify_group *group, · line 91

Quoted 7 lines
fs/notify/fanotify/fanotify.c +0 -2
@@ -87,8 +116,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
        if (response & FAN_AUDIT)
                audit_fanotify(response & ~FAN_AUDIT);
 
-       pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
-                group, event, ret);

Looks useful. Why remove?

OK, I'll leave it alone.

Quoted 22 lines
Quoted 5 lines
fs/notify/fanotify/fanotify_user.c +5 -0
+static void set_event_response(struct fsnotify_group *group,
+                              struct fanotify_perm_event_info *event,
+                              unsigned int response)
+{
+       spin_lock(&group->notification_lock);

fs/notify/fanotify/fanotify_user.c static int fill_event_metadata(struct fsnotify_group *group, · line 163

Quoted 10 lines
fs/notify/fanotify/fanotify_user.c +10 -0
+       /* Waiter got aborted by a signal? Free the event. */
+       if (unlikely(event->response == FAN_EVENT_CANCELED)) {
+               spin_unlock(&group->notification_lock);
+               fsnotify_destroy_event(group, &event->fae.fse);
+               return;
+       }
+       event->response = response | FAN_EVENT_ANSWERED;
+       spin_unlock(&group->notification_lock);
+}
+

I don't understand. AFAICS, all callers of set_event_response() call immediately after spin_unlock(&group->notification_lock), without any user wait involved.
I think it makes more sense for set_event_response() to assert the lock than it is to take it.

Quoted 1 line

Am I missing anything?

In case we need to destroy the event, we want to drop the notification_lock. So to avoid a situation where set_event_response() drops a lock it did not acquire (which is not very intuitive), I've decided for the less efficient scheme of dropping and retaking the lock.

But maybe with better function name and some asserts, we could live with dropping the lock inside the function without taking it.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#12 lore raw

Re: [PATCH 4/4] fanotify: Use interruptible wait when waiting for permission events

Quoted 76 lines · Jan Kara
Quoted 15 lines · Amir Goldstein
Quoted 13 lines · Jan Kara

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation). Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event.

Fix these problems by using interruptible sleep for waiting for response to fanotify event. This is slightly tricky though - we have to detect when the event got already reported to userspace as in that case we must not free the event.

Quoted 8 lines
Quoted 3 lines

Instead we push the responsibility for
freeing the event to the process that will write response to the event.

It a bit hard to follow this patch. Can we make the patch that shifts responsibility to free the event a separate patch? fsnotify_remove_queued_event() helper can tag along with it or separate patch as you wish.

I'll have a look how to best split this. It should be possible.

Quoted 5 lines
Quoted 2 lines

---

I think it would be good to add reported-by tand the links you provided in cover letter in this patch as well.

Good point. Will do.

Quoted 9 lines

fs/notify/fanotify/fanotify.c static int fanotify_get_response(struct fsnotify_group *group, · line 91

Quoted 7 lines
fs/notify/fanotify/fanotify.c +0 -2
@@ -87,8 +116,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
        if (response & FAN_AUDIT)
                audit_fanotify(response & ~FAN_AUDIT);
 
-       pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
-                group, event, ret);

Looks useful. Why remove?

OK, I'll leave it alone.

Quoted 22 lines
Quoted 5 lines
fs/notify/fanotify/fanotify_user.c +5 -0
+static void set_event_response(struct fsnotify_group *group,
+                              struct fanotify_perm_event_info *event,
+                              unsigned int response)
+{
+       spin_lock(&group->notification_lock);

fs/notify/fanotify/fanotify_user.c static int fill_event_metadata(struct fsnotify_group *group, · line 163

Quoted 10 lines
fs/notify/fanotify/fanotify_user.c +10 -0
+       /* Waiter got aborted by a signal? Free the event. */
+       if (unlikely(event->response == FAN_EVENT_CANCELED)) {
+               spin_unlock(&group->notification_lock);
+               fsnotify_destroy_event(group, &event->fae.fse);
+               return;
+       }
+       event->response = response | FAN_EVENT_ANSWERED;
+       spin_unlock(&group->notification_lock);
+}
+

I don't understand. AFAICS, all callers of set_event_response() call immediately after spin_unlock(&group->notification_lock), without any user wait involved.
I think it makes more sense for set_event_response() to assert the lock than it is to take it.

Quoted 1 line

Am I missing anything?

In case we need to destroy the event, we want to drop the notification_lock. So to avoid a situation where set_event_response() drops a lock it did not acquire (which is not very intuitive), I've decided for the less efficient scheme of dropping and retaking the lock.

Quoted 5 lines

But maybe with better function name and some asserts, we could live with dropping the lock inside the function without taking it.

Honza

Any more progress here? Thanks for your work on this, it's a real thorn in our side here.

Signature · 5 lines
Orion Poplawski
Manager of NWRA Technical Systems          720-772-5637
NWRA, Boulder/CoRA Office             FAX: 303-415-9702
3380 Mitchell Lane                       [email protected]
Boulder, CO 80301                 https://www.nwra.com/
author maintainer
#13 lore raw

Re: [PATCH 4/4] fanotify: Use interruptible wait when waiting for permission events

Quoted 80 lines · Orion Poplawski
Quoted 79 lines · Jan Kara
Quoted 15 lines · Amir Goldstein
Quoted 13 lines · Jan Kara

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation). Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event.

Fix these problems by using interruptible sleep for waiting for response to fanotify event. This is slightly tricky though - we have to detect when the event got already reported to userspace as in that case we must not free the event.

Quoted 8 lines
Quoted 3 lines

Instead we push the responsibility for
freeing the event to the process that will write response to the event.

It a bit hard to follow this patch. Can we make the patch that shifts responsibility to free the event a separate patch? fsnotify_remove_queued_event() helper can tag along with it or separate patch as you wish.

I'll have a look how to best split this. It should be possible.

Quoted 5 lines
Quoted 2 lines

---

I think it would be good to add reported-by tand the links you provided in cover letter in this patch as well.

Good point. Will do.

Quoted 9 lines

fs/notify/fanotify/fanotify.c static int fanotify_get_response(struct fsnotify_group *group, · line 91

Quoted 7 lines
fs/notify/fanotify/fanotify.c +0 -2
@@ -87,8 +116,8 @@ static int fanotify_get_response(struct fsnotify_group *group,
        if (response & FAN_AUDIT)
                audit_fanotify(response & ~FAN_AUDIT);
 
-       pr_debug("%s: group=%p event=%p about to return ret=%d\n", __func__,
-                group, event, ret);

Looks useful. Why remove?

OK, I'll leave it alone.

Quoted 22 lines
Quoted 5 lines
fs/notify/fanotify/fanotify_user.c +5 -0
+static void set_event_response(struct fsnotify_group *group,
+                              struct fanotify_perm_event_info *event,
+                              unsigned int response)
+{
+       spin_lock(&group->notification_lock);

fs/notify/fanotify/fanotify_user.c static int fill_event_metadata(struct fsnotify_group *group, · line 163

Quoted 10 lines
fs/notify/fanotify/fanotify_user.c +10 -0
+       /* Waiter got aborted by a signal? Free the event. */
+       if (unlikely(event->response == FAN_EVENT_CANCELED)) {
+               spin_unlock(&group->notification_lock);
+               fsnotify_destroy_event(group, &event->fae.fse);
+               return;
+       }
+       event->response = response | FAN_EVENT_ANSWERED;
+       spin_unlock(&group->notification_lock);
+}
+

I don't understand. AFAICS, all callers of set_event_response() call immediately after spin_unlock(&group->notification_lock), without any user wait involved.
I think it makes more sense for set_event_response() to assert the lock than it is to take it.

Quoted 1 line

Am I missing anything?

In case we need to destroy the event, we want to drop the notification_lock. So to avoid a situation where set_event_response() drops a lock it did not acquire (which is not very intuitive), I've decided for the less efficient scheme of dropping and retaking the lock.

But maybe with better function name and some asserts, we could live with dropping the lock inside the function without taking it.

Quoted 6 lines
Quoted 2 lines

Honza

Any more progress here? Thanks for your work on this, it's a real thorn in our side here.

I've sent v2 of the patches today to push things further.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author maintainer
#14 lore raw
In reply to Jan Kara
Quoted 17 lines · Jan Kara

Hello,

When waiting for response to fanotify permission events, we currently use uninterruptible waits. That makes code simple however it can cause lots of processes to end up in uninterruptible sleep with hard reboot being the only alternative in case fanotify listener process stops responding (e.g. due to a bug in its implementation) - reported e.g. in [1]. Uninterruptible sleep also makes system hibernation fail if the listener gets frozen before the process generating fanotify permission event (as reported e.g. here [2]).

This patch set modifies fanotify so that it will use interruptible wait when waiting for fanotify permission event response. Patches are based on current Linus' tree for the ease of testing (I plan to rebase them on top of Amir's pending changes later). I have also create LTP test which stresses handling of permission events while sending processes signals to test the new code - I'll send that separately later. Review, comments, and testing are welcome.

Quoted 3 lines

[1] https://lore.kernel.org/lkml/153474898224.6806.12518115530793064797.stgit@buzz/
[2] https://lore.kernel.org/lkml/[email protected]/

Patch adding LTP test I have created for excercising new code is attached.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Quoted 1 line

From 53bdff3951596f8084127998c085d4ff94667873 Mon Sep 17 00:00:00 2001

From: Jan Kara <[email protected]>
Date: Tue, 8 Jan 2019 17:01:40 +0100
Subject: [PATCH] syscalls/fanotify12: New test to test signal handling for
permission events

Test whether kernel survives when processes waiting for response to fanotify permission event are interrupted by a signal.

---
 testcases/kernel/syscalls/fanotify/.gitignore   |   1 +
 testcases/kernel/syscalls/fanotify/fanotify12.c | 217 ++++++++++++++++++++++++
 2 files changed, 218 insertions(+)
 create mode 100644 testcases/kernel/syscalls/fanotify/fanotify12.c
testcases/kernel/syscalls/fanotify/.gitignore +1 -0
index 3818e241ffcf..d583d459b4aa 100644
@@ -9,3 +9,4 @@
 /fanotify09
 /fanotify10
 /fanotify11
+/fanotify12
testcases/kernel/syscalls/fanotify/fanotify12.c new file +217 -0

217 lines not on this page · raw message

Signature · 1 line
2.16.4