Skip to contentSkip to conversation

Drivers: hv: utils: prevent crash when a utility driver is disabled host side

PATCH Superseded Vitaly Kuznetsov 2 participants 7 messages Nov 12, 2015 – Nov 16, 2015 Hyper-V/Azure core and drivers

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Drivers: hv: utils: prevent crash when a utility driver is disabled host side

I'm observing a crash when a utility driver is disabled host side (e.g. 'Guest services' is disabled live) when we have userspace daemon connected:

[   90.244859] general protection fault: 0000 [#1] SMP
...
[   90.800082] CPU: 3 PID: 473 Comm: hypervfcopyd Not tainted 4.3.0-rc7_netvsc_noalloc+ #1053
...
[   90.800082] Call Trace:
[   90.800082]  [<ffffffff81187008>] __fput+0xc8/0x1f0
[   90.800082]  [<ffffffff8118716e>] ____fput+0xe/0x10
[   90.800082]  [<ffffffff8107e4b3>] task_work_run+0x73/0x90
[   90.800082]  [<ffffffff81066995>] do_exit+0x335/0xa90
[   90.800082]  [<ffffffff8106716f>] do_group_exit+0x3f/0xc0
[   90.800082]  [<ffffffff81071abe>] get_signal+0x25e/0x5e0
[   90.800082]  [<ffffffff81015278>] do_signal+0x28/0x580
[   90.800082]  [<ffffffff81086656>] ? finish_task_switch+0xa6/0x180
[   90.800082]  [<ffffffff81443ebf>] ? __schedule+0x28f/0x870
[   90.800082]  [<ffffffffa01ebbaa>] ? hvt_op_read+0x12a/0x140 [hv_utils]
[   90.800082]  [<ffffffff8109eca0>] ? wake_atomic_t_function+0x70/0x70
...
[   90.800082] RIP  [<ffffffff810d3866>] module_put+0x16/0x90
[   90.800082]  RSP <ffff88003eb1bb88>
[   95.734261] ---[ end trace 0e09af6a6294a668 ]---

The problem is that hvutil_transport_destroy() which does misc_deregister() freeing the appropriate device is reachable by two paths: module unload and from util_remove(). While module unload path is protected by .owner in struct file_operations util_remove() path is not. Freeing the device while someone holds an open fd for it is a show stopper.

In general, it is not possible to revoke an fd from all users so the only way to solve the issue is to defer freeing the hvutil_transport structure asking the daemon to exit gracefully by responding -EBADF to all operations on unload.

Patch 1 fixes an unrelated issue I spotted, patch 2 renames outmsg_lock to 'lock' as we're gonna use it to protect test-and-set operations on 'mode', patch 3 introduces HVUTIL_TRANSPORT_DESTROY mode of operation, patch 4 fixes the issue itself.

Patches are rebased on previously sent Olaf's fixes.

Vitaly Kuznetsov (4):
Drivers: hv: utils: fix memory leak on on_msg() failure
Drivers: hv: utils: rename outmsg_lock
Drivers: hv: utils: introduce HVUTIL_TRANSPORT_DESTROY mode
Drivers: hv: utils: fix crash when device is removed from host side

 drivers/hv/hv_utils_transport.c | 110 +++++++++++++++++++++++++++++++---------
 drivers/hv/hv_utils_transport.h |   3 +-
 2 files changed, 88 insertions(+), 25 deletions(-)
Signature · 1 line
2.4.3
#1 lore raw

PATCH 1/4 Drivers: hv: utils: fix memory leak on on_msg() failure

inmsg should be freed in case of on_msg() failure to avoid memory leak.

---
 drivers/hv/hv_utils_transport.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/hv/hv_utils_transport.c +3 -2
index 24b2766..1d20451 100644
@@ -77,6 +77,7 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 {
 	struct hvutil_transport *hvt;
 	u8 *inmsg;
+	ssize_t ret = count;
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
@@ -85,10 +86,10 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 		return PTR_ERR(inmsg);
 
 	if (hvt->on_msg(inmsg, count))
-		return -EFAULT;
+		ret = -EFAULT;
 	kfree(inmsg);
 
-	return count;
+	return ret;
 }
 
 static unsigned int hvt_op_poll(struct file *file, poll_table *wait)
Signature · 1 line
2.4.3
#2 lore raw

Re: [PATCH 1/4] Drivers: hv: utils: fix memory leak on on_msg() failure

drivers/hv/hv_utils_transport.c static ssize_t hvt_op_write(struct file *file, const char __user *buf, · line 89

Quoted 6 lines · Vitaly Kuznetsov
drivers/hv/hv_utils_transport.c +1 -1
@@ -85,10 +86,10 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 		return PTR_ERR(inmsg);
 
 	if (hvt->on_msg(inmsg, count))
-		return -EFAULT;
+		ret = -EFAULT;

You fix this leak and then re-introduce it again directly in patch 3/4. Also it might be nice to preserve the error code.

	ret = hvt->on_msg(inmsg, count);

	kfree(inmsg);

	return ret ? ret : count;

regards,
dan carpenter

#3 lore raw

Re: [PATCH 1/4] Drivers: hv: utils: fix memory leak on on_msg() failure

Quoted 10 lines · Dan Carpenter

drivers/hv/hv_utils_transport.c static ssize_t hvt_op_write(struct file *file, const char __user *buf, · line 89

Quoted 6 lines · Vitaly Kuznetsov
drivers/hv/hv_utils_transport.c +1 -1
@@ -85,10 +86,10 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 		return PTR_ERR(inmsg);
 
 	if (hvt->on_msg(inmsg, count))
-		return -EFAULT;
+		ret = -EFAULT;

You fix this leak and then re-introduce it again directly in patch 3/4.

Such a shame ... Thanks for noticing!

Quoted 7 lines

Also it might be nice to preserve the error code.

	ret = hvt->on_msg(inmsg, count);

	kfree(inmsg);

	return ret ? ret : count;

It seems on_msg() hooks can only return -EINVAL now but why not ... let's do it in v2.

Signature · 1 line
  Vitaly
#4 lore raw

PATCH 2/4 Drivers: hv: utils: rename outmsg_lock

In reply to Vitaly Kuznetsov

As a preparation to reusing outmsg_lock to protect test-and-set openrations on 'mode' rename it the more general 'lock'.

---
 drivers/hv/hv_utils_transport.c | 14 +++++++-------
 drivers/hv/hv_utils_transport.h |  2 +-
 2 files changed, 8 insertions(+), 8 deletions(-)
drivers/hv/hv_utils_transport.c +7 -7
index 1d20451..5cce2d2 100644
@@ -27,11 +27,11 @@ static struct list_head hvt_list = LIST_HEAD_INIT(hvt_list);
 
 static void hvt_reset(struct hvutil_transport *hvt)
 {
-	mutex_lock(&hvt->outmsg_lock);
+	mutex_lock(&hvt->lock);
 	kfree(hvt->outmsg);
 	hvt->outmsg = NULL;
 	hvt->outmsg_len = 0;
-	mutex_unlock(&hvt->outmsg_lock);
+	mutex_unlock(&hvt->lock);
 	if (hvt->on_reset)
 		hvt->on_reset();
 }
@@ -47,7 +47,7 @@ static ssize_t hvt_op_read(struct file *file, char __user *buf,
 	if (wait_event_interruptible(hvt->outmsg_q, hvt->outmsg_len > 0))
 		return -EINTR;
 
-	mutex_lock(&hvt->outmsg_lock);
+	mutex_lock(&hvt->lock);
 	if (!hvt->outmsg) {
 		ret = -EAGAIN;
 		goto out_unlock;
@@ -68,7 +68,7 @@ static ssize_t hvt_op_read(struct file *file, char __user *buf,
 	hvt->outmsg_len = 0;
 
 out_unlock:
-	mutex_unlock(&hvt->outmsg_lock);
+	mutex_unlock(&hvt->lock);
 	return ret;
 }
 
@@ -197,7 +197,7 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 		return ret;
 	}
 	/* HVUTIL_TRANSPORT_CHARDEV */
-	mutex_lock(&hvt->outmsg_lock);
+	mutex_lock(&hvt->lock);
 	if (hvt->outmsg) {
 		/* Previous message wasn't received */
 		ret = -EFAULT;
@@ -211,7 +211,7 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 	} else
 		ret = -ENOMEM;
 out_unlock:
-	mutex_unlock(&hvt->outmsg_lock);
+	mutex_unlock(&hvt->lock);
 	return ret;
 }
 
@@ -242,7 +242,7 @@ struct hvutil_transport *hvutil_transport_init(const char *name,
 	hvt->mdev.fops = &hvt->fops;
 
 	init_waitqueue_head(&hvt->outmsg_q);
-	mutex_init(&hvt->outmsg_lock);
+	mutex_init(&hvt->lock);
 
 	spin_lock(&hvt_list_lock);
 	list_add(&hvt->list, &hvt_list);
drivers/hv/hv_utils_transport.h +1 -1
index 314c76c..bff4c92 100644
@@ -38,7 +38,7 @@ struct hvutil_transport {
 	u8 *outmsg;                         /* message to the userspace */
 	int outmsg_len;                     /* its length */
 	wait_queue_head_t outmsg_q;         /* poll/read wait queue */
-	struct mutex outmsg_lock;           /* protects outmsg */
+	struct mutex lock;                  /* protects struct members */
 };
 
 struct hvutil_transport *hvutil_transport_init(const char *name,
Signature · 1 line
2.4.3
#5 lore raw

PATCH 3/4 Drivers: hv: utils: introduce HVUTIL_TRANSPORT_DESTROY mode

In reply to Vitaly Kuznetsov

When Hyper-V host asks us to remove some util driver by closing the appropriate channel there is no easy way to force the current file descriptor holder to hang up but we can start to respond -EBADF to all operations asking it to exit gracefully.

As we're setting hvt->mode from two separate contexts now we need to use a proper locking.

---
 drivers/hv/hv_utils_transport.c | 69 ++++++++++++++++++++++++++++++++---------
 drivers/hv/hv_utils_transport.h |  1 +
 2 files changed, 56 insertions(+), 14 deletions(-)
drivers/hv/hv_utils_transport.c +55 -14
index 5cce2d2..984cba5 100644
@@ -27,11 +27,9 @@ static struct list_head hvt_list = LIST_HEAD_INIT(hvt_list);
 
 static void hvt_reset(struct hvutil_transport *hvt)
 {
-	mutex_lock(&hvt->lock);
 	kfree(hvt->outmsg);
 	hvt->outmsg = NULL;
 	hvt->outmsg_len = 0;
-	mutex_unlock(&hvt->lock);
 	if (hvt->on_reset)
 		hvt->on_reset();
 }
@@ -44,10 +42,17 @@ static ssize_t hvt_op_read(struct file *file, char __user *buf,
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
-	if (wait_event_interruptible(hvt->outmsg_q, hvt->outmsg_len > 0))
+	if (wait_event_interruptible(hvt->outmsg_q, hvt->outmsg_len > 0 ||
+				     hvt->mode != HVUTIL_TRANSPORT_CHARDEV))
 		return -EINTR;
 
 	mutex_lock(&hvt->lock);
+
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY) {
+		ret = -EBADF;
+		goto out_unlock;
+	}
+
 	if (!hvt->outmsg) {
 		ret = -EAGAIN;
 		goto out_unlock;
@@ -85,6 +90,9 @@ static ssize_t hvt_op_write(struct file *file, const char __user *buf,
 	if (IS_ERR(inmsg))
 		return PTR_ERR(inmsg);
 
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY)
+		return -EBADF;
+
 	if (hvt->on_msg(inmsg, count))
 		ret = -EFAULT;
 	kfree(inmsg);
@@ -99,6 +107,10 @@ static unsigned int hvt_op_poll(struct file *file, poll_table *wait)
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
 	poll_wait(file, &hvt->outmsg_q, wait);
+
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY)
+		return -EBADF;
+
 	if (hvt->outmsg_len > 0)
 		return POLLIN | POLLRDNORM;
 
@@ -108,26 +120,39 @@ static unsigned int hvt_op_poll(struct file *file, poll_table *wait)
 static int hvt_op_open(struct inode *inode, struct file *file)
 {
 	struct hvutil_transport *hvt;
+	int ret = 0;
+	bool issue_reset = false;
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
-	/*
-	 * Switching to CHARDEV mode. We switch bach to INIT when device
-	 * gets released.
-	 */
-	if (hvt->mode == HVUTIL_TRANSPORT_INIT)
+	mutex_lock(&hvt->lock);
+
+	if (hvt->mode == HVUTIL_TRANSPORT_DESTROY) {
+		ret = -EBADF;
+	} else if (hvt->mode == HVUTIL_TRANSPORT_INIT) {
+		/*
+		 * Switching to CHARDEV mode. We switch bach to INIT when
+		 * device gets released.
+		 */
 		hvt->mode = HVUTIL_TRANSPORT_CHARDEV;
+	}
 	else if (hvt->mode == HVUTIL_TRANSPORT_NETLINK) {
 		/*
 		 * We're switching from netlink communication to using char
 		 * device. Issue the reset first.
 		 */
-		hvt_reset(hvt);
+		issue_reset = true;
 		hvt->mode = HVUTIL_TRANSPORT_CHARDEV;
-	} else
-		return -EBUSY;
+	} else {
+		ret = -EBUSY;
+	}
 
-	return 0;
+	if (issue_reset)
+		hvt_reset(hvt);
+
+	mutex_unlock(&hvt->lock);
+
+	return ret;
 }
 
 static int hvt_op_release(struct inode *inode, struct file *file)
@@ -136,12 +161,15 @@ static int hvt_op_release(struct inode *inode, struct file *file)
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
-	hvt->mode = HVUTIL_TRANSPORT_INIT;
+	mutex_lock(&hvt->lock);
+	if (hvt->mode != HVUTIL_TRANSPORT_DESTROY)
+		hvt->mode = HVUTIL_TRANSPORT_INIT;
 	/*
 	 * Cleanup message buffers to avoid spurious messages when the daemon
 	 * connects back.
 	 */
 	hvt_reset(hvt);
+	mutex_unlock(&hvt->lock);
 
 	return 0;
 }
@@ -168,6 +196,7 @@ static void hvt_cn_callback(struct cn_msg *msg, struct netlink_skb_parms *nsp)
 	 * Switching to NETLINK mode. Switching to CHARDEV happens when someone
 	 * opens the device.
 	 */
+	mutex_lock(&hvt->lock);
 	if (hvt->mode == HVUTIL_TRANSPORT_INIT)
 		hvt->mode = HVUTIL_TRANSPORT_NETLINK;
 
@@ -175,6 +204,7 @@ static void hvt_cn_callback(struct cn_msg *msg, struct netlink_skb_parms *nsp)
 		hvt_found->on_msg(msg->data, msg->len);
 	else
 		pr_warn("hvt_cn_callback: unexpected netlink message!\n");
+	mutex_unlock(&hvt->lock);
 }
 
 int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
@@ -182,7 +212,8 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 	struct cn_msg *cn_msg;
 	int ret = 0;
 
-	if (hvt->mode == HVUTIL_TRANSPORT_INIT) {
+	if (hvt->mode == HVUTIL_TRANSPORT_INIT ||
+	    hvt->mode == HVUTIL_TRANSPORT_DESTROY) {
 		return -EINVAL;
 	} else if (hvt->mode == HVUTIL_TRANSPORT_NETLINK) {
 		cn_msg = kzalloc(sizeof(*cn_msg) + len, GFP_ATOMIC);
@@ -198,6 +229,11 @@ int hvutil_transport_send(struct hvutil_transport *hvt, void *msg, int len)
 	}
 	/* HVUTIL_TRANSPORT_CHARDEV */
 	mutex_lock(&hvt->lock);
+	if (hvt->mode != HVUTIL_TRANSPORT_CHARDEV) {
+		ret = -EINVAL;
+		goto out_unlock;
+	}
+
 	if (hvt->outmsg) {
 		/* Previous message wasn't received */
 		ret = -EFAULT;
@@ -268,6 +304,11 @@ err_free_hvt:
 
 void hvutil_transport_destroy(struct hvutil_transport *hvt)
 {
+	mutex_lock(&hvt->lock);
+	hvt->mode = HVUTIL_TRANSPORT_DESTROY;
+	wake_up_interruptible(&hvt->outmsg_q);
+	mutex_unlock(&hvt->lock);
+
 	spin_lock(&hvt_list_lock);
 	list_del(&hvt->list);
 	spin_unlock(&hvt_list_lock);
drivers/hv/hv_utils_transport.h +1 -0
index bff4c92..06254a1 100644
@@ -25,6 +25,7 @@ enum hvutil_transport_mode {
 	HVUTIL_TRANSPORT_INIT = 0,
 	HVUTIL_TRANSPORT_NETLINK,
 	HVUTIL_TRANSPORT_CHARDEV,
+	HVUTIL_TRANSPORT_DESTROY,
 };
 
 struct hvutil_transport {
Signature · 1 line
2.4.3
#6 lore raw

PATCH 4/4 Drivers: hv: utils: fix crash when device is removed from host side

In reply to Vitaly Kuznetsov

The crash is observed when a service is being disabled host side while userspace daemon is connected to the device:

[   90.244859] general protection fault: 0000 [#1] SMP
...
[   90.800082] Call Trace:
[   90.800082]  [<ffffffff81187008>] __fput+0xc8/0x1f0
[   90.800082]  [<ffffffff8118716e>] ____fput+0xe/0x10
...
[   90.800082]  [<ffffffff81015278>] do_signal+0x28/0x580
[   90.800082]  [<ffffffff81086656>] ? finish_task_switch+0xa6/0x180
[   90.800082]  [<ffffffff81443ebf>] ? __schedule+0x28f/0x870
[   90.800082]  [<ffffffffa01ebbaa>] ? hvt_op_read+0x12a/0x140 [hv_utils]

...

The problem is that hvutil_transport_destroy() which does misc_deregister() freeing the appropriate device is reachable by two paths: module unload and from util_remove(). While module unload path is protected by .owner in struct file_operations util_remove() path is not. Freeing the device while someone holds an open fd for it is a show stopper.

In general, it is not possible to revoke an fd from all users so the only way to solve the issue is to defer freeing the hvutil_transport structure.

---
 drivers/hv/hv_utils_transport.c | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
drivers/hv/hv_utils_transport.c +23 -3
index 984cba5..174c72a 100644
@@ -155,13 +155,22 @@ static int hvt_op_open(struct inode *inode, struct file *file)
 	return ret;
 }
 
+static void hvt_transport_free(struct hvutil_transport *hvt)
+{
+	misc_deregister(&hvt->mdev);
+	kfree(hvt->outmsg);
+	kfree(hvt);
+}
+
 static int hvt_op_release(struct inode *inode, struct file *file)
 {
 	struct hvutil_transport *hvt;
+	int mode_old;
 
 	hvt = container_of(file->f_op, struct hvutil_transport, fops);
 
 	mutex_lock(&hvt->lock);
+	mode_old = hvt->mode;
 	if (hvt->mode != HVUTIL_TRANSPORT_DESTROY)
 		hvt->mode = HVUTIL_TRANSPORT_INIT;
 	/*
@@ -171,6 +180,9 @@ static int hvt_op_release(struct inode *inode, struct file *file)
 	hvt_reset(hvt);
 	mutex_unlock(&hvt->lock);
 
+	if (mode_old == HVUTIL_TRANSPORT_DESTROY)
+		hvt_transport_free(hvt);
+
 	return 0;
 }
 
@@ -304,17 +316,25 @@ err_free_hvt:
 
 void hvutil_transport_destroy(struct hvutil_transport *hvt)
 {
+	int mode_old;
+
 	mutex_lock(&hvt->lock);
+	mode_old = hvt->mode;
 	hvt->mode = HVUTIL_TRANSPORT_DESTROY;
 	wake_up_interruptible(&hvt->outmsg_q);
 	mutex_unlock(&hvt->lock);
 
+	/*
+	 * In case we were in 'chardev' mode we still have an open fd so we
+	 * have to defer freeing the device. Netlink interface can be freed
+	 * now.
+	 */
 	spin_lock(&hvt_list_lock);
 	list_del(&hvt->list);
 	spin_unlock(&hvt_list_lock);
 	if (hvt->cn_id.idx > 0 && hvt->cn_id.val > 0)
 		cn_del_callback(&hvt->cn_id);
-	misc_deregister(&hvt->mdev);
-	kfree(hvt->outmsg);
-	kfree(hvt);
+
+	if (mode_old != HVUTIL_TRANSPORT_CHARDEV)
+		hvt_transport_free(hvt);
 }
Signature · 1 line
2.4.3