Skip to contentSkip to conversation

usbip: fix vhci races in connection tear down

PATCH In review Shuah Khan 2 participants 4 messages Mar 12, 2021 1 message not in the archive

Subsystems (2)
USB USB over IP driver
lore
Share & tools
lore

Shuah Khan

author maintainer
#0 lore raw

PATCH 1/1 usbip: fix vhci races in connection tear down

vhci_shutdown_connection() references connection state (tcp_socket, tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the lock.

Current connection tear down sequence:
Step 1: shutdown the socket
Step 2: stop rx thread and reset tcp_rx pointer
Step 3: stop tx thread and reset tcp_tx pointer
Step 4: Reset tcp_socket and sockfd

There are several race windows between these steps. In addition, device reset routine (vhci_device_reset) resets tcp_socket and sockfd holding the lock.

Fix these races:
- Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs only once.
- Change attach_store() to initialize in_disconnect to false while initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd)
- Change vhci_shutdown_connection() to check in_disconnect and bail out if disconnect is in progress.
- Change vhci_shutdown_connection() to
-- hold lock to save connection state pointers and unlock. -- Shutdown the socket and stop threads.
-- Hold lock to clear connection status and in_disconnect flag.
- Change vhci_device_reset() to reset tcp_socket and sockfd. if !in_disconnect

Tested syzbot and the reproducer did not trigger any issue.

Reported-and-tested-by [email protected]
Signed-off-by Shuah Khan <[email protected]>
---
 drivers/usb/usbip/usbip_common.h |  1 +
 drivers/usb/usbip/vhci_hcd.c     | 55 +++++++++++++++++++++++---------
 drivers/usb/usbip/vhci_sysfs.c   |  4 +++
 3 files changed, 45 insertions(+), 15 deletions(-)
drivers/usb/usbip/usbip_common.h +1 -0
index d60ce17d3dd2..f6261c5a8c91 100644
@@ -268,6 +268,7 @@ struct usbip_device {
 
 	struct task_struct *tcp_rx;
 	struct task_struct *tcp_tx;
+	bool in_disconnect; /* run device disconnect just once */
 
 	unsigned long event;
 	wait_queue_head_t eh_waitq;
drivers/usb/usbip/vhci_hcd.c +40 -15

85 lines not on this page · raw message

drivers/usb/usbip/vhci_sysfs.c +4 -0
index c4b4256e5dad..c4457026d5ad 100644
@@ -412,6 +412,10 @@ static ssize_t attach_store(struct device *dev, struct device_attribute *attr,
 	vdev->ud.tcp_rx     = tcp_rx;
 	vdev->ud.tcp_tx     = tcp_tx;
 	vdev->ud.status     = VDEV_ST_NOTASSIGNED;
+
+	/* used to run socket shutdown just once */
+	vdev->ud.in_disconnect = false;
+
 	usbip_kcov_handle_init(&vdev->ud);
 
 	spin_unlock(&vdev->ud.lock);
Signature · 1 line
2.27.0
#1 lore raw
Quoted 46 lines · Shuah Khan

vhci_shutdown_connection() references connection state (tcp_socket, tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the lock.

Current connection tear down sequence:
Step 1: shutdown the socket
Step 2: stop rx thread and reset tcp_rx pointer
Step 3: stop tx thread and reset tcp_tx pointer
Step 4: Reset tcp_socket and sockfd

There are several race windows between these steps. In addition, device reset routine (vhci_device_reset) resets tcp_socket and sockfd holding the lock.

Fix these races:
- Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs only once.
- Change attach_store() to initialize in_disconnect to false while initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd)
- Change vhci_shutdown_connection() to check in_disconnect and bail out if disconnect is in progress.
- Change vhci_shutdown_connection() to
-- hold lock to save connection state pointers and unlock. -- Shutdown the socket and stop threads.
-- Hold lock to clear connection status and in_disconnect flag.
- Change vhci_device_reset() to reset tcp_socket and sockfd. if !in_disconnect

Tested syzbot and the reproducer did not trigger any issue.

Reported-and-tested-by [email protected]
Signed-off-by Shuah Khan <[email protected]>
---
 drivers/usb/usbip/usbip_common.h |  1 +
 drivers/usb/usbip/vhci_hcd.c     | 55 +++++++++++++++++++++++---------
 drivers/usb/usbip/vhci_sysfs.c   |  4 +++
 3 files changed, 45 insertions(+), 15 deletions(-)
drivers/usb/usbip/vhci_hcd.c +0 -0
index 3209b5ddd30c..c1917efe5737 100644
@@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev)
 static void vhci_shutdown_connection(struct usbip_device *ud)
 {
 	struct vhci_device *vdev = container_of(ud, struct vhci_device, ud);

drivers/usb/usbip/vhci_hcd.c static void vhci_device_unlink_cleanup(struct vhci_device *vdev) · line 1019

Quoted 10 lines
drivers/usb/usbip/vhci_hcd.c +10 -0
+	unsigned long flags;
+	struct socket *socket;
+	struct task_struct *tcp_rx = NULL;
+	struct task_struct *tcp_tx = NULL;
+	int sockfd = 0;
+
+	spin_lock_irqsave(&ud->lock, flags);
+	if (vdev->ud.in_disconnect) {
+		pr_info("%s: Disconnect in progress for sockfd %d\n",
+			__func__, ud->sockfd);

Looks like you forgot to remove all you debug printks like this one before submitting.

drivers/usb/usbip/vhci_hcd.c static void vhci_device_unlink_cleanup(struct vhci_device *vdev) · line 1022

Quoted 3 lines
drivers/usb/usbip/vhci_hcd.c +3 -0
+		spin_unlock_irqrestore(&ud->lock, flags);
+		return;
+	}

Johan

author maintainer
#2 lore raw
Quoted 52 lines · Johan Hovold
Quoted 51 lines · Shuah Khan

vhci_shutdown_connection() references connection state (tcp_socket, tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the lock.

Current connection tear down sequence:
Step 1: shutdown the socket
Step 2: stop rx thread and reset tcp_rx pointer
Step 3: stop tx thread and reset tcp_tx pointer
Step 4: Reset tcp_socket and sockfd

There are several race windows between these steps. In addition, device reset routine (vhci_device_reset) resets tcp_socket and sockfd holding the lock.

Fix these races:
- Introduce in_disconnect flag to ensure vhci_shutdown_connection() runs only once.
- Change attach_store() to initialize in_disconnect to false while initializing connection status (tcp_socket, tcp_rx, tcp_tx, sockfd)
- Change vhci_shutdown_connection() to check in_disconnect and bail out if disconnect is in progress.
- Change vhci_shutdown_connection() to
-- hold lock to save connection state pointers and unlock. -- Shutdown the socket and stop threads.
-- Hold lock to clear connection status and in_disconnect flag.
- Change vhci_device_reset() to reset tcp_socket and sockfd. if !in_disconnect

Tested syzbot and the reproducer did not trigger any issue.

Reported-and-tested-by [email protected]
Signed-off-by Shuah Khan <[email protected]>
---
  drivers/usb/usbip/usbip_common.h |  1 +
  drivers/usb/usbip/vhci_hcd.c     | 55 +++++++++++++++++++++++---------
  drivers/usb/usbip/vhci_sysfs.c   |  4 +++
  3 files changed, 45 insertions(+), 15 deletions(-)
drivers/usb/usbip/vhci_hcd.c +5 -0
index 3209b5ddd30c..c1917efe5737 100644
@@ -1007,31 +1007,54 @@ static void vhci_device_unlink_cleanup(struct vhci_device *vdev)
  static void vhci_shutdown_connection(struct usbip_device *ud)
  {
  	struct vhci_device *vdev = container_of(ud, struct vhci_device, ud);
+	unsigned long flags;
+	struct socket *socket;
+	struct task_struct *tcp_rx = NULL;
+	struct task_struct *tcp_tx = NULL;
+	int sockfd = 0;
Quoted 9 lines

drivers/usb/usbip/vhci_hcd.c static void vhci_device_unlink_cleanup(struct vhci_device *vdev) · line 1019

Quoted 5 lines
drivers/usb/usbip/vhci_hcd.c +5 -0
+
+	spin_lock_irqsave(&ud->lock, flags);
+	if (vdev->ud.in_disconnect) {
+		pr_info("%s: Disconnect in progress for sockfd %d\n",
+			__func__, ud->sockfd);

Looks like you forgot to remove all you debug printks like this one before submitting.

Some printks were already in there and helped with debug. Yes I added a few more when I submitted for syzbot testing.

I will clean them up i v2.

thanks,
-- Shuah

A message that is not in the archive (sent off-list or not yet received): [email protected]

author maintainer
#4 lore raw
Quoted 11 lines · Hillf Danton
Quoted 10 lines · On Thu, 11 Mar 2021 19:27:37 -0700 Shuah Khan wrote:

vhci_shutdown_connection() references connection state (tcp_socket, tcp_rx, tcp_tx, sockfd) saved in usbpip_device without holding the lock.

Current connection tear down sequence:
Step 1: shutdown the socket
Step 2: stop rx thread and reset tcp_rx pointer
Step 3: stop tx thread and reset tcp_tx pointer
Step 4: Reset tcp_socket and sockfd

Quoted 7 lines
Quoted 3 lines

There are several race windows between these steps. In addition, device reset routine (vhci_device_reset) resets tcp_socket and sockfd holding the lock.

Can you specify the scenario where reset runs in race with teardown as both are parts of usbip_work on a singlethread workqueue?

Quoted 1 line

Hmm. I can't think of one. I was concerned about any async paths that potentially interfere with shutdown. With vhci_shutdown_connection() being so relaxed with locking, this is a cautious approach on my part. I am also keeping in mind that this problem shows up in a limited scope fuzzing test that doesn't trigger any other normal paths that would be active if there is real device on the other side.

As for the tcp_socket check in the reset routine, I am not positive what purpose it serves. I introduced the in_disconnect flag so shutdown and reset don't collide, in case I am missing some scenario in the normal path when we actually have a actual device attached.

With the other locking and error path problems in addressed, both shutdown and reset could be made simpler.

In any case, I think in_disconnect might be too big a hammer. I will redo the patch without it and also remove tcp_socket handling from the reset routine. I don't see USBIP_EH_RESET getting set without USBIP_EH_SHUTDOWN.

thanks,
-- Shuah