Skip to contentSkip to conversation

usbip: Implement SG support

PATCH Superseded Suwan Kim 2 participants 11 messages Aug 6, 2019 – Aug 8, 2019

Subsystems (2)

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Suwan Kim

author
#0 lore raw

v4 cover usbip: Implement SG support

There are bugs on vhci with usb 3.0 storage device. In USB, each SG list entry buffer should be divisible by the bulk max packet size. But with native SG support, this problem doesn't matter because the SG buffer is treated as contiguous buffer. But without native SG support, USB storage driver breaks SG list into several URBs and the error occurs because of a buffer size of URB that cannot be divided by the bulk max packet size. The error situation is as follows.

When USB Storage driver requests 31.5 KB data and has SG list which has 3584 bytes buffer followed by 7 4096 bytes buffer for some reason. USB Storage driver splits this SG list into several URBs because VHCI doesn't support SG and sends them separately. So the first URB buffer size is 3584 bytes. When receiving data from device, USB 3.0 device sends data packet of 1024 bytes size because the max packet size of BULK pipe is 1024 bytes. So device sends 4096 bytes. But the first URB buffer has only 3584 bytes buffer size. So host controller terminates the transfer even though there is more data to receive. So, vhci needs to support SG transfer to prevent this error.

In this patch, vhci supports SG regardless of whether the server's host controller supports SG or not, because stub driver splits SG list into several URBs if the server's host controller doesn't support SG.

To support SG, vhci_map_urb_for_dma() sets URB_DMA_MAP_SG flag in urb->transfer_flags if URB has SG list and this flag will tell stub driver to use SG list.

vhci sends each SG list entry to stub driver. Then, stub driver sees the total length of the buffer and allocates SG table and pages according to the total buffer length calling sgl_alloc(). After stub driver receives completed URB, it again sends each SG list entry to vhci.

If the server's host controller doesn't support SG, stub driver breaks a single SG request into several URBs and submits them to the server's host controller. When all the split URBs are completed, stub driver reassembles the URBs into a single return command and sends it to vhci.

Alan fixed vhci bug with the USB 3.0 storage device by modifying USB storage driver.
("usb-storage: Set virt_boundary_mask to avoid SG overflows") But the fundamental solution of it is to add SG support to vhci.

This patch works well with the USB 3.0 storage devices without Alan's patch, and we can revert Alan's patch if it causes some troubles.

Suwan Kim (2):
usbip: Skip DMA mapping and unmapping for urb at vhci
usbip: Implement SG support to vhci-hcd and stub driver

 drivers/usb/usbip/stub.h         |   7 +-
 drivers/usb/usbip/stub_main.c    |  55 ++++++---
 drivers/usb/usbip/stub_rx.c      | 204 ++++++++++++++++++++++---------
 drivers/usb/usbip/stub_tx.c      |  99 +++++++++++----
 drivers/usb/usbip/usbip_common.c |  59 ++++++---
 drivers/usb/usbip/vhci_hcd.c     |  34 +++++-
 drivers/usb/usbip/vhci_tx.c      |  63 ++++++++--
 7 files changed, 395 insertions(+), 126 deletions(-)
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH v4 1/2 usbip: Skip DMA mapping and unmapping for urb at vhci

vhci doesn’t do DMA for remote device. Actually, the real DMA operation is done by network card driver. vhci just passes virtual address of the buffer to the network stack, so vhci doesn’t use and need dma address of the buffer of the URB.

But HCD provides DMA mapping and unmapping function by default. Moreover, it causes unnecessary DMA mapping and unmapping which will be done again at the NIC driver and it wastes CPU cycles. So, implement map_urb_for_dma and unmap_urb_for_dma function for vhci in order to skip the DMA mapping and unmapping procedure.

When it comes to supporting SG for vhci, it is useful to use native SG list (urb->num_sgs) instead of mapped SG list because DMA mapping fnuction can adjust the number of SG list (urb->num_mapped_sgs). And vhci_map_urb_for_dma() prevents isoc pipe from using SG as hcd_map_urb_for_dma() does.

---
v3 - v4:
- Replace WARN_ON() with pr_err() in the error path.

v2 - v3
- Move setting URB_DMA_MAP_SG flag to the patch 2.
- Prevent isoc pipe from using SG buffer.

v1 - v2
- Add setting URB_DMA_MAP_SG flag in urb->transfer_flags to tell stub driver to use SG buffer.

---
 drivers/usb/usbip/vhci_hcd.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
drivers/usb/usbip/vhci_hcd.c +19 -0
index 000ab7225717..429e4e989f38 100644
@@ -1288,6 +1288,22 @@ static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev,
 	return 0;
 }
 
+static int vhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
+				gfp_t mem_flags)
+{
+	if (usb_endpoint_xfer_isoc(&urb->ep->desc) && urb->num_sgs) {
+		pr_err("SG is not supported for isochronous transfer\n");
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+static void vhci_unmap_urb_for_dma(struct usb_hcd *hcd, struct urb *urb)
+{
+	dev_dbg(hcd->self.controller, "vhci does not unmap urb for dma\n");
+}
+
 static const struct hc_driver vhci_hc_driver = {
 	.description	= driver_name,
 	.product_desc	= driver_desc,
@@ -1304,6 +1320,9 @@ static const struct hc_driver vhci_hc_driver = {
 
 	.get_frame_number = vhci_get_frame_number,
 
+	.map_urb_for_dma = vhci_map_urb_for_dma,
+	.unmap_urb_for_dma = vhci_unmap_urb_for_dma,
+
 	.hub_status_data = vhci_hub_status,
 	.hub_control    = vhci_hub_control,
 	.bus_suspend	= vhci_bus_suspend,
Signature · 1 line
2.20.1
maintainer
#2 lore raw

Re: [PATCH v4 1/2] usbip: Skip DMA mapping and unmapping for urb at vhci

Quoted 38 lines · Suwan Kim

vhci doesn’t do DMA for remote device. Actually, the real DMA operation is done by network card driver. vhci just passes virtual address of the buffer to the network stack, so vhci doesn’t use and need dma address of the buffer of the URB.

But HCD provides DMA mapping and unmapping function by default. Moreover, it causes unnecessary DMA mapping and unmapping which will be done again at the NIC driver and it wastes CPU cycles. So, implement map_urb_for_dma and unmap_urb_for_dma function for vhci in order to skip the DMA mapping and unmapping procedure.

When it comes to supporting SG for vhci, it is useful to use native SG list (urb->num_sgs) instead of mapped SG list because DMA mapping fnuction can adjust the number of SG list (urb->num_mapped_sgs). And vhci_map_urb_for_dma() prevents isoc pipe from using SG as hcd_map_urb_for_dma() does.

---
v3 - v4:
- Replace WARN_ON() with pr_err() in the error path.

v2 - v3
- Move setting URB_DMA_MAP_SG flag to the patch 2.
- Prevent isoc pipe from using SG buffer.

v1 - v2
- Add setting URB_DMA_MAP_SG flag in urb->transfer_flags to tell stub driver to use SG buffer.

---
  drivers/usb/usbip/vhci_hcd.c | 19 +++++++++++++++++++
  1 file changed, 19 insertions(+)
drivers/usb/usbip/vhci_hcd.c +0 -0
index 000ab7225717..429e4e989f38 100644

drivers/usb/usbip/vhci_hcd.c static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev, · line 1295

Quoted 8 lines
drivers/usb/usbip/vhci_hcd.c +5 -0
@@ -1288,6 +1288,22 @@ static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev,
  	return 0;
  }
 
+static int vhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
+				gfp_t mem_flags)
+{
+	if (usb_endpoint_xfer_isoc(&urb->ep->desc) && urb->num_sgs) {
+		pr_err("SG is not supported for isochronous transfer\n");

Any reason to not use dev_err()?

Looks good otherwise.

thanks,
-- Shuah

author
#3 lore raw

Re: [PATCH v4 1/2] usbip: Skip DMA mapping and unmapping for urb at vhci

Quoted 34 lines · shuah
Quoted 33 lines · Suwan Kim

vhci doesn’t do DMA for remote device. Actually, the real DMA operation is done by network card driver. vhci just passes virtual address of the buffer to the network stack, so vhci doesn’t use and need dma address of the buffer of the URB.

But HCD provides DMA mapping and unmapping function by default. Moreover, it causes unnecessary DMA mapping and unmapping which will be done again at the NIC driver and it wastes CPU cycles. So, implement map_urb_for_dma and unmap_urb_for_dma function for vhci in order to skip the DMA mapping and unmapping procedure.

When it comes to supporting SG for vhci, it is useful to use native SG list (urb->num_sgs) instead of mapped SG list because DMA mapping fnuction can adjust the number of SG list (urb->num_mapped_sgs). And vhci_map_urb_for_dma() prevents isoc pipe from using SG as hcd_map_urb_for_dma() does.

---
v3 - v4:
- Replace WARN_ON() with pr_err() in the error path.

v2 - v3
- Move setting URB_DMA_MAP_SG flag to the patch 2.
- Prevent isoc pipe from using SG buffer.

v1 - v2
- Add setting URB_DMA_MAP_SG flag in urb->transfer_flags to tell stub driver to use SG buffer.

---
  drivers/usb/usbip/vhci_hcd.c | 19 +++++++++++++++++++
  1 file changed, 19 insertions(+)
Quoted 14 lines

drivers/usb/usbip/vhci_hcd.c static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev, · line 1294

Quoted 12 lines
drivers/usb/usbip/vhci_hcd.c +5 -0
index 000ab7225717..429e4e989f38 100644
@@ -1288,6 +1288,22 @@ static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev,
  	return 0;
  }
+static int vhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
+				gfp_t mem_flags)
+{
+	if (usb_endpoint_xfer_isoc(&urb->ep->desc) && urb->num_sgs) {
+		pr_err("SG is not supported for isochronous transfer\n");

Any reason to not use dev_err()?

Because some codes in vhci_hcd.c use pr_err().There is no other reason. However, dev_err() seems more appropriate than pr_err(). I will replace pr_err() with dev_err(urb->dev->dev, "SG is ...") Is it ok?

Regards
Suwan Kim

maintainer
#4 lore raw

Re: [PATCH v4 1/2] usbip: Skip DMA mapping and unmapping for urb at vhci

Quoted 50 lines · Suwan Kim
Quoted 34 lines · shuah
Quoted 33 lines · Suwan Kim

vhci doesn’t do DMA for remote device. Actually, the real DMA operation is done by network card driver. vhci just passes virtual address of the buffer to the network stack, so vhci doesn’t use and need dma address of the buffer of the URB.

But HCD provides DMA mapping and unmapping function by default. Moreover, it causes unnecessary DMA mapping and unmapping which will be done again at the NIC driver and it wastes CPU cycles. So, implement map_urb_for_dma and unmap_urb_for_dma function for vhci in order to skip the DMA mapping and unmapping procedure.

When it comes to supporting SG for vhci, it is useful to use native SG list (urb->num_sgs) instead of mapped SG list because DMA mapping fnuction can adjust the number of SG list (urb->num_mapped_sgs). And vhci_map_urb_for_dma() prevents isoc pipe from using SG as hcd_map_urb_for_dma() does.

---
v3 - v4:
- Replace WARN_ON() with pr_err() in the error path.

v2 - v3
- Move setting URB_DMA_MAP_SG flag to the patch 2.
- Prevent isoc pipe from using SG buffer.

v1 - v2
- Add setting URB_DMA_MAP_SG flag in urb->transfer_flags to tell stub driver to use SG buffer.

---
   drivers/usb/usbip/vhci_hcd.c | 19 +++++++++++++++++++
   1 file changed, 19 insertions(+)
Quoted 14 lines

drivers/usb/usbip/vhci_hcd.c static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev, · line 1294

Quoted 12 lines
drivers/usb/usbip/vhci_hcd.c +5 -0
index 000ab7225717..429e4e989f38 100644
@@ -1288,6 +1288,22 @@ static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev,
   	return 0;
   }
+static int vhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
+				gfp_t mem_flags)
+{
+	if (usb_endpoint_xfer_isoc(&urb->ep->desc) && urb->num_sgs) {
+		pr_err("SG is not supported for isochronous transfer\n");

Any reason to not use dev_err()?

Quoted 5 lines

Because some codes in vhci_hcd.c use pr_err().There is no other reason. However, dev_err() seems more appropriate than pr_err(). I will replace pr_err() with dev_err(urb->dev->dev, "SG is ...") Is it ok?

Please. This way we will have the device information. pr_err() won't us that. In general I prefer dev_* if dev is available in the code path, which is the case here.

thanks,
-- Shuah

author
#5 lore raw

Re: [PATCH v4 1/2] usbip: Skip DMA mapping and unmapping for urb at vhci

Quoted 57 lines · shuah
Quoted 50 lines · Suwan Kim
Quoted 34 lines · shuah
Quoted 33 lines · Suwan Kim

vhci doesn’t do DMA for remote device. Actually, the real DMA operation is done by network card driver. vhci just passes virtual address of the buffer to the network stack, so vhci doesn’t use and need dma address of the buffer of the URB.

But HCD provides DMA mapping and unmapping function by default. Moreover, it causes unnecessary DMA mapping and unmapping which will be done again at the NIC driver and it wastes CPU cycles. So, implement map_urb_for_dma and unmap_urb_for_dma function for vhci in order to skip the DMA mapping and unmapping procedure.

When it comes to supporting SG for vhci, it is useful to use native SG list (urb->num_sgs) instead of mapped SG list because DMA mapping fnuction can adjust the number of SG list (urb->num_mapped_sgs). And vhci_map_urb_for_dma() prevents isoc pipe from using SG as hcd_map_urb_for_dma() does.

---
v3 - v4:
- Replace WARN_ON() with pr_err() in the error path.

v2 - v3
- Move setting URB_DMA_MAP_SG flag to the patch 2.
- Prevent isoc pipe from using SG buffer.

v1 - v2
- Add setting URB_DMA_MAP_SG flag in urb->transfer_flags to tell stub driver to use SG buffer.

---
   drivers/usb/usbip/vhci_hcd.c | 19 +++++++++++++++++++
   1 file changed, 19 insertions(+)
Quoted 14 lines

drivers/usb/usbip/vhci_hcd.c static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev, · line 1294

Quoted 12 lines
drivers/usb/usbip/vhci_hcd.c +5 -0
index 000ab7225717..429e4e989f38 100644
@@ -1288,6 +1288,22 @@ static int vhci_free_streams(struct usb_hcd *hcd, struct usb_device *udev,
   	return 0;
   }
+static int vhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
+				gfp_t mem_flags)
+{
+	if (usb_endpoint_xfer_isoc(&urb->ep->desc) && urb->num_sgs) {
+		pr_err("SG is not supported for isochronous transfer\n");

Any reason to not use dev_err()?

Quoted 5 lines

Because some codes in vhci_hcd.c use pr_err().There is no other reason. However, dev_err() seems more appropriate than pr_err(). I will replace pr_err() with dev_err(urb->dev->dev, "SG is ...") Is it ok?

Quoted 3 lines

Please. This way we will have the device information. pr_err() won't us that. In general I prefer dev_* if dev is available in the code path, which is the case here.

Ok. I will resend v5.

Regards
Suwan Kim

author
#6 lore raw

PATCH v4 2/2 usbip: Implement SG support to vhci-hcd and stub driver

In reply to Suwan Kim

There are bugs on vhci with usb 3.0 storage device. In USB, each SG list entry buffer should be divisible by the bulk max packet size. But with native SG support, this problem doesn't matter because the SG buffer is treated as contiguous buffer. But without native SG support, USB storage driver breaks SG list into several URBs and the error occurs because of a buffer size of URB that cannot be divided by the bulk max packet size. The error situation is as follows.

When USB Storage driver requests 31.5 KB data and has SG list which has 3584 bytes buffer followed by 7 4096 bytes buffer for some reason. USB Storage driver splits this SG list into several URBs because VHCI doesn't support SG and sends them separately. So the first URB buffer size is 3584 bytes. When receiving data from device, USB 3.0 device sends data packet of 1024 bytes size because the max packet size of BULK pipe is 1024 bytes. So device sends 4096 bytes. But the first URB buffer has only 3584 bytes buffer size. So host controller terminates the transfer even though there is more data to receive. So, vhci needs to support SG transfer to prevent this error.

In this patch, vhci supports SG regardless of whether the server's host controller supports SG or not, because stub driver splits SG list into several URBs if the server's host controller doesn't support SG.

To support SG, vhci_map_urb_for_dma() sets URB_DMA_MAP_SG flag in urb->transfer_flags if URB has SG list and this flag will tell stub driver to use SG list.

vhci sends each SG list entry to stub driver. Then, stub driver sees the total length of the buffer and allocates SG table and pages according to the total buffer length calling sgl_alloc(). After stub driver receives completed URB, it again sends each SG list entry to vhci.

If the server's host controller doesn't support SG, stub driver breaks a single SG request into several URBs and submits them to the server's host controller. When all the split URBs are completed, stub driver reassembles the URBs into a single return command and sends it to vhci.

Moreover, in the situation where vhci supports SG, but stub driver does not, or vice versa, usbip works normally. Because there is no protocol modification, there is no problem in communication between server and client even if the one has a kernel without SG support.

In the case of vhci supports SG and stub driver doesn't, because vhci sends only the total length of the buffer to stub driver as it did before the patch applied, stub driver only needs to allocate the required length of buffers regardless of whether vhci supports SG or not.

If stub driver needs to send data buffer to vhci because of IN pipe, stub driver also sends only total length of buffer as metadata and then sends real data as vhci does. Then vhci receive data from stub driver and store it to the corresponding buffer of SG list entry.

In the case of stub driver that supports SG, buffer is allocated by sgl_alloc(). However, stub driver that does not support SG allocates buffer using only kmalloc(). Therefore, if vhci supports SG and stub driver doesn't, stub driver has to allocate buffer with kmalloc() as much as the total length of SG buffer which is quite huge when vhci sends SG request, so it has big overhead in buffer allocation.

And for the case of stub driver supports SG and vhci doesn't, since the USB storage driver checks that vhci doesn't support SG and sends the request to stub driver by splitting the SG list into multiple URBs, stub driver allocates a buffer with kmalloc() as it did before this patch.

VUDC also works well with this patch. Tests are done with two USB gadget created by CONFIGFS USB gadget. Both use the BULK pipe.

1. Serial gadget
2. Mass storage gadget

* Serial gadget test

Serial gadget on the host sends and receives data using cat command on the /dev/ttyGS<N>. The client uses minicom to communicate with the serial gadget.

* Mass storage gadget test

After connecting the gadget with vhci, use "dd" to test read and write operation on the client side.

Read - dd if=/dev/sd<N> iflag=direct of=/dev/null bs=1G count=1 Write - dd if=<my file path> iflag=direct of=/dev/sd<N> bs=1G count=1

---
v3 - v4

- Rewrite the description about the vhci bug with USB 3.0 storage device.
- Add the description about the test with VUDC.
- Fix the error message in stub_recv_cmd_unlink().

v2 - v3

- Rewrite the commit log to make it more clear.
- Add the description about the mismatch situation in commit log.
- Run chechpatch.pl and fix errors with coding style and typos.
- Fix the error path of usbip_recv_xbuff() to stop receiving data after TCP error occurs.
- Consolidate the duplicated error path in usbip_recv_xbuff() and vhci_send_cmd_submit().
- Undo the unnecessary changes
* Undo the deletion of empty line in stub_send_ret_submit()
* Move memset() lines in stub_send_ret_submit() to the original position.

v1 - v2

- Add the logic that splits single SG request into several URBs in stub driver if server’s HC doesn’t support SG.

---
 drivers/usb/usbip/stub.h         |   7 +-
 drivers/usb/usbip/stub_main.c    |  55 ++++++---
 drivers/usb/usbip/stub_rx.c      | 204 ++++++++++++++++++++++---------
 drivers/usb/usbip/stub_tx.c      |  99 +++++++++++----
 drivers/usb/usbip/usbip_common.c |  59 ++++++---
 drivers/usb/usbip/vhci_hcd.c     |  15 ++-
 drivers/usb/usbip/vhci_tx.c      |  63 ++++++++--
 7 files changed, 376 insertions(+), 126 deletions(-)
drivers/usb/usbip/stub.h +6 -1
index 35618ceb2791..d11270560c24 100644
@@ -52,7 +52,11 @@ struct stub_priv {
 	unsigned long seqnum;
 	struct list_head list;
 	struct stub_device *sdev;
-	struct urb *urb;
+	struct urb **urbs;
+	struct scatterlist *sgl;
+	int num_urbs;
+	int completed_urbs;
+	int urb_status;
 
 	int unlinking;
 };
@@ -86,6 +90,7 @@ extern struct usb_device_driver stub_driver;
 struct bus_id_priv *get_busid_priv(const char *busid);
 void put_busid_priv(struct bus_id_priv *bid);
 int del_match_busid(char *busid);
+void stub_free_priv_and_urb(struct stub_priv *priv);
 void stub_device_cleanup_urbs(struct stub_device *sdev);
 
 /* stub_rx.c */
drivers/usb/usbip/stub_main.c +41 -14

78 lines not on this page · raw message

drivers/usb/usbip/stub_rx.c +146 -58

261 lines not on this page · raw message

drivers/usb/usbip/stub_tx.c +77 -22

150 lines not on this page · raw message

drivers/usb/usbip/usbip_common.c +41 -18

78 lines not on this page · raw message

drivers/usb/usbip/vhci_hcd.c +14 -1
index 429e4e989f38..0958024b8fae 100644
@@ -697,7 +697,8 @@ static int vhci_urb_enqueue(struct usb_hcd *hcd, struct urb *urb, gfp_t mem_flag
 	}
 	vdev = &vhci_hcd->vdev[portnum-1];
 
-	if (!urb->transfer_buffer && urb->transfer_buffer_length) {
+	if (!urb->transfer_buffer && !urb->num_sgs &&
+	     urb->transfer_buffer_length) {
 		dev_dbg(dev, "Null URB transfer buffer\n");
 		return -EINVAL;
 	}
@@ -1143,6 +1144,15 @@ static int vhci_setup(struct usb_hcd *hcd)
 		hcd->speed = HCD_USB3;
 		hcd->self.root_hub->speed = USB_SPEED_SUPER;
 	}
+
+	/*
+	 * Support SG.
+	 * sg_tablesize is an arbitrary value to alleviate memory pressure
+	 * on the host.
+	 */
+	hcd->self.sg_tablesize = 32;
+	hcd->self.no_sg_constraint = 1;
+
 	return 0;
 }
 
@@ -1296,6 +1306,9 @@ static int vhci_map_urb_for_dma(struct usb_hcd *hcd, struct urb *urb,
 		return -EINVAL;
 	}
 
+	if (urb->num_sgs)
+		urb->transfer_flags |= URB_DMA_MAP_SG;
+
 	return 0;
 }
 
drivers/usb/usbip/vhci_tx.c +51 -12

123 lines not on this page · raw message

Signature · 1 line
2.20.1
maintainer
#7 lore raw

Re: [PATCH v4 2/2] usbip: Implement SG support to vhci-hcd and stub driver

Quoted 81 lines · Suwan Kim

There are bugs on vhci with usb 3.0 storage device. In USB, each SG list entry buffer should be divisible by the bulk max packet size. But with native SG support, this problem doesn't matter because the SG buffer is treated as contiguous buffer. But without native SG support, USB storage driver breaks SG list into several URBs and the error occurs because of a buffer size of URB that cannot be divided by the bulk max packet size. The error situation is as follows.

When USB Storage driver requests 31.5 KB data and has SG list which has 3584 bytes buffer followed by 7 4096 bytes buffer for some reason. USB Storage driver splits this SG list into several URBs because VHCI doesn't support SG and sends them separately. So the first URB buffer size is 3584 bytes. When receiving data from device, USB 3.0 device sends data packet of 1024 bytes size because the max packet size of BULK pipe is 1024 bytes. So device sends 4096 bytes. But the first URB buffer has only 3584 bytes buffer size. So host controller terminates the transfer even though there is more data to receive. So, vhci needs to support SG transfer to prevent this error.

In this patch, vhci supports SG regardless of whether the server's host controller supports SG or not, because stub driver splits SG list into several URBs if the server's host controller doesn't support SG.

To support SG, vhci_map_urb_for_dma() sets URB_DMA_MAP_SG flag in urb->transfer_flags if URB has SG list and this flag will tell stub driver to use SG list.

vhci sends each SG list entry to stub driver. Then, stub driver sees the total length of the buffer and allocates SG table and pages according to the total buffer length calling sgl_alloc(). After stub driver receives completed URB, it again sends each SG list entry to vhci.

If the server's host controller doesn't support SG, stub driver breaks a single SG request into several URBs and submits them to the server's host controller. When all the split URBs are completed, stub driver reassembles the URBs into a single return command and sends it to vhci.

Moreover, in the situation where vhci supports SG, but stub driver does not, or vice versa, usbip works normally. Because there is no protocol modification, there is no problem in communication between server and client even if the one has a kernel without SG support.

In the case of vhci supports SG and stub driver doesn't, because vhci sends only the total length of the buffer to stub driver as it did before the patch applied, stub driver only needs to allocate the required length of buffers regardless of whether vhci supports SG or not.

If stub driver needs to send data buffer to vhci because of IN pipe, stub driver also sends only total length of buffer as metadata and then sends real data as vhci does. Then vhci receive data from stub driver and store it to the corresponding buffer of SG list entry.

In the case of stub driver that supports SG, buffer is allocated by sgl_alloc(). However, stub driver that does not support SG allocates buffer using only kmalloc(). Therefore, if vhci supports SG and stub driver doesn't, stub driver has to allocate buffer with kmalloc() as much as the total length of SG buffer which is quite huge when vhci sends SG request, so it has big overhead in buffer allocation.

And for the case of stub driver supports SG and vhci doesn't, since the USB storage driver checks that vhci doesn't support SG and sends the request to stub driver by splitting the SG list into multiple URBs, stub driver allocates a buffer with kmalloc() as it did before this patch.

VUDC also works well with this patch. Tests are done with two USB gadget created by CONFIGFS USB gadget. Both use the BULK pipe.

1. Serial gadget
2. Mass storage gadget

* Serial gadget test

Serial gadget on the host sends and receives data using cat command on the /dev/ttyGS<N>. The client uses minicom to communicate with the serial gadget.

Quoted 8 lines

* Mass storage gadget test

After connecting the gadget with vhci, use "dd" to test read and write operation on the client side.

Read - dd if=/dev/sd<N> iflag=direct of=/dev/null bs=1G count=1 Write - dd if=<my file path> iflag=direct of=/dev/sd<N> bs=1G count=1

Thanks for the test results.

Were you able to test with USB lowspeed devices?

thanks,
-- Shuah

author
#8 lore raw

Re: [PATCH v4 2/2] usbip: Implement SG support to vhci-hcd and stub driver

Quoted 87 lines · shuah
Quoted 86 lines · Suwan Kim

There are bugs on vhci with usb 3.0 storage device. In USB, each SG list entry buffer should be divisible by the bulk max packet size. But with native SG support, this problem doesn't matter because the SG buffer is treated as contiguous buffer. But without native SG support, USB storage driver breaks SG list into several URBs and the error occurs because of a buffer size of URB that cannot be divided by the bulk max packet size. The error situation is as follows.

When USB Storage driver requests 31.5 KB data and has SG list which has 3584 bytes buffer followed by 7 4096 bytes buffer for some reason. USB Storage driver splits this SG list into several URBs because VHCI doesn't support SG and sends them separately. So the first URB buffer size is 3584 bytes. When receiving data from device, USB 3.0 device sends data packet of 1024 bytes size because the max packet size of BULK pipe is 1024 bytes. So device sends 4096 bytes. But the first URB buffer has only 3584 bytes buffer size. So host controller terminates the transfer even though there is more data to receive. So, vhci needs to support SG transfer to prevent this error.

In this patch, vhci supports SG regardless of whether the server's host controller supports SG or not, because stub driver splits SG list into several URBs if the server's host controller doesn't support SG.

To support SG, vhci_map_urb_for_dma() sets URB_DMA_MAP_SG flag in urb->transfer_flags if URB has SG list and this flag will tell stub driver to use SG list.

vhci sends each SG list entry to stub driver. Then, stub driver sees the total length of the buffer and allocates SG table and pages according to the total buffer length calling sgl_alloc(). After stub driver receives completed URB, it again sends each SG list entry to vhci.

If the server's host controller doesn't support SG, stub driver breaks a single SG request into several URBs and submits them to the server's host controller. When all the split URBs are completed, stub driver reassembles the URBs into a single return command and sends it to vhci.

Moreover, in the situation where vhci supports SG, but stub driver does not, or vice versa, usbip works normally. Because there is no protocol modification, there is no problem in communication between server and client even if the one has a kernel without SG support.

In the case of vhci supports SG and stub driver doesn't, because vhci sends only the total length of the buffer to stub driver as it did before the patch applied, stub driver only needs to allocate the required length of buffers regardless of whether vhci supports SG or not.

If stub driver needs to send data buffer to vhci because of IN pipe, stub driver also sends only total length of buffer as metadata and then sends real data as vhci does. Then vhci receive data from stub driver and store it to the corresponding buffer of SG list entry.

In the case of stub driver that supports SG, buffer is allocated by sgl_alloc(). However, stub driver that does not support SG allocates buffer using only kmalloc(). Therefore, if vhci supports SG and stub driver doesn't, stub driver has to allocate buffer with kmalloc() as much as the total length of SG buffer which is quite huge when vhci sends SG request, so it has big overhead in buffer allocation.

And for the case of stub driver supports SG and vhci doesn't, since the USB storage driver checks that vhci doesn't support SG and sends the request to stub driver by splitting the SG list into multiple URBs, stub driver allocates a buffer with kmalloc() as it did before this patch.

VUDC also works well with this patch. Tests are done with two USB gadget created by CONFIGFS USB gadget. Both use the BULK pipe.

1. Serial gadget
2. Mass storage gadget

* Serial gadget test

Serial gadget on the host sends and receives data using cat command on the /dev/ttyGS<N>. The client uses minicom to communicate with the serial gadget.

* Mass storage gadget test

After connecting the gadget with vhci, use "dd" to test read and write operation on the client side.

Quoted 7 lines
Quoted 3 lines

Read - dd if=/dev/sd<N> iflag=direct of=/dev/null bs=1G count=1 Write - dd if=<my file path> iflag=direct of=/dev/sd<N> bs=1G count=1

Thanks for the test results.

Were you able to test with USB lowspeed devices?

I tested USB3 super-speed device and USB2 high-speed device. But I don't have full/low speed device that uses BULK transfer. In USB spec, low-speed device doesn't support BULK transfer.

Do I add test description about the USB3 super-speed and USB2 high-speed device to the commit log?

Regards
Suwan Kim

maintainer
#9 lore raw

Re: [PATCH v4 2/2] usbip: Implement SG support to vhci-hcd and stub driver

Quoted 92 lines · Suwan Kim
Quoted 91 lines · shuah
Quoted 89 lines · Suwan Kim

There are bugs on vhci with usb 3.0 storage device. In USB, each SG list entry buffer should be divisible by the bulk max packet size. But with native SG support, this problem doesn't matter because the SG buffer is treated as contiguous buffer. But without native SG support, USB storage driver breaks SG list into several URBs and the error occurs because of a buffer size of URB that cannot be divided by the bulk max packet size. The error situation is as follows.

When USB Storage driver requests 31.5 KB data and has SG list which has 3584 bytes buffer followed by 7 4096 bytes buffer for some reason. USB Storage driver splits this SG list into several URBs because VHCI doesn't support SG and sends them separately. So the first URB buffer size is 3584 bytes. When receiving data from device, USB 3.0 device sends data packet of 1024 bytes size because the max packet size of BULK pipe is 1024 bytes. So device sends 4096 bytes. But the first URB buffer has only 3584 bytes buffer size. So host controller terminates the transfer even though there is more data to receive. So, vhci needs to support SG transfer to prevent this error.

In this patch, vhci supports SG regardless of whether the server's host controller supports SG or not, because stub driver splits SG list into several URBs if the server's host controller doesn't support SG.

To support SG, vhci_map_urb_for_dma() sets URB_DMA_MAP_SG flag in urb->transfer_flags if URB has SG list and this flag will tell stub driver to use SG list.

vhci sends each SG list entry to stub driver. Then, stub driver sees the total length of the buffer and allocates SG table and pages according to the total buffer length calling sgl_alloc(). After stub driver receives completed URB, it again sends each SG list entry to vhci.

If the server's host controller doesn't support SG, stub driver breaks a single SG request into several URBs and submits them to the server's host controller. When all the split URBs are completed, stub driver reassembles the URBs into a single return command and sends it to vhci.

Moreover, in the situation where vhci supports SG, but stub driver does not, or vice versa, usbip works normally. Because there is no protocol modification, there is no problem in communication between server and client even if the one has a kernel without SG support.

In the case of vhci supports SG and stub driver doesn't, because vhci sends only the total length of the buffer to stub driver as it did before the patch applied, stub driver only needs to allocate the required length of buffers regardless of whether vhci supports SG or not.

If stub driver needs to send data buffer to vhci because of IN pipe, stub driver also sends only total length of buffer as metadata and then sends real data as vhci does. Then vhci receive data from stub driver and store it to the corresponding buffer of SG list entry.

In the case of stub driver that supports SG, buffer is allocated by sgl_alloc(). However, stub driver that does not support SG allocates buffer using only kmalloc(). Therefore, if vhci supports SG and stub driver doesn't, stub driver has to allocate buffer with kmalloc() as much as the total length of SG buffer which is quite huge when vhci sends SG request, so it has big overhead in buffer allocation.

And for the case of stub driver supports SG and vhci doesn't, since the USB storage driver checks that vhci doesn't support SG and sends the request to stub driver by splitting the SG list into multiple URBs, stub driver allocates a buffer with kmalloc() as it did before this patch.

VUDC also works well with this patch. Tests are done with two USB gadget created by CONFIGFS USB gadget. Both use the BULK pipe.

1. Serial gadget
2. Mass storage gadget

* Serial gadget test

Serial gadget on the host sends and receives data using cat command on the /dev/ttyGS<N>. The client uses minicom to communicate with the serial gadget.

* Mass storage gadget test

After connecting the gadget with vhci, use "dd" to test read and write operation on the client side.

Read - dd if=/dev/sd<N> iflag=direct of=/dev/null bs=1G count=1 Write - dd if=<my file path> iflag=direct of=/dev/sd<N> bs=1G count=1

Quoted 8 lines
Quoted 3 lines

Thanks for the test results.

Were you able to test with USB lowspeed devices?

I tested USB3 super-speed device and USB2 high-speed device. But I don't have full/low speed device that uses BULK transfer. In USB spec, low-speed device doesn't support BULK transfer.

I think I mentioned earlier that the reason to test with lowspeed is make sure there are no regressions due this change. You aren't testing BULK transfer, you are looking for any regressions.

Quoted 3 lines

Do I add test description about the USB3 super-speed and USB2 high-speed device to the commit log?

Please do.

thanks,
-- Shuah

author
#10 lore raw

Re: [PATCH v4 2/2] usbip: Implement SG support to vhci-hcd and stub driver

In reply to Suwan Kim
Quoted 187 lines · Suwan Kim

There are bugs on vhci with usb 3.0 storage device. In USB, each SG list entry buffer should be divisible by the bulk max packet size. But with native SG support, this problem doesn't matter because the SG buffer is treated as contiguous buffer. But without native SG support, USB storage driver breaks SG list into several URBs and the error occurs because of a buffer size of URB that cannot be divided by the bulk max packet size. The error situation is as follows.

When USB Storage driver requests 31.5 KB data and has SG list which has 3584 bytes buffer followed by 7 4096 bytes buffer for some reason. USB Storage driver splits this SG list into several URBs because VHCI doesn't support SG and sends them separately. So the first URB buffer size is 3584 bytes. When receiving data from device, USB 3.0 device sends data packet of 1024 bytes size because the max packet size of BULK pipe is 1024 bytes. So device sends 4096 bytes. But the first URB buffer has only 3584 bytes buffer size. So host controller terminates the transfer even though there is more data to receive. So, vhci needs to support SG transfer to prevent this error.

In this patch, vhci supports SG regardless of whether the server's host controller supports SG or not, because stub driver splits SG list into several URBs if the server's host controller doesn't support SG.

To support SG, vhci_map_urb_for_dma() sets URB_DMA_MAP_SG flag in urb->transfer_flags if URB has SG list and this flag will tell stub driver to use SG list.

vhci sends each SG list entry to stub driver. Then, stub driver sees the total length of the buffer and allocates SG table and pages according to the total buffer length calling sgl_alloc(). After stub driver receives completed URB, it again sends each SG list entry to vhci.

If the server's host controller doesn't support SG, stub driver breaks a single SG request into several URBs and submits them to the server's host controller. When all the split URBs are completed, stub driver reassembles the URBs into a single return command and sends it to vhci.

Moreover, in the situation where vhci supports SG, but stub driver does not, or vice versa, usbip works normally. Because there is no protocol modification, there is no problem in communication between server and client even if the one has a kernel without SG support.

In the case of vhci supports SG and stub driver doesn't, because vhci sends only the total length of the buffer to stub driver as it did before the patch applied, stub driver only needs to allocate the required length of buffers regardless of whether vhci supports SG or not.

If stub driver needs to send data buffer to vhci because of IN pipe, stub driver also sends only total length of buffer as metadata and then sends real data as vhci does. Then vhci receive data from stub driver and store it to the corresponding buffer of SG list entry.

In the case of stub driver that supports SG, buffer is allocated by sgl_alloc(). However, stub driver that does not support SG allocates buffer using only kmalloc(). Therefore, if vhci supports SG and stub driver doesn't, stub driver has to allocate buffer with kmalloc() as much as the total length of SG buffer which is quite huge when vhci sends SG request, so it has big overhead in buffer allocation.

And for the case of stub driver supports SG and vhci doesn't, since the USB storage driver checks that vhci doesn't support SG and sends the request to stub driver by splitting the SG list into multiple URBs, stub driver allocates a buffer with kmalloc() as it did before this patch.

VUDC also works well with this patch. Tests are done with two USB gadget created by CONFIGFS USB gadget. Both use the BULK pipe.

1. Serial gadget
2. Mass storage gadget

* Serial gadget test

Serial gadget on the host sends and receives data using cat command on the /dev/ttyGS<N>. The client uses minicom to communicate with the serial gadget.

* Mass storage gadget test

After connecting the gadget with vhci, use "dd" to test read and write operation on the client side.

Read - dd if=/dev/sd<N> iflag=direct of=/dev/null bs=1G count=1 Write - dd if=<my file path> iflag=direct of=/dev/sd<N> bs=1G count=1

---
v3 - v4

- Rewrite the description about the vhci bug with USB 3.0 storage device.
- Add the description about the test with VUDC.
- Fix the error message in stub_recv_cmd_unlink().

v2 - v3

- Rewrite the commit log to make it more clear.
- Add the description about the mismatch situation in commit log.
- Run chechpatch.pl and fix errors with coding style and typos.
- Fix the error path of usbip_recv_xbuff() to stop receiving data after TCP error occurs.
- Consolidate the duplicated error path in usbip_recv_xbuff() and vhci_send_cmd_submit().
- Undo the unnecessary changes
* Undo the deletion of empty line in stub_send_ret_submit()
* Move memset() lines in stub_send_ret_submit() to the original position.

v1 - v2

- Add the logic that splits single SG request into several URBs in stub driver if server’s HC doesn’t support SG.

---
 drivers/usb/usbip/stub.h         |   7 +-
 drivers/usb/usbip/stub_main.c    |  55 ++++++---
 drivers/usb/usbip/stub_rx.c      | 204 ++++++++++++++++++++++---------
 drivers/usb/usbip/stub_tx.c      |  99 +++++++++++----
 drivers/usb/usbip/usbip_common.c |  59 ++++++---
 drivers/usb/usbip/vhci_hcd.c     |  15 ++-
 drivers/usb/usbip/vhci_tx.c      |  63 ++++++++--
 7 files changed, 376 insertions(+), 126 deletions(-)
drivers/usb/usbip/stub.h +6 -1
index 35618ceb2791..d11270560c24 100644
@@ -52,7 +52,11 @@ struct stub_priv {
 	unsigned long seqnum;
 	struct list_head list;
 	struct stub_device *sdev;
-	struct urb *urb;
+	struct urb **urbs;
+	struct scatterlist *sgl;
+	int num_urbs;
+	int completed_urbs;
+	int urb_status;
 
 	int unlinking;
 };
@@ -86,6 +90,7 @@ extern struct usb_device_driver stub_driver;
 struct bus_id_priv *get_busid_priv(const char *busid);
 void put_busid_priv(struct bus_id_priv *bid);
 int del_match_busid(char *busid);
+void stub_free_priv_and_urb(struct stub_priv *priv);
 void stub_device_cleanup_urbs(struct stub_device *sdev);
 
 /* stub_rx.c */
drivers/usb/usbip/stub_main.c +21 -0
index 2e4bfccd4bfc..3ee636215e9c 100644
@@ -6,6 +6,7 @@
 #include <linux/string.h>
 #include <linux/module.h>
 #include <linux/device.h>
+#include <linux/scatterlist.h>
 
 #include "usbip_common.h"
 #include "stub.h"
@@ -288,6 +289,42 @@ static struct stub_priv *stub_priv_pop_from_listhead(struct list_head *listhead)
 	return NULL;
 }
 
+void stub_free_priv_and_urb(struct stub_priv *priv)
+{
+	struct urb *urb;
+	int i;
+
+	for (i = 0; i < priv->num_urbs; i++) {
+		urb = priv->urbs[i];
+
+		if (!urb)
+			return;
+
+		kfree(urb->setup_packet);
+		urb->setup_packet = NULL;
+
+
+		if (urb->transfer_buffer && !priv->sgl) {
+			kfree(urb->transfer_buffer);
+			urb->transfer_buffer = NULL;
+		}
+

drivers/usb/usbip/stub_main.c static struct stub_priv *stub_priv_pop_from_listhead(struct list_head *listhead) · line 321

Quoted 10 lines
drivers/usb/usbip/stub_main.c +10 -0
+		if (urb->num_sgs) {
+			sgl_free(urb->sg);
+			urb->sg = NULL;
+			urb->num_sgs = 0;
+		}
+
+		usb_free_urb(urb);
+	}
+
+	list_del(&priv->list);

I find list_del() error in my implementation. When client closes connection with server, stub driver calls stub_device_cleanup_urbs() to free all the privs and urbs. stub_device_cleanup_urbs() takes stub_priv from the lists calling stub_priv_pop() and calls stub_free_priv_and_urb() to free priv and urbs. But the error occurs when stub_free_priv_and_urb() tries to call list_del() to delete stub priv from the list because it was already deleted from the list in stub_priv_pop().

Quoted 28 lines
drivers/usb/usbip/stub_main.c +9 -7
+	if (priv->sgl)
+		sgl_free(priv->sgl);
+	kfree(priv->urbs);
+	kmem_cache_free(stub_priv_cache, priv);
+}
+
 static struct stub_priv *stub_priv_pop(struct stub_device *sdev)
 {
 	unsigned long flags;
@@ -314,25 +351,15 @@ static struct stub_priv *stub_priv_pop(struct stub_device *sdev)
 void stub_device_cleanup_urbs(struct stub_device *sdev)
 {
 	struct stub_priv *priv;
-	struct urb *urb;
+	int i;
 
 	dev_dbg(&sdev->udev->dev, "Stub device cleaning up urbs\n");
 
 	while ((priv = stub_priv_pop(sdev))) {
-		urb = priv->urb;
-		dev_dbg(&sdev->udev->dev, "free urb seqnum %lu\n",
-			priv->seqnum);
-		usb_kill_urb(urb);
+		for (i = 0; i < priv->num_urbs; i++)
+			usb_kill_urb(priv->urbs[i]);
 
-		kmem_cache_free(stub_priv_cache, priv);
-

drivers/usb/usbip/stub_main.c static struct stub_priv *stub_priv_pop(struct stub_device *sdev) · line 364

Quoted 10 lines
drivers/usb/usbip/stub_main.c +1 -7
-		kfree(urb->transfer_buffer);
-		urb->transfer_buffer = NULL;
-
-		kfree(urb->setup_packet);
-		urb->setup_packet = NULL;
-
-		usb_free_urb(urb);
+		stub_free_priv_and_urb(priv);
 	}
 }

I modified stub_free_priv_and_urb() and stub_device_cleanup_urbs() to support SG. In stub_device_cleanup_urbs(), the code is the same as stub_free_priv_and_urb() except that it does not call list_del() when freeing stub priv and urbs. So I removed the duplicated code and make stub_device_cleanup_urbs() call stub_free_priv_and_urb() to free stub priv and urbs. But when calling stub_free_priv_and_urb() in stub_device_cleanup_urbs(), it should avoid calling list_del() but it calls. I will fix it up and send v5.

Regards
Suwan Kim