Skip to contentSkip to conversation

hv_netvsc: Fix race condition on Multi-Send Data field

PATCH Applied Haiyang Zhang 4 participants 6 messages Dec 10, 2015 – Dec 14, 2015

Subsystems (3)
lore
Share & tools
lore

Haiyang Zhang

author maintainer
#0 lore raw

PATCH 1/1 hv_netvsc: Fix race condition on Multi-Send Data field

In commit 2a04ae8acb14 ("hv_netvsc: remove locking in netvsc_send()"), the locking for MSD (Multi-Send Data) field was removed. This could cause a race condition between RNDIS control messages and data packets processing, because these two types of traffic are not synchronized.
This patch fixes this issue by sending control messages out directly without reading MSD field.

---
 drivers/net/hyperv/netvsc.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/net/hyperv/netvsc.c +9 -0
index 02bab9a..059fc52 100644
@@ -867,6 +867,14 @@ int netvsc_send(struct hv_device *device,
 	packet->send_buf_index = NETVSC_INVALID_INDEX;
 	packet->cp_partial = false;
 
+	/* Send control message directly without accessing msd (Multi-Send
+	 * Data) field which may be changed during data packet processing.
+	 */
+	if (!skb) {
+		cur_send = packet;
+		goto send_now;
+	}
+
 	msdp = &net_device->msd[q_idx];
 
 	/* batch packets in send buffer if possible */
@@ -939,6 +947,7 @@ int netvsc_send(struct hv_device *device,
 		}
 	}
 
+send_now:
 	if (cur_send)
 		ret = netvsc_send_pkt(cur_send, net_device, pb, skb);
 
Signature · 1 line
1.7.4.1
#1 lore raw
Quoted 19 lines · Haiyang Zhang

In commit 2a04ae8acb14 ("hv_netvsc: remove locking in netvsc_send()"), the locking for MSD (Multi-Send Data) field was removed. This could cause a race condition between RNDIS control messages and data packets processing, because these two types of traffic are not synchronized.
This patch fixes this issue by sending control messages out directly without reading MSD field.

---
 drivers/net/hyperv/netvsc.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/net/hyperv/netvsc.c +0 -0
index 02bab9a..059fc52 100644
@@ -867,6 +867,14 @@ int netvsc_send(struct hv_device *device,
 	packet->send_buf_index = NETVSC_INVALID_INDEX;

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 877

Quoted 10 lines
drivers/net/hyperv/netvsc.c +8 -0
 	packet->cp_partial = false;
 
+	/* Send control message directly without accessing msd (Multi-Send
+	 * Data) field which may be changed during data packet processing.
+	 */
+	if (!skb) {
+		cur_send = packet;
+		goto send_now;
+	}
+

Is is supposed to be applied on top of some other patches? It doesn't compile on top of current net-next:

drivers/net/hyperv/netvsc.c: In function ‘netvsc_send’:
drivers/net/hyperv/netvsc.c:865:7: error: ‘skb’ undeclared (first use in this function)

  if (!skb) {
         ^

Did you mean to check rndis_msg instead (as skb is not defined here)?

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 952

Quoted 10 lines
drivers/net/hyperv/netvsc.c +1 -0
 	msdp = &net_device->msd[q_idx];
 
 	/* batch packets in send buffer if possible */
@@ -939,6 +947,7 @@ int netvsc_send(struct hv_device *device,
 		}
 	}
 
+send_now:
 	if (cur_send)
 		ret = netvsc_send_pkt(cur_send, net_device, pb, skb);

I suppose we untangle these two pathes completely: let rndis_filter_send_request() call netvsc_send_pkt() directly. Please see my patch attached (note: it should be split in 3 patches if submitted). If you like the idea I can send it.

Signature · 1 line
  Vitaly

commit b2784f827d2b7b19d3af6025bbe8be5b1450b88c

Author: Vitaly Kuznetsov <[email protected]>
Date:   Fri Dec 11 13:29:40 2015 +0100

netvsc: untangle rndis_filter_send_request and netvsc_send

drivers/net/hyperv/hyperv_net.h +4 -0
index f5b2145..03da20c 100644
@@ -189,6 +189,8 @@ int netvsc_device_remove(struct hv_device *device);
 int netvsc_send(struct hv_device *device,
 		struct hv_netvsc_packet *packet,
 		struct rndis_message *rndis_msg);
+int netvsc_send_pkt(struct hv_netvsc_packet *packet,
+		    struct netvsc_device *net_device);
 void netvsc_linkstatus_callback(struct hv_device *device_obj,
 				struct rndis_message *resp);
 void netvsc_xmit_completion(void *context);
@@ -211,6 +213,8 @@ int rndis_filter_receive(struct hv_device *dev,
 int rndis_filter_set_packet_filter(struct rndis_device *dev, u32 new_filter);
 int rndis_filter_set_device_mac(struct hv_device *hdev, char *mac);
 
+struct netvsc_device *netvsc_outbound_net_device(struct hv_device *device);
+
 #define NVSP_INVALID_PROTOCOL_VERSION	((u32)0xFFFFFFFF)
 
 #define NVSP_PROTOCOL_VERSION_1		2
drivers/net/hyperv/netvsc.c +6 -6
index 081f14f..eadbb03 100644
@@ -67,7 +67,7 @@ static void free_netvsc_device(struct netvsc_device *nvdev)
 	kfree(nvdev);
 }
 
-static struct netvsc_device *get_outbound_net_device(struct hv_device *device)
+struct netvsc_device *netvsc_outbound_net_device(struct hv_device *device)
 {
 	struct netvsc_device *net_device;
 
@@ -226,7 +226,7 @@ static int netvsc_init_buf(struct hv_device *device)
 	struct net_device *ndev;
 	int node;
 
-	net_device = get_outbound_net_device(device);
+	net_device = netvsc_outbound_net_device(device);
 	if (!net_device)
 		return -ENODEV;
 	ndev = net_device->ndev;
@@ -478,7 +478,7 @@ static int netvsc_connect_vsp(struct hv_device *device)
 		NVSP_PROTOCOL_VERSION_4, NVSP_PROTOCOL_VERSION_5 };
 	int i, num_ver = 4; /* number of different NVSP versions */
 
-	net_device = get_outbound_net_device(device);
+	net_device = netvsc_outbound_net_device(device);
 	if (!net_device)
 		return -ENODEV;
 	ndev = net_device->ndev;
@@ -740,7 +740,7 @@ static u32 netvsc_copy_to_send_buf(struct netvsc_device *net_device,
 	return msg_size;
 }
 
-static inline int netvsc_send_pkt(
+int netvsc_send_pkt(
 	struct hv_netvsc_packet *packet,
 	struct netvsc_device *net_device)
 {
@@ -850,7 +850,7 @@ int netvsc_send(struct hv_device *device,
 	struct hv_netvsc_packet *msd_send = NULL, *cur_send = NULL;
 	bool try_batch;
 
-	net_device = get_outbound_net_device(device);
+	net_device = netvsc_outbound_net_device(device);
 	if (!net_device)
 		return -ENODEV;
 
@@ -1063,7 +1063,7 @@ static void netvsc_send_table(struct hv_device *hdev,
 	int i;
 	u32 count, *tab;
 
-	nvscdev = get_outbound_net_device(hdev);
+	nvscdev = netvsc_outbound_net_device(hdev);
 	if (!nvscdev)
 		return;
 	ndev = nvscdev->ndev;
drivers/net/hyperv/rndis_filter.c +8 -1
index c8af172..9ee422d 100644
@@ -210,6 +210,11 @@ static int rndis_filter_send_request(struct rndis_device *dev,
 	int ret;
 	struct hv_netvsc_packet *packet;
 	struct hv_page_buffer page_buf[2];
+	struct netvsc_device *net_device;
+
+	net_device = netvsc_outbound_net_device(dev->net_dev->dev);
+	if (!net_device)
+		return -ENODEV;
 
 	/* Setup the packet to send it */
 	packet = &req->pkt;
@@ -239,8 +244,10 @@ static int rndis_filter_send_request(struct rndis_device *dev,
 
 	packet->completion_func = 0;
 	packet->xmit_more = false;
+	packet->send_buf_index = NETVSC_INVALID_INDEX;
+	packet->cp_partial = false;
 
-	ret = netvsc_send(dev->net_dev->dev, packet, NULL);
+	ret = netvsc_send_pkt(packet, net_device);
 	return ret;
 }
 
#2 lore raw
Quoted 35 lines · Vitaly Kuznetsov
Quoted 19 lines · Haiyang Zhang

In commit 2a04ae8acb14 ("hv_netvsc: remove locking in netvsc_send()"), the locking for MSD (Multi-Send Data) field was removed. This could cause a race condition between RNDIS control messages and data packets processing, because these two types of traffic are not synchronized.
This patch fixes this issue by sending control messages out directly without reading MSD field.

---
 drivers/net/hyperv/netvsc.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/net/hyperv/netvsc.c +0 -0
index 02bab9a..059fc52 100644
@@ -867,6 +867,14 @@ int netvsc_send(struct hv_device *device,
 	packet->send_buf_index = NETVSC_INVALID_INDEX;

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 877

Quoted 10 lines
drivers/net/hyperv/netvsc.c +8 -0
 	packet->cp_partial = false;
 
+	/* Send control message directly without accessing msd (Multi-Send
+	 * Data) field which may be changed during data packet processing.
+	 */
+	if (!skb) {
+		cur_send = packet;
+		goto send_now;
+	}
+

Is is supposed to be applied on top of some other patches? It doesn't compile on top of current net-next:

Quoted 7 lines

drivers/net/hyperv/netvsc.c: In function ‘netvsc_send’:
drivers/net/hyperv/netvsc.c:865:7: error: ‘skb’ undeclared (first use in this function)

  if (!skb) {
         ^

Did you mean to check rndis_msg instead (as skb is not defined here)?

Oh, my bad, it was me who was looking at the wrong branch... Sorry for the noise.

Quoted 8 lines
Quoted 8 lines
drivers/net/hyperv/netvsc.c +0 -0
 	msdp = &net_device->msd[q_idx];
 
 	/* batch packets in send buffer if possible */
@@ -939,6 +947,7 @@ int netvsc_send(struct hv_device *device,
 		}
 	}
 
Quoted 8 lines

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 952

Quoted 3 lines
drivers/net/hyperv/netvsc.c +1 -0
+send_now:
 	if (cur_send)
 		ret = netvsc_send_pkt(cur_send, net_device, pb, skb);

I suppose we untangle these two pathes completely: let rndis_filter_send_request() call netvsc_send_pkt() directly. Please see my patch attached (note: it should be split in 3 patches if submitted). If you like the idea I can send it.

This patch will need some changes but the suggestion still stands: let's untangle sending data and control messages.

Signature · 1 line
  Vitaly
author maintainer
#3 lore raw
Quoted 79 lines

-----Original Message-----
From: Vitaly Kuznetsov [mailto:[email protected]]
Sent: Friday, December 11, 2015 8:53 AM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]
Subject: Re: [PATCH net-next] hv_netvsc: Fix race condition on Multi- Send Data field

Quoted 3 lines · Vitaly Kuznetsov
Quoted 1 line · Haiyang Zhang

In commit 2a04ae8acb14 ("hv_netvsc: remove locking in netvsc_send()"),

the

Quoted 1 line
Quoted 1 line

locking for MSD (Multi-Send Data) field was removed. This could cause

a

Quoted 1 line
Quoted 1 line

race condition between RNDIS control messages and data packets

processing,

Quoted 11 lines

Review

Quoted 11 lines

because these two types of traffic are not synchronized. This patch fixes this issue by sending control messages out directly without reading MSD field.

---
 drivers/net/hyperv/netvsc.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
(quoted diff) +0 -0

b/drivers/net/hyperv/netvsc.c

Quoted 19 lines
Quoted 5 lines

index 02bab9a..059fc52 100644

drivers/net/hyperv/netvsc.c +0 -0
@@ -867,6 +867,14 @@ int netvsc_send(struct hv_device *device,
 	packet->send_buf_index = NETVSC_INVALID_INDEX;

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 877

Quoted 10 lines
drivers/net/hyperv/netvsc.c +8 -0
 	packet->cp_partial = false;
 
+	/* Send control message directly without accessing msd (Multi-Send
+	 * Data) field which may be changed during data packet processing.
+	 */
+	if (!skb) {
+		cur_send = packet;
+		goto send_now;
+	}
+

Is is supposed to be applied on top of some other patches? It doesn't compile on top of current net-next:

Quoted 2 lines

drivers/net/hyperv/netvsc.c: In function ‘netvsc_send’:
drivers/net/hyperv/netvsc.c:865:7: error: ‘skb’ undeclared (first use

in

Quoted 5 lines

this function)

  if (!skb) {
         ^

Did you mean to check rndis_msg instead (as skb is not defined here)?

Oh, my bad, it was me who was looking at the wrong branch... Sorry for the noise.

Quoted 8 lines
Quoted 8 lines
drivers/net/hyperv/netvsc.c +0 -0
 	msdp = &net_device->msd[q_idx];
 
 	/* batch packets in send buffer if possible */
@@ -939,6 +947,7 @@ int netvsc_send(struct hv_device *device,
 		}
 	}
 
Quoted 6 lines

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 952

Quoted 3 lines
drivers/net/hyperv/netvsc.c +1 -0
+send_now:
 	if (cur_send)
 		ret = netvsc_send_pkt(cur_send, net_device, pb, skb);

I suppose we untangle these two pathes completely: let rndis_filter_send_request() call netvsc_send_pkt() directly. Please

Quoted 6 lines

see

Quoted 2 lines

my patch attached (note: it should be split in 3 patches if submitted). If you like the idea I can send it.

This patch will need some changes but the suggestion still stands: let's untangle sending data and control messages.

Thanks for the suggestion.
I still prefer fixing this bug in the netvsc_send(), and keep the common entry point for sending data & control packets. So, the total lines of code is smaller. And, when we add more pre-processing steps for send in the future, we won't need to change multiple places.

Thanks,
- Haiyang

_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/driverdev-devel

#4 lore raw
In reply to Vitaly Kuznetsov
Quoted 80 lines

-----Original Message-----
From: devel [mailto:[email protected]] On Behalf Of Vitaly Kuznetsov
Sent: Friday, December 11, 2015 5:53 AM
To: Haiyang Zhang <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH net-next] hv_netvsc: Fix race condition on Multi-Send Data field

Quoted 3 lines · Vitaly Kuznetsov
Quoted 1 line · Haiyang Zhang

In commit 2a04ae8acb14 ("hv_netvsc: remove locking in netvsc_send()"),

the

Quoted 2 lines
Quoted 2 lines

locking for MSD (Multi-Send Data) field was removed. This could cause a race condition between RNDIS control messages and data packets

processing,

Quoted 30 lines
Quoted 16 lines

because these two types of traffic are not synchronized. This patch fixes this issue by sending control messages out directly without reading MSD field.

---
 drivers/net/hyperv/netvsc.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
drivers/net/hyperv/netvsc.c +0 -0
index 02bab9a..059fc52 100644
@@ -867,6 +867,14 @@ int netvsc_send(struct hv_device *device,
 	packet->send_buf_index = NETVSC_INVALID_INDEX;

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 877

Quoted 10 lines
drivers/net/hyperv/netvsc.c +8 -0
 	packet->cp_partial = false;
 
+	/* Send control message directly without accessing msd (Multi-Send
+	 * Data) field which may be changed during data packet processing.
+	 */
+	if (!skb) {
+		cur_send = packet;
+		goto send_now;
+	}
+

Is is supposed to be applied on top of some other patches? It doesn't compile on top of current net-next:

Quoted 7 lines

drivers/net/hyperv/netvsc.c: In function ‘netvsc_send’:
drivers/net/hyperv/netvsc.c:865:7: error: ‘skb’ undeclared (first use in this function)

  if (!skb) {
         ^

Did you mean to check rndis_msg instead (as skb is not defined here)?

Oh, my bad, it was me who was looking at the wrong branch... Sorry for the noise.

Quoted 8 lines
Quoted 8 lines
drivers/net/hyperv/netvsc.c +0 -0
 	msdp = &net_device->msd[q_idx];
 
 	/* batch packets in send buffer if possible */
@@ -939,6 +947,7 @@ int netvsc_send(struct hv_device *device,
 		}
 	}
 
Quoted 8 lines

drivers/net/hyperv/netvsc.c int netvsc_send(struct hv_device *device, · line 952

Quoted 3 lines
drivers/net/hyperv/netvsc.c +1 -0
+send_now:
 	if (cur_send)
 		ret = netvsc_send_pkt(cur_send, net_device, pb, skb);

I suppose we untangle these two pathes completely: let rndis_filter_send_request() call netvsc_send_pkt() directly. Please see my patch attached (note: it should be split in 3 patches if submitted). If you like the idea I can send it.

Quoted 2 lines

This patch will need some changes but the suggestion still stands: let's untangle sending data and control messages.

Control messages will have skb set to NULL and that is what Haiyang is doing to distinguish control packets.

K. Y

Quoted 11 lines

--
Vitaly
_______________________________________________
devel mailing list
[email protected]
https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fdriverde
v.linuxdriverproject.org%2fmailman%2flistinfo%2fdriverdev- devel%0a&data=01%7c01%7ckys%40microsoft.com%7c741026ba8cbb47a993 2c08d3023257b7%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=VdQiM NOzCtlRcsN5%2b%2faf%2bGrPNDHPmrvo2TuUs1T%2frlQ%3d

maintainer Applied
#5 lore raw
In reply to Haiyang Zhang

From: Haiyang Zhang <[email protected]>
Date: Thu, 10 Dec 2015 12:19:35 -0800

Quoted 9 lines

In commit 2a04ae8acb14 ("hv_netvsc: remove locking in netvsc_send()"), the locking for MSD (Multi-Send Data) field was removed. This could cause a race condition between RNDIS control messages and data packets processing, because these two types of traffic are not synchronized.
This patch fixes this issue by sending control messages out directly without reading MSD field.

Applied.