Skip to contentSkip to conversation

xen-netfront: drop skb when skb->len > 65535

PATCH In review Wei Liu 6 participants 12 messages Mar 1, 2013 – Mar 6, 2013

Subsystems (2)
lore
Share & tools
lore

Wei Liu

author
#0 lore raw

PATCH 1/1 xen-netfront: drop skb when skb->len > 65535

The `size' field of Xen network wired format is uint16_t, anything bigger than 65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

---
 drivers/net/xen-netfront.c |   12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/net/xen-netfront.c +12 -0
index 5527663..284059b 100644
@@ -547,6 +547,18 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	unsigned int len = skb_headlen(skb);
 	unsigned long flags;
 
+	/*
+	 * wired format of xen_netif_tx_request only supports skb->len
+	 * < 64K, because size field in xen_netif_tx_request is
+	 * uint16_t.
+	 */
+	if (unlikely(skb->len > (uint16_t)(~((uint16_t)0)))) {
+		net_alert_ratelimited(
+			"xennet: skb->len = %d, too big for wired format\n",
+			skb->len);
+		goto drop;
+	}
+
 	slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
 		xennet_count_skb_frag_slots(skb);
 	if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
Signature · 1 line
1.7.10.4
author
#1 lore raw

Oops, I made a mistake in Konrad's address.

Please remove the wrong address in future CC.

Wei.

Quoted 36 lines · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than 65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

---
 drivers/net/xen-netfront.c |   12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/net/xen-netfront.c +12 -0
index 5527663..284059b 100644
@@ -547,6 +547,18 @@ static int xennet_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	unsigned int len = skb_headlen(skb);
 	unsigned long flags;
 
+	/*
+	 * wired format of xen_netif_tx_request only supports skb->len
+	 * < 64K, because size field in xen_netif_tx_request is
+	 * uint16_t.
+	 */
+	if (unlikely(skb->len > (uint16_t)(~((uint16_t)0)))) {
+		net_alert_ratelimited(
+			"xennet: skb->len = %d, too big for wired format\n",
+			skb->len);
+		goto drop;
+	}
+
 	slots = DIV_ROUND_UP(offset + len, PAGE_SIZE) +
 		xennet_count_skb_frag_slots(skb);
 	if (unlikely(slots > MAX_SKB_FRAGS + 1)) {
#2 lore raw
In reply to Wei Liu
Quoted 9 lines
Quoted 1 line
Quoted 1 line · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than
65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

While fixing the frontend is certainly desirable, we can't expect everyone to deploy fixed netfronts in all their VMs - some OS versions used in there may even be out of service. So we ought to find a way to also more gracefully deal with the situation in netback, without re-opening the security issue that prompted those changes.

Jan

author
#3 lore raw
Quoted 10 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than
65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

Quoted 7 lines

While fixing the frontend is certainly desirable, we can't expect everyone to deploy fixed netfronts in all their VMs - some OS versions used in there may even be out of service. So we ought to find a way to also more gracefully deal with the situation in netback, without re-opening the security issue that prompted those changes.

Regarding the punishment bit, I think its worth discussing it a bit.

But the bug is always there, it drew no attention until revealed by XSA-39. It ought to be fixed anyway. :-)

Wei.

Quoted 2 lines

Jan

#4 lore raw
Quoted 19 lines · Wei Liu
Quoted 10 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than
65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

Quoted 7 lines

While fixing the frontend is certainly desirable, we can't expect everyone to deploy fixed netfronts in all their VMs - some OS versions used in there may even be out of service. So we ought to find a way to also more gracefully deal with the situation in netback, without re-opening the security issue that prompted those changes.

Quoted 1 line

Regarding the punishment bit, I think its worth discussing it a bit.

Yes, the trick is figuring out what to do without reintroducing the softlockup which XSA-39 fixed.

Perhaps we should allow silently consume (and drop) oversize skbs and only shutdown the rings if they also consume too many (FSVO too many) slots?

Quoted 2 lines

But the bug is always there, it drew no attention until revealed by XSA-39. It ought to be fixed anyway. :-)

I would have sworn that skb->len was also limited to 64k, but looking at the header I see it is actually an int and the only limit of that sort is related to MAX_SKB_FRAGS (which doesn't actually limit the total size).

OOI how big were the skbs you were seeing?

Not that it really matters but do we have a handle on why the prexisting bug didn't already cause connectivity issues? Does the retransmit (which I suppose must be happening) somehow end up using a smaller skb size?

BTW you mean "wire protocol" not "wired protocol" in the comments etc.

Ian.

author
#5 lore raw
Quoted 29 lines · Ian Campbell
Quoted 19 lines · Wei Liu
Quoted 10 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than
65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

Quoted 7 lines

While fixing the frontend is certainly desirable, we can't expect everyone to deploy fixed netfronts in all their VMs - some OS versions used in there may even be out of service. So we ought to find a way to also more gracefully deal with the situation in netback, without re-opening the security issue that prompted those changes.

Quoted 1 line

Regarding the punishment bit, I think its worth discussing it a bit.

Yes, the trick is figuring out what to do without reintroducing the softlockup which XSA-39 fixed.

Perhaps we should allow silently consume (and drop) oversize skbs and only shutdown the rings if they also consume too many (FSVO too many) slots?

Quoted 7 lines
Quoted 2 lines

But the bug is always there, it drew no attention until revealed by XSA-39. It ought to be fixed anyway. :-)

I would have sworn that skb->len was also limited to 64k, but looking at the header I see it is actually an int and the only limit of that sort is related to MAX_SKB_FRAGS (which doesn't actually limit the total size).

I had the impression that skb->len was limited to 64k, too. But it turned out I was wrong.

Quoted 2 lines

OOI how big were the skbs you were seeing?

As Nick ([email protected]) pointed out in his email, he saw size 65538. I can reproduce this as well by setting vif's mtu to 100 then run iperf. 100 was just a random number I came up with when I played with fragmentation.

Quoted 5 lines

Not that it really matters but do we have a handle on why the prexisting bug didn't already cause connectivity issues? Does the retransmit (which I suppose must be happening) somehow end up using a smaller skb size?

Not sure. I didn't have enough time to look into this yesterday. :-(

Quoted 2 lines

BTW you mean "wire protocol" not "wired protocol" in the comments etc.

Yes.

Wei.

Quoted 2 lines

Ian.

#6 lore raw
Quoted 33 lines · Wei Liu
Quoted 26 lines · Ian Campbell
Quoted 10 lines · Wei Liu
Quoted 9 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than
65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

Quoted 8 lines
Quoted 7 lines

While fixing the frontend is certainly desirable, we can't expect everyone to deploy fixed netfronts in all their VMs - some OS versions used in there may even be out of service. So we ought to find a way to also more gracefully deal with the situation in netback, without re-opening the security issue that prompted those changes.

Regarding the punishment bit, I think its worth discussing it a bit.

Yes, the trick is figuring out what to do without reintroducing the softlockup which XSA-39 fixed.

Perhaps we should allow silently consume (and drop) oversize skbs and only shutdown the rings if they also consume too many (FSVO too many) slots?

Quoted 6 lines
Quoted 2 lines

But the bug is always there, it drew no attention until revealed by XSA-39. It ought to be fixed anyway. :-)

I would have sworn that skb->len was also limited to 64k, but looking at the header I see it is actually an int and the only limit of that sort is related to MAX_SKB_FRAGS (which doesn't actually limit the total size).

Quoted 8 lines

I had the impression that skb->len was limited to 64k, too. But it turned out I was wrong.

Quoted 1 line

OOI how big were the skbs you were seeing?

As Nick ([email protected]) pointed out in his email, he saw size 65538. I can reproduce this as well by setting vif's mtu to 100 then run iperf. 100 was just a random number I came up with when I played with fragmentation.

Did you notice any GSO packets during your iperf test?

Thanks
Annie

#7 lore raw
In reply to Wei Liu
Quoted 6 lines · Wei Liu

As Nick ([email protected]) pointed out in his email, he saw size 65538. I can reproduce this as well by setting vif's mtu to 100 then run iperf. 100 was just a random number I came up with when I played with fragmentation.

Wei: How exactly did you set the MTU on the vif and what were your arguments to iperf? I tried this on a test host and was unable to trigger the XSA-39 protection.

As for a more graceful punishment, I noticed that xenvif_carrier_off() uses netif_carrier_off() to discard queued packets. Would doing a netif_carrier_off() and then a netif_carrier_on() be sufficient to drop the bad packets and keep netback from spinning? Would blinking the vif's carrier in this way allow the DomU to gracefully resume network traffic?

-Nick

author
#8 lore raw
Quoted 8 lines · Nick Pegg
Quoted 6 lines · Wei Liu

As Nick ([email protected]) pointed out in his email, he saw size 65538. I can reproduce this as well by setting vif's mtu to 100 then run iperf. 100 was just a random number I came up with when I played with fragmentation.

Quoted 4 lines

Wei: How exactly did you set the MTU on the vif and what were your arguments to iperf? I tried this on a test host and was unable to trigger the XSA-39 protection.

Sorry I didn't state this clearly. I tend to call the interface inside VM as vif as well. I did this inside a VM, ifconfig eth0 mtu 100. Nothing fancy added to iperf command line, just iperf -c XXX in VM and iperf -s in host (I limited the test time with -t though).

Quoted 6 lines

As for a more graceful punishment, I noticed that xenvif_carrier_off() uses netif_carrier_off() to discard queued packets. Would doing a netif_carrier_off() and then a netif_carrier_on() be sufficient to drop the bad packets and keep netback from spinning? Would blinking the vif's carrier in this way allow the DomU to gracefully resume network traffic?

I think the spinning has something to do with the ring, not packets -- I'm not very sure though, because I didn't take part in the discussion / development of XSA-39. Draining host queues is not likely to fix this.

Wei.

Quoted 2 lines

-Nick

#9 lore raw
Quoted 4 lines · Nick Pegg
Quoted 3 lines · Wei Liu
Quoted 2 lines

As Nick ([email protected]) pointed out in his email, he saw size

65538.

Quoted 1 line
Quoted 1 line
Quoted 1 line

I can reproduce this as well by setting vif's mtu to 100 then run

iperf.

Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

100 was just a random number I came up with when I played with fragmentation.

Wei: How exactly did you set the MTU on the vif and what were your arguments to iperf? I tried this on a test host and was unable to trigger the XSA-39 protection.

Sorry I didn't state this clearly. I tend to call the interface inside

VM as vif as

Quoted 3 lines

well. I did this inside a VM, ifconfig eth0 mtu 100.
Nothing fancy added to iperf command line, just iperf -c XXX in VM and iperf -s in host (I limited the test time with -t though).

We have recently hit the same problem (however in our case, larger NFS transfer was enough to trigger it) and I've found that it only happens if when MAX_SKB_FRAGS is different on domU and dom0 (as in, domU using older kernel (without this change
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commi
t/drivers/net/xen-netback/netback.c?id=48856286b64e4b66ec62b94e504d0b29c1a de664 )) can you confirm your domU kernel version?

author
#10 lore raw
Quoted 22 lines · Jacek Milewicz
Quoted 4 lines · Nick Pegg
Quoted 3 lines · Wei Liu
Quoted 2 lines

As Nick ([email protected]) pointed out in his email, he saw size

65538.

Quoted 1 line
Quoted 1 line
Quoted 1 line

I can reproduce this as well by setting vif's mtu to 100 then run

iperf.

Quoted 10 lines
Quoted 8 lines
Quoted 3 lines

100 was just a random number I came up with when I played with fragmentation.

Wei: How exactly did you set the MTU on the vif and what were your arguments to iperf? I tried this on a test host and was unable to trigger the XSA-39 protection.

Sorry I didn't state this clearly. I tend to call the interface inside

VM as vif as

Quoted 3 lines

well. I did this inside a VM, ifconfig eth0 mtu 100.
Nothing fancy added to iperf command line, just iperf -c XXX in VM and iperf -s in host (I limited the test time with -t though).

Quoted 8 lines

We have recently hit the same problem (however in our case, larger NFS transfer was enough to trigger it) and I've found that it only happens if when MAX_SKB_FRAGS is different on domU and dom0 (as in, domU using older kernel (without this change
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commi
t/drivers/net/xen-netback/netback.c?id=48856286b64e4b66ec62b94e504d0b29c1a de664 )) can you confirm your domU kernel version?

Actually I'm replying to your other email now. :-)

The root cause of your problem is a bit different than the one I discovered, but they ended up the same error path anyway.

My kernel should be 3.9-rc or something. I use it for both Dom0 and DomU.

Wei.

Quoted 1 line
#11 lore raw
In reply to Ian Campbell
Quoted 26 lines · Ian Campbell
Quoted 10 lines · Wei Liu
Quoted 9 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · Wei Liu

The `size' field of Xen network wired format is uint16_t, anything bigger than
65535 will cause overflow.

The punishment introduced by XSA-39 is quite harsh - DomU is disconnected when it's discovered to be sending corrupted skbs. However, it looks like Linux kernel will generate some bad skbs sometimes, so drop those skbs before sending to over netback to avoid being disconnected.

Quoted 8 lines
Quoted 7 lines

While fixing the frontend is certainly desirable, we can't expect everyone to deploy fixed netfronts in all their VMs - some OS versions used in there may even be out of service. So we ought to find a way to also more gracefully deal with the situation in netback, without re-opening the security issue that prompted those changes.

Regarding the punishment bit, I think its worth discussing it a bit.

Yes, the trick is figuring out what to do without reintroducing the softlockup which XSA-39 fixed.

Perhaps we should allow silently consume (and drop) oversize skbs and only shutdown the rings if they also consume too many (FSVO too many) slots?

Quoted 6 lines
Quoted 2 lines

But the bug is always there, it drew no attention until revealed by XSA-39. It ought to be fixed anyway. :-)

I would have sworn that skb->len was also limited to 64k, but looking at the header I see it is actually an int and the only limit of that sort is related to MAX_SKB_FRAGS (which doesn't actually limit the total size).

If compound page enabled, it is very possible that netback runs into following code.

                if (unlikely(frags >= MAX_SKB_FRAGS)) {
                        netdev_err(vif->dev, "Too many frags\n");
                        netbk_fatal_tx_err(vif);
                        return -frags;
                }

MAX_SKB_FRAGS does not actually limit the total size in theory. But it does limit the total size in netback since netback only supports one-page-sized fragments.

Thanks
Annie