Skip to contentSkip to conversation

Kernel panic eth2 mirred redirect to ifb0

PATCH In review Paweł Staszewski 5 participants 38 messages Dec 19, 2010 – Jan 3, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all

I have panic with kernel 2.6.37-rc6-git2 when use iproute2 mirred/redirect action

host1 (kernel 2.6.36.2)
netperf client -> eth3 (82598EB 10-Gigabit AT CX4) - directly connected to eth2 of host2 ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

ethtool -i eth3
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.1

host2 (kernel-2.6.37-rc6-git2)
netserver -> eth2 (82598EB 10-Gigabit AT CX4) - directly connected to eth3 of host1

ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

ethtool -i eth2
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.0

Normally without ifb and redirect netperf show:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.2 (192.168.0.2) port 0 AF_INET

Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

87380 16384 16384 10.00 9042.14

Steps to reproduce panic:
ip link set dev ifb0 up

tc qdisc add dev eth2 ingress

tc filter add dev eth2 parent ffff: protocol ip prio 10 u32 \
match ip src 0.0.0.0/0 flowid 1:1 \
action mirred egress redirect dev ifb0

After this when i start netperf on host1 I have panic (screenshot in attached image).

Thanks
Pawel

#1 lore raw

W dniu 2010-12-19 12:35, Paweł Staszewski pisze:

Quoted 75 lines

Hi all

I have panic with kernel 2.6.37-rc6-git2 when use iproute2 mirred/redirect action

host1 (kernel 2.6.36.2)
netperf client -> eth3 (82598EB 10-Gigabit AT CX4) - directly connected to eth2 of host2 ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

ethtool -i eth3
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.1

host2 (kernel-2.6.37-rc6-git2)
netserver -> eth2 (82598EB 10-Gigabit AT CX4) - directly connected to eth3 of host1

ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

ethtool -i eth2
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.0

Normally without ifb and redirect netperf show:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.2 (192.168.0.2) port 0 AF_INET

Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

87380 16384 16384 10.00 9042.14

Steps to reproduce panic:
ip link set dev ifb0 up

tc qdisc add dev eth2 ingress

tc filter add dev eth2 parent ffff: protocol ip prio 10 u32 \
match ip src 0.0.0.0/0 flowid 1:1 \
action mirred egress redirect dev ifb0

Quoted 4 lines

After this when i start netperf on host1 I have panic (screenshot in attached image).

Thanks
Pawel

In next attached image is second screenshot (different - trace) - but with the same configuration

Regards
Pawel

#2 lore raw
In reply to Paweł Staszewski
Quoted 74 lines · Paweł Staszewski

Hi all

I have panic with kernel 2.6.37-rc6-git2 when use iproute2 mirred/redirect action

host1 (kernel 2.6.36.2)
netperf client -> eth3 (82598EB 10-Gigabit AT CX4) - directly connected to eth2 of host2
ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

ethtool -i eth3
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.1

host2 (kernel-2.6.37-rc6-git2)
netserver -> eth2 (82598EB 10-Gigabit AT CX4) - directly connected to eth3 of host1

ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

ethtool -i eth2
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.0

Normally without ifb and redirect netperf show:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.2 (192.168.0.2) port 0 AF_INET

Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

87380 16384 16384 10.00 9042.14

Steps to reproduce panic:
ip link set dev ifb0 up

tc qdisc add dev eth2 ingress

Quoted 7 lines

tc filter add dev eth2 parent ffff: protocol ip prio 10 u32 \
match ip src 0.0.0.0/0 flowid 1:1 \
action mirred egress redirect dev ifb0

After this when i start netperf on host1 I have panic (screenshot in attached image).

Unfortunately, we miss the start of panic messages. Could you try to get them ?

#3 lore raw

W dniu 2010-12-19 16:43, Eric Dumazet pisze:

Quoted 80 lines · Paweł Staszewski
Quoted 79 lines

Hi all

I have panic with kernel 2.6.37-rc6-git2 when use iproute2 mirred/redirect action

host1 (kernel 2.6.36.2)
netperf client -> eth3 (82598EB 10-Gigabit AT CX4) - directly connected to eth2 of host2
ethtool -k eth3
Offload parameters for eth3:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

ethtool -i eth3
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.1

host2 (kernel-2.6.37-rc6-git2)
netserver -> eth2 (82598EB 10-Gigabit AT CX4) - directly connected to eth3 of host1

ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

ethtool -i eth2
driver: ixgbe
version: 2.0.84-k2
firmware-version: 1.12-2
bus-info: 0000:03:00.0

Normally without ifb and redirect netperf show:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.2 (192.168.0.2) port 0 AF_INET

Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec
   87380  16384  16384    10.00    9042.14

Steps to reproduce panic:
ip link set dev ifb0 up

tc qdisc add dev eth2 ingress

tc filter add dev eth2 parent ffff: protocol ip prio 10 u32 \
match ip src 0.0.0.0/0 flowid 1:1 \
action mirred egress redirect dev ifb0

Quoted 5 lines
Quoted 2 lines

After this when i start netperf on host1 I have panic (screenshot in attached image).

Unfortunately, we miss the start of panic messages. Could you try to get them ?

In attached images

Regards
Pawel

Quoted 7 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

2010/12/20 Paweł Staszewski <[email protected]>:

Quoted 7 lines

W dniu 2010-12-19 16:43, Eric Dumazet pisze:

Quoted 4 lines

Unfortunately, we miss the start of panic messages. Could you try to get them ?

In attached images

It seems the kernel panic at:

        if (skb_shared(skb))
                BUG();

in pskb_expand_head().

It maybe related to my patch:
http://git.kernel.org/?p=linux/kernel/git/davem/net-next-2.6.git;a=commitdiff;h=210d6de78c5d7c785fc532556cea340e517955e1

You can try to revert it and test again.

However, the bug is a misuse of pskb_expand_head().

Signature · 2 lines
Regards,
Changli Gao([email protected])
#5 lore raw

W dniu 2010-12-19 17:22, Changli Gao pisze:

Quoted 12 lines

2010/12/20 Paweł Staszewski<[email protected]>:

Quoted 6 lines

W dniu 2010-12-19 16:43, Eric Dumazet pisze:

Quoted 3 lines

Unfortunately, we miss the start of panic messages. Could you try to get them ?

In attached images

It seems the kernel panic at:

         if (skb_shared(skb))
                 BUG();
Quoted 9 lines

in pskb_expand_head().

It maybe related to my patch:
http://git.kernel.org/?p=linux/kernel/git/davem/net-next-2.6.git;a=commitdiff;h=210d6de78c5d7c785fc532556cea340e517955e1

You can try to revert it and test again.

However, the bug is a misuse of pskb_expand_head().

patching file net/sched/act_mirred.c
Hunk #1 FAILED at 169.
Hunk #2 succeeded at 195 (offset 10 lines).
1 out of 2 hunks FAILED -- saving rejects to file net/sched/act_mirred.c.rej

***************
*** 169,181 ****

                goto out;
        }

- at = G_TC_AT(skb->tc_verd);

-       skb2 = skb_act_clone(skb, GFP_ATOMIC, m->tcf_action);
        if (skb2 == NULL)
                goto out;

        m->tcf_bstats.bytes += qdisc_pkt_len(skb2);
        m->tcf_bstats.packets++;
        if (!(at & AT_EGRESS)) {
                if (m->tcfm_ok_push)
                        skb_push(skb2, skb2->dev->hard_header_len);
--- 169,181 ----
                goto out;
        }
+       skb2 = skb_act_clone(skb, GFP_ATOMIC);
        if (skb2 == NULL)
                goto out;

        m->tcf_bstats.bytes += qdisc_pkt_len(skb2);
        m->tcf_bstats.packets++;
+       at = G_TC_AT(skb->tc_verd);
        if (!(at & AT_EGRESS)) {
                if (m->tcfm_ok_push)
                        skb_push(skb2, skb2->dev->hard_header_len);

for sch_generic.h was ok.

#6 lore raw
Quoted 60 lines · Paweł Staszewski

W dniu 2010-12-19 17:22, Changli Gao pisze:

Quoted 15 lines

2010/12/20 Paweł Staszewski<[email protected]>:

Quoted 7 lines

W dniu 2010-12-19 16:43, Eric Dumazet pisze:

Quoted 4 lines

Unfortunately, we miss the start of panic messages. Could you try to get
them ?

In attached images

It seems the kernel panic at:

         if (skb_shared(skb))
                 BUG();

in pskb_expand_head().

Quoted 8 lines

It maybe related to my patch:
http://git.kernel.org/?p=linux/kernel/git/davem/net-next-2.6.git;a=commitdiff;h=210d6de78c5d7c785fc532556cea340e517955e1

You can try to revert it and test again.

However, the bug is a misuse of pskb_expand_head().

patching file net/sched/act_mirred.c
Hunk #1 FAILED at 169.
Hunk #2 succeeded at 195 (offset 10 lines).
1 out of 2 hunks FAILED -- saving rejects to file
net/sched/act_mirred.c.rej

***************
*** 169,181 ****

                goto out;
        }
(quoted diff) +2 -3
-       at = G_TC_AT(skb->tc_verd);
-       skb2 = skb_act_clone(skb, GFP_ATOMIC, m->tcf_action);
        if (skb2 == NULL)
                goto out;
 
        m->tcf_bstats.bytes += qdisc_pkt_len(skb2);
        m->tcf_bstats.packets++;
        if (!(at & AT_EGRESS)) {
                if (m->tcfm_ok_push)
                        skb_push(skb2, skb2->dev->hard_header_len);
--- 169,181 ----
                goto out;
        }
 
+       skb2 = skb_act_clone(skb, GFP_ATOMIC);
        if (skb2 == NULL)
                goto out;
 
        m->tcf_bstats.bytes += qdisc_pkt_len(skb2);
        m->tcf_bstats.packets++;
+       at = G_TC_AT(skb->tc_verd);
        if (!(at & AT_EGRESS)) {
                if (m->tcfm_ok_push)
                        skb_push(skb2, skb2->dev->hard_header_len);
Quoted 1 line

for sch_generic.h was ok.

Should be enough to try after reverting this sch_generic.h change only.

Jarek P.

#7 lore raw
Quoted 5 lines · Jarek Poplawski
Quoted 1 line · Paweł Staszewski

for sch_generic.h was ok.

Should be enough to try after reverting this sch_generic.h change only.

Hmm... Sorry, I meant the change inside skb_act_clone(). I'll send a patch.

Jarek P.

#8 lore raw
Quoted 8 lines · Jarek Poplawski
Quoted 5 lines · Jarek Poplawski
Quoted 1 line · Paweł Staszewski

for sch_generic.h was ok.

Should be enough to try after reverting this sch_generic.h change only.

Hmm... Sorry, I meant the change inside skb_act_clone(). I'll send a patch.

Here it is.

Jarek P.

include/net/sch_generic.h +1 -7
index ea1f8a8..8763ccc 100644
@@ -608,13 +608,7 @@ static inline u32 qdisc_l2t(struct qdisc_rate_table* rtab, unsigned int pktlen)
 static inline struct sk_buff *skb_act_clone(struct sk_buff *skb, gfp_t gfp_mask,
 					    int action)
 {
-	struct sk_buff *n;
-
-	if ((action == TC_ACT_STOLEN || action == TC_ACT_QUEUED) &&
-	    !skb_shared(skb))
-		n = skb_get(skb);
-	else
-		n = skb_clone(skb, gfp_mask);
+	struct sk_buff *n = skb_clone(skb, gfp_mask);
 
 	if (n) {
 		n->tc_verd = SET_TC_VERD(n->tc_verd, 0);
#9 lore raw

W dniu 2010-12-19 23:26, Jarek Poplawski pisze:

Quoted 10 lines
Quoted 5 lines · Jarek Poplawski
Quoted 3 lines · Jarek Poplawski
Quoted 1 line · Paweł Staszewski

for sch_generic.h was ok.

Should be enough to try after reverting this sch_generic.h change only.

Hmm... Sorry, I meant the change inside skb_act_clone(). I'll send a patch.

Here it is.

Jarek P.

Yes reverting this patch solves the problem.

Thanks
Paweł

#10 lore raw

2010/12/20 Paweł Staszewski <[email protected]>:

Quoted 4 lines
Quoted 1 line

Yes reverting this patch solves the problem.

I am not sure reverting is the right fix. Maybe you can try this patch attached instead.

BTW: there are some others NIC drivers and net_sched actions don't take care of shared skbs.

Signature · 2 lines
Regards,
Changli Gao([email protected])
drivers/net/ixgbe/ixgbe_main.c +9 -0
index ca9036d..602cd32 100644
@@ -6096,6 +6096,15 @@ static int ixgbe_tso(struct ixgbe_adapter *adapter,
 	u32 mss_l4len_idx, l4len;
 
 	if (skb_is_gso(skb)) {
+		if (skb_shared(skb)) {
+			struct sk_buff *nskb;
+
+			nskb = skb_clone(skb, GFP_ATOMIC);
+			if (!nskb)
+				return -ENOMEM;
+			kfree_skb(skb);
+			skb = nskb;
+		}
 		if (skb_header_cloned(skb)) {
 			err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
 			if (err)
#11 lore raw
Quoted 6 lines · Changli Gao

2010/12/20 Paweł Staszewski <[email protected]>:

Quoted 4 lines
Quoted 1 line

Yes reverting this patch solves the problem.

Quoted 6 lines

I am not sure reverting is the right fix. Maybe you can try this patch attached instead.

BTW: there are some others NIC drivers and net_sched actions don't take care of shared skbs.

This patch cant be right as is.

        if (skb_is_gso(skb)) {

+ if (skb_shared(skb)) {
+ struct sk_buff *nskb;
+
+ nskb = skb_clone(skb, GFP_ATOMIC);
+ if (!nskb)
+ return -ENOMEM;
+ kfree_skb(skb); // HERE
+ skb = nskb;
+ }

You free original skb, while caller might dereference it later. So you must pass cloned skb back to caller.

(line 6393 :)
count = ixgbe_tx_map(adapter, tx_ring, skb, tx_flags, first);

#12 lore raw
Quoted 15 lines · Eric Dumazet

Le lundi 20 décembre 2010 à 16:56 +0800, Changli Gao a écrit :

This patch cant be right as is.

       if (skb_is_gso(skb)) {
+               if (skb_shared(skb)) {
+                       struct sk_buff *nskb;
+
+                       nskb = skb_clone(skb, GFP_ATOMIC);
+                       if (!nskb)
+                               return -ENOMEM;
+                       kfree_skb(skb); // HERE
+                       skb = nskb;
+               }

Quoted 7 lines

You free original skb, while caller might dereference it later. So you must pass cloned skb back to caller.

(line 6393 :)
 count = ixgbe_tx_map(adapter, tx_ring, skb, tx_flags, first);

Yes, you are right. I just wonder where shared skbs are allowed. Is there any rule?

Signature · 2 lines
Regards,
Changli Gao([email protected])
#13 lore raw
Quoted 19 lines · Changli Gao
Quoted 18 lines · Eric Dumazet

Le lundi 20 décembre 2010 à 16:56 +0800, Changli Gao a écrit :

This patch cant be right as is.

       if (skb_is_gso(skb)) {
drivers/net/ixgbe/ixgbe_main.c +9 -0
+               if (skb_shared(skb)) {
+                       struct sk_buff *nskb;
+
+                       nskb = skb_clone(skb, GFP_ATOMIC);
+                       if (!nskb)
+                               return -ENOMEM;
+                       kfree_skb(skb); // HERE
+                       skb = nskb;
+               }

You free original skb, while caller might dereference it later. So you must pass cloned skb back to caller.

Quoted 8 lines
Quoted 4 lines

(line 6393 :)
count = ixgbe_tx_map(adapter, tx_ring, skb, tx_flags, first);

Yes, you are right. I just wonder where shared skbs are allowed. Is there any rule?

Shared skbs are allowed for sure (pktgen is a provider of such skbs).

#14 lore raw
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · Changli Gao

Yes, you are right. I just wonder where shared skbs are allowed. Is there any rule?

Shared skbs are allowed for sure (pktgen is a provider of such skbs).

But pktgen doesn't use common dev_queue_xmit() path. It looks like some places checking segmentation call pskb_expand_head() assuming skb isn't shared. Btw, it seems ifb should have GSO features similarly to loopback. Anyway, until all this is verified, IMHO we should revert Changli's mirred patch in stable.

Jarek P.

#15 lore raw
Quoted 6 lines · Jarek Poplawski
Quoted 5 lines · Eric Dumazet
Quoted 3 lines · Changli Gao

Yes, you are right. I just wonder where shared skbs are allowed. Is there any rule?

Quoted 8 lines
Quoted 1 line

Shared skbs are allowed for sure (pktgen is a provider of such skbs).

But pktgen doesn't use common dev_queue_xmit() path. It looks like some places checking segmentation call pskb_expand_head() assuming skb isn't shared. Btw, it seems ifb should have GSO features similarly to loopback. Anyway, until all this is verified, IMHO we should revert Changli's mirred patch in stable.

I thought Changli question was : is a skb given to a device ndo_start_xmit() handler is allowed to be shared ;)

I believe Changli patch should be reverted, because not doing a skb_clone() should be known by caller.

To avoid the skb_clone(), we must tell caller what we did, so that caller dont even try to reuse skb (even freeing it)

#16 lore raw
Quoted 16 lines · Eric Dumazet
Quoted 6 lines · Jarek Poplawski
Quoted 5 lines · Eric Dumazet
Quoted 3 lines · Changli Gao

Yes, you are right. I just wonder where shared skbs are allowed. Is there any rule?

Quoted 8 lines
Quoted 1 line

Shared skbs are allowed for sure (pktgen is a provider of such skbs).

But pktgen doesn't use common dev_queue_xmit() path. It looks like some places checking segmentation call pskb_expand_head() assuming skb isn't shared. Btw, it seems ifb should have GSO features similarly to loopback. Anyway, until all this is verified, IMHO we should revert Changli's mirred patch in stable.

Quoted 8 lines

I thought Changli question was : is a skb given to a device ndo_start_xmit() handler is allowed to be shared ;)

I believe Changli patch should be reverted, because not doing a skb_clone() should be known by caller.

To avoid the skb_clone(), we must tell caller what we did, so that caller dont even try to reuse skb (even freeing it)

To tell the truth, Changli could assume we don't need to tell anything, because this mirred skb is almost not shared (except the refcount ;-).

Jarek P.

#17 lore raw
Quoted 2 lines · Jarek Poplawski

To tell the truth, Changli could assume we don't need to tell anything, because this mirred skb is almost not shared (except the refcount ;-).

Btw, since there was ifb involved, I doubt this skb could hit ixgbe xmit before unsharing, and patching the driver could matter. Anyway, Changli, I guess, Pawel needs some instructions on this last patch?

Jarek P.

#18 lore raw

W dniu 2010-12-20 12:58, Jarek Poplawski pisze:

Quoted 10 lines · Jarek Poplawski
Quoted 2 lines

To tell the truth, Changli could assume we don't need to tell anything, because this mirred skb is almost not shared (except the refcount ;-).

Btw, since there was ifb involved, I doubt this skb could hit ixgbe xmit before unsharing, and patching the driver could matter. Anyway, Changli, I guess, Pawel needs some instructions on this last patch?

Jarek P.

With this patch:

drivers/net/ixgbe/ixgbe_main.c +9 -0
index ca9036d..602cd32 100644
@@ -6096,6 +6096,15 @@ static int ixgbe_tso(struct ixgbe_adapter *adapter,
 	u32 mss_l4len_idx, l4len;
 
 	if (skb_is_gso(skb)) {
+		if (skb_shared(skb)) {
+			struct sk_buff *nskb;
+
+			nskb = skb_clone(skb, GFP_ATOMIC);
+			if (!nskb)
+				return -ENOMEM;
+			kfree_skb(skb);
+			skb = nskb;
+		}
 		if (skb_header_cloned(skb)) {
 			err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
 			if (err)

I have the same panic as without.

This patch was added to clean 2.6.37-rc6 - without previous patch for sch_generic.h

Attached image with kernel panic.

Thanks
Pawel

#19 lore raw
Quoted 6 lines · Paweł Staszewski

I have the same panic as without.

This patch was added to clean 2.6.37-rc6 - without previous patch for sch_generic.h

Attached image with kernel panic.

This one is definitely the nicest ;-)

Thanks,
Jarek P.

#20 lore raw
Quoted 9 lines · Jarek Poplawski
Quoted 6 lines · Paweł Staszewski

I have the same panic as without.

This patch was added to clean 2.6.37-rc6 - without previous patch for sch_generic.h

Attached image with kernel panic.

This one is definitely the nicest ;-)

Indeed, we finally can see where it happens ;)

#21 lore raw
Quoted 4 lines · Eric Dumazet
Quoted 3 lines · Jarek Poplawski
Quoted 2 lines · Paweł Staszewski

I have the same panic as without.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

This patch was added to clean 2.6.37-rc6 - without previous patch for sch_generic.h

Attached image with kernel panic.

This one is definitely the nicest ;-)

Indeed, we finally can see where it happens ;)

Shouldnt ifb adds some dev->features, like NETIF_F_FRAGLIST / NETIF_F_SG ?

#22 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · Jarek Poplawski
Quoted 6 lines · Paweł Staszewski

I have the same panic as without.

This patch was added to clean 2.6.37-rc6 - without previous patch for sch_generic.h

Attached image with kernel panic.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

This one is definitely the nicest ;-)

Indeed, we finally can see where it happens ;)

Shouldnt ifb adds some dev->features, like NETIF_F_FRAGLIST / NETIF_F_SG ?

IMHO it should, (probably even more, like loopback) but we should consider mirred can xmit to other than ifb too.

Jarek P.

#23 lore raw

2010/12/20 Jarek Poplawski <[email protected]>:

Quoted 4 lines

IMHO it should, (probably even more, like loopback) but we should consider mirred can xmit to other than ifb too.

I also think so. And when making ifb a multiqueue NIC, I tried to add these dev features to ifb. :)

Signature · 2 lines
Regards,
Changli Gao([email protected])
#24 lore raw

PATCH 1/1 ifb: add performance flags to dev->features

Quoted 9 lines · Changli Gao

2010/12/20 Jarek Poplawski <[email protected]>:

Quoted 4 lines

IMHO it should, (probably even more, like loopback) but we should consider mirred can xmit to other than ifb too.

I also think so. And when making ifb a multiqueue NIC, I tried to add these dev features to ifb. :)

This has litle to do with your multiqueue work, its more an effect of GRO being more and more deployed.

I dont see dev->features being changed in one of your previous patches. I did dummy case in commit 6d81f41c58c6

[PATCH net-next-2.6] ifb: add performance flags to dev->features

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unecessary split of some packets (GRO for example)

Cc Changli Gao <[email protected]>
Cc Jarek Poplawski <[email protected]>
Cc Pawel Staszewski <[email protected]>
---
 drivers/net/ifb.c |    3 +++
 1 file changed, 3 insertions(+)
drivers/net/ifb.c +3 -0
index 124dac4..c761551 100644
@@ -136,6 +136,9 @@ static void ifb_setup(struct net_device *dev)
 	ether_setup(dev);
 	dev->tx_queue_len = TX_Q_LIMIT;
 
+	dev->features |= NETIF_F_NO_CSUM | NETIF_F_SG |
+			 NETIF_F_FRAGLIST | NETIF_F_HIGHDMA |
+			 NETIF_F_TSO;
 	dev->flags |= IFF_NOARP;
 	dev->flags &= ~IFF_MULTICAST;
 	dev->priv_flags &= ~IFF_XMIT_DST_RELEASE;
#25 lore raw

Re: [PATCH net-next-2.6] ifb: add performance flags to dev->features

Quoted 11 lines · Eric Dumazet

 drivers/net/ifb.c |    3 +++

 1 file changed, 3 insertions(+)
drivers/net/ifb.c +0 -0
index 124dac4..c761551 100644

drivers/net/ifb.c static void ifb_setup(struct net_device *dev) · line 144

Quoted 9 lines
drivers/net/ifb.c +3 -0
@@ -136,6 +136,9 @@ static void ifb_setup(struct net_device *dev)
        ether_setup(dev);
        dev->tx_queue_len = TX_Q_LIMIT;
 
+       dev->features |= NETIF_F_NO_CSUM | NETIF_F_SG |
+                        NETIF_F_FRAGLIST | NETIF_F_HIGHDMA |
+                        NETIF_F_TSO;
        dev->flags |= IFF_NOARP;
        dev->flags &= ~IFF_MULTICAST;
        dev->priv_flags &= ~IFF_XMIT_DST_RELEASE;

I think we can assign these features to vlan_features.

dev->vlan_features |= dev->features.

Although I don't know all the features related to GSO, I just added all of them. For pseudo NIC, I think it is safe to do so.

+ dev->features |= NETIF_F_SG | NETIF_F_NO_CSUM | NETIF_F_TSO |
+ NETIF_F_UFO | NETIF_F_GSO_ROBUST | NETIF_F_TSO_ECN |
+ NETIF_F_TSO6 | NETIF_F_FSO;

Signature · 2 lines
Regards,
Changli Gao([email protected])
maintainer
#26 lore raw

Re: [PATCH net-next-2.6] ifb: add performance flags to dev->features

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Mon, 20 Dec 2010 15:25:20 +0100

Quoted 17 lines · Changli Gao
Quoted 9 lines

2010/12/20 Jarek Poplawski <[email protected]>:

Quoted 4 lines

IMHO it should, (probably even more, like loopback) but we should consider mirred can xmit to other than ifb too.

I also think so. And when making ifb a multiqueue NIC, I tried to add these dev features to ifb. :)

This has litle to do with your multiqueue work, its more an effect of GRO being more and more deployed.

I dont see dev->features being changed in one of your previous patches. I did dummy case in commit 6d81f41c58c6

Quoted 7 lines

[PATCH net-next-2.6] ifb: add performance flags to dev->features

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unecessary split of some packets (GRO for example)

Changli has suggested that these flags can also be set in ->vlan_features, please address that if you expect me to apply the change.

#27 lore raw

Re: [PATCH net-next-2.6] ifb: add performance flags to dev->features

Quoted 3 lines · David Miller

Changli has suggested that these flags can also be set in ->vlan_features, please address that if you expect me to apply the change.

Hmm, I am just scratching my head to actually setup vlans on top of ifb, to test such a change ?

#28 lore raw

Re: [PATCH net-next-2.6] ifb: add performance flags to dev->features

Quoted 8 lines · Eric Dumazet
Quoted 3 lines · David Miller

Changli has suggested that these flags can also be set in ->vlan_features, please address that if you expect me to apply the change.

Hmm, I am just scratching my head to actually setup vlans on top of ifb, to test such a change ?

Ingress is before vlans handler so these features and the NETIF_F_HW_VLAN_TX flag seem useful for ifb considering dev_hard_start_xmit() checks.

Jarek P.

#29 lore raw

PATCH v2 1/1 ifb: add performance flags

Quoted 3 lines · Jarek Poplawski

Ingress is before vlans handler so these features and the NETIF_F_HW_VLAN_TX flag seem useful for ifb considering dev_hard_start_xmit() checks.

OK, here is v2 of the patch then, thanks everybody.

[PATCH v2 net-next-2.6] ifb: add performance flags

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unnecessary split of some packets (GRO for example)

Changli suggested to also set vlan_features, Jarek suggested to add NETIF_F_HW_VLAN_TX as well.

Cc Changli Gao <[email protected]>
Cc Jarek Poplawski <[email protected]>
Cc Pawel Staszewski <[email protected]>
---
 drivers/net/ifb.c |    6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/ifb.c +6 -0
index 124dac4..66ca7bf 100644
@@ -126,6 +126,9 @@ static const struct net_device_ops ifb_netdev_ops = {
 	.ndo_validate_addr = eth_validate_addr,
 };
 
+#define IFB_FEATURES (NETIF_F_NO_CSUM | NETIF_F_SG  | NETIF_F_FRAGLIST | \
+		      NETIF_F_HIGHDMA | NETIF_F_TSO | NETIF_F_HW_VLAN_TX)
+
 static void ifb_setup(struct net_device *dev)
 {
 	/* Initialize the device structure. */
@@ -136,6 +139,9 @@ static void ifb_setup(struct net_device *dev)
 	ether_setup(dev);
 	dev->tx_queue_len = TX_Q_LIMIT;
 
+	dev->features |= IFB_FEATURES;
+	dev->vlan_features |= IFB_FEATURES;
+
 	dev->flags |= IFF_NOARP;
 	dev->flags &= ~IFF_MULTICAST;
 	dev->priv_flags &= ~IFF_XMIT_DST_RELEASE;
#30 lore raw

Re: [PATCH v2 net-next-2.6] ifb: add performance flags

Quoted 9 lines · Eric Dumazet
Quoted 3 lines · Jarek Poplawski

Ingress is before vlans handler so these features and the NETIF_F_HW_VLAN_TX flag seem useful for ifb considering dev_hard_start_xmit() checks.

OK, here is v2 of the patch then, thanks everybody.

Quoted 7 lines

[PATCH v2 net-next-2.6] ifb: add performance flags

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unnecessary split of some packets (GRO for example)

Changli suggested to also set vlan_features,

He also suggested more GSO flags of which especially NETIF_F_TSO6 seems interesting (wrt GRO)?

Jarek P.

Quoted 36 lines

Jarek suggested to add NETIF_F_HW_VLAN_TX as well.

Cc Changli Gao <[email protected]>
Cc Jarek Poplawski <[email protected]>
Cc Pawel Staszewski <[email protected]>
---
 drivers/net/ifb.c |    6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/ifb.c +6 -0
index 124dac4..66ca7bf 100644
@@ -126,6 +126,9 @@ static const struct net_device_ops ifb_netdev_ops = {
 	.ndo_validate_addr = eth_validate_addr,
 };
 
+#define IFB_FEATURES (NETIF_F_NO_CSUM | NETIF_F_SG  | NETIF_F_FRAGLIST | \
+		      NETIF_F_HIGHDMA | NETIF_F_TSO | NETIF_F_HW_VLAN_TX)
+
 static void ifb_setup(struct net_device *dev)
 {
 	/* Initialize the device structure. */
@@ -136,6 +139,9 @@ static void ifb_setup(struct net_device *dev)
 	ether_setup(dev);
 	dev->tx_queue_len = TX_Q_LIMIT;
 
+	dev->features |= IFB_FEATURES;
+	dev->vlan_features |= IFB_FEATURES;
+
 	dev->flags |= IFF_NOARP;
 	dev->flags &= ~IFF_MULTICAST;
 	dev->priv_flags &= ~IFF_XMIT_DST_RELEASE;
maintainer
#31 lore raw

Re: [PATCH v2 net-next-2.6] ifb: add performance flags

From: Jarek Poplawski <[email protected]>
Date: Mon, 3 Jan 2011 20:37:03 +0100

Quoted 12 lines · Eric Dumazet
Quoted 11 lines · Jarek Poplawski
Quoted 4 lines

Ingress is before vlans handler so these features and the NETIF_F_HW_VLAN_TX flag seem useful for ifb considering dev_hard_start_xmit() checks.

OK, here is v2 of the patch then, thanks everybody.

[PATCH v2 net-next-2.6] ifb: add performance flags

Quoted 8 lines
Quoted 5 lines

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unnecessary split of some packets (GRO for example)

Changli suggested to also set vlan_features,

He also suggested more GSO flags of which especially NETIF_F_TSO6 seems interesting (wrt GRO)?

I think at least TSO6 would very much be appropriate here.

#32 lore raw

Re: [PATCH v2 net-next-2.6] ifb: add performance flags

Quoted 19 lines · David Miller

From: Jarek Poplawski <[email protected]>
Date: Mon, 3 Jan 2011 20:37:03 +0100

Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Jarek Poplawski
Quoted 4 lines

Ingress is before vlans handler so these features and the NETIF_F_HW_VLAN_TX flag seem useful for ifb considering dev_hard_start_xmit() checks.

OK, here is v2 of the patch then, thanks everybody.

[PATCH v2 net-next-2.6] ifb: add performance flags

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unnecessary split of some packets (GRO for example)

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Changli suggested to also set vlan_features,

He also suggested more GSO flags of which especially NETIF_F_TSO6 seems interesting (wrt GRO)?

I think at least TSO6 would very much be appropriate here.

Yes, why not, I am only wondering why loopback / dummy (and others ?) only set NETIF_F_TSO :)

Since I want to play with ECN, I might also add NETIF_F_TSO_ECN ;)

For other flags, I really doubt it can matter on ifb ?

[PATCH v3 net-next-2.6] ifb: add performance flags

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unnecessary split of some packets (GRO for example)

Changli suggested to also set vlan_features, NETIF_F_TSO6, NETIF_F_TSO_ECN.

Jarek suggested to add NETIF_F_HW_VLAN_TX as well.

Cc Changli Gao <[email protected]>
Cc Jarek Poplawski <[email protected]>
Cc Pawel Staszewski <[email protected]>
---
 drivers/net/ifb.c |    7 +++++++
 1 file changed, 7 insertions(+)
drivers/net/ifb.c +7 -0
index 124dac4..e07d487 100644
@@ -126,6 +126,10 @@ static const struct net_device_ops ifb_netdev_ops = {
 	.ndo_validate_addr = eth_validate_addr,
 };
 
+#define IFB_FEATURES (NETIF_F_NO_CSUM | NETIF_F_SG  | NETIF_F_FRAGLIST	| \
+		      NETIF_F_TSO_ECN | NETIF_F_TSO | NETIF_F_TSO6	| \
+		      NETIF_F_HIGHDMA | NETIF_F_HW_VLAN_TX)
+
 static void ifb_setup(struct net_device *dev)
 {
 	/* Initialize the device structure. */
@@ -136,6 +140,9 @@ static void ifb_setup(struct net_device *dev)
 	ether_setup(dev);
 	dev->tx_queue_len = TX_Q_LIMIT;
 
+	dev->features |= IFB_FEATURES;
+	dev->vlan_features |= IFB_FEATURES;
+
 	dev->flags |= IFF_NOARP;
 	dev->flags &= ~IFF_MULTICAST;
 	dev->priv_flags &= ~IFF_XMIT_DST_RELEASE;
maintainer
#33 lore raw

Re: [PATCH v2 net-next-2.6] ifb: add performance flags

From: Eric Dumazet <[email protected]>
Date: Mon, 03 Jan 2011 21:35:22 +0100

Quoted 5 lines · David Miller
Quoted 1 line

I think at least TSO6 would very much be appropriate here.

Yes, why not, I am only wondering why loopback / dummy (and others ?) only set NETIF_F_TSO :)

TSO6 probably didn't exist when the current set were added, at least in the loopback case that's almost certainly the reason.

Quoted 10 lines

Since I want to play with ECN, I might also add NETIF_F_TSO_ECN ;)

For other flags, I really doubt it can matter on ifb ?

[PATCH v3 net-next-2.6] ifb: add performance flags

IFB can use the full set of features flags (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_TSO | NETIF_F_NO_CSUM | NETIF_F_HIGHDMA) to avoid unnecessary split of some packets (GRO for example)

Quoted 6 lines

Changli suggested to also set vlan_features, NETIF_F_TSO6, NETIF_F_TSO_ECN.

Jarek suggested to add NETIF_F_HW_VLAN_TX as well.

I'll apply this, thanks Eric.

#34 lore raw
In reply to Paweł Staszewski

2010/12/20 Paweł Staszewski <[email protected]>:

Quoted 24 lines

With this patch:

drivers/net/ixgbe/ixgbe_main.c +9 -0
index ca9036d..602cd32 100644
@@ -6096,6 +6096,15 @@ static int ixgbe_tso(struct ixgbe_adapter *adapter,
        u32 mss_l4len_idx, l4len;
 
        if (skb_is_gso(skb)) {
+               if (skb_shared(skb)) {
+                       struct sk_buff *nskb;
+
+                       nskb = skb_clone(skb, GFP_ATOMIC);
+                       if (!nskb)
+                               return -ENOMEM;
+                       kfree_skb(skb);
+                       skb = nskb;
+               }
                if (skb_header_cloned(skb)) {
                        err = pskb_expand_head(skb, 0, 0, GFP_ATOMIC);
                        if (err)
Quoted 8 lines

I have the same panic as without.

This patch was added to clean 2.6.37-rc6 - without previous patch for sch_generic.h

Attached image with kernel panic.

Sigh. We have to revert my patch first. :(

Signature · 2 lines
Regards,
Changli Gao([email protected])
#35 lore raw
Quoted 1 line · Changli Gao

Sigh. We have to revert my patch first. :(

Yes, please, send the patch. And don't worry: I hope you'll find the way to add it again ;-)

Cheers,
Jarek P.

#36 lore raw
In reply to Paweł Staszewski
Quoted 4 lines · Paweł Staszewski

W dniu 2010-12-19 23:26, Jarek Poplawski pisze:

Quoted 3 lines
Quoted 2 lines · Jarek Poplawski
Quoted 1 line · Jarek Poplawski

Paweł Staszewski wrote:

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line

for sch_generic.h was ok.

Should be enough to try after reverting this sch_generic.h change only.

Hmm... Sorry, I meant the change inside skb_act_clone(). I'll send a patch.

Here it is.

Jarek P.

Yes reverting this patch solves the problem.

Thanks !

I tried to reproduce the crash without success. Might be related to 10Gb speed. (I only have 1Gb here right now)

#37 lore raw

W dniu 2010-12-20 09:06, Eric Dumazet pisze:

Quoted 10 lines · Paweł Staszewski
Quoted 9 lines

W dniu 2010-12-19 23:26, Jarek Poplawski pisze:

Quoted 8 lines
Quoted 5 lines · Jarek Poplawski
Quoted 3 lines · Jarek Poplawski
Quoted 1 line · Paweł Staszewski

for sch_generic.h was ok.

Should be enough to try after reverting this sch_generic.h change only.

Hmm... Sorry, I meant the change inside skb_act_clone(). I'll send a patch.

Here it is.

Quoted 9 lines
Quoted 3 lines
Quoted 2 lines

Jarek P.

Yes reverting this patch solves the problem.

Thanks !

I tried to reproduce the crash without success. Might be related to 10Gb speed. (I only have 1Gb here right now)

Yes this is only ixgbe 10Gbit issue on e1000e all is working correctly.

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html