Skip to contentSkip to conversation

loopback: clear pfmemalloc on outgoing skb's

PATCH In review Josef Bacik 3 participants 11 messages Feb 1, 2017 – Feb 3, 2017

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 loopback: clear pfmemalloc on outgoing skb's

I was seeing random disconnects while testing NBD over loopback. This turned out to be because NBD sets pfmemalloc on it's socket, however the receiving side is a user space application so does not have pfmemalloc set on its socket. This means that sk_filter_trim_cap will simply drop this packet, under the assumption that the other side will simply retransmit. Well we do retransmit, and then the packet is just dropped again for the same reason. To keep this from happening simply clear skb->pfmemalloc on transmit so that we don't drop the packet on the receive side.

---
 drivers/net/loopback.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/net/loopback.c +7 -0
index 1e05b7c..13c9126 100644
@@ -81,6 +81,13 @@ static netdev_tx_t loopback_xmit(struct sk_buff *skb,
 	 */
 	skb_dst_force(skb);
 
+	/* If our transmitter was a pfmemalloc socket we need to clear
+	 * pfmemalloc here, otherwise the receiving socket may not be
+	 * pfmemalloc, and if this is a tcp packet then it'll get dropped and
+	 * all traffic will halt.
+	 */
+	skb->pfmemalloc = false;
+
 	skb->protocol = eth_type_trans(skb, dev);
 
 	/* it's OK to use per_cpu_ptr() because BHs are off */
Signature · 1 line
2.7.4
#1 lore raw
Quoted 19 lines · Josef Bacik

I was seeing random disconnects while testing NBD over loopback. This turned out to be because NBD sets pfmemalloc on it's socket, however the receiving side is a user space application so does not have pfmemalloc set on its socket. This means that sk_filter_trim_cap will simply drop this packet, under the assumption that the other side will simply retransmit. Well we do retransmit, and then the packet is just dropped again for the same reason. To keep this from happening simply clear skb->pfmemalloc on transmit so that we don't drop the packet on the receive side.

---
 drivers/net/loopback.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/net/loopback.c +0 -0
index 1e05b7c..13c9126 100644

drivers/net/loopback.c static netdev_tx_t loopback_xmit(struct sk_buff *skb, · line 90

Quoted 10 lines
drivers/net/loopback.c +7 -0
@@ -81,6 +81,13 @@ static netdev_tx_t loopback_xmit(struct sk_buff *skb,
 	 */
 	skb_dst_force(skb);
 
+	/* If our transmitter was a pfmemalloc socket we need to clear
+	 * pfmemalloc here, otherwise the receiving socket may not be
+	 * pfmemalloc, and if this is a tcp packet then it'll get dropped and
+	 * all traffic will halt.
+	 */
+	skb->pfmemalloc = false;
+

I am not sure this is a proper fix.

Presumably if the socket was able to store packets in its write queue, fact that it sends it to loopback or an Ethernet link should not matter.

Only in RX path the pfmemalloc thing is really important.

So I would rather not set skb->pfmemalloc for skbs allocated for the write queue, and more exactly the fast clone.

This would actually speed up the stack a bit.

net/core/skbuff.c +1 -1
index 734c71468b013838516cfe8c744dcd0e797a6e2b..f91b81340dc5be80e0c26f9835d9192f35b75ad7 100644
@@ -271,7 +271,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
                atomic_set(&fclones->fclone_ref, 1);
 
                fclones->skb2.fclone = SKB_FCLONE_CLONE;
-               fclones->skb2.pfmemalloc = pfmemalloc;
+               fclones->skb2.pfmemalloc = 0;
        }
 out:
        return skb;
#2 lore raw
Quoted 48 lines · Eric Dumazet
Quoted 19 lines · Josef Bacik

I was seeing random disconnects while testing NBD over loopback. This turned out to be because NBD sets pfmemalloc on it's socket, however the receiving side is a user space application so does not have pfmemalloc set on its socket. This means that sk_filter_trim_cap will simply drop this packet, under the assumption that the other side will simply retransmit. Well we do retransmit, and then the packet is just dropped again for the same reason. To keep this from happening simply clear skb->pfmemalloc on transmit so that we don't drop the packet on the receive side.

---
 drivers/net/loopback.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/net/loopback.c +0 -0
index 1e05b7c..13c9126 100644

drivers/net/loopback.c static netdev_tx_t loopback_xmit(struct sk_buff *skb, · line 90

Quoted 10 lines
drivers/net/loopback.c +7 -0
@@ -81,6 +81,13 @@ static netdev_tx_t loopback_xmit(struct sk_buff *skb,
 	 */
 	skb_dst_force(skb);
 
+	/* If our transmitter was a pfmemalloc socket we need to clear
+	 * pfmemalloc here, otherwise the receiving socket may not be
+	 * pfmemalloc, and if this is a tcp packet then it'll get dropped and
+	 * all traffic will halt.
+	 */
+	skb->pfmemalloc = false;
+

I am not sure this is a proper fix.

Presumably if the socket was able to store packets in its write queue, fact that it sends it to loopback or an Ethernet link should not matter.

Only in RX path the pfmemalloc thing is really important.

So I would rather not set skb->pfmemalloc for skbs allocated for the write queue, and more exactly the fast clone.

This would actually speed up the stack a bit.

net/core/skbuff.c +0 -0
index 734c71468b013838516cfe8c744dcd0e797a6e2b..f91b81340dc5be80e0c26f9835d9192f35b75ad7 100644

net/core/skbuff.c struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, · line 274

Quoted 5 lines
net/core/skbuff.c +1 -1
@@ -271,7 +271,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
                atomic_set(&fclones->fclone_ref, 1);
 
                fclones->skb2.fclone = SKB_FCLONE_CLONE;
-               fclones->skb2.pfmemalloc = pfmemalloc;
+               fclones->skb2.pfmemalloc = 0;

It turns out this part was not needed in current kernel, because __copy_skb_header() will copy pfmemalloc, since it is included in the headers_start/headers_end section of skb.

So this patch is not solving your issue.

#3 lore raw

PATCH 1/1 net: remove useless pfmemalloc setting

From: Eric Dumazet <[email protected]>

When __alloc_skb() allocates an skb from fast clone cache, setting pfmemalloc on the clone is not needed.

Clone will be properly initialized later at skb_clone() time, including pfmemalloc field, as it is included in the headers_start/headers_end section which is fully copied.

---
 net/core/skbuff.c |    1 -
 1 file changed, 1 deletion(-)
net/core/skbuff.c +0 -1
index 26c1344cc23e31d4248d23fc007bcab4b034c5c9..4f8f2a1a66b5a36705e8f9966f7abd751932dcc6 100644
@@ -271,7 +271,6 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask,
 		atomic_set(&fclones->fclone_ref, 1);
 
 		fclones->skb2.fclone = SKB_FCLONE_CLONE;
-		fclones->skb2.pfmemalloc = pfmemalloc;
 	}
 out:
 	return skb;
maintainer Applied
#4 lore raw

Re: [PATCH net-next] net: remove useless pfmemalloc setting

From: Eric Dumazet <[email protected]>
Date: Wed, 01 Feb 2017 18:13:23 -0800

Quoted 10 lines

From: Eric Dumazet <[email protected]>

When __alloc_skb() allocates an skb from fast clone cache, setting pfmemalloc on the clone is not needed.

Clone will be properly initialized later at skb_clone() time, including pfmemalloc field, as it is included in the headers_start/headers_end section which is fully copied.

Applied.

#5 lore raw
In reply to Eric Dumazet
Quoted 9 lines · Eric Dumazet

I am not sure this is a proper fix.

Presumably if the socket was able to store packets in its write queue, fact that it sends it to loopback or an Ethernet link should not matter.

Only in RX path the pfmemalloc thing is really important.

So I would rather not set skb->pfmemalloc for skbs allocated for the write queue, and more exactly the fast clone.

Also note that any looped packet would have the problem : veth, macvlan, ipvlan, ...

#6 lore raw
In reply to Eric Dumazet
Quoted 49 lines · Eric Dumazet
Quoted 37 lines · Josef Bacik

I was seeing random disconnects while testing NBD over loopback.  This turned
out to be because NBD sets pfmemalloc on it's socket, however the receiving side
is a user space application so does not have pfmemalloc set on its socket.  This
means that sk_filter_trim_cap will simply drop this packet, under the assumption
that the other side will simply retransmit.  Well we do retransmit, and then the
packet is just dropped again for the same reason.  To keep this from happening
simply clear skb->pfmemalloc on transmit so that we don't drop the packet on the
receive side.

---
 drivers/net/loopback.c | 7 +++++++

 1 file changed, 7 insertions(+)
drivers/net/loopback.c +0 -0
index 1e05b7c..13c9126 100644
@@ -81,6 +81,13 @@ static netdev_tx_t loopback_xmit(struct sk_buff
 *skb,
  	 */
  	skb_dst_force(skb);

drivers/net/loopback.c +1 -0
+	/* If our transmitter was a pfmemalloc socket we need to

clear

drivers/net/loopback.c +1 -0
+	 * pfmemalloc here, otherwise the receiving socket may not

be

drivers/net/loopback.c +1 -0
+	 * pfmemalloc, and if this is a tcp packet then it'll get

dropped and

drivers/net/loopback.c

Quoted 4 lines
drivers/net/loopback.c +4 -0
+	 * all traffic will halt.
+	 */
+	skb->pfmemalloc = false;
+

I am not sure this is a proper fix.

Presumably if the socket was able to store packets in its write queue,
fact that it sends it to loopback or an Ethernet link should not matter.

Quoted 6 lines

Only in RX path the pfmemalloc thing is really important.

So I would rather not set skb->pfmemalloc for skbs allocated for the write queue, and more exactly the fast clone.

This would actually speed up the stack a bit.

The problem is we set skb->pfmemalloc a bunch of different places, such as __skb_fill_page_desc, which appears to be used in both the RX and TX path, so we can't just kill it there.  Do we want to go through and audit each one, provide a way for callers to indicate if we care about pfmemalloc and solve this problem that way?  I feel like that's more likely to bite us in the ass down the line, and somebody who doesn't know the context is going to come along and change it and regress us to the current situation.  The only place this is a problem is with loopback, and my change is contained to this one weird case.  Thanks,

Josef

#7 lore raw
Quoted 9 lines · Josef Bacik

The problem is we set skb->pfmemalloc a bunch of different places, such as __skb_fill_page_desc, which appears to be used in both the RX and TX path, so we can't just kill it there. Do we want to go through and audit each one, provide a way for callers to indicate if we care about pfmemalloc and solve this problem that way? I feel like that's more likely to bite us in the ass down the line, and somebody who doesn't know the context is going to come along and change it and regress us to the current situation. The only place this is a problem is with loopback, and my change is contained to this one weird case. Thanks,

I mentioned this in another mail :

Same issue will happen with veth, or any kind of driver allowing skb being given back to the stack in RX.

So your patch on loopback is not the definitive patch.

We probably should clear pf->memalloc directly in TCP write function.

Note that I clear it on the clone, not in original skb.

(It might be very useful to keep skb->pfmemalloc on original skbs in write queue, at least for debugging purposes)

net/ipv4/tcp_output.c +1 -0
index 8ce50dc3ab8cac821b8a2c3e0d31f0aa42f5c9d5..010280f1592d3bd195315882c364bdbbd4a1c2ec 100644
@@ -944,6 +944,7 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
                        skb = skb_clone(skb, gfp_mask);
                if (unlikely(!skb))
                        return -ENOBUFS;
+               skb->pfmemalloc = 0;
        }
 
        inet = inet_sk(sk);
author Acked-by
#8 lore raw
Quoted 46 lines · Eric Dumazet
Quoted 17 lines · Josef Bacik

The problem is we set skb->pfmemalloc a bunch of different places, such
as __skb_fill_page_desc, which appears to be used in both the RX and TX
path, so we can't just kill it there.  Do we want to go through and audit each one, provide a way for callers to indicate if we care about
pfmemalloc and solve this problem that way?  I feel like that's more
likely to bite us in the ass down the line, and somebody who doesn't
know the context is going to come along and change it and regress us to
the current situation.  The only place this is a problem is with loopback, and my change is contained to this one weird case.  Thanks,

I mentioned this in another mail :

Same issue will happen with veth, or any kind of driver allowing skb being given back to the stack in RX.

So your patch on loopback is not the definitive patch.

We probably should clear pf->memalloc directly in TCP write function.

Note that I clear it on the clone, not in original skb.

(It might be very useful to keep skb->pfmemalloc on original skbs in write queue, at least for debugging purposes)

net/ipv4/tcp_output.c +0 -0

index 8ce50dc3ab8cac821b8a2c3e0d31f0aa42f5c9d5..010280f1592d3bd195315882c36 4bdbbd4a1c2ec 100644

net/ipv4/tcp_output.c +1 -0
@@ -944,6 +944,7 @@ static int tcp_transmit_skb(struct sock *sk,
 struct sk_buff *skb, int clone_it,
                         skb = skb_clone(skb, gfp_mask);
                 if (unlikely(!skb))
                         return -ENOBUFS;
+               skb->pfmemalloc = 0;
         }
Quoted 4 lines


        inet = inet_sk(sk);

Yup this fixes my problem, you can add

when you send it.  Thanks,

Josef

#9 lore raw

PATCH 1/1 tcp: clear pfmemalloc on outgoing skb

In reply to Josef Bacik

From: Eric Dumazet <[email protected]>

Josef Bacik diagnosed following problem :

I was seeing random disconnects while testing NBD over loopback. This turned out to be because NBD sets pfmemalloc on it's socket, however the receiving side is a user space application so does not have pfmemalloc set on its socket. This means that sk_filter_trim_cap will simply drop this packet, under the assumption that the other side will simply retransmit. Well we do retransmit, and then the packet is just dropped again for the same reason.

It seems the better way to address this problem is to clear pfmemalloc in the TCP transmit path. pfmemalloc strict control really makes sense on the receive path.

---
 net/ipv4/tcp_output.c |    7 +++++++
 1 file changed, 7 insertions(+)
net/ipv4/tcp_output.c +7 -0
index 8ce50dc3ab8cac821b8a2c3e0d31f0aa42f5c9d5..a22545c8917cdc2382717e6176f7bb384e1d91f2 100644
@@ -966,6 +966,13 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
 	 */
 	skb->ooo_okay = sk_wmem_alloc_get(sk) < SKB_TRUESIZE(1);
 
+	/* If we had to use memory reserve to allocate this skb,
+	 * this might cause drops if packet is looped back :
+	 * Other socket might not have SOCK_MEMALLOC.
+	 * Packets not looped back do not care about pfmemalloc.
+	 */
+	skb->pfmemalloc = 0;
+
 	skb_push(skb, tcp_header_size);
 	skb_reset_transport_header(skb);
 
maintainer Applied
#10 lore raw

Re: [PATCH net-next] tcp: clear pfmemalloc on outgoing skb

From: Eric Dumazet <[email protected]>
Date: Thu, 02 Feb 2017 20:40:08 -0800

Quoted 13 lines

From: Eric Dumazet <[email protected]>

Josef Bacik diagnosed following problem :

I was seeing random disconnects while testing NBD over loopback. This turned out to be because NBD sets pfmemalloc on it's socket, however the receiving side is a user space application so does not have pfmemalloc set on its socket. This means that sk_filter_trim_cap will simply drop this packet, under the assumption that the other side will simply retransmit. Well we do retransmit, and then the packet is just dropped again for the same reason.

Quoted 6 lines

It seems the better way to address this problem is to clear pfmemalloc in the TCP transmit path. pfmemalloc strict control really makes sense on the receive path.

Applied.