Skip to contentSkip to conversation

xsk: various CPU barrier and {READ, WRITE}_ONCE fixes

PATCH Superseded Björn Töpel 5 participants 11 messages Aug 26, 2019 – Sep 3, 2019

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover xsk: various CPU barrier and {READ, WRITE}_ONCE fixes

This is a four patch series of various barrier, {READ, WRITE}_ONCE cleanups in the AF_XDP socket code. More details can be found in the corresponding commit message.

For an AF_XDP socket, most control plane operations are done under the control mutex (struct xdp_sock, mutex), but there are some places where members of the struct is read outside the control mutex. This, as pointed out by Daniel in [1], requires proper {READ, WRITE}_ONCE-correctness [2] [3]. To address this, and to simplify the code, the state variable (introduced by Ilya), is now used a point of synchronization ("is the socket in a valid state, or not").

Thanks,
Björn

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lwn.net/Articles/793253/
[3] https://github.com/google/ktsan/wiki/READ_ONCE-and-WRITE_ONCE

v1->v2:
Removed redundant dev check. (Jonathan)

Björn Töpel (4):
xsk: avoid store-tearing when assigning queues
xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state
xsk: avoid store-tearing when assigning umem
xsk: lock the control mutex in sock_diag interface

 net/xdp/xsk.c      | 61 +++++++++++++++++++++++++++++++---------------
 net/xdp/xsk_diag.c |  3 +++
 2 files changed, 45 insertions(+), 19 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/4 xsk: avoid store-tearing when assigning queues

From: Björn Töpel <[email protected]>

Use WRITE_ONCE when doing the store of tx, rx, fq, and cq, to avoid potential store-tearing. These members are read outside of the control mutex in the mmap implementation.

Fixes 37b076933a8e ("xsk: add missing write- and data-dependency barrier")
---
 net/xdp/xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/xdp/xsk.c +1 -1
index ee4428a892fa..f3351013c2a5 100644
@@ -409,7 +409,7 @@ static int xsk_init_queue(u32 entries, struct xsk_queue **queue,
 
 	/* Make sure queue is ready before it can be seen by others */
 	smp_wmb();
-	*queue = q;
+	WRITE_ONCE(*queue, q);
 	return 0;
 }
 
Signature · 1 line
2.20.1
#2 lore raw

PATCH v2 2/4 xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

The state variable was read, and written outside the control mutex (struct xdp_sock, mutex), without proper barriers and {READ, WRITE}_ONCE correctness.

In this commit this issue is addressed, and the state member is now used a point of synchronization whether the socket is setup correctly or not.

This also fixes a race, found by syzcaller, in xsk_poll() where umem could be accessed when stale.

Reported-by [email protected]
Fixes 77cd0d7b3f25 ("xsk: add support for need_wakeup flag in AF_XDP rings")
---
 net/xdp/xsk.c | 57 ++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 40 insertions(+), 17 deletions(-)
net/xdp/xsk.c +40 -17
index f3351013c2a5..8fafa3ce3ae6 100644
@@ -162,10 +162,23 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
 	return err;
 }
 
+static bool xsk_is_bound(struct xdp_sock *xs)
+{
+	if (READ_ONCE(xs->state) == XSK_BOUND) {
+		/* Matches smp_wmb() in bind(). */
+		smp_rmb();
+		return true;
+	}
+	return false;
+}
+
 int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
 {
 	u32 len;
 
+	if (!xsk_is_bound(xs))
+		return -EINVAL;
+
 	if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
 		return -EINVAL;
 
@@ -362,7 +375,7 @@ static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len)
 	struct sock *sk = sock->sk;
 	struct xdp_sock *xs = xdp_sk(sk);
 
-	if (unlikely(!xs->dev))
+	if (unlikely(!xsk_is_bound(xs)))
 		return -ENXIO;
 	if (unlikely(!(xs->dev->flags & IFF_UP)))
 		return -ENETDOWN;
@@ -378,10 +391,15 @@ static unsigned int xsk_poll(struct file *file, struct socket *sock,
 			     struct poll_table_struct *wait)
 {
 	unsigned int mask = datagram_poll(file, sock, wait);
-	struct sock *sk = sock->sk;
-	struct xdp_sock *xs = xdp_sk(sk);
-	struct net_device *dev = xs->dev;
-	struct xdp_umem *umem = xs->umem;
+	struct xdp_sock *xs = xdp_sk(sock->sk);
+	struct net_device *dev;
+	struct xdp_umem *umem;
+
+	if (unlikely(!xsk_is_bound(xs)))
+		return mask;
+
+	dev = xs->dev;
+	umem = xs->umem;
 
 	if (umem->need_wakeup)
 		dev->netdev_ops->ndo_xsk_wakeup(dev, xs->queue_id,
@@ -417,10 +435,9 @@ static void xsk_unbind_dev(struct xdp_sock *xs)
 {
 	struct net_device *dev = xs->dev;
 
-	if (!dev || xs->state != XSK_BOUND)
+	if (xs->state != XSK_BOUND)
 		return;
-
-	xs->state = XSK_UNBOUND;
+	WRITE_ONCE(xs->state, XSK_UNBOUND);
 
 	/* Wait for driver to stop using the xdp socket. */
 	xdp_del_sk_umem(xs->umem, xs);
@@ -495,7 +512,9 @@ static int xsk_release(struct socket *sock)
 	local_bh_enable();
 
 	xsk_delete_from_maps(xs);
+	mutex_lock(&xs->mutex);
 	xsk_unbind_dev(xs);
+	mutex_unlock(&xs->mutex);
 
 	xskq_destroy(xs->rx);
 	xskq_destroy(xs->tx);
@@ -589,19 +608,18 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
 		}
 
 		umem_xs = xdp_sk(sock->sk);
-		if (!umem_xs->umem) {
-			/* No umem to inherit. */
+		if (!xsk_is_bound(umem_xs)) {
 			err = -EBADF;
 			sockfd_put(sock);
 			goto out_unlock;
-		} else if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
+		}
+		if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
 			err = -EINVAL;
 			sockfd_put(sock);
 			goto out_unlock;
 		}
-
 		xdp_get_umem(umem_xs->umem);
-		xs->umem = umem_xs->umem;
+		WRITE_ONCE(xs->umem, umem_xs->umem);
 		sockfd_put(sock);
 	} else if (!xs->umem || !xdp_umem_validate_queues(xs->umem)) {
 		err = -EINVAL;
@@ -626,10 +644,15 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
 	xdp_add_sk_umem(xs->umem, xs);
 
 out_unlock:
-	if (err)
+	if (err) {
 		dev_put(dev);
-	else
-		xs->state = XSK_BOUND;
+	} else {
+		/* Matches smp_rmb() in bind() for shared umem
+		 * sockets, and xsk_is_bound().
+		 */
+		smp_wmb();
+		WRITE_ONCE(xs->state, XSK_BOUND);
+	}
 out_release:
 	mutex_unlock(&xs->mutex);
 	rtnl_unlock();
@@ -869,7 +892,7 @@ static int xsk_mmap(struct file *file, struct socket *sock,
 	unsigned long pfn;
 	struct page *qpg;
 
-	if (xs->state != XSK_READY)
+	if (READ_ONCE(xs->state) != XSK_READY)
 		return -EBUSY;
 
 	if (offset == XDP_PGOFF_RX_RING) {
Signature · 1 line
2.20.1
#3 lore raw

Re: [PATCH bpf-next v2 2/4] xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

Quoted 102 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The state variable was read, and written outside the control mutex (struct xdp_sock, mutex), without proper barriers and {READ, WRITE}_ONCE correctness.

In this commit this issue is addressed, and the state member is now used a point of synchronization whether the socket is setup correctly or not.

This also fixes a race, found by syzcaller, in xsk_poll() where umem could be accessed when stale.

Reported-by [email protected]
Fixes 77cd0d7b3f25 ("xsk: add support for need_wakeup flag in AF_XDP rings")
---
 net/xdp/xsk.c | 57 ++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 40 insertions(+), 17 deletions(-)
net/xdp/xsk.c +27 -8
index f3351013c2a5..8fafa3ce3ae6 100644
@@ -162,10 +162,23 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
 	return err;
 }
 
+static bool xsk_is_bound(struct xdp_sock *xs)
+{
+	if (READ_ONCE(xs->state) == XSK_BOUND) {
+		/* Matches smp_wmb() in bind(). */
+		smp_rmb();
+		return true;
+	}
+	return false;
+}
+
 int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
 {
 	u32 len;
 
+	if (!xsk_is_bound(xs))
+		return -EINVAL;
+
 	if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
 		return -EINVAL;
 
@@ -362,7 +375,7 @@ static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len)
 	struct sock *sk = sock->sk;
 	struct xdp_sock *xs = xdp_sk(sk);
 
-	if (unlikely(!xs->dev))
+	if (unlikely(!xsk_is_bound(xs)))
 		return -ENXIO;
 	if (unlikely(!(xs->dev->flags & IFF_UP)))
 		return -ENETDOWN;
@@ -378,10 +391,15 @@ static unsigned int xsk_poll(struct file *file, struct socket *sock,
 			     struct poll_table_struct *wait)
 {
 	unsigned int mask = datagram_poll(file, sock, wait);
-	struct sock *sk = sock->sk;
-	struct xdp_sock *xs = xdp_sk(sk);
-	struct net_device *dev = xs->dev;
-	struct xdp_umem *umem = xs->umem;
+	struct xdp_sock *xs = xdp_sk(sock->sk);
+	struct net_device *dev;
+	struct xdp_umem *umem;
+
+	if (unlikely(!xsk_is_bound(xs)))
+		return mask;
+
+	dev = xs->dev;
+	umem = xs->umem;
 
 	if (umem->need_wakeup)
 		dev->netdev_ops->ndo_xsk_wakeup(dev, xs->queue_id,
@@ -417,10 +435,9 @@ static void xsk_unbind_dev(struct xdp_sock *xs)
 {
 	struct net_device *dev = xs->dev;
 
-	if (!dev || xs->state != XSK_BOUND)
+	if (xs->state != XSK_BOUND)
 		return;
-
-	xs->state = XSK_UNBOUND;
+	WRITE_ONCE(xs->state, XSK_UNBOUND);
 
 	/* Wait for driver to stop using the xdp socket. */
 	xdp_del_sk_umem(xs->umem, xs);
@@ -495,7 +512,9 @@ static int xsk_release(struct socket *sock)
 	local_bh_enable();
 
 	xsk_delete_from_maps(xs);
+	mutex_lock(&xs->mutex);
 	xsk_unbind_dev(xs);
+	mutex_unlock(&xs->mutex);
 
 	xskq_destroy(xs->rx);
 	xskq_destroy(xs->tx);

net/xdp/xsk.c static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) · line 611

Quoted 6 lines
net/xdp/xsk.c +1 -2
@@ -589,19 +608,18 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
 		}
 
 		umem_xs = xdp_sk(sock->sk);
-		if (!umem_xs->umem) {
-			/* No umem to inherit. */
+		if (!xsk_is_bound(umem_xs)) {

This changes the error code a bit.
Previously:
umem exists + xs unbound --> EINVAL
no umem + xs unbound --> EBADF
xs bound to different dev/q --> EINVAL

With this change:
umem exists + xs unbound --> EBADF
no umem + xs unbound --> EBADF
xs bound to different dev/q --> EINVAL

Just a note. Not sure if this is important.

Quoted 46 lines
net/xdp/xsk.c +12 -7
 			err = -EBADF;
 			sockfd_put(sock);
 			goto out_unlock;
-		} else if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
+		}
+		if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
 			err = -EINVAL;
 			sockfd_put(sock);
 			goto out_unlock;
 		}
-
 		xdp_get_umem(umem_xs->umem);
-		xs->umem = umem_xs->umem;
+		WRITE_ONCE(xs->umem, umem_xs->umem);
 		sockfd_put(sock);
 	} else if (!xs->umem || !xdp_umem_validate_queues(xs->umem)) {
 		err = -EINVAL;
@@ -626,10 +644,15 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
 	xdp_add_sk_umem(xs->umem, xs);
 
 out_unlock:
-	if (err)
+	if (err) {
 		dev_put(dev);
-	else
-		xs->state = XSK_BOUND;
+	} else {
+		/* Matches smp_rmb() in bind() for shared umem
+		 * sockets, and xsk_is_bound().
+		 */
+		smp_wmb();
+		WRITE_ONCE(xs->state, XSK_BOUND);
+	}
 out_release:
 	mutex_unlock(&xs->mutex);
 	rtnl_unlock();
@@ -869,7 +892,7 @@ static int xsk_mmap(struct file *file, struct socket *sock,
 	unsigned long pfn;
 	struct page *qpg;
 
-	if (xs->state != XSK_READY)
+	if (READ_ONCE(xs->state) != XSK_READY)
 		return -EBUSY;
 
 	if (offset == XDP_PGOFF_RX_RING) {
#4 lore raw

Re: [PATCH bpf-next v2 2/4] xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

Quoted 6 lines · Ilya Maximets

This changes the error code a bit.
Previously:

    umem exists + xs unbound    --> EINVAL
    no umem     + xs unbound    --> EBADF
    xs bound to different dev/q --> EINVAL
Quoted 7 lines

With this change:

    umem exists + xs unbound    --> EBADF
    no umem     + xs unbound    --> EBADF
    xs bound to different dev/q --> EINVAL

Just a note. Not sure if this is important.

Note that this is for *shared* umem, so it's very seldom used. Still, you're right, that strictly this is an uapi break, but I'd vote for the change still. I find it hard to see that anyone relies on EINVAL/EBADF for shared umem bind.

Opinions? :-)

Björn

#5 lore raw

Re: [PATCH bpf-next v2 2/4] xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

Quoted 15 lines · Björn Töpel
Quoted 6 lines · Ilya Maximets

This changes the error code a bit.
Previously:

    umem exists + xs unbound    --> EINVAL
    no umem     + xs unbound    --> EBADF
    xs bound to different dev/q --> EINVAL
Quoted 7 lines

With this change:

    umem exists + xs unbound    --> EBADF
    no umem     + xs unbound    --> EBADF
    xs bound to different dev/q --> EINVAL

Just a note. Not sure if this is important.

Quoted 6 lines

Note that this is for *shared* umem, so it's very seldom used. Still, you're right, that strictly this is an uapi break, but I'd vote for the change still. I find it hard to see that anyone relies on EINVAL/EBADF for shared umem bind.

Opinions? :-)

I'd agree - if it isn't documented somewhere, it's not an API break. :)

Signature · 1 line
Jonathan
Acked-by on v2 2/4
#6 lore raw

Re: [PATCH bpf-next v2 2/4] xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

In reply to Björn Töpel
Quoted 10 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The state variable was read, and written outside the control mutex (struct xdp_sock, mutex), without proper barriers and {READ, WRITE}_ONCE correctness.

In this commit this issue is addressed, and the state member is now used a point of synchronization whether the socket is setup correctly or not.

Quoted 7 lines

This also fixes a race, found by syzcaller, in xsk_poll() where umem could be accessed when stale.

Reported-by [email protected]
Fixes 77cd0d7b3f25 ("xsk: add support for need_wakeup flag in AF_XDP rings")
maintainer
#7 lore raw

Re: [PATCH bpf-next v2 2/4] xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

In reply to Björn Töpel
Quoted 10 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The state variable was read, and written outside the control mutex (struct xdp_sock, mutex), without proper barriers and {READ, WRITE}_ONCE correctness.

In this commit this issue is addressed, and the state member is now used a point of synchronization whether the socket is setup correctly or not.

Quoted 7 lines

This also fixes a race, found by syzcaller, in xsk_poll() where umem could be accessed when stale.

Reported-by [email protected]
Fixes 77cd0d7b3f25 ("xsk: add support for need_wakeup flag in AF_XDP rings")

Sorry for the delay.

Quoted 112 lines
---
  net/xdp/xsk.c | 57 ++++++++++++++++++++++++++++++++++++---------------
  1 file changed, 40 insertions(+), 17 deletions(-)
net/xdp/xsk.c +31 -13
index f3351013c2a5..8fafa3ce3ae6 100644
@@ -162,10 +162,23 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
  	return err;
  }
 
+static bool xsk_is_bound(struct xdp_sock *xs)
+{
+	if (READ_ONCE(xs->state) == XSK_BOUND) {
+		/* Matches smp_wmb() in bind(). */
+		smp_rmb();
+		return true;
+	}
+	return false;
+}
+
  int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
  {
  	u32 len;
 
+	if (!xsk_is_bound(xs))
+		return -EINVAL;
+
  	if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
  		return -EINVAL;
 
@@ -362,7 +375,7 @@ static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len)
  	struct sock *sk = sock->sk;
  	struct xdp_sock *xs = xdp_sk(sk);
 
-	if (unlikely(!xs->dev))
+	if (unlikely(!xsk_is_bound(xs)))
  		return -ENXIO;
  	if (unlikely(!(xs->dev->flags & IFF_UP)))
  		return -ENETDOWN;
@@ -378,10 +391,15 @@ static unsigned int xsk_poll(struct file *file, struct socket *sock,
  			     struct poll_table_struct *wait)
  {
  	unsigned int mask = datagram_poll(file, sock, wait);
-	struct sock *sk = sock->sk;
-	struct xdp_sock *xs = xdp_sk(sk);
-	struct net_device *dev = xs->dev;
-	struct xdp_umem *umem = xs->umem;
+	struct xdp_sock *xs = xdp_sk(sock->sk);
+	struct net_device *dev;
+	struct xdp_umem *umem;
+
+	if (unlikely(!xsk_is_bound(xs)))
+		return mask;
+
+	dev = xs->dev;
+	umem = xs->umem;
 
  	if (umem->need_wakeup)
  		dev->netdev_ops->ndo_xsk_wakeup(dev, xs->queue_id,
@@ -417,10 +435,9 @@ static void xsk_unbind_dev(struct xdp_sock *xs)
  {
  	struct net_device *dev = xs->dev;
 
-	if (!dev || xs->state != XSK_BOUND)
+	if (xs->state != XSK_BOUND)
  		return;
-
-	xs->state = XSK_UNBOUND;
+	WRITE_ONCE(xs->state, XSK_UNBOUND);
 
  	/* Wait for driver to stop using the xdp socket. */
  	xdp_del_sk_umem(xs->umem, xs);
@@ -495,7 +512,9 @@ static int xsk_release(struct socket *sock)
  	local_bh_enable();
 
  	xsk_delete_from_maps(xs);
+	mutex_lock(&xs->mutex);
  	xsk_unbind_dev(xs);
+	mutex_unlock(&xs->mutex);
 
  	xskq_destroy(xs->rx);
  	xskq_destroy(xs->tx);
@@ -589,19 +608,18 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
  		}
 
  		umem_xs = xdp_sk(sock->sk);
-		if (!umem_xs->umem) {
-			/* No umem to inherit. */
+		if (!xsk_is_bound(umem_xs)) {
  			err = -EBADF;
  			sockfd_put(sock);
  			goto out_unlock;
-		} else if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
+		}
+		if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
  			err = -EINVAL;
  			sockfd_put(sock);
  			goto out_unlock;
  		}
-
  		xdp_get_umem(umem_xs->umem);
-		xs->umem = umem_xs->umem;
+		WRITE_ONCE(xs->umem, umem_xs->umem);
  		sockfd_put(sock);
  	} else if (!xs->umem || !xdp_umem_validate_queues(xs->umem)) {
  		err = -EINVAL;
@@ -626,10 +644,15 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
  	xdp_add_sk_umem(xs->umem, xs);
 
  out_unlock:

net/xdp/xsk.c static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) · line 653

Quoted 10 lines
net/xdp/xsk.c +6 -3
-	if (err)
+	if (err) {
  		dev_put(dev);
-	else
-		xs->state = XSK_BOUND;
+	} else {
+		/* Matches smp_rmb() in bind() for shared umem
+		 * sockets, and xsk_is_bound().
+		 */
+		smp_wmb();

You write with what this barrier matches/pairs, but would be useful for readers to have an explanation against what it protects. I presume to have things like xs->umem public as you seem to guard it behind xsk_is_bound() in xsk_poll() and other cases? Would be great to have a detailed analysis of all this e.g. in the commit message so one wouldn't need to guess; right now it feels this is doing many things at once and w/o further explanation of why READ_ONCE() or others are omitted sometimes. Would be great to get a lot more clarity into this, perhaps splitting it up a bit might also help.

net/xdp/xsk.c static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) · line 655

Quoted 2 lines
net/xdp/xsk.c +2 -0
+		WRITE_ONCE(xs->state, XSK_BOUND);
+	}

Thanks,
Daniel

#8 lore raw

Re: [PATCH bpf-next v2 2/4] xsk: add proper barriers and {READ, WRITE}_ONCE-correctness for state

Quoted 145 lines · Daniel Borkmann
Quoted 10 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The state variable was read, and written outside the control mutex (struct xdp_sock, mutex), without proper barriers and {READ, WRITE}_ONCE correctness.

In this commit this issue is addressed, and the state member is now used a point of synchronization whether the socket is setup correctly or not.

Quoted 7 lines

This also fixes a race, found by syzcaller, in xsk_poll() where umem could be accessed when stale.

Reported-by [email protected]
Fixes 77cd0d7b3f25 ("xsk: add support for need_wakeup flag in AF_XDP rings")

Sorry for the delay.

Quoted 112 lines
---
  net/xdp/xsk.c | 57 ++++++++++++++++++++++++++++++++++++---------------
  1 file changed, 40 insertions(+), 17 deletions(-)
net/xdp/xsk.c +31 -13
index f3351013c2a5..8fafa3ce3ae6 100644
@@ -162,10 +162,23 @@ static int __xsk_rcv_zc(struct xdp_sock *xs, struct xdp_buff *xdp, u32 len)
      return err;
  }
 
+static bool xsk_is_bound(struct xdp_sock *xs)
+{
+     if (READ_ONCE(xs->state) == XSK_BOUND) {
+             /* Matches smp_wmb() in bind(). */
+             smp_rmb();
+             return true;
+     }
+     return false;
+}
+
  int xsk_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
  {
      u32 len;
 
+     if (!xsk_is_bound(xs))
+             return -EINVAL;
+
      if (xs->dev != xdp->rxq->dev || xs->queue_id != xdp->rxq->queue_index)
              return -EINVAL;
 
@@ -362,7 +375,7 @@ static int xsk_sendmsg(struct socket *sock, struct msghdr *m, size_t total_len)
      struct sock *sk = sock->sk;
      struct xdp_sock *xs = xdp_sk(sk);
 
-     if (unlikely(!xs->dev))
+     if (unlikely(!xsk_is_bound(xs)))
              return -ENXIO;
      if (unlikely(!(xs->dev->flags & IFF_UP)))
              return -ENETDOWN;
@@ -378,10 +391,15 @@ static unsigned int xsk_poll(struct file *file, struct socket *sock,
                           struct poll_table_struct *wait)
  {
      unsigned int mask = datagram_poll(file, sock, wait);
-     struct sock *sk = sock->sk;
-     struct xdp_sock *xs = xdp_sk(sk);
-     struct net_device *dev = xs->dev;
-     struct xdp_umem *umem = xs->umem;
+     struct xdp_sock *xs = xdp_sk(sock->sk);
+     struct net_device *dev;
+     struct xdp_umem *umem;
+
+     if (unlikely(!xsk_is_bound(xs)))
+             return mask;
+
+     dev = xs->dev;
+     umem = xs->umem;
 
      if (umem->need_wakeup)
              dev->netdev_ops->ndo_xsk_wakeup(dev, xs->queue_id,
@@ -417,10 +435,9 @@ static void xsk_unbind_dev(struct xdp_sock *xs)
  {
      struct net_device *dev = xs->dev;
 
-     if (!dev || xs->state != XSK_BOUND)
+     if (xs->state != XSK_BOUND)
              return;
-
-     xs->state = XSK_UNBOUND;
+     WRITE_ONCE(xs->state, XSK_UNBOUND);
 
      /* Wait for driver to stop using the xdp socket. */
      xdp_del_sk_umem(xs->umem, xs);
@@ -495,7 +512,9 @@ static int xsk_release(struct socket *sock)
      local_bh_enable();
 
      xsk_delete_from_maps(xs);
+     mutex_lock(&xs->mutex);
      xsk_unbind_dev(xs);
+     mutex_unlock(&xs->mutex);
 
      xskq_destroy(xs->rx);
      xskq_destroy(xs->tx);
@@ -589,19 +608,18 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
              }
 
              umem_xs = xdp_sk(sock->sk);
-             if (!umem_xs->umem) {
-                     /* No umem to inherit. */
+             if (!xsk_is_bound(umem_xs)) {
                      err = -EBADF;
                      sockfd_put(sock);
                      goto out_unlock;
-             } else if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
+             }
+             if (umem_xs->dev != dev || umem_xs->queue_id != qid) {
                      err = -EINVAL;
                      sockfd_put(sock);
                      goto out_unlock;
              }
-
              xdp_get_umem(umem_xs->umem);
-             xs->umem = umem_xs->umem;
+             WRITE_ONCE(xs->umem, umem_xs->umem);
              sockfd_put(sock);
      } else if (!xs->umem || !xdp_umem_validate_queues(xs->umem)) {
              err = -EINVAL;
@@ -626,10 +644,15 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
      xdp_add_sk_umem(xs->umem, xs);
 
  out_unlock:

net/xdp/xsk.c static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) · line 653

Quoted 10 lines
net/xdp/xsk.c +6 -3
-     if (err)
+     if (err) {
              dev_put(dev);
-     else
-             xs->state = XSK_BOUND;
+     } else {
+             /* Matches smp_rmb() in bind() for shared umem
+              * sockets, and xsk_is_bound().
+              */
+             smp_wmb();
Quoted 9 lines

You write with what this barrier matches/pairs, but would be useful for readers to have an explanation against what it protects. I presume to have things like xs->umem public as you seem to guard it behind xsk_is_bound() in xsk_poll() and other cases? Would be great to have a detailed analysis of all this e.g. in the commit message so one wouldn't need to guess; right now it feels this is doing many things at once and w/o further explanation of why READ_ONCE() or others are omitted sometimes. Would be great to get a lot more clarity into this, perhaps splitting it up a bit might also help.

I'll address that. Thanks for the review!

Björn

Quoted 5 lines

net/xdp/xsk.c static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) · line 655

Quoted 2 lines
net/xdp/xsk.c +2 -0
+             WRITE_ONCE(xs->state, XSK_BOUND);
+     }

Thanks,
Daniel

#9 lore raw

PATCH v2 3/4 xsk: avoid store-tearing when assigning umem

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

The umem member of struct xdp_sock is read outside of the control mutex, in the mmap implementation, and needs a WRITE_ONCE to avoid potentional store-tearing.

Fixes 423f38329d26 ("xsk: add umem fill queue support and mmap")
---
 net/xdp/xsk.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/xdp/xsk.c +1 -1
index 8fafa3ce3ae6..e3e99ee5631b 100644
@@ -716,7 +716,7 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
 
 		/* Make sure umem is ready before it can be seen by others */
 		smp_wmb();
-		xs->umem = umem;
+		WRITE_ONCE(xs->umem, umem);
 		mutex_unlock(&xs->mutex);
 		return 0;
 	}
Signature · 1 line
2.20.1
#10 lore raw

PATCH v2 4/4 xsk: lock the control mutex in sock_diag interface

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

When accessing the members of an XDP socket, the control mutex should be held. This commit fixes that.

Fixes a36b38aa2af6 ("xsk: add sock_diag interface for AF_XDP")
---
 net/xdp/xsk_diag.c | 3 +++
 1 file changed, 3 insertions(+)
net/xdp/xsk_diag.c +3 -0
index d5e06c8e0cbf..c8f4f11edbbc 100644
@@ -97,6 +97,7 @@ static int xsk_diag_fill(struct sock *sk, struct sk_buff *nlskb,
 	msg->xdiag_ino = sk_ino;
 	sock_diag_save_cookie(sk, msg->xdiag_cookie);
 
+	mutex_lock(&xs->mutex);
 	if ((req->xdiag_show & XDP_SHOW_INFO) && xsk_diag_put_info(xs, nlskb))
 		goto out_nlmsg_trim;
 
@@ -117,10 +118,12 @@ static int xsk_diag_fill(struct sock *sk, struct sk_buff *nlskb,
 	    sock_diag_put_meminfo(sk, nlskb, XDP_DIAG_MEMINFO))
 		goto out_nlmsg_trim;
 
+	mutex_unlock(&xs->mutex);
 	nlmsg_end(nlskb, nlh);
 	return 0;
 
 out_nlmsg_trim:
+	mutex_unlock(&xs->mutex);
 	nlmsg_cancel(nlskb, nlh);
 	return -EMSGSIZE;
 }
Signature · 1 line
2.20.1