Skip to contentSkip to conversation

dumpcap: fix mbuf pool ring type

PATCH Superseded Stephen Hemminger 4 participants 18 messages Aug 4, 2023 – Nov 12, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dumpcap: fix mbuf pool ring type

The ring used to store mbufs needs to be multiple producer, multiple consumer because multiple queues might on multiple cores might be allocating and same time (consume) and in case of ring full, the mbufs will be returned (multiple producer).

Bugzilla ID: 1271

Fixes cb2440fd77af ("dumpcap: fix mbuf pool ring type")
---
 app/dumpcap/main.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
app/dumpcap/main.c +3 -4
index 64294bbfb3e6..991174e95022 100644
@@ -691,10 +691,9 @@ static struct rte_mempool *create_mempool(void)
 			data_size = mbuf_size;
 	}
 
-	mp = rte_pktmbuf_pool_create_by_ops(pool_name, num_mbufs,
-					    MBUF_POOL_CACHE_SIZE, 0,
-					    data_size,
-					    rte_socket_id(), "ring_mp_sc");
+	mp = rte_pktmbuf_pool_create(pool_name, num_mbufs,
+				     MBUF_POOL_CACHE_SIZE, 0,
+				     data_size, rte_socket_id());
 	if (mp == NULL)
 		rte_exit(EXIT_FAILURE,
 			 "Mempool (%s) creation failed: %s\n", pool_name,
Signature · 1 line
2.39.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Friday, 4 August 2023 18.16

The ring used to store mbufs needs to be multiple producer, multiple consumer because multiple queues might on multiple cores might be allocating and same time (consume) and in case of ring full, the mbufs will be returned (multiple producer).

Quoted 4 lines

Bugzilla ID: 1271

Fixes cb2440fd77af ("dumpcap: fix mbuf pool ring type")

---

#2 lore raw
In reply to Stephen Hemminger
Quoted 5 lines · Stephen Hemminger

The ring used to store mbufs needs to be multiple producer, multiple consumer because multiple queues might on multiple cores might be allocating and same time (consume) and in case of ring full, the mbufs will be returned (multiple producer).

I think I get the idea, but can you rephrase please?

Quoted 3 lines

Bugzilla ID: 1271

Fixes cb2440fd77af ("dumpcap: fix mbuf pool ring type")

This Fixes: tag looks wrong.

Quoted 10 lines
---
 app/dumpcap/main.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
app/dumpcap/main.c +0 -0
index 64294bbfb3e6..991174e95022 100644

app/dumpcap/main.c static struct rte_mempool *create_mempool(void) · line 696

Quoted 10 lines
app/dumpcap/main.c +3 -4
@@ -691,10 +691,9 @@ static struct rte_mempool *create_mempool(void)
                        data_size = mbuf_size;
        }
 
-       mp = rte_pktmbuf_pool_create_by_ops(pool_name, num_mbufs,
-                                           MBUF_POOL_CACHE_SIZE, 0,
-                                           data_size,
-                                           rte_socket_id(), "ring_mp_sc");
+       mp = rte_pktmbuf_pool_create(pool_name, num_mbufs,
+                                    MBUF_POOL_CACHE_SIZE, 0,
+                                    data_size, rte_socket_id());

Switching to rte_pktmbuf_pool_create() still leaves the user with the possibility to shoot himself in the foot (I was thinking of setting some --mbuf-pool-ops-name EAL option).

This application has explicit requirements in terms of concurrent access (and I don't think the mempool library exposes per driver capabilities in that regard).
The application was enforcing the use of mempool/ring so far.

I think it is safer to go with an explicit
rte_pktmbuf_pool_create_by_ops(... "ring_mp_mc").
WDYT?

Quoted 6 lines
app/dumpcap/main.c +0 -1
        if (mp == NULL)
                rte_exit(EXIT_FAILURE,
                         "Mempool (%s) creation failed: %s\n", pool_name,
--

2.39.2

Thanks.

Signature · 1 line
David Marchand
#3 lore raw
Quoted 46 lines

From: David Marchand [mailto:[email protected]]
Sent: Monday, 2 October 2023 09.34

Quoted 5 lines · Stephen Hemminger

The ring used to store mbufs needs to be multiple producer, multiple consumer because multiple queues might on multiple cores might be allocating and same time (consume) and in case of ring full, the mbufs will be returned (multiple producer).

I think I get the idea, but can you rephrase please?

Quoted 3 lines

Bugzilla ID: 1271

Fixes cb2440fd77af ("dumpcap: fix mbuf pool ring type")

This Fixes: tag looks wrong.

Quoted 10 lines
---
 app/dumpcap/main.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
app/dumpcap/main.c +0 -0
index 64294bbfb3e6..991174e95022 100644

app/dumpcap/main.c static struct rte_mempool *create_mempool(void) · line 696

Quoted 10 lines
app/dumpcap/main.c +3 -4
@@ -691,10 +691,9 @@ static struct rte_mempool *create_mempool(void)
                        data_size = mbuf_size;
        }
 
-       mp = rte_pktmbuf_pool_create_by_ops(pool_name, num_mbufs,
-                                           MBUF_POOL_CACHE_SIZE, 0,
-                                           data_size,
-                                           rte_socket_id(), "ring_mp_sc");
+       mp = rte_pktmbuf_pool_create(pool_name, num_mbufs,
+                                    MBUF_POOL_CACHE_SIZE, 0,
+                                    data_size, rte_socket_id());

Switching to rte_pktmbuf_pool_create() still leaves the user with the possibility to shoot himself in the foot (I was thinking of setting some --mbuf-pool-ops-name EAL option).

Quoted 8 lines

This application has explicit requirements in terms of concurrent access (and I don't think the mempool library exposes per driver capabilities in that regard).
The application was enforcing the use of mempool/ring so far.

I think it is safer to go with an explicit
rte_pktmbuf_pool_create_by_ops(... "ring_mp_mc").
WDYT?

<feature creep>
Or perhaps one of "ring_mt_rts" or "ring_mt_hts", if any of those mbuf pool drivers are specified on the command line; otherwise fall back to "ring_mp_mc".

Actually, I prefer Stephen's suggestion of using the default mbuf pool driver. The option is there for a reason.

However, David is right: We want to prevent the user from using a thread-unsafe mempool driver in this use case.

And I guess there might be other use cases than this one, where a thread-safe mempool driver is required. So adding a generalized function to get the "upgraded" (i.e. thread safe) variant of a mempool driver would be nice. </feature creep>

Feel free to ignore my suggested feature creep, and go ahead with David's suggestion instead.

Quoted 13 lines
Quoted 8 lines
app/dumpcap/main.c +0 -1
        if (mp == NULL)
                rte_exit(EXIT_FAILURE,
                         "Mempool (%s) creation failed: %s\n", pool_name,
--

2.39.2

Thanks.

--
David Marchand

#4 lore raw
Quoted 16 lines · On Mon, 2 Oct 2023 10:42:53 +0200 Morten Brørup <[email protected]>…
Quoted 4 lines

Switching to rte_pktmbuf_pool_create() still leaves the user with the possibility to shoot himself in the foot (I was thinking of setting some --mbuf-pool-ops-name EAL option).

Quoted 8 lines

This application has explicit requirements in terms of concurrent access (and I don't think the mempool library exposes per driver capabilities in that regard).
The application was enforcing the use of mempool/ring so far.

I think it is safer to go with an explicit
rte_pktmbuf_pool_create_by_ops(... "ring_mp_mc").
WDYT?

<feature creep>
Or perhaps one of "ring_mt_rts" or "ring_mt_hts", if any of those mbuf pool drivers are specified on the command line; otherwise fall back to "ring_mp_mc".

Quoted 6 lines

Actually, I prefer Stephen's suggestion of using the default mbuf pool driver. The option is there for a reason.

However, David is right: We want to prevent the user from using a thread-unsafe mempool driver in this use case.

And I guess there might be other use cases than this one, where a thread-safe mempool driver is required. So adding a generalized function to get the "upgraded" (i.e. thread safe) variant of a mempool driver would be nice. </feature creep>

If the user overrides the default mbuf pool type, then it will need to be thread safe for the general case of driver as well (or they are on single cpu).

I think the patch should be applied as is.

#5 lore raw
Quoted 32 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Monday, 6 November 2023 20.24

Quoted 1 line · On Mon, 2 Oct 2023 10:42:53 +0200 Morten Brørup <[email protected]>…
Quoted 1 line

Switching to rte_pktmbuf_pool_create() still leaves the user with

the

Quoted 8 lines
Quoted 8 lines

possibility to shoot himself in the foot (I was thinking of setting some --mbuf-pool-ops-name EAL option).

This application has explicit requirements in terms of concurrent access (and I don't think the mempool library exposes per driver capabilities in that regard).
The application was enforcing the use of mempool/ring so far.

Quoted 6 lines
Quoted 3 lines

I think it is safer to go with an explicit
rte_pktmbuf_pool_create_by_ops(... "ring_mp_mc").
WDYT?

<feature creep>
Or perhaps one of "ring_mt_rts" or "ring_mt_hts", if any of those

mbuf pool drivers are specified on the command line; otherwise fall back to "ring_mp_mc".

Quoted 2 lines

Actually, I prefer Stephen's suggestion of using the default mbuf

pool driver. The option is there for a reason.

Quoted 2 lines

However, David is right: We want to prevent the user from using a

thread-unsafe mempool driver in this use case.

Quoted 2 lines

And I guess there might be other use cases than this one, where a

Quoted 8 lines

thread-safe mempool driver is required. So adding a generalized function to get the "upgraded" (i.e. thread safe) variant of a mempool driver would be nice.

Quoted 1 line

</feature creep>

If the user overrides the default mbuf pool type, then it will need to be thread safe for
the general case of driver as well (or they are on single cpu).

If they have chosen a thread safe pool type, using the chosen type will work for dumpcap too. I think we all agree on that.

I'm not sure I understand your single-cpu argument, so let me try to paraphrase:

If their application only uses one EAL thread, and they have chosen "ring_sc_sp", dumpcap also work, because mempool accesses are still single-threaded. Is this correctly understood?

Or are you saying that if they want to use dumpcap, they must choose a thread safe pool type for their application (regardless if the application is single-threaded or not)?

Quoted 3 lines

I think the patch should be applied as is.

#6 lore raw
Quoted 10 lines · On Mon, 6 Nov 2023 22:50:50 +0100 Morten Brørup <[email protected]>…
Quoted 9 lines
Quoted 1 line

And I guess there might be other use cases than this one, where a

thread-safe mempool driver is required. So adding a generalized function to get the "upgraded" (i.e. thread safe) variant of a mempool driver would be nice.

Quoted 1 line

</feature creep>

If the user overrides the default mbuf pool type, then it will need to be thread safe for
the general case of driver as well (or they are on single cpu).

Quoted 7 lines

If they have chosen a thread safe pool type, using the chosen type will work for dumpcap too. I think we all agree on that.

I'm not sure I understand your single-cpu argument, so let me try to paraphrase:

If their application only uses one EAL thread, and they have chosen "ring_sc_sp", dumpcap also work, because mempool accesses are still single-threaded. Is this correctly understood?

Or are you saying that if they want to use dumpcap, they must choose a thread safe pool type for their application (regardless if the application is single-threaded or not)?

There is no command line of EAL nature in dumpcap. This is intentional.
QED: overriding default pool type is not going to be a possible

#7 lore raw
Quoted 28 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Tuesday, 7 November 2023 03.36

Quoted 3 lines · On Mon, 6 Nov 2023 22:50:50 +0100 Morten Brørup <[email protected]>…
Quoted 3 lines
Quoted 1 line

And I guess there might be other use cases than this one, where a

thread-safe mempool driver is required. So adding a generalized function to get the "upgraded" (i.e. thread safe) variant of a

mempool

Quoted 4 lines
Quoted 4 lines

driver would be nice.

Quoted 1 line

</feature creep>

If the user overrides the default mbuf pool type, then it will need

to

Quoted 4 lines
Quoted 2 lines

be thread safe for
the general case of driver as well (or they are on single cpu).

If they have chosen a thread safe pool type, using the chosen type

will work for dumpcap too. I think we all agree on that.

Quoted 2 lines

I'm not sure I understand your single-cpu argument, so let me try to

paraphrase:

Quoted 2 lines

If their application only uses one EAL thread, and they have chosen

"ring_sc_sp", dumpcap also work, because mempool accesses are still single-threaded. Is this correctly understood?

Quoted 1 line
Quoted 7 lines
Quoted 1 line

Or are you saying that if they want to use dumpcap, they must choose

a thread safe pool type for their application (regardless if the application is single-threaded or not)?

There is no command line of EAL nature in dumpcap. This is intentional.
QED: overriding default pool type is not going to be a possible

The preferred mbuf pool type can configured in the primary process by EAL params. If so configured, it is stored in a memzone named "mbuf_user_pool_ops". And if it is set there, the secondary process will also use it as its preferred mbuf pool type.

#8 lore raw
Quoted 11 lines · On Tue, 7 Nov 2023 08:22:37 +0100 Morten Brørup <[email protected]>…
Quoted 8 lines
Quoted 2 lines

Or are you saying that if they want to use dumpcap, they must choose

a thread safe pool type for their application (regardless if the application is single-threaded or not)?

There is no command line of EAL nature in dumpcap. This is intentional.
QED: overriding default pool type is not going to be a possible

The preferred mbuf pool type can configured in the primary process by EAL params. If so configured, it is stored in a memzone named "mbuf_user_pool_ops". And if it is set there, the secondary process will also use it as its preferred mbuf pool type.

If user decides to use a thread unsafe mempool, wouldn't it break in every PMD as well.

#9 lore raw
Quoted 16 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Tuesday, 7 November 2023 17.41

Quoted 2 lines · On Tue, 7 Nov 2023 08:22:37 +0100 Morten Brørup <[email protected]>…
Quoted 2 lines
Quoted 2 lines

Or are you saying that if they want to use dumpcap, they must

choose

Quoted 7 lines
Quoted 6 lines

a thread safe pool type for their application (regardless if the application is single-threaded or not)?

There is no command line of EAL nature in dumpcap. This is intentional.
QED: overriding default pool type is not going to be a possible

Quoted 8 lines
Quoted 1 line

The preferred mbuf pool type can configured in the primary process by

EAL params. If so configured, it is stored in a memzone named "mbuf_user_pool_ops".

Quoted 1 line

And if it is set there, the secondary process will also use it as its

preferred mbuf pool type.

If user decides to use a thread unsafe mempool, wouldn't it break in every PMD as well.

Yes, except if the application only uses one thread for packet processing. Then thread safety is not necessary.

#10 lore raw
Quoted 5 lines · On Tue, 7 Nov 2023 18:38:32 +0100 Morten Brørup <[email protected]>…
Quoted 2 lines

If user decides to use a thread unsafe mempool, wouldn't it break in every PMD as well.

Yes, except if the application only uses one thread for packet processing. Then thread safety is not necessary.

If application only used single thread then single consumer pool would work fine for dumpcap. Only the primary thread would ever allocate. But both secondary and primary could be freeing mbuf.

#11 lore raw
Quoted 15 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Wednesday, 8 November 2023 17.51

Quoted 1 line · On Tue, 7 Nov 2023 18:38:32 +0100 Morten Brørup <[email protected]>…
Quoted 1 line

If user decides to use a thread unsafe mempool, wouldn't it break

in

Quoted 3 lines
Quoted 1 line

every PMD as well.

Yes, except if the application only uses one thread for packet

processing. Then thread safety is not necessary.

Quoted 1 line
Quoted 5 lines

If application only used single thread then single consumer pool would work fine
for dumpcap. Only the primary thread would ever allocate. But both secondary and
primary could be freeing mbuf.

I think it is an acceptable risk. Perhaps mention somewhere that dumpcap breaks the application if it uses non-thread safe mbuf ops.

Optionally, dumpcap could compare the return value of rte_mbuf_best_mempool_ops() [1] with the few known unsafe ops. However, this is probably going to be forgotten, if new non-thread safe mempool ops are ever added to DPDK.

[1]: https://elixir.bootlin.com/dpdk/latest/source/lib/mbuf/rte_mbuf_pool_ops.c#L88

#12 lore raw
In reply to Morten Brørup
Quoted 10 lines · On Tue, 7 Nov 2023 08:22:37 +0100 Morten Brørup <[email protected]>…
Quoted 7 lines
Quoted 1 line

Or are you saying that if they want to use dumpcap, they must choose

a thread safe pool type for their application (regardless if the application is single-threaded or not)?

There is no command line of EAL nature in dumpcap. This is intentional.
QED: overriding default pool type is not going to be a possible

The preferred mbuf pool type can configured in the primary process by EAL params. If so configured, it is stored in a memzone named "mbuf_user_pool_ops". And if it is set there, the secondary process will also use it as its preferred mbuf pool type.

I notice that no other app or example is using the create_by_ops except pdump/pcapng/dumpcap.

~/DPDK/main/examples $ git grep rte_pktmbuf_pool_create_by_ops

~/DPDK/main/app $ git grep rte_pktmbuf_pool_create_by_ops

dumpcap/main.c: mp = rte_pktmbuf_pool_create_by_ops(pool_name, num_mbufs,
pdump/main.c:                   mbuf_pool = rte_pktmbuf_pool_create_by_ops(mempool_name,
test/test_pcapng.c:     mp = rte_pktmbuf_pool_create_by_ops("pcapng_test_pool", IOV_MAX + NUM_PACKETS,
#13 lore raw
In reply to David Marchand
Quoted 4 lines · On Mon, 2 Oct 2023 09:33:50 +0200 David Marchand <[email protected]>…

Switching to rte_pktmbuf_pool_create() still leaves the user with the possibility to shoot himself in the foot (I was thinking of setting some --mbuf-pool-ops-name EAL option).

Quoted 8 lines

This application has explicit requirements in terms of concurrent access (and I don't think the mempool library exposes per driver capabilities in that regard).
The application was enforcing the use of mempool/ring so far.

I think it is safer to go with an explicit
rte_pktmbuf_pool_create_by_ops(... "ring_mp_mc").
WDYT?

The dumpcap command does not have EAL command line arguments that can be changed by user.

#14 lore raw

PATCH v2 1/1 dumpcap: fix mbuf pool ring type

In reply to Stephen Hemminger

The internal buffer pool used for copies of mbufs captured needs to be thread safe. If capturing on multiple interfaces or multiple queues, the same pool will be used (consumers). And if the capture ring gets full, the queues will need to put back the capture buffer which leads to multiple producers.

Since this is the same use case as normal drivers and the default pool time can not be overridden on dumpcap command line, it is OK to use the default pool type.

Bugzilla ID: 1271

Fixes cbb44143be74 ("app/dumpcap: add new packet capture application")

---
v2 - reword commit message

 app/dumpcap/main.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
app/dumpcap/main.c +3 -4
index 64294bbfb3e6..991174e95022 100644
@@ -691,10 +691,9 @@ static struct rte_mempool *create_mempool(void)
 			data_size = mbuf_size;
 	}
 
-	mp = rte_pktmbuf_pool_create_by_ops(pool_name, num_mbufs,
-					    MBUF_POOL_CACHE_SIZE, 0,
-					    data_size,
-					    rte_socket_id(), "ring_mp_sc");
+	mp = rte_pktmbuf_pool_create(pool_name, num_mbufs,
+				     MBUF_POOL_CACHE_SIZE, 0,
+				     data_size, rte_socket_id());
 	if (mp == NULL)
 		rte_exit(EXIT_FAILURE,
 			 "Mempool (%s) creation failed: %s\n", pool_name,
Signature · 1 line
2.39.2
#15 lore raw

PATCH v3 1/1 dumpcap: fix mbuf pool ring type

In reply to Stephen Hemminger

The internal buffer pool used for copies of mbufs captured needs to be thread safe. If capturing on multiple interfaces or multiple queues, the same pool will be used (consumers). And if the capture ring gets full, the queues will need to put back the capture buffer which leads to multiple producers.

Bugzilla ID: 1271

Fixes cbb44143be74 ("app/dumpcap: add new packet capture application")

---
v3 - just change ops, don't use default pool ops

 app/dumpcap/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
app/dumpcap/main.c +1 -1
index 64294bbfb3e6..4f581bd341d8 100644
@@ -694,7 +694,7 @@ static struct rte_mempool *create_mempool(void)
 	mp = rte_pktmbuf_pool_create_by_ops(pool_name, num_mbufs,
 					    MBUF_POOL_CACHE_SIZE, 0,
 					    data_size,
-					    rte_socket_id(), "ring_mp_sc");
+					    rte_socket_id(), "ring_mp_mc");
 	if (mp == NULL)
 		rte_exit(EXIT_FAILURE,
 			 "Mempool (%s) creation failed: %s\n", pool_name,
Signature · 1 line
2.39.2
Reviewed-by on v3 1/1
#16 lore raw
Quoted 9 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Wednesday, 8 November 2023 18.48

The internal buffer pool used for copies of mbufs captured needs to be thread safe. If capturing on multiple interfaces or multiple queues, the same pool will be used (consumers). And if the capture ring gets full, the queues will need to put back the capture buffer which leads to multiple producers.

Quoted 4 lines

Bugzilla ID: 1271

Fixes cbb44143be74 ("app/dumpcap: add new packet capture application")

---

#17 lore raw

09/11/2023 08:21, Morten Brørup:

Quoted 9 lines
Quoted 9 lines

From: Stephen Hemminger [mailto:[email protected]]
Sent: Wednesday, 8 November 2023 18.48

The internal buffer pool used for copies of mbufs captured needs to be thread safe. If capturing on multiple interfaces or multiple queues, the same pool will be used (consumers). And if the capture ring gets full, the queues will need to put back the capture buffer which leads to multiple producers.

Quoted 5 lines
Quoted 3 lines

Bugzilla ID: 1271

Fixes cbb44143be74 ("app/dumpcap: add new packet capture application")

Applied, thanks.