How to fight forwarding performance regression on large mempool sizes.

Good day everyone,

While working on IP packet defragmenting I had to enlarge mempool size. I did this to provide large enough time window for assembling a fragment sequence. Unfortunately, I got a performance regression: if I enlarge mempool size from 2**12 to 2**20 MBufs, packet performance for not fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I made only a single measure, so the data are noisy, but the trend is evident:
SIZE 4096 - 8.47mpps
SIZE 8192 - 8.26mpps
SIZE 16384 - 8.29mpps
SIZE 32768 - 8.31mpps
SIZE 65536 - 8.12mpps
SIZE 131072 - 7.93mpps
SIZE 262144 - 6.22mpps
SIZE 524288 - 5.72mpps
SIZE 1048576 - 5.63mpps

And I need even larger sizes.

I want to ask for an advice, how best to tackle this? One way I'm thinking about is to make two mempools, one large for fragments (we may accumulate a big number of them) and one small for full packets, we just forward them burst by burst. Is it possible to configure RSS to distribute packets between queues according to this scheme? Perhaps, there are better ways?

Thanks,
Dmitry

Good day everyone,

While working on IP packet defragmenting I had to enlarge mempool size. I did this to provide large enough time window for assembling a fragment sequence. Unfortunately, I got a performance regression: if I enlarge mempool size from 2**12 to 2**20 MBufs, packet performance for not fragmented packets drops from ~8.5mpps to ~5.5mpps for single core. I made only a single measure, so the data are noisy, but the trend is evident:
SIZE 4096 - 8.47mpps
SIZE 8192 - 8.26mpps
SIZE 16384 - 8.29mpps
SIZE 32768 - 8.31mpps
SIZE 65536 - 8.12mpps
SIZE 131072 - 7.93mpps
SIZE 262144 - 6.22mpps
SIZE 524288 - 5.72mpps
SIZE 1048576 - 5.63mpps

And I need even larger sizes.

I want to ask for an advice, how best to tackle this? One way I'm
thinking about is to make two mempools, one large for fragments (we
may accumulate a big number of them) and one small for full packets,
we just forward them burst by burst. Is it possible to configure RSS
to distribute packets between queues according to this scheme? Perhaps, there are better ways?

Thanks,
Dmitry

Hi Dmitry,

The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB huge pages already (rather than 2MB)?

I would think that it would not use over 1GB until you ask for 512K mbufs or more.

Dmitry,
One other question - what version of DPDK are you doing on? -Venky

Quoted 9 lines · Robert Sanford

Hi Dmitry,

The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB huge pages already (rather than 2MB)?

I would think that it would not use over 1GB until you ask for 512K mbufs or more.

Hi Robert,

Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs are 2240 bytes of size. So something strange indeed happens then we move from ~200MB to ~400MB. Any ideas?

Regards,
Dmitry

Quoted 9 lines · Robert Sanford

Hi Dmitry,

The biggest drop-off seems to be from size 128K to 256K. Are you using 1GB huge pages already (rather than 2MB)?

I would think that it would not use over 1GB until you ask for 512K mbufs or more.

Hi Robert,

Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs are 2240 bytes of size. So something strange indeed happens then we move from ~200MB to ~400MB. Any ideas?

Regards,
Dmitry

One more point, if you're not doing this already: Allocate 2^N-1 mbufs, not 2^N. According to the code and comments: "The optimum size (in terms of memory usage) for a mempool is when n is a power of two minus one: n = (2^q - 1)."