Skip to contentSkip to conversation

CMA alloc failed with 6 4-port X710 QAD4 NICs

Intel-wired-lan Ben Greear 3 participants 5 messages Aug 12, 2019 – Aug 14, 2019

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

Hello,

I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.

dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.

Any idea how much CMA memory is needed for this config?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#1 lore raw

We probably need a lot of continguous memory. Offhand, I don't know, but with >1500 queue pairs, we need a lot of memory in general.

I don't think we've had any more than 8 NICs in a system, and when I asked some people how many NICs we can use they gave me a more practical answer: 1 less than the config that makes the system unusable.

I don' t think there is a catch-all answer, so the questions we'd have are:
1) how much memory do you have
2) how many cards do you have in the system
3) how many option roms are in the cards in the system
4) you're not running a 32-bit OS, are you?

Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Ben Greear
Sent: Monday, August 12, 2019 2:15 PM
To: intel-wired-lan <[email protected]>
Subject: [Intel-wired-lan] CMA alloc failed with 6 4-port X710 QAD4 NICs

Hello,

I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.

dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.

Any idea how much CMA memory is needed for this config?

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#2 lore raw

Oh, and someone gave me the standard "check the bios settings" answer too, and enable memory over 8GB. Probably more of a desktop setting than a server setting, but I don't know what system you're running.

Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Fujinaka, Todd
Sent: Monday, August 12, 2019 2:38 PM
To: Ben Greear <[email protected]>; intel-wired-lan <[email protected]>
Subject: Re: [Intel-wired-lan] CMA alloc failed with 6 4-port X710 QAD4 NICs

We probably need a lot of continguous memory. Offhand, I don't know, but with >1500 queue pairs, we need a lot of memory in general.

I don't think we've had any more than 8 NICs in a system, and when I asked some people how many NICs we can use they gave me a more practical answer: 1 less than the config that makes the system unusable.

I don' t think there is a catch-all answer, so the questions we'd have are:
1) how much memory do you have
2) how many cards do you have in the system
3) how many option roms are in the cards in the system
4) you're not running a 32-bit OS, are you?

Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Ben Greear
Sent: Monday, August 12, 2019 2:15 PM
To: intel-wired-lan <[email protected]>
Subject: [Intel-wired-lan] CMA alloc failed with 6 4-port X710 QAD4 NICs

Hello,

I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.

dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.

Any idea how much CMA memory is needed for this config?

Thanks,
Ben

Signature · 11 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan at osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
#3 lore raw
In reply to Ben Greear
Quoted 11 lines · Ben Greear

Hello,

I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.

dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.

Any idea how much CMA memory is needed for this config?

Thanks,
Ben

You are going to need to go much bigger since you have 24 ports. Each port can consume several MB of contiguous space due to all the dma_alloc_coherent calls. It also depends on the number of queues you have enabled on the system as each queue pair will consume 24K of contiguous space as well.

My advise would be to look at increasing it to somewhere in the range of 64MB to 128MB.

Another alternative, assuming that the drivers are still working after making a fall-back allocation out of non-CMA space would be to just add __GFP_NOWARN to the flags passed to dma_alloc_coherent. That should mute the warnings on allocation failures and just fall back to the warning provided by the caller.

- Alex

author
#4 lore raw
Quoted 19 lines · Alexander Duyck
Quoted 11 lines · Ben Greear

Hello,

I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.

dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.

Any idea how much CMA memory is needed for this config?

Thanks,
Ben

You are going to need to go much bigger since you have 24 ports. Each port can consume several MB of contiguous space due to all the dma_alloc_coherent calls. It also depends on the number of queues you have enabled on the system as each queue pair will consume 24K of contiguous space as well.

Quoted 8 lines

My advise would be to look at increasing it to somewhere in the range of 64MB to 128MB.

Another alternative, assuming that the drivers are still working after making a fall-back allocation out of non-CMA space would be to just add __GFP_NOWARN to the flags passed to dma_alloc_coherent. That should mute the warnings on allocation failures and just fall back to the warning provided by the caller.

cma=512M made the warnings go away, but last I heard, the user could not get
link on any of the ports with or without that change. I'm not sure the root cause of that at this point, possibly it is cabling or some higher software stack issue.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com