CMA alloc failed with 6 4-port X710 QAD4 NICs
Hello,
I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.
dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.
Any idea how much CMA memory is needed for this config?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Fujinaka
We probably need a lot of continguous memory. Offhand, I don't know, but with >1500 queue pairs, we need a lot of memory in general.
I don't think we've had any more than 8 NICs in a system, and when I asked some people how many NICs we can use they gave me a more practical answer: 1 less than the config that makes the system unusable.
I don' t think there is a catch-all answer, so the questions we'd have are:
1) how much memory do you have
2) how many cards do you have in the system
3) how many option roms are in the cards in the system
4) you're not running a 32-bit OS, are you?
Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Ben Greear
Sent: Monday, August 12, 2019 2:15 PM
To: intel-wired-lan <[email protected]>
Subject: [Intel-wired-lan] CMA alloc failed with 6 4-port X710 QAD4 NICs
Hello,
I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.
dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.
Any idea how much CMA memory is needed for this config?
Thanks,
Ben
Signature · 7 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com _______________________________________________ Intel-wired-lan mailing list Intel-wired-lan at osuosl.org https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Fujinaka
Oh, and someone gave me the standard "check the bios settings" answer too, and enable memory over 8GB. Probably more of a desktop setting than a server setting, but I don't know what system you're running.
Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Fujinaka, Todd
Sent: Monday, August 12, 2019 2:38 PM
To: Ben Greear <[email protected]>; intel-wired-lan <[email protected]>
Subject: Re: [Intel-wired-lan] CMA alloc failed with 6 4-port X710 QAD4 NICs
We probably need a lot of continguous memory. Offhand, I don't know, but with >1500 queue pairs, we need a lot of memory in general.
I don't think we've had any more than 8 NICs in a system, and when I asked some people how many NICs we can use they gave me a more practical answer: 1 less than the config that makes the system unusable.
I don' t think there is a catch-all answer, so the questions we'd have are:
1) how much memory do you have
2) how many cards do you have in the system
3) how many option roms are in the cards in the system
4) you're not running a 32-bit OS, are you?
Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Ben Greear
Sent: Monday, August 12, 2019 2:15 PM
To: intel-wired-lan <[email protected]>
Subject: [Intel-wired-lan] CMA alloc failed with 6 4-port X710 QAD4 NICs
Hello,
I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.
dmesg is full of cma_alloc errors. I increased CMA space to 32MB, but still no luck.
Any idea how much CMA memory is needed for this config?
Thanks,
Ben
Signature · 11 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com _______________________________________________ Intel-wired-lan mailing list Intel-wired-lan at osuosl.org https://lists.osuosl.org/mailman/listinfo/intel-wired-lan _______________________________________________ Intel-wired-lan mailing list Intel-wired-lan at osuosl.org https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
Quoted 11 lines · Ben Greear
Hello,
I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.
dmesg is full of
cma_allocerrors. I increased CMA space to 32MB, but still no luck.Any idea how much CMA memory is needed for this config?
Thanks,
Ben
You are going to need to go much bigger since you have 24 ports. Each port can consume several MB of contiguous space due to all the dma_alloc_coherent calls. It also depends on the number of queues you have enabled on the system as each queue pair will consume 24K of contiguous space as well.
My advise would be to look at increasing it to somewhere in the range of 64MB to 128MB.
Another alternative, assuming that the drivers are still working after making a fall-back allocation out of non-CMA space would be to just add __GFP_NOWARN to the flags passed to dma_alloc_coherent. That should mute the warnings on allocation failures and just fall back to the warning provided by the caller.
- Alex
Quoted 19 lines · Alexander Duyck
Quoted 11 lines · Ben Greear
Hello,
I have a user trying to boot up our 4.20+ kernel with 6 4-port 10G NICs in it.
dmesg is full of
cma_allocerrors. I increased CMA space to 32MB, but still no luck.Any idea how much CMA memory is needed for this config?
Thanks,
BenYou are going to need to go much bigger since you have 24 ports. Each port can consume several MB of contiguous space due to all the
dma_alloc_coherentcalls. It also depends on the number of queues you have enabled on the system as each queue pair will consume 24K of contiguous space as well.
Quoted 8 lines
My advise would be to look at increasing it to somewhere in the range of 64MB to 128MB.
Another alternative, assuming that the drivers are still working after making a fall-back allocation out of non-CMA space would be to just add
__GFP_NOWARNto the flags passed todma_alloc_coherent. That should mute the warnings on allocation failures and just fall back to the warning provided by the caller.
cma=512M made the warnings go away, but last I heard, the user could not get
link on any of the ports with or without that change. I'm not sure the root cause of that at this point, possibly it is cabling or some higher software stack issue.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com