Question on huge pages and running DPDK apps inside containers
Hi,
I am encountering a problem running DPDK apps inside containers, and need your help:
I create containers by doing:
lxc-create -t ubuntu -n fpc0
lxc-start -n fpc0
(Reserve hugepages and follow all other steps to run the DPDK app on host)
Then run a standard dpdk app, on host, it works fine, inside container, it runs into this problem:
ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$ ./testpmd -c f -n 4 -- -i rte_eal_init : 857 opt : 99 opt : 110 EAL: No free hugepages reported in hugepages-2048kB PANIC in rte_eal_init(): Cannot get hugepage information 6: [./testpmd() [0x402941]] 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f14f50ec5]] 4: [./testpmd() [0x40237e]] 3: [./testpmd() [0x478edb]] 2: [./testpmd() [0x4022ad]] 1: [./testpmd() [0x47e3c3]] Aborted ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$
However, running this same app on the host, or doing lxc-execute works.
While trying to debug my own dpdk app, I see that reading the same file for no of huge pages (
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
), on the host, its value is read 2048, while inside the container it is 0. This is same file.
Also, if you cat the contents of this file, it is actually 0! (nr_hugepages is 2048, but the DPDK code does not read this file.)
cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages 0
In the below logs, the 2 prints "EAL: Using .." and "EAL: path ..." are added by me in DPDL eal library.
On host:rte_eal_init : 859
opt : 99
opt : 110
EAL: Using free_hugepages
EAL: path /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 2048
Inside container:
pt : 99
opt : 110
EAL: Using free_hugepages
EAL: path /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 0
Has anyone run into this problem and know what's the issue?
Anjali
A message that is not in the archive (sent off-list or not yet received): CFDAB971.2ABC8%[email protected]
Do you know if the host's hugepages are mapped into the container?
Seeing as containers are meant to provide isolation, it seems to make sense that the host would not automatically share hugepages with a container, but I'm not sure.
Jeff
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Anjali Kulkarni
Sent: Thursday, July 03, 2014 7:41 AM
To: [email protected]
Subject: [dpdk-dev] Question on huge pages and running DPDK apps inside containers
Hi,
I am encountering a problem running DPDK apps inside containers, and need your help:
I create containers by doing:
lxc-create -t ubuntu -n fpc0
lxc-start -n fpc0
(Reserve hugepages and follow all other steps to run the DPDK app on host)
Then run a standard dpdk app, on host, it works fine, inside container, it runs into this problem:
ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$ ./testpmd -c f -n 4 -- -i rte_eal_init : 857 opt : 99 opt : 110 EAL: No free hugepages reported in hugepages-2048kB PANIC in rte_eal_init(): Cannot get hugepage information 6: [./testpmd() [0x402941]] 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f14f50ec5]] 4: [./testpmd() [0x40237e]] 3: [./testpmd() [0x478edb]] 2: [./testpmd() [0x4022ad]] 1: [./testpmd() [0x47e3c3]] Aborted ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$
However, running this same app on the host, or doing lxc-execute works.
While trying to debug my own dpdk app, I see that reading the same file for no of huge pages (
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
), on the host, its value is read 2048, while inside the container it is 0. This is same file.
Also, if you cat the contents of this file, it is actually 0! (nr_hugepages is 2048, but the DPDK code does not read this file.)
cat /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages 0
In the below logs, the 2 prints "EAL: Using .." and "EAL: path ..." are added by me in DPDL eal library.
On host:rte_eal_init : 859
opt : 99
opt : 110
EAL: Using free_hugepages
EAL: path /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 2048
Inside container:
pt : 99
opt : 110
EAL: Using free_hugepages
EAL: path /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 0
Has anyone run into this problem and know what's the issue?
Anjali
A message that is not in the archive (sent off-list or not yet received): 4032A54B6BB5F04B8C08B6CFF08C59285544C729-AtyAts71sc9Qxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org
Thanks - I got a confirmation from some Intel folks earlier that Intel DPDK apps do work inside containers. Also, the weird thing is, testpmd application runs in the container using lxc-execute. Lxc-execute runs the app inside container. It works, see below:
root@anjali:/dpdk/x86_64-default-linuxapp-gcc/app# lxc-execute -n foo -- ./testpmd -c f -n 4 -- -i lxc: call to cgmanager_move_pid_abs_sync(cpuset) failed: Escape request from different namespace requires a proxy rte_eal_init : 859 opt : 99 opt : 110 EAL: Using free_hugepages EAL: path /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages EAL: No of hugepages is 2048 rte_eal_init : 884 EAL: No of hugepages is 0 EAL: No of hugepages is 1 EAL: No of hugepages is 2 EAL: No of hugepages is 3 EAL: Setting up hugepage memory... EAL: Ask a virtual area of 0x8388608 bytes EAL: Virtual area found at 0x7f7c39c00000 (size = 0x800000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7c39800000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7c39400000 (size = 0x200000) EAL: Ask a virtual area of 0x293601280 bytes EAL: Virtual area found at 0x7f7c27a00000 (size = 0x11800000) EAL: Ask a virtual area of 0x1841299456 bytes EAL: Virtual area found at 0x7f7bb9c00000 (size = 0x6dc00000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7bb9800000 (size = 0x200000) EAL: Ask a virtual area of 0x2143289344 bytes EAL: Virtual area found at 0x7f7b39a00000 (size = 0x7fc00000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7b39600000 (size = 0x200000) EAL: Requesting 1024 pages of size 2MB from socket 0 EAL: Requesting 1024 pages of size 2MB from socket 1 EAL: TSC frequency is ~2593752 KHz EAL: No of hugepages is 32902 EAL: No of hugepages is 3584 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3586 EAL: No of hugepages is 32902 EAL: No of hugepages is 3586 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3587 EAL: No of hugepages is 32902 EAL: No of hugepages is 3587 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3588 EAL: No of hugepages is 32902 EAL: No of hugepages is 3588 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3592 EAL: No of hugepages is 32902 EAL: No of hugepages is 3592 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3594 EAL: No of hugepages is 32902 EAL: No of hugepages is 3594 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3624 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3626 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3628 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7486 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7482 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7483 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7469 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7440 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7462 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 9294 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7489 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7426 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7458 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7460 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 5409 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 5409 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 4347 EAL: No of hugepages is 5421 EAL: No of hugepages is 35198 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 4347 EAL: No of hugepages is 5421 EAL: No of hugepages is 35198 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 7531 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 6659 EAL: No of hugepages is 4432 EAL: No of hugepages is 6659 EAL: No of hugepages is 4432 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3592 EAL: No of hugepages is 32902 EAL: No of hugepages is 3592 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3624 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3626 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3628 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 4347 EAL: No of hugepages is 32902 EAL: No of hugepages is 3 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 4347 EAL: No of hugepages is 32902 EAL: No of hugepages is 3 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3712 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3728 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3760 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3761 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3762 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3763 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3764 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3765 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3766 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3767 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3613 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3636 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3713 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3638 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3784 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3785 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3786 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3776 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3777 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3778 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3779 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3614 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3615 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3808 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3810 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3812 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3814 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3809 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3811 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3813 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3815 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3744 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3632 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3752 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3697 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3754 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3755 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3756 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 32902 EAL: No of hugepages is 3757 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 1 EAL: No of hugepages is 6659 EAL: No of hugepages is 8192 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3712 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3728 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3760 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3761 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3762 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3763 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3764 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3765 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3766 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3767 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3613 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3636 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3713 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3638 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3784 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3785 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3786 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3776 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3777 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3778 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3779 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3614 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3615 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3808 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3810 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3812 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3814 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3809 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3811 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3813 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3815 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3744 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3632 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3752 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3697 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3754 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3755 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3756 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 EAL: No of hugepages is 32902 EAL: No of hugepages is 3757 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0
...
When I logged inside the container and ran the app, I saw this issue.
Anjali
Quoted 82 lines · Jeffrey B
Do you know if the host's hugepages are mapped into the container? Seeing as containers are meant to provide isolation, it seems to make sense that the host would not automatically share hugepages with a container, but I'm not sure.
Jeff
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Anjali Kulkarni
Sent: Thursday, July 03, 2014 7:41 AM
To: [email protected]
Subject: [dpdk-dev] Question on huge pages and running DPDK apps inside containersHi,
I am encountering a problem running DPDK apps inside containers, and need your help:
I create containers by doing:
lxc-create -t ubuntu -n fpc0
lxc-start -n fpc0(Reserve hugepages and follow all other steps to run the DPDK app on host)
Then run a standard dpdk app, on host, it works fine, inside container, it runs into this problem:
ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$ ./testpmd -c f -n 4 -- -i rte_eal_init : 857 opt : 99 opt : 110 EAL: No free hugepages reported in hugepages-2048kB PANIC in rte_eal_init(): Cannot get hugepage information 6: [./testpmd() [0x402941]] 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f14f50ec5]] 4: [./testpmd() [0x40237e]] 3: [./testpmd() [0x478edb]] 2: [./testpmd() [0x4022ad]] 1: [./testpmd() [0x47e3c3]] Aborted ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$However, running this same app on the host, or doing lxc-execute works.
While trying to debug my own dpdk app, I see that reading the same file for no of huge pages (
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages), on the host, its value is read 2048, while inside the container it is
0. This is same file.
Also, if you cat the contents of this file, it is actually 0! (nr_hugepagesis 2048, but the DPDK code does not read this file.)cat
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages0In the below logs, the 2 prints "EAL: Using .." and "EAL: path ..." are added by me in DPDL eal library.
On host:
rte_eal_init: 859
opt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 2048Inside container:
pt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 0Has anyone run into this problem and know what's the issue?
Anjali
Ignore the previous email, it was too long. Pls see this one only:
Thanks - I got a confirmation from some Intel folks earlier that Intel DPDK apps do work inside containers. Also, the weird thing is, testpmd application runs in the container using lxc-execute. Lxc-execute runs the app inside container. It works, see below:
root@anjali:/dpdk/x86_64-default-linuxapp-gcc/app# lxc-execute -n foo -- ./testpmd -c f -n 4 -- -i lxc: call to cgmanager_move_pid_abs_sync(cpuset) failed: Escape request from different namespace requires a proxy rte_eal_init : 859 opt : 99 opt : 110 EAL: Using free_hugepages EAL: path /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages EAL: No of hugepages is 2048 rte_eal_init : 884 EAL: No of hugepages is 0 EAL: No of hugepages is 1 EAL: No of hugepages is 2 EAL: No of hugepages is 3 EAL: Setting up hugepage memory... EAL: Ask a virtual area of 0x8388608 bytes EAL: Virtual area found at 0x7f7c39c00000 (size = 0x800000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7c39800000 (size = 0x200000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7c39400000 (size = 0x200000) EAL: Ask a virtual area of 0x293601280 bytes EAL: Virtual area found at 0x7f7c27a00000 (size = 0x11800000) EAL: Ask a virtual area of 0x1841299456 bytes EAL: Virtual area found at 0x7f7bb9c00000 (size = 0x6dc00000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7bb9800000 (size = 0x200000) EAL: Ask a virtual area of 0x2143289344 bytes EAL: Virtual area found at 0x7f7b39a00000 (size = 0x7fc00000) EAL: Ask a virtual area of 0x2097152 bytes EAL: Virtual area found at 0x7f7b39600000 (size = 0x200000) EAL: Requesting 1024 pages of size 2MB from socket 0 EAL: Requesting 1024 pages of size 2MB from socket 1 EAL: TSC frequency is ~2593752 KHz EAL: No of hugepages is 32902 EAL: No of hugepages is 3584 EAL: No of hugepages is 5421 EAL: No of hugepages is 35207 EAL: No of hugepages is 0 Š
Š
When I logged inside the container and ran the app, I saw this issue.
Anjali
Quoted 82 lines · Jeffrey B
Do you know if the host's hugepages are mapped into the container? Seeing as containers are meant to provide isolation, it seems to make sense that the host would not automatically share hugepages with a container, but I'm not sure.
Jeff
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Anjali Kulkarni
Sent: Thursday, July 03, 2014 7:41 AM
To: [email protected]
Subject: [dpdk-dev] Question on huge pages and running DPDK apps inside containersHi,
I am encountering a problem running DPDK apps inside containers, and need your help:
I create containers by doing:
lxc-create -t ubuntu -n fpc0
lxc-start -n fpc0(Reserve hugepages and follow all other steps to run the DPDK app on host)
Then run a standard dpdk app, on host, it works fine, inside container, it runs into this problem:
ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$ ./testpmd -c f -n 4 -- -i rte_eal_init : 857 opt : 99 opt : 110 EAL: No free hugepages reported in hugepages-2048kB PANIC in rte_eal_init(): Cannot get hugepage information 6: [./testpmd() [0x402941]] 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f14f50ec5]] 4: [./testpmd() [0x40237e]] 3: [./testpmd() [0x478edb]] 2: [./testpmd() [0x4022ad]] 1: [./testpmd() [0x47e3c3]] Aborted ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$However, running this same app on the host, or doing lxc-execute works.
While trying to debug my own dpdk app, I see that reading the same file for no of huge pages (
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages), on the host, its value is read 2048, while inside the container it is
0. This is same file.
Also, if you cat the contents of this file, it is actually 0! (nr_hugepagesis 2048, but the DPDK code does not read this file.)cat
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages0In the below logs, the 2 prints "EAL: Using .." and "EAL: path ..." are added by me in DPDL eal library.
On host:
rte_eal_init: 859
opt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 2048Inside container:
pt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 0Has anyone run into this problem and know what's the issue?
Anjali
Can you try deleting the hugepage files on the host before trying to run the app in a container?
Quoted 76 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Anjali Kulkarni
Sent: Thursday, July 03, 2014 7:41 AM
To: [email protected]
Subject: [dpdk-dev] Question on huge pages and running DPDK apps inside containersHi,
I am encountering a problem running DPDK apps inside containers, and need your help:
I create containers by doing:
lxc-create -t ubuntu -n fpc0
lxc-start -n fpc0(Reserve hugepages and follow all other steps to run the DPDK app on host)
Then run a standard dpdk app, on host, it works fine, inside container, it runs into this problem:
ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$ ./testpmd -c f -n 4 -- -i rte_eal_init : 857 opt : 99 opt : 110 EAL: No free hugepages reported in hugepages-2048kB PANIC in rte_eal_init(): Cannot get hugepage information 6: [./testpmd() [0x402941]] 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f14f50ec5]] 4: [./testpmd() [0x40237e]] 3: [./testpmd() [0x478edb]] 2: [./testpmd() [0x4022ad]] 1: [./testpmd() [0x47e3c3]] Aborted ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$However, running this same app on the host, or doing lxc-execute works.
While trying to debug my own dpdk app, I see that reading the same file for no of huge pages (
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages), on the host, its value is read 2048, while inside the container it is 0. This is
same file.
Also, if you cat the contents of this file, it is actually 0! (nr_hugepagesis 2048, but the DPDK code does not read this file.)cat
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages0In the below logs, the 2 prints "EAL: Using .." and "EAL: path ..." are added by me in DPDL eal library.
On host:
rte_eal_init: 859
opt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 2048Inside container:
pt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 0Has anyone run into this problem and know what's the issue?
Anjali
A message that is not in the archive (sent off-list or not yet received): 59AF69C657FD0841A61C55336867B5B02CF13B78-kPTMFJFq+rELt2AQoY/[email protected]
Richard, thanks.
Looking at lxc-execute, I see this, which seems to be the key in running the applications created via lxc-create and those run lxc-execute:
Note
"lxc-execute is intended to run application containers that share the host's
root file system, and not to run system containers that you create
using
lxc-create. Use lxc-start to run system containers."
Anjali
Quoted 88 lines · Bruce
Can you try deleting the hugepage files on the host before trying to run the app in a container?
Quoted 85 lines
-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Anjali Kulkarni
Sent: Thursday, July 03, 2014 7:41 AM
To: [email protected]
Subject: [dpdk-dev] Question on huge pages and running DPDK apps inside containersHi,
I am encountering a problem running DPDK apps inside containers, and need
your help:I create containers by doing:
lxc-create -t ubuntu -n fpc0
lxc-start -n fpc0(Reserve hugepages and follow all other steps to run the DPDK app on host)
Then run a standard dpdk app, on host, it works fine, inside container, it runs
into this problem:ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$ ./testpmd -c f -n 4 -- -i rte_eal_init : 857 opt : 99 opt : 110 EAL: No free hugepages reported in hugepages-2048kB PANIC in rte_eal_init(): Cannot get hugepage information 6: [./testpmd() [0x402941]] 5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7f14f50ec5]] 4: [./testpmd() [0x40237e]] 3: [./testpmd() [0x478edb]] 2: [./testpmd() [0x4022ad]] 1: [./testpmd() [0x47e3c3]] Aborted ubuntu@fpc0:/dpdk/x86_64-default-linuxapp-gcc/app$However, running this same app on the host, or doing lxc-execute works.
While trying to debug my own dpdk app, I see that reading the same file for no
of huge pages (
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages), on the host, its value is read 2048, while inside the container it
is 0. This is
same file.
Also, if you cat the contents of this file, it is actually 0! (nr_hugepagesis 2048,
but the DPDK code does not read this file.)cat
/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages0In the below logs, the 2 prints "EAL: Using .." and "EAL: path ..." are added by me
in DPDL eal library.On host:
rte_eal_init: 859
opt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 2048Inside container:
pt : 99
opt : 110
EAL: Usingfree_hugepages
EAL: path/sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages
EAL: No of hugepages is 0Has anyone run into this problem and know what's the issue?
Anjali