【For help】 configure crossbar build tool in CMakelist.txt

Hi, all

I am working for debugging userspace code of hns in rdma-core.git.

I have met a cmake question now. please give a help for me.

if I use crossbar build tool aarch64-linux-gnu-gcc for building the directory(provider/hns), what i should do it ?

My modification currently according to the others as fllows:

in the file : provider/hns/CMakelist.txt

set(CMAKE_C_COMPILER /opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/aarch64-linux-gnu-gcc)

but the modification is fail

-- The C compiler identification is GNU 4.8.4
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- cmake_share:
-- GNU:
-- BUILD_BIN_DIR:/home/ubuntu/rdma-core/build/bin
-- CMAKE_INSTALL: /usr/local/var/run
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- rdma_build: Debug;Release;RelWithDebInfo;MinSizeRel
-- BUILD_INCLUDE_DIR:/home/ubuntu/rdma-core/build/include
-- Performing Test SUPPORTS_GNU99
-- Performing Test SUPPORTS_GNU99 - Success
-- Performing Test HAVE_C_WARNINGS
-- Performing Test HAVE_C_WARNINGS - Success
-- Performing Test HAVE_C_WMISSING_PROTOTYPES
-- Performing Test HAVE_C_WMISSING_PROTOTYPES - Success
-- Performing Test HAVE_C_WMISSING_DECLARATIONS
-- Performing Test HAVE_C_WMISSING_DECLARATIONS - Success
-- Performing Test HAVE_C_WWRITE_STRINGS
-- Performing Test HAVE_C_WWRITE_STRINGS - Success
-- Performing Test HAVE_C_WFORMAT_2
-- Performing Test HAVE_C_WFORMAT_2 - Success
-- Performing Test HAVE_C_WORKING_SHADOW
-- Performing Test HAVE_C_WORKING_SHADOW - Success
-- Performing Test HAVE_C_WORKING_MISSING_FIELD_INITIALIZERS
-- Performing Test HAVE_C_WORKING_MISSING_FIELD_INITIALIZERS - Failed
-- Performing Test HAVE_C_WNO_MISSING_FIELD_INITIALIZERS
-- Performing Test HAVE_C_WNO_MISSING_FIELD_INITIALIZERS - Success
-- Performing Test HAVE_NO_STRICT_ALIASING
-- Performing Test HAVE_NO_STRICT_ALIASING - Success
-- Performing Test HAS_CLOEXEC
-- Performing Test HAS_CLOEXEC - Success
-- Performing Test HAVE_FUNC_ATTRIBUTE_ALWAYS_INLINE
-- Performing Test HAVE_FUNC_ATTRIBUTE_ALWAYS_INLINE - Success
-- Performing Test SUPPORTS_AS_NEEDED
-- Performing Test SUPPORTS_AS_NEEDED - Success
-- Performing Test SUPPORTS_NO_UNDEFINED
-- Performing Test SUPPORTS_NO_UNDEFINED - Success
-- Performing Test _LDSYMVER_SUCCESS
-- Performing Test _LDSYMVER_SUCCESS - Success
-- Found LDSymVer: GNU
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- checking for modules 'libnl-3.0;libnl-route-3.0'
-- found libnl-3.0, version 3.2.21
-- found libnl-route-3.0, version 3.2.21
-- Performing Test HAVE_WORKING_IF_H
-- Performing Test HAVE_WORKING_IF_H - Failed
-- Check size of long
-- Check size of long - done
-- Looking for rdma/rdma_user_rxe.h
-- Looking for rdma/rdma_user_rxe.h - not found
-- Looking for valgrind/memcheck.h
-- Looking for valgrind/memcheck.h - found
-- Looking for valgrind/drd.h
-- Looking for valgrind/drd.h - found
-- Performing Test LIBC_HAS_LIBRT
-- Performing Test LIBC_HAS_LIBRT - Success
-- Performing Test HAVE_C_WSTRICT_PROTOTYPES
-- Performing Test HAVE_C_WSTRICT_PROTOTYPES - Success
-- Performing Test HAVE_C_WOLD_STYLE_DEFINITION
-- Performing Test HAVE_C_WOLD_STYLE_DEFINITION - Success
-- Performing Test HAVE_C_WREDUNDANT_DECLS
-- Performing Test HAVE_C_WREDUNDANT_DECLS - Failed
-- c_complier: /opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/aarch64-linux-gnu-gcc -- Missing Optional Items:

--  netlink/route/link.h and net/if.h NOT co-includable (old headers)
--  rdma/rdma_user_rxe.h NOT found (old system kernel headers)
--  -Wmissing-field-initializers does NOT work
--  -Wredundant-decls does NOT work

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/rdma-core/build
[115/164] Building C object providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o
FAILED: /usr/bin/cc -Dhns_rdmav2_EXPORTS -std=gnu99 -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wshadow -Wno-missing-field-initializers -Wstrict-prototypes -Wold-style-definition -O2 -g -fPIC -Iinclude -I/usr/include/libnl3 -MMD -MT providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o -MF "providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o.d" -o providers/hns/CMakeFiles/hns-rdmav2.dir/hns_roce_u_hw_v1.c.o -c ../providers/hns/hns_roce_u_hw_v1.c ../providers/hns/hns_roce_u_hw_v1.c: Assembler messages:
../providers/hns/hns_roce_u_hw_v1.c:830: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:264: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:413: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:586: Error: number of operands mismatch for `ds'
../providers/hns/hns_roce_u_hw_v1.c:630: Error: number of operands mismatch for `ds'
[115/164] Building C object ibacm/CMakeFiles/ibacm.dir/src/acm.c.o
ninja: build stopped: subcommand failed.

.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

You cannot cross compile only a part of the project, you must cross compile everything.

$ mkdir build; cd build
$ CC=aarch64-linux-gnu-gcc cmake -GNinja
$ ninja

I haven't extensively tested cross compiling, you may run into problems, in particular I know pkgconfig for libnl doesn't work reliably when cross compiling.

You may need to build with -DENABLE_RESOLVE_NEIGH=0 if you don't have a cross compiled libnl3 available.

This is not the right approach, the hns provider CMakelist should detect the compiler is not ARM64 and just do nothing.

Perhaps something like this:

CHECK_C_SOURCE_COMPILES("
#ifndef __ARM64__
#error Failed
#endif
int main(int argc,const char *argv[]) { return 1; }" HAVE_ARCH_ARM64)

if (HAVE_ARCH_ARM64)
[..]
endif()

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi, Jason

Indeed, you are right! when use the cmd: CC= aarch64-linux-gnu-gcc cmake -GNinja ..

the libnl doesn't work

the print log as follows:

ubuntu@62fe1357a077:~/rdma-core/build$ ninja
[27/164] Linking C executable bin/iwpmd
FAILED: : && /opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/aarch64-linux-gnu-gcc -std=gnu99 -Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -Wmissing-prototypes -Wmissing-declarations -Wwrite-strings -Wformat=2 -Wshadow -Wno-missing-field-initializers -Wstrict-prototypes -Wold-style-definition -O2 -g -Wl,--as-needed -Wl,--no-undefined iwpmd/CMakeFiles/iwpmd.dir/iwarp_pm_common.c.o iwpmd/CMakeFiles/iwpmd.dir/iwarp_pm_helper.c.o iwpmd/CMakeFiles/iwpmd.dir/iwarp_pm_server.c.o -o bin/iwpmd ccan/libccan.a -lnl-route-3 -lnl-3 -lpthread && :
/opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/../lib/gcc/aarch64-linux-gnu/4.9.2/../../../../aarch64-linux-gnu/bin/ld: cannot find -lnl-route-3
/opt/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux/bin/../lib/gcc/aarch64-linux-gnu/4.9.2/../../../../aarch64-linux-gnu/bin/ld: cannot find -lnl-3
collect2: error: ld returned 1 exit status
[27/164] Building C object libibverbs/CMakeFiles/ibverbs.dir/cmd.c.o
ninja: build stopped: subcommand failed.

when use the cmd: CC=aarch64-linux-gnu-gcc cmake -GNinja -DENABLE_RESOLVE_NEIGH=0 ..

the provider/hns/CMakelist.txt can be called correctly and generate the library libhns-rdmav2.so(not be tested), but it will generate warning with mlx5

the print log as follows:

Building C object providers/mlx5/CMakeFiles/mlx5-rdmav2.dir/cq.c.o ../providers/mlx5/cq.c:414:13: warning: function declaration isn't a prototype [-Wstrict-prototypes] static void mlx5_stall_poll_cq()
^
../providers/mlx5/cq.c: In function 'mlx5_stall_poll_cq':
../providers/mlx5/cq.c:414:13: warning: old-style function definition [-Wold-style-definition]

In summary, I have two questions:

1. if don't use libnl(use -DENABLE_RESOLVE_NEIGH=0), it will not confluent the function of library?

2. the rdma-core.git use the default build method by used build.sh, and i need modify it by added the building item or modify the README.md for hns, is the right approach?

Thanks

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks, you may be the first person to compile on ARM64, so I'm glad to hear that is your only warning, I will get it fixed.

Correct, RoCEE will not work entirely as expected if you try and run the libibverbs from such a build.

No.

If you native compile on an actual ARM64 machine you will not have problems, simply follow the README.md directions to install the required packages on the actual machine.

Your problem is cross compiling. Cross compiling is hard. You need to make libnl3 available in your cross compiler's 'sysroot'. I can't really give you generic good directions for that..

One option is to cross compile libnl and 'cross-install' it into the compiler tree.

Another option is to download the libnl3 ARM64 packages from Debian and then unpack and rename directories 'just so' to make it work.

But if you don't intend to run the build binaries (eg when you go to test, you will build natively on ARM64), then don't worry about it.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi, Jason
I am working for test the userspace code of hns in rdma-core.git with your guide. I have met some quesiton for debugging in my environment.
after add some print lines in load_driver() in libibverbs/init.c, as follows:

   printf("[%s, %d] %s, %d\n", __func__, __LINE__, VERBS_PROVIDER_DIR, sizeof(VERBS_PROVIDER_DIR));
        /* If configured with a provider plugin path then try that next */
        if (sizeof(VERBS_PROVIDER_DIR) >= 1) {
                if (asprintf(&so_name, VERBS_PROVIDER_DIR "/lib%s" DLOPEN_TRAILER, name) <
                    0)
                        goto out_asprintf;
                printf("[%s, %d] so_name: %s\n", __func__, __LINE__, so_name);// @1
                printf("[%s, %d] so_name: %s\n", __func__, __LINE__, DLOPEN_TRAILER);//
                dlhandle = dlopen(so_name, RTLD_NOW);
                free(so_name);
                if (dlhandle)
                        return;
        }
        /* Otherwise use the system libary search path. This is the historical
           behavior of libibverbs */
        if (asprintf(&so_name, "lib%s" DLOPEN_TRAILER, name) < 0)
                goto out_asprintf;
        printf("[%s, %d] so_name: %s\n", __func__, __LINE__, so_name);//@2
        printf("[%s, %d] so_name: %s\n", __func__, __LINE__, DLOPEN_TRAILER);//

using the cmd:
CC=aarch64-linux-gnu-gcc cmake -GNinja -DENABLE_RESOLVE_NEIGH=0 -DVERBS_PROVIDER_DIR='' .. ninja

after obtained the libhisi-rdmav2.so and hisi.driver, i put them into the /lib and /etc/libibverbs.d/
in my test environment.

Now, it is test fail. some question are not understanded.
1. when set the VERBS_PROVIDER_DIR to empty, the sizeof(VERBS_PROVIDER_DIR) should be 0 and the branch should not be run
2. the value of so_name should be /libhisi-rdmav2.so in @1 and libhisi-rdmav2.so in @2. in fact, the value of so_name is /libhisi and libhisi

the test print log as follows:

-rdmav2.soer, 218] so_name: /libhisi
[load_driver, 219] so_name: -rdmav2.so
-rdmav2.soer, 229] so_name: libhisi
[load_driver, 230] so_name: -rdmav2.so

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

This is a mistake, sizeof('') is 1, so the if should be (> 1)' I will fix it.

If you notice the -rdmav2.so has been placed at the start of the line, this suggest you have a spurious '\r' character. This would come from the .driver file.

Since you did not use the cmake install process you must have written the .driver file yourself and used a DOS text editor. UNIX line endings are required.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks, i see it.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html