rdma-core, cmake/ninja question
Hey Jason, newb question:
How do I force the rdma-core to build with /usr as the install prefix, /etc as the config dir, and /usr/lib64 as the lib dir? It seems to default to /usr/local..
Thanks!
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Steve Wise
Hey Jason, newb question:
How do I force the rdma-core to build with /usr as the install prefix, /etc as the config dir, and
/usr/lib64as the lib dir? It seems to default to/usr/local..
Yes, that is the GNU standard - it is what configure did.
The equivilant to configure --prefix is:
-DCMAKE_INSTALL_PREFIX=/usr
You often also need.
-DCMAKE_INSTALL_SYSCONFDIR=/etc
There are many paths that are configurable, each distro sets them as appropriate via the packaging scripts. For instance this is what RPM tells cmake to do on centos6:
/usr/bin/cmake -DCMAKE_VERBOSE_MAKEFILE=ON
-DCMAKE_INSTALL_PREFIX:PATH=/usr
-DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64
-DINCLUDE_INSTALL_DIR:PATH=/usr/include
-DLIB_INSTALL_DIR:PATH=/usr/lib64
-DSYSCONF_INSTALL_DIR:PATH=/etc
-DSHARE_INSTALL_PREFIX:PATH=/usr/share
-DLIB_SUFFIX=64
-DCMAKE_SKIP_RPATH:BOOL=ON
-DBUILD_SHARED_LIBS:BOOL=ON
-DCMAKE_BUILD_TYPE=Release
-DCMAKE_INSTALL_BINDIR:PATH=/usr/bin
-DCMAKE_INSTALL_SBINDIR:PATH=/usr/sbin
-DCMAKE_INSTALL_LIBDIR:PATH=/usr/lib64
-DCMAKE_INSTALL_LIBEXECDIR:PATH=/usr/libexec
-DCMAKE_INSTALL_LOCALSTATEDIR:PATH=/var
-DCMAKE_INSTALL_SHAREDSTATEDIR:PATH=/var/lib
-DCMAKE_INSTALL_INCLUDEDIR:PATH=/usr/include
-DCMAKE_INSTALL_INFODIR:PATH=/usr/share/info
-DCMAKE_INSTALL_MANDIR:PATH=/usr/share/man
-DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc
-DCMAKE_INSTALL_SYSTEMD_SERVICEDIR:PATH=/tmp/
-DCMAKE_INSTALL_INITDDIR:PATH=/etc/rc.d/init.d
Each distro is a little different. This is all the same stuff as configure, just different names..
I recommend *against* running 'make install' to /usr. This is a big package now, it is hard to unwind it once it installs over your system. I really should finish the run in place patch :|
If you do this:
$ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfig
Then you can install to /usr/local/ and things will work as you expect. Note, this depends on a verbs patch that is still only in my preview tree.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20160922154814.GB15212-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 5 lines
I recommend *against* running 'make install' to /usr. This is a big package now, it is hard to unwind it once it installs over your system. I really should finish the run in place patch :|
Quoted 7 lines
If you do this:
$ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfigThen you can install to
/usr/local/and things will work as you expect. Note, this depends on a verbs patch that is still only in my preview tree.
So I have a dev system setup with everything installed from the distro in /usr. I have my rdma-core git tree. I change libcxgb4 to fix a bug. I want to test it. I install it to /usr/local and add /etc/ld.so.conf.d/usrlocal.conf. But that still doesn't work because ldconfig still finds libcxgb4 in /usr/lib64 first...
How can I force ldconfig to look in /usr/local/lib64 first?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 13 lines · Steve Wise
Quoted 5 lines
I recommend *against* running 'make install' to /usr. This is a big package now, it is hard to unwind it once it installs over your system. I really should finish the run in place patch :|
Quoted 7 lines
If you do this:
$ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfigThen you can install to
/usr/local/and things will work as you expect. Note, this depends on a verbs patch that is still only in my preview tree.
Quoted 5 lines
So I have a dev system setup with everything installed from the distro in /usr. I have my rdma-core git tree. I change libcxgb4 to fix a bug. I want to test it. I install it to
/usr/localand add/etc/ld.so.conf.d/usrlocal.conf. But that still doesn't work because ldconfig still finds libcxgb4 in/usr/lib64first...
Apply this patch:
https://github.com/jgunthorpe/rdma-plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8f
It hardwires the search path into libverbs and libverbs will look into the place it was installed before searching the system library path. That will eliminate the problem you mentioned above.
Further, after that patch is applied you can do a rough imitation of 'run-in-place' like this:
$ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -DSYSCONF_INSTALL_DIR:PATH=/etc
(untested) That hardwires the build directory into libibverbs, so it will try to load all drivers from there first.
Then use
$ export LD_LIBRARY_PATH=`pwd`/lib
And all verbs using programs started from that shell will use the full new library set and the new providers without having to do make install at all.
I plan to automate this basic approach for 'run in place', please let me know if it works for you and makes sense.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20160922163143.GC6994-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 1 line
Quoted 1 line
So I have a dev system setup with everything installed from the distro in
/usr.
Quoted 1 line
Quoted 1 line
I have my rdma-core git tree. I change libcxgb4 to fix a bug. I want to
test
Quoted 1 line
Quoted 1 line
it. I install it to
/usr/localand add/etc/ld.so.conf.d/usrlocal.conf.
But
Quoted 12 lines
Quoted 2 lines
that still doesn't work because ldconfig still finds libcxgb4 in
/usr/lib64first...Apply this patch:
https://github.com/jgunthorpe/rdma-
plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8fIt hardwires the search path into libverbs and libverbs will look into the place it was installed before searching the system library path. That will eliminate the problem you mentioned above.
Quoted 6 lines
Further, after that patch is applied you can do a rough imitation of 'run-in-place' like this:
$ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib - DSYSCONF_INSTALL_DIR:PATH=/etc
And how do I build this after the cmake?
Quoted 11 lines
(untested) That hardwires the build directory into libibverbs, so it will try to load all drivers from there first.
Then use
$ export LD_LIBRARY_PATH=`pwd`/libAnd all verbs using programs started from that shell will use the full new library set and the new providers without having to do make install at all.
Quoted 8 lines
I plan to automate this basic approach for 'run in place', please let me know if it works for you and makes sense.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Steve Wise
Quoted 9 lines
Apply this patch:
https://github.com/jgunthorpe/rdma-
plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8fIt hardwires the search path into libverbs and libverbs will look into the place it was installed before searching the system library path. That will eliminate the problem you mentioned above.
Quoted 8 lines
Quoted 6 lines
Further, after that patch is applied you can do a rough imitation of 'run-in-place' like this:
$ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib - DSYSCONF_INSTALL_DIR:PATH=/etcAnd how do I build this after the cmake?
Sorry, you always need the -GNinja, I was just being brief
I also mistyped the ETC name.
I tested this:
$ mkdir build ; cd build
$ cmake .. -GNinja -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib -DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc
$ ninja
$ strace bin/ibv_devinfo 2>&1 | grep -i rdmav2
open("/tmp/rdma-core/build/lib/libmlx4-rdmav2.so", O_RDONLY|O_CLOEXEC) = 3Note LD_LIBRARY_PATH is needed to run things from outside build/bin/
I have some thoughts to make this simpler, call it a proof of concept for run-from-build ..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20160922192932.GA25659-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 33 lines · Steve Wise
Quoted 10 lines
Quoted 10 lines
Apply this patch:
https://github.com/jgunthorpe/rdma-
plumbing/commit/64ed4fd30a2157bdaf0a8f7f1b04d2dad02c1b8fIt hardwires the search path into libverbs and libverbs will look into the place it was installed before searching the system library path. That will eliminate the problem you mentioned above.
Quoted 8 lines
Quoted 6 lines
Further, after that patch is applied you can do a rough imitation of 'run-in-place' like this:
$ cmake .. -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib - DSYSCONF_INSTALL_DIR:PATH=/etcAnd how do I build this after the cmake?
Sorry, you always need the -GNinja, I was just being brief
I also mistyped the ETC name.
I tested this:
$ mkdir build ; cd build $ cmake .. -GNinja -DVERBS_PROVIDER_DIR:PATH=`pwd`/lib - DCMAKE_INSTALL_SYSCONFDIR:PATH=/etc $ ninja $ strace bin/ibv_devinfo 2>&1 | grep -i rdmav2 open("/tmp/rdma-core/build/lib/libmlx4-rdmav2.so", O_RDONLY|O_CLOEXEC) = 3
Quoted 5 lines
Note
LD_LIBRARY_PATHis needed to run things from outside build/bin/I have some thoughts to make this simpler, call it a proof of concept for run-from-build ..
No worries. I'm just thick headed. :) Its working for me. I've made a change to libcxgb4 and can test by running in the build tree. I compile with and without my change and see that the correct libcxgb4 is being used. Ship it! :)
So if I have a libcxgb4 change to submit, I should submit it against rdma-core, ja?
Thanks Jason!
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Steve Wise
So if I have a libcxgb4 change to submit, I should submit it against rdma-core, ja?
Yes, Doug or Leon will apply it.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 12 lines
Quoted 11 lines
I recommend *against* running 'make install' to /usr. This is a big package now, it is hard to unwind it once it installs over your system. I really should finish the run in place patch :|
If you do this:
$ echo /usr/local/lib64/ > /etc/ld.so.conf.d/usrlocal.conf && ldconfigThen you can install to
/usr/local/and things will work as you expect. Note, this depends on a verbs patch that is still only in my preview tree.
Quoted 7 lines
So I have a dev system setup with everything installed from the distro in /usr. I have my rdma-core git tree. I change libcxgb4 to fix a bug. I want to test it. I install it to
/usr/localand add/etc/ld.so.conf.d/usrlocal.conf. But that still doesn't work because ldconfig still finds libcxgb4 in/usr/lib64first...How can I force ldconfig to look in
/usr/local/lib64first?
Actually, I just adjusted the search order with LD_LIBRARY_PATH and got what I needed.
Thanks!
Steve.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html