OFED-4.8, rdma-core, and library paths

From what I can tell, it uses the ofed_info command, if it exists, to determine

Hey,

I think we have an issue with the new rdma-core packaging and OFED-4.8. I think OFED-4.8 installs provider libraries in a different location than previous releases of the provider libs. The result of this, I think, is that OFED-4.8 installed over a destro with its own provider libs installed will result in two versions of the libs installed, and further, the system might end up using the distro provider libs with newer OFED drivers, which could be problematic. I believe the OFED installer uninstalls previous OFED rpms, but not distro rpms.
From what I can tell, it uses the ofed_info command, if it exists, to determine
which rpms to uninstall. So if there is no previous OFED installed, then ofed_info will not exist so the distro rdma rpms will not be uninstalled. Prior to OFED-4.8, I think this was somewhat benign, because OFED would install the provider libs over the currently installed distro libs, and thus nobody noticed. But now the rdma-core package puts the provider libs in a different location, thus exposing this issue.

What do folks think about this? Should OFED-4.8 try and uninstall rdma cmds/libs regardless of where they came from? Perhaps optionally. Or should this just be documented so the admin is required to deal with it? I think if we leave OFED as-is, we'll end up with lots of support issues where old libs are being loaded causing problems.

Thoughts? Am I missing something in the OFED-4.8 installer that avoids this issue?

Steve.

Quoted 5 lines

Thoughts? Am I missing something in the OFED-4.8 installer that avoids this issue?

Quoted 1 line

Steve.

Hi Steve,
OFED's install script uninstalls previous OFED versions and corresponding in-box RPMs as well.
The list of in-box RPMs that should be uninstalled is maintained in the install.pl script itself. ofed_info is not used by the install.pl script. So, there should be no issue here.

I think we have an issue with the new rdma-core packaging and OFED-4.8. I think OFED-4.8 installs provider libraries in a different location than previous releases of the provider libs.

The result of this, I think, is that OFED-4.8 installed over a destro with its own provider libs installed will result in two versions of the libs installed,

and further, the system might end up using the distro provider libs with newer OFED drivers, which could be problematic.

What do folks think about this? Should OFED-4.8 try and uninstall rdma cmds/libs regardless of where they came from? Perhaps optionally. Or should this just be documented so the admin is required to deal with it? I think if we leave OFED as-is, we'll end up with lots of support issues where old libs are being loaded causing problems.

Hm, possibly yes. ibverbs first checks the new location, if the provider is not there then it will fall back to a naked dlopen which could find providers in the system library path if there was a .driver file for it.

However, starting in rdma-core 13 the distro providers will not be link compatible with new libibverbs and will silently fail to load.

Quoted 2 lines

What do folks think about this? Should OFED-4.8 try and uninstall rdma cmds/libs regardless of where they came from? Perhaps optionally. Or

Quoted 1 line

this just be documented so the admin is required to deal with it? I think

Quoted 1 line

leave OFED as-is, we'll end up with lots of support issues where old libs

Quoted 1 line

being loaded causing problems.

I'm not sure it is likely someone will hit this. The user would need a distro packaged provider that is not included in rdma-core. AFAIK no such thing exists...

Do you know of another way to trigger wrong loading?

Quoted 5 lines

Hm, possibly yes. ibverbs first checks the new location, if the provider is not there then it will fall back to a naked dlopen which could find providers in the system library path if there was a .driver file for it.

Hmm, so it will load the provider libraries directly specifying the full path? IE 'ldconfig -p' doesn't matter?

Quoted 6 lines · Steve Wise

Quoted 5 lines

Hm, possibly yes. ibverbs first checks the new location, if the provider is not there then it will fall back to a naked dlopen which could find providers in the system library path if there was a .driver file for it.

Quoted 2 lines

Hmm, so it will load the provider libraries directly specifying the full path? IE 'ldconfig -p' doesn't matter?

As the first try, yes. That is the usual way to locate plugins. Typically you don't want the system linker searching plugin directories.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 7 lines · Jason Gunthorpe

Quoted 6 lines · Steve Wise

Quoted 5 lines

Hm, possibly yes. ibverbs first checks the new location, if the provider is not there then it will fall back to a naked dlopen which could find providers in the system library path if there was a .driver file for it.

Quoted 6 lines

Quoted 2 lines

Hmm, so it will load the provider libraries directly specifying the full path? IE 'ldconfig -p' doesn't matter?

As the first try, yes. That is the usual way to locate plugins. Typically you don't want the system linker searching plugin directories.

Jason,
I have a slightly different question, but it is still in context of plugins and linkers.

In v0 of DV, you asked from us to create normal shared library for mlx5, e.g. libmlx5.so. In order to be visible to "gcc -lmlx5"i command, it should be placed in the same directory as libibverbs.so, while plugins should be placed in libibverbs folder.

I did it by using symlinks
$l /usr/lib64/libibverbs

lrwxrwxrwx    1 root root   10 Feb  7 18:09 libmlx5-rdmav2.so -> ../libmlx5.so
$l /usr/lib64 | grep mlx5
lrwxrwxrwx    1 root root   12 Feb  7 18:09 libmlx5.so -> libmlx5.so.1
lrwxrwxrwx    1 root root   17 Feb  7 18:09 libmlx5.so.1 -> libmlx5.so.1.0.13

It works, but I don't know if it is right approach from distro/packaging perspective. Is it ok?

Quoted 31 lines · Leon Romanovsky

Quoted 7 lines · Jason Gunthorpe

Quoted 6 lines · Steve Wise

Quoted 5 lines

Hm, possibly yes. ibverbs first checks the new location, if the provider is not there then it will fall back to a naked dlopen which could find providers in the system library path if there was a .driver file for it.

Quoted 6 lines

Quoted 2 lines

Hmm, so it will load the provider libraries directly specifying the full path? IE 'ldconfig -p' doesn't matter?

As the first try, yes. That is the usual way to locate plugins. Typically you don't want the system linker searching plugin directories.

Jason,
I have a slightly different question, but it is still in context of plugins and linkers.

In v0 of DV, you asked from us to create normal shared library for mlx5, e.g. libmlx5.so. In order to be visible to "gcc -lmlx5"i command, it should be placed in the same directory as libibverbs.so, while plugins should be placed in libibverbs folder.

I did it by using symlinks
$l /usr/lib64/libibverbs

lrwxrwxrwx    1 root root   10 Feb  7 18:09 libmlx5-rdmav2.so -> ../libmlx5.so
$l /usr/lib64 | grep mlx5
lrwxrwxrwx    1 root root   12 Feb  7 18:09 libmlx5.so -> libmlx5.so.1
lrwxrwxrwx    1 root root   17 Feb  7 18:09 libmlx5.so.1 -> libmlx5.so.1.0.13

Quoted 2 lines

It works, but I don't know if it is right approach from distro/packaging perspective. Is it ok?

I think this is basically OK. The symlink should point to the real file (libmlx5.so.1.0.13) though.

Also, make sure that cmake computes the .. properly, something like

execute_process("realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

You'll want to create a new rdma_dv_provider function to handle all of this.

Jason

I made it (rdma_shared_provider function), but have a very hard time to properly create ".." symlink, because during the build (in place too) the output is placed in build/lib in flat structure and symlinks need to be without "..". But during installation phase, these symlinks should be changed to ".." and it doesn't work for me in automatic way :(

Quoted 5 lines · Leon Romanovsky

I made it (rdma_shared_provider function), but have a very hard time to properly create ".." symlink, because during the build (in place too) the output is placed in build/lib in flat structure and symlinks need to be without "..". But during installation phase, these symlinks should be changed to ".." and it doesn't work for me in automatic way :(

For symlinks build/ and installed are two different flows, they don't intermix

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

For install, use realpath like this:

execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

rdma_install_symlink("${LINK_PATH}/libibverbs-dv-mlx5.so.1.0.13" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")

The rdma_install_symlink helper takes care of the install step.

Jason

Quoted 11 lines · Jason Gunthorpe

Quoted 5 lines · Leon Romanovsky

I made it (rdma_shared_provider function), but have a very hard time to properly create ".." symlink, because during the build (in place too) the output is placed in build/lib in flat structure and symlinks need to be without "..". But during installation phase, these symlinks should be changed to ".." and it doesn't work for me in automatic way :(

For symlinks build/ and installed are two different flows, they don't intermix

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

Quoted 4 lines

For install, use realpath like this:

execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

It doesn't work for me :(
-- verbs_provder_dir = /usr/lib64/libibverbs
-- install_libdir = /usr/lib64
-- link_path =

lrwxrwxrwx 1 root root     18 Feb  7 21:43 /lib64/libibverbs/libmlx5-rdmav2.so -> /libmlx5.so.1.0.13
lrwxrwxrwx 1 root root     10 Feb  7 21:43 /lib64/libmlx5-rdmav2.so -> libmlx5.so
lrwxrwxrwx 1 root root     12 Feb  7 21:43 /lib64/libmlx5.so -> libmlx5.so.1
lrwxrwxrwx 1 root root     17 Feb  7 21:43 /lib64/libmlx5.so.1 -> libmlx5.so.1.0.13
-rwxr-xr-x 1 root root 666637 Feb  7 21:43 /lib64/libmlx5.so.1.0.13
lrwxrwxrwx 1 root root     18 Feb  7 21:43 /usr/lib64/libibverbs/libmlx5-rdmav2.so -> /libmlx5.so.1.0.13
lrwxrwxrwx 1 root root     10 Feb  7 21:43 /usr/lib64/libmlx5-rdmav2.so -> libmlx5.so
lrwxrwxrwx 1 root root     12 Feb  7 21:43 /usr/lib64/libmlx5.so -> libmlx5.so.1
lrwxrwxrwx 1 root root     17 Feb  7 21:43 /usr/lib64/libmlx5.so.1 -> libmlx5.so.1.0.13
-rwxr-xr-x 1 root root 666637 Feb  7 21:43 /usr/lib64/libmlx5.so.1.0.13

Quoted 4 lines

rdma_install_symlink("${LINK_PATH}/libibverbs-dv-mlx5.so.1.0.13" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")

The rdma_install_symlink helper takes care of the install step.

Quoted 1 line

Jason

Quoted 4 lines

Quoted 2 lines

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

Quoted 5 lines

Quoted 3 lines

For install, use realpath like this:

execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

It doesn't work for me :(

add_custom_target(share_link ALL DEPENDS "${DEST}" COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"

	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")

add_dependencies(share_link ${DEST})

  install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")

  rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")

Quoted 4 lines · Leon Romanovsky

Quoted 4 lines

Quoted 2 lines

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

That will clash with the legacy providers, don't recommend it.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

For install, use realpath like this:

execute_process(COMMAND "realpath --relative-to ${VERBS_PROVIDER_DIR} ${CMAKE_INSTALL_LIBDIR}/libmlx5.so.1.0.13" OUTPUT_VARIBALE LINK_PATH)

Quoted 1 line

Quoted 1 line

It doesn't work for me :(

Ah realpath needs the paths to exist.. Sigha

Another alternative is this:

python -c "import os; print(os.path.relpath('/usr/lib64/libmlx4.so','/usr/lib64/libibverbs'));"

Quoted 4 lines

add_custom_target(share_link ALL DEPENDS "${DEST}" COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"

	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")

add_dependencies(share_link ${DEST})

The doesn't need a build time, just do:

    execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
                 "lib${DEST}.so.${VERSION}"
		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")

It creates a dangling link until compilation, which is fine..

Quoted 2 lines

  install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")

This line isn't needed, it is part of the next function

Quoted 2 lines

  rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/libmlx5-rdmav2.so")

Should be

   rdma_install_symlink("${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")a

And most distros will not allow an absolute link under /usr which is why you need to make a relpath work and use that instead of ${CMAKE_INSTALL_LIBDIR}

Jason

Quoted 5 lines · Jason Gunthorpe

Ah realpath needs the paths to exist.. Sigha

Another alternative is this:

python -c "import os; print(os.path.relpath('/usr/lib64/libmlx4.so','/usr/lib64/libibverbs'));"

It prints to the screen, but doesn't update OUTPUT_VARAIBLE :(
115 execute_process(COMMAND python -c "import os; print(os.path.relpath('/usr/lib64/libmlx5.so','/usr/lib64/libibverbs'));"
116 OUTPUT_VARIBALE LINK_PATH)
117 message(STATUS "link_path = ${LINK_PATH}")

..
-- Performing Test HAVE_C_WREDUNDANT_DECLS - Failed ../libmlx5.so
-- link_path =
-- Missing Optional Items:

Quoted 5 lines · Jason Gunthorpe

Quoted 3 lines · Leon Romanovsky

Quoted 3 lines

Quoted 2 lines

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

Quoted 3 lines

Quoted 1 line

Quoted 1 line

Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

That will clash with the legacy providers, don't recommend it.

Is it real scenario? Will these legacy providers co-exists with rdma-core library? I really don't like libibverbs-dv-mlx5 name, it is too long and have feature name in the name.

Quoted 5 lines

    execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
                 "lib${DEST}.so.${VERSION}"
		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")

It creates a dangling link until compilation, which is fine..

In non-ninja builds, it doesn't create libmlx5-rdmav2.so symlink in build/lib, because libmlx5.so is not created yet. This is why I ended with custom target.

Quoted 3 lines

Quoted 1 line

  install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")

This line isn't needed, it is part of the next function

Not for build/lib

Quoted 6 lines · Leon Romanovsky

Quoted 5 lines · Jason Gunthorpe

Quoted 3 lines · Leon Romanovsky

Quoted 3 lines

Quoted 2 lines

Compile the library to build/lib/libibverbs-dv-mlx5.so.1.0.13
and setup a symlink build/lib/libmlx5-rdmav2.so -> libibverbs-dv-mlx5.so.1.0.13

Quoted 7 lines

Quoted 3 lines

Quoted 1 line

Quoted 1 line

Just as a note, I'm calling it libmlx5 and not libibverbs-dv-mlx5.

That will clash with the legacy providers, don't recommend it.

Is it real scenario? Will these legacy providers co-exists with rdma-core library? I really don't like libibverbs-dv-mlx5 name, it is too long and have feature name in the name.

I'd save having the 'dv' in the name is sort of the point...

The possible issue would be if an app links to this new library and is run on an old system with the old type of library - then you'd get a really ugly linker error message. It is best to avoid re using library names.

That said, the old name is libmlx4-rdmav2.so so it should be OK..

Quoted 9 lines

Quoted 5 lines

    execute_process(COMMAND "${CMAKE_COMMAND}" -E create_symlink
                 "lib${DEST}.so.${VERSION}"
		 "${BUILD_LIB}/lib${DEST}-rdmav2.so")

It creates a dangling link until compilation, which is fine..

In non-ninja builds, it doesn't create libmlx5-rdmav2.so symlink in build/lib, because libmlx5.so is not created yet. This is why I ended with custom target.

Eh? That doesn't sound right.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

  install(FILES "${BUILD_LIB}/lib${DEST}-rdmav2.so" DESTINATION "${CMAKE_INSTALL_LIBDIR}")

This line isn't needed, it is part of the next function

Not for build/lib

? Do not install something calleld lib*-rdmav2.so to /usr/lib/

Jason

On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:

Thanks for the help, my final version which works correctly for build in place, install from sources and packages for centos6/centos6 is below:

# Create a special provider with exported symbols in it function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
# Installed driver file
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n") install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")

  # Uninstalled driver file
  file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
  file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")

  # Create a static provider library
  if (ENABLE_STATIC)
    add_library(${DEST} STATIC ${ARGN})
    set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
    install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")

    list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
    set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")

endif()

  # Create the plugin shared library
  add_library(${DEST} SHARED ${ARGN})
  # Even though these are modules we still want to use Wl,--no-undefined
  set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
  rdma_set_library_map(${DEST} ${VERSION_SCRIPT})

target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
set_target_properties(${DEST} PROPERTIES
SOVERSION ${SOVERSION}
VERSION ${VERSION}
LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
add_custom_target(share_link ALL DEPENDS "${DEST}" COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"

	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")

add_dependencies(share_link ${DEST})

  install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
  execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
    ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
    OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
  rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")

and buildlib/relpath
import os
import sys

print(os.path.relpath(sys.argv[1], sys.argv[2]))

But I still have issues with DEB package.
It creates absolute (and wrong) symlink instead of relative one.

-rw-r--r-- 1 leonro leonro 46K Feb 8 17:13 libmlx4-rdmav2.so
lrwxrwxrwx 1 leonro leonro 65 Feb 8 17:13 libmlx5-rdmav2.so -> /home/leonro/src/build-deb/lib/x86_64-linux-gnu/libmlx5.so.1.0.13

    ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}

Quoted 4 lines · Leon Romanovsky

On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:

Thanks for the help, my final version which works correctly for build in place, install from sources and packages for centos6/centos6 is below:

Looks good

Quoted 34 lines

# Create a special provider with exported symbols in it function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
# Installed driver file
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n") install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")

  # Uninstalled driver file
  file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
  file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")

  # Create a static provider library
  if (ENABLE_STATIC)
    add_library(${DEST} STATIC ${ARGN})
    set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
    install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")

    list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
    set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")

endif()

  # Create the plugin shared library
  add_library(${DEST} SHARED ${ARGN})
  # Even though these are modules we still want to use Wl,--no-undefined
  set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
  rdma_set_library_map(${DEST} ${VERSION_SCRIPT})

target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
set_target_properties(${DEST} PROPERTIES
SOVERSION ${SOVERSION}
VERSION ${VERSION}
LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
add_custom_target(share_link ALL DEPENDS "${DEST}" COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"

Quoted 2 lines

	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")

add_dependencies(share_link ${DEST})

Except this really shouldn't be a rule. The non-rule method is used everywhere else (eg man pages), so it must work here, it doesn't make sense that ninja vs make would be any different. When you get everything working put it back to execute_process..

Quoted 11 lines

  install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")
  execute_process(COMMAND python ${CMAKE_SOURCE_DIR}/buildlib/relpath
    ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}
    OUTPUT_VARIABLE DEST_LINK_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
  rdma_install_symlink("${DEST_LINK_PATH}" "${VERBS_PROVIDER_DIR}/lib${DEST}-rdmav2.so")

and buildlib/relpath
import os
import sys

print(os.path.relpath(sys.argv[1], sys.argv[2]))

Sure, yes, doing the escaping right would probably be quite hard, that is the usual nightmare with shell stuff unfortunately.

Quoted 2 lines

But I still have issues with DEB package.
It creates absolute (and wrong) symlink instead of relative one.

Hmm. Can you update your github? I'll look for you.

Quoted 3 lines

-rw-r--r-- 1 leonro leonro 46K Feb 8 17:13 libmlx4-rdmav2.so
lrwxrwxrwx 1 leonro leonro 65 Feb 8 17:13 libmlx5-rdmav2.so -> /home/leonro/src/build-deb/lib/x86_64-linux-gnu/libmlx5.so.1.0.13

Most likely this means that DEST_LINK_PATH passed to rdma_install_symlink is not correct. Which suggests python relpath is not working..

relpath in python is done textually, from these args:

Quoted 1 line

    ${CMAKE_INSTALL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}

But

set(VERBS_PROVIDER_DIR "${CMAKE_INSTALL_FULL_LIBDIR}/libibverbs"

So if CMAKE_INSTALL_FULL_LIBDIR != CMAKE_INSTALL_LIBDIR (which depends on how the distro build script configures cmake) then it will fail.

Switch to:

     ${CMAKE_INSTALL_FULL_LIBDIR}/lib${DEST}.so.${VERSION} ${VERBS_PROVIDER_DIR}

Jason

Quoted 1 line

Hmm. Can you update your github? I'll look for you.

I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
https://github.com/rleon/rdma-core/commits/m/dv-v1
Updated cover letter with changelog:
https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977fbf95aa7ee47a69e
And all cmake/packages stuff in the patch:
https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d568476314c1f65ddc

Quoted 8 lines · Leon Romanovsky

Quoted 1 line

Hmm. Can you update your github? I'll look for you.

I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
https://github.com/rleon/rdma-core/commits/m/dv-v1
Updated cover letter with changelog:
https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977fbf95aa7ee47a69e
And all cmake/packages stuff in the patch:
https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d568476314c1f65ddc

This should take care of most things

https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16b7056f1effa9136e8afe46

The debian packaging still needs some adjusting.

I suspect it is against Debian policy to include libmlx.so in the providers package, it probably needs to have its own package and its own -dev package.

Jason

Quoted 11 lines

Quoted 2 lines

Hmm. Can you update your github? I'll look for you.

I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
https://github.com/rleon/rdma-core/commits/m/dv-v1
Updated cover letter with changelog:
https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977f
bf95aa7ee47a69e
And all cmake/packages stuff in the patch:
https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d56
8476314c1f65ddc

This should take care of most things

https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16
b7056f1effa9136e8afe46

The debian packaging still needs some adjusting.

I suspect it is against Debian policy to include libmlx.so in the providers package, it probably needs to have its own package and its own -dev package.

Am Mittwoch, den 08.02.2017, 13:56 -0700 schrieb Jason Gunthorpe:

Quoted 20 lines · Leon Romanovsky

Quoted 11 lines

Quoted 2 lines

Hmm. Can you update your github? I'll look for you.

I pushed latest code to m/dv-1 branch, it is not for inclusion yet.
https://github.com/rleon/rdma-core/commits/m/dv-v1
Updated cover letter with changelog:
https://github.com/rleon/rdma-core/commit/a332669511d5bbbd3dcdd977f
bf95aa7ee47a69e
And all cmake/packages stuff in the patch:
https://github.com/rleon/rdma-core/commit/fcc4996c7fa5169fe599c2d56
8476314c1f65ddc

This should take care of most things

https://github.com/jgunthorpe/rdma-plumbing/commit/490b7fb184c414ab16
b7056f1effa9136e8afe46

The debian packaging still needs some adjusting.

Quoted 3 lines

I suspect it is against Debian policy to include libmlx.so in the providers package, it probably needs to have its own package and its own -dev package.

When other packages should be able to link against libmlx.so and need header files for that, it is better to have separate binary packages (libmlx and libmlx-dev) for it.

--
Benjamin Drung
System Developer
Debian & Ubuntu Developer

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: [email protected]
URL: http://www.profitbricks.com

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506B. Geschäftsführer: Andreas Gauger, Achim Weiss.

Quoted 4 lines · Leon Romanovsky

On Wed, Feb 08, 2017 at 10:33:35AM -0700, Jason Gunthorpe wrote:

Thanks for the help, my final version which works correctly for build in place, install from sources and packages for centos6/centos6 is below:

Looks good

Quoted 34 lines

# Create a special provider with exported symbols in it function(rdma_shared_provider DEST VERSION_SCRIPT SOVERSION VERSION)
# Installed driver file
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" "driver ${DEST}\n") install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${DEST}.driver" DESTINATION "${CONFIG_DIR}")

  # Uninstalled driver file
  file(MAKE_DIRECTORY "${BUILD_ETC}/libibverbs.d/")
  file(WRITE "${BUILD_ETC}/libibverbs.d/${DEST}.driver" "driver ${BUILD_LIB}/lib${DEST}\n")

  # Create a static provider library
  if (ENABLE_STATIC)
    add_library(${DEST} STATIC ${ARGN})
    set_target_properties(${DEST} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
    install(TARGETS ${DEST} DESTINATION "${CMAKE_INSTALL_LIBDIR}")

    list(APPEND RDMA_STATIC_LIBS ${DEST}-rdmav2 ${DEST})
    set(RDMA_STATIC_LIBS "${RDMA_STATIC_LIBS}" CACHE INTERNAL "")

endif()

  # Create the plugin shared library
  add_library(${DEST} SHARED ${ARGN})
  # Even though these are modules we still want to use Wl,--no-undefined
  set_target_properties(${DEST} PROPERTIES LINK_FLAGS ${CMAKE_SHARED_LINKER_FLAGS})
  rdma_set_library_map(${DEST} ${VERSION_SCRIPT})

target_link_libraries(${DEST} LINK_PRIVATE ${COMMON_LIBS_PIC})
target_link_libraries(${DEST} LINK_PRIVATE ibverbs)
target_link_libraries(${DEST} LINK_PRIVATE ${CMAKE_THREAD_LIBS_INIT})
set_target_properties(${DEST} PROPERTIES
SOVERSION ${SOVERSION}
VERSION ${VERSION}
LIBRARY_OUTPUT_DIRECTORY "${BUILD_LIB}")
add_custom_target(share_link ALL DEPENDS "${DEST}" COMMAND ${CMAKE_COMMAND} -E create_symlink "lib${DEST}.so.${VERSION}"

Quoted 2 lines

	  "${BUILD_LIB}/lib${DEST}-rdmav2.so")

add_dependencies(share_link ${DEST})

Except this really shouldn't be a rule. The non-rule method is used everywhere else (eg man pages), so it must work here, it doesn't make sense that ninja vs make would be any different. When you get everything working put it back to execute_process..