[Build] CUDA version linkage

[bbrouwers]

Describe the issue

I'm have succesfully compiled ONNXRuntime 1.18.0 with CUDA from source for Nvidia Jetson Xavier without TensorRT (latest JetPack 5 version). As per instructions I installed the CUDA12 (development) libraries, updated GCC to v10.5. Compiling succeeds using the following instruction: ./build.sh --config Release --update --build --parallel --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --compile_no_warning_as_error.

However when running my application linked to the static binaries produced it throws the following exception: libonnxruntime_providers_cuda.so with error: libcudart.so.11.0

This indicates that somehow it is still looking for CUDA 11 Runtime. How does this happen and how can I resolve this issue so it looks for CUDA 12 instead?

Urgency

Blocking issue for us since it prevents doing our tests

Target platform

aarch64

Build script

As shipped in repo. Instruction: ./build.sh --config Release --update --build --parallel --use_cuda --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu --compile_no_warning_as_error

During configuration the following information is shown, indicating CUDA 12 is properly found:

-- Found CUDAToolkit: /usr/local/cuda/targets/aarch64-linux/include (found version "12.2.140")
-- The CUDA compiler identification is NVIDIA 12.2.140 with host compiler GNU 10.5.0
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done

$ ls -la /usr/local/cuda
lrwxrwxrwx 1 root root 22 Feb 22 20:36 /usr/local/cuda -> /etc/alternatives/cuda
$ ls -la /etc/alternatives/cuda
lrwxrwxrwx 1 root root 20 Feb 26 19:18 /etc/alternatives/cuda -> /usr/local/cuda-12.2

Error / output

Running ONNXRuntime 1.18.0 which is linked / compiled with CUDA 12 the following runtime exception is thrown: libonnxruntime_providers_cuda.so with error: libcudart.so.11.0. This indicates that CUDA 11 is linked to

Visual Studio Version

No response

GCC / Compiler Version

10.5

[tianleiwu]

Follow https://onnxruntime.ai/docs/build/eps.html#nvidia-jetson-tx1tx2nanoxavierorin

To make sure you are using cuda 12.2, you can try the following:

sudo apt-get install -y cuda-compat-12-2
export LD_LIBRARY_PATH="/usr/local/cuda-12.2/lib64:/usr/local/cuda-12.2/compat:$LD_LIBRARY_PATH"
export CUDACXX="/usr/local/cuda-12.2/bin/nvcc"
pip install cmake
./build.sh --config Release --parallel --build_wheel --use_cuda --cuda_home /usr/local/cuda-12.2 --cudnn_home /usr/lib/aarch64-linux-gnu --compile_no_warning_as_error --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=native onnxruntime_BUILD_UNIT_TESTS=OFF

[bbrouwers]

Hi @tianleiwu, thank you for you help! I finally figured out what was going on. Apparently there was an older left over version of libonnxruntime_providers_cuda.so in the LD search paths. The application just found that version before the newer one and thus rendering the error.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.