Ytukyg

I've installed CUDA 7.5 toolkit, and Tensorflow inside anaconda env. The CUDA driver is also installed. The folder containing the so libraries is in LD_LIBRARY_PATH. When I import tensorflow I get the following error:

Couldn't open CUDA library libcuda.so. LD_LIBRARY_PATH:
/usr/local/cuda-7.5/lib64

In this folder, there exist a file named libcudart.so (which is actually a symbolic link to libcudart.so.7.5). So (just as a guess) I created a symbolic link to libcudart.so named libcuda.so. Now the library is found by Tensorflow, but as soon as I call tensorflow.Session() I get the following error:

F tensorflow/stream_executor/cuda/cuda_driver.cc:107] Check failed: f
!= nullptr could not find cuInitin libcuda DSO; dlerror:
/usr/local/cuda-7.5/lib64/libcudart.so.7.5: undefined symbol: cuInit

Any ideas?

For future reference, here is what I found out and what I did to solve this problem.
The system is Ubuntu 14.04 64 bit. The NVIDIA driver version that I was trying to install was 367.35. The installation resulted in an error towards the end, with message:

ERROR: Unable to load the kernel module 'nvidia-drm'

However the CUDA samples compiled and run with no problem, so the driver was at least partially installed correctly. However, when I checked the version using:

cat /proc/driver/nvidia/version

The version I got was different (I don't remember exactly but some 352 sub-version).
So I figured out I better remove all traces of the driver and re-install. I followed the instructions in the accepted answer here: https://askubuntu.com/questions/206283/how-can-i-uninstall-a-nvidia-driver-completely, except for the command that makes sure nouveau driver will be loaded in boot.

I finally reinstalled the most up-to-date NVIDIA driver (367.35). The installation finished with no errors and Tensorflow was able to load all libraries.

I think the problem began when someone who worked on the installation before me used apt-get to install the driver, and not a run script. Not sure however.

PS during installation there is a warning:

The distribution-provided pre-install script failed! Are you sure
you want to continue?

Looking at the logs I could locate this pre-install script, and its content is simply:

# Trigger an error exit status to prevent the installer from overwriting

# Ubuntu's nvidia packages.

exit 1

so it seems ok to install despite this warning.

I had this error on a couple of Ubuntu 16.04 machines. I tried just updating the NVIDIA drivers and Cuda toolkit hoping that apt would take care of replacing the missing file, but that didn't happen.

Here's a hopefully clear explanation of how I fixed an error like:

...libcuda.so.1: cannot open shared object file: No such file or directory

You are missing this libcuda.so.1 file apparently.
If you look at other SO posts, you will discover that libcuda.so.1 is actually a symbolic link (fancy Unix term for a thing that looks like a file but actually is just a pointer to another file). Specifically, it is a symbolic link to a libcuda.so.# file that is part of the NVIDIA graphics drivers!!! (not part of the Cuda toolkit). So if you do find wherever the package manager has put the libcuda.so.1 file on your system, you'll see it's pointing to this driver-related file:

$ ls /usr/lib/x86_64-linux-gnu/libcuda.so.1 -la

lrwxrwxrwx 1 root root 17 Oct 25 14:29 /usr/lib/x86_64-linux-gnu/libcuda.so.1 -> libcuda.so.410.73

Okay, so you need to make a symbolic link like the one you found, but where?
I.e., where is Tensorflow looking for this libcuda.so.1? Obviously not where your package manager stuck it.
It turns out that Tensorflow looks in the "load library path".
You can see this path like so:

$ echo $LD_LIBRARY_PATH

and what you get back should include the installed Cuda toolkit:

/usr/local/cuda/lib64

(The exact path might vary on your system)
If not, you need to add the toolkit to $LD_LIBRARY_PATH using some shell command like this (from the NVIDIA Toolkit install manual):

export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

(If you don't find anything in /usr/local/cuda you might not have the toolkit installed.)

Now that you know where Tensorflow looks on the $LD_LIBRARY_PATH for Cuda toolkit, you can add a symbolic link to the toolkit directory.

sudo ln -s /usr/lib/x86_64-linux-gnu/libcuda.so.410.73  /usr/local/cuda/lib64/libcuda.so.1

Or you can just listen to other posts that don't explain what's going on but instead tell you to try installing more things in a bunch of different ways. Didn't work for me though :(