Builds and runs on Ubuntu 16.04, fingers crossed.

Author: Holst, Henrik

README.md

@@ -3,10 +3,6 @@
 </div>
 -----------------
 
-|  **`Linux CPU`**   |  **`Linux GPU PIP`** | **`Mac OS CPU`** |  **`Android`** |
-|-------------------|----------------------|------------------|----------------|
-| [![Build Status](http://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu)](http://ci.tensorflow.org/job/tensorflow-master-cpu) | [![Build Status](http://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-gpu_pip)](http://ci.tensorflow.org/job/tensorflow-master-gpu_pip) | [![Build Status](http://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-mac)](http://ci.tensorflow.org/job/tensorflow-master-mac) | [![Build Status](http://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-android)](http://ci.tensorflow.org/job/tensorflow-master-android) |
-
 **TensorFlow** is an open source software library for numerical computation using
 data flow graphs.  Nodes in the graph represent mathematical operations, while
 the graph edges represent the multidimensional data arrays (tensors) that flow
@@ -18,38 +14,140 @@ organization for the purposes of conducting machine learning and deep neural
 networks research.  The system is general enough to be applicable in a wide
 variety of other domains, as well.
 
-**If you'd like to contribute to tensorflow, be sure to review the [contribution
-guidelines](CONTRIBUTING.md).**
+## Building TensorFlow in Ubuntu 16.04 LTS
+
+Install NVIDIA CUDA packages from Universe repository:
+
+    $ sudo apt install nvidia-cuda-toolkit
+
+Install NVIDIA cuDNN 4.0:
 
-**We use [github issues](https://github.com/tensorflow/tensorflow/issues) for
-tracking requests and bugs, but please see
-[Community](tensorflow/g3doc/resources/index.md#community) for general questions
-and discussion.**
+    $ sudo tar -C /usr/local -xzf /mnt/dl/mirror/cudnn-7.0-linux-x64-v4.0-prod.tgz
+
+Check out the TensorFlow 0.8.0, modified to work on Ubuntu 16.04 LTS
+and with older NVIDIA GPUs with compute capability 3.0:
+
+    $ git checkout [email protected]/hholst/ea-tensorflow.git
+
+Create an Anaconda environment containing the build tools needed:
+```
+$ conda create -n ea-tensorflow python=3.5 swig numpy
+Using Anaconda Cloud api site https://api.anaconda.org
+Fetching package metadata: ....
+Solving package specifications: .........
 
-## Installation
-*See [Download and Setup](tensorflow/g3doc/get_started/os_setup.md) for instructions on how to install our release binaries or how to build from source.*
+Package plan for installation in environment /home/hholst/anaconda3/envs/ea-tensorflow:
 
-People who are a little bit adventurous can also try our nightly binaries:
+The following NEW packages will be INSTALLED:
 
-* Linux CPU only: [Python 2](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=cpu-slave/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-cp27-none-linux_x86_64.whl) ([build history](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=cpu-slave/)) / [Python 3](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=cpu-slave/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-cp34-cp34m-linux_x86_64.whl) ([build history](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=cpu-slave/))
-* Linux GPU: [Python 2](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-working/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-cp27-none-linux_x86_64.whl) ([build history](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=gpu-working/)) / [Python 3](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=gpu-working/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-cp34-cp34m-linux_x86_64.whl) ([build history](http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=gpu-working/))
-* Mac CPU only: [Python 2](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=mac-slave/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-py2-none-any.whl) ([build history](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=mac-slave/)) / [Python 3](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=mac-slave/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.8.0-py3-none-any.whl) ([build history](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=mac-slave/))
-* [Android](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-android/TF_BUILD_CONTAINER_TYPE=ANDROID,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=NO_PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=android-slave/lastSuccessfulBuild/artifact/bazel-out/local_linux/bin/tensorflow/examples/android/tensorflow_demo.apk) ([build history](http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-android/TF_BUILD_CONTAINER_TYPE=ANDROID,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=NO_PIP,TF_BUILD_PYTHON_VERSION=PYTHON2,label=android-slave/))
+    mkl:        11.3.1-0     
+    numpy:      1.11.0-py35_0
+    openssl:    1.0.2h-0     
+    pcre:       8.31-0       
+    pip:        8.1.1-py35_1 
+    python:     3.5.1-0      
+    readline:   6.2-2        
+    setuptools: 20.7.0-py35_0
+    sqlite:     3.9.2-0      
+    swig:       3.0.8-1      
+    tk:         8.5.18-0     
+    wheel:      0.29.0-py35_0
+    xz:         5.0.5-1      
+    zlib:       1.2.8-0      
+
+Proceed ([y]/n)? 
+
+Linking packages ...
+[      COMPLETE      ]|############################################################################| 100%
+#
+# To activate this environment, use:
+# $ source activate ea-tensorflow
+#
+# To deactivate this environment, use:
+# $ source deactivate
+#
+```
+
+You also need to activate the new Anaconda environment:
+
+    $ source activate ea-tensorflow
+
+*Optional:* You can re-run the `./configure` script.
+Observe that we override the compute capability
+to include support for CUDA compute capability 3.0!
+
+```
+$ ./configure
+Please specify the location of python. [Default is /home/hholst/anaconda3/envs/ea-tensorflow/bin/python]: 
+Do you wish to build TensorFlow with GPU support? [Y/n] 
+GPU support will be enabled for TensorFlow
+Please specify which gcc nvcc should use as the host compiler. [Default is /usr/bin/gcc]: 
+Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 
+Please specify the location where CUDA  toolkit is installed. Refer to README.md for more details. [Default is /usr]: 
+Please specify the Cudnn version you want to use. [Leave empty to use system default]: 
+Please specify the location where cuDNN  library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 
+Please specify a list of comma-separated Cuda compute capabilities you want to build with.
+You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
+Please note that each additional compute capability significantly increases your build time and binary size.
+[Default is: "3.5,5.2"]: 3.0,3.5,5.2
+Setting up Cuda include
+Setting up Cuda lib
+Setting up Cuda bin
+find: File system loop detected; ‘/usr/bin/X11’ is part of the same file system loop as ‘/usr/bin’.
+find: File system loop detected; ‘/usr/bin/X11’ is part of the same file system loop as ‘/usr/bin’.
+Configuration finished
+```
+
+### Compiling
+
+Build TensorFlow
+
+    $ bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
+
+Run test
+
+    $ bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu
+
+Build pip package
+
+    $ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
+
+Install pip package:
+
+    $ pip install /tmp/tensorflow_pkg/tensorflow-0.8.0-py3-none-any.whl
+
+### *Try your first TensorFlow program*
+
+NOTE: Make sure you're not standing inside the `ea-tensorflow` git repository 
+when you are starting python. Doing so might cause problems with CUDA runtime library.
 
-#### *Try your first TensorFlow program*
 ```python
 $ python
-
+Python 3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:16:01) 
+[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
+Type "help", "copyright", "credits" or "license" for more information.
 >>> import tensorflow as tf
->>> hello = tf.constant('Hello, TensorFlow!')
->>> sess = tf.Session()
->>> sess.run(hello)
-Hello, TensorFlow!
+I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
+I tensorflow/stream_executor/dso_loader.cc:99] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH: 
+I tensorflow/stream_executor/cuda/cuda_dnn.cc:1562] Unable to load cuDNN DSO
+I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
+I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
+I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
+>>> sess = tf.InteractiveSession()
+I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
+name: GeForce GTX 980
+major: 5 minor: 2 memoryClockRate (GHz) 1.2155
+pciBusID 0000:03:00.0
+Total memory: 4.00GiB
+Free memory: 3.24GiB
+I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
+I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
+I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:03:00.0)
 >>> a = tf.constant(10)
 >>> b = tf.constant(32)
 >>> sess.run(a+b)
 42
->>>
+>>> 
 ```
 
 ##For more information

configure

@@ -40,11 +40,11 @@ echo "$SWIG_PATH" > tensorflow/tools/swig/swig_path
 ## Set up Cuda-related environment settings
 
 while [ "$TF_NEED_CUDA" == "" ]; do
-  read -p "Do you wish to build TensorFlow with GPU support? [y/N] " INPUT
+  read -p "Do you wish to build TensorFlow with GPU support? [Y/n] " INPUT
   case $INPUT in
     [Yy]* ) echo "GPU support will be enabled for TensorFlow"; TF_NEED_CUDA=1;;
     [Nn]* ) echo "No GPU support will be enabled for TensorFlow"; TF_NEED_CUDA=0;;
-    "" ) echo "No GPU support will be enabled for TensorFlow"; TF_NEED_CUDA=0;;
+    "" ) echo "GPU support will be enabled for TensorFlow"; TF_NEED_CUDA=1;;
     * ) echo "Invalid selection: " $INPUT;;
   esac
 done
@@ -86,7 +86,7 @@ while true; do
 
   fromuser=""
   if [ -z "$CUDA_TOOLKIT_PATH" ]; then
-    default_cuda_path=/usr/local/cuda
+    default_cuda_path=/usr
     read -p "Please specify the location where CUDA $TF_CUDA_VERSION toolkit is installed. Refer to README.md for more details. [Default is $default_cuda_path]: " CUDA_TOOLKIT_PATH
     fromuser="1"
     if [ -z "$CUDA_TOOLKIT_PATH" ]; then
@@ -98,10 +98,10 @@ while true; do
   else
     TF_CUDA_EXT=".$TF_CUDA_VERSION"
   fi
-  if [ -e $CUDA_TOOLKIT_PATH/lib64/libcudart.so$TF_CUDA_EXT ]; then
+  if [ -e $CUDA_TOOLKIT_PATH/lib/x86_64-linux-gnu/libcudart.so$TF_CUDA_EXT ]; then
     break
   fi
-  echo "Invalid path to CUDA $TF_CUDA_VERSION toolkit. $CUDA_TOOLKIT_PATH/lib64/libcudart.so$TF_CUDA_EXT cannot be found"
+  echo "Invalid path to CUDA $TF_CUDA_VERSION toolkit. $CUDA_TOOLKIT_PATH/x86_64-linux-gnu/libcudart.so$TF_CUDA_EXT cannot be found"
   if [ -z "$fromuser" ]; then
     exit 1
   fi
@@ -119,7 +119,7 @@ while true; do
 
   fromuser=""
   if [ -z "$CUDNN_INSTALL_PATH" ]; then
-    default_cudnn_path=${CUDA_TOOLKIT_PATH}
+    default_cudnn_path=/usr/local/cuda
     read -p "Please specify the location where cuDNN $TF_CUDNN_VERSION library is installed. Refer to README.md for more details. [Default is $default_cudnn_path]: " CUDNN_INSTALL_PATH
     fromuser="1"
     if [ -z "$CUDNN_INSTALL_PATH" ]; then

tensorflow/core/common_runtime/gpu/gpu_device.cc

@@ -689,8 +689,8 @@ struct CudaVersion {
 
 // "configure" uses the specific name to substitute the following string.
 // If you change it, make sure you modify "configure" as well.
-std::vector<CudaVersion> supported_cuda_compute_capabilities = {
-    CudaVersion("3.5"), CudaVersion("5.2")};
+// Unofficial setting. DO NOT SUBMIT!!!
+std::vector<CudaVersion> supported_cuda_compute_capabilities = {CudaVersion("3.0"), CudaVersion("3.5"), CudaVersion("5.2"),};
 
 }  // namespace
 

tensorflow/tensorflow.bzl

@@ -262,7 +262,7 @@ def tf_cuda_cc_tests(tests, deps, tags=[], size="medium"):
 def tf_gpu_kernel_library(srcs, copts=[], cuda_copts=[], deps=[], hdrs=[],
                           **kwargs):
   cuda_copts = ["-x", "cuda", "-DGOOGLE_CUDA=1",
-                "-nvcc_options=relaxed-constexpr", "-nvcc_options=ftz=true",
+                "-nvcc_options=expt-relaxed-constexpr", "-nvcc_options=ftz=true",
                 "--gcudacc_flag=-ftz=true"] + cuda_copts
   native.cc_library(
       srcs = srcs,
@@ -475,7 +475,7 @@ def tf_custom_op_library(name, srcs=[], gpu_srcs=[], deps=[]):
   if gpu_srcs:
     basename = name.split(".")[0]
     cuda_copts = ["-x", "cuda", "-DGOOGLE_CUDA=1",
-                  "-nvcc_options=relaxed-constexpr", "-nvcc_options=ftz=true",
+                  "-nvcc_options=expt-relaxed-constexpr", "-nvcc_options=ftz=true",
                   "--gcudacc_flag=-ftz=true"]
 
     native.cc_library(

third_party/gpus/crosstool/CROSSTOOL

@@ -50,6 +50,9 @@ toolchain {
   # Use "-std=c++11" for nvcc. For consistency, force both the host compiler
   # and the device compiler to use "-std=c++11".
   cxx_flag: "-std=c++11"
+  # EA: gcc 5.3 tweaks.
+  cxx_flag: "-D_MWAITXINTRIN_H_INCLUDED"
+  cxx_flag: "-D_FORCE_INLINES=1"
   linker_flag: "-lstdc++"
   linker_flag: "-B/usr/bin/"
 
@@ -105,6 +108,10 @@ toolchain {
   compiler_flag: "-Wunused-but-set-parameter"
   # But disable some that are problematic.
   compiler_flag: "-Wno-free-nonheap-object" # has false positives
+  # EA: Disable some pedantic warnings.
+  compiler_flag: "-Wno-unused-function"
+  compiler_flag: "-Wno-unused-local-typedefs"
+  compiler_flag: "-Wno-unused-variable"
 
   # Keep stack frames for debugging, even in opt mode.
   compiler_flag: "-fno-omit-frame-pointer"

third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc

@@ -250,7 +250,8 @@ def InvokeNvcc(argv, log=False):
 
   # "configure" uses the specific format to substitute the following string.
   # If you change it, make sure you modify "configure" as well.
-  supported_cuda_compute_capabilities = [ "3.5", "5.2" ]
+  # Unofficial setting. DO NOT SUBMIT!!!
+  supported_cuda_compute_capabilities = ["3.0", "3.5", "5.2",]
   nvccopts = ''
   for capability in supported_cuda_compute_capabilities:
     capability = capability.replace('.', '')

third_party/gpus/cuda/cuda_config.sh

@@ -101,7 +101,10 @@ function CheckAndLinkToSrcTree {
   # the same. This could happen if invoked from the source tree by accident.
   if [ ! $(readlink -f $PWD) == $(readlink -f $OUTPUTDIR/third_party/gpus/cuda) ]; then
     mkdir -p $(dirname $OUTPUTDIR/third_party/gpus/cuda/$FILE)
-    ln -sf $PWD/$FILE $OUTPUTDIR/third_party/gpus/cuda/$FILE
+    # EA: don't overwrite local changed files (ie., host_config.h)
+    if [ ! -f $OUTPUTDIR/third_party/gpus/cuda/$FILE -o -L $OUTPUTDIR/third_party/gpus/cuda/$FILE ]; then
+      ln -sf $PWD/$FILE $OUTPUTDIR/third_party/gpus/cuda/$FILE
+    fi
   fi
 }
 
@@ -120,8 +123,8 @@ fi
 # Actually configure the source tree for TensorFlow's canonical view of Cuda
 # libraries.
 
-if test ! -e ${CUDA_TOOLKIT_PATH}/lib64/libcudart.so$TF_CUDA_VERSION; then
-  CudaError "cannot find ${CUDA_TOOLKIT_PATH}/lib64/libcudart.so$TF_CUDA_VERSION"
+if test ! -e ${CUDA_TOOLKIT_PATH}/lib/x86_64-linux-gnu/libcudart.so$TF_CUDA_VERSION; then
+  CudaError "cannot find ${CUDA_TOOLKIT_PATH}/lib/x86_64-linux-gnu/libcudart.so$TF_CUDA_VERSION"
 fi
 
 if test ! -d ${CUDNN_INSTALL_PATH}; then
@@ -175,12 +178,12 @@ function LinkAllFiles {
 mkdir -p $OUTPUTDIR/third_party/gpus/cuda
 echo "Setting up Cuda include"
 LinkAllFiles ${CUDA_TOOLKIT_PATH}/include $OUTPUTDIR/third_party/gpus/cuda/include || exit -1
-echo "Setting up Cuda lib64"
-LinkAllFiles ${CUDA_TOOLKIT_PATH}/lib64 $OUTPUTDIR/third_party/gpus/cuda/lib64 || exit -1
+echo "Setting up Cuda lib"
+LinkAllFiles ${CUDA_TOOLKIT_PATH}/lib/x86_64-linux-gnu $OUTPUTDIR/third_party/gpus/cuda/lib64 || exit -1
 echo "Setting up Cuda bin"
 LinkAllFiles ${CUDA_TOOLKIT_PATH}/bin $OUTPUTDIR/third_party/gpus/cuda/bin || exit -1
-echo "Setting up Cuda nvvm"
-LinkAllFiles ${CUDA_TOOLKIT_PATH}/nvvm $OUTPUTDIR/third_party/gpus/cuda/nvvm || exit -1
+#echo "Setting up Cuda nvvm"
+#LinkAllFiles ${CUDA_TOOLKIT_PATH}/nvvm $OUTPUTDIR/third_party/gpus/cuda/nvvm || exit -1
 
 # Set up symbolic link for cudnn
 ln -sf $CUDNN_HEADER_PATH/cudnn.h $OUTPUTDIR/third_party/gpus/cuda/include/cudnn.h || exit -1