Tag: CUDA

CUDART Error in Singularity Container Workaround

Singularity is a Linux container system similar (and compatible to) Docker. It’s advantage over Docker is that is was designed to allow users without superuser privileges to run containers within their environment. I recently encountered the following error when running a Nvidia CUDA application within a Singularity container using “singularity run -nv <container>:

CUDART: cudaGetDeviceCount(&deviceCount); = 999 (unknown error)

Workaround: After running /usr/local/cuda/samples/1_Utilities/deviceQuery/deviceQuery outside of the container, the CUDA application within the container ran without any problem.

Fix/Solution: TODO

 

Finding the NUMA Node of a Nvidia CUDA GPU in Linux

For some applications it might be useful to pin their CPU processes to the NUMA node which is connected to the GPU. To find out which GPU is located at which NUMA node one can use the following script:

for i in $(nvidia-smi -x -q | grep "gpu id" | awk -F "\"" '{print $2}' | awk -F ":" '{print "0000:"  $2 ":"  $3}' | tr '[:upper:]' '[:lower:]'); do echo $i; cat "/sys/bus/pci/devices/$i/numa_node"; done

Pinning the CPU process to the right NUMA node can speed up your application significantly on all Nvidia GPUs like the double precision HPC GPUs Tesla V100, A100 and A10, the professional Quadro RTX GPUs as well as all CUDA capable GeForce GPUs.

 

NVIDIA CUDA on Ubuntu: unsupported GNU version! gcc versions later than 5 are not supported!

After installing CUDA on Ubuntu, compiling CUDA applications with nvcc results in an error similar to this:

In file included from /usr/local/cuda-8.0/bin/../targets/x86_64-linux/include/cuda_runtime.h:78:0,
from <command-line>:0:
/usr/local/cuda-8.0/bin/../targets/x86_64-linux/include/host_config.h:119:2: error: #error — unsupported GNU version! gcc versions later than 5 are not supported!
#error — unsupported GNU version! gcc versions later than 5 are not supported!
^~~~~}}}

Ubuntu comes with a more up-to-date GCC then CUDA can handle. To solve this issue we install GCC 4.9:

sudo apt install gcc-4.9 g++-4.9

Now GCC 4.9 and the up-to-date GCC are installed on the system. To be able to use the up-to-date GCC without CUDA, one can setup and use the “update-alternatives” system for GCC. In this case we replace the default GCC 6 compiler:

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 50 --slave /usr/bin/g++ g++ /usr/bin/g++-4.9

Now we can list the available compilers with

update-alternatives --list gcc

and set the GCC 4.9 as default with

update-alternatives --set gcc /usr/bin/gcc-4.9