Category: Linux

CUDART Error in Singularity Container Workaround

Singularity is a Linux container system similar (and compatible to) Docker. It’s advantage over Docker is that is was designed to allow users without superuser privileges to run containers within their environment. I recently encountered the following error when running a Nvidia CUDA application within a Singularity container using “singularity run -nv <container>:

CUDART: cudaGetDeviceCount(&deviceCount); = 999 (unknown error)

Workaround: After running /usr/local/cuda/samples/1_Utilities/deviceQuery/deviceQuery outside of the container, the CUDA application within the container ran without any problem.

Fix/Solution: TODO

Finding the NUMA Node of a Nvidia CUDA GPU in Linux

For some applications it might be useful to pin their CPU processes to the NUMA node which is connected to the GPU. To find out which GPU is located at which NUMA node one can use the following script:

for i in $(nvidia-smi -x -q | grep "gpu id" | awk -F "\"" '{print $2}' | awk -F ":" '{print "0000:"  $2 ":"  $3}' | tr '[:upper:]' '[:lower:]'); do echo $i; cat "/sys/bus/pci/devices/$i/numa_node"; done

Pinning the CPU process to the right NUMA node can speed up your application significantly on all Nvidia GPUs like the double precision HPC GPUs Tesla V100, A100 and A10, the professional Quadro RTX GPUs as well as all CUDA capable GeForce GPUs.

How to set C, C++ or Fortran compiler for CMake

To use a different compiler (e.g. Intel Compiler, CLANG or PGI) or a different version then CMake uses by default, one can either set environment variables or modify
the CMakeLists.txt file.

CMake evaluates the environment variables CC for the C compiler, CXX for the C++ compiler and FC
for the Fortran compiler:

CC=/path/to/icc cmake ..
CXX=/path/to/icpc cmake ..
FC=/path/to/ifort cmake ..

For a more permanent solution, one can also edit the CMakeLists.txt file:

SET(CMAKE_C_COMPILER /path/to/pgcc)
SET(CMAKE_CXX_COMPILER /path/to/pgc++)
SET(CMAKE_FC_COMPILER /path/to/pgfortran)

BTW: The environment variables LDFLAGS, CFLAGS, CXXFLAGS or FFLAGS are also evaluated by CMake.

SPACK and Intel Parallel Studio: “error while loading shared libraries:”

Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. With Spack, you can build a package with multiple versions, configurations, platforms, and compilers, and all of these builds can coexist on the same machine.

However, when using the Intel Compiler as compiler, I got the following error for some packages:

 error while loading shared libraries: cannot open shared object file: No such file or directory

To solve this, edit your ~/.spack/linux/compilers.yaml file and set the extra_rpaths to your Intel Compiler libraries directory:

- compiler:
    environment: {}
    extra_rpaths: [/opt/intel/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/]
    flags: {}
    modules: []
    operating_system: centos7
      cc: /opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64/icc
      cxx: /opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64/icpc
      f77: /opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64/ifort
      fc: /opt/intel/compilers_and_libraries_2018.1.163/linux/bin/intel64/ifort
    spec: intel@18.0.1
    target: x86_64

BASH: Nach jedem xten Zeichen ein Zeichen in einen String einfügen

Mit sed "s/.\{x\}/& /g" kann man in eine Zeichenkette alle x Zeichen ein Leerzeichen automatisiert einfügen. Aber auch andere Zeichen sind möglich. So kann man bspw. alle zwei Zeichen einen Doppelpunkt in einen String einfügen:

#~> echo "AA11CC22EE" | sed "s/.\{2\}/&:/g"

Sollte das letzte Zeichen stören, kann man dies wie folgt entfernen:
#~> a = $(echo "AA11CC22EE" | sed "s/.\{2\}/&:/g")
#~> echo ${a%?}


When cmake complains about a missing MHD library, install the microhttpd library development files.


 yum install libmicrohttpd-devel

Or download the up-to-date release from and compile and install it with:

tar -xzf libmicrohttpd-latest.tar.gz
cd libmicrohttpd-*
./configure && make install