Read Blog Post

Read Blog Post

Contributed on May 19, 2024

27

58

How to Run Docker with GPU Support

Blog: Imixs Workflow Blog

In this short tutorial I will explain how to run a Docker container with GPU support. This becomes important if you want to run AI applications such as the Imixs-AI project with Docker. My tutorial is based on Debian 12 (Bookworm) and the usage of a NVIDA graphic card. I assume that you already have installed Debian and the Docker engine. Read the official install guide about how to install the Docker engine on Debian.

Install NVIDIA Driver

In the following section I show how install the proprietary NVIDIA Drivers with the Cuda-Toolkit on Debian Bookworm. The Cuda toolkit includes GPU-accelerated libraries do run high-performance, GPU-accelerated applications. There are also open source drivers available, but I did not test this.

Frist update your repo and

sudo apt update
sudo apt upgrade
# Remove previous installed nvida drivers (optional)
sudo apt autoremove nvidia* --purge
# Enable Contrib and Non-Free Repositories on Debian
sudo apt install software-properties-common -y
sudo add-apt-repository contrib non-free-firmware

Next Import the Nvidia APT Repository. This repo allows access to additional Nvidia tools like nvida-smi

sudo apt install dirmngr ca-certificates software-properties-common apt-transport-https dkms curl -y
sudo curl -fSsL https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/3bf863cc.pub | sudo gpg --dearmor | sudo tee /usr/share/keyrings/nvidia-drivers.gpg > /dev/null 2>&1
echo 'deb [signed-by=/usr/share/keyrings/nvidia-drivers.gpg] https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/ /' | sudo tee /etc/apt/sources.list.d/nvidia-drivers.list
sudo apt update

Finally Install Nvidia Drivers on Debian via DEFAULT APT Repository. I assume you have a 64-bit system here.

sudo apt update
sudo apt install linux-headers-amd64 nvidia-detect

Now you can check the Nvidia support with the nvida-detect command. This will give you hints how to install the driver:

nvidia-detect
Detected NVIDIA GPUs:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)

Checking card:  NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
Your card is supported by all driver versions.
Your card is also supported by the Tesla drivers series.
Your card is also supported by the Tesla 470 drivers series.
It is recommended to install the
    nvidia-driver
package.

The output reveals that the machine features a GeForce GTX 1080 card and recommends installing the nvidia-driver package. Now you can finally install the recommended package together with the cuda toolkit.

sudo apt install nvidia-driver nvidia-smi linux-image-amd64 cuda

I install in addition the Nvida Service-Management-Interface which allows you to verify your installation. Finally reboot your system.

Note: it may happen that you need a hard reset on your machine. I don’t know exactly why but it could be something with driver conflicts.

Verify Installation

To verify your installation run nvidia-smi which shows you some insights of your environment:

# nvidia-smi
Sun Mar 31 10:46:20 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1080        Off |   00000000:01:00.0 Off |                  N/A |
| 36%   42C    P0             39W /  180W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuring Docker with GPU Support

To get things done in addition it is necessary to install the ‘NVIDIA Container Toolkit’. This will add the GPU support for your Docker engine.

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey |sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg 
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list 
&& sudo apt-get update
$ sudo apt-get install -y nvidia-container-toolkit    
$ sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test you setup with

nvidia-container-cli -k -d /dev/tty info

This should not show any errors.

Now you can start a test container to verify your new Docker GPU support:

docker run --gpus all nvidia/cuda:12.3.1-base-ubuntu20.04  nvidia-smi

This test container should just show the nvidia-smi output form above.

That’s it! Now you have a Docker Environment with GPU Support.