Using Nvidia GPU in your lxc container

Recently, I acquired a second-hand Nvidia Tesla M40 to experiment with AGI applications. With some additional second-hand x79 motherboard and CPU components on hand, I decided to assemble them into a dedicated AI server.

Hardware requirements

A functioning computer with an x16 PCI-E slot.
An Nvidia GPU.

Software requirements

Proxmox Installation: Download and install Proxmox on your computer.
Nvidia Linux Driver: Download the Nvidia Linux driver from Nvidia's website. I’m using version 550 for this guide.

Start a http server on your local machine

To run the Nvidia driver installer on both the Proxmox host and the LXC container, we need to transfer the downloaded file to both systems. On my Ubuntu PC, I navigate to the Downloads folder and run the command:

Disabling nouveau driver

The nouveau driver is an open-source alternative for Nvidia GPUs. To use the official driver, we need to disable it:

In the Proxmox interface, select the Proxmox node and click the Shell button.
Create the file /etc/modprobe.d/nvidia-installer-disable-nouveau.conf and add the following lines:
```
blacklist nouveau
options nouveau modeset=0
```
Reboot the system reboot.

Install Nvidia driver

Once the system is back up, install the essential packages for the Nvidia driver and run the installation script:

apt install build-essential pve-headers-$(uname -r)
wget http://[your pc's IP]:8080/NVIDIA-Linux-x86_64-<VERSION>.run
chmod +x ./NVIDIA-Linux-x86_64-<VERSION>.run
./NVIDIA-Linux-x86_64-<VERSION>.run

Next, add the following lines to /etc/modules-load.d/modules.conf:

nvidia
nvidia_uvm

Update the initramfs with:

update-initramfs -u

Create the file /etc/udev/rules.d/70-nvidia.rules and include:

# /etc/udev/rules.d/70-nvidia.rules
# Create /nvidia0, /dev/nvidia1 … and /nvidiactl when nvidia module is loaded
KERNEL=="nvidia", RUN+="/bin/bash -c '/usr/bin/nvidia-smi -L && /bin/chmod 666 /dev/nvidia*'"
# Create the CUDA node when nvidia_uvm CUDA module is loaded
KERNEL=="nvidia_uvm", RUN+="/bin/bash -c '/usr/bin/nvidia-modprobe -c0 -u && /bin/chmod 0666 /dev/nvidia-uvm*'"

Finally, reboot again with reboot.

Enabling LXC Access to the GPU

Open a shell on your Proxmox host and run:

modprobe nvidia-uvm
ls /dev/nvidia* -l

You should see output similar to:

crw-rw-rw- 1 root root 195,   0 Sep 25 16:01 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Sep 25 16:01 /dev/nvidiactl
crw-rw-rw- 1 root root 235,   0 Sep 25 16:01 /dev/nvidia-uvm
crw-rw-rw- 1 root root 235,   1 Sep 25 16:01 /dev/nvidia-uvm-tools

/dev/nvidia-caps:
total 0
cr-------- 1 root root 238, 1 Sep 25 16:01 nvidia-cap1
cr--r--r-- 1 root root 238, 2 Sep 25 16:01 nvidia-cap2

Take note of the numbers after the owner group (the second root). In my case, they are 195, 235, and 238.
Add the following to the LXC config for the container you want to access the GPU:

lxc.cgroup.devices.allow: c <number from previous step>:* rwm
lxc.cgroup.devices.allow: c <number from previous step>:* rwm
lxc.cgroup.devices.allow: c <number from previous step>:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-caps/nvidia-cap1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-caps/nvidia-cap2 none bind,optional,create=file

Repeat for each device as necessary, then start the LXC container.

Install Nvidia driver on the lxc container.

Open the console in your LXC container, log in, and run:

dpkg --add-architecture i386
apt update
apt install libc6:i386
wget http://[your pc's IP]:8080/NVIDIA-Linux-x86_64-<VERSION>.run
chmod +x ./NVIDIA-Linux-x86_64-<VERSION>.run
./NVIDIA-Linux-x86_64-<VERSION>.run --no-kernel-module

After rebooting the container, verify GPU access by typing nvidia-smi.

What I'm running

InvokeAI for text-to-image generation.
Ollama for chatbots.

Note: For InvokeAI, I start the application via command line, so all dependencies must be installed manually. However, Ubuntu 24.04 has an incompatible Python version, necessitating another container running Ubuntu 22.04.