Overview
This guide covers deploying the DocBot inference server on an NVIDIA Jetson Orin Nano Super Developer Kit (aarch64 / ARM64, JetPack SDK, 8 GB shared CPU+GPU RAM).
note
The standard x86_64 CUDA images will not work on Jetson. You must use the L4T-based Jetson image and the Jetson-specific Compose configuration described in this guide.
Prerequisites
| Requirement | Notes |
|---|---|
| Jetson Orin Nano Super Developer Kit | Other Orin/Nano variants should also work; 8 GB version recommended |
| JetPack SDK ≥ 6.x (L4T R36.x) | Install via NVIDIA SDK Manager or JetPack download page |
| Docker Engine ≥ 20.10 | Included in JetPack 6; verify with docker --version |
| NVIDIA Container Runtime for L4T | Enables GPU access inside containers on Jetson (see setup below) |
| Internet access | Required to pull the L4T base image and pip packages during the first build |
| Registry credentials | Username + token provided by your vendor (for backend/UI/rag-db images) |
| License key | A license.key file provided by your vendor |
Set Up the NVIDIA Container Runtime for L4T
The NVIDIA Container Runtime allows Docker containers to access the Jetson GPU. JetPack 6 typically installs nvidia-container-toolkit automatically, but you may need to configure it as the Docker default runtime:
# Install (if not already present)
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
# Configure Docker to use the NVIDIA runtime by default
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# Verify — you should see GPU info output:
docker run --rm --runtime=nvidia \
nvcr.io/nvidia/l4t-base:r36.3.0 nvidia-smi