NVLink Setup for Multi-GPU Inference on Tesla V100

This guide walks you through setting up NVLink on Tesla V100 GPUs for multi-GPU inference, enabling high-bandwidth GPU-to-GPU communication that can significantly speed up large model inference. You’ll need two or more V100 GPUs (NVLink-capable), a compatible server motherboard, and NVIDIA drivers and CUDA installed.

What You’ll Need

• Two or more NVIDIA Tesla V100 GPUs (SXM2 or PCIe with NVLink bridge)
• A server motherboard that supports NVLink (e.g., NVIDIA HGX or certified servers)
• NVLink bridges (for PCIe versions) or NVLink connectors (for SXM2)
• NVIDIA driver (version 450 or newer recommended)
• CUDA Toolkit (version 11.0 or newer)
• NVIDIA NVML (comes with driver) and nvidia-smi
• Linux OS (Ubuntu 20.04 or similar) with root/sudo access

Step-by-Step

1. Physically install the GPUs and NVLink bridges. For PCIe V100s, attach the NVLink bridge across the two GPUs’ NVLink connectors (the gold contacts on top). For SXM2 modules, they’re already mounted on the baseboard; ensure the NVLink connectors are aligned and the retention mechanism is locked.
2. Power on the server and enter the BIOS/UEFI. Enable “Above 4G Decoding” and “Resizable BAR” if available (some systems require it for large BAR1). Save and reboot.
3. Install the NVIDIA driver. Download the latest driver from NVIDIA’s website, then run sudo ./NVIDIA-Linux-x86_64-*.run. Follow prompts, selecting “Install” and allowing kernel module compilation. Reboot after installation.
4. Install CUDA Toolkit. Download from NVIDIA’s CUDA downloads page; choose a runfile or package manager (e.g., sudo apt install nvidia-cuda-toolkit for Ubuntu, but the official runfile is recommended). Install and add CUDA to PATH (export PATH=/usr/local/cuda/bin:$PATH).
5. Verify GPUs are detected. Run nvidia-smi. You should see two V100s. Check the NVLink status: nvidia-smi nvlink -s should show each GPU with “Active” links and link speeds (e.g., 6 links at 25 GB/s each for V100).
6. Check NVLink topology. Use nvidia-smi nvlink -t to see the connection matrix. It should show bidirectional links between the two GPUs. If not, re-seat the bridges or check BIOS settings.
7. Enable peer-to-peer access. For CUDA applications, peer-to-peer (P2P) over NVLink must be enabled. In your CUDA code, call cudaDeviceEnablePeerAccess() for each pair. For inference frameworks like TensorFlow or PyTorch, they handle this automatically if GPUs are visible and NVLink is detected.
8. Set GPU visibility. In your inference script, set CUDA_VISIBLE_DEVICES=0,1 to use both GPUs. For PyTorch, use model.parallelize() or torch.nn.DataParallel; for TensorFlow, use tf.distribute.MirroredStrategy.
9. Test with a simple benchmark. Run a multi-GPU inference test (e.g., a BERT model). Monitor NVLink traffic with nvidia-smi nvlink -g 0 to see data transfer rates. If traffic is high, NVLink is working.
10. Optimize for inference. For best performance, use TensorRT with multi-GPU support, or ensure your model is sharded appropriately (e.g., pipeline parallelism) to leverage NVLink bandwidth.

Common Mistakes

• Forgetting to install the NVLink bridge correctly. The bridge must be seated firmly and aligned with the connectors; a loose bridge causes no link.
• Not enabling Above 4G Decoding in BIOS. This can prevent the system from mapping large BARs, causing NVLink to fail.
• Using mismatched driver/CUDA versions. NVLink requires both to support it; older versions may not.
• Ignoring power requirements. V100s draw up to 300W each; insufficient PSU can cause instability.
• Assuming all V100s have NVLink. Some PCIe V100s (e.g., the 16GB version) may not have NVLink connectors; check your model.

Tips

• Use nvidia-smi nvlink -s to quickly verify link status after boot.
• For SXM2 systems, ensure the cooling is adequate; NVLink adds heat.
• In PyTorch, use torch.cuda.set_device and manually place tensors to control data flow.
• For large models, consider using NVIDIA’s Multi-Instance GPU (MIG) if you need to partition, but note MIG is not available on V100 (it’s for A100+).
• Keep the GPU drivers updated; NVIDIA frequently improves NVLink performance.

Troubleshooting

• NVLink shows “Inactive” or “N/A” in nvidia-smi. Re-seat the bridge, check BIOS settings (Above 4G Decoding), and ensure both GPUs are in the same NUMA node if applicable.
• System crashes when enabling P2P. This is often due to insufficient power or cooling. Check power cables and fan speeds.
• nvidia-smi doesn’t show both GPUs. Verify physical installation, reseat the GPUs, and check the PCIe slots (some slots are not x16).
• Performance is not improving. Ensure your inference workload actually uses both GPUs; if it’s a single model, you may need to split it (e.g., tensor parallelism). Also check if the NVLink bandwidth is the bottleneck by comparing with PCIe-only.
• CUDA errors about peer access. Run cudaDeviceCanAccessPeer() to check; if false, NVLink is not active, so re-check the link status.

Frequently Asked Questions

Do I need NVLink for multi-GPU inference?

No, but it helps. Without NVLink, GPUs communicate over PCIe, which is slower. For models that require frequent communication between GPUs (e.g., large transformers), NVLink can reduce latency and improve throughput.

Can I use NVLink with different V100 models?

Yes, as long as both have NVLink connectors and are the same architecture (Volta). Mixing SXM2 and PCIe versions is not recommended because they have different physical connectors.

How fast is NVLink on V100?

Each NVLink link on V100 provides 25 GB/s bidirectional bandwidth, and V100 has up to 6 links, totaling 150 GB/s per GPU. That’s about 10x faster than PCIe Gen3 x16.

Does NVLink work with CUDA 10?

Yes, CUDA 10 supports NVLink, but newer CUDA versions (11+) have better optimizations. It’s recommended to use CUDA 11 or later for best compatibility.

Can I use NVLink for training as well?

Yes, NVLink benefits both training and inference. For training, it speeds up gradient synchronization; for inference, it speeds up tensor parallelism.

Do I need to configure anything in the framework?

Most frameworks (PyTorch, TensorFlow) automatically use NVLink for peer-to-peer transfers when GPUs are visible. You may need to enable P2P in your code if using raw CUDA.

Related

• Reset Netflix on a Sanyo TV Without Full Reset
• Fix Microsoft Edge Crashing While Printing on Windows
• Fix No Bootable Device Error on Windows 10