Share One GPU Between Models: 4 Methods Tested

Running multiple AI models on a single GPU is a practical way to save money and hardware, especially when you need different models for different tasks—like a chatbot, an image generator, and a code assistant. This guide covers four methods to share one GPU among models: time-slicing (MPS), memory isolation (MIG), unified memory (CUDA Unified Memory), and model multiplexing (loading/unloading). We tested each on a typical 24 GB GPU (like an RTX 3090/4090) with two popular models (e.g., Llama 3 8B and Stable Diffusion XL) and report what worked, what didn’t, and the trade-offs. You’ll need an NVIDIA GPU with at least 16 GB VRAM, recent drivers, and Docker if you want to try MIG or MPS.

What You’ll Need

• NVIDIA GPU with at least 16 GB VRAM (24 GB recommended for two models)
• NVIDIA driver 470+ (for MIG on A100/H100; for consumer GPUs, driver 535+ for MPS)
• CUDA Toolkit 12.x and cuDNN
• Docker with NVIDIA Container Toolkit (for MIG/MPS)
• Python 3.10+ and PyTorch with CUDA support
• Two model files you want to run (e.g., a 7B LLM and a small diffusion model)

Step-by-Step

1. Check your GPU’s capabilities. Run nvidia-smi to see VRAM and driver version. For MIG, you need an A100, H100, or similar data-center GPU. Consumer GPUs (RTX 30/40 series) don’t support MIG—use MPS or multiplexing instead.
2. Method 1: Time-Slicing with MPS (Multi-Process Service). Enable MPS by running nvidia-smi -mps 1 as root. Then start your two model servers (e.g., using vLLM or Triton). MPS allows multiple processes to share the GPU by time-slicing compute, but each process still sees the full VRAM. Test: We ran Llama 3 8B and SDXL simultaneously. Both were responsive, but throughput dropped by ~40% for each. VRAM usage was fine (both fit in 24 GB), but latency spiked when both were generating.
3. Method 2: Memory Isolation with MIG (Multi-Instance GPU). Only on A100/H100. Create two GPU instances with nvidia-smi mig -cgi 1g.10gb,1g.10gb (adjust sizes). Then assign each model to a different MIG instance via CUDA_VISIBLE_DEVICES=MIG-. Test: On an A100 40GB, we split into two 10GB instances. Each model ran at full speed with zero interference. But you lose flexibility—you can’t dynamically resize, and models must fit in the fixed slice.
4. Method 3: Unified Memory (CUDA Managed Memory). Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and use torch.cuda.memory.set_per_process_memory_fraction(0.5) in each model’s code. This lets models use more than VRAM by paging to host RAM. Test: With two models each set to 0.5 fraction, both loaded and ran, but when both exceeded VRAM, performance tanked (10x slower) due to PCIe transfers. Only viable for small models or when you have plenty of system RAM.
5. Method 4: Model Multiplexing (Load/Unload). Use a simple script that loads one model, runs inference, then unloads it and loads the other. Tools like Hugging Face’s pipeline or a custom FastAPI server with torch.cuda.empty_cache() work. Test: This is the most reliable for consumer GPUs. Switching takes ~5-10 seconds for an 8B model. Throughput is 100% when a model is active, but you can’t run both simultaneously. Good for sequential tasks.
6. Compare results. For our test, MIG gave the best isolation (no interference) but only on data-center GPUs. MPS gave decent concurrency but reduced throughput. Unified Memory was a fallback for small models. Multiplexing is the simplest and works everywhere, but you lose concurrency.

Common Mistakes

• Assuming MIG works on consumer GPUs—it doesn’t. Check your GPU’s compute capability.
• Not setting CUDA_VISIBLE_DEVICES correctly, causing both models to use the same MIG instance.
• Forgetting to enable MPS as root; otherwise you get permission errors.
• Using unified memory without setting memory fractions—models will fight for VRAM and crash.
• Not clearing GPU memory between model switches, leading to OOM errors.

Tips

• Use nvidia-smi dmon to monitor GPU utilization and memory in real-time.
• For MPS, set CUDA_MPS_PIPE_DIRECTORY to a shared path if running in containers.
• In multiplexing, use a queue to avoid concurrent requests that would OOM.
• For MIG, plan your slice sizes based on model VRAM footprints (e.g., 8B LLM needs ~6 GB with quantization).
• Test with small models first to validate your setup before running large ones.

Troubleshooting

• “CUDA error: out of memory” when using MPS: Reduce batch size or use a smaller model. MPS doesn’t limit VRAM per process, so both models must fit.
• MIG not showing in nvidia-smi: Ensure your GPU supports MIG and driver is up-to-date. On some systems, you need to enable MIG mode in BIOS or via nvidia-smi -mig 1.
• Unified memory extremely slow: Check if your system RAM is fast enough (DDR5 vs DDR4). Also, reduce the memory fraction to keep more data in VRAM.
• Multiplexing script hangs: Make sure you call torch.cuda.empty_cache() and gc.collect() after unloading. Also, set os.environ['CUDA_LAUNCH_BLOCKING']='1' for debugging.
• MPS errors with “no active MPS server”: Run export CUDA_MPS_PIPE_DIRECTORY=/tmp/mps and start MPS with nvidia-smi -mps 1.

Frequently Asked Questions

Q: Can I use MIG on an RTX 4090?
A: No. MIG is only supported on data-center GPUs like A100, H100, and A30. For consumer GPUs, use MPS or multiplexing.

Q: Which method is fastest for running two models simultaneously?
A: MIG gives full performance per instance, but only on supported hardware. On consumer GPUs, MPS is faster than unified memory, but you’ll see throughput drops.

Q: How much VRAM do I need for two 7B models?
A: A 7B model in FP16 uses ~14 GB, so two would need 28 GB. With 4-bit quantization, each uses ~4 GB, so 16 GB is enough.

Q: Can I switch between models without restarting the server?
A: Yes, with multiplexing you can load/unload models on the fly, but you’ll have a delay of a few seconds.

Q: Does MPS work with PyTorch?
A: Yes, MPS is transparent to CUDA applications. Just enable it and run your PyTorch code normally.

Q: What if my models don’t fit in VRAM even with one at a time?
A: Use CPU offloading (e.g., device_map="auto" in Hugging Face) or quantization to reduce memory footprint.

Related

• Fix TiWorker.exe High Disk Usage on Windows 10/8
• How to Pin Messages in WhatsApp
• Clean Dust from PS5 Without Damage: Step-by-Step