Local AI Inference on AMD Ryzen AI Max

This guide walks you through setting up local AI inference on a PC powered by an AMD Ryzen AI Max processor. You’ll end up with a working system that can run large language models and other AI workloads entirely on your own hardware, without sending data to the cloud. We’ll cover the necessary software, model selection, and how to get the best performance from the integrated NPU and GPU.

What You’ll Need

A PC with an AMD Ryzen AI Max 300-series processor (e.g., Ryzen AI Max 395 or 390).
At least 32 GB of system memory (64 GB or more is recommended for larger models).
Windows 11 (version 24H2 or later) or a recent Linux distribution (Ubuntu 24.04 or similar).
An internet connection for downloading software and models.
Sufficient free storage space (models can range from 1 GB to over 50 GB).

Step-by-Step

Update your drivers and BIOS. Visit your motherboard or PC manufacturer’s support page and install the latest AMD chipset drivers and BIOS. This ensures the NPU and integrated GPU are fully supported.
Install Python (if not already present). Download Python 3.11 or 3.12 from python.org. During installation, check the box “Add Python to PATH.”
Install the AMD ROCm software stack. For Windows, download and install the AMD ROCm for Windows installer from AMD’s official site. For Linux, follow the ROCm installation instructions for your distribution. This provides the low-level libraries needed for GPU and NPU acceleration.
Set up a virtual environment (recommended). Open a terminal (Command Prompt or PowerShell on Windows) and run: python -m venv ai-env, then activate it with ai-env\Scripts\activate (Windows) or source ai-env/bin/activate (Linux).
Install the AI inference framework. The easiest option is to use Ollama (for LLMs) or LM Studio (for a graphical interface). Alternatively, install PyTorch with ROCm support: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2. For NPU-specific acceleration, install the AMD NPU driver and the ONNX Runtime with the ROCm execution provider.
Download a model. With Ollama, run ollama run llama3.2 to pull and run a small model. For larger models, use ollama pull qwen2.5:14b or similar. In LM Studio, browse the model library and download a quantized model (e.g., a 7B or 13B parameter model in GGUF format).
Configure the runtime to use the GPU. In Ollama, set the environment variable OLLAMA_GPU_LAYERS=100 to offload as many layers as possible to the GPU. In LM Studio, go to Settings > Hardware and select the AMD GPU as the compute device.
Run your first inference. In Ollama, type ollama run llama3.2 and ask a question. In LM Studio, load the model and start a chat. You should see output generated locally.
Verify NPU usage (optional). Use the Task Manager (Windows) or the ROCm system profiler to confirm that the NPU is being utilized. Some frameworks like ONNX Runtime can target the NPU directly with the right execution provider.

Common Mistakes

Not updating drivers. Old drivers can cause the GPU or NPU to be unrecognized, leading to CPU-only inference (very slow).
Using an incompatible Python version. Some AI libraries are not yet compatible with Python 3.13. Stick with 3.11 or 3.12.
Choosing a model that is too large. A 70B parameter model requires around 40 GB of VRAM. If your system has 32 GB of shared memory, it will run painfully slowly or not at all. Start with a 7B or 13B model.
Forgetting to activate the virtual environment. If you get “command not found” errors, you likely skipped the activation step.
Assuming the NPU is automatically used. Many LLM frameworks default to the GPU. You often need to explicitly enable the NPU with a specific execution provider.

Tips

Use quantized models. Models like Llama 3.2 7B Q4_K_M are much smaller and faster than the full-precision version, with only a slight drop in quality.
Monitor memory usage. The Ryzen AI Max uses unified memory, so the GPU and CPU share the same RAM. Keep an eye on total memory usage to avoid system slowdowns.
Experiment with context length. Longer context windows consume more memory. If you run out of memory, reduce the context length in the model settings.
Use the AMD NPU for specific tasks. The NPU is excellent for vision models and lightweight inference. For heavy LLM generation, the GPU is usually faster.
Keep your software updated. AMD frequently releases new ROCm versions that improve performance and add support for more models.

Troubleshooting

Model runs but is extremely slow. Check if the GPU is being used. Run ollama ps to see the compute device. If it says “CPU,” set OLLAMA_GPU_LAYERS to a high number. In LM Studio, verify the GPU is selected in the hardware settings.
Error: “No ROCm-capable device found.” This usually means the ROCm drivers are not installed correctly. Reinstall the AMD ROCm software and ensure your GPU is listed in the ROCm support matrix.
Out of memory errors. Reduce the model size (e.g., from 13B to 7B) or lower the context length. Close other applications that consume RAM.
NPU not showing up in Task Manager. Update your chipset drivers and BIOS. The NPU may be disabled in the BIOS; check for an option like “NPU Enable” and turn it on.
Python import errors when using PyTorch. Make sure you installed the ROCm version of PyTorch, not the standard CUDA build. Uninstall and reinstall with the correct index URL.
LM Studio shows “No compute device available.” Go to Settings > Hardware and manually select the AMD GPU. If it’s not listed, reinstall the ROCm runtime and restart LM Studio.

Frequently Asked Questions

Do I need a dedicated GPU?

No. The Ryzen AI Max’s integrated GPU is powerful enough for most local AI inference tasks. The NPU adds extra efficiency for certain workloads.

What is the fastest way to run LLMs on this hardware?

Use the GPU with a quantized model (e.g., Q4_K_M) and keep the context length moderate. The NPU is not always faster for LLM generation.

Can I use the NPU for Stable Diffusion?

Yes, but support is still evolving. You may need to use ONNX Runtime with the NPU execution provider and a specific model export. The GPU is generally more reliable for image generation.

How much RAM do I need?

For 7B models, 16 GB is enough. For 13B models, 32 GB is recommended. For 30B+ models, 64 GB or more is advised.

Will this work on Linux?

Yes, but you may need to use a recent kernel and install ROCm manually. Windows is generally more plug-and-play for this processor.

Can I run multiple models at once?

Yes, but they share the same memory. Running two 7B models simultaneously may use over 20 GB of RAM, so be mindful of your total memory.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →