Run a Private AI Assistant Locally: Step-by-Step

Running a private AI assistant on your own computer means you get chatbot-style help without sending your conversations to a cloud service. Everything stays on your hardware, which is useful for sensitive notes, code, or just avoiding subscription fees. This guide walks you through installing and running a capable open-source model using free tools, with no prior machine-learning experience required.

What You’ll Need

A computer with at least 8 GB of RAM (16 GB or more recommended).
Roughly 10–20 GB of free disk space for the model files.
An internet connection for the initial download (the assistant works offline afterward).
macOS, Windows, or Linux. The steps below work on all three with small adjustments noted.

Step-by-Step

Install Ollama. Go to ollama.com and download the installer for your operating system. Run it and follow the prompts. On macOS and Windows, it installs as a background service; on Linux, use the provided curl command. Verify it works by opening a terminal and typing ollama --version — you should see a version number.
Pick a model. Ollama gives you access to many open-source models. For most personal use, start with Llama 3.2 (3B) if you have 8 GB RAM, or Llama 3.1 (8B) if you have 16 GB+. You can also choose Mistral or Phi for different strengths. The model name is what you’ll type in the next step.
Download and run the model. In a terminal, type ollama run llama3.2 (or your chosen model name). This downloads the model (several GB) and then drops you into a chat prompt. Type a test question like “What is the capital of France?” and press Enter. If you get a sensible answer, the core assistant works.
Exit the chat. Type /bye or press Ctrl+D (Cmd+D on macOS) to leave the chat session. The model stays downloaded and ready for future use.
Install a chat interface (optional but recommended). The terminal chat works, but a web interface is easier to use. Open a second terminal and run ollama serve (if it isn’t already running). Then install Open WebUI by following its official instructions: typically pip install open-webui then open-webui serve. This starts a local web app at http://localhost:8080 where you can chat with your model in a nicer interface.
Set up the interface to use your model. In Open WebUI, create an account (local only), then go to Settings > Models and ensure your downloaded model (e.g., llama3.2) is selected. You can now chat through the browser.
Test offline. Disconnect your internet (or turn off Wi-Fi) and try chatting again. If the assistant still responds, everything is local. If it errors, check that Ollama is running and the model is loaded.

Common Mistakes

Choosing too large a model. A 70B model needs 32+ GB RAM and will run painfully slow or crash on typical laptops. Stick to 3B–8B models unless you have a high-end machine.
Not enough RAM. Ollama doesn’t warn you; it just slows down or fails. Close other heavy apps before starting.
Forgetting to run ollama serve. On some setups the service doesn’t auto-start. If the web interface can’t connect, start the server manually.
Using the wrong model name. Typos like llama3.2 vs llama3 cause errors. Check the exact name on the Ollama library page.

Tips

Use a smaller model (like Phi-3 mini) if you want faster responses on older hardware.
Keep the model files on an SSD if possible — loading is much faster than from a hard drive.
Use the ollama pull command separately if you want to download the model without entering chat.
Set up a system prompt (in Open WebUI) to make the assistant act as a specific helper, like a writing editor or coding buddy.
Update Ollama regularly with ollama update to get the latest performance improvements and model support.

Troubleshooting

“Error: pull model manifest” — Usually a network issue. Check your internet connection, try again, or use a VPN if you’re in a restricted region.
Model runs but answers are slow. Reduce the model size, close other programs, or check if your CPU is thermal throttling. On a Mac, ensure you’re using Apple Silicon (M1/M2/M3) for best performance.
Web interface won’t load. Make sure Ollama is running (ollama serve) and that you’re visiting the correct URL (usually localhost:8080). Try a different browser or incognito mode.
Out of memory error. Use a smaller model or add a swap file. On Windows, increase virtual memory; on macOS, the system manages it automatically.
Command not found. On Windows, you may need to restart your terminal or computer after installing. On Linux, ensure the install path is in your PATH.

Frequently Asked Questions

Q: Is this really private?
A: Yes, once the model is downloaded, all inference happens on your computer. No data leaves your machine as long as you don’t use cloud-based features.

Q: Can I use this on a laptop with 8 GB RAM?
A: Yes, but stick to 3B or 4B models. You’ll get decent responses, though not as fast as on bigger hardware.

Q: Do I need a GPU?
A: No, but it helps. Ollama uses your CPU if no GPU is present. A modern CPU can run small models at usable speeds.

Q: Can I run multiple models?
A: Yes, download as many as you want. Each takes disk space, but you can switch between them in the interface or by running ollama run with a different name.

Q: How do I update a model?
A: Run ollama pull <model-name> again — it will fetch the latest version if available.

Q: What if I want a model with more knowledge?
A: Try a larger model like Llama 3.1 70B, but only if you have 32+ GB RAM. Otherwise, consider a fine-tuned variant that fits your memory.

General steps for current mainstream versions; menus can differ by device and version. See the disclaimer or report an error.

More in this bed

The whole Tech How-To bed →