Ollama is a command-line tool that downloads and runs open-source large language models on your own hardware. No cloud account. No API keys. No per-token billing. You type a command, and a model like Llama 3.2 or Mistral starts answering questions right on your laptop.
What Ollama actually does
Ollama wraps several open-weight models into a single, consistent interface. It handles model downloads, quantization, and the server process that serves responses. The heavy lifting—the actual neural network inference—happens on your CPU or GPU. The tool itself is a thin layer of convenience.
Under the hood, Ollama uses llama.cpp and related projects to run models efficiently. It also quantizes models, which means it compresses their weights to fit in less memory. A model that needs 8 GB of RAM at full precision might run in 4 GB after quantization. That trade-off costs a little accuracy but makes local inference practical on ordinary machines.
Installing Ollama
Installation is straightforward on macOS and Linux. A single curl command does the job:
curl -fsSL https://ollama.com/install.sh | sh
Windows users download an installer from the Ollama website. After installation, verify everything works with ollama --version. The command should print a version number and return to the prompt.
Pulling your first model
Models are distributed through Ollama's library. You pull one with the pull command:
ollama pull llama3.2
This downloads the model weights, which can take a few minutes depending on your connection. The smallest useful models are around 1–3 GB. Larger ones, like a 70B-parameter model, can exceed 40 GB. Start small.
Running a model
Once pulled, you can chat with the model directly in your terminal:
ollama run llama3.2
You get an interactive prompt. Type a question, press Enter, and the model streams its answer token by token. Type /bye to exit. That's the whole workflow for basic use.
You can also pass a single prompt without entering interactive mode:
ollama run llama3.2 "Explain recursion in one sentence."
The built-in API
Ollama runs a local HTTP server on port 11434. That means any program on your machine can talk to the model over a standard REST endpoint. A minimal request looks like this:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Write a haiku about a cat."
}'
The response is JSON, and you can stream it or collect it whole. This API is what makes Ollama useful beyond the terminal. You can build a small Python script, a Discord bot, or a web app that calls the same endpoint. The model stays local, so your data never leaves the machine.
Choosing a model
The Ollama library lists dozens of models. Llama 3.2 and Mistral are solid general-purpose choices. CodeLlama and Qwen2.5-Coder handle programming tasks well. Phi-3 is a compact model that runs on modest hardware. If you have a GPU with 8 GB of VRAM, you can comfortably run 7B-parameter models. Without a GPU, a 3B model on a modern CPU still responds in a few seconds.
Experiment. Pull two or three models and compare their answers on the same prompt. They differ noticeably in tone, accuracy, and speed.
Hardware and privacy notes
Local inference is slower than cloud APIs. A 7B model on CPU might generate 10–20 tokens per second, which feels like reading text rather than receiving it. A GPU speeds that up several times over. Memory matters more than raw speed: the model must fit in RAM or VRAM, or the system will swap and crawl.
The privacy benefit is real. Your prompts, your documents, and your code never leave your computer. That matters for confidential work, for offline environments, and for anyone who simply prefers not to send their data to a third party.
Ollama is not a replacement for frontier models like GPT-4 or Claude. It is a practical way to run capable, open-weight models on hardware you already own. Start with a small model, test it against your own tasks, and scale up only if you need to.