What is local AI?

When you use most AI tools, your words travel to a company’s computers, get processed there, and the answer comes back. Local AI skips that trip. The model (the AI system that writes the replies) sits on your machine and runs on your machine.

Skip straight to setting up a local model

What you get

Privacy that is structural, not promised. Nothing is sent anywhere, so there is no policy to trust and no breach to worry about. For anything sensitive (client work, personal notes, half-formed ideas), that is a genuine difference.

No bill. No subscription, no per-use charge, no anxiety about how much that long document cost to summarise.

It works offline. On a plane, on bad hotel wifi, or when a service is having an outage.

What you give up

Quality. A model small enough to run on a laptop is meaningfully less capable than the large ones. For rewriting, summarising and drafting, the gap is small. For careful reasoning and long documents, the gap is obvious. Most people end up using both: a local model for anything private, routine or offline, and a hosted one for the hard problems.

Speed, sometimes. On a recent Mac, responses feel immediate. On an older Windows laptop, they can be slow enough to break your concentration.

Convenience. Someone else’s model is always up to date. Yours is whatever you downloaded.

Can your computer do it?

The number that matters is memory, not processor speed.

• Under 8GB: technically possible, practically frustrating. Not recommended.
• 8–16GB: small models run fine. This covers most everyday text work.
• 16GB and up: comfortable, and you can run larger models that are noticeably better.

Apple Silicon Macs are unusually good at this because the memory is shared with the graphics chip. On Windows and Linux, a dedicated graphics card makes the biggest difference. You will also need disk space: each model is a few gigabytes.

The easiest way to get started

The setup guide walks through it by hand in about ten minutes. If you would rather not work out which model your machine can cope with, hand the whole job to an AI assistant that runs on your own computer and can see what is installed.

Not sure whether yours can reach your machine, or whether a browser tab counts? What an AI agent is explains the difference, then come back here.

Paste this, as it is. Nothing to fill in.

Please inspect this computer and tell me whether it can usefully run an AI model locally.
Check how much memory it has, how much free disk space there is, and whether there is a
graphics chip worth using. Check whether anything for running local models is already
installed before you install anything new.

First tell me the best arrangement for this machine: which local model runner suits it, and
roughly what size of model this hardware will actually be comfortable with. Say so plainly
if the honest answer is that this machine is not worth it.

Then, once you have explained the plan, set it up for me rather than handing me commands.
Install the runner, and download one small model to start with. Leave the larger ones until
I have decided the quality is good enough.

Make it stick somewhere sensible: an existing skill, a new skill, or the relevant AGENTS.md
or agent instructions file. Note which model is installed and when it makes sense to use the
local one rather than a hosted one. Read what is already there first, and do not overwrite
unrelated guidance or credentials.

Tell me how much disk space a download will take before you start it. Do not remove anything
already on this machine without asking. Test it by asking the model one short question and
showing me the reply.

When you are finished, tell me what you changed, where you wrote it down, how I start it
again tomorrow, and how I remove it all if I decide against it.

Know what “working” looks like. A successful setup has three signs:

• The model answers a question with no internet connection involved. Turning wifi off for a moment is a fair test, and a convincing one.
• The assistant tells you which model it installed and how big it is, so you know what is now sitting on your disk.
• It tells you where it wrote the setup down, so the next private or offline job uses the local model without you asking again.

If it reports that your machine is not up to it, believe it. That is a useful answer, not a failure. A model squeezed onto a computer without the memory for it is slow enough that nobody keeps using it, and the hosted tools are still there.