All articles

Local AI for Businesses: Hardware, Costs and Limits

What local AI can do in an office, which graphics card or server it needs, what running it costs, and where home-built Ollama setups fall short.

Local AI8 min read

Local AI means the language model runs on a machine you own, sitting in your office or your own data centre. The question an employee types never leaves the company network, and the answer is computed on the same machine. There is no OpenAI or Google account involved and no charge per request.

For firms that hold confidential files, that is the main reason to consider it. A fiduciary office, a law firm or a medical practice cannot simply send client or patient data to a US provider. With local AI, that question does not arise.

What local AI does with your own documents

A language model on its own knows nothing about your firm. It has never seen your engagement letters, your internal guidelines or last March's file note. Local AI becomes useful once it can search your files.

It works like this: a program reads your documents (PDF, Word, email, scanned paper with text recognition) and builds a search index. When someone asks a question, the system first finds the relevant passages and hands only those to the language model. The model writes an answer from them and cites the source. The technical term is retrieval-augmented generation, or RAG.

Typical everyday questions:

  • "What notice period is in our engagement letter with Keller AG?"
  • "How have we treated private use of company cars over the last few years?"
  • "Do we have an internal template for requesting a deadline extension?"
  • "Summarise the correspondence with the tax office in this file."

Each answer points to the document and page, so whoever uses it can check it in seconds.

The hardware local AI needs

The figure that matters most is graphics memory (VRAM), or unified memory on Apple machines. The model has to fit into it completely, otherwise it slows to a crawl.

Open-weight models are described by their number of parameters (7B, 14B, 32B and so on). In office use they usually run compressed (quantised). At the common 4-bit setting, they need roughly this much memory, according to an Ollama hardware guide:

Model sizeMemory needed (4-bit, approx.)Fits on
7–8B5–6 GBGraphics card with 8–12 GB
14B9–10 GBGraphics card with 16 GB
32Babout 20 GBGraphics card with 24–32 GB
70Bover 40 GBWorkstation card or a Mac with lots of memory

On top of that comes memory for context: the document excerpts and the conversation so far. With several people using it at once, that share grows, so leave headroom.

For questions about your own documents, models in the 14B to 32B range are often enough, because the facts come from your files and the model only has to summarise them. With long contracts, mixed languages (German, French, English) or demanding summaries, larger models do noticeably better.

Three realistic hardware options

A workstation with one graphics card. A PC with an NVIDIA card carrying 24 to 32 GB of VRAM can run a 32B model. The RTX 5090 with 32 GB launched at USD 1,999, but in 2026 it sells well above that. For a small team, this is a sensible starting point.

A Mac Studio. Apple machines share memory between processor and graphics. A Mac Studio with M3 Ultra and 96 GB is available in Switzerland from around CHF 3,950 and can load larger models. It is quiet and frugal with power, but answers more slowly than a strong NVIDIA card once many people use it at the same time.

A server with a professional graphics card. For 20 to 50 users or for 70B models, cards such as the NVIDIA RTX PRO 6000 with 96 GB come into play. It launched at around USD 8,500; by mid-2026 Nvidia's own store listed it at USD 13,250. Add the server chassis, storage and installation.

Graphics card prices moved a lot in 2026 because demand for AI hardware is high. Get current quotes before you set a budget.

Running costs

With local AI you pay for the hardware once. After that you pay for:

  • electricity (depending on load and the card)
  • maintenance: operating system, drivers, model updates, security patches
  • time from your IT person or IT partner
  • replacing the hardware after a few years

There is no bill per request. Whether that works out cheaper than a cloud subscription depends on how many people use it and how often. For a worked example, see The true cost of AI: cloud tokens vs. on-premise.

Local, hybrid and cloud compared

LocalHybridCloud (e.g. ChatGPT Enterprise)
Where are the documents?Your serverYour serverWith the provider
Where does the model run?Your serverCloud providerCloud provider
What leaves the building?NothingThe question and a few short excerptsThe question and any uploaded files
Graphics card needed?YesNoNo
Cost modelHardware once, plus operationServer plus cost per requestSubscription per user
Works without internet?YesNoNo

Hybrid is a reasonable middle ground for many firms. What matters is how it is built. Done well, documents, search index and logs stay on your server, and only the question plus three to five short excerpts go to the cloud model. Done badly, whole files get uploaded. More on this in Not all hybrid RAG is equal.

Compared with ChatGPT, a local model is weaker at general knowledge and free writing. For questions about your own documents the gap is smaller, because the content comes from your files.

Where home-built Ollama setups go wrong in an office

Ollama is a popular free program for running open-weight models on your own computer. For a first test on your laptop it is excellent. In an office with several people, the gaps show quickly.

No user accounts. Ollama has no built-in login. Anyone who can reach its address on the network can use the model. In 2025 Cisco Talos found more than 1,100 Ollama servers reachable from the open internet, around a fifth of them serving models without any authorisation.

Document permissions. Once you connect documents, the AI sees everything that was indexed. The apprentice at reception might then get excerpts from payroll or from client files she has no right to see. Your file system permissions have to be checked for every answer. Simple DIY setups do not do that.

Many users at once. A model that answers smoothly for one person becomes sluggish with ten requests in parallel. Without a queue and suitable hardware, requests time out.

Updates. New model versions, security fixes in Ollama itself, driver updates: someone has to keep track. Often that is the person who set the system up one weekend and has since left.

No logs. Who asked what, and when? For professionals bound by confidentiality, and for evidence towards clients, that can matter.

New documents. Your files change every day. The search index has to keep up, or the AI answers from an outdated state.

Checklist before you buy

  • Which repositories should the AI search (file server, SharePoint, DMS, email)?
  • How many people will use it at the same time, and how many in total?
  • Do permissions need to be enforced per file or per client?
  • Which languages are your documents in?
  • May any text leave the building (hybrid is an option), or none at all (local)?
  • Who looks after updates, backups and outages?
  • Do you need a log of queries?
  • What is the budget for hardware, and for ongoing operation?

Local AI with KADARAG

KADARAG is an AI assistant for your document store that runs on your own server. It covers the parts where home-built setups usually fail: login, permissions taken from your existing file rights, continuous indexing of new documents, answers with page-level citations, and logs. You can run it fully local without internet, or in hybrid mode without a graphics card, where only the question and short excerpts go to a cloud model.

Next step

Take the checklist above and work through it with your IT person. That narrows the hardware down to one option. If you would like to go through it with us, book an intro call.

See KADARAG answer your own questions

30 minutes, no commitment. We show you how KADARAG works with documents like yours and whether local or hybrid fits.