All articles

AI Without Internet: When Offline AI Makes Sense

Offline AI runs entirely on your own server with no outside connection. When you need it, what works offline today, and where the limits are.

Local AI7 min read

Most AI tools only work with an internet connection. Your question is sent to a data centre in the US or Europe, processed there, and the answer comes back. For many tasks that is fine. For some offices and departments it is not an option. This article covers when AI without internet makes sense, what works offline today, and what you should expect.

What "offline" means for AI

With offline AI, every component runs on hardware you control: the language model, the search across your documents, the database of processed text and the logs. While it is in use, the system makes no outside connection.

In practice there are two levels:

  • Local with maintenance access. The server sits in your office or a Swiss data centre. Questions and documents never leave your network, but the server may open a controlled connection from time to time for software updates.
  • Fully isolated (air-gapped). The network has no internet connection at all. Updates and new models arrive on a storage device that is checked beforehand.

For most law firms and fiduciary offices, the first level is enough. The second is more common in government, defence supply chains and critical infrastructure.

When running offline makes sense

When professional secrecy applies

Lawyers, notaries, auditors and their auxiliaries are bound by professional secrecy under Art. 321 of the Swiss Criminal Code. Fiduciaries and tax advisers are often subject to the professional duty of confidentiality under Art. 62 of the Data Protection Act. Whether, and on what terms, cloud services are compatible with these duties has not been conclusively settled. If nothing leaves the network, the question does not arise.

When the network is already isolated

Some organisations deliberately separate their internal network from the internet, for example for engineering data, security documentation or patient records. A cloud tool cannot be used there. An AI that runs offline can.

When your clients require it

Some clients, such as banks, insurers or public bodies, set out in their contracts where data may be processed. Read the clauses on data sharing and subcontractors in your engagement letters. If they say no data goes to third parties, a cloud model is effectively ruled out.

When you want to rule out foreign access

Providers based in the US can be compelled under the CLOUD Act to hand data to US authorities, even when the servers are in Europe. With an offline system, there is no such provider in the chain.

What is realistic offline today

A few years ago, open language models were clearly weaker than the cloud offerings. That has changed. One example: in August 2025 OpenAI released the open models gpt-oss-120b and gpt-oss-20b under the Apache 2.0 licence. According to the model card, the larger one runs on a single GPU with 80 GB of memory and the smaller one within 16 GB. There are also open models from other providers, including Mistral, Google and Alibaba.

For working with your own documents, that means:

TaskFeasible offline?
Finding a passage in a document ("Where is the notice period in the Huber engagement letter?")Yes, well
Answering with a source reference (document and page)Yes, well
Summarising a tax assessment or a court decisionYes, well
Asking in English about documents written in German, French or ItalianYes, with a multilingual model
Drafting a letter from an existing templateYes
Long, multi-step analysis across many documentsPossible, but slower and needs more hardware
Questions about events or legal changes not covered in your documentsNo

Most everyday questions in an office belong in the upper rows. A fiduciary preparing a VAT return at year-end who wants to know how a client booked a particular item last year needs the right document. General world knowledge does little here.

The limits of offline AI

An open model on your own hardware does not match the largest cloud models in every respect. On complex, multi-step tasks, the cloud models are usually ahead. For answering questions from your documents, the gap matters much less, because most of the answer comes from the passages the system has found.

Other points to plan for:

  • Hardware. A server with a suitable graphics card is an up-front investment. For small offices, it is often more expensive on cost alone than a cloud model, as our article on AI token costs shows.
  • Maintenance. Someone has to install updates, monitor the hardware and test backups. On an isolated network, this is done by hand.
  • No current knowledge from outside. The system knows only your documents and what the model learned in training. New legislation has to be filed as a document before the AI can take it into account.
  • Scanned paperwork. The AI cannot read scanned letters without text recognition, so text recognition has to run locally as well.

What to settle before you start

Before buying an offline AI system, it is worth working through a few practical questions with your IT partner:

  1. Where will the server stand, and who has physical access?
  2. How will updates and new models reach the system?
  3. Will the AI's access rights follow your existing folder permissions, so that not everyone sees every file?
  4. Where are questions and answers logged, and for how long?
  5. Who responds if the system goes down?

If you conclude that full offline operation is too much for your office, there is a middle path. What exactly goes over the internet in that case is explained in Hybrid AI: what exactly leaves your network?

KADARAG in offline mode

In local mode, KADARAG runs entirely on your server, without an internet connection if you wish. Search, database, language model and logs stay in-house. Every answer cites the document and page, and access rights follow your existing file permissions. If you start with hybrid, you can switch to local mode later.

Your next step

Take two or three of your most important engagement letters and check the clauses on data sharing. Also ask your IT partner whether they can look after a server with a graphics card. With those two answers, we can assess in an introductory call whether offline operation suits you.

See KADARAG answer your own questions

30 minutes, no commitment. We show you how KADARAG works with documents like yours and whether local or hybrid fits.