In this guide 6 sections
Use a hosted AI service when your documents can lawfully leave your network under a signed data processing agreement. Run a model yourself only for the tier that cannot. Most firms end up with both: hosted for ordinary files, self hosted for contracts, payroll and patient or client records.
Sort documents into sensitivity tiers first
The choice is rarely about the whole company. It is about each folder. Brochures, public tender notices and training manuals sit at one end. Employment contracts, bank statements, medical files and unreleased price lists sit at the other.
Ask the compliance owner or the person who signs customer contracts to label three tiers. Open, internal and restricted is enough. Each tier then gets its own answer, and the argument about hosted versus self hosted stops being all or nothing.
- Open: published material, safe on any hosted service
- Internal: SOPs, meeting notes, supplier quotes
- Restricted: payroll, health data, legal files, source code
- Anything a customer contract forbids sharing with a subprocessor
What a hosted API actually asks you to trust
A hosted model receives your prompt and the document text over the internet, inside the vendor’s cloud region. You are trusting the data processing agreement, the subprocessor list and the vendor’s retention settings. Read the clause on training use before anything else.
Data residency matters when a regulator or a customer says records must stay in one country. Many vendors let you pin a region, and some offer zero retention for API traffic. Get both in writing, not from a sales call.
What running your own model costs you
A self hosted model keeps text on servers you control, in your own cloud tenancy or a rack in the office. Nothing leaves. That comfort has a bill: a GPU server, drivers, a serving layer such as vLLM or Ollama, patching, monitoring and someone on call when it stalls at month-end.
Open weight models have also been behind the best hosted ones on hard reasoning and long contracts. For summaries, search over manuals and form extraction the gap is small. For tricky clause comparison it can show.
Budget for upkeep, not just the purchase. GPUs age, models get replaced every few months and your retrieval index needs rebuilding each time.
- GPU hardware or reserved cloud instances
- A named engineer who owns updates and outages
- Evaluation set to compare each new model release
- Backups of the vector index and prompt templates
Questions to put to any vendor in writing
Prompt logging is the quiet risk. Ask whether prompts and answers are stored, for how long, who at the vendor can read them and whether abuse monitoring can be switched off for your account. Ask the same of any app layer sitting between you and the model.
Then ask about the plumbing. Where is the key stored? Can you rotate it? Is there an audit log of which employee sent what? A short reply is a warning sign.
- Is customer data ever used to train or tune models?
- Which region processes and stores the request?
- How long are prompts and outputs retained?
- Which subprocessors touch the traffic?
- What is the breach notice period in the agreement?
A split setup most firms can run
Route by tier. Open and internal material goes to the hosted service through a small gateway that strips names and account numbers before sending. Restricted material goes to a local model behind your firewall, reached through the same chat screen so staff do not choose.
Start with one pilot folder and twenty real questions. Score hosted and local answers side by side, then show the results to whoever signs off on risk.
AI hosting decision sheet
A worksheet with three columns for open, internal and restricted documents. Rows list the folder, owner, law or contract limiting sharing, region required, retention allowed and chosen option. A second block holds eight vendor questions with a reply, a date and a pass or fail box.
Open a blank worksheet to printQuestions owners ask
Is a self hosted model always safer than a hosted one?
No. A neglected server with open ports and no patching can leak more than a well run vendor. Safety depends on access control, logging and updates, wherever the model runs.
Do we need a data processing agreement for a hosted API?
If the documents contain personal data, most privacy laws expect one. It should name the subprocessors, the region and the retention rules. Ask your counsel which law applies to you.
Can staff paste documents into a public chatbot instead?
Consumer chat tools usually sit under different terms from business APIs, often with weaker retention promises. Block them for restricted files and give staff an approved screen instead.
How much worse is a local model on answer quality?
It varies by task. Lookup and extraction are close, while long multi-document reasoning lags. Test with your own twenty questions rather than trusting a leaderboard.