How to fine-tune a private AI model on your own documents
8 min read · Updated October 2026
Fine-tuning a small open-weight model on your own data gives you an assistant that writes the way your team writes and knows your vocabulary. This guide covers when fine-tuning helps, how it works alongside retrieval, and how to do it without sending your documents to a public AI provider.
What fine-tuning changes
Fine-tuning adjusts a model's weights using examples from your own work. It is best at teaching tone, format and vocabulary: how your team phrases a reply, which terms you use, how long an answer should be.
It is not the best way to teach facts that change. A price list or a policy updated last week belongs in retrieval, where the model looks it up at question time.
Fine-tuning and retrieval work together
Retrieval-augmented generation (RAG) indexes your documents and passes the most relevant passages to the model with each question. That keeps answers current and lets the model cite its sources.
A fine-tuned model with retrieval gets both: answers in your voice, grounded in your latest documents.
Choosing a base model
Small open-weight models between 3B and 14B parameters are enough for most company knowledge work, and they answer faster and cost less to run than large general models.
Pick by task. General assistants do well on 7B to 9B models; heavy reasoning or long documents benefit from 14B.
Preparing your data
Remove duplicates, outdated drafts and anything you would not want quoted back. Flag personal details such as email addresses, phone numbers and ID numbers, and decide whether each should be redacted, kept or excluded.
Add a small set of example answers that sound right. These shape tone more than thousands of raw documents.
Keeping the model current
Documents change, so schedule retraining weekly or daily, and keep new material in retrieval between runs.
Ratings help too. When people mark answers as good or not quite, those ratings can steer the next training run toward what your team wants.
Doing it without a public AI provider
Running this yourself means GPUs, pipelines and security reviews. Monotonic does it in an isolated, encrypted environment: data is cleaned and trained on in isolation, and never sent to a third-party model provider.