Your Unsloth Fine-Tune Breaks in Ollama at 2048 Tokens

The adapter trained clean. Loss came down, the held-out prompts looked right, and inference inside the notebook did what you wanted. Then you exported to GGUF, ran ollama create, and the same prompt came back as gibberish, or as an answer that never stopped.

The weights didn't change in that step. Two pieces of configuration that don't live inside the GGUF file did: the chat template and the context window. Unsloth's own troubleshooting page names the template as the most common cause. The context window is easier to miss, because Ollama has a default and the default is small.

Table of contents

• What the GGUF file doesn't carry
• Chat template mismatch
• Default context window of 2048
• How much of the window survives
• VRAM cost of a larger context
• Reading the Modelfile
• Diagnosis order
• FAQ
• Sources

What the GGUF file doesn't carry

A GGUF file holds tensors and metadata. It doesn't decide how your conversation gets turned into one string of tokens before it reaches those tensors. That is the chat template, and in Ollama it lives in the Modelfile under TEMPLATE, next to the PARAMETER stop lines that tell the runtime when to quit.

Unsloth generates that Modelfile on the Ollama export path, which people miss when they hand-write one instead. Write your own FROM ./model.gguf with no TEMPLATE and no stop parameters, and Ollama falls back to whatever it can infer. Inference still runs, but the model sees a prompt formatted differently from every example it trained on.

Chat template mismatch

Unsloth's docs list the symptoms: poor results, gibberish, endless or infinite generations, repeated output. All four are what a model does when the tokens around your question aren't the ones it was tuned to expect.

Three things have to match between training and serving:

Nobody checks the BOS row. Unsloth says the engine may add an unnecessary start-of-sequence token or fail to add one, so test both cases instead of only the obvious one.

The fix is to use the same template you trained with. Unsloth recommends training through their conversational notebooks, which force a known template, and lists Qwen-3 14B, Gemma-3 4B, Llama-3.2 3B, Phi-4 14B and Mistral v0.3 7B versions of it.

Default context window of 2048

Ollama's Modelfile reference gives num_ctx a default of 2048 tokens. If your Modelfile doesn't set it, that is the window your model runs in, whatever it was trained at.

This produces no error text. The prompt is cut to fit, the model answers from the part that survived, and the answer looks plausible because it is a real answer to a shorter question. A system prompt plus a few examples plus a long document can pass 2,048 tokens without feeling long, and what falls off the front is usually the system prompt with your formatting rules.

So a model that "forgot its instructions after fine-tuning" often forgot nothing. It never received them.

How much of the window survives

Take a Modelfile with no num_ctx line and work out what survives of the trained window:

Fine-tune at 8,192 to handle real documents, serve on the default, and three quarters of the capability you paid GPU hours for is out of reach. For English prose, 2,048 tokens is roughly 1,500 words. The ratio shifts with the tokenizer, and shifts a lot on code or non-Latin scripts.

VRAM cost of a larger context

Setting PARAMETER num_ctx 8192 works, but the KV cache grows with the context length you allow. A model that loaded comfortably at 2,048 can start spilling to system RAM at 8,192. Generation then slows down instead of failing, which is harder to notice.

Estimate the quantised weights and the KV cache at your intended context in the AI VRAM Calculator. Training-time and serving-time memory are different problems with different answers. The longer version of that split is in Unsloth vs Ollama: Fine-Tuning vs Running Local Models, and the sequence-length effect on training memory is in Every QLoRA VRAM Table Leaves Out Sequence Length.

Set num_ctx to the largest prompt you actually send, not the largest your GPU tolerates.

Reading the Modelfile

Ollama prints the full Modelfile of any registered model, which is the fastest way to see what your export produced:

ollama show --modelfile your-model-name

Look for a TEMPLATE block, the stop parameters, and an explicit context size. A Llama-3 style entry looks like this in Ollama's own reference output:

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>"""
PARAMETER stop "<|start_header_id|>"
PARAMETER stop "<|end_header_id|>"
PARAMETER stop "<|eot_id|>"

The stop lines are the same special tokens the template uses. If ollama show --modelfile returns a FROM line, a temperature and nothing else, you've found the bug before running a single test prompt.

Adding the context window to your own file takes one line:

FROM ./custom-model.Q4_K_M.gguf

PARAMETER num_ctx 8192
PARAMETER temperature 0.4

Ollama wants the GGUF path absolute or relative to the Modelfile. Split models keep their original shard filenames and take a wildcard, FROM ./model-*.gguf, or one FROM per shard.

Diagnosis order

Go from the cheapest check to the most expensive:

1. Run ollama show --modelfile and look for TEMPLATE. A missing template explains gibberish on its own.
2. Check the stop parameters against the special tokens in the template. Missing stops explain generation that never ends.
3. Check for a num_ctx line. No line means 2,048, whatever you trained at.
4. Send a deliberately short prompt, well under 2,048 tokens. If the short prompt behaves and the long one doesn't, the window is the problem, not the template.
5. Only then question the training run.

Step four separates the two failure modes and takes about a minute. Most of the time you never reach step five.

To compare raw JSON between the notebook and the Ollama API response, the JSON Formatter makes the diff readable without pasting internal data into a web service that logs it.

FAQ

Why does my model work in Unsloth but produce gibberish in Ollama?

Almost always a chat template mismatch, which Unsloth names as the most common cause. The model was tuned on one prompt format and is served another, so the role markers and the EOS token no longer sit where it expects them. Check the TEMPLATE block in ollama show --modelfile first.

What is the default context length in Ollama?

2,048 tokens. Ollama's Modelfile reference lists that as the default for num_ctx. It applies whenever your Modelfile doesn't set the parameter, regardless of the sequence length the model was trained on.

Why does my fine-tuned model ignore its system prompt?

If the full prompt exceeds the served context window, it gets cut to fit, and the system prompt at the front is what disappears. There is no error. Test with a short prompt before assuming the training failed.

Why does my model generate endlessly?

Either the EOS token is wrong or the PARAMETER stop entries are missing. Unsloth ties gibberish on longer generations to an incorrect EOS token. The stop parameters in the Modelfile should match the special tokens used in the template.

Sources

• Ollama documentation: Modelfile reference
• Unsloth documentation: Saving models to Ollama
• Unsloth documentation: Errors and troubleshooting