Saluki 27B: 7.89 GB File, 9.5 GiB at 32K

Underdog released Saluki 27B on 7 October 2026. It's a 2-bit GGUF of Qwen3.8-27B, and the file is 7.89 GB, against about 54 GB for the full model. It runs in stock llama.cpp, and the pitch is that it keeps tool calling intact: 88 of 120 on Underdog's tool-calling test, where the full-size model scores 84.

The file size reads like "fits an 8 GB card". It doesn't, once you give it a context window an agent can use. This post works out what Saluki actually needs in memory at 8K, 32K, 64K and 128K, lists what the 2-bit squeeze costs and what it keeps, and gives you a llama-server setup for tool calls you can copy.

Table of Contents

• What Saluki is
• 7.89 GB is not 7.89 GiB
• The KV cache at agent context lengths
• Which card it actually fits
• What 2-bit keeps and what it loses
• Saluki vs the other 2-bit builds
• A tool-calling setup you can copy
• Three catches before you download
• FAQ
• Conclusion
• Sources

What Saluki is

Saluki is not a new model. Underdog took ISTA-DASLab's Qwen3.8-27B-GSQ-RCO-IQ2_XS file, which assigns a separate quantization type to every tensor, and modified it to hold up better on tool use. The NOTICE file in the repo says so. The license stays Apache 2.0, the same as Qwen3.8-27B.

Underdog calls it a preview checkpoint and says the full training write-up comes later.

7.89 GB is not 7.89 GiB

The 7.89 GB figure is decimal gigabytes: 7,898,369,152 bytes. GPU memory is counted in binary units, and an "8 GB" card holds 8 GiB. In those units the file is 7.36 GiB.

That leaves about 0.64 GiB on an 8 GiB card before anything else loads. llama.cpp also needs compute buffers and the CUDA context, and those don't fit in 0.64 GiB alongside a useful KV cache. You can run Saluki on an 8 GB card with some layers offloaded to system RAM, but you lose a lot of speed.

The KV cache at agent context lengths

Qwen3.8-27B uses a hybrid stack. Of its 64 layers, 48 are Gated DeltaNet (linear attention) layers that keep a fixed-size state, and only 16 are full-attention layers with a KV cache that grows per token. The config.json gives 4 KV heads and a head dimension of 256 for those layers.

So the cache per token at FP16 is:

2 (key and value) x 16 layers x 4 KV heads x 256 x 2 bytes = 65,536 bytes, or 64 KiB per token.

The DeltaNet state adds a fixed 144 MiB: 48 layers x 48 value heads x 128 x 128 x 4 bytes in FP32. It doesn't grow with context.

Totals are weights plus KV cache plus the DeltaNet state, before compute buffers. The vision add-on adds 0.86 GiB at F16 or 0.59 GiB at Q8_0 when you load it.

The hybrid stack is why this is manageable. If all 64 layers kept a KV cache, 32K of context would cost 8 GiB on its own, more than the weights.

Which card it actually fits

Underdog's README runs Saluki with -c 32768. At that setting:

• 8 GB card (RTX 4060, 3070): no room for full GPU offload. Use partial offload, or drop to about 4K of context with a q8_0 cache.
• 12 GB card (RTX 3060 12 GB, 4070): 32K with FP16 cache (9.50 GiB) fits with headroom for buffers. 64K fits with a q8_0 cache (9.62 GiB). This is the card Saluki suits best.
• 16 GB card or a 16 GB Apple Silicon Mac: 64K at FP16 cache, or 128K with q8_0 (11.75 GiB). Underdog's "runs on a 16 GB laptop" line holds at 32K. On a Mac, macOS keeps part of that unified memory, so expect less than the full 16 GB.
• 24 GB card: you don't need 2-bit. Q4_K_M of the full model is 16.24 GiB and leaves room for a 32K cache.

You can check these figures in the AI VRAM Calculator. Pick the Qwen3.8 27B preset, which already counts only the 16 attention layers. The calculator's lowest weight precision is INT4, so for a 2-bit file, read the KV cache line from the calculator and add the 7.36 GiB file size yourself. For the general method behind these numbers, see the local LLM VRAM guide.

What 2-bit keeps and what it loses

Underdog ran Saluki and the full-size model in the same harness for the agent tests. For the other tests, Underdog compared its own run against the public numbers for the full model, which came from a different harness.

The pattern is clear. Picking the right tool and following instructions survive the 2-bit squeeze, and on this test set they come out slightly above the full model. Multi-step reasoning doesn't hold up as well: AIME 2025 keeps 82% and MuSR 85%. Coding sits in between, at 30 of 50 issues against 33. 27 of Saluki's fixes were issues the full model also fixed.

Underdog itself says that 120 tasks is a modest test and that a few tasks of difference is run-to-run variation. Read "88 vs 84" as "about the same as the full model on tool selection". It doesn't mean Saluki is smarter.

Saluki vs the other 2-bit builds

Underdog tested other small builds of the same model on the same 120 tasks, each in its own runtime:

So the choice depends on the agent you're building:

• Many tools, one call per step (a coding agent, a file assistant): Saluki. On the two tool-selection categories it got 47 of 48, against Mia's 31.
• Several calls in one reply ("check the weather in three cities"): Mia leads clearly, 69 against 42. But Mia is in ExLlamaV3 format and needs ExLlamaV3. It won't load in llama.cpp, Ollama or LM Studio.
• General chat and reasoning at 2-bit: ISTA's own IQ2_XS is the file Saluki was built from, and its README reports 96.67 on AIME25 against 100.00 for BF16 in ISTA's harness. If you can spare about 2 GB more, ISTA's IQ3_XXS (10.09 GB) matches BF16 on AIME25 in that harness.

A tool-calling setup you can copy

Download the file and start the server with the settings from Underdog's README:

huggingface-cli download ConwayResearch/Underdog-Saluki-27B-1.0 \
  Underdog-Saluki-27B-1.0-IQ2-mix.gguf --local-dir .

# 12 GB card: 32K context, FP16 KV cache
llama-server -m Underdog-Saluki-27B-1.0-IQ2-mix.gguf \
  --jinja -ngl 99 -fa on -c 32768

--jinja loads the Qwen3.8 chat template, which handles tool calls and thinking. Leave it out and tool calls come back as plain text. To fit 64K on the same card, add --cache-type-k q8_0 --cache-type-v q8_0. That brings the cache from 4.00 GiB to 2.12 GiB.

The server speaks the OpenAI chat API on port 8080. For fast, direct tool calls, Underdog recommends thinking off and temperature 0:

{
  "model": "saluki",
  "temperature": 0,
  "chat_template_kwargs": { "enable_thinking": false },
  "messages": [
    { "role": "user", "content": "What's the weather in Pune?" }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Current weather for a city",
        "parameters": {
          "type": "object",
          "properties": { "city": { "type": "string" } },
          "required": ["city"]
        }
      }
    }
  ]
}

Send it with curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d @request.json. If you're building tool schemas by hand, run them through the JSON formatter first. One stray comma and the request fails before it reaches the model.

For planning or reasoning steps, switch thinking back on with temperature 0.6, top_p 0.95 and top_k 20. Expect long thinking before the answer.

Three catches before you download

Parallel calls need a lenient parser. About a fifth of Saluki's multi-call replies have small formatting slips. On 23 of the 100 parallel tasks, the reply had no call a strict checker could read. With a parser that forgives small slips, the score goes from 42 to 55. If your agent framework rejects malformed calls outright, expect retries.

The text and the card disagree on images. Underdog's blog post says "This release doesn't take images". The Hugging Face card ships two mmproj vision files and a --mmproj command. The main file is text only either way. Test the vision add-on on your own images before you count on it.

Letter-level instructions are its weak spot. Underdog lists palindromes, vowel rules and alphabetical ordering as the instructions it handles worst. Validation rules like "start every line with a capital letter" belong in your code, not the prompt.

FAQ

How much VRAM does Saluki 27B need? The weights are 7.36 GiB. At the README's 32K context with an FP16 KV cache, the total is 9.50 GiB before compute buffers, so plan for a 12 GB card. With a q8_0 cache, 64K fits in 9.62 GiB.

Can Saluki run on an 8 GB GPU? Not fully on the GPU at a useful context. The file alone takes 7.36 GiB of an 8 GiB card. Use partial offload to system RAM and accept slower generation, or pick a 12 GB card.

Is Saluki better than the full Qwen3.8-27B? On Underdog's tool-calling test it scored 88 against 84, which Underdog says is partly run-to-run variation. It fixed fewer SWE-bench issues (30 vs 33) and keeps about 82 to 85% of the full model's competition math score.

Does Saluki work in Ollama and LM Studio? Underdog says it runs in stock llama.cpp and apps built on it, and it's a standard GGUF. Ollama and LM Studio both use llama.cpp. The tool-call template comes from the GGUF's embedded chat template, so check that tool calls come back as structured tool_calls and not plain text.

What license is Saluki under? Apache 2.0, the same as Qwen3.8-27B and the ISTA-DASLab GGUF it was built from. Commercial use is allowed.

Conclusion

Saluki gets Qwen3.8-27B's tool selection into a 7.36 GiB file that runs in plain llama.cpp, and on Underdog's tests it gives up little on tool use and instruction following. It loses ground on competition math and multi-step reasoning. The file size undersells the memory bill: at 32K context the total is 9.50 GiB, so it's a 12 GB card model, or a 16 GB laptop model, but not an 8 GB one. If your agent mostly picks one tool per step, it's the strongest llama.cpp option at this size. If it fires several calls at once, Mia does that better, but only under ExLlamaV3.

Sources

• Hugging Face, ConwayResearch/Underdog-Saluki-27B-1.0 model card, file list, GGUF metadata and NOTICE: file sizes, 262,144 context, llama-server commands, sampling settings, benchmark tables, limitations, Apache 2.0 and the ISTA-DASLab base file.
• Underdog, "Meet Underdog Saluki 27B" (underdog.ai/saluki, 7 October 2026): release date, Underdog Bench comparison with Mia, ISTA and Ternary Bonsai 2, per-category tool scores, parallel-call strict and lenient scores, SWE-bench and WebWalkerQA results, IFEval run, "doesn't take images" note.
• Hugging Face, ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF model card and file list: GSQ-RCO method, file sizes, AIME25/GPQA-Diamond/LiveCodeBench results against BF16.
• Hugging Face, Qwen/Qwen3.8-27B config.json: 64 layers, 16 full-attention layers, 4 KV heads, head dim 256, linear-attention head sizes.
• Hugging Face, bartowski/Qwen3.8-27B-GGUF file list: Q4_K_M at 17,442,399,968 bytes.
• KV cache, DeltaNet state, GiB conversions and card-fit figures are calculated by ToolMintX from those sources. Compute buffers and runtime overhead are not included.