small-language-models

Small Language Models (SLMs)

Confidence 0.75 · 22 sources · last confirmed 2026-11-04

Language models chosen by deployment footprint rather than capability ceiling: small enough to run on hardware you already own, at latency low enough to serve one user’s agentic requests in real time. The wiki tracks SLMs as a design variable for agents — the claim that most of what an agent asks a model to do is narrow enough that a generalist frontier model is over-provisioned for it.

The position was stated as a research argument by NVIDIA Research and Georgia Tech (Belcak et al., 2025), demonstrated as a working model family by Salesforce AI Research ( APIGen-MT), measured on an academic instrument by UC Berkeley (BFCL), and carried to a practitioner audience by Sokolenko at PyCon DE 2026.

The definition is device-indexed, so it moves

The canonical working definition (WD1) is deliberately not a parameter count:

A SLM is a LM that can fit onto a common consumer electronic device and perform inference with latency sufficiently low to be practical when serving the agentic requests of one user. — Belcak et al. 2025

An LLM is then simply “a model that is not a SLM.” Any number attached to this is a dated instance of the definition, not the definition:

Stated	Threshold	Hardware assumed
Belcak et al., mid-2025	”most models below 10bn parameters”	consumer device, 2025
Sokolenko, 2026	10–30B, 4-bit quantized, in 16–32 GB of consumer RAM	laptop, 2026

The wiki holds the definition, not the number. Sokolenko adds a field observation the papers do not: 8B models “are still stupid and dumb” for agentic work; things get interesting around 30B at 4-bit quantization — including the point at which the model reliably terminates its loop.

Why agents are the case where small works

Five arguments, in rough order of how well they hold up.

1. Agents expose a narrow slice of the model. The strongest one, and it is really an observation about the harness: “an AI agent is essentially a heavily instructed and externally choreographed gateway to a language model… restricted to operate within a small section of its otherwise large pallet of skills.” The work the harness does to constrain a generalist is evidence a specialist would have sufficed.
2. Format alignment beats format versatility. Agentic output is parsed by code, so one output format produced reliably is worth more than six produced occasionally. A model post-trained on a single convention is better suited, not merely cheaper.
3. Economics. Serving a 7B model is cited at 10–30× cheaper than a 70–175B model in latency, energy and FLOPs; fine-tuning is a few GPU-hours rather than weeks. Sokolenko reframes this as an experimentation argument rather than a unit-cost one — the binding constraint is that you cannot iterate on a model that will not fit on your machine.
4. Heterogeneity is free. Any call in an agentic system can pick any model, and a model can be a tool called by another model. So SLM-first does not mean LLM-never: the recommended shape is SLMs by default, frontier LLMs invoked selectively.
5. The agent generates its own specialisation data. Instrument the non-HCI calls and you accumulate exactly the narrow instruction data needed to fine-tune the model that replaces the incumbent on that call — a live example being APIGen-MT’s synthetic-trajectory pipeline. Cf. self-improving agents.

The four capabilities that actually matter

The corpus’s cleanest short answer to what an agentic model has to be good at — and the reason generalist benchmarks are the wrong instrument here:

1. Task decomposition — breaking a prompt into subtasks. “This is what reasoning is.”
2. Tool calling — initiating a call, passing parameters, formatting output.
3. Glue-code generation — for cases where no tool covers the problem.
4. Instruction following — staying inside guardrails.

The headline claim drawn from these: for these four, an SLM of size x is now roughly as capable as a generalist LLM of ten times that size.

What the evidence actually shows

The strongest and best-measured instance is xLAM-2 on BFCL — and reading it carefully is a lesson in how to read every claim in this area:

• On single-turn function calling, xLAM-2-70b (88.48) and GPT-4o (88.1) are indistinguishable. That category is saturated, so parity there means little.
• On multi-turn, xLAM-2-70b scores 75.12 against 41–47 for GPT-4o and 36 for o1. The entire margin lives here — and multi-turn is exactly the category BFCL’s own authors identify as unsolved.
• Even the 1B model, ranked 36th overall, beats o1 and gpt-4o on multi-turn.
• But on τ-bench overall, xLAM-2-70b (56.2) loses to Claude 3.5 Sonnet new (60.1), o1 (63.9) and Claude 3.7 with an optimised prompt (69.8).

So: narrow, real, and measured on the axis that matters — not general parity. And every rank is dated: xLAM-2 held BFCL ranks 1–2 in April 2025 and roughly position 18 of ~110 by December 2025.

The counter-arguments worth keeping

• Scaling laws and the “semantic hub.” Larger models are better at general language understanding, full stop, and may integrate meaning across domains in a way small models structurally cannot. The rebuttal is that scaling-law studies hold architecture constant, ignore task-specific fine-tuning, and — the sharper move — that agentic systems decompose problems until the subtasks are simple enough that general abstraction buys nothing.
• Centralised inference may win on economy of scale anyway. Load-balancing a fleet of specialised endpoints is harder than balancing one generalist endpoint, and self-hosting adds infrastructure and staffing costs routinely omitted from per-token comparisons. The proponents concede this is unresolved — “the jury is still out.” This is the honest state of the economic case.
• Inertia may simply win. Roughly USD 57bn of capital is committed to centralised LLM inference, tooling was built for that paradigm, and SLM development still evaluates itself on generalist benchmarks. Named as barriers, not refutations — but barriers with balance sheets behind them.

The practical stack (2026)

What running an SLM agent locally actually takes, per Sokolenko:

Layer	2026 choice	Reason given
Model	xLAM-2 32B, 4-bit quantized	open weights, top-20 on BFCL, fits 16 GB
Local inference	llama.cpp	uses Apple-silicon GPUs; simple Hugging Face naming
Remote inference	Together AI, Hugging Face	when the workload graduates to production
Framework	LangChain / LlamaIndex	orchestration; routes tool choice off function names, argument names and docstrings

And the migration path from an existing LLM agent, per Belcak et al. §6: log non-HCI calls → curate and de-identify (10k–100k examples) → cluster into recurring tasks → select a candidate SLM per task → LoRA/QLoRA fine-tune (optionally distilling from the incumbent LLM) → iterate.

Related concepts

ai-agents · agent-harness · foundation-models · ai-benchmarks · open-source-ai · react-reasoning-acting

Debates and supersession

• What “small” means is contested by construction. The definition is device-indexed and therefore moves with consumer hardware; treat 10B (2025) and 10–30B (2026) as the same definition evaluated in different years, and expect the number to move again. Nothing here is superseded — the number is.
• The economic case is not settled, by the proponents’ own admission. Whether per-token savings survive the loss of centralised load-balancing plus self-hosted infrastructure and staffing is “highly case-specific.” No source in the corpus resolves it, and any page citing the 10–30× figure should carry that caveat.
• The capability case rests on benchmark transfer, which the same argument warns against. Belcak et al. name reliance on the wrong benchmarks as adoption barrier B2 — and then support their own capability claim with a roster of benchmark results, four of eight from their own lab. Consistent, but it means V1 stands or falls on whether BFCL-style agentic benchmarks predict deployment.
• A same-vendor tension worth holding open. NVIDIA’s CEO argues owners should build frontier-scale intelligence while NVIDIA Research argues most agentic work is over-provisioned. Different layers of the stack, opposite rhetorical directions, same company.
• The gap nobody has closed: no source measures an SLM-based agent against an LLM-based one on the same task, under the same harness, with the same evaluation. Until one does, “as good as models ten times the size” is a benchmark claim, not a deployment claim — and the ai-benchmarks page’s scaffold-versus-model confound applies in full.
• Loop termination may be size-conditioned. Sokolenko reports that below ~30B, agentic loops “can never terminate… because the SLMs are incentivized to call as many tools as they can.” If that holds, the runaway-loop failure in Google Cloud’s loop-engineering taxonomy is partly a model-capability threshold rather than purely a harness-design problem. Single-source, unmeasured, and worth confirming.

The loop-count argument for a cheap model (added 2026-09-15)

[[2026-09-14-google-cloud-agent-factory-agent-harnesses-explained|Google Cloud’s Agent Factory]] reaches this page’s conclusion from an axis the corpus had not recorded: not device fit, not privacy, not experimentation cost, but how many times the model runs per task.

“Agents don’t work like traditional single-turn prompt boxes. They run in continuous loops, so an agent might inspect a directory, update a function, execute unit tests, and then repeat that sequence 20, 40, or even 60 times just to finish one task. So when you have something that uses that many sequential hops, speed and cost compound dramatically. So a fast, cost-effective model isn’t a downgrade — it is the exact thing that makes real-time agentic loops practical and responsive.”

The claim is narrower and more testable than the SLM position paper’s: nothing here says a small model is as capable, only that per-invocation latency and cost multiply by the loop count, so the cheapest model that clears the task bar wins on wall-clock and spend. It is the vendor’s argument for Gemini 3.8 Flash as a daily driver, and it is the same reasoning agent-harness records under heterogeneous routing — with the loop count supplying the multiplier that makes the routing decision material.

Note this is a vendor recommending its own cheaper tier, with no measurements attached.

A security practitioner’s bet on small, local models (added 2026-09-19)

Alim (PyData, Sep 2026) makes the SLM case from an unexpected direction, and labels it speculation: “small language models that runs on consumer hardware is going to be the future … so small that … they run natively on macOS, on iOS, on your consumer laptops and phones.” His reason is competitive rather than technical. The frontier labs’ moat is size, and “that has not been working just because Chinese labs are able to produce very effective, smaller, much cheaper models,” so “the power is going to be taken away from large corporations and … be in the consumer’s hands.”

It adds no evidence to this page (see Belcak et al. for the careful version), and confidence is unchanged. It is worth recording for its vantage: someone whose job is securing software sees local, owned models as the likely end state. That bears on the security argument open-source-ai records, where open models measured worse on package hallucination. If deployment moves to the device, that gap becomes an endpoint-security problem.

A specialised model that does not generate text (added 2026-09-22)

This page argues from size: a small model is enough for the narrow calls an agent makes. Three LangChain sources on TypeSafe AI’s Jev make the same heterogeneous-system argument from output type. Jev is a “System One” decision model. It takes a state and typed questions (choice / score / yes-no) and returns probabilities, with no text generation (Runkle & Lovell; Runkle video). The claim is that many agent decisions are classifications: which model to route to, whether a tool call is risky, whether an answer passes a rubric. So they can move off the generalist model. “Use an LLM for open-ended reasoning and generation, and Jev for fast, structured decisions along the way.”

Jev’s size is not disclosed, so it is not evidence for the small-model claim. It is evidence for the specialised-model one, which is the core of Belcak et al.’s portfolio argument. The one measurement (Shea & Roche, five cases, used as an eval judge) is instructive about the economics. Jev cost $0.00035 per call against GPT-5.6 Luna’s $0.00039, but GPT-5.6 Terra’s $0.00289 and Claude Sonnet 4.6’s $0.02811. Latency was ~5–6× lower than every LLM. Against a cheap generalist tier, the per-call saving nearly disappears; the latency saving does not. That fits the loop-count argument above: in a loop, latency compounds. Confidence unchanged; all three sources come from the vendor distributing the integration.

The buyer’s version: frontier models on routine work (added 2026-09-28)

Lukic & Goydan (BCG, Aug 2026) make the routing argument from the cost side, not the builder’s side. Paul Goydan: “A frontier model costs 30% more than a good enough model. Yet… we see more often than not very basic rudimentary tasks are being handed over to frontier models. It’s like giving your teenage driver a brand-new Ferrari.” He calls model selection “a whole new capability” for companies, alongside controlling context-window size. The 30% figure is BCG’s and is far smaller than list-price gaps between frontier and small models, so the direction of the claim is more useful than the number.

Decision models beside frontier models, as a vendor’s roadmap (added 2026-10-05)

Chase at Interrupt NYC (LangChain, 24 Sep 2026) places TypeSafe AI’s Jev in a portfolio rather than presenting it as a replacement: “absolutely there will be these frontier models that are driving these really complex harnesses… But there’s also a ton of other decisions that need to be made.” He names three uses, each tied to latency:

• Evals in the loop: “if you can score it quickly according to some criteria as it’s running… you can catch mistakes in real time.”
• Guardrails: “One of the downsides of guardrails is that it always adds latency.”
• Routing among agents, models, skills or tools.

Two additions to the section above. First, LangChain’s gateway now hosts a second decision model, SemIf, described as open source. Chase claims “a massive explosion in these open source decision models since Jev launched” but names no other. Second, the slide’s worked example shows the output type concretely: a support message scored 0.999 on an is_urgent question by jev-1.13.0. Jev’s size is still undisclosed and the 20–200x faster, 40–400x cheaper range is still TypeSafe’s, so this page’s specialised, not necessarily small reading stands. Confidence unchanged.

Routing by task at an application company (added 2026-10-12)

Ramineni (Fireflies, Oct 2026) describes routing across “almost 7 to 10 different model providers, speech providers”, chosen per task: async versus real-time speech models, a separate model for live sales coaching, and a low-latency model for voice agents that must cope with background noise and a rambling caller. His summary of the design rule: “I don’t need to distill the entire world to go solve something as simple as summarization. But if I’m trying to do a more agentic difficult task … that’s where I might need to use a more powerful model.” He also wants models that run on the user’s phone or desktop for an always-on assistant. The argument matches Goydan’s buyer-side version above, here from a company that runs the routing. No figures; confidence unchanged.

The same routing argument, from McKinsey (added 2026-10-25)

McKinsey’s Explainer on the economics of enterprise AI (Oct 2026) restates the buyer’s case above. Wasim Lala’s client moved from Sonnet to Opus within months although “all those users didn’t need… a much more advanced model”; McKinsey’s own Cursor users pick the newest frontier model when they can. A badly designed agent can vary “up to 30x” in tokens, and many steps of an IT ticket workflow are fixed rules where generative AI “would be a waste of the token cost.” A bank’s control plane now asks its users “why didn’t you route this particular request to an open weight model?” Two consultancies (BCG above, McKinsey here) reach the same advice from cost reviews.

Routing from your own benchmark (added 2026-10-25)

Lloyd (Warp, Oct 2026) shows routing decided by a firm’s own data rather than by public benchmarks. Replaying Warp’s recorded coding tasks against different models gave a cost-against-correctness curve: GPT-5.6 Sol cost $6.23 per task on average at 95% correct; GLM-5.3 Flash cost about a third as much and was “just a little less correct.” “We would probably choose GLM-5.3 Flash for this, honestly,” and the result feeds a routing rule for similar tasks. One vendor’s benchmark on its own work.

Small models trained on synthetic questions (added 2026-10-30)

Hack (ClickUp, on Navigators, Oct 2026) describes a recipe for retrieval subagents: a large model reads a document and writes question-answer pairs, then a model of “a couple hundred million parameters” is trained or RL-trained to find the answers. The data is effectively unlimited, and the result is cheap and specific to one firm’s tools and index. The executive function stays with a frontier model. Chase (Interrupt London, Oct 2026) sees supervised fine-tuning used mainly for narrow tasks such as classification, extraction and summarisation.

A constellation of specialist models, and local models judged not enough (added 2026-11-04)

• Hippocratic AI on Google Cloud (Oct 2026): Polaris is more than 30 specialist models from 30 billion to 400 billion parameters, and a separate open model family (Gemma) is used to evaluate them.
• Van der Voort (Remote) on BNR (Oct 2026): Remote runs local models for meeting transcription, but finds them “not nearly good enough for the kind of work we do” and relies mainly on Anthropic’s models.