A small language model can run fully offline on a motorcycle today. Quantised to 4 bits, a 1 to 3 billion parameter model such as Llama 3.2 needs about 2 to 4 GB of memory, and a 2024 Android flagship runs the 1B version at about 50 tokens a second. The best computer for it is the phone already on your handlebar; a Raspberry Pi 5 works but manages only 2 to 15 tokens a second, and any box on the bike has to survive heat and an unsteady 12 V supply.
- 50.2 tokens/s and 0.3 s to first token: Llama 3.2 1B, 4-bit SpinQuant, on a OnePlus 12 (Meta).
- Bits ÷ 8 × parameters gives the weight memory: a 3.21B model at 4 bits is about 1.6 GB before the cache.
- 5 to 15 tokens/s for models up to 1.5B on a Raspberry Pi 5, 2 to 5 for 3B, at about 10 W peak.
- 80 to 85°C: where a Pi 5 starts throttling. A sealed box in the sun gets there.
- Good for commands and tool choice; bad for facts it cannot check, like whether a pass is open.
"Small language model" has no strict definition, but in practice it means roughly 0.5 to 4 billion parameters: small enough to fit in a phone's memory next to everything else. That is a hundred times smaller than the frontier models behind chat apps, and the gap shows in general knowledge. What survives the shrinking is the part a rider actually needs: understanding a spoken command and picking the right action.
Why would a rider want an offline language model at all?
Because the places worth riding to are often the places where a cloud assistant stops answering. An offline model keeps working with no signal, sends nothing off the bike, and answers without a network round trip.
In 2022 I rode my carburetted Pulsar NS200 from Bengaluru to Gurudongmar in North Sikkim, the highest point I have ridden to. Coming down, I planned to reach Zuluk the same day, but I had no phone or maps with me. I missed a left turn, kept going, and ended up in Rangpo instead, then carried on to Siliguri, about six to seven hours from the lake. Nothing broke and nobody was hurt. It was simply a day when the one thing I needed was an answer to "which way", and nothing on the bike could give it.
Sapta: Gurudongmar by bike, what thin air did to my carb NS200A language model would not have known the road on its own. What it can do is sit in front of things that do work offline, such as a downloaded map, the bike's fuel data and your saved route, and let you ask for them by voice. That is the honest job description, and it shapes every choice below.

Which small language models can run offline?
Five open families matter in late 2026: Llama 3.2, Gemma 3n, Phi-3, Qwen3 and, on iPhones, Apple's own on-device model. All of them have versions that fit in 2 to 4 GB.
| Model | Parameters | Memory (maker's figure) | Notes from the maker |
|---|---|---|---|
| Llama 3.2 1B | 1.23B | 1.9 GB RSS (4-bit SpinQuant) | Hindi among 8 official languages; 8k context when quantised |
| Llama 3.2 3B | 3.21B | 3.7 GB RSS (4-bit SpinQuant) | 19.7 tokens/s on a OnePlus 12 |
| Gemma 3n E2B / E4B | 5B / 8B raw | As little as 2 GB / 3 GB | Takes audio input; speech recognition and translation; 140 text languages |
| Phi-3-mini | 3.8B | About 1.8 GB at 4 bits | Over 12 tokens/s on an iPhone 14, fully offline |
| Qwen3 0.6B / 1.7B / 4B | 0.6B to 4B | Not stated | 32K context, 119 languages, Apache 2.0, switchable "thinking" mode |
| Apple on-device model | About 3B | 2 bits per weight | Compressed with quantisation-aware training; open to apps via the Foundation Models framework |
For an Indian rider, two details stand out. Llama 3.2 officially supports Hindi, and Gemma 3n accepts audio directly, so one model can both hear and answer. Qwen3's tiny 0.6B is the one to try on weak hardware.
Microsoft trained phi-3-mini, a 3.8-billion-parameter model, on 3.3 trillion tokens of heavily filtered web and synthetic data. It scored 69% on the MMLU knowledge benchmark. Quantised to 4 bits, it took about 1.8 GB of memory and generated more than 12 tokens a second natively on an iPhone 14 with the A16 chip, fully offline. The limit, which the authors state: a model this size cannot store much factual knowledge, so it does worse on trivia-style questions unless paired with search.
Read the paper (free)How much memory does a small language model need?
Weights take parameters × bits ÷ 8 bytes; the conversation cache adds a slice that grows with context. Here is the working for the two Llama 3.2 models, using the numbers in their published configs.
Step 1: the weights
- Llama 3.2 1B at 4 bits: \( 1.23 \times 10^9 \times 0.5 = 0.62 \) GB.
- Llama 3.2 3B at 4 bits: \( 3.21 \times 10^9 \times 0.5 = 1.6 \) GB.
- Same 3B at 16 bits: 6.4 GB, which is why unquantised models do not fit on most phones.
Step 2: the key-value cache
Here L is layers, hkv key-value heads, d head dimension and C the context in tokens, stored at 16 bits.
- 1B (16 layers, 8 KV heads, head size 64) at 4,096 tokens: \( 2 \times 16 \times 8 \times 64 \times 4096 \times 2 = 134 \) MB.
- 3B (28 layers, 8 KV heads, head size 128) at 4,096 tokens: \( 2 \times 28 \times 8 \times 128 \times 4096 \times 2 = 470 \) MB.
So the 3B model needs about 2.1 GB by this formula. Meta's measured figure is 3.7 GB, because the embedding table, activations and the runtime itself sit on top. Budget roughly double the weight memory and you will rarely be caught short.
Use this to check what you just read, with any model's numbers from its config file:
Memory and speed for a small language model
What does quantisation cost in quality?
Less than you would fear at 4 bits, and more as you go lower. Quantisation stores each weight in fewer bits; good methods first reshape the numbers so that the rounding hurts less.
Meta's quantised Llama 3.2 models averaged 56% smaller files, 41% less memory, 2.5x faster decoding and 4.2x faster prefill than the 16-bit originals on the same phones. Apple goes further, to 2 bits per weight, but only by training the model to expect the rounding.
The authors learn rotation matrices that spread out the few extreme values inside a model, so that weights, activations and the KV cache can all be cut to 4 bits. With everything at 4 bits, the gap to full precision on zero-shot reasoning tasks shrank to 2.9 points on LLaMA-2 7B, and on the harder-to-quantise LLaMA-3 8B the gap fell by up to 45.1% compared with an earlier method, QuaRot. This is the method behind Meta's on-phone Llama 3.2 numbers. The limit: benchmark accuracy, not a test of spoken commands in noise.
Read the paper (free)Phone, Raspberry Pi 5 or Jetson: which runs it best on a bike?
The phone wins on speed per watt and on convenience; a Jetson wins on raw throughput; a Pi 5 is the cheapest way to learn. The phone also already has the screen, GPS, mobile data and a battery.
See the data as a table
| Decode speed (tokens/s) | |
|---|---|
| Llama 3.2 1B, 4-bit SpinQuant (OnePlus 12) | 50.2 |
| Llama 3.2 1B, 4-bit QLoRA (OnePlus 12) | 45.8 |
| Llama 3.2 3B, 4-bit SpinQuant (OnePlus 12) | 19.7 |
| Llama 3.2 1B, 16-bit (OnePlus 12) | 19.2 |
| Phi-3-mini 3.8B, 4-bit (iPhone 14) | 12 |
| Llama 3.2 3B, 16-bit (OnePlus 12) | 7.6 |
| Platform | Memory | Power | Small-model speed |
|---|---|---|---|
| 2024 Android flagship (OnePlus 12) | Phone RAM | Phone battery | 50.2 tok/s (1B, 4-bit), 19.7 (3B) |
| Raspberry Pi 5 | 1 to 16 GB LPDDR4X | 5 V / 5 A USB-C supply; about 10 W peak in inference | 5 to 15 tok/s up to 1.5B; 2 to 5 for 3B |
| Jetson Orin Nano Super | 8 GB LPDDR5, 102 GB/s | 7 to 25 W | 67 INT8 TOPS; no maker figure for these models |
The authors ran 25 quantised open models on a Raspberry Pi 4, a Raspberry Pi 5 and an Orange Pi 5 Pro, using Ollama and Llamafile, and logged speed, memory and power. On the Pi 5, models up to 1.5B parameters held 5 to 15 tokens a second, 3B models 2 to 5, and models under 360M passed 20; peak draw during inference was about 10 W. They conclude such boards reliably handle models up to about 1.5B. The limit: results are given as ranges, and the paper is inconsistent about whether Llamafile ran on the Pi at all.
Read the paper (free)
What do power and heat do to a computer on a motorcycle?
Power is easy; heat is the real limit. A 10 W board draws under 1 A from a 12 V system, but it sits in sun, engine heat and a sealed box.
- Power: the Pi 5 asks for a 5 V, 5 A USB-C supply with Power Delivery, the official one rated 27 W. A weak bike USB socket will brown it out under load. Wire a fused, switched supply, and see my motorcycle battery care guide before adding loads.
- Heat: Raspberry Pi's own documentation says the Pi 5 throttles its cores progressively from 80°C and throttles the GPU too at 85°C. Its fan curve starts at 50°C.
- Vibration: use a board with no loose connectors, screw-down storage and a damped mount.
- Water: a sealed box traps heat; a vented one lets in rain. This is the strongest argument for keeping the computing in the phone.
Is a small model fast enough for spoken replies?
On a recent phone, yes; on a Pi with a 3B model, no. Spoken replies should be short, so the deciding numbers are time to first token and tokens per second for about 25 tokens.
The full chain from your voice to the answer, including speech recognition and text-to-speech, is broken down stage by stage in how to build a voice assistant for your motorcycle helmet.
What is a small language model actually good for on a ride?
Turning loose speech into a precise action, which small models do well, rather than knowing facts, which they do badly.
| Task | Small model on its own? | Why |
|---|---|---|
| "Find fuel ahead" to a map search | Good | Picks a tool and fills in arguments |
| Read and summarise a message | Good | Short text in, short text out |
| Fuel range from tank and km/l | Use plain code | Arithmetic should never be guessed |
| "Is Nathu La open today?" | Bad | Needs live data; the model will make something up |
| Hindi or Hinglish commands | Fair | Llama 3.2 supports Hindi; test your own phrases |
A small model on a bike is a translator from speech to actions, not an encyclopedia.This post, from the Phi-3 report's own caveat on factual knowledge
A sensible offline stack
- Pick the model: Llama 3.2 1B or Qwen3 1.7B at 4 bits for speed; Gemma 3n E2B if you want speech in and text out from one model.
- Download offline data: maps for the region, fuel stations along your route, contacts.
- Write tools as code: range, nearest fuel, next turn, read last message.
- Constrain the output: a tool call or one sentence under 25 words.
- Test with no SIM: put the phone in flight mode before the ride and try every command.

Where the industry is taking this, with radar and rider aids rather than chat, is covered in AI in motorcycles: what is real in 2026 and the thought experiment what if a motorcycle had real AI on board. My broader notes on putting models to work are on Applied AI.
What's new
- 25 quantised models benchmarked on Raspberry Pi 4, Pi 5 and Orange Pi 5 Pro arXiv, Oct 2025
- Google releases Gemma 3n, running in as little as 2 GB with audio input Google Developers Blog, Jun 2025
- Apple opens its 3B, 2-bit on-device model to app developers Apple Machine Learning Research, Jun 2025
- Qwen3 ships 0.6B to 4B open models with 119 languages Qwen team, Apr 2025
Questions people ask
Can a Raspberry Pi 5 run Llama 3.2 offline?
Yes. The 1B model sits in the 5 to 15 tokens-a-second band reported for models up to 1.5B on a Pi 5, and the 3B model in the 2 to 5 band. That is fine for text, slow for conversation.
How much RAM do I need for a 3B model?
About 1.6 GB of weights at 4 bits plus about 0.5 GB of cache at 4,096 tokens, and Meta measured 3.7 GB in total for Llama 3.2 3B on a phone. An 8 GB device is comfortable; 4 GB is tight with other apps open.
Does running a model drain the phone battery on a ride?
The model only runs while answering, a second or two at a time, so the drain is small compared with screen-on navigation. Keep the phone on a charger for long days anyway, and watch its temperature in direct sun.
Is a 1B model smart enough to be useful?
For choosing actions and short summaries, yes. For facts, no: Microsoft's Phi-3 report says models this size cannot hold much factual knowledge. Give it tools and offline data rather than asking it to remember.
Which is better offline, Gemma 3n or Llama 3.2?
Gemma 3n E2B takes audio directly and fits in about 2 GB, so it can replace a separate speech model. Llama 3.2 1B is faster on Meta's published phone numbers and lists Hindi support. Test both on your phrases.
Sources
- Meta: Llama 3.2 1B model card (2024) (parameters, languages, OnePlus 12 speed and memory)
- Meta AI: Introducing quantized Llama models (2024) (56% smaller, 2.5x decode)
- Llama 3.2 3B config file and 1B config file (layers, KV heads, head size)
- Abdin et al.: Phi-3 Technical Report, arXiv (2024)
- Liu et al.: SpinQuant, arXiv / ICLR 2025 (2024)
- Nguyen and Nguyen: An Evaluation of LLMs Inference on Popular Single-board Computers, arXiv (2025)
- Google: Introducing Gemma 3n, the developer guide (2025)
- Google: Gemma 3n E2B model card (audio input, context)
- Qwen team: Qwen3 (2025)
- Apple: Foundation language models, 2025 updates (2025)
- Raspberry Pi: Raspberry Pi 5 product page (CPU, RAM, power)
- Raspberry Pi documentation: throttling and fan behaviour
- NVIDIA: Jetson Orin Nano Super developer kit (TOPS, memory, power)
- Motor Vehicles Act 1988, Section 184 (as amended 2019)


