Running a small language model on a motorcycle: what fits

A 1-billion-parameter model now answers at 50 tokens a second on a phone, with no signal. Here is the memory maths, the speed on a phone, a Raspberry Pi and a Jetson, and what the heat and power cost on a bike.

A motorcycle handlebar with a round display and a smartphone holder
Photo: Shixart1985, CC BY 2.0, via Wikimedia Commons

A small language model can run fully offline on a motorcycle today. Quantised to 4 bits, a 1 to 3 billion parameter model such as Llama 3.2 needs about 2 to 4 GB of memory, and a 2024 Android flagship runs the 1B version at about 50 tokens a second. The best computer for it is the phone already on your handlebar; a Raspberry Pi 5 works but manages only 2 to 15 tokens a second, and any box on the bike has to survive heat and an unsteady 12 V supply.

Key takeaways
  • 50.2 tokens/s and 0.3 s to first token: Llama 3.2 1B, 4-bit SpinQuant, on a OnePlus 12 (Meta).
  • Bits ÷ 8 × parameters gives the weight memory: a 3.21B model at 4 bits is about 1.6 GB before the cache.
  • 5 to 15 tokens/s for models up to 1.5B on a Raspberry Pi 5, 2 to 5 for 3B, at about 10 W peak.
  • 80 to 85°C: where a Pi 5 starts throttling. A sealed box in the sun gets there.
  • Good for commands and tool choice; bad for facts it cannot check, like whether a pass is open.

"Small language model" has no strict definition, but in practice it means roughly 0.5 to 4 billion parameters: small enough to fit in a phone's memory next to everything else. That is a hundred times smaller than the frontier models behind chat apps, and the gap shows in general knowledge. What survives the shrinking is the part a rider actually needs: understanding a spoken command and picking the right action.

Why would a rider want an offline language model at all?

Because the places worth riding to are often the places where a cloud assistant stops answering. An offline model keeps working with no signal, sends nothing off the bike, and answers without a network round trip.

The story

In 2022 I rode my carburetted Pulsar NS200 from Bengaluru to Gurudongmar in North Sikkim, the highest point I have ridden to. Coming down, I planned to reach Zuluk the same day, but I had no phone or maps with me. I missed a left turn, kept going, and ended up in Rangpo instead, then carried on to Siliguri, about six to seven hours from the lake. Nothing broke and nobody was hurt. It was simply a day when the one thing I needed was an answer to "which way", and nothing on the bike could give it.

Sapta: Gurudongmar by bike, what thin air did to my carb NS200

A language model would not have known the road on its own. What it can do is sit in front of things that do work offline, such as a downloaded map, the bike's fuel data and your saved route, and let you ask for them by voice. That is the honest job description, and it shapes every choice below.

Aerial view of four people on motorcycles on a desert road in Ladakh
Riders on a desert road in Ladakh. Out here mobile coverage can vanish, which is the case for a model that works offline.Photo: Ville Hyvönen, CC BY-SA 2.0, via Wikimedia Commons

Which small language models can run offline?

Five open families matter in late 2026: Llama 3.2, Gemma 3n, Phi-3, Qwen3 and, on iPhones, Apple's own on-device model. All of them have versions that fit in 2 to 4 GB.

ModelParametersMemory (maker's figure)Notes from the maker
Llama 3.2 1B1.23B1.9 GB RSS (4-bit SpinQuant)Hindi among 8 official languages; 8k context when quantised
Llama 3.2 3B3.21B3.7 GB RSS (4-bit SpinQuant)19.7 tokens/s on a OnePlus 12
Gemma 3n E2B / E4B5B / 8B rawAs little as 2 GB / 3 GBTakes audio input; speech recognition and translation; 140 text languages
Phi-3-mini3.8BAbout 1.8 GB at 4 bitsOver 12 tokens/s on an iPhone 14, fully offline
Qwen3 0.6B / 1.7B / 4B0.6B to 4BNot stated32K context, 119 languages, Apache 2.0, switchable "thinking" mode
Apple on-device modelAbout 3B2 bits per weightCompressed with quantisation-aware training; open to apps via the Foundation Models framework

For an Indian rider, two details stand out. Llama 3.2 officially supports Hindi, and Gemma 3n accepts audio directly, so one model can both hear and answer. Qwen3's tiny 0.6B is the one to try on weak hardware.

Phi-3 Technical Report: A Highly Capable Language Model Locally on Your PhoneMarah Abdin et al. (Microsoft) · arXiv · 2024

Microsoft trained phi-3-mini, a 3.8-billion-parameter model, on 3.3 trillion tokens of heavily filtered web and synthetic data. It scored 69% on the MMLU knowledge benchmark. Quantised to 4 bits, it took about 1.8 GB of memory and generated more than 12 tokens a second natively on an iPhone 14 with the A16 chip, fully offline. The limit, which the authors state: a model this size cannot store much factual knowledge, so it does worse on trivia-style questions unless paired with search.

Read the paper (free)

How much memory does a small language model need?

Weights take parameters × bits ÷ 8 bytes; the conversation cache adds a slice that grows with context. Here is the working for the two Llama 3.2 models, using the numbers in their published configs.

Step 1: the weights

\[ M_{w} = N \times \frac{b}{8} \]
  1. Llama 3.2 1B at 4 bits: \( 1.23 \times 10^9 \times 0.5 = 0.62 \) GB.
  2. Llama 3.2 3B at 4 bits: \( 3.21 \times 10^9 \times 0.5 = 1.6 \) GB.
  3. Same 3B at 16 bits: 6.4 GB, which is why unquantised models do not fit on most phones.

Step 2: the key-value cache

\[ M_{kv} = 2 \times L \times h_{kv} \times d \times C \times 2\ \text{bytes} \]

Here L is layers, hkv key-value heads, d head dimension and C the context in tokens, stored at 16 bits.

  1. 1B (16 layers, 8 KV heads, head size 64) at 4,096 tokens: \( 2 \times 16 \times 8 \times 64 \times 4096 \times 2 = 134 \) MB.
  2. 3B (28 layers, 8 KV heads, head size 128) at 4,096 tokens: \( 2 \times 28 \times 8 \times 128 \times 4096 \times 2 = 470 \) MB.

So the 3B model needs about 2.1 GB by this formula. Meta's measured figure is 3.7 GB, because the embedding table, activations and the runtime itself sit on top. Budget roughly double the weight memory and you will rarely be caught short.

weights 1.6 GBcache 0.47 GBruntime, embeddings≈1.6 GBLlama 3.2 3B, 4-bit12 GB phonemodel 3.7 GBapps, OS, camera,maps, musicmeasured 3.7 GB (Meta)
Where the memory goes for Llama 3.2 3B at 4 bits. Weights and cache from the formula above; the measured total from Meta's model card. The phone column is to scale for a 12 GB device.

Use this to check what you just read, with any model's numbers from its config file:

Memory and speed for a small language model

What does quantisation cost in quality?

Less than you would fear at 4 bits, and more as you go lower. Quantisation stores each weight in fewer bits; good methods first reshape the numbers so that the rounding hurts less.

Meta's quantised Llama 3.2 models averaged 56% smaller files, 41% less memory, 2.5x faster decoding and 4.2x faster prefill than the 16-bit originals on the same phones. Apple goes further, to 2 bits per weight, but only by training the model to expect the rounding.

SpinQuant: LLM quantization with learned rotationsZechun Liu et al. (Meta) · ICLR 2025 · arXiv 2024

The authors learn rotation matrices that spread out the few extreme values inside a model, so that weights, activations and the KV cache can all be cut to 4 bits. With everything at 4 bits, the gap to full precision on zero-shot reasoning tasks shrank to 2.9 points on LLaMA-2 7B, and on the harder-to-quantise LLaMA-3 8B the gap fell by up to 45.1% compared with an earlier method, QuaRot. This is the method behind Meta's on-phone Llama 3.2 numbers. The limit: benchmark accuracy, not a test of spoken commands in noise.

Read the paper (free)

Phone, Raspberry Pi 5 or Jetson: which runs it best on a bike?

The phone wins on speed per watt and on convenience; a Jetson wins on raw throughput; a Pi 5 is the cheapest way to learn. The phone also already has the screen, GPS, mobile data and a battery.

Decode speed of small models on phones
See the data as a table
Decode speed (tokens/s)
Llama 3.2 1B, 4-bit SpinQuant (OnePlus 12)50.2
Llama 3.2 1B, 4-bit QLoRA (OnePlus 12)45.8
Llama 3.2 3B, 4-bit SpinQuant (OnePlus 12)19.7
Llama 3.2 1B, 16-bit (OnePlus 12)19.2
Phi-3-mini 3.8B, 4-bit (iPhone 14)12
Llama 3.2 3B, 16-bit (OnePlus 12)7.6

Source: Meta Llama 3.2 model card (ExecuTorch, OnePlus 12); Abdin et al. 2024 (Phi-3, iPhone 14, reported as more than 12)

PlatformMemoryPowerSmall-model speed
2024 Android flagship (OnePlus 12)Phone RAMPhone battery50.2 tok/s (1B, 4-bit), 19.7 (3B)
Raspberry Pi 51 to 16 GB LPDDR4X5 V / 5 A USB-C supply; about 10 W peak in inference5 to 15 tok/s up to 1.5B; 2 to 5 for 3B
Jetson Orin Nano Super8 GB LPDDR5, 102 GB/s7 to 25 W67 INT8 TOPS; no maker figure for these models
An Evaluation of LLMs Inference on Popular Single-board ComputersTung (Thomas) Nguyen and Tuyen Nguyen · arXiv · 2025

The authors ran 25 quantised open models on a Raspberry Pi 4, a Raspberry Pi 5 and an Orange Pi 5 Pro, using Ollama and Llamafile, and logged speed, memory and power. On the Pi 5, models up to 1.5B parameters held 5 to 15 tokens a second, 3B models 2 to 5, and models under 360M passed 20; peak draw during inference was about 10 W. They conclude such boards reliably handle models up to about 1.5B. The limit: results are given as ranges, and the paper is inconsistent about whether Llamafile ran on the Pi at all.

Read the paper (free)
An Nvidia Jetson Nano 2 development kit
An Nvidia Jetson Nano 2 development kit, one of the small boards a rider could run a model on.Photo: Ubahnverleih, CC0, via Wikimedia Commons

What do power and heat do to a computer on a motorcycle?

Power is easy; heat is the real limit. A 10 W board draws under 1 A from a 12 V system, but it sits in sun, engine heat and a sealed box.

  • Power: the Pi 5 asks for a 5 V, 5 A USB-C supply with Power Delivery, the official one rated 27 W. A weak bike USB socket will brown it out under load. Wire a fused, switched supply, and see my motorcycle battery care guide before adding loads.
  • Heat: Raspberry Pi's own documentation says the Pi 5 throttles its cores progressively from 80°C and throttles the GPU too at 85°C. Its fan curve starts at 50°C.
  • Vibration: use a board with no loose connectors, screw-down storage and a damped mount.
  • Water: a sealed box traps heat; a vented one lets in rain. This is the strongest argument for keeping the computing in the phone.

Is a small model fast enough for spoken replies?

On a recent phone, yes; on a Pi with a 3B model, no. Spoken replies should be short, so the deciding numbers are time to first token and tokens per second for about 25 tokens.

Phone, 1BPhone, 3BPi 5, ≤1.5BPi 5, 3B0.8 s2.0 s1.7 to 5 s5 to 12.5 s25-token reply. Phone: first token + 25 ÷ speed (Meta). Pi: 25 ÷ 5 to 15 tok/s (Nguyen 2025), no first-token time added.
How long a short spoken reply takes to generate. Phone figures: 0.3 s + 25/50.2 for 1B, 0.7 s + 25/19.7 for 3B. Pi ranges use the paper's tokens-per-second bands. Speech-to-text and text-to-speech come on top.

The full chain from your voice to the answer, including speech recognition and text-to-speech, is broken down stage by stage in how to build a voice assistant for your motorcycle helmet.

What is a small language model actually good for on a ride?

Turning loose speech into a precise action, which small models do well, rather than knowing facts, which they do badly.

TaskSmall model on its own?Why
"Find fuel ahead" to a map searchGoodPicks a tool and fills in arguments
Read and summarise a messageGoodShort text in, short text out
Fuel range from tank and km/lUse plain codeArithmetic should never be guessed
"Is Nathu La open today?"BadNeeds live data; the model will make something up
Hindi or Hinglish commandsFairLlama 3.2 supports Hindi; test your own phrases
A small model on a bike is a translator from speech to actions, not an encyclopedia.This post, from the Phi-3 report's own caveat on factual knowledge

A sensible offline stack

  1. Pick the model: Llama 3.2 1B or Qwen3 1.7B at 4 bits for speed; Gemma 3n E2B if you want speech in and text out from one model.
  2. Download offline data: maps for the region, fuel stations along your route, contacts.
  3. Write tools as code: range, nearest fuel, next turn, read last message.
  4. Constrain the output: a tool call or one sentence under 25 words.
  5. Test with no SIM: put the phone in flight mode before the ride and try every command.
A Raspberry Pi 5 single-board computer
A Raspberry Pi 5, a small computer that can run a quantised language model on the bench or in a bike.Photo: RetroEditor, CC BY 4.0, via Wikimedia Commons

Where the industry is taking this, with radar and rider aids rather than chat, is covered in AI in motorcycles: what is real in 2026 and the thought experiment what if a motorcycle had real AI on board. My broader notes on putting models to work are on Applied AI.

Questions people ask

Can a Raspberry Pi 5 run Llama 3.2 offline?

Yes. The 1B model sits in the 5 to 15 tokens-a-second band reported for models up to 1.5B on a Pi 5, and the 3B model in the 2 to 5 band. That is fine for text, slow for conversation.

How much RAM do I need for a 3B model?

About 1.6 GB of weights at 4 bits plus about 0.5 GB of cache at 4,096 tokens, and Meta measured 3.7 GB in total for Llama 3.2 3B on a phone. An 8 GB device is comfortable; 4 GB is tight with other apps open.

Does running a model drain the phone battery on a ride?

The model only runs while answering, a second or two at a time, so the drain is small compared with screen-on navigation. Keep the phone on a charger for long days anyway, and watch its temperature in direct sun.

Is a 1B model smart enough to be useful?

For choosing actions and short summaries, yes. For facts, no: Microsoft's Phi-3 report says models this size cannot hold much factual knowledge. Give it tools and offline data rather than asking it to remember.

Which is better offline, Gemma 3n or Llama 3.2?

Gemma 3n E2B takes audio directly and fits in about 2 GB, so it can replace a separate speech model. Llama 3.2 1B is faster on Meta's published phone numbers and lists Hindi support. Test both on your phrases.

Sources

  1. Meta: Llama 3.2 1B model card (2024) (parameters, languages, OnePlus 12 speed and memory)
  2. Meta AI: Introducing quantized Llama models (2024) (56% smaller, 2.5x decode)
  3. Llama 3.2 3B config file and 1B config file (layers, KV heads, head size)
  4. Abdin et al.: Phi-3 Technical Report, arXiv (2024)
  5. Liu et al.: SpinQuant, arXiv / ICLR 2025 (2024)
  6. Nguyen and Nguyen: An Evaluation of LLMs Inference on Popular Single-board Computers, arXiv (2025)
  7. Google: Introducing Gemma 3n, the developer guide (2025)
  8. Google: Gemma 3n E2B model card (audio input, context)
  9. Qwen team: Qwen3 (2025)
  10. Apple: Foundation language models, 2025 updates (2025)
  11. Raspberry Pi: Raspberry Pi 5 product page (CPU, RAM, power)
  12. Raspberry Pi documentation: throttling and fan behaviour
  13. NVIDIA: Jetson Orin Nano Super developer kit (TOPS, memory, power)
  14. Motor Vehicles Act 1988, Section 184 (as amended 2019)
small language modelLlama 3.2Gemma 3nPhi-3Qwen3quantisationRaspberry Pi 5Jetson Orin Nano
Sapta

Sapta (Saptarshi Sarkar) is a mechanical engineer turned marketer in Bengaluru and a solo touring rider. He writes the seven Saptaverse blogs. Work with him or read more of RoadHook Machines.

Keep reading in RoadHook Machines

From the other blogs