Run a language model in the browser with WebLLM

A hosted model costs money per request and receives everything the visitor types. A model in the browser costs nothing per request once downloaded, and the conversation never leaves the device. WebLLM makes that work for small models wherever the browser has WebGPU.

The chat below downloads a one-billion-parameter model only after the visitor clicks, shows a progress bar while it loads, and streams replies the visitor can stop. It is a plain Vite project with Vue 3.5 and @mlc-ai/web-llm 0.2.79, in <script setup lang="ts">.

What WebLLM is

WebLLM is an inference engine (the part that runs a trained model) built to work inside a web page. It downloads the model's weights as files and runs the maths on the graphics chip through WebGPU, the browser API that exposes the GPU to JavaScript. Your code talks to it through an API shaped like OpenAI's, engine.chat.completions.create({ messages, stream: true }), so streaming, JSON mode and a system prompt work the way they do against a hosted API.

WebLLM does not load a model in its original Hugging Face form. Each model is compiled for WebGPU by the MLC project and listed, with its memory needs, in prebuiltAppConfig.model_list. The 0.2.79 package ships 141 entries, from families including Llama, Phi, Gemma, Mistral, Qwen and SmolLM.

You pick a model by its id, and the id tells you the quantisation, meaning how the weights are compressed. Llama-3.2-1B-Instruct-q4f16_1-MLC is Llama 3.2 with one billion parameters, weights stored in 4 bits and calculations done in 16-bit floats. The q4f32_1 variant of the same model does the calculations in 32-bit floats for GPUs that lack 16-bit shader support, and needs more memory for it.

What the visitor needs

Three things, and your component should check the first and say the other two out loud.

WebGPU. Chrome and Edge have shipped it since version 113 on Windows, macOS and ChromeOS, and Chrome 121 brought it to Android 12 and later on devices with Qualcomm and ARM GPUs. Safari 26, released in September, has it too. Firefox 141 ships it on Windows only. The quick test is one expression, 'gpu' in navigator, and a browser that fails it needs a fallback: a hosted model, or simply no assistant.

GPU memory. The model list gives a figure per model. The 1B Llama in q4f16_1 wants about 880 MB, the 3B version about 2.3 GB and the 8B Llama 3.1 about 5 GB; the list marks only the first two as fit for limited devices such as phones. A machine without enough memory does not fail politely, so start with the 1B model and offer larger ones only where you know the hardware.

A download. The first load fetches the weights, several hundred megabytes for the 1B model, and the browser keeps them afterwards. That is why the code below never starts the download until the visitor has clicked a button that says how large it is.

Run the engine in a Web Worker

Inference is heavy. Run it on the main thread and typing in the input box stutters while the model thinks. WebLLM ships a handler for a dedicated Web Worker, so the model lives in its own thread and the page stays responsive:

import { WebWorkerMLCEngineHandler } from '@mlc-ai/web-llm'

const handler = new WebWorkerMLCEngineHandler()

self.onmessage = (event: MessageEvent) => {
  handler.onmessage(event)
}

That is the whole file: the handler creates the real engine inside the worker and answers the page's messages. The main thread creates the worker and asks WebLLM for an engine that forwards every call to it. That engine implements the same MLCEngineInterface as an in-page engine, so the rest of the code does not know a worker exists.

A composable that loads on click

The composable owns the engine, the loading state and the progress figure. Nothing is downloaded until load() is called:

import { ref, shallowRef } from 'vue'
import {
  CreateWebWorkerMLCEngine,
  hasModelInCache,
  type ChatCompletionMessageParam,
  type MLCEngineInterface,
} from '@mlc-ai/web-llm'

export const MODEL_ID = 'Llama-3.2-1B-Instruct-q4f16_1-MLC'

type Status = 'unsupported' | 'idle' | 'loading' | 'ready' | 'thinking'

export function useLocalModel() {
  const engine = shallowRef<MLCEngineInterface | null>(null)
  const status = ref<Status>('gpu' in navigator ? 'idle' : 'unsupported')
  const progress = ref(0)
  const progressText = ref('')
  const cached = ref(false)

  hasModelInCache(MODEL_ID).then((value) => {
    cached.value = value
  })

  async function load() {
    status.value = 'loading'
    const worker = new Worker(new URL('../workers/llm.worker.ts', import.meta.url), { type: 'module' })
    engine.value = await CreateWebWorkerMLCEngine(worker, MODEL_ID, {
      initProgressCallback: (report) => {
        progress.value = report.progress
        progressText.value = report.text
      },
    })
    status.value = 'ready'
  }

  async function ask(messages: ChatCompletionMessageParam[], onToken: (token: string) => void) {
    if (!engine.value) throw new Error('Call load() before ask()')
    status.value = 'thinking'
    const chunks = await engine.value.chat.completions.create({
      messages,
      stream: true,
      max_tokens: 400,
      temperature: 0.7,
    })
    for await (const chunk of chunks) {
      onToken(chunk.choices[0]?.delta.content ?? '')
    }
    status.value = 'ready'
    return engine.value.getMessage()
  }

  function stop() {
    engine.value?.interruptGenerate()
  }

  return { status, progress, progressText, cached, load, ask, stop }
}

What the less obvious lines do:

• The engine sits in a shallowRef, not a ref: nothing in the template reads inside it, so a deep reactive proxy around a library's object adds overhead and buys nothing. The model and its GPU memory live in the worker; the engine on the page is a thin messenger.
• new Worker(new URL('../workers/llm.worker.ts', import.meta.url), { type: 'module' }) is the form Vite's docs recommend for workers: Vite finds the worker file and bundles it with its imports. It does so only when new URL() sits directly inside new Worker(). { type: 'module' } makes it a module worker, so the file can use import.
• CreateWebWorkerMLCEngine() takes the worker, the model id and a config object. Its initProgressCallback fires repeatedly during the load with a progress between 0 and 1 and a text such as "Fetching param cache" or "Loading model from cache", which is exactly what a progress bar and a status line need.
• hasModelInCache() answers whether the weights are already on this device, so the button can say "load" instead of "download" on a return visit.

With stream: true, create() resolves to an async iterable of chunks. Each chunk carries the next piece of the reply, usually one token (a word or part of a word), in choices[0].delta.content, and getMessage() returns the assembled text once the loop ends. interruptGenerate() stops a reply mid-sentence, which a chat needs as much as a send button.

Streaming replies into a chat component

The component decides what to show for each status and keeps the conversation history, because the model has no memory between calls: every request carries the whole exchange.

<script setup lang="ts">
import { ref } from 'vue'
import type { ChatCompletionMessageParam } from '@mlc-ai/web-llm'
import { useLocalModel } from '../composables/useLocalModel'

const { status, progress, progressText, cached, load, ask, stop } = useLocalModel()

const history = ref<ChatCompletionMessageParam[]>([
  { role: 'system', content: 'You are a concise assistant. Answer in two or three sentences.' },
])
const draft = ref('')
const reply = ref('')

async function send() {
  const question = draft.value.trim()
  if (!question || status.value !== 'ready') return
  draft.value = ''
  history.value.push({ role: 'user', content: question })
  reply.value = ''
  const messages = history.value.map((message) => ({ ...message }))
  const full = await ask(messages, (token) => {
    reply.value += token
  })
  history.value.push({ role: 'assistant', content: full })
  reply.value = ''
}
</script>

<template>
  <section>
    <p v-if="status === 'unsupported'">This browser has no WebGPU, so the assistant cannot run here.</p>

    <button v-else-if="status === 'idle'" type="button" @click="load">
      {{ cached ? 'Load the assistant (already on this device)' : 'Download the assistant (several hundred MB, kept in this browser)' }}
    </button>

    <template v-else-if="status === 'loading'">
      <progress :value="progress" max="1">{{ Math.round(progress * 100) }}%</progress>
      <p>{{ progressText }}</p>
    </template>

    <template v-else>
      <ul>
        <li v-for="(message, index) in history.slice(1)" :key="index">
          <strong>{{ message.role }}:</strong> {{ message.content }}
        </li>
        <li v-if="reply"><strong>assistant:</strong> {{ reply }}</li>
      </ul>
      <form @submit.prevent="send">
        <label for="question">Your question</label>
        <input id="question" v-model="draft" type="text" :disabled="status === 'thinking'" />
        <button type="submit" :disabled="status === 'thinking'">Send</button>
        <button type="button" :disabled="status !== 'thinking'" @click="stop">Stop</button>
      </form>
    </template>
  </section>
</template>

The button in the idle state is the consent. It names the size, it says the files stay in the browser, and nothing is fetched before it is pressed. During the reply, reply grows token by token and renders as it grows; when the loop finishes, the full text moves into history and reply empties, so the message is not shown twice.

Send plain data to the worker, not reactive state

One line in send() looks redundant: the map() that copies every message before calling ask(). Remove it and the first question throws.

const full = await ask(history.value, (token) => {
  reply.value += token
})

history.value is a reactive proxy, because ref() wraps arrays and objects in reactive(). The engine sends your request to the worker with postMessage(), which copies its argument with the structured clone algorithm. That algorithm cannot copy a Proxy, so Chrome reports a DataCloneError saying the object could not be cloned. Copying each message into a plain object first gives the engine data it can send:

const messages = history.value.map((message) => ({ ...message }))
const full = await ask(messages, (token) => {
  reply.value += token
})

The same rule applies to anything you post to a worker, a service worker or another window from a Vue app: pass plain data, never reactive state.

Cache the weights, free the GPU and handle errors

After the first load the weights sit in the browser's Cache Storage, the same store service workers use, and the next load reads them from disk instead of the network. The cache belongs to your origin, so a visitor who downloaded the model on another site downloads it again on yours. If you prefer IndexedDB, set useIndexedDBCache: true in an appConfig passed to the engine, and pass the same appConfig to hasModelInCache(), or it checks Cache Storage instead.

GPU memory is not freed when a component unmounts. Register onBeforeUnmount(() => engine.value?.unload()) inside useLocalModel(), and the model is released when the chat goes away; the cached files stay for next time.

Loading and answering can both fail, and the composable as written then leaves status stuck at loading or thinking. Give each function a try/catch that resets status and tells the person what happened. Two failures to expect:

• load() throws DeviceLostError when the GPU is lost during the load, often because it ran out of memory: offer a smaller model or the fallback. A browser that passes 'gpu' in navigator but has no usable GPU fails here too, with "Unable to find a compatible GPU".
• ask() throws ContextWindowSizeExceededError once the conversation outgrows the context window, the most text the model can take in at once: 4,096 tokens for the 1B Llama in the prebuilt list. Drop the oldest messages (keep the system prompt) or start a new chat.

Because the engine runs in a worker, an error reaches the page as a string that starts with its name (DeviceLostError: The WebGPU device was lost…), so check the text, not instanceof.

Limits: speed, quality and battery

Speed depends on the GPU. On a recent laptop with a discrete or Apple GPU a 1B model answers at a comfortable reading pace; on integrated graphics or a phone it can be slower than a person reads. Measure before you promise anything: engine.runtimeStatsText() gives the last reply's speed in tokens per second. In 0.2.79 it also logs a warning that it will soon be deprecated; the replacement is to add stream_options: { include_usage: true } to the create() call, and the last chunk's usage.extra carries the same figures. A quick test on the oldest machine you support is worth more than any benchmark.

Quality is that of a one-billion-parameter model. It can rephrase, summarise a paragraph, draft a short message and answer simple questions from text you give it. It will also state wrong facts with confidence, lose track of long conversations and ignore instructions a larger model would follow. Keep the system prompt short, keep the task narrow, and never present its output as a source of facts.

Battery and heat are real. A GPU at full load for a minute warms a laptop and drains a phone, which is one more reason the download and the first question should both be the visitor's choice, and why the Stop button is not optional.