Models in the browser: WebLLM and Transformers.js

A notes app wants an assistant that answers questions about the person's own notes, works offline and costs nothing per question. A small open model, one anyone may download, can run in the browser on the visitor's graphics chip, so no note leaves the device. The bill moves to the visitor: hundreds of megabytes to download, memory, battery, and a device that may not run it at all. Most of the work is making that bill fair.

The Code tab holds that assistant in a Nuxt app (Nuxt is a Vue framework that renders pages on the server first). You need one idea from the lesson Chrome's built-in AI APIs: check that the device can run a model, and download it only after a click. There Chrome supplied the model; here your page brings its own.

Two libraries for two sizes of job

Two libraries run models in a page, each for one size of job.

WebLLM, the @mlc-ai/web-llm package, runs chat models inside an engine, an object with an API shaped like OpenAI's: engine.chat.completions.create({ messages, stream: true }). It needs WebGPU, the browser API that gives a page access to the graphics chip, the GPU. The notes assistant's model, a small Llama 3.2, is about 700 MB to download.

Transformers.js, the @huggingface/transformers package, runs small task models: sentiment, search by meaning, speech to text, image labels. By default it runs on the CPU through WebAssembly, so it also works where WebGPU does not; pass device: 'webgpu' to use the graphics chip.

A task model fits a smaller job than the notes assistant: a shop's review form that only labels each review positive or negative. useSentiment.ts, in the Code tab, loads a sentiment model for it and sets dtype, the number format of the weights (the numbers the model learned). The default is 8-bit q8 on WebAssembly but 32-bit fp32 on WebGPU, 68 MB against 268 MB, so set it yourself:

const load = () =>
  import('@huggingface/transformers').then(({ pipeline }) =>
    pipeline('sentiment-analysis', 'Xenova/distilbert-base-uncased-finetuned-sst-2-english', {
      dtype: 'q8',
    }),
  )

classify(), in the same file, loads it on the first review and returns its label, POSITIVE or NEGATIVE. A task model is a tenth of the download: keep chat models for free-form answers.

Check the device after mount

Before offering a 700 MB download, the assistant checks the device.

The notes app keeps the model's state in a composable, useLocalModel.ts (a function components call to share state), and shows it in a panel, LocalAssistant.vue, which says "Checking this device…" until the check ends.

navigator.gpu.requestAdapter(), WebGPU's way to ask for a graphics chip (an adapter), can return null even where WebGPU exists, when the chip is blocked or missing. The server render has no browser to ask, so LocalAssistant.vue calls checkDevice() in onMounted, which runs only in the browser:

async function checkDevice() {
  const adapter = 'gpu' in navigator ? await navigator.gpu.requestAdapter() : null
  if (!adapter) {
    status.value = 'no-webgpu'
    return
  }
  modelId.value = adapter.features.has('shader-f16')
    ? 'Llama-3.2-1B-Instruct-q4f16_1-MLC'
    : 'Llama-3.2-1B-Instruct-q4f32_1-MLC'
  cached.value = await caches.has('webllm/model')
  status.value = 'idle'
}

With no adapter, the panel says "This device cannot run the assistant. Questions go to our server instead." Otherwise caches.has() looks for a model an earlier visit stored, and the panel shows Download, or Start when it finds one.

The two model ids differ only in f16 and f32: 16-bit or 32-bit arithmetic. An f16 build uses less memory, 879 MB against 1,129 MB, but needs the chip's shader-f16 feature. Without it, WebLLM stops with This model requires WebGPU extension shader-f16, which is not enabled in this browser. So pick the build from the adapter.

Download once, after a click, in a worker

On a device that passes, a click starts the model in four steps.

load() downloads and starts the model, and only a button that says what will happen calls it: "Download the assistant (about 700 MB, kept in this browser)". The heart of load() starts a worker and the engine:

const { CreateWebWorkerMLCEngine } = await import('@mlc-ai/web-llm')
const worker = new Worker(new URL('../workers/llm.worker.ts', import.meta.url), { type: 'module' })
engine.value = await CreateWebWorkerMLCEngine(worker, modelId.value, {
  initProgressCallback: report => (progress.value = report.progress),
})

The dynamic import() keeps WebLLM's 6 MB of JavaScript out of the page until the click, and initProgressCallback feeds the panel's percentage. If the GPU runs out of memory or the disk is full, the panel says "The assistant could not start:" and the reason.

The model runs in a Web Worker, a separate thread started from llm.worker.ts, so the page stays responsive while it thinks. The engine sits in a shallowRef, so Vue does not track its insides.

WebLLM keeps the weights in the browser's Cache Storage, under webllm/model. The next visit skips the download but still loads 700 MB from disk into GPU memory, which takes seconds: keep the progress bar.

ask() streams the answer into the panel, and the Stop button's stop() ends a rambling answer early with engine.interruptGenerate().

What goes wrong

The first mistake checks for WebGPU in <script setup>, which also runs on the server. Without a guard, the Nuxt server render fails with TypeError (500): Cannot use 'in' operator to search for 'gpu' in undefined, because the server has no navigator. A guard stops the 500 and causes something quieter:

<script setup lang="ts">
const supported = ref(typeof navigator !== 'undefined' && 'gpu' in navigator)
</script>

Its template shows "This browser has no WebGPU" when supported is false, which it always is on the server. The browser computes true and wants the Download button, so Vue, taking over the server's HTML (hydration), warns Hydration node mismatch: rendered on server: <p> … expected on client: button and then Hydration completed but contains mismatches. Render a neutral "Checking this device…" on both sides and decide after mount:

<script setup lang="ts">
const { status, checkDevice } = useLocalModel()
onMounted(checkDevice)
</script>

Both sides now render the same first state, and the warning is gone.

The second mistake creates the engine inside the composable function:

export function useLocalModel() {
  const engine = shallowRef<MLCEngineInterface | null>(null)
  // …
}

Picture a note summary, a second component that calls useLocalModel() as LocalAssistant.vue does. Each call makes its own engine, so both load the model into GPU memory, twice 879 MB; on a laptop with little memory the second load can fail with The WebGPU device was lost while loading the model. Move the state to the top of the file:

const engine = shallowRef<MLCEngineInterface | null>(null)

export function useLocalModel() {
  return { status, modelId, cached, progress, error, checkDevice, load, ask, stop }
}

Every caller now shares one engine and one status. On the server, state at the top of a file is shared by every visitor: safe here, because only the browser writes it.

What to remember

• WebLLM runs chat models on WebGPU, a 700 MB download for a small one. Transformers.js runs task models of tens of megabytes, on WebAssembly by default; set dtype either way.
• Check requestAdapter() in onMounted, and pick the q4f16 build only when the adapter has shader-f16.
• Download only after a click on a button that states the size, show progress, handle errors, and run the model in a worker with a Stop button.
• Cached weights can be evicted, and a return visit still loads them into GPU memory.
• Render the same first state on server and browser, and keep one engine at the top of the composable's file.