Step 8 — Browser inference boundaries with ONNX Runtime Web

Small classification, embedding, and OCR-assistance models can run in the browser instead of on a server. This can reduce server calls and user-data transfer, but introduces model download, memory, battery, and device-performance costs.

Select an execution provider by capability

Do not assume WebGPU. Detect support and model operators at startup: prefer WebGPU when compatible, fall back to WebAssembly, then use a server API or disable the optional feature. Do not preload the model on the first screen; load it after an action that actually needs inference.

Inference choice

1

Feature request → WebGPU capability → WASM capability → server fallback

2

Model fetch → verify size and hash → create session → bounded inference

Treat download and memory as budgets

Measure model size, uncompressed memory, first-inference latency, and repeated latency on a low-end mobile device. Key Cache Storage by version and hash, and retire an old model only after validating the new one. Be able to release the session when the tab is hidden or memory is constrained.

Completion evidence

• The core interface remains usable in an unsupported browser.
• The model is not downloaded before use, and size plus integrity are verified.
• WebGPU, WASM, and server fallback results stay within an accepted tolerance.
• The feature remains disabled by default when transfer or inference exceeds its budget.

Related term: ONNX Runtime Web

• Previous step
• Back to the course