AI_dev Europe 2024

We will discuss new ways to support cross-platform GPU / AI workloads in container ecosystems, specifically with Docker’s support for the WebGPU standard. WebGPU allows container applications to access the host GPUs and other AI accelerators through a portable API. There is no more need to build Docker images for GPU vendors and their proprietary drivers. We will demonstrate how the WasmEdge project builds on the WebGPU standard to create portable LLM inference applications in Rust, and have those apps seamless managed and orchestrated by Docker.

Community impact:

A major advantage of open-source LLMs is to run them on each user’s private computers. Those include personal laptops, AI PCs, private or hybrid or edge cloud servers, edge devices, or even mobile devices. However, the heterogeneous hardware and software accelerators on those private computers or devices also pose great challenges for today’s AI developers. For example, today, an LLM app requires separate Docker images for all combinations of Nvidia CUDA, CUDNN, AMD ROCm, or Intel AVX etc. Each of those platform-dependent Docker images need to be developed, tested, and deployed separately.

WebGPU and Wasm provide an opportunity to abstract away and unify these underlying AI hardware and software stacks. With Docker support, they allow developers to create truly portable AI applications, and allow ops to manage those applications using their familiar Docker and Kubernetes tools.