What is NVIDIA NIM?

NVIDIA NIM (NVIDIA Inference Microservices) is a software stack that packages optimized inference engines, industry-standard APIs, and model support into containers for local deployment.

This runs through WSL2 on Windows and makes it easier to deploy and run local LLM models on RTX AI PCs. RealTimeX includes a desktop integration that simplifies installation, setup, and ongoing use of supported NIM runtimes.

Privacy

NVIDIA NIM models run fully locally on your machine using your own GPU. RealTimeX does not send any data to NVIDIA or any other third party in order to run NIM models. After a model is installed, it is present on your local machine and RealTimeX will use this local engine for inference.

NVIDIA NIM on RTX is not to be confused with NVIDIA's cloud-based NIM offering. This is a completely separate product and service designed to run NIM on your local RTX GPU.

How does it work?

A NIM is a single model plus software stack packaged into a container designed and maintained by NVIDIA. It is specifically designed to run on NVIDIA RTX GPUs. In RealTimeX, we use NIM to run the LLM models for chat, agents, and all other tasks that require inference.

See the NVIDIA NIM system requirements for the full list of requirements to run NIM models on your system.

What models are supported?

RealTimeX supports all of the models that are available in the NIM containers. You can see the full list of models on build.nvidia.com (opens in a new tab).

How do I install it?

If your machine is compatible and the provider is available in your build, RealTimeX will present a UI to install and manage NIM containers when you select the NVIDIA NIM LLM provider.

Once the official NIM installer has finished, you will be able to use NVIDIA NIM models in RealTimeX.

See the NVIDIA NIM x RealTimeX Walkthrough for the full walkthrough.

Definitions

• NIM: Nvidia Inference Microservice - a single LLM or Model + software stack, packaged into a container designed and maintained by NVIDIA.
• WSL2: Windows Subsystem for Linux 2 - a compatibility layer that allows you to run Linux binaries on Windows 11. You will not need to directly interact with WSL2 - the NIM installer will handle this for you and RealTimeX will use it automatically.
• NIM Installer: The pre-built NVIDIA NIM installer that runs in the RealTimeX Desktop client to unlock the use of NIM models in RealTimeX.
• NIM Manager: The RealTimeX UI that allows you to install, update, and run a NIM.