Skip to main content
Connect LLM Controls with NVIDIA NIM on an RTX Windows system with Windows Subsystem for Linux 2 (WSL2) installed. NVIDIA NIM (NVIDIA Inference Microservices) provides containers to self-host GPU-accelerated inferencing microservices. In this example, you connect a model component in LLM Controls to a deployed mistral-nemo-12b-instruct NIM on an RTX Windows system with WSL2. For more information on NVIDIA NIM, see the NVIDIA documentation.

​
Prerequisites

  • NVIDIA NIM WSL2 installed
  • A NIM container was deployed according to the model’s instructions. Prerequisites vary between models. For example, to deploy the mistral-nemo-12b-instruct NIM, follow the instructions for Windows on RTX AI PCs (Beta) on your model’s deployment overview
  • Windows 11 build 23H2 or later
  • At least 12 GB of RAM

​
Use the NVIDIA NIM in a flow

To connect the NIM you’ve deployed with LLM Controls, add the NVIDIA model component to a flow.
  1. Create a basic prompting flow.
  2. Replace the OpenAI model component with the NVIDIA component.
  3. In the NVIDIA component’s Base URL field, add the URL where your NIM is accessible. If you followed your model’s deployment instructions, the value is http://localhost:8000/v1.
  4. In the NVIDIA component’s NVIDIA API Key field, add your NVIDIA API Key.
  5. Select your model from the Model Name dropdown.
  6. Open the Playground and chat with your NIM model..