NCA-AIIO: AI Infrastructure and OperationsDemo

An AI cluster uses three distinct network fabrics: out-of-band infrastructure management, combined storage and cluster-access traffic, and a dedicated inter-node accelerator-communication fabric for distributed training and inference.

DGX OS is NVIDIA’s customized, optimized installation of Ubuntu Linux, with additional drivers, system-specific configurations, and diagnostic and monitoring tools.

NVIDIA GPUDirect RDMA enables direct data exchange between GPU memory and third-party PCIe peer devices, such as network interfaces, bypassing CPU host-memory staging. This reduces communication overhead and improves the latency and effective bandwidth of remote GPU memory access.

NVIDIA cuDNN provides GPU-accelerated, highly tuned primitives for deep neural networks, including convolution, matrix multiplication, pooling, normalization, and attention operations.

NVIDIA MIG can securely partition supported GPUs into up to seven separate GPU instances, each with isolated GPU resources. Individual GPU models may support fewer instances, but seven is the maximum capability.

A DGX H100 has one dedicated out-of-band management connection: the BMC 1 GbE RJ45 interface. NVIDIA’s DGX H100 network documentation describes the OOB port as the port used for BMC access.

NVIDIA Data Center GPU Manager (DCGM) provides GPU monitoring, configuration management, policy management, health diagnostics, accounting, and NVSwitch monitoring for cluster and datacenter environments.

NVIDIA Merlin is a framework for building high-performing recommender systems at scale, covering workflows such as data preparation, training, and inference.

CUDA is NVIDIA’s parallel computing platform and programming model. It enables GPU programming through supported high-level languages and language extensions that express parallel execution. cuML and cuGraph are specialized libraries built within the CUDA ecosystem.

A DGX H100 system’s 1 GbE RJ45 interface is for BMC out-of-band management. Its in-band management and storage connectivity uses higher-speed Ethernet interfaces rather than 1 GbE connections.

NVIDIA Triton Inference Server is inference-serving software that enables deployment of AI models from multiple frameworks across cloud, data-center, edge, and embedded environments, making it the platform intended for scalable production model serving.

RDMA-capable network adapters perform hardware offloading and enable direct memory transfers with minimal CPU and operating-system involvement. This reduces host CPU overhead and network latency.

GPUDirect Storage enables direct DMA transfers between GPU memory and storage, bypassing CPU bounce buffers. This increases bandwidth and reduces latency and CPU utilization.

GPUs are optimized to perform many similar operations simultaneously across many cores, making them well suited to parallel workloads. CPUs are optimized for fast, low-latency execution of serial and general-purpose tasks.

High-powered GPUs provide the parallel computing capacity needed to train and run modern AI models efficiently, while large amounts of data supply the examples those models require to learn useful patterns.

Training uses large volumes of data to learn model parameters. Inference uses the trained model to make predictions on new inputs and commonly has real-time processing or low-latency requirements.

GPU utilization is the percentage of time that the GPU’s cores are actively processing work during the measured interval. GPU memory usage is a separate memory-utilization metric.

A DPU offloads infrastructure functions from the host CPU, accelerates their processing, and isolates them from application workloads. This supports networking, storage, security, and management services with improved performance and isolation.

GPUDirect Storage provides a direct DMA data path between GPU memory and local or remote storage, avoiding CPU memory bounce buffers and reducing CPU overhead.

nvidia-smi is NVIDIA’s System Management Interface command-line utility and reports per-GPU utilization metrics, including GPU and memory utilization.

Slurm is a cluster workload manager that schedules queued jobs and allocates compute resources, such as nodes and processors, to those jobs.

OpenSM provides the InfiniBand Subnet Manager function required to initialize, discover, activate, and manage an InfiniBand subnet. NVIDIA documentation states that one Subnet Manager must run for each InfiniBand subnet.

NVIDIA Data Center GPU Manager (DCGM) is designed for administration of NVIDIA datacenter GPUs and includes GPU behavior monitoring, health checks, diagnostics, configuration management, and GPU metrics. NVIDIA DCGM Overview

NVIDIA Data Center GPU Manager (DCGM) provides GPU behavior monitoring, configuration management, policy oversight, health and diagnostics, and related administration capabilities for NVIDIA datacenter environments.