NCP-AIO: AI OperationsDemo

High availability for an edge cluster provides redundant capacity and failover so workloads can continue when an individual edge server fails. Secure NFS only addresses storage access, over-the-air updates distribute application changes, and MIG partitions GPU resources within a server; none supplies server-level failover.

Multi-Instance GPU (MIG) support partitions a compatible physical GPU into separate GPU instances with isolated compute and memory paths. This provides predictable performance and prevents one application’s workload from interfering with another application sharing the same GPU.

Slurm schedules GPUs that are configured as GPU GRES resources. Explicitly defining only the GPUs intended for compute use in gres.conf, with matching GRES configuration in slurm.conf, keeps reserved devices out of the schedulable GPU set. Slurm supports manually specifying a subset of system GPUs in gres.conf when autodetection is disabled.

Existing DOCA driver software must be uninstalled before a new DOCA version is installed, preventing conflicts with the prior DOCA or MLNX_OFED installation.

Setting ConstrainDevices=yes in Slurm's cgroup.conf constrains a job's permitted device files according to its allocated GRES resources. With the task/cgroup plugin configured, this prevents a job allocated one GPU from accessing the node's other GPUs.

iostat reports CPU and disk input/output statistics, enabling investigation of disk utilization, throughput, and I/O wait when storage performance may be slowing model training.

The L1 Researcher role provides researcher-level access to create and manage research workloads that use GPU resources, without granting permissions to administer cluster settings or users. NVIDIA Run:ai classifies L1 Researcher as a legacy researcher role, distinct from administrative and management roles.

scontrol show job displays detailed job records, including the scheduler-assigned reason a pending job has not started, such as unavailable resources, an unmet dependency, or a policy limit. sacct is accounting-oriented, while sinfo reports node and partition information.

sstat displays status and resource-usage information for running Slurm jobs and job steps, including CPU, task, RSS, and virtual-memory metrics. The -j option takes a job ID with an optional step ID in the form <job(.step)>.

scontrol show job <jobid> returns detailed job information, including the Reason field that identifies why the scheduler has not started a pending job, such as unavailable resources, dependencies, or policy limits.

The runai-adm Administrator CLI requires Kubernetes access through a configured kubectl environment and a Kubernetes profile with cluster-administrative privileges. Those permissions allow the CLI to carry out cluster-wide administrative operations, including automated node-role configuration and log collection.

The NVIDIA AI Enterprise pre-configured GPU Operator uses a prebuilt vGPU driver image available to entitled enterprise customers and is configured to use the NVIDIA License System (NLS).

InfiniBand provides a high-bandwidth, low-latency, RDMA-capable interconnect for inter-node communication, which is critical for the frequent data exchanges in large-scale distributed AI training. NVIDIA identifies InfiniBand as the scale-out compute fabric for massive multi-node AI training clusters.

NVIDIA DOCA is the BlueField DPU programming framework. Its SDK provides APIs and libraries for building custom BlueField applications and services across networking, security, and storage, and NVIDIA provides DOCA storage applications and NVMe-oF capabilities for block-storage workloads.

NVIDIA Fleet Command is a hybrid-cloud platform designed to securely deploy, manage, and scale AI applications across edge devices and servers.

GPUDirect Storage uses direct DMA data paths between storage and GPU memory; for networked storage, compatible RDMA-capable network devices, drivers, and configuration are required to preserve that path and its performance benefits. GPU and CPU clock rates or additional VRAM do not validate the RDMA data path.

NCCL and NVSHMEM control GPU communication transports and their runtime configuration. NCCL peer-to-peer settings determine whether direct GPU-to-GPU communication over NVLink or PCIe is used, so an unsuitable configuration can degrade communication performance. NCCL Environment Variables

The GPU polling subsystem in NVIDIA Fabric Manager is set by default to poll every 30 seconds, providing periodic monitoring of GPU and NVLink fabric state used for partition management.

UFM must be able to exchange management traffic with InfiniBand switches. A firewall on the UFM server can block that traffic; NVIDIA’s installation guidance specifies disabling the firewall or allowing the required ports. Disabling the firewall therefore restores the necessary communication path.

The -o (--output) option directs a batch script’s standard output to the specified file. Unless a separate error file is configured with -e, Slurm directs standard error to that same output file by default.

Slurm jobs should explicitly request the required GPU count with a GRES request such as --gres=gpu:<number>, allowing Slurm to allocate and track the GPUs required by each job. Its backfill scheduler considers GRES requirements when evaluating pending jobs and can schedule work into available resource windows, improving utilization and scheduling opportunities for multi-GPU workloads.

docker logs CONTAINER-NAME fetches a container’s logs; by default, Docker logs include output written to STDOUT and STDERR. It is the applicable listed command for viewing a container’s I/O output.

Lustre metadata requests are served by the Metadata Server and Metadata Target, whereas Object Storage Targets hold file-data objects. Separating the metadata-intensive and large sequential I/O workload classes into different storage pools isolates their data-placement and storage resources, reducing contention between the competing access patterns. Increasing OSTs adds capacity for file data rather than metadata operations, while reducing striping would diminish parallel throughput for large sequential I/O.

ngc config set configures the NGC CLI by prompting for the API key and relevant organization and team settings, enabling authenticated access to NGC services, including Base Command Platform.