Merge pull request #2502 from madeline-underwood/dgx

Dgx_JA to review

Author: jasonrandrews

content/learning-paths/laptops-and-desktops/dgx_spark_llamacpp/1_gb10_introduction.md

@@ -1,241 +1,49 @@
 ---
-title: Verify Grace Blackwell system readiness for AI inference
+title: Explore Grace Blackwell architecture for efficient quantized LLM inference
 weight: 2
 
 ### FIXED, DO NOT MODIFY
 layout: learningpathall
 ---
 
-## Introduction to Grace Blackwell architecture
+## Overview
 
-In this session, you will explore the architecture and system design of the [NVIDIA DGX Spark](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/) platform, a next-generation Arm-based CPU–GPU hybrid for large-scale AI workloads.
+In this Learning Path you will explore the architecture and system design of NVIDIA DGX Spark, a next-generation Arm-based CPU-GPU hybrid for large-scale AI workloads. The NVIDIA DGX Spark is a personal AI supercomputer that brings data center-class AI computing directly to the developer desktop. The NVIDIA GB10 Grace Blackwell Superchip fuses CPU and GPU into a single unified compute engine.
 
-You will also perform hands-on verification steps to ensure your DGX Spark environment is properly configured for subsequent GPU-accelerated LLM sessions.
+The GB10 platform combines:
+- The NVIDIA Grace CPU, featuring 10 Arm [Cortex-X925](https://www.arm.com/products/cortex-x) and 10 [Cortex-A725](https://www.arm.com/products/silicon-ip-cpu/cortex-a/cortex-a725) cores built on the Armv9 architecture, offering exceptional single-thread performance and power efficiency
+- The NVIDIA Blackwell GPU, equipped with next-generation CUDA cores and 5th-generation Tensor Cores, optimized for FP8 and FP4 precision workloads
+- A 128 GB unified memory subsystem, enabling both CPU and GPU to share the same address space with NVLink-C2C, eliminating data-transfer bottlenecks
 
-The NVIDIA DGX Spark is a personal AI supercomputer that brings data center–class AI computing directly to the developer desktop.
-The NVIDIA GB10 Grace Blackwell Superchip fuses CPU and GPU into a single unified compute engine.
+This GB10 platform design delivers up to 1 petaFLOP (1,000 TFLOPs) of AI performance at FP4 precision. DGX Spark is a compact yet powerful development platform that lets you build and test AI models locally before scaling them to larger systems.
 
-The NVIDIA Grace Blackwell DGX Spark (GB10) platform combines:
-- The NVIDIA Grace CPU, featuring 10 Arm [Cortex-X925](https://www.arm.com/products/cortex-x) and 10 [Cortex-A725](https://www.arm.com/products/silicon-ip-cpu/cortex-a/cortex-a725) cores built on the Armv9 architecture, offering exceptional single-thread performance and power efficiency.
+You can find out more about Nvidia DGX Spark on the [NVIDIA website](https://www.nvidia.com/en-gb/products/workstations/dgx-spark/).
 
-- The NVIDIA Blackwell GPU, equipped with next-generation CUDA cores and 5th-generation Tensor Cores, optimized for FP8 and FP4 precision workloads.
-- A 128 GB unified memory subsystem, enabling both CPU and GPU to share the same address space with NVLink-C2C, eliminating data-transfer bottlenecks.
+## Benefits of Grace Blackwell for quantized LLM inference
 
-This design delivers up to one petaFLOP (1,000 TFLOPs) of AI performance at FP4 precision.
-DGX Spark is a compact yet powerful development platform for modern AI workloads.
+Quantized Large Language Models (LLMs), such as those using Q4, Q5, or Q8 precision, benefit from the hybrid architecture of the Grace Blackwell Superchip which brings several key advantages to quantized LLM workloads. The unified CPU-GPU design eliminates traditional bottlenecks while providing specialized compute capabilities for different aspects of inference.
 
-DGX Spark represents a major step toward NVIDIA’s vision of AI Everywhere, empowering developers to prototype, fine-tune, and deploy large-scale AI models locally while seamlessly connecting to cloud or data-center environments when needed.
+On Arm-based systems, quantized LLM inference is especially efficient because the Grace CPU delivers high single-thread performance and energy efficiency, while the Blackwell GPU accelerates matrix operations using Arm-optimized CUDA libraries. The unified memory architecture means you don't need to manually manage data movement between CPU and GPU, which is a common challenge on traditional x86-based platforms. This is particularly valuable when working with large models or running multiple inference tasks in parallel, as it reduces latency and simplifies development.
 
-### Why Grace Blackwell for quantized LLMs?
+## Grace Blackwell features and their impact on quantized LLMs
 
-Quantized Large Language Models (LLMs), such as those using Q4, Q5, or Q8 precision, benefit from the hybrid architecture of the Grace Blackwell Superchip.
+The table below shows how specific hardware features enable efficient quantized model inference:
 
-| **Feature** | **Impact on Quantized LLMs** |
+| **Feature** | **Impact on quantized LLMs** |
 |--------------|------------------------------|
-| Grace CPU (Arm Cortex-X925 / A725) | Handles token orchestration, memory paging, and lightweight inference efficiently with high IPC (instructions per cycle). |
-| Blackwell GPU (CUDA 13, FP4/FP8 Tensor Cores) | Provides massive parallelism and precision flexibility, ideal for accelerating 4-bit or 8-bit quantized transformer layers. |
-| High Bandwidth + Low Latency | NVLink-C2C delivers 900 GB/s of bidirectional bandwidth, enabling synchronized CPU–GPU workloads. |
-| Unified 128 GB Memory (NVLink-C2C) | CPU and GPU share the same memory space, allowing quantized model weights to be accessed without explicit data transfer. |
-| Energy-Efficient Arm Design | Armv9 cores maintain strong performance-per-watt, enabling sustained inference for extended workloads. |
+| Grace CPU (Arm Cortex-X925 / A725) | Handles token orchestration, memory paging, and lightweight inference efficiently with high instructions per cycle (IPC) |
+| Blackwell GPU (CUDA 13, FP4/FP8 Tensor Cores) | Provides massive parallelism and precision flexibility, ideal for accelerating 4-bit or 8-bit quantized transformer layers |
+| High bandwidth and low latency | NVLink-C2C delivers 900 GB/s of bidirectional bandwidth, enabling synchronized CPU-GPU workloads |
+| Unified 128 GB memory (NVLink-C2C) | CPU and GPU share the same memory space, allowing quantized model weights to be accessed without explicit data transfer |
+| Energy-efficient Arm design | Armv9 cores maintain strong performance-per-watt, enabling sustained inference for extended workloads |
 
+## Overview of a typical quantized LLM workflow
 
 In a typical quantized LLM workflow:
-- The Grace CPU orchestrates text tokenization, prompt scheduling, and system-level tasks.
-- The Blackwell GPU executes the transformer layers using quantized matrix multiplications for optimal throughput.
-- Unified memory allows models like Qwen2-7B or LLaMA3-8B (Q4_K_M) to fit directly into the shared memory space — reducing copy overhead and enabling near-real-time inference.
+- The Grace CPU orchestrates text tokenization, prompt scheduling, and system-level tasks
+- The Blackwell GPU executes the transformer layers using quantized matrix multiplications for optimal throughput
+- Unified memory allows models like Qwen2-7B or LLaMA3-8B (Q4_K_M) to fit directly into the shared memory space, reducing copy overhead and enabling near-real-time inference
 
 Together, these features make the GB10 a developer-grade AI laboratory for running, profiling, and scaling quantized LLMs efficiently in a desktop form factor.
 
 
-### Inspecting your GB10 environment
-
-Let's verify that your DGX Spark system is configured and ready for building and running quantized LLMs.
-
-#### Step 1: Check CPU information
-
-Run the following command to print the CPU information:
-
-```bash
-lscpu
-```
-
-Expected output:
-
-```output
-Architecture:             aarch64
-  CPU op-mode(s):         64-bit
-  Byte Order:             Little Endian
-CPU(s):                   20
-  On-line CPU(s) list:    0-19
-Vendor ID:                ARM
-  Model name:             Cortex-X925
-    Model:                1
-    Thread(s) per core:   1
-    Core(s) per socket:   10
-    Socket(s):            1
-    Stepping:             r0p1
-    CPU(s) scaling MHz:   89%
-    CPU max MHz:          4004.0000
-    CPU min MHz:          1378.0000
-    BogoMIPS:             2000.00
-    Flags:                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 as
-                          imddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 f
-                          lagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
-  Model name:             Cortex-A725
-    Model:                1
-    Thread(s) per core:   1
-    Core(s) per socket:   10
-    Socket(s):            1
-    Stepping:             r0p1
-    CPU(s) scaling MHz:   99%
-    CPU max MHz:          2860.0000
-    CPU min MHz:          338.0000
-    BogoMIPS:             2000.00
-    Flags:                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 as
-                          imddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 f
-                          lagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
-Caches (sum of all):      
-  L1d:                    1.3 MiB (20 instances)
-  L1i:                    1.3 MiB (20 instances)
-  L2:                     25 MiB (20 instances)
-  L3:                     24 MiB (2 instances)
-NUMA:                     
-  NUMA node(s):           1
-  NUMA node0 CPU(s):      0-19
-Vulnerabilities:          
-  Gather data sampling:   Not affected
-  Itlb multihit:          Not affected
-  L1tf:                   Not affected
-  Mds:                    Not affected
-  Meltdown:               Not affected
-  Mmio stale data:        Not affected
-  Reg file data sampling: Not affected
-  Retbleed:               Not affected
-  Spec rstack overflow:   Not affected
-  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
-  Spectre v1:             Mitigation; __user pointer sanitization
-  Spectre v2:             Not affected
-  Srbds:                  Not affected
-  Tsx async abort:        Not affected
-```
-
-The Grace CPU implements the Armv9-A instruction set and supports advanced vector extensions, making it ideal for quantized LLM inference and tensor operations.
-
-The following table summarizes the key specifications of the Grace CPU and explains their relevance to quantized LLM inference.
-
-| **Category** | **Specification** | **Description / Impact for LLM Inference** |
-|---------------|-------------------|---------------------------------------------|
-| Architecture | Armv9-A (64-bit, aarch64) | Modern Arm architecture supporting advanced vector and AI extensions. |
-| Core Configuration | 20 cores total — 10× Cortex-X925 (Performance) + 10× Cortex-A725 (Efficiency) | Heterogeneous CPU design balancing high performance and power efficiency. |
-| Threads per Core | 1 | Optimized for deterministic scheduling and predictable latency. |
-| Clock Frequency | Up to **4.0 GHz** (Cortex-X925)<br>Up to **2.86 GHz** (Cortex-A725) | High per-core speed ensures strong single-thread inference for token orchestration. |
-| Cache Hierarchy | L1: 1.3 MiB × 20<br>L2: 25 MiB × 20<br>L3: 24 MiB × 2 | Large shared L3 cache enhances data locality for multi-threaded inference workloads. |
-| Instruction Set Features** | SVE / SVE2, BF16, I8MM, AES, SHA3, SM4, CRC32 | Vector and mixed-precision instructions accelerate quantized (Q4/Q8) math operations. |
-| NUMA Topology | Single NUMA node (node0: 0–19) | Simplifies memory access pattern for unified memory workloads. |
-| Security & Reliability | Not affected by Meltdown, Spectre, Retbleed, or similar vulnerabilities | Ensures stable and secure operation for long-running inference tasks. |
-
-Its SVE2, BF16, and INT8 matrix multiplication (I8MM) capabilities make it ideal for quantized LLM workloads, providing power-efficient foundation for both CPU-only inference and CPU–GPU hybrid processing.
-
-You can also verify the operating system running on your DGX Spark by using the following command:
-
-```bash
-lsb_release -a
-```
-
-Expected output:
-
-```log
-No LSB modules are available.
-Distributor ID:	Ubuntu
-Description:	Ubuntu 24.04.3 LTS
-Release:	24.04
-Codename:	noble
-```
-As shown above, DGX Spark runs on Ubuntu 24.04 LTS, a developer-friendly Linux distribution.
-It provides excellent compatibility with AI frameworks, compiler toolchains, and system utilities—making it an ideal environment for building and deploying quantized LLM workloads.
-
-#### Step 2: Verify Blackwell GPU and driver
-
-After confirming your CPU configuration, verify that the Blackwell GPU inside the GB10 Grace Blackwell Superchip is available and ready for CUDA workloads.
-
-```bash
-nvidia-smi
-```
-
-You will see output similar to:
-
-```output
-Wed Oct 22 09:26:54 2025       
-+-----------------------------------------------------------------------------------------+
-| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
-+-----------------------------------------+------------------------+----------------------+
-| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
-| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
-|                                         |                        |               MIG M. |
-|=========================================+========================+======================|
-|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
-| N/A   32C    P8              4W /  N/A  | Not Supported          |      0%      Default |
-|                                         |                        |                  N/A |
-+-----------------------------------------+------------------------+----------------------+
-
-+-----------------------------------------------------------------------------------------+
-| Processes:                                                                              |
-|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
-|        ID   ID                                                               Usage      |
-|=========================================================================================|
-|    0   N/A  N/A            3094      G   /usr/lib/xorg/Xorg                       43MiB |
-|    0   N/A  N/A            3172      G   /usr/bin/gnome-shell                     16MiB |
-+-----------------------------------------------------------------------------------------+
-```
-
-The `nvidia-smi` tool reports GPU hardware specifications and provides valuable runtime information, including driver status, temperature, power usage, and GPU utilization. This information helps verify that the system is ready for AI workloads.
-
-The table below provides more explanation of the `nvidia-smi` output:
-
-| **Category** | **Specification (from nvidia-smi)** | **Description / Impact for LLM Inference** |
-|---------------|--------------------------------------|---------------------------------------------|
-| GPU Name** | NVIDIA GB10 | Confirms the system recognizes the Blackwell GPU integrated into the Grace–Blackwell Superchip. |
-| Driver Version | 580.95.05 | Indicates that the system is running the latest driver package required for CUDA 13 compatibility. |
-| CUDA Version | 13.0 | Confirms that the CUDA runtime supports GB10 (sm_121) and is ready for accelerated quantized LLM workloads. |
-| Architecture / Compute Capability | Blackwell (sm_121) | Supports FP4, FP8, and BF16 Tensor Core operations optimized for LLMs. |
-| Memory | Unified 128 GB LPDDR5X (shared with CPU via NVLink-C2C) | Enables zero-copy data access between Grace CPU and GPU for unified inference memory space. |
-| Power & Thermal Status | ~4W at idle, 32°C temperature | Confirms the GPU is powered on and thermally stable while idle. |
-| GPU-Utilization | 0% (Idle) | Indicates no active compute workloads; GPU is ready for new inference jobs. |
-| Memory Usage | Not Supported (headless GPU configuration) | DGX Spark operates in headless compute mode; display memory metrics may not be exposed. |
-| Persistence Mode | On | Ensures the GPU remains initialized and ready for rapid inference startup. |
-
-
-#### Step 3: Check CUDA Toolkit
-
-To build the CUDA version of llama.cpp, the system must have a CUDA toolkit installed.
-
-The `nvcc --version` command confirms that the CUDA compiler is available and compatible with CUDA 13.
-This ensures that CMake can correctly detect and compile the GPU-accelerated components.
-
-```bash
-nvcc --version
-```
-
-You will see output similar to:
-
-```output
-nvcc: NVIDIA (R) Cuda compiler driver
-Copyright (c) 2005-2025 NVIDIA Corporation
-Built on Wed_Aug_20_01:57:39_PM_PDT_2025
-Cuda compilation tools, release 13.0, V13.0.88
-Build cuda_13.0.r13.0/compiler.36424714_0
-```
-
-{{% notice Note %}}
-The nvcc compiler is required only during the CUDA-enabled build process; it is not needed at runtime for inference.
-{{% /notice %}}
-
-This confirms that the CUDA 13 toolkit is installed and ready for GPU compilation.
-If the command is missing or reports an older version (e.g., 12.x), you should update to CUDA 13.0 or later to ensure compatibility with the Blackwell GPU (sm_121).
-
-At this point, you have verified that:
-- The Grace CPU (Arm Cortex-X925 / A725) is correctly recognized and supports Armv9 extensions.
-- The Blackwell GPU is active with driver 580.95.05 and CUDA 13 runtime.
-- The CUDA toolkit 13.0 is available for building the GPU-enabled version of llama.cpp.
-
-Your DGX Spark environment is now fully prepared for the next section,  where you will build and configure both CPU and GPU versions of llama.cpp, laying the foundation for running quantized LLMs efficiently on the Grace Blackwell platform.