Hprc banner tamu.png

Difference between revisions of "Grace:Intro"

From TAMU HPRC
Jump to: navigation, search
(→‎Namesake)
(→‎Hardware Overview)
 
(52 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
<H1>Grace: A Dell x86 HPC Cluster</H1>
 
<H1>Grace: A Dell x86 HPC Cluster</H1>
 
__TOC__
 
__TOC__
−
=== Hardware Overview ===
 
  
−
----
+
== Hardware Overview ==
 +
 
 
[[Image:Grace-racks.jpg|right|400px|caption]]
 
[[Image:Grace-racks.jpg|right|400px|caption]]
  
Line 17: Line 17:
 
|-
 
|-
 
| Total Compute Cores/Nodes:
 
| Total Compute Cores/Nodes:
−
| 44,656 cores<br>925 nodes
+
| 45,376 cores<br>940 nodes
 
|-
 
|-
 
| Compute Nodes:
 
| Compute Nodes:
−
| 800 48-core compute nodes, each with 384GB RAM <br> 100 48-core GPU nodes, each with two A100 40GB GPU accelerator and 384GB RAM <br>8 38-core GPU nodes, each with two RTX 6000 24GB GPU accellator and 384 GB RAM<br>8 48-cores GPU nodes, each with 4 T4 16GB GPU accelerator<br> 8 80-core large memory nodes, each with 3TB RAM
+
| 800 48-core compute nodes, each with 384GB RAM <br> 100 48-core GPU nodes, each with two A100 40GB GPUs and 384GB RAM <br>9 48-core GPU nodes, each with two RTX 6000 24GB GPUs and 384GB RAM<br>8 48-core GPU nodes, each with 4 T4 16GB GPUs and 384GB RAM<br> 15 48-core GPU nodes, each with two A40 48GB GPUs and 384GB RAM<br> 8 80-core large memory nodes, each with 3TB RAM
 
|-
 
|-
 
| Interconnect:
 
| Interconnect:
−
| Mellanox HDR Infiniband
+
| Mellanox HDR 100 InfiniBand
 
|-
 
|-
 
| Peak Performance:
 
| Peak Performance:
−
| 2,638.5 TFLOPS
+
| 6.3 PFLOPS
 
|-
 
|-
 
| Global Disk:
 
| Global Disk:
−
| 5PB (usable) via DDN appliance for general use <br>?PB (raw) via Lenovo's DSS purchased by and dedicated for ??
+
| 5PB (usable) via DDN Lustre appliances for general use <br>1.4PB (usable) via Lenovo DSS GPFS appliance (purchased by and dedicated for Dr. Junjie Zhang's CryoEM Lab)<br>1.9PB (usable) via Lenovo DSS GPFS appliance (purchased by and dedicated for Dr. Ping Chang's iHESP Lab)
 
|-
 
|-
 
| File System:
 
| File System:
Line 44: Line 44:
 
|}
 
|}
  
−
Grace is an Intel x86-64 Linux cluster with 950 compute nodes (44,656 total cores) and 5 login nodes.  There are 800 compute nodes with 384 GB of memory, and 117 GPU nodes with 384 GB of memory.  Among the 117 GPU nodes, there are 100 GPU nodes two A100 40 GB GPU cards, 9 GPU nodes with two RTX 6000 24GB GPU cards, 8 GPU nodes with four T4 16GB GPU cards.  These 800 compute nodes and 117 GPU nodes are a dual socket server with two Intel 6248R 3.0GHz 24-core processors.  There are 8 compute nodes with 3 TB of memory and four Intel 6248 2.5 GHz 20-core processors.
+
Grace is an Intel x86-64 Linux cluster with 940 compute nodes (45,376 total cores) and 5 login nodes.  There are 800 compute nodes with 384 GB of memory, and 117 GPU nodes with 384 GB of memory.  Among the 132 GPU nodes, there are 100 GPU nodes two A100 40 GB GPU cards, 9 GPU nodes with two RTX 6000 24GB GPU cards, 8 GPU nodes with four T4 16GB GPU cards, and 15 GPU nodes with two A40 48GB GPUs.  These 800 compute nodes and 117 GPU nodes are a dual socket server with two Intel 6248R 3.0GHz 24-core processors, commonly known as Cascade Lake.  There are 8 compute nodes with 3 TB of memory and four Intel 6248 2.5 GHz 20-core processors.
 +
 
 +
The interconnecting fabric is a two-level fat-tree based on HDR 100 InfiniBand. 
  
−
The interconnecting fabric is a two-level fat-tree based on EDR Infiniband.  High performance mass storage of 5 petabyte (usable) capacity is made available to all nodes by DDN.
+
High performance mass storage of 5 petabyte (usable) capacity is made available to all nodes by the DDN Lustre storage.  Also, 3.3 PB of Lenovo DSS GPFS storage is for their respective research labs.
  
 
Get details on using this system, see the [[Grace | User Guide for Grace]].
 
Get details on using this system, see the [[Grace | User Guide for Grace]].
−
 
  
 
== Compute Nodes ==
 
== Compute Nodes ==
Line 60: Line 61:
 
! General 384GB <br> Compute
 
! General 384GB <br> Compute
 
! GPU A100<br> Compute
 
! GPU A100<br> Compute
−
! GPU RTX6000<br> Compute
+
! GPU RTX 6000<br> Compute
 
! GPU T4<br> Compute
 
! GPU T4<br> Compute
−
! Large memory 3TB<br> Compute
+
! GPU A40<br> Compute
 +
! Large Memory 3TB<br> Compute
 
|-
 
|-
 
| Total Nodes
 
| Total Nodes
Line 69: Line 71:
 
| 9
 
| 9
 
| 8
 
| 8
 +
| 15
 
| 8
 
| 8
 
|-
 
|-
 
| Processor Type
 
| Processor Type
−
| colspan=4 | Intel Xeon 6248R 3.0GHz 24-core
+
| colspan=5 | Intel Xeon 6248R (Cascade Lake), 3.0GHz, 24-core
−
| Intel 6248 2.5 GHz 20-core
+
| Intel Xeon 6248 (Cascade Lake), 2.5 GHz, 20-core
 
|-
 
|-
 
| Sockets/Node
 
| Sockets/Node
−
| colspan=4 | 2
+
| colspan=5 | 2
 
|  4
 
|  4
 
|-
 
|-
 
| Cores/Node
 
| Cores/Node
−
| colspan=4 | 48
+
| colspan=5 | 48
 
| 80
 
| 80
 
|-
 
|-
 
| Memory/Node
 
| Memory/Node
−
| colspan=4 | 384 GB DDR4, 3200 MHz
+
| colspan=5 | 384GB DDR4, 3200 MHz
−
| 3 TB DDR4, 3200 MHz
+
| 3TB DDR4, 3200 MHz
 
|-
 
|-
 
| Accelerator(s)
 
| Accelerator(s)
 
| N/A
 
| N/A
−
| 2 NVIDIA A100 40GB Accelerator
+
| 2 NVIDIA A100 40GB GPU
−
| 2 NVIDIA RTX6000 24GB Accelerator
+
| 2 NVIDIA RTX 6000 24GB GPU
−
| 4 NVIDIA T4 16GB Accelerator
+
| 4 NVIDIA T4 16GB GPU
 +
| 2 NVIDIA A40 48GB GPU
 
| N/A
 
| N/A
 
|-
 
|-
 
| Interconnect
 
| Interconnect
−
| colspan=5 | Mellanox HDR Infiniband
+
| colspan=6 | Mellanox HDR100 InfiniBand
 
|-
 
|-
 
|Local Disk Space
 
|Local Disk Space
−
| colspan=5 | 1.6TB NVMe
+
| colspan=6 | 1.6TB NVMe (/tmp), 480GB SSD
 
|}
 
|}
−
 
  
 
== Usable Memory for Batch Jobs ==
 
== Usable Memory for Batch Jobs ==
Line 108: Line 111:
 
The table below contains information regarding the approximate limits of Grace memory hardware and our suggestions on its use.
 
The table below contains information regarding the approximate limits of Grace memory hardware and our suggestions on its use.
  
 +
{| class="wikitable" style="text-align: center;"
 +
|+ Memory Limits of Nodes
 +
!
 +
! 384GB Nodes (Regular and GPU)
 +
! 3TB Nodes
 +
|-
 +
| Node Count
 +
| 932
 +
| 8
 +
|-
 +
| Number of Cores
 +
| 48 Cores
 +
| 80 Cores
 +
|-
 +
| Memory Limit<br>Per Core
 +
| 7500 MB <br> ~7.5 GB
 +
| 37120 MB <br> 36.25 GB
 +
|-
 +
| Memory Limit<br>Per Node
 +
| 368640 MB <br> 360 GB
 +
| 2969600 MB <br> 2900 GB
 +
|-
 +
|}
  
 
== Login Nodes ==
 
== Login Nodes ==
  
−
The '''grace.hprc.tamu.edu''' hostname can be used to access the Grace cluster. This translates into one of the five login nodes, '''grace[1-5].hprc.tamu.edu'''. To access a specific login node use its corresponding host name (e.g., grace2.tamu.edu).  All login nodes have 10 GbE connections to the TAMU campus network and direct access to all global parallel (Lustre-based) file systems. The table below provides more details about the hardware configuration of the login nodes.
+
The '''grace.hprc.tamu.edu''' hostname can be used to access the Grace cluster. This translates into one of the five login nodes, '''grace[1-5].hprc.tamu.edu'''. To access a specific login node use its corresponding host name (e.g., grace2.hprc.tamu.edu).  All login nodes have 10 GbE connections to the TAMU campus network and direct access to all global parallel (Lustre-based) file systems. The table below provides more details about the hardware configuration of the login nodes.
  
 
{| class="wikitable" style="text-align: center;"  
 
{| class="wikitable" style="text-align: center;"  
 
|+ Table 2: Details of Login Nodes
 
|+ Table 2: Details of Login Nodes
 
!
 
!
−
! No Accelerator
+
! NVIDIA A100 GPU
−
! NVIDIA A100 Accelerator
+
! NVIDIA RTX 6000 GPU
−
! NVIDIA RTX6000 Accelerator
+
! NVIDIA T4 GPU
−
! NVIDIA T4 Accelerator
+
! No GPU
 
|-
 
|-
−
| HostNames
+
| Hostnames
−
| grace1.hprc.tamu.edu<br>grace2.hprc.tamu.edu
+
| grace1.hprc.tamu.edu
 +
| grace2.hprc.tamu.edu
 
| grace3.hprc.tamu.edu
 
| grace3.hprc.tamu.edu
−
| grace4.hprc.tamu.edu
+
| grace4.hprc.tamu.edu<br>grace5.hprc.tamu.edu
−
| grace5.hprc.tamu.edu
 
 
|-
 
|-
 
| Processor Type
 
| Processor Type
Line 131: Line 157:
 
|-
 
|-
 
| Memory
 
| Memory
−
| colspan=4 | 384 GB DDR4 3200 MHz
+
| colspan=4 | 384GB DDR4 3200 MHz
 
|-
 
|-
 
| Total Nodes
 
| Total Nodes
 +
| 1
 +
| 1
 +
| 1
 
| 2
 
| 2
−
| colspan=3 | 1
 
 
|-
 
|-
 
| Cores/Node
 
| Cores/Node
Line 141: Line 169:
 
|-
 
|-
 
| Interconnect
 
| Interconnect
−
| colspan=4 | Mellanox HDR Infiniband
+
| colspan=4 | Mellanox HDR100 InfiniBand
 
|-
 
|-
 
| Local Disk Space
 
| Local Disk Space
−
| colspan=4 | per node: two 480 GB SSD drives
+
| colspan=4 | per node: two 480 GB SSD drives, 1.6 TB NVMe
 
|}
 
|}
  
 
== Mass Storage ==
 
== Mass Storage ==
  
−
5PB (usable) with Lustre provided by DDN
+
* 5PB (usable) with Lustre provided by DDN ES200NV appliance and two ES7990X appliances
−
 
+
* 1.4PB (usable) with GPFS provided by Lenovo's DSS-G220 appliance
 +
* 1.9PB (usable) with GPFS provided by Lenovo's DSS-G230 appliance
  
 
== Interconnect ==
 
== Interconnect ==
  
 +
Two level fat tree topology with Mellanox HDR100:
  
 +
* There are 5 core switches and 11 leaf switches.
 +
* Each leaf switch has 2 Mellanox HDR InfiniBand (200Gb/s) uplinks to each core switch.
 +
* There are up to 80 compute nodes attached to each leaf switch.
 +
* Each login or compute node has a single Mellanox HDR100 InfiniBand (100Gb/s) link to a leaf switch.
 +
* The DDN storage has 12 total HDR100 links.
 +
* The Lenovo DSS-G220 storage (Dr. Junjie Zhang's CryoEM Lab) has 8 HDR100 links.
 +
* The Lenovo DSS-G230 storage (Dr. Ping Chang's iHESP Lab) has 8 EDR links (100Gb/s).
  
 
== Namesake ==
 
== Namesake ==

Latest revision as of 10:22, 24 March 2023

Grace: A Dell x86 HPC Cluster

Hardware Overview

caption
System Name: Grace
Host Name: grace.hprc.tamu.edu
Operating System: Linux (CentOS 7)
Total Compute Cores/Nodes: 45,376 cores
940 nodes
Compute Nodes: 800 48-core compute nodes, each with 384GB RAM
100 48-core GPU nodes, each with two A100 40GB GPUs and 384GB RAM
9 48-core GPU nodes, each with two RTX 6000 24GB GPUs and 384GB RAM
8 48-core GPU nodes, each with 4 T4 16GB GPUs and 384GB RAM
15 48-core GPU nodes, each with two A40 48GB GPUs and 384GB RAM
8 80-core large memory nodes, each with 3TB RAM
Interconnect: Mellanox HDR 100 InfiniBand
Peak Performance: 6.3 PFLOPS
Global Disk: 5PB (usable) via DDN Lustre appliances for general use
1.4PB (usable) via Lenovo DSS GPFS appliance (purchased by and dedicated for Dr. Junjie Zhang's CryoEM Lab)
1.9PB (usable) via Lenovo DSS GPFS appliance (purchased by and dedicated for Dr. Ping Chang's iHESP Lab)
File System: Lustre and GPFS
Batch Facility: Slurm by SchedMD
Location: West Campus Data Center
Production Date: Spring 2021

Grace is an Intel x86-64 Linux cluster with 940 compute nodes (45,376 total cores) and 5 login nodes. There are 800 compute nodes with 384 GB of memory, and 117 GPU nodes with 384 GB of memory. Among the 132 GPU nodes, there are 100 GPU nodes two A100 40 GB GPU cards, 9 GPU nodes with two RTX 6000 24GB GPU cards, 8 GPU nodes with four T4 16GB GPU cards, and 15 GPU nodes with two A40 48GB GPUs. These 800 compute nodes and 117 GPU nodes are a dual socket server with two Intel 6248R 3.0GHz 24-core processors, commonly known as Cascade Lake. There are 8 compute nodes with 3 TB of memory and four Intel 6248 2.5 GHz 20-core processors.

The interconnecting fabric is a two-level fat-tree based on HDR 100 InfiniBand.

High performance mass storage of 5 petabyte (usable) capacity is made available to all nodes by the DDN Lustre storage. Also, 3.3 PB of Lenovo DSS GPFS storage is for their respective research labs.

Get details on using this system, see the User Guide for Grace.

Compute Nodes

A description of the four types of compute nodes is below:

Table 1 Details of Compute Nodes
General 384GB
Compute 		GPU A100
Compute 		GPU RTX 6000
Compute 		GPU T4
Compute 		GPU A40
Compute 		Large Memory 3TB
Compute
Total Nodes 800 100 9 8 15 8
Processor Type Intel Xeon 6248R (Cascade Lake), 3.0GHz, 24-core Intel Xeon 6248 (Cascade Lake), 2.5 GHz, 20-core
Sockets/Node 2 4
Cores/Node 48 80
Memory/Node 384GB DDR4, 3200 MHz 3TB DDR4, 3200 MHz
Accelerator(s) N/A 2 NVIDIA A100 40GB GPU 2 NVIDIA RTX 6000 24GB GPU 4 NVIDIA T4 16GB GPU 2 NVIDIA A40 48GB GPU N/A
Interconnect Mellanox HDR100 InfiniBand
Local Disk Space 1.6TB NVMe (/tmp), 480GB SSD

Usable Memory for Batch Jobs

While nodes on Grace have either 384GB or 3TB of RAM, some of this memory is used to maintain the software and operating system of the node. In most cases, excessive memory requests will be automatically rejected by SLURM.

The table below contains information regarding the approximate limits of Grace memory hardware and our suggestions on its use.

Memory Limits of Nodes
384GB Nodes (Regular and GPU) 3TB Nodes
Node Count 932 8
Number of Cores 48 Cores 80 Cores
Memory Limit
Per Core 		7500 MB
~7.5 GB 		37120 MB
36.25 GB
Memory Limit
Per Node 		368640 MB
360 GB 		2969600 MB
2900 GB

Login Nodes

The grace.hprc.tamu.edu hostname can be used to access the Grace cluster. This translates into one of the five login nodes, grace[1-5].hprc.tamu.edu. To access a specific login node use its corresponding host name (e.g., grace2.hprc.tamu.edu). All login nodes have 10 GbE connections to the TAMU campus network and direct access to all global parallel (Lustre-based) file systems. The table below provides more details about the hardware configuration of the login nodes.

Table 2: Details of Login Nodes
NVIDIA A100 GPU NVIDIA RTX 6000 GPU NVIDIA T4 GPU No GPU
Hostnames grace1.hprc.tamu.edu grace2.hprc.tamu.edu grace3.hprc.tamu.edu grace4.hprc.tamu.edu
grace5.hprc.tamu.edu
Processor Type Intel Xeon 6248R 3.0GHz 24-core
Memory 384GB DDR4 3200 MHz
Total Nodes 1 1 1 2
Cores/Node 48
Interconnect Mellanox HDR100 InfiniBand
Local Disk Space per node: two 480 GB SSD drives, 1.6 TB NVMe

Mass Storage

  • 5PB (usable) with Lustre provided by DDN ES200NV appliance and two ES7990X appliances
  • 1.4PB (usable) with GPFS provided by Lenovo's DSS-G220 appliance
  • 1.9PB (usable) with GPFS provided by Lenovo's DSS-G230 appliance

Interconnect

Two level fat tree topology with Mellanox HDR100:

  • There are 5 core switches and 11 leaf switches.
  • Each leaf switch has 2 Mellanox HDR InfiniBand (200Gb/s) uplinks to each core switch.
  • There are up to 80 compute nodes attached to each leaf switch.
  • Each login or compute node has a single Mellanox HDR100 InfiniBand (100Gb/s) link to a leaf switch.
  • The DDN storage has 12 total HDR100 links.
  • The Lenovo DSS-G220 storage (Dr. Junjie Zhang's CryoEM Lab) has 8 HDR100 links.
  • The Lenovo DSS-G230 storage (Dr. Ping Chang's iHESP Lab) has 8 EDR links (100Gb/s).

Namesake

"Grace" is named for Grace Hopper.

Retrieved from "https://hprc.tamu.edu/w/index.php?title=Grace:Intro&oldid=13418"